Tokens, Latency and Trust: The AI Metrics Leaders in Australia and New Zealand Need to Understand

AI SECURITY AND GOVERNANCE

You don't need to be an AI engineer to run a good vendor review. But when a vendor pitch is built entirely on numbers your team can't interrogate, that's an operational risk, not a technical detail.

Why AI performance metrics are now a leadership issue

Every AI vendor pitch now comes with a slide of numbers. Time to first token. Tokens per second. Cache hit rate. Cost per million tokens. For most leaders and executives across Australia and New Zealand managing IT, security or operations, these numbers wash past as technical noise, somewhere between the demo and the pricing page.

That's a problem regulators have now named directly. The Australian Prudential Regulation Authority's letter to industry on artificial intelligence, published 30 April 2026, observed an overreliance on vendor presentations and summaries without sufficient examination of key AI risks such as unpredictable model behaviour and the impact on critical operations. That observation lands squarely on the desks of the people actually running AI deployments day to day: IT leaders, security teams and operational managers who sign off on vendor contracts and carry the consequences when performance or security assumptions don't hold up.

ASIC's companion letter of 8 May 2026 (26-092MR) reinforces the same point, requiring AI and cyber risk to be discussed at the highest governance level of every AFS licensee and market participant, which means the operational detail behind that discussion has to come from somewhere. It comes from the leaders and executives closest to the technology.

This blog is a plain English glossary of the metrics that actually matter in AI performance and cost, and why each one carries a practical operational and security implication for your team.

The metrics that describe how fast an AI system responds

Behind every AI chat interface, copilot or customer-facing assistant sits an inference process: the model reading a prompt and generating a response, one token at a time. A token is roughly three-quarters of a word (or is that 'w-o-r' 😆). How fast that process runs, and how consistently, is measured by a small set of standard metrics.

Different use cases put weight on different metrics. A customer-facing chat assistant lives or dies on TTFT: research suggests a response has to start appearing in under roughly half a second to feel responsive. A batch process summarising thousands of aged care incident reports overnight cares far more about total throughput than about first-token speed. Knowing which metric matters for which workload is the difference between a sensible SLA and one that sounds impressive but measures the wrong thing.

Inference: the engine behind every AI answer

"Inference" is simply the term for a trained AI model being run against new input to produce an output, as distinct from "training," which is the earlier, far more expensive process of building the model in the first place. Every time a staff member sends a prompt or an AI agent processes a case, that's an inference request. It's the ongoing, per-use cost and performance layer your team should be actively monitoring, because unlike training costs, which the vendor absorbs, inference costs scale directly with how heavily your organisation uses the tool.

Prefix caching: the cost lever vendors don't always explain

When an AI model processes a prompt, it builds an internal working memory of that prompt called a KV cache. If the next request shares the same starting text, such as a recurring system instruction, a policy document, or a long case file being asked several questions in a row, the model can reuse that cached memory instead of reprocessing it from scratch. This is prefix caching, also called prompt caching.

The economics are significant. Major providers charge substantially less, in some cases around a 90 per cent discount, for cached tokens compared with fresh ones, and cached responses generate noticeably faster. For a team running AI across repeated workflows, such as a fractional CISO reviewing the same client risk register, or an aSOC analyst re-querying the same incident context, prefix caching is the difference between a sustainable AI cost base and a runaway one.

Your team doesn't need to manage cache configuration directly, but procurement and platform owners should be asking whether the vendor's pricing model and architecture actually take advantage of it, because the gap between a well cached and poorly cached deployment can be an order of magnitude in ongoing cost.

The security risk hiding inside prefix caching

This is where a pure performance metric becomes a genuine security question, and it's the reason this topic sits inside Insicon Cyber's AI Security and Governance practice rather than purely in the IT budget conversation.

Because cached prompts respond measurably faster than uncached ones, the response time itself leaks information. Researchers have demonstrated that in multi-tenant AI deployments, where a cache is shared across different users or customers, an attacker can send crafted prompts and measure the time to first token to work out whether a prefix has been seen before, in effect fingerprinting or partially reconstructing another user's private input purely from timing. This is a timing side-channel attack, and it has been documented against several commercial AI API providers.

For teams in financial services, aged care and healthcare, where AI is increasingly touching client records, medical notes and case files, this is precisely the kind of AI-specific attack pathway APRA's letter calls out directly, alongside prompt injection, data leakage and insecure integrations. It's also a clean, concrete example of why identity and access management and security testing need to be extended specifically to cover AI implementations, not just traditional application layers.

Token costs and the hidden line item on your AI bill

AI usage is billed by the token, both for what goes in (the prompt, including any documents or context) and what comes out (the response). Input and output tokens are often priced differently, cached and uncached tokens are priced differently again, and a single conversation with a large document attached can consume orders of magnitude more tokens than a short question.

This matters in a very practical way: a pilot that looks cheap with five staff and short prompts can become a materially different cost line once it's rolled out across a business unit processing full case files or long compliance documents. When you're approving AI budgets or reviewing a vendor contract, ask for a cost-per-transaction model, not just a per-seat licence fee, because per-seat pricing frequently hides the token economics underneath it.

What APRA and ASIC expect in practice

APRA's letter is explicit that entities should map and maintain visibility over their full AI supply chain, including material third and fourth-party dependencies, with contractual arrangements that address audit rights, model updates and incident notification. It also notes that few entities had tested exit or substitution strategies for critical AI providers, and that some were heavily dependent on a single provider across multiple use cases.

Performance and cost metrics sit directly inside that supplier risk work. A vendor that can't explain its latency guarantees, its caching architecture, or its cost-per-transaction model under real production load is a vendor your organisation cannot meaningfully audit, and a concentration risk you cannot substitute out of quickly if something goes wrong. This is operational work, not a once-a-year governance exercise.

Questions for your next AI vendor review

• What is our guaranteed time to first token and end to end latency under expected peak load, not a controlled demo?
• Is our prompt or context data cached, and if so, is that cache isolated to our organisation or shared across other tenants?
• What is our real cost per transaction once input tokens, output tokens and document context are all accounted for, at expected production volume?
• What security testing has been performed specifically against AI-specific attack pathways, including timing-based and prompt injection risks?
• If this provider became unavailable or was compromised, what is our tested fallback or substitution plan?

How Insicon Cyber helps

Insicon Cyber's AI Security and Governance practice, delivered through our team of Australia-based staff, gives IT, security and operational leaders across Australia and New Zealand the practical assurance APRA and ASIC now expect. This includes reviewing AI vendor architecture and contractual protections, testing for AI-specific attack pathways, and turning technical performance metrics into the vendor questions your team needs answered before the next AI deployment goes live.

Secure AI. Governed AI. Compliant AI.

Sources

• APRA, Letter to Industry on Artificial Intelligence, 30 April 2026: https://www.apra.gov.au/apra-letter-to-industry-on-artificial-intelligence-ai
• ASIC, Open Letter to AFS Licensees and Market Participants, 26-092MR, 8 May 2026: https://download.asic.gov.au/media/xhrf1w0e/26-092mr-open-letter-to-afs-licensees-and-market-participants.pdf
• ASD, Frontier models and their impact on cyber security: https://www.cyber.gov.au/about-us/view-all-content/news/frontier-models-and-their-impact-on-cyber-security
• IBM, Time to First Token: https://www.ibm.com/think/topics/time-to-first-token
• Databricks, LLM Inference Performance Engineering, Best Practices: https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices
• NVIDIA, Metrics, NIM LLMs Benchmarking: https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
• BentoML, LLM Inference Handbook, Prefix Caching: https://bentoml.com/llm/inference-optimization/prefix-caching
• DigitalOcean, How KV Caching Slashes LLM Inference Costs at Scale: https://www.digitalocean.com/community/conceptual-articles/how-kv-caching-slashes-llm-inference-costs-at-scale
• Auditing Prompt Caching in LLM APIs (Stanford, arXiv): https://arxiv.org/pdf/2502.07776
• PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systems (arXiv): https://arxiv.org/html/2603.10726v2
• InputSnatch: Stealing Input in LLM Services via Timing Side-Channel Attacks (arXiv): https://arxiv.org/abs/2411.18191
• Akamai, State of the Internet, Financial Services Security Trends 2026: https://www.akamai.com/security