AI SECURITY AND GOVERNANCE
You don't need to be an AI engineer to run a good vendor review. But when a vendor pitch is built entirely on numbers your team can't interrogate, that's an operational risk, not a technical detail.
Every AI vendor pitch now comes with a slide of numbers. Time to first token. Tokens per second. Cache hit rate. Cost per million tokens. For most leaders and executives across Australia and New Zealand managing IT, security or operations, these numbers wash past as technical noise, somewhere between the demo and the pricing page.
That's a problem regulators have now named directly. The Australian Prudential Regulation Authority's letter to industry on artificial intelligence, published 30 April 2026, observed an overreliance on vendor presentations and summaries without sufficient examination of key AI risks such as unpredictable model behaviour and the impact on critical operations. That observation lands squarely on the desks of the people actually running AI deployments day to day: IT leaders, security teams and operational managers who sign off on vendor contracts and carry the consequences when performance or security assumptions don't hold up.
ASIC's companion letter of 8 May 2026 (26-092MR) reinforces the same point, requiring AI and cyber risk to be discussed at the highest governance level of every AFS licensee and market participant, which means the operational detail behind that discussion has to come from somewhere. It comes from the leaders and executives closest to the technology.
This blog is a plain English glossary of the metrics that actually matter in AI performance and cost, and why each one carries a practical operational and security implication for your team.
Behind every AI chat interface, copilot or customer-facing assistant sits an inference process: the model reading a prompt and generating a response, one token at a time. A token is roughly three-quarters of a word. How fast that process runs, and how consistently, is measured by a small set of standard metrics.
TTFT: Time to First TokenThe delay between submitting a request and the first word appearing. This is what makes a chatbot feel instant or sluggish. It's driven by prompt length, system load and how much of the prompt is already cached. |
TPOT: Time Per Output TokenThe average time to generate each subsequent word after the first. Sometimes called inter-token latency or ITL. This is what determines whether a response streams smoothly or stutters, and it's the number that governs how long a long report or code file takes to finish generating. |
End to end latency (E2E)The full time from request to the last token of a complete response. Some vendor documentation calls this Time to Last Token (TTLT). This is the number that matters for anything where a partial answer is useless: code generation, form completion, an automated compliance check that has to finish before the next system step can run. |
TPS: Tokens Per Second (throughput)How many tokens the whole system can generate per second across every user at once. This is the capacity number: it determines whether a platform holds up under real transaction volume rather than a controlled demo with one user in the room. |
Different use cases put weight on different metrics. A customer-facing chat assistant lives or dies on TTFT: research suggests a response has to start appearing in under roughly half a second to feel responsive. A batch process summarising thousands of aged care incident reports overnight cares far more about total throughput than about first-token speed. Knowing which metric matters for which workload is the difference between a sensible SLA and one that sounds impressive but measures the wrong thing.
Where do you actually stand?
APRA wrote to industry on AI governance on 30 April 2026. ASIC issued 26-092MR on 8 May 2026. The ASD Essential Eight remains the baseline requirement for government suppliers and insurers. An independent readiness assessment gives you a documented position against the obligation that actually binds you. Two to four weeks. Founder reviewed. Every time.
"Inference" is simply the term for a trained AI model being run against new input to produce an output, as distinct from "training," which is the earlier, far more expensive process of building the model in the first place. Every time a staff member sends a prompt or an AI agent processes a case, that's an inference request. It's the ongoing, per-use cost and performance layer your team should be actively monitoring, because unlike training costs, which the vendor absorbs, inference costs scale directly with how heavily your organisation uses the tool.
When an AI model processes a prompt, it builds an internal working memory of that prompt called a KV cache. If the next request shares the same starting text, such as a recurring system instruction, a policy document, or a long case file being asked several questions in a row, the model can reuse that cached memory instead of reprocessing it from scratch. This is prefix caching, also called prompt caching.
The economics are significant. Major providers charge substantially less, in some cases around a 90 per cent discount, for cached tokens compared with fresh ones, and cached responses generate noticeably faster. For a team running AI across repeated workflows, prefix caching is the difference between a sustainable AI cost base and a runaway one.
This is where a pure performance metric becomes a genuine security question. Because cached prompts respond measurably faster than uncached ones, the response time itself leaks information. Researchers have demonstrated that in multi-tenant AI deployments, an attacker can send crafted prompts and measure the time to first token to work out whether a prefix has been seen before, in effect fingerprinting or partially reconstructing another user's private input purely from timing. This is a timing side-channel attack, and it has been documented against several commercial AI API providers.
For teams in financial services, aged care and healthcare, where AI is increasingly touching client records, medical notes and case files, this is precisely the kind of AI-specific attack pathway APRA's letter calls out directly.
AI usage is billed by the token, both for what goes in and what comes out. Input and output tokens are often priced differently, cached and uncached tokens are priced differently again, and a single conversation with a large document attached can consume orders of magnitude more tokens than a short question.
APRA's letter is explicit that entities should map and maintain visibility over their full AI supply chain, including material third and fourth-party dependencies, with contractual arrangements that address audit rights, model updates and incident notification. Performance and cost metrics sit directly inside that supplier risk work.
Insicon Cyber's AI Security and Governance practice, delivered through our team of Australia-based staff, gives IT, security and operational leaders across Australia and New Zealand the practical assurance APRA and ASIC now expect. This includes reviewing AI vendor architecture and contractual protections, testing for AI-specific attack pathways, and turning technical performance metrics into the vendor questions your team needs answered before the next AI deployment goes live.
Secure AI. Governed AI. Compliant AI.