AI Security Assessment: Public Sector Education Chatbot

The Engagement

A large Australian state government education department engaged Insicon Cyber to assess a newly released version of an AI chatbot deployed across schools. With a broader rollout and potential licensing on the horizon, the department needed independent assurance before proceeding.

The Customer Problem

The department faced a core uncertainty: their existing security testing did not tell them whether the AI system was safe to scale. Traditional penetration testing and red teaming do not adequately address AI-specific risks such as prompt injection, jailbreaks, unsafe outputs, and data leakage. Those risks carried heightened stakes here, because the system operated in an environment involving minors.

What Insicon Cyber Delivered

Insicon Cyber ran a comprehensive assessment spanning:

• Application security and secure code scanning
• Identity and access controls
• Cloud posture review
• API integration security
• Large-scale adversarial AI testing aligned to MITRE ATLAS and OWASP LLM guidance

In partnership with F5, Insicon Cyber also evaluated runtime AI guardrails, comparing system behaviour with protections enabled and disabled to measure their real-world effectiveness.

The Findings

The assessment uncovered serious vulnerabilities. It demonstrated that existing guardrails could be bypassed using techniques that traditional testing would not detect. The AI system carried risk the department could not have identified through conventional methods.

The Outcome

Insicon Cyber delivered a clear remediation roadmap, executive-level reporting, and evidence-based assurance the department could act on and take to its board before broader rollout and licensing.

Key Takeaways

AI-Specific Security Challenges

• Traditional testing fails to identify AI risks like prompt injection, jailbreaks, unsafe outputs, and data leakage.
• These risks carry heightened stakes in sensitive environments, such as those involving minors or protected data.

Comprehensive Security Assessment

• The assessment covered application security, identity controls, cloud posture, API integrations, and adversarial AI testing.
• Adversarial AI testing was aligned to recognised frameworks: MITRE ATLAS and OWASP LLM guidance.
• Secure code scanning was included alongside the runtime and infrastructure review.

Runtime AI Guardrail Evaluation

• Evaluated AI guardrails under enabled and disabled states to reveal bypass techniques undetectable by traditional tests.
• Delivered in partnership with F5, measuring the real-world effectiveness of runtime protections.

Outcomes and Business Impact

• Provided remediation roadmap, executive reporting, and assurance to guide secure AI deployment and sales positioning.
• Gave the customer board-ready evidence of due diligence before broader rollout and licensing.

Learn More

• AI Security & Governance practice: https://insiconcyber.com/ai-security-governance
• AI Assurance (Test it. Protect it.): https://insiconcyber.com/ai-assurance
• Board Cyber Advisory: https://insiconcyber.com/board-cyber-advisory
• Contact Insicon Cyber: https://insiconcyber.com/contact