Securing AI Coding Agents with 0.983 F1 Threat Detection in Benchmarks

A security layer that scans AI coding agent activity for threats in real time

0.983 F1 score
In benchmarking, a combined accuracy measure (precision 1.00, recall 0.966)
Zero false positives
And 100% recall on malicious prompts in benchmarking
13 threat categories
Scanned on every agent interaction

About the customer

Our client is a US generative AI company that helps enterprises adopt AI coding agents safely. Tools such as Claude Code, Cursor, Codex and KIRO now read source code, run commands and write files on their own. The client provides the security controls, policy enforcement, compliance checks and codebase context that let enterprises trust these agents with their corporate code.

Company
Confidential Client
Country
USA
Industry
B2B
Customer size
1000+

What needed solving

Developer-level access

AI coding agents act with the same access as the developer who runs them

New attack surface

This exposes enterprises to prompt injection, jailbreaks, and leakage of secrets and source code to third-party LLMs

Risky commands and code

Destructive shell commands and insecure or malicious generated code can reach production unnoticed

No company context

Agents lack knowledge of the company’s own architecture and conventions, and protection had to work in real time without slowing developers down with false alarms

How we built it

01

Security layer

An agentic, LLM-driven layer inside the client’s AI gateway scans every prompt, model response, tool call and tool result across 13 threat categories.

02

Policy enforcement

Block, warn or allow policies, set per customer.

03

Secure code audit

Generated code is audited against OWASP Top 10:2025 and OWASP ASVS v5.0, widely used application security standards.

04

Codebase RAG

Company repositories are indexed into a vector store, which serves repository profiles and coding guidelines to agents.

05

PR scanning and Skills audit

Pull request scanning with verified auto-fix, plus an audit of agent Skills for malicious content and leaked secrets.

06

Benchmarking and deployment

Benchmarked on 387 cases mirroring real Claude Code, Cursor, Codex and KIRO traffic. Supports seven LLM providers, including on-premises and air-gapped (fully offline) deployments.

Result

Enterprises gained the confidence to roll out AI coding agents

Full policy control and audit trails
In benchmarking: 1.00 precision, 0.966 recall and 0.983 F1
Zero false positives and 100% recall on malicious prompts
LLM costs kept low through prompt caching and deterministic codebase context

Technologies

OpenAI GPTAnthropic ClaudeAWS BedrockGoogle GeminivLLMLangGraphFastAPIMongoDBQdrantTree-sitterOpenTelemetryLangSmithLangfuseDockerKubernetesGitHub

Domains

Generative AIAI Coding AgentsAgentic AI SecurityAI SDLC GovernanceLLM SecurityRetrieval Augmented Generation (RAG)Application SecurityCompliance Automation (OWASP)

Explore similar work

Face profile built from scattered text, representing verified LLM answers

LLM Response Verification for Accurate, Compliant Enterprise AI

Wall of archive boxes, representing enterprise knowledge search with RAG

Trusted, Cited AI Answers from Corporate Data, with Permissions Enforced

Have a Query? Let's Have a Conversation

Do you have a project in mind? We’d love to know what it is. Email us or send a message from our contact page.

Location..
Visit Us At
Islamabad, Pakistan.
Drop a line..
Mail Address
info@vannguardai.com

Have a similar problem?

Tell us what you are trying to solve and our team will get back to you.

Talk to our team