AASTF

Agentic AI Security Testing Framework

Every AI agent a business deploys is a new employee with system access and no background check. AASTF is my open-source, MIT-licensed answer: it security-tests whole agent systems — model, tools, memory, and planning loops — with 130+ attack scenarios mapped to the OWASP Top 10 for Agentic Applications, turning agent risk into a repeatable, reportable test result.

Challenge

Most AI security testing stops at the prompt: send adversarial inputs to a model and inspect its replies. Agent systems have a far larger attack surface — tools that execute actions, memory that persists across turns, and planning loops that chain decisions together. A harness that only sees the model misses how those pieces can be abused in combination.

Approach

AASTF tests the whole agent system: model plus tools, memory, and planning. It ships 130+ attack scenarios mapped to the OWASP Top 10 for Agentic Applications, executes them against sandboxed tool backends so no test causes real side effects, and classifies every result with a three-class verdict — VULNERABLE, REFUSAL_ECHO, or SAFE. A LangGraph adapter is supported out of the box, with experimental adapters for CrewAI, OpenAI Agents, and PydanticAI, plus 25 dedicated MCP security scenarios.

Outcome

The framework is open source under the MIT license. Results export as SARIF for CI/CD pipelines, and an EU AI Act readiness score summarizes where an agent system stands — making agent security testing repeatable instead of a one-off audit.

Key Features

• 130+ built-in attack scenarios
• Mapped to the OWASP Top 10 for Agentic Applications
• Three-class verdicts: VULNERABLE, REFUSAL_ECHO, and SAFE
• Sandboxed tool backends — tests run with no real side effects
• Framework adapters: LangGraph, plus experimental CrewAI, OpenAI Agents, and PydanticAI
• 25 MCP security scenarios
• SARIF output for CI/CD
• EU AI Act readiness scoring

Stack & Methods

• Python
• OWASP Top 10 for Agentic Applications mapping
• LangGraph adapter; experimental CrewAI, OpenAI Agents, and PydanticAI adapters
• Sandboxed tool backends
• SARIF output for CI/CD
• EU AI Act readiness scoring
• MIT license