Sources: IBM Cost of a Data Breach Report 2025 (adapted for AI-specific findings); NIST Secure Software Development Framework cost analysis 2025.

AI-Specific Security Incident Data

• OWASP's 2025 Top 10 for LLM Applications survey found that prompt injection remains the most commonly identified critical vulnerability across audited enterprise LLM applications, present in the majority of applications that had not undergone AI-specific threat modeling before deployment

• Organizations that conduct AI-specific threat modeling before development report 60–70% fewer critical security findings during pre-production testing compared to organizations conducting security review only after development is complete (OWASP, 2025)

• Excessive agency AI agents granted broader tool and data permissions than their function requires was identified as a contributing factor in a majority of documented AI agent security incidents reviewed in 2025, reinforcing that permission scoping at design time is a primary security control, not an afterthought (OWASP AI Security Project, 2025)

Early Integration Impact

• Early security integration reduces vulnerabilities in production systems significantly organizations applying secure-by-design practices throughout the AI development lifecycle report substantially fewer critical findings in production penetration testing compared to organizations applying security review only at the pre-launch stage (NIST AI RMF implementation data, 2025)

• Teams that scope AI agent permissions using least-privilege principles from initial design report a measurably smaller blast radius in the rare cases where an agent is successfully compromised or manipulated, compared to teams that retrofit permission restrictions after a broader-access agent is already in production

How to Implement Secure-by-Design AI: A 6-Step Framework

Step 1: Conduct AI-Specific Threat Modeling Before Model or Architecture Selection

Before selecting a foundation model, an orchestration framework, or a deployment architecture, run threat modeling specifically against the OWASP Top 10 for LLM Applications categories:

1. Map your planned AI system's data flows what untrusted input does it process, what trusted instructions does it follow, what tools or systems can it access

2. Identify which OWASP LLM risk categories apply to your specific use case a customer-facing chatbot has different prompt injection exposure than an internal document summarization tool

3. Document the specific architectural decisions that mitigate each identified risk before development begins, not as a remediation list to address later

Step 2: Design Clear Separation Between System Instructions and Untrusted Input

Architect your prompt construction so that system instructions (trusted, defining the AI's intended behavior) and user or external input (untrusted, potentially containing injection attempts) are structurally distinguishable to the model, rather than concatenated as undifferentiated text:

1. Use structured prompting formats (system/user/assistant role separation, supported natively by most modern LLM APIs) rather than building prompts as single concatenated strings

2. Treat any content retrieved from external sources documents, web pages, emails, database records as untrusted input requiring the same scrutiny as direct user input, since retrieval-augmented generation introduces injection surface area that's frequently overlooked

3. Implement input validation and sanitization specifically targeting known prompt injection patterns, while recognizing this is a defense-in-depth layer, not a complete solution on its own

Step 3: Scope Every AI Agent Permission to Least Privilege From Initial Design

For any AI system with tool-use or agentic capability, define permission scope before granting any access:

1. List every tool, API, and data source the AI system genuinely needs for its specific function not what might be convenient to have available

2. Implement permission boundaries at the orchestration layer, not as a configuration option the AI system itself could potentially be manipulated into bypassing

3. Require explicit approval workflows for any permission expansion as the system's capability grows, treating scope expansion with the same rigor as the initial design review

Step 4: Implement Secure Output Handling Before Any Downstream System Consumes AI Outputs

AI-generated output that feeds into downstream systems code that gets executed, SQL queries that get run, HTML that gets rendered requires the same validation and sanitization discipline as any other untrusted input to those downstream systems:

1. Never directly execute AI-generated code, queries, or commands without validation appropriate to that specific execution context

2. Sanitize AI-generated output that will be rendered in a web context to prevent injection attacks where the AI's output itself becomes an attack vector

3. Implement output filtering for sensitive information that may have leaked into a response from training data or retrieved context, particularly relevant for systems with access to regulated or confidential data sources

Step 5: Conduct Adversarial Testing Before Production Deployment

Test the system against deliberate adversarial inputs before launch, and on a recurring basis afterward:

1. Run prompt injection test suites covering known attack patterns from the OWASP LLM Top 10 and current threat intelligence on emerging techniques

2. Conduct jailbreak testing specifically against any safety or behavioral constraints the system is meant to enforce

3. Test data extraction attempts can an attacker craft inputs that cause the system to reveal system instructions, training data fragments, or other users' context

Step 6: Implement Continuous Production Monitoring Specific to AI Security Events

Deploy monitoring that captures AI-specific anomaly signals distinct from traditional application monitoring:

1. Monitor for unusual tool-call patterns from AI agents calls outside normal scope, unusual frequency, or sequences inconsistent with the agent's defined function

2. Monitor output patterns for signs of successful prompt injection or jailbreak unexpected format shifts, content inconsistent with the system's intended behavior

3. Establish a defined incident response procedure specifically for AI security events, since the investigation and remediation steps for a compromised AI agent differ meaningfully from a traditional application security incident

Which Tools and Frameworks Deliver Best Results for Secure-by-Design AI in 2026?

For AI-specific threat modeling and risk frameworks:
OWASP Top 10 for LLM Applications provides the standard, regularly updated risk taxonomy for systematic AI threat modeling the foundational reference every secure-by-design AI program should assess against. NIST AI Risk Management Framework (AI RMF) provides a broader governance structure for documenting AI security considerations throughout the development lifecycle, increasingly referenced in regulatory compliance contexts.

For prompt injection testing and adversarial evaluation:
Garak (open-source) provides automated LLM vulnerability scanning specifically designed to test for prompt injection, jailbreak susceptibility, and other OWASP LLM Top 10 risk categories. Promptfoo provides prompt testing and evaluation infrastructure that integrates adversarial test cases into CI/CD pipelines, enabling automated regression testing against known attack patterns as the system evolves.

For AI agent permission scoping and orchestration security:
LangGraph and Microsoft AutoGen, covered in our agentic AI architecture guide, both support permission scoping at the orchestration layer when configured deliberately the framework provides the mechanism, but least-privilege scoping requires explicit design decisions rather than default configuration.

For output validation and sanitization:
Standard web application security tools (OWASP-recommended sanitization libraries for the relevant output context HTML, SQL, shell commands) apply directly to AI-generated output requiring the same validation discipline as any other untrusted content reaching those execution contexts.

For AI security monitoring:
Lakera Guard and Robust Intelligence (now part of Cisco) provide specialized AI runtime security monitoring designed to detect prompt injection attempts and anomalous model behavior in production distinct from general application performance monitoring, which doesn't natively understand AI-specific attack signatures.

For model supply chain security:
Hugging Face's model scanning and signed model provenance features help verify the integrity of pre-trained models and dependencies before integration, addressing the model supply chain risk category in the OWASP taxonomy.

Explore our DevSecOps Services and AI Governance capabilities for organizations building secure-by-design AI development practices integrated into existing engineering workflows.

What Goes Wrong With AI Security Programs and How Secure-by-Design Prevents Each Failure

Failure 1: Treating AI Security as a Pre-Launch Penetration Test

Organizations that apply security review only as a final gate before launch consistently discover prompt injection and permission scope issues that require architectural rework, not configuration changes at a point in the development cycle where that rework is most expensive and most likely to be skipped under launch deadline pressure. Secure-by-design AI moves the same questions earlier, when the answer is a design decision rather than a delayed launch.

Failure 2: Granting AI Agents Broad Permissions for Development Convenience

Development teams that grant AI agents broad tool and data access during development because it's faster than configuring granular permissions while building and testing frequently ship that broad access configuration to production unchanged, because narrowing permissions after the system works as-is feels like unnecessary risk to a working deployment. Excessive agency was identified as a contributing factor in the majority of documented AI agent security incidents specifically because this convenience-driven permission creep is so common. Define least-privilege scope before development begins, not as a hardening pass before launch.

Failure 3: Concatenating Untrusted Input Directly Into System Prompts

Teams building AI applications quickly, particularly those new to LLM-specific architecture patterns, frequently construct prompts by directly concatenating user input or retrieved document content into the same string as system instructions creating exactly the undifferentiated trust boundary that makes prompt injection straightforward to execute. This is rarely a deliberate decision; it's frequently the simplest implementation path that secure-by-design threat modeling, applied before development, would have flagged as a structural risk requiring a different prompt architecture from the start.

Failure 4: Monitoring AI Systems With Traditional APM Tools Alone

Organizations that deploy AI applications with only traditional application performance monitoring uptime, latency, error rates lack visibility into AI-specific security signals: unusual tool-call patterns, output anomalies indicating successful manipulation, or behavioral drift from intended system behavior. A successful prompt injection attack may produce an AI response that looks functionally normal to traditional APM (the application didn't crash, response time was normal) while representing a genuine security incident invisible to monitoring that doesn't understand AI-specific failure modes.

Frequently Asked Questions

What Is Secure-by-Design?

Secure-by-design is the practice of integrating security requirements into a system's architecture from the earliest design decisions, rather than applying security review as a separate, later validation step against an already-built system. Applied to AI specifically, secure-by-design means threat modeling AI-specific risks prompt injection, data leakage, excessive agent permissions, model supply chain integrity before model selection and architecture decisions are finalized, then carrying security considerations through data pipeline design, prompt architecture, permission scoping, adversarial testing, and production monitoring. The core principle is that security becomes a property of the system's structure, making certain classes of vulnerability architecturally difficult or impossible rather than relying entirely on post-launch detection and patching.

Why Is Secure-by-Design Important for AI Systems Specifically?

Secure-by-design is particularly important for AI systems because the most critical AI-specific risks prompt injection, excessive agent permissions, insecure handling of untrusted retrieved content are architectural problems that late-stage security review cannot reliably catch or cheaply fix. Unlike many traditional software vulnerabilities that a code-level patch can resolve, prompt injection vulnerabilities frequently stem from fundamental decisions about how user input and system instructions are combined, requiring redesign rather than patching once a system is already in production. Remediation cost for AI security issues discovered in production runs 20–40x higher than addressing the same risk during the design phase, making secure-by-design a direct cost and risk reduction strategy, not just a theoretical best practice.

How Is Secure-by-Design Implemented for AI Applications?

Secure-by-design AI is implemented across six stages: AI-specific threat modeling against the OWASP Top 10 for LLM Applications before model or architecture selection; secure prompt architecture with clear separation between trusted system instructions and untrusted user or retrieved input; least-privilege permission scoping for any AI agent tool or data access, defined before development begins; secure output handling treating AI-generated content as untrusted input to any downstream system that consumes it; adversarial testing prompt injection, jailbreak, and data extraction attempts before production deployment and on a recurring basis afterward; and continuous production monitoring specifically tuned to AI security signals like anomalous tool-call patterns and output deviations that traditional application monitoring does not capture.

Threat Model Before You Select a Model. Scope Permissions Before You Grant Convenience.

Secure-by-design AI delivers its strongest protection when security questions are asked at the point where the answer is a design decision, not a production incident threat modeling before architecture selection, permission scoping before agent deployment, and adversarial testing before launch rather than as a response to a discovered breach.

The development and security teams building the safest enterprise AI applications in 2026 share one operational discipline: they ran AI-specific threat modeling against a systematic framework like the OWASP Top 10 for LLM Applications before committing to a model or architecture, not as a checklist item completed after the system already worked. That sequencing produced AI systems with substantially fewer critical findings in pre-production testing, and a measurably smaller blast radius in the rare cases where something still went wrong because least-privilege permission scoping was a design decision, not a hardening afterthought.

Run AI-specific threat modeling against your current or planned AI applications this month, scoring each against the OWASP Top 10 for LLM Applications categories. Audit your existing AI agents' tool and data permissions against what their specific function actually requires, not what was convenient to grant during development. Implement adversarial testing prompt injection and jailbreak attempts before your next AI feature launch, and build it into recurring testing afterward rather than treating it as a one-time pre-launch gate.

To build secure-by-design AI development practices integrated into your existing engineering workflows, from threat modeling through production monitoring, explore our DevSecOps Services and AI Governance capabilities structured for development and security teams that need AI security delivered as an architectural property, not a pre-launch checklist.

PARTNER WITH AGAMISOFT