A practical security model for LLM applications that combines governance, least privilege, adversarial testing, and production monitoring.
Introduction: AI security is application security plus new failure modes
A generative AI feature inherits the risks of every component around it: identity, APIs, cloud infrastructure, source data, dependencies, and user interface. It also introduces probabilistic behavior and natural-language attack surfaces. An LLM may follow malicious instructions hidden in a document, reveal sensitive context, select an unsafe tool, or produce convincing misinformation.
Secure design therefore cannot be delegated to a system prompt. OWASP's 2025 guidance and NIST's Generative AI Profile provide useful risk language; implementation still requires engineering controls across the complete lifecycle.
1. Establish ownership and risk tiers
Maintain an inventory of models, providers, datasets, prompts, indexes, tools, integrations, and business owners. Classify each use case by data sensitivity, autonomy, external exposure, reversibility, and impact on people. A public writing assistant and an agent that changes financial records should not share the same approval path.
Define prohibited uses, acceptable data, required testing, human oversight, incident ownership, and retirement criteria. ISO/IEC 42001 can help organizations manage AI policies and continuous improvement within a formal management system.
2. Minimize and control data
Send only the information required for the task. Redact secrets and unnecessary personal data before model processing. Review provider retention, model-training use, residency, subprocessors, and deletion controls. Protect prompts, responses, embeddings, evaluation data, and traces according to their sensitivity.
Tenant isolation must extend into caches, vector indexes, logs, and evaluation tooling. Test access controls with cross-tenant and role-change scenarios rather than assuming application permissions automatically propagate.
3. Design for prompt injection
Prompt injection occurs when untrusted input changes model behavior in an unintended way. It may arrive directly from a user or indirectly through a document, webpage, email, image, or tool result. Retrieval does not remove this risk.
Separate instructions from untrusted content, label data boundaries, constrain output with schemas, and keep sensitive decisions in deterministic code. Restrict tools and data for each request using the authenticated user's permissions. Treat model output as untrusted input to downstream systems.
4. Contain tool and agent actions
Give every tool the minimum permissions needed. Validate arguments server-side, use allowlists for destinations, and prevent arbitrary code, query, or URL execution. Apply transaction limits, rate limits, timeouts, budgets, and idempotency controls. Require confirmation for sensitive or irreversible actions.
Where possible, use a plan-review-execute pattern: the model proposes an action, policy code evaluates it, a person approves when required, and the system executes through a narrow API. Log each stage with the acting user, agent, policy decision, and result.
5. Secure the AI supply chain
Models, adapters, datasets, embedding libraries, plugins, and hosted services form a supply chain. Track provenance and versions, scan dependencies, review model licenses and data terms, pin deployable artifacts, and evaluate updates before promotion. Limit administrative access to prompts and indexes because silent changes can alter system behavior without a code release.
6. Test behavior adversarially
Test direct and indirect prompt injection, sensitive-data extraction, role confusion, excessive agency, insecure output handling, denial of service, poisoned knowledge, and cross-tenant access. Include encoded, multilingual, and obfuscated inputs. Evaluate both whether the model refuses and whether the surrounding application blocks unsafe effects.
Automate repeatable tests in delivery pipelines, but retain expert red-team exercises for complex systems. Convert incidents and near misses into regression cases.
7. Monitor and prepare to respond
Capture security-relevant traces without creating a new sensitive-data repository. Monitor unusual token volume, repeated refusals, anomalous tool sequences, retrieval of restricted sources, policy overrides, cost spikes, and changes in output quality. Define a kill switch that can disable a tool, model, tenant, or feature independently.
Incident plans should cover evidence preservation, model or prompt rollback, index rebuilds, credential rotation, provider coordination, user notification, and reassessment of affected outputs.
Conclusion: Put deterministic controls around probabilistic systems
Generative AI can be operated responsibly when its uncertainty is acknowledged in the architecture. Governance determines acceptable risk; least privilege limits impact; evaluations expose failure modes; monitoring detects change; and human approval protects consequential decisions. Firefly combines AI engineering with secure software delivery so innovation reaches production with controls that can be explained, tested, and audited.
Authoritative sources
Primary references used to research and validate this article.