AI SECURITY

You have an AI strategy. Who owns its security?

A practical guide for CTOs and engineering leaders: define the AI Security Architect’s responsibilities, protect connected systems and build security into delivery.

An AI strategy needs a security owner

An internal assistant is still a business system. Its risk depends on the information it receives, the people it serves and the actions it can perform. A document summariser and an agent authorised to change customer records need different safeguards. Start by defining the business impact of a wrong answer, an unauthorised disclosure or an unwanted action. A job title is not a security programme: an accountable owner needs engineering support, authority and a clear escalation route.

Cloud security remains essential. AI adds another layer.

Identity management, secure deployment, patching, encryption and recovery remain part of the foundation. AI introduces additional questions about untrusted language, model behaviour and connected tools. A private deployment does not automatically make a system safe. Nor does a stronger model remove the need for application controls. Review the complete path from user input to retrieval, model response and downstream execution.

Understand the risks before choosing the controls

Different failure modes need different responses. Do not use “prompt injection” as a catch-all for every AI incident. Record the relevant threats for each use case and explain the business consequence in language decision-makers can assess.

  • Prompt injection: instructions in a message or retrieved document attempt to redirect the assistant away from its intended task.
  • Data or model poisoning: manipulated training, fine-tuning or embedding inputs can compromise the integrity of a system. This differs from instructions arriving during a live conversation.
  • Sensitive-data exposure: retrieval, responses, logs or integrations can disclose information to the wrong person or destination.
  • Unsafe execution: an application may accept generated content as a command, or give an agent broader permissions than its task requires.
  • Supplier and consumption risks: dependencies, model changes and repeated requests can affect reliability, security and operating cost.

Define the AI Security Architect by the work

The role connects security architecture with AI engineering and business risk. It can be a dedicated position or an explicitly assigned responsibility within an existing team. What matters is whether the person can challenge the design and translate identified risks into testable requirements.

  • Map data flows, trust boundaries, model providers, retrieval stores and tool permissions.
  • Agree security requirements and release criteria with product and engineering owners.
  • Review how identities, sensitive information and third-party components are handled.
  • Coordinate adversarial testing, remediation evidence and incident preparation.
  • Maintain a record of accepted risks, decision owners and triggers for reassessment.

Keep authority outside the conversation

Treat generated requests as proposals to the application, not proof of permission. A backend service should check who initiated an action, which records they may access and whether the operation is permitted. Give integrations narrowly scoped identities; keep credentials outside prompts. For consequential changes, present the exact action for approval before execution. Reviewers need enough context to make that approval meaningful.

Protect retrieval before information reaches the model

Search quality and access control solve different problems. A useful answer can still expose a confidential document. Check user and tenant permissions before returning retrieved material; preserve source ownership and provenance as content changes. Control who can add documents to the knowledge base. Review retention across providers, indexes and logs, rather than assuming that deleting an original file removes every copy.

An example: the support agent that can issue refunds

Imagine a customer-support agent that reads tickets and proposes refunds. A malicious ticket tries to instruct it to pay a different account. The security question is not simply whether the model recognises the instruction. The payment service must independently enforce the permitted customer, amount and destination. A proposal exceeding the agreed limit should require an authorised reviewer. This illustrative scenario shows why limiting business authority matters even when a model behaves unexpectedly.

Test the boundaries, not just the quality of answers

A fluent demo is not evidence of a secure release. Build a controlled evaluation set around the system’s actual permissions and potential harm. Repeat relevant checks when models, prompts, retrieval sources or integrations change.

  • Attempt cross-user and cross-tenant retrieval using synthetic information.
  • Place conflicting instructions in test documents and inspect whether protected operations remain blocked.
  • Check that invalid tool arguments and unauthorised destinations are rejected by application code.
  • Exercise request, time and spending limits, including repeated agent steps.
  • Capture the expected result, observed behaviour, remediation and retest outcome.

Prepare to contain an incident

Name the people who can disable a connector, revoke access or roll back a release. Keep enough diagnostic evidence to investigate the sequence of events without unnecessarily retaining confidential prompts or personal data. Practise a response for suspected disclosure and unwanted actions. Monitoring should show failed authorisation, unexpected tool use and unusual consumption—not merely whether the model endpoint is available.

Hire for evidence of capability

Avoid building a wish list that combines every cloud, machine-learning, compliance and security specialism into one impossible vacancy. Define the architecture problems the hire must own, then assess relevant experience. A useful interview exercise is a small AI application design: ask the candidate to identify trust boundaries, prioritise risks and explain how they would verify the controls. Look for collaboration and judgement alongside technical depth.

Questions to ask before the next release

Use these questions to make ownership and release readiness concrete. The goal is an informed decision about residual risk, not a claim that every possible attack has been eliminated.

  • Who owns this use case and who can accept its remaining risks?
  • What information can it retrieve, and whose permissions are checked?
  • What can it change or send, and where are those limits enforced?
  • Which abuse scenarios have been tested, and what remains unresolved?
  • Who can contain an incident, and has that response been rehearsed?

Connect the hiring brief with the security requirement

If you are planning an AI security hire, begin with your architecture, delivery stage and internal capabilities. ULTISOURCE can discuss a requirement-led talent search and the scope of relevant cybersecurity advisory support. Specialist availability and engagement deliverables should be confirmed in that conversation. This guide describes evaluation considerations; it does not certify a system or guarantee protection against attacks.

FURTHER READING

References & further reading

This guide offers practical evaluation questions. The sources below provide the underlying risk-management and assessment principles; they do not endorse ULTISOURCE or its products.

Discuss AI security talent.

Start a conversation
YOUR NEXT CHAPTER

What are you
building next?

Let’s start a conversation

YOUR PRIVACY

We currently use only storage needed for website security and administration. No optional analytics or advertising cookies are enabled, so there are no optional cookies to accept or reject.

Necessary security and administrationAdministrator sign-in uses a secure session cookie lasting up to eight hours. Cloudflare Turnstile checks help protect our forms against spam.Always active
Analytics and marketingNo analytics or advertising services are currently connected.Not used

If we introduce optional cookies, we will ask for your choice before enabling them.

Read our cookie policy