Six Questions Every CISO Should Ask Before Buying an AI Product

Every AI vendor pitch now includes some version of "enterprise-grade security," "SOC 2 compliant," and "full data control." Some of that is true in a narrow, technical sense. Some of it is meaningless in practice. Some of it is simply aspirational — a roadmap item described as a current capability.

Your job in the evaluation isn't to take the claim at face value. It's to ask the question specific enough that the answer either holds up or falls apart. Here are six that do that.

1. Where does our data actually go, and who inside your company can see it?

Why it matters: "We take data privacy seriously" is not an answer — it's a sentence designed to end the conversation. What you actually need is the full path: where your data is processed, where it's stored, how long it stays there, and who — including the vendor's own support and engineering staff — has the ability to read it.

What a real answer includes:

• A specific description of where data is processed and stored, down to the region or environment
• Confirmation that vendor employees can't casually access request or response content, even for troubleshooting
• Clear, configurable retention limits rather than an indefinite default
• Documentation you can actually verify — architecture diagrams or contract language, not just a claim on a trust page

Red flags:

• Vague reassurance without specifics
• A support process that requires staff to view raw customer traffic to diagnose issues
• Terms of service that reserve the right to use your data to improve the vendor's models

2. Who controls the encryption keys — you, or them?

Why it matters: Nearly every vendor will tell you data is encrypted at rest. That's the easy part. The real question is who holds the keys. If the vendor holds them, the vendor can decrypt your data — and so can their cloud provider, and so can anyone who serves them a legal request, without you ever being in the loop.

What a real answer includes:

• Customer-managed keys, held in infrastructure you control, not the vendor's
• Support for integrating with the key management systems you already use
• A real answer to "what happens to our data if we delete our keys" — ideally, it becomes permanently unreadable
• Key rotation that doesn't require re-encrypting everything or taking the product offline

Red flags:

• "We encrypt everything" with no mention of who controls the keys
• No option to bring your own key management
• An inability to explain what happens to your data after you close the account

3. Can we independently verify that your audit logs haven't been altered?

Why it matters: Audit logs only matter if you can trust them. A vendor that controls both your data and the record of what happened to it is, in principle, in a position to edit the story after the fact — intentionally or not. Regulators are increasingly asking for tamper-evident records, not just logs that exist.

What a real answer includes:

• Cryptographically signed log entries, not just timestamped database rows
• A chained structure where altering one record breaks the integrity of everything after it
• Signing keys that you control, so the vendor genuinely can't quietly regenerate a "clean" version of history
• A way to independently verify the log's integrity yourself, without depending on the vendor's own dashboard

Red flags:

• "Stored in a highly durable cloud service" offered as if durability were the same thing as tamper-evidence
• No cryptographic signing at all
• No tooling that lets you check the record independently

4. What happens to our data and our history if you get breached, acquired, or run out of money?

Why it matters: Vendors get breached. Startups run out of runway. Products get folded into other products after an acquisition. None of that is hypothetical in this market right now, and you need a real answer for what your exposure looks like in each case — including whether you can walk away with your data and your compliance record intact, without the vendor's cooperation.

What a real answer includes:

• A self-hosted or customer-environment deployment option, if your risk tolerance calls for it — so a vendor-side breach doesn't automatically mean your data was exposed too
• Full data export in a standard, portable format, not one that only the vendor's own tools can read
• Audit history that belongs to you, accessible independent of the vendor's portal
• A contractual, time-bound commitment to delete your data on termination, with written confirmation

Red flags:

• No option to keep processing within your own environment
• Proprietary export formats that create vendor lock-in on your own compliance records
• Silence in the contract about what happens to your data after termination

5. How does this handle the threats that are specific to AI, not just software in general?

Why it matters: Standard security controls were built for standard software risks. AI products introduce a different threat surface — inputs designed to manipulate the model, tools invoked in ways nobody intended, and data leaking through outputs rather than through a conventional breach. A vendor who only talks about general application security hasn't addressed the part of the risk that's actually new.

What a real answer includes:

• Detection for manipulated or adversarial input, including content the model processes indirectly — documents, search results, retrieved data — not just what the user types directly
• Validation and permission scoping on anything the AI can act on, with limits on what it's allowed to touch
• Detection and redaction of sensitive data in both directions — what goes in and what comes back out
• Specific controls for autonomous, multi-step behavior, not just single-turn conversation safety

Red flags:

• Security described purely in terms of standard web application protections
• No mention of manipulated-input risk or tool misuse at all
• Sensitive-data detection that only covers what the user submits, not what the system generates or retrieves

6. What actually happens when the model gets something wrong?

Why it matters: Every AI product will eventually produce an output that's wrong, biased, or acted on when it shouldn't have been. The differentiator isn't whether that happens — it's whether the vendor has a defined process for catching it, telling you about it, and fixing it, instead of routing it through a generic support queue.

What a real answer includes:

• A defined incident process specific to model behavior, separate from general uptime SLAs
• Clear commitments on how fast you'd be notified if something the AI did affected you
• Evidence that this process has actually been used, not just documented for the sales deck

Red flags:

• Model errors treated as ordinary support tickets
• No contractual language addressing what happens when the AI component specifically fails
• An answer that focuses entirely on the model's general safety training, without addressing what's been verified for this product

How to Actually Use These Questions

Ask them of the vendor's engineering team, not just their sales team. Sales will give you a confident answer regardless of what's true underneath it — that's the job. Engineers will either confirm the capability with specifics or, more usefully, tell you where the gap actually is.

For each of these six, push for three things: a written answer you can put in your vendor risk file, a reference customer who can speak to it directly, and a live technical demonstration rather than a slide describing it. Vendors who can produce all three, for all six questions, are the ones worth taking seriously. The rest are still selling you a story — just a well-designed one.