The short answer. A software company can reliably reconstruct what its own AI feature did only if it recorded the evidence at the time. At minimum, that means five records: the user’s input, the data the feature retrieved, the prompt, the model version, and the output. Software teams investigate incidents with change logs, access logs, and by re-running the request. An AI feature can change its behavior with no code change, often reads data under its own account, and can give a different answer on a second run.
A company that cannot reconstruct an AI incident has to assume the worst about it. That means broader customer notification, a longer feature outage, and a harder renewal conversation.
Five questions below show whether a portfolio company can investigate an incident in its own AI features.
Research Grounding
Gartner, September 24, 2026. Gartner predicts that by 2028, 50% of all enterprise cybersecurity incident response efforts will focus on incidents involving custom-built AI-driven applications. Its first recommendation is a complete inventory of the AI-enabled applications in use.
IBM, July 29, 2026. IBM’s 2026 Cost of a Data Breach study covered 602 organizations that had a breach between March 2025 and February 2026. More than 20% reported a breach targeting AI models or applications.
IBM, September 2026. In IBM’s analysis of the same study, 21% of organizations reported an incident involving an AI model or application, up from 13% a year earlier. Of the organizations with an AI-related breach, 92% lacked what IBM classifies as proper AI access controls.
IBM’s dollar figures describe the organizations it surveyed. A smaller company’s costs will differ. The direction still applies to a lower-mid-market portfolio. AI features are becoming a common source of incidents, and their access controls are often missing.
For a software company, the AI applications in Gartner’s prediction include the AI features inside its own product. Its customers' security teams will be the ones investigating.
The PE Translation
Many portfolio software companies have added AI features in the last two years. They include summaries on a record, search boxes that answer questions, and assistants that draft emails. Each one sends the customer’s data to a model.
A board reviewing these features will normally ask whether customers use them and what they cost. What happens when one of them goes wrong needs its own review.
An AI incident at a software company often starts with a customer. The customer may report that the feature showed one user another customer’s data, took an action nobody asked for, or followed instructions hidden in a document it was asked to summarize. The customer’s security team then asks what happened, which records were affected, and whether it can happen again.
Engineering teams answer those questions with methods built for ordinary software. An AI feature weakens each of them.
WHAT AN INVESTIGATION RELIES ON | WHAT AN AI FEATURE CHANGES |
|---|---|
The change log shows what changed and when | Behavior can change with no code change: the provider updates the model, someone edits a prompt in an admin screen, or the data the feature reads changes |
Access logs show who saw which records | The feature often reads data under its own service account, so the log shows the feature rather than the person who asked |
The team re-runs the request to reproduce the problem | The same request can produce a different answer on the next run, so a re-run is not reliable evidence of what happened |
Access control decides who may use the feature | The feature also has to treat the content it reads as untrusted, because instructions hidden in a document or email can steer it, an attack known as prompt injection |
Re-running the request does not reliably show what an AI feature did. The investigation depends on records captured when the original request happened.
Cost. A team that cannot show which records were affected has to treat every customer who used the feature as possibly affected. Notification, legal review and support time scale with that number.
Revenue. The incident becomes part of the renewal conversation at the customers who were told about it. A customer that cannot get a clear account of what happened may ask for the feature to be switched off, and the feature’s revenue goes with it.
Valuation. A buyer’s technical team will ask for an inventory of AI features, the data each can reach and what is recorded. Missing records become a remediation line in the buyer’s plan after closing.
Operator Experience
The cheapest control on an AI feature is a limit on what data it can reach, decided when the feature is designed.
In the AI work I led, our rule was to bring the AI to the data, never the data to the AI. We set rules on which data could go to which model. When we built an agent that resolved support tickets, wrote code, and opened pull requests, it worked only from logs, code, and prior tickets. It had no access to customer databases.
A limit like that reduces risk more reliably than monitoring does. An agent that cannot read customer records cannot leak them, whatever instructions reach it.
In my experience, many AI features in mid-market products were built faster than that. Two artifacts show quickly how much risk a feature has. Both can be checked in an afternoon.
The retrieval filter. Many AI features search an index of customer documents before answering. Ask an engineer to show where that search filters by customer, and whether the filter is applied when the query runs. A filter applied after the results come back, or applied only in the screen, is how one customer’s data reaches another customer’s answer. Where strict filters slow the vector database, a separate index for each customer keeps the isolation.
The request record. Pick one AI response a customer saw last week. Ask the team to produce the user’s input, the data the feature retrieved, the prompt, the model version, and the output. If any of the five is missing, the team may not be able to reliably reconstruct what the feature did.
The request record holds customer data, so it needs the same access controls and retention rules as the product database. It should exclude secrets such as passwords and API keys. It also has to follow the deletion terms in customer contracts and privacy laws. A team that logs everything to a general-purpose logging tool has created a second copy of its most sensitive data.
Board Question
If a customer told us tomorrow that our AI feature showed them another customer’s data, how long would it take us to show whether it happened?
Five Questions to Ask About Every AI Feature in the Product
QUESTION | WHO ANSWERS | WHAT A WEAK ANSWER SOUNDS LIKE |
|---|---|---|
1. Which features in our product call a model, and what data can each one reach? | CTO with the head of product | Just the assistant. |
2. For any AI response a customer questions, can we retrieve the input, the retrieved data, the prompt, the model version and the output? | CTO | We log errors. |
3. Does each feature read data with the requesting user’s permissions, and filter by customer when the query runs? | Lead architect or security lead | The service account has read access to the index. |
4. Can we switch a feature off for one customer, or for everyone, within minutes and without a deployment? | Platform lead | We’d push a hotfix. |
5. Does our incident response plan cover AI features, and who decides to switch one off? | Security lead with the CEO | It’s the same as any other incident. |
The fourth weak answer means containment depends on a release. A release usually takes hours, and the feature keeps running while it is prepared. Where AI sits inside a core workflow, such as ticket routing, the switch also needs a non-AI fallback, such as a default queue.
The fifth question is the one that ends in a decision. Switching off an AI feature affects customers who depend on it, so the authority to do it should be agreed before an incident rather than during one.
Three Decisions
1. Produce an inventory of AI features and the data each can reach
Give the CTO two weeks. The inventory should name each feature, the model and provider it uses, the data it can read and the owner. This is Gartner’s first recommendation, applied to the company’s own product.
2. Record each AI request in a form an investigator can use
For every AI feature, record the input, the retrieved data, the prompt, the model version and the output. Protect that record like the product database.
3. Build a per-customer switch, and test for cross-customer leakage
Every AI feature should have a switch that works per customer and for all customers, without a deployment. Then ask a security tester to try to make each feature reveal another customer’s data, including through instructions hidden in an uploaded document.
One Number for the Next Operating Review
AI feature traceability: the percentage of AI features in production for which the team can produce all five request records for any response in the last 90 days.
The number shows how many AI features the company could investigate today. Report it with the AI roadmap, so each new feature arrives with its records in place.
Board Takeaway
An AI feature can change behavior without a code change, and a re-run may not reproduce what it did. The company can reliably investigate an AI incident only with records made at the time.
Before approving the next AI feature, ask whether the current ones could be investigated tomorrow.
Portco Brief is a weekly briefing for PE operating partners and portfolio company executives focused on technology, AI, and value creation. If this was forwarded to you, subscribe at portcobrief.com.
Sources
Gartner, Gartner Security & Risk Management Summit EMEA 2026: Day 3 Highlights, September 24, 2026.
IBM, IBM Study: One in Four Malicious Breaches are AI-Enabled, Costing Companies $6 Million on Average, July 29, 2026.
IBM, Every AI agent followed the rules, and the data still leaked, September 2026.