AI assurance is the work of measuring, evaluating, and communicating the trustworthiness of AI systems. Independent verification organizations (IVOs) are outside organizations brought in to do part of that work. Dozens of assurance providers don’t consider themselves IVOs yet, but that is changing fast. Through SB 813, California is building a system to designate AI assurance organizations to perform this work for specific purposes, and it is moving quickly. That makes it important to clarify the elements we all need to align on when we talk about the scope of IVO work. Last week we looked at the type of assurance work. This week we look at where that work happens across the AI life cycle.
In our last post, An AI Evaluator Is Not an Auditor, we separated four kinds of assurance work: investigation, evaluation, assessment, and audit. This post applies to all of them. IVOs may do any of this work, and we refer to it as “IVO work” throughout.
If you deploy, insure, or evaluate AI, “independently checked” can mean very different things. An IVO might examine a system before release, investigate a harm after it happens, review how a company governs itself, or judge whether a deployment fits a particular business. Each job asks a different question and produces a different kind of evidence, so what an IVO can tell you depends on the job it was hired to do.
The four examples below show IVO work at different stages, from the lab to your business. They are not a complete list, and more types of IVO work will emerge as the field matures. They show that managing AI risk happens in layers, and each layer builds on the one before. If an early layer has gaps, later layers may miss the problem or be unable to fix it. Many risks also show up at more than one stage. Cybersecurity, for example, matters at every stage, so the risks listed are examples, not a complete list.
1. Internal deployment and development
Is the system trustworthy for use and testing inside the lab?
This work happens while a system is still being built and tested, before anyone outside the lab has access, and it requires deep access to the developer’s systems. It could include risks such as an AI system pursuing goals its developers did not intend, people losing the ability to direct or shut it down, cybersecurity weaknesses, and recursive self-improvement (RSI), AI systems building their own successors with less and less human involvement.
This is the closest example to the embedded evaluator conversation. It matters because it is the stage where outsiders can see the least, so without an IVO, they have to rely on the company’s own account of how these risks are being managed.
2. Pre-release
Is the system trustworthy enough to release to external users?
This work happens before a system is made available to people outside the company. It can include risks that, once the system is released, would be very hard to contain, such as help with building chemical, biological, radiological, or nuclear (CBRN) weapons, manipulation, and flaws from how the system was trained, such as telling users what they want to hear or being built to keep them hooked.
3. Post-release societal
Is the system trustworthy for use by external users?
This can include risks such as mental health effects, bias and fairness, and misuse. Many of these harms only appear once millions of people use a system in ways developer tests could not fully anticipate, so this work has to happen post-system release, and be ongoing.
4. Use-case level commercial
Is the system trustworthy for deployment to employees and customers?
This work happens inside businesses that deploy systems and agents built on a system. It can include risks such as reliability, privacy, and confidentiality, and it depends on context: the same system can be fine for one use case in a particular company and unfit for another.
5. Across every stage: governance and the development process
Testing a finished system can only cover so much, so IVO work can also look at the organization behind it and the processes it follows. On governance, that can include who decides how much risk is acceptable, whether the company’s risk policies are adequate and actually followed, how pay and culture affect the way people respond to bad news, and whether internal checks and public commitments hold up. On the development process, it can include where training data comes from, who approves training runs and changes to them, how testing and red-teaming (deliberately trying to break the system) are done, and who decides when a system is ready to release.
What this means in practice
Ensuring an AI system is trustworthy for your specific use case only goes as far as the earlier layers of safety allow. It can help you understand and reduce risks tied to your deployment, but it cannot make up for gaps in the system. In our last post we suggested four questions to ask before relying on any IVO work: What were the criteria? Who set them? What information will this give me? Who is the performer independent of? They apply here too, and we can add one more: which layer of the system did this check, and what unchecked layers does it leave to someone else?
If you deploy AI: Ask vendors which layers have had independent verification work, and treat gaps as your responsibility until shown otherwise.
If you insure AI risk: A claim of assurance means little until you know which layer it covers. Evidence from different layers answers different questions.
If you provide IVO work: Say plainly which question you were engaged to answer, what you looked at, and what you cannot conclude.
What is still unsettled
The IVO field is broad, and some questions have no answers yet. Who decides what IVO work should check? Who pays for it? Who confirms that IVOs are qualified? And how do findings from different kinds of IVO work fit together so a buyer or insurer can actually use them? Answering them is the work of building a trustworthy assurance ecosystem, and it is what PACT AI is currently focused on.