What Europe's Draft Annex 22 Actually Asks Of Human Oversight Of AI

By Sachin Bhandari, TrustBridge Compliance

I advised a European pharmaceutical client that had built an artificial intelligence (AI) tool to classify deviations in its good manufacturing practice (GMP) operations. The tool read the incident description, asked the person raising it a few questions, and proposed a risk classification along with the content needed for the corrective and preventive action (CAPA). Each classification went to the quality team for approval, and each one looked reasonable on its own.

Then we trended the classifications by site during a periodic review. Deviations from one site were always classified as medium or low. When we looked closer, we found the model had never been trained on that site's data. It had learned from a sterile manufacturing site, and this site did packaging. The model looked for the words it knew, such as "sterile," and when it couldn't find them it scored the event low, or medium at best. Some of those deviations should have been classified as high risk.

Every single classification had looked plausible to the people who approved it. The error only showed when we put a few months of results side by side. After that review, the client tightened change control so that using the tool at any site it hasn't been trained on now needs a formal change and an assessment first. A reviewer looks at one output at a time, and some of the errors that matter only show up when you look at a few hundred together. The draft of the new Annex 22 to the European Union (EU) GMP guidelines spends a lot of its text on that difference between checking single items and keeping a whole process under control.

What The Draft Says About Human Review

The European Commission and the Pharmaceutical Inspection Co-operation Scheme (PIC/S) consulted on draft Annex 22, "Artificial Intelligence," from July 7 to Oct. 7, 2025, alongside revised drafts of Chapter 4 and Annex 11.¹ The European Medicines Agency (EMA) said in February 2026 that the draft drew about 1,300 comments and that it expected a final document by the end of the year.² As of the publication date of this article, Annex 22 is still a consultation draft, so everything below refers to that draft, and the wording may still change.

The draft covers AI models in critical GMP applications, meaning those with direct impact on patient safety, product quality, or data integrity. It applies to static models with deterministic output and says dynamic and probabilistic models, generative AI, and large language models "should not be used in critical GMP applications."³ Where generative AI is used in noncritical work, the draft wants qualified people to be responsible for checking that its outputs suit the intended use, an arrangement it calls human-in-the-loop.

The two clauses I'd read twice are 3.3 and 10.5. Section 3.3 says that where a model gives input to a decision a human operator makes, "and where the effort to test such model has been diminished, the description of the intended use should include the responsibility of the operator. In this case, the training and consistent performance of the operator should be monitored like any other manual process." Section 10.5 adds that records should be kept from this process, and that depending on criticality and the level of model testing, this "may imply a consistent review and/or test of every output from the model, according to a procedure."³

So the draft frames human review as a trade. You can test the model less, and in exchange the person becomes a documented, monitored part of the process. Their responsibility goes into the intended use description, which section 3.1 says a process subject matter expert (SME) must approve before acceptance testing. Their consistency gets watched like any manual inspection, and their reviews are recorded. In my experience that's the harder half of the trade, because most teams have far more practice testing software than showing that a person reviews consistently.

The draft gives people a defined job at several other points, shown in Figure 1. A process SME sets and approves the acceptance criteria before testing (4.2), and those criteria must be at least as high as the performance of the process the model replaces (4.3, "No decrease"). That means you need a measured baseline for the people doing the job today. Someone reviews the features driving the model's decisions as part of approving test results (8.2). And a confidence threshold decides which cases reach a person, since the draft suggests that low-confidence outcomes could be flagged as "undecided" (9.2).³

Figure 1: Draft Annex 22 gives people a defined job at several points in the AI model lifecycle, and it treats the reviewer added to reduce model testing as part of the validated process. Clause numbers refer to the July 2025 consultation draft (reference 3). Figure by the author. Click on image to enlarge.

The draft also has three open questions that each company will have to answer for itself. How far can testing be "diminished" before the human review is doing most of the work, and how does a team show the trade is fair? How should the operator's "consistent performance" be measured? And how should a team manage automation bias, the well-studied tendency of people to accept a confident machine's answer? The draft doesn't use that term.

Regulators are asking the same questions. The EMA's report of its June 2026 workshop with industry on Annex 22, published in early October, records regulators asking "how to monitor the performance of the human performing oversight; how to manage automation bias; what information and competence a human reviewer needs" and "how oversight can be evidenced during inspection."⁴ The report sums up that session as support for language requiring "meaningful, risk-based and documented oversight," and among its main messages it lists that oversight "cannot compensate for inadequate validation."⁴ It's a record of discussion with no regulatory force, but it shows where the drafting group's attention is.

When A Reviewer Reduces Risk

In the live deployments I've worked on, a reviewer reduces risk when four conditions hold at the same time. Drop any one and the review turns into a confirmation step.

The reviewer can see what the model saw, along with the confidence score and the main features behind the output. A reviewer shown only a result and a green tick can do little more than agree with it.

The reviewer knows the process well enough to recognize a wrong answer, with the training you'd expect of someone doing the task by hand. The workshop report puts the bar in a sensible place: The reviewer has to understand the process, the output, and the decision context "sufficiently to challenge the AI."⁴

The reviewer has the authority to overrule the model, and the standard operating procedure (SOP) makes overruling a normal, quick action. If disagreeing takes a deviation form and a talk with the manager, people will disagree less often.

And the reviewer has the time. If the throughput target assumes 40 seconds per case, the review will take 40 seconds whatever the SOP says.

The U.S. Food and Drug Administration (FDA) made the accountability half of this point in April 2026, in what appears to be its first warning letter citing AI use in drug manufacturing. The firm had used AI to create its drug specifications, procedures, and master production and control records. The FDA wrote that "any output or recommendations from an AI agent must be reviewed and cleared by an authorized human representative of your firm's QU," its quality unit.⁵ The letter puts accountability with the quality unit, and draft Annex 22 goes a step further by asking you to show that the review works.

What The SOP And The Records Need To Contain

Two documents show whether oversight exists. The SOP shows it was designed, and the records show it happened.

The SOP should name the decision the reviewer owns, what they'll see, what they're checking for, and what happens when they disagree. It should also say how "undecided" cases get routed and who can suspend the model.

The records need to capture the reviewer's reasoning as well as the signature. I'd want to see overrides with the reason, disagreements that were escalated and how they closed, and what people decided on the "undecided" cases. Then someone has to trend those records by subgroup, the way section 3.2 asks you to split the input space by characteristics such as site, equipment, material, or defect type.³ In the deviation example, the trend by site is what found the problem. A systematic error usually shows up in one subgroup and stays hidden in the totals.

A Checklist For Monday Morning

I'd ask these questions this week of any AI-assisted step in a GMP operation.

• Does the intended use description name the human's responsibility, and did a process SME approve it before acceptance testing?
• If you reduced model testing because a person reviews the output, is that trade written down with a rationale?
• Do you have a measured baseline for the manual process the model replaced, so you can show "no decrease"?
• Can the reviewer see the inputs, confidence score, and main features behind each output, and do low-confidence cases reach a person as "undecided"?
• Is each reviewer's performance monitored the way you'd monitor a manual visual inspection, with training records and a periodic check?
• Do your records capture overrides, escalations, and their reasons, and does someone trend them by subgroup at least monthly?
• Does the intended use say which sites, products, and input types the model was trained on, and does change control stop it from being used anywhere else?
• Does the throughput target leave the reviewer enough time per case to do the review your SOP describes?

Where you can answer yes to all of them, your reviewer is a control you can defend to an inspector. Where you can't, the review design is the place to start. The final text may still change, and fixing the design now costs far less than fixing it after an inspector has asked.

References

1. Pharmaceutical Inspection Co-operation Scheme (PIC/S). "Joint stakeholders consultation on the revision of Chapter 4 on Documentation, Annex 11 on Computerised Systems and the new Annex 22 on Artificial Intelligence of the PIC/S and EU GMP Guides." July 7, 2025. https://picscheme.org/en/news/joint-stakeholders-consultation-on-the-revision-of-chapter-4
2. European Medicines Agency (EMA). "Summary notes: meeting of the HMA/EMA group focused on AI with industry stakeholders." February 2026. https://www.ema.europa.eu/en/documents/minutes/summary-notes-meeting-hma-ema-group-focused-ai-industry-stakeholders-february-2026_en.pdf
3. European Commission. EudraLex Volume 4, "Annex 22: Artificial Intelligence." Consultation draft, July 2025. https://health.ec.europa.eu/document/download/5f38a92d-bb8e-4264-8898-ea076e926db6_en?filename=mp_vol4_chap4_annex22_consultation_guideline_en.pdf
4. European Medicines Agency (EMA). "Good manufacturing practice (GMP): Multistakeholder workshop on expert contributions to artificial intelligence guidance development (Annex 22)." Workshop report, EMA/156789/2026, published October 2026. https://www.ema.europa.eu/en/documents/report/report-multistakeholder-workshop-expert-contributions-artificial-intelligence-guidance-development-annex-22_en.pdf
5. U.S. Food and Drug Administration (FDA). Warning Letter 320-26-58, Purolea Cosmetics Lab. April 2, 2026. https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/warning-letters/purolea-cosmetics-lab-722591-04022026

About The Author:

Sachin Bhandari is the founder and principal consultant of TrustBridge Compliance, an independent practice that helps pharma, biotech, and contract manufacturing companies validate AI and computerized systems for GMP use. He has more than 25 years in pharma quality IT, nearly 20 of them in validation, across computer system validation, computer software assurance, data integrity, and enterprise quality systems. He sits on the GAMP (Good Automated Manufacturing Practice) Global Steering Committee, contributed to the industry review of draft EU GMP Annex 22, and wrote Validating AI in GxP Environments: A Practitioner's Handbook. Reach him at sachin.bhandari@trustbridge-compliance.com or trustbridge-compliance.com.