A retailer can use a recommendation model without explaining every suggestion. If the model gets one wrong, a customer sees a product they do not want. A safety team cannot work that way. If an AI marks a case as non-serious and it was serious, a report can miss its deadline and a safety signal can be delayed. In pharmacovigilance, every AI-supported decision has to be something your team can explain, defend and correct.
This post looks at why explainability is a requirement in drug safety, what regulators and international bodies have said about it, and what explainable AI looks like in day-to-day work.
Inspectors ask “why”
Pharmacovigilance inspections test whether decisions follow your procedures. An inspector can pick a case and ask why it was classed as non-serious, why a term was coded the way it was, or why an article was judged not to contain a valid case. “The model said so” is not an answer.
The record-keeping rules point the same way. In the US, 21 CFR 11.10(e) requires secure, computer-generated, time-stamped audit trails that independently record the date and time of operator entries and actions that create, modify or delete electronic records, and record changes must not obscure what was recorded before. When AI takes part in creating or changing a safety record, your audit trail has to show what it did.
What regulators and CIOMS have said
Several recent documents address AI in the medicines lifecycle. None of them asks for blind trust; all of them ask for evidence.
- CIOMS Working Group XIV published its report on AI in pharmacovigilance on 4 December 2025. Its seven guiding principles are a risk-based approach, human oversight, validity and robustness, transparency and explainability, data privacy, fairness and equity, and governance and accountability.
- EMA’s reflection paper on the use of AI in the medicinal product lifecycle was adopted by the CHMP and CVMP in September 2024. In the same month EMA and the Heads of Medicines Agencies published guiding principles for large language models, which include critical thinking and cross-checking of outputs.
- FDA’s draft guidance of January 2025, Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products, proposes a seven-step, risk-based framework for establishing the credibility of an AI model for a defined context of use: define the question of interest, define the context of use, assess model risk, develop a credibility assessment plan, execute it, document the results and deviations, and determine whether the model is adequate for that context.
One caution about “explainable AI”
Explainability is easy to promise and harder to deliver. The CIOMS report is candid about this: it notes that explainable AI methods have limitations, because they return plausible hypotheses about how a complex model reached a result, not a guarantee of the pathway it actually used.
That has a practical consequence. A heat map or a list of “important words” produced after the fact may not be enough to defend a decision. What holds up better is an answer that is traceable by design: linked to the source text it came from, tied to an explicit rule or criterion, accompanied by a measure of confidence, and recorded in the audit trail.
What good looks like
Source traceability
Every value the AI extracts, such as a patient’s age, a product name, a dose or an event, should link back to the page, line or box it came from. A reviewer should be able to check it in seconds, without searching the document.
Rule traceability
Every assessment should show which rule or criterion it applied. If a case is classed as serious because the patient was admitted to hospital, the system should say so, using the seriousness criteria your SOPs follow. Rules that are explicit can be reviewed, tested and changed under change control.
Honest confidence
The system should show how sure it is, and results below a threshold you set should go to a person automatically. Thresholds belong in your SOPs, not in a vendor’s default settings.
A complete audit trail
Every AI action should be recorded with the model version that produced it, the time, and the person who accepted, changed or rejected it. When the model is updated, you should be able to tell which records were affected.
How much explanation is enough?
Not every use of AI carries the same risk, and the level of explanation and oversight should follow the risk. FDA’s draft guidance frames model risk as a combination of two factors: model influence, meaning how much the AI output drives the decision, and decision consequence, meaning the harm that could follow if the output is wrong. The same idea fits pharmacovigilance well.
An AI that ranks literature hits so reviewers see likely cases first, while every record is still reviewed, has low influence and a recoverable consequence. An AI that classifies seriousness with no human review has high influence and a serious consequence if it is wrong. Most real uses sit in between, and your risk assessment, not the vendor, should decide where.
Why reviewers need it too
Explainability is not only for inspectors. If a reviewer cannot see why a term was suggested or why a case was routed to them, they have two choices. They can re-check everything, which removes the benefit of the AI. Or they can trust it blindly, which removes the control. Neither is acceptable. A visible link to the source and the rule lets the reviewer check what matters and move on.
Why validation depends on it
Validation needs something to test. Explicit rules and traceable outputs can be tested against expected results, documented and re-tested after every change. Behaviour that cannot be explained is much harder to validate, and much harder to keep validated as models are updated. The CIOMS report also recommends continuous monitoring of performance to detect drift, which again depends on being able to see what the model did and why.
Questions to ask any AI vendor
- Can a reviewer see, for every extracted value, exactly where it came from?
- Does every assessment show the rule or criterion it applied?
- Are confidence thresholds configurable, and does uncertain work go to a person automatically?
- Is every AI action in the audit trail, with the model version?
- How are model updates validated and released, and who approves them?
- Which steps are human-in-the-loop and which are human-on-the-loop, and can we change that?
Sources
- 21 CFR 11.10, Controls for closed systems
- CIOMS Working Group XIV, Artificial Intelligence in Pharmacovigilance (report published 4 December 2025)
- Sagittal, “AI in PV: what the CIOMS WG XIV report means in practice” (list of the seven principles)
- EMA, Artificial intelligence (reflection paper, LLM guiding principles, AI workplan)
- NSF summary of FDA draft guidance, Considerations for the Use of AI to Support Regulatory Decision-Making for Drug and Biological Products (January 2025)