T8 — AI Operational Monitoring Plan — free template

Document Control

Field	Entry
Document ID	T8-AI-XXXX
Version	0.1
Agent / model	name, registry ID, version
Linked documents	T2-…, T3-…, T6-…, T7-…, T9-…, T10-…
Tier / model risk	Tier 2 / 3; Low / Medium / High
Monitoring owner (role)	role
Effective from	go-live date

Approvals

Role	Name	Signature	Date
System / agent owner	name	e-signature	date
Process / clinical SME	name	e-signature	date
Quality / CSV	name	e-signature	date

Revision History

Version	Date	Author	Change summary
0.1	date	author	Initial plan

Instructions

1. Monitoring Scope

Field	Entry
Released scope being monitored (from T7 §8)
Validated performance being protected (from T7 §4)	metric, value, lower bound, per critical subgroup
Input sample space and subgroups (from T3)
Data sources for monitoring	system logs, audit trail, QMS, LIMS, reference re-checks
Dashboard / report location

2. Performance Monitoring (Annex 22 §10.3)

KPI ID	Metric	Ground-truth source	Frequency	Action limit	Response
PERF-01	sensitivity on critical class	manual re-inspection of AQL sample / re-challenge kit			playbook step
PERF-02	precision / false-reject rate	confirmed dispositions
PERF-03	periodic re-challenge with qualified set	locked challenge set / defect kit	quarterly + after maintenance	below validated value

3. Input Drift Monitoring (Annex 22 §10.4)

KPI ID	Metric	Frequency	Alert limit	Response
DRIFT-01	embedding distance / population stability index vs. training distribution
DRIFT-02	out-of-distribution or "undecided" rate		e.g., > 2× validated baseline
DRIFT-03	share of inputs from sources not in the validated scope (new site, supplier, document type)		any	scope check → T9 or change control
DRIFT-04	class prevalence (SPC p-chart per class)	per batch / weekly	Western Electric rules	route to process deviation first; check precision at new prevalence

4. Confidence Calibration

KPI ID	Metric	Frequency	Alert limit	Action limit	Owner	Response
CAL-01	Brier score / expected calibration error on confirmed outcomes
CAL-02	distribution of cases across confidence bands vs. validation		shift > x pp

5. Human-in-the-Loop Monitoring (Annex 22 §3.3, §10.5)

KPI ID	Metric	Frequency	Alert limit	Action limit	Response
HITL-01	acceptance rate, rolling window		< 60% or > 98% sustained		low: fitness review; high: seeded-error check
HITL-02	modify / override rate, and direction (AI said reject → human accepted, and vice versa)
HITL-03	seeded-error catch rate (known errors mixed into normal work)	quarterly	below T7 HITL result	below baseline	retrain reviewers; restrict
HITL-04	edit distance / time-on-review per item		sharp fall		automation-bias check

6. Agent Operations (agents and LLM components only)

KPI ID	Metric	Frequency	Alert limit	Action limit	Owner	Response
OPS-01	unsupported-claim rate caught by checker			any reaching a signed record
OPS-02	disallowed tool-call attempts / prompt-injection detections		any		Security
OPS-03	cost per decision (tokens + compute)			circuit breaker
OPS-04	latency p95 / decision velocity
OPS-05	degraded-mode activations (connector loss → confidence cap)
OPS-06	configuration integrity (hash check of model, prompt, thresholds, index)	daily	any mismatch	any mismatch		stop; investigate as unauthorised change — Annex 22 §10.2

7. Response Playbook and Escalation

Level	Trigger	Response	Decision owner	Record
0 — Investigate	Any alert limit	Document the investigation; confirm the signal; check for process causes before blaming the model	Monitoring owner	Monitoring log
1 — Restrict	Action limit on a critical KPI, or unexplained alert repeated	Restrict to advisory or to the unaffected scope; increase human review to 100% of affected outputs	System owner + QA	Deviation
2 — Retrain / reconfigure	Confirmed drift or degradation with a known cause	Change via T9 (if in envelope) or full change control; regression on the locked test set	System owner + SME + QA	Change control
3 — Retire / suspend	Critical error reached a GxP record, or performance cannot be restored	Suspend the AI function; revert to the manual process; impact assessment on past decisions	QA	Deviation / CAPA

8. Failure-Mode Coverage

T2 FM	Rating	KPI ID(s)	Covered
FM-01	High	PERF-01, PERF-03	Y/N
FM-04
FM-06		HITL-01, HITL-03

9. Reporting

Audience	Content	Frequency
System owner	All KPIs, open investigations	weekly / monthly
Quality	Action-limit events, deviations, trend summary	monthly
Periodic review (T10)	Full trend record	per tier
Management review	Portfolio summary across AI functions	quarterly

Appendix A — Worked Example: In-line AVI Camera for Lyophilised Vials (all numbers illustrative)Illustrative worked example

Section 1. Deep-learning classifier sorting each vial as accept, reject:particle, reject:crack, reject:fill-height, reject:cake-defect, or undecided. Released scope: Line 3, 10 mL clear glass vials from Supplier A, products P1–P2. Validated performance (T7): particle sensitivity 0.97 (lower bound 0.94) against manual Knapp reject-zone efficiency 0.91; false-reject rate 1.2%.

KPI ID	Metric	Ground truth / source	Frequency	Alert limit	Action limit	Owner	Response
PERF-01	Critical defects found in manual re-inspection of an AQL sample of accepted units	Qualified manual inspectors	Per batch	1 critical defect in any batch	2 in a rolling 10 batches	QA	L0 → L1: 100% manual re-inspection of affected batches; deviation
PERF-02	False-reject rate (rejects confirmed good on manual review)	Manual review of rejects	Per batch	> 2.0%	> 3.0%	Production	L0 → L2 if persistent
PERF-03	Knapp re-challenge with the qualified defect kit	Kit with known reject probabilities	Quarterly and after any camera maintenance	Particle detection < 0.95	< 0.94 (validated lower bound)	Validation	L1 → L2
DRIFT-01	Image-embedding distance vs. training distribution	Automated	Daily	> 95th percentile of validation range for 2 days	> 99th percentile	Automation / data science	Check lighting, lens, supplier; L1 if unexplained
DRIFT-02	`undecided` rate	Automated	Per batch	> 0.6% (2× validated 0.3%)	> 1.5%	MS&T	L0 → L1
DRIFT-03	Units from a container lot or supplier not in scope	Batch record / ERP	Per batch	Any	Any	QA	Stop AI inspection for that lot; manual inspection; T9 or change control
DRIFT-04	Reject rate per class (p-chart)	Automated	Per batch	Western Electric rules	3σ breach	Production / QA	Treat as a process signal first; confirm precision holds
CAL-01	Share of accepted units with confidence 0.90–0.95	Automated	Weekly	+5 pp vs. validation	+10 pp	Data science	L0
HITL-01	Agreement of manual reviewers with AI rejects	Reject review log	Weekly	< 70% or > 99% sustained	—	QA	Low: fitness review. High: seeded-reject check
HITL-03	Seeded defects in reject review caught by manual reviewers	Seeded units	Quarterly	< operator qualification level	—	QA training	Retrain reviewers
OPS-06	Model and threshold hash check	Automated	Daily	Any mismatch	Any mismatch	Automation	Stop; investigate unauthorised change

Response example. Week 14: DRIFT-01 crosses the alert limit for 2 days and undecided rises to 0.7%. Investigation finds an LED panel at 82% of rated output after 11,000 hours. LED replaced under maintenance; PERF-03 re-challenge after maintenance passes (particle detection 0.97). No change to the model. Logged as a Level 0 event and carried to T10.

Appendix B — Site AI Monitoring Policy (reference)Policy reference

B.1 Scope. Every AI function in the AI inventory (vendor-delivered, built, or approved general-purpose tool) with a tier of 2 or 3 has a T8 plan. Tier 1 functions are reviewed at periodic review only.

B.2 Five metric families. Every Tier 2–3 plan covers all five, or records why a family does not apply:

Family	Watches	T8 section
1. Inputs / drift	Inputs still inside the validated sample space: distribution distance, OOD / undecided rate, out-of-scope sources, class prevalence	§3
2. Outputs / performance	The validated claim still holds: critical-class sensitivity, precision or false-reject rate, re-challenge, calibration	§2, §4
3. Human-in-the-loop	Reviewers still reviewing and AI still helping: acceptance (both tails), override and modification in both directions, seeded-error catch rate, edit distance / time-on-review	§5
4. System / vendor	Configuration is what was validated: model, prompt, index, threshold hashes; vendor version and deprecation date; probe-set check; latency; cost; degraded-mode activations	§6
5. Data integrity	Every AI-touched record attributable and complete: output stored as generated, model version, sources, reviewer action; no unattributed AI-originated fields; retention of AI context artifacts	§6 (OPS) and the record system

B.3 Minimums by tier.

Tier	Minimum	Frequency	Reference check (ground truth)
3 (GxP-critical)	All five families; at least one drift score, one OOD/undecided rate, one out-of-scope check; critical-class sensitivity and false-reject rate; calibration; acceptance and override in both directions; seeded-error catch rate; configuration hash; vendor version and deprecation; audit-trail completeness	Inputs and configuration daily or per batch; performance per batch where a reference exists; HITL weekly; calibration weekly	Per batch or campaign on the critical class; qualified re-challenge quarterly and after maintenance
2 (GxP, HITL)	Families 1, 3, 4, 5 and one performance proxy: undecided or out-of-scope rate; checker findings per item; acceptance and edit rate; seeded-error catch rate; configuration hash and vendor version; audit-trail completeness	Weekly; configuration daily	Quarterly sample audit against an adjudicated reference; seeded-error check quarterly
1 (non-GxP / authoring aid)	Inventory entry current; acceptable-use SOP compliance; vendor version; shadow-use detection	At periodic review	None required

B.4 Limits. Alert and action limits are derived, not copied. For a performance metric the action limit is the lower bound of the validated confidence interval (T7) and the alert limit sits between the point estimate and that bound. Limits guarding a High failure mode (T2) are tighter and escalate faster. For HITL metrics the action limit on the seeded-error catch rate is the measured human baseline. Drift scores, undecided rates, override rates and edit distances are charted (individuals/EWMA or p-charts) with run rules as the alert condition, and seasonal baselines where cycles exist. A limit change is a change to the monitoring-configuration CI.

B.5 Ownership, escalation and response ladder. Level 0 Investigate (monitoring owner; alert limit or run-rule violation; check process and data-path causes before suspecting the model). Level 1 Restrict to advisory (system owner + QA; action limit on a critical KPI or repeated unexplained alert; 100% human review of affected outputs). Level 2 Retrain / reconfigure (system owner + SME + QA; T9 if inside the envelope, else full change control; regression on the locked test set; PIR). Level 3 Retire / suspend (QA; critical error reached a GxP record or performance cannot be restored; revert to manual; impact assessment on past decisions). Escalation SLAs are set per tier in T8 Section 7. Every Level 1–3 event feeds T10 and re-triggers T2.

B.6 Tie-in to change control. Alert-limit breach → Event → Incident (Level 0). Action-limit breach → Problem (Level 1). Problem with a root cause in a configuration item → Change Request (Level 2), Standard inside T9, Normal outside it. Vendor deprecation notice → planned change. Monitoring configuration is a CI.

B.7 Alert-fatigue controls. (a) Tiering: only Tier 3 action-limit breaches page in real time; alert-limit breaches go to a daily digest; Tier 2 to a weekly review. (b) Suppression windows: bounded, with cause, approver and end date recorded; the metric is checked when the window closes; no open-ended suppression. (c) De-duplication: alerts sharing one physical cause are correlated into one Incident. (d) Usefulness review: per KPI, alerts raised / led to a finding / closed as false alarm, reported at every T10.

B.8 Monitoring the monitors. At each T10: re-derive limits from current validated performance and observed metric distributions; review false-alarm and missed-signal rates; confirm every High failure mode in the current T2 maps to a live KPI; confirm reference checks ran at the stated frequency; confirm intended use still matches actual use. Output: "limits confirmed" or a change to the monitoring-configuration CI.