Životni ciklus modela

Sažetak

Modeli se poboljšavaju iz odluka kliničara: što su zadržali, promijenili ili odbacili. Te se odluke izvoze bez identiteta, koriste za doradu malih modela, vrednuju na zaključanim skupovima i objavljuju samo kao potpisana izdanja koja odobravaju tri uloge, kroz sjenu, pilot i kliničku fazu.

Tehnička specifikacija vodi se na engleskom jeziku; naslovi i sažeci poglavlja prevedeni su na hrvatski.

The loop

Clinician decisions
accept · edit · reject
De-identified export
pairs · triples
Adapter training
QLoRA SFT → DPO
Evaluation
locked sets · sign-off

Clinical
monitored
Pilot
named cohort
Shadow
no UI exposure
Signed release
Ed25519 · 3 approvals

Connections (8)

Clinician decisions → De-identified export
De-identified export → Adapter training
Adapter training → Evaluation
Evaluation → Signed release
Signed release → Shadow
Shadow → Pilot
Pilot → Clinical
Clinical ⇢ Clinician decisions · feedback

Clinician decisions become training data; nothing trained reaches a clinician without evaluation, signing and staged exposure.

Training data

The training signal is the difference between what the model drafted and what the clinician finalised. scripts/export-clinical-dpo-dataset.ts produces, per adapter, preference pairs (the finalised text as chosen, the draft it replaced as rejected) and supervised examples, plus contrastive triples for the embedding model and premise–hypothesis–label triples for entailment.

Every example is scrubbed for PII twice, and anything the PHI-leak rule still flags is quarantined rather than exported.
Clinical and research data are physically separate. Imaging datasets live only in the research database, built from de-identified DICOM under written agreements, with blinded readers and adjudication.
No patient data trains a hosted provider’s model: Vertex AI calls run under the provider’s enterprise data terms, and training happens only on the self-hosted estate.

Training

SOAP structuring
Base
Qwen3-8B
Purpose
Notes from transcripts
Note critic
Base
Qwen3-8B
Purpose
Second read of a draft note
Clinical information extraction
Base
Qwen3-8B
Purpose
Structured values from text
Memory extraction
Base
Qwen3-8B
Purpose
Patient memory items
Clinical critic
Base
MedGemma 27B
Purpose
High-risk answer review
Radiograph verifier
Base
MedGemma 4B
Purpose
Imaging region-of-interest check

Adapter	Base	Purpose
SOAP structuring	Qwen3-8B	Notes from transcripts
Note critic	Qwen3-8B	Second read of a draft note
Clinical information extraction	Qwen3-8B	Structured values from text
Memory extraction	Qwen3-8B	Patient memory items
Clinical critic	MedGemma 27B	High-risk answer review
Radiograph verifier	MedGemma 4B	Imaging region-of-interest check

The recipe is QLoRA supervised fine-tuning followed by Direct Preference Optimisation on clinician edits (rank 16, alpha 32, β = 0.1). The objective:

L_DPO = −E_(x, y_w, y_l) [ log σ( β·log(π_θ(y_w|x) / π_ref(y_w|x)) − β·log(π_θ(y_l|x) / π_ref(y_l|x)) ) ]

y_w is the clinician-finalised text, y_l the draft it replaced, π_ref the supervised model.

The training workflow packages adapters with their hashes into a release bundle; scripts/train-clinical-llm-adapters.py validates the bundle, hashes every weight file and emits the release manifest that is then signed.

Evaluation

Model-evaluation harness
What it measures
Per-endpoint thresholds, rubrics, adjudication worksheets and a JSON schema for every evidence report; an upgrade-approval workflow
Locked evaluation sets
What it measures
Fixed cases a model must pass before an upgrade; the clinician-authored validation corpus is being written
Golden gate
What it measures
Embedding recall@5, entailment contradiction recall and accuracy — a release cannot be signed below threshold
Scribe, Copilot, concierge, treatment-plan and voice evaluations
What it measures
Dedicated runners; Scribe and Copilot evaluations have their own CI workflows
Dictation word error rate
What it measures
Recognition quality on clinical dictation
Perio recognition benchmark
What it measures
Recognition engines scored through the real perio command engine; runs in CI as a cut-over gate
Evidence benchmark
What it measures
Top-5 precision ≥ 95 %, recall ≥ 95 %, support-direction precision ≥ 98 %, at least two blinded reviewers with ≥ 80 % agreement

Instrument	What it measures
Model-evaluation harness	Per-endpoint thresholds, rubrics, adjudication worksheets and a JSON schema for every evidence report; an upgrade-approval workflow
Locked evaluation sets	Fixed cases a model must pass before an upgrade; the clinician-authored validation corpus is being written
Golden gate	Embedding recall@5, entailment contradiction recall and accuracy — a release cannot be signed below threshold
Scribe, Copilot, concierge, treatment-plan and voice evaluations	Dedicated runners; Scribe and Copilot evaluations have their own CI workflows
Dictation word error rate	Recognition quality on clinical dictation
Perio recognition benchmark	Recognition engines scored through the real perio command engine; runs in CI as a cut-over gate
Evidence benchmark	Top-5 precision ≥ 95 %, recall ≥ 95 %, support-direction precision ≥ 98 %, at least two blinded reviewers with ≥ 80 % agreement

Release and exposure

draft → candidate
Meaning
Built and registered; not runnable for clinicians
locked
Meaning
Weights, adapters, prompts and thresholds frozen; hashes recorded
validated
Meaning
Evaluation passed against locked sets
approved
Meaning
Signed (Ed25519) with machine-learning, quality and clinical approval
retired
Meaning
Withdrawn; history kept for traceability

Stage	Meaning
draft → candidate	Built and registered; not runnable for clinicians
locked	Weights, adapters, prompts and thresholds frozen; hashes recorded
validated	Evaluation passed against locked sets
approved	Signed (Ed25519) with machine-learning, quality and clinical approval
retired	Withdrawn; history kept for traceability

Imaging models follow the full state machine above, with exposure modes shadow (runs on real traffic, nothing shown), pilot (a named cohort) and clinical; activation is a recorded database action, never a configuration flag.
A significant change — new data, new threshold, new model — reopens the release under a written change-control procedure.
The clinical LLM worker loads only a signed release package and hashes every response, so what ran can be proven afterwards.

Monitoring after release

Every call is in the AI decision record: model actually used, prompt version, tokens, cost, latency and whether PII was detected.
Clinicians can report an AI output from the interface; reports flow into a platform review queue.
Post-market surveillance signals are defined in the QMS registries, and a live drift-monitoring plan in docs/model-evaluation/.
Shadow mode is used beyond models — turn routing, safety rules, contradiction checks and consent enforcement all ran or run in shadow before they enforce.

ProductionDe-identified export of clinician decisions
Built
Rolling outAdapter training and self-hosted deployment
In progress now
ProductionEvaluation harness, golden gate, fingerprint invalidation
Built; upgrade gate armed at clinical launch
ProductionSigned release control plane
Imaging and the clinical LLM worker
Rolling outClinician-authored validation corpora
Being authored

SOURCE OF TRUTH

scripts/export-clinical-dpo-dataset.ts
scripts/train-clinical-llm-adapters.py
scripts/eval-golden.ts
scripts/modelEvalRun.ts
docs/model-evaluation/
server/config/clinicalEvalGate.ts
tests/ai/modelEvalFingerprintScope.test.ts
docs/adr/0006-imaging-model-release-boundary.md