AI

Charles Trinh — Human-in-the-Loop Review Design | Las Vegas, NV · Las Vegas, Nevada · Henderson · Clark County, Nevada · North Las Vegas, Nevada

A review screen with source evidence, recommendation and decision controls
Charles Trinh — Human-in-the-Loop Review Design

A review point is where an AI output meets a person who can reject it. The design decides whether that person actually reviews or just clicks. I build review points that catch what models miss and measure how well they do.

What review actually means

“Human-in-the-loop” can mean anything from rubber-stamping an output to a clinician who independently recreates it. The first is liability-shifting, not review. I define review per workflow as a decision with authority, a named reviewer who may reject, evidence for and against the output, and a record of the decision. Absent all four, a click moves consequences from the model to a reviewer who rarely has the information to bear them.

The failure mode in review design is the accept-everything button. It appears in almost every system I audit in the first hour. A button labelled “accept” that submits the output without opening the evidence panel, or one that clears when the reviewer clicks twice in quick succession, is a defect. The fix is not to hide the button behind a confirmation dialog, which reviewers quickly learn to click through. The fix is to make the evidence visible in the same screen as the decision and to give rejection the same visual weight as acceptance.

The review authority question is a legal and clinical one, not mine. The clinical leadership should define for each task whether the reviewer must independently verify, review the evidence and make a judgement, or spot-check the output. That definition becomes the design, and I write it into the screen and the record so the reviewer knows what is expected of them. Where it is not written, reviewers infer it from the interface, and usually the interface suggests the reviewer does not need to read much.

Review capacity and routing

Review capacity is finite and normally over-estimated. Twenty cases an hour in the morning is not twenty in the afternoon, and not twenty when the cases are hard. I estimate capacity from observed data across a working day and target sixty to seventy percent of it, not the peak. A queue running at ninety-five percent of capacity slows every case and grows less careful as it grows longer.

The routing rule decides which output reaches which reviewer. For a documentation workflow it may depend on note type, specialty and reviewer training; for a claims integrity flag, on dollar amount and pattern class. I write the rule as an explicit specification, then measure actual queue flow against it. Where the queues differ in shape from the specification, the rule has an implementation defect and reviewers usually notice it first.

Routing to the wrong reviewer is a common failure that looks like poor model performance until the queue is disaggregated. A reviewer with no experience in the code set will reject a valid flag they do not understand, and that rejection feeds back as model imprecision. I measure acceptance by reviewer and by specialty and treat reviewer-level variation as a training signal rather than a model signal, until training has been done and rejected.

Interface design that makes review real

The interface should put the evidence in front of the reviewer at the moment of decision. For a clinical summary, that is the relevant source chart section side by side with the model's output. For a coding flag, that is the claim, the evidence chain and the code definition. For an FWA pattern, it is the specific claims and the benchmark. The evidence must load with the decision screen, not in a second application the reviewer opens only when worried. When evidence requires a second application, most reviewers will not open it.

Rejection must be easy to do and must carry a reason. A rejection that is harder to perform than acceptance inverts the incentives, and no amount of guidance about professional responsibility overcomes a design that favours the fast path. I design the rejection control to accept a structured reason in one step and to remember the reason class per reviewer and per task. Those reasons become the training data for the next version of the model and for the review guidelines.

The acceptance decision also needs a moment of friction: not a modal that can be clicked through, but a visual state that changes based on how the reviewer has interacted. If the reviewer has not opened the evidence, the accept control should say so. If the reviewer has opened the evidence and made no modification, the accept is faster. Over time this pattern gives reviewers a real choice between fast acceptance and careful review, and it makes careful review the affordance that the interface supports rather than one it works around.

Measuring review quality

Review quality is measured against a small sample independently re-reviewed by a second reviewer. That second review produces a rate at which reviewers accept outputs the second reviewer would reject, and a rate at which reviewers reject outputs the second reviewer would accept. Both are costs. The first is the acceptance of bad output, which is the safety cost; the second is the wasting of a reviewer's time and of the model's good output. Both belong in the programme's measurement, and both are invisible to the reviewer being measured.

Agreement between reviewers on the same output is a measure of the task's ambiguity and of the guideline's clarity. Where agreement is high, the review is repeatable. Where it is low, I look at the guideline first. A low-agreement task with a detailed guideline often means the guideline has a section that reviewers interpret differently, and the fix is to change the guideline rather than to train the model more. That fix is cheap and frequently not tried.

Time on task is another signal. A reviewer who takes twenty seconds per case is not reviewing carefully, and one who takes twenty minutes is probably not using the interface the way it was designed. Measuring actual time on task, and the pattern of which evidence was opened, tells the clinician leadership whether review is happening. Where it is not, the design is the suspect, and my first move is to look at the review screen, not the reviewers.

Fatigue, coverage and the honest service target

Fatigue is real and should be planned for. A reviewer misses more in the fifth hour than the first, and the missed-judgement rate by time of day is measurable. I stagger the load so hard cases arrive fresher and set a target a person can hold carefully across a shift. Under-planned fatigue is a hidden cause of safety events in AI workflows and is almost never instrumented.

Service targets should match the capacity of the team providing coverage. A twenty-four hour acceptance target with one reviewer, no night coverage and no weekend cover is not a target; it is an aspiration. Where the target is contractual, the honest reply is to describe the coverage that would meet it, the cost of that coverage and the risk that comes with running below it. Where the target is internal, I set the target so that it can be met by the team as staffed.

Provider turnover is a coverage risk that planners underestimate. A reviewer leaving the programme removes their training, their specialty knowledge and their history of the queue. I keep the review guidelines and the recorded decisions as training material and build the programme's knowledge on the artefacts, not on individuals. That is partly a staffing argument and partly a safety one, and both matter.

The record of every decision

Every review decision should be recorded as a structured event: the output reviewed, the evidence presented, the reviewer, the time on task, the decision, the reason, and any modification made. That record supports the clinical or compliance response to a safety event, the audit trail for a program-integrity review, and the model evaluation's feedback loop. Building it from the start is cheaper and more complete than adding it after the first incident, and it is the single most important artefact a review programme can produce.

The record also makes the programme auditable. A reviewer whose acceptance rate departs from their peers can be traced to a training gap, a routing mismatch, a model-quality difference or something else. A model whose acceptance rate departs from its offline score can be traced to the strata where the divergence sits. Neither comparison exists without a structured decision event, and both are what a clinical leader asks for first.

Who owns the record is a governance decision. In a clinical workflow, the reviewer's signature and the record of the decision belong with the chart. In an administrative workflow, the record belongs with the case file. In either case, the record should be retrievable by someone with the right role without a developer, and it should be retained for the length of the corresponding record's retention. Those rules are set by compliance and I implement them.

The pattern reused across the batch

This design is used in clinical-documentation-ai for the clinician's edit of a draft, in risk-adjustment-ai for the capture of a suspect condition, in fraud-waste-abuse-analytics for the review of a flag and in the ai-vendor-diligence review of a vendor's control claims. The pieces are the same, the review authority differs and the record context differs. Reuse of the pattern is why the second workflow costs much less than the first, and the record format is the main reason to start it early.

My background is engineering; judgement about clinical or compliance authority belongs to your clinical and compliance leadership. The interface and its measurement are mine, and a working review screen with a measured accept-reject rate is what makes an AI workflow safe to run rather than merely safe to demonstrate. The gap between those two states is almost always in the review screen, and that is the part I own.

Engagements start with a review of the existing screen, a measurement of the actual decision rate and an interface rebuild where the interface does not support the review. The rebuild usually takes two to four weeks per workflow, and the measurement continues for the life of the programme. The record outlives the reviewer and outlives the model, and that is why I build it to last.

Proof

  • ONC (g)(10) self-test PASS, 317 of 317 subtests, April 2026 — public suite URL
  • 67 public hostnames, 51 serving, one AWS account
  • 104 clearinghouse enrolments across 53 jurisdictions
  • USCDI v7 proposal — Patient Digital Identity Credential — listed at isp.healthit.gov
  • SAM.gov UEI MVKQW9DKKGB8 · CAGE 20KM5 · NPI 1194645564
  • AWS Business Associate Addendum active since 18 August 2026

Where this work is delivered

Engagements run from Las Vegas and reach across the valley: clients in Las Vegas, NV; Las Vegas, Nevada; Henderson; Clark County, Nevada; and North Las Vegas, Nevada — plus remote delivery nationwide.

How an engagement runs

1. Trace one real case

Follow a single patient or claim end to end with a timestamp at every hop, before any document is written.

2. Written findings

A findings list naming the defect, the system, the owner and the date.

3. Monthly review

Review the list monthly and check whether the dates held.

Frequently asked

What is this service?

This page describes charles trinh — human-in-the-loop review design — the scope, the failure modes it prevents and how the work is measured. It sits in the Security and Compliance category at trinh.io.

Where is the work delivered?

Engagements run from Las Vegas, Nevada across the valley — Las Vegas, Henderson, North Las Vegas, Spring Valley, Paradise, Winchester, Sunrise Manor, Enterprise, Summerlin, Clark County — and remotely anywhere in the United States. On-site work in the Las Vegas valley, remote delivery elsewhere.

What does it cost?

Standard rate $195 per hour, committed $165 per hour at twenty hours or more per week for three months or more; $1,600 standard day, $1,350 committed day. Scoped in writing before work starts, fixed fee where the deliverable is defined.

Who is Charles Trinh?

A technology architect in Las Vegas, Nevada, working in healthcare identity, consent and interoperability. Not a physician, not a medical practice, and not affiliated with any other Charles Trinh.

How does an engagement start?

With one trace of a real case and a written findings list naming the defect, the system, the owner and the date. The first call is twenty minutes and free: 702-559-3489.

Engagement

Scoped in writing before work starts; fixed fee where the deliverable is defined. First call is 20 minutes and free. 702-559-3489 · charles@trinh.io

Talk to Charles

Record

Profile

Healthcare technology architect working where federal regulation becomes production infrastructure. Built and operates a FHIR R4 clinical platform — SMART App Launch authorization server, patient and provider portals, SQLite — on a hardened single-host AWS estate of 67 public hostnames. The platform returned a self-test PASS on the ONC (g)(10) Standardized API test suite in April 2026, 317 of 317 subtests, publicly verifiable. Designed a consent architecture in which a permissioned ledger records patient grant, refusal and revocation as immutable events while all clinical data stays off-chain and encrypted per patient — no PHI on the ledger by design. Filed a proposed USCDI v7 data element with ONC on verifiable patient identity. Translates the Federal Register, HHS/CMS guidance and HITRUST requirements into architecture, controls and audit evidence.

Selected engineering scale

  • Self-managed estate of 67 public hostnames, 51 serving, on one AWS account.
  • FHIR R4 API with a SMART App Launch authorization server; patient and provider portals; an AI service layer; Hyperledger Fabric peer, orderer and CA; IPFS; a 337-table SQLite clinical store.
  • Two permissioned distributed ledgers in production, one operating a consent registry that records grant, refusal and revocation with no PHI on chain.
  • X12 005010 pipelines across eight transaction sets.
  • Designed, built, deployed and operated end to end by one person.

Credentials and standards work

  • Inferno (g)(10) self-test PASS, 317 of 317 subtests, April 2026 — the ONC Standardized API test suite. Standards validated: US Core 6.1.0, USCDI v3, SMART App Launch 2.0.0, Bulk Data 2.0.0. Public suite URL.
  • USCDI v7 public comment submission, April 2026 — proposed Patient Digital Identity Credential under the Provenance data class, listed publicly on the ONC Interoperability Standards Platform.
  • SAM.gov UEI MVKQW9DKKGB8 · CAGE 20KM5 · NPI 1194645564. AWS Business Associate Addendum active 18 August 2026.

Technical environment

AWS us-east-1 (compute, storage, messaging, SMS, email, voice) · Linux and systemd hardening · nginx · Fabric and EVM ledgers · FHIR R4 · X12 005010 · HL7 · SQLite · self-hosted mail, drive, search and monitoring · Git-based change control and CI release gates.

TRINH — Charles Trinh, technology architect
TRINH

Charles Trinh

Fractional CTO/CIO and Technology Architect — Healthcare Data, Identity & Regulated Infrastructure · Las Vegas, NV

Las Vegas, Nevada · US citizen · Fractional and project engagements

(702) 559-3489 Email Charles Trinh