AI
Charles Trinh — AI Evaluation and Benchmarking | Las Vegas, NV · Las Vegas, Nevada · Henderson · Clark County, Nevada · North Las Vegas, Nevada

A benchmark is only as good as the task it measures. Most healthcare AI evaluations measure a proxy for the task, on a sample from the wrong distribution, and call it a result. I build evaluations that measure the task.
The task, the proxy and the gap
Every evaluation begins by writing down the task. What decision does the model support, who makes it, and what does a wrong answer cost. From that I derive the evaluation task, which is usually narrower and more specific than what the model does in the abstract. An evaluation of a triage assistant should measure whether it routes each real patient's encounter to the correct level of care, on the population the model will see, not whether it produces well-formed triage text. A structure check is a useful filter; it is not the task.
The gap between the benchmark and the task is where evaluations fail. I have seen models scored on a synthetic test set the vendor published, deployed on real encounters where the input distribution differed in ways the test set never contained — different note lengths, different vocabulary, different patient demographics. The number was high; the model underperformed. Every benchmark I build is constructed from a sample from the real production distribution, even if that sample is smaller than the vendor's benchmark.
Task specification is a written document, not an implicit understanding. It names the input, the required output, the scoring rule and the review procedure. Where the requirement is clinical, the clinical team writes the input and output definitions, and I write the scoring and the review procedure. A task with no written specification produces disagreement between reviewers, and that disagreement usually inflates or deflates the scores depending on which reviewer started first.
Sampling that matches the production mix
Sampling is a decision with consequences for every number the evaluation produces. A random sample from the population understates the conditions where the model fails most, because those conditions are rare. Stratified sampling over the conditions where failure would harm the most — the longest encounters, the emergency admissions, the subgroups where the risk register says performance is uncertain — produces a clearer picture. The trade-off is that the aggregate score from a stratified sample does not reflect the production aggregate, so I report the stratified result and a reweighted aggregate.
The sample size is set from a power calculation, not from an intuition about how much data is enough. Where two models or two versions are compared, the sample needs to distinguish the differences that matter to the operation, not the smallest difference the metrics can resolve at all. A benchmark that says “model A is 2.1 points better than model B” with a confidence interval of +/− 3 points has said that A and B are comparable, whether that was the intended message or not.
Samples age. A sample taken last year on last year's population mix, provider operations and patient population is the right benchmark for last year's model, not this year's. I keep the harness and rerun the benchmark on a new sample on a schedule. The scores are not comparable across samples unless the sample construction is identical, and I fix the construction so the comparison is valid.
Reference labels and the reviewers' disagreement
Labels are the highest-cost part of an evaluation and where it most often goes wrong. A label is the answer a clinician would give for the same task, produced without seeing the model's output. If the label is produced after the model output, reviewers anchor on it, and the evaluation measures agreement with the reviewer rather than accuracy. I label first, I hide the model's output from the labelers, and I sign the reference before scoring.
Inter-rater agreement sets the ceiling: if two clinicians agree ninety percent of the time, the task itself is ambiguous beyond that point and no model can be judged above it. I report kappa or equivalent beside every model score. Low agreement means the label definition needs work before scoring begins; high agreement means score differences are more likely to reflect the model rather than the label noise around it.
The label set should include the negative cases and the boundary cases. A positive-only evaluation measures a model's ability to find positives, which is a different and easier task than telling positives from negatives. I deliberately include cases where a clinician decided not to act and cases that sit right on the boundary of the definition. Models that perform well on positive cases only can collapse on the full task, and it is the negative and boundary cases that expose the collapse.
Stratified reporting and the uncertainty number
An aggregate score is a summary and always hides something. The stratified report shows performance for each population and each condition the risk register identifies. Where a stratum is too small for a reliable estimate, I say so and report it with a wide confidence interval or an explicit note, rather than as a point estimate. Silent omission of a small stratum is the most common deception in benchmarking, and it is often not intentional.
Confidence intervals are required on every score, and in most healthcare evaluation contexts so are paired nonparametric tests for comparing models — a bootstrap on the same sample is standard. I compute these routinely and include them in the report. A model difference under 1 point on a test set of 200 cases is not a finding, and presenting it as one is the main way a benchmark is allowed to mislead a go/no-go committee without anyone lying.
Metrics matter. Accuracy, sensitivity, specificity, positive predictive value, negative predictive value and the confusion matrix are all reported for classification tasks. Where the task is generation, the metrics include content overlap against the reference, but they also include human review outcomes and inter-rater agreement on those outcomes. F1 and BLEU scores — or their modern equivalents — measure a proxy; they should be reported and not confused with the result.
Benchmark artefacts and reproducibility
Every benchmark I build is designed to be rerun on the same sample. The sample, the labels, the labeler identities and the model outputs are stored with a hash, so that a rerun on the same input produces verifiably the same result. That makes the benchmark an artefact with a version, and each reported score points at a specific version. An organisation cannot compare this year to last year without it.
The harness has a scoring script, a sample builder, a label collection tool and a report generator. Each is separately versioned, each has a test and each runs in your pipeline. When a new task is added, the harness extends rather than re-implements, and the reuse keeps the cost of the second evaluation much lower than the first. On a program with many models, that cost difference matters for whether the programme proceeds at all.
Reproducibility includes reproducibility across machines. A scored result that depends on a Python version, a library version or a GPU that no longer exists is not reproducible in a meaningful sense. I pin the environment and record it in the artefact metadata. Rebuilding the environment for a rerun on a newer stack is checked, not assumed, because a result that comes out slightly different is the most difficult case to diagnose.
Reporting to a committee, honestly
Most evaluations end in a report and a decision. The report I deliver has a one-page summary with the task, the measurement, the stratified results, the uncertainty, and a clear statement of what the measurement does not cover. The last part is the one that anchors the committee's expectations. A report that measures the model but does not say what was not measured invites the committee to assume coverage that does not exist.
The summary also states the decision the measurement supports and the decision it does not. A benchmark of a classification model measures the classification task; it does not measure whether the workflow built around it is safe, useful or well-designed. If the committee needs the workflow decision, that is a different evaluation, and I say so before the report is read rather than after the deployment.
The benchmark I build is the reference implementation for the organisation's own evaluation practice. Once one is running in a repository with a sample, a harness and a report, other tasks can be modelled on it without starting from scratch. The programme compounds, and the value of the second and third evaluations is usually much greater than the value of the first.
Where this practice fits
This is the practice behind the other nine pages in this batch. It is the harness used in
clinical-documentation-ai, ambient-scribe-evaluation,
risk-adjustment-ai and fraud-waste-abuse-analytics, with a task
specification in place of a general evaluation. The specificity is the point. An evaluation
of a risk adjustment model, an FWA pattern and a clinical documentation model share the
harness; they are not the same evaluation, and folding them into one programme usually
produces a score nobody trusts.
Where a standard mandates the evaluation, be it a survey, an accreditation review or a payer requirement, the harness reports against the standard's expectations explicitly. I have engineered the ONC Health IT Certification test suite, all 317 of 317 (g)(10) subtests passing on a mock stack, and the same approach applies to model evaluation harnesses: the criteria are written before the evaluation begins and cited in the report. That style is repeatable and auditable.
My role is engineering, and the task definitions and clinical judgements come from clinicians and clinical leadership. The result is a harness, a sample, a report and a record, runnable in your repository. It is a working practice, not a standards body or a certification, and I say so clearly because evaluation work is one of the places where an inflated claim is most tempting and least useful.
Proof
- ONC (g)(10) self-test PASS, 317 of 317 subtests, April 2026 — public suite URL
- 67 public hostnames, 51 serving, one AWS account
- 104 clearinghouse enrolments across 53 jurisdictions
- USCDI v7 proposal — Patient Digital Identity Credential — listed at isp.healthit.gov
- SAM.gov UEI MVKQW9DKKGB8 · CAGE 20KM5 · NPI 1194645564
- AWS Business Associate Addendum active since 18 August 2026
Where this work is delivered
Engagements run from Las Vegas and reach across the valley: clients in Las Vegas, NV; Las Vegas, Nevada; Henderson; Clark County, Nevada; and North Las Vegas, Nevada — plus remote delivery nationwide.
How an engagement runs
1. Trace one real case
Follow a single patient or claim end to end with a timestamp at every hop, before any document is written.
2. Written findings
A findings list naming the defect, the system, the owner and the date.
3. Monthly review
Review the list monthly and check whether the dates held.
Frequently asked
What is this service?
This page describes charles trinh — ai evaluation and benchmarking — the scope, the failure modes it prevents and how the work is measured. It sits in the AI and Automation category at trinh.io.
Where is the work delivered?
Engagements run from Las Vegas, Nevada across the valley — Las Vegas, Henderson, North Las Vegas, Spring Valley, Paradise, Winchester, Sunrise Manor, Enterprise, Summerlin, Clark County — and remotely anywhere in the United States. On-site work in the Las Vegas valley, remote delivery elsewhere.
What does it cost?
Standard rate $195 per hour, committed $165 per hour at twenty hours or more per week for three months or more; $1,600 standard day, $1,350 committed day. Scoped in writing before work starts, fixed fee where the deliverable is defined.
Who is Charles Trinh?
A technology architect in Las Vegas, Nevada, working in healthcare identity, consent and interoperability. Not a physician, not a medical practice, and not affiliated with any other Charles Trinh.
How does an engagement start?
With one trace of a real case and a written findings list naming the defect, the system, the owner and the date. The first call is twenty minutes and free: 702-559-3489.
Engagement
Scoped in writing before work starts; fixed fee where the deliverable is defined. First call is 20 minutes and free. 702-559-3489 · charles@trinh.io
Talk to Charles
- Phone: 702-559-3489
- Email: charles@trinh.io
- Based in Las Vegas, Nevada · working remotely across the United States
Record
Profile
Healthcare technology architect working where federal regulation becomes production infrastructure. Built and operates a FHIR R4 clinical platform — SMART App Launch authorization server, patient and provider portals, SQLite — on a hardened single-host AWS estate of 67 public hostnames. The platform returned a self-test PASS on the ONC (g)(10) Standardized API test suite in April 2026, 317 of 317 subtests, publicly verifiable. Designed a consent architecture in which a permissioned ledger records patient grant, refusal and revocation as immutable events while all clinical data stays off-chain and encrypted per patient — no PHI on the ledger by design. Filed a proposed USCDI v7 data element with ONC on verifiable patient identity. Translates the Federal Register, HHS/CMS guidance and HITRUST requirements into architecture, controls and audit evidence.
Selected engineering scale
- Self-managed estate of 67 public hostnames, 51 serving, on one AWS account.
- FHIR R4 API with a SMART App Launch authorization server; patient and provider portals; an AI service layer; Hyperledger Fabric peer, orderer and CA; IPFS; a 337-table SQLite clinical store.
- Two permissioned distributed ledgers in production, one operating a consent registry that records grant, refusal and revocation with no PHI on chain.
- X12 005010 pipelines across eight transaction sets.
- Designed, built, deployed and operated end to end by one person.
Credentials and standards work
- Inferno (g)(10) self-test PASS, 317 of 317 subtests, April 2026 — the ONC Standardized API test suite. Standards validated: US Core 6.1.0, USCDI v3, SMART App Launch 2.0.0, Bulk Data 2.0.0. Public suite URL.
- USCDI v7 public comment submission, April 2026 — proposed Patient Digital Identity Credential under the Provenance data class, listed publicly on the ONC Interoperability Standards Platform.
- SAM.gov UEI MVKQW9DKKGB8 · CAGE 20KM5 · NPI 1194645564. AWS Business Associate Addendum active 18 August 2026.
Technical environment
AWS us-east-1 (compute, storage, messaging, SMS, email, voice) · Linux and systemd hardening · nginx · Fabric and EVM ledgers · FHIR R4 · X12 005010 · HL7 · SQLite · self-hosted mail, drive, search and monitoring · Git-based change control and CI release gates.