Charles Trinh · Notes · Health data

Why "fail small" should be the test for health data

Every health data integration eventually breaks. The only question worth asking in procurement is how much breaks with it.

Conformance testing answers a narrower question: can the system do the thing once, in a sandbox, under observation, with someone watching the log? It says nothing about what happens at three in the morning when a clearinghouse returns a batch of 999 rejects, when a consent is revoked halfway through a claim run, or when one host in a sixty-seven-host estate develops a bad disk. Those are the moments that produce a breach notification, a payer audit finding, or a headline — not the happy path that earned the badge.

So the test I apply to healthcare architecture is containment: when this component fails, what is the largest thing it can take with it?

Four rules that follow from it

No PHI where the blast radius is largest. In a system I built, consent events live on a permissioned ledger and clinical data does not. Grant, refusal and revocation are recorded on-chain as immutable events; the chart itself stays off-chain, encrypted per patient. That means ledger compromise is not a PHI breach. It is a design decision made once, and it decides the size of every future incident.

A revocation is an event, not a deletion. Deleting a consent row leaves no evidence that a refusal ever existed, which is precisely what an auditor asks to see. Recording it as a state transition preserves the history and lets downstream systems be rebuilt to match. If you cannot reconstruct who could see what on a given date, you do not have a consent architecture, you have a settings page.

Transaction sets fail independently. Eight X12 005010 sets run through the estate. A failed eligibility check must not block claims submission, and a stalled remittance batch must not exhaust the retry budget of the queue behind it. Separate queues, separate retry ceilings, separate alarms. The failure of one should be visible and boring.

Authentication fails closed; availability fails open on non-PHI surfaces. A consent check that errors must deny. A status page that errors should degrade to static, not vanish — the difference between "we cannot tell you" and "we are down" is the whole communications posture during an incident.

Then test it, deliberately

Three drills, run quarterly. Revoke one consent mid-run and confirm no further PHI leaves the system on the next request — not the next day, the next request. Take one of the fifty-one serving hosts out of rotation and confirm the other fifty keep serving with no manual intervention. Feed the pipeline a malformed 837 and confirm the batch quarantines with a readable rejection rather than the worker stalling and the queue silently aging.

None of those require a red team. They require that someone designed for the failure before it happened.

Which brings it back to procurement

When you evaluate a vendor, ask the one question the demo cannot survive: the last time your system had an incident, how many patients were in scope? An answer in the dozens is an architecture. An answer of "all of them" is a system designed to work, not designed to fail.

Failing small is not resilience theatre. It is the difference between an incident and a reportable breach — and it is decided in the architecture, years before the incident.

Charles Trinh is a technology architect in Las Vegas who builds FHIR and X12 infrastructure for providers and payers. The views are his own.

Charles Trinh · Las Vegas, Nevada · trinh.io · press kit
TRINH — Charles Trinh, technology architect
TRINH

Charles Trinh

Fractional CTO/CIO and Technology Architect — Healthcare Data, Identity & Regulated Infrastructure · Las Vegas, NV

Las Vegas, Nevada · US citizen · Fractional and project engagements

(702) 559-3489 Email Charles Trinh