DATA

Charles Trinh — Healthcare ETL and Pipeline Engineering | Las Vegas, NV · Las Vegas, Nevada · Henderson · Clark County, Nevada · North Las Vegas, Nevada

Pipeline run history showing staged loads, contracts and reconciliation checks per source
Charles Trinh — Healthcare ETL and Pipeline Engineering

A pipeline that fails loudly at 02:00 is better than one that completes quietly and loads yesterday's file twice. I build healthcare pipelines where silent success is treated as a defect.

Silent success is the failure mode

The costliest pipeline failures in healthcare I have seen were green in the scheduler. The FHIR export job completed, the row count matched, and the data was yesterday's file because the upstream vendor sent it again with no changes. The quality dashboard read success. The next morning an analyst noticed patients vanishing, three days after the first disappearance. A pipeline that treats silent success as acceptable fails at a slower speed than one that crashes.

I build pipelines with explicit contracts for every source. Each run declares the expected source window, the expected row count band, a checksum over the input, and the identities of the records included. On any breach the run fails with a descriptive error and the scheduler turns it red. The contract file is versioned alongside the pipeline and is the first thing I write for a new source.

Idempotency is the second requirement. Every step can be re-run on the same input and produces the same output, or a recorded no-op if the input already loaded. I achieve this with a natural key per source and a merge on that key, not with append-and-dedupe. A pipeline that cannot be retried safely is not a pipeline; it is a scheduled job that someone else has to clean up after.

Landing, staging and conformed layers

I split the pipeline into three layers: landing, staging, conformed. Landing is a write-once archive of the source's bytes with a manifest recording the arrival time, hash, and sender. Staging parses, types and normalises, and produces a row per source record. Conformed applies the identity rules and the cross-source joins that produce the warehouse's shape. Each layer is separately rebuildable, and none of them is allowed to write back into the layer above it, so a bug in one layer cannot corrupt the evidence the lower layers preserve.

The layers are physically separate storages where possible. Staging tables are dropped and rebuilt from landing on demand; conformed tables are incrementally maintained with the source keys visible. When a parsing bug is found, the fix is applied to staging and conformed is rebuilt from staging without returning to the source. A rebuild costs minutes rather than a vendor request, which is the difference between a fix today and a fix next week. I measure the rebuild time for every source, because that rebuild is the recovery path the runbook depends on.

For HL7 v2 sources in particular, the parser is deterministic and strict: an unexpected segment, a missing required field, or an unsupported OBX data type fails rather than being silently skipped. The HL7 v2 interfaces work covers the interface design. In the pipeline the rule is that parser fail is visible, and the failure includes the message control ID so the source can be traced to a single message.

Late and out-of-order data

Healthcare data arrives out of order as a rule rather than an exception. Lab results within hours, encounter end notifications within a day, provider notes often within days. The pipeline handles three classes: on-time, late within a defined window, and beyond the window. On-time flows through the normal path. Late flows through staging with the period reassigned to the source-effective date. Beyond-window records are quarantined with an operator notified, because they often indicate a source problem.

Period reassignment is essential. If a lab result dated in March arrives in May and is attributed to May, every month-on-month report is wrong from that point onwards. I write the reassignment into the staging contract with a per-source tolerance (for example, seven days). The tolerance is measured against the source's actual lag and recorded in the contract, not guessed. When that lag shifts, the contract flags it even if no record is out of order, because a source that used to arrive in hours and now takes days is telling the organisation something.

Reports that read from the warehouse carry a current-as-of line read from the pipeline's own run records. The line breaks when a source is stale rather than inaccuracy in the rows. The reader can see stale in time to act on it, and the next scheduled correction is on the record. When a source recovers, the next run recomputes the period and the current-as-of line moves forward; the runbook says explicitly which pages to check for the pattern of stale-then-recovered, so that a recovery does not read as a spike in the trend.

The bulk export path

Many modern sources now expose FHIR bulk data. For those, the pipeline needs to manage asynchronous job start, polling, partial file arrival, and final completion. The bulk FHIR export work covers this. The pipeline rules are the same: no load begins until the export manifest says the job is complete, and each file is checksummed before it is loaded. A partial export loaded as complete is the largest single source of duplicate and contradictory rows I encounter.

For sources that ship a full export each time rather than a delta, the pipeline must be able to detect which records changed and load only the changed ones. I achieve this by hashing the resource and comparing to the previous import, so unchanged pages skip the upstream transforms entirely. On a million-resource export, this takes the daily pipeline from hours to minutes. The hash table itself is part of landing, so a reload can be reconstructed without re-requesting the export.

Rate limits matter for sources that expose a public API. The pipeline has a single request queue per source and a budget per hour, and the failure mode is a back-off that extends the run. Rate-limit errors do not close the run; they extend it. A pipeline that gives up after three failures on a source with a per-hour limit spends the rest of the day unloaded and no human is told why, which is a failure that looks like a slow source.

Testing and the CI path

Every pipeline has a test harness with fixture sources. Fixtures are real samples, de-identified, and versioned, covering the normal path, known edge cases, malformed records, and the late-arrival case. A change to the pipeline runs the harness before merging. The harness is fast and runs in the local development container, so the author learns about regressions in minutes rather than in production.

Integration tests run on a staging environment against a live source sample. The staging environment is a full copy of the schema, but not full data volume; it is a smoke test of the contract and the deployment. The final test runs against the source in production with the run contracts set to alert-on-violation rather than block, for a period of a week, before the pipeline is switched to enforcing mode.

Deployments are single-step and the deployment artefact includes the pipeline code, the contract files and the tests. A rollback restores the previous version with the previous contracts. A rollback that does not restore the contracts is a common defect; the contracts changed with the code and the rollback puts back behaviour that the new contract no longer describes., and the flag is visible in the same run record that produces the reports.

Operations and incident response

Production pipelines have a runbook: what to check when the scheduler says green but the analyst says the numbers are wrong, how to trace a record from source to warehouse, how to recover a source that went silent, how to replay from landing, how to quarantine a source. The runbook names a primary and a secondary operator and their hours. I write it with the operators, not for them; a runbook written by the person who has never carried the pager tends to miss the first five minutes of an incident.

Alerts fire on contract violations, source silence, and late runs beyond a tolerance. The tolerances are set against the source's own behaviour and measured rather than assumed. A source that sends at 03:00 to 04:00 daily alerts at 04:30, not at 02:00 because it has not sent yet. The alert route is the operator's phone. Alerts that fire into an unmonitored email list are the equivalent of no alert.

Post-incident reviews are written for every failure with the record of what the contract caught and what it missed. The pipeline is updated so that the next failure is caught earlier, if possible before the source reaches production. Where the pipeline is the right tool and the source is the wrong one, I raise the source issue upstream and log the workaround. The balance to strike is that failures are visible enough that the right team is paged, and the record is enough that a fix is possible without the original author present.

Proof

  • ONC (g)(10) self-test PASS, 317 of 317 subtests, April 2026 — public suite URL
  • 67 public hostnames, 51 serving, one AWS account
  • 104 clearinghouse enrolments across 53 jurisdictions
  • USCDI v7 proposal — Patient Digital Identity Credential — listed at isp.healthit.gov
  • SAM.gov UEI MVKQW9DKKGB8 · CAGE 20KM5 · NPI 1194645564
  • AWS Business Associate Addendum active since 18 August 2026

Where this work is delivered

Engagements run from Las Vegas and reach across the valley: clients in Las Vegas, NV; Las Vegas, Nevada; Henderson; Clark County, Nevada; and North Las Vegas, Nevada — plus remote delivery nationwide.

How an engagement runs

1. Trace one real case

Follow a single patient or claim end to end with a timestamp at every hop, before any document is written.

2. Written findings

A findings list naming the defect, the system, the owner and the date.

3. Monthly review

Review the list monthly and check whether the dates held.

Frequently asked

What is this service?

This page describes charles trinh — healthcare etl and pipeline engineering — the scope, the failure modes it prevents and how the work is measured. It sits in the Security and Compliance category at trinh.io.

Where is the work delivered?

Engagements run from Las Vegas, Nevada across the valley — Las Vegas, Henderson, North Las Vegas, Spring Valley, Paradise, Winchester, Sunrise Manor, Enterprise, Summerlin, Clark County — and remotely anywhere in the United States. On-site work in the Las Vegas valley, remote delivery elsewhere.

What does it cost?

Standard rate $195 per hour, committed $165 per hour at twenty hours or more per week for three months or more; $1,600 standard day, $1,350 committed day. Scoped in writing before work starts, fixed fee where the deliverable is defined.

Who is Charles Trinh?

A technology architect in Las Vegas, Nevada, working in healthcare identity, consent and interoperability. Not a physician, not a medical practice, and not affiliated with any other Charles Trinh.

How does an engagement start?

With one trace of a real case and a written findings list naming the defect, the system, the owner and the date. The first call is twenty minutes and free: 702-559-3489.

Engagement

Scoped in writing before work starts; fixed fee where the deliverable is defined. First call is 20 minutes and free. 702-559-3489 · charles@trinh.io

Talk to Charles

Record

Profile

Healthcare technology architect working where federal regulation becomes production infrastructure. Built and operates a FHIR R4 clinical platform — SMART App Launch authorization server, patient and provider portals, SQLite — on a hardened single-host AWS estate of 67 public hostnames. The platform returned a self-test PASS on the ONC (g)(10) Standardized API test suite in April 2026, 317 of 317 subtests, publicly verifiable. Designed a consent architecture in which a permissioned ledger records patient grant, refusal and revocation as immutable events while all clinical data stays off-chain and encrypted per patient — no PHI on the ledger by design. Filed a proposed USCDI v7 data element with ONC on verifiable patient identity. Translates the Federal Register, HHS/CMS guidance and HITRUST requirements into architecture, controls and audit evidence.

Selected engineering scale

  • Self-managed estate of 67 public hostnames, 51 serving, on one AWS account.
  • FHIR R4 API with a SMART App Launch authorization server; patient and provider portals; an AI service layer; Hyperledger Fabric peer, orderer and CA; IPFS; a 337-table SQLite clinical store.
  • Two permissioned distributed ledgers in production, one operating a consent registry that records grant, refusal and revocation with no PHI on chain.
  • X12 005010 pipelines across eight transaction sets.
  • Designed, built, deployed and operated end to end by one person.

Credentials and standards work

  • Inferno (g)(10) self-test PASS, 317 of 317 subtests, April 2026 — the ONC Standardized API test suite. Standards validated: US Core 6.1.0, USCDI v3, SMART App Launch 2.0.0, Bulk Data 2.0.0. Public suite URL.
  • USCDI v7 public comment submission, April 2026 — proposed Patient Digital Identity Credential under the Provenance data class, listed publicly on the ONC Interoperability Standards Platform.
  • SAM.gov UEI MVKQW9DKKGB8 · CAGE 20KM5 · NPI 1194645564. AWS Business Associate Addendum active 18 August 2026.

Technical environment

AWS us-east-1 (compute, storage, messaging, SMS, email, voice) · Linux and systemd hardening · nginx · Fabric and EVM ledgers · FHIR R4 · X12 005010 · HL7 · SQLite · self-hosted mail, drive, search and monitoring · Git-based change control and CI release gates.

TRINH — Charles Trinh, technology architect
TRINH

Charles Trinh

Fractional CTO/CIO and Technology Architect — Healthcare Data, Identity & Regulated Infrastructure · Las Vegas, NV

Las Vegas, Nevada · US citizen · Fractional and project engagements

(702) 559-3489 Email Charles Trinh