INTEGRATION

Charles Trinh — Epic Integration Architecture | Las Vegas, NV · Las Vegas, Nevada · Henderson · Clark County, Nevada · North Las Vegas, Nevada

Diagram of an Epic community interconnect with two landing zones and a mirrored non-production environment
Charles Trinh — Epic Integration Architecture

Epic interfaces fail at the seams, not in the middle. I map every landing zone, every non-production mirror and every retry path before I sign off on a single message flow.

What an Epic integration actually depends on

An Epic integration is not one interface. It is a tenant interconnection, a set of OAuth scopes granted by the customer's security team, a community interconnect handshake, and an application that lives inside the customer's non-production mirror for at least six weeks before anything reaches production. I start every engagement by listing those four dependencies and finding out which one is already broken. Usually it is the mirror: it lags the production release by a version or two, and the first interface test is run against schemas that no longer match.

The artefact that proves a build is wired is the X-Request-ID echoed back in the response header and the epic-specific AuditLogRequest entry that appears in the customer's non-production web logs within thirty seconds. If the header comes back but the log entry does not, the app is hitting a stub. If neither comes back, the OAuth token was minted against a scope set that the interconnection does not accept, and the whole chain needs re-authorising.

None of this is revealed by a successful 200 OK on a health endpoint. I have seen half a dozen implementations where the health check passed on a route that was served by a mock, and the customer was paying for interfaces that had never touched Epic. The integration ticket is only closed when a real patient, in the non-production mirror, produces a real Encounter resource through the production-scope token.

Landing zones and the ECS interconnect

Epic's community interconnect is the hop between the customer's Epic and the region the customer has contracted into. A landing zone lives in that region and terminates TLS with a certificate Epic issues. I have seen organisations put the landing zone behind their own load balancer and then wonder why Epic's validation fails, when the validation certificate is bound to the landing zone endpoint and not the load balancer in front of it. The landing zone has to be reachable from Epic's egress range directly, or the interconnect silently fails after the first heartbeat.

The other failure mode is the DNS records. Epic issues three CNAMEs for the interconnect, and they must resolve from Epic's own resolvers, not just from yours. One organisation's zone was correct internally but the parent domain had a wildcard CAA record that prevented Epic from obtaining the certificate; the interconnect sat in pending for three weeks while their team checked the landing zone over and over. I now check dig +trace from an external resolver on the first day of the build, not the last.

Landing zone capacity is another trap. The default sizing is generous for one interface but too small for a multi-interface programme, and Epic does not queue messages that exceed the zone's connection limits. They are rejected and the customer sees failed deliveries in the interconnect monitor that do not correspond to anything on the application side. If you are planning more than one integration, size the landing zone for the total, and add headroom for the nightly batch that everyone forgets to account for.

Non-production mirrors and why they matter more than production

Every serious Epic integration is tested against the customer's non-production mirror first. The mirror is refreshed on a schedule the customer controls, usually quarterly, and it contains real patient data that has been redacted. The redaction is not total: I have seen full dates of birth and real MRNs in the mirror, and I have seen synthetic patients with fabricated clinical content. Assuming either is a compliance problem. The rule I apply is that no mirror data leaves the customer's environment, and no extract from the mirror is used to seed a lower environment unless the customer's privacy office has signed off.

The mirror is also where the release cycle bites. Epic releases are quarterly, and the mirror upgrades shortly after the production release. If your test plan is scheduled between the production upgrade and the mirror upgrade, you are testing against a version that will be gone before go-live. I have pushed organisations to align their test window with the mirror's current version instead of the production version, because the mirror version is what they will be validating against during the build, whether they planned it or not.

Access to the mirror is granted to named accounts, not service accounts. That means your automated test harness cannot authenticate the way it would in production, and the integration test has to be run by a human with a mirror account or by a named service account provisioned specifically for it. Most delays I have seen in Epic programmes trace back to an assumption that the service account in production would work in the mirror. It never does, and the fix is a ticket that sits in the customer's queue for a month.

OAuth scopes and the token that hides the problem

Epic uses a scoped OAuth 2.0 model. The customer's security office grants a set of scopes per application, and the token issued from the authorisation server only carries those scopes. The failure I see most often is a token with the right scope but a patient context that the application was not granted. The call returns a 403 with a message that says the patient is not in the allowed set, but the message is generic; the actual reason is in the customer's AuditLog, not in the API response, and the application developer reads the response and chases the wrong problem.

I build the scope map before the build begins: every scope the application will need, the FHIR operation each one gates, and the human owner at the customer who can approve it. The scope map goes into the application's README and into the customer's integration ticket. When a developer adds a call that needs a new scope, the pull request fails a check that compares the requested scopes against the scope map, and the reviewer sees immediately that the change is blocked. I have seen one programme ship a feature that was blocked in production for two weeks because a scope was granted by the wrong team.

Token refresh is the second trap. Epic tokens are short-lived, and a long-running job that starts with a valid token will fail halfway through when the token expires. The application has to handle 401 from any call by refreshing and retrying, not just at the start of the job. I have seen batch jobs fail at exactly the same point every night because the refresh handler only fired on the first request, and the second request an hour later was still trying to use the expired token.

The retry design that decides whether you are blamed

Failed Epic messages are not retried by Epic. They sit in the interconnect and are failed exactly once; it is the receiving application's job to decide whether to retry, and how. The design question is not whether to retry but what the retry envelope looks like. I use a three-tier design: immediate retry with exponential backoff for transient transport failures, deferred retry for server errors with a fixed schedule, and dead-letter for anything that has failed a validation check that a retry cannot fix.

The retry envelope has to carry a stable message ID across attempts, or the receiving application will create duplicate resources. Epic does not deduplicate on Encounter or Observation; it accepts every resource the application posts and only the application's business logic decides what is a duplicate. I have seen an organisation send the same Observation four times because the retry envelope regenerated the ID. The customer's analytics team spent a week de-duplicating the data before anyone noticed the interface was the cause.

The dead-letter queue has to be owned. A queue that nobody watches is a defect. The owner is the interface analyst, not the application support desk, and the alert on dead-letter depth has to route to them within ten minutes. I write the alert threshold into the runbook and confirm the routing during go-live. If the analyst is out of office, the threshold routes to a defined backup, not to a shared mailbox that nobody opens. That one line has saved more than one programme from a silent failure that would have run for a quarter.

Where the integration cost actually lands

The bill for an Epic integration is rarely in the licensing. It is in the security review, the mirror access tickets, the scope-map meetings and the release re-test after every Epic upgrade. A realistic budget for a single Epic-integrated application includes one full quarter of non-production mirror time, a security review that takes four to six weeks with a director-level escalation likely, and a re-test every quarter after go-live that the customer owns and your team supports.

That re-test is often where the cost surprises organisations. They budget for build and go-live and not for the steady-state. Epic is built to upgrade, and every upgrade has a regression list of interface behaviour changes. I write the regression list from the first upgrade and keep it in the integration ticket, so the re-test is a checklist and not a discovery exercise. The checklist is the second artefact I hand over, after the scope map, because it is the only thing that makes the interface cheap to maintain.

If you want to see how these interfaces relate to the wider interoperability programme, the healthcare interoperability strategy work and the ONC (g)(10) readiness programme describe how Epic's FHIR APIs fit into a broader conformance plan. Integrating with Epic is not a standalone job; it is one part of the organisation's data architecture, and the decisions here are the same decisions those services exist to make.

What I hand over when the build closes

The first deliverable is the scope map, with every scope linked to a FHIR operation and a named approver. The second is the mirror test plan, with the mirror version flagged and the test window recorded. The third is the retry design, the dead-letter owner and the alert threshold. The fourth is the regression checklist from the last Epic upgrade, with every interface behaviour change and how it will be re-tested. Together these are the difference between an interface that survives a year and one that breaks in the first upgrade.

None of them require me to log into the customer's production. I produce them in the mirror and hand them over, and the customer runs the final go-live. That separation is deliberate. The customer owns the interfaces; I help them build the ones they can own. The delivery model is a small number of engagements run by an engineer who is also the person answering the phone when something breaks during a release.

Proof

  • ONC (g)(10) self-test PASS, 317 of 317 subtests, April 2026 — public suite URL
  • 67 public hostnames, 51 serving, one AWS account
  • 104 clearinghouse enrolments across 53 jurisdictions
  • USCDI v7 proposal — Patient Digital Identity Credential — listed at isp.healthit.gov
  • SAM.gov UEI MVKQW9DKKGB8 · CAGE 20KM5 · NPI 1194645564
  • AWS Business Associate Addendum active since 18 August 2026

Where this work is delivered

Engagements run from Las Vegas and reach across the valley: clients in Las Vegas, NV; Las Vegas, Nevada; Henderson; Clark County, Nevada; and North Las Vegas, Nevada — plus remote delivery nationwide.

How an engagement runs

1. Trace one real case

Follow a single patient or claim end to end with a timestamp at every hop, before any document is written.

2. Written findings

A findings list naming the defect, the system, the owner and the date.

3. Monthly review

Review the list monthly and check whether the dates held.

Frequently asked

What is this service?

This page describes charles trinh — epic integration architecture — the scope, the failure modes it prevents and how the work is measured. It sits in the Identity and Consent category at trinh.io.

Where is the work delivered?

Engagements run from Las Vegas, Nevada across the valley — Las Vegas, Henderson, North Las Vegas, Spring Valley, Paradise, Winchester, Sunrise Manor, Enterprise, Summerlin, Clark County — and remotely anywhere in the United States. On-site work in the Las Vegas valley, remote delivery elsewhere.

What does it cost?

Standard rate $195 per hour, committed $165 per hour at twenty hours or more per week for three months or more; $1,600 standard day, $1,350 committed day. Scoped in writing before work starts, fixed fee where the deliverable is defined.

Who is Charles Trinh?

A technology architect in Las Vegas, Nevada, working in healthcare identity, consent and interoperability. Not a physician, not a medical practice, and not affiliated with any other Charles Trinh.

How does an engagement start?

With one trace of a real case and a written findings list naming the defect, the system, the owner and the date. The first call is twenty minutes and free: 702-559-3489.

Engagement

Scoped in writing before work starts; fixed fee where the deliverable is defined. First call is 20 minutes and free. 702-559-3489 · charles@trinh.io

Talk to Charles

Record

Profile

Healthcare technology architect working where federal regulation becomes production infrastructure. Built and operates a FHIR R4 clinical platform — SMART App Launch authorization server, patient and provider portals, SQLite — on a hardened single-host AWS estate of 67 public hostnames. The platform returned a self-test PASS on the ONC (g)(10) Standardized API test suite in April 2026, 317 of 317 subtests, publicly verifiable. Designed a consent architecture in which a permissioned ledger records patient grant, refusal and revocation as immutable events while all clinical data stays off-chain and encrypted per patient — no PHI on the ledger by design. Filed a proposed USCDI v7 data element with ONC on verifiable patient identity. Translates the Federal Register, HHS/CMS guidance and HITRUST requirements into architecture, controls and audit evidence.

Selected engineering scale

  • Self-managed estate of 67 public hostnames, 51 serving, on one AWS account.
  • FHIR R4 API with a SMART App Launch authorization server; patient and provider portals; an AI service layer; Hyperledger Fabric peer, orderer and CA; IPFS; a 337-table SQLite clinical store.
  • Two permissioned distributed ledgers in production, one operating a consent registry that records grant, refusal and revocation with no PHI on chain.
  • X12 005010 pipelines across eight transaction sets.
  • Designed, built, deployed and operated end to end by one person.

Credentials and standards work

  • Inferno (g)(10) self-test PASS, 317 of 317 subtests, April 2026 — the ONC Standardized API test suite. Standards validated: US Core 6.1.0, USCDI v3, SMART App Launch 2.0.0, Bulk Data 2.0.0. Public suite URL.
  • USCDI v7 public comment submission, April 2026 — proposed Patient Digital Identity Credential under the Provenance data class, listed publicly on the ONC Interoperability Standards Platform.
  • SAM.gov UEI MVKQW9DKKGB8 · CAGE 20KM5 · NPI 1194645564. AWS Business Associate Addendum active 18 August 2026.

Technical environment

AWS us-east-1 (compute, storage, messaging, SMS, email, voice) · Linux and systemd hardening · nginx · Fabric and EVM ledgers · FHIR R4 · X12 005010 · HL7 · SQLite · self-hosted mail, drive, search and monitoring · Git-based change control and CI release gates.

TRINH — Charles Trinh, technology architect
TRINH

Charles Trinh

Fractional CTO/CIO and Technology Architect — Healthcare Data, Identity & Regulated Infrastructure · Las Vegas, NV

Las Vegas, Nevada · US citizen · Fractional and project engagements

(702) 559-3489 Email Charles Trinh