In my last post I argued that interoperability is hard. This one is about the part of the problem I think we underinvest in most: testing.
Here is the uncomfortable version. In HL7, there is no true end-to-end interoperability testing. Connectathons give us a place to try out the IGs, and for many use cases, that's certainly good enough. But they fall short on complex, multi-party scenarios, and they offer no glide path from "it worked at the Connectathon" to "it works in production." That last mile is often left to implementers to figure out on their own.
This is not a criticism of HL7. Testing has never been its core mission. As FHIR implementations move from development into production, the gap begins to matter more, especially with regulation such as CMS-0057-F pushing this forward.
At the last HL7 Working Group Meeting, I sat in on an education session focused on testing, coming from the HL7 FAST Accelerator work. If you are not familiar with FAST, it's one of the Accelerators that has been around in some form for almost ten years, and as a FHIR Accelerator for the last four. Its focus is scaling FHIR, and this recent initiative takes a laser focus on scaling testing.
Why interoperability testing is hard
Healthcare systems don't get to meet requirements individually (or collectively) and call their work finished. They must support varying workflows across provider organizations, geography, payer system, specialty/sub-specialty, user type/role, and more. The variety is too much for a standard QA test plan or a pass/fail on data structure validation.
Payer-to-Payer exchange under CMS-0057-F is a good example: two payers can each pass conformance testing against the Da Vinci PDex IG and still fail to exchange anything useful with each other for several reasons:
- Member matching logic differences
- Consent and opt-in management
- Endpoint discovery inconsistencies, or altogether absent
- Data completeness variations
None of that surfaces in a validation exercise. It often shows up the first week in production. Good interoperability testing needs configurable workflows and dynamic test scripts. It needs real-world test partners paired with each other. And it needs engineers on both sides actually talking.
Conformance is not interoperability
Conformance proves you can speak the language, but interoperability proves you can communicate. And yes, this is something we've all heard again and again, and yet it's more important than ever to continue this messaging as the industry continues to shift and change, incorporating providers, payers, and patients more into a single vision of shared interoperability workflows.
Tools like Inferno and the HL7 FHIR Validator will catch structural errors, but they will miss things like how partners interpret optional elements needed for specific interoperability workflows. Passing conformance is the entry ticket, not the finish line. Too many go-live plans treat it as the finish line for production-ready interoperability solutions.
A testing maturity ladder
It helps to think of interoperability testing as a ladder. Each rung builds on the one below it, proving greater levels of interoperability as the systems climb.
| Step | Description | Who owns it |
|---|---|---|
| 6. Production Monitoring | Live traffic watched for errors, drift, and data quality issues | Implementers |
| 5. Pilot Site | Limited production exchange with real members or patients | Implementers |
| 4. Paired Partner Testing | Two real systems run the full workflow with their own data | Implementers |
| 3. Connectathon Scenarios | Scenario-based tests against several test partners | HL7+Tools+Community |
| 2. IG Conformance | Inferno-level test kits that check systems at the FHIR IG level (i.e., a collection of FHIR Profiles) | HL7+Tools+Community |
| 1. Resource & Profile Validation | The validators that check data structures and FHIR Profile-level conformance | HL7+Tools+Community |
The bottom rungs involve HL7, publicly available and subscription-based tooling, and the overall community, whereas the top three rungs are where the interoperability happens in the real world. This ladder is our glide path to production-level interoperability.
Being a good testing partner
Whatever formal structure emerges, testing quality still comes down to how partners show up. My short list:
- Stage your system before you arrive. Load test data that matches the stated scenarios, and confirm your endpoints, credentials, and authorization flows work before day one.
- If there's no test scenario, write one, and share it. A half-page testing scenario your partner can react to beats an open-ended "let's see what happens."
- Publish your capabilities. Your capability statement should reflect what you plan to test today, not your roadmap. But use some leeway for stretch goals to explore new areas.
- Log everything. Turn up those log levels! Capture requests, responses, and errors; AND understand how to search your logs. This helps to reproduce errors afterward.
- Document lessons learned. Your interpretation of IGs may be different than your test partner's. Write these things down, send them to IG authors, and think through how that impacts your production implementation.
- Be helped, but also be helpful. If your system is more mature, give some of your time to the partner who's just getting started. The ecosystem only works if the newer entrants can connect too. Teamwork makes the dreamwork.
Bottom line
There’s never been a better time to fix this. Interoperability-focused regulation and incentive programs are creating real demand for connected systems, and vendors, providers, and payers are well positioned to move beyond conformance into real-world testing.
We don’t need to wait for a formal program to start closing the gap between development and production. Conformance is our entry point. From there, we can build reusable scenarios and test data, and keep sandboxes available for cross-organization testing. When FAST asks for participation, show up, because the shared infrastructure we create AS A COMMUNITY will be our ticket to success.