Local Kafka-to-FHIR Pipeline for Real-Time ECG Vitals

Transforming real-time ECG vitals through Kafka into standardized FHIR Observations stored in MongoDB.

You can get the source for this project and run it in your own AWS environment by downloading it here.

In today’s healthcare environment, the ability to ingest high-velocity streams of patient data and transform them into a common clinical standard has become a critical business imperative. When you’re monitoring ICU bedside vitals, integrating wearables for remote patient monitoring, or aggregating data across multiple devices, you need a pipeline that can handle real-time throughput without dropping a beat. Healthcare organizations also must conform to interoperable models like HL7 FHIR to exchange data seamlessly between EMRs, analytics platforms, and decision-support tools. Rapid local prototyping of these integrations accelerates development cycles and reduces the risk of costly errors in production.

To address these challenges, I built an end-to-end demo you can run entirely on your laptop with Docker Compose. A Python-based “random-vitals-producer” emits synthetic ECG, blood pressure, respiratory rate, and SpO₂ measurements every 10 milliseconds, compresses and batches them, then pushes the payload into an Apache Kafka topic. A companion “fhir-consumer” service reads from that topic, validates each JSON record, maps it into a FHIR Observation resource - complete with LOINC coding and units - and persists the result as a JSON document in MongoDB. Because everything runs locally, you gain the confidence to test, debug, and tune performance before moving to Kubernetes or managed cloud services.

Under the hood, seven Docker containers live on a single Docker network. ZooKeeper and the Kafka broker form the durable, scalable message backbone. Kafka Manager offers a web UI for inspecting topics, partitions, and consumer lags. The producer uses LZ4 compression and batching to simulate real-world device throughput. MongoDB stores the FHIR resources as native JSON documents, and mongo-express gives you a browser interface on port 8081 to explore what you’ve captured. Each piece is loosely coupled yet seamlessly integrated via environment variables and Docker networking.

On the consumer side, the fhir-consumer spins up a Kafka consumer group that polls for the “vitals-json” topic, gracefully handles JSON parsing errors, and applies a configurable mapping to turn raw measurements into standard FHIR Observation components. Patient identifiers map to the subject field, timestamps to effectiveDateTime, and a lookup table injects proper display names, units, and LOINC codes into each measurement. The result is a clean, FHIR-compliant JSON document ready for downstream clinical applications.

This demo not only proves out high-throughput healthcare data ingestion and standardization on your local machine, but also serves as a blueprint for enterprise rollouts. By swapping Docker Compose for Kubernetes, adding TLS/SASL on Kafka, or leveraging managed cloud services, you can scale securely and reliably in production. To explore the full source and start experimenting with your own Kafka/FHIR integration, visit https://github.com/mjones3/kafka-vitals-fhir-pipeline.