Building an AI model on healthcare data is one of the most valuable things a health system can do. It is also one of the most regulated. A HIPAA-compliant AI pipeline is the set of architectural decisions and data processes that enable a health system, healthcare AI vendor, or research team to train and deploy machine learning models on clinical data—without exposing protected health information (PHI) or triggering HIPAA's business associate obligations.
A pipeline becomes HIPAA-compliant in one of two ways:
- De-identifying PHI before it enters the training process, under HIPAA's Safe Harbor or Expert Determination standards.
- Handling PHI under a fully executed set of Business Associate Agreements and a Data Use Agreement, without de-identification.
Most AI teams find that de-identifying early in the pipeline, rather than managing PHI under a BAA all the way through training and deployment, produces a cleaner, more auditable, and less operationally fragile compliance posture.
This article walks through where PHI creates risk in a typical healthcare AI pipeline, the two main compliance paths available, and a step-by-step approach to building a pipeline that holds up to audit. For the broader compliance landscape beyond model training, including chatbots and other deployed AI systems, see our overview of AI chatbot compliance under HIPAA, GDPR, and state law.
The HIPAA problem with healthcare AI pipelines
HIPAA applies to PHI regardless of whether a human or an AI system is doing the processing. The Department of Health and Human Services' Office for Civil Rights has signaled this directly through its proposed update to the HIPAA Security Rule, which expects AI software that creates, receives, maintains, or transmits ePHI to be tracked in a covered entity's technology asset inventory like any other system touching protected health information. The rule has not been finalized, but the direction is clear: if PHI is ingested by an AI system to train it or generate output, HIPAA's rules apply to that data exactly as they would to a human-staffed process.
In a typical healthcare AI pipeline, PHI risk concentrates at three points.
- Ingestion. Raw clinical data, including clinical notes, discharge summaries, radiology reports, and structured EHR fields, enters the pipeline carrying the patient's name, dates, identifiers, and other protected fields.
- Training. That data, or anything derived from it, gets baked into model weights, a step that is difficult to reverse if PHI later needs to be removed.
- Output. A model trained on identifiable data can, in some cases, reproduce fragments of training data in its responses, a phenomenon Carlini et al. document in peer-reviewed research on language model memorization.
Any AI vendor or subcontractor that creates, receives, maintains, or transmits PHI as part of this pipeline is a HIPAA business associate, full stop, and requires a signed BAA before that data can lawfully reach them.
The two paths to compliance: de-identification vs. data use agreements
Healthcare AI teams generally have two structurally different ways to make a pipeline HIPAA-compliant.
|
De-identification |
Data use agreements (DUAs) |
| How it works |
Strip or transform the 18 HIPAA identifiers (Safe Harbor) or use statistical analysis to certify very low re-identification risk (Expert Determination) before data enters the pipeline. |
Keep data identifiable, but govern its use, access, and onward disclosure through a signed contractual agreement with the receiving party. |
| Data exits HIPAA scope? |
Yes, once properly de-identified, the data is no longer PHI for HIPAA purposes. |
No, the data remains PHI throughout, and HIPAA's full Security Rule and Privacy Rule obligations stay in force. |
| Best for |
Model training, fine-tuning, and most downstream analytics, where direct identifiers are not needed for the model to learn. |
Narrow research collaborations or limited data sets where some identifiers (e.g., dates, geography) must be retained for the specific analysis. |
| Operational burden |
Front-loaded: requires a validated de-identification process, but downstream handling is simpler once data is out of PHI scope. |
Ongoing: requires BAAs, access controls, audit logging, and breach procedures for as long as the data remains identifiable. |
DUAs have real limitations for model training specifically. A Limited Data Set under a DUA can still contain dates and geographic detail below the state level, which means the data remains PHI and the full weight of HIPAA's Security Rule, audit logging, and breach notification requirements travels with it through every stage of training, validation, and deployment. De-identified data, once properly validated, exits that scope entirely, which is why most healthcare AI teams treat de-identification as the cleaner default and reserve DUAs for the narrower cases where specific identifiers must be retained for the research question at hand.
Step-by-step: building a HIPAA-compliant AI data pipeline
A standard HIPAA-compliant AI pipeline follows six stages. The de-identification layer sits early, before data ever reaches the training environment.
1. Raw clinical data
What happens: Clinical notes, discharge summaries, radiology reports, and structured EHR fields are extracted from source systems.
Compliance obligation: Data remains full PHI. Access must be limited to authorized workforce members under the covered entity's existing Security Rule controls.
What can go wrong: Raw exports get copied to shared drives or analytics environments without access logging, creating an unmanaged PHI footprint before the pipeline even starts.
2. De-identification layer
What happens: PHI identifiers are detected and redacted or replaced across unstructured and structured fields, using either Safe Harbor's 18-identifier removal or Expert Determination's statistical risk certification.
Compliance obligation: This is the step that determines whether downstream data is still PHI. The de-identification method and its accuracy must be documented and, for Expert Determination, certified by a qualified expert.
What can go wrong: General-purpose, non-healthcare-tuned de-identification tools miss identifiers embedded in unstructured clinical narrative, such as a physician's name inside a dictated note, leaving residual PHI that the team believes has been removed.
3. De-identified training dataset
What happens: The validated, de-identified dataset is assembled and versioned for use in model training.
Compliance obligation: Once properly de-identified, this dataset is no longer PHI for HIPAA purposes, which materially reduces the compliance burden for everything downstream.
What can go wrong: Teams treat the de-identified dataset as PHI-free without documenting the validation method, leaving no audit trail if regulators or partners later ask how de-identification was performed.
4. Model training
What happens: The model is trained or fine-tuned on the de-identified training dataset.
Compliance obligation: Because the input data is de-identified, training itself does not create new PHI exposure, provided no identifiable data leaks back in through auxiliary metadata or logging.
What can go wrong: Training pipelines log raw input batches for debugging purposes, inadvertently retaining a parallel copy of data that was supposed to be de-identified.
5. Model validation
What happens: The trained model is evaluated for accuracy, bias, and (where relevant) its propensity to reproduce memorized training content.
Compliance obligation: Validation should include testing for output-level leakage, confirming the model does not regurgitate identifiable fragments even though its training data was de-identified.
What can go wrong: Validation focuses only on clinical accuracy metrics and skips privacy-specific testing, missing a memorization risk that surfaces later in production.
6. Production deployment
What happens: The validated model is deployed for clinical or operational use, potentially processing new, identifiable patient data at inference time.
Compliance obligation: If the deployed model processes new PHI at inference time, that live data flow is a separate HIPAA consideration from the training data and may require its own BAA and Security Rule controls.
What can go wrong: Teams assume that because training data was de-identified, the entire deployed system is outside HIPAA scope, missing that inference-time PHI is a distinct exposure point.
The de-identification layer's position matters more than any other single design decision in this pipeline. Performed before data enters the training environment, it determines the HIPAA status of every stage that follows. Performed late, or skipped, PHI exposure compounds across training, validation, and deployment, each of which then independently requires its own Security Rule safeguards and BAA coverage.
The de-identification layer: where it sits and what it processes
The de-identification layer sits between raw clinical data extraction and the training environment, ideally inside the covered entity's own infrastructure rather than in a third-party cloud service that would itself need to be a business associate. It needs to process the full range of healthcare data formats: structured EHR fields, semi-structured lab results, and, most challenging, unstructured free text like clinical notes, discharge summaries, and physician dictation, where identifiers like names, dates, and facility references are embedded in natural language rather than discrete fields.
Output from this layer should be either Safe Harbor-compliant, with all 18 HIPAA identifiers removed, or accompanied by Expert Determination documentation if certain fields like dates or sub-state geography are retained for analytical value. This is also the most common point where general-purpose tools fail: identifying a Social Security number in a structured field is straightforward, but reliably identifying a patient's name embedded in three different ways across a six-paragraph clinical note requires healthcare-specific natural language processing, not generic pattern matching.
Common mistakes that break HIPAA compliance in AI pipelines
- De-identifying after training instead of before. Once PHI has been used to train a model, the model itself may carry traces of that data. De-identification has to happen upstream of training, not as a cleanup step afterward.
- Treating structured-field redaction as sufficient. Removing the patient name field from a database table does not remove the patient's name if it also appears inside a free-text clinical note elsewhere in the same record.
- Skipping Expert Determination documentation. Teams that retain dates or geographic detail beyond Safe Harbor's limits, often necessary for clinically meaningful research, sometimes do so without the required statistical certification, leaving the data in an undocumented and arguably non-compliant state.
- Assuming a BAA with one vendor covers the whole pipeline. A pipeline often involves multiple vendors and subcontractors, cloud infrastructure, the AI platform, monitoring tools, and each one that touches PHI needs its own BAA.
- No process for inference-time PHI. A model trained on de-identified data can still receive identifiable PHI at inference time in production, which is a separate compliance surface that needs its own safeguards.
Validation and audit requirements: proving your pipeline is compliant
HIPAA does not require a specific certification for AI pipelines, but it does require covered entities and business associates to be able to demonstrate, on request, that their de-identification and data handling practices meet the standard they claim. In practice, that means three categories of documentation.
- De-identification methodology. Document whether Safe Harbor or Expert Determination was used, the specific identifiers removed or retained, and, for Expert Determination, the qualified expert's written certification of re-identification risk.
- Accuracy validation. For automated de-identification tools, maintain evidence of accuracy testing on representative healthcare data, since general-purpose tools have been shown to perform meaningfully worse on clinical narrative than healthcare-specific systems.
- Access and audit logs. Maintain logs showing who accessed PHI at each pipeline stage prior to de-identification, consistent with HIPAA Security Rule audit control requirements.
For organizations pursuing FDA clearance for AI as a medical device, or submitting research protocols to an Institutional Review Board (IRB), Expert Determination-ready documentation is particularly valuable: it gives reviewers a defensible, statistically grounded basis for evaluating re-identification risk, rather than asking them to take an informal assurance at face value.
How Limina fits into your pipeline
Limina's data de-identification platform sits at the de-identification layer described above, deployed in-VPC or on-premises so clinical data never has to leave a health system's own infrastructure to be de-identified. It processes clinical notes, radiology reports, discharge summaries, and other unstructured PHI formats across 52 languages and more than 50 entity types, identifying and replacing PII, PHI, and PCI before that data reaches a training pipeline.
Limina achieves 99.5 percent+ accuracy on real healthcare data, compared to 60 to 70 percent for general-purpose cloud de-identification tools, and produces expert determination-ready outputs that support the documentation FDA submissions, IRB protocols, and HIPAA audits typically require. Integration happens through an API that sits ahead of the training environment, so the de-identification step becomes a defined, auditable stage in the pipeline rather than an informal, undocumented process.
Building your pipeline
A HIPAA-compliant AI pipeline is not primarily a legal document. It is an architectural decision about where PHI gets removed, and whether that removal happens early enough, and accurately enough, to hold up under audit, FDA review, or an IRB protocol. Getting the de-identification layer right at the start saves a health system from re-architecting the entire pipeline later.
Limina helps health systems and healthcare AI teams de-identify clinical data in-VPC, producing expert determination-ready outputs that support HIPAA-compliant AI pipelines from ingestion through deployment. Teams get a defined, auditable de-identification stage instead of an informal cleanup step bolted onto the end of the pipeline.
Talk to an Expert to see how it fits your pipeline.
Explore Limina's data de-identification platform, or read our companion piece on AI chatbot compliance under HIPAA, GDPR, and state law for the compliance picture beyond model training.
