Evaluating Healthcare AI Vendors Under HIPAA and BAA Constraints

When a healthcare team wants to add an LLM, transcription service, retrieval system, or AI-assisted workflow, the first question is often: "Which vendors are HIPAA compliant?" That question collapses several decisions into one label and makes it easy to miss the actual data path.

The useful starting point is narrower: What exact information will this service create, receive, maintain, or transmit, under which account and product configuration, for what purpose, and under what agreement?

I have used this approach while evaluating LLM and clinical transcription options for healthcare products and while shaping an AI-assisted patient-history workflow. It keeps architecture connected to the product's real data rather than to a vendor comparison spreadsheet.

Sources reviewed: HHS business-associate guidance, HHS cloud-computing guidance, HHS sample BAA provisions, HHS Security Rule guidance, and current first-party AI/vendor documentation where specific products are discussed.

Determine whether the service is acting as a business associate

HHS guidance explains that a person or entity can be a business associate when it performs functions or provides services involving the use or disclosure of PHI on behalf of a covered entity. When a cloud service provider creates, receives, maintains, or transmits ePHI on behalf of a covered entity or business associate, HHS says the provider is a business associate and the parties must enter into a HIPAA-compliant BAA.

That remains true when the cloud provider stores only encrypted ePHI and does not hold the encryption key. Conversely, merely selling software does not automatically create a business-associate relationship when the vendor has no access to PHI.

Primary references: HHS business-associate guidance and HHS guidance on HIPAA and cloud computing.

Draw the AI data flow

List what enters the service and what comes back. For a clinical transcription feature, that might include audio, patient or encounter identifiers, generated text, confidence or timing data, logs, and support artifacts. For an LLM workflow, it may include prompts, retrieved documents, structured patient facts, model output, evaluation traces, and cached responses.

Include less visible paths:

• Application and API logs
• Prompt or conversation history
• Human support access
• Abuse monitoring and safety systems
• Backups and disaster-recovery copies
• Analytics, observability, and evaluation tools
• Subprocessors used by the selected product or region

The result should make it clear whether PHI can reach the vendor and which contractual and technical controls are required.

Verify BAA coverage for the exact service

A company may offer a BAA for some products, account tiers, configurations, regions, or APIs but not others. Confirm the legal entity, purchased service, account, data location, subprocessors, and features covered by the agreement. Marketing language such as "HIPAA eligible" should not be treated as evidence that the current architecture and contract are covered.

HHS sample provisions describe a BAA as defining permitted and required PHI uses and disclosures, safeguards, incident reporting, subcontractor obligations, access to records, and return or destruction of PHI at termination. Contract review belongs with qualified legal and compliance stakeholders; engineering must provide them with an accurate data-flow and service inventory.

Reference: HHS sample BAA provisions.

Inspect retention, training, and administrative access

Ask how long inputs and outputs are retained, whether retention can be disabled or configured, whether data is used to train or improve models, who can access it for support, and how deletion requests propagate through active systems and backups.

Do not stop at a general privacy policy. The relevant terms may differ for enterprise APIs, consumer chat products, preview features, logs, fine-tuning, file storage, evaluation products, and support channels. Architecture should prevent staff from moving PHI into an uncovered interface simply because it uses the same vendor name.

Evaluate security and operations, not only model quality

Design clinical and AI failure boundaries

A signed BAA does not make an output clinically correct. Healthcare AI still needs product controls around hallucination, omission, unsafe sequencing, biased behavior, prompt injection, untrusted retrieved content, and changes between model versions.

For an AI-assisted clinical workflow, define what the model may recommend, what deterministic rules it cannot override, what requires human review, how unsuitable output is rejected, and how the workflow continues when the AI service fails. Evaluation should use representative, appropriately governed test cases and measure the behavior that matters to the actual workflow.

What to capture before approving a vendor

1. Purpose of the AI capability and the users affected.
2. Exact inputs, outputs, identifiers, logs, storage, and subprocessors.
3. Whether PHI is involved and the responsible compliance determination.
4. BAA coverage for the selected legal entity, service, account, and configuration.
5. Retention, training, deletion, support access, and regional controls.
6. Security, audit, incident, availability, model-change, and exit requirements.
7. Clinical oversight, deterministic constraints, fallback behavior, and evaluation plan.
8. Named product, security, privacy, legal, and operational owners.