incident
Watson for Oncology recommended unsafe care
IBM’s flagship cancer assistant suggested treatments that would have seriously harmed patients because it was trained on hypothetical data rather than real cases.
Healthcare Assistants
Assist care teams safely.
Keep PHI contained.
Protect clinical and patient-facing copilots with PHI-safe boundaries, guideline grounding, and escalation controls for unsafe advice.
solution workbench
incidents
3
products
2
evidence
full trace
failure modes
Watson for Oncology recommended unsafe care
IBM’s flagship cancer assistant suggested treatments that would have seriously harmed patients because it was trained on hypothetical data rather than real cases.
Mental health bot gave dangerous dieting advice
The National Eating Disorders Association shut down “Tessa” after it told users to cut calories and lose weight—precisely the guidance clinicians warn against.
Clinicians pasted PHI into public ChatGPT
Universities and hospital compliance teams cautioned doctors that feeding patient notes to OpenAI could violate HIPAA, since the vendor retains and trains on those prompts.
controls
Asset Management
Runtime Security
Guideline citations and PHI logs
Clinical safety checks
Protect clinical and patient-facing copilots with PHI-safe boundaries, guideline grounding, and escalation controls for unsafe advice.
Test the realistic attack paths
3 field failure modes become adversarial campaigns tailored to this deployment.
Convert findings into controls
Asset Management, Runtime Security keep the workflow bounded after launch.
Built for this workflow
Controls that match
the deployment.
- Exam-room copilots that summarize the visit into discrete EHR sections, highlight missing documentation, and prep prior-auth packets.
- Patient-facing navigators that triage symptoms, refill requests, or discharge questions with clear disclaimers and escalation paths.
- Care-management assistants scanning registries for gaps in therapy, clinical trials, or social-determinant outreach.
healthcare-copilots.yaml
# Apply the solution playbook. # $ ga solutions apply healthcare-copilots deployment: healthcare-copilots assets: - EHR context - guidelines - patient messages test_against: - Watson for Oncology recommended unsafe care - Mental health bot gave dangerous dieting advice - Clinicians pasted PHI into public ChatGPT runtime_controls: - AI Security Asset Management - AI Runtime Security evidence: traces,citations,owners
Field evidence
Failure modes worth testing.
Healthcare Assistants deployments fail when the model gets more trust than the workflow can safely absorb. These examples become concrete tests, not generic awareness copy.
incident
Mental health bot gave dangerous dieting advice
The National Eating Disorders Association shut down “Tessa” after it told users to cut calories and lose weight—precisely the guidance clinicians warn against.
incident
Clinicians pasted PHI into public ChatGPT
Universities and hospital compliance teams cautioned doctors that feeding patient notes to OpenAI could violate HIPAA, since the vendor retains and trains on those prompts.
How General Analysis helps
Product controls for this use case.
The playbook connects discovery, automated red teaming, and runtime protection so controls stay specific to the deployment instead of becoming a generic policy layer.
General Analysis
AI Security Asset Management
Catalogue every model, retrieval index, and dataset touching PHI, enforce encryption/retention policies, and keep training plus evaluation within HIPAA-compliant enclaves.
General Analysis
AI Runtime Security
Apply controls derived from red-team findings: dosing/contraindication checks, guideline citations, and escalation thresholds before any action.
How the playbook runs
Map
Identify the assets and owners
Inventory EHR context, guidelines, patient messages and the identities, tools, and data paths attached to the workflow.
Attack
Replay the relevant incidents
Turn field failures into adversarial prompts, multi-turn tests, tool-use probes, and policy traps for this deployment.
Enforce
Ship controls into production
Apply clinical safety checks, clinician-reviewed actions, and escalation rules where the workflow needs them.
Prove
Keep evidence attached
Guideline citations and PHI logs
FAQ
Questions teams ask before launch.
Practical answers for deploying healthcare assistants with controls that security, legal, and operators can inspect.
Knowledge packs are configured to sync with formulary updates, FDA label changes, society guidelines, and institutional protocols on a schedule you control. When a source document is updated, the system re-indexes affected content and flags any copilot responses that cited the now-outdated version. Runtime guardrails enforce mandatory review cycles so that clinical content is never served past its expiration date—outdated guidance is retired automatically and replaced with a notice directing clinicians to the updated source or a human specialist.