On November 16, 2026, Anthropic added two encryption-key events to its beta organization webhooks: external_key.failed and external_key.recovered. Teams running the Claude API with customer-managed encryption keys can use these notifications to investigate loss of key access. They still need a separate decision about when applications may resume.
Consider a key shared by a support agent and a document-processing service. The key becomes usable again, but one workspace remains excluded by its KMS policy. An incident handler that closes on a recovery notification would leave part of the application broken.
This guide joins Anthropic's documented controls into a proposed incident procedure for the direct Claude API. General Analysis has not tested it against a live tenant. It covers key access and recovery; keep runtime guardrails and application authorization in place when traffic returns.
Record the dependencies before an alert arrives#
Maintain a small inventory linking each application to its Claude organization, workspace, external-key ID, KMS key, and owner. Include the credential route used to make requests, without storing the credential itself. A responder should be able to identify the affected service without asking an agent to search production secrets.
Use the workspace list API with include_default=true; the default is false. Follow pagination through has_more, using the returned last_id as the next after_id. Match external_key_id to find the key's dependents. Include archived workspaces in the inventory if retained data matters, but keep them separate from active services to probe. Preserve compartment_id too: AWS policy conditions use that encryption identifier, not the wrkspc_ ID.
Store the inventory outside the monitored application, with the time of its last complete refresh. If a page fetch fails, retain the previous snapshot and mark the new enumeration incomplete.
For webhook setup, the create-endpoint reference requires anthropic-beta: webhooks-2026-10-07. Both missing enablement and a missing beta header can produce a 404. HIPAA-ready organizations are excluded. Register an approved HTTPS receiver for both key event types and capture its signing secret securely at creation; later reads do not return it. Keep administrative credentials in the operations service, outside the agent runtime.
See how your AI systems hold up under real attacks
General Analysis maps AI applications and agents, red teams prompts, retrieval, tools, MCP servers, browser actions, permissions, and business workflows, then turns findings into evidence your team can reproduce and retest.
Keep the alert path independent of the failing key#
On Claude Platform, a CMEK attached to the Default Workspace encrypts organization-webhook names, descriptions, and signing secrets. When Anthropic cannot use that key, dependent endpoints can lose events. Claude Platform CMEK coverage documents this alongside the fields that remain under Anthropic's key.
An external receiver cannot compensate for a notification that the sender cannot produce. Route application errors and cloud KMS alerts into the incident process independently. Keep the receiver's own queue, credentials, and monitoring accessible during the outage. Document who can inspect those systems when the application team loses access.
Verify incoming signatures against the original request body and enforce timestamp tolerance with a Standard Webhooks library. Check the expected organization before accepting an event. Persist one intake record per event ID, then queue investigation work. Keep the signed event, receipt time, and subsequent observations separate so retries cannot overwrite the incident history.
Anthropic requires a 2xx response within five seconds. Its delivery contract allows duplicates and limited retries; disabled endpoints lose events. Acknowledge only after durable intake, and monitor queue age as well as receiver availability.
Validate the key, then check its workspaces#
For either notification, Anthropic recommends validating the key, then checking every workspace using it. Validation covers at most one workspace. Events may be absent during KMS outages or low traffic, and recovery need not produce a notification. Schedule reconciliation as a second trigger for this procedure.
The following request is adapted from the validation endpoint. Use an Admin API key in ANTHROPIC_API_KEY and the affected ekey_ identifier in EXTERNAL_KEY_ID. Run it from your authorized operations environment. It performs an encrypt/decrypt check against your KMS; this is a diagnostic operation, not a local syntax test.
Code source: Anthropic external-key validation reference.
curl --fail-with-body --max-time 45 \
-X POST \
"https://api.anthropic.com/v1/organizations/external_keys/${EXTERNAL_KEY_ID}/validate" \
-H 'anthropic-version: 2026-10-21' \
-H "Authorization: Bearer ${ANTHROPIC_API_KEY}"
Read the response body. HTTP 200 can contain status: "failure"; the endpoint waits up to 30 seconds and reports the error or timeout there. Preserve the result rather than converting transport success into a green health check. A client timeout leaves the diagnosis unresolved.
Probe each active dependent workspace with a harmless Messages request through its own authorized credential route. Use a synthetic prompt without tools. Save the workspace, request time, response ID when available, and outcome; a probe using another workspace's key would test the wrong access path.
Anthropic's key lifecycle documentation allows up to one hour for revocation to take effect. An early success cannot establish that revocation failed. After a failure notification, the webhook guide calls for repeating successful workspace probes an hour later. Record the notification time separately from the suspected change time: an event's created_at is its first send attempt, not the KMS failure time.
Choose the response from the failure evidence#
The error meanings below come from Anthropic's CMEK troubleshooting guide. The operational decisions are recommendations for your runbook.
| Observed error | Meaning | Operator decision |
|---|---|---|
cmek_key_disabled (400) | The KMS refused access; a disabled key or missing policy permission can cause it. | Establish whether revocation was intentional before restoring access. Check the affected workspace's policy scope. |
cmek_key_network_blocked (400) | KMS network controls rejected the request. | Review the network change and documented provider addresses. Do not broaden access just to clear the alert. |
cmek_key_unreachable (503) | Anthropic could not reach the KMS key. | Investigate availability and use bounded retries; keep the affected jobs visible. |
If validation succeeds but one workspace still fails, compare that workspace's compartment ID with the KMS policy. On AWS, Anthropic specifically identifies missing workspace conditions as a cause. Preserve the intended restrictions while fixing the mismatch.
Decide whether the incident concerns an accidental outage or an intentional containment action. An automation that always re-enables a key on failure could undo a security team's revocation.
Close the incident with a work record#
Restoring key access leaves application work to reconcile. Anthropic says requests that failed while the key was disabled are not automatically retried, and retention periods continue during the outage. Key lifecycle and recovery limits.
Use an incident record with these fields:
- The intended outcome and the owner who approved it.
- The key and affected-workspace inventory, including any incomplete enumeration.
- Validation results and workspace probes, with their observation times.
- Jobs that failed, completed, or remain uncertain, and the decision for each retry.
- The remaining application hold, or the evidence and owner permitting resumption.
For an agent that can write to another service, reconcile downstream effects before replaying a failed job. A new successful model request cannot tell you whether an earlier business action already happened. Resume through the application's existing authorization and duplicate-action controls.
Use cloud KMS records for key operations. Anthropic's CMEK architecture documentation assigns configuration changes to the Compliance API Activity Feed, while key operations belong in the cloud audit log. Our Compliance API guide explains the retained evidence path; the API trace coverage guide explains why a trace feed needs its own reconciliation.
Before enabling automated incident closure, exercise the receiver with local fixtures for duplicate notifications, delayed delivery, a validation failure inside HTTP 200, and one unresolved workspace. Require the last case to keep the incident open. These are proposed acceptance checks, not measured results. Use an isolated, approved environment for any later integration test; production key disruption is unnecessary for checking your handler's decisions.
Frequently asked questions
- Does a successful key validation prove every workspace recovered?
No. Anthropic's webhook guidance says validation checks at most one workspace. Review the workspaces using that key, including the Default Workspace, and verify their access separately before closing the incident.
- Should a key-recovery alert automatically resume an agent?
Keep resumption under application policy. Confirm why access was removed, whether the intended workspace can now make requests, and which jobs failed or have uncertain outcomes. Key recovery does not establish that retrying a business action is safe.

