---
name: Session Log - 2026-04-22 - Consent-flow incident response and drill
description: Incident response after consent-revocation events stopped propagating to the cache for 18 minutes. Root cause was an upstream subscription failure on consent-service side. Worst-case stale cache exposure was 11 minutes; no enriched-on-revoked records shipped due to output guardrail re-check.
type: session-log
---

# Session Log - 2026-04-22 - Consent-flow incident response and drill

- **Last reviewed:** 2026-04-22
- **Review cadence:** ad-hoc (point-in-time)
- **Status:** complete

## What this session was

Incident-driven session. At 14:32 UTC, monitoring caught a `consent-revocations subscription stale` P1 alert; subscription had been silent for 18 minutes. Triage and response covered ~3 hours; followed by a synthetic drill confirming end-to-end propagation works correctly under steady-state conditions.

## What was produced

| Artifact | Path | Status |
|----------|------|--------|
| Incident timeline | `sessions/2026-04-22-consent-flow-incident.md` | this file |
| Updated `cross-platform/consent-flow.md` | last-verified-live date refreshed | modified |
| Updated `runbooks/consent-flow-incident.md` | drill procedure adjusted to include subscription-staleness check | modified |
| Updated `policies/philosophy.md` last-reviewed date | reflecting incident-driven review | modified |
| Compliance Engineering inbox entry | `inbox/from-compliance-engineering.md` | new entry; 2026-Q3 SOC 2 evidence capture pre-scheduled |

## Decisions made

- **Decision:** Add subscription-staleness alert to monitoring base (`cross-platform/monitoring-base.md`) as a Group 4 (Drift) panel; fires P1 if subscription quiet for > 5 min. Routine hygiene; documented inline in alert-routing.yaml.
- **Decision:** Output guardrail's consent re-check stays in place (already operating as-designed; this incident validated its value). No change.

No ADR-worthy decisions; ADR-0003's pattern held under stress.

## Findings

- **Finding:** Cache infrastructure has a single point of subscription-staleness detection (the `consent-revocations` topic subscription). When the subscription fails silently upstream (consent-service event publisher hiccup), this team's cache cannot detect the failure for up to the TTL.
  - **Why it matters:** ADR-0003 named subscription-event invalidation as a key mitigation. Silent subscription failure undermines that mitigation
  - **Next move:** Add proactive heartbeat monitoring: consent-service emits a `consent-heartbeat` event every 60 seconds; absence > 5 minutes triggers cache flush + P1 alert. Negotiation with Compliance Engineering needed
  - **Recorded where:** `inbox/from-compliance-engineering.md` (proposal sent); BACKLOG.md entry pending Compliance approval

- **Finding:** Worst-case exposure during this incident was 11 minutes of stale cache (subscription stale 18 min; cache TTL 5 min; first cache-miss after subscription resumed re-fetched from consent-service correctly). Acceptable per ADR-0003's risk model, but the heartbeat mitigation would tighten worst case.

## State changes

- **Layer maturity changes:** L7 consent-flow concern stayed `exists-mature` (pattern held under stress); no demotion warranted
- **Relationship status changes:** consent-revocations topic subscription transient `declared-but-inactive` for 18 min; restored to `live` at 14:50 UTC; last-verified-live confirmed at 14:55 UTC after synthetic revocation test
- **Pinned-version changes:** none

## Incident timeline

| Time (UTC) | Event |
|-----------|-------|
| 14:14 | Last successful consent-revocation event delivery (per consent-service log) |
| 14:32 | `consent-revocations subscription stale` P1 alert fires (quiet > 18 min) |
| 14:33 | RevOps-Ops oncall acknowledges; pages RevOps-AI eng lead |
| 14:36 | RevOps-AI eng lead diagnoses: subscription healthy on our side, no events arriving |
| 14:39 | Page Compliance Engineering oncall (consent-service owner) |
| 14:43 | Compliance Engineering oncall confirms: consent-service event publisher had a transient deadlock; restart in progress |
| 14:48 | Consent-service publisher restored; events resume |
| 14:50 | Subscription resumes receiving events; first event in 36 minutes processed |
| 14:55 | Synthetic revocation test on test subject; propagation 3 min 14 sec; pass |
| 15:30 | Incident closed; runbook updated |

## Concurrent-actor notes

Three actors during incident: RevOps-AI eng lead (driver), RevOps-Ops oncall (paged), Compliance Engineering oncall (paged). Coordination via `#revops-ai-incidents` Slack channel; full transcript preserved.

No concurrent-write conflicts; only the eng lead modified artifacts during the incident.

## Handoff posture

- **Where things stand:** Incident closed; pattern from ADR-0003 held; output guardrail re-check confirmed in place; no enriched-on-revoked records shipped
- **Where to start:** Negotiation with Compliance Engineering on heartbeat-event proposal (driver: RevOps-AI eng lead; assistive: Compliance Engineering)
- **Pending blockers:** Compliance Engineering capacity to review and possibly add heartbeat event publisher

## Failure modes encountered

- **Caught: untested relationships.** The consent-revocations subscription was reported as `live` in the most recent quarterly review (2026-04-15); under stress it became `declared-but-inactive` for 18 minutes silently. Standard reinforced: subscription liveness must be continuously verified, not point-in-time. The new subscription-staleness alert encodes this trigger.

## Session metadata

- **Effort:** ~3 hours during incident; ~45 minutes runbook + artifacts update afterward
- **Format:** Incident-driven; multi-team Slack-coordinated
- **Operators:** RevOps-AI eng lead, RevOps-Ops oncall, Compliance Engineering oncall
- **Tools used:** {{observability_platform}} (panels), {{alerting_destination}} (PagerDuty), Slack, runbook reference

## Cross-references

- ADRs written or modified: none (ADR-0003 pattern validated; no supersession needed)
- BACKLOG entries created: heartbeat-event proposal pending Compliance approval
- Inbox entries created: `inbox/from-compliance-engineering.md` outbound proposal; `inbox/from-{{platform_team_name}}.md` (informational; cache infrastructure feedback)
- Artifacts modified: `cross-platform/consent-flow.md`, `cross-platform/monitoring-base.md`, `runbooks/consent-flow-incident.md`
- Prior session this builds on: `sessions/2026-04-15-quarterly-review.md`
- Related ADR: `decisions/ADR-0003-consent-flow-integration.md` (pattern validated)
