---
name: Alerting Fabric - mql-enrichment-service
description: Unified alert routing across all three platforms. Tiered (P0 customer-impacting, P1 degradation, P2 drift). PagerDuty + Slack destinations. Owner: RevOps-Ops oncall.
type: cross-platform-artifact
layer: L7-concern
Owner: RevOps-Ops; oncall lead rotates weekly
---

# Alerting Fabric - mql-enrichment-service

- **Last reviewed:** 2026-04-27 (synthetic P1 firing drill)
- **Review cadence:** quarterly + on routing rule change
- **Status:** exists-mature

## What this means here

A unified alert routing path so any L5 platform can emit a tiered alert that reaches the right responder within a defined time. Without unified alerting, each platform would have its own alerting destination; cross-platform incidents would surface fragmentarily across multiple channels.

## Implementation

- **Alert destinations:** {{alerting_destination}} (PagerDuty for P0 / P1; Slack channel `#revops-ai-alerts` for P2)
- **Routing rules:** `cross-platform/alert-routing.yaml` - source platform + alert type + tier maps to destination
- **Tier definitions:**
  - **P0 customer-impacting:** SDR cannot work the queue (enrichment service down), or PII boundary breach detected, or consent-revoked enrichment shipped (red line trip)
  - **P1 degradation:** cost-ceiling breach pattern, eval threshold miss, latency target miss > 30 minutes, mirror sync drift > 21 days, guardrail trip rate above baseline
  - **P2 drift:** intent-signal data staleness rate above 5%, SDR thumbs-down rate above baseline, telemetry schema validation failures
- **Acknowledgement SLA:** P0 < 5 min, P1 < 30 min, P2 next business day

## Participants

All three platforms emit alerts; alerts route by source + type + tier through `alert-routing.yaml`.

| Platform | Alert types emitted |
|----------|---------------------|
| mql-enrichment-service | guardrail-trip-{red-line-1,2,3}, cost-ceiling-breach, eval-threshold-miss, mirror-sync-drift, schema-validation-failure |
| mql-router | routing-failure, edge-case-fallthrough, cost-ceiling-breach, latency-target-miss |
| sdr-handoff-helper | sdr-feedback-thumbs-down-pattern, canary-cohort-disengagement, latency-target-miss |

## Synthetic firing drill

**Last drill:** 2026-04-27

**Procedure:** Emit a synthetic P1 alert (`mql-router cost-ceiling-breach test`) and time end-to-end delivery to PagerDuty oncall.

**Result:** alert reached oncall in 2 min 14 sec from emission. Acknowledgement received in 3 min 22 sec from emission (within P1 SLA).

**Pass criteria:** P1 alert reaches oncall < 10 min; acknowledged < 30 min.

All criteria met. Recurs quarterly; P0 drill scheduled 2026-05-15.

## Alert hygiene

Alerts are reviewed quarterly for:

- Alerts firing > 3x weekly with no action taken (candidates for retirement or threshold adjustment)
- Alerts that should have fired during incidents but did not (gaps)
- Alert messages without a runbook link (every alert must point to a runbook)

Latest review (2026-04-15 quarterly): retired one P2 alert that had not fired in 6 months; tightened threshold on `latency-target-miss` after observing flapping.

## Last verified live

2026-04-27 synthetic drill (above). Continuous evidence: real alerts have fired and been acknowledged in production; the audit trail in `revops-ai-alerts` Slack channel and PagerDuty incident log shows steady-state alert-and-respond behavior.

## Inheritance and dependencies

Inherits from: {{cutting_edge_saas_company}} corporate alerting infrastructure (PagerDuty schedule, Slack channels). Predates this deployment.

Consumed by: all three L5 dossiers; every guardrail trip and cost ceiling breach routes through this fabric.

## Cross-references inside this instance

- `cross-platform/L7-glue.md`
- `cross-platform/alert-routing.yaml` (the routing rules)
- `policies/philosophy.md` (L2) - red line trips emit through this fabric
- `policies/framework-contract.md` (L3) - cost-ceiling breaches route through this fabric
- `runbooks/` - every alert points to a runbook
