---
name: security-hardening-deploy
description: Safe sequencing for shipping security hardening changes to a live public-facing surface. One layer at a time, verify after each, rollback on any signal of breaking legitimate traffic. Invoke when closing findings from public-endpoint-review or implementing controls from a threat model.
---

# Security Hardening Deploy

Hardening is dangerous work. Each layer CAN break legitimate submissions if deployed wrong. This skill enforces a sequence that minimizes blast radius, keeps rollback trivial, and produces a trace the operator can grep if something breaks later.

## When to Use

- Implementing remediations for findings from `public-endpoint-review`
- Shipping new controls from a fresh threat model
- Responding to sustained abuse (emergency hardening)
- As part of the appsec-reviewer subagent's output when the review proposes concrete fixes

## Principle

> Every hardening change is deployed in its smallest survivable increment, verified in production, and rolled back at the first sign of a legitimate user being blocked.

Hardening that blocks legitimate users is worse than no hardening - it produces silent conversion loss plus the false confidence that "we're protected."

## Sequence

### Phase 1: Baseline

Before changing anything, capture the current state. This is the rollback destination and the "what legitimate behavior looks like" reference.

```bash
# Probe each endpoint with representative legitimate + invalid payloads
# Save the responses (body + status + headers) to local/baseline-YYYY-MM-DD/
# Never commit this directory

curl -sD - -X POST https://SITE/api/ENDPOINT \
  -H 'Content-Type: application/json' \
  -H 'Origin: https://SITE' \
  -d @valid-payload.json \
  > local/baseline-YYYY-MM-DD/valid.txt

curl -sD - -X POST https://SITE/api/ENDPOINT \
  -H 'Content-Type: application/json' \
  -d '{}' \
  > local/baseline-YYYY-MM-DD/empty.txt
```

Keep the baseline for the duration of the hardening cycle. After each phase, diff against baseline to confirm legitimate behavior still works.

### Phase 2: Observability first

Ship structured logging BEFORE any rejection logic. If a later phase breaks something, the logs are the audit trail that tells the operator what got rejected and why.

- Deploy log emission on every request (success + error)
- Include trace ID, timestamp, endpoint, status, reason, source IP
- Do NOT include PII in logs beyond what's needed for debugging
- Confirm logs are flowing in Cloudflare Logs (or equivalent sink)
- Wait for 10-30 legitimate requests before moving on - validates the log format in production

### Phase 3: Passive checks (log-only, no reject)

Ship the new controls in LOG-ONLY mode first. The Worker inspects the request and emits a log line saying "would have rejected because X" but still processes the request normally.

Typical controls to ship log-only first:

- Origin check - log when Origin doesn't match allowlist, but don't reject yet
- Honeypot - log when honeypot field is non-empty, but don't reject yet
- Rate limit - count requests per IP, log when over threshold, but don't reject yet
- Turnstile verification - verify the token if present, log the verdict, but don't require a token yet

Leave each control in log-only mode for at least 24-48 hours (or as long as needed to see a meaningful traffic sample). Look at the logs:

- Are the rejections 100% what you expect (bots) or are legitimate users hitting them?
- Is the origin allowlist complete? (Users on iOS sometimes send different origins; preview deploys have different origins; browser extensions can strip Origin.)
- Is the rate-limit threshold right? (Legitimate users never hit it; set it so.)

Tune thresholds and allowlists based on what the logs show. This is the phase where you save yourself from a production outage.

### Phase 4: Enforce, one layer at a time

After log-only data confirms a control is safe, flip it from log-only to enforce. One layer per deploy. Not "origin + honeypot + rate-limit" in one commit - that's three deploys.

Order of enforcement (lowest risk to highest):

1. **Honeypot** - rejection is invisible (fake 200 success). No legitimate user is ever affected. Ship first.
2. **Input length caps / validation tightening** - if the log-only phase showed no legitimate submissions hitting the cap, safe to enforce. Rejection is an explicit 400 with the cap communicated.
3. **Origin check** - if the log-only phase showed no legitimate submissions with non-allowlist origins, safe to enforce. Rejection is 403.
4. **Rate limit** - highest risk because "legitimate" bursty traffic can trigger. Enforce only after log-only shows threshold is safe. Rejection is 429 + Retry-After.
5. **Turnstile verification** - requires frontend deploy of the widget first. Enforce server-side verification only after the widget has been serving for 24+ hours and generating valid tokens.
6. **CSP tightening** - most likely to break something (inline scripts, external resources). Report-only mode first (`Content-Security-Policy-Report-Only`), monitor violations, tighten based on report data, then enforce.

After each enforcement, verify:

- Baseline legitimate payload still gets 200
- The rejection path returns the expected status
- Logs show the expected reason codes

### Phase 5: Monitor

After all layers enforce, monitor for 1-2 weeks. Specifically watch for:

- Legitimate-looking emails / payloads in rejection logs (false positives)
- Sustained failure rate on any one reject reason (could indicate a misconfigured rule)
- Drop in overall submission volume (could indicate legitimate users silently giving up)

Adjust thresholds or allowlists if false positives surface. Document the adjustment in the site's THREAT-MODEL.md change history.

## Rollback triggers

Roll back immediately (not after investigation, not after a ticket) if:

- Legitimate submissions stop arriving (check Airtable for new records vs prior-day baseline)
- A user-facing error rate spikes
- A canary submission (you POST a known-good test payload) gets rejected
- The Worker throws an unhandled exception in logs

Rollback = revert the most recent hardening deploy. The goal is to restore service; diagnosis happens AFTER rollback, not BEFORE.

## Commands

### Deploy a Worker change

```bash
cd /path/to/site
# Before deploy, lint + check syntax
node --check _worker.js

# Deploy
npx wrangler pages deploy . --project-name SITE-PROJECT --commit-dirty=true

# Wait for the deploy to go live (~30-60s)
# Then verify against baseline
```

### Revert a Worker change

```bash
# Option A: deploy the prior commit
git checkout HEAD~1 -- _worker.js
npx wrangler pages deploy . --project-name SITE-PROJECT --commit-dirty=true
git checkout HEAD -- _worker.js  # restore the in-repo version

# Option B: deploy a specific known-good commit
git show GOOD_COMMIT:_worker.js > _worker.js
npx wrangler pages deploy . --project-name SITE-PROJECT --commit-dirty=true
```

Keep the rollback one-liner bookmarked during a hardening cycle.

### Add a Cloudflare Rate Limiting rule via API

```bash
# Requires CLOUDFLARE_API_TOKEN in env with Zone:Edit on the target zone
ZONE_ID="..."
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/rulesets" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Rate limit /api/*",
    "kind": "zone",
    "phase": "http_ratelimit",
    "rules": [{
      "action": "block",
      "ratelimit": {
        "characteristics": ["ip.src"],
        "period": 60,
        "requests_per_period": 10,
        "mitigation_timeout": 600
      },
      "expression": "(http.request.uri.path matches \"^/api/\")",
      "description": "Rate limit all /api/* to 10 req/min per IP"
    }]
  }'
```

Start generous (e.g., 10 req/min); tighten based on log-only phase observations.

## Post-cycle checklist

When a hardening cycle closes, verify:

- [ ] All findings from the review are marked closed in the findings doc
- [ ] The site's THREAT-MODEL.md change history records what was deployed
- [ ] ADR written if the overall posture is a new decision (typically the first hardening cycle per surface)
- [ ] Baseline directory deleted (it's in `local/` and gitignored; don't let it rot)
- [ ] Rollback commands tested (at least once, against a canary deploy, to confirm they work before you need them under pressure)
- [ ] Monitoring timeframe noted; calendar reminder for the 2-week check

## Anti-patterns

- **All-at-once deploys** - ship honeypot + origin + rate-limit + Turnstile in one commit. Something breaks, you don't know which layer. Fix the process before the code.
- **No baseline** - hardening without a captured baseline means no objective rollback target. "I think it worked before" is not a rollback plan.
- **Enforce-first** - skipping log-only because "it's obviously safe." Log-only takes a day; rollback after an outage takes more. Budget the day.
- **Silent rollback** - rolling back without updating the findings doc or threat model. The next reviewer thinks the control is live when it isn't.
- **Threshold drift** - setting rate limit to 1 req/min "just to be safe," then never tuning it. Legitimate bursty users bounce. Real bots pace themselves anyway.

## Reference

- Standard: `Operations/.ops/standards/application-security.md`
- Companion skills: `public-endpoint-review`, `threat-model-walkthrough`, `publish-security`
- Cloudflare Rate Limiting API: https://developers.cloudflare.com/ruleset-engine/
- Cloudflare Turnstile: https://developers.cloudflare.com/turnstile/
- Wrangler deploy docs: https://developers.cloudflare.com/workers/wrangler/commands/#deploy
