# Retry Handling, Backoff, and Timeout Management

This reference covers the complete retry and timeout strategy for Airtable automation scripts. These patterns were developed through production experience with concurrent automations hitting shared API rate limits.

---

## When to Retry vs. Bail

Not every failure deserves a retry. Retries consume fetch() budget, burn clock, and can make cascading failures worse.

**Always retry:**
- HTTP 429 (rate limit) or API-specific rate limit codes (Marketo 606, 615)
- HTTP 500, 502, 503, 504 (server errors, usually transient)
- Network timeouts or connection resets
- HTTP 408 (request timeout)

**Never retry:**
- HTTP 400 (bad request) with a clear validation error
- HTTP 401/403 after a token refresh has already been attempted
- API-specific auth failures (Marketo 601, 602) after one refresh attempt
- HTTP 404 (resource not found)
- API-specific "method not supported" (Marketo 605)
- API-specific "daily quota exhausted" (Marketo 607)

**Retry once, then bail:**
- Auth failures on first encounter (attempt token refresh, retry, then give up)
- Malformed response (non-JSON, unexpected structure) - could be a transient proxy issue

---

## Backoff Strategies

### Why Backoff Matters

Without backoff, a retry fires immediately. If the failure was a rate limit, the immediate retry hits the same limit. With multiple concurrent scripts, this creates a thundering herd: every script retries at the same instant, all get rate-limited again, all retry again.

### Exponential Backoff (Baseline)

Delay doubles with each attempt:

```
Attempt 1: 2s
Attempt 2: 4s
Attempt 3: 8s
```

Formula: `delay = BASE_DELAY * Math.pow(2, attempt)`

Problem: deterministic. If two scripts fail at the same time, they retry at the same intervals.

### Full Jitter

Randomize the entire delay:

```
Attempt 1: random(0, 2s)
Attempt 2: random(0, 4s)
Attempt 3: random(0, 8s)
```

Formula: `delay = Math.random() * BASE_DELAY * Math.pow(2, attempt)`

Maximum spread, but can occasionally retry very fast (close to 0ms), which defeats the purpose when the root cause is a rate limit.

### Equal Jitter (Recommended)

Guarantee at least half the backoff, randomize the rest:

```
Attempt 1: random(1s, 2s)
Attempt 2: random(2s, 4s)
Attempt 3: random(4s, 8s)
```

Formula:
```javascript
var maxDelay = BASE_DELAY * Math.pow(2, attempt);
var half = maxDelay / 2;
var delay = half + Math.random() * half;
```

**This is the recommended strategy for Airtable scripts** because:
- The minimum delay (half) ensures you always wait a meaningful amount
- The random component prevents stampeding across concurrent automations
- The maximum delay stays bounded (no runaway waits)
- It is simple to implement and reason about

### Decorrelated Jitter

Each retry bases its delay on the previous delay, not the attempt count:

```javascript
var delay = Math.min(MAX_DELAY, Math.random() * previousDelay * 3);
```

Good for long retry chains, but overkill when you only retry 3 times. Skip this for Airtable scripts.

---

## Time-Budget-Aware Retries

The standard retry pattern ignores a critical constraint: Airtable scripts have 180 seconds total. A retry that waits 8 seconds is useless if there are only 6 seconds left.

### The Decision Framework

Before each retry wait, check whether the wait plus the expected operation time fits within the remaining budget:

```javascript
function shouldRetry(attempt, maxRetries, estimatedOperationMs) {
  if (attempt >= maxRetries) return false;

  var maxDelay = BASE_DELAY * Math.pow(2, attempt);
  var half = maxDelay / 2;
  var worstCaseWait = maxDelay;  // use max, not jittered, for planning

  var needed = worstCaseWait + estimatedOperationMs + 5000;  // 5s safety buffer
  if (timeRemaining() < needed) {
    console.log("Skipping retry " + (attempt + 1) + ": need " + needed + "ms, have " + timeRemaining() + "ms");
    return false;
  }
  return true;
}
```

### Full Retry Helper (Production Pattern)

This combines equal jitter, time-budget awareness, error classification, and fetch counting:

```javascript
var BASE_DELAY_MS = 2000;
var MAX_RETRIES = 3;

async function fetchWithRetry(url, headers, label) {
  label = label || url.slice(0, 80);

  for (var attempt = 0; attempt < MAX_RETRIES; attempt++) {
    // Time budget check before attempt
    if (timeRemaining() < 8000) {
      console.log("[" + elapsed() + "s] Time budget too low for " + label + " (" + timeRemaining() + "ms remaining)");
      throw new Error("Time budget exhausted before " + label);
    }

    try {
      var res = await trackedFetch(url, { headers: headers });
      var raw = await res.text();

      console.log("[" + elapsed() + "s] " + label + " HTTP " + res.status + " (" + raw.length + " chars)");

      var json;
      try {
        json = JSON.parse(raw);
      } catch (e) {
        throw new Error("Non-JSON from " + label + " (HTTP " + res.status + "): " + raw.slice(0, 200));
      }

      // API-level success check (Marketo pattern)
      if (json.success === false) {
        var errCode = (json.errors && json.errors[0] && json.errors[0].code) || "";
        var errMsg = (json.errors && json.errors[0] && json.errors[0].message) || "Unknown";

        // Non-retryable errors
        if (errCode === "601" || errCode === "602") {
          throw new Error("Auth failure (" + errCode + "): " + errMsg);
        }
        if (errCode === "605") {
          throw new Error("Method not supported: " + errMsg);
        }
        if (errCode === "607") {
          throw new Error("Daily quota exhausted: " + errMsg);
        }

        // Retryable errors
        if (errCode === "606" || errCode === "615") {
          if (!shouldRetry(attempt, MAX_RETRIES, 3000)) {
            throw new Error("Rate limited (" + errCode + ") with insufficient time to retry");
          }
          var half = BASE_DELAY_MS * Math.pow(2, attempt) / 2;
          var jitter = half + Math.random() * half;
          console.log("[" + elapsed() + "s] " + label + " rate limited (" + errCode + "), waiting " + Math.round(jitter) + "ms (attempt " + (attempt + 1) + "/" + MAX_RETRIES + ")");
          busyWait(jitter);
          continue;
        }

        throw new Error("API error (" + errCode + "): " + errMsg);
      }

      return json;  // success

    } catch (err) {
      // Network or parse errors
      if (attempt < MAX_RETRIES - 1 && shouldRetry(attempt, MAX_RETRIES, 3000)) {
        var half = BASE_DELAY_MS * Math.pow(2, attempt) / 2;
        var jitter = half + Math.random() * half;
        console.log("[" + elapsed() + "s] " + label + " attempt " + (attempt + 1) + "/" + MAX_RETRIES + ": " + err.message);
        console.log("   Retrying in " + Math.round(jitter) + "ms");
        busyWait(jitter);
      } else {
        throw err;
      }
    }
  }

  throw new Error("Exhausted " + MAX_RETRIES + " retries for " + label);
}
```

---

## Timeout Management

### The 180-Second Ceiling

Airtable kills the script at 180 seconds. There is no warning, no graceful shutdown hook, no final callback. Any in-flight `fetch()` call is aborted. Any in-progress Airtable write is discarded. The automation history shows "Script timed out."

### The 170s / 10s Split

Reserve the last 10 seconds for cleanup:

```javascript
var TIME_LIMIT_MS = 170000;  // stop new work at 170s
var RUN_START = Date.now();

function timeRemaining() {
  return TIME_LIMIT_MS - (Date.now() - RUN_START);
}
```

The 10-second buffer covers:
- One Airtable write to record final status (typically 1-3s)
- One Airtable write to clear the trigger flag (1-3s)
- Console log output for the run summary
- Network latency variance

### Dynamic Cost Estimation

Before starting an expensive operation, estimate how long it will take and compare against the remaining budget:

```javascript
// Before a batch write
var ESTIMATED_WRITE_MS_PER_RECORD = 50;
var batchCost = batch.length * ESTIMATED_WRITE_MS_PER_RECORD;
if (timeRemaining() < batchCost + 5000) {
  console.log("Skipping batch write: need ~" + batchCost + "ms, have " + timeRemaining() + "ms");
  break;
}

// Before an API page fetch
var ESTIMATED_FETCH_MS = 3000;  // conservative for a single API call
if (timeRemaining() < ESTIMATED_FETCH_MS + 10000) {
  console.log("Stopping pagination: insufficient time for another page");
  break;
}

// Before starting a new day in a multi-day scan
var ESTIMATED_DAY_SCAN_MS = 30000;  // based on observed averages
if (timeRemaining() < ESTIMATED_DAY_SCAN_MS) {
  console.log("Stopping: not enough time for another day scan");
  break;
}
```

Estimates should be conservative (assume the worst case). Underestimating leads to timeouts. Overestimating just means the script stops a bit early, which is fine because the chained automation or next re-trigger picks up where it left off.

### Timeout in Multi-Step Pipelines

Each scripting step in an automation gets its own 180-second budget. A 4-step automation chain gets 4 x 180s = 720s total. But the steps run sequentially, not in parallel. If step 1 times out, step 2 does not run (unless step 1 handles the timeout gracefully and sets outputs).

**The graceful timeout pattern:**

```javascript
var timedOut = false;

for (var i = startOffset; i < items.length; i++) {
  if (timeRemaining() < 15000) {
    console.log("Time budget reached at index " + i);
    timedOut = true;
    break;
  }
  // ... process item ...
}

// Always set outputs, even on timeout
output.set("exitReason", timedOut ? "timeout" : "completed");
output.set("lastOffset", i);
output.set("processedCount", i - startOffset);
```

The next script in the chain reads `exitReason` and `lastOffset` to decide whether to continue.

### Timeout vs. Fetch Limit Interaction

A script can hit either limit first. The management pattern handles both:

```javascript
for (var page = 0; page < MAX_PAGES; page++) {
  // Check both limits
  if (timeRemaining() < 15000) {
    exitReason = "timeout";
    break;
  }
  if (fetchCount >= FETCH_LIMIT - 2) {  // reserve 2 for cleanup
    exitReason = "fetchLimit";
    break;
  }

  var data = await trackedFetchWithRetry(url);
  // ... process ...
}
```

Reserve 1-2 fetch calls for cleanup writes (status field updates, trigger flag clears) when approaching the 50-call limit.

---

## Concurrent Automation Stampeding

When Airtable triggers the same automation for 20 records simultaneously, all 20 scripts start at roughly the same time. If they all hit a rate limit at the same time, deterministic backoff means they all retry at the same time, creating a cycle.

**Equal jitter breaks this cycle** by spreading retries across a random window. With 20 scripts and 2s base delay:
- Deterministic: all 20 retry at exactly 2s, 4s, 8s
- Equal jitter: retries spread across 1-2s, 2-4s, 4-8s per script

### Pre-Call Governance Delay

For scripts that run in large batches (20+ concurrent), add a small random delay before the first API call to stagger the initial burst:

```javascript
var PRE_CALL_DELAY = Math.floor(Math.random() * 2000);  // 0-2s
console.log("Pre-call governance delay: " + PRE_CALL_DELAY + "ms");
busyWait(PRE_CALL_DELAY);
```

This costs 0-2 seconds of budget but prevents 20 simultaneous first calls from immediately triggering rate limits.

### Cross-Automation Awareness

Airtable does not provide a way for one automation to know that other automations are running. You cannot check "how many scripts are active right now." The only signals are:
- Rate limit errors from the external API (reactive)
- System Panel status fields showing "In Progress" for other syncs (requires a read at script start)
- Automation scheduling to avoid overlaps (preventive)

The most reliable approach: design each script to handle rate limits gracefully via jitter, and avoid scheduling overlapping automation triggers against the same external API.
