Diagnostics: Explain a Failure · 25 min · free lesson

Bound the Retry Budget

Retry transient failures within attempt and time limits.

Core idea

A retry needs a transient failure, remaining attempts and a deadline budget.

Diagram for Bound the Retry Budget: a Rust code example flows to the lesson rule and the key observation.
The worked implementation exercises retry transient failures within attempt and time limits. Its tests demonstrate the stated finite contract.

Understand it

Retries amplify load when a dependency is already failing. Decide whether an error is transient, cap total attempts, and stop when the proposed delay plus one minimum operation budget no longer fits the deadline. Retrying authentication failures or malformed input does not repair them.

Here failed_attempts counts attempts already made, including the one that just failed. Retry only transient failures after at least one attempt, while the count is below the total maximum. Require delay + operation_budget to fit in remaining time, using checked arithmetic so an overflow cannot create a false large budget. Jitter and exponential delay selection are separate policies. Before retrying a write, use the idempotency contract from the HTTP stage; a timing budget does not prevent duplicated effects.

The route project is Task Ledger: first keep a reproducible source history, then expose its data through a CLI and HTTP boundary, persist it in SQL, diagnose failures and prepare a release. This lesson owns one contract in that project. Run the worked example before changing the starter and explain which input makes the tempting shortcut fail.

Terminal practice: Run python3 scripts/retry_probe.py against the desktop readiness endpoint. Stop the service and repeat; inspect the bounded attempt count and request ID. This probe retries an idempotent client GET, rather than automatically retrying service writes.

The compiler lab is independent of the reading question. Edit solution.rs; the app supplies a separate caller with normal, empty and adversarial inputs. A successful unchanged demonstration does not prove your implementation. After the lab, apply the same contract to the route companion project. Its README separates the on-device adapter from the desktop Cargo/TCP profile.

Read the code

pub fn retry(
    transient: bool,
    failed_attempts: u32,
    maximum: u32,
    delay: u64,
    operation_budget: u64,
    remaining: u64,
) -> bool {
    transient
        && failed_attempts > 0
        && failed_attempts < maximum
        && delay
            .checked_add(operation_budget)
            .is_some_and(|cost| cost <= remaining)
}

fn main() {
    assert_eq!(retry(true, 1, 3, 10, 20, 30), true, "case 1");
    assert_eq!(retry(true, 0, 3, 0, 0, 0), false, "case 2");
    assert_eq!(retry(false, 1, 3, 0, 0, 100), false, "case 3");
    assert_eq!(retry(true, 3, 3, 0, 0, 100), false, "case 4");
    assert_eq!(
        retry(true, 1, 3, u64::MAX, 1, u64::MAX),
        false,
        "case 5"
    );
    assert_eq!(retry(true, 1, 3, 10, 20, 29), false, "case 6");
    println!("worked example passed");
}

The worked implementation exercises retry transient failures within attempt and time limits. Its tests demonstrate the stated finite contract.

Practice starter

pub fn retry(transient: bool, failed_attempts: u32, maximum: u32, delay: u64, operation_budget: u64, remaining: u64) -> bool {
    todo!("implement the contract")
}

Open local example in Playground ↗

Try it

Return true only for a transient failure with 1<=failed_attempts<maximum and checked delay+operation_budget<=remaining. Overflow refuses the retry. Keep the public signature. Open the compiler lab and edit solution.rs; do not edit the private verification caller.

What success looks like: The separate caller accepts the changed solution and rejects unchanged TODO, a wrong implementation and a printed success message.

Check your understanding

What else is needed before retrying a possibly committed write?

Reveal the answer
  1. An idempotency policy
  2. Only a longer timeout
  3. Always retry forever until success

Timeouts and budgets do not prove the previous attempt had no external effect.

Go deeper

Common misconception: Only a longer timeout

A retry needs a transient failure, remaining attempts and a deadline budget.

  1. Identify the boundary: Retry transient failures within attempt and time limits. Write down its accepted inputs and observable output.
  2. Exercise the failure: Timeouts and budgets do not prove the previous attempt had no external effect.
  3. Check the independent caller: Return true only for a transient failure with 1<=failed_attempts<maximum and checked delay+operation_budget<=remaining. Overflow refuses the retry.

Transfer challenge: Apply this contract to Task Ledger, introduce the failure described in the lesson, and show the failing check before fixing it.

Connections

  • Measure the Tail, Not Just the Average: Compute an explicitly defined latency percentile
  • Make Checks Block Delivery: Require all declared release checks