Error Handling & Resilience

10.1 Normalized error shape

Whatever fails underneath, an internal bug, an external API failure, a validation failure, the API layer normalizes it into one consistent error shape and code system before it reaches the frontend. The frontend handles one contract, not a different shape per failure type.

One error envelope, regardless of source: a code, a human-readable message, and enough structured detail (e.g. which field failed validation) for the client to act on it without parsing free text.
Codes are stable and documented, not ad hoc strings invented per endpoint. A client should be able to branch on a code ("not_found", "validation_failed", "rate_limited") the same way regardless of which endpoint produced it.
Internal detail never leaks into the response. A stack trace, a database error message, or an internal file path is not part of the normalized shape, only a safe, stable code and message are. Log the internal detail (see 6.2); return the normalized shape.

Rule: an endpoint that returns its underlying error (a raw database error, a raw exception message) instead of the normalized shape is incomplete, not just imperfect, because it forces the frontend to special- case that endpoint instead of handling errors through the one shared contract.

10.2 UI-level error conventions

Error boundaries, not-found states, and loading/suspense patterns are standardized components used consistently across features, not implemented ad hoc per page.

Error boundaries catch rendering failures at a deliberate granularity (a feature/section boundary, not just one at the root of the app), so one broken widget doesn’t take down an entire page.
Not-found states are a distinct, designed UI state, not a blank page or a raw error message, used consistently for “this resource doesn’t exist” across every feature that can hit that case.
Loading and suspense states follow the same pattern established in 4.2’s PR checklist: every view that fetches or mutates data has a designed loading state, not a blank screen while data resolves.

Rule: a new feature does not introduce a bespoke error, not-found, or loading pattern when a standardized one already exists for that situation. If none of the existing patterns fit, that’s a gap in the shared components, add to them rather than building a one-off.

10.3 External API failure handling

This is the concrete mechanism behind the design-for-failure principle in 8.1: every external call has a timeout and a default failure behavior decided up front. This subsection defines how.

Retries use backoff, not a tight loop. A failed call to an external dependency retries with exponential backoff (and jitter, to avoid synchronized retry storms across concurrent requests), up to a bounded number of attempts, never retried indefinitely.
Circuit breaking for consistently failing dependencies. Once a dependency’s failure rate crosses a threshold, stop calling it for a cooldown window and fail fast instead, rather than letting every request pile up waiting on a dependency that’s already down. This is part of the required middleware baseline in 6.2.
Fallback behavior matches the choice made in 8.1: graceful degradation (serve stale/cached data) where that’s acceptable, a normalized hard failure (10.1) where it isn’t.

Rule: retry, backoff, and circuit-breaking logic lives in the typed provider wrapper for that dependency (see 7.4), not duplicated at every call site. A call site that needs different resilience behavior than the wrapper provides is a sign the wrapper needs a new method or option, not a local workaround.

10.4 Background and asynchronous work

Queue technology matches the actual need, not a default reached for regardless of workload:
- A Redis-backed queue or the current stack’s message queue (Azure Service Bus, see 2.2) for work that needs durability, retries, and ordering guarantees.
- A hosted/managed job runner (Azure Container Apps Jobs, see 2.2) for scheduled or event-driven work that doesn’t need a persistent queue in front of it.
- Azure Function Apps (see 2.2) for small, lightweight event-driven triggers, a blob upload triggering processing, a timer trigger, a small webhook receiver, where the work is simple enough that a full container is unnecessary overhead. Once that trigger’s logic grows into something with real dependencies and its own deployment lifecycle, move it to a proper service under Container Apps rather than letting it grow indefinitely inside a function.
- A simple cron job only for periodic maintenance work with no real-time trigger and no need for retry semantics beyond “it runs again next cycle.”
What belongs in a background job, not inline in a request: anything that takes more than roughly 1-2 seconds of work. This is the same threshold 6.3 applies when deciding a handler is a long-running operation.
Retry policy includes dead-letter handling. A job that keeps failing after its retry budget is exhausted moves to a dead-letter queue or equivalent for manual inspection, it does not retry forever, and it does not silently disappear.
Job handlers must be idempotent. A retried job is a normal occurrence, not an edge case, so every background job handler is safe to run twice. This is the same requirement as 8.7; it applies here because retries are exactly the mechanism that can invoke a handler more than once.

Rule: a background job without a dead-letter path or without idempotency is not production-ready, regardless of how reliably it happens to succeed today. Both are review blockers, not follow-up tickets, for the same reason idempotency is a blocker in 8.7: the failure mode only shows up under retry, which means it shows up in production, not in normal testing.

10.5 Timeout budget propagation

8.1 requires every external call to have a timeout. This subsection covers what that means across a chain of calls, a handler that calls one service, which calls another, rather than a single, isolated call.

A downstream timeout fits within the caller’s remaining budget, not a fresh, independent duration. If a handler has a 5-second overall budget and it’s already spent 2 seconds before calling a downstream service, that downstream call gets a timeout of roughly the remaining 3 seconds, not its own independent 5-second timeout stacked on top. A chain of calls each independently set to the same full duration can let the total time genuinely exceed what the original caller (or user) is actually willing to wait for, defeating the point of having set a timeout in the first place.
Propagate the remaining budget, not just a fixed per-call constant. The concrete mechanism is passing the deadline (an absolute time, or the remaining duration) into each downstream call, through the same typed wrapper pattern that already owns retry and circuit-breaking behavior (see 7.4), so a caller doesn’t have to manually recompute the remaining budget at every call site.
A request that’s already past its budget doesn’t start a new downstream call at all. If the remaining budget is at or near zero by the time a handler would make another downstream call, fail fast with the normalized error shape (per 10.1) rather than starting a call that’s already certain to be cut off partway through or exceed the overall budget regardless of what timeout you give it.

Rule: a service that calls another service, which itself calls a further service, needs a stated answer for how the timeout budget is divided across that chain before it ships. A chain of independently timed calls with no shared budget is exactly how a single slow request degrades into a much longer one than any individual timeout suggests.

10.6 Bulkheading and failure isolation

This is a different concern from circuit breaking (10.3): circuit breaking stops calling a dependency once it’s clearly failing. Bulkheading is about containing the resource cost of a dependency that’s slow but not outright failing, so it can’t exhaust a shared resource pool and starve requests that don’t even depend on it.

A slow dependency shouldn’t be able to consume every available connection, thread, or concurrency slot. If every in-flight request is waiting on the same slow external call, and that call holds a connection or a worker slot for its full (long) duration, an unrelated request that doesn’t call that dependency at all can still be starved simply because there’s no capacity left to serve it.
Give a slow or unreliable dependency its own bounded resource pool (a separate connection pool, a capped number of concurrent in-flight calls to it) rather than sharing the same pool every dependency draws from. This is the same isolation principle as the per-database connection pool already required in 6.6, applied to any dependency with a real risk of becoming a bottleneck, not just the database.
This matters most for dependencies with a history of being slow or unreliable, not applied uniformly to every external call regardless of risk; a fast, reliable, low-volume dependency doesn’t need its own dedicated resource pool, the cost of that isolation should match the actual risk the dependency poses.

Rule: a dependency identified as unreliable or prone to latency spikes (from an incident, from the circuit-breaking history in 10.3) gets a bounded, isolated resource allocation before it’s allowed to keep sharing a pool with everything else, an unreliable dependency sharing an unbounded resource pool is a known, recurring cause of an unrelated feature going down during an unrelated dependency’s incident.

10.7 User-facing error message quality

10.1 defines the error envelope’s shape (a code, a message, structured detail). This subsection defines what makes the message itself good, once you’re writing the actual text a user sees.

State what happened and what the user can do about it, not just that something failed. “Something went wrong” tells the user nothing actionable; “We couldn’t save your changes because the connection was lost. Try again” tells them what happened and gives them a next step. Where there’s a genuine next step (retry, check a specific field, try again later), state it.
No technical internals in user-facing text. A stack trace, an internal error code meant for engineers, a raw exception message, or a database error string does not appear in what the user sees, this is the user-facing side of the internal-detail-never-leaks rule already stated in 10.1; that rule covers the response payload, this covers what’s actually rendered from it.
One consistent tone and vocabulary across the product. Error copy for the same class of problem (a validation failure, a permissions failure, a not-found) reads consistently everywhere it appears, per the centralized-copy convention in 5.7, not reworded slightly differently by whoever happened to write that particular feature.
Match specificity to what the user can actually act on. A validation error names the specific field and what’s wrong with it, since the user can fix that. A downstream failure the user has no control over (an external provider being down) doesn’t need to name the provider, “the payment couldn’t be processed, please try again shortly” is more useful than exposing an internal dependency name the user has no context for.

Rule: an error message that only says an action “failed” with no indication of why or what to do next is incomplete, the same standard this document already applies to a missing error state in the 4.2 checklist: having an error state at all is necessary but not sufficient if the message inside it doesn’t actually help the user.