Backend & Data Architecture

6.1 Service and request structure

Regardless of which communication paradigm handles a given request (see 7.2), every backend request passes through the same three internal layers, matching the layer responsibilities defined in 3.2:

Handler (procedure/controller/route). The entry point for a request. Its only job is to authenticate, validate input against the shared schema (see 9.3), call the relevant domain service, and map the result to a response shape (see 6.8 below). A handler contains no business logic, if it’s making a decision more complex than “call this service and shape the response,” that decision belongs in the domain service layer instead.
Domain service. The business logic itself, framework-agnostic and callable the same way whether it’s invoked from a handler, a background job (see 10.4), or a test. This is where the design patterns and dependency injection conventions in Section 8 apply.
Data access. The only layer that talks to a database or external provider directly (see 6.6 and 7.4).

Rule: if a handler function is more than a few lines, or you can’t describe what it does without the word “and,” that’s a sign business logic has leaked into the handler. Move it into a domain service; the handler’s job is routing and shaping, not deciding.

Logging and observability baseline. This states what a backend service must log, not which tool it uses, see 2.2 for the current logging/tracing stack.

Log at the handler and domain-service boundary, not inside every low-level function, enough to reconstruct what happened during an incident without flooding logs with noise.
Every log line for a request carries a correlation/request ID (see 11.5) so a single request can be traced across service calls.
Never log secrets, tokens, or full request bodies containing sensitive data, see 11.4. A logging statement is a place a secret can leak just as easily as a code path can.

6.2 Standard middleware and observability format

Every backend service is required to have the following in place before it handles production traffic. This is a baseline, not a wish list, a service missing one of these is not production-ready regardless of how complete its business logic is.

Required middleware stack:

Authentication. Session/token verification, centralized in middleware, not duplicated per route or procedure. See 11.2 for where this is enforced.
Rate limiting. Required on sensitive operations (login, OTP, search, and any other operation named in 11.5), owned by the engineering team even where infra-level protection also exists.
Circuit breaking. Required around calls to external dependencies that can fail or degrade (the typed provider wrappers from 7.4), so a failing dependency doesn’t cascade into the whole service failing. See 10.3 for the retry/backoff/circuit-breaking convention this pairs with.
Request logging. Every request logged with method, path, status code, and duration at minimum, see the observability format below for the required shape.
Request ID / correlation ID. Assigned or propagated per request (from an inbound header if present, generated otherwise), bound to every log line and trace for that request. See 11.5.
CORS and request size limits. See 11.5 for the specific policy; every service enforces both, not just the ones that serve a browser client directly.

Rule: these middlewares are registered once, centrally, at service startup, in the same order across services (request ID and tracing first, so every later middleware and every log line in the request has a trace/request ID to attach to; auth and rate limiting before any handler code runs). A service should never leave one of these to be opted into per route.

Observability format standard. The current standard, provided by the AI/ML team’s “Heimdall” observability package, combines OpenTelemetry, structured logging, and Azure Monitor. A service adopting this standard must satisfy the following, regardless of which language/framework implementation it uses:

Structured logs, not plain strings. Every log line is a structured event (a JSON object in production, e.g. {"event": "item_created", "item_id": "...", "level": "info", "timestamp": "..."}), not a free-text message. This is what makes logs queryable instead of just readable.
Every log line carries trace_id and span_id from the active OpenTelemetry span, so any log line can be pivoted to its parent trace. This is the log-trace correlation the observability stack exists to provide, don’t disable it to save a few lines of setup.
Every request carries a request ID, read from an inbound X-Request-Id header when present or generated otherwise, bound to the log context and echoed back in the response headers.
One server span per request, named {METHOD} {path} (e.g. POST /v1/items), opened as early in the middleware chain as possible so it wraps the entire request lifecycle, and correctly parented to an upstream span when a W3C traceparent header is present.
Secrets and full request bodies are never logged, per 11.4, regardless of how convenient it is for local debugging, this must hold in every environment including local.
Noisy framework-level loggers are quieted deliberately (e.g. suppressing duplicate access logs from a framework’s own request logging once the standard middleware is doing it), not left to spam output alongside the structured logs.

Rule: if a new service can’t yet adopt the Heimdall package directly (e.g. a different language runtime), it must still produce the same observable contract, structured JSON logs, trace/span correlation IDs, one server span per request, so that Azure Monitor and any log queries work the same way regardless of which service emitted the data.

6.3 API contract conventions

These conventions apply at the handler layer (6.1) regardless of which communication paradigm owns the endpoint (see 7.2).

Contract-first, not implementation-first. Define the request/response schema (the Zod schema for a tRPC procedure, or the request/response DTO for a REST endpoint, see 9.3) before writing the handler body. The schema is the source of truth for what the endpoint accepts and returns; the implementation is built to satisfy it, not the other way around.

Writing the schema first forces the shape of the contract to be decided deliberately, instead of being whatever the implementation happened to produce.
A schema change is a visible, reviewable diff on its own line in the PR, making a breaking change to a contract obvious in review rather than buried inside handler logic.
This is what keeps 7.7’s additive-only versioning rule enforceable in practice: a reviewer can see the schema diff and ask “is this additive?” directly.

Pagination on every list endpoint. Any endpoint returning a collection accepts cursor-based pagination parameters (see 9.4 for why cursor over offset) and enforces a maximum page size on the server, regardless of what the client requests.

A client-requested page size above the server maximum is clamped to the maximum, not rejected and not honored as-is, an unbounded or excessive page size is a resource-exhaustion risk, not just a performance nuisance.
An endpoint returning a collection with no pagination at all is not acceptable once that collection can grow without bound, decide pagination at design time, not after the first slow-query incident.

Long-running operations do not block the request/response cycle. If an operation can’t reliably complete within a normal request timeout (report generation, bulk import/export, any multi-step processing job), it is not run inline in the handler:

The handler enqueues the work (see 10.4 for background job criteria) and returns immediately, 202 Accepted with a job ID the client can poll, or a webhook/notification callback when the work completes.
The actual work runs as a worker in Azure Container Apps Jobs (see 2.2), not as a long-held connection or a background setTimeout/thread inside the request-serving process.
A true fire-and-forget operation (the caller doesn’t need the result at all, only confirmation the operation was accepted) returns 202 Accepted or 201 Created immediately, with no polling contract needed, only once it’s genuinely fire-and-forget; if the caller will ever need the outcome, use the job ID + polling/webhook pattern above instead of pretending it’s fire-and-forget.

Rule: a handler that can take more than a couple of seconds under normal conditions is a long-running operation by definition, and must follow the pattern above rather than being left inline “because it’s usually fast.” Ties back to 10.4’s ~1-2 second threshold for what belongs in a background job.

Bulk and batch operations report per-item outcomes. An endpoint that accepts multiple items in one request (a bulk create, update, or import) is not all-or-nothing by default:

Each item in the batch succeeds or fails independently, and the response reports a per-item result (e.g. {id, status, error?} per item), not a single overall success/failure for the whole batch. A caller submitting 100 records should be able to tell which 3 failed without resubmitting the other 97.
Only require all-or-nothing behavior when the operation is genuinely a single logical transaction (see the transaction boundary rule in 6.6 below), state that explicitly in the endpoint’s contract (6.3 above) so callers know which behavior to expect, don’t leave it to be discovered by trial and error.
Batch endpoints are still subject to a maximum batch size, enforced on the server, for the same resource-exhaustion reason as the pagination limit above.

API documentation is attached to the contract, not written separately. Because the schema is the source of truth (contract-first, above), document the endpoint at the schema, procedures and Zod schemas carry field-level descriptions inline, and REST endpoints intended for external/partner consumption (see 7.2) generate their documentation, including OpenAPI/Swagger output, from that same schema rather than a hand-maintained document that can drift out of sync with the actual contract.

Generated Swagger/OpenAPI output is only as good as the descriptions in the schema it’s generated from, a field with no description in the Zod schema produces a field with no description in the generated docs. Add the description at the source, not by hand-editing generated output, hand-edits to generated documentation are overwritten on the next generation and drift out of sync immediately.
All schema descriptions, generated API documentation, and API response content (error messages, field descriptions, log messages surfaced through the API) are written in English, per 4.1. This applies whether the consumer is another internal service or an external partner.

Rule: if an endpoint’s behavior can’t be understood from its schema plus its inline descriptions, the schema is under-specified, add the missing description or field rather than compensating with an external document only some readers will find.

6.4 Health checks

Every service exposes two endpoints for platform health monitoring, distinct in what they check:

Liveness (/health): answers “is this process running and able to respond at all.” It does not check downstream dependencies, a database outage should not make a service report itself as dead when the process itself is healthy, that would cause the platform to needlessly restart a service that can’t fix the underlying problem by restarting.
Readiness (/ready): answers “is this service ready to serve real traffic,” and does check its critical dependencies (database connectivity, required external services). A service that’s up but can’t reach its database should report not-ready, so traffic can be routed away from it or a rollout can be held.

Rule: both endpoints are unauthenticated (platform health checks don’t carry credentials) but reveal nothing about internal state beyond a status, no stack traces, no dependency connection strings, no version details that aren’t already public. A health check endpoint is still a publicly reachable endpoint and follows the same information-disclosure discipline as any other (see Section 11).

6.5 Service boundary criteria

A separate service is justified by a concrete operational reason, not by popularity. Splitting a monolith into services adds real cost, network calls, deployment coordination, distributed failure modes, so the split must pay for that cost. Split out a service only when at least one of these is true:

Independent scaling need. The component has a load profile meaningfully different from the rest of the system (e.g. a CPU-intensive processing job that would otherwise force the whole application to scale to match a workload only it needs).
Distinct ownership. A different team owns it end to end, with its own release cadence and on-call responsibility, and coupling its deploys to the rest of the system would create cross-team blocking.
Different reliability requirements. The component has a meaningfully different availability or latency requirement than the rest of the system (e.g. a best-effort batch job vs a user-facing request path), and bundling them means the stricter requirement forces unnecessary operational overhead onto the looser one, or vice versa.

Rule: “microservices are the modern way to build software” is not a valid justification on its own. If none of the three criteria above apply, keep the code in the existing service, a well-structured monolith (see 3.2 on layer responsibility) is not technical debt, it is the correct default until one of these criteria is met.

6.6 Database and persistence standards

SQL vs NoSQL decision criteria. Choose based on the actual shape and access pattern of the data, not familiarity or trend:

Favor a relational (SQL) store when the data has strong, stable relationships that need to be queried in combination, when multi-record transactional consistency is a hard requirement (e.g. financial records), or when the schema is well-understood and unlikely to churn per feature.
Favor a document (NoSQL) store when the data is naturally hierarchical or varies in shape between records, when the dominant access pattern is “fetch this whole object,” and when schema iteration speed matters more than enforcing a fixed shape at the database layer.

Current answer: MongoDB with Mongoose is the approved default (see 2.2), chosen for iteration speed and fit with our current domain shapes. A service whose data is genuinely relational and transactional in nature is not required to force that data into a document model just to match the default, propose the exception per the process in 2.5.

Analytical queries over large files are a separate case, not an exception to the rule above. Running SQL queries against a large Excel/spreadsheet file (DuckDB with Parquet, see 2.2) is not an application’s primary data store, it’s a query engine for a specific analytical or reporting need. Reach for it when the task is “query this large file efficiently,” not as a general-purpose alternative to MongoDB for a service’s actual persisted data.

Migration tooling and review process.

Every schema change ships as a migration file, not an ad hoc script run once against production. A migration is code, reviewed like any other PR.
A migration PR must state, explicitly: what changes, whether it is backward compatible with the currently deployed application version, and the rollback plan if it fails partway.
A migration that is not backward compatible with the previous application version must follow the zero-downtime pattern in 7.8 (expand, migrate, contract) rather than changing the schema and the code in one atomic step.

Connection pooling. Every service maintains one connection pool per database, created once at boot and reused across requests, never a new connection per request. Pool size is a config value (see 6.10 below), sized to the service’s expected concurrency, not left at a library default without consideration.

Transaction boundaries. A transaction should wrap exactly the set of writes that must succeed or fail together as one logical operation, no more and no less. Don’t wrap unrelated writes in one transaction for convenience, and don’t split a single logical operation across multiple transactions, that reintroduces the partial-failure problem the transaction exists to prevent.

Denormalization criteria. Denormalize (duplicate data across records to avoid a join or a cross-document fetch) only when a measured read pattern requires it, not preemptively. Denormalized data needs an explicit consistency story, either it’s acceptable for it to go briefly stale, or there’s a defined process (a background job, an event handler) that keeps it in sync. Denormalizing without deciding that story is how data silently drifts out of sync in a way nobody notices until support questions come in.

N+1 query prevention. Never issue a database query inside a loop over a collection of parent records to fetch each one’s related data one-by-one. This is an authoring rule, not just a performance guideline (see 12 for the broader performance treatment):

Batch-load related data with a single query (an IN query keyed on the collected parent IDs, or the ORM/query-builder’s equivalent), then join the result back onto the parent records in memory.
If the same batch-loading need recurs across multiple call sites (e.g. loading a User for many different record types), centralize it as a reusable loader, don’t reimplement the same batching logic per feature.
Treat a query issued inside a loop as a review blocker, the same way an unhandled promise rejection would be, not a style preference to fix later, because the cost of an N+1 query scales with data volume and tends to only surface as a production incident once real data volume arrives.

6.7 Query standards

This subsection is the concrete “how you write a query” companion to 6.6’s persistence standards.

Queries live in the data access layer, nowhere else. Per 6.1, the data access layer is the only layer allowed to talk to the database directly. A query string, an aggregation pipeline, or a raw driver call does not appear inside a domain service or a handler, it’s wrapped in a named data access function (findActiveUsersInTenant, not an inline query at the call site).

Default to the ORM/query-builder’s typed methods; a raw query is the exception, not the starting point. Mongoose’s typed query methods (or the equivalent for a documented relational exception, see 6.6) are the default because they preserve the type-safety criterion from 2.1. Drop to a raw query or raw aggregation pipeline only when the typed API genuinely cannot express what’s needed, and say why in the PR, a raw query without that justification is a review flag, not a style preference.

Every query has a timeout. This is the database-specific case of the general rule in 8.1: every external dependency call, including a database query, has a bound on how long it can run. An unbounded query can hang a request indefinitely the same way an unbounded external API call can.

Never build a query by string concatenation or interpolation. All query inputs are parameterized or passed through the ORM/query-builder’s own parameter binding, never interpolated directly into a query string or a $where-style expression. This is the query-writing side of the input validation and sanitization requirement in 11.3: a parameterized query is what actually prevents injection, validation alone is not a substitute for it.

Reused filters are defined once, as a composable fragment, not copy-pasted per query. A filter that recurs across multiple queries (tenant scoping, “active records only,” a common date-range bound) is defined once, e.g. as a shared aggregation stage or query-builder helper, and reused, rather than hand-written again at every call site where it’s needed. This matters most for tenant scoping specifically: see 7.6, which requires this enforcement to be structural, not left to a developer remembering to repeat a filter correctly every time.

Break a complex query into named, readable stages, not one dense expression. For MongoDB aggregation pipelines, this means staging the pipeline into named, single-purpose steps (a $match stage, a $lookup stage, a $group stage) that can be read and reasoned about individually, the same benefit a CTE (WITH clause) provides in a relational database. For the documented relational exception in 6.6, prefer CTEs over deeply nested subqueries for the same reason: a query that reads top-to-bottom as a sequence of named steps is reviewable; a deeply nested one is not. This is suggested by Zalman Faresi D of Data & Prompt Engineer Team.

Check the query plan before shipping a new query against a table expected to grow. For a query that will run frequently or against a table with real (or realistically growing) volume, check its execution plan (confirm it uses the intended index, per 12.4) as part of writing the query, not after it shows up in a slow-query log once the table has grown large enough to matter.

Rule: a PR introducing a new query against a table expected to grow states, briefly, which index it relies on and that the query is parameterized. A query without either isn’t ready to merge.

6.8 Data modeling

Distinguish three shapes explicitly. Conflating them is a common source of leaky abstractions:

Domain model: the shape business logic actually operates on, expressed in the language of the business (e.g. an Order with a computed total, a status state machine). This shape exists in code and may not map one-to-one with anything stored or transmitted.
DB model: the shape actually persisted (a Mongoose schema, a SQL table). It’s allowed to differ from the domain model, e.g. storing a denormalized field the domain model computes on the fly.
API DTO/schema: the shape sent to or received from a client. It’s allowed to be a subset, a reshaped version, or a combination of domain models, whatever the consumer actually needs, and is never the same object as the DB model passed through directly. See 9.3 for the shared-schema (Zod) convention that governs this layer; this subsection defines what it is, 9.3 defines how it’s validated and shared between client and server.

Rule: a data access function returns a domain model, not a raw DB document. An API handler maps a domain model to a DTO before it leaves the server. Never let a Mongoose document (or equivalent) or a raw database row serialize directly into an API response, that couples your public contract to your storage implementation and makes either one hard to change without breaking the other.

6.9 Standard table columns

Every table (or collection, for a document store) includes this set of columns, regardless of what domain data it otherwise holds. This standard is credited to Adnan Abdul Malik of Operation Maintenance Lead. Column names follow the same camelCase convention as any other field name (see 3.3), consistent with how Mongoose schemas are conventionally defined.

id: the primary identifier for the record.
recordStatus: the record’s lifecycle state (A and D or Active and Deactive). Use this for soft deletes and status transitions instead of physically deleting a row, so history and referential integrity are preserved; see 8.1 for when a hard failure vs a soft, recoverable state is the correct choice.
changeNo: a version/change counter for the record, incremented on every update. Use this for optimistic concurrency control, a write can check it hasn’t changed since it was read, rather than silently overwriting a concurrent update.
createDate: when the record was created.
createBy: who (or what process) created the record.
changeDate: when the record was last modified.
changeBy: who (or what process) last modified the record.

Rule: these columns are populated by the data access layer automatically, not left to individual feature code to remember to set on every write. A new table missing one of these columns is a schema review blocker, not a follow-up ticket, since these columns are what makes an audit trail and concurrency control possible after the fact, they can’t be reconstructed retroactively once data has already been written without them.

6.10 Configuration management

Configuration varies per environment (local, staging, production) and must never be hardcoded into application logic.

All config is read from environment variables (or an equivalent environment-scoped source), never hardcoded, and never committed to the repository, see 11.1 for the public/server-only boundary this implies.
Configuration is validated at boot, not at first use. Parse and validate all required environment variables when the service starts, using the same schema library as the rest of the stack (Zod, see 9.3). A service with a missing or malformed required config value should fail to start with a clear error, not start successfully and fail later under production load when that value is first read.

Rule: if a misconfiguration can only be discovered by a runtime error in a code path that isn’t hit until traffic arrives, that’s a configuration validation gap. Boot-time validation exists specifically so misconfiguration is a deploy-time failure, not a production incident.

6.11 Parameterize operational values, don’t hardcode them

6.10 covers environment-specific config (connection strings, secrets, per-environment values). This subsection covers a different, equally common source of technical debt: an operational or business value, a limit, a threshold, a retry count, a timeout duration, a page size, written as a literal number or string directly in the code that uses it, instead of named, centralized, and adjustable.

A value that could reasonably change is a named constant, a config value, or a database-backed setting, not a literal. A magic number buried in a conditional (if (items.length > 47)) forces the next engineer to guess whether 47 is meaningful or arbitrary, and to hunt down every place it’s duplicated if it needs to change. Give it a name that states what it is (MAX_BATCH_SIZE), in one place.
Match the mechanism to how often the value actually needs to change, the same tiered thinking as feature flags in 8.9: a value that’s fixed at deploy time is a named constant or environment config (6.10); a value an operator needs to tune without a redeploy (a rate limit threshold, a batch size under active tuning) is database-backed config, following the same DB-backed tier already defined for flags.
Don’t duplicate the same value at multiple call sites. A limit that’s genuinely used in more than one place (a max page size enforced both at the API layer and in a background job) is defined once and imported, not restated as the same literal in each location, where it will silently drift the first time only one of the copies gets updated.
This is a modularity and reuse concern as much as a configuration one. A function or module that hardcodes an assumption specific to one caller (a provider name, a fixed limit, a specific collection name) isn’t actually reusable even if its signature looks generic; parameterize what varies by caller and pass it in, rather than encoding one caller’s specifics into shared code.

Rule: a PR introducing a new numeric or string literal that represents a limit, threshold, timeout, or other operationally meaningful value, with no name and no clear reason it can never change, is a review flag. The question to ask is not “does this work today” but “will the next person who needs to change this know where to find it,” if the answer is no, it’s hardcoded technical debt, not a finished implementation.

6.12 Service-to-service authentication

Centralized authentication (see 11.2) covers a user-facing client calling a service. This subsection covers a different boundary: one internal service calling another, where there’s no end user in the request at all. A call to an external, third-party provider is a different case again, see 7.4 for the typed wrapper pattern that applies there instead.

Current mechanism: a shared service token. A service calling another internal service authenticates with a service-issued token (distinct from a user’s session/access token), verified by the receiving service the same way any other credential is verified, centrally, not with an ad hoc per-route check (per the same principle as 11.2).
A service token identifies the calling service, not a user. Don’t reuse or forward a user’s token to authenticate a service-to-service call that happens on that user’s behalf, the calling service authenticates as itself. If the receiving service needs to know which user originated the request, pass that as data in the request (and authorize against it explicitly), not by impersonating the user’s credential.
Service tokens are secrets, managed the same way as any other secret in 11.4: never in code, never logged, stored in the secrets store, rotatable without a code deploy.
A service-to-service call still goes through the typed provider wrapper pattern in 7.4 when calling another internal service over HTTP, the wrapper owns attaching the service token, not each call site re-implementing it.

Rule: a service-to-service call authenticated with anything other than the shared service token (a forwarded user token, a hardcoded credential, an unauthenticated internal call “because it’s inside our own network”) is a review flag. Network-level trust is not a substitute for authenticating the calling service.

6.13 Dependency vulnerability scanning

A vulnerable dependency is a security risk introduced by an update, not a bug in code this team wrote, and is caught the same way any other CI gate catches a problem before merge, not discovered later in a security audit.

npm audit runs in CI on every PR, alongside lint and format (see 4.3), checking new and changed dependencies for known vulnerabilities against the npm advisory database.
A high or critical severity finding blocks merge. A PR that introduces or upgrades to a dependency with a high/critical vulnerability does not merge without either resolving it (a patched version, a different dependency) or an explicit, reviewed justification for why it’s acceptable (the vulnerable code path is unreachable in how the dependency is actually used, a fix isn’t yet available and the risk is accepted knowingly), the same review-flag pattern as the any-suppression justification in 4.1.
A moderate or low finding is visible, not blocking, tracked the same way any other non-blocking finding is, worth addressing, not worth holding a PR for by default (see the blocking vs. non-blocking distinction in 4.5).
This applies to existing dependencies too, not just new ones. A scheduled, periodic audit run (not gated on a specific PR touching that dependency) surfaces vulnerabilities in dependencies nobody has recently touched, since a dependency can become vulnerable after it’s already in the codebase, a new CVE published against an existing, unchanged package.

Rule: an unresolved high/critical vulnerability finding with no documented justification is treated as a production incident waiting to happen, not a background task, the same urgency this document already applies to a secret found in source control (see 11.4).

6.14 Deciding sync, fire-and-forget, or a worker

This subsection is the single decision path for a question this document otherwise answers in pieces across 6.3 and 10.4: does this operation run inline, does the caller wait for a result, and if it runs in the background, what actually hosts it. Work through these in order:

Does it fit inside a normal request, roughly 1-2 seconds? If yes, run it inline in the handler. This is the threshold 6.3 and 10.4 both already state, don’t reach for a background job or a worker for something that comfortably fits inside a request/response cycle.
If it doesn’t fit inline, does the caller need the outcome? If the caller needs a result, a confirmation beyond “accepted,” or needs to know if it failed, it is not fire-and-forget, use the job ID + polling or webhook/notification pattern from 6.3, regardless of how the work is actually hosted (step 3). If the caller genuinely doesn’t need the outcome at all, only confirmation the operation was accepted, it’s a true fire-and-forget operation.
Whichever of those two it is, where does the work actually run? This is a separate question from step 2, sync-wait vs. fire-and-forget describes what the caller experiences; this step decides what actually executes the work, using the queue technology criteria in 10.4: Azure Service Bus when the work needs durability, retries, and ordering; a worker in Azure Container Apps Jobs for scheduled or event-driven work that doesn’t need a persistent queue in front of it; Azure Function Apps for small, simple, lightweight triggers; a cron job for periodic maintenance with no real-time trigger.

Worked example. A report-generation feature: it takes 30 seconds (fails step 1, not inline), and the user needs the finished report (fails step 2’s fire-and-forget test, they need the outcome), so it’s a job-ID/polling operation. That work then runs as a worker in Azure Container Apps Jobs (step 3), because it’s a substantial, self-contained unit of work, not a small enough trigger to justify a Function App and not something that needs Service Bus’s durability/ordering guarantees on its own.

Rule: these are three separate questions, don’t conflate “it’s async” with “it’s fire-and-forget,” and don’t conflate “it needs a result” with “it needs Service Bus.” An operation can be async and still need the caller to see the outcome (job ID + polling); a worker hosted in Container Apps Jobs can back either a fire-and-forget or a polled operation. Decide each question independently using the criteria above, don’t default the whole decision based on answering only one of them.

6.15 Graceful shutdown

Every service handles a shutdown signal (SIGTERM) by draining, not by exiting immediately. This is what makes the zero-downtime deploy pattern in 7.8 actually zero-downtime in practice: a rollout that terminates instances abruptly drops in-flight requests regardless of how carefully the schema migration itself was staged.

Stop accepting new work, finish what’s in flight. On receiving SIGTERM, a service stops accepting new requests (fails readiness, see 6.4, so the platform routes new traffic elsewhere) while letting in-flight requests complete normally, up to a bounded grace period.
Background work drains the same way. A worker or job runner processing a message when shutdown begins finishes that message (or returns it to the queue cleanly, if it can’t finish in time) rather than being killed mid-processing, since a message killed mid-processing without being returned to the queue is a silent data-loss bug, not just a rough edge.
Connections close cleanly, not abruptly. The database connection pool (see 6.6) and any open external connections are closed after in-flight work completes, not left to be forcibly terminated by the process exiting.
The grace period is bounded, matching the platform’s actual termination timeout (Azure Container Apps’ shutdown grace period). A service that assumes an unbounded drain window will eventually be force-killed mid-request anyway; design the drain to complete within the real, finite window the platform actually gives it.

Rule: a service with no SIGTERM handler is not production-ready, the same review-blocking standard this document already applies to a service missing the required middleware baseline in 6.2. Every rollout, not just an occasional one, sends this signal, an untested shutdown path is a guaranteed, not hypothetical, source of dropped requests.

6.16 Distributed locking

Optimistic concurrency (the changeNo column, see 6.9) protects a single record from a lost update between two concurrent writers. It does not solve a different problem: coordinating a critical section across multiple running instances of the same service, so exactly one instance does something, not all of them at once.

Use it for exactly-once-across-instances work: a scheduled job that must run on only one instance even though the job runner has multiple replicas, a periodic cleanup task, anything where every instance running it concurrently would duplicate work or corrupt state, rather than merely being redundant.
The lock lives in the shared datastore already in use, not a new piece of infrastructure introduced solely for locking: a document in MongoDB with a unique constraint and a TTL/expiry (see 2.2) acquired via an atomic find-and-update, or the equivalent atomic operation for a documented relational exception (see 6.6). Introducing a dedicated distributed-lock service for this is disproportionate to the problem in the current stack; evaluate that only if a real need outgrows what the database-backed lock provides.
Every lock has a TTL/expiry. A lock held by an instance that crashes or is killed mid-task must not block every future run indefinitely, an expiring lock recovers automatically; a lock with no expiry turns one crashed instance into an outage for the locked operation.
Don’t reach for a distributed lock to solve a single-record concurrency problem. If the actual need is “don’t let two requests overwrite the same record,” that’s optimistic concurrency (6.9), not a distributed lock, a lock introduces real complexity (expiry tuning, failure-mode handling) that a version check doesn’t need.

Rule: a scheduled or periodic job that can run on more than one service instance and isn’t idempotent enough to tolerate concurrent execution (see 8.7) needs a distributed lock before it ships, not after duplicate runs are noticed in production.