System Architecture & Service Communication

7.1 Frontend/backend boundary

Define the boundary by responsibility, not by where code physically lives in the repository. The frontend is responsible for presentation and user interaction (see 3.2); the backend is responsible for business rules, data access, and anything that must be trusted (auth checks, validation that matters, anything a client could otherwise bypass). This holds even when frontend and backend code live in the same monorepo, the same language, or even the same file boundary in a framework that blurs server/client code, the conceptual boundary is about trust and responsibility, not file location.

Rule: never trust a decision made on the client as the final word. If a rule matters (a permission check, a price, a validation), it is enforced again on the server regardless of what the client already checked. Client-side versions of these checks exist for user experience, not as the actual guarantee.

7.2 Choosing a communication paradigm

This is the core decision framework for how a client talks to a backend, or how one internal service talks to another. Pick per API, not per project, using this order:

1. Internal RPC-style (tRPC today). The default when both client and server are in the same type system and only your own frontend(s) consume the API. This gives full type safety across the boundary with no schema duplication (see 2.2), and is the cheapest option to build and maintain when there’s no external consumer to accommodate.

2. REST. Escalate to REST the moment an external party consumes the API:

A partner or third party needs to call it.
It needs public API documentation.
It’s a webhook, webhooks are always REST by convention, since the receiving system rarely shares your internal type system or tooling.
A mobile team (or any consumer) unfamiliar with your internal stack needs to integrate against it.

3. GraphQL (Apollo). Only justified when one of these is true:

Clients have meaningfully different data-shape needs from the same backend (e.g. a thin mobile payload vs a deeply nested dashboard query), such that REST would require either over-fetching or maintaining multiple bespoke endpoints per client shape.
You’re aggregating multiple internal services into one queryable graph for multiple frontend consumers, and a client needs to query across that aggregation in one round trip.

GraphQL is an opt-in exception justified by real pain, never a default choice. It carries real, non-optional cost that must be explicitly accepted before adopting it: N+1 query risk (requiring a DataLoader or equivalent batching layer from day one, not retrofitted after the first incident), and the need for per-query-cost rate limiting, since a single GraphQL query can trigger arbitrarily expensive nested resolution that a flat REST endpoint can’t.

Rule: document which paradigm owns which domain. For example, “internal dashboards use tRPC, partner integrations use REST.” An engineer building a new endpoint should be able to look this up rather than guessing per feature. When a new domain doesn’t clearly map to an existing entry, decide it once, using the framework above, and add it to the mapping rather than deciding it silently and moving on.

7.3 Event-driven and asynchronous messaging

Direct request/response is the default. Reach for pub/sub or asynchronous messaging instead of a direct call when:

More than one consumer needs to react to the same event, and that number is expected to grow (once N services need to react to the same event, a direct call from the producer to every consumer becomes an N-way coupling problem).
The producer shouldn’t be blocked on, or coupled to the availability of, the consumer’s processing (e.g. firing an event on order creation that several unrelated systems react to independently).
Work needs to be durably queued and processed asynchronously rather than inline in a request, see 10.4 for the criteria on what belongs in a background job.

Current answer: not needed today. Our current service communication is direct request/response. This is the honest current state, not a gap, revisit this once the “N services react to the same event” criterion above is actually met, don’t introduce a message bus speculatively ahead of that.

7.4 External API calls

This subsection covers a call to an external, third-party provider (payment processor, identity provider, third-party data source). A call from one of our own internal services to another is a different case, see 6.12 for the service-to-service authentication pattern that applies instead.

Every call to an external provider goes through a typed wrapper module dedicated to that provider, see the Factory pattern in 8.3. Never call fetch (or an equivalent) directly against a third-party API scattered across feature code.

The wrapper module owns the provider’s base URL, auth, and request/response shapes in one place.
Responses are validated at the boundary using the shared schema library (Zod, see 9.3) before anything downstream trusts their shape, an external provider changing its response shape without warning should fail loudly at the wrapper, not silently corrupt data three layers deeper in the codebase.
Retry, timeout, and circuit-breaking behavior (see 10.3) lives in the wrapper, not duplicated at every call site.

Rule: if you find yourself writing a second fetch call to the same provider outside its wrapper module, that’s a signal the wrapper is missing a needed method, add it there, don’t duplicate the call.

7.5 Data flow

At a high level, a request flows in one direction through these layers:

Client → API layer → services (domain logic) → external providers / database.

The client initiates a request through whichever paradigm owns that domain (7.2).
The API layer is the trust boundary: authentication, authorization, and input validation happen here (see Section 11), before anything reaches domain logic.
Services (domain logic, see 3.2) apply business rules and orchestrate whatever the request needs, calling out to the data access layer or external provider wrappers (7.4) as needed.
External providers and the database are the outermost layer, never called directly from the client or from presentation-layer code.

Data flows back up through the same layers in reverse, each layer returning its own appropriate shape, a domain model up from services, a DTO out from the API layer, per 6.8.

7.6 Multi-tenancy

This subsection is conditional: it applies only if a product actually serves multiple tenants from one codebase and one set of infrastructure. If that’s not your situation, skip it, don’t build multi-tenant isolation speculatively ahead of an actual multi-tenant requirement (see 8.4 YAGNI).

Where multi-tenancy is used:

Isolation model: shared database with a tenant_id field on tenant-scoped records is the cheapest and most common model, and the default unless a specific tenant has a contractual or regulatory requirement for physical data isolation.
Enforce scoping at the ORM/query-builder layer, not by trusting every developer to remember a WHERE tenant_id = ... clause on every query. A query helper or middleware that automatically injects tenant scoping is required; relying on human discipline across every query in the codebase is not an acceptable isolation guarantee.
A dedicated test suite must prove tenant isolation, specifically, that tenant A can never read or write tenant B’s data, run against every tenant-scoped resource. This is not covered by ordinary feature tests and must exist as its own suite, treated as a security requirement, not a nice-to-have (see Section 11).

7.7 API versioning

Prefer additive-only changes: add a new optional field, a new endpoint, or a new procedure rather than changing the meaning or shape of an existing one. Additive changes don’t break existing consumers and don’t require a version bump.

When a breaking change is genuinely unavoidable:

Create a new named procedure or endpoint (e.g. getUserV2, or a versioned REST path) rather than mutating the old one in place.
Document a deprecation notice period for the old version, stating the removal date, and communicate it to known consumers before that date arrives.
The old version keeps working, unchanged, for the entire deprecation window. A breaking change is never shipped as a silent mutation of an endpoint or procedure another consumer already depends on.

7.8 Schema migrations under zero-downtime deploys

Because deploys aren’t instantaneous and multiple versions of the application may briefly run concurrently, a schema change that isn’t backward compatible with the previous application version will break requests during the rollout window. Use the expand/contract pattern instead of changing a schema in a single breaking step:

Expand: add the new column or field alongside the old one. Both old and new application code can run against this schema.
Migrate readers/writers: deploy application code that writes to both the old and new field (dual-write), or backfill the new field from the old one.
Migrate readers fully: deploy application code that reads only from the new field, once you’ve confirmed the backfill/dual-write has fully caught up.
Contract: remove the old field, only once no running version of the application still reads or writes it.

Rule: a migration PR that combines adding a new required field and removing the old one in a single deploy is a zero-downtime violation, regardless of how small the change looks. Split it into the expand and contract steps above, deployed separately, with the migrate step in between.

7.9 Caching architecture

This is the system-level companion to 9.2’s fetch-level caching and revalidation, and to 2.2’s in-memory lru-cache entry: this subsection covers caching as a system component, not per-fetch client behavior.

Current answer: no shared/distributed cache. A distributed cache (Redis or equivalent) is not in use today, its operational cost isn’t justified by a current need. Process-local caching (lru-cache, see 2.2) is the current mechanism for hot, cheap-to-recompute data, and it’s sufficient as long as a value being cached per-instance rather than shared across instances is an acceptable tradeoff (some duplicate computation across instances, no cross-instance invalidation problem to solve).

Reach for a shared/distributed cache only once one of these is true, not speculatively ahead of an actual need:

Per-instance caching is producing a measurable, current problem: meaningfully inconsistent behavior across instances (a user seeing different cached results depending on which instance served them), or redundant expensive computation across instances that a shared cache would eliminate.
A value needs to be invalidated consistently across every running instance the moment it changes, and process-local caching means an invalidation has to somehow reach every instance individually, which process-local caching has no mechanism for.

If a shared cache is introduced:

Cache-aside is the default pattern: the application reads from the cache, falls back to the source of truth (the database) on a miss, and populates the cache with the result. Reach for write-through (writing to cache and source of truth together) only where staleness on a cache miss is genuinely unacceptable, cache-aside’s simplicity is worth keeping otherwise.
Cache invalidation has a named owner per cached value, the same discipline already required for fetch-level cache tags in 9.2: whatever mutation changes the underlying data is responsible for invalidating the cached value, not left to a TTL alone to eventually resolve staleness.

Rule: introducing a distributed cache follows the same tool-evaluation process as any other stack addition (2.5), it is not a default reached for because a system “should probably have caching,” the current in-memory approach is a deliberate choice, not an oversight, until one of the criteria above is actually met.

7.10 Rate limiting architecture

11.5 states that rate limiting on sensitive operations is required and engineering-owned. This subsection covers where that’s architecturally enforced and how.

Enforced per-service, at the middleware layer (see 6.2’s required middleware stack), not at a shared gateway that every service would need to depend on. This matches the current architecture, direct request/response between services (see 7.3) with no API gateway layer in front of them today; introducing a gateway purely to centralize rate limiting is not justified by that need alone.
A sliding window (or token bucket) counter, not a fixed window. A fixed window (e.g. “100 requests per calendar minute”) allows a burst of up to double the limit right at the window boundary; a sliding window or token bucket approach smooths that out and is the expected default.
The limit key matches what’s actually being protected: per-user or per-account for an authenticated operation (login attempts, actions on one account), per-IP for an unauthenticated or pre-authentication operation (a login endpoint itself, OTP requests), matching the sensitive-operation list in 11.5.
A rate-limited request returns a normalized rate_limited error (per 10.1), not a generic failure, so a client can distinguish “you’re being throttled, retry later” from any other failure and back off accordingly.

Rule: if a future architectural change introduces a shared API gateway (for a reason unrelated to rate limiting, e.g. centralizing authentication or request routing across many services), rate limiting moves to the gateway at that point; don’t introduce a gateway solely to centralize rate limiting when the per-service middleware approach already satisfies the requirement.

7.11 Idempotency key mechanism

8.7 establishes idempotency as a requirement for webhook handlers, payment callbacks, and retried calls. This subsection defines the concrete mechanism: how an idempotency key is generated, transmitted, and enforced.

The caller generates the key, not the server, since the whole point is that a retried call (the same logical operation, sent again after a timeout or a network failure) carries the same key as the original attempt. A server-generated key would be different on every attempt and defeat the purpose.
Transmitted as an Idempotency-Key header on the request, a client-generated unique value (a UUID) scoped to one logical operation, the same convention across every endpoint that requires idempotency, so a client integrating against multiple endpoints doesn’t need a different mechanism per endpoint.
The server checks the key before applying effects. On receiving a request with an idempotency key, the handler checks whether that key has already been processed (a record keyed on the idempotency key, with a bounded retention window) before doing the actual work. If it has, return the original response without reapplying the effect; if not, apply the effect and record the key with its result.
The stored result, not just a “seen it” flag, is what’s returned on a repeat. A retried request expects the same response as the original succeeded with (e.g. the same created resource’s ID), not a generic “already processed” acknowledgment that leaves the caller unable to get the actual result of the original attempt.

Rule: an endpoint identified as requiring idempotency per 8.7 that doesn’t implement the Idempotency-Key mechanism above is incomplete, the header convention and the check-before-apply pattern are what make the principle in 8.7 enforceable in practice, not just a stated intention.

7.12 Service discovery and inter-service addressing

Now that internal services authenticate to each other via a shared service token (see 6.12), this subsection covers the other half of a service-to-service call: how a calling service knows where to send it.

Current mechanism: configuration-based addressing. A service’s address (its base URL) is read from environment configuration (see 6.10), resolved per environment (local, staging, production point to different addresses for the same logical service), not hardcoded, per the parameterization principle in 6.11.
A service’s address is read once, at the point the typed service-to-service wrapper is constructed (mirroring the external API wrapper pattern in 7.4), not looked up ad hoc at every call site.
This is sufficient as long as the number of internal services stays small and their addresses change infrequently. A dedicated service registry or DNS-based service discovery mechanism is unnecessary complexity at the current scale; revisit this once the number of internal services calling each other, or the frequency of their addresses changing, makes manually maintained configuration a real operational burden, not before.

Rule: a service address hardcoded directly at a call site, instead of read from configuration through the service’s typed wrapper, is a review flag, the same class of issue as any other hardcoded operational value under 6.11.

7.13 Backpressure and load shedding

This is a different failure mode from the one 10.3’s circuit breaking addresses: circuit breaking protects a service from a dependency that’s failing; this subsection covers what a service does when it itself is overwhelmed by more load than it can currently handle.

Reject explicitly, don’t silently degrade into unbounded queueing. A service under sustained overload that keeps accepting every incoming request into an ever-growing internal queue (rather than rejecting new work once a threshold is reached) trades a fast, clear failure for a slow, cascading one, eventually every request times out instead of a clear subset being rejected immediately.
A service that must shed load returns a normalized 503-class error (per 10.1), distinct from the rate_limited response in 7.10: rate limiting protects against one client’s excessive usage, load shedding protects the service itself when aggregate load, across all callers, exceeds capacity.
Concurrency limits at the handler or connection-pool level are the practical mechanism: a bounded number of in-flight requests or database connections (see the connection pooling rule in 6.6) naturally sheds load once the bound is reached, rather than requiring bespoke load-shedding logic per endpoint.

Rule: a service with no bound on concurrent in-flight work is not production-ready, the same review-blocking standard already applied to a service with no graceful shutdown handling (6.15); an unbounded service degrades into cascading failure under load instead of failing predictably at a known limit.