From the journey Design Reliable Microservices
What makes a system distributed?
The question and reading below are what WeLearn generated for this lesson. Try the question before you read.
Maya presses “Place order” in a food-delivery app at her apartment. The screen spins for a while, then says it could not confirm the order; the restaurant has not called or messaged her.
What should Maya expect may have happened to her order?
Tutor's follow-up
From the reading
The Message Boundary
One observation, several possible realities
Maya presses “Place order.” Her phone sends work beyond the app screen: the delivery service must receive the order, record it, contact the restaurant, and return a confirmation. The screen eventually reports that it could not confirm the order. That report establishes only one fact: Maya’s app did not receive a confirmation before it stopped waiting.
The same fact is compatible with several realities. The request may never have left Maya’s phone. It may have reached the delivery service but not the restaurant. The restaurant may still be deciding whether to accept it. The kitchen may already have started the meal while the confirmation is delayed or lost. Visible progress on the phone does not select among these cases, and missing confirmation does not prove that no remote action occurred.
The uncertainty exists because the participants do not share one execution state. Maya’s app, the delivery service, and the restaurant system each continue according to their own processing and failures. They coordinate by sending messages, and each participant directly observes only its own state and the messages that arrive.
Independent execution → message exchange → local observation → uncertain global state is the mechanism that makes distribution consequential. The first two links identify a distributed system; the latter two explain why its behavior cannot be analyzed as one ordinary program.
Distribution begins at the message boundary
A system is distributed when multiple independently executing components coordinate toward a task by exchanging messages over a network. Such a component is commonly called a node. A node may be a physical machine, a virtual machine, a container, a process, a browser, or a device. The definition depends on independent execution and message exchange, not on the number of boxes visible in an architecture diagram.
In Maya’s order, the phone app and the remote delivery service are nodes because either can continue, pause, restart, or fail without executing as part of the other’s instruction stream. Their network connection carries messages but does not create shared memory or a shared point of view. The restaurant service adds another node and another message boundary.
This criterion classifies less obvious cases. A program whose modules call one another within one process is not distributed merely because it is large; the modules share the process’s execution and failure. Two processes on the same machine can form a distributed system when they communicate through sockets or another message mechanism and retain independent lifecycles. Conversely, several processor cores executing one tightly coupled program are not classified as distributed solely from the hardware count.
Machine count remains a useful clue because networked nodes often occupy different machines, regions, or organizations. It is not the deciding property. The deciding property is whether coordination crosses a boundary at which execution and knowledge are no longer shared.
The mechanism now has a concrete boundary: independent nodes exchange messages, and no node gains direct access to another node’s current state. The remaining difficulty lies in how that boundary changes the meaning of a call.
A remote call does not preserve local-call semantics
An application interface can make a remote request resemble an ordinary function call. The syntax may be nearly identical: provide arguments, wait, and receive either a result or an error. That resemblance holds during normal operation, but it does not establish the same semantics.
For a local function call, the caller and callee normally share a process. If the callee returns, the caller directly receives that return through local execution. If the process stops, both sides usually share the failure. The call therefore has a comparatively direct relationship between execution and observation.
Maya’s order crosses independent components. Suppose the app sends an order at 12:00:00 and stops waiting at 12:00:05. Variable delay alone can make the confirmation arrive at 12:00:06. A lost request can leave the restaurant unaware of the order. A received request followed by a lost response can leave the restaurant preparing food while Maya sees failure. A service crash can occur before recording the order or after recording it but before replying. The observation at 12:00:05 is identical in all these cases: no confirmation has arrived.
A timeout is the caller’s decision to stop waiting after a deadline. It reports the absence of a timely response, not the remote operation’s outcome. Retrying immediately therefore does not merely repeat a known failure. If the first request never arrived, a retry may create the intended order; if the first request succeeded but its response was lost, the retry may create a duplicate. Later readings will develop the mechanisms that constrain this risk, but the need for them already follows from the message boundary.
Remote communication introduces variable delay, uncertain delivery, and independent failure. These properties turn a call result from a direct observation of shared execution into evidence that must be interpreted from one node’s local position.
Partial failure makes system state observer-dependent
A partial failure occurs when some nodes or communication paths fail while others continue operating. Maya’s phone can lose connectivity while the restaurant remains healthy. The delivery service can reach Maya but not the restaurant. The restaurant can accept orders while its confirmation path is unavailable. None of these states is equivalent to the entire system being down.
Partial failure also produces asymmetric knowledge. The restaurant may know that it accepted order 731, while Maya does not know whether order 731 exists. The delivery service may know that it sent the order but not whether the restaurant persisted it. Each statement can be accurate because each node reports from different local evidence.
This is the completed mechanism: independent execution → message exchange → local observation → uncertain global state. Distribution changes design because a node must often act before it can determine the whole system’s state. In Maya’s case, the app must choose a policy for an outcome it cannot yet classify—for example, keep waiting or report that the order remains unconfirmed. If a dependency is known to be unavailable, the system may instead refuse new work or provide only functions that do not require that dependency. No implementation can remove the need for such decisions merely by making remote calls look local.
Distribution is nevertheless useful. Independent nodes can add capacity, place computation near users or data, isolate organizational responsibilities, and allow some service to continue when one component fails. The same independence creates coordination costs: messages take time, participants can disagree temporarily, and partial failure prevents any one observation from serving as complete proof of global state.
An architecture therefore enters distributed-systems reasoning at the first operationally significant message boundary between independent components. Naming the nodes and network path identifies where distribution exists; tracing what each node can observe identifies why reliability requires a different design model.