/ˈpaɪp.laɪn/
noun — “a relay race for data, where every runner hands off cleanly.”
Pipeline is a structured sequence of processes or commands where the output of one stage becomes the input of the next, forming a continuous flow of data from start to finish. In computing, a Pipeline allows complex tasks to be broken into smaller, specialized steps that operate in coordination rather than isolation.
In a Command Line Interface, a Pipeline is typically constructed using the pipe operator (|), which connects the Standard Output of one command directly to the Standard Input of another. For example, a command sequence like cat file.txt | grep error | sort forms a Pipeline in which each program performs one focused operation before passing its results forward. No intermediate files are required; the data flows through memory as an I/O Stream.
Under the hood, the operating system creates a Pipe between each adjacent stage of the Pipeline. Each process runs concurrently, meaning upstream commands can continue producing output while downstream commands consume it. This parallelism makes a Pipeline both efficient and elegant. Rather than waiting for one step to finish completely, stages overlap in execution, reducing total processing time.
The power of a Pipeline lies in composability. Each stage adheres to a simple contract: read input, transform it, write output. Because of this discipline, tools can be recombined in unexpected ways. Text filters, format converters, compressors, and network utilities can all participate in a Pipeline without knowing anything about the larger workflow. This design philosophy encourages small, modular programs that cooperate through shared data streams.
Pipeline is not limited to shell commands. In software engineering, a Pipeline may describe build systems, data processing chains, graphics rendering paths, or continuous integration workflows. A CI/CD Pipeline, for example, moves code through stages like compilation, testing, packaging, and deployment. Each stage depends on the successful output of the previous one, forming a controlled progression from raw source code to running software.
Error handling within a Pipeline can be subtle. If an early stage fails, downstream stages may receive incomplete or no data. Exit codes propagate through the chain, and careful scripting is required to detect failures reliably. In shell environments, understanding how exit statuses interact with a Pipeline is crucial to building robust automation.
From a systems perspective, a Pipeline exemplifies streaming architecture. Instead of loading all data into memory at once, it processes data incrementally. This approach scales better for large datasets and reduces memory pressure. It also aligns with network programming, where data arrives as a continuous stream rather than a finished block.
Conceptually, a Pipeline is a production line for information: each worker performs one precise task, and the assembly advances step by step until the final product emerges.
Pipeline is like turning a pile of raw ingredients into a finished meal — one deliberate transformation at a time.
See Pipe, I/O Stream, Redirection, Process Management, Exit Code.