Pipe-filter : An Architecture for Processing Data

The pipe-filter architectural pattern is very common and used to process data. It is flexible and scalable. This pattern has existed in other areas. However, it is the perfect solution for software problems that need steps or filters.

The Pipe-Filter Pattern Defined

Think of a start and an endpoint. Data has to go from beginning to end, and work will be done on it along the way. The work done is the filters, and we can place any number of filters along the journey. Each one is stand-alone in a sense as it takes in data, does the work, and then spits it out on the other end. We can change countless steps like this together as one might do with Lego blocks.

Each Step A Island One of the best ways to create a solution that is easy to maintain and scalable is to take well-defined steps that can easily be swapped out. We can even add steps that load balance or provide other performance enhancements between them. That is the strength of the pipe-filter pattern. We can add, remove, and update filters with ease without rebuilding the system. We have well-defined and consistent inputs and outputs when each step takes data in, does the processing, and then spits it out. Those same inputs and outputs from step to step make each filter interchangeable. Likewise, we can start simple with a single filter, test it, and then move to a more complex filter series. Even better, we can always reduce the solution to a single filter and validate it. Challenges The pipe-filter pattern has drawbacks related to the strengths. For example, there may be pre-requisites in a solution that limit interchangeability. Likewise, a bunch of filters can conflict or reduce performance. Therefore, we need to be wise in our design and not fall back on adding more filters. Any solution can be over-architected. Thus, be intentional in designing both the filters and the flow. A good design does the proper amount of work at each step. That work also does not require assumptions about the data it is processing.