Journey to Event Driven

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

The world is changing. New problems need to be solved. Companies now run global businesses that span the globe and hop between clouds in real-time, breaking down data silos to create seamless applications that synergize the organization. This continuous state of change means that legacy architectures are insufficient or unsuitable to meet the needs of the modern organization. Applications must be able to run 24×7 with 5-9s (uptime of 99.999%), as well as be superelastic, global, and cloud-native. Enter event-driven architecture (EDA), a type of software architecture that ingests, processes, stores, and reacts to real-time data as it’s being generated, opening new capabilities in the way businesses run.

Traditional vs Event-Driven

At some point, it becomes obvious that we need to go back to basics, back to the first principles of system design, and start again. The common element of all these new world problems is that they revolve around the notion of real-time events. These events drive actions and reactions, and transform between different streams, splitting, merging, and evolving like the pathways of your brain.

Importance of Event Driven Architecture
To understand the importance of being event-driven, we’ll examine why events have become so pivotal in our thinking today. We will then evaluate the qualities and how events have become a first-class concern for the modern organization, as awareness of events underpins event-first thinking and design. I’ll then extrapolate design concepts against different perspectives of event patterns, such as the event command and event first, and why event-driven programming fulfills foundational requirements when choosing an architecture that supports evolution and elasticity.

Overview
An opinionated history of “events” – Why do they matter?
Adoption journey of the “event”
Considerations of the event-driven architecture
Transitioning to event-first thinking
Event-first versus event-command patterns for event-driven design
Event-command pattern
The cost of the event-first approach
Benefits of the event-first approach
To read the other articles in this series, see:
Journey to Event Driven – Part 2: Programming Models for the Event-Driven Architecture 
Journey to Event Driven – Part 3: The Affinity Between Events, Streams and Serverless
Journey to Event Driven – Part 4: Four Pillars of Event Streaming Microservices

An opinionated history of “events” – Why do they matter?
It’s taken the last 25 years to get to the point where we consider the event as a core principle in system design. Throughout that period, much of the world was stuck thinking that remote object calling was the right way of doing things. REST and web services started by exposing the API as a concern, but that was done by a synchronous protocol.
In 2002, I built a commercial messaging system that adopted principles from the SEDA: An Architecture for Well-Conditioned, Scalable Internet Services paper—the idea being that we move everything to become asynchronous; local and remote events are then used to drive the intelligence and adaptation of an application. It really shaped the way I understood events to be a core design principle as well as how to think about system design and data in general.

There has been a revolution; organizations must become real time; to become real time, they must become event driven

It took the era of big data for us to realize that while having lots of data is useful, things get slower as you have more of it. And yet, we collect more. Over the last several years, there has been a revolution; organizations must become real time; to become real time, they must be event driven. This need is fueled by the increasingly digital world that we live in where everything is connected. Events are a fundamental aspect that drives this ecosystem; an organization is but a cog that when viewed at any level needs to act and react to internal and external events.

The value of events is that a sequence of related events represent behavior (e.g., an item was added and then removed from a shopping cart, an error recurs every 24 hours or users always clicks through a site in a particular order). A sequence of related events is commonly called a stream. Streams can come from an IoT device, a user visiting a website, changes to a database or many other sources. When thinking of APIs, we start with the event rather than the highly coupled concept of a command, e.g., an event-X has occurred, rather than command-Y should be executed. This thinking underpins event-driven streaming systems.

A stream of events captures temporal behavior
Adoption journey of the “event”
There are many paths to the inception of an event-driven architecture within an organization. For some, it is a natural pattern evolving from domain-driven design (DDD) principles, the related and powerful event-storming practice, decoupled systems design or just event-driven microservices. The point is they can come from any level, whether business or technical, but it will be a conscious adoption that doesn’t just happen by accident.

Considerations of the event-driven architecture
There are many considerations when evaluating the event-driven architecture. Events initially start out to as atomic and drive reactionary callbacks (functions). As a use case evolves, workflows develop and the criticality of the application generally increases along with the nature in which an event is considered. This raises numerous questions. Is ordering important? Do I want to ensure transactionality? How do I trust the execution? Security? Lineage? Where did the event come from?

The key realization for the adoption of event-first thinking is that an event represents a fact, something happened; it is immutable and therefore changes how we think about our domain model

The basic attributes of an event include time, source, key, header, metadata and payload. One problem with events is that they are everywhere, and the extreme lack of consistency makes them very difficult to work with at scale.

The other consideration is that events do not exist in isolation. Due to their very nature, an event tends to be part of a flow of information, a stream. We define a stream as an unbounded sequence of related events that are associated through the use of an “eventKey,” a key that ties multiple events together.
In the real world, the nature of the “event” can be described as:
Atomic: something happened (bid on an item, send an email, device temperature)
Related: a stream or sequence of events (tracking a pricing change, device metrics changes over time)
Behavioral: the accumulation of facts captures behavior
Behavioral is the least obvious but most important aspect of event-first thinking. For example, in a shopping cart, if an item was added and removed—and this has occurred among many users—then we have captured a behavior. By analyzing the sequence across many users, it is possible to discern whether they changed their mind due to targeted advertising, finding a cheaper price elsewhere or perhaps because it is close to an annual event like Christmas, a birthday, etc. Ideally, the system should detect the behavior and react to it, as capturing events has a great deal of implicit value.

Transitioning to event-first thinking
Event-first thinking changes how you think about what you are building
The adoption of event-driven architectures tends to result from an “aha” moment, which involves moving from simple, use cases to a deeper understanding of processes, operations and how to reason about the architecture.
You may start off creating the event streams required to build an application to fulfill your original design goals, but quickly find that the event streams themselves enable you to build much richer, totally unexpected analysis and functionality.
How is the event treated within a system? All of the following questions get asked as the event-driven architecture is developed:
Is it observable, and are the flows of streams behaving as expected?
Is it trusted, meaning transactional, exactly one or at least once? Will it scale?
Is stateless processing, such as filtering, projection, cleaning or enrichment required? Is stateful processing, such as aggregations or stateful sequence processing required?
Is a materialized view against the stream required? How many transactions per second are required for a windowed view?
Do we want to scale/fan/map out (parallelize), fan in/collect and, build materialized views?
Does it support error handling, such as error flows and dead letter queue?
Does it send and transform events from one stream into others (stream processing)?
Does it react and drive intelligence from the state collected from a stream (stream processing)?
In terms of increased adoption, we generally see these patterns as characteristic of an event-driven organization:
Data pipelines (ETL, integration)
Monitoring and alerting, including log collection and aggregation
Event-driven microservices
Enterprise wide event-driven microservices
IoT
Customer 360
Digital transformation
Real-time enterprise
Central nervous system/digital nervous system

By embracing event-first thinking, we naturally inherit the foundations of event streaming platforms like event sourcing, replayability, stream processing and dataflow design, amongst others. We should also incorporate event-first thinking into an architecture that leverages a serverless stack to be event driven, multi-cloud and elastic.

There are several ways to be event driven. Without becoming tangled up in the marketing noise, we can boil it down to a couple of approaches. The first reuses messaging principles and sends pure events like item-purchased, while the second reuses the conventional, more familiar approach of sending commands as events, like user.purchaseItem(item). Both can be event driven, but the design principles are fundamentally different. Let’s expand on this thinking to understand the impact of both approaches.
The importance of events and event first thinking:
Capture facts
Capture behavior
Provide a representation of the real world
Model use cases of how we think
Supports repeated evaluation and processing (a time machine)
Provide horizontal scaling
Speak the same language as the business

Event-first versus event-command patterns for event-driven design

You can build the same application using either the event-first or event-command approach, but the event-first approach is generally recommended. If we rewind onto familiar ground for an example, we see that traditional REST applications are an event → command pattern. They are coupled in multiple ways: Firstly, the endpoint is known (i.e., the service address). Secondly, the method being called is also known (i.e., an API call to doStuff), and lastly, the calls tended to return a value; they are synchronous. Newer frameworks will allow this call to be handled synchronously or asynchronously—it is called the event-command pattern.
Alternatively, with the event-first pattern, the paradigm shift is to discard all of those considerations and just send an event as we would in the traditional messaging sense; don’t do anything else; have no API coupling to a remote service. The approach is unique in that it processes the event as a reaction; the emitter doesn’t call on a specific function; the API has been removed and it instead just sends an event. The emitter of the event doesn’t know which processors (or functions) are going to consume it, and the event becomes the API. This decoupling allows the set of consuming apps to change over time without any upstream changes required in the emitter.

Event-first analog: I walk into a room, generate an “entered room” event and the light turns on. This is a reaction to an event.

Event-command analog: I walk into a room, flip the light switch and the light turns on. This is a command.

In the event-first analog, I have no knowledge and don’t ask for the lights to be turned on;. Instead, a sensor detects my presence; it has the responsibility. In the event-command analog, I have the responsibility of knowing how to turn on the light and also making it happen.

The event-first approach forces an inversion of responsibility; it is a fundamental paradigm shift of how applications are developed
Let’s apply it to a business function.

Event-command pattern
In the event-command pattern, the service knows the endpoints and API of the other services and calls them. For example:

Event-first pattern
With the event-first approach, each service listens for inbound events and outputs zero or more events after performing some (re)action. For example:

At this point, the two approaches don’t look that different. If anything, the event-first approach looks more complex, as it has to more moving parts, e.g., publishing/consuming events via streams. The benefit is that these streams provide decoupling.
The power of the event-first approach becomes clear when we try to enhance the system. Let’s say we want to add the ability to check inventory levels and reorder stock when levels are low, to perform analysis on the cause of failed purchases and help improve conversion rates.
For the event-command-based system, we need to change the main purchasing application to call these new systems. But for the pure event-first solution, we can simply add new processors of the existing events, such as why(item) and checkInventory().

Notice how the event-driven approach allows many processors to react to the same “purchased” event (published via the /user-purchases topic)? This flexibility means functionality can run in parallel and be added, extended, upgraded or replaced without taking the system down. Decoupling is the key. The “event” also affords the opportunity to propagate versioning, security tokens, correlation IDs and other useful information.

The cost of the event-first approach
The tradeoff of the event-first/event-driven architecture is the amount of scaffolding required to inspire confidence. In the old days when transactional consistency shaped our expectations, and NoSQL pushed us to eventual consistency with CAP theorem tradeoffs, it was known that data wouldn’t (and shouldn’t) get lost. Event orientation requires not only different thinking in replacing the command model but also the ability to support traceability, failure paths, scaling and explanation about why things have gone wrong. These challenges are easily solvable and are also quite fun; I will revisit this topic later on.

Benefits of the event-first approach
The pure event-first approach described above demonstrates:
Decoupling: Processors don’t know anything about upstream or downstream processors
Encapsulation: There are clean boundaries between processors.
Inverted knowledge: Knowledge and responsibility are reversed.
Evolutionary change: The system and events can change over time.
Event sourcing: When using a log and log-aware stream processors, we gain the ability to potentially rebuild and restore application state. It’s not a free lunch; the application state needs to be captured and managed.

Evolutionary architecture is a natural benefit from event-first thinking and event-driven architectures
When you use the pure, event-driven approach, the architecture can change over time as different processors may react to events, which can be reprocessed while the data model evolves simultaneously. Perhaps the domain model has changed, or functionality is being extended. None of the originating events are lost, and the behavior is captured in the stream. With these foundations, you then have a system that supports architectural evolution as well as data and domain evolution.
.
To support the event-first approach you need a suitable platform.

The ideal solution is one where the stack is simple, the surface area is small and I the developer retain control.

Event-driven, reactive architecture
Simple event-driven architectures were introduced many years ago. We first saw them gain mainstream popularity as part of the ESB phase. The approach of this model is to send messages (events) to different services that can react and execute logic. It serves as a conduit to drive distributed logic (event processing), and messaging is a remote invocation layer. The actor model is a variation on this idea where it is a means of distributed computation. The event-driven, reactive model was further standardized and adopted when the Reactive Manifesto was in full swing.
Systems built as Reactive Systems are more flexible, loosely-coupled and scalable. This makes them easier to develop and amenable to change. They are significantly more tolerant of failure and when failure does occur they meet it with elegance rather than disaster. Reactive Systems are highly responsive, giving users effective interactive feedback.
Reactive Manifesto, 2014
The Reactive Manifesto has influenced many frameworks and platforms, including Akka (an actor-based library circa 1973), Vert.x (reactive microservices) and Lagom (an opinionated, reactive microservice framework). Although the principles behind event-driven frameworks are sound, those behind event sourcing, CQRS and hydrating application state are separate concerns so we often see them handled explicitly as an orthogonal concern (e.g., operational processes) or externally (think GitHub for your applications state).
Scaling mechanism
The Akka scaling mechanism uses a cluster service which pools actors from remote nodes that join as cluster members. Inactive actors are persisted to disk to minimize idle resource requirements. The cluster is elastic, provides resilience and provides the distributed runtime for making the Akka system scale. Scaling actors is achieved by increasing the number of actors—effectively increasing the event processing capability of the application, not the data. Persistent actors will store their set of events to a journal (log). Akka Streams then changed tact as streaming became the core mechanism to drive processors in a more data-centric manner.
Event-driven, streaming architecture
With the event-driven streaming architecture, the central concept is the event stream, where a key is used to create a logical grouping of events as a stream. We think of streams and events much like database tables and rows; they are the basic building blocks of a data platform. Streams represent the core data model, and stream processors are the connecting nodes that enable flow creation resulting in a streaming data topology. Stream processor patterns enable filtering, projections, joins, aggregations, materialized views and other streaming functionality. Unlike the previous model, events are front and center.
The event streaming platform is effectively a data platform and therefore behaves accordingly. Data models need to be developed in relation to use cases. A streaming application can be thought of as a dataflow system. For example, a stream can be used to capture patterns such as a user’s clickstream, a set of events against a transaction, shopping cart behavior, etc. In this case, “events” drive the event stream.

An event streaming application depicting auction-bid functionality
The streaming topology shows a flow of data through an organization, representing the real-time DNA of your business. A proven approach is to capture domain knowledge as events using domain-driven design (DDD) and event storming. This approach works well because the events-centric view forces the use of a ubiquitous language; the business, technologist, developers and ops all have a common understanding of the system’s functions.

An event streaming app: streams and stream processors modeling an auction platform bid functionality
Stream processors can also be used to materialize tables from the stream of events to enable a view of the data (e.g., how much money was in my account at time X). The term is called turning the database inside out. Stream processors are used for constructing the flow from other streams in various patterns, either executing and reacting to logic or splitting, merging and maintaining materialized views, such that when they are all combined (fan in), they represent a domain model in the form of a digital nervous system.
Like database transactions, stream processors can also transact against the event log, support correctness semantics like ordering, exactly once semantics (EoS), idempotence, temporal reasoning and more. Unlike a traditional database, the platform scales horizontally to thousands of processors, observe real-time events, replay streams on demand to perform historical event processing and catch up with the live, data-in-motion view.

Materialized views scaling against a stream
Streams processors store their fair share of data locally; in combination, they form a distributed data layer
The stream is like a database table, whereas the event streaming platform is a data platform. As such, we model the domain with event-first thinking. A common use case that trips up those who are new to the concept is payment processing. A traditional OLTP system uses a two-phase commit to move money from a debit account to a credit account. It is very simple but presents scalability challenges. In the event-driven world, we’d build a scalable model where the event streams flow between event processors. By the end of the flow, the payment is transacted.

Event-driven payment processing
Let’s walk through the flow of the above diagram:
Payment requests flow from the /payment-inflight topic into the balance in-flight processor. It maintains a view of the total balance, including those currently being processed, similar to a read-uncommitted view. It sends the payment request into the /payment topic, keyed by the “from” account and setting the event type to DEBIT.
The payment processing picks up the DEBIT request and validates that the payment can be processed, updating the user’s balance in the materialized view. It then emits another event, keyed against the “to” account and setting the event type to CREDIT.
The payment processor receives the CREDIT event, updating the user’s balance in the materialized view, and passes on the payment-complete event.
The Balance Conf’d stream processor updates the total confirmed balance similar to the read-committed value.
The two balance processors are different instances of the same stream processor. The difference is that while one is aggregating the system payment, balance as a total, the in-flight processor is consuming the payment-complete values. The balance processor tables can be queried using interactive queries, and the difference is the current amount being processed by the system.
Scaling mechanism
Scaling is the opportunity to send concurrent streams from thousands of producers to thousands of consumers. Populating a stream with real data as shown below lets us understand how the data is distributed in order to support horizontal scaling. To achieve scaling the underlying event streaming platform will use partitioning (sharding) to act as buckets for streams to be stored in. The broker sizing is a function of the total sum of partitions across the number of servers and replicas.
To map a stream of events in a single partition we use an eventKey. The eventKey acts to identify a set of related events as a stream. Events never occur in isolation. There is always a context for triggering the event, and the eventKey captures this identifier. An example might be an IoT device, transaction-id or some other natural key. To determine which partition is used for storage, the key is mapped into a key space. This key mapping uses a partition-assignment strategy (the default provider uses a hashing function).
It is important to note that a partition can only be consumed by a single consumer; however, a consumer can consume from multiple partitions. Consumers in this context are anything that requests data; they could be stream processors, Java or .NET applications or ksqlDB server nodes.

Horizontal scaling is achieved via partitions. Each partition will hold multiple streams (streams are color coded).
I won’t go into this topic any further, but suffice it to say, key space is of fundamental importance (as well as having sufficient partitions). While we see partitions becoming cheaper, the number of partitions (shards) should always be much greater than the number of stream processors or FaaS functions required to support scaling of unexpected workloads.
Comparing persistence models
Persistence with the streaming model is significantly different to other approaches (actor, reactive, external DB). In non-event-streaming platforms, the persistence model is part of the framework or is a deliberate intent to interact with an object. For example, with Akka there is the “actor entity” (using a persistent actor or distributed data). In this approach, you program an entity, and there is a binding to the underlying persistence layer, such as the following:

creditCardActor.get(123).deposit(456).
The actor can also emit an event to a view layer thereby enabling CQRS.
The streaming model doesn’t use an entity as a programming model in the same way. It’s more in line with a data processing approach, where the incoming stream represents events. The event processor updates the local state accordingly and emits related events.
For example:

/deposits-topic { card:123 deposit:456 } →  
	cardProcessor.updateCard(123).deposit(456) →
		 /card-balance-topic topic event:{ card:123, balance:456}

The entity state is maintained by the event processor using the the incoming stream, and updates are sent to the output stream. The entity is implicit in the stream, and the materialized views provide a particular view against the stream, such as min, max, avg and current value.
With stream processing, the log stores the truth for the entity; it is organized as a stream. The log (stream) also stores every version of the entity.
Let’s summarize the persistence aspects:
The event captures facts about an entity within the domain.
The event stream is a journal of events, it is the transaction log.
The event stream captures behavior.
Event streams provide a model for scaling horizontally as well as performing local processing.
Materialized views let the processor maintain a local state. Jay Kreps provides an in-depth view and rationalization on how this works with streaming.
External sources can be materialized using connectors providing yet another streaming source of data accessible by the same data plane (streams and stream processors).
Because the orientation of the event streaming platform is event data, it brings with it a wealth of other functionality that I would encourage the interested reader to research further: Streams can be queried using SQL (ksqlDB); stream processors can perform data-local processing (colocated microservices and data) to support throughput and everything becomes observable by leveraging other streams (metrics, logs). I’m limited to space so will briefly mention Confluent Schema Registry, data evolution (schema evolution), change data capture (data virtualization), event collaboration and coordination primitives, streaming patterns, evolutionary architectures, disposable architectures and event tracing (correlation IDs).
Do I need to use a microservices framework?
Many people start looking at stream processing through the lens of microservices. They believe that streaming should accompany their existing microservice framework. This is not the case; the event streaming platform will replace it. We think of stream processors, producers and consumers as microservices.
The event streaming platform provides the means of storing data as a stream and streaming events at scale. In order to facilitate this functionality the underlying protocol needs to ensure qualities such as offline status, different consumer rates, high throughput, low latency and elasticity (amongst others). The consumer model is one which exposes the protocol to the developer; this means the consumer will have the opportunity to ACK, NACK message receipt and interact with the correctness of accepting a message payload.

Stepping through an example of an event streaming app
Let’s combine all of the preceding concepts into a solution architecture. I‘m going to present a single use case that captures the essence of everything discussed thus far—namely, how I compose a business function of streams and stream processors.
This approach addresses:
Core function: Building an event streaming model for item bid activity and placement
Trust: Run it on rails with instrumentation and throughput monitoring
Control: Offline and rolling restart
1. Core function: Building the event streaming model for item bid activity and analytics

A user bidding app, which is part of the auction platform
Let’s step through the above event streaming application:
This FaaS function will enrich, filter, geo-encode and validate the item before it is accepted into the items stream. FaaS is a good fit because it is at the edge, and there won’t be competing events, i.e., the same user won’t submit the same item within milliseconds of each other. Once submitted, we enter the world of streaming. A stream processor instance is running a queryable KTable of items that can be used to query against available items.
Our next flow is the bidding against certain items. At this point, the client has queried the available items and is submitting bids to the /auctions/bids topic via a FaaS, which enriches the event to a geo-IP grid or provides address validation. The stream processor joins this stream with the items, creating an audit of bidding history stored on the /auctions/bids/history topic.
To determine when an item has expired (sold, unsold, reserve not met), each stream processor runs a future against its locally held item table (materialized view) so that upon completion, the original item is updated to break the join between steps 1 and 2 (i.e., join where item.status != complete).
The change of item condition emits an new event to /auction/items-complete that executes the item-complete processor. It looks at the bidding history where a series of FaaS events are triggered.
The FaaS analytic function group in the top right of the diagram receives the entire history of related events, generates bid offer spread analytics and notifies the users of how the auction played out. The notification FaaS function simply emails the bidding users in response to the initial set of success or failure events. In other words, you either purchased this item or you were outbid.
This functionality can be extended in many ways. We can leverage the live bidding stream for ad placement to attract bidders to similar items as the bidding time comes to a close, or set their expectations about the historically successful price range of similar items.
2. Trust: Run it on rails with Instrumentation and throughput monitoring
It’s always important to understand if the event streaming application is performing, if it is meeting throughput requirements and provides sufficient insight about how it is meeting business SLAs. More importantly, many newcomers to event-first design get extremely concerned about knowing that an asynchronous application is actually performing, failing or just working as expected; they don’t trust it. This is because event streaming apps need to be “run on rails.” You not only monitor the happy path but also track all other aspects like error handling with dead letter queues, business metrics and flow metrics.

Instrumentation plane tracking application-wide metrics
Business metrics are collected by stream processors to aggregate the data patterns and mapping them onto end-to-end business flows; these flows are identified by the source and elapsed milliseconds. Outliers are identified by tracking end-to-end latencies and drilling down to flow level metrics.
Flow metrics are the tracking individual components (microservices) where outliers and other analytics can be identified.
Dead letter queues are just queues that are used where data quality or corrupt messages are detected. In cases where event formats evolve, the dead letter queue might be used in back testing to understand compatibility issues.
3. Control: Offline and rolling restart

Restarting stream processing microservices
Offline
Streaming apps tend to run continuously. It is critical that events are not missed, and that any stateful operations are resumed without compromising the output stream. Event streaming platforms support the ability to allow consumers to be turned off while the producer still generates messages; they are stored in the log. Upon resuming processing responsibility, the consumer’s process will pick up from where it left off. Stateful operations will also be resumed using the event sourcing model, and state will be pushed into the changelog streams.
The appealing aspect of the event streaming platform is that not only are all streams replicated across multiple locations on the backend broker infrastructure but also that all paths of failure and accuracy are catered for. Offline resilience means that replacement containers can be spun up in a new location with a new application version, and they will simply resume processing.
Rolling restart
Maintenance and operational requirements mean that the applications will inevitably be redeployed and updated. In a 24×7 environment, we leverage the event streaming platform’s consumer group protocol. This protocol ensures that members (consumers) who join and leave the group are tracked and that partitions within a stream will be redirected or rebalanced amongst the consumers as the population changes.
As mentioned previously, stateful operations are also persisted as streams at the partition or event-key level. Flexing consumers up and down, or cycling through a set of consumers on a controlled fashion like a Kubernetes operator means that rolling restart and elastic scaling are supported natively. As in the offline scenarios, the new consumer processors simply event source the related state and carry on processing.

Being event first or event driven is about recognizing the value of events. It’s about going back to first principles, challenging every preconceived idea of how to build distributed systems. Being event first means we change our methodology to an interplay between modeling real world events as DDD, modeling events to capture those use cases and implementing the dataflow of those events for individual use cases. We also develop our monitoring and instrumentation as part of the implementation in a different event plane.
All of these elements compound upon each other with the repeated notion that the event captures a fact, a series of related events is a stream and that a stream tells a story. This is how we think about system design and architecture.

Streams capture events; a series of events capture behavior; they tell a story

Data evolution
Another benefit of the event streaming systems is that we can continuously extend the functionality. We can add new processors to extract different sets of intelligence stored within the event streaming patterns. This might be backtesting ad placement, driving and calibrating machine learning, identifying user behavior by geography or even tracking and identifying malicious users (HammerSnipe/auction stealing).
By storing only the events and never the commands, we have a wealth of capability that not only allows the system to be refined, extended and proven but also supports evolutionary change. As the domain or requirements change over time the platform can support the evolution of the events where new fields are added, stream processors might be updated or scaled and the architecture might change the event stream, which brings us nicely to the future state where we see streaming in the cloud.

The future of programming
Evolutionary architectures represent the next generation of thinking about cloud. The ability to compose data-driven, reactive architectures that leverage FaaS and elastic stream processing while pushing down infrastructure concerns is the journey we are on.

There are many facets to this journey, but this story is one which will take time to unfurl and become clear. We have seen many architecture revolutions in the past (e.g., big data and NoSQL), but it’s only when the principles of being event first are correctly applied to the data platform do the qualities of evolvability become attainable. Only then does it become possible for data and logic to change independently without compromising the future runtime.

We have now baked into our ideology that events are the future. Making this a reality is fun, but we are on a journey, one which will unfold as we continue to discover renewed value of going back to first principles of distributed computing.

FaaS as part of the event-driven, streaming architecture
Fundamentally speaking, you can leverage AWS Lambda integration with Amazon Kinesis or the Kafka Connect AWS Lambda Sink Connector to trigger events from streams. So it depends on the orientation of your development methodology. Do you want to stick to the cloud vendor platform or adopt a vendor-agnostic approach? Many organizations favor the latter, which is the approach I will use.

The above event stream shows FaaS functions on the edge processing part of our application. Real-time stateful operations are handled by stream processors; they interact with the log and with correctness guarantees due to the underlying streaming platform protocol. This model ensures temporal guarantees of stream-table joins, i.e., event time correctness.
Let’s step through the data flow above and focus on the FaaS functionality:
When item events are sent to the system, they are received via a FaaS function. This function will validate, enrich and either reject the event, or write it into the /auction/items stream
Similarly, bidding events enter the system via a FaaS function whose job is to validate, enrich and then store the event in the /auction/bids stream
As events pass through the dataflow, an item-complete event is eventually emitted. This triggers a join reaction, and the output is a series of events to notify bidders and calculate analytics.

The synergy of FaaS and the event streaming platform is a natural fit when we think about domain modeling and organizational needs as they change over time. Whether it’s business processes, data models, technology, cloud or just the organization itself, evolution is inevitable and needs to be embraced. We aren’t quite there yet, but it’s undeniable that FaaS is moving at a fast pace, along with streaming and cloud.
In the context of building event-driven microservices with Kafka or event streaming apps; they are all names for the same thing. The core concept that enables the functionality is the “event” and the ability to record, replay and react to events. Building systems around this principle is fine for streams processors and clients that speak the event streaming platform protocol; however, FaaS doesn’t. As such, we need to ensure that we FaaS functions are operated in such a way so as not to compromise the stream of events but instead be used for the right type of data workloads. Hence, I developed the Faas principles above.

It is not uncommon for those embarking on the journey to event-driven, asynchronous architecture to ask:

“Hey, so I’m writing these events, how do I know what’s happening and whether they’re working as intended? I don’t trust my application when it is asynchronous—I really want transactions.”

Let’s explore what this really means. We need to build something real, such as a payment system. Why a payment system, you ask? Well, pretty much every business out there needs to get paid, so it seems like a very common use case. It is easy to understand but also scary for developers who must trust an action that moves money from one account to another only using a transaction.

Transactions give you all or nothing (atomic) behavior. They provide a guaranteed level of trust; they are simple. When we do replace transactions with events, it can seem complicated. Yes, we can transact across partitions in Apache Kafka®, and with traditional database transactions you get many “trust” semantics for free; however, they are pushed down into the database runtime. The problem is that many traditional OLTP databases are also relatively slow when they need to scale. ?

Not only does the event streaming model expose the developer to building transactional semantics, scaling and error handling, etc., but it also means that they have more control, a richer ability to extend and enrich the system. Although this sounds like it adds complexity, the opposite is true.

We also have the opportunity to add layers of functionality to the payment system as it evolves. Some banks run up to 30 processing stages, which stretches well beyond the realms of a single, atomic transaction,

“We believe that the major contributor to this complexity in many systems is the handling of state and the burden that this adds when trying to analyse and reason about the system”

Four pillars of event streaming
Recognizing architectural pillars and patterns allows us to move away from complexity.

To adopt event-first thinking, we need to stop, take a breath and deeply understand the world of transaction processing, which we often take for granted, and then map its mechanics to the event-driven streaming world. (Note: We view event streaming as the next generation of event driven.) There are many orthogonal concepts we also need to understand, let alone the core business function of moving the money: How much money is being transferred? What is the latency? How do I upgrade or evolve microservices? Can I scale it up and down? What happens to bad payments? Can it be extended for different payment types? Which teams are going to run my system? Where is it going to run?
For this reason, I’m not only going to focus on the payment processing pipeline but also work through the other functionality required to architect a production-worthy solution. By nature anything that is event driven provides loose coupling. With event-first design, the data becomes the API which, like any production system, needs to support change and evolution (i.e., Avro or Protobuf). Furthermore, in embracing loose coupling, we inherit a clean separation of concerns (SoC), which provides the ability to develop and operate various aspects of the system architecture independently.
These aspects can be organized into a set of complementary pillars:
Business function: The payment processing pipeline
Instrumentation plane: How much money is transferred? How quickly are payments processed?
Control plane: Patterns to start, stop, pause, coordinate and autoscale
Operational plane: Dead letter queues (DLQs), error tracking & handling, logging and NoOps

1. Business function
The business function represents the core functionality being built. It provides business value that is immediate and beneficial to the organization. Examples include payment processing, logistics, data processing (ETL) and so on. This function can be developed as a set of events and related dataflows across one or more bounded contexts.

2. Instrumentation plane
The role of the instrumentation plane is to capture metrics that prove that the business function is performing sufficiently. Metrics may have a variety of uses—they might be used to drive alerts, assist in capacity planning or drive infrastructure automation functions such as auto-scaling.
In our case, business level monitoring is the ability to track metrics such as, payments in progress, failed payments and payment throughput. These could also be mapped onto application metrics, including the supporting dataflow volume, latency tracking and throughput. In the next level down, they can be mapped to the underlying broker infrastructure metrics, such as consumer lag, throughput per topic and partition hotspots, in addition to operating system metrics like CPU, network I/O, disk I/O, load average, etc.

3. Control plane
An often overlooked aspect of many systems is its ability to control the flow of events. Unlike batch systems, event streaming applications continuously process data. This means that while it might be possible to perform rolling upgrades, the system sometimes needs to halt processing, perform an action, such as an upgrade or logic change, and then resume processing. The control plane becomes essential when outages have occurred and a large-scale system must get back online in a coordinated fashion, perhaps with incremental, restricted functionality.
In the context of a payment system, you might need to drain the current set of payments for a topology change. But what if there is a regional outage—how do you bring the system back online without overloading downstream systems?

4. Operational plane
Developing the system for the “happy path” is easy—it’s catering for the remaining paths when simplicity can become compromised. Relying on pushdown semantics of container “restart on failure,” workload feedback cycles to drive elasticity and other underlying functionality—we need to understand all of this and more. We need to build a model for capturing operational processes, such as rolling restarts and wipe and update processes, and develop these in automation scripts to drive the infrastructure automation.
Ansible has emerged as a leading automation language and provides a wide range of plugins for configuration management, orchestration as well as anything else that might be needed when working with physical servers. For those with a Kubernetes ecosystem, the use of Helm, Helm Charts and Operator scripts is quite popular.

The most important aspect of operations is not that you have dead letter queues or the like, but rather that you have trusted processes. These processes must be deterministic; they must be reliable. The only way of validating operational processes is to use automation and testing. Testing requires appropriate instrumentation that serves as a validation mechanism. Testing operational processes becomes a lot easier when the systems being operated are decoupled and, most importantly, have all its events captured in Kafka. This means that Kafka is used not only to store and emit events but also playback scenarios, so you can pinpoint when things started going wrong.

Pillar 1 – Business function: Payment processing pipeline

Goal
To solve the business problem/objective(To handle millions of payments)

Requirements
Although I normally write these as user stories, I’m just going to cut to the chase with a collection of fictional requirements. The system needs to report how much money is inflight, process both sides of the payment and also confirm the total amount of payments successfully processed. In other words, it must:
Scale to millions of payments
Perform asynchronous processing
Track payments inflight (total value of payments being processed)
Remove/debit money from an account when handling a debit event
Add/credit money to an account when handling a credit event
Track payments confirmed today (total value of payments successfully processed)
.
Enhancements
The purpose of the demo is to demonstrate the four key architectural principles in practice. It can be improved with regard to:
Error handling: Implement sagas to handle unwinding on failed payments, perhaps when corrupted or fraudulent activity is detected
Pre-payment validation: Add a preliminary stage to support scenarios such as detecting and handling suspicious or fraudulent activity
Restrict dependent payments: When it makes sense, only allow a single debit payment for a single account to be inflight at any point in time
Limits: Introduce limit checks against accounts with low balances

Pillar 2 – Instrumentation plane: Business metrics
There are a plethora of monitoring tools that provide valuable functionality. Most of them are great for bottom-up metrics or infrastructure metrics.
Operating system and application infrastructure metrics are useful for overall knowledge, but the most important question to answer is: Am I meeting business goals?
The goal for our instrumentation plane is tracking business metrics. We need to know with certainty that the system is performing correctly and how system performance impacts the business. Are our users happy? Are we responding quickly enough to meet demand? Behavioral problems can often be identified through these business metrics because they tend to result from a repeated series of underlying interactions that comprise the function, the flow of events. Another aspect is that business metrics can be used to drive adaptive behavior. Examples include scaling up the processing capability, applying backpressure and modifying system SLAs.

Goal
To meet the system’s business goals/key results/measurable goals
Requirements
Let’s assume we have the following requirements, some of which should be quite obvious:
Process payments in < 200 milliseconds (latency)
Throughput > 1,000 per second
Report the total count and dollar amount (the demo code uses a one-minute window)
Event flow model

Pillar 3 – Control plane: Flow control, start, pause, bootstrap, scale and rate limit

Goal
To gracefully control the business system (Start, Stop, Pause, Restart) (To pause and resume payment processing)
Requirements
Pause processing and allow existing payments to be completed (drained)
Resume processing from the paused state
Event flow model

Pillar 4 – Operational plane: Event logging, DLQs and automation
Application logs: Each microservice will use a log appender to selectively send log messages to a specific topic that is relevant to the application context. The log message will also contain metadata about the runtime, perhaps including the input topic and output topic, consumer group as well as the source event information. These logs are relevant to business events and should be tied to the bounded context.
Error/warning logs: Similar to above, but the focus is on error/warning log events and sending to the relevant topic. It is also useful to build a log context that accumulates relevant information to enrich the log message if an error/warning does occur.
Audit logs: Each microservice will capture a security context (e.g., user, group, credential or reason) as well as event context (i.e., event ID, event origin, source and destination topic) and emit to a relevant, contextually determined audit log topic.
Lineage: Similar to above, except that the event will contain trace information about its journey—for example, each processor, or the context it has accessed or modified on its dataflow.
Dead letter queues: These provide a place to store raw events that cannot be processed. Processing failure can be due to any number of reasons, from deserialization failure to invalid values or invalid references (join failure, etc). In any case, the purpose of dead letter queues is to make you aware of the error so you can fix it. The dead letter queue is the last line of defense and might indicate a failing CI/CD process, in which a scenario has occurred that was not catered for. The dead letter queue should always be empty. In some cases, a saga pattern is needed to unwind the preceding state that caused the error. Remember, events must be processed idempotently.
These patterns all need to be associated with the bounded context in which they occur.

I started this post by talking about complexity. Complexity in system design is inevitable in any serious, large-scale system. To tackle complexity, you need control. To have control, you need to observe. To observe, you need to instrument. While control, observability and instrumentation are commonplace in many traditional systems, their implementation is subtly different in the world of events. We demonstrated this by reimagining a traditional payment system as a set of discrete stream processors. We saw how, despite being stateful, such a system could be scaled to accommodate digital-native workloads. Moreover, we saw how to layer control and observability over systems of this type, identifying four key pillars in the process—pillars that not only apply to payment processing but also to any event streaming system we may choose to build. These pillars are:
Core business flow: business functionality
Instrumentation plane: metrics and observability
Control plane: start, stop and scale
Operations plane: application support, error handling and dead letter queues

Unstarred

Dunk (exclude from study)

Journey to Event Driven

Review Questions

Todos

Sensemaker