Apache Kafka

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

https://kafka.apache.org/documentation/

Distributed Logs that are logs of events that are a better fit as a data structure for microservice and real time analytics/insight

Why Logs
Write once read many
Immutable
Offsets help find data (LtR)
Simple
Idempotent when read (Not deleted, or consumed)
unbounded

Distributed Systems
Availability: Failure is not an option. Kafka uses replication through commit logs
Scalability
Performance

Kafka Replication
Leader gets writes and followers continuously copy from leader

Kafka Partitioning
The way Kafka provides scalability through parallelization 
For each topic you choose your number of partitions and it's replicating factor
Replicas for each partition will always be stored on different brokers
Each partition represents a subset of a topic (sharding). Each partition is stored in a single location in the cluster.
Spread out partitions help you take advantage of spread out computing resources

Why Segments
Problem: How do we delete data we no longer need to retain?
Logs are unbounded. Logs are immutable
How do we delete from a file we might also be writing to simultaneously?
Solution: Separate log into segments. Delete segments we don't need

Data Retention
Time Based
log compaction - recent record for each key

Consumer Group
Scalability through Parallelism
Better Performance
Higher Throughput
Consumer offset is a marker for where last a consumer was reading from a log so it can pick back up from in case of failure.

Kafka Connect framework for integrations 
Kafka Connect applications are streaming applications. Real time. Constant flow

Compacted topic keys are required

Producer
Actions
Send from Application code
Producer API 
Actions
Serialize (JSON, CSV, Avro, Protobufs, XML)
Partition
Buffer + Batch to Broker
All customizable
Response
Acknowledge
Fail
Retry

Kafka doesn't care what you send to it as long as it's been converted into a byte stream beforehand. Data stored as a binary blob. Byte arrays

Partition Algorithms
Semantic partitioning. Equation is | hash(key) % numPartitions
Useful for ordering, aggregation
Round robin

Order is only possible on the partition level

There is technically a 1-1 relationship between consumers in a consumer group and partitions. If there are more consumers than partitions, that consumer will be idle

Use cases

https://www.kai-waehner.de/blog/2020/10/20/apache-kafka-event-streaming-use-cases-architectures-examples-real-world-across-industries/

Testing

https://aiven.io/blog/create-your-own-data-stream-for-kafka-with-python-and-faker

Tip #5: Use the power of record headers
Apache Kafka 0.11 introduced the concept of record headers. Record headers give you the ability to add some metadata about the Kafka record, without adding any extra information to the key/value pair of the record itself. Consider if you wanted to embed some information in a message, such as an identifier for the system from which the data originated. Perhaps you want this for lineage and audit purposes and in order to facilitate routing of the data downstream.
Why not just append this information to the key? Then you could extract the part needed and you would be able to route data accordingly. But adding artificial data to the key poses two potential problems.
First, if you are using a compacted topic, adding information to the key would make the record incorrectly appear as unique. Thus, compaction would not function as intended.
For the second issue, consider the effect if one particular system identifier dominates in the records sent. You now have a situation where you could have significant key skew. Depending on how you are consuming from the partitions, the uneven distribution of keys could have an impact on processing by increasing latency.
These are two situations where you might want to use headers. The original KIP proposing headers provides some additional cases as well:
Automated routing of messages based on header information between clusters
Enterprise APM tools (e.g., Appdynamics or Dynatrace) need to stitch in “magic” transaction IDs for them to provide end-to-end transaction flow monitoring.
Audit metadata is recorded with the message, for example, the client-id that produced the record.
Business payload needs to be encrypted end to end and signed without tamper, but ecosystem components need access to metadata to achieve tasks.

https://www.confluent.io/blog/5-things-every-kafka-developer-should-know/

Tip #4: Master the command line tools
The Apache Kafka binary installation includes several tools located in the bin directory. While you’ll find several tools in that directory, I want to show you the four tools that I think will have the most impact on your day-to-day work. I’m referring to the console-consumer, console-producer, dump-log, and delete-records.

Tip #3: Avoid “stop-the-world” consumer group rebalances by using cooperative rebalancing
Kafka is a distributed system, and one of the key things distributed systems need to do is deal with failures and disruptions—not just anticipate failures, but fully embrace them. A great example of how Kafka handles this expected disruption is the consumer group protocol, which manages multiple instances of a consumer for a single logical application. If an instance of a consumer stops, by design or otherwise, Kafka will rebalance and make sure another instance of the consumer takes over the work.

Tip #2: Learn about the new sticky partitioner in the producer API
Kafka uses partitions to increase throughput and spread the load of messages to all brokers in a cluster. Kafka records are in a key/value format, where the keys can be null. Kafka producers don’t immediately send records, instead placing them into partition-specific batches to be sent later. Batches are an effective means of increasing network utilization. There are three ways the partitioner determines into which partition the records should be written.
The partition can be explicitly provided in the ProducerRecord object via the overloaded ProducerRecord constructor. In this case, the producer always uses this partition.
If no partition is provided, and the ProducerRecord has a key, the producer takes the hash of the key modulo the number of partitions. The resulting number from that calculation is the partition that the producer will use.
If there is no key and no partition present in the ProducerRecord, then previously Kafka used a round robin approach to assign messages across partitions. The producer would assign the first record in the batch to partition zero, the second to partition one, and so on, until the end of the partitions. The producer would then start over with partition zero and repeat the entire process for all remaining records.

Building Streaming Microservices with Apache Kafka - Tim Berglund
Kafka
The fundamental abstraction of Kafka is a log
Pub/Sub and Storing
EDA/Event Sourcing roots are from Domain Driven Design
Stream processing, 
Events
Serve as notifications. Thing as happened
Carry state. Data. Information rich
Streaming
Stream Pipeline
Materialized Views
Built in Scaling
Compute on write
Example
App Opened 
App Crashed
App Opened/Crashed. = Stream join for unstable applications

Event-driven Architectures on Apache Kafka with Spring Boot

Kafka streaming with Spark and Flink

Apache Spark is an open-source platform for distributed batch and stream processing, providing features for advanced analytics with high speed and availability.

On the other hand, Apache Flink is an open-source framework for distributed stream data processing, mostly focused on providing low latency and high fault tolerance data processing.

there are two types of stream processing
Native streaming (Flink): data records are processed as soon as they arrive, without waiting a specific amount of time for other records;
Micro-batching (Spark): data records are grouped into small batches and processed together with some seconds of delay.

https://aws.amazon.com/msk/what-is-kafka/#:~:text=Why%20would%20you%20use%20Kafka,that%20consumes%20streams%20of%20data.

https://developers.redhat.com/articles/2022/05/03/fine-tune-kafka-performance-kafka-optimization-theorem

Unstarred

Dunk (exclude from study)

Apache Kafka

Review Questions

Todos

Sensemaker