Sizing, Performance and Cost

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

How to read this
Each layer opens with Foundations that teach the concept from first principles, then moves to patterns. Every pattern uses the same eight fields: overview, decision variables, mechanism, Confluent view, failure modes, resource constraints, observability, and competitive angle.
This domain builds directly on the Kafka internals deep dive. Batching, partitions, replication, consumer groups, and tiered storage all reappear here as inputs to sizing and cost.
Prices change often and vary by cloud, region, and contract. This document teaches the structure of cost, not current list prices. Items marked (verify) should be checked against current Confluent pricing pages, cloud provider pricing, and internal enablement before you quote them.
The equation that ties this domain together
Almost every sizing, performance, and cost question reduces to the same chain:
  WORKLOAD                 x   EFFICIENCY                =   RESOURCES          ->   COST
  ---------                    ----------                    ---------               ----
  bytes in per second          compression ratio             throughput units        capacity units
  read fan-out                 batch size                    storage                 throughput
  retention                    request rate per MB           partitions              storage
  partitions, connections      client behaviour              connections, requests   networking
                                                             network paths           people and risk
The workload is what the business needs. Efficiency is how well clients and topic design use the platform. Resources are what must be provisioned or consumed. Cost is what the customer pays, including the parts that never appear on an invoice.
An elite SE can move along this chain in both directions. Forward, they turn a workload into a defensible bill. Backward, they explain a surprising bill by tracing it to a workload or efficiency problem, and then fix it.
6.1 Workload characterisation
The one-sentence story: you cannot size, price, or benchmark what you have not measured, and most customers do not actually know their numbers, so the SE's first job is to turn vague descriptions into a precise, defensible workload profile.
Foundations
Why sizing starts with the workload
A common mistake is to start a sizing conversation with the platform: "how many brokers?" or "which cluster type?". That question has no answer until you know what the workload demands. Every capacity unit, every price, and every performance claim is a function of the workload, so a weak workload profile poisons everything downstream.
The workload profile is also where trust is won or lost. A customer's platform team will check your numbers against their own dashboards. If your profile matches what they see, your sizing and cost model inherit that credibility. If it does not, nothing you say afterwards will be believed.
The six dimensions of a Kafka workload
A Kafka workload is not one number. It has six dimensions, and any one of them can be the one that determines capacity and cost.
  1. WRITE THROUGHPUT    bytes/s and records/s produced
  2. READ THROUGHPUT     bytes/s consumed  = write throughput x fan-out
  3. RETENTION           how long data is kept -> stored bytes
  4. PARTITIONS          total partitions, including compacted ones
  5. CONNECTIONS         concurrent client connections and new connections/s
  6. REQUESTS            produce, fetch, and metadata requests per second
The first three are about data volume. The last three are about coordination overhead, and they are driven as much by client behaviour and topic design as by data volume. Many sizing mistakes come from modeling only the first dimension.
Write throughput: bytes and records
Write throughput (ingress) is usually the first number customers offer, often as "we do about 2 billion events a day". That number needs translating twice: into bytes and into a rate.
  2,000,000,000 events/day / 86,400 s  ~ 23,000 events/s average
  average event size 1 KB              ~ 23 MB/s average (uncompressed)
  compression ratio 4:1                ~ 6 MB/s average (compressed, on the wire)
The record rate matters separately from the byte rate. Ten thousand 100-byte records per second and ten 100 KB records per second are both 1 MB/s, but the first generates far more per-record overhead and, if poorly batched, far more requests. Always capture both, plus the message size distribution (median and large-message tail), because a few very large messages can hit size limits or distort batching.
Compression deserves special attention. Kafka stores and transmits data as the producer compressed it, so compressed bytes are what drive network, storage, and most throughput billing. Customers often quote uncompressed sizes from their application's point of view. Always ask which they mean.
Average, peak, and burst
Daily averages hide what actually sizes a system. Real traffic follows business rhythms: daytime peaks, end-of-month batch jobs, Black Friday, a marketing push, a backfill.
  ingress MB/s over a typical day

400 |                        ___                burst (backfill / replay)
   300 |                 ______/   \___      ___
   200 |          ______/              \____/   \
   100 |   ______/                               \______
     0 |__/____________________________________________\__
         00   03   06   09   12   15   18   21   24  (hour)

average ~ 150 MB/s    daily peak ~ 300 MB/s    burst ~ 400 MB/s
Three numbers matter:
Average drives throughput-based charges and long-run storage.
Sustained peak (the busiest hour) drives capacity, because the system must handle it without throttling.
Burst (short spikes, replays, backfills) drives headroom and how quickly capacity must scale.
The ratio of peak to average determines how much an elastic model saves compared with a provisioned one. A workload that peaks at 3× its average for two hours a day is paying for idle capacity most of the time on fixed infrastructure.
Read throughput and fan-out
Every consumer group reads its own full copy of the data it subscribes to. Read throughput (egress) is therefore write throughput multiplied by fan-out, the number of consumer groups reading each topic.
  topic "orders": 50 MB/s in

consumer groups: fraud, fulfillment, analytics-sink, search-index, audit
  fan-out = 5

egress = 50 MB/s x 5 = 250 MB/s
Fan-out is one of the most underestimated numbers in sizing. Customers know what they produce, but reads are spread across many teams, and new consumers get added over time. Ask about every consumer, including connectors, stream processing jobs, mirroring, and ad hoc tools. Replays and backfills also add bursts of egress.
Kafka was designed for high fan-out, which is why capacity units typically allow roughly three times as much egress as ingress. A workload with fan-out above that ratio is egress-bound.
Retention and the storage footprint
Storage is a function of throughput and time. Using compressed ingress:
  stored bytes (pre-replication)  = average ingress x retention
  stored bytes (post-replication) = pre-replication x replication factor
For example, 6 MB/s average compressed ingress kept for 7 days is about 3.6 TB before replication and about 10.9 TB after three-way replication. Confluent Cloud bills storage on the post-replication volume, so the billed figure is typically about three times what was written.
Retention also has a design side. Ask why the customer keeps data as long as they do. Often the honest answer is "it was the default", or "we want replay for incidents", which may need only days, not weeks. Compacted topics are different: their size is driven by the number of distinct keys and value sizes, not by time.
Partitions, connections, and requests
The coordination dimensions often surprise teams that sized only for data volume.
Partitions: count every partition the workload needs, including compacted partitions and the internal topics of self-managed Connect workers or Kafka Streams apps. In Confluent Cloud, limits count pre-replication (leader) partitions, and some cluster types have a separate, lower limit for compacted partitions.
Connections: each client instance typically opens connections to multiple brokers. A microservices estate with thousands of pods, each with a producer and a consumer, can reach tens of thousands of connections even at modest throughput. Connection attempts per second matter too: frequent restarts, autoscaling, and short-lived clients create connection churn.
Requests: every produce, fetch, and metadata call is a request. Request rate is driven by batching and fetch settings far more than by data volume (layer 6.3 covers this).
The binding dimension
Capacity units bundle limits across all six dimensions. Whichever dimension needs the most units decides the size. That is the binding dimension.
A worked example on a Confluent Cloud Enterprise cluster, using the published planning limits per eCKU (about 60 MB/s ingress, 180 MB/s egress, 3,000 partitions, 18,000 connections, and 7,500 requests per second; check current docs before quoting):
  peak workload                       units needed
  -------------                       ------------
  ingress     150 MB/s   / 60     =   2.5
  egress      600 MB/s   / 180    =   3.3   <-- binding dimension
  partitions  1,200      / 3,000  =   0.4
  connections 6,000      / 18,000 =   0.3
  requests    20,000/s   / 7,500  =   2.7

peak capacity ~ 4 eCKU, driven by egress (fan-out of 4)
The binding dimension also tells you where optimization pays off. In this example, reducing fan-out or compressing better would shrink capacity; reducing partitions would not. Published limits are planning guidelines that interact (using one dimension heavily reduces what others can achieve), so treat this as a first estimate, not a guarantee.
Growth and change over time
A workload profile is a snapshot. Ask how each dimension is expected to change: new use cases, customer growth, new consumers, regulatory retention changes, a migration of more apps onto the platform. Size the initial footprint for today, but make sure topic design (especially partition counts on keyed topics) and commitments account for the next one to three years.
Where the numbers come from
Workload data comes in three levels of quality, and good SEs push toward the highest available:
Measured: metrics from an existing Kafka or messaging cluster (broker bytes in and out, partition counts, connection counts, request rates). The most credible.
Derived: cloud bills, log volumes, database change rates, or application event counts converted into bytes and rates.
Estimated: application teams' educated guesses. Useful early, but label them clearly and replace them as soon as possible.
For a self-managed Kafka estate, the key broker metrics are BytesInPerSec and BytesOutPerSec (excluding replication traffic), the global partition count, socket server connection counts and creation rates, and request rates by type (consumer fetches separate from follower fetches, which are internal replication).
The mental model to carry forward
A workload is six dimensions, not one, and the binding dimension decides capacity. Capture averages, peaks, and bursts, compressed and uncompressed sizes, and fan-out, and label every number with where it came from. A precise workload profile is the foundation of every credible sizing, performance, and cost conversation that follows.
Patterns
6.1.1 Throughput profiling
Overview: Converting business-level volume ("events per day") into the byte and record rates, at average, peak, and burst, that capacity and cost depend on.
Decision variables: Measurement window (hourly peaks vs daily averages), compressed vs uncompressed basis, message size distribution, seasonality, planned backfills or migrations.
Mechanism: Start from event counts or measured broker metrics. Convert to bytes using actual message sizes. Apply the expected compression ratio (measure it on real payloads if possible). Identify the busiest hour and the largest bursts. Express results as MB/s and records/s at average, sustained peak, and burst.
Confluent view: Elastic cluster types bill for the maximum eCKU used each hour, so hourly peaks drive capacity charges while total bytes drive ingress and egress charges. Dedicated clusters are sized to the sustained peak plus headroom. The Confluent cost estimator and sizing workbooks expect these inputs.
Failure modes: Sizing on daily averages and throttling at peak. Using uncompressed numbers and overestimating cost several times over. Ignoring replays and backfills, which can briefly multiply traffic. Missing a large-message tail that exceeds message size limits.
Resource constraints: Ingress capacity per unit and per partition (per-partition guidelines are a few MB/s to around 12 MB/s depending on cluster type), producer CPU for compression.
Observability: Broker BytesInPerSec and MessagesInPerSec at one-minute resolution, producer outgoing-byte-rate, record-send-rate, compression-rate-avg; in Confluent Cloud, received bytes and records via the Metrics API.
Competitive angle: Competitors' calculators are equally sensitive to these inputs. When comparing quotes, insist every vendor uses the same peak, average, compression, and fan-out assumptions, or the comparison is meaningless.
6.1.2 Fan-out and egress modeling
Overview: Determining how many times each byte is read, which multiplies egress, network, and request load.
Decision variables: Number of consumer groups per topic, connectors and stream processors, cross-region mirroring, ad hoc and tooling consumers, replay frequency, whether consumers read whole topics or filtered subsets.
Mechanism: For each topic, list every consumer group. Egress = Σ (topic ingress × consumer groups reading it). Add mirroring (each cluster link or mirror reads the data again), stream processing (which reads input and writes new output topics that are themselves consumed), and expected replays.
Confluent view: Capacity units allow about three times as much egress as ingress, reflecting typical fan-out. Egress is billed per GB read. Stream processing in Flink can reduce downstream fan-out by filtering or aggregating once and publishing smaller derived topics.
Failure modes: Forgetting connectors, Flink jobs, and monitoring tools as consumers. Teams each creating their own consumer of a large raw topic just to filter it. Underestimating replay storms after incidents. Fan-out growing over time as new teams adopt the platform.
Resource constraints: Egress capacity, broker network, request rate from many fetching consumers.
Observability: BytesOutPerSec (excluding follower fetch), number of active consumer groups per topic, per-group consumed bytes; Confluent Cloud sent bytes metric.
Competitive angle: Fan-out is where log-based systems beat queue-based ones economically, since reads do not duplicate storage. In cloud-provider comparisons, check whether egress to consumers in other zones incurs separate cloud data transfer charges.
6.1.3 Retention and storage footprint
Overview: Translating retention requirements into stored bytes and cost, and challenging retention that does not serve a purpose.
Decision variables: retention.ms and retention.bytes per topic, compacted vs delete topics, compliance minimums and maximums, replay requirements, tiered storage availability.
Mechanism: Pre-replication storage ≈ average compressed ingress × retention for delete topics; distinct keys × average record size for compacted topics. Multiply by replication factor for physical storage. In Confluent Cloud, storage is billed per GB-hour on post-replication volume.
Confluent view: Confluent Cloud offers effectively unlimited storage per partition on most cluster types, backed by tiered object storage, so long retention is a pricing question rather than a hardware problem. This supports replay, reprocessing, and "Kafka as a source of truth" positioning.
Failure modes: Default 7-day retention applied everywhere without thought. Very long retention set "just in case" on high-volume topics. Compacted topics growing unexpectedly due to high key cardinality. Storage costs surprising finance because post-replication billing was not explained.
Resource constraints: Storage cost, and in self-managed Kafka, broker disk size and recovery time.
Observability: Partition and topic size, retained bytes metric in the Metrics API, storage line items in the Costs API.
Competitive angle: Some competitors price storage very low (especially object-storage-native designs) while others charge separately for extended retention. Normalize on post-replication vs logical storage before comparing prices.
6.1.4 Metadata, connection, and request profile
Overview: Profiling the coordination dimensions (partitions, connections, connection attempts, requests) that can bind capacity independently of data volume.
Decision variables: Total partitions and compacted partitions, number of client instances and their connection patterns, client lifecycle (long-lived services vs short-lived jobs and functions), batching and fetch settings.
Mechanism: Count partitions per topic, including internal topics for self-managed Connect and Kafka Streams. Estimate connections as client instances × brokers each connects to (in managed services, measure rather than guess). Measure request rates from existing clusters, separating consumer fetches from internal follower fetches.
Confluent view: Confluent Cloud limits partitions (pre-replication), connections, connection attempts, and requests per eCKU or CKU, and Confluent began enforcing request and connection limits more strictly on elastic cluster types during 2026. Partitions are not billed directly, but they can drive eCKU count.
Failure modes: A microservices estate binding on connections at low throughput. Serverless functions creating a new connection per invocation and hitting connection-attempt limits. Topic sprawl binding on partitions. Untuned clients binding on requests.
Resource constraints: Broker network threads, request handler threads, memory per connection and partition.
Observability: Connection count and creation rate, request rate by type, partition counts vs limits, the Confluent Cloud eCKU usage chart (which shows which dimension is driving scaling).
Competitive angle: Competitors often advertise throughput but bury partition and connection limits. Ask for all six dimensions in any comparison.
6.1.5 Discovery: gathering credible numbers
Overview: The process of collecting workload data from a customer quickly and credibly.
Decision variables: Data source availability (existing cluster, cloud bills, application telemetry), customer willingness to share metrics, time available before a proposal.
Mechanism:
Ask for broker metrics exports from existing clusters (one to two weeks at fine resolution, including a busy period).
Gather topic inventories: partitions, retention, cleanup policy, replication factor.
List consumer groups and connectors per topic.
Validate application estimates against measured data where possible.
Label every number with its source and confidence, and review the profile with the customer's platform team.
Confluent view: Confluent provides migration assessment tooling and sizing workbooks (verify current internal tools). A shared workload profile reviewed with the customer becomes the basis for sizing, pricing, and later the migration plan.
Failure modes: Accepting one headline number and extrapolating. Not reviewing the profile with the platform team, then losing credibility when they spot errors. Using peak-day data as the steady state, or quiet-week data as the peak.
Resource constraints: Customer time and access; metric retention in their monitoring systems.
Observability: The customer's existing monitoring stack (Prometheus, Datadog, Grafana, cloud provider metrics) is the source of truth.
Competitive angle: The SE with the most accurate workload profile usually frames the evaluation. Competitors who quote from headline numbers are easy to outflank by showing a more precise, customer-validated model.
6.2 Performance mechanics
The one-sentence story: throughput, latency, and durability trade against each other in predictable ways, and an SE who can explain exactly where time goes in a Kafka pipeline can diagnose performance problems and expose misleading benchmarks.
Foundations
Three properties that pull against each other
Every Kafka performance conversation involves three properties, plus a fourth that customers care about just as much:
Throughput: how much data moves per second.
Latency: how long one record takes to travel from producer to consumer.
Durability: how safe an acknowledged record is (1.3 in the internals deep dive).
Cost: what resources are consumed to achieve the other three.
                  THROUGHPUT
                     /\
                    /  \      bigger batches, compression
                   /    \     -> more throughput, lower cost,
                  /      \       a few ms more latency
                 /        \
          LATENCY -------- DURABILITY
     smaller batches,       acks=all, min.insync.replicas=2
     acks=1                 -> safer, adds a replication round trip
     -> lower latency,
        lower throughput, more cost
No configuration maximizes all three. Every tuning recommendation is a choice about which property to favour for a given workload. A payments topic favours durability; a telemetry firehose favours throughput and cost; a trading signal favours latency. The SE's job is to identify which matters most and tune for it, rather than applying one set of defaults everywhere.
Why batching raises throughput
Every request has a fixed cost regardless of how much data it carries: network round trip, request parsing, locking, disk append, replication bookkeeping, response. Batching spreads that fixed cost across many records.
  fixed overhead per request = 1 unit,  per-record cost = 0.01 units

1 record per request:      1 + 0.01      = 1.01 units per record
  100 records per request:   (1 + 1) / 100 = 0.02 units per record
  1,000 records per request: (1 + 10)/1000 = 0.011 units per record
The gain is enormous at first and then flattens. Going from 1 to 100 records per request cuts cost per record roughly fiftyfold; going from 100 to 1,000 cuts it less than twofold. This is why moderate batching (a few milliseconds of linger, tens to hundreds of kilobytes per batch) captures most of the benefit, and why tiny batches are so destructive.
Batching also compresses better (more repetition within a batch), which reduces network, storage, and billable bytes. In managed services, it lowers the request rate, which is itself a capacity dimension.
Pipelining and in-flight requests
A single producer connection can have several requests in flight at once (max.in.flight.requests.per.connection). The relationship between throughput, latency, and concurrency follows Little's law:
  throughput = data in flight / round-trip latency

one 64 KB batch in flight, 5 ms round trip:   64 KB / 5 ms   ~ 12.8 MB/s
  five 64 KB batches in flight, 5 ms round trip: 320 KB / 5 ms ~ 64 MB/s
This explains two common observations. First, higher network latency (cross-region, or a slow link) directly reduces per-connection throughput unless batches grow or more requests are in flight. Second, a producer that waits synchronously for each send has exactly one record in flight and terrible throughput.
Where the time goes: end-to-end latency
End-to-end latency is the time from the application calling send() to a consumer's application code receiving the record. It is a sum of stages, and knowing them lets you find the real bottleneck instead of guessing.
  PRODUCER
  1. queue time in accumulator      (waiting for linger.ms or a full batch)
  2. serialization + compression
  3. network to leader
  BROKER
  4. request queue + append         (broker load, disk)
  5. replication to ISR             (only with acks=all; one follower round trip)
  6. high watermark advances        (record becomes visible to consumers)
  7. response to producer           (affects producer latency, not consumer visibility)
  CONSUMER
  8. fetch wait                     (fetch.max.wait.ms / fetch.min.bytes)
  9. network to consumer
  10. decompression + deserialization
  11. queue in poll() buffer        (records fetched but not yet processed)
  12. application processing
In a healthy cluster within one region, broker-side stages (4 to 6) typically take single-digit to low tens of milliseconds. In practice, end-to-end latency is often dominated by client-side stages: linger and batch fill on low-traffic producers, consumer fetch waits, and especially consumer-side queueing when processing cannot keep up. When a customer reports "Kafka latency", measure each stage before blaming the brokers.
Latency distributions and tails
Average latency is nearly useless for real systems. Users and downstream services experience the slow requests, and when one business operation fans out into many Kafka round trips, the slowest one decides the total.
  latency distribution for produce requests

count
   |  ####
   |  ######
   |  ########
   |  ##########
   |  ###########                                      ^
   |  ############ ##                                  | the tail:
   |  ############ #### ##   #       #             #   | p99, p99.9
   +----------------------------------------------------> latency
      p50 = 4 ms        p99 = 25 ms             p99.9 = 180 ms
Common causes of tail latency in Kafka:
Garbage collection pauses in JVM clients or brokers.
Rebalances pausing consumption (1.5 in the internals deep dive).
Leader elections after broker restarts or failures, forcing clients to refresh metadata and retry.
Catch-up reads from lagging consumers pulling cold data from disk and evicting hot data from the page cache.
Throttling when a client or cluster reaches its quota or capacity limits.
Noisy neighbours in shared infrastructure, whether self-managed multi-tenant clusters or poorly isolated services.
Slow followers with acks=all, since the leader waits for every ISR member.
Always ask for p99 and p99.9 targets, and always measure tails over long enough periods to capture rare events like rolling restarts.
The consumer side of performance
Consumer performance usually depends less on Kafka and more on processing. Kafka can deliver records far faster than most applications can process them when processing involves database writes, API calls, or model inference.
Two levers matter most:
Parallelism: a consumer group's maximum parallelism equals the partition count. If each record takes 5 ms to process, one consumer thread handles about 200 records per second; 10,000 records per second needs about 50 threads, and therefore at least 50 partitions or a within-partition parallel processing pattern.
Fetch behaviour: fetch.min.bytes and fetch.max.wait.ms make the broker wait to accumulate data before responding, trading a little latency for fewer, larger fetches. max.partition.fetch.bytes and max.poll.records control how much data each fetch and poll returns, and therefore consumer memory and processing time per poll.
Throttling: how a managed service protects itself
When a client exceeds its quota, or a cluster approaches its capacity, Kafka does not reject requests outright. Brokers delay their responses, a mechanism called throttling. Clients slow down automatically because they wait longer for responses.
Throttling shows up in client metrics as produce-throttle-time-avg and fetch-throttle-time-avg. Non-zero values mean the client is being slowed by policy, not by a fault. In Confluent Cloud, high cluster load can lead to throttling and delayed new connections to keep the cluster available. Recognizing throttling quickly prevents long, misguided investigations into "slow brokers".
Benchmarks: why they mislead
Because performance depends on so many variables, benchmarks are easy to make misleading, intentionally or not. The variables that swing results most:
Message size and compressibility (random bytes do not compress; real JSON does).
Batching settings (linger.ms, batch.size).
Durability settings (acks, replication factor, min.insync.replicas, fsync policy).
Fan-out (zero consumers vs many).
Partition count and key distribution.
Duration (minutes hide garbage collection, compaction, segment rolls, and cache effects that appear over hours).
Whether latency is measured correctly under load ("coordinated omission": a load generator that waits for slow responses before sending more hides tail latency).
A benchmark is meaningful only if it mirrors the customer's real workload on all of these. Section 6.2.4 turns this into a methodology.
The mental model to carry forward
Performance is a set of trades among throughput, latency, durability, and cost. Batching amortizes fixed overhead; pipelining keeps the network busy; replication adds a round trip for safety. End-to-end latency is a chain of stages, often dominated by clients rather than brokers, and tails matter more than averages. Always identify which property the workload values most, measure each stage, and distrust any benchmark that does not look like the customer's workload.
Patterns
6.2.1 Tuning profiles: throughput, latency, durability
Overview: Pre-defined configuration profiles that favour one property, used as starting points for each workload class.
Decision variables: Business priority for the topic, acceptable end-to-end latency, durability requirement, message rate per partition.
Mechanism:
Throughput profile: larger batch.size (e.g. 100 to 200 KB), linger.ms of 10 to 100 ms, efficient compression (lz4 or zstd), larger fetch.min.bytes, moderate partition counts.
Latency profile: low linger.ms (0 to 5 ms), smaller batches, minimal fetch wait, fast or no compression, avoid queueing in consumer processing.
Durability profile: acks=all, idempotence on, min.insync.replicas=2 with replication factor 3, generous delivery.timeout.ms, handle send failures explicitly.
Most production workloads combine the durability profile with throughput-leaning batching.
Confluent view: Confluent publishes optimization guidance organized around exactly these goals (throughput, latency, durability, availability). In Confluent Cloud, durability settings are largely fixed by the service, so client batching and compression are the main levers.
Failure modes: One configuration copied across every service regardless of needs. Latency-tuned producers on high-volume topics flooding brokers with requests. Throughput-tuned linger on latency-critical alerting topics.
Resource constraints: Producer memory for larger batches, CPU for compression, request rate capacity.
Observability: batch-size-avg, record-queue-time-avg, request-latency-avg, records-per-request-avg, end-to-end latency measured with timestamps or tracing headers.
Competitive angle: Competitors often present one headline number achieved with a throughput profile and another with a latency profile, implying both at once. Ask which profile produced which number.
6.2.2 End-to-end latency budgeting
Overview: Breaking a latency requirement ("under 200 ms end to end") into per-stage budgets so each component can be designed and measured.
Decision variables: Target percentile (p50, p99, p99.9), stages in the pipeline (producer, Kafka, stream processing, sink), network distances, processing complexity.
Mechanism: List every stage from source event to final effect. Assign each a budget. Measure actuals using event timestamps and tracing headers. Optimize the largest contributor first, which is frequently consumer-side queueing or stream processing windows, not broker time.
Confluent view: For latency-sensitive workloads, Confluent recommends Enterprise or Dedicated clusters with tuned clients. Freight clusters trade latency (on the order of seconds) for cost and are unsuitable for tight budgets. Flink windowing and checkpoint intervals also enter the budget for processed results.
Failure modes: Blaming Kafka for latency added by batch-oriented sinks or window sizes. Ignoring cross-region hops. Measuring only producer acknowledgement latency and calling it end to end.
Resource constraints: Lower latency usually means more requests, smaller batches, and higher cost per byte.
Observability: Producer request-latency-avg and request-latency-max, consumer fetch-latency-avg, records lag converted to time, application traces across stages.
Competitive angle: Ultra-low-latency claims from thread-per-core engines are real in some benchmarks, but end-to-end application latency is usually dominated by stages those engines do not change. Bring the conversation back to the customer's full pipeline.
6.2.3 Tail latency management
Overview: Controlling p99 and p99.9 latency, which drives user-facing performance and SLA compliance.
Decision variables: Tail latency targets, client runtime (JVM tuning, language), consumer group protocol, deployment practices, isolation of noisy workloads.
Mechanism: Identify tail causes (GC, rebalances, leader elections, catch-up reads, throttling) using correlated metrics. Mitigate each: modern garbage collectors and heap sizing in JVM clients; the new consumer group protocol and static membership; controlled shutdowns and rolling deployments; tiered storage to isolate historical reads; capacity headroom to avoid throttling.
Confluent view: Confluent Cloud handles broker-side tail causes (balancing, broker replacement, upgrades) and isolates tenants on elastic types. Client-side causes remain the customer's responsibility, which is where SE guidance adds value.
Failure modes: Tails spiking during every deployment due to rebalances. Batch replay jobs on the same cluster as latency-critical consumers. Running clusters hot, so any burst triggers throttling.
Resource constraints: Headroom costs money; isolation may mean separate clusters.
Observability: Latency histograms (p99, p99.9) rather than averages, throttle time metrics, rebalance rate, leader election rate, GC pause metrics in clients.
Competitive angle: Ask competitors for tail latency over long runs including rolling restarts and failures, not just steady-state p99 over a few minutes.
6.2.4 Benchmarking methodology
Overview: Designing benchmarks that reflect the customer's real workload, so results are meaningful and defensible.
Decision variables: Message sizes and payloads, key distribution, partition counts, durability settings, fan-out, duration, failure scenarios, client libraries and versions.
Mechanism:
Use real or realistic payloads (captures true compression ratio).
Match production durability (acks=all, replication factor 3, min.insync.replicas=2).
Match fan-out with real consumer groups.
Run for hours, including a broker or zone failure and a rolling restart.
Measure end-to-end latency percentiles with a load generator that avoids coordinated omission (open-loop, fixed-rate load).
Run the same test on every candidate platform with identical client settings.
Confluent view: Confluent supports customer benchmarks during POCs and publishes benchmark methodology. Tools include kafka-producer-perf-test (newer versions add a warm-up option), the Open Messaging Benchmark, and custom harnesses.
Failure modes: Benchmarks with acks=1 and no consumers. Random payloads that cannot compress. Ten-minute runs. Comparing a tuned competitor setup against default Confluent settings, or the reverse.
Resource constraints: Load generator capacity (an underpowered generator becomes the bottleneck), test duration, cost of running benchmarks.
Observability: Throughput achieved vs target, latency histograms, throttle metrics, broker or cluster load metrics during the run.
Competitive angle: Benchmark wars are common in Kafka deals. The winning move is usually to propose a fair, customer-specific benchmark and its rules up front, which neutralizes cherry-picked vendor results.
6.2.5 Consumer throughput and fetch tuning
Overview: Making consumers keep up, which usually means scaling processing rather than tuning Kafka.
Decision variables: Per-record processing time, partition count, consumer instances and threads, max.poll.records, fetch.min.bytes, fetch.max.wait.ms, max.partition.fetch.bytes, ordering requirements.
Mechanism: Required parallelism ≈ record rate × processing time per record. If parallelism exceeds partition count, either add partitions (with care for keyed topics), use a within-partition parallel processing pattern that preserves per-key order, or batch processing (e.g. bulk database writes). Tune fetch settings to balance latency and request rate.
Confluent view: Confluent's open-source Parallel Consumer library supports per-key concurrency within a partition. Share groups (Kafka 4.2) serve workloads that can drop ordering. Flink scales processing independently of client code.
Failure modes: Adding consumers beyond partition count. Slow processing exceeding max.poll.interval.ms and causing rebalance loops. Huge fetch sizes causing out-of-memory errors.
Resource constraints: Consumer CPU, memory, downstream system capacity (often the true bottleneck).
Observability: Records lag and lag in time, records-consumed-rate, time-between-poll-max, fetch-latency-avg, downstream system latency.
Competitive angle: Queue-based competitors scale consumers without partition limits. Counter with share groups and parallel consumer patterns for the cases that need it, while keeping ordering and replay elsewhere.
6.2.6 Throttling and quota behaviour
Overview: Understanding how clients behave when they hit quotas or cluster capacity, and distinguishing throttling from faults.
Decision variables: Client quotas, cluster capacity and maximum eCKU settings, headroom policy, alerting thresholds.
Mechanism: Brokers compute whether a client exceeds its byte rate or request rate quota and delay responses accordingly. Clients do not error; they slow down. At cluster level, high load can cause throttling and delayed connection acceptance. Capping maximum eCKU on elastic clusters controls cost but makes throttling more likely at peaks.
Confluent view: Confluent Cloud exposes throttling via client metrics and cluster load metrics, and enforces published request and connection limits more strictly on elastic cluster types since 2026. Setting a maximum eCKU is a deliberate cost-versus-headroom choice.
Failure modes: Teams interpret throttling as an outage and escalate. A maximum eCKU set too low causes throttling during predictable peaks. One noisy application consumes shared quota.
Resource constraints: Capacity units and quotas.
Observability: produce-throttle-time-avg and produce-throttle-time-max, fetch-throttle-time-avg, cluster load metric, eCKU usage chart.
Competitive angle: All managed services throttle at limits; what differs is how high the limits are, how quickly capacity scales, and how transparent the signals are.
6.3 Client behaviour as a cost driver
The one-sentence story: on modern Kafka, especially managed Kafka, clients often consume more capacity through requests, connections, and metadata than through data volume, so fixing client behaviour is frequently the cheapest capacity and cost win available.
Foundations
Why clients, not data, often bind capacity
Layer 6.1 introduced three coordination dimensions: partitions, connections, and requests. Unlike bytes, these are not set by the business; they are set by how applications are written and configured. Two teams moving the same data can differ tenfold in requests and connections.
  same workload: 50 MB/s from 200 producer instances

Team A (tuned)                      Team B (defaults, poor patterns)
  long-lived producers                new producer per HTTP request
  linger.ms=20, ~100 KB batches       linger.ms=0, ~1 KB batches
  ~500 produce requests/s             ~50,000 produce requests/s
  ~stable connections                 thousands of new connections/s

capacity bound by: ingress          capacity bound by: requests + connections
In a self-managed cluster, Team B burns broker CPU and network threads and causes latency for everyone. In a managed service, Team B pays for capacity units it does not need for its data. Either way, client behaviour is a first-class sizing input.
How a client connects
A Kafka client does not talk to one endpoint. It connects in stages:
Bootstrap: connect to one of the bootstrap servers and request cluster metadata.
Metadata: learn which brokers lead which partitions for the topics it uses.
Broker connections: open a connection to every broker that leads a partition it produces to or consumes from, plus the group coordinator for consumers.
Refresh: periodically, and whenever leadership changes, request metadata again.
  client
    |---- bootstrap: "tell me about the cluster" ----> any broker
    |<--- metadata: P0 on B1, P1 on B2, P2 on B3 ----
    |---- connect --> B1
    |---- connect --> B2
    |---- connect --> B3
    |---- connect --> group coordinator (consumers)
So one client instance typically holds several connections, roughly one per broker it needs. A producer and a consumer in the same process each hold their own. Across thousands of instances, connection counts climb into the tens of thousands even at modest throughput.
The cost of a new connection
Opening a connection is expensive compared with using one. A secure connection requires a TCP handshake, a TLS handshake (asymmetric cryptography, CPU-heavy on the server side), and authentication (SASL, OAuth token validation, or mTLS certificate checks), followed by metadata requests.
  new connection cost (server side)

TCP handshake         small
  TLS handshake         significant CPU (key exchange, certificate validation)
  authentication        token or credential validation
  metadata requests     cluster and topic metadata lookups
  -----------------------------------------------------
  then: cheap, reused for thousands of requests
Long-lived clients pay this cost once. Short-lived clients pay it constantly. That is why connection attempts per second is its own capacity dimension in Confluent Cloud, separate from total connections, and why connection churn is one of the most damaging client anti-patterns.
Where requests come from
Every interaction is a request, and many are invisible to application developers:
Produce requests: one per batch group sent to a broker. Rate depends on batching.
Fetch requests: consumers and connectors continuously fetch, even when no data is arriving. A fetch with no data waits up to fetch.max.wait.ms (500 ms by default) and returns empty, after which the client fetches again.
Heartbeats: consumers in groups send heartbeats every few seconds.
Offset commits: one per commit, which can be per poll or more often.
Metadata requests: at startup, periodically, and after errors or leadership changes.
Admin requests: topic creation, config lookups, monitoring tools polling the cluster.
Idle clients are not free. A quick calculation:
  2,000 consumer instances on quiet topics

fetch:      each fetches every ~0.5 s (empty fetch wait) -> ~4,000 requests/s
  heartbeat:  each every ~3 s                              -> ~670 requests/s
  commits:    auto-commit every 5 s                        -> ~400 requests/s
  ---------------------------------------------------------------------------
  ~5,000 requests/s with almost no data moving
With a published planning figure of about 7,500 requests per second per Enterprise eCKU, that idle estate alone uses most of one unit's request budget. Real fetch rates vary with configuration and how many brokers each consumer talks to, so measure rather than assume, but the order of magnitude makes the point.
Batching decides request rate
For producers, request rate is roughly bytes per second divided by bytes per request:
  50 MB/s ingress

1 KB per request:    50,000 requests/s
  16 KB per request:   ~3,200 requests/s
  100 KB per request:  ~500 requests/s
This is the same batching lever from 6.2, now viewed as cost. It explains why linger.ms=0 with low per-partition traffic is so expensive, and why the Kafka 4.0 change of the default linger.ms to 5 ms matters. It also shows why spreading a modest stream across many partitions hurts: batches are per partition, so more partitions mean smaller batches and more requests.
Metadata and sprawl
Every topic and partition is metadata that clients request and brokers serve. Large numbers of topics and partitions inflate metadata responses, slow client startup, and increase controller work. Common sources of sprawl:
A topic per customer, per tenant, or per device.
Auto-created topics from misconfigured clients.
Very high partition counts set "for future scale".
Internal topics from many Kafka Streams applications and self-managed Connect clusters (repartition topics, changelogs, config, offset, and status topics).
In Confluent Cloud, partitions are not billed directly, but they count toward per-unit limits (with a lower sub-limit for compacted partitions on some cluster types), so sprawl can raise capacity requirements on its own.
Client libraries are not identical
Kafka clients exist for most languages, broadly in two families: the official Java client, and clients built on librdkafka (a C library used by Python, Go, .NET, Node.js, Rust and other clients). They speak the same protocol but differ in defaults and behaviour:
Batching defaults: librdkafka has long defaulted to a 5 ms linger; the Java client defaulted to 0 ms before Kafka 4.0.
Partitioner defaults: the Java client uses murmur2 hashing for keys; librdkafka defaults to a different hash unless configured, which breaks key-to-partition consistency across languages (1.2 in the internals deep dive).
Configuration names and semantics: similar settings can have different names or slightly different meanings.
Feature timing: new protocol features (new consumer group protocol, share groups) reach the Java client first, with other clients following.
Old client versions are a hidden cost too. They miss efficiency improvements and newer protocols, and Kafka 4.0 removed support for very old protocol versions entirely.
HTTP and serverless patterns
Some applications cannot hold long-lived Kafka connections: serverless functions, edge devices, browser-based tools, legacy systems. They often produce through an HTTP layer (a REST proxy or a managed REST produce API) or open a fresh Kafka client per invocation.
HTTP produce adds per-request overhead and usually prevents efficient batching. A new client per invocation is worse: it adds connection setup, authentication, and metadata requests to every single event. Both patterns are legitimate when used deliberately, but they need to be priced and capacity-planned explicitly. In Confluent Cloud, REST produce carries an additional ingress charge and its own limits.
The mental model to carry forward
Capacity is consumed by connections, connection churn, requests, and metadata, not just bytes. Long-lived, well-batched clients are cheap; short-lived, unbatched, chatty clients are expensive at any data volume. When a cluster or bill looks too big for the data it moves, look at client behaviour first.
Patterns
6.3.1 Request-rate inflation from poor batching
Overview: Producers sending tiny batches generate far more requests than the data requires, consuming request capacity and broker CPU.
Decision variables: linger.ms, batch.size, partition count per topic, keyed vs sticky partitioning, latency requirements.
Mechanism: Request rate ≈ bytes per second ÷ bytes per request. Low per-partition traffic with no linger produces near-single-record batches. Raising linger to 5 to 50 ms and reducing unnecessary partitions increases records per request dramatically.
Confluent view: Requests are a capacity dimension per eCKU or CKU, enforced more strictly on elastic types since 2026. Client tuning to reduce requests is a legitimate cost optimization and often reduces eCKU usage directly.
Failure modes: Requests becoming the binding dimension at low throughput. Broker CPU saturation in self-managed clusters. Higher latency for everyone due to request queueing.
Resource constraints: Request handler and network threads (self-managed), request limits per capacity unit (managed).
Observability: Producer records-per-request-avg, batch-size-avg, request-rate; broker or cluster request rate by type; Confluent Cloud eCKU usage chart showing requests driving scale.
Competitive angle: Competitors that price purely on throughput may look cheaper for request-heavy workloads; competitors with request-based limits will hit the same problem. Fixing batching benefits the customer on any platform, which builds trust.
6.3.2 Connection storms and churn
Overview: Large numbers of new connections per second, from short-lived clients, crash loops, or synchronized restarts, overwhelm connection handling.
Decision variables: Client lifecycle (singleton producer per process vs per request), autoscaling behaviour, deployment strategy, retry and backoff settings, authentication method.
Mechanism: Each new connection costs TLS and authentication work. A deployment that restarts thousands of pods at once, or an app that creates a producer per request, produces a spike of connection attempts. Retries without backoff amplify the storm during incidents.
Confluent view: Connection attempts per second are a separate capacity dimension with a fixed limit per unit, and Confluent began enforcing connection limits more strictly on elastic cluster types in 2026. Guidance: one long-lived producer per process, connection reuse, exponential backoff (reconnect.backoff.ms and reconnect.backoff.max.ms), staggered rollouts.
Failure modes: Delayed or refused connections, cascading failures as clients retry aggressively, authentication service overload (for OAuth), throttling across the cluster.
Resource constraints: Broker CPU for TLS, connection attempt limits, identity provider capacity.
Observability: Connection creation rate, total connections, client connection-creation-rate and connection-close-rate, authentication failure rates.
Competitive angle: Every Kafka service has connection limits; few advertise them prominently. Ask competitors for connection-attempt limits when a customer has large autoscaling or serverless estates.
6.3.3 Idle consumer overhead
Overview: Consumers on quiet topics still generate fetch, heartbeat, and commit requests, which add up across large estates.
Decision variables: Number of consumer instances, fetch.max.wait.ms, heartbeat interval, commit frequency, whether every instance needs to consume every topic.
Mechanism: Empty fetches return after fetch.max.wait.ms and are immediately reissued. Heartbeats and periodic commits add steady background traffic. Thousands of over-provisioned consumer instances multiply this.
Confluent view: Right-size consumer instance counts, consolidate tiny consumers, and use reasonable fetch waits. Fully managed connectors and Flink run their own consumers efficiently, which can reduce the sprawl of small custom consumers.
Failure modes: Request capacity consumed by idle traffic; unnecessary connections; rebalance overhead in very large groups.
Resource constraints: Request limits, connection limits, coordinator load.
Observability: Fetch request rate vs bytes out (many fetches with few bytes signals idle overhead), consumer group sizes, heartbeat and commit rates.
Competitive angle: Push-based or serverless competitors may have lower idle costs for very sparse workloads. Acknowledge this for genuinely sparse topics, and show how consolidation fixes it on Kafka.
6.3.4 Metadata floods and topic or partition sprawl
Overview: Too many topics and partitions, or clients that request metadata too often, increase coordination cost and can bind partition limits.
Decision variables: Topic-per-tenant vs shared topics with keys, partition counts, auto topic creation, Kafka Streams topology design, internal topic counts.
Mechanism: Every metadata response grows with the topics a client uses. Clients refresh metadata periodically (metadata.max.age.ms) and after errors; misbehaving clients that error repeatedly request metadata in tight loops. Each partition counts toward capacity limits regardless of traffic.
Confluent view: Partitions are not billed directly in Confluent Cloud but count toward per-unit limits, including a lower compacted-partition sub-limit on Enterprise and Freight. Self-managed Connect internal topics count; fully managed connector state does not. Topic design reviews often recover significant capacity.
Failure modes: Hitting partition limits during migration; slow client startup; metadata request spikes during incidents; controller strain in self-managed clusters.
Resource constraints: Partition limits per unit, controller memory and CPU (self-managed), client memory for metadata.
Observability: Partition and topic counts vs limits, metadata request rate, auto-created topic audits.
Competitive angle: Some competitors advertise very high partition limits; validate with realistic metadata and failover behaviour. For topic-per-tenant designs, discuss shared topics with keys plus access control as an alternative.
6.3.5 Client library and version differences
Overview: Different client libraries and versions behave differently, affecting efficiency, correctness, and feature availability.
Decision variables: Languages in use, client versions, partitioner consistency across languages, upgrade cadence, support requirements.
Mechanism: Java and librdkafka-based clients differ in default linger, batch sizing, partitioners, and feature timing. Old clients miss performance improvements, idempotence defaults, and new protocols, and very old protocol versions were removed in Kafka 4.0.
Confluent view: Confluent maintains and supports clients across major languages (librdkafka-based clients for Python, Go, .NET, and others) and publishes recommended configurations. A client inventory is part of every migration assessment.
Failure modes: Cross-language key routing mismatches; one team's old client creating disproportionate load; features (new consumer protocol, share groups) unavailable in some languages.
Resource constraints: Engineering time for upgrades across many teams.
Observability: Client version and software name reported in connection metadata (where available), per-client request rates, error rates by client.
Competitive angle: Kafka-compatible engines rely on the same client ecosystem, so client issues travel with the customer. Confluent's role as a major contributor to both client families is a fair support differentiator.
6.3.6 HTTP and serverless producers
Overview: Producing from environments that cannot hold persistent Kafka connections, via REST APIs or per-invocation clients.
Decision variables: Event volume from these sources, latency needs, availability of a long-lived intermediary, cost per request, security model.
Mechanism: REST produce wraps records in HTTP requests; each request carries overhead and often a single record. Per-invocation Kafka clients pay connection, authentication, and metadata costs per event. Mitigations: batch on the client side before sending, reuse clients across warm function invocations, or route through a long-lived gateway or collector that batches.
Confluent view: Confluent Cloud offers a REST produce API (with an additional per-GB ingress charge and its own throughput and request limits) and Confluent REST Proxy for self-managed setups. For high-volume serverless sources, SEs recommend a batching intermediary.
Failure modes: Unexpected cost from per-event HTTP overhead; connection-attempt limits hit by bursty function scaling; latency spikes from cold starts.
Resource constraints: REST API limits, connection attempts, request rate.
Observability: REST produce throughput and request metrics, connection creation rate, function invocation counts versus events produced.
Competitive angle: Cloud-native event services (such as Kinesis, Event Hubs, or Pub/Sub) offer HTTP-native ingestion that may suit sparse serverless sources well. Position Kafka for the shared, replayable backbone, with appropriate ingestion paths for serverless edges.
6.4 Confluent Cloud pricing model
The one-sentence story: Confluent Cloud bills separately for capacity and for usage, across several dimensions, so an SE must be able to build a bill line by line from a workload profile and explain every line to an engineer and a finance analyst alike.
Foundations
The shape of a consumption bill
Traditional infrastructure is bought as capacity: servers, disks, licences. Confluent Cloud is billed on consumption, split into two kinds of charges:
Capacity charges: paying for the ability to handle load, measured in eCKUs or CKUs per hour.
Usage charges: paying for what actually flows and is stored, measured in GB written, GB read, and GB stored.
On top of the Kafka cluster sit charges for other platform components (connectors, Flink, governance, Cluster Linking, Tableflow) and, separately, the customer's own cloud provider bill for networking. A credible cost model covers all of these, or it will be wrong in exactly the places finance looks.
  TOTAL COST OF A CONFLUENT CLOUD DEPLOYMENT

Confluent bill                                   Customer's cloud bill
  --------------                                   ---------------------
  Kafka capacity     eCKU-hours or CKU-hours       private endpoint hours / data
  Kafka ingress      GB written                    data transfer to and from Confluent
  Kafka egress       GB read                       (depends on networking and zones)
  Kafka storage      GB-hours (post-replication)
  Connectors         task-hours + GB processed
  Flink              CFU-minutes
  Cluster Linking    link-hours + GB
  Governance         base + per schema + per rule
  Tableflow          topic-hours + GB processed
  Support plan
Confluent measures storage and throughput in binary gigabytes (GiB, 2^30 bytes), and applies a regional multiplier to base prices depending on cloud provider and region.
Elastic capacity: how eCKU billing works
Basic, Standard, Enterprise, and Freight clusters are elastic. Capacity scales automatically with load up to a ceiling, and billing follows usage hour by hour. The rules, from Confluent's billing documentation:
Each hour, you are billed for the maximum number of eCKUs used during that hour across all dimensions.
If the cluster uses any capacity, a minimum applies (for example, 2 eCKU on Standard and Enterprise for the 99.99% SLA).
At genuinely zero consumption (no topics or partitions, no traffic, no connections or requests), you pay nothing for capacity.
You can set a maximum eCKU to cap cost, accepting that the cluster will throttle if load exceeds it.
  eCKUs used within each hour (sampled)             billed

hour 09:00  2 2 3 3 2 2 3 2 2 2 2 2               3
  hour 10:00  3 3 4 4 4 5 4 4 3 3 3 3               5
  hour 11:00  2 2 2 2 2 2 2 2 2 2 2 2               2
  hour 02:00  1 1 1 1 1 1 1 1 1 1 1 1               2  (minimum for 99.99% SLA)
Two consequences matter for modeling. First, short spikes within an hour set that hour's bill, so smoothing bursts (for example, rate-limiting a backfill) can reduce cost. Second, a workload with a strong daily cycle pays much less on elastic capacity than on capacity provisioned for peak around the clock.
Provisioned capacity: CKUs
Dedicated clusters use CKUs: a fixed amount of capacity chosen at creation and billed every hour whether used or not. You can expand or shrink a Dedicated cluster, and billing changes from the following hour.
Dedicated suits large, steady, predictable workloads, single-tenant requirements, and networking or feature needs that elastic types do not cover. Multi-zone Dedicated clusters require at least 2 CKUs. Because capacity is reserved, the economics resemble provisioned infrastructure: high utilization is efficient, idle capacity is waste.
What a capacity unit contains
Each eCKU or CKU bundles limits across the six workload dimensions from 6.1. Published planning figures per unit (check current documentation before quoting):
Basic eCKU: about 5 MB/s ingress, 15 MB/s egress, 30 partitions, 20 connections, 100 requests per second.
Standard eCKU: about 25 MB/s ingress, 75 MB/s egress, 250 partitions, 1,000 connections, 1,500 requests per second.
Enterprise eCKU: about 60 MB/s ingress, 180 MB/s egress, 3,000 partitions (up to 1,000 compacted), 18,000 connections, 500 connection attempts per second, 7,500 requests per second.
Dedicated CKU: about 60 MB/s ingress, 180 MB/s egress, 4,500 partitions, 18,000 connections, 500 connection attempts per second, 15,000 requests per second.
Freight eCKU: about 60 MB/s ingress, 180 MB/s egress, 3,000 partitions (up to 1,000 compacted), 18,000 connections, 15,000 requests per second.
Some limits are fixed (storage and partitions pre-replication, connection attempts); others are guidelines that interact (ingress, egress, connections, requests). Using one guideline dimension heavily reduces what others can achieve. Clusters also have maximum sizes: for example, Standard up to 10 eCKU, Enterprise up to 32 eCKU (with networking-dependent caps), Freight up to 152 eCKU, and Dedicated up to 252 CKUs on AWS by request.
Partitions are not billed as a separate line item, but leader partitions count toward per-unit limits, so a partition-heavy workload can raise eCKU count on its own.
Usage charges: throughput and storage
Usage charges follow the data:
Ingress: per GB written, covering produce traffic and other incoming requests.
Egress: per GB read, covering consume traffic, including reads by tools such as the console's message browser.
Storage: per GB stored per hour, on the post-replication volume. Because Confluent Cloud keeps three replicas, billed storage is typically about three times the data written and retained.
Throughput is measured on what crosses the wire, which for Kafka is compressed data. Managed connectors are an exception worth remembering: connector throughput is billed on decompressed data, so connector GB can look larger than cluster GB for the same flow.
Networking: two bills, not one
Networking is the most commonly mis-modeled part of a Confluent Cloud cost. There are two sources of networking cost:
On the Confluent bill: private networking options carry their own charges (verify current private networking pricing for PrivateLink, Private Network Interface, peering, and transit gateway options).
On the customer's cloud bill: data transfer between the customer's applications and Confluent, private endpoint hourly and data processing charges, and cross-zone or cross-region traffic on their side.
  customer VPC (cloud bill)                     Confluent Cloud (Confluent bill)
  +---------------------------+                 +---------------------------+
  | producers / consumers     |                 | Kafka cluster             |
  |      |                    |   private       |                           |
  |      +--> private endpoint ===============> |  ingress / egress GB      |
  |           (hourly + per-GB                  |  private networking fees  |
  |            charges here)  |                 |                           |
  +---------------------------+                 +---------------------------+
Networking choices also affect cluster type and size limits. For example, Enterprise clusters on AWS PrivateLink are capped at 10 eCKU, while Private Network Interface allows up to 32 eCKU. Settle networking requirements early in a deal; changing them late can force a different cluster type.
Platform add-ons
Beyond the Kafka cluster, other components carry their own dimensions:
Managed connectors: cost per task per hour plus cost per GB processed (decompressed). Dedicated clusters can carry a Connect capacity charge.
Flink: cost per CFU (Confluent Flink Unit) per minute; compute pools scale with running statements, and you pay only while queries run.
Cluster Linking: cost per link per hour (higher for privately networked clusters and for cross-cloud private links), plus per-GB charges on data sent and received, including synced metadata.
Stream Governance: a base charge plus per-schema and per-rule charges, depending on package.
Tableflow: cost per topic-hour plus per GB processed, with extra processing charges for upsert tables.
Audit logs: per GB read from audit log topics.
Support plan: prorated monthly.
These are often small individually, but in a platform deal (Kafka plus connectors plus Flink plus governance) they can rival the cluster cost. Model them explicitly.
Commitments and purchasing
List prices are a starting point. Larger customers typically sign annual or multi-year commitments in exchange for discounts, then consume against the commitment. Confluent Cloud can also be purchased through cloud marketplaces, which can let customers burn down existing cloud spend commitments (verify current marketplace and commit terms with deal desk).
The structure of a commitment matters as much as its size. Under-committing loses discount; over-committing leaves unused balance. Ramp schedules that match migration timelines protect both sides.
The mental model to carry forward
A Confluent Cloud bill is capacity (eCKU or CKU hours) plus usage (GB in, GB out, GB stored) plus platform add-ons, with a parallel networking bill from the customer's cloud provider. Capacity is set by the binding dimension of the workload; usage is set by bytes after compression; networking is set by architecture. Build every estimate line by line from the workload profile, show the formulas, and confirm current prices before quoting.
Patterns
6.4.1 Cluster type selection
Overview: Matching each workload to the right cluster type based on SLA, networking, scale, latency, features, and cost model.
Decision variables: Production vs development, required uptime SLA, public vs private networking, peak capacity needed, latency tolerance, feature needs (transactions, Cluster Linking as destination, share consumers, specific connectors), cloud and region availability.
Mechanism:
Basic: development and testing; 99.5% SLA; small limits.
Standard: production on public networking; 99.9% SLA, or 99.99% with at least 2 eCKU; up to 10 eCKU.
Enterprise: production with private networking; 99.9% or 99.99% (2 eCKU minimum); fast scaling to 10 eCKU and on-demand beyond, up to 32 eCKU depending on networking.
Dedicated: provisioned, single-tenant, highest limits and broadest networking and feature options; 99.95% single-zone, 99.99% multi-zone.
Freight: cost-optimized, high-throughput, relaxed latency (on the order of seconds) because data is written directly to object storage; 99.99% SLA; select AWS regions; some features (for example share consumers) not supported, and clients must be tuned for it.
Confluent view: The typical enterprise estate uses several types: Enterprise or Dedicated for latency-sensitive transactional data, Freight for logs and telemetry, Basic for development. Placing workloads correctly is one of the largest cost levers an SE controls.
Failure modes: Production on Basic. Latency-sensitive or transactional workloads placed on Freight. Choosing Enterprise with AWS PrivateLink and later needing more than 10 eCKU. Feature gaps discovered after migration.
Resource constraints: Per-type maximum eCKU or CKU, per-unit limits, region availability.
Observability: eCKU usage chart and cluster load against type limits; feature support documentation.
Competitive angle: Cloud-provider services offer provisioned and serverless tiers with different limits and SLAs. Compare the specific tier that matches each workload, not a single headline offering.
6.4.2 Capacity modeling (eCKU and CKU)
Overview: Estimating capacity units hour by hour from the workload's binding dimension, and choosing elastic vs provisioned.
Decision variables: Hourly peak by dimension, daily and weekly shape, minimum eCKU for SLA, maximum eCKU cap, growth, Dedicated vs elastic.
Mechanism: For each hour, units needed = max over dimensions of (usage ÷ per-unit limit). Elastic: sum the hourly maximums (with minimums applied). Dedicated: provision for the sustained peak plus headroom and multiply by hours. Compare totals, including headroom and growth.
Confluent view: Elastic types make the bill follow load, which typically favours spiky workloads. Dedicated favours steady, high-utilization workloads or those needing its limits and features. The eCKU usage chart in the console shows which dimension drives scaling.
Failure modes: Modeling only throughput and missing a binding partition, connection, or request dimension. Ignoring the hourly-maximum rule, so short bursts inflate cost. Setting a maximum eCKU below predictable peaks and throttling production.
Resource constraints: Maximum cluster sizes, scaling speed (Enterprise fast-scales to 10 eCKU, then roughly 20 minutes per eCKU on demand; Freight adds about 4 eCKU every 10 minutes).
Observability: eCKU usage over time with normalized dimension usage, cluster load metric for Dedicated, throttle metrics.
Competitive angle: Provisioned competitors force peak sizing; serverless competitors may scale differently or cap partitions lower. Show hourly cost curves side by side for the customer's real traffic shape.
6.4.3 Throughput and storage charges
Overview: Modeling usage charges from bytes in, bytes out, and bytes stored.
Decision variables: Compressed ingress, fan-out, retention per topic, replication (fixed at three in Confluent Cloud), regional multiplier, compression ratio.
Mechanism: Monthly ingress GB = average compressed ingress × seconds per month. Egress GB = ingress × fan-out (plus replays). Storage GB-hours = retained pre-replication bytes × 3 × hours. Multiply each by unit price and regional multiplier.
Confluent view: Usage charges reward efficient clients and sensible retention. Compression directly reduces ingress, egress, and storage charges, making it the simplest cost lever.
Failure modes: Using uncompressed sizes; forgetting post-replication storage billing; missing egress from tools, connectors, and replays; ignoring regional multipliers.
Resource constraints: None technically, but storage and egress can dominate cost for long-retention or high-fan-out workloads.
Observability: Costs API line items (KAFKA_NETWORK_WRITE, KAFKA_NETWORK_READ, KAFKA_STORAGE), Metrics API received, sent, and retained bytes.
Competitive angle: Some competitors bundle throughput into capacity pricing; others charge per GB like Confluent. Normalize to total monthly cost for the same workload rather than comparing unit prices.
6.4.4 Networking charges on both bills
Overview: Accounting for networking costs on the Confluent bill and on the customer's own cloud bill.
Decision variables: Public vs private connectivity, private networking option, client placement across zones and regions, cross-cloud traffic, Cluster Linking topology.
Mechanism: List every network path: producers to cluster, cluster to consumers, connectors to sources and sinks, cluster links between clusters. For each, identify who bills it (Confluent or the cloud provider) and the per-hour and per-GB components. Include cross-zone traffic on the customer side for clients not aligned with cluster zones.
Confluent view: Private networking choices also determine cluster type options and size limits. Confluent pays cloud egress for data it sends over Cluster Linking and charges per GB for it. Zone-aligned clients and fetch-from-follower can reduce cross-zone costs on Freight.
Failure modes: Quotes that omit the customer's private endpoint and data transfer charges. Networking selected late, forcing cluster type changes. Cross-region consumers creating large unexpected egress.
Resource constraints: Networking-dependent eCKU caps (for example, Enterprise with AWS PrivateLink limited to 10 eCKU).
Observability: Cloud provider cost reports for endpoints and data transfer, Confluent Costs API networking lines, per-client traffic by zone.
Competitive angle: Self-managed and in-account deployments shift networking costs into the customer's cloud bill, where they are easy to overlook. Cloud-provider services may not charge for in-cluster replication traffic but still charge for client cross-zone traffic (verify per provider). Make every option's networking explicit.
6.4.5 Platform add-ons
Overview: Modeling connectors, Flink, governance, Cluster Linking, Tableflow, and audit logs as part of the total platform cost.
Decision variables: Number of connectors and tasks, connector throughput (decompressed), Flink statements and CFU usage, number of schemas and rules, number of cluster links and mirrored GB, Tableflow topics and GB.
Mechanism: Connectors: tasks × hours × task price + decompressed GB × throughput price. Flink: CFU-minutes × price. Cluster Linking: links × hours × link price + GB sent and received. Governance: base + schemas + rules. Tableflow: topic-hours + GB processed.
Confluent view: Add-ons replace components customers would otherwise build and operate (Connect clusters, Flink clusters, schema infrastructure, replication tools). The value case compares add-on cost with the infrastructure and people cost of self-managing them.
Failure modes: Quoting Kafka only and surprising the customer with connector and Flink costs. Underestimating connector throughput because it is billed on decompressed data. Idle or paused connectors still accruing task-hour charges.
Resource constraints: Connector task limits per cluster, Flink compute pool limits.
Observability: Costs API line items per product, connector throughput metrics, Flink CFU usage.
Competitive angle: Competitors often price Kafka alone, leaving the customer to run Connect, Flink, and schema registries separately. Compare full platform cost, including people.
6.4.6 Commitments and purchasing structure
Overview: Structuring how the customer buys: pay-as-you-go, annual commitments, marketplace purchasing, and ramps.
Decision variables: Confidence in usage forecasts, migration timeline, existing cloud spend commitments, procurement preferences, discount targets.
Mechanism: Build a month-by-month consumption forecast from the migration plan (workloads move over time). Size the commitment to forecast with modest buffer. Consider ramped commitments and marketplace purchasing to align with cloud commits (verify current terms).
Confluent view: Accurate forecasts tied to a credible migration plan make commitments safer for both sides and avoid renewals dominated by unused balance or overage.
Failure modes: Over-commitment from optimistic migration timelines. Under-commitment forgoing discount. Commitments that ignore add-on products the customer will adopt.
Resource constraints: Customer budget cycles and procurement timelines.
Observability: Consumption burn-down against commitment, monthly cost trends.
Competitive angle: Cloud-provider services may let customers spend existing commitments directly. Marketplace purchasing of Confluent can neutralize this objection.
6.5 TCO vs self-managed and alternatives
The one-sentence story: the real competitor in most sophisticated accounts is "keep running it ourselves", which looks cheap because its largest costs never appear on an invoice, so the SE must make those costs visible in a model the customer's own finance team will accept.
Foundations
Why self-managed Kafka looks cheaper than it is
When a platform team compares Confluent Cloud with their existing Kafka, they usually compare a Confluent quote with their cloud compute and storage bill for the brokers. That comparison almost always favours self-managed Kafka, because it leaves out most of what self-managed Kafka actually costs.
  The TCO iceberg

visible on an invoice
        ------------------------
          broker compute
          broker storage
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ waterline
          cross-zone replication and client traffic
          peak over-provisioning and idle headroom
          controllers, Connect, Schema Registry, monitoring
          DR clusters and non-production environments
          engineers to operate, upgrade, secure, and be on call
          outages, data loss, and security incidents
          delayed projects while the team runs infrastructure
Total cost of ownership (TCO) is the full cost of delivering the capability, visible or not. Building a TCO model is not about inflating the self-managed number; it is about making the comparison complete. A good model uses the customer's own data and assumptions, so they own the conclusion.
The four cost categories
Every Kafka TCO model has four categories:
Infrastructure: compute, storage, networking, and supporting services.
People: the engineering time to build, operate, upgrade, secure, and support the platform.
Risk: the expected cost of outages, data loss, security incidents, and failed upgrades.
Opportunity: the value of what those engineers and that time could deliver instead.
Infrastructure is the easiest to quantify and usually the smallest share. People and risk are harder to quantify but often larger. Opportunity cost is the most powerful in executive conversations and the hardest to put in a spreadsheet.
Infrastructure: what self-managed Kafka really consumes
A self-managed cluster in a public cloud consumes far more than broker instances:
Brokers sized for peak: because scaling stock Kafka means moving data (1.7 and 1.8 in the internals deep dive), clusters are provisioned for the highest expected load plus headroom, around the clock.
Storage with headroom: retained data × replication factor, plus free space to survive traffic spikes and broker failures, often on premium block storage.
Controllers: dedicated KRaft controller nodes (or ZooKeeper for older clusters).
Supporting components: Kafka Connect clusters, Schema Registry, REST proxies, monitoring and alerting stacks, log aggregation, balancing tools.
Additional environments: development, staging, performance testing, and disaster recovery clusters, often each a substantial fraction of production.
Networking: cross-zone replication and client traffic, which deserves its own section.
Cross-zone networking: the hidden line item
Production Kafka spreads replicas across three availability zones for resilience. In major public clouds, data transferred between zones is billed (on AWS, roughly $0.01 per GB in each direction, so about $0.02 per GB crossing; verify current pricing). Self-managed Kafka generates a lot of cross-zone traffic:
  Self-managed, 3 zones, RF=3, 100 MB/s ingress, fan-out 3, no zone-aware clients

replication:   each byte copied to 2 other zones         -> 200 MB/s cross-zone
  producers:     2/3 of producers write to a leader
                 in another zone                            -> ~67 MB/s cross-zone
  consumers:     300 MB/s reads, 2/3 from another zone      -> ~200 MB/s cross-zone
  ---------------------------------------------------------------------------
  total          ~467 MB/s cross-zone

per month:     467 MB/s x 2,592,000 s ~ 1,210 TB ~ 1.2 PB
  at ~$0.02/GB:  roughly $24,000 per month in cross-zone transfer alone
These figures are approximate and depend on provider pricing, compression, and client placement, but the order of magnitude is typical. Many teams have never isolated this cost because it is buried in a general data transfer line on their cloud bill. Follower fetching (1.3.4) can cut the consumer portion; nothing in classic replicated Kafka removes the replication portion. This is exactly the cost that diskless designs such as Freight and WarpStream target.
Utilization: paying for peak all the time
Self-managed clusters run at low average utilization because they must handle peaks and survive failures without scaling.
  utilization of a self-managed cluster over a week

100% |- - - - - - - - - - - - - - - - - - - - - - - - capacity
       |
   60% |      /\        /\        /\        /\        /\     peaks
   30% |  ___/  \______/  \______/  \______/  \______/  \___ average
       |
    0% +-------------------------------------------------------
          Mon      Tue      Wed      Thu      Fri      (weekend)

headroom for failures + peak provisioning -> average utilization often 20-40%
At 30% average utilization, the customer pays for roughly three times the capacity they use on average. An elastic service bills closer to actual usage. This is often the single largest infrastructure difference in a TCO model, and the customer's own monitoring data can prove it.
People: the largest cost most teams undercount
Running Kafka well requires specialized skills and continuous effort:
Capacity planning, scaling, and partition rebalancing.
Version upgrades, including major migrations like ZooKeeper to KRaft.
Security: patching, certificate rotation, authentication and ACL management, audits.
Monitoring, alerting, and 24/7 on-call rotations.
Incident response and post-incident reviews.
Supporting application teams: onboarding, topic design reviews, debugging client issues.
Running the surrounding components: Connect, Schema Registry, mirroring, stream processing clusters.
  people cost = (FTEs on the platform) x (fully loaded cost per FTE)
              + (on-call burden and attrition risk)
              + (time other teams spend waiting or self-supporting)
The key inputs come from the customer: how many engineers spend how much of their time on Kafka, and what a fully loaded engineer costs them. A managed service does not eliminate platform work (topic design, client guidance, and governance remain), so a credible model shows a reduction, not zero.
Risk: pricing what could go wrong
Risk is quantified as expected cost: probability × impact. For a Kafka platform, relevant events include:
Outages: hours of downtime per year × revenue or productivity impact per hour.
Data loss: probability of loss events × cost of reconstruction, regulatory exposure, and reputation.
Security incidents: exposure from unpatched versions or misconfigured access.
Failed or delayed upgrades: cost of running unsupported versions, emergency work, and blocked features.
The most credible inputs are the customer's own incident history. Ask for their last year of Kafka-related incidents and their durations. Compare their achieved availability with a published SLA, rather than asserting that self-managed is unreliable.
Opportunity: what the time could buy
Engineers running infrastructure are not building products. In executive conversations, the strongest TCO argument is often not "you will save money" but "your best engineers will spend their time on the work that differentiates your business". For digital natives with expensive, scarce engineering talent, this framing usually lands harder than infrastructure savings.
Comparing against other managed options
TCO against self-managed Kafka is one comparison. Against other managed Kafka services, the frame shifts:
Cloud-provider managed Kafka typically runs Apache Kafka on provisioned instances (with serverless variants). Customers still size brokers, manage partitions and often balancing, and may plan scaling events. Some providers do not charge for in-cluster replication traffic but do charge client cross-zone traffic (verify per provider). People cost is lower than self-managed but not as low as fully elastic services.
Kafka-compatible engines often claim lower infrastructure cost through more efficient implementations. Compare on the customer's workload with identical durability settings, and include ecosystem, support, and migration costs.
Diskless and object-storage-native designs dramatically cut networking and storage cost for latency-tolerant workloads. Confluent's own answers are Freight and WarpStream, so the SE can meet this argument with a portfolio rather than a rebuttal.
The mental model to carry forward
TCO = infrastructure + people + risk + opportunity. Self-managed Kafka's infrastructure is inflated by peak provisioning, headroom, extra environments, and cross-zone traffic; its people and risk costs rarely appear on any invoice. Build the model from the customer's own data, show every assumption, present ranges rather than single numbers, and let their finance team check the math.
Patterns
6.5.1 Self-managed infrastructure cost model
Overview: Reconstructing the true infrastructure cost of an existing or proposed self-managed Kafka estate.
Decision variables: Broker instance types and counts, storage type and size, controller nodes, supporting components, number of environments, DR strategy, cloud pricing and discounts.
Mechanism: Inventory all clusters and environments. For each: compute (brokers, controllers, Connect, Schema Registry, monitoring), storage (provisioned, not used), networking (cross-zone, cross-region, internet egress), and supporting services. Use the customer's actual cloud bill and discounts where possible.
Confluent view: The model often reveals that non-production environments, DR clusters, and supporting components together rival production broker costs. Confluent Cloud's elastic types can make non-production environments especially cheap.
Failure modes: Counting only production brokers. Using list prices when the customer has steep cloud discounts (or vice versa). Forgetting monitoring and logging costs.
Resource constraints: Access to the customer's cloud billing data.
Observability: Cloud cost reports with resource tagging, instance and volume inventories, cluster utilization metrics.
Competitive angle: Customers with large cloud discounts may have cheaper infrastructure than list prices suggest; acknowledge this and shift weight to people, risk, and utilization.
6.5.2 Cross-zone networking cost
Overview: Isolating the cost of data crossing availability zones in self-managed Kafka.
Decision variables: Ingress rate, replication factor, number of zones, fan-out, client zone placement, follower fetching, cloud provider pricing.
Mechanism: Cross-zone bytes ≈ replication (ingress × (RF − 1)) + producer traffic to remote-zone leaders + consumer traffic from remote-zone leaders. Multiply by the provider's per-GB cross-zone price in both directions. Compare against data transfer lines in the cloud bill.
Confluent view: In Confluent Cloud, replication inside the service is not billed to the customer as cloud data transfer; the customer pays Confluent ingress and egress. Freight and WarpStream eliminate inter-zone replication entirely for latency-tolerant workloads.
Failure modes: Teams unaware of this cost because it is aggregated in their bill. Overstating it by ignoring compression or zone-aware consumers. Ignoring that cross-zone client traffic can also exist with a managed service if clients are not zone-aligned.
Resource constraints: None technical; purely economic.
Observability: Cloud provider data transfer cost reports filtered by resource, VPC flow logs, broker replication bytes metrics.
Competitive angle: This is the core economic argument of diskless competitors. Meet it with Confluent's own diskless options and by distinguishing workloads that genuinely need low latency.
6.5.3 People and operations cost
Overview: Quantifying engineering effort to run Kafka and its surrounding platform.
Decision variables: FTEs and fraction of time on Kafka, fully loaded cost per FTE, on-call structure, number of supported teams, upgrade cadence, components operated.
Mechanism: Interview the platform team: list recurring tasks (upgrades, scaling, patching, incidents, support) and estimate hours per month for each. Convert to FTEs and cost. Estimate the post-migration effort (topic governance, client guidance, cost management) and show the difference.
Confluent view: Managed services shift the platform team from operating infrastructure to enabling application teams. Position this as redeploying talent rather than eliminating jobs, which matters for internal champions.
Failure modes: Claiming 100% people savings. Ignoring the political sensitivity of headcount discussions. Using generic industry FTE estimates instead of the customer's own.
Resource constraints: Customer willingness to share staffing data.
Observability: Ticket volumes, incident counts, on-call pages, time-tracking or sprint allocation data.
Competitive angle: Cloud-provider managed services reduce some operations work but leave sizing, partition management, and scaling decisions with the customer. Quantify the residual work for each option.
6.5.4 Risk and incident cost
Overview: Converting outage, data loss, and security risk into expected annual cost.
Decision variables: Historical incident frequency and duration, revenue or productivity impact per hour, regulatory exposure, SLA requirements, upgrade backlog.
Mechanism: Expected annual risk cost = Σ (probability of event × impact). Use the customer's incident history for frequency and their business metrics for impact. Compare achieved availability with the target SLA of the proposed service.
Confluent view: Confluent Cloud's uptime SLAs (up to 99.99% on qualifying configurations), automated failure handling, and continuous patching reduce several risk categories. Multi-region designs with Cluster Linking address regional risk.
Failure modes: Fear-driven arguments without data. Ignoring risks that also apply to managed services (client bugs, application errors, regional outages without a multi-region design).
Resource constraints: Availability of incident data and impact estimates.
Observability: Incident records, postmortems, availability metrics, security audit findings.
Competitive angle: Compare SLA levels and exclusions across managed options, and compare against the customer's own measured availability.
6.5.5 Utilization and over-provisioning
Overview: Quantifying the waste from provisioning for peak and failure headroom on inelastic infrastructure.
Decision variables: Peak-to-average ratio, required failure headroom, scaling lead time, growth buffer.
Mechanism: Measure average and peak utilization across CPU, network, and disk from the customer's monitoring. Provisioned cost ÷ average utilized capacity shows the effective cost per used unit. Compare with an elastic model billed hourly on actual usage.
Confluent view: Elastic cluster types bill the hourly maximum actually used, so strongly cyclical workloads benefit most. For steady workloads, Dedicated at high utilization may be more efficient, and saying so builds credibility.
Failure modes: Claiming elastic savings for flat workloads. Ignoring minimum capacity requirements for SLA. Ignoring scaling speed limits for sudden spikes.
Resource constraints: Scaling speed and maximum capacity per cluster type.
Observability: Utilization time series from existing clusters, eCKU usage charts after migration.
Competitive angle: Serverless competitor offerings also bill on usage; compare their limits, scaling behaviour, and unit costs at the customer's traffic shape.
6.5.6 Building a finance-grade TCO model
Overview: Assembling infrastructure, people, risk, and opportunity costs into a model that survives scrutiny from the customer's finance team.
Decision variables: Time horizon (typically three years), migration ramp, discount rates, growth assumptions, which categories finance accepts.
Mechanism:
Use the customer's own numbers wherever possible and label every assumption.
Present ranges (conservative, expected, optimistic) rather than single values.
Include migration costs and dual-running periods honestly.
Separate hard savings (infrastructure) from soft savings (people redeployment, risk reduction).
Run sensitivity analysis on the biggest assumptions.
Review the model with the platform team and finance before presenting it to executives.
Confluent view: Confluent provides TCO calculators and value engineering support (verify current internal resources). The most effective models are co-built with the customer so they defend the conclusion internally.
Failure modes: Vendor-built models with opaque assumptions. Ignoring migration cost. Overstating soft savings. A single headline number that collapses under one challenged assumption.
Resource constraints: Customer time and data access; finance team engagement.
Observability: Post-migration actual costs versus the model, used to build credibility for future expansions.
Competitive angle: Competitors' TCO claims often rest on list-price infrastructure comparisons. A transparent, customer-validated model wins against a headline number.
6.6 Cost optimisation
The one-sentence story: helping a customer spend less on the platform builds the trust that drives renewals and expansion, and it is done systematically by tracing every bill line to its workload driver and pulling the right lever.
Foundations
Why an SE should optimize the customer's bill
It can feel counterintuitive for a vendor to help a customer spend less. In consumption businesses it is the opposite. Customers who feel their spend is efficient expand onto the platform; customers who feel overcharged look for alternatives at renewal. A bill that grows only because of waste is fragile revenue.
Cost optimization also creates credibility for the next conversation. An SE who found 30% savings in the current estate is trusted when proposing new use cases, Flink adoption, or migrations of more workloads.
From bill line to driver to lever
Optimization starts from the bill and works backwards along the chain introduced at the top of this document: cost comes from resources, which come from workload and efficiency.
  BILL LINE              DRIVER                          LEVERS
  ---------              ------                          ------
  capacity (eCKU/CKU) -> binding dimension at peak  ->  fix the binding dimension,
                                                        smooth peaks, right cluster type
  ingress GB          -> compressed bytes written   ->  compression, payload size,
                                                        filter at source
  egress GB           -> bytes x fan-out            ->  derived topics, fewer
                                                        redundant consumers
  storage GB-hours    -> bytes x retention x 3      ->  retention right-sizing,
                                                        compaction where suitable
  connectors          -> tasks x hours, GB          ->  right-size tasks, remove
                         (decompressed)                 idle connectors
  Flink               -> CFU-minutes                ->  efficient queries, state TTL
  networking          -> paths and zones            ->  zone alignment, locality,
                                                        private networking choices
The discipline is to find the largest lines first, identify the driver of each, and choose the lever with the best savings per unit of effort and risk.
The six families of levers
Almost every Kafka cost optimization falls into one of six families:
Fewer bytes: compression, compact serialization formats, trimming unused fields, filtering irrelevant events at the source.
Fewer copies: reducing redundant fan-out by publishing filtered or aggregated derived topics once, instead of many consumers each reading the raw stream.
Less time: retention that matches real replay and compliance needs.
Less coordination: better batching, long-lived connections, fewer unnecessary partitions (layer 6.3).
Better placement: the right cluster type for each workload's latency and feature needs (layer 6.4).
Shorter paths: keeping traffic within zones and regions where possible, and choosing networking deliberately.
A seventh, smoothing peaks, applies specifically to elastic billing: because each hour bills its maximum eCKU, spreading a backfill or batch job over more time can lower capacity charges without changing total bytes.
Serialization format: an underrated lever
Payload format changes bytes before compression even starts. JSON repeats every field name in every record and stores numbers as text. Binary formats with schemas (Avro, Protobuf) store field names once in the schema and encode values compactly.
  one order event

JSON:      {"order_id":"8812734","customer_id":"55102","amount":129.95,
              "currency":"USD","status":"PAID","ts":1760947200000}   ~120 bytes
  Avro:      field names live in the schema; values only              ~35 bytes

after batch compression the gap narrows, but binary formats usually
  remain meaningfully smaller and cheaper to parse
The exact ratio depends on the data, so measure on real payloads. Moving to schema-based binary formats also brings data contracts and Schema Registry governance (domain 2), so the cost argument and the quality argument reinforce each other.
Fan-out consolidation
A common pattern in growing estates: a large raw topic, and many teams each consuming all of it to extract a small slice. Every one of those consumers pays full egress.
  Before                                    After

raw clickstream (200 MB/s)                raw clickstream (200 MB/s)
    -> team A consumes all, keeps 5%          -> one Flink job filters and routes
    -> team B consumes all, keeps 10%             -> topic A (10 MB/s)  -> team A
    -> team C consumes all, keeps 3%              -> topic B (20 MB/s)  -> team B
    -> team D consumes all, keeps 20%             -> topic C (6 MB/s)   -> team C
                                                  -> topic D (40 MB/s)  -> team D
  egress ~800 MB/s                          egress ~200 MB/s (Flink) + ~76 MB/s
This trades some processing cost (the filtering job and the extra derived topics) for a large egress reduction, and it often improves data quality because the filtering logic is defined once. This is the same "shift-left" idea developed in domain 11.
The optimization process
A repeatable process keeps optimization credible and safe:
Measure: pull cost by line item and usage metrics by cluster, topic, and client.
Attribute: map costs to teams, applications, and topics (service accounts and naming conventions help).
Prioritize: estimate savings and effort for each candidate lever; start with high-savings, low-risk changes.
Implement: change one thing at a time where possible, with rollback plans.
Verify: confirm savings in the next billing period and that latency, durability, and reliability did not regress.
The last step matters. Optimization that quietly lowers durability (for example switching producers to acks=1) or causes throttling at peaks destroys more value than it saves.
The mental model to carry forward
Every cost line has a driver, and every driver has levers: fewer bytes, fewer copies, less time, less coordination, better placement, shorter paths, and smoother peaks. Start from the largest lines, verify savings, and never trade away reliability silently. Proactive optimization is a renewal and expansion strategy, not a discount.
Patterns
6.6.1 Compression and serialization efficiency
Overview: Reducing bytes on the wire and at rest through producer compression and compact formats.
Decision variables: Compression codec, batch sizes (which affect compression ratio), serialization format (JSON vs Avro or Protobuf), unused fields, large embedded blobs.
Mechanism: Enable producer-side compression (lz4 or zstd are common choices) and ensure batches are large enough to compress well. Move high-volume topics to schema-based binary formats. Remove unused fields and move large blobs to object storage with references in the event (the claim-check pattern).
Confluent view: Compression reduces ingress, egress, and storage charges simultaneously, and often lowers eCKU needs when throughput is binding. Schema Registry supports the move to binary formats with compatibility controls. Note that connector throughput is billed on decompressed bytes, so compression helps less on connector lines.
Failure modes: Topic-level compression mismatched with producer codec, causing broker recompression in self-managed clusters. Tiny batches yielding poor compression. Format migrations breaking consumers without schema compatibility planning.
Resource constraints: Producer and consumer CPU for compression and decompression.
Observability: Producer compression-rate-avg, bytes in versus records in, Costs API ingress and egress trends.
Competitive angle: Efficiency gains apply on any platform, so recommending them builds trust. In competitive benchmarks, ensure compression is enabled equally on every candidate.
6.6.2 Cross-zone and cross-region traffic reduction
Overview: Cutting data transfer charges by keeping traffic local where possible.
Decision variables: Client zone placement, fetch-from-follower support, region locality of producers and consumers, cluster placement, cross-region replication needs.
Mechanism: Align clients with cluster zones where supported (for example Freight clients can be zone-aligned and use fetch from follower). Keep heavy consumers in the same region as the cluster. Replicate across regions selectively (only topics that need it) rather than everything.
Confluent view: Confluent Cloud handles intra-service replication; customers control client placement and cross-region design. Cluster Linking per-GB charges make selective mirroring an important design choice.
Failure modes: Mirroring all topics to DR regions by default. Consumers in remote regions reading high-volume topics. Zone-alignment misconfigurations negating savings.
Resource constraints: Application deployment constraints (some services cannot move zones or regions).
Observability: Cloud data transfer reports, Cluster Linking throughput metrics, per-client zone and region metadata.
Competitive angle: Diskless competitors emphasize cross-zone savings; show which of the customer's workloads truly benefit and route them to Freight or WarpStream.
6.6.3 Retention right-sizing and compaction
Overview: Matching retention to actual replay, recovery, and compliance needs.
Decision variables: Per-topic retention, replay requirements, regulatory minimums and maximums, whether a topic is better served by compaction, downstream archival in table formats.
Mechanism: Inventory retention by topic and volume. For the largest topics, ask what replay window is actually needed. Shorten where safe; use compaction for entity-state topics; materialize long-term history to open table formats (domain 11) instead of keeping it in hot topics if the use is analytical.
Confluent view: Storage is billed post-replication per GB-hour, so retention cuts reduce cost three times over relative to logical bytes. Confluent still supports long or infinite retention where replay value justifies it.
Failure modes: Cutting retention below what incident recovery requires. Compaction applied to event topics that need full history. Compliance violations from shortening mandated retention.
Resource constraints: Business and regulatory requirements.
Observability: Retained bytes by topic, storage line items, consumer replay patterns (how far back consumers actually read).
Competitive angle: Object-storage-native competitors make long retention cheap; Confluent's tiered storage and Tableflow provide comparable economics for history.
6.6.4 Filter and aggregate early
Overview: Reducing downstream volume by filtering, routing, and aggregating once near the source.
Decision variables: Overlap between consumers' needs, processing complexity, latency requirements, ownership of derived topics.
Mechanism: Identify raw topics with many consumers each using small subsets. Introduce a stream processing step (often Flink SQL) that filters, routes, or aggregates into derived topics. Migrate consumers to derived topics and track egress reduction.
Confluent view: Flink in Confluent Cloud (billed per CFU-minute) is the natural tool. Positioned as shift-left data products that improve quality as well as cost.
Failure modes: Processing costs exceeding egress savings for low-fan-out topics. Unclear ownership of derived topics. Consumers needing fields filtered out.
Resource constraints: Flink compute, team adoption.
Observability: Egress per topic before and after, Flink CFU usage, consumer group counts per raw topic.
Competitive angle: Warehouse vendors propose filtering after landing data; show the cost of moving unneeded data into the warehouse first.
6.6.5 Workload placement across cluster types
Overview: Putting each workload on the cluster type that matches its latency, feature, and SLA needs at the lowest cost.
Decision variables: Latency tolerance, need for transactions or other features, throughput, networking, SLA.
Mechanism: Segment workloads: latency-critical transactional (Enterprise or Dedicated), high-volume latency-tolerant (Freight, or WarpStream for in-account deployment), development and test (Basic or small Standard). Migrate segments with Cluster Linking where appropriate.
Confluent view: A deliberate multi-cluster-type strategy is often the largest single saving in a large estate. It also counters diskless competitors directly.
Failure modes: Moving workloads to Freight that need low latency or unsupported features. Fragmenting into too many clusters and adding operational and networking overhead.
Resource constraints: Region availability (Freight is in select AWS regions), feature support per type.
Observability: Per-cluster cost and utilization, latency metrics after moves.
Competitive angle: Competitors offering one architecture force a single trade-off for every workload; a portfolio lets each workload sit at its own point on the latency-cost curve.
6.6.6 Consolidation versus isolation
Overview: Deciding when to merge small clusters for efficiency and when to separate workloads for isolation.
Decision variables: Cluster utilization, minimum capacity per cluster, blast radius tolerance, noisy-neighbour risk, compliance boundaries, team autonomy.
Mechanism: Many small, underused clusters each carry minimums and fixed costs; consolidating raises utilization. Conversely, a single giant cluster mixing critical and bulk workloads risks interference and larger blast radius; isolating critical workloads protects them.
Confluent view: Elastic clusters make small clusters cheap at low usage, but SLA minimums (for example 2 eCKU for 99.99%) still apply. SEs recommend consolidation for underused production clusters and isolation for critical or regulated workloads.
Failure modes: Consolidating regulated and unregulated data together. Hitting per-cluster limits after consolidation. Proliferating clusters per team without need.
Resource constraints: Per-cluster maximums, partition and connection limits.
Observability: Utilization per cluster, minimum-capacity charges, cross-workload latency interference.
Competitive angle: Self-managed estates often have many underused clusters because each team stood up its own; consolidation onto managed clusters is a strong savings story.
6.6.7 Peak smoothing and capacity caps
Overview: Lowering elastic capacity charges by flattening hourly peaks and capping maximum capacity deliberately.
Decision variables: Flexibility of batch jobs and backfills, maximum eCKU setting, tolerance for throttling, scheduling.
Mechanism: Because each hour bills its maximum eCKU, rate-limit backfills, replays, and bulk loads, or schedule them in quieter hours. Set maximum eCKU to cap runaway cost, accepting throttling beyond it.
Confluent view: Maximum eCKU is a cost-control feature to use carefully; it can cause throttling and workload impact when reached, and Confluent may contact customers whose capped clusters repeatedly exceed limits.
Failure modes: Caps set below predictable peaks, throttling production. Smoothing that delays time-sensitive pipelines.
Resource constraints: Time windows for batch work, scaling speed limits.
Observability: Hourly eCKU usage, throttle metrics, backfill throughput.
Competitive angle: Provisioned competitors cannot save money by smoothing peaks; elastic billing makes this lever unique.
6.6.8 Cost attribution and chargeback
Overview: Making costs visible per team, application, or business unit so owners can optimize their own usage.
Decision variables: Attribution granularity, chargeback vs showback, service account and naming conventions, reporting cadence.
Mechanism: Use separate service accounts per application, consistent topic naming by domain, and usage metrics per principal and topic to allocate shared cluster costs. Report regularly to teams.
Confluent view: Confluent exposes cost and usage data through the Costs API and Metrics API (including per-principal usage metrics where available) (verify current chargeback features). Platform teams use these to run internal chargeback.
Failure modes: Shared service accounts making attribution impossible. Allocation models seen as unfair, creating internal friction. No owner for shared topics.
Resource constraints: Platform team effort to build and maintain reporting.
Observability: Usage by principal and topic, cost trends per team.
Competitive angle: Attribution helps platform teams justify the platform internally, which protects the account and supports expansion.

Unstarred

Dunk (exclude from study)

Sizing, Performance and Cost

Review Questions

Todos

Sensemaker