data engineeringprojectsstreaming

Build an Event-Driven Analytics Stack with ClickHouse, Kafka, and Materialized Views

UUnknown

2026-03-05

10 min read

Deploy a low-latency analytics pipeline with Kafka producers, ClickHouse ingestion, and materialized views to power live dashboards in 2026.

Stop waiting for nightly ETL: build a low-latency analytics pipeline with ClickHouse + Kafka + materialized views

If you run dashboards that are always minutes—or hours—behind reality, you know the pain: slow ETL, complex batch windows, and dashboards that don’t reflect current user activity. In 2026, teams expect analytics that are near real-time, scalable, and cheap to operate. This guide walks you through a production-ready project to deploy an event-driven analytics stack using Kafka producers, ClickHouse ingestion, and ClickHouse materialized views (and projections) to power live dashboards with low latency.

Why this stack matters in 2026

Two important trends make this architecture compelling today:

ClickHouse adoption has accelerated—driven by its OLAP performance and investments since 2025—making it a mainstream choice for fast analytics at large scale.
Event-driven analytics (stream-first ETL) is now the default for observability and product analytics: streaming can reduce time-to-insight from hours to seconds, which matters for incident response and product experimentation.

ClickHouse’s rapid growth and investment (notable funding in 2025) and cloud-managed offerings have pushed it into many production analytics stacks in 2026.

This project focuses on practical steps: get Kafka producers to push events, configure ClickHouse to ingest from Kafka, use materialized views (and projections) to maintain live aggregates, and connect to dashboards like Grafana or Superset.

Architecture overview: components and flow

At a high level the pipeline is:

Producers send events to Kafka topics (JSON or Avro).
ClickHouse consumes events via the Kafka table engine (or via Kafka Connect sink).
A ClickHouse materialized view (or projection) transforms and writes to a MergeTree table—maintaining live aggregates and precomputations.
Dashboards query the aggregated tables for low-latency visuals.

Why prefer ClickHouse materialized views? They run inside ClickHouse and avoid external stream-processing frameworks for many use cases. For heavier stream-processing logic (joins across large, evolving state), consider Flink/ksqlDB alongside this stack.

Prerequisites and deployment options

Before you start, pick an operational footprint that matches your team:

Small teams: Managed cloud ClickHouse (ClickHouse Cloud, Altinity.Cloud) + Confluent Cloud or Aiven for Kafka.
On-prem or k8s: ClickHouse Operator (or Altinity’s operator) + Strimzi for Kafka.
Hybrid: Self-hosted ClickHouse with Kafka Connect-based sinks for simpler operational separation.

Required tools: Kafka cluster, ClickHouse cluster (v23+ recommended for projections and stability in 2026), a dashboard tool (Grafana, Superset), and a lightweight producer (Python/Go).

Step-by-step project guide

1) Define the event shape

Choose a canonical event schema. Use JSON or Avro — Avro + schema registry is recommended for long-term stability. Example event for product telemetry:

{
  "event_id": "uuid-123",
  "ts": "2026-01-18T12:34:56Z",
  "user_id": "u-1000",
  "project": "billing",
  "action": "checkout",
  "amount": 29.99
}

2) Produce events to Kafka (Python example)

Use an idempotent producer and set the message key to a meaningful value (user_id or event_id) to get partition affinity. Here's a simple Python example using confluent-kafka:

from confluent_kafka import Producer
import json

p = Producer({
  'bootstrap.servers': 'kafka:9092',
  'enable.idempotence': True
})

event = {
  'event_id': 'uuid-123',
  'ts': '2026-01-18T12:34:56Z',
  'user_id': 'u-1000',
  'project': 'billing',
  'action': 'checkout',
  'amount': 29.99
}

p.produce('events', key=event['user_id'], value=json.dumps(event))
p.flush()

3) Create a ClickHouse Kafka engine table

The Kafka engine makes ClickHouse directly consume messages. Use JSONEachRow or Avro/Schema Registry where appropriate. Example DDL (JSONEachRow):

CREATE TABLE events_kafka (
  kafka_key String,
  kafka_value String
) ENGINE = Kafka
SETTINGS
  kafka_broker_list = 'kafka:9092',
  kafka_topic_list = 'events',
  kafka_group_name = 'clickhouse-events-group',
  kafka_format = 'JSONEachRow';

Notes:

kafka_group_name controls the consumer group for offsets stored in ClickHouse.
For Avro, use kafka_format='AvroConfluent' and provide schema registry settings.

4) Create a target table (MergeTree) for raw events

Store parsed events in a MergeTree or ReplacingMergeTree for deduplication. Use a sorting key that matches your most common query patterns (time + project, for example).

CREATE TABLE events_raw (
  event_id String,
  ts DateTime('UTC'),
  user_id String,
  project String,
  action String,
  amount Float64
) ENGINE = ReplacingMergeTree(event_id)
ORDER BY (project, toYYYYMMDD(ts), ts);

Using ReplacingMergeTree with event_id allows deduplication during merges if a producer retries. For strict deduplication, include a version column.

5) Create a materialized view to populate the target table

Create a materialized view that parses kafka_value and inserts into the raw table. This view runs inside ClickHouse and is low-latency.

CREATE MATERIALIZED VIEW mv_events_to_raw TO events_raw AS
SELECT
  JSONExtractString(kafka_value, 'event_id') AS event_id,
  parseDateTimeBestEffort(JSONExtractString(kafka_value, 'ts')) AS ts,
  JSONExtractString(kafka_value, 'user_id') AS user_id,
  JSONExtractString(kafka_value, 'project') AS project,
  JSONExtractString(kafka_value, 'action') AS action,
  JSONExtractDouble(kafka_value, 'amount') AS amount
FROM events_kafka;

Important options:

Don't use POPULATE on production tables unless you need to ingest historic data at creation time (it can block resources).
Monitor the system.kafka table for offsets and errors.

6) Build live aggregates with materialized views or projections

Materialized views are great for simple transformations and inserts. For pre-aggregations, use a materialized view that writes to an aggregates table. For 2026 workloads, consider projections (ClickHouse feature) as they often perform better for complicated aggregations and reduce maintenance.

Example: maintain minute-level totals per project and action.

CREATE TABLE events_minute_agg (
  agg_minute DateTime,
  project String,
  action String,
  total_count UInt64,
  total_amount Float64
) ENGINE = SummingMergeTree()
ORDER BY (project, action, agg_minute);

CREATE MATERIALIZED VIEW mv_events_minute TO events_minute_agg AS
SELECT
  toStartOfMinute(ts) AS agg_minute,
  project,
  action,
  count() AS total_count,
  sum(amount) AS total_amount
FROM events_raw
GROUP BY agg_minute, project, action;

Tips:

For fast dashboards, materialize at the granularity your UI needs (minute, 10s, 1s).
Consider a TTL policy if you only need recent aggregates to reduce storage.

Operational tuning for low-latency

Low latency requires tuning at multiple layers. Focus on these knobs:

Kafka: producer linger.ms, acks=all, partitioning by key to even load. Monitor consumer lag.
ClickHouse ingestion: kafka_max_block_size, kafka_max_wait_ms, max_insert_block_size. Reduce buffering if you need lower tail latency.
Table engines: use ReplacingMergeTree or SummingMergeTree for dedup/agg. Use projections where appropriate for faster reads.
Network and disk: NVMe or fast network for hot data; compress older parts to cheaper storage with tiered storage if needed.

Example ClickHouse settings for lower latency (server config or session):

SET max_insert_block_size = 1048576;
SET max_partitions_per_insert_block = 100;

Handling late-arriving events and deduplication

Real streams have late events. Strategies:

Use event timestamps (ts) as the source of truth and include watermarking logic in materialized view queries or downstream consumers.
Deduplicate using ReplacingMergeTree with event_id or a version column.
For stateful joins and exactly-once semantics, integrate a stream processor (Flink, Kafka Streams) before ClickHouse.

Observability and monitoring

Instrument three layers: producers, Kafka, ClickHouse. Key metrics:

Producer acks, retries, and producer latency.
Kafka: consumer group lag (per partition), under-replicated partitions, input throughput.
ClickHouse: system.merges, system.parts, system.mutations, system.kafka (offsets/errors), inserts per second, query latency.

Sample queries to inspect ClickHouse ingestion:

-- check kafka engine offsets and errors
SELECT * FROM system.kafka WHERE table = 'events_kafka';

-- monitor active merges/parts
SELECT table, total_rows, active FROM system.parts WHERE database = 'default' AND table LIKE 'events%';

-- check recent inserts
SELECT
  table,
  sum(rows) AS rows_inserted,
  sum(bytes) AS bytes_inserted
FROM system.insert_log
WHERE event_time > now() - INTERVAL 1 HOUR
GROUP BY table;

Use Grafana dashboards to combine Kafka and ClickHouse metrics for single-pane observability.

Scaling and reliability patterns

Production patterns to adopt:

Partition events by logical sharding key (project or region) for parallelism.
Use ClickHouse replication (ReplicatedMergeTree) for high availability and multi-node reads.
Shard Kafka topics based on throughput; set partitions count to match consumer parallelism.
Use schema registry + Avro or Protobuf to evolve schemas safely.

When to use materialized views vs projections vs external stream processors

Choose based on complexity and latency needs:

Materialized views: Best for straightforward parsing and single-table aggregations written directly into ClickHouse. Low operational overhead.
Projections: Prefer for heavy-read workloads where precomputed, compact projections reduce query time. In 2026, projections are more widely used for production pre-aggregations.
External stream processors (Flink/Kafka Streams): Use these when you need complex stateful processing, windowing with late event handling, or cross-topic joins before materialization.

Real-world example: live observability dashboard

Scenario: you want an SRE dashboard showing errors per service in the last 5 minutes, updated every few seconds.

Producers send log events to Kafka with key=service.
ClickHouse Kafka table ingests messages.
Materialized view writes to a SummingMergeTree table aggregated by 10-second buckets.
Grafana queries the aggregate table (it’s very fast because of the SummingMergeTree and small time buckets).

DDL for a 10-second bucketed aggregate:

CREATE TABLE svc_errors_10s (
  bucket DateTime,
  service String,
  error_count UInt64
) ENGINE = SummingMergeTree()
ORDER BY (service, bucket);

CREATE MATERIALIZED VIEW mv_svc_errors_10s TO svc_errors_10s AS
SELECT
  toStartOfInterval(ts, INTERVAL 10 SECOND) AS bucket,
  service,
  countIf(1, level = 'error') AS error_count
FROM events_raw
GROUP BY bucket, service;

Testing, CI, and starter templates

Create repeatable tests for the pipeline:

Unit tests for producer serialization and deserialization.
Integration tests running Kafka + ClickHouse (use Docker Compose for local CI) to verify end-to-end ingestion and aggregation.
Performance tests that replay realistic event traces at target throughput and assert query latency SLAs.

Starter template (suggested repo layout):

/k8s - manifests for Kafka (Strimzi) + ClickHouse Operator manifests
/clickhouse - DDLs and sample queries
/producers - example producers (Python, Go)
/dashboards - Grafana JSON models
/tests - Docker Compose integration harness

Cost and tradeoffs

Pros:

Low latency: dashboards reflect activity in seconds.
Operational simplicity: ClickHouse materialized views reduce the need for separate stream processing for many tasks.
High throughput at low cost compared to cloud OLAP warehouses for similar latency.

Cons and tradeoffs:

Complex stateful processing still needs specialized stream engines.
Operational expertise in ClickHouse and Kafka is required to tune for extreme loads.
Storage costs if you retain high-cardinality raw events for long periods—use tiered/archival storage.

Advanced strategies and future-proofing (2026+)

Plan for scale and evolution:

Adopt schema evolution best practices with a schema registry (Avro/Protobuf) to avoid breaking consumers.
Use projections for complex queries where possible—2026 usage shows better read performance and simpler query plans.
Prepare for hybrid storage: hot recent data in ClickHouse clusters, colder historic data in object storage with ClickHouse tiered reads.
Build dashboards that can gracefully degrade (from per-second to per-minute) depending on SLA and load.

Checklist: production readiness

Idempotent, keyed producers with schema registry
Kafka monitoring and partition strategy
ClickHouse replication and backup strategy
Materialized views for transforms + projections for heavy loads
CI tests for end-to-end pipeline
Dashboard and alerting for ingestion and query SLAs

Actionable takeaways

Start small: get a Kafka topic -> ClickHouse Kafka table -> materialized view -> simple Grafana chart working end-to-end.
Iterate on retention and aggregation granularity based on dashboard needs, not just storage fears.
Use ReplacingMergeTree for simple deduplication; upgrade to external stream processors if your logic becomes stateful or requires complex windowing.
Monitor Kafka consumer lag and ClickHouse insert throughput as your core SLA metrics.

Wrap-up

In 2026, the combination of Kafka + ClickHouse provides an accessible path to real-time analytics: low-latency ingestion, efficient storage, and powerful materialized views (and projections) to keep dashboards fresh. This stack reduces batch ETL complexity while giving you control over correctness and performance. Start with the minimal pipeline in this guide, add observability, and evolve to projections or external stream processors only when you need more advanced semantics.

Ready to try it? Clone a starter repo with DDLs, Docker Compose integration tests, and Grafana dashboards to get a working end-to-end pipeline in an afternoon. Build the template, run replay tests, and measure how quickly your dashboards go from minutes to seconds.

Call to action

Get the starter template, sample DDLs, and a troubleshooting checklist from our repo: start the project, run the sample producer, and drop a note in thecoding.club community to share results and optimizations. If you want, I can generate a Docker Compose starter for you now—say which producer language (Python/Go/Node) and I'll output a ready-to-run stack.

Unknown

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.