Classification of Solutions by Flow Typology
For the complete, navigable and filterable list (sort by stars/date, licenses, top/flop), see the site automatically generated from the catalogs:
docs/(GitHub Pages). This document remains an editorial analysis.
Objective: Classify all open-source solutions according to their data processing mode: Streaming, Micro-Batching, and Batching.
Introduction
Modern data processing architectures can be classified according to three main paradigms based on the latency and frequency of data processing:
1. Streaming (Real-Time)
Continuous processing of data on the fly, event by event, with a latency on the order of milliseconds to seconds.
Characteristics:
- Ultra-low latency (< 1 second)
- Continuous event-based processing
- Ideal for real-time alerts, fraud detection, IoT
- Event-driven architecture
- Usage examples: Real-time monitoring, algorithmic trading, anomaly detection
2. Micro-Batching (Near Real-Time)
Processing in small batches of data at very short intervals, with a latency on the order of a few seconds to a few minutes.
Characteristics:
- Low latency (1 second - 5 minutes)
- Accumulation of data over short intervals
- Trade-off between throughput and latency
- Ideal for near real-time analytics
- Usage examples: Real-time dashboards, recent aggregations, business metrics
3. Batching (Batch Processing)
Processing of large volumes of data at scheduled intervals (hourly, daily, weekly), with a latency on the order of minutes to hours.
Characteristics:
- High acceptable latency (> 5 minutes)
- Processing of large volumes
- Optimized for throughput
- Ideal for historical analyses, ETL, reports
- Usage examples: Daily reports, data warehousing, ML training
Classification Matrix
Classification Criteria
| Criterion | Streaming | Micro-Batching | Batching |
|---|---|---|---|
| Latency | < 1 second | 1 sec - 5 min | > 5 minutes |
| Batch size | 1 event | Small batches (KB-MB) | Large batches (GB-TB) |
| Frequency | Continuous | High (seconds) | Low (hours/days) |
| Use Case | Real-time | Near real-time | Historical analytics |
| Complexity | High | Medium | Low |
| Throughput | Medium | Medium-High | Very high |
Complete Classification of Solutions
Legend
- ● Primary: Main use case of the tool
- ◐ Secondary: Supported but not optimal use case
- ○ Not supported: Not applicable or not recommended
1 Ingestion & Transport
Data Replication
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Airbyte | ○ | ◐ | ● | Batch ETL/ELT, incremental sync |
| Apache Camel | ● | ● | ● | Versatile framework, all modes supported |
| Apache Gobblin | ○ | ◐ | ● | Batch ingestion for Hadoop |
| Apache NiFi | ● | ● | ● | Flow-based, supports all modes |
| Apache SeaTunnel | ◐ | ● | ● | Batch + streaming integration, multi-connector |
| Bento (Benthos fork) | ● | ● | ◐ | Stream processor, optimized for real-time |
| CloudQuery | ○ | ◐ | ● | Cloud/API ELT to SQL, incremental batch |
| data load tool (dlt) | ○ | ◐ | ● | Python ETL, mainly batch |
| Embulk | ○ | ◐ | ● | Pluggable bulk data loader, batch |
| Meltano | ○ | ◐ | ● | Orchestrated ELT, batch with Singer taps |
| Singer | ○ | ◐ | ● | Data exchange standard, batch-oriented |
| Starlake (Starflow) | ○ | ◐ | ● | Declarative ELT + orchestration (YAML), batch-first |
Recommendations by use case:
- Streaming: Apache NiFi, Bento, Apache Camel
- Micro-Batching: Apache NiFi, Apache Camel
- Batching: Airbyte, Meltano, dlt, Gobblin
Event/Stream Processing
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Apache Kafka | ● | ● | ◐ | Event streaming platform, native real-time |
| Apache Pekko | ● | ● | ◐ | Actors/streams toolkit (Akka fork), real-time |
| Apache Pulsar | ● | ● | ◐ | Messaging + streaming, multi-tenant |
| Apache RocketMQ | ● | ● | ◐ | Distributed messaging, low latency |
| Arroyo | ● | ● | ○ | SQL stream processing, native real-time |
| Eclipse Mosquitto | ● | ◐ | ○ | Lightweight MQTT broker, real-time IoT |
| EMQX | ● | ◐ | ○ | Scalable distributed MQTT broker, IoT |
| NATS | ● | ◐ | ○ | Ultra-lightweight messaging, microsecond latency |
| Numaflow | ● | ● | ◐ | Kubernetes-native stream processing |
| RabbitMQ | ● | ● | ◐ | Message broker, AMQP protocol |
| Rudderstack | ● | ● | ◐ | Customer data platform, event streaming |
| Snowplow | ● | ● | ◐ | Event analytics, real-time tracking |
Recommendations by use case:
- Streaming: All (Kafka, Pulsar, NATS leaders)
- Micro-Batching: Kafka, Pulsar, RabbitMQ
- Batching: Not recommended (not optimal)
Log Collection and Processing
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Fluent Bit | ● | ● | ◐ | Ultra-lightweight, edge/containers |
| Fluentd | ● | ● | ◐ | Unified logging layer, real-time |
| Grafana Alloy | ● | ● | ◐ | OpenTelemetry/Prometheus collector, real-time |
| Grafana Loki | ● | ● | ◐ | Log aggregation, real-time ingestion |
| Graylog | ● | ● | ● | Log management platform, all modes |
| Logstash | ● | ● | ● | Elastic Stack, versatile pipeline |
| OpenTelemetry Collector | ● | ● | ◐ | Telemetry collection (logs/metrics/traces), real-time |
| rsyslog | ● | ● | ◐ | High-performance syslog processing, real-time |
| syslog-ng | ● | ● | ◐ | Log collection/routing, real-time |
| Vector | ● | ● | ◐ | High-performance observability, Rust-based |
Recommendations by use case:
- Streaming: Vector, Fluent Bit, Fluentd
- Micro-Batching: All
- Batching: Graylog, Logstash
Change Data Capture (CDC)
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Apache Flink CDC | ● | ● | ○ | CDC streaming on Flink, snapshot + incremental |
| Canal | ● | ◐ | ○ | MySQL binlog CDC (Alibaba), real-time |
| Debezium | ● | ◐ | ○ | CDC streaming, real-time binlog |
| Databus | ● | ◐ | ○ | LinkedIn CDC, low-latency streaming |
| Maxwell | ● | ◐ | ○ | MySQL CDC, JSON streaming to Kafka |
| PeerDB | ● | ● | ◐ | Real-time Postgres CDC, incremental sync |
| Sequin | ● | ● | ○ | Postgres CDC to streams, real-time |
Recommendations by use case:
- Streaming: All (CDC = streaming by nature)
- Micro-Batching: Possible but not optimal
- Batching: Not suitable
2 Storage
File Layer
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Apache Arrow | ◐ | ● | ● | In-memory columnar format, fast interchange |
| Apache Kudu | ● | ● | ● | Columnar storage, real-time updates + analytics |
| Avro | ● | ● | ● | Serialization, all use cases |
| DuckLake | ○ | ◐ | ● | Lakehouse format (DuckDB), batch analytics |
| Lance | ◐ | ● | ● | Columnar format for ML/vectors, random access |
| ORC | ○ | ◐ | ● | Columnar storage, batch-optimized |
| Parquet | ○ | ◐ | ● | Columnar storage, analytics-optimized |
| Vortex | ◐ | ● | ● | Next-generation compressed columnar format |
Recommendations by use case:
- Streaming: Avro (lightweight, schema evolution)
- Micro-Batching: Avro, ORC, Parquet
- Batching: ORC, Parquet (excellent compression)
Object Storage
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Apache Ozone | ◐ | ● | ● | Scalable object store for big data (Hadoop) |
| Ceph | ◐ | ● | ● | Distributed object/block/file storage |
| Garage | ◐ | ● | ● | Lightweight geo-distributed S3 object store |
| JuiceFS | ◐ | ● | ● | Distributed file system over object storage |
| MinIO | ◐ | ● | ● | S3-compatible, optimized for all use cases |
| SeaweedFS | ◐ | ● | ● | Distributed object/file store, small files |
Recommendations by use case:
- Streaming: Possible for storing results
- Micro-Batching: Excellent for data lake
- Batching: Ideal for massive storage
Metadata Layer
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Apache Gravitino | ○ | ◐ | ● | Unified multi-source metadata, batch |
| Apache XTable | ○ | ◐ | ● | Table format interoperability (Iceberg/Hudi/Delta) |
| Delta Lake | ◐ | ● | ● | ACID on data lake, streaming + batch |
| Hive Metastore | ○ | ◐ | ● | Metadata management, batch-oriented |
| Hudi | ● | ● | ● | Streaming + batch upserts, very versatile |
| Iceberg | ◐ | ● | ● | Table format, analytics-optimized |
| lakeFS | ○ | ◐ | ● | Git-like data lake versioning, batch workflows |
| Lakekeeper | ○ | ◐ | ● | Iceberg REST catalog, batch analytics |
| Nessie | ○ | ◐ | ● | Git-like versioning, batch workflows |
| Paimon | ● | ● | ● | Streaming data lake, very performant streaming |
| Polaris | ○ | ◐ | ● | Unified catalog, batch analytics |
| Unity Catalog | ○ | ◐ | ● | Multi-format governance catalog, batch |
Recommendations by use case:
- Streaming: Hudi, Paimon (streaming leaders)
- Micro-Batching: All except Hive Metastore and Nessie
- Batching: All
Vector Storage
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Chroma | ◐ | ● | ● | Embedded vector DB, batch/micro indexing |
| Milvus | ◐ | ● | ● | Scalable vector DB, upserts + indexing |
| pgvector | ◐ | ● | ● | Postgres vector extension, real-time upserts |
| Qdrant | ◐ | ● | ● | Rust vector DB, upserts + search |
| Weaviate | ◐ | ● | ● | Vector DB + semantic search, batch import |
Recommendations by use case:
- Streaming: Upserts possible (pgvector, Qdrant, Milvus) but not optimal
- Micro-Batching: All (vector indexing and queries)
- Batching: All (massive embedding ingestion)
Data Modeling
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Big Functions | ○ | ○ | ● | BigQuery UDFs, batch analytics |
| dbt core | ○ | ◐ | ● | SQL transformations, batch-first |
| GraphQL | ● | ● | ● | API query language, real-time |
| SQL Mesh | ○ | ◐ | ● | Data transformations, batch workflows |
Recommendations by use case:
- Streaming: GraphQL (real-time API)
- Micro-Batching: dbt core (with Airflow), GraphQL
- Batching: All
3 Query & Processing
Query Engine
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Apache Calcite | ◐ | ● | ● | Query optimization framework, versatile |
| Apache Drill | ○ | ◐ | ● | Schema-free SQL, batch-oriented |
| chDB | ○ | ● | ● | Embedded ClickHouse (in-process OLAP) |
| Datafusion | ◐ | ● | ● | Rust query engine, high performance |
| DuckDB | ○ | ● | ● | Embedded OLAP, fast analytics |
| Hydra | ○ | ◐ | ● | Postgres columnar extension |
| Velox | ◐ | ● | ● | Reusable C++ execution engine, high performance |
| PostgreSQL | ● | ● | ● | OLTP database, all modes |
| Presto | ○ | ◐ | ● | Distributed SQL, interactive queries |
| Quack on Demand | ○ | ● | ● | DuckDB fleet gateway, federated queries |
| Trino | ○ | ◐ | ● | Presto fork, MPP SQL engine |
Recommendations by use case:
- Streaming: PostgreSQL (with streaming extensions)
- Micro-Batching: DuckDB, Datafusion, Calcite
- Batching: All (Presto/Trino leaders for data lakes)
Stream Processing
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Apache Flink | ● | ● | ◐ | True streaming engine, market leader |
| Apache Kafka | ● | ● | ◐ | Streaming platform (also in Ingestion) |
| Apache Samza | ● | ● | ◐ | Stream processing, Kafka-based |
| Apache Storm | ● | ◐ | ○ | Real-time computation, mature |
| Apache StreamPipes | ● | ● | ◐ | Self-service streaming/IoT toolbox |
| Arroyo | ● | ● | ○ | SQL stream processing, native real-time |
| Bytewax | ● | ● | ◐ | Python stream processing (Rust), dataflow |
| Faust-streaming | ● | ● | ◐ | Python stream processing, Kafka-based |
| Feldera | ● | ● | ◐ | Incremental view maintenance, real-time |
| Materialize | ● | ● | ◐ | Streaming SQL database, real-time |
| Numaflow | ● | ● | ◐ | Kubernetes-native stream processing |
| Pathway | ● | ● | ◐ | Real-time Python framework, batch + stream |
| Quix Streams | ● | ● | ◐ | Python stream processing, Kafka-native |
| RisingWave | ● | ● | ◐ | Streaming SQL database, real-time |
| Redpanda | ● | ● | ◐ | Kafka-compatible, C++ performance |
| Timeplus Proton | ● | ● | ◐ | Streaming SQL + analytics, real-time |
Recommendations by use case:
- Streaming: Flink (leader), Storm, Materialize, Kafka
- Micro-Batching: All
- Batching: Not optimal (use Spark)
Batch Processing
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| AmphiETL | ○ | ◐ | ● | Low-code ETL, batch-first |
| Apache Beam | ● | ● | ● | Unified batch + stream, very versatile |
| Apache Hop | ○ | ◐ | ● | Data orchestration, batch ETL |
| Apache Sedona | ○ | ◐ | ● | Distributed geospatial (Spark/Flink), batch |
| Apache Spark | ◐ | ● | ● | Unified engine, micro-batch streaming |
| dbt core | ○ | ◐ | ● | SQL transformations, batch |
| Duckle | ○ | ◐ | ● | Lightweight batch processing on DuckDB |
| Fugue | ○ | ◐ | ● | Unified Spark/Dask/Ray abstraction, batch |
| Odyssée | ○ | ◐ | ● | Batch data processing |
| Ray | ◐ | ● | ● | Distributed Python compute, batch + micro-batch |
| Talaxie | ○ | ◐ | ● | Talend open-source, batch ETL |
Recommendations by use case:
- Streaming: Apache Beam (with Flink runner)
- Micro-Batching: Spark Structured Streaming, Beam
- Batching: All (Spark leader)
Dataframe Processing
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| cuDF (RAPIDS) | ○ | ● | ● | GPU dataframes, accelerated batch |
| Daft | ○ | ● | ● | Distributed dataframe (Rust), batch + micro-batch |
| Dask | ○ | ● | ● | Parallel computing, batch + micro-batch |
| Ibis Project | ○ | ◐ | ● | Portable dataframe API, batch |
| Modin | ○ | ● | ● | Scalable distributed Pandas, batch |
| Pandas | ○ | ◐ | ● | In-memory dataframes, small batches |
| Polars | ○ | ● | ● | Fast Rust dataframes, batch-optimized |
| Vaex | ○ | ● | ● | Out-of-core dataframes, large batch volumes |
| Xarray | ○ | ◐ | ● | Labeled N-dimensional arrays, scientific batch |
Recommendations by use case:
- Streaming: Not suitable (in-memory)
- Micro-Batching: Dask, Polars (very performant)
- Batching: All
Datawarehouse & OLAP
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Apache Hive | ○ | ◐ | ● | Data warehouse, batch SQL |
| Apache Impala | ○ | ● | ● | MPP SQL, interactive queries |
| Apache Kylin | ○ | ◐ | ● | OLAP cube engine, batch pre-aggregation |
| ClickHouse | ● | ● | ● | OLAP database, excellent streaming ingestion |
| Doris | ◐ | ● | ● | MPP database, real-time analytics |
| Druid | ● | ● | ◐ | Real-time OLAP, streaming-optimized |
| Pinot | ● | ● | ◐ | Real-time OLAP, LinkedIn, sub-second queries |
| StarRocks | ◐ | ● | ● | MPP OLAP, unified batch + real-time |
Recommendations by use case:
- Streaming: ClickHouse, Druid, Pinot (real-time OLAP leaders)
- Micro-Batching: All except Hive and Kylin
- Batching: All
Time-Series Database
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Apache IoTDB | ● | ● | ● | IoT TSDB, real-time ingestion + analytics |
| CrateDB | ● | ● | ● | Distributed SQL time-series, real-time ingestion |
| GreptimeDB | ● | ● | ● | Cloud-native TSDB, real-time metrics/logs |
| InfluxDB | ● | ● | ● | Reference TSDB, real-time ingestion |
| QuestDB | ● | ● | ● | High-performance SQL TSDB, real-time ingestion |
| TDengine | ● | ● | ● | IoT/industrial TSDB, massive real-time ingestion |
| TimescaleDB | ● | ● | ● | Postgres time-series extension, real-time + SQL |
| VictoriaMetrics | ● | ● | ● | Scalable metrics TSDB, real-time ingestion |
Recommendations by use case:
- Streaming: All (native real-time ingestion, IoT/metrics)
- Micro-Batching: All (aggregations and downsampling)
- Batching: All (historical analytical queries)
4 Analysis & Output
Framework
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| AntV G2 | ● | ● | ● | Viz grammar, real-time updates |
| ApexCharts | ● | ● | ● | Interactive JS charting, real-time updates |
| Apache ECharts | ● | ● | ● | Viz library, websocket for streaming |
| Bokeh | ● | ● | ● | Interactive viz, streaming callbacks |
| Chart JS | ● | ● | ● | JavaScript charting, real-time updates |
| Cube | ◐ | ● | ● | Headless BI, pre-aggregation + refresh |
| D3JS | ● | ● | ● | Data viz, streaming transition animations |
| deck.gl | ● | ● | ● | WebGL geospatial viz, real-time layers |
| ggplot2 | ○ | ◐ | ● | R graphics grammar, batch |
| Great Tables | ○ | ◐ | ● | Presentation tables, batch reporting |
| Holoviz | ○ | ◐ | ● | Jupyter viz, batch-oriented |
| kepler.gl | ◐ | ● | ● | Large-scale geospatial viz, batch/refresh |
| Lonboard | ◐ | ● | ● | Python geospatial viz (deck.gl), batch |
| Matplotlib | ○ | ◐ | ● | Static viz, batch |
| Nivo | ● | ● | ● | React viz components (D3), real-time |
| Perspective | ● | ● | ● | Interactive streaming analytics, real-time |
| Plotly | ● | ● | ● | Interactive viz, Dash for streaming |
| Plotnine | ○ | ◐ | ● | Python graphics grammar (ggplot), batch |
| Recharts | ● | ● | ● | React charting components, real-time |
| Seaborn | ○ | ◐ | ● | Statistical viz, batch |
| Vega/Altair | ○ | ◐ | ● | Declarative viz, batch |
| visx | ● | ● | ● | React viz primitives (D3), real-time |
Recommendations by use case:
- Streaming: D3JS, Plotly, Chart JS, Bokeh
- Micro-Batching: All
- Batching: All
High-Code / Low-Code / No-Code
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Apache Zeppelin | ○ | ● | ● | Notebooks, micro-batch queries |
| Plotly Dash | ● | ● | ● | Real-time web apps, callbacks |
| JupyterLab | ○ | ◐ | ● | Notebooks, batch-oriented |
| Panel | ● | ● | ● | Python dashboards, streaming callbacks |
| Streamlit | ● | ● | ● | Interactive apps, streaming auto-rerun |
| Taipy | ● | ● | ● | Data pipelines + GUI, all modes |
| Gradio | ● | ● | ● | ML/data app UI, streaming outputs |
| Marimo | ○ | ● | ● | Reactive Python notebook, micro-batch |
| NiceGUI | ● | ● | ● | Real-time Python UI, websocket |
| Quarto | ○ | ◐ | ● | Scientific publishing, batch reporting |
| Reflex | ● | ● | ● | Full-Python web apps, real-time state |
| Solara | ● | ● | ● | Reactive Python web apps, real-time |
| Voila | ○ | ◐ | ● | Notebooks as apps, batch-oriented |
| H2O Wave | ● | ● | ● | Real-time ML/data apps, real-time |
| Shiny (R) | ● | ● | ● | Interactive R apps, reactive real-time |
| Evidence | ○ | ● | ● | BI from SQL, scheduled refresh |
| Grafana | ● | ● | ● | Monitoring dashboards, real-time |
| Kibana | ● | ● | ● | Elastic viz, streaming logs |
| PyGWalker | ○ | ◐ | ● | Interactive EDA, batch dataframes |
| Metabase | ○ | ● | ● | BI tool, auto-refresh dashboards |
| Superset | ○ | ● | ● | BI platform, scheduled queries |
| Datasette | ○ | ◐ | ● | SQLite exploration/publishing, batch |
| Observable Framework | ○ | ● | ● | Static/reactive data dashboards, batch build |
| Vizro | ○ | ● | ● | Low-code dashboards (Plotly/Dash), refresh |
| Querybook | ○ | ● | ● | Collaborative SQL notebook, micro-batch queries |
| WrenAI | ○ | ● | ● | Conversational BI (text-to-SQL), batch |
| Appsmith | ○ | ● | ● | No-code app builder, CRUD/dashboards |
| Budibase | ○ | ● | ● | Low-code app builder, CRUD/dashboards |
| DataEase | ○ | ● | ● | No-code BI, dashboards + refresh |
| ToolJet | ○ | ● | ● | Low-code app builder, dashboards |
Recommendations by use case:
- Streaming: Grafana, Kibana, Streamlit, Dash, Panel, NiceGUI, Reflex, H2O Wave, Shiny
- Micro-Batching: All
- Batching: All
Web Analytics
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Ackee | ● | ● | ◐ | Privacy-respecting analytics, real-time |
| GoatCounter | ● | ● | ◐ | Lightweight analytics, real-time tracking |
| Matomo | ● | ● | ● | Real-time tracking + batch reports |
| Open Web Analytics | ● | ● | ● | Web analytics, tracking + reports |
| OpenReplay | ● | ● | ◐ | Session replay, real-time capture |
| Plausible | ● | ● | ◐ | Lightweight analytics, real-time |
| Posthog | ● | ● | ● | Product analytics, event streaming |
| Rybbit | ● | ● | ◐ | Lightweight web analytics, real-time tracking |
| Umami | ● | ● | ◐ | Simple privacy-first analytics, real-time |
Recommendations by use case:
- Streaming: All (real-time tracking)
- Micro-Batching: All
- Batching: Matomo, Posthog, Open Web Analytics
5 Platform Management
Workflow Manager
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Airflow | ○ | ◐ | ● | Batch orchestration, scheduled DAGs |
| Astronomer Cosmos | ○ | ◐ | ● | dbt integration in Airflow, batch |
| Dagster | ○ | ◐ | ● | Data orchestration, asset-based |
| Flyte | ○ | ◐ | ● | Kubernetes ML/data orchestration, batch |
| Hamilton | ○ | ◐ | ● | Dataflow/DAG micro-framework, batch |
| Kedro | ○ | ◐ | ● | Data science pipeline framework, batch |
| Kestra | ◐ | ● | ● | Versatile orchestration, event-driven |
| Metaflow | ○ | ◐ | ● | ML orchestration (Netflix), batch |
| Prefect | ○ | ◐ | ● | Workflow orchestration, batch-first |
| Temporal | ◐ | ● | ● | Durable workflow orchestration, event-driven |
| Windmill | ◐ | ● | ● | Script/workflow orchestration, triggers + batch |
| ZenML | ○ | ◐ | ● | MLOps pipelines, batch |
Recommendations by use case:
- Streaming: Kestra, Temporal, Windmill (event triggers)
- Micro-Batching: Kestra, Temporal, Windmill, Airflow (short intervals)
- Batching: All (main use case)
Governance & Quality (new tools)
Covers the Platform categories: Data Quality, Governance, Automation, Green IT, Compliance. Most of these tools run in batch mode (controls, audits, generation); a few automation tools support event-driven.
| Solution | Streaming | Micro-Batching | Batching | Notes |
|---|---|---|---|---|
| Pandera | ○ | ◐ | ● | Dataframe validation, batch controls |
| SQLFluff | ○ | ○ | ● | SQL linter, batch quality control |
| Evidently | ○ | ◐ | ● | ML/data drift monitoring, batch evaluation |
| OpenLineage | ◐ | ● | ● | Data lineage, pipeline event capture |
| CKAN | ○ | ◐ | ● | Open data catalog portal, batch |
| Apache Ranger | ◐ | ● | ● | Centralized security/authorization, continuous enforcement |
| Apache Egeria | ○ | ◐ | ● | Open governance/metadata, batch |
| Node-RED | ● | ● | ◐ | Flow-based IoT automation, event-driven |
| Activepieces | ◐ | ● | ● | No-code automation, triggers + batch |
| Huginn | ◐ | ● | ● | Automation agents, triggers + polling |
| Kepler | ○ | ◐ | ● | Energy consumption measurement (eBPF), batch metrics |
| Scaphandre | ○ | ◐ | ● | Energy metrology, batch collection/metrics |
| Microsoft Presidio | ○ | ◐ | ● | PII detection/anonymization, batch processing |
| Faker | ○ | ○ | ● | Fake data generation, batch |
| OPA | ◐ | ● | ● | Policy engine, per-request + batch evaluation |
| OpenFGA | ◐ | ● | ● | Fine-grained authorization, per-request evaluation |
| Cerbos | ◐ | ● | ● | Decoupled authorization, per-request evaluation |
| SDV | ○ | ○ | ● | Synthetic data generation, batch |
| Permify | ◐ | ● | ● | Fine-grained authorization, per-request evaluation |
Recommendations by use case:
- Streaming: Node-RED (event-driven), per-request policy engines (OPA, OpenFGA, Cerbos, Permify)
- Micro-Batching: Automation tools (Activepieces, Huginn), lineage (OpenLineage)
- Batching: Majority (quality, governance, anonymization, generation, Green IT)
Summary Tables
By Category and Dominant Mode
Number of tools with the mode as primary use case (●), by category. In parentheses: total number of solutions classified in the category.
| Category | Streaming | Micro-Batching | Batching | Total |
|---|---|---|---|---|
| Ingestion & Transport | 32 tools | 26 tools | 13 tools | 41 |
| Storage | 5 tools | 21 tools | 35 tools | 35 |
| Query & Processing | 29 tools | 45 tools | 45 tools | 63 |
| Analysis & Output | 33 tools | 49 tools | 55 tools | 61 |
| Platform Management | 1 tool | 12 tools | 30 tools | 31 |
| Total | 100 tools | 153 tools | 178 tools | 263 |
Multi-Mode Solutions (3/3)
(excluding 2026 new additions; many new tools — TSDB, NiceGUI, Perspective, Apache Kudu, etc. — are also 3/3.)
Tools that excellently support all three modes:
- Apache Camel (Ingestion)
- Apache NiFi (Ingestion)
- Graylog (Log Collection)
- Logstash (Log Collection)
- Hudi (Storage - Metadata Layer)
- Paimon (Storage - Metadata Layer)
- Apache Beam (Batch Processing)
- Apache Spark (Batch Processing - via Structured Streaming)
- ClickHouse (Datawarehouse & OLAP)
- PostgreSQL (Query Engine - with extensions)
- Apache ECharts (Framework)
- D3JS (Framework)
- Plotly (Framework)
- Streamlit (High-Code)
- Grafana (Low-Code)
- Matomo (Web Analytics)
- Posthog (Web Analytics)
Total: 17 truly versatile solutions
Leaders by Mode
(excluding 2026 new additions; editorial ranking kept as is.)
Streaming Champions
| Category | Leader #1 | Leader #2 | Leader #3 |
|---|---|---|---|
| Ingestion | Apache Kafka | Apache Pulsar | NATS |
| CDC | Debezium | Maxwell | Databus |
| Stream Processing | Apache Flink | Kafka Streams | Materialize |
| Storage | Hudi | Paimon | Delta Lake |
| OLAP Real-time | Druid | Pinot | ClickHouse |
| Visualization | Grafana | Kibana | Plotly Dash |
Micro-Batching Champions
| Category | Leader #1 | Leader #2 | Leader #3 |
|---|---|---|---|
| Ingestion | Apache NiFi | Apache Camel | - |
| Processing | Apache Spark | Apache Beam | Flink |
| Storage | Delta Lake | Iceberg | Hudi |
| Query Engine | DuckDB | Datafusion | Presto/Trino |
| OLAP | ClickHouse | Doris | StarRocks |
| Orchestration | Kestra | Airflow | Dagster |
Batching Champions
| Category | Leader #1 | Leader #2 | Leader #3 |
|---|---|---|---|
| ETL/ELT | Airbyte | dbt core | Meltano |
| Processing | Apache Spark | Apache Beam | Hop |
| Storage Format | Parquet | ORC | - |
| Dataframes | Polars | Dask | Pandas |
| OLAP | ClickHouse | StarRocks | Hive |
| BI Tools | Superset | Metabase | Evidence |
Recommended Architectures
Lambda Architecture (Batch + Streaming)
Ingestion: Kafka + Airbyte
Stream Layer: Flink → Druid/Pinot
Batch Layer: Spark → Iceberg/Delta Lake
Query: Trino (data lake) + Druid (real-time)
Viz: Superset + Grafana
Use Cases: Real-time + historical analytics, e-commerce, IoT
Kappa Architecture (Streaming Only)
Ingestion: Kafka
Processing: Flink → Materialize
Storage: Hudi/Paimon
Query: ClickHouse
Viz: Grafana + Dash
Use Cases: Real-time IoT, monitoring, fraud detection
Modern Batch Architecture
Ingestion: Airbyte
Orchestration: Airflow/Dagster
Processing: dbt + Spark
Storage: Iceberg on MinIO
Query: Trino
Viz: Superset
Use Cases: Modern data warehouse, BI, ML
Micro-Batching Architecture
Ingestion: NiFi (1-minute batches)
Processing: Spark Structured Streaming
Storage: Delta Lake
Query: DuckDB + StarRocks
Viz: Metabase (auto-refresh)
Use Cases: Near real-time dashboards, business KPIs
Selection Guide
Choosing by Required Latency
| Need | Target Latency | Recommended Mode | Example Stack |
|---|---|---|---|
| Critical alerts | < 100ms | Streaming | NATS + Flink + Druid |
| Real-time dashboards | < 1 sec | Streaming | Kafka + Flink + ClickHouse |
| Near real-time analytics | 1-5 min | Micro-Batching | Spark Streaming + Delta Lake |
| Hourly reports | 1 hour | Batching | Airflow + dbt + Iceberg |
| Daily reports | 24 hours | Batching | Airbyte + Spark + Parquet |
Choosing by Volume
| Daily Volume | Recommended Mode | Justification |
|---|---|---|
| < 10 GB | Streaming/Micro-Batching | Little data, prioritize latency |
| 10-100 GB | Micro-Batching | Good latency/cost trade-off |
| 100 GB - 1 TB | Micro-Batching/Batching | Optimize throughput |
| > 1 TB | Batching | Cost and efficiency paramount |
Choosing by Use Case
| Use Case | Mode | Recommended Stack |
|---|---|---|
| System monitoring | Streaming | Vector → ClickHouse → Grafana |
| Fraud detection | Streaming | Kafka → Flink → Druid → Alerts |
| E-commerce analytics | Lambda | Kafka + Flink (stream) + Spark (batch) |
| Data Warehouse | Batching | Airbyte → dbt → Iceberg → Trino |
| IoT sensors | Streaming | MQTT → Kafka → Flink → TimescaleDB |
| BI Dashboards | Micro-Batching | dbt → StarRocks → Superset (5min refresh) |
| ML Training | Batching | Spark → Parquet → Notebooks |
| Customer 360 | Lambda | CDC (stream) + ETL (batch) → Delta Lake |
Cross-References
Notes
Evolution of Paradigms
Current trend: Convergence toward unified batch + streaming solutions
- Apache Beam: Unified API
- Apache Spark Structured Streaming: Advanced micro-batching
- Flink: Batch on streaming engine
- Iceberg/Hudi/Delta Lake: Unified storage
2025 recommendation: Favor versatile solutions to reduce operational complexity.
Document created on: 2025-12-09 Last updated: 2026-06-25 Version: 2.1 Status: [OK] Complete - 263 solutions classified