Flow × Function Matrix - Two-Dimensional Classification

Complete navigable/filterable list (stars, dates, licenses, top/flop): site generated from the catalogs → docs/ (GitHub Pages). This document is an editorial analysis.

Objective: Classify solutions along two dimensions: Flow Typology (Streaming/Micro-Batching/Batching) AND Function in the data architecture.

Introduction

This two-dimensional classification allows selecting tools according to:

1. The temporal axis: Streaming, Micro-Batching, Batching
2. The functional axis: Collection, Transport, Storage, Processing, Analysis, Governance

The Two Dimensions

Dimension 1: Flow Typology

Dimension 2: Data Function

Global Matrix: Flow × Function

Overview

Total solutions analyzed: 263

Detailed Classification by Function

FUNCTION: COLLECTION

Initial capture of data from sources (APIs, databases, files, logs, events)

Collection × Flow Matrix

Collection Leaders: (excluding new additions)

• Streaming: Debezium (CDC), Vector (logs), NiFi (versatile)
• Micro-Batching: NiFi, Camel, Graylog
• Batching: Airbyte (ETL), Meltano, dlt

FUNCTION: TRANSPORT

Reliable delivery and routing of data between systems

Transport × Flow Matrix

Transport Leaders: (excluding new additions)

• Streaming: Kafka (standard), NATS (lightweight), Pulsar (multi-DC)
• Micro-Batching: Kafka, Pulsar, RabbitMQ
• Batching: NiFi, Camel

FUNCTION: STORAGE

Persistence, organization and access to data

Storage × Flow Matrix

Storage Leaders: (excluding new additions)

• Streaming: Hudi, Paimon, ClickHouse
• Micro-Batching: Delta Lake, Iceberg, Hudi, MinIO
• Batching: Parquet, ORC, Iceberg, MinIO

FUNCTION: PROCESSING

Transformation, aggregation and enrichment of data

Processing × Flow Matrix

Processing Leaders: (excluding new additions)

• Streaming: Flink (leader), Materialize, Druid, Pinot
• Micro-Batching: Spark, Beam, DuckDB, Polars
• Batching: Spark, Trino, dbt, Presto

FUNCTION: ANALYSIS

Visualization, exploration and business intelligence

Analysis × Flow Matrix

Analysis Leaders: (excluding new additions)

• Streaming: Grafana (monitoring), Kibana (logs), Dash, Streamlit
• Micro-Batching: Superset, Metabase, Grafana
• Batching: Superset, Metabase, Matplotlib

FUNCTION: GOVERNANCE

Orchestration, quality, security, compliance

Governance × Flow Matrix

Governance Leaders: (excluding new additions)

• Streaming: Kestra (event triggers)
• Micro-Batching: Kestra, Great Expectations
• Batching: Airflow (standard), Dagster, DataHub

Recommendation Matrices

Matrix 1: By Use Case × Function

Matrix 2: Full Stack by Flow × All

PURE STREAMING STACK (Latency < 1s)

Architecture Example:

MySQL → Debezium → Kafka → Flink → Hudi → Trino → Grafana
                             ↓
                         ClickHouse → Superset

MICRO-BATCHING STACK (Latency 1s-5min)

Architecture Example:

APIs → NiFi → Kafka → Spark Streaming → Delta Lake → Trino → Superset
                                            ↓
                                        StarRocks → Grafana (refresh 1min)

BATCHING STACK (Latency > 5min)

Architecture Example:

Sources → Airbyte → MinIO/S3 (Raw) → Spark → Iceberg (Processed)
                                       ↓
                                      dbt → Iceberg (Curated) → Trino → Superset
                                                                           ↓
                                                                      Airflow (orchestration)

Summary Tables

Top 10 Multi-Function Solutions

Functional Coverage by Flow

Insight: Micro-batching offers the best overall coverage (204 tool-function combinations).

Gap Analysis by Function

Recommendations:

• Streaming: Needs orchestration (Kestra in development)
• Batching: Needs real-time transport (not critical)

Two-Dimensional Selection Guide

Step 1: Identify the Function

Question: What is the position in the pipeline?

• Start (Source) → Collection
• Middle (Routing) → Transport
• Base (Data) → Storage
• Middle (Transform) → Processing
• End (Insights) → Analysis
• Cross-cutting (Ops) → Governance

Step 2: Identify the Flow

Question: What is the maximum acceptable latency?

• < 1 second → Streaming
• 1 sec - 5 min → Micro-Batching
• > 5 minutes → Batching

Step 3: Consult the Matrix

Refer to the corresponding section (e.g., “Collection × Streaming”)

Decision Example

Need: E-commerce dashboards with real-time metrics (sales, inventory)

1. Function: Analysis (end of pipeline)
2. Required latency: 10 seconds → Micro-Batching
3. Analysis × Micro-Batch Matrix:
   • Options: Superset ●, Grafana ●, Metabase ●
4. Choice: Superset (rich BI) + Grafana (monitoring)

Two-Dimensional Architecture Patterns

Pattern 1: Full Lambda (All Functions)

 COLLECTION
├─ Streaming: Debezium (CDC)
└─ Batching: Airbyte (Bulk)

 TRANSPORT
└─ Streaming: Kafka

 STORAGE
├─ Hot: ClickHouse (streaming)
└─ Cold: Iceberg (batch)

 PROCESSING
├─ Streaming: Flink
└─ Batching: Spark + dbt

 ANALYSIS
├─ Real-time: Grafana
└─ BI: Superset

 GOVERNANCE
└─ Batching: Airflow

Pattern 2: Simplified Kappa (Streaming Only)

 COLLECTION: Fluent Bit
 TRANSPORT: NATS
 STORAGE: Paimon
 PROCESSING: Flink
 ANALYSIS: Grafana
 GOVERNANCE: Kestra (event-driven)

Pattern 3: Modern Data Stack (Batch + BI)

 COLLECTION: Airbyte
 STORAGE: Iceberg on MinIO
 PROCESSING: dbt
 ANALYSIS: Superset
 GOVERNANCE: Airflow + DataHub

Heatmap: Solution Maturity

Maturity Legend

• ● Mature (production-ready, large adoption)
• ◐ Growing (production possible, moderate adoption)
• ○ Emerging (early-stage, limited adoption)

Insights:

• Mature zone: Batch (all functions) and Micro-Batch
• In-development zone: Streaming Governance, Streaming Storage
• Solid zone: Transport and Processing (all flows)

Cross-References

• Classification by data flow
• Ingestion and Transport
• Storage
• Query and Processing
• Analysis and Output
• Platform Management

Classification Methodology

Function Criteria

Each tool is classified according to its primary function:

1. Collection: Initial capture from sources
2. Transport: Reliable delivery between systems
3. Storage: Persistence and organization
4. Processing: Transformation and enrichment
5. Analysis: Visualization and BI
6. Governance: Orchestration and quality

Flow Criteria

Classification according to 3 modes based on latency observed in production:

• ● Primary: Optimal and recommended usage
• ◐ Secondary: Supported but not optimal
• ○ Not supported: Not recommended

Validation

Each classification is based on:

• Official documentation
• Production use cases
• Community benchmarks
• Established architecture patterns

Document created on: 2025-12-09 Last updated: 2026-06-25 Version: 2.1 Status: [OK] Complete - 263 solutions × 6 functions × 3 flows