Home

Query and Processing Tools

Query and processing tools are essential components of any data analytics infrastructure. They enable organizations to extract insights from large volumes of data, perform complex computations, and support decision-making processes. These tools can be broadly categorized into six main areas: query engines, stream processing, batch processing, dataframe processing, datawarehouse & OLAP, and time-series databases.

They can be broadly categorized into six main areas:

Available Tools

Here is a summary table of the main query and processing tools we have identified.

Query Engine

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
Apache Calcite Query Engine 25/06/2014 5216 2550 341 N/A 18/11/2026 Yes Apache-2.0 https://github.com/apache/calcite
Apache Drill Query Engine 05/09/2012 2027 984 164 29/06/2025 12/11/2026 Yes Apache-2.0 https://github.com/apache/drill
chDB Query Engine 25/02/2023 2930 139 20 18/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/chdb-io/chdb
Datafusion Query Engine 17/04/2021 9614 2660 427 N/A 18/11/2026 Yes Apache-2.0 https://github.com/apache/arrow-datafusion
DuckDB Query Engine 26/06/2018 43166 4221 340 21/10/2026 18/11/2026 Yes MIT https://github.com/duckdb/duckdb
Hydra Query Engine 22/07/2022 3042 106 16 01/04/2024 10/02/2025 No Apache-2.0 https://github.com/hydradatabase/hydra
PostgreSQL Query Engine 21/09/2010 22870 6114 42 N/A 18/11/2026 Yes PostgreSQL https://github.com/postgres/postgres
Presto Query Engine 09/08/2012 16778 5554 307 04/11/2026 19/11/2026 Yes Apache-2.0 https://github.com/prestodb/presto
Quack on Demand Query Engine 27/05/2026 155 11 5 15/11/2026 18/11/2026 No Apache-2.0 https://github.com/starlake-ai/quack-on-demand
Trino Query Engine 19/01/2019 13528 3880 330 16/10/2026 19/11/2026 Yes Apache-2.0 https://github.com/trinodb/trino
Velox Query Engine 23/07/2021 4270 1672 298 N/A 19/11/2026 Yes Apache-2.0 https://github.com/facebookincubator/velox

Stream Processing

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
Apache Flink Stream Processing 07/06/2014 26480 14193 285 N/A 18/11/2026 Yes Apache-2.0 https://github.com/apache/flink
Apache Kafka Stream Processing 15/08/2011 34233 15706 354 N/A 18/11/2026 Yes Apache-2.0 https://github.com/apache/kafka
Apache Samza Stream Processing 14/03/2015 844 332 133 N/A 20/08/2026 Yes Apache-2.0 https://github.com/apache/samza
Apache Storm Stream Processing 05/11/2013 6703 4024 284 11/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/apache/storm
Apache StreamPipes Stream Processing 22/04/2018 763 252 93 15/12/2025 18/11/2026 No Apache-2.0 https://github.com/apache/streampipes
Arroyo Stream Processing 31/03/2023 5083 391 43 17/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/ArroyoSystems/arroyo
Bytewax Stream Processing 04/02/2022 2074 114 33 25/11/2024 13/11/2026 Yes Apache-2.0 https://github.com/bytewax/bytewax
Faust-streaming Stream Processing 22/10/2020 1892 206 141 28/10/2026 28/10/2026 Yes BSD-3-Clause https://github.com/faust-streaming/faust
Feldera Stream Processing 11/05/2023 2180 181 67 17/11/2026 19/11/2026 No MIT https://github.com/feldera/feldera
Materialize Stream Processing 22/02/2019 6407 519 167 19/11/2026 19/11/2026 Yes BSL-1.1 https://github.com/MaterializeInc/materialize
Numaflow Stream Processing 20/05/2022 2838 187 109 29/10/2026 18/11/2026 Yes Apache-2.0 https://github.com/numaproj/numaflow
Pathway Stream Processing 27/11/2022 61982 1704 43 01/08/2026 18/11/2026 Yes BSL-1.1 https://github.com/pathwaycom/pathway
Quix Streams Stream Processing 17/11/2022 1583 120 41 15/10/2026 18/11/2026 Yes Apache-2.0 https://github.com/quixio/quix-streams
Redpanda Stream Processing 02/11/2020 12673 811 175 22/08/2026 12/11/2026 Yes BSL-1.1 https://github.com/redpanda-data/redpanda
RisingWave Stream Processing 28/01/2022 9424 868 226 05/11/2026 19/11/2026 Yes Apache-2.0 https://github.com/risingwavelabs/risingwave
Timeplus Proton Stream Processing 14/08/2023 2288 112 291 10/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/timeplus-io/proton

Batch Processing

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
AmphiETL Batch Processing 20/03/2024 1411 108 10 N/A 24/08/2026 Yes Elastic-2.0 https://github.com/amphi-ai/amphi-etl
Apache Beam Batch Processing 02/02/2016 8722 4821 306 02/11/2026 19/11/2026 Yes Apache-2.0 https://github.com/apache/beam
Apache Hop Batch Processing 24/09/2019 1542 493 140 27/10/2026 18/11/2026 Yes Apache-2.0 https://github.com/apache/hop
Apache Sedona Batch Processing 24/04/2015 2430 798 179 05/08/2026 18/11/2026 Yes Apache-2.0 https://github.com/apache/sedona
Apache Spark Batch Processing 25/02/2014 44563 29596 331 N/A 19/11/2026 Yes Apache-2.0 https://github.com/apache/spark
dbt core Batch Processing 10/03/2016 14192 2681 383 12/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/dbt-labs/dbt-core
Duckle Batch Processing 21/05/2026 1577 121 12 09/11/2026 18/11/2026 No Apache-2.0 https://github.com/SouravRoy-ETL/duckle
Fugue Batch Processing 24/03/2020 2168 103 23 12/11/2026 12/11/2026 Yes Apache-2.0 https://github.com/fugue-project/fugue
Odyssée Batch Processing 21/11/2024 0 0 2 N/A 26/06/2025 No N/A https://gitlab.adullact.net/dgfip/odyssee
Ray Batch Processing 25/10/2016 44324 8290 411 18/11/2026 19/11/2026 Yes Apache-2.0 https://github.com/ray-project/ray
Talaxie Batch Processing 28/05/2024 12 6 140 N/A 20/10/2024 No N/A https://github.com/Talaxie/tdi-studio-se

Dataframe Processing

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
cuDF (RAPIDS) Dataframe Processing 07/05/2017 9801 1165 313 20/10/2026 19/11/2026 Yes Apache-2.0 https://github.com/rapidsai/cudf
Daft Dataframe Processing 25/04/2022 5853 590 193 04/11/2026 09/11/2026 Yes Apache-2.0 https://github.com/Eventual-Inc/Daft
Dask Dataframe Processing 04/01/2015 13958 2031 416 24/08/2026 17/11/2026 Yes BSD-3-Clause https://github.com/dask/dask
Ibis Project Dataframe Processing 17/04/2015 6704 789 227 07/02/2026 05/11/2026 Yes Apache-2.0 https://github.com/ibis-project/ibis
Modin Dataframe Processing 21/06/2018 10392 680 137 02/10/2025 10/02/2026 Yes Apache-2.0 https://github.com/modin-project/modin
Pandas Dataframe Processing 24/08/2010 50569 20753 414 04/11/2026 18/11/2026 Yes BSD-3-Clause https://github.com/pandas-dev/pandas
Polars Dataframe Processing 13/05/2020 40379 3243 444 25/10/2026 19/11/2026 Yes MIT https://github.com/pola-rs/polars
Vaex Dataframe Processing 27/09/2014 8519 601 70 29/03/2018 06/11/2026 Yes MIT https://github.com/vaexio/vaex
Xarray Dataframe Processing 30/09/2013 4238 1530 421 29/09/2026 18/11/2026 Yes Apache-2.0 https://github.com/pydata/xarray

Datawarehouse & OLAP

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
Apache Hive Datawarehouse & OLAP 21/05/2009 6040 4795 259 N/A 18/11/2026 Yes Apache-2.0 https://github.com/apache/hive
Apache Impala Datawarehouse & OLAP 13/04/2016 1295 565 190 06/10/2026 19/11/2026 Yes Apache-2.0 https://github.com/apache/impala
Apache Kylin Datawarehouse & OLAP 03/01/2015 3786 1502 60 06/04/2025 13/11/2026 Yes Apache-2.0 https://github.com/apache/kylin
ClickHouse Datawarehouse & OLAP 02/06/2016 51004 9288 312 18/11/2026 19/11/2026 Yes Apache-2.0 https://github.com/ClickHouse/ClickHouse
Doris Datawarehouse & OLAP 10/08/2017 16242 4027 330 15/10/2026 19/11/2026 Yes Apache-2.0 https://github.com/apache/doris
Druid Datawarehouse & OLAP 23/10/2012 14098 3821 355 01/10/2026 18/11/2026 Yes Apache-2.0 https://github.com/apache/druid
Pinot Datawarehouse & OLAP 19/05/2014 6178 1537 367 23/10/2026 19/11/2026 Yes Apache-2.0 https://github.com/apache/pinot
StarRocks Datawarehouse & OLAP 04/09/2021 12267 2682 392 03/11/2026 19/11/2026 Yes Apache-2.0 https://github.com/StarRocks/starrocks

*Criteria: >40 contributors, >500 stars, and recent releases/commit

Time-Series Database

Time-series databases are optimized for ingesting and querying large volumes of timestamped data at high throughput. They are commonly used for metrics, IoT sensor telemetry, and observability workloads.

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
Apache IoTDB Time-Series Database 24/11/2018 6400 N/A 250 N/A 13/06/2026 Yes Apache-2.0 https://github.com/apache/iotdb
CrateDB Time-Series Database 10/04/2013 4400 N/A 150 N/A 13/06/2026 Yes Apache-2.0 https://github.com/crate/crate
GreptimeDB Time-Series Database 11/04/2022 6400 N/A 300 N/A 12/06/2026 Yes Apache-2.0 https://github.com/GreptimeTeam/greptimedb
InfluxDB Time-Series Database 26/09/2013 31500 N/A 40 N/A 12/06/2026 Yes MIT https://github.com/influxdata/influxdb
QuestDB Time-Series Database 28/04/2014 17000 N/A 160 N/A 13/06/2026 Yes Apache-2.0 https://github.com/questdb/questdb
TDengine Time-Series Database 01/01/2019 24900 N/A 200 N/A 13/06/2026 Yes AGPL-3.0 https://github.com/taosdata/TDengine
TimescaleDB Time-Series Database 07/03/2017 22900 N/A 338 N/A 13/06/2026 Yes Apache-2.0 https://github.com/timescale/timescaledb
VictoriaMetrics Time-Series Database 30/09/2018 17100 N/A 405 N/A 12/06/2026 Yes Apache-2.0 https://github.com/VictoriaMetrics/VictoriaMetrics

*Criteria: >40 contributors, >500 stars, and recent releases/commit

Tool Details

Query Engine

  1. Apache Calcite: Dynamic data management framework providing query optimization, data federation, and more.
  2. Apache Drill: Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage.
  3. chDB: In-process SQL OLAP engine powered by ClickHouse, embeddable directly into applications like DuckDB.
  4. Datafusion: Fast query engine for Apache Arrow, written in Rust.
  5. DuckDB: In-process SQL OLAP database management system, designed to be fast and efficient for analytical queries.
  6. GraphQL: Query language for APIs and a runtime for executing those queries with existing data.
  7. Hydra: Unified interface for constructing and executing complex analytical queries across different query engines and data sources.
  8. PostgreSQL: Powerful, open source object-relational database system with a strong reputation for reliability and data integrity.
  9. Presto: Distributed SQL query engine for big data, designed for fast analytic queries against data of any size.
  10. Trino: Fast distributed SQL query engine for big data analytics, designed to efficiently query vast amounts of data.
  11. Velox: Unified C++ execution engine library that accelerates data management systems and query engines.
  12. Quack on Demand: Open-source control plane running autoscaling fleets of DuckDB nodes on Kubernetes via an Arrow Flight SQL gateway, with per-tenant isolation, access controls and federated queries from any ODBC/JDBC/ADBC client.

Stream Processing

  1. Apache Flink: Stateful computations over data streams, providing precise control of time and state.
  2. Apache Kafka: Distributed event streaming platform capable of handling trillions of events a day.
  3. Apache Samza: Distributed stream processing framework that uses Apache Kafka for messaging, and Hadoop YARN for fault tolerance.
  4. Apache Storm: Distributed real-time computation system for processing fast, large streams of data.
  5. Apache StreamPipes: Self-service toolbox for building, deploying, and managing industrial IoT data stream processing pipelines.
  6. Arroyo: Distributed stream processing engine that runs stateful SQL queries on high-volume real-time data.
  7. Bytewax: Python-native stream processing framework built on a Rust dataflow engine for stateful workloads.
  8. Faust-streaming: Python stream processing library that ports Kafka Streams ideas to the Python ecosystem.
  9. Feldera: Incremental computation engine that continuously evaluates SQL queries over changing data.
  10. Materialize: Streaming database that makes it easy to build real-time applications on streaming data.
  11. Numaflow: Kubernetes-native platform for running scalable and reliable stream processing pipelines.
  12. Pathway: Python framework for building real-time data and AI pipelines over streaming and batch data; licensed under BSL-1.1.
  13. Quix Streams: Python library for building real-time stream processing applications on top of Kafka.
  14. Redpanda: Modern streaming platform compatible with Kafka API, built for mission-critical workloads with high performance.
  15. RisingWave: Distributed SQL streaming database for building real-time event-driven applications and materialized views.
  16. Timeplus Proton: Streaming SQL engine powered by ClickHouse for fast real-time and historical analytics.

Batch Processing

  1. AmphiETL: Cloud-native ETL platform built for modern data teams, offering scalable data transformations and integrations.
  2. Apache Beam: Unified programming model for batch and streaming data processing, offering language-specific SDKs.
  3. Apache Hop: Data orchestration and data engineering platform designed for visual development of data pipelines and workflows.
  4. Apache Sedona: Cluster computing system for processing large-scale spatial data with distributed geospatial operations.
  5. Apache Spark: Fast and general-purpose cluster computing system, providing high-level APIs in Java, Scala, Python and R.
  6. dbt core: Transforms data in warehouses by allowing analysts and engineers to define models using SQL SELECT statements.
  7. Duckle: Lightweight ETL tool built around DuckDB for portable batch data processing.
  8. Fugue: Unified interface for distributed computing that lets users run Python, Pandas, and SQL code on Spark, Dask, and Ray.
  9. Odyssée: Web-based data preparation tool by the French public finances directorate (DGFiP) that lets technical and non-technical users build datasets, query them via an assistant or SQL (powered by Trino), and chain queries into reproducible treatments orchestrated with Kestra. Hosted on the ADULLACT GitLab forge; no license file declared at time of listing.
  10. Ray: Unified framework for scaling Python and AI applications, from data processing to distributed training and serving.

Dataframe Processing

  1. cuDF (RAPIDS): GPU DataFrame library that provides a Pandas-like API for loading, joining, and aggregating data on NVIDIA GPUs.
  2. Daft: Distributed DataFrame library for multimodal data, built in Rust for fast Python-native data processing.
  3. Dask: Flexible library for parallel computing in Python, scaling Python and Pandas workflows efficiently.
  4. Ibis Project: Expression compiler for analytics, bridging different query engines with a unified Python API for data analytics.
  5. Modin: Drop-in replacement for Pandas that transparently distributes DataFrame operations across all available cores.
  6. Pandas: Powerful Python data manipulation and analysis library, offering data structures for efficiently storing large datasets.
  7. Polars: Fast multi-threaded DataFrame library for Rust and Python, designed as a faster alternative to Pandas.
  8. Vaex: Out-of-core DataFrame library for lazy, memory-efficient exploration of very large tabular datasets.
  9. Xarray: Library for working with labeled multi-dimensional arrays, bringing Pandas-like semantics to N-dimensional data.

Datawarehouse & OLAP

  1. Apache Hive: Data warehouse software facilitating reading, writing, and managing large datasets in distributed storage using SQL.
  2. Apache Impala: Massively Parallel Processing (MPP) SQL query engine for data stored in Hadoop clusters.
  3. Apache Kylin: Extreme OLAP engine for big data that allows for sub-second queries on datasets with trillions of rows.
  4. ClickHouse: Open-source column-oriented database management system for real-time analytics using SQL.
  5. Doris: High-performance real-time analytical database based on MPP architecture.
  6. Druid: High performance real-time analytics database designed for workflows where fast queries and ingest really matter.
  7. StarRocks: High-performance analytical database that enables real-time, multi-dimensional, and highly concurrent data analysis.

Time-Series Database

  1. Apache IoTDB: Time-series database purpose-built for IoT scenarios, offering efficient storage and querying of device telemetry.
  2. CrateDB: Distributed SQL database combining the scalability of NoSQL with relational queries for time-series and machine data.
  3. GreptimeDB: Cloud-native time-series database unifying metrics, logs, and events for observability workloads.
  4. InfluxDB: Purpose-built time-series database optimized for high-write and query loads of metrics and events.
  5. QuestDB: High-performance time-series database with SQL support, designed for fast ingestion and real-time analytics.
  6. TDengine: High-performance time-series database built for IoT, connected cars, and industrial monitoring; licensed under AGPL-3.0.
  7. TimescaleDB: Time-series database built as a PostgreSQL extension for scalable storage and analytics; licensed under Apache-2.0 and the Timescale License (TSL).
  8. VictoriaMetrics: Fast and cost-efficient time-series database and monitoring solution compatible with Prometheus.

These tools offer a wide range of capabilities for querying and processing data in various scenarios. When choosing a tool, consider factors such as:

Remember that different categories of tools can be combined to create comprehensive data processing pipelines:

The choice of tools can significantly impact the performance and capabilities of your data analytics infrastructure. It’s often beneficial to combine multiple tools to address different aspects of your data processing needs while maintaining a balance between functionality, complexity, and maintainability.

The Challenge of Choice

The open-source community has developed numerous solutions for various aspects of data handling, including: