Data Storage Tools and Formats

Data storage is a fundamental aspect of any data management strategy. It involves organizing and preserving data in various formats and systems to ensure efficient access, retrieval, and analysis. In the context of big data and modern analytics, choosing the right storage solution is crucial for performance, scalability, and data integrity.

They can be broadly categorized into five main areas:

Available Tools and Formats

Here is a summary table of the main data storage tools and formats we have identified.

File Layer

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
Apache Arrow File Layer 17/02/2016 17314 4465 371 03/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/apache/arrow
Apache Kudu File Layer 29/01/2016 1919 665 131 06/12/2024 18/11/2026 Yes Apache-2.0 https://github.com/apache/kudu
Avro File Layer 21/05/2009 3314 1795 390 23/08/2026 15/11/2026 Yes Apache-2.0 https://github.com/apache/avro
DuckLake File Layer 03/03/2025 3376 303 92 N/A 13/11/2026 Yes MIT https://github.com/duckdb/ducklake
Lance File Layer 07/07/2022 7363 955 350 17/11/2026 19/11/2026 Yes Apache-2.0 https://github.com/lancedb/lance
ORC File Layer 06/05/2015 774 519 148 16/07/2026 09/11/2026 Yes Apache-2.0 https://github.com/apache/orc
Parquet File Layer 10/06/2014 3093 1589 278 11/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/apache/parquet-mr
Vortex File Layer 27/02/2024 3329 267 99 06/11/2026 19/11/2026 No Apache-2.0 https://github.com/spiraldb/vortex

Object Storage

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
Apache Ozone Object Storage 02/10/2019 1336 657 310 30/10/2026 18/11/2026 Yes Apache-2.0 https://github.com/apache/ozone
Ceph Object Storage 01/09/2011 17245 6574 288 N/A 18/11/2026 Yes LGPL-2.1 https://github.com/ceph/ceph
Garage Object Storage 17/11/2021 4900 190 101 N/A 17/11/2026 Yes AGPL-3.0 https://github.com/deuxfleurs-org/garage
JuiceFS Object Storage 08/01/2021 14753 1363 210 22/10/2026 19/11/2026 Yes Apache-2.0 https://github.com/juicedata/juicefs
MinIO Object Storage 14/01/2015 61244 8352 410 16/10/2025 12/02/2026 Yes AGPL-3.0 https://github.com/minio/minio
SeaweedFS Object Storage 14/07/2014 36145 3130 402 16/11/2026 19/11/2026 Yes Apache-2.0 https://github.com/seaweedfs/seaweedfs

Metadata Layer

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
Apache Gravitino Metadata Layer 23/04/2023 3400 997 369 30/09/2026 19/11/2026 Yes Apache-2.0 https://github.com/apache/gravitino
Apache XTable Metadata Layer 21/07/2023 1352 226 56 24/08/2026 12/11/2026 No Apache-2.0 https://github.com/apache/incubator-xtable
Delta Lake Metadata Layer 22/04/2019 9086 2225 387 05/10/2026 19/11/2026 Yes Apache-2.0 https://github.com/delta-io/delta
Hive Metastore Metadata Layer 21/05/2009 6040 4795 259 N/A 18/11/2026 Yes Apache-2.0 https://github.com/apache/hive
Hudi Metadata Layer 14/12/2016 6386 2542 388 17/10/2026 19/11/2026 Yes Apache-2.0 https://github.com/apache/hudi
Iceberg Metadata Layer 19/11/2018 9445 3698 406 30/09/2026 19/11/2026 Yes Apache-2.0 https://github.com/apache/iceberg
LakeFS Metadata Layer 12/09/2019 5586 493 121 01/10/2026 18/11/2026 Yes Apache-2.0 https://github.com/treeverse/lakeFS
Lakekeeper Metadata Layer 05/04/2024 1512 180 65 15/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/lakekeeper/lakekeeper
Nessie Metadata Layer 09/04/2020 1534 204 87 06/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/projectnessie/nessie
Paimon Metadata Layer 12/01/2022 3452 1437 390 12/10/2026 18/11/2026 Yes Apache-2.0 https://github.com/apache/paimon
Polaris Metadata Layer 29/05/2024 2121 575 196 14/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/apache/polaris
Unity Catalog (OSS) Metadata Layer 13/06/2024 3581 705 141 13/10/2026 19/11/2026 Yes Apache-2.0 https://github.com/unitycatalog/unitycatalog

Vector Storage

Vector storage systems store and index high-dimensional embeddings to enable similarity search at scale. They are a core building block for AI applications such as retrieval-augmented generation (RAG), semantic search, and recommendation engines.

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
Chroma Vector Storage 05/10/2022 29712 2609 191 01/04/2025 07/11/2026 Yes Apache-2.0 https://github.com/chroma-core/chroma
Milvus Vector Storage 16/09/2019 46688 4335 344 11/11/2026 19/11/2026 Yes Apache-2.0 https://github.com/milvus-io/milvus
pgvector Vector Storage 20/04/2021 23730 1400 23 N/A 18/11/2026 Yes PostgreSQL https://github.com/pgvector/pgvector
Qdrant Vector Storage 30/05/2020 35633 2824 217 16/11/2026 16/11/2026 Yes Apache-2.0 https://github.com/qdrant/qdrant
Weaviate Vector Storage 30/03/2016 16954 1440 162 16/11/2026 18/11/2026 Yes BSD-3-Clause https://github.com/weaviate/weaviate

*Criteria: >40 contributors, >500 stars, and recent releases/commit

Data Modeling

Tool Subcategory Creation Date Stars Forks Contributors Last Release Latest Commit Meets Criteria* License Link
Big Functions Data Modeling 24/08/2022 762 69 38 15/05/2025 23/08/2026 No MIT https://github.com/unytics/bigfunctions
dbt core Data Modeling 10/03/2016 14192 2681 383 12/11/2026 18/11/2026 Yes Apache-2.0 https://github.com/dbt-labs/dbt-core
GraphQL Data Modeling 01/07/2015 14596 1161 132 12/11/2026 13/11/2026 Yes MIT https://github.com/graphql/graphql-spec
SQL Mesh Data Modeling 23/09/2022 3362 493 183 22/10/2026 17/11/2026 Yes Apache-2.0 https://github.com/TobikoData/sqlmesh

*Criteria: >40 contributors, >500 stars, and recent releases/commit

Tool and Format Details

File Layer

  1. Apache Arrow: A language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations and zero-copy data sharing across systems.
  2. Apache Kudu: A columnar storage engine for the Hadoop ecosystem that enables fast analytics on rapidly changing data, combining low-latency random access with efficient analytical scans.
  3. Avro: A row-based storage format, Avro is a data serialization system that provides rich data structures and a compact, fast, binary data format.
  4. DuckLake: An open table format that stores lakehouse metadata in a standard SQL database while keeping data in open formats, simplifying lakehouse management.
  5. Lance: A modern columnar data format optimized for machine learning and AI workloads, offering fast random access, versioning, and efficient vector search.
  6. ORC: (Optimized Row Columnar) A highly efficient way to store Hive data. It was designed to overcome limitations of other Hive file formats.
  7. Parquet: A columnar storage file format available to any project in the Hadoop ecosystem. Parquet is built from the ground up with complex nested data structures in mind.
  8. Vortex: A next-generation columnar file format and in-memory layout designed for high compression and fast random access, aimed at analytical and AI workloads.

Object Storage

  1. Apache Ozone: A scalable, distributed object store for Hadoop and cloud-native environments, providing both S3 and Hadoop-compatible APIs for storing billions of objects.
  2. Ceph: A highly scalable, software-defined distributed storage platform that delivers object, block, and file storage from a single unified cluster.
  3. Garage: A lightweight, self-hosted S3-compatible distributed object storage system designed for geo-distributed deployments on commodity hardware, licensed under AGPL-3.0.
  4. JuiceFS: A distributed POSIX-compatible file system built on top of object storage, providing high-performance shared storage for cloud-native and data-intensive workloads.
  5. MinIO: High-performance, S3-compatible object storage built for large-scale AI/ML, data lake, and datalake analytics workloads. Designed for multi-cloud environments with native Kubernetes integration. Serves as production-ready, self-hosted alternative to AWS S3, Azure Blob Storage, and Google Cloud Storage. Used by enterprises worldwide for modern data infrastructure.
  6. SeaweedFS: A fast distributed storage system for blobs, objects, files, and a data lake, offering S3-compatible APIs and optimized for handling billions of small files.

Metadata Layer

  1. Apache Gravitino: A high-performance, geo-distributed metadata lake that manages metadata across data sources, enabling unified data cataloging and governance for data and AI assets.
  2. Apache XTable: A cross-table interoperability project that provides omni-directional translation between Apache Hudi, Apache Iceberg, and Delta Lake table formats.
  3. Delta Lake: An open-source storage layer bringing ACID transactions to Apache Spark and big data workloads, enabling reliable data lake operations with time travel, schema enforcement, and batch/streaming unification.
  4. Hive Metastore: A centralized metadata repository service that stores semantic and technical metadata about data assets, providing metadata management and schema validation for Hadoop ecosystem components.
  5. Hudi: A data lake storage system enabling atomic publishing, record-level updates/deletes, and incremental data processing. It offers snapshot isolation and provides efficient upsert and delete capabilities.
  6. Iceberg: A table format for massive analytic datasets, offering schema evolution, hidden partitioning, and snapshot isolation. Supports efficient reads and writes with partition pruning and metadata handling.
  7. LakeFS: An open-source data versioning system that brings Git-like branching, committing, and merging operations to object-storage-based data lakes.
  8. Lakekeeper: An open-source Apache Iceberg REST catalog implementation, providing centralized table management and governance for Iceberg-based lakehouses.
  9. Nessie: A Git-like version control system for data lakes enabling branch-based development, metadata versioning, and time travel capabilities across multiple table formats and data lake configurations.
  10. Paimon: A streaming data lake platform optimized for high-speed data ingestion and real-time analytics, featuring changelog tracking and efficient incremental computation on dynamic datasets.
  11. Polaris: A unified metadata management system providing data discovery, lineage tracking, and governance capabilities across data lakes and warehouses with integrated security controls.
  12. Unity Catalog (OSS): An open-source, universal catalog for data and AI that provides centralized governance, access control, and lineage across tables, files, functions, and ML models.

Vector Storage

  1. Chroma: An open-source embedding database designed to make it easy to build LLM applications with memory, offering simple APIs for storing and querying vector embeddings.
  2. Milvus: A cloud-native, highly scalable vector database built for managing and searching massive volumes of embeddings to power similarity search and AI applications.
  3. pgvector: An open-source vector similarity search extension for PostgreSQL that adds vector data types and indexing; it is a PostgreSQL extension, not a standalone DB.
  4. Qdrant: A high-performance vector similarity search engine and database written in Rust, providing production-ready filtering, payloads, and an easy-to-use API.
  5. Weaviate: An open-source vector database that stores both objects and vectors, combining vector search with structured filtering and built-in machine learning model integrations.

Data Modeling

  1. dbt core: A command-line tool that enables data analysts and engineers to transform data in their warehouses more effectively.
  2. GraphQL: A query language for APIs and a runtime for executing those queries with your existing data. It provides a complete and understandable description of the data in your API.
  3. SQL Mesh: An open-source tool for building and managing data transformations, with a focus on data modeling and lineage.

When choosing storage and data modeling tools, consider factors such as data volume, query patterns, integration with existing systems, scalability requirements, and the specific needs of your data team. For file formats, think about compression, schema evolution capabilities, and compatibility with your processing engines. For metadata layers, consider transaction support and real-time requirements. For data modeling tools, consider the complexity of your data relationships, the need for version control, and collaboration features.

It’s often beneficial to combine multiple tools. For example, you might use Parquet for base storage, Delta Lake for transaction support, and dbt for transformation and modeling. The key is to create a flexible, scalable data infrastructure that supports your current needs and can evolve with your organization’s data strategy.

Remember, the choice of storage tools can significantly impact query performance, data governance, and the overall efficiency of your data operations. It’s worth investing time in selecting the right combination of tools for your specific use case.

The Challenge of Choice

The open-source community has developed numerous solutions for various aspects of data handling, including: