lance-format /lance
Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
★ 7,324⑂ 944👁 49⑃ 7,587● 3,215▤ 698
+13
Daily
7,324
stars
944
forks
49
watchers
7,587
prs
3,215
issues
698
releases
README
The Open Lakehouse Format for Multimodal AI High-performance vector search, full-text search, random access, and feature engineering capabilities for the lakehouse. Compatible with Pandas, DuckDB, Polars, PyArrow, Ray, Spark, and more integrations on the way. Documentation • Community • Discord • Mailing List [CI]: https://github.com/lance-format/lance/actions/workflows/rust.yml [CI Badge]: https://github.com/lance-format/lance/actions/workflows/rust.yml/badge.svg [Docs]: https://lance.org [Docs Badge]: https://img.shields.io/badge/docs-passing-brightgreen [crates.io]: https://crates.io/crates/lance [crates.io badge]: https://img.shields.io/crates/v/lance.svg [Python versions]: https://pypi.org/project/pylance/ [Python versions badge]: https://img.shields.io/pypi/pyversions/pylance [![CI Badge]][CI] [![Docs Badge]][Docs] [![crates.io badge]][crates.io] [![Python versions badge]][Python versions] Lance is an open lakehouse format for multimodal AI. It contains a file format, table format, and catalog spec that allows you to build a complete lakehouse on top of object storage to power your AI workflows. Lance is perfect for: 1. Building search engines and feature stores with hybrid search capabilities. 2. Large-scale ML training requiring high performance IO and random access. 3. Storing, querying, and managing multimodal data including images, videos, audio, text, and embeddings. The key features of Lance include: - Expressive hybrid search: Combine vector similarity search, full-text search (BM25), and SQL analytics on the same dataset with accelerated secondary indices. - Lightning-fast random access: 100x faster than Parquet or Iceberg for random access without sacrificing scan performance. - Native multimodal data support: Store images, videos, audio, text, and embeddings in a single unified format with efficient blob encoding and lazy loading. - Data evolution: Efficiently add columns with backfilled values without full table rewrites, perfect for ML feature engineering. - Zero-copy versioning: Automatic versioning with ACID transactions, time travel, tags, and branches—no extra infrastructure needed. - Rich ecosystem integrations: Apache Arrow, Pandas, Polars, DuckDB, Apache Spark, Ray, Trino, Apache Flink, and open catalogs (Apache Polaris, Unity Catalog, Apache Gravitino). For more details, see the full Lance format specification. [!TIP] Lance is in active development and we welcome contributions. Please see our contributing guide for more information. File format stability Lance releases frequently because the SDKs, integrations, and performance work are moving quickly. This does not mean the Lance file format changes incompatibly in every release. The Lance file format is identified by the data_storage_version stored in each dataset, and stable storage versions are a long-term compatibility contract. - Once a dataset is written with a stable data_storage_version, future Lance releases will continue to support rea
Top issues
#5194Lance commits are progressively slower once a lot of fragments are written30 comments · 2025-11-07#4237Find a way to aware accurate fragment id in operation of transaction17 comments · 2025-07-15#4163Column Statistics14 comments · 2025-07-07#3461[Feature] lance support mosaicml streaming14 comments · 2025-02-19#7584Support HDFS as a storage backend13 comments · 2026-07-02#5130Zonemap index reading too many files13 comments · 2025-11-03#4918Bitmap index should support spilling13 comments · 2025-10-09#4772Should we provide a query interface from rowId to rowAddr to the non-Rust side?13 comments · 2025-09-19
Discussions
#10948Experimental listelement scalar index: indexing fields inside list<struct> elements0 comments · 2026-11-11#10935Proposal: 128-bit bitpacking for Decimal128 in Lance 2.33 comments · 2026-11-11#10801A stress harness for storage faults, process kills and concurrent commits: in-tree or a separate repo?0 comments · 2026-11-06#10599[VOTE] Release Candidate v14.0.3-rc.15 comments · 2026-11-02#10590Is a Vamana sub-index for IVF wanted in Lance?0 comments · 2026-11-02#10416[VOTE] Release Candidate v14.0.2-rc.13 comments · 2026-10-28#10395Expiring pins for ongoing work2 comments · 2026-10-28#10366[VOTE] Release Candidate v14.0.1-rc.14 comments · 2026-10-27
Latest release
v14.0.3 v14.0.3 2026-11-15What's Changed Bug Fixes 🐛 fix(blob): decode legacy blob pages in 2.2 files after promotion by @jackylee-ch in https://github.com/lance-format/lance/pull/10454 fix(blob): promote legacy blob input to v2 on fragment and update writes by @zhangyue19921010 in https://github.com/lance-format/lance/pull/10586 Full Changelog: https://github.com/lance-format/lance/compare/v14.0.2...v14.0.3