ParquetDB: A Lightweight Python Parquet-Based Database

A customer hands you a directory of JSON exports. Your evaluation job needs a few fields from every record, but each run parses the whole export into Python objects before doing useful work. ParquetDB addresses this kind of workflow: it provides create, read, update and delete operations over a directory of Parquet files, using PyArrow to represent and process the data in memory.

The distinction between those layers matters. Parquet supplies the columnar file format. Arrow supplies typed arrays and processing machinery. ParquetDB supplies a Python interface and manages tasks such as assigning record IDs, aligning schemas and rewriting files after changes. You can work with evolving, nested records without first running a separate database server.

The paper compares ParquetDB with SQLite and MongoDB, then applies it to a large materials dataset. Its strongest lessons concern access patterns. Reading selected columns and handling large batches can suit this design. Frequent small updates incur file-management costs, and the indexed alternatives win the paper's point-lookup comparison.

Read this as a study of a useful storage tradeoff. Before using it for application state, examine recovery and concurrent access as carefully as read speed. The authors explicitly describe manual recovery after a crash or power failure.

An FDE often owns the intermediate data between a customer's systems and an AI application: evaluation cases, enriched document metadata, feature snapshots or offline results. That data may be too large to reparse comfortably on every run, yet small enough for one machine. A portable directory can make an experiment easier to hand over, provided the recipient also has its schema and provenance.

ParquetDB makes the cost of that convenience visible. Adding a field may require rewriting existing data. Updating one record may touch much more than that record. A read can be inexpensive because it selects a narrow column set, even when another query against the same dataset is slow.

Start with the customer's operations: which fields each job reads, how often records change, how many processes write, and what an acknowledged write must survive. For offline evaluation snapshots, this paper offers a useful design to investigate. For a live agent's payment status or approval queue, the recovery requirements should determine the storage choice first. The Parquet interview question is a good introduction if columnar storage is unfamiliar.