Skip to content

Release 1.0.2 with the drop-empty fix#59

Draft
clayote wants to merge 5 commits into
lllangWV:mainfrom
clayote:1.0.2
Draft

Release 1.0.2 with the drop-empty fix#59
clayote wants to merge 5 commits into
lllangWV:mainfrom
clayote:1.0.2

Conversation

@clayote

@clayote clayote commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

Though I was able to work around bug #55 in 1.0.1 like this:

import pyarrow as pa
from parquetdb import ParquetDB
from pyarrow import compute as pc

db = ParquetDB('parquetdb')
if db.is_empty():
    return
unwanted = pa.table([[]], schema=pa.schema([("id", pa.int64())]))
for datum in data:
    exprs: list[pc.Expression] = []
    for k, v in datum.items():
        expr = pc.field(k) == pc.scalar(v)
        exprs.append(expr)
    selected = db.read(filters=exprs, columns=["id"])
    unwanted = pa.concat_tables([unwanted, selected])
if unwanted.num_rows == db.n_rows:
    db.drop_dataset()
else:
    db.delete(ids=unwanted["id"])

This does seem like an unnecessary piece of boilerplate, and might affect performance in apps that delete data often.

So I've made a fork of ParquetDB that applies the real fix, #55, and only that fix. This PR points to that fork, but I don't think you want to merge it, as such, since you've done significant refactoring after 1.0.1.

Please clone my branch 1.0.2 and do a release.

Fixes #58.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release 1.0.2

1 participant