Skip to content

Fix column misalignment when appending to an existing CSV - #19

Open
Benjapieres wants to merge 1 commit into
zohaibbashir:mainfrom
Benjapieres:fix/append-column-misalignment
Open

Fix column misalignment when appending to an existing CSV#19
Benjapieres wants to merge 1 commit into
zohaibbashir:mainfrom
Benjapieres:fix/append-column-misalignment

Conversation

@Benjapieres

Copy link
Copy Markdown

Problem

save_places_to_csv() drops every column whose values are all identical, so the output schema depends on the scraped data. On the --append path the header is only written for a new file, so a later run with a different set of surviving columns writes rows that don't line up with the header already on disk.

This is easy to hit in normal use: any batch where every business happens to share a field (all the same phone_number, all store_delivery = "No") drops that column, and the next --append run with varied data keeps it.

When the appended rows keep more columns than the header has, the file stops being parseable at all:

pandas.errors.ParserError: Error tokenizing data. C error: Expected 5 fields in line 4, saw 6

Reproduction

Run 1 — two places sharing a phone number, so phone_number is dropped and the header has 5 columns. Run 2 with --append — two places with different phone numbers, so the column survives and 6-field rows go under the 5-field header:

name,address,reviews_count,reviews_average,place_type
Alpha,1 St,,,Cafe
Beta,2 St,,,Deli
Gamma,3 St,+1-555-1111,,,Bar     <- 6 fields under a 5-field header
Delta,4 St,+1-555-2222,,,Pub

Phone numbers land under place_type, and the CSV can no longer be read back.

Fix

Appending now reads the existing header and reindexes onto it, so rows always conform to the file on disk and absent fields become empty cells. Columns present in the scrape but missing from the file are logged rather than silently shifting the row. Column-dropping stays on the fresh-write path, where it's safe and still does what the README advertises.

Same scenario after the fix:

name,address,place_type
Alpha,1 St,Cafe
Beta,2 St,Deli
Gamma,3 St,Bar
Delta,4 St,Pub

Second bug found while testing

The drop also ran on single-row frames, where every column is constant by definition — so it dropped all of them. The default -t 1 wrote a row with no columns:

$ python main.py -t 1
# result.csv -> '\n'

It's now skipped for frames under two rows.

Notes

Behavior is unchanged for the default (non-append) path with 2+ results, which is what most users hit. No dependency or Python-version changes are included here.

🤖 Generated with Claude Code

save_places_to_csv() drops every column whose values are all identical,
so the output schema depends on the scraped data. On the --append path
the header is only written for a new file, which means a later run with
a different set of surviving columns writes rows that do not line up
with the header already on disk.

When the appended rows keep more columns than the header has, the file
becomes unparseable rather than merely wrong:

    pandas.errors.ParserError: Error tokenizing data.
    C error: Expected 5 fields in line 4, saw 6

Appending now reads the existing header and reindexes onto it, so rows
always conform to the file on disk and absent fields become empty cells.
Columns present in the scrape but missing from the file are logged
instead of silently shifting the row.

Dropping uninformative columns is also skipped for single-row frames,
where every column is constant by definition. Previously the default
-t 1 wrote a row with no columns at all.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Kahtaf

Kahtaf commented Aug 14, 2026

Copy link
Copy Markdown

Reproduced both defects at c702488 (Python 3.12.3, Playwright 1.44.0, Ubuntu arm64)

Append: logged Saved 3 places, exit 0, but wrote three 11-column rows under the existing 9-column header (lines 12–14).

-t 1: logged Saved 1 places, wrote a 2-byte file.

Both come from the pre-write drop of every column with only one unique value, the schema ends up depending on the batch. At -t 1 every column has one unique value so they all get dropped; on append the new batch keeps a different column set than the file it's appending to. Either way the log still says success, so downstream picks up a corrupt CSV silently.

Full run + evidence: https://gittested.com/reviews/google-maps-scrapper/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants