Fix column misalignment when appending to an existing CSV - #19
Open
Benjapieres wants to merge 1 commit into
Open
Fix column misalignment when appending to an existing CSV#19Benjapieres wants to merge 1 commit into
Benjapieres wants to merge 1 commit into
Conversation
save_places_to_csv() drops every column whose values are all identical,
so the output schema depends on the scraped data. On the --append path
the header is only written for a new file, which means a later run with
a different set of surviving columns writes rows that do not line up
with the header already on disk.
When the appended rows keep more columns than the header has, the file
becomes unparseable rather than merely wrong:
pandas.errors.ParserError: Error tokenizing data.
C error: Expected 5 fields in line 4, saw 6
Appending now reads the existing header and reindexes onto it, so rows
always conform to the file on disk and absent fields become empty cells.
Columns present in the scrape but missing from the file are logged
instead of silently shifting the row.
Dropping uninformative columns is also skipped for single-row frames,
where every column is constant by definition. Previously the default
-t 1 wrote a row with no columns at all.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 14, 2026
|
Reproduced both defects at Append: logged
Both come from the pre-write drop of every column with only one unique value, the schema ends up depending on the batch. At Full run + evidence: https://gittested.com/reviews/google-maps-scrapper/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
save_places_to_csv()drops every column whose values are all identical, so the output schema depends on the scraped data. On the--appendpath the header is only written for a new file, so a later run with a different set of surviving columns writes rows that don't line up with the header already on disk.This is easy to hit in normal use: any batch where every business happens to share a field (all the same
phone_number, allstore_delivery = "No") drops that column, and the next--appendrun with varied data keeps it.When the appended rows keep more columns than the header has, the file stops being parseable at all:
Reproduction
Run 1 — two places sharing a phone number, so
phone_numberis dropped and the header has 5 columns. Run 2 with--append— two places with different phone numbers, so the column survives and 6-field rows go under the 5-field header:Phone numbers land under
place_type, and the CSV can no longer be read back.Fix
Appending now reads the existing header and reindexes onto it, so rows always conform to the file on disk and absent fields become empty cells. Columns present in the scrape but missing from the file are logged rather than silently shifting the row. Column-dropping stays on the fresh-write path, where it's safe and still does what the README advertises.
Same scenario after the fix:
Second bug found while testing
The drop also ran on single-row frames, where every column is constant by definition — so it dropped all of them. The default
-t 1wrote a row with no columns:It's now skipped for frames under two rows.
Notes
Behavior is unchanged for the default (non-append) path with 2+ results, which is what most users hit. No dependency or Python-version changes are included here.
🤖 Generated with Claude Code