Skip to content

feat: add NDJSON and YAML exporters to ScrapeResult + extension-aware save() - #147

Open
sharyaropensource wants to merge 1 commit into
mldsveda:mainfrom
sharyaropensource:feat/ndjson-yaml-exporters
Open

feat: add NDJSON and YAML exporters to ScrapeResult + extension-aware save()#147
sharyaropensource wants to merge 1 commit into
mldsveda:mainfrom
sharyaropensource:feat/ndjson-yaml-exporters

Conversation

@sharyaropensource

Copy link
Copy Markdown

Fixes #145

Summary

Adds line-oriented / config-friendly exporters to ScrapeResult and extends save() to infer them from file extensions.

New methods

to_ndjson() -> str

  • Emits one JSON object per line over data (the NDJSON contract, no metadata/errors envelope)
  • Uses ensure_ascii=False so unicode is preserved literally
  • Empty data -> "" (matches to_csv empty guard)

to_yaml() -> str

  • Dumps the full {data, metadata, errors} envelope as YAML
  • Guarded by optional PyYAML import using the same ImportError pattern as to_dataframe
  • Install with pip install pyscrappy[yaml]

Extended save() mappings

Extension Method
.ndjson, .jsonl to_ndjson()
.yaml, .yml to_yaml()
.json, .csv, .md unchanged

Changes

  • src/pyscrappy/core/models.py: Added to_ndjson(), to_yaml(), updated save() and class docstring
  • tests/test_core/test_models.py: 13 new tests covering NDJSON round-trip, unicode, empty data, YAML round-trip, missing PyYAML error, and all new save() extensions

Test plan

  • All 33 tests pass (20 existing + 13 new)
  • NDJSON: round-trip parse each line, unicode preserved, empty data -> empty string
  • YAML: round-trip via yaml.safe_load, missing PyYAML raises ImportError
  • save(): .ndjson/.jsonl/.yaml/.yml all write correctly

… save()

- Add to_ndjson(): one JSON object per line, ensure_ascii=False, empty data → ''
- Add to_yaml(): optional PyYAML dep, same ImportError guard as to_dataframe()
- Extend save() to handle .ndjson/.jsonl → to_ndjson, .yaml/.yml → to_yaml
- Add tests: NDJSON round-trip, unicode, empty data, YAML round-trip,
  missing PyYAML error, save() extensions for all new formats

Fixes mldsveda#145
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ScrapeResult: add NDJSON (and YAML) exporters + extension-aware save()

1 participant