[Feature] Create Export Endpoint for CSV/JSON Data Downloads with Streaming

[KarenZita01]

Description

Users and integrators need the ability to export meter reading data, analytics summaries, and system reports in common data formats (CSV and JSON) for offline analysis, reporting, and integration with external tools. This issue creates export endpoints that generate downloadable files with proper content types, streaming for large datasets, and configurable column selection and filtering.

The export system must support: Format Options — CSV (with header row, configurable delimiter and quoting) and JSON (array of objects or newline-delimited JSON); Streaming — for large datasets (10,000+ records), responses should stream data using Node.js streams to avoid loading the entire dataset into memory; Column Selection — query parameter ?fields=col1,col2,col3 to select specific columns; Filtering — support the same filter parameters as list endpoints (date range, meter IDs, etc.) so users can export filtered subsets; Filename — Content-Disposition header with a meaningful filename based on the export type and date range (e.g., meter-readings-2026-01-01-to-2026-06-01.csv).

The CSV generation should use a streaming CSV serializer (e.g., csv-stringify from the csv package) that writes data row by row. The JSON export should support both pretty-printed JSON (for small datasets) and newline-delimited JSON (for streaming large datasets). All export endpoints should respect authentication (JWT or API key) and enforce appropriate scope permissions.

Technical Context & Impact

• Dependencies: csv-stringify (or csv package) for streaming CSV generation. No other new dependencies — Node.js built-in streams handle the rest.
• Architecture: New src/services/exporter.js with format-specific export functions. New src/routes/exports.js with endpoints under /api/exports/. The exporter service uses Node.js Readable streams and pipes to the HTTP response.
• Impact: Essential for data portability and offline analysis. Without this feature, users would need to paginate through list endpoints manually to gather data. Streaming support is critical for production scalability.

Step-by-Step Implementation Guide

1. Install CSV Dependency: Run npm install csv-stringify (or @fast-csv/format for faster streaming). Review the API — it should support stringify(records, options) returning a readable stream or transforming an input stream.
2. Create Export Service: Write src/services/exporter.js exporting: exportToCSV(dataStream, columns, options) — returns a transform stream that converts JSON objects to CSV rows; exportToJSON(dataStream, options) — returns a stream that outputs JSON array or NDJSON; generateFilename(type, startDate, endDate, format) — generates a descriptive filename.
3. Create Export Routes: Write src/routes/exports.js with GET /api/exports/readings — export meter readings with query params (format, startDate, endDate, meterIds, fields); GET /api/exports/analytics/{summaryType} — export analytics summaries; GET /api/exports/system-report — export system-wide report combining multiple data types. Apply authentication and validation middleware.
4. Implement Streaming Logic: For each export route, create a readable stream from the data source (using the repository pattern from Issue [Refactor] Implement Repository Pattern for Clean Data Access Layer Abstraction #22). Pipe through the CSV or JSON transform. Set proper headers: Content-Type (text/csv, application/json), Content-Disposition (attachment with filename).
5. Handle Error Cases: If the data source returns no records, return an empty file with appropriate headers rather than an error. If an invalid field is requested in ?fields=, return a 400 error listing invalid fields.
6. Write Tests: Create tests/unit/exporter.test.js testing CSV and JSON generation with various column selections and data types. Test streaming with large datasets. Create tests/integration/exports.test.js testing full HTTP response — verify headers, content type, and content.

Verification & Testing Steps

1. Call GET /api/exports/readings?format=csv with a valid JWT — expect a 200 response with Content-Type: text/csv and Content-Disposition: attachment. The response body should be well-formed CSV with headers.
2. Call the same endpoint with ?format=json — expect Content-Type: application/json and a JSON array of reading objects.
3. Call with ?fields=id,timestamp,value — verify the CSV/JSON only contains the specified columns.
4. Call with a large dataset (10,000+ records) — verify the response starts streaming immediately (check Transfer-Encoding: chunked header) and completes successfully without timeout.
5. Call with an invalid field name (?fields=nonexistent) — expect a 400 error listing nonexistent as invalid.
6. Import the downloaded CSV into Excel or Google Sheets — verify columns map correctly and data is parseable.