Skip to content

Rest duckdb source - #1514

Draft
ahuang11 wants to merge 16 commits into
mainfrom
rest_duckdb_source
Draft

Rest duckdb source#1514
ahuang11 wants to merge 16 commits into
mainfrom
rest_duckdb_source

Conversation

@ahuang11

@ahuang11 ahuang11 commented Dec 10, 2025

Copy link
Copy Markdown
Contributor

Builds upon #1518

from lumen.sources.rest_duckdb import RESTDuckDBSource

# Create a source with multiple REST tables
source = RESTDuckDBSource(
    uri=':memory:',
    tables={
        # https://mesonet.agron.iastate.edu/cgi-bin/request/daily.py?stations=AMW&sts=2019-01-01&ets=2019-01-31&format=csv
        'daily': {
            'url': 'https://mesonet.agron.iastate.edu/cgi-bin/request/daily.py',
            'url_params': {'stations': 'ABR', 'sts': '2025-12-08', 'ets': '2025-12-09', 'network': 'SD_ASOS', 'format': 'csv'},
        },
        # https://mesonet.agron.iastate.edu/cgi-bin/request/raob.py?station=KABR&sts=2025-12-08T15%3A49&ets=2025-12-09T15%3A49
        'raob': {
            'url': 'https://mesonet.agron.iastate.edu/cgi-bin/request/raob.py',
            'url_params': {'station': 'KABR', 'sts': '2025-12-08T15:49', 'ets': '2025-12-09T15:49', 'format': 'csv'},
        },
    }
)

source.get("daily", url_params={'stations': 'CMI', 'sts': '2025-12-08', 'ets': '2025-12-09', 'network': 'IL_ASOS', 'format': 'csv'})
print(source.render_table_url('daily', url_params={'stations': 'CMI', 'sts': '2025-12-08', 'ets': '2025-12-09', 'network': 'IL_ASOS', 'format': 'csv'}))
print(source.create_sql_expr_source(tables={"daily2": "SELECT * FROM daily LIMIT 5 OFFSET 1"}).get("daily2"))

@ahuang11
ahuang11 requested a review from philippjfr December 10, 2025 02:43
@codecov

codecov Bot commented Dec 10, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.68874% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.75%. Comparing base (fde1d77) to head (3a8fca0).
⚠️ Report is 258 commits behind head on main.

Files with missing lines Patch % Lines
lumen/sources/rest_duckdb.py 86.66% 12 Missing ⚠️
lumen/tests/sources/test_rest_duckdb.py 99.14% 2 Missing ⚠️
lumen/sources/duckdb.py 95.45% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1514      +/-   ##
==========================================
+ Coverage   62.11%   62.75%   +0.64%     
==========================================
  Files         150      152       +2     
  Lines       23001    23434     +433     
==========================================
+ Hits        14286    14706     +420     
- Misses       8715     8728      +13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ahuang11
ahuang11 changed the base branch from main to make_tables_persist December 10, 2025 22:24
@ahuang11 ahuang11 mentioned this pull request Dec 10, 2025
Base automatically changed from make_tables_persist to main December 11, 2025 08:30
@ahuang11
ahuang11 marked this pull request as draft March 24, 2026 02:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant