docs(hub): document refresh a single resource in triggers - #4324
docs(hub): document refresh a single resource in triggers#4324ShreyasGS wants to merge 1 commit into
Conversation
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
docs | 01f0ada | Commit Preview URL Branch Preview URL |
Aug 07 2026, 01:54 PM |
|
|
||
| ### Refresh a single resource | ||
|
|
||
| For a targeted rebuild, combine `refresh="always"` with `with_resources(*resources)` and drive the resource list from configuration so one job covers every table. The job below drops and reloads only the tables named at runtime: |
There was a problem hiding this comment.
refresh="always" does not affect the scoping here. the body sets pipeline.refresh = "drop_resources" and never reads run_context["refresh"].
what it does affect: start_run sees policy == "always" and clears prev_completed_run on this job and everything transitively downstream in the freshness graph (block nodes excluded). those jobs get run_context["refresh"] = True on their next run and their interval window restarts, so the effect reaches past the resources named here.
either remove refresh="always", or read run_context["refresh"] in the body and document the downstream effect.
|
|
||
| - **First run:** the job has never completed successfully, which is the natural moment for a backfill. | ||
| - **Explicit request:** you pass `--refresh` to `dlthub run`, `dlthub local run`, or `dlthub job trigger`. | ||
| - **Cascade:** an upstream job with `refresh="always"` (or a `--refresh` triggered on an upstream) propagates the signal through the freshness graph. `refresh="block"` on a job severs the cascade at that node. |
There was a problem hiding this comment.
block also ignores an explicit --refresh on the job itself, not just the cascade. start_run skips the clear and logs the request as ignored. add that to the bullet above.
also: the policy table says always originates on every run, but the cascade fires at run start, not on completion. a failed refresh run still clears prev_completed_run downstream.
| dlthub job trigger tag:refresh | ||
| ``` | ||
|
|
||
| See [doing a full or partial refresh](../../general-usage/incremental-loading.md#doing-a-full-or-partial-refresh) for `drop_data` vs `drop_resources` vs `drop_sources` semantics. Use `drop_data` when the schema is still correct and only the rows should be replaced. Use `drop_resources` when the schema changed and the table should be recreated. |
There was a problem hiding this comment.
add the drop_sources caveat here instead of only linking it: it ignores with_resources and drops the whole source.
| resources = ["inventory"] | ||
| ``` | ||
|
|
||
| The section key is `[jobs.<module>.<job>]`. Use `[jobs.<module>]` to configure every job in the module. Since configuration is per profile, `dev.config.toml` can point at a small table for fast local runs while `prod.config.toml` names the real one. |
There was a problem hiding this comment.
the middle segment is the module name, and the snippet above has no filename. in jobs.py the key would be [jobs.jobs.refresh_tables]. name the file in the snippet, or state that the segment is the module.
| Override on the CLI for a one-off: | ||
|
|
||
| ```sh | ||
| dlthub local run refresh_tables -c 'resources=["purchases"]' |
There was a problem hiding this comment.
-c only exists on dlthub local run, there is no equivalent on dlthub run. state that on the platform the resource list comes from the profile config.toml uploaded at deploy, and that this line is the local check before deploying.
| Then trigger every refresh job on the platform with a tag selector: | ||
|
|
||
| ```sh | ||
| dlthub job trigger tag:refresh |
There was a problem hiding this comment.
job trigger only matches deployed jobs and does not sync. add dlthub deploy before this.
| ``` | ||
| Above we tell `dlt` to truncate all tables belonging to resources in `data_source()` if the refresh signal got passed in the `refresh` flag. | ||
|
|
||
| ### Refresh a single resource |
There was a problem hiding this comment.
this uses expose={"tags": [...]} and job trigger tag:... before the section that introduces them. the cascade and explicit-request bullets also repeat the policy table and the dlthub run backfill --refresh line above. suggest keeping only the first-run bullet and moving this subsection below tags and bulk triggering.
| `run_context["refresh"]` is `True` in three cases: | ||
|
|
||
| - **First run:** the job has never completed successfully, which is the natural moment for a backfill. | ||
| - **Explicit request:** you pass `--refresh` to `dlthub run`, `dlthub local run`, or `dlthub job trigger`. |
There was a problem hiding this comment.
the first-run rule above is platform state. locally there is no completion history, refresh is resolved from --refresh and the declared policy only. say the list describes platform runs, and keep dlthub local run to the explicit case.
2 participants
Description
Adds a "Refresh a single resource" subsection under
hub/pipeline-operations/triggers.md#refresh-cascade, plus a short "three sources of refresh" bullet list to make signal origins explicit.Why
Users following the
refresh="always"pattern intriggers.mdhad no on-page guide for the common case of "rebuild only these named tables". They'd hitKeyError-shaped confusion or reach fordrop_sources(which wipes the whole source). The addition covers the missing patterns and links out to the canonical refresh-mode reference.What's new in
triggers.md#refresh-cascaderun_context["refresh"]: first run (never completed), explicit--refreshon CLI, or cascade from an upstreamrefresh="always"(withrefresh="block"severing).@run.pipeline(..., refresh="always")combined withpipeline.run(source.with_resources(*resources), ...)andpipeline.refresh = "drop_resources".[jobs.<module>.<job>]in.dlt/<profile>.config.toml, sodevandprodcan name different tables without a code change.dlthub local run refresh_tables -c 'resources=["..."]'for one-off runs.dlthub job trigger tag:refresh.drop_datavsdrop_resourcesvsdrop_sourcessemantics.