[MAINTENANCE] Teach integer batch parameters across the documentation - #12066
Draft
joshua-stauffer wants to merge 12 commits into
Draft
[MAINTENANCE] Teach integer batch parameters across the documentation#12066joshua-stauffer wants to merge 12 commits into
joshua-stauffer wants to merge 12 commits into
Conversation
Numeric batch parameters are moving to a single integer contract across every datasource family. This adds the shared contract the family boundaries will call, with no call sites yet: - is_digit_string: what counts as a digit-string (ASCII decimal only, so bools, signed, whitespace-padded and Unicode-digit forms are excluded) - normalize_batch_parameters: coerces digit-string values of declared numeric parameters to int, returning a new dict and never mutating its input. Nothing-coercible input returns the identical object with no warning, which keeps type-correct calls byte-for-byte unchanged. - batch_parameter_values_match: equality extended with int-to-digit-string numeric equivalence, so 4 and '04' denote the same partition while string-to-string comparison stays exact. - numeric_parameter_names_of: fail-closed lookup of a partitioner's declared numeric parameters. A kind that declares nothing is exempt, which is what keeps a string column literally named 'year' out of coercion by construction. Coercion emits exactly one GxDeprecationWarning, naming integers as the replacement and 2.0 as the removal target. GxDeprecationWarning subclasses UserWarning deliberately: a bare DeprecationWarning is suppressed by Python's default filters whenever the calling code lives in an imported module, which would hide the migration notice from precisely the users who need it. The warning is attributed to the first stack frame outside both this package and the standard library. Skipping only this package's frames would attribute to functools.py on the checkpoint path, which both misreports the location and poisons that module's warning registry so later occurrences are swallowed process-wide. Naming the coerced keys without their values keeps the message textually stable, so the interpreter's registry collapses repeated usages within one run into a single notice.
Classification of a "numeric batch parameter" follows the partitioner's kind, never the parameter's key name. Each numeric kind now declares its own parameter names: the yearly/monthly/daily file partitioners, the six numeric SQL partitioners, and the three dataframe partitioners. Key-name classification would be actively wrong. A column-value partitioner names its parameter after the column, so a string column literally named "year" produces a "year" key whose value is legitimately non-numeric. Declaring by kind means that column is exempt by construction rather than by a special case that a future edit could forget. Exemption is therefore the default and the declarations are deliberately absent from the whole-path, column-value, multi-column-value, and converted-datetime kinds. Each declaration sits on the concrete class rather than a shared base, because the single-column base is shared between numeric and exempt kinds and a declaration there would silently un-exempt them. Every declaration derives from the class's own param_names rather than restating it, so the two cannot drift apart.
The two file matching sites compared request values against regex captures
with raw equality. Captures are zero-padded exactly as the filename spells
them ('04'), so an integer request value could never match one.
Both sites now compare through the shared equivalence helper, so 4 and '04'
denote the same partition. Rendering the integer back to a padded string was
rejected as the alternative: it is lossy for months 1-9 and ambiguous under a
variable-width regex, and it would have required the captures themselves to
change, churning batch identifiers for workflows that work today.
Applying the tolerance unconditionally is safe because both sites only ever
compare against regex captures, which are always strings. String-to-string
pairs therefore keep today's exact equality on the fast path, so '04' still
does not match '4'. Threading the partitioner through the batch filter to
scope the tolerance more narrowly would have been signature churn with no
behavioral difference.
The None-wildcard short-circuit and all sorting paths are unchanged; ordering
reads captured values, never request values.
The marker check pairs each deprecation warning with a 'deprecated-v' comment by counting occurrences of the literal name. Two things defeat that count once a deprecation category lives in this package rather than being the builtin. A category defined here has to be imported before it can be raised, and the import line reads as a second occurrence, demanding a second marker for a single deprecation. The count now ignores import lines, so a marker still pairs with an emission. The module that declares the categories is excluded outright: it raises none of them, and its own docstring names the builtin while explaining why the category deliberately subclasses UserWarning instead. The check still fails when an emission genuinely lacks its marker.
File-based assets required strings for numeric batch parameters and raised InvalidBatchRequestError on an integer, while SQL assets required integers for the same keys. One checkpoint spanning both families was therefore unreachable from a single batch_parameters dict. Integers are now accepted for numeric parameters, and digit-strings are coerced at this boundary with the deprecation warning. Which parameters count as numeric is the intersection of what the partitioner declares and what the regex actually captures. Both sides are load-bearing: the declared names are the authoritative universe of request keys, while group names alone would over-reach onto unnamed groups and the injected path, and the declared names alone would reach parameters the regex never captures. Booleans stay rejected even though Python treats them as integers, since a boolean is never a meaningful year or month. The existing skip of falsy values is preserved exactly as it was: 0, False and the empty string bypass the type check today, and tightening that here would be a behavior change unrelated to the typed contract. Coercion applies to the values used for matching only. Selected batches keep their zero-padded string captures in identifiers and metadata, so batch IDs and ordering are unchanged for requests that work today. The two tests that pinned integer rejection now pin the integer contract. Selection is exercised across fixed- and variable-width regexes with a single-digit month, which is what distinguishes numeric equivalence from rendering the integer back to a padded string -- the latter passes December and fails months one through nine.
SQL assets take integers for numeric batch parameters because their candidate values come from the database as integers. A string matched nothing and surfaced as a bare no-available-batches error, which reads as absent data. Digit-strings are now coerced at the SQL request boundary with the deprecation warning, so they select the numerically equivalent batches. Integer requests are untouched. Classification runs on the resolved partitioner implementation rather than the requested partitioner: the requested kinds carry no parameter names at all, and resolution is also what applies a dialect's overrides, which is how the sqlite converted-datetime kind stays exempt. The matcher itself keeps exact equality. Requests now arrive carrying integers and candidates are integers, so tolerance there would be dead code -- and its absence is what guarantees a column-value partitioner over a column named 'year' never starts matching '01' against 1. The stringification of date-typed candidates is likewise untouched, since it is what makes string input work for those columns today. Resolution stays behind the same guard as the rest of the method: with no values to coerce there is nothing to classify, so building a request does not become the step that reports an unimplemented partitioner.
A no-match on a SQL asset raised a bare "No available batches found.", which reads as absent data. A malformed parameter produced the identical message, so the common case of a typo sent engineers to debug healthy pipelines. The exception gains an optional message; constructed bare it still produces the original text, so existing raise sites and catchers are unaffected. The class and the raise site are unchanged. Three situations are now distinguishable. An empty table or column says no candidate batches exist at all. Candidates that exist but do not match report how many were checked and against which options. When a numeric parameter carries a string that cannot be read as an integer, that is very likely the real explanation, so it is named on top with the parameter and its value. The diagnostics are composed from the candidate list the matching pass already walks. That pass queries the database, so the raise site consumes what was captured rather than asking again. Nothing raises earlier than before: a request carrying an uninterpretable value still builds, and the error surfaces only when batches are fetched.
Directory assets have no matching step: batch parameters flow straight into batch identifiers and on into the downstream date-part comparisons. A string value therefore produced an empty batch with no error and no warning at all, which is the least diagnosable of the three families' failure modes. Digit-strings are now coerced at the directory asset boundary with the deprecation warning, so they select the numerically equivalent batches. Integer parameters are unchanged. The path option these assets inject into batch parameters is never coerced and never warned on, and neither is the partitioner's column name. Neither is a declared numeric parameter, so classification excludes both by construction rather than by a special case that a later edit could forget. Identifiers for parameters that previously arrived as strings now hold integers, which changes their batch IDs. Those calls never produced a batch, so no working workflow's identifiers move.
The warning relied on Python's per-module warning registry to avoid repeating itself. That registry is discarded whenever any code mutates the global warning filters, and pandas does exactly that while materializing a DataFrame. So in a checkpoint holding a file-based and a SQL validation definition, the same warning fired once per definition, and whether it did depended on the order the definitions happened to run in. A checkpoint with many definitions would have printed the same line many times, which is the noise the warning was designed to avoid. Emissions are now tracked here, keyed by the message together with the user code location the warning is attributed to. Each distinct place in a user's code that needs migrating is reported exactly once, however often it runs, and unrelated filter changes elsewhere in the process cannot affect it. The stack walk that finds the user frame now returns that frame's location alongside the stacklevel, so the key costs no extra walk. Attribution and the emitted category are unchanged. The tracking outlives a single call by design, so the suite resets it between tests. Without that, parametrized cases sharing a source line would share a key and only the first would warn.
This is the workflow the typed contract exists to make reachable. A checkpoint applies one batch parameter dict unchanged to every validation definition it holds, so while file assets required strings and SQL assets required integers, no dict satisfied both and the checkpoint could not be built at all. One context holds a pandas filesystem source and a sqlite file under a temporary path, with one validation definition each, driven by a single all-integer dict. The assertion is that exactly two results come back and both succeeded. The failure being replaced is silent absence rather than an error, so a test that only checked for a raised exception, or only that the results present had succeeded, would pass against the broken behavior. The month is single-digit deliberately. Rendering an integer back to a padded string would satisfy December and fail every month from January to September, so a test using month twelve would report success for an implementation that is wrong most of the year. The digit-string run asserts a single warning across both definitions, with the file-based definition first: that ordering is what puts a pandas batch load between the two coercion sites, which is the case that previously produced a duplicate.
The example already showed an integer, which stays canonical. It now also records that digit strings still work, that they warn, and that support for them ends in 2.0, so the migration target is visible where the type is documented rather than only in the warning a user has already triggered.
The examples taught that numeric batch parameters take strings for file-based assets, which was true only of that family and is no longer the contract. Every teaching surface now passes integers uniformly, and the prose telling readers the parameter format depends on the asset type is gone, because it no longer does. The change is parameter typing only. Where a snippet asserts the options it built, the expectation moves with the values so the snippet stays internally consistent. The older connection guides are updated for typing alone; other ways they have aged are left as they were rather than modernized here.
✅ Deploy Preview for niobium-lead-7998 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates every documentation surface that teaches batch parameters — the runnable examples
registered in the docs-tests gate, the non-registered snippets, and the older filesystem
connection guides — to pass integers for numeric batch parameters uniformly across file,
SQL, and directory sources. The prose telling readers that the accepted type depends on
the asset family is removed, because it no longer does.
This PR must merge only with the release that ships the integer contract
Expected 1.21.0. Please do not merge it before that release.
Published documentation builds from released lines. The contract these pages teach is not
released yet: on the currently shipped version, passing an integer to a file-based batch
parameter raises
InvalidBatchRequestError, while the string forms these pages currentlyteach keep working — with a deprecation warning — throughout the transition window.
Merging ahead of the release would hand readers guidance that fails on the version they
have installed. Published guidance must never precede released behavior.
This branch was forked from the head of the feature branch with
developas its base, soonce the feature merges, this diff collapses to a documentation-only change and needs no
rebase and no force-push.
Scope
Parameter typing only. The older connection guides under
oss/guides/have aged in otherways — several still show pre-1.0 call shapes — and those are deliberately left alone
rather than modernized here, so this diff stays reviewable as one mechanical change.
Where a snippet asserts the options it built, the expected value moves with the supplied
value, so each snippet stays internally consistent.
Verification
this was forked from.
internal consistency; parameter values and any dependent assertions were updated
together.
ruff checkandruff format --checkclean on every changed Python file.docs/docusaurus/;versioned_docs/is untouched.