Sortby bug - #2620
Draft
norberttech wants to merge 31 commits into
Draft
Conversation
- write whole Rows in FilesystemBuckets and FloeSerializer - fail loudly on bucket append schema mismatch - replace FloeStreamWriter::unionSchema with Rows::schema - pad missing selected columns with null_entry, not str_entry - drop Row/Rows schema memoization, keep it lazy - repurpose serializer batch size tests to the read side
- assert every written value against its column type, before any bytes - drop DynamicEncoder - mixed, union and structure extras are no longer writable - format v2: uuid payloads length-prefixed, v1 files unsupported - dispatch encoders and decoders on Definition instead of Type - aggregates take their result type from the column, not the value - prune group-by spill columns without null-filling them
…values - divide() always returns float - float or numeric-string operand forces a float result - sum() stops narrowing a whole float sum back to int - extract RunningSum, shared by Sum and SumAccumulator - landing divide example uses equals() instead of same()
…torage - file sources and sinks take a Filesystem, guarded against their Path - mount table leaves Config and FlowContext - spilling algorithms take a BucketsStorage instead of a filesystem - every governed operation can override its algorithm - FilesystemStreams replaced by a per-sink FilesSink with two endings - save mode moves from DataFrame onto FileLoader
- Extractor gains schema() and withSchema(); all 35 implement both - extract() hydrates every batch against that schema, so one source yields one shape - Cache gains schema(), stored beside the rows it belongs to - move TypeMerge to flow-php/types as TypeWidener, drop its inference-only widening flag - TypeDetector never returns a type that rejects its own value - per-extractor metadata columns replace Config::putInputIntoRows()
- sortBy() accepts a bare column name, like groupBy() - Parquet/Memory schema() idempotent and side-effect-free, no memo into the declared-schema property - Parquet/Floe extract() hydrates against schema()'s promise, keeping _input_file_uri under a declared schema - FilesExtractor no longer reports extension "0" as null - GoogleSheetExtractor uses the shared MetadataColumns trait - http.md documents response_body/request_body as raw text
- FunctionTree contract (children/withChildren/resolved) + returns(): Type on all 134 scalar functions; ReferenceResolver resolves ref() leaves against a Schema, assertResolved() gates unresolved trees - EntryReference split into immutable UnresolvedReference / ResolvedReference; as()/asc()/desc() return copies; 22 nominal positions widened to Reference - ExecutionMode::LENIENT removed: all 191 invalidResult() sites throw, optional() is the only leniency door; Coalesce/OnEach stop swallowing errors, Exists keeps its catch deliberately - 13 argument-dependent functions constrained (Round, ArraySort, Regex flags, CallUserFunc required return type, Uuid, DOM siblings, ArrayGet wildcard rejection, Cast constructor-resolved target, json_pretty removed) - flow-php/types: PromotingUnifier/StrictUnifier over one ContainerUnification, Nullability + type_bare(), OptionalType double-wrap collapses; nullability declared as top-level OptionalType (54 nullable / 80 NOT NULL, gate-tested) - ScalarFunctionChain abstract class converted to trait; doc tooling re-pointed at the trait, dsl/api manifests regenerated with zero chain flips
- comparisons dispatch on values, propagate NULL and assert comparability at bind; 25 predicates return ?bool, filter() drops NULL rows like SQL WHERE - Parameter arms throw on malformed values, propagate NULL, validate once per call - declaration/value mismatches closed: ToDateTime false->null, XPath elements only, JsonDecode throws on scalar JSON, DOMElementParent root parent->null - shared ScalarUnification core extracted under both type unifiers - CallUserFunc ctor aligned to (callable, returnType, parameters); unreachable aggregate catches removed - fixture map carries expected types so the meta-test asserts the exact declared type of all 134 functions
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 1.x #2620 +/- ##
=============================================
- Coverage 86.51% 73.35% -13.17%
- Complexity 23323 28382 +5059
=============================================
Files 1837 1968 +131
Lines 72461 84835 +12374
=============================================
- Hits 62688 62227 -461
- Misses 9773 22608 +12835
🚀 New features to boost your workflow:
|
- AggregatingFunction: result(EntryFactory) replaced by outputName()/value()/returns(); WindowFunction gains returns(); both extend FunctionTree - consumers bind before rows flow: scalar transformers per batch, GroupBy and WindowProcessor once per run; produced values are cast to the declared Definition - filter()/until()/branch() reject non-boolean predicates at bind; unknown partitionBy()/orderBy() refs refused instead of collapsing partitions - Window and all window functions are immutable; over() returns a copy - Reference extends ScalarFunction; resolver looks up aliased refs by source column - array_expand/array_values/on_each derive structure operand types via field unification; ScalarResult deleted
- StructureType holds list<StructureElement>, replacing the required/optional buckets - declared field order is part of the type: equality, widening and unification are positional - structure_v2 wire shape, in PHP and in the Rust extension - type_structure() accepts structure_element() markers; the optional_elements argument is gone - new flow-php/mago-types-bridge derives structure shapes for the Mago analyzer - Schema::conformOrderTo() reorders a schema to an authority, used when resuming a Floe stream
- delete Entry and its 19 implementations; Row holds name-keyed values - Rows carries one Schema; merge() rejects disagreeing batches - schema owns column order, rows are never rekeyed - partition columns declared per dataset, nullable where a path lacks them - native hydrator folds column metadata; extension 0.3.0 - replace 31 *_entry() DSL functions with *_schema() declarations
- remove DataFrame::map(), to_callable() and the Callback transformers/loader - narrow call() and CallUserFunc to ScalarFunction, dropping the callable arm - reject Closure values in lit() and TypeDetector - replace AddRowIndex and collectRefs closures with named transformers - carry the callable as CallUserFunc's first child so refs resolve at plan time - drop the map() docs page and regenerate the DSL completers
- group keys keep their input nullability instead of being widened - a key absent under NOT NULL refuses; under a nullable column it is null - key extraction loses its null-on-missing flag, one rule for joins and grouping - a row omitting a nullable join key is unmatched rather than refused - resolving any multi-member union throws the domain exception, null|T is ?T
- partitioning becomes a loader concern: PartitioningLoader, Partitioning, PartitionRouter, RowPartitions; partition columns stripped from file bodies - RepartitionProcessor replaces PartitioningProcessor, making window-partition contiguity a guarantee instead of an accident - Parquet and Floe discover Hive partitions on read; schema() reads one file instead of every file, with unionByName() as the opt-in - Rows drops its Partitions object, and DropPartitionsTransformer with it - Floe stops carrying a partitions frame and footer table; byte-level golden fixtures pin the format - Partition values become nullable, spelled __HIVE_DEFAULT_PARTITION__ on disk
- Rows::__construct conforms every row to the schema and throws SchemaMismatchException naming the offending row - Rows::trusted() bypasses the check for engine paths that produced the schema and the rows together - reorderEntries(), RowPadding and FieldOrderConformer deleted - the gate now orders columns and fills nullable gaps - RowProjection renamed to RowProjector, with Row::project() reshaping without validating
- nine file extractors take the tail from one FileColumns - partition values typed once per file, not once per row - partition_types() stops being a no-op in seven extractors - Type::cast() refuses what it used to coerce, Rust mirrored - a Hive null partition reads back as null, not '' - extract() releases every reader it opens
…emselves - extractors derive their schema from a memoised zero-row probe instead of throwing - Client::describe() reports a query's result columns (BC break on the interface) - ResultCaster gains array, oid and timetz arms; arrays keep their SQL NULLs - duplicate output names cast by name, last wins, instead of by position - TimeType::cast() accepts HH:MM:SS clock time
- derive the schema from the query result, memoised per read - thread it into extract(); an undescribable query no longer reads - describe by preparing on MySQL, by a zero-row probe on PostgreSQL and SQLite - map native driver types per engine; SQLite floors to nullable string - stop leaking LIMIT/OFFSET onto the caller's query builder
- bounded sample, type fold and floor behind infer_schema() - candidate type set instead of an all-strings flag - unionByName decides the column set across files - string ladder: numerics before dates, one shared date_parse() - BC: digit and month-precision strings no longer infer as dates
- wrap a refused value in SchemaMismatchException, PHP and native alike - keep exceptions the types package did not raise unwrapped
…duces - schema() answers before extract() runs, one schema per run - inferSchema(infer_schema()->sampleSize()->filesToSniff()->types()->allStrings()->unionByName()) - diverging column sets throw, naming both sources and the remedy - close every stream on STOP, limit and hydrator throw - CSVReadOptions, CSVDialect, CSVHeader, CSVSourceOpener, CSVOpenSource, CSVFileReader - FileColumns gains tail() and withoutTail(), so partition columns are never typed from the body
…roduce - schema() answers from a bounded sample instead of throwing - every batch of a run carries that one schema - one JsonFileReader serves both extractors, selected by JsonFormat - inferSchema(infer_schema()->...) exposes the sampling knobs - 0-byte documents are skipped and streams close on every exit path - a row-less first source no longer freezes an empty column set
- schema() answers from a bounded sample instead of throwing; every batch carries that one schema - inferSchema(infer_schema()...) knob, and a diverging header now throws - read split into WorkbookReader/WorkbookSheet/WorkbookSampler, one fresh reader per open - text narrows to uuid, json and timezone; cells a workbook can type stay as typed - to_excel() writes real date cells, so the format options are now Excel number formats
…ma it produces - schema() answers from a bounded sample instead of throwing; every batch carries that one schema - inferSchema(infer_schema()...) knob; a header that changed since the sample throws - read split into GoogleSheetReadOptions/Reader/Sampler, the sampler implements core's SchemaSampler - empty cells decode to null by default; withEmptyToNull(false) restores "" - samples 100 rows, and a sheet the sample read in full is not fetched again - fix(flow-php/etl): a row-less read keeps the source schema instead of re-running the pipeline
- Hydrator loses cast(); hydrate(array, Schema) is the only door and Schema is mandatory - array_to_row/array_to_rows take Schema as parameter 2, required - Type::cast() refuses integer overflow, wall-clock date strings, null into string and scalar into a container; temporal to int/float now emits seconds - inference survives only in ScalarFunctionTransformer's array_unpack branch; pivot and on_each declare their output through PivotSchema and OnEachElementSchema - Floe stops storing the datetime class and refuses a null on a NOT NULL column - unifyAll() reads a bare NullType as nullable, fixing coalesce/greatest/least
…produce - resolve schema() through the inference fold instead of Schema::merge per row, memoised - describe a one-shot iterable by spilling it, folding exactly, then replaying; ::nonRewindable() has no producers left - from_array() takes a filesystem and a spill root; all three extractors take inferSchema() - BC: every inferred column is nullable and [] floors to json - keep enums out of the type floor, make a null structure element nullable, and widen duplicateRow()'s output schema - silence the postgresql-bundle test kernel's logger so a deliberate 500 stops printing [critical] into the suite
- remove DataFrame::autoCast(), AutoCastTransformer and Flow\Types\Type\AutoCaster - extract Flow\ETL\Extractor\InfersSchema over the eight extractors declaring inferSchema() - swap the CLI --schema-auto-cast flag for --schema-sample-size/-files-to-sniff/-all-strings/-union-by-name - bind partition filters against the extractor's schema instead of guessing per path - keep TypeDetector's numeric widening inside containers: [33, 65.5] is list<float> - make isIn() agree with equals() on numeric operands and refuse incomparable elements at bind
…g the pipeline - bind() on Transformer and Processor, walked by PlanBinder - schema() and printSchema() are no longer triggers - extractors declare the schema they produce and their repeatability - pivot values, join and group-by shapes resolved at bind time - SimilarNames suggests columns only within a distance threshold
- DataFrameExtractor answers from the wrapped frame's bind, reading no row - FeedExtractor is seeded with a required constructor Schema, so a nested transformation binds at the first fed batch instead of at row time - delete PipelineExtractor, from_pipeline(), GeneratorExtractor, Schema::fromPipeline() and SchemaNotDerivableException::pipeline(), which reaches zero producers - guard Pipeline::bind() and Pipeline::process() against re-entrancy, so a plan that reads back from itself throws instead of segfaulting - re-pin dtolnay/rust-toolchain and correct setup-chrome's version comment
norberttech
force-pushed
the
sortby-bug
branch
from
September 7, 2026 19:44
df271d7 to
cbbcc85
Compare
- a global aggregate over zero rows emits one row of typed defaults - pivot resolves its aggregate before asking it to describe itself, so min/max/first/last/collect work - pivot execution moves off GroupBy into PivotShape, PivotedTable and PivotAggregation - SkipRows skips the failed batch instead of emitting it half transformed - until() stops at the first row failing its predicate
- restructure the example corpus and fix every broken example - execute the whole corpus through the WASM playground in a browser - add dbal/postgresql reading and writing examples, pagination and query extraction - verify documentation fences call only symbols that exist - drop the CLI example runner and its committed output baselines - fix Parquet FLOAT32 decode losing precision
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.