fix(etl): count documents, not rows, in joining_props - #319
Open
grzelaka-roche wants to merge 1 commit into
Open
grzelaka-roche wants to merge 1 commit into
grzelaka-roche wants to merge 1 commit into
Conversation
A joining index is read at its intermediate step, before its own translate_final flattens it to one row per document, so a document linked to several parents is still one row per link at that point. `fn: count` counted those rows: an expression file linked to 23 samples added 23 to the study's data_file_count instead of 1. Count over the key of the joining index instead, and deduplicate the projection on that key so `list` and `sum` also see each document once. Both are skipped when the frame does not carry that key, falling back to the previous row count rather than silently merging distinct documents. Also in the same path: - drop the re-join of a key-only frame onto the aggregates, which multiplied every row of the joined index by its document count - a group with no matching document now reads 0 instead of null. This is visible in Elasticsearch: guppy sees 0 where the field used to be missing - build the aggregation expressions from the props that were actually projected, so a joining_props block mixing props with and without `fn` no longer builds agg(None)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A joining index is read at its intermediate step, before its own translate_final flattens it to one row per document, so a document linked to several parents is still one row per link at that point.
fn: countcounted those rowsCount over the key of the joining index instead, and deduplicate the projection on that key so
listandsumalso see each document once. Both are skipped when the frame does not carry that key, falling back to the previous row count rather than silently merging distinct documents.Also in the same path:
fnno longer builds agg(None)Link to JIRA ticket if there is one:
New Features
Breaking Changes
Bug Fixes
Improvements
Dependency updates
Deployment changes