Sync with dbt labs main - #2
Merged
Merged
Conversation
This PR contains assorted updates related to `DataflowPlanBuilder`: * Rename from `FindSourceNodeRecipeParameterSet` to `FindSourceNodeRecipeInput`. * Rename from `for_group_by_source_node` -> `output_group_by_metric_instances`. * Simplify `FindSourceNodeRecipeInput` by removing fields that are no longer used.
To simplify classes, this PR removes `WhereFilterSpecSet` as it seems a tuple of `WhereFilterSpec` works as well.
…bt-labs#1978) This PR updates immutable classes so that instead of the mutable `PydanticMetricTimeWindow`, the classes use an immutable dataclass equivalent. This ensures immutability and also has performance benefits when those classes are used as keys (from better hashing performance).
…t-labs#1979) This PR renames `WhereConstraintNode` to `WhereFilterNode` for consistency with how we use the term `filter`. This also renames some other classes related to filters.
This PR updates the rendering of multiple `WHERE` filters to use the SQL model classes instead of joining with ` AND `. This results in a more readable `WHERE` clause in the rendered statement as it is rendered using default rules. It also simplifies later rendering improvements. e.g. Before: ``` WHERE ((metric_time__day >= '2020-01-02') AND (metric_time__day <= '2020-01-02')) AND (booking__ds__month > '2020-01-01') ``` After: ``` WHERE ( metric_time__day >= '2020-01-02' ) AND ( metric_time__day <= '2020-01-02' ) AND ( booking__ds__month > '2020-01-01' ) ```
The class `LinklessEntitySpec` is redundant with the addition of `EntityReference`, so this PR removes it. The removal of `LinklessEntitySpec` allows for the removal of custom hash / equality methods that make collision cases harder to understand.
As per title - please view by commit.
This PR adds a `.create()` initializer to simplify creation with `iterable` arguments.
This PR adds `MetricEvaluationPlan`, a graph that models the evaluation of metrics in a query. A query for multiple metrics is currently handled as a full-outer join of subqueries that compute each metric. Similarly, the query for a derived metric has subqueries that compute each of the input metrics. In the evaluation plan, the nodes are the individual queries that compute a specific set of metrics, and the edges point to nodes that are the input queries. The metric evaluation plan is later used in the `DataflowPlanBuilder` to guide construction of the dataflow plan. The revised `DataflowPlanBuilder` follows the dependencies in the evaluation plan instead of recursively following the metric definitions. This enables different planners to be used - one that follows the existing logic and another that makes optimizations to reduce the number of full-outer joins.
This PR adds a metric evaluation planner that creates the plan using DFS traversal of the metric definition. This mirrors the current approach used in the dataflow plan builder.
…bt-labs#1986) CTEs are generated from dataflow nodes that appear multiple times in the plan. This PR adds a cache to `ComputeMetricsBranchCombiner` so that the same output nodes are generated for the same input nodes, enabling correct CTE generation for plans that are processed by the combiner. Snapshot changes show additional CTEs that better simplify queries.
CTEs are currently generated for the largest common branches. This PR updates CTE generation to recursively inspect those branches for additional common sub-branches.
…abs#1989) This PR updates `DataflowPlanBuilder` to use the metric evaluation plan generated by `DepthFirstSearchMetricEvaluationPlanner`. This should match the order of evaluation in the current builder, so no snapshot changes are expected.
This PR updates the test for the metric evaluation planner to include the rendered SQL associated with the query in the test snapshots. The rendered SQL can help with snapshot review.
This PR adds PassThroughMetricEvaluationPlanner, which builds evaluation plans by passing input metrics through intermediate nodes when possible. This can reduce the number of full outer joins required for multi-metric queries.
…1994) This PR adds the `DataflowPlanOptimization.PASSTHROUGH_METRIC_EVALUATION` option so that the passthrough metric evaluation planner can be specified in a request and rolled out via a feature flag. The passthrough planner can reduce the number of full outer joins in some multi-metric queries.
This PR introduces `DataflowPlanOptionSet` to group the options used when building a dataflow plan. As part of that change, it removes the builder’s `_optimizations` instance variable. It also fixes a caching issue where the selected dataflow plan optimizations were not included in the cache key. That fix changes some generated IDs in test snapshots, but does not change query semantics.
# Why Our `CONTRIBUTING.md` lacked clear standards around contribution quality, which has led to increased review burden — PRs mixing concerns, missing tests, missing changelog entries, and interface changes arriving without prior discussion. Closes [DI-3578](https://dbtlabs.atlassian.net/browse/DI-3578). # What - **New section: "Before you write any code"** — requires contributors to open a GitHub issue and discuss with the Semantic Layer engineering team before implementing any interface change (APIs, schemas, CLI commands, etc.). PRs without prior discussion on interface changes will be declined. - **Lint is now a hard pre-PR requirement** — reframed from "run at any time" to a blocking requirement, with CI enforcement called out explicitly. - **Tests are required** — added explicit callout that PRs adding behavior or fixing bugs must include tests; reviewers will ask for them if missing. - **PR scope guidance strengthened** — added bold callout that a PR should do one thing, with concrete heuristic ("also" / "while I was in here" = signal to split); added guidance to use commits to tell the story of a PR. - **Rebase step clarified** — replaced vague "merge your changes into your fork" with explicit `git fetch upstream && git rebase upstream/main` instruction. - **Changelog timing fixed** — corrected "once your PR is created" to "before opening your PR". # Notes Paul Yang should do a final review and improvement pass per the Jira ticket. Drafted by Claude Sonnet 4.6 under the direction of @theyostalservice [DI-3578]: https://dbtlabs.atlassian.net/browse/DI-3578?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
…bt-labs#1988) This PR adds tests that demonstrate a bug in queries of an nested offset metric with a custom grain. This is a bug that existed prior to the FOJ optimization work. A PR to fix this issue is pending.
I recently updated my dev environment (e.g. updated package versions, changed how `hatch` was installed) and saw that there was an error running `make regenerate-test-snapshots`. The issue was due to nested `hatch` invocations: the script was run via `hatch` and then executed additional `hatch` commands targeting a different environment. This PR removes dependencies from the top-level script so that it can be run without `hatch`. Please view by commit.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This PR removes an unused helper in the dataflow plan builder.
…ic output (dbt-labs#2004) This PR fixes an issue in the dataflow plan builder where for time-offset derived metrics, the base grain of time dimensions at a custom grain were getting passed to the outputs even if not requested. The issue and fix are highlighted with `tests_metricflow/query_rendering/test_custom_granularity.py::test_nested_offsets_with_custom_grain`. In addition, the fix also resolves an issue with the test `tests_metricflow/query_rendering/test_custom_granularity.py::test_offset_to_grain_metric`. Please view by commit.
We're collapsing [DSI](https://github.com/dbt-labs/dbt-semantic-interfaces/) into MetricFlow. This is the final sync to do that, and in the coming weeks we'll archive the DSI repository.
In dbt-labs#2001 we did the final sync of DSI to MetricFlow. Crazy how things come full circle. In 2023 we moved DSI to its own repository, with dbt-labs#540 being the cut over. Now, basically 3 years later, DSI moves back to MetricFlow as metricflow-semantic-interfaces. To be clear this isn't a change of "oh we liked separate projects, and now want a monorepo", the separation was necessary for licensing purposes at that time. In an ideal world, these never would have been separate repositories. Now that the licensing conflict is gone, things get to come back together 🙂
…labs#2005) Removes `dbt-semantic-interfaces` from `dbt-metricflow` dependencies now that `metricflow-semantic-interfaces` has been migrated into the metricflow repo
This PR moves / renames `DirectoryPathAnchor` to the `toolkit` module to simplify importing from `toolkit`.
…abs#2008) This PR adds a helper that consolidates some methods required for time-offset metrics. There are plans to move more methods into the helper as `DataflowPlanBuilder` has grown to 1000's of lines. The methods are later used to restructure and fix filter issues in the `DataflowPlanBuilder`.
…t-labs#2007) This PR adds several cases that demonstrate the current issues with how filters are applied in queries for time-offset metrics. The tests include context on the cases, but in general, the issue is that filters on the aggregation time dimension / metric time are applied before the time offset is applied. This causes rows to be filtered out as the filter for `metric_time` describes the condition after the time offset.
This PR removes a few unused dependencies from `metricflow` and also shifts some test-related dpendencies.
This PR updates the import of graphviz to be local as it's an optional dependency and only needed for demo / development / debugging cases.
dbt-labs#1653 added a CI test for the `dbt-metricflow` package in a release branch, but it's missing from `main`. This PR adds it.
This PR removes the `graphviz` dependency from `dbt-metricflow` as it should be handled as an optional one.
…labs#2039) This PR updates the lint and test flow for `dbt-metricflow` to use the hatch environment for the package. Previously, the environment for `metricflow` was used, which could cause issues since `dbt-metricflow` depends on a specific of `metricflow`.
The release process for `metricflow` / `dbt-metricflow` has a number of steps: 1. Create a branch / PR that contains the change log, attribution, and updated `metricflow` version. 2. Create a branch / PR that contains the updated `metricflow` dev version. 3. Merge PRs from 1, 2 and push a git tag for the new `metricflow`. 4. Trigger the workflow for publish the package to PyPi. 5. Repeat 1-4 but for `dbt-metricflow` with the updated `metricflow` dependency. 6. Publish release notes at https://github.com/dbt-labs/metricflow/releases * Running the steps manually was tedious and error-prone. * To address this, this PR includes the first part of a CLI tool that automates this process. * The CLI tool handles all the steps in the terminal, except for approving the PRs (merge is handled by the tool) and approving the PyPi workflow to publish the package. * This tool was recently used for the last release, and it greatly simplified the process. * An API key is required to handle GitHub operations. * The tool writes state to a file so that subsequent steps (or a re-run) can act appropriately. * Downside is that the tool is complicated, especially with the interfaces to support tests. Arguably a little out of hand. * It was mostly written with the aid of AI. * If the first step looks good, I'll cut out the PRs for the other steps (6 others). * Please view by commit.
Decouples CI for metricflow and dbt-metricflow package. Required for independently bumping metricflow to support 3.14
Following up on dbt-labs#2040, this PR adds the remaining release tool steps. Please view by commit.
This PR updates the release tool with some refinements: * Adds a warning if a PR already exists and will be updated. * Adds a warning if a step was already run. * Moves the `--metricflow-repo` argument to before the command to make running subsequent steps easier. * Prints the banner message requiring user action as the last item shown.
This PR updates the mypy configuration so that different cache directories are used between metricflow and dbt-metricflow. The resolves some mypy-related cache errors that would occasionally appear due to the different requirements in the two packages.
This PR adds retries to `test_format_report_to_text_table` as it can be flaky due to different runtimes.
This PR updates the CLI to allow multiple `--where` arguments. This provides a UX improvement and can be used to push different predicates to different levels.
This PR updates DuckDB dependency in the development environment to `>= 1.5` as the Python 3.14 wheels for some older versions are not available. Without the wheel, CI checks for Python 3.14 are slow as the package needs to be built locally.
Release `metricflow` 0.211.0 with updated changelog and attribution.
Update `metricflow` version to 0.212.0.dev0 to reflect in-progress development.
) Release `dbt-metricflow` 0.13.0 with updated dependency `metricflow==0.211.0`. Release PR: dbt-labs#2044
Update `dbt-metricflow` version to 0.14.0.dev0 to reflect in-progress development.
…#2055) During development, the `dbt-metricflow` package is configured to depend on the `metricflow` package in the repo i.e. (`metricflow @ {root:parent:uri}`. In step 4 of the release, the dependency is changed to a published version of `metricflow`. To restore the dependency for development, this PR updates step 5 to add a commit to revert the change.
This PR removes old release helpers since the new release tool is ready to use.
The `fast_frozen_dataclass` decorator had a case for handling Python versions < 3.10. Now that the minimum Python version is 3.10, the case can be removed. Due to indentation changes, the diff looks more complex than it actually is.
Resolves dbt-labs#2066 This PR updates the rendering of user-supplied SQL in filter expressions to include parentheses to address operator precedence issues when multiple filters are provided. This fixes problematic rendering from: ``` WHERE booking__ds__day = '2020-01-01' OR booking__ds__day = '2020-01-02' AND booking__is_instant ``` to: ``` WHERE ( booking__ds__day = '2020-01-01' OR booking__ds__day = '2020-01-02' ) AND ( booking__is_instant ) ```
This PR updates the output column ordering classes to revised names and removes some validations / fallbacks that seemed excessive. For context, there are 3 ways that the output columns can be ordered: * Columns are grouped by the spec type, but columns in each group are in an arbitrary order that depends on how MF generates the SQL (current default, `LegacyTypeGroupedOrderer`). * Columns are grouped by the spec type, but columns in each group are in the same order as the inputs (planned new default, `TypeGroupedOrderer`). * Columns are in the order of the inputs (`InputOrderPreservingOrderer`).
To aid rollout, this PR adds an option to the query request to control whether the legacy column orderer should be used.
Keerthi-Kamarthi
marked this pull request as ready for review
June 16, 2026 14:44
andrewgazda
approved these changes
Jun 16, 2026
andrewgazda
left a comment
andrewgazda
left a comment
There was a problem hiding this comment.
Blindly approving this because obviously I'm not reading 1800 files, but let's make sure everything works still on staging before this goes to prod.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.