Skip to content

Add support for user-defined macro functions with arbitrary expressions#4318

Open
hazefully wants to merge 13 commits into
FoundationDB:mainfrom
hazefully:implement-scalar-functions
Open

Add support for user-defined macro functions with arbitrary expressions#4318
hazefully wants to merge 13 commits into
FoundationDB:mainfrom
hazefully:implement-scalar-functions

Conversation

@hazefully

@hazefully hazefully commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

This PR extends support for user-defined macro functions (user-defined functions which return a value instead of a relational expression) to support arbitrary expressions as the function's body instead of only field values. In addition, the following is also now supported for user-defined macro functions:

  • Providing arguments to macro functions using named arguments in addition to positional arguments.
  • Having default values for macro functions parameters.
  • Specifying an explicit return type for a macro function using the keyword RETURNS
  • Defining the function body using both the keyword RETURN, in addition to the already supported keyword AS. Both keywords are identical in how the body is handled.

See the new tests added to the YAML test file user-defined-macro-function-tests.yamsql for examples on what macro functions are now supported and how they can be used.

In addition, this PR adds support for the YAML tests option supported_version to setup blocks to make it possible to only run setup blocks starting from some version in f880733. It also adds additional tests for macro functions to the file case-sensitivity.yamsql in 3b05bf8.

@hazefully hazefully added the enhancement New feature or request label Jul 1, 2026
@hazefully hazefully force-pushed the implement-scalar-functions branch from f3c3bee to e60a7d9 Compare July 1, 2026 16:16
@hazefully hazefully force-pushed the implement-scalar-functions branch from e60a7d9 to ad58abd Compare July 1, 2026 16:57
@hazefully hazefully added the Run mixed-mode Label to add to Pull Requests to have it run mixed mode tests label Jul 1, 2026
@hazefully hazefully marked this pull request as ready for review July 1, 2026 23:10
@hatyo hatyo self-requested a review July 2, 2026 10:18
Comment thread yaml-tests/src/test/resources/sql-functions.yamsql
Comment thread fdb-record-layer-core/src/main/proto/record_query_plan.proto
Comment thread yaml-tests/src/test/resources/user-defined-macro-function-tests.yamsql Outdated
return function.validateCall(arguments.toNamedArgumentInvocation());
return function.validateCall(namedArguments);
}
// todo: this should go throw validateCall for unnamed arguments, however that function is not considering

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please restore the todo (with the corrected rewording) this has to be consolidated and resolved at some point.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the todo because before this change validateCall was not being called for functions with unnamed arguments, but now it is. However, validateCall still doesn't check if the type of the arguments matches the expected function parameters types, as this is still delegated to the encapsulation logic, and the comment at L89 reflects that.

Is this something we want to change at a later time (i.e. move the type checking up from encapsulation time to function resolution time) or is this a design choice at this time?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're leaning more towards keeping the encapsulation logic intact and maybe move part of the type checking to the function resolution logic. this heavily depends on whether we want to support function overloading in the future or not.

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown

📊 Metrics Diff Analysis Report

Summary

  • New queries: 16
  • Dropped queries: 0
  • Plan changed + metrics changed: 23
  • Plan unchanged + metrics changed: 0
ℹ️ About this analysis

This automated analysis compares query planner metrics between the base branch and this PR. It categorizes changes into:

  • New queries: Queries added in this PR
  • Dropped queries: Queries removed in this PR. These should be reviewed to ensure we are not losing coverage.
  • Plan changed + metrics changed: The query plan has changed along with planner metrics.
  • Metrics only changed: Same plan but different metrics

The last category in particular may indicate planner regressions that should be investigated.

New Queries

Count of new queries by file:

  • yaml-tests/src/test/resources/user-defined-macro-function-tests.metrics.yaml: 16

Plan and Metrics Changed

These queries experienced both plan and metrics changes. This generally indicates that there was some planner change
that means the planning for this query may be substantially different. Some amount of query plan metrics change is expected,
but the reviewer should still validate that these changes are not excessive.

Total: 23 queries

Statistical Summary (Plan and Metrics Changed)

task_count:

  • Average change: -23.8
  • Median change: -16
  • Standard deviation: 11.7
  • Range: -57 to -16
  • Queries changed: 23
  • No regressions! 🎉

transform_count:

  • Average change: -7.0
  • Median change: -5
  • Standard deviation: 2.9
  • Range: -15 to -5
  • Queries changed: 23
  • No regressions! 🎉

transform_yield_count:

  • Average change: -2.8
  • Median change: -2
  • Standard deviation: 1.1
  • Range: -6 to -2
  • Queries changed: 23
  • No regressions! 🎉

insert_new_count:

  • Average change: -6.2
  • Median change: -4
  • Standard deviation: 3.2
  • Range: -15 to -4
  • Queries changed: 23
  • No regressions! 🎉

There were no queries with significant regressions detected.

Minor Changes (Plan and Metrics Changed)

In addition, there were 23 queries with minor changes.

@hazefully hazefully requested a review from hatyo July 3, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Run mixed-mode Label to add to Pull Requests to have it run mixed mode tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants