[analytics-backend-datafusion] Normalize CHAR/VARCHAR literals to Str in VirtualTable conversion - #22554
Conversation
PR Reviewer Guide 🔍(Review updated until commit 830d097)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 830d097 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 0663a61
Suggestions up to commit 4e3d130
Suggestions up to commit efeee4b
Suggestions up to commit 2e6d777
Suggestions up to commit f07f2eb
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #22554 +/- ##
============================================
- Coverage 71.42% 71.35% -0.08%
+ Complexity 76786 76737 -49
============================================
Files 6148 6148
Lines 357980 357980
Branches 52177 52177
============================================
- Hits 255689 255437 -252
- Misses 81937 82184 +247
- Partials 20354 20359 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…version isthmus's visit(Values) types each schema column from the Calcite RelDataType but derives each row literal from the string value, emitting fixed_char(n). A multi-row literal table (e.g. makeresults data=) then trips VirtualTableScan's row-conforms-to-schema check (FixedChar(3) vs FixedChar(5), or FixedChar vs Str). That check runs during isthmus POJO construction, before any proto is produced, so a proto-level rewrite is too late. Override visit(Values) on the SubstraitRelVisitor subclass in createVisitor and build the VirtualTableScan directly, mapping every CHAR/VARCHAR column to a length-independent Str in both the base schema and every row literal. Numeric and boolean columns are left to isthmus's own LiteralConverter and TypeConverter. Empty (zero-row) Values delegate to super.visit unchanged. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
da000e4 to
6ab8d92
Compare
|
Persistent review updated to latest commit 6ab8d92 |
|
Persistent review updated to latest commit f07f2eb |
Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
f07f2eb to
2e6d777
Compare
|
Persistent review updated to latest commit 2e6d777 |
Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit efeee4b |
…rted) Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit 4e3d130 |
…via fields) Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit 0663a61 |
|
Persistent review updated to latest commit 830d097 |
Description
An inline literal-row plan (a Calcite
LogicalValueswith string columns, e.g. fromPPL
makeresults format=csv data=...) fails to convert to Substrait for theDataFusion runtime. When a string column has rows of differing length it throws,
during Substrait POJO construction,
AssertionError: Row field type (FixedChar{length=3}) does not match schema field type (FixedChar{length=5}),which the convertor rewraps as
IllegalStateException: Substrait conversion rejected the plan: ....Root cause. isthmus types each row's string literal from the value itself
(
FixedCharLiteral.getType()returnsfixedChar(value().length()), so'Bob'isFixedChar(3)), while the column schema comes from the CalciteCHAR(maxLen)type(
FixedChar(5)).VirtualTableScan's@Value.Checkthen asserts each cell type isexactly equal to the column type, with no widening, so a shorter cell is rejected.
This runs at the RelNode-to-POJO layer, before any proto is generated, so a
proto-level rewrite is too late; and the
CAST(...):VARCHARthat themakeresultsProjectadds sits above theValuesand does not help. (Equal-length rows producematching
FixedChar(n)on both sides and are unaffected.)Fix. Override
visit(Values)on theSubstraitRelVisitorsubclass increateVisitor(the seam already used forvisit(Aggregate)) and map everyCHAR/VARCHAR column to a length-independent
Strin both the base schema and everyrow literal (
ExpressionCreator.string). Numeric/boolean columns are left toisthmus; empty (zero-row) Values delegate to
super.visit. This is a general fix forany inline-literal Values routed to the DataFusion backend, not specific to
makeresults.Reproduction
On a cluster with
cluster.pluggable.dataformat: composite(so scan-lessmakeresultsroutes to the analytics engine):Logical plan (
_explain); theLogicalValuescells areCHAR(n), which is why theProjectinsertsCAST($n):VARCHAR:HTTP 500, theAssertionErrorabove (thrown atVirtualTableScan.validateRowConformsToSchema).HTTP 200,datarows: [["Alice","30"],["Bob","25"]].Full stack trace (before the fix)
Source references (substrait-java 0.89.1)
isthmus/.../expression/LiteralConverter.java—CHAR->ExpressionCreator.fixedChar(value);VARCHARwithout precision ->ExpressionCreator.string(Str).core/.../expression/Expression.java—FixedCharLiteral.getType()returnsfixedChar(value().length());StrLiteral.getType()returnsSTRING.core/.../relation/VirtualTableScan.java—check()/validateRowConformsToSchema()assertrowFieldType.equals(schemaFieldType).Testing
:sandbox:plugins:analytics-backend-datafusion:testpassing.cluster.pluggable.dataformat=composite):the query above returns
HTTP 200after the change and the assertion before it; amixed string/int literal table also converts, with the int column staying numeric.
analytics-engine-restIT suite shows no new failures; the only failures arepre-existing on
main(FilterDelegationGoldenIT,SystemFunctionsIT.testTypeofArithmetic).Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.