Skip to content

Commit 779b245

Browse files
authored
[AURON #1863] Support native Flink UNIX_TIMESTAMP: converter integration (#2448)
# Which issue does this PR close? Part of #1863. Not `Closes`, because the issue description also covers the 0-input form, which this PR does not implement. This is the Flink Java side of `UNIX_TIMESTAMP`. The native function it calls merged in #2409. Together they cover the 1-input and 2-input forms from the issue description. The 0-input form is still outstanding. It is a different function rather than a missing branch: Flink binds the niladic form to `DateTimeUtils.unixTimestamp()`, which reads the clock per record and parses nothing, so it needs its own design pass. The converter rejects it explicitly and falls back to Flink's engine, and #1863 stays open to track it. # Rationale for this change This completes native support for Flink's `UNIX_TIMESTAMP` by wiring the Flink Calc converter to emit the native function added in #2409. Without it the native function is unreachable, and any Calc containing `UNIX_TIMESTAMP` falls back to Flink's engine for the whole Calc. # What changes are included in this PR? The converter recognizes `UNIX_TIMESTAMP` and lowers the 1-argument and 2-argument forms to the native `Flink_UnixTimestamp` node. `UNIX_TIMESTAMP` resolves to `SqlKind.OTHER_FUNCTION`, so it is matched by reference identity on the operator before the supported-kinds switch, the same way `TRY_CAST` is handled. A format scanner translates the supported subset of Java date-format letters (`yyyy MM dd HH mm ss` and literals) to the native format. Anything outside that subset falls back: other pattern letters, unsupported run-lengths, a non-literal format argument, the 0-argument form, and a numeric field adjacent to another numeric field where the run-length would not survive translation (for example `yyyyMd`). Falling back keeps results correct rather than risking a silent divergence. The session time zone is resolved at plan time and passed into the node. This required completing the config threading in the standalone-Calc path, which passed a persisted config that does not carry `table.local-time-zone`. That gap had no effect until now: `UNIX_TIMESTAMP` is the first time-zone-sensitive expression the converter supports, and the earlier ones (arithmetic, comparison, logical, cast) never read the session zone. The effective node config is threaded through instead, so the configured zone reaches the native evaluation. # Are there any user-facing changes? Yes. `UNIX_TIMESTAMP(string)` and `UNIX_TIMESTAMP(string, format)` now execute on the native engine when the format is a supported literal pattern. Unsupported patterns and the 0-argument form continue to run on Flink's engine, with the same results as before. # How was this patch tested? Unit tests for the scanner (accept/reject, quote escaping, the adjacency rule), the converter (node shape, format translation, time-zone propagation), and the operator-identity invariant. Fallback tests assert that unsupported inputs actually fall back rather than silently producing a native plan. An end-to-end ITCase runs `UNIX_TIMESTAMP(ts)` with a non-UTC session zone and confirms the native result matches the expected epoch values. The executed native plan shows the function and the resolved zone, confirming the query runs natively rather than falling back. 160 tests pass in `auron-flink-planner` on the rebased branch, with spotless clean and 0 checkstyle violations. # Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No Generated-by: Claude Code (Claude Opus 5)
1 parent 14d17f7 commit 779b245

11 files changed

Lines changed: 875 additions & 10 deletions

File tree

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,234 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
package org.apache.auron.flink.table.planner.converter;
18+
19+
import java.util.ArrayList;
20+
import java.util.List;
21+
import java.util.Objects;
22+
import java.util.Optional;
23+
24+
/**
25+
* Translates a {@code java.text.SimpleDateFormat} pattern into the native parser's
26+
* {@code strftime}-style format string, or reports that the pattern cannot be translated.
27+
*
28+
* <p>The native {@code Flink_UnixTimestamp} function replicates {@code SimpleDateFormat} lenient
29+
* parsing only for a fixed set of numeric fields. This converter is the plan-time gate: it accepts
30+
* a pattern only when every token maps to a field the native parser handles identically, and
31+
* returns {@link Optional#empty()} otherwise so the whole {@code Calc} falls back to Flink's engine.
32+
*
33+
* <p>Accepted fields and their native specifiers:
34+
* <ul>
35+
* <li>{@code yyy} / {@code yyyy} &rarr; {@code %Y} (year)
36+
* <li>{@code M} / {@code MM} &rarr; {@code %m} (month)
37+
* <li>{@code d} / {@code dd} &rarr; {@code %d} (day of month)
38+
* <li>{@code H} / {@code HH} &rarr; {@code %H} (hour of day)
39+
* <li>{@code m} / {@code mm} &rarr; {@code %M} (minute)
40+
* <li>{@code s} / {@code ss} &rarr; {@code %S} (second)
41+
* </ul>
42+
*
43+
* <p>Non-alphabetic characters are literals (a literal {@code %} is emitted as {@code %%}).
44+
* {@code SimpleDateFormat} single-quote escaping applies, including the doubled {@code ''} that
45+
* denotes a literal quote. Every other ASCII letter and every unlisted run length forces a fall
46+
* back, because Java reserves all letters and the omitted forms either depend on a locale or on the
47+
* clock at parse time.
48+
*
49+
* <p>Adjacency rule: run length is erased by the translation ({@code M} and {@code MM} both become
50+
* {@code %m}), yet the native parser reads each numeric field at a canonical width (year 4; month,
51+
* day, hour, minute, second 2) while Java's lenient scan window equals the run length. When two
52+
* numeric fields are adjacent with no literal separator, those two widths must agree, so the left
53+
* field's run length must equal its canonical width; otherwise the pattern falls back to avoid a
54+
* silent divergence (e.g. {@code yyyyMd} on {@code 20201010} yields month 1 in Java but month 10
55+
* natively).
56+
*/
57+
public final class FlinkDateTimeFormatConverter {
58+
59+
private FlinkDateTimeFormatConverter() {
60+
// utility class
61+
}
62+
63+
/**
64+
* Translates the given Java {@code SimpleDateFormat} pattern to the native {@code strftime}-style
65+
* format string.
66+
*
67+
* @param javaPattern the Java date-time format pattern, never {@code null}. Null is rejected
68+
* rather than reported as untranslatable: {@link Optional#empty()} means the user wrote a
69+
* pattern outside the native surface and the {@code Calc} should fall back, whereas a null
70+
* pattern means the caller never resolved one, which is a plumbing bug that a silent
71+
* fallback would hide.
72+
* @return the translated native format string, or {@link Optional#empty()} if any part of the
73+
* pattern is outside the natively supported surface
74+
* @throws NullPointerException if {@code javaPattern} is null
75+
*/
76+
public static Optional<String> translate(String javaPattern) {
77+
Objects.requireNonNull(javaPattern, "format pattern must not be null");
78+
List<Token> tokens = scan(javaPattern);
79+
if (tokens == null) {
80+
return Optional.empty();
81+
}
82+
if (!adjacencyValid(tokens)) {
83+
return Optional.empty();
84+
}
85+
StringBuilder out = new StringBuilder();
86+
for (Token token : tokens) {
87+
out.append(token.rendered);
88+
}
89+
return Optional.of(out.toString());
90+
}
91+
92+
/**
93+
* Walks the pattern into a token list, accumulating literal runs and emitting one token per
94+
* pattern-letter field. Returns {@code null} on any unsupported letter, unsupported run length,
95+
* or unterminated quote.
96+
*/
97+
private static List<Token> scan(String pattern) {
98+
List<Token> tokens = new ArrayList<>();
99+
StringBuilder literal = new StringBuilder();
100+
int i = 0;
101+
int n = pattern.length();
102+
while (i < n) {
103+
char c = pattern.charAt(i);
104+
if (c == '\'') {
105+
if (i + 1 < n && pattern.charAt(i + 1) == '\'') {
106+
literal.append('\'');
107+
i += 2;
108+
continue;
109+
}
110+
i++;
111+
boolean closed = false;
112+
while (i < n) {
113+
char q = pattern.charAt(i);
114+
if (q == '\'') {
115+
if (i + 1 < n && pattern.charAt(i + 1) == '\'') {
116+
literal.append('\'');
117+
i += 2;
118+
continue;
119+
}
120+
closed = true;
121+
i++;
122+
break;
123+
}
124+
literal.append(q);
125+
i++;
126+
}
127+
if (!closed) {
128+
return null;
129+
}
130+
} else if (isAsciiLetter(c)) {
131+
int j = i;
132+
while (j < n && pattern.charAt(j) == c) {
133+
j++;
134+
}
135+
Field field = fieldFor(c, j - i);
136+
if (field == null) {
137+
return null;
138+
}
139+
flushLiteral(tokens, literal);
140+
tokens.add(Token.field(field, j - i));
141+
i = j;
142+
} else {
143+
literal.append(c);
144+
i++;
145+
}
146+
}
147+
flushLiteral(tokens, literal);
148+
return tokens;
149+
}
150+
151+
private static void flushLiteral(List<Token> tokens, StringBuilder literal) {
152+
if (literal.length() > 0) {
153+
tokens.add(Token.literal(literal.toString()));
154+
literal.setLength(0);
155+
}
156+
}
157+
158+
/**
159+
* Checks the adjacency rule over the token list: for every field immediately followed by another
160+
* field (no intervening literal), the left field's run length must equal its canonical width.
161+
*/
162+
private static boolean adjacencyValid(List<Token> tokens) {
163+
for (int i = 0; i + 1 < tokens.size(); i++) {
164+
Token left = tokens.get(i);
165+
Token right = tokens.get(i + 1);
166+
if (left.field != null && right.field != null && left.runLength != left.field.canonicalWidth) {
167+
return false;
168+
}
169+
}
170+
return true;
171+
}
172+
173+
private static boolean isAsciiLetter(char c) {
174+
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z');
175+
}
176+
177+
private static Field fieldFor(char letter, int runLength) {
178+
switch (letter) {
179+
case 'y':
180+
return runLength == 3 || runLength == 4 ? Field.YEAR : null;
181+
case 'M':
182+
return runLength == 1 || runLength == 2 ? Field.MONTH : null;
183+
case 'd':
184+
return runLength == 1 || runLength == 2 ? Field.DAY : null;
185+
case 'H':
186+
return runLength == 1 || runLength == 2 ? Field.HOUR : null;
187+
case 'm':
188+
return runLength == 1 || runLength == 2 ? Field.MINUTE : null;
189+
case 's':
190+
return runLength == 1 || runLength == 2 ? Field.SECOND : null;
191+
default:
192+
return null;
193+
}
194+
}
195+
196+
/** A supported numeric field: its native specifier and the width the native parser reads. */
197+
private enum Field {
198+
YEAR("%Y", 4),
199+
MONTH("%m", 2),
200+
DAY("%d", 2),
201+
HOUR("%H", 2),
202+
MINUTE("%M", 2),
203+
SECOND("%S", 2);
204+
205+
private final String specifier;
206+
private final int canonicalWidth;
207+
208+
Field(String specifier, int canonicalWidth) {
209+
this.specifier = specifier;
210+
this.canonicalWidth = canonicalWidth;
211+
}
212+
}
213+
214+
/** A scanned token: either a field (non-null {@link #field}) or a literal run. */
215+
private static final class Token {
216+
private final Field field;
217+
private final int runLength;
218+
private final String rendered;
219+
220+
private Token(Field field, int runLength, String rendered) {
221+
this.field = field;
222+
this.runLength = runLength;
223+
this.rendered = rendered;
224+
}
225+
226+
static Token field(Field field, int runLength) {
227+
return new Token(field, runLength, field.specifier);
228+
}
229+
230+
static Token literal(String raw) {
231+
return new Token(null, 0, raw.replace("%", "%%"));
232+
}
233+
}
234+
}

auron-flink-extension/auron-flink-planner/src/main/java/org/apache/auron/flink/table/planner/converter/FlinkNodeConverterUtils.java

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,14 @@
1616
*/
1717
package org.apache.auron.flink.table.planner.converter;
1818

19+
import java.util.List;
1920
import org.apache.auron.flink.utils.SchemaConverters;
21+
import org.apache.auron.protobuf.ArrowType;
2022
import org.apache.auron.protobuf.PhysicalCastNode;
2123
import org.apache.auron.protobuf.PhysicalExprNode;
24+
import org.apache.auron.protobuf.PhysicalScalarFunctionNode;
2225
import org.apache.auron.protobuf.PhysicalTryCastNode;
26+
import org.apache.auron.protobuf.ScalarFunction;
2327
import org.apache.calcite.rel.type.RelDataType;
2428
import org.apache.calcite.rel.type.RelDataTypeFactory;
2529
import org.apache.calcite.rel.type.RelDataTypeSystem;
@@ -140,6 +144,27 @@ public static PhysicalExprNode wrapInCast(PhysicalExprNode expr, RelDataType tar
140144
.build();
141145
}
142146

147+
/**
148+
* Assembles a {@link PhysicalScalarFunctionNode} that routes to Auron's ext-function registry
149+
* ({@link ScalarFunction#AuronExtFunctions}) by name. The arguments must already be converted to
150+
* native expression nodes.
151+
*
152+
* @param name the registry name of the ext function (e.g. {@code "Flink_UnixTimestamp"})
153+
* @param args the already-converted argument expressions, in call order
154+
* @param returnType the native Arrow return type of the function
155+
* @return a {@link PhysicalExprNode} wrapping the scalar-function call
156+
*/
157+
public static PhysicalExprNode buildExtScalarFunctionNode(
158+
String name, List<PhysicalExprNode> args, ArrowType returnType) {
159+
return PhysicalExprNode.newBuilder()
160+
.setScalarFunction(PhysicalScalarFunctionNode.newBuilder()
161+
.setName(name)
162+
.setFun(ScalarFunction.AuronExtFunctions)
163+
.addAllArgs(args)
164+
.setReturnType(returnType))
165+
.build();
166+
}
167+
143168
private static boolean notApproxType(SqlTypeName typeName) {
144169
return typeName != SqlTypeName.FLOAT && typeName != SqlTypeName.DOUBLE;
145170
}

0 commit comments

Comments
 (0)