TCP J4: Add Dynamic support - #460
Conversation
ccde30a to
4d3c00a
Compare
TriageCategory: Summary What this impacts
Concerns
Required reviewer action
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (2)
ClickHouse.Driver.Tcp/Types/Codecs/TupleColumnCodec.cs:367
- In the flat tuple write path,
BuildStatereturns pooledchildBoxedbuffers infinally, but if one of the child codecs’BeginWrite(...)calls throws, any earlier child states are leaked (they may hold rented buffers) because they’re never disposed on the exceptional path.
try
{
for (int row = 0; row < length; row++)
{
var tuple = (ITuple)column.GetValue(start + row);
for (int i = 0; i < arity; i++)
{
childBoxed[i][row] = tuple[i];
}
}
for (int i = 0; i < arity; i++)
{
childColumns[i] = childFlatBuilders[i](column.Name, children[i].TypeName, childBoxed[i], length);
childStates[i] = children[i].BeginWrite(childColumns[i], 0, length);
}
return new TupleWriteState { ChildColumns = childColumns, ChildStart = 0, Length = length, ChildStates = childStates };
ClickHouse.Driver.Tcp/Types/Codecs/MapColumnCodec.cs:392
- In the jagged map path, if one of the inner
BeginWrite(...)calls throws (especiallyvalueCodec.BeginWriteafterkeyStateis created), the already-created inner state is leaked. The exception path should dispose any created inner state before returning the pooled key/value buffers.
var keyColumn = ArrayColumn<TKey>.OverBuffer(column.Name, keyCodec.TypeName, flatKeys, total);
var valueColumn = ArrayColumn<TValue>.OverBuffer(column.Name, valueCodec.TypeName, flatValues, total);
IColumnWriteState keyState = keyCodec.BeginWrite(keyColumn, 0, total);
IColumnWriteState valueState = valueCodec.BeginWrite(valueColumn, 0, total);
return new MapWriteState(keyColumn, valueColumn, pairBase: 0, total, keyState, valueState, flatKeys, flatValues);
}
catch
{
ArrayPool<TKey>.Shared.Return(flatKeys, clearArray: RuntimeHelpers.IsReferenceOrContainsReferences<TKey>());
ArrayPool<TValue>.Shared.Return(flatValues, clearArray: RuntimeHelpers.IsReferenceOrContainsReferences<TValue>());
throw;
| var fieldColumns = new IColumn[children.Length]; | ||
| var fieldStates = new IColumnWriteState[children.Length]; | ||
| for (int f = 0; f < children.Length; f++) | ||
| { | ||
| children[f].WriteColumn(writer, nested.GetField(f), elementBase, elementCount); | ||
| fieldColumns[f] = nested.GetField(f); | ||
| fieldStates[f] = children[f].BeginWrite(fieldColumns[f], elementBase, elementCount); | ||
| } | ||
|
|
||
| return new NestedWriteState { FieldColumns = fieldColumns, ElementBase = elementBase, ElementCount = elementCount, FieldStates = fieldStates }; |
4d3c00a to
34ba7b1
Compare
|
Thanks @copilot — all four inline findings plus the two low-confidence ones were valid (the same pooled-buffer / partial-child-state leak class). Addressed in the latest push:
These mirror the try/catch cleanup already applied to |
34ba7b1 to
e602f97
Compare
e602f97 to
33505ed
Compare
33505ed to
b244505
Compare
b244505 to
285d486
Compare
42231e7 to
9e891ea
Compare
9e891ea to
1bf1cb5
Compare
| private static string InferFromClrType(Type type) | ||
| { | ||
| if (type is not null && Scalars.TryGetValue(type, out string scalar)) | ||
| { | ||
| return scalar; | ||
| } | ||
|
|
||
| throw new NotSupportedException( | ||
| $"No ClickHouse type is inferred for a Dynamic array whose element CLR type is '{type?.ToString() ?? "unknown"}' with no element to disambiguate it."); | ||
| } |
e91cc73 to
909b359
Compare
523dc5d to
bad13c6
Compare
5925968 to
c4f7c01
Compare
c4f7c01 to
24c45ef
Compare
24c45ef to
de7677c
Compare
de7677c to
aae2f5c
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
Suppressed comments (6)
ClickHouse.Driver.Tcp/Types/Codecs/DynamicTypeInference.cs:102
- Returning the original
Arrayis unsafe when value-based inference chooses a different CLR element shape. For example,object[] { 1 }infersArray(Int32)but remains anobject[]; the write planner later builds theInt32[]child andBuildFlatColumn<int[]>throwsInvalidCastException.int?[] { 1, null }similarly infers non-nullableArray(Int32). Canonicalize the array to the inferred codec shape (including nullability), or reject these shapes here with a clear error.
if (value is Array array)
{
return (InferArrayOrMap(array), value);
ClickHouse.Driver.Tcp/Types/Codecs/DynamicTypeInference.cs:117
- Map inference uses only
KeyValuePair<K,V>'s declared generic arguments, so supported value-dependent/null-capable shapes cannot be inferred even from populated maps. For example,KeyValuePair<string, IPAddress>[]throws becauseInferFromClrType(IPAddress)cannot distinguish IPv4/IPv6, andKeyValuePair<string, uint?>[]also throws. Inspect present pairs and validate a homogeneous inferred key/value type, with declared-type fallback only for empty maps.
Type elementType = array.GetType().GetElementType();
if (elementType is not null && elementType.IsGenericType && elementType.GetGenericTypeDefinition() == typeof(KeyValuePair<,>))
{
Type[] pair = elementType.GetGenericArguments();
return $"Map({InferFromClrType(pair[0])}, {InferFromClrType(pair[1])})";
ClickHouse.Driver.Tcp/Types/DynamicColumn.cs:129
Valuesmaterializes and caches rows independently, while all current round-trip/decoder assertions callGetValue, so this new accessor is never executed. AGENTS.md:165-170 requires dedicated coverage for independently materializedValuescaches. Add a test asserting value order, NULL handling, and repeated cached access.
public ReadOnlySpan<object> Values
ClickHouse.Driver.Tcp/Types/ColumnCodecRegistry.cs:162
- Registering Dynamic adds user-visible TCP client behavior, but this PR does not update either release document. AGENTS.md:303-305 requires every behavioral client change to update both
CHANGELOG.mdandRELEASENOTES.md; add a concise TCP Dynamic support entry to each.
AddFactory("Dynamic", static (TypeNode node, in ResolveContext context, ColumnCodecRegistry registry) => DynamicColumnCodec.Create(node, context, registry));
ClickHouse.Driver.Tcp/Types/Codecs/DynamicTypeInference.cs:176
- The suggested raw-
longworkaround cannot representDateTime64inside a Dynamic composite:Infer(long)always selectsInt64, soDateTimeOffset[]is rejected whilelong[]becomesArray(Int64), notArray(DateTime64(9)). There is also noClickHouseDateTime64carrier in the TCP project. This leaves the workaround promised in the PR description unavailable; add an unambiguous carrier/typed value mechanism or support recursive coercion.
(string typeName, object canonical) = Infer(element);
if (canonical.GetType() != element.GetType())
{
throw new NotSupportedException(
$"A Dynamic composite element of CLR type '{element.GetType()}' would be coerced to '{canonical.GetType()}', which is not supported inside a composite. Use the canonical element type directly (e.g. a raw long count for a DateTime64, ClickHouseDecimal for a decimal).");
ClickHouse.Driver.Tcp/Types/Codecs/DynamicColumnCodec.cs:93
- This validation accepts repeated arguments and values outside ClickHouse's valid range, e.g.
Dynamic(max_types=1, max_types=2),Dynamic(max_types=-1), andDynamic(max_types=255). ClickHouse accepts at most one unsignedmax_typesvalue in the range 0–254, so these codecs pass client validation only to fail at the server. Enforce both arity and range here.
foreach (TypeNode argument in node.Arguments)
{
if (!TryParseMaxTypes(argument.Name, out _))
{
throw new FormatException(
$"Dynamic type '{node}' has unsupported argument '{argument.Name}'; only 'max_types=N' is recognized.");
}
}
Adds the ClickHouse `Dynamic` column type — a column whose per-row value type is discovered at runtime — over the native protocol, reading and writing only the FLATTENED serialization (wire version 3), gated on `output_format_native_use_flattened_dynamic_and_json_serialization = 1`. The leading version word is validated and any non-flat encoding rejected. - Dense `DynamicColumn` (runtime type-name list + widened discriminators + per-type child columns) as the zero-copy read/write source; an ergonomic `IColumn<object>` write source infers each value's ClickHouse type via a self-contained `DynamicTypeInference` (scalars, IPv4/IPv6 by address family, date-times, decimals, and Array/Map/Tuple recursion), normalizing a value to its codec's element type so it round-trips. - Introduces the per-operation write-state contract (`IColumnCodec.BeginWrite` + `IColumnWriteState` + state-aware `WriteStatePrefix`/`WriteColumn` overloads that default to the state-free ones, so leaf codecs are untouched). The block writer runs BeginWrite -> prefix -> body -> Dispose. This lets a data-dependent prefix (the Dynamic type list) and the element-flattening composites do their flatten/scatter once across both phases. - Full nesting both directions: the Array/Tuple/Map/Nested codecs now project their flattened inner sub-column into the child's prefix and body, so `Array(Dynamic)`, `Tuple(...Dynamic...)`, `Map(K, Dynamic)`, `Nested(...Dynamic...)` and composite values inside a Dynamic all round-trip. Variant rejects a Dynamic alternative (server-disallowed). - Like Variant, there is no `Nullable(Dynamic)` — NULL rides the discriminator (value `num_types`, not 255). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dynamic infers its alternatives from the data at write time, so it cannot densify up front (its Densify stays the identity) — but the per-type columns it builds from the boxed values are the ergonomic form, and the resolved codecs now write and measure only the dense wire shape. So both data-dependent paths densify each per-type column before delegating: BuildState densifies each bucket before the child's BeginWrite (storing the dense column for the body phase), and the per-row measure densifies its one-row probe before pricing it. This makes a Dynamic value that is itself an Array/Tuple/Nullable round-trip end-to-end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dynamic discovers its per-type children from the ergonomic source and writes each straight through its codec, with no densify pre-pass or byte measurement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR 460 feedback: - Reject a heterogeneous Dynamic array (e.g. IPv4 mixed with IPv6 in an IPAddress[]) with a clear NotSupportedException at inference time, rather than letting a later element fail the bucket cast mid-write. - Infer an empty nested composite (an empty Array(Array(T)) or Array(Map(...))) structurally from its CLR element type instead of throwing, so empty nested arrays are usable inside Dynamic. - Null-check the BuildFlatColumn reflection lookup in FlatBuilderFor, throwing an actionable InvalidOperationException like VariantColumnCodec rather than a NullReferenceException. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ReadColumn_DocumentedBytes_ReconstructsValuesAndNull asserted only
RowCount + GetValue on the golden vector, which the "Dynamic [scalars + null]"
case in InsertRoundTripCase reproduces against a real server with the same
{String, UInt64} type list and NULL discriminator.
Rather than delete it, folded it into
ReadColumn_DocumentedBytes_SurfacesTheRuntimeTypeList -- same setup, and
keeping the value assertions next to the type list and discriminators guards
the decoder against drifting even if a future server emits a different type
ordering. Net one test, no coverage lost.
Added the Dynamic(max_types=2) case. That is the one shape where the server
reshapes the dynamic structure -- types past N go to the shared/overflow
bucket -- and Create_MaxTypesArgument_IsAccepted only checks the TypeName
string, so the client's read of the reshaped structure had no coverage at all.
Verified against ClickHouse 26.6 -- 1009 tests pass, integration included.
Co-Authored-By: Claude <noreply@anthropic.com>
Same defect as Variant, which Dynamic's indexer mirrors: it read discriminators[row] out of a buffer that is normally a pooled array longer than the column, so an out-of-range row's behavior depended on the leftover value — a stale NULL discriminator reported the row as an existing NULL, anything else fell through to the exactly-sized local index and threw. Indexing through the RowCount-sliced Discriminators span makes both a bounds failure, and the constructor now validates the buffer rather than letting the local-index walk fault on a short one. Co-Authored-By: Claude <noreply@anthropic.com>
IDynamicColumn already described the wire layout, so this promotes it and documents the two things that distinguish Dynamic from Variant for a consumer reading it columnar: the runtime type list is discovered per block rather than declared, so TypeNames is how a caller knows which typed column to read a child as; and because the list is discovered, NULL is encoded as TypeCount — one past the last type — instead of Variant's fixed 255. Like Variant, Dynamic has no useful materialized element type, so its IColumn<T> surface is IColumn<object> and the columnar view is the typed way in.
Same regression as the variant branch: making IDynamicColumn public turned the dense write path's trust boundary into a public extension point. The planner trusts invariants only DynamicColumn's constructor establishes — the type-name list matches the child-column count, and every discriminator is a valid type index or the NULL marker — so gate on the concrete class, as every other composite does. The interface stays public as a read view. Two consequences of the same exposure, fixed alongside: TypeNames returned the backing string[] as IReadOnlyList<string>, which is castable back and mutable. A caller could rewrite a decoded column's type list, after which a re-insert resolves a codec from the rewritten name while the child column still holds the original values. Hand out a read-only wrapper instead. BuildDenseState rented its discriminator slice before the per-type walk that indexes by discriminator, with nothing between the rent and the try that would return it. Reordered so the rent comes after the fallible indexing. Unreachable again now the gate is concrete, but it cost nothing to stop relying on that.
Same two additions as the variant branch, plus a note that TypeNames is read-only and does not hand out its backing storage.
Last of the six composites. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Drop the null-state fallback from the Dynamic codec, as the write-state contract now takes the state as required in the state-aware overloads. Cover the dense slice plan, which had no test at all. The plan derives each type's child-column run start from the local index of that type's first in-slice row, and at start 0 every one of those is 0 - so a slice starting at row 0 cannot tell a correct plan from one that ignores the offset. Both new tests slice past row 0: - a codec test asserting the exact bytes of rows [3, 5) of a five-row column whose two types each hold two values, so each run is written from offset 1. Replacing the run start with 0 fails this test and nothing else. - an integration case inserting the dense column as blocks of two rows, the Dynamic counterpart of the Variant case beside it. Co-Authored-By: Claude <noreply@anthropic.com>
DynamicWire held three members that only DynamicColumnCodec used: the flattened serialization version, the type-count ceiling, and the discriminator-width rule. A separate file for them added a hop without adding a boundary, so move them into the codec — the version and the ceiling as private constants, the width rule as an internal static method the tests still reach. The width test moves with it, from DynamicTypeInferenceTests to DynamicColumnCodecTests, where it now belongs. Co-Authored-By: Claude <noreply@anthropic.com>
Inference for a Map inside a Dynamic used only the CLR generic arguments of KeyValuePair<K, V>. That helper knows the scalar table and arrays, so a type that needs a value to resolve — an IPAddress (address family) or a ClickHouseDecimal (scale) — threw NotSupportedException as a map key or value, although the same type works as an array element or a tuple element. Scan the present pairs instead, as the array branch already does, and keep the CLR types as the fallback for an empty map. The two branches now share one Agree helper, which also gives the map the mixed-type rejection the array had: without it, a map of mixed IPv4 and IPv6 keys would trade a clear error for an InvalidCastException in the bucket projection. The pair scan runs through a cached per-pair-type delegate, the pattern DynamicColumnCodec already uses, so it does not box each pair or reflect over Key/Value once per entry. Also correct a comment that named ClickHouseDateTime64, a type that does not exist; the coerced value is the raw Int64 nanosecond count. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 55416b7. Configure here.
| { | ||
| Type[] pair = elementType.GetGenericArguments(); | ||
| (string key, string value) = PairScannerFor(elementType)(array); | ||
| return $"Map({key ?? InferFromClrType(pair[0])}, {value ?? InferFromClrType(pair[1])})"; |
There was a problem hiding this comment.
Boxed map write type mismatch
Medium Severity
Map inference now derives key and value ClickHouse types from present pair values, so a KeyValuePair<string, object>[] can resolve to something like Map(String, IPv4). Infer still returns that original pair array, whose CLR type does not match MapColumnCodec.ElementType (KeyValuePair<string, IPAddress>[]). The Dynamic write path then fails with an InvalidCastException in BuildFlatColumn when projecting the bucket, instead of a clear inference rejection. Strongly typed pair arrays still work; the boxed-value path newly succeeds at inference and breaks on write.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 55416b7. Configure here.


Adds the ClickHouse
Dynamiccolumn type over the native protocol, stacked on #459 (Variant).Dynamicis a column whose per-row value type is discovered at runtime. This reads and writes only the FLATTENED serialization (wire version 3), gated on the query settingoutput_format_native_use_flattened_dynamic_and_json_serialization = 1; the leadingUInt64version word is validated and any non-flat encoding rejected as a protocol error. (A follow-up will have the client set that setting automatically — tracked in the TODO.)What's here
Dynamic(FLATTENED v3): version + runtime type-name list + per-type prefixes as the state prefix, then discriminators (width scales with type count; NULL =num_types) + one dense run per type. Verified against a real 26.6 server; the documented-bytes unit tests use a real server capture.DynamicColumn(type-name list + widened discriminators + per-type child columns) — the zero-copy read/write source; an ergonomicIColumn<object>write source infers each value's ClickHouse type via a self-containedDynamicTypeInference(the TCP project can't reference the main driver): scalars, IPv4/IPv6 by address family, date-times, decimals, andArray/Map/Tuplerecursion, normalizing each value to its codec's element type.IColumnCodec.BeginWrite+IColumnWriteState, with state-awareWriteStatePrefix/WriteColumnoverloads that default to the existing state-free ones (leaf codecs untouched). The block writer runsBeginWrite → prefix → body → Dispose, so a data-dependent prefix (the Dynamic type list) and the element-flattening composites do their flatten/scatter once across both phases.Array/Tuple/Map/Nestednow project their flattened inner sub-column into the child's prefix and body, soArray(Dynamic),Tuple(...Dynamic...),Map(K, Dynamic),Nested(...Dynamic...)and composite values inside a Dynamic all round-trip.Variantrejects aDynamicalternative (server-disallowed).Nullable(Dynamic)— NULL rides the discriminator.Testing
Array(Dynamic),Tuple(Dynamic, String),Tuple(Dynamic, Dynamic),Map(String, Dynamic),Nested(a Dynamic, b String),Array(Tuple(Dynamic, String)).DynamicColumnCodec~90% line coverage (remaining gaps are the >255-type discriminator-width branches).Notes / deferred
DateTimeOffset[]inside a Dynamic) throws a clearNotSupportedException; use the canonical CLR type (ClickHouseDateTime64/ClickHouseDecimal).Map; passKeyValuePair<K, V>[], the shape the map codec reads back. InferringMap(K, V)from anyIDictionary/IReadOnlyDictionarywas tried as an experiment and dropped together with the dictionary write path in TCP I4: Add Map(K, V) support #443 — the interface walk needed a per-type plan cache and several rejections (ambiguous key/value pairs, multidimensional arrays) to stay correct, which was not worth it for a second spelling of a type the client already accepts.🤖 Generated with Claude Code