feat(cubesql): Support COPY ... FROM STDIN

[MazterQyou]

Check List

• Tests have been run in packages where changes have been made if available
• Linter has been run for changed code
• Tests for the changes have been added if not covered yet
• Docs have been added / updated if required

Description of Changes Made

This PR adds support for COPY ... FROM STDIN command, allowing adding data to temporary tables. Related tests are included.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
cubed3	🟢 Ready	View Preview	Aug 12, 2026, 4:29 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[claude]

Claude finished @MazterQyou's task in 9m 44s —— View job

---

Review: 2 high, 3 medium, 4 low — 6 inline comments posted

Full review — findings, verification notes, and todo list

Todos

• Gather context (diff, existing review threads — none found)
• Review protocol layer (pg-srv/protocol.rs, buffer.rs)
• Review COPY parsing/planning (compile/copy.rs, router.rs, plan.rs)
• Review data ingest (sql/postgres/copy.rs, temp_tables.rs, shim.rs)
• Review tests & docs
• Post findings

General

This is unusually careful work. The decoder mirrors CopyReadLineText / CopyReadAttributesText / CopyReadAttributesCSV down to error codes, CONTEXT/HINT fields and limit_printout_length; rows are correctly assembled across arbitrary CopyData boundaries and the chunked test helper exercises that; an error mid-copy discards everything so a failed COPY leaves the table untouched; and the reserve rewrite in temp_tables.rs fixes a real pre-existing bug (the old nested fetch_update could double-count the server total on CAS retry) and now rolls the session counter back when the total limit rejects. Comments explain why, not what. Docs are appropriately concise and in the right place.

Findings below; the substantive ones are also inline on the diff.

High

1. CREATE TEMPORARY TABLE performs its side effect during planning — router.rs:765. create_empty_table_to_plan calls TempTableManager::save from plan_query, but plan_query also runs on Parse (shim.rs:1127), on Bind (shim.rs:1006), under EXPLAIN, and as a compile-only check for DECLARE. Any driver that uses the extended protocol for DDL creates the table at Parse and then fails at Bind with relation "t" already exists, before Execute. All tests use simple_query, which plans once, so this is uncovered.

2. Trailing lone backslash drops a field and then spins — copy.rs:245. split_line(0) leaves the backslash in the buffer with scanned reset to 0, so finish() re-reads the same state forever. For a single-column table it appends empty rows until check_memory_limit trips ~2M iterations later and reports a bogus memory-limit error; for wider tables the row loses its trailing backslash. Suggested one-line fix inline.

Medium

3. parse_decimal rejects valid numeric values — copy.rs:961. Leading zeros count against precision (007 → numeric(2,0) rejected) and sub-1 values consume the whole precision (0.05 → numeric(2,2) rejected). Separately, PG rounds a too-long fraction to the declared scale rather than erroring, which combined with the Decimal(38, 10) default for bare NUMERIC makes >10 decimals an error.

4. NOT NULL is accepted but never enforced — copy.rs:447. Unlisted columns are filled with None and \N is appended as-is regardless of the field's nullable flag. Best case the constraint is silently ignored; worst case RecordBatch::try_new fails with an opaque internal error instead of PG's 23502.

5. FORCE_NOT_NULL / FORCE_NULL names aren't identifier-folded — compile/copy.rs:289. The COPY column list uses normalize_ident, these use the raw ident.value, so FORCE_NOT_NULL (A) fails with column "A" … does not exist while COPY t (A) works.

Low / nits

6. timestamptz silently becomes naive — router.rs:1078; values carrying an offset fail to parse. Inline.

7. IF NOT EXISTS is silently ignored — the .. in the CreateTable pattern swallows if_not_exists (and column.collation), so CREATE TEMPORARY TABLE IF NOT EXISTS t (…) still errors when t exists.

8. Decode budget is the full session limit, not the remaining one — handle_copy_in passes TempTableManager::session_memory_limit(), and accepted_bytes / built_bytes are each compared against it independently, so peak transient memory can approach ~2× the configured limit on top of what the session already holds. The real enforcement is the reserve inside append, which only runs after the entire stream has been buffered and built — a copy into a nearly-full session does all the work before failing.

9. Minor: is_utf8_encoding doesn't accept PG's unicode alias; CopyInResponse::serialize returns None (silently writing no message, hanging the client) when columns exceed i16::MAX, unreachable in practice but an error would be better than a hang; FORCE_NOT_NULL columns aren't required to appear in the copy column list, which PG rejects.

Verification notes

I did not build or run the test suite here (the DataFusion fork build is prohibitively long in this environment) — findings 1–5 come from reading the call graph, and the reasoning for each is spelled out in the inline comments so they can be checked quickly. Findings 1 and 2 are deterministic from the code paths shown rather than heuristic.

· branch `cubesql/copy-command`

[codecov]

Codecov Report

❌ Patch coverage is 89.61260% with 244 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.28%. Comparing base (33c8828) to head (b66ff70).

Files with missing lines	Patch %	Lines
rust/cubesql/cubesql/src/sql/postgres/copy.rs	90.90%	94 Missing ⚠️
rust/cubesql/cubesql/src/compile/router.rs	75.34%	53 Missing ⚠️
rust/cubesql/cubesql/src/compile/copy.rs	87.60%	31 Missing ⚠️
rust/cubesql/pg-srv/src/protocol.rs	70.00%	27 Missing ⚠️
rust/cubesql/cubesql/src/compile/test/test_copy.rs	96.93%	17 Missing ⚠️
rust/cubesql/cubesql/src/compile/plan.rs	0.00%	12 Missing ⚠️
rust/cubesql/cubesql/src/sql/temp_tables.rs	94.56%	5 Missing ⚠️
rust/cubesql/cubesql/src/sql/postgres/shim.rs	96.34%	3 Missing ⚠️
rust/cubesql/cubesql/src/compile/test/mod.rs	0.00%	1 Missing ⚠️
rust/cubesql/cubesql/src/sql/postgres/extended.rs	92.30%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #11538       +/-   ##
===========================================
+ Coverage   59.35%   84.28%   +24.92%     
===========================================
  Files         223      260       +37     
  Lines       17989    83859    +65870     
  Branches     3641        0     -3641     
===========================================
+ Hits        10677    70678    +60001     
- Misses       6793    13181     +6388     
+ Partials      519        0      -519     

Flag	Coverage Δ
cube-backend	?
cubesql	84.28% <89.61%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[claude]

Claude finished @MazterQyou's task in 5m 13s —— View job

---

Re-review after force-push: all 6 prior findings fixed — 1 medium, 5 low remain (2 inline comments posted)

Full review — todos, what changed, and remaining findings

Todos

• Gather context (diff vs. prior reviewed commit dfb05cf, existing review threads)
• Check which prior findings were addressed
• Review the reworked code (planning, decoder, protocol, temp tables, docs)
• Resolve stale threads (6 resolved)
• Post new findings

Prior findings — all addressed

#	Finding	Status
1	CREATE TEMPORARY TABLE created the table during planning	Fixed — new QueryPlan::CreateEmptyTempTable carries the schema; save now happens at execution (extended.rs:593), with an explicit comment at router.rs:766. COPY itself is now executed from Execute via Portal::take_copy_from, so the extended protocol works end to end.
2	Trailing lone backslash dropped a field and spun	Fixed at copy.rs:246, with test_trailing_backslash_at_end_of_data covering both the multi-column and single-column cases
3	parse_decimal rejected valid numeric values	Fixed — the fraction is now rounded half-away-from-zero to the declared scale and precision is left to the builder as an overflow. Spot-checked 0.005→0.01 (scale 2), 1.5→2 (scale 0), 9.999→ overflow at numeric(3,2); all match PG
4	NOT NULL accepted but unenforced	Fixed at copy.rs:478 with 23502 and PG's wording; test_not_null_is_enforced covers both the explicit \N and the unlisted-column case
5	FORCE_NOT_NULL/FORCE_NULL names not folded	Fixed — column_names now goes through normalize_ident, and both lists are validated against the schema at plan time
6	timestamptz silently became naive	Fixed — WITH TIME ZONE is now rejected at DDL time, and parse_timestamp ignores an offset the way PG's timestamp input does

Six stale threads resolved.

Remaining findings

Medium

1. CSV ESCAPE cannot escape itself, and the unquoting desynchronizes from the scan — copy.rs:759. PG peeks for escapec || quotec; this peeks only for the quote. Harmless with the default ESCAPE (same character as QUOTE), but with ESCAPE '\' the row 1,"a\\" is one complete row to take_line and comes out of split_csv_line as a\" instead of a\ — a wrong value loaded silently, not an error. ESCAPE is not exercised by any end-to-end test. Suggested fix inline.

Low / nits

2. CopyFail's doc comment describes CopyDone — inline. (Two comments there: the first anchored a line low and its suggestion block would replace the #[derive]; the second is the correct one.)
3. IF NOT EXISTS is still silently ignored — the .. in the CreateTable pattern at router.rs:198 swallows if_not_exists, so CREATE TEMPORARY TABLE IF NOT EXISTS t (…) still errors when t exists rather than being a no-op. Same for DROP TABLE IF EXISTS, which is pre-existing.
4. Decode budget is the full session limit, not the remaining one — handle_copy_in passes TempTableManager::session_memory_limit() (shim.rs:1794), and accepted_bytes/built_bytes are each compared against it independently. A copy into a nearly-full session buffers and builds the whole stream before reserve inside append rejects it.
5. is_utf8_encoding doesn't accept PG's unicode alias; CopyInResponse::serialize returns None (writing nothing and hanging the client) when columns exceed i16::MAX — unreachable in practice, but an error beats a hang; FORCE_NOT_NULL columns are validated against the schema but not required to appear in the COPY column list, which PG rejects.
6. Small input-function divergences: parse_bool doesn't take PG's unique prefixes (tr, fal, ye), and char(n) is not blank-padded to its declared width.

General

The rework is clean and the fixes went in at the right level rather than being patched at the edges — moving the table creation into a plan variant also got COPY itself onto the extended protocol path, Portal::take_copy_from leaves the portal in a Finished state so a repeated Execute can't re-run the copy, and stray CopyData/CopyDone/CopyFail outside a copy are dropped in the main loop the way PG does. reserve's two-counter ordering with the session rollback on total-limit rejection is right, and the comment explaining why the counters aren't nested is the kind that saves the next reader real time. New tests are targeted at the exact regressions rather than added for coverage's sake.

Verification notes

I did not build or run the test suite (the DataFusion fork build is prohibitively long in this environment). The medium finding is traced through both take_line and split_csv_line against PG's CopyReadAttributesCSV, with the concrete input and both outputs spelled out in the inline comment so it can be checked quickly.

· branch `cubesql/copy-command`

[claude]

With a non-default ESCAPE, the escape character cannot escape itself, and the unquoting disagrees with the scan.

PostgreSQL's CopyReadAttributesCSV peeks for either character after the escape:

if (c == escapec)
{
    if (cur_ptr < line_end_ptr)
    {
        char nextc = *cur_ptr;
        if (nextc == escapec || nextc == quotec)
        {
            *output_ptr++ = nextc;
            cur_ptr++;
            continue;
        }
    }
}

Here only options.quote is peeked for. With the default ESCAPE (which is QUOTE) the two are the same character, so nothing is lost; with WITH (FORMAT csv, ESCAPE '\') they diverge, and the divergence is not just a missing unescape — it desynchronizes this splitter from take_line.

For the row 1,"a\\" (field a followed by two backslashes, ESCAPE '\'):

• take_line (line 215) sees \ inside the quote, skips two bytes, then closes the quote on " — one complete row, which is right.
• split_csv_line sees \ with peek \ ≠ quote → pushes \; then \ with peek " == quote → pushes " and consumes it, so in_quote is never cleared. The field comes out as a\".

PostgreSQL stores a\. So this silently loads a wrong value rather than erroring. ESCAPE isn't exercised anywhere in test_copy.rs, and the one unit test that sets options.escape = '\\' (test_the_scan_never_runs_past_the_data) only checks the unterminated-quote error.

Suggested change

	if char == options.escape && chars.peek() == Some(&options.quote) {
	if char == options.escape
	&& chars
	.peek()
	.is_some_and(|next| *next == options.quote || *next == options.escape)
	{

[claude]

Nit: the doc comment describes CopyDone, not CopyFail — this message means the client is aborting the copy and carries the reason.

Suggested change

	#[derive(Debug, PartialEq)]
	/// (F) Sent by the client to abort a `COPY ... FROM STDIN`, carrying the reason.

[claude]

Nit (this is the line my adjacent comment meant — please ignore that one's suggestion block, it anchored one line too low and would replace the #[derive]):

This doc comment describes CopyDone, not CopyFail — CopyFail means the client is aborting the copy, and the message is the reason. Suggested wording: /// (F) Sent by the client to abort a COPY ... FROM STDIN, carrying the reason.