[iluebbe]

Problem

I tested CSV and text import against Excel 365 on Windows (de-DE UI). The test covered File › Open, Data › From Text/CSV (data.fromTextCsv), paste with Text to Columns, and NUMBERVALUE / TEXTSPLIT. It used about 60 variant files: delimiters , ; TAB |, quoting, line ends, BOMs, en-US / de-DE / fr / de-CH / en-IN / zh-CN numbers and dates, and Windows-1252 / UTF-16 / GB18030. Nine bugs came up. Main 84acf62 is the baseline; I found no existing issue or PR for any of them.

#	Path	Input	Excel 365	GridCraft main
1	From Text/CSV	;-separated file (r2;1.234.567,89, 12,5 %)	splits on ; (File › Open in GridCraft does too)	detects only tab or comma: one column per line, and the decimal commas split numbers (1.234.567,89 → "r2;1.234.567" and 89, 12,5 % → 0.05)
2	From Text/CSV	=1+1, =HYPERLINK("http://example.com","click")	text (Power Query; GridCraft's File › Open keeps text too)	live formulas: 2 and a working hyperlink; an unquoted comma inside the formula splits it
3	From Text/CSV	2024-03-04, 12.5%, $1,234.50	date, percent and currency formats	45355, 0.125, 1234.5 (General)
4	From Text/CSV	a CSV starting with [ID],Name	CSV	refused: "invalid parameters for data.getData: expected value at line 1 column 2" (read as JSON)
5	all import paths	's-Hertogenbosch, 't Hooft, '007	the apostrophe is kept	stripped (s-Hertogenbosch, 007)
6	Text to Columns	Pipe 12" long,5,EUR; 5'10" tall,180,cm	Pipe 12" long | 5 | EUR; 5'10" tall | 180 | cm	one cell Pipe 12 long,5,EUR; one cell 5'10 tall,180,cm (a quote anywhere toggles quoting)
7	File › Open, From Text/CSV	first line sep=| (or sep=;, sep=,)	splits on that character and hides the line	shows a sep= row; a pipe file stays one column
8	typing and all import paths	1,5E3, -1,5E3, 12,3e4, 1,2e3	text	15000, -15000, 1230000, 12000 (a German 1,5E3, meaning 1500, becomes 15000)
8		1,000E3, 1,234E2	1000000, 123400	text
9	From Text/CSV	delimiter: "；" (U+FF1B)	–	cast as u8, so it never splits; a character whose low byte is 0x2C would split at commas

Cause

• From Text/CSV (crates/engine/src/cmd/extra.rs):
  • It picked tab or comma from the first line instead of File › Open's sniff_delimiter.
  • It truncated the delimiter as u8.
  • It treated anything starting with [ as JSON.
  • It wrote every value back through range.setValues, which parses each field as typed input, so formulas went live and number formats were lost.
• Apostrophe: CSV import and Text to Columns used parse_input (crates/core/src/parse.rs), where a leading ' is the typed-text prefix.
• Text to Columns quoting: split_delimited (crates/engine/src/cmd/data.rs) was a third tokenizer, next to csv.rs (File › Open) and parse_tsv (paste). It toggled quoting on any " and dropped the quote.
• Exponent: parse_plain checked digit groups on the whole number including the exponent, so 5E3 counted as a group of three.

Fix

• One tokenizer: CSV open, paste (parse_tsv) and Text to Columns now share one tokenizer in crates/xlsx/src/csv.rs (split_records / split_fields with a Delimited options struct). The copies in data.rs and edit.rs are gone.
  • A quote opens a quoted field only at the start of a field.
  • Doubled qualifiers inside quotes still read as one (Preserve doubled text qualifiers in Text to Columns #119).
• From Text/CSV:
  • Without a delimiter it detects the delimiter as File › Open does.
  • Both honour a sep=X first line (when the delimiter is detected) and hide it.
  • CsvOptions.delimiter is a char ('\0' means detect), so any character works; a delimiter param longer than one character is refused.
• From Text/CSV writes the cells read_csv produced, values with their number formats, never through typing. JSON strings are read like CSV fields. Both use the workbook's date system.
• JSON only when it is JSON: text is treated as JSON only when it parses as a JSON array of objects, or when the file is .json.
• Apostrophe: a new parse_imported in core keeps a leading apostrophe as text. Typing 'abc still forces text.
• Exponent: parse_plain checks thousands groups on the mantissa only.
• docs/file-format-parity.md: CSV rows and a revision row.

Not changed here:

• Import locale options (decimal and thousands separators, date order, file origin, trailing minus). With German, Indian or Chinese files those give en-US readings: 0,001 → 1, 04.03.2024 → 3 April. Those belong to the regional-format work in M12: Excel-style languages (12 languages, local formulas, regional formats) #191, M12: regional formats (typed and displayed numbers, dates and times) #447 and Data: Automatic Data Conversion options for typing, pasting and opening CSV (#506) #583.
• =1+1 in File › Open: it stays text. Excel's double-click open makes a formula, but keeping text looks deliberate (safety), so I left it as is.
• Pipe detection: | is not added to automatic detection. Excel's File › Open doesn't split on pipes either; sep=| covers those files.

Testing

• New tests:

  • core: thousands_separators_belong_to_the_mantissa, plus apostrophe cases for parse_imported.
  • xlsx: a_quote_inside_a_field_is_a_character, a_sep_line_names_the_delimiter, any_character_can_be_the_delimiter, plus apostrophe cases.
  • engine: text_to_columns_keeps_a_leading_apostrophe, text_to_columns_reads_a_quote_inside_a_field_as_a_character, from_text_csv_detects_the_delimiter_like_file_open, from_text_csv_keeps_field_text_and_number_formats, from_text_csv_reads_a_bracketed_header_as_csv.

• cargo xtask ci: fmt, clippy, test, assets, layers and wasm pass. parity and keytips --check differ only in line endings on this Windows checkout (core.autocrlf); main does the same here.

• Re-run of the Excel comparison: cells that differ from Excel 365, main → this branch. No count went up anywhere.

  Variant	Path	Before → after
  sep=, / sep=; / sep=|	File › Open	8 / 8 / 7 → 0 / 0 / 0
  sep=, / sep=; / sep=|	From Text/CSV	8 / 7 / 7 → 0 / 0 / 0
  apostrophes	File › Open / From Text/CSV	4 / 8 → 0 / 0
  exponents with separators	all paths	6 → 0
  quotes inside fields	Text to Columns	10 → 0
  ;, de-DE, cp1252, de-CH, fr files	From Text/CSV (against Excel de-DE)	38 / 48 / 12 / 10 / 12 → 2 / 20 / 2 / 1 / 4

  The remaining differences are the locale options above.

• Overlaps:

  • Data: Automatic Data Conversion options for typing, pasting and opening CSV (#506) #583, Data: “Remove trailing zeros after the decimal separator” conversion option (#656) #657 and M12: regional formats (typed and displayed numbers, dates and times) #447 also touch parse_plain in crates/core/src/parse.rs. This change is limited to the group check.
  • CSV: open Arabic (Windows-1256) and Cyrillic (Windows-1251) ANSI files #706 touches csv.rs; CsvOptions.delimiter is now a char.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SAgnkqJiy3v24Hp25dQLRj

[iluebbe]

Closing this pull request. The problem, Excel's measurements and a fix approach are now in #1446.