Reject empty sourced feature-oracle suite

[timsehn]

Summary

• Fail test/vc_oracle_feature_interaction_test.sh when zero oracle cases run (pass + fail == 0).
• Prevents empty or missing sourced family fragments from reporting success with 0 passed, 0 failed and exit 0.

Background

Found while testing #1781: emptying the five family sources left the runner at 0 passed, 0 failed with exit 0 because the only failure gate was $fail -gt 0. The both-empty guard in vc_oracle_common.sh only applies when an oracle call actually runs.

Test plan

• Reproduced: empty family files → 0 passed, 0 failed, exit 0 (before fix)
• After fix: empty family files → error message + exit 1
• Family files restored; guard does not trip when any cases execute

[github-actions]

Version-Control Performance Ceilings

Runs: median of 3 executions per benchmark, excluding fixture setup. The
working-set fixtures use 800 committed tables with 125 rows each
(100000 rows total); data-dirty cases update 1 row in 40
tables, and schema-dirty cases update 1 row plus add a column in 20 tables.
Branch tests use 300 branches, checkout uses a clean 400-table
branch switch over 100000 rows, and merge
tests use 100000-row tables with 2000 changed or conflicting
rows per side.

Benchmark	Median ms	Ceiling ms	Result
status_clean_many_tables	65	200	PASS
status_dirty_many_tables	68	200	PASS
diff_regular_working_one_table	61	150	PASS
diff_regular_working_many_tables	72	200	PASS
diff_stat_working_many_tables	72	200	PASS
diff_schema_working_many_tables	73	200	PASS
branch_list_many_branches	20	100	PASS
branch_create_delete	35	100	PASS
checkout_branch_clean	117	200	PASS
merge_data_no_conflicts	42	150	PASS
merge_schema_no_conflicts	21	100	PASS
merge_data_conflicts	78	250	PASS
merge_data_conflicts_with_resolve	78	250	PASS

[github-actions]

Sysbench-Style Benchmark: Doltlite vs SQLite

In-Memory

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	24,248	30,175	1.24
oltp_range_select	10,417	14,074	1.35
oltp_sum_range	9,482	12,766	1.35
oltp_order_range	2,539	2,962	1.17
oltp_distinct_range	3,565	4,165	1.17
oltp_index_scan	3,829	5,518	1.44
select_random_points	10,217	11,659	1.14
select_random_ranges	2,980	4,073	1.37
covering_index_scan	4,099	4,335	1.06
groupby_scan	29,512	34,405	1.17
index_join	5,918	8,562	1.45
index_join_scan	3,422	4,877	1.43
types_table_scan	1,064,198	1,404,119	1.32
table_scan	1,233,645	1,509,373	1.22
oltp_read_only	104,043	126,680	1.22
Average			1.27

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	176,334	249,628	1.42
oltp_insert	15,226	29,018	1.91
oltp_update_index	50,264	93,839	1.87
oltp_update_non_index	34,155	60,545	1.77
oltp_delete_insert	45,861	74,438	1.62
oltp_write_only	22,230	45,991	2.07
types_delete_insert	23,923	40,856	1.71
oltp_read_write	66,476	113,643	1.71
Average			1.76

File-Backed

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	91,895	53,860	0.59
oltp_range_select	17,482	16,603	0.95
oltp_sum_range	16,943	15,490	0.91
oltp_order_range	3,431	3,405	0.99
oltp_distinct_range	4,395	4,532	1.03
oltp_index_scan	11,024	8,354	0.76
select_random_points	17,411	14,477	0.83
select_random_ranges	10,001	6,612	0.66
covering_index_scan	11,448	6,975	0.61
groupby_scan	30,236	35,097	1.16
index_join	10,176	10,629	1.04
index_join_scan	4,290	5,272	1.23
types_table_scan	1,052,254	1,382,256	1.31
table_scan	1,201,516	1,498,075	1.25
oltp_read_only	203,066	162,604	0.80
Average			0.94

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	188,471	268,176	1.42
oltp_insert	24,711	35,690	1.44
oltp_update_index	77,050	115,824	1.50
oltp_update_non_index	60,884	80,137	1.32
oltp_delete_insert	70,960	95,159	1.34
oltp_write_only	50,625	64,401	1.27
types_delete_insert	40,438	53,310	1.32
oltp_read_write	96,267	131,620	1.37
Average			1.37

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	45,858	53,437	1.17
oltp_range_select	12,896	16,751	1.30
oltp_sum_range	12,599	15,930	1.26
oltp_order_range	3,165	3,588	1.13
oltp_distinct_range	4,161	4,643	1.12
oltp_index_scan	6,646	8,302	1.25
select_random_points	13,227	14,509	1.10
select_random_ranges	5,503	6,517	1.18
covering_index_scan	7,017	7,042	1.00
groupby_scan	29,971	35,060	1.17
index_join	7,636	10,285	1.35
index_join_scan	3,900	5,232	1.34
types_table_scan	1,070,868	1,392,281	1.30
table_scan	1,251,433	1,501,689	1.20
oltp_read_only	134,372	165,702	1.23
Average			1.21

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert_ac	23,041	90,482	3.93
oltp_insert_ac	27,010	103,498	3.83
oltp_update_index_ac	28,538	117,537	4.12
oltp_update_non_index_ac	23,219	91,702	3.95
oltp_delete_insert_ac	24,921	107,351	4.31
oltp_write_only_ac	25,397	97,710	3.85
types_delete_insert_ac	23,384	93,936	4.02
oltp_read_write_ac	30,003	106,104	3.54
Average			3.94

100000 rows, median of 5 invocations per test; autocommit writes use 9, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (2.5x individual, 2x average; autocommit writes: 10x / 5x)

All tests within ceilings.

[github-actions]

Sysbench-Style Benchmark (composite PK): Doltlite vs SQLite

Companion to the classic Sysbench-Style Benchmark. Every workload here
runs against tables with a 2-column INTEGER PRIMARY KEY(a, b) WITHOUT ROWID.
Individual ratios gated at 2.5×; section averages gated at 2×. Autocommit writes use 10× / 5×.

In-Memory

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	26,046	30,074	1.15
oltp_range_select	15,281	17,382	1.14
oltp_sum_range	13,421	16,648	1.24
oltp_order_range	2,869	3,213	1.12
oltp_distinct_range	3,655	4,128	1.13
oltp_index_scan	3,617	4,563	1.26
select_random_points	21,297	24,282	1.14
select_random_ranges	5,770	6,608	1.15
covering_index_scan	3,378	3,323	0.98
groupby_scan	29,867	33,444	1.12
index_join	6,270	8,044	1.28
index_join_scan	3,331	4,805	1.44
types_table_scan	880,702	1,029,585	1.17
table_scan	1,012,401	1,133,601	1.12
oltp_read_only	113,973	135,192	1.19
Average			1.18

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	189,990	261,147	1.37
oltp_insert	14,952	27,292	1.83
oltp_update_index	53,216	89,710	1.69
oltp_update_non_index	39,927	63,339	1.59
oltp_delete_insert	38,990	72,988	1.87
oltp_write_only	21,457	44,363	2.07
types_delete_insert	25,810	40,772	1.58
oltp_read_write	75,001	118,056	1.57
Average			1.70

File-Backed

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	90,373	51,891	0.57
oltp_range_select	22,278	19,383	0.87
oltp_sum_range	19,903	19,230	0.97
oltp_order_range	3,593	3,481	0.97
oltp_distinct_range	4,407	4,421	1.00
oltp_index_scan	10,355	7,360	0.71
select_random_points	28,396	27,009	0.95
select_random_ranges	11,980	9,088	0.76
covering_index_scan	10,105	6,010	0.59
groupby_scan	30,649	33,915	1.11
index_join	9,831	10,036	1.02
index_join_scan	4,022	5,120	1.27
types_table_scan	876,261	1,037,197	1.18
table_scan	1,016,812	1,131,172	1.11
oltp_read_only	205,909	165,840	0.81
Average			0.93

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	249,312	331,640	1.33
oltp_insert	29,553	49,547	1.68
oltp_update_index	164,230	181,294	1.10
oltp_update_non_index	137,532	129,351	0.94
oltp_delete_insert	135,020	141,923	1.05
oltp_write_only	97,610	102,477	1.05
types_delete_insert	89,154	93,517	1.05
oltp_read_write	150,632	188,662	1.25
Average			1.18

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	47,004	52,583	1.12
oltp_range_select	18,102	19,549	1.08
oltp_sum_range	16,208	19,450	1.20
oltp_order_range	3,328	3,470	1.04
oltp_distinct_range	4,077	4,419	1.08
oltp_index_scan	6,198	7,427	1.20
select_random_points	24,260	26,630	1.10
select_random_ranges	8,177	9,076	1.11
covering_index_scan	5,915	6,011	1.02
groupby_scan	30,246	33,841	1.12
index_join	7,969	10,019	1.26
index_join_scan	3,739	5,111	1.37
types_table_scan	877,016	1,037,647	1.18
table_scan	1,015,796	1,125,109	1.11
oltp_read_only	145,731	166,819	1.14
Average			1.14

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert_ac	28,685	79,312	2.76
oltp_insert_ac	33,307	93,604	2.81
oltp_update_index_ac	31,702	105,067	3.31
oltp_update_non_index_ac	29,026	89,581	3.09
oltp_delete_insert_ac	29,379	114,381	3.89
oltp_write_only_ac	29,852	91,113	3.05
types_delete_insert_ac	28,014	80,492	2.87
oltp_read_write_ac	36,991	106,075	2.87
Average			3.08

100000 rows, median of 5 invocations per test; autocommit writes use 9, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (2.5x individual, 2x average; autocommit writes: 10x / 5x)

All tests within ceilings.

[github-actions]

Sysbench-Style Benchmark (BLOB PK): Doltlite vs SQLite

Companion to the classic Sysbench-Style Benchmark. Every workload here
runs against tables with a 16-byte big-endian BLOB PRIMARY KEY.
Individual ratios gated at 2.5×; section averages gated at 2×. Autocommit writes use 10× / 5×.

In-Memory

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	30,694	35,045	1.14
oltp_range_select	13,526	14,520	1.07
oltp_sum_range	12,031	14,139	1.18
oltp_order_range	3,156	3,296	1.04
oltp_distinct_range	4,126	4,428	1.07
oltp_index_scan	4,545	6,104	1.34
select_random_points	17,942	20,480	1.14
select_random_ranges	4,308	5,443	1.26
covering_index_scan	4,556	4,715	1.03
groupby_scan	34,108	37,178	1.09
index_join	6,767	9,478	1.40
index_join_scan	4,225	5,681	1.34
types_table_scan	1,133,964	1,334,429	1.18
table_scan	1,385,454	1,476,627	1.07
oltp_read_only	121,802	137,708	1.13
Average			1.17

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	239,832	342,676	1.43
oltp_insert	20,468	40,282	1.97
oltp_update_index	70,007	132,478	1.89
oltp_update_non_index	50,258	86,509	1.72
oltp_delete_insert	50,251	106,031	2.11
oltp_write_only	28,852	64,575	2.24
types_delete_insert	33,174	53,735	1.62
oltp_read_write	83,834	137,236	1.64
Average			1.83

File-Backed

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	114,709	63,670	0.56
oltp_range_select	22,155	17,306	0.78
oltp_sum_range	20,881	17,129	0.82
oltp_order_range	4,012	3,630	0.90
oltp_distinct_range	4,989	4,801	0.96
oltp_index_scan	12,705	9,106	0.72
select_random_points	26,508	23,482	0.89
select_random_ranges	12,812	8,294	0.65
covering_index_scan	13,354	7,858	0.59
groupby_scan	35,077	37,415	1.07
index_join	11,191	11,518	1.03
index_join_scan	5,272	6,049	1.15
types_table_scan	1,150,856	1,326,891	1.15
table_scan	1,457,660	1,492,854	1.02
oltp_read_only	243,093	178,000	0.73
Average			0.87

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	258,335	364,114	1.41
oltp_insert	31,406	52,112	1.66
oltp_update_index	103,599	167,097	1.61
oltp_update_non_index	79,065	108,730	1.38
oltp_delete_insert	83,101	134,297	1.62
oltp_write_only	55,847	88,652	1.59
types_delete_insert	53,197	72,388	1.36
oltp_read_write	112,035	161,061	1.44
Average			1.51

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	58,157	64,235	1.10
oltp_range_select	16,962	17,545	1.03
oltp_sum_range	15,600	17,185	1.10
oltp_order_range	3,564	3,627	1.02
oltp_distinct_range	4,559	4,776	1.05
oltp_index_scan	7,846	9,058	1.15
select_random_points	21,336	23,260	1.09
select_random_ranges	7,178	8,251	1.15
covering_index_scan	8,236	7,761	0.94
groupby_scan	34,383	37,355	1.09
index_join	8,934	11,387	1.27
index_join_scan	4,705	6,131	1.30
types_table_scan	1,155,099	1,330,976	1.15
table_scan	1,467,224	1,492,648	1.02
oltp_read_only	162,201	177,868	1.10
Average			1.10

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert_ac	16,378	65,227	3.98
oltp_insert_ac	19,303	86,042	4.46
oltp_update_index_ac	19,958	95,698	4.79
oltp_update_non_index_ac	15,808	76,341	4.83
oltp_delete_insert_ac	17,914	89,738	5.01
oltp_write_only_ac	19,323	88,885	4.60
types_delete_insert_ac	16,894	78,419	4.64
oltp_read_write_ac	23,841	93,350	3.92
Average			4.53

100000 rows, median of 5 invocations per test; autocommit writes use 9, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (2.5x individual, 2x average; autocommit writes: 10x / 5x)

All tests within ceilings.

[github-actions]

Sysbench-Style Benchmark (TEXT PK): Doltlite vs SQLite

Companion to the classic Sysbench-Style Benchmark. Every workload here
runs against tables with a 32-char hex TEXT PRIMARY KEY (UUID-shaped).
Individual ratios gated at 2.5×; section averages gated at 2×. Autocommit writes use 10× / 5×.

In-Memory

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	32,283	39,531	1.22
oltp_range_select	14,902	15,499	1.04
oltp_sum_range	12,516	14,873	1.19
oltp_order_range	3,107	3,248	1.05
oltp_distinct_range	4,110	4,452	1.08
oltp_index_scan	4,805	6,872	1.43
select_random_points	19,215	21,869	1.14
select_random_ranges	4,085	5,394	1.32
covering_index_scan	4,884	4,669	0.96
groupby_scan	31,526	35,509	1.13
index_join	7,005	10,036	1.43
index_join_scan	5,136	5,818	1.13
types_table_scan	1,252,003	1,390,652	1.11
table_scan	1,537,978	1,490,561	0.97
oltp_read_only	132,365	145,741	1.10
Average			1.15

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	232,118	366,617	1.58
oltp_insert	22,427	40,743	1.82
oltp_update_index	78,950	140,375	1.78
oltp_update_non_index	51,673	90,003	1.74
oltp_delete_insert	56,254	108,942	1.94
oltp_write_only	30,902	64,481	2.09
types_delete_insert	34,807	57,684	1.66
oltp_read_write	95,723	148,671	1.55
Average			1.77

File-Backed

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	101,599	64,566	0.64
oltp_range_select	22,731	18,087	0.80
oltp_sum_range	21,378	17,756	0.83
oltp_order_range	3,965	3,598	0.91
oltp_distinct_range	5,069	4,820	0.95
oltp_index_scan	12,152	9,398	0.77
select_random_points	28,110	25,593	0.91
select_random_ranges	11,568	7,810	0.68
covering_index_scan	13,157	7,386	0.56
groupby_scan	33,574	36,006	1.07
index_join	11,973	11,759	0.98
index_join_scan	6,282	6,287	1.00
types_table_scan	1,219,126	1,369,377	1.12
table_scan	1,543,142	1,496,864	0.97
oltp_read_only	236,611	186,301	0.79
Average			0.87

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	247,644	392,896	1.59
oltp_insert	73,449	52,925	0.72
oltp_update_index	119,147	176,224	1.48
oltp_update_non_index	91,224	116,029	1.27
oltp_delete_insert	94,271	138,047	1.46
oltp_write_only	90,913	86,793	0.95
types_delete_insert	56,834	76,211	1.34
oltp_read_write	134,088	171,664	1.28
Average			1.26

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	55,882	65,229	1.17
oltp_range_select	17,109	18,047	1.05
oltp_sum_range	16,197	17,642	1.09
oltp_order_range	3,457	3,630	1.05
oltp_distinct_range	4,541	4,778	1.05
oltp_index_scan	7,702	9,321	1.21
select_random_points	22,042	25,782	1.17
select_random_ranges	6,971	7,994	1.15
covering_index_scan	8,801	7,457	0.85
groupby_scan	33,266	36,395	1.09
index_join	9,578	11,785	1.23
index_join_scan	5,676	6,275	1.11
types_table_scan	1,235,451	1,378,096	1.12
table_scan	1,536,263	1,489,848	0.97
oltp_read_only	166,255	184,099	1.11
Average			1.09

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert_ac	20,731	78,453	3.78
oltp_insert_ac	26,009	91,743	3.53
oltp_update_index_ac	26,635	114,894	4.31
oltp_update_non_index_ac	23,318	92,988	3.99
oltp_delete_insert_ac	25,071	104,399	4.16
oltp_write_only_ac	25,961	94,286	3.63
types_delete_insert_ac	20,913	92,628	4.43
oltp_read_write_ac	29,394	111,823	3.80
Average			3.96

100000 rows, median of 5 invocations per test; autocommit writes use 9, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (2.5x individual, 2x average; autocommit writes: 10x / 5x)

All tests within ceilings.

[itoqa]

Commit: c7ccc12: 5 test cases ran, 5 passed ✅.

Summary

Coverage spans the product’s feature-interaction logic across merge, history, schema, query, and stress behaviors, including successful end-to-end comparisons, detection of mismatched results, and safeguards for empty inputs. The exercised paths show healthy handling of both normal operation and edge conditions, including correct failure signaling when no cases or incorrect results are present.

Safe to merge — the exercised behavior shows no PR-attributable regressions or new failures, and the observed outcomes are consistent with the expected handling of successful, mismatched, and empty inputs. No merge-blocking application issue was identified.

Tests run by Ito

Result	Severity	Type	Description
✅	—	Accounting	The intact feature-interaction oracle suite executed 752 cases with 752 passed, 0 failed, no empty-suite diagnostic, and exit status 0 using real DoltLite and Dolt binaries.
✅	—	Accounting	The runner correctly treated one mismatched oracle comparison as an executed failure rather than an empty suite: it reported 0 passed and 1 failed, named accounting_mismatch_case, omitted the empty-suite diagnostic, and exited with status 1.
✅	—	Empty	The empty disposable suite reported 0 passed and 0 failed, printed the no-oracle-cases diagnostic, and exited with status 1 as expected.
✅	—	Empty	The CI-style wrapper propagated the empty suite's exit status 1, retained the 0 passed and 0 failed summary and no-oracle-cases diagnostic, and recorded the raw status as 1.
✅	—	Family	The intact merge, history, schema, query, and stress families were sourced and executed, reaching aggregate validation with 752 counted oracle cases. The run's 0 passed and 752 failed result correctly produced exit status 1 for the observed oracle failures, while the empty-suite diagnostic was not emitted.

Tip

Reply with @itoqa to send us feedback on this test run.