This document outlines the standard operating procedure for all AI agents contributing to this project. Adhering to these conventions ensures a systematic, verifiable, and robust development process.
When you suggest commands in your regular response (NOT inside a SEARCH/REPLACE block), format them inside bash blocks:
cmd1
cmd2These commands are then presented to the user, who can accept them for execution, and the results are returned to you.
Key tools available for you to suggest:
rg(ripgrep): Your primary tool for fast, powerful code searching. Use it to find definitions, locate files, or understand code relationships.tree: List files to understand directory structures.head: Inspect the beginning of files to quickly understand their structure and content.ls: Check file sizes to determine if they should be read or inspected.psql: Run arbitrary SQL for debugging or inspection (e.g.,echo 'SELECT * FROM sql_saga.era;' | psql -d sql_saga_regress).
For file system operations and large-scale edits, prefer suggesting shell commands over generating SEARCH/REPLACE blocks where appropriate. This is faster and more efficient.
- Use
rmto delete files andgit mvto move or rename them. - For simple content replacement (e.g., replacing an entire file's contents),
echo "new content" > filenamecan be used instead of a largeSEARCH/REPLACEblock. - For large-scale, repetitive search-and-replace operations across multiple files, powerful tools like
ruplacerandrenamerare available and should be used.
- All shell commands must be single-line. The execution environment splits commands by newline.
- Do not suggest multi-line
echocommands. To write multi-line content to a file,echoa single-line string with\ncharacters to a temporary file first.
- C99 Compliance: All C code must be compatible with the C99 standard. The build process uses the
-Wdeclaration-after-statementflag, which will generate a warning if variable declarations are mixed with code.- Rule: Declare all variables at the beginning of a block (immediately after a
{). Do not mix declarations and executable statements.
- Rule: Declare all variables at the beginning of a block (immediately after a
- PostgreSQL Coding Conventions: Adhere to the formatting and naming conventions outlined in the official PostgreSQL Documentation. This includes conventions for variable names, function names, and code layout.
- You must never add an
Ebefore the echo string, as it outputs. So instead ofecho E'...\n...'you must useecho '...\n...', the\nworks regardless.
-
Function/Procedure Definitions:
- Use the function/procedure name in the literal string quote for the body (e.g.,
AS $my_function_name$). - Specify
LANGUAGE plpgsql(or other) before the body. - Use the long form for parameters for documentation clarity (e.g.,
param_name param_type). - Example:
CREATE FUNCTION public.example(email text) RETURNS void LANGUAGE plpgsql AS $example$ BEGIN -- Use function_name.parameter_name for clarity if needed, e.g., example.email SELECT * FROM some_table st WHERE st.email = example.email; END; $$;
- Use the function/procedure name in the literal string quote for the body (e.g.,
-
Function Calls: For calls with 3+ arguments, use named arguments (e.g.,
arg1 => val1). -
String Literals for
format():- Always prefer dollar-quoting (e.g.,
format($$ ... $$)) for the main dynamic SQL string. This avoids having to escape single quotes inside the SQL. - Nesting: When nesting dollar-quoted strings (e.g., a dynamic SQL string that itself contains another
format()call), use named dollar quotes for the outer string to avoid conflicts. The convention is to use a descriptive name like$SQL$or$jsonb_expr$. This is especially common inside function bodies, which already use a named quote (e.g.,$function_name$). - For
format()calls with multiple parameters, especially if parameters repeat, use numbered placeholders for clarity:%1$Ifor the 1st parameter as an identifier,%2$Lfor the 2nd as a literal,%3$sfor the 3rd as a plain string, etc.- Example:
format($$Testing %3$s, %2$s, %1$s$$, 'one' /* %1 */, 'two' /* %2 */, 'three' /* %3 */);
- Prefer parameter binding with
EXECUTE ... USINGfor large arrays or values rather than interpolating with%Lwhere possible.
Good
-- Dollar-quoted format() string; identifier and literal are numbered; batch array is passed via USING. EXECUTE format($$ UPDATE public.%1$I AS dt SET last_completed_priority = %2$L WHERE dt.row_id = ANY($1) AND dt.action = 'skip' $$, v_data_table_name /* %1$I */, v_step.priority /* %2$L */) USING p_batch_row_ids;
Also good (embedding quotes safely without doubling them)
EXECUTE format($$ UPDATE public.%1$I AS dt SET error = COALESCE(dt.error, '{}'::jsonb) || jsonb_build_object('status_code', 'Provided status_code not found and no default available'), state = 'error' WHERE dt.row_id = ANY($1) AND dt.status_code IS NOT NULL $$, v_data_table_name) USING p_batch_row_ids;Also good (nested dollar-quoting)
-- The outer string uses a named quote ($jsonb_expr$) to avoid conflicting with the inner $$ v_error_sql := $jsonb_expr$ jsonb_build_object('latitude', CASE ... THEN format($$Value %1$s out of range$$, lat_val) ... END) $jsonb_expr$;
Avoid
-- Hard to read and easy to break: manual '' escaping inside single-quoted format string EXECUTE format('UPDATE public.%s SET note = ''it'''''s broken'' WHERE id = %s', tbl, id);
Notes
- Use
%Ifor identifiers,%Lfor SQL literals, and%sfor raw string insertion. - Keep the SQL readable by aligning numbered placeholders with inline comments that show which parameter they refer to.
- Always prefer dollar-quoting (e.g.,
-
Table Aliases: Prefer explicit
ASfor table aliases, e.g.,FROM my_table AS mt. For common data table aliases in import procedures,AS dtis preferred. -
Temporal Logic: When writing conditions involving time, always order the components chronologically for readability (e.g.,
start <= point AND point < end). Avoid non-chronological forms likepoint >= start. -
Temporary Table Management:
- To ensure procedures are idempotent and to avoid noisy
NOTICEmessages in logs, use the following pattern to clean up temporary tables at the beginning of a procedure:-- The explicit 'pg_temp.' schema ensures we only check for session-local tables. IF to_regclass('pg_temp.my_temp_table') IS NOT NULL THEN DROP TABLE my_temp_table; END IF; CREATE TEMP TABLE my_temp_table (...) ON COMMIT DROP;
- This pattern has several advantages:
- Silent Operation: It avoids the
NOTICE: table "..." does not exist, skippingmessage thatDROP TABLE IF EXISTSwould generate on the first run. - Co-location: It keeps the cleanup logic directly beside the creation logic, improving readability.
- Debuggability: If the code does not behave as expected, then a test running in the same transaction can inspect those temporary tables to determine where the faulty logic lies.
- Silent Operation: It avoids the
- To ensure procedures are idempotent and to avoid noisy
Makefileand Source Files: TheMakefileuses a glob pattern (src/[0-9][0-9]_*.sql) to automatically discover and concatenate SQL source files. When adding a new feature, simply create a new numbered.sqlfile in thesrc/directory (e.g.,src/31_new_feature.sql). It will be included in the build automatically without any need to edit theMakefile.
x_idis a foreign key to tablexx_identis an external identifier, not originating from the databasex_atis a TIMESTAMPTZ (with timezone)x_onis a DATE- Temporal Columns: To ensure consistency and intuitive use, all temporal periods must follow the
[)semantic (inclusive start, exclusive end). This aligns with the default behavior of PostgreSQL's native range types (e.g.,daterange) and ensures compatibility with operators likeOVERLAPS.- Column names for the start of a period must be named
valid_from(or a similarly descriptive name ending in_from). - Column names for the end of a period must be named
valid_until(or a similarly descriptive name ending in_until). - Metadata columns in
sql_sagathat store these column names will be namedvalid_from_column_nameandvalid_until_column_name.
- Column names for the start of a period must be named
With each new message from the user, especially one containing test results or command output, you must clear all previous assumptions and hypotheses. Your analysis must be based only on the new information provided in the latest message. Do not refer back to your own previous hypotheses if they have been contradicted by data. Treat each interaction as a fresh start to the analysis cycle.
All development work is an iterative process of forming hypotheses and verifying them with real-world data. The core principle is to replace assumption with verification.
All development work, especially bug fixing, must follow a rigorous, hypothesis-driven cycle to ensure progress is verifiable, correct, and does not waste time on flawed assumptions. A task is not complete until the final step of this cycle has been successfully executed. A hypothesis is not confirmed until it is supported by direct, empirical observation.
This cycle is designed to optimize for the shortest overall development time, not just the shortest "happy path". In a complex system, the probability of any given change being perfectly correct on the first attempt is low. A process that optimizes for the happy path (e.g., CHANGE -> CLEANUP -> DONE) results in a very long and costly sad path when the change is incorrect (INCORRECT CHANGE -> CLEANUP -> ERROR -> RE-INVESTIGATE -> ...).
Our process deliberately adds verification steps (CHANGE -> VERIFY -> CLEANUP -> DONE), which makes the happy path slightly longer. However, it makes the sad path significantly shorter (INCORRECT CHANGE -> VERIFY -> RE-HYPOTHESIZE -> ...). By catching errors early and cheaply, we minimize wasted effort and ensure the average time to a correct solution is much lower.
This cycle is designed to optimize for the shortest overall development time, not just the shortest "happy path". In a complex system, the probability of any given change being perfectly correct on the first attempt is low. A process that optimizes for the happy path (e.g., CHANGE -> CLEANUP -> DONE) results in a very long and costly sad path when the change is incorrect (INCORRECT CHANGE -> CLEANUP -> ERROR -> RE-INVESTIGATE -> ...).
Our process deliberately adds verification steps (CHANGE -> VERIFY -> CLEANUP -> DONE), which makes the happy path slightly longer. However, it makes the sad path significantly shorter (INCORRECT CHANGE -> VERIFY -> RE-HYPOTHESIZE -> ...). By catching errors early and cheaply, we minimize wasted effort and ensure the average time to a correct solution is much lower.
- Action: Before making any code changes, clearly state your hypothesis about the root cause of the problem in
tmp/journal.md. This creates a locally persistent log of your thought process. - Example: "Hypothesis: The import job hangs because the batch selection query is inefficient and confuses the query planner."
- Action: Ensure a test case exists that isolates the bug. This can be an existing pg_regress test or a temporary SQL script (
tmp/debug_*.sql) that demonstrates the failure (e.g., a query that is slow or returns incorrect data).
- Action: Before proposing a permanent fix, create a temporary, non-destructive script (e.g.,
tmp/verify_fix.sql) to test your proposed change. This script must use tools likeEXPLAIN (ANALYZE, BUFFERS)or read-onlySELECTqueries to gather performance data or check logic without altering the database state. - Example: Create a script that runs
EXPLAIN ANALYZEon a simplified query to prove it is fast and returns the expected number of rows.
- Action: Suggest the user run the verification script from Step 3. Do not proceed until you have observed the results. This step is mandatory.
- Standard Command:
psql -d sql_saga_regress < tmp/verify_fix.sql
- Action: Carefully inspect the output from the verification script.
- If Successful: The prototype confirms the hypothesis (e.g., the new query is fast). You can now proceed to propose the permanent change.
- If Unsuccessful: The hypothesis was incorrect. The prototype failed (e.g., the query was still slow or returned zero rows). Analyze the new data, update
tmp/journal.mdwith a new hypothesis, and return to Step 1.
- Action: Only after the prototype has been successfully verified in Step 5, propose the specific code changes (using
SEARCH/REPLACEblocks) for the permanent files (e.g., migrations, functions). State the expected outcome. - Example: "This change replaces the complex query with the simplified version that was verified to be fast in
tmp/verify_fix.sql."
- Action: After the user applies the permanent changes, request that they run the relevant test suite to ensure the fix works and has not introduced any regressions.
- Standard Command Format: Always use the following command structure to run tests. This ensures the extension is installed before testing and that any failures are immediately diffed for analysis.
- Debugging Tip: For complex SQL or C functions, prefer adding
RAISE DEBUGstatements overRAISE NOTICE. In the corresponding test file, temporarily wrap the relevant commands withSET client_min_messages TO DEBUG;andRESET client_min_messages;. This provides targeted diagnostic output without permanently cluttering the test results. - Command:
make install && make test ...; make diff-fail-all - Test Output Review: You must never propose
SEARCH/REPLACEblocks forexpected/*.outfiles. If a test fails due to intended changes, instruct the user to review and accept the new output interactively by runningmake diff-fail-all vim. - Usage:
- To run all tests:
make install && make test; make diff-fail-all - To run fast tests (excluding benchmarks):
make install && make test fast; make diff-fail-all - To run specific tests:
make install && make test TESTS="01_install 51_quoted_identifiers"; make diff-fail-all
- To run all tests:
- Note on Self-Contained Tests: The
01_install.sqltest handles setup required by older tests. Newer tests should be self-contained.- Include
\i sql/include/test_setup.sqlat the beginning to create necessary roles and permissions. - Include
\i sql/include/test_teardown.sqlat the end to clean up. - For tests that verify transactional behavior, the
BEGIN/ROLLBACKblock should wrap only the test logic, not the setup/teardown of non-transactional objects like roles. - When running a specific older test,
01_installmust still be run first. - Test Isolation Patterns: To ensure tests are robust and independent, use
SAVEPOINTs to manage transactional state within a test file.- For successful test cases: Wrap each distinct scenario in a
SAVEPOINT <name>andRELEASE SAVEPOINT <name>block. This creates a nested transaction for the scenario. If it succeeds, its changes are committed to the main transaction.SAVEPOINT scenario_A; -- ... test logic for scenario A ... RELEASE SAVEPOINT scenario_A; - For expected failures: To test a command that is expected to raise an
ERROR, wrap it in aSAVEPOINTandROLLBACK TO SAVEPOINTblock. This allows the test to capture and display the raw error message without aborting the entire test script, enabling subsequent verification steps. This is the preferred pattern for negative testing.SAVEPOINT expect_error; -- This command is expected to fail SELECT my_buggy_function(); ROLLBACK TO SAVEPOINT expect_error; -- This command will now execute successfully SELECT 'The transaction is still active';
- For successful test cases: Wrap each distinct scenario in a
- Include
- Debugging Tip: For complex SQL or C functions, prefer adding
The temporal_merge planner includes a permanent trace column in its output (temporal_merge_plan). This is a critical architectural feature for debugging and must be handled with a strict protocol.
-
Trace is Permanent and Performant: The
tracecolumn must never be removed from the planner's output or from any regression test that queries it. Its presence, even asNULL, is required for consistency and to allow for easy activation of tracing. The implementation is designed to have near-zero performance impact when tracing is disabled. The planner uses two patterns to achieve this:- Short-circuiting
||operator: When tracing is off, the trace seed isNULL. Subsequenttrace || jsonb_build_object(...)operations are short-circuited by the PostgreSQL planner, which knows thatNULL || anythingisNULLand therefore does not execute the expensivejsonb_build_objectcall. CASEstatements: In other areas, an explicitCASE WHEN ...statement is used, which also guarantees the expensive trace-building logic is skipped. Both approaches ensure that the architectural decision to have a permanenttracecolumn incurs no runtime overhead when tracing is not active.
- Short-circuiting
-
Trace is Toggled via GUC: Tracing is activated for a specific operation by setting the session-level GUC:
SET sql_saga.temporal_merge.enable_trace = true;. Whenfalse(the default), the planner populates thetracecolumn withNULL. -
Debugging Protocol: When a
temporal_mergetest fails and the root cause is not immediately obvious from the plan output, the following steps are mandatory: a. Enable Trace: In the test file, wrap the failingCALL sql_saga.temporal_merge(...)withSET sql_saga.temporal_merge.enable_trace = true;andSET sql_saga.temporal_merge.enable_trace = false;. b. Isolate Trace in Diffs: Run the test and inspect the diff in trace to gain understanding. The expected has NULL in trace, and the actual has the trace due to the GUC variable setting. TheSELECTs must remain unchanged, such that when the the issue is fixed and the trace is disabled again it will mach. This technique ensures that if the plan's data is wrong, the test will fail, and thediffoutput will contain the full trace from the actual plan for analysis. -
Trace Aggregation: When multiple atomic time segments are coalesced into a single final plan operation, their individual traces must be aggregated into a
jsonbarray usingjsonb_agg. This provides a complete diagnostic record of all atomic segments that were merged into the final operation. The trace for each atomic segment includes the Allen Interval Relation (s_t_relation) between the covering source and target rows, providing detailed insight into the initial temporal interaction that created the segment.
- Action: Only after the fix has been successfully validated in Step 7, update
todo.mdto move the task to a "done" state (e.g.,[x]).
- Embrace Falsifiability: Treat every hypothesis and plan as provisional until proven by direct, empirical observation. Frame plans as "Current Plan" or "Next Step," not "Final Plan." The goal is to be prepared to be wrong and to allow evidence to guide the development process. Optimism should be rooted in the rigor of the process itself, not in the assumed correctness of any single solution. This mindset is the primary defense against hubris and wasted effort.
- Fail Fast:
- Functionality that is expected to work should fail immediately and clearly if an unexpected state or error occurs.
- Do not mask or work around problems; instead, provide sufficient error or debugging information to facilitate a solution. This is crucial for maintaining system integrity and simplifying troubleshooting, especially in backend processes and SQL procedures.
- Declarative Transparency:
- Where possible, store the inputs and intermediate results of complex calculations directly on the relevant records. This makes the system's state self-documenting and easier to debug, inspect, and trust, rather than relying on dynamic calculations that can appear magical.
- Example: The
temporal_mergeplanner was refactored to separate ephemeral and non-ephemeral data into distinct payloads (ephemeral_payload,data_payload). This makes the core logic clearer, as the complex coalescing operations now work on a cleandata_payload, instead of requiring repeated, on-the-fly stripping of ephemeral keys. This separation of concerns makes the algorithm's intent explicit. - Example: The
time_points_unifiedCTE is the key to this. It uses aFIRST_VALUEwindow function to establish a single, authoritativecorr_entfor each entity, which is then used as a stable partitioning key throughout the rest of the plan. This prevents the timeline for a single conceptual entity from being fragmented. - Example: The planner's logic for determining if a source row represents a "new" entity is a critical declarative step. The
new_entflag is derived from information that is already available (stable_key IS NULLandtarget_entity_exists), rather than being passed down through complex state. This makes the logic for partitioning timelines robust and correct across multiple batches.
When a task is complex or has a history of regressions (e.g., the rename_following trigger), a more rigorous, data-driven protocol is required to avoid speculative fixes. This protocol is a practical application of the "Observe first, then change" principle.
-
Establish a Baseline (Observe):
- Action: Before writing any new logic, instrument the existing, passing code with diagnostic logging (
RAISE NOTICE). - Goal: Capture the exact inputs, outputs, and event sequences that occur during the relevant tests. This baseline data serves as a definitive specification for the new logic. The tests are expected to fail due to the new logging output.
- Action: Before writing any new logic, instrument the existing, passing code with diagnostic logging (
-
Verify New Logic in Read-Only Mode (Verify):
- Action: Implement the new logic, but keep it "read-only." It should calculate its decisions but only log them, without altering the program's control flow (e.g., no early
RETURNstatements). - Goal: Run the tests again. Analyze the new logs to compare the baseline data against the logic's decisions. This step proves the logic is correct against real-world test data before it is activated.
- Action: Implement the new logic, but keep it "read-only." It should calculate its decisions but only log them, without altering the program's control flow (e.g., no early
-
Activate the Verified Logic (Implement):
- Action: Only after the logic has been verified in read-only mode, remove all diagnostic logging and enable the new control flow (e.g., add the early
RETURNstatement). - Goal: Run the tests a final time to confirm that the now-active logic works as intended and that no regressions have been introduced.
- Action: Only after the logic has been verified in read-only mode, remove all diagnostic logging and enable the new control flow (e.g., add the early
This protocol transforms debugging from a cycle of "guess-and-check" into a methodical, scientific process of data gathering and verification, dramatically reducing the number of failed attempts.
For complex, multi-step tasks like major refactoring, a detailed plan should be maintained in tmp/journal.md. This file outlines the sequence of steps, the actions required for each step, and the expected outcome. It serves as a roadmap for the task, ensuring a systematic approach. The journal should be cleared when you begin on a new major todo item. To clear the journal, you must propose the shell command echo -n > tmp/journal.md (or the relevant journal file). You must not attempt to clear it using a SEARCH/REPLACE block, as its contents are ephemeral and cannot be reliably searched. This helps for bug fixing and iterative development by logging hypotheses and outcomes, providing a low-level history of the debugging process, and to spot patterns of failure or repetition, upon wich you can suggest a clearing of the context for improved focus.
This section documents incorrect assumptions that have been disproven through testing. Reviewing these can help avoid repeating past mistakes.
-
MVCC and Transaction Visibility in PL/pgSQL: The set of rows visible to a
pl/pgsqlfunction is determined when the function begins execution. Within a single function, aSET-based query and aLOOPthat executes queries will both operate on the same data snapshot. It is a flawed assumption to think that aLOOPis somehow more robust to transaction visibility issues than a single complex query within the same function. If a set-based query is failing, the bug is in the query's logic, not in a fundamental limitation of MVCC for set-based operations. -
regclassType Casting in C Triggers: Theregclassdata type is a symbolic reference to a relation OID, not just the OID itself. When a C function using SPI executes a query involving aregclasscolumn, the PostgreSQL backend may attempt to format this value, which involves looking up the object's name. This behavior can be dangerous and lead to subtle bugs.- The Problem in
sql_dropEvent Triggers: Ansql_droptrigger fires after the object has been deleted from the system catalogs. If an SPI query in the trigger function selects aregclasscolumn that refers to a now-deleted object, the backend's attempt to look up the object's name will fail. This causesSPI_executeto return an error, leading to unpredictable behavior if not handled precisely. This can manifest as cascading, incorrect error messages or the appearance that queries are returning rows when they have actually failed. - The Problem in Regular Triggers: In regular DML triggers, using
regclassin aWHEREclause (e.g.,WHERE table_oid = $1) can be sensitive to the currentsearch_pathand rely on implicit casting rules. This can make queries less robust. - The Solution: For maximum robustness in any C-based trigger function using SPI:
- Always
CASTtooidin SQL: In all SPI queries, explicitly castregclasscolumns tooidfor bothSELECTlists andWHEREclauses (e.g.,SELECT table_oid::oid ...,... WHERE table_oid::oid = $1). - Use
OIDOIDfor Parameters: When usingSPI_execute_with_args, specify the parameter type asOIDOIDfor relation identifiers. - Perform Lookups in C: Fetch raw OIDs from the database. Any conversion from an OID to a textual name for error messages must be done manually in the C code (e.g., using
regclassout) after the query has successfully completed.
- Always
- The Problem in
-
IGNORE NULLSin Window Functions: PostgreSQL does not support theIGNORE NULLSclause for window functions likeLAG,LEAD, orFIRST_VALUE. This is a common pitfall.- The Problem: Attempting to use
FIRST_VALUE(my_col) IGNORE NULLS OVER (...)will result in a syntax error. - The Solution: The standard and most robust workaround is a "gaps-and-islands" approach. This involves creating a new grouping key using a conditional
SUM()window function that increments every time a non-NULLvalue is encountered. This partitions the data into "islands" where a value can be safely propagated to adjacentNULLs using an aggregate likeMAX()or a carefully framedFIRST_VALUE(). - Example: The
diff_with_propagated_idsCTE in thetemporal_mergeplanner is a canonical example of this pattern, using both forward (look_behind_grp) and backward (look_ahead_grp) running sums to find the nearest non-NULLvalue in either direction.
- The Problem: Attempting to use
When repeated iterations of the hypothesis-driven cycle fail to resolve a persistent and complex bug (such as a memory corruption crash), it is a sign that the underlying assumptions are wrong and a change in strategy is required.
- Formulate a new meta-hypothesis: State that the complexity of the current implementation is the source of the problem.
- Strip to a non-crashing stub: Drastically simplify the faulty code to a minimal, stable state that is guaranteed not to crash, even if it returns an incorrect result. This changes the failure mode from a crash to a predictable test failure.
- Verify stability: Run the tests to confirm that the server is now stable and the crash is gone. This establishes a new, reliable baseline.
- Incrementally re-introduce logic (Logical Binary Search): Re-introduce functionality piece by piece, running tests at each step.
- Start by restoring the first half of the original logic. If the system remains stable, the bug is in the second half. If it crashes, the bug is in the first half.
- Continue this process, dividing the suspicious code block in half with each iteration, until the single line or small section of code causing the instability is isolated.
- This process is a logical "binary search" on the code's functionality.
- Resume the normal cycle: Once the root cause is identified, resume the standard hypothesis-driven cycle to fix the specific issue.