fix: gate off-orglist purge on governance freshness, not sync state

README.md

@@ -235,10 +235,10 @@ In the `CHIA_ROOT` directory (usually `~/.chia/mainnet` on Linux), CADT will add
   * **DATALAYER_FILE_SERVER_URL**: Publicly available URL and port where Chia Datalayer [files are served](#datalayer-http-file-serving), including schema (http:// or https://). If serving DataLayer files from S3, this would be the public URL of the S3 bucket. Port can be omitted if using standard ports for http or https requests.
   * **AUTO_SUBSCRIBE_FILESTORE**: Subscribing to the filestore for any organization is optional. To automatically subscribe and sync the filestore to every organization you subscribe to, set this to `true`.
   * **AUTO_MIRROR_EXTERNAL_STORES**: When set to true (the default), CADT will automatically create mirrors for each store you are subscribed to. Mirroring all subscriptions using the `DATALAYER_FILE_SERVER_URL` will make the entire CADT network more resilient and distributed. Note: `DATALAYER_FILE_SERVER_URL` must also be set to a valid URL or IP address for mirrors to be created. Both settings are required for external store mirroring to function.
-  * **ONLY_CADT_SUBSCRIPTIONS**: When `true` (the default), CADT keeps DataLayer subscriptions aligned with the governance **orgList** in both directions. Organizations removed from the orgList are first unsubscribed from DataLayer, then removed from this node after unsubscribe is confirmed and the purge grace period has elapsed — the organization record and **all** of its local data (projects, units, and every related record it created) are deleted from the database. The deletion is unconditional: it does **not** check whether another organization references that data. Organizations on the orgList that are not subscribed are subscribed (including orgs re-added after a prior removal, including orgs previously removed via the API delete flow). The home organization and governance body store are never auto-removed. Reconciliation runs only after a successful governance sync provides a non-empty **orgList**; empty or stale cached governance data does not trigger removals. Set to `false` to disable orglist-driven subscribe/remove reconciliation. While enabled, a manual unsubscribe of an org still listed on the orgList will be reverted on the next sync cycle.
+  * **ONLY_CADT_SUBSCRIPTIONS**: When `true` (the default), CADT keeps DataLayer subscriptions aligned with the governance **orgList** in both directions. Organizations removed from the orgList are first unsubscribed from DataLayer, then removed from this node after unsubscribe is confirmed and the purge grace period has elapsed — the organization record and **all** of its local data (projects, units, and every related record it created) are deleted from the database. The deletion is unconditional: it does **not** check whether another organization references that data. Organizations on the orgList that are not subscribed are subscribed (including orgs re-added after a prior removal, including orgs previously removed via the API delete flow). The home organization and governance body store are never auto-removed. Reconciliation runs only while a recent governance sync has confirmed a non-empty **orgList**; empty or stale cached governance data does not trigger removals. "Recent" means within five `GOVERNANCE_SYNC_TASK_INTERVAL` periods (clamped to between 10 and 60 minutes), so a node whose governance sync has been failing stops removing organizations until it recovers. Set to `false` to disable orglist-driven subscribe/remove reconciliation. While enabled, a manual unsubscribe of an org still listed on the orgList will be reverted on the next sync cycle.
   * **LOG_LEVEL**: Controls verbosity of logging. Common settings are `info` and `debug`. Setting to `silly` will log all queries.
   * **TASKS**: Section for configuring sync intervals.
-    * **GOVERNANCE_SYNC_TASK_INTERVAL**: Syncs picklist, orgList, and glossary from the governance node. Default 120 seconds (2 minutes).
+    * **GOVERNANCE_SYNC_TASK_INTERVAL**: Syncs picklist, orgList, and glossary from the governance node. Default 120 seconds (2 minutes). Also sets how long a successful sync keeps the `ONLY_CADT_SUBSCRIPTIONS` reconciliation eligible to run — see that setting for details.
     * **ORGANIZATION_META_SYNC_TASK_INTERVAL**: Subscribes to default organizations and refreshes metadata for already-imported organizations. Default 120 seconds (2 minutes).
     * **PICKLIST_SYNC_TASK_INTERVAL**: Syncs picklist from the governance node. Default 120 seconds (2 minutes).
     * **MIRROR_CHECK_TASK_INTERVAL**: Checks if our DataLayer is advertising our `DATALAYER_FILE_SERVER_URL` as a mirror for all subscriptions when `AUTO_MIRROR_EXTERNAL_STORES` is true. Default 900 seconds (15 minutes).

src/models/governance/governance.model.js

@@ -188,7 +188,8 @@ class Governance extends Model {
 
   static async sync() {
     logger.debug('[v1]: running governance model sync()');
-    markGovernanceNotReady('v1');
+    // Readiness is a freshness timestamp, not a sync-in-progress flag; do not
+    // clear it here. See utils/governance-readiness.js.
 
     // Check simulator/dev mode first to match V2 behavior and avoid errors
     // in test/dev environments that may not have GOVERNANCE_BODY_ID configured.
@@ -275,6 +276,10 @@ class Governance extends Model {
         }
         if (hasOrgList) {
           markGovernanceReady('v1');
+        } else {
+          // A download carrying no orgList cannot vouch for the cached one, so
+          // revoke readiness instead of letting the purge run against it.
+          markGovernanceNotReady('v1');
         }
         return;
       }
@@ -340,6 +345,10 @@ class Governance extends Model {
       }
       if (hasOrgList) {
         markGovernanceReady('v1');
+      } else {
+        // A download carrying no orgList cannot vouch for the cached one, so
+        // revoke readiness instead of letting the purge run against it.
+        markGovernanceNotReady('v1');
       }
     } catch (error) {
       logger.error(

src/models/v2/governance-v2.model.js

@@ -465,7 +465,8 @@ class GovernanceV2 extends Model {
    */
   static async sync() {
     loggerV2.debug('[v2]: running V2 governance model sync()');
-    markGovernanceNotReady('v2');
+    // Readiness is a freshness timestamp, not a sync-in-progress flag; do not
+    // clear it here. See utils/governance-readiness.js.
 
     const { USE_SIMULATOR, USE_DEVELOPMENT_MODE } = getConfig().APP;
 
@@ -556,6 +557,10 @@ class GovernanceV2 extends Model {
         }
         if (hasOrgList) {
           markGovernanceReady('v2');
+        } else {
+          // A download carrying no orgList cannot vouch for the cached one, so
+          // revoke readiness instead of letting the purge run against it.
+          markGovernanceNotReady('v2');
         }
         return;
       }
@@ -622,6 +627,10 @@ class GovernanceV2 extends Model {
       }
       if (hasOrgList) {
         markGovernanceReady('v2');
+      } else {
+        // A download carrying no orgList cannot vouch for the cached one, so
+        // revoke readiness instead of letting the purge run against it.
+        markGovernanceNotReady('v2');
       }
     } catch (error) {
       loggerV2.error(

src/tasks/sync-default-organizations-v2.js

@@ -121,10 +121,14 @@ const task = new Task('sync-default-organizations-v2', async () => {
         }
       });
 
+      // Read once: readiness is time-based, so two separate calls could
+      // straddle its expiry and leave neither branch taken.
+      const governanceReady = isGovernanceReady('v2');
+
       if (
         onlyCadtSubscriptions &&
         defaultOrgList.length > 0 &&
-        isGovernanceReady('v2')
+        governanceReady
       ) {
         const { GOVERNANCE_BODY_ID } = getConfigV2().GOVERNANCE;
         const allowSet = buildOrgListAllowSet(defaultOrgList, GOVERNANCE_BODY_ID);
@@ -154,10 +158,10 @@ const task = new Task('sync-default-organizations-v2', async () => {
       } else if (
         onlyCadtSubscriptions &&
         defaultOrgList.length > 0 &&
-        !isGovernanceReady('v2')
+        !governanceReady
       ) {
         loggerV2.debug(
-          '[v2]: ONLY_CADT_SUBSCRIPTIONS: skipping off-orglist purge until governance sync completes',
+          '[v2]: ONLY_CADT_SUBSCRIPTIONS: skipping off-orglist purge until a recent governance sync confirms the orgList is fresh',
         );
       }
     }

src/tasks/sync-default-organizations.js

@@ -110,10 +110,14 @@ const task = new Task('sync-default-organizations', async () => {
         }
       });
 
+      // Read once: readiness is time-based, so two separate calls could
+      // straddle its expiry and leave neither branch taken.
+      const governanceReady = isGovernanceReady('v1');
+
       if (
         onlyCadtSubscriptions &&
         defaultOrgRecords.length > 0 &&
-        isGovernanceReady('v1')
+        governanceReady
       ) {
         const { GOVERNANCE_BODY_ID } = CONFIG.GOVERNANCE;
         const allowSet = buildOrgListAllowSet(defaultOrgRecords, GOVERNANCE_BODY_ID);
@@ -145,10 +149,10 @@ const task = new Task('sync-default-organizations', async () => {
       } else if (
         onlyCadtSubscriptions &&
         defaultOrgRecords.length > 0 &&
-        !isGovernanceReady('v1')
+        !governanceReady
       ) {
         logger.debug(
-          '[v1]: ONLY_CADT_SUBSCRIPTIONS: skipping off-orglist purge until governance sync completes',
+          '[v1]: ONLY_CADT_SUBSCRIPTIONS: skipping off-orglist purge until a recent governance sync confirms the orgList is fresh',
         );
       }
     }

src/tasks/sync-governance-body-v2.js

@@ -7,10 +7,7 @@ import { getConfig, getConfigV2 } from '../utils/config-loader.js';
 import { loggerV2 } from '../config/logger.js';
 import { GovernanceV2 } from '../models/v2/index.js';
 import { OrganizationsV2 } from '../models/v2/index.js';
-import {
-  markGovernanceNotReady,
-  markGovernanceReady,
-} from '../utils/governance-readiness.js';
+import { markGovernanceReady } from '../utils/governance-readiness.js';
 
 const CONFIG = getConfig().APP;
 const CONFIG_V2 = getConfigV2();
@@ -27,7 +24,8 @@ const task = new Task('sync-governance-meta-v2', async () => {
       return;
     }
 
-    markGovernanceNotReady('v2');
+    // Readiness is a freshness timestamp, not a sync-in-progress flag; do not
+    // clear it here. See utils/governance-readiness.js.
     await assertDataLayerAvailable();
     await assertWalletIsSynced();
 

src/tasks/sync-governance-body.js

@@ -8,10 +8,7 @@ import {
 import { getConfig } from '../utils/config-loader';
 import { logger } from '../config/logger.js';
 import { Organization } from '../models';
-import {
-  markGovernanceNotReady,
-  markGovernanceReady,
-} from '../utils/governance-readiness.js';
+import { markGovernanceReady } from '../utils/governance-readiness.js';
 
 const CONFIG = getConfig();
 
@@ -27,7 +24,8 @@ const task = new Task('sync-governance-meta', async () => {
       return;
     }
 
-    markGovernanceNotReady('v1');
+    // Readiness is a freshness timestamp, not a sync-in-progress flag; do not
+    // clear it here. See utils/governance-readiness.js.
     await assertDataLayerAvailable();
     await assertWalletIsSynced();
 

src/utils/governance-readiness.js

@@ -1,25 +1,102 @@
 'use strict';
 
-const ready = {
-  v1: false,
-  v2: false,
+import { getConfig } from './config-loader.js';
+
+// Governance readiness is a freshness signal, not a "sync in progress" flag. It
+// answers "was governance data last confirmed good recently enough to destroy
+// local state against it?". The off-orglist purge in the
+// sync-default-organizations tasks is gated on it, and that purge unsubscribes
+// from and deletes every local record for organizations missing from the cached
+// orgList, so a wrong answer here is expensive in both directions.
+//
+// Two rules follow, and both are load-bearing:
+//
+// 1. Never clear readiness because a sync has *started*. The producer
+//    (tasks/sync-governance-body{,-v2}.js) and the consumer
+//    (tasks/sync-default-organizations{,-v2}.js) are separate jobs that share a
+//    default task interval, so they fire on the same scheduler tick. A value
+//    cleared at the start of a sync is what the consumer reads for that entire
+//    cycle, which closes the purge gate permanently rather than momentarily.
+//    Readiness may only move backwards on evidence that governance is stale,
+//    never on the mere fact that a refresh is underway.
+//
+// 2. Staleness is time-based. A boolean can only express "succeeded at least
+//    once since process start", which would let a node whose governance sync
+//    has been broken for hours keep purging against a long-stale orgList.
+//    Comparing against the last confirmed-good timestamp blocks that node while
+//    never dipping false during a healthy refresh.
+const lastReadyAt = {
+  v1: null,
+  v2: null,
+};
+
+// Readiness expires after this many governance sync intervals. Several
+// intervals rather than one so a single slow or failed sync does not gate the
+// purge off. The window is clamped at both ends: the floor keeps an
+// aggressively short configured interval from making readiness flap, and the
+// ceiling keeps a long one from stretching "recently validated" into days,
+// which would defeat rule 2 above.
+const STALE_AFTER_SYNC_INTERVALS = 5;
+const MIN_READINESS_MAX_AGE_MS = 10 * 60 * 1000;
+const MAX_READINESS_MAX_AGE_MS = 60 * 60 * 1000;
+// The ceiling itself has a floor, expressed in sync intervals. Without it, an
+// interval longer than the ceiling would expire readiness before the next
+// scheduled sync could ever renew it, turning the gate into a permanent off
+// switch and silently disabling reconciliation instead of merely pausing it.
+const MIN_INTERVALS_BEFORE_EXPIRY = 2;
+const DEFAULT_SYNC_INTERVAL_SECONDS = 120;
+
+const getReadinessMaxAgeMs = () => {
+  // TASKS lives under the shared APP section, which getConfig and getConfigV2
+  // populate identically, so this one read serves both versions.
+  const configured = Number(
+    getConfig()?.APP?.TASKS?.GOVERNANCE_SYNC_TASK_INTERVAL,
+  );
+  const intervalMs =
+    (Number.isFinite(configured) && configured > 0
+      ? configured
+      : DEFAULT_SYNC_INTERVAL_SECONDS) * 1000;
+
+  const ceiling = Math.max(
+    MAX_READINESS_MAX_AGE_MS,
+    intervalMs * MIN_INTERVALS_BEFORE_EXPIRY,
+  );
+
+  return Math.min(
+    Math.max(
+      intervalMs * STALE_AFTER_SYNC_INTERVALS,
+      MIN_READINESS_MAX_AGE_MS,
+    ),
+    ceiling,
+  );
 };
 
 export const markGovernanceReady = (version) => {
-  if (Object.prototype.hasOwnProperty.call(ready, version)) {
-    ready[version] = true;
+  if (Object.prototype.hasOwnProperty.call(lastReadyAt, version)) {
+    lastReadyAt[version] = Date.now();
   }
 };
 
+// Only for cases where governance data is known to be bad — currently a
+// download that carries no orgList, which cannot vouch for the cached one.
+// Not for signalling that a refresh has begun; see rule 1 above.
 export const markGovernanceNotReady = (version) => {
-  if (Object.prototype.hasOwnProperty.call(ready, version)) {
-    ready[version] = false;
+  if (Object.prototype.hasOwnProperty.call(lastReadyAt, version)) {
+    lastReadyAt[version] = null;
   }
 };
 
-export const isGovernanceReady = (version) => ready[version] === true;
+export const isGovernanceReady = (version) => {
+  const readyAt = lastReadyAt[version];
+
+  if (typeof readyAt !== 'number') {
+    return false;
+  }
+
+  return Date.now() - readyAt <= getReadinessMaxAgeMs();
+};
 
 export const resetGovernanceReadiness = () => {
-  ready.v1 = false;
-  ready.v2 = false;
+  lastReadyAt.v1 = null;
+  lastReadyAt.v2 = null;
 };

tests/integration/orglist-subscription-reconcile-v1.spec.js

@@ -1,4 +1,5 @@
 import { expect } from 'chai';
+import sinon from 'sinon';
 import { Op } from 'sequelize';
 import { prepareDb } from '../../src/database';
 import { Organization, Project, Staging, Meta, Governance } from '../../src/models';
@@ -70,6 +71,12 @@ describe('orglist-subscription-reconcile (V1)', function () {
     resetGovernanceReadiness();
   });
 
+  // A test aborted by timeout never reaches its own restore, which would leave
+  // a stub installed for the rest of the file.
+  afterEach(function () {
+    sinon.restore();
+  });
+
   it('should unsubscribe and delete orgs after the grace cycle using V1 field names', async function () {
     await Organization.create({
       orgUid: ORG_A,
@@ -492,6 +499,58 @@ describe('orglist-subscription-reconcile (V1)', function () {
     expect(isGovernanceReady('v1')).to.equal(false);
   });
 
+  it('should expire governance readiness once the last confirmed-good sync is stale', function () {
+    const clock = sinon.useFakeTimers({ now: Date.now(), toFake: ['Date'] });
+
+    try {
+      markGovernanceReady('v1');
+      expect(isGovernanceReady('v1')).to.equal(true);
+
+      // Readiness holds for five default 120s sync intervals, so a handful of
+      // failed syncs does not gate the purge off.
+      clock.tick(9 * 60 * 1000);
+      expect(isGovernanceReady('v1')).to.equal(true);
+
+      clock.tick(2 * 60 * 1000);
+      expect(isGovernanceReady('v1')).to.equal(false);
+    } finally {
+      clock.restore();
+    }
+  });
+
+  it('should keep governance readiness while a sync is in flight', async function () {
+    markGovernanceReady('v1');
+
+    let releaseUpsert;
+    const upsertGate = new Promise((resolve) => {
+      releaseUpsert = resolve;
+    });
+    let upsertStarted;
+    const upsertReached = new Promise((resolve) => {
+      upsertStarted = resolve;
+    });
+
+    const upsertStub = sinon.stub(Governance, 'upsert').callsFake(async () => {
+      upsertStarted();
+      await upsertGate;
+    });
+
+    try {
+      const syncPromise = Governance.sync();
+      await upsertReached;
+
+      // A sync that has started but not finished must not lower readiness: the
+      // purge gate is polled by a task on the same interval, so it would read
+      // the cleared value for the whole cycle.
+      expect(isGovernanceReady('v1')).to.equal(true);
+
+      releaseUpsert();
+      await syncPromise;
+    } finally {
+      upsertStub.restore();
+    }
+  });
+
   it('should mark V1 governance ready after fallback governance sync', async function () {
     markGovernanceNotReady('v1');