Add size constraints to CTIM collection fields (XDR-46972)

[ereteog]

Epic XDR-46972
Related flanders#58
Related ctia#1514

Add pred/max-len constraints to collection fields in sighting, incident, and common schemas.

Changes

• Bump flanders to 1.1.1-SNAPSHOT (adds :spec support on SequenceOfType, SetOfType, and MapType)
• Add default-collection-max-len (500) constant in common.cljc for shared collection fields
• Constrain external_ids and external_references in base entity entries
• Add specific limits for sighting fields:
  • columns: 100, rows: 10,000
  • targets: 2,000, observables: 5,000, relations: 10,000
• Constrain incident fields (categories, assignees, tactics, techniques) to 500
• Add relation_info.actions limit of 1,000
• Add {:max-len len} metadata to pred/max-len for downstream 413 error reporting
• Replace unbounded f/any-str with typed string constraints (ShortString, MedString, LongString)

§ QA

No QA is needed. All 139 existing tests (including generative tests) pass.

§ Release Notes

intern: Add size constraints to CTIM collection fields.

§ Squashed Commits

[yogsototh]

I have only two concerns:

• why these numbers, to me they are a bit like "magic numbers" and it will eventually breaks in PROD refusing the creation of many entities suddenly without a warning which may create issues with already working integrations. Personally, I think it may be a bit too much to constrain statically here instead of directly into the CTIA application where we could apply different size constraints depending on the source for example. Which will give the time for an integration to adapt to the new constraints. And perhaps instead of a hard reject, we may also integrate a mechanism to just cut the strings/arrays to keep the first one.

• We should check if adding these constraints does not affect Swagger and Swagger UI. From memory plumatic/schemas and swagger do not support very well certain kind of external constraints like this one. So while the API will work, Swagger UI may be broken, or not show the correct schemas anymore.

[yogsototh]

Verdict: CHANGES REQUESTED -- 1 CRITICAL, 2 HIGH, 3 MEDIUM

This is a clean, well-scoped tightening and the core max-len predicate is correct (inclusive >= bound, nil-safe via string? guards). The blocking issues are about the dependency and where the constraints actually take effect, not the schema design itself.

Findings

• CRITICAL [cross-cutting] -- SNAPSHOT dependency on unreleased flanders: project.clj pins threatgrid/flanders "1.1.1-SNAPSHOT". flanders 1.1.1 has no git tag, no GitHub release, and Clojars returns 404 for it. The :spec-on-collection-types support that every f/seq-of/f/map constraint in this PR relies on exists only on flanders master, not in 1.1.0. Merging this to master makes the ctim build non-reproducible (a SNAPSHOT is mutable / can be GC'd), blocks cutting any stable ctim release, and means the collection constraints are inert until flanders 1.1.1 actually ships. Release flanders 1.1.1 to Clojars and pin the released coordinate before merge. (See inline on project.clj.)

• HIGH [cross-cutting] -- Collection caps are not enforced in the schema/Swagger target: flanders schema.cljc (master) does not read :spec for SequenceOfType/SetOfType/MapType -- it is consumed only by the clojure.spec target. So the new collection-length caps (external_ids/external_references, sighting targets/observables/relations/data-table, incident categories/assignees/tactics/techniques, relation_info.actions) are present in spec but absent from the plumatic-schema object and the generated Swagger/JSON Schema. This confirms the prior reviewer's Swagger concern at the source level. The string-length caps (ShortString/MedString/LongString) do propagate to both targets. Decide whether flanders should honor :spec on collection types in the schema target, or document the collection caps as spec-only and accept that Swagger and any plumatic-based validation will not enforce them.

• HIGH [correctness/cross-cutting] -- Backward-incompatible narrowing rejects previously-valid data: replacing unbounded f/any-str with bounded types is a hard reject for documents that validated before. Highest risk is open_IOC -> MedString (2048) -- the field's own docstring calls it "an XML blob of an openIOC indicator," and OpenIOC XML routinely exceeds 2048 chars; SIOC and snort_sig (both MedString 2048) are similar. No migration path, grace period, or source-conditional limit is present. Recommend auditing existing PROD data against each new bound and confirming open_IOC/SIOC at 2048 is intended. (See inline on indicator.cljc.)

• MEDIUM [security/correctness] -- relation_info.actions cap is trivially bypassable: the :spec only bounds :actions when it is sequential?; a non-sequential value (JSON object -> map, or a string) returns true and passes unbounded, and only the literal :actions key is inspected in an otherwise free-form map. As a DoS-size guard on an already-open field this is best-effort, but the cap as written does not bound relation_info. (See inline on common.cljc.)

• MEDIUM [quality/cross-cutting] -- Asymmetric bounding: the embedded SightingDataTable bounds :columns/:rows, but the standalone DataTable entity (data_table.cljc:53-56) leaves the structurally identical fields unbounded -- the first-class entity is the larger surface and is left open. Same pattern with incident :detection_sources, uncapped while its sibling collection fields were capped. (See inline on incident.cljc.)

• MEDIUM [tests] -- No failure-path tests for the new bounds: the diff touches only src/. None of the 12 new collection caps (nor the boundary/off-by-one, nor the 3-branch relation_info spec, nor the new {:max-len} metadata) has a test asserting rejection or at-limit acceptance. The existing generative tests only assert normally-sized documents are accepted (test.check default sizes never approach 500-10000), so "139 tests pass" does not exercise any new bound. Suggest at-N (accept) / at-N+1 (reject) tests per bound.

PASS: Coding Style (only minor :spec indentation drift at sighting.cljc:48,51 and incident.cljc:171; CLJS require parity is correct in all files -- no build break).

Also left 4 inline comments.

Non-blocking follow-ups

• max-len now returns (with-meta fn {:max-len len}); no consumer reads it in this PR, and the metadata is lost once the predicate is composed inside cs/and (which is how every string type uses it). Confirm the intended consumer reads it off the bare predicate.
• Centralize or document the convention for where size constants live (common.cljc defaults vs sighting.cljc-local) and add the P99.9 rationale (currently only in commit messages) as in-code comments.
• Observable :value is intentionally left f/any-str (per commit message: process_args up to ~32K); noted as deliberate, not a defect.
• count-based max-len is not guarded against a non-counted lazy seq reaching the relation_info spec (in-process/EDN callers only; JSON input is unaffected).

-- auto-review-team (BOT_APPROVABLE gate)

[yogsototh]

CRITICAL -- SNAPSHOT dependency on unreleased flanders: 1.1.1-SNAPSHOT has no flanders git tag, no GitHub release, and Clojars 404s for it. The :spec support on SequenceOfType/SetOfType/MapType that this entire PR depends on exists only on flanders master, not in 1.1.0. Merging this to master makes the build non-reproducible (a SNAPSHOT is mutable and can be GC'd), blocks any stable ctim release, and leaves these constraints inert until flanders 1.1.1 ships. Release flanders 1.1.1 to Clojars and pin the released version before merge.

[yogsototh]

HIGH -- Bound likely too small for documented content: this field's docstring describes it as "an XML blob of an openIOC indicator," and OpenIOC XML documents routinely exceed 2048 chars, so MedString will reject most real open_IOC values. SIOC (line 49) and snort_sig (line 41) share the same 2048 cap and the same risk. More broadly, replacing unbounded f/any-str with a hard cap rejects documents that validated before, with no migration path. Recommend auditing existing PROD data against these bounds and confirming the values are intended for these blob fields.

[yogsototh]

MEDIUM -- Actions cap is bypassable: the limit only applies when :actions is sequential?. A non-sequential value -- a JSON object (decodes to a map) or a string -- makes (not (sequential? actions)) return true, so it passes unbounded. Only the literal :actions key is inspected; the rest of this free-form f/any map is uncapped. If the goal is to bound oversized relation_info, consider rejecting non-sequential :actions rather than waving them through, and confirm :actions is the real key used in production payloads.

[yogsototh]

MEDIUM -- Uncapped sibling: :detection_sources kept the plain [c/MedString] form while its neighbours :categories, :assignees, :tactics, and :techniques all received (f/seq-of ... :spec (pred/max-len default-collection-max-len)) in this PR. If the size cap is meant to cover incident collection fields, this one was missed; if it is intentionally unbounded, a one-line note would make that explicit. (Same asymmetry exists between the bounded embedded SightingDataTable and the unbounded standalone DataTable entity in data_table.cljc.)