Stack Palantir engineering context into Zyra agents and copilot

.agents/skills/zyra-adaptive-intelligence/SKILL.md

@@ -25,6 +25,16 @@ Use a bounded five-phase loop:
 - For security work: stay within authorized defensive analysis and do not bypass access controls or obtain restricted information.
 - When evidence is incomplete, say what is unknown and identify the next verification step.
 
+## Engineering Fleet Bridge
+
+For data engineering, ontology, application development, AIP/LLM integration, deployment, or cross-stack architecture, combine this adaptive loop with `.agents/skills/zyra-engineering-fleet/SKILL.md` and `server/engineering-context.ts`.
+
+Map adaptive phases into the engineering decision loop:
+
+`ABSORB/INDEX -> INSPECT/MODEL -> IDEATE -> PLAN/DECOMPOSE -> VERIFY -> VALIDATE/OBSERVE -> REFLECT -> REPAIR/AUDIT`
+
+Use the smallest useful bounded fleet from intake, pipeline, quality, ontology, application, security, release, and observer roles. Parallel work is allowed only when dependencies are explicit. The runtime owns tools and permissions; consequential writes and releases remain approval-gated.
+
 ## Adaptive Strategy Selection
 
 Maintain lightweight strategy outcomes such as:

.agents/skills/zyra-architecture/SKILL.md

@@ -19,6 +19,7 @@ description: Zyra platform architecture, module map, and conventions. Use when m
 | `auth.ts` | JWT middleware: `requireAuth`, `requireRole`, token gen/verify |
 | `storage.ts` | `DatabaseStorage` implementing `IStorage` — all DB operations |
 | `intelligence.ts` | ZyraCopilot engine, CVE database, threat correlation |
+| `engineering-context.ts` | Clean-room data/application engineering stack, fleet roles, decision loop, and LLM system context |
 | `metrics.ts` | Prometheus-style counters/histograms, request middleware |
 | `caasm.ts` | CAASM risk scoring, identity correlation, API routes |
 | `enterprise.ts` | SIEM configs, retention policies, workspaces |
@@ -28,6 +29,16 @@ description: Zyra platform architecture, module map, and conventions. Use when m
 | `stripe.ts` | Stripe Checkout sessions, graceful degradation |
 | `soar.ts` | SOAR playbook execution engine |
 
+## Engineering Stack
+
+For data engineering, ontology, application development, AIP/LLM work, deployment, and cross-stack architecture, apply `.agents/skills/zyra-engineering-fleet/SKILL.md` and `server/engineering-context.ts`.
+
+Architecture path:
+
+`SOURCE/INTAKE -> PIPELINE -> QUALITY -> ONTOLOGY -> APPLICATION -> AIP/LLM -> SECURITY -> RELEASE -> AUDIT`
+
+Use the smallest useful bounded role set from intake, pipeline, quality, ontology, application, security, release, and observer. Model dependencies before parallel execution. Treat schemas, datasets, ontology objects, actions, APIs, tools, variables, events, and UI state as explicit contracts. Keep consequential writes and production promotion inside approval policy, rollback, and evidence gates.
+
 ## Adding a New Feature
 1. Add table(s) to `shared/schema.ts` with insert schema + types
 2. Add CRUD methods to `IStorage` interface and `DatabaseStorage` in `server/storage.ts`
@@ -36,6 +47,8 @@ description: Zyra platform architecture, module map, and conventions. Use when m
 5. Add sidebar entry in `client/src/components/layout/sidebar.tsx`
 6. Use TanStack Query v5 (object form only) for data fetching
 7. Add `data-testid` attributes to all interactive and display elements
+8. For LLM-facing features, use `buildEngineeringSystemContext()` rather than duplicating engineering policy text
+9. Run `npm test`, `npm run check`, dependency audit, and production build before promotion
 
 ## Critical Conventions
 - `apiRequest(method, url, data?)` returns `Response` — call `.json()` to parse

.agents/skills/zyra-engineering-fleet/SKILL.md

@@ -0,0 +1,51 @@
+# Zyra Engineering Fleet
+
+Use this skill for data engineering, ontology, application development, AIP/LLM integration, release engineering, and cross-stack architecture work.
+
+## Clean-room knowledge sources
+
+- Palantir Learn Data Engineer guide: https://learn.palantir.com/data-engineer-guide/1388785
+- Palantir Learn Application Developer guide: https://learn.palantir.com/application-developer-guide/1481796
+- Public Palantir Foundry documentation for pipelines, data expectations, ontology, application building, Workshop state/events, and AIP tool context.
+
+Do not copy proprietary certification-guide text or claim Palantir affiliation, certification, tenant access, private APIs, or product internals.
+
+## Fleet roles
+
+For complex missions, assign the smallest useful bounded role set:
+
+1. Intake — inspect sources, constraints, provenance, ownership, desired outcome.
+2. Pipeline — ingestion, normalization, transforms, data products, freshness.
+3. Quality — preconditions, postconditions, tests, integrity, failure policy.
+4. Ontology — canonical objects, properties, links, actions, semantic contracts.
+5. Application — lowest-complexity UI/workflow/API that satisfies the use case.
+6. Security — least privilege, secret handling, write boundaries, abuse cases.
+7. Release — CI, downstream impact, rollback, health, promotion evidence.
+8. Observer — lineage, evidence, unresolved risks, task status, completion proof.
+
+## Decision loop
+
+`INSPECT -> MODEL -> PLAN -> DECOMPOSE -> EXECUTE -> VALIDATE -> OBSERVE -> REPAIR -> APPROVE -> RELEASE -> AUDIT`
+
+Independent roles may work in parallel only when their dependencies are explicit. The runtime owns tools and permissions. Read access and write access must remain distinguishable. Consequential external writes, destructive changes, production promotion, credentials, and payment-related operations remain approval-gated by the surrounding system.
+
+## Data engineering rules
+
+- Work backward from the operational ontology/workflow.
+- Separate source cleanup, reusable transforms, canonical outputs, and consuming workflows.
+- Treat published datasets as contracts with ownership, schema, provenance, freshness, and downstream consumers.
+- Encode critical assumptions as executable quality checks and stop invalid data before it propagates.
+- Prefer batch when it meets requirements; adopt incremental or streaming only when scale or latency justifies complexity.
+- Include health, lineage, build evidence, incident ownership, and rollback in production design.
+
+## Application development rules
+
+- Start from objects, links, properties, and governed actions before UI details.
+- Treat components/widgets as explicit input/output units with typed state and events.
+- Route domain mutations through governed actions, not arbitrary client-side writes.
+- Give LLMs only the variables, objects, searches, functions, tools, applications, and actions needed for the workflow.
+- Prefer the lowest-complexity surface that meets the use case; escalate to custom code only when justified.
+
+## Completion gate
+
+Do not report completion until applicable tests, data expectations, security checks, downstream-impact checks, health checks, rollback readiness, and execution evidence are present.

docs/PALANTIR_ENGINEERING_STACK.md

@@ -0,0 +1,82 @@
+# Zyra Palantir Engineering Stack
+
+Zyra uses a clean-room engineering knowledge layer derived from public Palantir Learn links supplied by the project owner and public Palantir Foundry documentation. It does not copy certification-guide text, proprietary code, private APIs, or Palantir product assets.
+
+## Source guides
+
+- https://learn.palantir.com/data-engineer-guide/1388785
+- https://learn.palantir.com/application-developer-guide/1481796
+
+## Stack
+
+```text
+SOURCE / INTAKE
+    ↓
+PIPELINE
+    ↓
+QUALITY / DATA EXPECTATIONS
+    ↓
+ONTOLOGY
+objects • properties • links • actions
+    ↓
+APPLICATION
+state • variables • events • UI • APIs
+    ↓
+AIP / LLM
+bounded context • tools • functions • governed actions
+    ↓
+SECURITY / APPROVAL
+    ↓
+RELEASE / HEALTH / LINEAGE / AUDIT
+```
+
+## Agentic fleet
+
+Zyra decomposes complex engineering work into bounded specialist roles:
+
+- `intake`: sources, provenance, constraints, ownership, outcome
+- `pipeline`: ingestion, normalization, transforms, freshness, publication
+- `quality`: tests, expectations, invariants, fail policy
+- `ontology`: canonical objects, properties, links, actions
+- `application`: lowest-complexity UI/workflow/API implementation
+- `security`: least privilege, boundaries, secrets, write controls
+- `release`: CI, health, downstream impact, rollback, promotion
+- `observer`: lineage, evidence, unresolved risk, completion status
+
+The deterministic shared context is implemented in `server/engineering-context.ts`.
+
+## Decision loop
+
+```text
+INSPECT
+→ MODEL
+→ PLAN
+→ DECOMPOSE
+→ EXECUTE
+→ VALIDATE
+→ OBSERVE
+→ REPAIR
+→ APPROVE
+→ RELEASE
+→ AUDIT
+```
+
+The fleet is a decision-support and orchestration pattern. It does not give an LLM unrestricted execution. Tools remain explicitly bounded, read/write access stays distinguishable, consequential writes/releases follow application policy and approval gates, and completion requires evidence.
+
+## Application principles
+
+- Start from the ontology/domain model before UI detail.
+- Treat widgets/components as explicit input/output units with typed state and events.
+- Route domain mutations through governed actions rather than arbitrary client-side writes.
+- Give LLMs only the context, tools, variables, objects, functions, searches, and actions required by the current workflow.
+- Prefer lower-complexity application surfaces until custom code is justified.
+
+## Data principles
+
+- Separate source cleanup, reusable transforms, canonical/ontology outputs, and consuming workflows.
+- Publish stable contracts with ownership, schema, freshness, provenance, and known downstream consumers.
+- Encode critical assumptions as executable quality gates and prevent invalid data from propagating.
+- Use batch unless incremental or streaming requirements justify extra operational complexity.
+- Treat lineage, health, build evidence, and rollback as production features.
+
+This document and `server/engineering-context.ts` are XUNIA/Zyra implementations of public engineering concepts, not Palantir Foundry components and not evidence of Palantir affiliation or tenant access.

package.json

@@ -8,7 +8,7 @@
     "build": "tsx script/build.ts",
     "start": "NODE_ENV=production node dist/index.cjs",
     "check": "tsc",
-    "test": "tsx --test server/zyra-shield-core.test.ts",
+    "test": "tsx --test server/zyra-shield-core.test.ts server/engineering-context.test.ts",
     "db:push": "drizzle-kit push"
   },
   "dependencies": {

server/engineering-context.test.ts

@@ -0,0 +1,43 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import {
+  ENGINEERING_DECISION_LOOP,
+  ENGINEERING_GUIDE_SOURCES,
+  ENGINEERING_STACK_STAGES,
+  buildEngineeringSystemContext,
+  engineeringMissionPlan,
+  selectEngineeringFleet,
+} from "./engineering-context";
+
+test("engineering sources include both supplied Palantir Learn guides", () => {
+  assert.equal(ENGINEERING_GUIDE_SOURCES.length, 2);
+  assert.ok(ENGINEERING_GUIDE_SOURCES.some(url => url.includes("data-engineer-guide")));
+  assert.ok(ENGINEERING_GUIDE_SOURCES.some(url => url.includes("application-developer-guide")));
+});
+
+test("data mission selects pipeline, quality, ontology, security, release, and observer roles", () => {
+  const ids = selectEngineeringFleet("Build a streaming data pipeline with schema validation").map(role => role.id);
+  assert.deepEqual(ids, ["intake", "pipeline", "quality", "ontology", "security", "release", "observer"]);
+});
+
+test("application mission selects ontology and application roles", () => {
+  const ids = selectEngineeringFleet("Build a React application with ontology objects and actions").map(role => role.id);
+  assert.deepEqual(ids, ["intake", "ontology", "application", "security", "release", "observer"]);
+});
+
+test("production mission requires approval", () => {
+  const plan = engineeringMissionPlan("Deploy the application to production");
+  assert.equal(plan.approvalRequired, true);
+  assert.ok(plan.completionGates.includes("execution evidence recorded"));
+});
+
+test("stack and decision loop are complete", () => {
+  assert.equal(ENGINEERING_STACK_STAGES.at(-1), "AUDIT");
+  assert.equal(ENGINEERING_DECISION_LOOP.at(0), "INSPECT");
+  assert.equal(ENGINEERING_DECISION_LOOP.at(-1), "AUDIT");
+  const context = buildEngineeringSystemContext("test");
+  assert.match(context, /bounded specialist fleet/i);
+  assert.match(context, /approval-gated/i);
+  assert.match(context, /Palantir affiliation/i);
+});

server/engineering-context.ts

@@ -0,0 +1,111 @@
+export const ENGINEERING_GUIDE_SOURCES = [
+  "https://learn.palantir.com/data-engineer-guide/1388785",
+  "https://learn.palantir.com/application-developer-guide/1481796",
+] as const;
+
+export const ENGINEERING_STACK_STAGES = [
+  "SOURCE/INTAKE",
+  "PIPELINE",
+  "QUALITY",
+  "ONTOLOGY",
+  "APPLICATION",
+  "AIP/LLM",
+  "SECURITY",
+  "RELEASE",
+  "AUDIT",
+] as const;
+
+export const ENGINEERING_DECISION_LOOP = [
+  "INSPECT",
+  "MODEL",
+  "PLAN",
+  "DECOMPOSE",
+  "EXECUTE",
+  "VALIDATE",
+  "OBSERVE",
+  "REPAIR",
+  "APPROVE",
+  "RELEASE",
+  "AUDIT",
+] as const;
+
+export const ENGINEERING_FLEET = [
+  { id: "intake", mission: "Inspect sources, provenance, constraints, ownership, and outcomes." },
+  { id: "pipeline", mission: "Design ingestion, normalization, transforms, publication, and freshness contracts." },
+  { id: "quality", mission: "Define tests, expectations, integrity checks, and fail-closed invariants." },
+  { id: "ontology", mission: "Model canonical objects, properties, links, actions, and semantic contracts." },
+  { id: "application", mission: "Build the lowest-complexity workflow, UI, or API surface that meets the need." },
+  { id: "security", mission: "Check least privilege, data boundaries, write controls, secrets, and abuse cases." },
+  { id: "release", mission: "Verify CI, health, downstream impact, rollback, and promotion evidence." },
+  { id: "observer", mission: "Track lineage, evidence, unresolved risk, status, and claimed completion." },
+] as const;
+
+const DATA_TERMS = ["data", "dataset", "pipeline", "ingestion", "transform", "etl", "streaming", "batch", "schema"];
+const APP_TERMS = ["app", "application", "frontend", "backend", "ui", "widget", "workflow", "react", "api"];
+const ONTOLOGY_TERMS = ["ontology", "object", "objects", "link", "links", "semantic", "action"];
+const HIGH_IMPACT_TERMS = ["production", "deploy", "release", "delete", "migrate", "payment", "credential", "external action"];
+
+function escapeRegex(term: string): string {
+  return term.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+}
+
+function includesAny(input: string, terms: readonly string[]): boolean {
+  const lower = input.toLowerCase();
+  return terms.some(term => {
+    if (term.includes(" ")) return lower.includes(term);
+    return new RegExp(`(?<![a-z0-9])${escapeRegex(term)}(?![a-z0-9])`).test(lower);
+  });
+}
+
+export function selectEngineeringFleet(objective: string) {
+  const selected = new Set<string>(["intake"]);
+  const isData = includesAny(objective, DATA_TERMS);
+  const isApp = includesAny(objective, APP_TERMS);
+  const isOntology = includesAny(objective, ONTOLOGY_TERMS);
+
+  if (isData) {
+    selected.add("pipeline");
+    selected.add("quality");
+  }
+  if (isData || isApp || isOntology) selected.add("ontology");
+  if (isApp) selected.add("application");
+  selected.add("security");
+  selected.add("release");
+  selected.add("observer");
+
+  return ENGINEERING_FLEET.filter(role => selected.has(role.id));
+}
+
+export function engineeringMissionPlan(objective: string) {
+  return {
+    objective: objective.trim(),
+    stages: [...ENGINEERING_STACK_STAGES],
+    decisionLoop: [...ENGINEERING_DECISION_LOOP],
+    roles: selectEngineeringFleet(objective),
+    approvalRequired: includesAny(objective, HIGH_IMPACT_TERMS),
+    completionGates: [
+      "source and provenance known",
+      "contracts and ontology reviewed",
+      "tests and data expectations pass",
+      "security checks pass",
+      "downstream impact reviewed",
+      "rollback path exists",
+      "execution evidence recorded",
+    ],
+  };
+}
+
+export function buildEngineeringSystemContext(scope = "general engineering"): string {
+  return [
+    "Use the Zyra/XUNIA clean-room Palantir Engineering Stack for relevant engineering work.",
+    `Scope: ${scope}.`,
+    `Stages: ${ENGINEERING_STACK_STAGES.join(" -> ")}.`,
+    `Decision loop: ${ENGINEERING_DECISION_LOOP.join(" -> ")}.`,
+    `Bounded specialist fleet: ${ENGINEERING_FLEET.map(role => role.id).join(", ")}.`,
+    "Ground work in explicit data/schema/ontology/action/tool/UI contracts, lineage, quality gates, and evidence.",
+    "Give LLMs only the context and tools required for the workflow; distinguish read from write access.",
+    "Consequential writes and releases stay approval-gated, reversible, and auditable.",
+    "Prefer the lowest-complexity application surface that satisfies the workflow.",
+    "Do not claim Palantir affiliation, certification, tenant access, or proprietary implementation details.",
+  ].join(" ");
+}

server/intelligence.ts

@@ -1,3 +1,4 @@
+import { buildEngineeringSystemContext } from "./engineering-context";
 import { storage } from "./storage";
 
 export async function fetchCveDatabase(orgId: string) {
@@ -119,10 +120,14 @@ export async function analyzeSecurityImage(imageBase64: string, mimeType: string
     body: JSON.stringify({
       model: "google/gemma-3-27b-it",
       messages: [
+        {
+          role: "system",
+          content: `You are ZyraCopilot, an expert cybersecurity AI analyst. ${buildEngineeringSystemContext("cybersecurity vision analysis")}`,
+        },
         {
           role: "user",
           content: [
-            { type: "text", text: `You are ZyraCopilot, an expert cybersecurity AI analyst. ${prompt}` },
+            { type: "text", text: prompt },
             { type: "image_url", image_url: { url: dataUrl } },
           ],
         },