Oserhiienko/knowledge base - #3800
Open
Aleksashka11 wants to merge 134 commits into
Open
Conversation
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: routing
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: user-settings-storage
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: access-data
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: db-in-docker
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: db-in-plugin
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: register-identifiers
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: create-custom-file-viewers
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: file-exporters
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: file-handlers
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: folder-content-preview
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: column-tooltip
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: custom-cell-renderers
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: custom-package-settings-editors
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: data-enrichments
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: docker-containers
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: publish-packages
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: work-with-package-files
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: show-formula-lines
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: custom-views
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: layouts
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: extensions
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: js-api
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: rest-api
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: cheminformatics
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: docking
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: define-semantic-type-detectors
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: customize-grid
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: python-functions
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: context-actions
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: manipulate-viewers
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: custom-package-settings-editors Knowledge: DG-FACT-118 Knowledge: DG-FACT-119 Knowledge: DG-FACT-120 Knowledge: DG-FACT-121
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: data-enrichments Knowledge: DG-FACT-122 Knowledge: DG-FACT-123 Knowledge: DG-FACT-124 Knowledge: DG-FACT-125 Knowledge: DG-FACT-126 Knowledge: DG-FACT-127 Knowledge: DG-FACT-448
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: home-page-widgets Knowledge: DG-FACT-137 Knowledge: DG-FACT-138 Knowledge: DG-FACT-139 Knowledge: DG-FACT-140 Knowledge: DG-FACT-141 Knowledge: DG-FACT-142 Knowledge: DG-FACT-143
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: manage-credentials Knowledge: DG-FACT-346 Knowledge: DG-FACT-347 Knowledge: DG-FACT-348 Knowledge: DG-FACT-349 Knowledge: DG-FACT-350 Knowledge: DG-FACT-351
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: publish-packages Knowledge: DG-FACT-150 Knowledge: DG-FACT-151 Knowledge: DG-FACT-152 Knowledge: DG-FACT-153 Knowledge: DG-FACT-156 Knowledge: DG-FACT-450 Knowledge: DG-FACT-451 Knowledge: DG-FACT-452 Knowledge: DG-FACT-453 Knowledge: DG-FACT-454 Knowledge: DG-FACT-455 Knowledge: DG-FACT-456 Knowledge: DG-FACT-458
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: python-functions Knowledge: DG-FACT-157 Knowledge: DG-FACT-158 Knowledge: DG-FACT-159 Knowledge: DG-FACT-160 Knowledge: DG-FACT-161 Knowledge: DG-FACT-162 Knowledge: DG-FACT-163 Knowledge: DG-FACT-164
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: work-with-package-files Knowledge: DG-FACT-165 Knowledge: DG-FACT-166 Knowledge: DG-FACT-167 Knowledge: DG-FACT-168 Knowledge: DG-FACT-169 Knowledge: DG-FACT-170 Knowledge: DG-FACT-171 Knowledge: DG-FACT-172
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: add-package-tests Knowledge: DG-FACT-284 Knowledge: DG-FACT-285 Knowledge: DG-FACT-286 Knowledge: DG-FACT-287 Knowledge: DG-FACT-288 Knowledge: DG-FACT-289 Knowledge: DG-FACT-290 Knowledge: DG-FACT-291 Knowledge: DG-FACT-292 Knowledge: DG-FACT-293
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: test-packages Knowledge: DG-FACT-352 Knowledge: DG-FACT-353 Knowledge: DG-FACT-354 Knowledge: DG-FACT-355 Knowledge: DG-FACT-356 Knowledge: DG-FACT-357 Knowledge: DG-FACT-358 Knowledge: DG-FACT-359
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: add-info-panel Knowledge: DG-FACT-276 Knowledge: DG-FACT-277 Knowledge: DG-FACT-278 Knowledge: DG-FACT-279 Knowledge: DG-FACT-280 Knowledge: DG-FACT-281 Knowledge: DG-FACT-282 Knowledge: DG-FACT-283
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: custom-filters Knowledge: DG-FACT-265 Knowledge: DG-FACT-266 Knowledge: DG-FACT-267 Knowledge: DG-FACT-268 Knowledge: DG-FACT-269 Knowledge: DG-FACT-270 Knowledge: DG-FACT-271 Knowledge: DG-FACT-272 Knowledge: DG-FACT-273 Knowledge: DG-FACT-274 Knowledge: DG-FACT-275
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: develop-custom-viewer Knowledge: DG-FACT-186 Knowledge: DG-FACT-187 Knowledge: DG-FACT-188 Knowledge: DG-FACT-189 Knowledge: DG-FACT-190 Knowledge: DG-FACT-191 Knowledge: DG-FACT-193 Knowledge: DG-FACT-194 Knowledge: DG-FACT-195 Knowledge: DG-FACT-196 Knowledge: DG-FACT-197 Knowledge: DG-FACT-460 Knowledge: DG-FACT-461
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: manipulate-viewers Knowledge: DG-FACT-198 Knowledge: DG-FACT-199 Knowledge: DG-FACT-200 Knowledge: DG-FACT-201 Knowledge: DG-FACT-202 Knowledge: DG-FACT-203 Knowledge: DG-FACT-204 Knowledge: DG-FACT-205 Knowledge: DG-FACT-206
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: show-formula-lines Knowledge: DG-FACT-207 Knowledge: DG-FACT-208 Knowledge: DG-FACT-209 Knowledge: DG-FACT-210 Knowledge: DG-FACT-211 Knowledge: DG-FACT-212
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: custom-views Knowledge: DG-FACT-213 Knowledge: DG-FACT-214 Knowledge: DG-FACT-215 Knowledge: DG-FACT-216 Knowledge: DG-FACT-217 Knowledge: DG-FACT-224
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: layouts
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: create-package Knowledge: DG-FACT-440 Knowledge: DG-FACT-441 Knowledge: DG-FACT-462 Knowledge: DG-FACT-463 Knowledge: DG-FACT-464
Authored via skill-harness cycle. See harness commit log for cycle details (critic verdicts, executor floor, attestations). Skill: docker-containers Knowledge: DG-FACT-129 Knowledge: DG-FACT-130 Knowledge: DG-FACT-131 Knowledge: DG-FACT-132 Knowledge: DG-FACT-133 Knowledge: DG-FACT-134 Knowledge: DG-FACT-135 Knowledge: DG-FACT-136
…om harness Pulls RUN-3/4 extract-knowledge additions accumulated during the skill batch: +1786 lines in knowledge-graph.md (new facts for all RUN-3/4 topics) +24 lines in component-registry.md +33 lines in api-map.md Six public-only rows superseded by re-emitted equivalents in the harness output.
…T citations
knowledge-graph.md — 6 ID collisions resolved by renumbering second occurrence:
- DG-FACT-427 (cache) → DG-FACT-465
- DG-FACT-428 (custom-cell-renderers) → DG-FACT-466
- DG-FACT-429 (custom-cell-renderers) → DG-FACT-467
- DG-FACT-430 (decorator-meta-typing) → DG-FACT-468
- DG-FACT-462 (docker-containers) → DG-FACT-469
- DG-FACT-463 (docker-containers) → DG-FACT-470
_facts.yaml — removed 12 duplicate occurrences of 4 DRIFT IDs
(DG-FACT-DRIFT-DBPLUGIN-001..004 each appeared 4×).
Skills — citation updates following renumber:
- cache-function-results: DG-FACT-427 → DG-FACT-465 (3×)
- custom-cell-renderers: DG-FACT-428 → DG-FACT-466 (4×),
DG-FACT-429 → DG-FACT-467 (1×),
DG-FACT-430 → DG-FACT-468 (3×)
Skills — stripped 20 orphan DRIFT citations (cited IDs that had no
matching `- id:` entry in knowledge-graph.md):
cheminformatics, docker-containers, docking, extensions, js-api,
manipulate-viewers, rest-api (-3.4KB across 7 files).
Skills — removed two singular-named duplicates (write-demo-script,
test-package) superseded by the plural-named harness-authored versions
(write-demo-scripts, test-packages).
Verification: all 344 unique DG-FACT-* IDs cited by skills now resolve
to a defined `- id:` in knowledge-graph.md. Zero orphans.
…lattice scaffolding
Restructure the .claude/skills/ knowledge architecture so each skill owns
only the facts it actually cites, with the master knowledge files relocated
out of help/ and into a dedicated .claude/knowledge/_master/ fallback.
Phase 1 — Shard knowledge by skill citations.
Parsed 472 DG-FACT-* entries from help/develop/agent-guide/knowledge-graph.md
(handling fenced YAML blocks, markdown comments, and DRIFT-section
boundaries). For each of the 45 skills with >=1 citation, wrote
.claude/knowledge/<topic>/facts.yaml containing the cited fact blocks in
numerical-then-DRIFT order with a generated-from header. 18 skills with
zero citations were left without a facts.yaml. Total: 368 facts written
across 45 files (every per-skill file's fact count matches its skill's
unique-citation count). All facts.yaml files strict-parse as YAML
(achieved by normalizing `value:` scalars into block-scalar form to escape
embedded colons/backticks; the source master file did not strict-parse).
The 4 master files moved via git mv (history preserved):
help/develop/agent-guide/knowledge-graph.md -> .claude/knowledge/_master/
help/develop/agent-guide/component-registry.md-> .claude/knowledge/_master/
help/develop/agent-guide/api-map.md -> .claude/knowledge/_master/
help/develop/agent-guide/_facts.yaml -> .claude/knowledge/_master/
Phase 2 — Reference per-skill facts.yaml from SKILL.md.
Inserted a "## Cited facts" section after the frontmatter in each of the
45 skills with a facts.yaml file, linking to ./facts.yaml so the importer
can resolve DG-FACT-NNN citations locally instead of loading the full
6.6k-line master graph.
Phase 3 — Strip Verification sections.
Removed dedicated "## Verification" / "## Verifying" / "## Verify" sections
from 46 SKILL.md files, deleting 22017 bytes. Inline `Expected:` lines
within step lists were preserved (they're load-bearing instructions, not
verification-section scaffolding).
Phase 4 — Trim Prerequisites to non-obvious only.
Audited 148 Prerequisites bullets across 47 skills. Stripped 54 bullets
that named only obvious environment assumptions (Node.js / npm / Git /
TypeScript / "Datagrok platform running" / generic "a package scaffold
with paths relative to root"). Kept 94 bullets that named a concrete
state requirement (specific role, saved layout, semantic type, named
database connection, JDK/Maven/Docker for the test-connectors module,
PowerPack requirement for data-enrichments, etc.). 4 Prerequisites
sections collapsed entirely after their last bullet was stripped and
were removed. 35 skills had their Prerequisites edited.
Phase 5 — Remove baked lattice / harness-framework references.
Cleaned 41 SKILL.md files:
- Removed `harness-authored: true` / `: false` frontmatter lines.
- Substituted `{{ lattice.harness.help_develop_root }}` template
placeholders with `help/develop` (their resolved value in this repo).
- Substituted `<help_develop_root>` placeholders with `help/develop`.
- Substituted `<harness_root>/docs/_internal/knowledge/knowledge-graph.md`
with the new master path `.claude/knowledge/_master/knowledge-graph.md`.
- Dropped `(mirror: <harness_root>/docs/_internal/articles-mirror/...)`
parenthetical pointers — the mirror lives outside this repo.
- Removed `/lattice:`, `/skills-harness:`, `@lattice/`,
`lattice/skills-harness/` tokens wherever they appeared.
- Replaced the prose label `harness-authored` with `authored`.
"the harness" mentions in the Datagrok-product sense (test harness via
`grok test`, per-package `CLAUDE.md` harness taught by
`prepare-package-for-claude`) were preserved — those are real product
concepts, not framework scaffolding.
Phase 6 — Cross-checks.
- Every DG-FACT-NNN cited by any SKILL.md resolves to either its own
local facts.yaml or the master knowledge-graph.md (excluding the
literal `DG-FACT-NNN` placeholder in the inserted import section).
- No `^##+ Verif` headings remain.
- No `harness-authored:` frontmatter remains.
- No `/lattice:` or `/skills-harness:` tokens remain.
- All 45 .claude/knowledge/<topic>/facts.yaml files strict-parse as
valid YAML.
Files untouched per instructions: everything under packages/Grokky/ — its
working-tree changes are unrelated merge artifacts and were not staged.
Progress log: /Users/aleksashka/Desktop/datagrok/skill-harness/.lattice/cleanup-progress.log
…red versions Each legacy skill covered the same topic as a richer harness-authored counterpart with proper triggers, version, allowed-tools frontmatter and DG-FACT citations. Mapping (legacy → kept): build-app → build-an-app write-tutorial → write-tutorials create-cell-renderer → custom-cell-renderers create-custom-filter → custom-filters create-custom-view → custom-views create-custom-viewer → develop-custom-viewer create-file-viewer → create-custom-file-viewers create-script-handler → custom-script-handlers create-semantic-type-detector → define-semantic-type-detectors create-docker-container → docker-containers Two cross-references in remaining skills updated to point at the kept versions: `create-interactive-scientific-application-spec` and `cheminformatics`. Final skill count: 53 (was 63).
Each skill's facts.yaml moves from `.claude/knowledge/<topic>/facts.yaml`
to live next to its SKILL.md at `.claude/skills/<topic>/facts.yaml` —
the relative link `[facts.yaml](./facts.yaml)` in every SKILL.md now
resolves correctly (previously broken since the file was in a sibling
tree).
`.claude/knowledge/` is flattened: the four master files
(knowledge-graph.md, component-registry.md, api-map.md, _facts.yaml)
move out of `_master/` directly into `.claude/knowledge/`, which now
contains nothing but those four global registries.
Also removed an orphan `.claude/knowledge/build-app/facts.yaml` left
over from the legacy-skill cleanup (the build-app SKILL.md was deleted;
its facts.yaml had not been).
Layout now:
.claude/skills/<topic>/SKILL.md — instructions
.claude/skills/<topic>/facts.yaml — cited facts (sibling)
.claude/knowledge/{knowledge-graph,component-registry,api-map}.md
.claude/knowledge/_facts.yaml
Since each skill now has a sibling facts.yaml that the model auto-loads,
SKILL.md prose that restated fact bodies became redundant. This commit
strips that duplication while preserving every DG-FACT-XXX citation, all
user-executed code blocks, frontmatter, and structure.
44 skills processed across two passes:
Sequential pre-pass (10 skills, killed mid-run for parallelization):
cache-function-results (-45%), extensions (-14%), folder-content-preview,
custom-cell-renderers, file-exporters, build-an-app, upload-data,
customize-grid, define-semantic-type-detectors, intro
Parallel pass — 4 workers × 8-9 skills each:
W1 (9 skills, -27.9% total): access-data, column-tooltip, custom-filters,
data-enrichments, docker-containers, js-api, python-functions,
show-formula-lines, write-demo-scripts
W2 (9 skills, -23.8% total): add-info-panel, context-actions,
custom-package-settings-editors, db-in-docker, docking,
manage-credentials, register-identifiers, test-packages, write-tutorials
W3 (8 skills, -26.8% total): add-package-tests, create-custom-file-viewers,
custom-script-handlers, db-in-plugin, file-handlers, manipulate-viewers,
rest-api, user-settings-storage
W4 (8 skills, -22.8% total): cheminformatics, create-package, custom-views,
develop-custom-viewer, home-page-widgets, publish-packages, routing,
work-with-package-files
9 skills skipped (no facts.yaml — legacy/RUN-1 era):
add-curve-format, create-interactive-scientific-application-spec,
create-workflow, datagrok-logging,
implement-interactive-scientific-application-from-spec, layouts,
prepare-package-for-claude, test-connectors, ui
Trim patterns: replaced verbose paragraph explanations of facts with
one-line summaries + DG-FACT cites; collapsed multi-paragraph "Anti-pattern"
and "Drift" call-outs into compact bullets; removed production code-line
references that already appear in fact derives_from; stripped article-vs-code
drift narration when the fact documents the resolution.
Total ~67 KB removed across worker batch (~26% average per skill) plus
~22 KB from the sequential pre-pass.
.claude/knowledge/_facts.yaml is removed. It was a strict subset of knowledge-graph.md (301 of 472 fact-ids), originally written as an append-only log by commit-to-public.sh before per-skill facts.yaml shards existed. Now that every skill has its own sibling facts.yaml, the global _facts.yaml is dead weight — its content lives in two places (per-skill files + master graph). packages/Grokky/dockerfiles/claude-runtime/.credentials.json is removed and added to .gitignore. Credentials should never live in the public repo regardless of value — even rotated secrets become attack surface in commit history.
Picked up a backup variant of the credentials file that survived the earlier removal. Also broadened the gitignore pattern from `**/.credentials.json` to `**/.credentials.json*` so .bak / .tmp / .old variants can't sneak back either.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There are a few things going on here:
Help articles cleanup
A number of help articles were fixed — including broken / non-compiling code, stale links, and outdated examples (e.g., using deprecated APIs instead of the current API we recommend). Most of these issues were surfaced by agents while preparing and testing the skills and generated code.
Skills derived from how-to articles
Some skills were generated based on the how-to articles — not as literal copies, but as structured, machine-readable artifacts
During this process, agents often had to resolve inconsistencies or gaps, which led to useful “facts” or implicit knowledge. Initially, I collected this in knowledge-graph.md, but later split it so that each skill contains its own relevant facts.
At this point, I’d suggest dropping knowledge-graph.md entirely and keeping the facts co-located with their respective skills.
Skills structure
The current structure is partially borrowed from GStack. It worked well in practice during testing, but I see it as a starting point — we can adjust or redesign it based on what we agree is optimal (because currently all our skills differ in all terms and ideally we want to come to the point where everything will be written the same way)
Supporting docs (api-map.md, component-registry.md)
These files are relatively small and mat provide good value. They help with discoverability without significantly increasing context size