Add CI + production deploy + BYOK embedder - #18
Merged
Conversation
Gives app_ef a CI pipeline that validates every change, ships the app to apps.thorwhalen.com via the tw_platform / enlace stack on each push to main, and threads a per-request OpenAI key through to ef's embedder so the deployed backend holds no key of its own. # BYOK embedder * backend/app/main.py: map create_corpus's embedder_api_key parameter to the X-OpenAI-Key request header via qh's param_overrides. Header-mapped params are excluded from the OpenAPI request body, so the generated TS body type does not carry the key. * frontend/src/api/openaiKey.ts: SSOT for the localStorage key + the request-header name; assistant/engine.ts and api/client.ts both import it. * frontend/src/api/client.ts: createCorpus sends X-OpenAI-Key when a key is stored in localStorage; search/retrieve/explore reuse the corpus's bound embedder and need no header. * The flow degrades cleanly: no key + no server env → hashing fallback (lexical search), still works offline. * Pins ef[openai]>=0.1.17 (which carries embedder_api_key) and qh>=0.0.14 (which carries the HEADER param_override) in backend/pyproject.toml. * openapi.json + openapi.d.ts regenerated; embedder_api_key is absent from the request schema (only present in the human description). # CI pipeline (.github/workflows/ci.yml) * frontend job — checkout, pnpm install, typecheck (app + e2e), Playwright e2e, production build with VITE_PUBLIC_BASE/VITE_API_BASE matching production. * backend job — pip install -e ./backend[test] (pulls ef + qh from PyPI), pytest. * deploy job — on push to main, triggers tw_platform's deploy.yml via `gh workflow run -f app=app_ef`. deploy.py stays the single source of deploy truth; this job only triggers it. Needs TW_PLATFORM_DEPLOY_TOKEN (Actions: write on tw_platform). # Deployment wiring * server.py — thin enlace entry shim at the repo root, puts backend/ on sys.path and re-exports app.main:app. * app.toml — enlace app config: access=protected:user (allowed_users TBD; empty by default so the app fails closed until populated), build runs pnpm in frontend/, frontend_dir=frontend/dist, [python].packages lists ef[openai] + qh. * vite.config.ts — `base: VITE_PUBLIC_BASE ?? '/'`, explicit outDir 'dist'. * @zodal/* deps move from `link:` to npm version pins (^0.1.2 / ^0.1.0, current vs local source), so the build is self-contained — no sibling checkouts in CI or on the deploy runner; mirrors how acture is already consumed. Removes the deps:build / prebuild / predev scripts and the vite server.fs.allow widening. * Rename VITE_API_BASE_URL → VITE_API_BASE (the tw_platform convention). In production deploy.py injects VITE_API_BASE=/api/app_ef and VITE_PUBLIC_BASE=/app_ef/. * README.md + frontend/README.md updated for the new env var, npm-pinned zodal, BYOK, and CI/deploy. # Tests * backend/tests/test_app.py: BYOK threading (header → embedder_api_key), keyless fallback, embedder_api_key absent from the request schema. * All existing backend + frontend tests stay green. # Prerequisites (manual) * Set the TW_PLATFORM_DEPLOY_TOKEN repo secret (Actions: write on tw_platform). * Grant the deploy GitHub App contents:read on app_ef. * Add the user email(s) to app.toml's allowed_users (or switch access to protected:shared) before merging. * tw_platform PR #21 (registering app_ef) must merge — itself stacked on the tw_platform deploy-workflow PR (#13/#20). Refs the deploy-design discussion (no single issue — multi-repo work).
… root The wads template's Python .gitignore had unscoped 'lib/' and 'lib64/' entries, which silently matched any 'lib/' directory in the tree — including frontend/src/lib/. As a result frontend/src/lib/utils.ts (the standard shadcn 'cn()' helper) was never tracked: local typecheck passed because the file was on disk, but the first CI run failed with TS2307 'Cannot find module @/lib/utils' across nine components. - Scope the Python ignores to the repo root: lib/ → /lib/, lib64/ → /lib64/. The Python build artifacts only ever appear at the top level; nested lib/ directories (frontend, future packages) must not be dropped. - Track frontend/src/lib/utils.ts. Refs #18.
app_ef is a public repo, so committing per-user email allow-lists is undesirable. Switch to a shared-password gate: access = "protected:shared" shared_password_env = "APP_EF_SHARED_PASSWORD" The env var must hold the argon2id HASH of the password (not plaintext), generated with 'python -m enlace_auth hash-password'. Set on the deploy server before the first real deploy; otherwise the login form fails closed. Refs #18.
Owner
Author
|
Manual prereqs (GitHub App + repo secrets) tracked in thorwhalen/tw_platform#22. Once the App exists, the agent will set |
Owner
Author
Inheriting from tw_platform#22 — remaining app_ef onboarding checklisttw_platform#22 is being closed (its core deliverable — the GitHub App + Outstanding for this PR to be mergeable + functional
Verification once the above is in place
Refs tw_platform#22 (closed), tw_platform#21 (closed — needs reprise), tw_platform#13 (parent). |
Conflict: frontend/src/assistant/engine.ts import block. main renamed the store helper `appContext` -> `assistantContext` (it marks the dispatch `origin: 'assistant'`) and kept the OpenAI key location as an inline `KEY_STORAGE` const. This branch had extracted that constant into `@/api/openaiKey` as `OPENAI_KEY_STORAGE`, so the assistant engine and the API client share one source of truth without importing each other. Resolved by taking both: main's `assistantContext` import and this branch's `OPENAI_KEY_STORAGE` import. The inline `KEY_STORAGE` is dropped -- the file body already calls `assistantContext()` and `OPENAI_KEY_STORAGE`, so this is what the merged code actually references. tsc --noEmit passes. Claude-Session: https://claude.ai/code/session_01Dq9GZDVCd3oGmEeehN3Cgm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bundled deliverable: app_ef gets a CI pipeline, a production deploy to
apps.thorwhalen.com/app_ef/via the tw_platform / enlace stack, and abring-your-own-key embedder so the deployed backend holds no OpenAI key.
Per the plan agreed with the user. Independent of #17 (orthogonal; both branch from
main).What ships
BYOK embedder
EfService.create_corpus's newembedder_api_key(inef@0.1.17, already merged + published) is mapped onto theX-OpenAI-Keyrequest header viaqh'sparam_overrides. Header-mapped params are excluded from the OpenAPI request body — the TS body type does not carry the key.frontend/src/api/openaiKey.tsis the SSOT for the localStorage key + header name;client.tssends the header oncreateCorpuswhen a key is stored.search/retrieve/explorereuse the corpus's bound embedder and need no header.CI pipeline (
.github/workflows/ci.yml)VITE_PUBLIC_BASE=/app_ef/+VITE_API_BASE=/api/app_ef.pip install -e ./backend[test](pullsef[openai]>=0.1.17+qh>=0.0.14from PyPI), pytest.main, triggers tw_platform'sdeploy.yml(gh workflow run -f app=app_ef).deploy.pystays the single source of deploy truth.Deployment wiring
server.py— thin enlace entry shim (putsbackend/onsys.path, re-exportsapp.main:app).app.toml—access="protected:shared"+shared_password_env="APP_EF_SHARED_PASSWORD",[build]runs pnpm infrontend/,frontend_dir=frontend/dist,[python].packages = [ef[openai]>=0.1.17, qh>=0.0.14].vite.config.ts—base: VITE_PUBLIC_BASE ?? '/', explicitoutDir: 'dist'.@zodal/*deps move from locallink:to npm version pins (^0.1.2/^0.1.0— published versions are current vs local source). Build is now self-contained: no sibling checkouts in CI or on the deploy runner. Mirrors howactureis already consumed. Droppeddeps:build/prebuild/predevscripts + thevite server.fs.allowwidening.VITE_API_BASE_URL→VITE_API_BASE(the tw_platform convention).Fixes since first CI run
.gitignorehad unscopedlib/andlib64/(Python build-artifact ignores from the wads template) which silently swallowedfrontend/src/lib/. As a resultfrontend/src/lib/utils.ts(the standard shadcncn()helper) was never tracked, and the first CI run failed withTS2307: Cannot find module '@/lib/utils'across nine components. Scoped both ignores to the repo root (/lib/,/lib64/) and tracked the file. Local typecheck passed before this fix because the file was on disk.Tests
embedder_api_keyabsent from the request schema.Prerequisites — must be in place before merging
13-ci-deploy-workflow) — theworkflow_dispatchdeploy workflow. Must merge to tw_platformmainfirst.mainonce #20 merges, then merge it.TW_PLATFORM_DEPLOY_TOKENsecret in this repo — fine-grained PAT (or GitHub App token) with Actions: write onthorwhalen/tw_platform.thorwhalen/app_ef(so tw_platform'sdeploy.ymlcan check it out).APP_EF_SHARED_PASSWORDserver env var — set the argon2id hash of the chosen shared password before the first real deploy. Generate withpython -m enlace_auth hash-passwordon any machine withenlace_authinstalled; copy the printed hash (starts with$argon2id$...) into the server's environment (systemdEnvironment=line or wherevertw_platformsources app secrets). Without this, the login form fails closed.Done already (no action needed)
ef@0.1.17— published to PyPI, carriesembedder_api_key.qh@0.0.14— already on PyPI with the headerparam_overridesupport.protected:shared(no per-user emails committed to this public repo).