Modernize for LangSmith Deployment: SaaS and self-hosted, hybrid removed - #14
Merged
Merged
Conversation
The repo was ~10 months stale and its deployment script could only ever have worked against one hosting model. Deployment - Split the control plane client by hosting model. The API restricts the deployment source per model: Cloud SaaS requires `source: github`, self-hosted requires `source: external_docker`. The old script sent external_docker for everything, so the SaaS path could not work. - Replace the hardcoded internal `gtm.smith.langchain.dev` host with resolved hosts: api.host.langchain.com (+ eu/apac/aws regions) for SaaS, and <host>/api-host for self-hosted. - Send the now-required X-Tenant-Id workspace header. - Read the deployment URL from the API's `url` field instead of guessing https://<name>.langchain.dev. - Add revision polling, request timeouts and host validation. Security - Stop printing the request payload (which included the OpenAI key value) and response.request.headers (which included X-Api-Key) into CI logs. - Pass credentials to scripts via the environment rather than argv. - Replace the preview workflow's `workflow_run` trigger with `workflow_call`. `workflow_run` executes in the base repo with access to every secret while building a Docker image from PR-head code, exposing credentials to forks. - Add explicit fork guards, per-job permissions and job timeouts. - Pass branch names through env instead of interpolating them into shell. Dependencies - LangChain 0.3 -> 1.3, LangGraph 0.5 -> 1.2, langgraph-cli 0.0.19 -> 0.4.31, langsmith 0.4 -> 0.11, openevals 0.1 -> 0.2. - Drop langchain-community, which was sunset upstream with no replacement. Only SQLDatabase.get_usable_table_names and .run were used; both are now backed by SQLAlchemy directly, removing 6 packages. - Bump all GitHub Actions and pre-commit hooks to current versions. Docs and tests - Remove the Hybrid deployment model throughout; document SaaS and self-hosted as distinct paths with their required config. - Add tests/deployment covering both target paths against a mocked control plane, including that secrets never reach stdout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by deploying against a real self-hosted LangSmith instance. Deployment hang - agents/simple_text2sql.py built the Chinook database at module import, which fetches a SQL file over the network. The Agent Server imports the graph during startup, so on a cluster with restricted egress that import blocks and the deployment times out after 600s with no useful error. Observed exactly this on a live instance. - SQLDatabase now accepts an engine factory and resolves it on first use, so importing the graph performs no I/O. - Cache get_engine_for_chinook_db and get_detailed_table_info. The schema is static, but generate_sql rebuilt the whole database on every call. - Add raise_for_status so a failed fetch reports the HTTP error. CI output was invisible - Python block-buffers stdout when it is a pipe, so a ten-minute deployment printed nothing until it finished and the job looked hung. All three CLIs now enable line buffering. PR comment could claim success on a failed deploy - A deployment sits at READY while its newest revision is DEPLOY_FAILED, so the comment heading rendered a green tick over a failed deploy. The heading now leads with the worse of the two states. Unhelpful failure reason - This control plane version omits status_message from the revision response, which surfaced as "no detail provided". Point at the revision server logs instead. Also regenerate the Dockerfile with langgraph-cli 0.4.31; the committed one was produced by an old CLI and installed local deps with an `-e /deps/*` glob. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only the self-hosted path builds images, and it defaults to Docker Hub. Spell out how to point it at GHCR, ECR, Google Artifact Registry or ACR instead. ECR gets its own note because docker/login-action cannot authenticate to it with a static username and password; it needs aws-actions/amazon-ecr-login. Also flag that the cluster must be able to pull from whichever registry is chosen, which means an imagePullSecret for a private one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Creating a deployment also creates a LangSmith tracing project of the same name, but deleting the deployment does not delete the project. The name stays claimed, so reopening a pull request whose preview was already cleaned up fails with a bare 409 that does not say what to do. Hit this against a live self-hosted instance while re-testing a preview. Raise NameConflictError (a ControlPlaneError, so existing handling is unchanged) naming the leftover project and how to clear it, and document the case in the pipeline troubleshooting guide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A preview failed to deploy on a live self-hosted cluster with "0/2 available nodes have sufficient CPU" while the cluster sat at 5-11% utilisation. The queue runs as its own deployment and queue_cpu defaults to cpu, so asking for 1 CPU actually reserves 2 before Postgres and Redis are counted. Nothing in the script exposed that, and there was no way to lower it. - Add --queue-cpu and --queue-memory-mb, and document that leaving them unset doubles the CPU request. - patch_deployment now accepts source_config so resources can be changed on an existing deployment instead of deleting and recreating it. Recreating is worse than it sounds, because the deleted deployment's tracing project keeps the name reserved. - Only send resource_spec when patching. integration_id, repo_url, deployment_type and listener_id are fixed at creation, and resending them would have broken every SaaS revision. Covered by a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerfiles/Dockerfile used langchain/langgraph-api:3.11-wolfi, a floating tag. Rebuilding the same commit later produces a different Agent Server, and because the base image's /api/constraints.txt bounds what this project can install, it also changes the resolved dependency set. A freshly built image today carries langgraph-api 0.13.0; one built from this repo last October carried 0.4.38. That gap matters on self-hosted, where the Agent Server has to be compatible with the LangSmith platform running it, so the pin carries a comment saying to match it to your platform version. Note that `base_image` in langgraph.json cannot express this: the CLI appends `:<python>-<distro>` to whatever it is given, so a value with a tag produces a malformed `repo:0.13.0:3.11-wolfi`. The pin has to live in the Dockerfile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A deployment provisions four workloads -- agent, queue, Postgres and Redis -- each with its own CPU and memory request. The script only ever set the first two, so a preview stuck at "0/2 available nodes have sufficient CPU" could not be made to fit: lowering the agent and queue left the Postgres StatefulSet at its default, and that was the pod failing to schedule. - Add --resource-spec, a JSON blob merged over the named flags, covering db_cpu, db_memory_mb, db_storage_gi, redis_cpu, redis_memory_mb and anything else the API grows later, without adding twenty flags. - Add --action wait, which blocks on a deployment's newest revision. Previously the only way to watch an existing revision was to start another deploy, which created a redundant revision. The control plane enforces minimums (redis_memory_mb must be at least 512), and those come back as a clear 400 that the client now surfaces verbatim. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oes not scale Four attempts to deploy a preview to a live self-hosted cluster all failed the same way: ~600s in DEPLOYING, then DEPLOY_FAILED with no reason attached. The cause was not the image. Deploying a known-good image as a new deployment failed identically while the existing deployment stayed healthy, which isolates it to cluster capacity. Write down what that cost to learn: - A deployment provisions four workloads, not one. queue_cpu defaults to cpu, so --cpu 1 reserves two cores, and db_* / redis_* are not touched by --cpu at all. - With a preview per pull request, every open PR costs its own Postgres and Redis. Point them at shared datastores with POSTGRES_URI_CUSTOM and REDIS_URI_CUSTOM instead. - Add the known-good-image check to troubleshooting, since it separates "my image is broken" from "the cluster is full" in one step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five preview deployments to a live self-hosted cluster reported DEPLOY_FAILED after ten minutes with no reason. The pods were healthy and serving the whole time. The operator log had the answer: admission webhook "vscaledobject.kb.io" denied the request: HPA name "keda-hpa-text2sql-agent-pr-9004-f8b7994671e1515582608a584da304d4" is 64 characters long; must be no more than 63 Self-hosted autoscaling uses KEDA, which derives an HPA named keda-hpa-<deployment>-<32-char-hash>. That overhead is 42 characters, so the deployment name gets 21. Postgres, Redis, the queue and the agent all reconcile and serve traffic; only the autoscaler is rejected, so the revision is never marked ready and dies on the platform timeout. A working deployment reported as failed, with nothing in the API to say why. This is not specific to that cluster. Previews are named text2sql-agent-pr-<n> = 18 + digits, so the default prefix silently breaks the first time a repository reaches PR #1000. Validate the length before deploying and fail immediately with the arithmetic and the fix. Renaming the same deployment from 22 characters to 13, changing nothing else, took it to DEPLOYED with both KEDA ScaledObjects READY. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit validated the KEDA name limit everywhere, which meant an over-long deployment created before the rule existed could no longer be deleted or inspected -- the check fired on cleanup and refused to look it up. That is precisely backwards: those are the deployments most in need of removing. Split it into validate_deployment_name (character set, used for any lookup) and validate_new_deployment_name (also length, used only on create). Verified against the live cluster: text2sql-agent-pr-9004 now deletes cleanly, while creating text2sql-agent-pr-1000 is still refused with the explanation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit made an over-long name fail loudly instead of silently, but the default prefix was still one that outgrows the limit: text2sql-agent-pr-<n> fits only while pull request numbers stay below 1000. Reporting a bug the default walks into is not a fix. Change the default prefix to `text2sql`, which leaves room for a six-digit pull request number (text2sql-pr-999999 derives a 60-character HPA, against a limit of 63). Expose DEPLOYMENT_NAME_PREFIX in both deploy workflows so it can be overridden with a repository variable, and stop hardcoding the deployment name in the reporting steps, which would otherwise drift from the prefix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
.gitignore covered .env, .env.local and .env.*.local. A file named .env.saas, .env.prod or .env.staging was tracked, which is exactly the sort of file people create when juggling credentials for more than one environment -- and I created one while testing. Ignore .env.* outright and keep .env.example as the single explicit exception. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Organisations that no longer issue provider keys to developers could not run
this agent at all: it hardcoded ChatOpenAI("gpt-4o-mini") and the pipeline
forwarded OPENAI_API_KEY.
build_llm now supports three routes, auto-detected in this order and
overridable with LLM_PROVIDER:
gateway LangSmith LLM Gateway. OpenAI-compatible, authenticates with
LANGSMITH_API_KEY, needs no provider key. Model IDs are
provider-qualified, e.g. anthropic/claude-haiku-4-5-20251001.
LangSmith Cloud injects LANGSMITH_API_KEY into deployments, so
this works there with no secret to forward -- in fact the control
plane rejects LANGSMITH_API_KEY as a reserved variable name.
anthropic Anthropic directly, with ANTHROPIC_API_KEY.
openai OpenAI directly, the previous behaviour.
Verified live: the full text2sql graph answers correctly through the gateway
against anthropic/claude-haiku-4-5-20251001 with no provider key present.
Also stop passing override=True to load_dotenv. It let a stale local .env
silently beat environment variables that CI or a deployment set deliberately,
which cost real debugging time here -- a self-hosted key in .env overrode the
Cloud key and surfaced as an opaque 403 from the gateway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Document the gateway, direct-Anthropic and direct-OpenAI routes in the README, and record two things learned deploying to Cloud: - LANGSMITH_API_KEY is a reserved deployment variable. The control plane returns 400 if you forward it, because Cloud injects its own. The gateway route therefore needs no credential forwarded at all -- only the base URL and model. - Gateway model IDs are provider-qualified (anthropic/claude-...), unlike the bare IDs used when calling a provider directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A revision that is crash-looping or stuck building holds the queue: later revisions sit in QUEUED behind it until it times out, so one bad deploy stalls every push after it. Hit this on Cloud. A revision built before the gateway change could not import the graph, crash-looped, and held DEPLOYING for twenty minutes while the fix sat QUEUED behind it. POSTing to the revision's interruption endpoint moved the old one to INTERRUPTED and the queued one to BUILDING immediately. Add `--action interrupt`, which cancels the newest revision when it is still in progress and no-ops when it has already settled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two loose ends from a coherence pass over the branch: - helpers/create_datasets.py and examples/langgraph_client.py still passed override=True to load_dotenv. Only the agent had been fixed, so the same footgun remained in the two other entry points: a stale local .env silently beating environment variables set deliberately. - The interrupt action was the only CLI action missing from the docs. Give it a section, since the failure it addresses -- a stuck revision holding the queue and stalling every later push -- is not obvious from the outside. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first real run of this pipeline failed at startup, before any job began: A reusable workflow cannot request more permissions than the job calling it. preview-deployment.yml needs pull-requests: write to post its deployment status comment, but the calling job inherited the caller's workflow-level default of contents: read, leaving pull-requests: none. GitHub rejects the entire run for that, with no job-level error to point at. Grant it explicitly on the calling job. actionlint does not catch this -- it is enforced by GitHub at run start, so only actually opening a pull request surfaces it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every job failed at "Set up job" with: Unable to resolve action `astral-sh/setup-uv@v10`, unable to find version `v10` astral-sh/setup-uv publishes only full semver tags; there is no floating v10. I had confirmed the v10.0.1 *release* existed and assumed the major tag followed, which is the convention for actions/* but not for this one. Pin all twelve references to v10.0.1, and verify every action reference in the workflows resolves against the tags API rather than the releases API. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the repo current (it was ~10 months stale), fixes the SaaS deployment
path, and removes the hybrid hosting model. Both hosting models were verified
end to end against live infrastructure.
The main bug
The SaaS path could never have worked. The script sent
source: external_dockerfor every deployment, which the control plane acceptsonly for self-hosted. Cloud requires
source: github. The client now resolveshost and source per hosting model:
api.host.langchain.com(+ regional)<host>/api-hostgithub— Cloud builds itexternal_docker— you build itIt also sends the now-required
X-Tenant-Idheader, and replaces a hardcodedinternal
gtm.smith.langchain.devhost.Verified live
DEPLOYED, KEDA autoscalers ready, serving/okDEPLOYED, and the deployed agent answered a realquestion — "We have 2 albums by AC/DC." — via Anthropic through the LLM
Gateway with no provider key present
Found by deploying, not by testing
reconciles and serves traffic but the autoscaler is rejected, so the revision
is never marked ready and fails on timeout with no reason. The old default
prefix would have broken this pipeline at PR #1000.
QUEUEDbehindit. Added an
interruptaction.queue_cpusilently doubles the CPU request, anddb_cpu/redis_cpuwere unreachable, making constrained clusters undeployable.
LANGSMITH_API_KEYis reserved on Cloud — forwarding it returns 400.Security
response request headers (including
X-Api-Key) into CI logsworkflow_runtrigger, which ran in the baserepo with every secret while building PR-head code, exposing credentials to
forks
github.head_ref; added request timeouts and hostvalidation
Also
langchain-communityfor plain SQLAlchemyNote for reviewers
The deploy jobs need two secrets that are not yet set:
LANGSMITH_WORKSPACE_IDandLANGSMITH_GITHUB_INTEGRATION_ID.