Add tpu v7x-16 for buildkite agent by meiyeh123 · Pull Request #401 · vllm-project/ci-infra

meiyeh123 · 2026-07-03T07:08:08Z

Description

This changes add tpu v7x-16 for buildkite by using Terraform

Addition:

Use 'ci-bk' infix for multi-host TPU VM names and labels to avoid conflicts with existing v7x TPU resources
Include the hostname in Buildkite agent tags to better identify the agent.

FYI:
The multi-host does not support attaching read-write mode disk (ref), and there isn't any way to scale up the vm boot disk size from even using the official gcloud CLI.
Therefore, a better solution for accessing the model weights is using runai_streamer instead of downloading the weights to the local disk.

Tests

buildkite (run the multihost test with the agent)

Checklist

Before submitting this PR, please make sure:

I have performed a self-review of my code.
I have necessary comments in my code, particularly in hard-to-understand areas.
I have made or will make corresponding changes to any relevant documentation.

Signed-off-by: Dennis Yeh <dennis.yeh@cienet.com>

theminghuang · 2026-07-09T17:27:13Z

Looks like this is very similar to ci_v7x/main.tf file, do we see a way to update that startup script and reuse them?
Also it would be great that user don't have to remember if the instance they are creating are multi-host so that they don't accidentally set accelerator_type tpu7x-2 to a multi host module. https://screenshot.googleplex.com/C9Tjr6H4FAMcQyQ this seems to suggest anything larger than 8 cores is multi host

Done. Combine them and also extract it to startup-script.sh.tftpl

theminghuang · 2026-07-09T17:59:09Z

+        UserKnownHostsFile /dev/null
+EOF
+
+      # Write the ephemeral private key generated by Terraform


The comment suggests the ssh key is the ephemeral but I think each time we run the cicd tests on multi hosts they need to communicate with each other. Would the key expire after certain time?

The key is considered ephemeral because its lifecycle is tied strictly to the Terraform deployment. It is
generated dynamically during apply, exists only within the state and the VM, and is discarded once the
infrastructure is destroyed. This avoids the need for long-term secret management in GCP Secret Manager.

But we can tweak the comments if you think it's better.

Got it, yeah lets remove the ephemeral part since it's a bit misleading imo.

theminghuang

We also store cache (other than model weights that can be solved with runai_streamer) in the attached disk so I think attach disk is required

theminghuang · 2026-07-09T18:30:39Z

We also store cache (other than model weights that can be solved with runai_streamer) in the attached disk so I think attach disk is required

is READ_WRITE_SINGLE an option https://screenshot.googleplex.com/BQzwHVFX7LEacet? I think each host get it's own disk if I'm not mistaken and it can claim the sole ownership

meiyeh123 · 2026-07-10T02:53:25Z

We also store cache (other than model weights that can be solved with runai_streamer) in the attached disk so I think attach disk is required

is READ_WRITE_SINGLE an option https://screenshot.googleplex.com/BQzwHVFX7LEacet? I think each host get it's own disk if I'm not mistaken and it can claim the sole ownership

I don't think so. There's a note below:
https://screenshot.googleplex.com/8CBqCCHV66smRHf

And we have tested it. Multi-host actually does not work with single-writer mode

Signed-off-by: dennis yeh <dennis.yeh@cienet.com>

theminghuang · 2026-07-10T04:51:57Z

+        UserKnownHostsFile /dev/null
+EOF
+
+      # Write the ephemeral private key generated by Terraform


Got it, yeah lets remove the ephemeral part since it's a bit misleading imo.

theminghuang · 2026-07-10T05:00:29Z

+      echo "Setting up backward compatibility symlink for JAX cache..."
+      # Create the persistent directory first
+      sudo mkdir -p /mnt/disks/persist/tpu_jax_cache
+      sudo chmod 777 /mnt/disks/persist/tpu_jax_cache
+
+      # Forcefully intercept old CI jobs writing to /tmp and redirect them to the persistent disk
+      sudo rm -rf /tmp/tpu_jax_cache
+      sudo ln -s /mnt/disks/persist/tpu_jax_cache /tmp/tpu_jax_cache
+
+      # ==========================================
+      # 2. Automated Disk Garbage Collection (Cron)
+      # ==========================================
+      echo "Setting up daily cron job for JAX cache cleanup..."
+      echo -e '0 2 * * * root find /mnt/disks/persist/tpu_jax_cache -type f -mtime +30 -delete > /dev/null 2>&1\n30 2 * * * root find /mnt/disks/persist/tpu_jax_cache -type d -empty -delete > /dev/null 2>&1' | sudo tee /etc/cron.d/tpu_cache_cleanup
+      sudo chmod 0644 /etc/cron.d/tpu_cache_cleanup


Should we remove these and other setup related to /mnt/disks/persist as we don't really have a mount disk? And it also make it clear that for tests running on multi host should not use cache (or it will fully occupy the boot disk)

Signed-off-by: Dennis Yeh <dennis.yeh@cienet.com>

- Use 'ci-bk' infix for multi-host TPU VM names and labels to avoid conflicts with existing v7x TPU resources - Include the hostname in Buildkite agent tags to better identify the agent. Signed-off-by: Dennis Yeh <dennis.yeh@cienet.com>

meiyeh123 force-pushed the dennis/add-v7x-16 branch from 1ac72c0 to ff6080b Compare July 3, 2026 07:13

CienetStingLin approved these changes Jul 7, 2026

View reviewed changes

meiyeh123 force-pushed the dennis/add-v7x-16 branch 2 times, most recently from c00f595 to b396299 Compare July 8, 2026 09:46

meiyeh123 marked this pull request as draft July 8, 2026 09:46

meiyeh123 added 3 commits July 9, 2026 10:55

add ci_v7x_16, and copy original modules to ci_v7x_multi

63f8b8b

Signed-off-by: Dennis Yeh <dennis.yeh@cienet.com>

for v7x-16

49d9f10

Signed-off-by: Dennis Yeh <dennis.yeh@cienet.com>

Use the tls_private_key for workers to ssh

d663939

Signed-off-by: Dennis Yeh <dennis.yeh@cienet.com>

meiyeh123 force-pushed the dennis/add-v7x-16 branch from b396299 to 5ad9f07 Compare July 9, 2026 07:52

meiyeh123 requested a review from CienetStingLin July 9, 2026 07:53

fix EOF indent

1c5e97c

Signed-off-by: Dennis Yeh <dennis.yeh@cienet.com>

meiyeh123 force-pushed the dennis/add-v7x-16 branch from 5ad9f07 to 1c5e97c Compare July 9, 2026 07:54

meiyeh123 marked this pull request as ready for review July 9, 2026 07:55

theminghuang reviewed Jul 9, 2026

View reviewed changes

Lumosis approved these changes Jul 9, 2026

View reviewed changes

theminghuang requested changes Jul 9, 2026

View reviewed changes

intergrate the single and multi host module

0bea70f

Signed-off-by: dennis yeh <dennis.yeh@cienet.com>

meiyeh123 force-pushed the dennis/add-v7x-16 branch from ce3b6a6 to 0bea70f Compare July 10, 2026 04:41

theminghuang reviewed Jul 10, 2026

View reviewed changes

tweak comment and does attached disk things only when single host

3d94c15

Signed-off-by: Dennis Yeh <dennis.yeh@cienet.com>

meiyeh123 force-pushed the dennis/add-v7x-16 branch from 6c69cc6 to 3d94c15 Compare July 10, 2026 09:22

meiyeh123 marked this pull request as draft July 10, 2026 09:22

meiyeh123 added 2 commits July 13, 2026 15:09

dynamic get disk name

2186b93

Signed-off-by: Dennis Yeh <dennis.yeh@cienet.com>

remove redundent module

e2a186a

Signed-off-by: Dennis Yeh <dennis.yeh@cienet.com>

meiyeh123 force-pushed the dennis/add-v7x-16 branch from 1f1cfdb to e2a186a Compare July 13, 2026 07:29

meiyeh123 requested a review from theminghuang July 13, 2026 07:30

meiyeh123 marked this pull request as ready for review July 13, 2026 07:31

theminghuang approved these changes Jul 13, 2026

View reviewed changes

CienetStingLin approved these changes Jul 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add tpu v7x-16 for buildkite agent#401

Add tpu v7x-16 for buildkite agent#401
meiyeh123 wants to merge 9 commits into
vllm-project:mainfrom
meiyeh123:dennis/add-v7x-16

meiyeh123 commented Jul 3, 2026 •

edited

Loading

Uh oh!

theminghuang Jul 9, 2026

Uh oh!

meiyeh123 Jul 13, 2026

Uh oh!

theminghuang Jul 9, 2026

Uh oh!

meiyeh123 Jul 10, 2026

Uh oh!

theminghuang Jul 10, 2026

Uh oh!

theminghuang left a comment

Uh oh!

theminghuang commented Jul 9, 2026

Uh oh!

meiyeh123 commented Jul 10, 2026

Uh oh!

theminghuang Jul 10, 2026

Uh oh!

theminghuang Jul 10, 2026

Uh oh!

meiyeh123 Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

meiyeh123 commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Tests

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theminghuang left a comment

Choose a reason for hiding this comment

Uh oh!

theminghuang commented Jul 9, 2026

Uh oh!

meiyeh123 commented Jul 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

meiyeh123 commented Jul 3, 2026 •

edited

Loading