Pre download all repositories

[JesseZZZZZ]

Hi, authors. Thank you for Codescout, which is a wonderful work. After reading your code, it seems like you clone the repository in SWE-smith during training, and a repo is deleted once the current step finishes? I wonder if this process significantly slows the training process down. Do you think pre-downloading all repos is possible? If the current logic is reserved, what is the time consumed for this online cloning process?

I wonder, perhaps you can use some tricks to reduce this cost? For instance, can you clone the repos used for step 11 in the updating process of step 10?

Looking forward to your response~

[adityasoni9998]

Hi @JesseZZZZZ - yes we clone repositories on the fly and delete them after the rollout exits. I am not fully sure about the exact time it takes to clone the repo but it is definitely not the biggest bottleneck in my understanding. There are a few ways you can optimize this:

• Shallow clones and/or just clone specific branch only for a task. For SWE-smith the instance_id in HF dataset has a corresponding branch name in the repository on GitHub (see example)
• Use prebuilt docker images/apptainer images with env pre-configured. This is relatively more lightweight for SWE-smith than other datasets like SWE-Gym since SWE-smith has 1 docker image per repository whereas others have 1 docker image per instance. The repo will be pre-cloned in /testbed and you can just checkout the correct commit.
• Just pre-clone the repo for each instance_id once and cache it before running the training - since we don't install dependencies this is not a terrible idea, and then share the repo across the entire group of 8 rollouts for each instance id. The reason we don't share repos across rollouts is that the agent may mistakenly edit the codebase which can interfere with other rollouts, but we never observed this error behaviour.

Do you find clones to significantly slow down the trainer? During our experiments, it took 48 hours to train a 14B model and 12 hours to train a 4B model. And honestly, this was fast enough to not exhaust all our compute resources so we did not optimize this :) The clusters we used had reasonable network bandwidth.

I have been recently using CodeScout as a smoke test to try out SkyRL-Tinker and AReaL backends with Apptainer runtime via platoon - the codebase is still under development but if you think that is helpful for your work, I can share it with you.