GitLab Bulk Cloner

A bash script to clone all repositories from a GitLab group and its subgroups recursively. Perfect for backing up your organization's repositories or setting up a local development environment.

Features

• 🔍 Recursive Group Cloning: Automatically discovers and clones repositories from all subgroups
• 📁 Organized Structure: Creates directories matching your GitLab group hierarchy
• ⚡ Smart Skipping: Skips repositories that already exist locally
• 🔐 Secure: Supports both command-line arguments and environment variables for tokens
• 🌐 GitLab Compatible: Works with GitLab.com and self-hosted GitLab instances

Prerequisites

• bash (version 4.0 or higher)
• curl - for making API requests
• jq - for parsing JSON responses
• git - for cloning repositories
• A GitLab Personal Access Token with read_api and read_repository scopes

Installing Dependencies

macOS:

brew install curl jq

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install curl jq

Linux (Fedora/RHEL):

sudo dnf install curl jq

Installation

1. Clone this repository:

git clone https://github.com/yourusername/gitlab-bulk-cloner.git
cd gitlab-bulk-cloner

2. Make the script executable:

chmod +x clone.sh

Usage

Method 1: Command-Line Arguments

./clone.sh <GITLAB_URL> <ROOT_GROUP> <TOKEN>

Example:

./clone.sh https://gitlab.com myorganization glpat-xxxxxxxxxxxxx

Method 2: Environment Variables

export GITLAB_URL="https://gitlab.com"
export ROOT_GROUP="myorganization"
export GITLAB_TOKEN="glpat-xxxxxxxxxxxxx"
./clone.sh

Method 3: Inline Environment Variables

GITLAB_URL=https://gitlab.com ROOT_GROUP=myorganization GITLAB_TOKEN=glpat-xxxxx ./clone.sh

Examples

GitLab.com

./clone.sh https://gitlab.com mygroup glpat-xxxxxxxxxxxxx

Self-Hosted GitLab Instance

./clone.sh https://gitlab.example.com mygroup glpat-xxxxxxxxxxxxx

Using Environment Variables

export GITLAB_URL="https://gitlab.com"
export ROOT_GROUP="mygroup"
export GITLAB_TOKEN="glpat-xxxxxxxxxxxxx"
./clone.sh

How It Works

1. Fetches Group ID: Queries the GitLab API to find the root group ID
2. Discovers Subgroups: Recursively finds all subgroups under the root group
3. Creates Directory Structure: Creates local directories matching the GitLab group hierarchy
4. Clones Repositories: Clones all repositories from each group and subgroup
5. Skips Existing: Automatically skips repositories that already exist locally

Directory Structure

The script creates a directory structure that mirrors your GitLab groups:

.
├── mygroup/
│   ├── project1/
│   ├── project2/
│   └── subgroup1/
│       ├── project3/
│       └── project4/

Creating a GitLab Personal Access Token

1. Go to your GitLab profile settings
2. Navigate to Access Tokens (or Personal Access Tokens)
3. Create a new token with the following scopes:
   • read_api - Required to read group and project information
   • read_repository - Required to clone repositories
4. Copy the token (it starts with glpat-)

Note: Keep your token secure and never commit it to version control!

Troubleshooting

Error: "Group not found"

• Verify the ROOT_GROUP name is correct (case-sensitive)
• Ensure your token has access to the group
• Check that the GitLab URL is correct

Error: "curl is required but not installed"

• Install curl using your package manager (see Prerequisites)

Error: "jq is required but not installed"

• Install jq using your package manager (see Prerequisites)

Error: "Permission denied"

• Make sure the script is executable: chmod +x clone.sh
• Verify your token has the correct permissions

Repositories Not Cloning

• Check your token has read_repository scope
• Verify you have access to the repositories
• Check your network connection

Security Best Practices

• ✅ Use environment variables instead of command-line arguments when possible
• ✅ Never commit your GitLab token to version control
• ✅ Use tokens with minimal required scopes
• ✅ Rotate your tokens regularly
• ✅ Consider using .env files (not included in this script, but you can add it)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

This project is open source and available under the MIT License.

Acknowledgments

• Built for the GitLab community
• Uses the GitLab API v4

Support

If you encounter any issues or have questions, please open an issue on GitHub.

---

Made with ❤️ for the GitLab community