Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
071622e
create architecture
mandylan75 Jun 30, 2026
77027fd
rename maim file
mandylan75 Jul 1, 2026
9519995
Apply suggestions from code review
mandylan75 Jul 2, 2026
ea1a3a2
lots of fixes
mandylan75 Jul 2, 2026
41a9278
reorganized functions
mandylan75 Jul 2, 2026
e5107b9
implement copilots suggestions
mandylan75 Jul 2, 2026
3f7b9f5
Apply suggestions from code review
mandylan75 Jul 6, 2026
05e2edb
add gh and hf subparsers
mandylan75 Jul 6, 2026
701a94b
Merge branch 'feature/issue-58/consolidate-code' of https://github.co…
mandylan75 Jul 6, 2026
dc09387
tests for package
mandylan75 Jul 7, 2026
83e56e1
incorporate feedback and add pyproject.toml from PR #63
mandylan75 Jul 7, 2026
c7f31d1
add pip install -e in pr-test workflow
mandylan75 Jul 7, 2026
a8e4cd9
Update .github/workflows/pr-tests.yml
mandylan75 Jul 7, 2026
8da9abf
add .[dev] in pr workflow
mandylan75 Jul 7, 2026
1a5a388
Apply suggestions from code review
mandylan75 Jul 8, 2026
59cad37
Apply suggestions from code review
mandylan75 Jul 10, 2026
2e07edf
add argparse in __main__.py
mandylan75 Jul 10, 2026
78ab39c
transfer doi badge check from legacy to package
mandylan75 Jul 21, 2026
c52a6a5
add unit tests for package & organize test suite
mandylan75 Jul 22, 2026
acb8b7e
Merge remote-tracking branch 'origin/main' into feature/issue-58/cons…
mandylan75 Jul 22, 2026
629bc4b
Merge remote-tracking branch 'origin/main' into feature/issue-58/cons…
mandylan75 Jul 24, 2026
888f151
transfer 73 logic over to package
mandylan75 Aug 12, 2026
41a4815
refactor/ transfer paper function prioritization implementation from …
mandylan75 Aug 12, 2026
e86fb56
change sheet col count to 2 for first sync_new_columns() test
mandylan75 Aug 12, 2026
7a066ac
Merge branch 'feature/issue-58/consolidate-code' of https://github.co…
mandylan75 Aug 12, 2026
3ca1b83
fix: remove redundant dict branch in ensure_string_value
Copilot Aug 12, 2026
d418e99
fix prevent formula injection and conditional format rule accumulation
mandylan75 Aug 13, 2026
fd355e0
Merge branch 'feature/issue-58/consolidate-code' of https://github.co…
mandylan75 Aug 13, 2026
2951846
Merge branch 'main' into feature/issue-58/consolidate-code
mandylan75 Aug 13, 2026
89d470f
fix onlyy manage rules matching exporter's TEXT_EQ no signature in co…
mandylan75 Aug 14, 2026
0422728
Merge branch 'feature/issue-58/consolidate-code' of https://github.co…
mandylan75 Aug 14, 2026
155365f
remove temporary local testing files
mandylan75 Aug 18, 2026
58a9616
add all fallback to gh_repo_type
mandylan75 Aug 20, 2026
4ae80ba
replaced a test function that defaults to all to fix error
mandylan75 Aug 20, 2026
5cf9b23
remove legacy exporter scripts
mandylan75 Aug 21, 2026
d802337
remove legacy tests
mandylan75 Aug 21, 2026
ee83a6c
update README.md for package
mandylan75 Aug 25, 2026
b00f3b5
update workflows to package instead of legacy script
mandylan75 Aug 27, 2026
c89d3d5
Apply suggestions from code review
mandylan75 Aug 27, 2026
a7fbd0e
update test function in test_huggingface.py
mandylan75 Aug 27, 2026
142fc77
update README.md presentation
mandylan75 Aug 28, 2026
4e74442
remove redundancy in running exporter locally at step 5
mandylan75 Aug 31, 2026
8a8b924
update README wording
mandylan75 Aug 31, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/gh-repo-exporter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,10 @@ jobs:
with:
python-version: "3.x"

- name: Install dependencies
- name: Install package
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install .

- name: Write Google credentials
run: printf "%s" '${{ secrets.GOOGLE_SERVICE_ACCOUNT_JSON }}' > service_account.json
Expand All @@ -47,4 +47,4 @@ jobs:
GOOGLE_CREDENTIALS_PATH: service_account.json
GH_ORG_NAME: ${{ vars.GH_ORG_NAME }}
SPREADSHEET_ID: ${{ vars.SPREADSHEET_ID }}
run: python gh_repo_exporter.py --repo-type "${{ github.event.inputs.repo_type || 'all' }}"
run: repo-exporter github --repo-type "${{ github.event.inputs.repo_type || 'all' }}"
6 changes: 3 additions & 3 deletions .github/workflows/hf-repo-exporter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,10 @@ jobs:
with:
python-version: "3.x"

- name: Install dependencies
- name: Install package
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install .

- name: Write Google credentials
run: printf "%s" '${{ secrets.GOOGLE_SERVICE_ACCOUNT_JSON }}' > service_account.json
Expand All @@ -38,4 +38,4 @@ jobs:
GOOGLE_CREDENTIALS_PATH: service_account.json
HF_ORG_NAME: ${{ vars.HF_ORG_NAME }}
SPREADSHEET_ID: ${{ vars.SPREADSHEET_ID }}
run: python hf_repo_exporter.py
run: repo-exporter huggingface
2 changes: 1 addition & 1 deletion .github/workflows/pr-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
- name: install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -e ".[dev]"
shell: bash

- name: run test
Expand Down
2 changes: 1 addition & 1 deletion .zenodo.json
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
"title": "Repository Exporter",
"version": "1.1.1",
"license": "MIT",
"publication_date": "2025-08-12",
"publication_date": "2026-08-12",
"grants": [
{
"id": "021nxhr62::2118240"
Expand Down
64 changes: 46 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Repository Exporter [![DOI](https://zenodo.org/badge/1080019710.svg)](https://doi.org/10.5281/zenodo.17835081)

Python scripts that gather metadata for all repositories in a provided GitHub or Hugging Face organization and automatically export the data into a desired Google Sheet (using a Google Cloud Console Service Account) for easy viewing and analysis.
Python package that gathers metadata for all repositories in a provided GitHub or Hugging Face organization and automatically exports the data into a designated Google Sheet (using a Google Cloud Console Service Account) for easy viewing and analysis.

## Contents
- [Features](#features)
Expand All @@ -10,6 +10,7 @@ Python scripts that gather metadata for all repositories in a provided GitHub or
- [Create a Hugging Face Token](#create-a-hugging-face-token)
- [Set up Google Cloud Service Account Access](#set-up-google-cloud-service-account-access)
- [Run repo exporter locally](#run-repo-exporter-locally)
- [Command-line options](#command-line-options)
- [Environment Variables](#environment-variables)
- [Testing](#testing)

Expand All @@ -23,7 +24,7 @@ Python scripts that gather metadata for all repositories in a provided GitHub or
- Creator and top 4 contributors (`N/A` creator means it was either a transferred repository or a forked repository and `None (<GitHub Username>)` means there was no full name attached to their GitHub account)
- Number of stars and number of branches for GitHub repositories, for Hugging Face repos, the analogous number of likes and open Discussions/PRs are collected
- Standard file/metadata checks:
- **For GitHub:** `README.md`, license, `.gitignore`, package requirements (`requirements.txt`, `environment.yaml`, etc.), `CITATION.cff`, `.zenodo.json`, and `CONTRIBUTING.md` files
- **For GitHub:** `README.md`, license, `.gitignore`, package requirements (`requirements.txt`, `environment.yaml`, etc.), `CITATION.cff`, `.zenodo.json`, `CONTRIBUTING.md`, and `AGENTS.md` files
- **For Hugging Face:** `README.md` ([dataset or model card](https://imageomics.github.io/Collaborative-distributed-science-guide/wiki-guide/About-Templates/)/Space README) and license (read from `yaml`)
- DOI for the repository (HF generated or from Zenodo for GitHub repos)
- Primary Programming Language (**GitHub only**)
Expand Down Expand Up @@ -104,38 +105,64 @@ Once configured, the workflow can be run by following the [Usage Instructions](#
conda activate repo-exporter
```

3. Create a `.env` file in the root of the project to configure required [environment variables](#environment-variables). See [`.env.example`](.env.example) for an example; the default sheet names are included and can be removed.
3. Create a `.env` file in the root of the project to configure required [environment variables](#environment-variables). See [`.env.example`](.env.example) for an example; the default sheet names are included and can be removed. Any value in `.env` can also be overridden with a CLI flag. See [Command-line options](#command-line-options) below.

4. Install Python dependencies:
```
pip install -r requirements.txt
```

5. Run the exporters

You can run **either exporter individually** or **both**, depending on your needs:
4. Install the package (this installs the `repo-exporter` CLI along with all dependencies, as defined in `pyproject.toml`):
```
pip install -e .
```

5. Each exporter run individually with the `repo-exporter` command, specifying the platform as a subcommand:

- **Run only the GitHub repository exporter**
- **Run the GitHub repository exporter**
```
python gh_repo_exporter.py
repo-exporter github
```

- **Run only the Hugging Face repository exporter**
- **Run the Hugging Face repository exporter**
```
python hf_repo_exporter.py
repo-exporter huggingface
```

- **Run both exporters (wait for one to finish before running the other)**
```
python hf_repo_exporter.py
python gh_repo_exporter.py
- Check the installed version at any time with:
```
repo-exporter --version
```

### Command-line options

All options fall back to the corresponding [environment variable](#environment-variables) when omitted, so you can mix and match `.env` values with one-off CLI overrides.

**`repo-exporter github [options]`**

| Flag | Overrides | Description |
|---|---|---|
| `--org` | `GH_ORG_NAME` | GitHub organization name |
| `--token` | `GH_TOKEN` | GitHub personal access token |
| `--repo-type` | `GH_REPO_TYPE` | Repo type filter: `all`, `public`, `private`, `forks`, `sources`, `member` (default: `all`) |
| `--spreadsheet-id` | `SPREADSHEET_ID` | Google Sheets spreadsheet ID |
| `--sheet-name` | `GH_SHEET_NAME` | Sheet tab name (default: `GH-Repos`) |
| `--credentials-path` | `GOOGLE_CREDENTIALS_PATH` | Path to `service_account.json` (default: `service_account.json`) |

**`repo-exporter huggingface [options]`**

| Flag | Overrides | Description |
|---|---|---|
| `--org` | `HF_ORG_NAME` | Hugging Face organization name (**case-sensitive**) |
| `--token` | `HF_TOKEN` | Hugging Face token |
| `--spreadsheet-id` | `SPREADSHEET_ID` | Google Sheets spreadsheet ID |
| `--sheet-name` | `HF_SHEET_NAME` | Sheet tab name (default: `HF-Repos`) |
| `--credentials-path` | `GOOGLE_CREDENTIALS_PATH` | Path to `service_account.json` (default: `service_account.json`) |

Run `repo-exporter github --help` or `repo-exporter huggingface --help` to see these options from the command line.

## Environment Variables

> [!NOTE]
> `SPREADSHEET_ID` and Google service account credentials are required for both GitHub and Hugging Face exports, as well as the platform-prefaced variables (e.g., `GH_ORG_NAME` and `HF_ORG_NAME`)
> * Both exporters support optional custom sheet name variables (`GH_SHEET_NAME` and `HF_SHEET_NAME`; these must be *distinct*).
> * Any of these values may instead be passed as a CLI flag. See [Command-line options](#command-line-options).

> [!WARNING]
> Tokens and Google service account credentials must be saved as repository *secrets*. All other values may be saved as environment variables assuming your spreadsheet is appropriately protected.
Expand All @@ -145,6 +172,7 @@ Once configured, the workflow can be run by following the [Usage Instructions](#
* Set `GH_ORG_NAME` to your GitHub organization name (for API calls).
* Set `SPREADSHEET_ID` to the Google Sheet ID used by the exporter.
* `GH_SHEET_NAME` is optional. If not provided, the exporter uses "GH-Repos".
* `GH_REPO_TYPE` is optional. If not provided, the exporter uses "all" (also accepts `public`, `private`, `forks`, `sources`, `member`).
* `GH_TOKEN` is required to access GitHub repositories.

### Hugging Face exporter
Expand Down
Loading