Data Management Scripts

A toolkit for researchers to audit, inventory, and clean their data directories. All scripts are self-contained Python 3 — no installation required beyond the standard library. An optional openpyxl install enables formatted Excel output.

---

Repository Layout

data-management-scripts/
├── README.md
├── requirements.txt
└── scripts/
    └── src/
        ├── scan_directory.py
        ├── summarize_filetypes.py
        ├── find_large_files.py
        ├── check_path_length.py
        └── run_all.py

All commands below are run from the repository root and reference scripts under scripts/src/. Reports are written relative to wherever you run the command from (not the scripts' location) — see Output Directory Layout.

---

Setup

Requires Python 3.9+. These scripts run fine with just the standard library (CSV output only); the steps below also set up openpyxl so --excel works.

# 1. Clone the repo
git clone git@github.com:nuitrcs/directory-scanning.git
cd directory-scanning

# 2. Create and activate a virtual environment
python3 -m venv .venv

# macOS / Linux:
source .venv/bin/activate

# Windows (PowerShell):
.venv\Scripts\Activate.ps1

# Windows (cmd.exe):
.venv\Scripts\activate.bat

# 3. Install dependencies
pip install -r requirements.txt

Once activated, python inside the venv will have openpyxl available. When you're done, run deactivate to leave the virtual environment. You'll need to re-run the source/Activate step (not the venv creation) each time you open a new terminal session.

If you'd rather skip the venv and just want Excel output available globally:

pip install -r requirements.txt --user

---

Scripts

Script	Purpose
scan_directory.py	Full per-folder inventory with recursive size roll-ups
summarize_filetypes.py	Storage breakdown by file type / extension
find_large_files.py	Surface the N largest files for cleanup candidates
check_path_length.py	Flag paths that would break SharePoint's 400-char limit after a OneDrive → SharePoint move
run_all.py	Run all four reports in one command

---

Quick Start

# Full suite — one command, all reports
python scripts/src/run_all.py /path/to/your/data

# With Excel output
python scripts/src/run_all.py /path/to/your/data --excel

# Include the SharePoint path-length check in the full suite
python scripts/src/run_all.py /path/to/your/data --site TeamSite --library "Shared Documents"

# Individual scripts
python scripts/src/scan_directory.py /path/to/your/data
python scripts/src/summarize_filetypes.py /path/to/your/data
python scripts/src/find_large_files.py /path/to/your/data --top 50 --min-size 100MB
python scripts/src/check_path_length.py /path/to/your/data --site TeamSite --library "Shared Documents"

Reports land in ./reports/ by default (configurable with --output).

---

Script Reference

scan_directory.py — Directory Inventory

The main report. Walks every folder and produces a CSV row per directory with:

• Folder path and depth relative to the scan root
• Direct file/folder counts and size (items immediately inside that folder)
• Recursive totals — all nested files and size rolled up
• Last modified timestamp of the most-recently-changed file inside
• Per-category file counts and sizes (Documents, Images, Code, Scientific Data, etc.)

python scripts/src/scan_directory.py <path> [options]

Options:
  --output DIR      Where to write reports (default: ./reports)
  --depth N         Limit scan depth (default: unlimited)
  --excel           Also write a formatted .xlsx file
  --top N           Print N largest folders to terminal (default: 10)
  --follow-symlinks Follow symbolic links (may cause loops)
  --no-summary      Skip the summary JSON file

Output columns (CSV):

Column	Description
Path	Folder path relative to scan root
Folder Name	Leaf folder name
Depth	How many levels below the root
Direct Subfolders	Folders immediately inside this folder
Direct Files	Files immediately inside this folder
Direct Size	Space used by files directly in this folder
Total Subfolders (recursive)	All nested subfolders
Total Files (recursive)	All nested files
Total Size	All nested storage combined
Last Modified (any file)	Most recent file modification inside
Files: Documents	Count of document-type files (recursive)
Size: Documents	Storage used by documents (recursive)
(one pair per category)	…

---

summarize_filetypes.py — File Type Summary

Aggregates storage across the whole tree by file type. Good for a quick "what is filling up this drive?" answer.

python scripts/src/summarize_filetypes.py <path> [options]

Options:
  --output DIR       Where to write reports
  --excel            Also write a formatted .xlsx file
  --by-extension     Group by raw extension (.pdf, .csv) instead of category
  --follow-symlinks

File type categories:

Category	Extensions
Documents	pdf, doc/x, txt, md, tex, rtf …
Spreadsheets/Tables	xls/x, csv, tsv, ods
Images	jpg, png, tif, heic, raw, nef …
Video	mp4, mov, avi, mkv …
Audio	mp3, wav, flac …
Code/Scripts	py, r, rmd, ipynb, sas, sh, sql …
Archives	zip, tar, gz, 7z, rar …
Scientific Data	h5, nc, mat, nii, fastq, bam, vcf, fcs …
Other / No Extension	Everything else

---

find_large_files.py — Largest Files

Lists the N biggest individual files — the fastest route to reclaiming space.

python scripts/src/find_large_files.py <path> [options]

Options:
  --output DIR       Where to write reports
  --top N            How many files to return (default: 25)
  --min-size SIZE    Only include files ≥ SIZE, e.g. 10MB, 1GB (default: 0)
  --excel            Also write a formatted .xlsx with color-coded size ranks
  --follow-symlinks

---

check_path_length.py — SharePoint Path-Length Check

OneDrive folders migrating into a SharePoint document library get their local prefix rewritten: .../My Files/... becomes sites/<site_name>/<document library>/.... Since the new prefix is usually longer, paths that were fine on OneDrive can break SharePoint's ~400 character path limit after the move. This script simulates that rename for every file (and folder) under a path and reports anything that would exceed the limit — before you move anything.

python scripts/src/check_path_length.py <path> --site <site_name> --library <library_name> [options]

Options:
  --site NAME        Destination SharePoint site name (required)
  --library NAME     Destination document library name (required)
  --limit N          Character limit to check against (default: 400)
  --marker TEXT      Local path segment replaced by 'sites/<site>/<library>'
                     (default: "My Files")
  --warn-margin N    Flag paths within N chars of the limit as NEAR LIMIT (default: 20)
  --over-only        Only include OVER LIMIT / NEAR LIMIT rows in the report
  --files-only       Skip checking folder paths, files only
  --output DIR       Where to write reports (default: ./reports)
  --excel            Also write a formatted .xlsx file
  --follow-symlinks

Notes:

• The 400-char count is the site-relative path only (sites/<site>/<library>/...), not the full https://tenant.sharepoint.com/... URL.
• If --marker ("My Files" by default) isn't found in a scanned item's path, the script falls back to using that item's path relative to the scanned root — so it still works if you point it at a subfolder that no longer contains "My Files".

---

run_all.py — Run Everything

Calls all four scripts against a single path and drops all reports into one timestamped folder (reports/run_YYYYMMDD_HHMMSS/).

python scripts/src/run_all.py <path> [options]

Options:
  --output DIR       Parent output directory (default: ./reports)
  --excel            Enable Excel output for all reports
  --depth N          Max depth for directory inventory
  --top-files N      Largest-files list length (default: 25)
  --min-size SIZE    Min file size for large-files report
  --by-extension     Raw extension grouping in file-type summary
  --follow-symlinks
  --site NAME        Destination SharePoint site name — enables the path-length
                     check report (requires --library too)
  --library NAME     Destination document library name (requires --site too)
  --path-limit N     Character limit for the path-length check (default: 400)
  --marker TEXT      Local path segment replaced in the path-length check
                     (default: "My Files")

---

Optional: Excel Output

--excel needs openpyxl, which is already in requirements.txt. If you followed Setup and activated the venv, it's already installed. Otherwise:

pip install openpyxl

Excel reports include:

• Frozen header row with auto-filter
• Alternating row colors
• Auto-sized columns
• Color-coded size tiers (find_large_files)

---

Requirements

• Python 3.9 or newer (no external packages needed for CSV output)
• openpyxl for Excel output — installed via pip install -r requirements.txt (see Setup)

---

Output Directory Layout

reports/
└── run_20240315_143022/          ← run_all.py groups into a dated folder
    ├── directory_inventory_*.csv
    ├── directory_inventory_*_summary.json
    ├── filetype_summary_*.csv
    ├── large_files_*.csv
    └── path_length_check_*.csv   ← only when --site and --library are passed

Individual scripts write directly into reports/ (or --output).