Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 67 additions & 7 deletions NV-Segment-CT/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,28 +16,88 @@ cd NV-Segment-CTMR/NV-Segment-CT;
pip install -r requirements.txt;
```

Model weights are prepared automatically during inference. The first run downloads the checkpoint from Hugging Face into the local Hugging Face cache and links it at `models/model.pt`;
Model weights are prepared automatically during inference. The first run downloads the checkpoint from Hugging Face into the local Hugging Face cache and links it at `models/model.pt`; later runs reuse the cached weights.

## 1.1 **NV-Segment-CT** [[Github]](https://github.com/NVIDIA-Medtech/NV-Segment-CTMR/tree/main/NV-Segment-CT) [[Huggingface]](https://huggingface.co/nvidia/NV-Segment-CT)

### Automatic Segmentation (support multi-gpu batch processing)

[class definition](https://github.com/NVIDIA-Medtech/NV-Segment-CTMR/blob/main/NV-Segment-CTMR/configs/label_dict.json)
[class definition](https://github.com/NVIDIA-Medtech/NV-Segment-CTMR/blob/main/NV-Segment-CT/configs/label_dict.json)

#### Single image inference to segment everything (automatic)

The output will be saved to `{output_dir}/spleen_03/spleen_03_{output_postfix}{output_ext}`.

```bash
# CT sementation
# Make sure conda environment is activated
conda activate vista3d-nv
cd NV-Segment-CT

# Automatic Segment everything
python -m monai.bundle run --config_file configs/inference.json --input_dict "{'image':'example/spleen_03.nii.gz'}"
```

#### Single image inference to segment specific class (automatic)

The detailed automatic segmentation class index can be found [here](../configs/label_dict.json).

```bash
# Automatic Segment specific class
python -m monai.bundle run --config_file configs/inference.json --input_dict "{'image':'example/spleen_03.nii.gz','label_prompt':[3]}"
# Automatic Batch segmentation for the whole folder
```

#### Batch inference with multiGPU support (automatic)

The `configs/batch_inference.json` defines the batch inference, you can:

1. Segment all NIfTI files within a folder and subfolders

- `configs/batch_inference.json` builds `input_list` with `scripts/batch_inference_utils.build_input_list()`:
- Recursively discovers `**/*.nii.gz` under `--input_dir`.
- **Resume (default):** with `batch_resume_skip_existing: true` in `batch_inference.json`, only volumes whose expected output is **missing or empty** under `--output_dir` are queued (same layout as `SaveImaged`). Re-run the **same** command to finish leftovers. Set `batch_resume_skip_existing` to false to segment every discovered file again.
- **Discovery filters:** edit `batch_skip_dir_names` (exact parent folder name) and/or `batch_skip_dir_prefixes` (parent folder name starts with...) in `configs/batch_inference.json` (JSON arrays of strings).
- **Optional keys** in `configs/batch_inference.json` (defaults in the file): `batch_skip_dir_names`, `batch_skip_dir_prefixes`, `batch_resume_skip_existing`, `batch_use_input_list_cache`, `batch_cache_wait_sec`.
- **`batch_use_input_list_cache`:** With `torchrun` (multi-process), only rank 0 walks the tree to build `input_list` and writes a small JSON cache under the system temp directory; other ranks read that file so you do not repeat a huge filesystem scan on every GPU. Set to `false` if you want every rank to compute the list itself (simpler, slower on large cohorts). Single-process runs are unaffected in practice.
- **`batch_cache_wait_sec`:** When `batch_use_input_list_cache` is `true`, non-zero ranks wait up to this many seconds for rank 0's cache file. Increase if rank 0's scan is slow; decrease only if the list is always built quickly.
- **Which classes to segment:** edit **`everything_labels`** in **`configs/inference.json`**. See `configs/label_dict.json` and `docs/inference.md`.
- If **resume** leaves nothing to run (all outputs already present), the run **exits successfully** with `[nvseg] batch: nothing to run (resume); ok` (avoids a zero-length dataloader / `DistributedSampler` failure). If **no** `*.nii.gz` files are discovered under `input_dir`, you get a short `[nvseg] batch: no *.nii.gz...` error.
- Rank 0 logs: `[nvseg] batch resume (skip existing outputs): N volume(s) (...)`.
- **Multi-GPU:** `--nproc_per_node` must be less than or equal to the number of volumes in `input_list` after filtering.
- **Outputs:** With `data_root_dir` and `separate_folder: true`, `input_dir/patient1/scan.nii.gz` -> `output_dir/patient1/scan/scan_trans.nii.gz`. If `models/model.pt` is missing, inference prepares it automatically from Hugging Face.
- Advanced: edit `should_skip_path_by_parent_rules()` in `scripts/batch_inference_utils.py` for custom path rules.

2. Segment based on a filelist.txt file, you can change the `input_list` in `configs/batch_inference.json`

```json
"input_list": "$sorted([os.path.abspath(line.strip()) for line in open('/absolute/path/to/filelist.txt') if line.strip() and not line.strip().startswith('#')])",
```

##### Single-GPU Batch Inference

```bash
# Make sure conda environment is activated
conda activate vista3d-nv
cd NV-Segment-CT

python -m monai.bundle run --config_file="['configs/inference.json', 'configs/batch_inference.json']" --input_dir="example/" --output_dir="example/"
# Automatic Batch segmentation for the whole folder with multi-gpu support. mgpu_inference.json is below. change nproc_per_node to your GPU number.
torchrun --nproc_per_node=2 --nnodes=1 -m monai.bundle run --config_file="['configs/inference.json', 'configs/batch_inference.json', 'configs/mgpu_inference.json']" --input_dir="example/" --output_dir="example/"
```

Note: For more details about batch processing, please refer to NV-Segment-CTMR readme.md
##### Multi-GPU batch inference (cohorts, resume, optional folder filters)

```bash
conda activate vista3d-nv
cd NV-Segment-CT

# Example: multi-GPU batch (same command for first run or resume)
torchrun --nproc_per_node=2 --nnodes=1 -m monai.bundle run \
--config_file="['configs/inference.json', 'configs/batch_inference.json', 'configs/mgpu_inference.json']" \
--input_dir="example/" \
--output_dir="example/"
```

```text
Note: if using a finetuned checkpoint whose label_mapping maps custom labels to global indexes "2, 20, 21", remove the `subclass` dict from inference.json since those values defined in `subclass` will trigger the wrong subclass segmentation.
```

### Interactive segmentation

Expand Down
2 changes: 1 addition & 1 deletion NV-Segment-CTMR/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ cd NV-Segment-CTMR/NV-Segment-CTMR
pip install -r requirements.txt
```

Model weights are prepared automatically during inference. The first run downloads the checkpoint from Hugging Face into the local Hugging Face cache and links it at `models/model.pt`;
Model weights are prepared automatically during inference. The first run downloads the checkpoint from Hugging Face into the local Hugging Face cache and links it at `models/model.pt`; later runs reuse the cached weights.

## Automatic Segmentation (support multi-gpu batch processing)

Expand Down
Loading