diff --git a/README.md b/README.md index b658e96..87dd845 100755 --- a/README.md +++ b/README.md @@ -7,10 +7,10 @@ This repository is based on [Marigold](https://marigoldmonodepth.github.io), CVP [![Hugging Face Model](https://img.shields.io/badge/🤗%20Hugging%20Face-Model-green)](https://huggingface.co/developy/ApDepth) [![Hugging Face Demo](https://img.shields.io/badge/🤗%20Hugging%20Face-Demo-purple)](https://huggingface.co/spaces/developy/ApDepth) -[**Haruko386**](https://haruko386.github.io/), + ![cover](doc/cover.png) @@ -18,7 +18,7 @@ This repository is based on [Marigold](https://marigoldmonodepth.github.io), CVP ## 📢 News - **2026-04-06:** `ApDepth V2-0` is released! -- **2026-04-03:** We officially release the complete code for **ApDepth**! The repository now includes the full coarse-to-fine two-stage training pipeline and evaluation scripts. +- **2026-04-03:** We officially release the code for **ApDepth**! Stage 1 feature-alignment training is maintained in [ApDepth_Stage1](https://github.com/Haruko386/ApDepth_Stage1); this repository provides Stage 2 training, inference and evaluation. - **2026-01-15:** We successfully introduce a spatial-preserving **Conv Adapter** and a **Cosine Similarity Loss** to enhance feature alignment, alongside a **Pixel-level $L_1$ Loss** to establish an accurate global metric scale. - **2025-10-25:** Inspired by DepthMaster, we propose a two-stage loss function training strategy based on `ApDepth V1-0`. In the first stage, we perform foundational training using MSE loss. In the second stage, we learn edge structures through FFT loss. Based on this, we introduce `ApDepth V1-1`. - **2025-10-09:** We propose a novel diffusion-based depth estimation framework guided by pre-trained models. @@ -182,6 +182,10 @@ You can get the result under `output/eval` ## 🏋️ Training +The workflow is **Stage 1 feature alignment → Stage 2 main training → optional +[VKITTI far-depth post-training](#post-training)**. Complete Stage 2, including +its FFT refinement, before starting the optional post-training run. + Based on the previously created environment, install extended requirements: ```bash @@ -197,56 +201,114 @@ export BASE_CKPT_DIR=YOUR_CHECKPOINT_DIR # directory of pretrained checkpoint Download Stable Diffusion v2 [checkpoint](https://huggingface.co/sd2-community/stable-diffusion-2-1) into `${BASE_CKPT_DIR}` -Download the checkpoint of [Depth-Anything-V2](https://github.com/DepthAnything/Depth-Anything-V2) into `DA2/checkpoints/`. Stage 1 uses the ViT-G checkpoint by default at `DA2/checkpoints/depth_anything_v2_vitg.pth` as the external semantic encoder. +Download the ViT-G checkpoint of [Depth-Anything-V2](https://github.com/DepthAnything/Depth-Anything-V2) to `DA2/checkpoints/depth_anything_v2_vitg.pth`. The ApDepth pipeline uses it to generate the depth prior during training and inference. -Prepare for [Hypersim](https://github.com/apple/ml-hypersim) and [Virtual KITTI 2](https://europe.naverlabs.com/research/computer-vision/proxy-virtual-worlds-vkitti-2/) datasets and save into `${BASE_DATA_DIR}`. Please refer to [this README](script/dataset_preprocess/hypersim/README.md) for Hypersim preprocessing. Stage 1 uses `config/dataset/dataset_apdepth_train_s1.yaml`; the main training stage uses `config/dataset/dataset_train.yaml`. +Prepare [Hypersim](https://github.com/apple/ml-hypersim) and [Virtual KITTI 2](https://europe.naverlabs.com/research/computer-vision/proxy-virtual-worlds-vkitti-2/) under `${BASE_DATA_DIR}`. Please refer to [this README](script/dataset_preprocess/hypersim/README.md) for Hypersim preprocessing. Configure the training paths in `config/dataset/dataset_train.yaml` and prepare the validation datasets listed in `config/dataset/dataset_val.yaml` and `config/dataset/dataset_vis.yaml`. ------------ **Stage 1: feature alignment pre-training** -Stage 1 has been integrated into this repository. It trains the denoising U-Net and DINOv2 adapter in feature space before the main ApDepth training stage. +Stage 1 is maintained separately in +[Haruko386/ApDepth_Stage1](https://github.com/Haruko386/ApDepth_Stage1). +Follow that repository's setup and training instructions. This repository contains +only the main Stage 2 trainer; it does not include Stage 1 code or dependencies. + +------------ + +**Stage 2: main ApDepth training** + +To initialize Stage 2 from an existing Stage 1 checkpoint, start a new run with +`--init_checkpoint`. This loads only the U-Net; optimizer and iteration counters +start fresh using the Stage 2 config. `config/train_apdepth.yaml` keeps far-depth +supervision disabled throughout the main training stage: ```bash -bash script/apdepth_train_s1.sh +python train.py --config config/train_apdepth.yaml \ + --init_checkpoint /path/to/stage1/checkpoint/iter_020000 --no_wandb ``` -Equivalent direct command: +The initialization directory must contain +`unet/diffusion_pytorch_model.safetensors`. `--init_checkpoint` rejects `.bin` +checkpoints. + +To train the main stage directly from the base model without Stage 1 initialization: ```bash -python apdepth_train_s1.py \ - --config config/apdepth_train_s1.yaml \ - --base_data_dir ${BASE_DATA_DIR} \ - --base_ckpt_dir ${BASE_CKPT_DIR} \ - --output_dir output/stage1 \ - --no_wandb +python train.py --config config/train_apdepth.yaml --no_wandb ``` -Resume Stage 1 from a checkpoint, e.g. +Resume from a checkpoint, e.g. ```bash -python apdepth_train_s1.py --resume_run output/stage1/checkpoint/latest --no_wandb +python train.py --resume_run output/train_apdepth/checkpoint/latest --no_wandb ``` ------------ -**Stage 2: main ApDepth training** +## 🌤️ Optional post-training after Stage 2 + +**Start this step only after Stage 2 training has finished.** The new VKITTI +far-depth supervision feature is enabled by `config/train_sky_finetune.yaml` for +a separate post-training run initialized from the completed Stage 2 U-Net. + +Measured depths in [80, 655.35] m supply an additional clipped far-depth target +before VAE encoding; missing depth remains excluded. Evaluation masks and +normalization quantiles are unchanged. The separately averaged far latent MSE +and pixel L1 terms have weight 0.5. For a post-training ablation, set +`far_depth_supervision.enabled: false` in the post-training config. + +Use the same environment, datasets and base checkpoints prepared for Stage 2. +First audit the native VKITTI depth and target coverage: ```bash -python train.py --config config/train_apdepth.yaml --no_wandb +python -m script.audit_far_supervision --config config/train_sky_finetune.yaml \ + --base_data_dir "${BASE_DATA_DIR}" --samples 20 ``` -Resume from a checkpoint, e.g. +Check the RGB, far masks, target images and `coverage.json` in `output/far_audit`. +If preprocessing replaced the far-plane values with zero, restore the original +depth files. Zero depth is not treated as sky. Adjust each dataset config's `dir` +to match the actual layout under `BASE_DATA_DIR`. + +Start a new post-training run from the final Stage 2 checkpoint (adjust the path +if Stage 2 was saved elsewhere): ```bash -python train.py --resume_run output/train_apdepth/checkpoint/latest --no_wandb +python train.py --config config/train_sky_finetune.yaml \ + --init_checkpoint output/train_apdepth/checkpoint/iter_021000 --no_wandb ``` ------------- +This resets the optimizer, learning-rate schedule and iteration counter, then +runs 6000 new optimization steps at LR 5e-6 with 100 warmup steps and saves +backups every 1000 steps in `output/train_sky_finetune/checkpoint`. It retains the +90% Hypersim / 10% VKITTI sampling, original batch settings and periodic validation. +These 6000 steps use latent MSE + pixel L1 reconstruction plus the new far-depth +loss; the Stage 2 FFT refinement has already finished. Initialization requires +`unet/diffusion_pytorch_model.safetensors` in the supplied checkpoint directory. + +If this post-training run is interrupted, resume its own checkpoint: + +```bash +python train.py --resume_run output/train_sky_finetune/checkpoint/latest --no_wandb +``` + +`--resume_run` restores the saved config and training state. Starting post-training +requires `--init_checkpoint` with the post-training config above; resuming a +Stage 2 checkpoint would continue Stage 2 with its saved settings. +`--resume_run` and `--init_checkpoint` cannot be combined. + +Monitor `train/far_loss`, `train/far_pixel_ratio` and `train/far_latent_ratio`. +Compare KITTI and NYU against the completed Stage 2 model after post-training; this change +alone does not establish a metric improvement. -**Evaluating results** +## Evaluating trained checkpoints -Only the U-Net is updated and saved during training. To use the inference pipeline with your training result, replace `unet` folder in `train_apdepth` checkpoints with that in the `checkpoint` output folder. Then refer to [this section](#evaluation) for evaluation. +Stage 2 and optional post-training update and save the U-Net. For inference, copy a complete +ApDepth pipeline checkpoint to a new directory, then replace that copy's `unet/` +folder with the selected training checkpoint's `unet/`. The pipeline's VAE, +tokenizer, text encoder and scheduler are still required; a training checkpoint +alone is not a complete inference pipeline. Then refer to [evaluation](#evaluation). > [!IMPORTANT] > diff --git a/apdepth_train_s1.py b/apdepth_train_s1.py deleted file mode 100644 index 6b3a3ce..0000000 --- a/apdepth_train_s1.py +++ /dev/null @@ -1,330 +0,0 @@ -# Last modified: 2025-07-13 -# -# Copyright 2025 Jiawei Wang, SJZU. All rights reserved. -# -# This file has been modified from the original version. -# Original copyright (c) 2023 Bingxin Ke, ETH Zurich. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -------------------------------------------------------------------------- -# If you find this code useful, we kindly ask you to cite our paper in your work. -# Please find bibtex at: https://github.com/Haruko386/ApDepth#-citation -# More information about the method can be found at https://haruko386.github.io/apdepth -# -------------------------------------------------------------------------- - -import argparse -import logging -import os -import shutil -from datetime import datetime, timedelta -from typing import List - -import torch -from omegaconf import OmegaConf -from torch.utils.data import ConcatDataset, DataLoader -from tqdm import tqdm - - -from apdepth import ApDepthPipeline -from src.dataset import BaseDepthDataset, DatasetMode, get_dataset -from src.dataset.mixed_sampler import MixedBatchSampler -from src.trainer import get_trainer_cls -from src.util.config_util import ( - find_value_in_omegaconf, - recursive_load_config, -) -from src.util.depth_transform import ( - DepthNormalizerBase, - get_depth_normalizer, -) -from src.util.logging_util import ( - config_logging, - init_wandb, - load_wandb_job_id, - log_slurm_job_id, - save_wandb_job_id, - tb_logger, -) -from src.util.slurm_util import get_local_scratch_dir, is_on_slurm - -if "__main__" == __name__: - t_start = datetime.now() - print(f"start at {t_start}") - - # -------------------- Arguments -------------------- - parser = argparse.ArgumentParser(description="Train your cute model!") - parser.add_argument( - "--config", - type=str, - default="config/apdepth_train_s1.yaml", - help="Path to config file.", - ) - parser.add_argument( - "--resume_run", - action="store", - default=None, - help="Path of checkpoint to be resumed. If given, will ignore --config, and checkpoint in the config", - ) - parser.add_argument( - "--output_dir", type=str, default=None, help="directory to save checkpoints" - ) - parser.add_argument("--no_cuda", action="store_true", help="Do not use cuda.") - parser.add_argument( - "--exit_after", - type=int, - default=-1, - help="Save checkpoint and exit after X minutes.", - ) - parser.add_argument("--no_wandb", action="store_true", help="run without wandb") - parser.add_argument( - "--do_not_copy_data", - action="store_true", - help="On Slurm cluster, do not copy data to local scratch", - ) - parser.add_argument( - "--base_data_dir", type=str, default="/root/Dataset", help="directory of training data" - ) - parser.add_argument( - "--base_ckpt_dir", - type=str, - default="/root/ApDepth/pretrained_checkpoint", - help="directory of pretrained checkpoint", - ) - parser.add_argument( - "--add_datetime_prefix", - action="store_true", - help="Add datetime to the output folder name", - ) - - args = parser.parse_args() - print("\n=== Arguments Summary ===") - max_len = max(len(arg) for arg in vars(args)) - for arg in vars(args): - print(f"{arg.ljust(max_len)} : {getattr(args, arg)}") - resume_run = args.resume_run - output_dir = args.output_dir - base_data_dir = ( - args.base_data_dir - if args.base_data_dir is not None - else os.environ["BASE_DATA_DIR"] - ) - base_ckpt_dir = ( - args.base_ckpt_dir - if args.base_ckpt_dir is not None - else os.environ["BASE_CKPT_DIR"] - ) - - # -------------------- Initialization -------------------- - # Resume previous run - if resume_run is not None: - print(f"Resume run: {resume_run}") - out_dir_run = os.path.dirname(os.path.dirname(resume_run)) - job_name = os.path.basename(out_dir_run) - # Resume config file - cfg = OmegaConf.load(os.path.join(out_dir_run, "config.yaml")) - else: - # Run from start - cfg = recursive_load_config(args.config) - - # Output dir - if output_dir is not None: - out_dir_run = output_dir - job_name = os.path.basename(args.config) - else: - job_name = os.path.basename(args.config).split(".")[0] - out_dir_run = os.path.join("./output", job_name) - os.makedirs(out_dir_run, exist_ok=False) - - cfg_data = cfg.dataset - - # Other directories - out_dir_ckpt = os.path.join(out_dir_run, "checkpoint") - if not os.path.exists(out_dir_ckpt): - os.makedirs(out_dir_ckpt) - out_dir_tb = os.path.join(out_dir_run, "tensorboard") - if not os.path.exists(out_dir_tb): - os.makedirs(out_dir_tb) - out_dir_eval = os.path.join(out_dir_run, "evaluation") - if not os.path.exists(out_dir_eval): - os.makedirs(out_dir_eval) - out_dir_vis = os.path.join(out_dir_run, "visualization") - if not os.path.exists(out_dir_vis): - os.makedirs(out_dir_vis) - - # -------------------- Logging settings -------------------- - config_logging(cfg.logging, out_dir=out_dir_run) - logging.debug(f"config: {cfg}") - - # Initialize wandb - # if not args.no_wandb: - # if resume_run is not None: - # wandb_id = load_wandb_job_id(out_dir_run) - # wandb_cfg_dic = { - # "id": wandb_id, - # "resume": "must", - # **cfg.wandb, - # } - # else: - # wandb_cfg_dic = { - # "config": dict(cfg), - # "name": job_name, - # "mode": "online", - # **cfg.wandb, - # } - # wandb_cfg_dic.update({"dir": out_dir_run}) - # wandb_run = init_wandb(enable=True, **wandb_cfg_dic) - # save_wandb_job_id(wandb_run, out_dir_run) - # else: - # init_wandb(enable=False) - - # Tensorboard (should be initialized after wandb) - tb_logger.set_dir(out_dir_tb) - - log_slurm_job_id(step=0) - - # -------------------- Device -------------------- - cuda_avail = torch.cuda.is_available() and not args.no_cuda - device = torch.device("cuda" if cuda_avail else "cpu") - logging.info(f"device = {device}") - - # -------------------- Snapshot of code and config -------------------- - if resume_run is None: - _output_path = os.path.join(out_dir_run, "config.yaml") - with open(_output_path, "w+") as f: - OmegaConf.save(config=cfg, f=f) - logging.info(f"Config saved to {_output_path}") - # Copy and tar code on the first run - _temp_code_dir = os.path.join(out_dir_run, "code_tar") - _code_snapshot_path = os.path.join(out_dir_run, "code_snapshot.tar") - os.system( - f"rsync --relative -arhvz --quiet --filter=':- .gitignore' --exclude '.git' . '{_temp_code_dir}'" - ) - os.system(f"tar -cf {_code_snapshot_path} {_temp_code_dir}") - os.system(f"rm -rf {_temp_code_dir}") - logging.info(f"Code snapshot saved to: {_code_snapshot_path}") - - # -------------------- Copy data to local scratch (Slurm) -------------------- - if is_on_slurm() and (not args.do_not_copy_data): - # local scratch dir - original_data_dir = base_data_dir - base_data_dir = os.path.join(get_local_scratch_dir(), "ApDepth_data") - # copy data - required_data_list = find_value_in_omegaconf("dir", cfg_data) - required_data_list = list(set(required_data_list)) - logging.info(f"Required_data_list: {required_data_list}") - for d in tqdm(required_data_list, desc="Copy data to local scratch"): - ori_dir = os.path.join(original_data_dir, d) - dst_dir = os.path.join(base_data_dir, d) - os.makedirs(os.path.dirname(dst_dir), exist_ok=True) - if os.path.isfile(ori_dir): - shutil.copyfile(ori_dir, dst_dir) - elif os.path.isdir(ori_dir): - shutil.copytree(ori_dir, dst_dir) - logging.info(f"Data copied to: {base_data_dir}") - - # -------------------- Gradient accumulation steps -------------------- - eff_bs = cfg.dataloader.effective_batch_size - accumulation_steps = eff_bs / cfg.dataloader.max_train_batch_size - assert int(accumulation_steps) == accumulation_steps - accumulation_steps = int(accumulation_steps) - - logging.info( - f"Effective batch size: {eff_bs}, accumulation steps: {accumulation_steps}" - ) - - # -------------------- Data -------------------- - loader_seed = cfg.dataloader.seed - if loader_seed is None: - loader_generator = None - else: - loader_generator = torch.Generator().manual_seed(loader_seed) - - # Training dataset - depth_transform: DepthNormalizerBase = get_depth_normalizer( - cfg_normalizer=cfg.depth_normalization - ) - train_dataset: BaseDepthDataset = get_dataset( - cfg_data.train, - base_data_dir=base_data_dir, - mode=DatasetMode.TRAIN, - augmentation_args=cfg.augmentation, - depth_transform=depth_transform, - ) - logging.debug("Augmentation: ", cfg.augmentation) - if "mixed" == cfg_data.train.name: - dataset_ls = train_dataset - assert len(cfg_data.train.prob_ls) == len( - dataset_ls - ), "Lengths don't match: `prob_ls` and `dataset_list`" - concat_dataset = ConcatDataset(dataset_ls) - mixed_sampler = MixedBatchSampler( - src_dataset_ls=dataset_ls, - batch_size=cfg.dataloader.max_train_batch_size, - drop_last=True, - prob=cfg_data.train.prob_ls, - shuffle=True, - generator=loader_generator, - ) - train_loader = DataLoader( - concat_dataset, - batch_sampler=mixed_sampler, - num_workers=cfg.dataloader.num_workers, - ) - else: - train_loader = DataLoader( - dataset=train_dataset, - batch_size=cfg.dataloader.max_train_batch_size, - num_workers=cfg.dataloader.num_workers, - shuffle=True, - generator=loader_generator, - ) - - # -------------------- Model -------------------- - _pipeline_kwargs = cfg.pipeline.kwargs if cfg.pipeline.kwargs is not None else {} - model = ApDepthPipeline.from_pretrained( - os.path.join(base_ckpt_dir, cfg.model.pretrained_path), **_pipeline_kwargs - ) - - # -------------------- Trainer -------------------- - # Exit time - if args.exit_after > 0: - t_end = t_start + timedelta(minutes=args.exit_after) - logging.info(f"Will exit at {t_end}") - else: - t_end = None - - trainer_cls = get_trainer_cls(cfg.trainer.name) - logging.debug(f"Trainer: {trainer_cls}") - trainer = trainer_cls( - cfg=cfg, - model=model, - train_dataloader=train_loader, - device=device, - base_ckpt_dir=base_ckpt_dir, - out_dir_ckpt=out_dir_ckpt, - out_dir_eval=out_dir_eval, - out_dir_vis=out_dir_vis, - accumulation_steps=accumulation_steps, - ) - - # -------------------- Checkpoint -------------------- - if resume_run is not None: - trainer.load_checkpoint( - resume_run, load_trainer_state=True, resume_lr_scheduler=True - ) - - # -------------------- Training & Evaluation Loop -------------------- - try: - trainer.train(t_end=t_end) - except Exception as e: - logging.exception(e) diff --git a/config/apdepth_train_s1.yaml b/config/apdepth_train_s1.yaml deleted file mode 100644 index dd7fad8..0000000 --- a/config/apdepth_train_s1.yaml +++ /dev/null @@ -1,92 +0,0 @@ -base_config: -- config/logging.yaml -- config/wandb.yaml -- config/dataset/dataset_apdepth_train_s1.yaml -- config/dataset/dataset_val.yaml -- config/dataset/dataset_vis.yaml -- config/model_sdv2.yaml - - -pipeline: - name: ApDepthPipeline - kwargs: - scale_invariant: true - shift_invariant: true - -depth_normalization: - type: scale_shift_depth - clip: true - norm_min: -1.0 - norm_max: 1.0 - min_max_quantile: 0.02 - -augmentation: - lr_flip_p: 0.5 - -dataloader: - num_workers: 32 - effective_batch_size: 32 - max_train_batch_size: 4 - seed: 2024 # to ensure continuity when resuming from checkpoint - -# Training settings -trainer: - name: ApDepthTrainerS1 - dinov2_pretrained_path: DA2/checkpoints/depth_anything_v2_vitg.pth - training_noise_scheduler: - pretrained_path: stable-diffusion-2 - init_seed: 2024 # use null to train w/o seeding - save_period: 250 - backup_period: 18000 - validation_period: 4000000 - visualization_period: 4000000 - -gt_depth_type: depth_raw_norm -gt_mask_type: valid_mask_raw - -max_epoch: 10000 # a large enough number -max_iter: 20000 # usually converges at around 20k - -optimizer: - name: Adam - -loss: - name: mse_loss - kwargs: - reduction: mean - -loss_feat_align: - lamda: 1 - -lr: 3.0e-05 -lr_scheduler: - name: IterExponential - kwargs: - total_iter: 25000 - final_ratio: 0.01 - warmup_steps: 100 - -# Validation (and visualization) settings -validation: - ensemble_size: 1 # simplified setting for on-training validation - processing_res: 0 - match_input_res: false - resample_method: bilinear - main_val_metric: abs_relative_difference - main_val_metric_goal: minimize - init_seed: 2024 - -eval: - alignment: least_square_sqrt_disp - align_max_res: null - eval_metrics: - - abs_relative_difference - - squared_relative_difference - - rmse_linear - - rmse_log - - log10 - - delta1_acc - - delta2_acc - - delta3_acc - - i_rmse - - silog_rmse diff --git a/config/dataset/dataset_apdepth_train_s1.yaml b/config/dataset/dataset_apdepth_train_s1.yaml deleted file mode 100644 index 6fc4431..0000000 --- a/config/dataset/dataset_apdepth_train_s1.yaml +++ /dev/null @@ -1,18 +0,0 @@ -dataset: - train: - name: mixed - prob_ls: [0.9, 0.1] - dataset_list: - - name: hypersim - disp_name: hypersim_train - dir: Hypersim/precessed/train - filenames: data_split/hypersim/filename_list_train_filtered.txt - resize_to_hw: - - 480 - - 640 - - name: vkitti - disp_name: vkitti_train - dir: vkitti - filenames: data_split/vkitti/vkitti_train.txt - kitti_bm_crop: true - valid_mask_crop: null \ No newline at end of file diff --git a/config/train_apdepth.yaml b/config/train_apdepth.yaml index 6433f29..feeed27 100644 --- a/config/train_apdepth.yaml +++ b/config/train_apdepth.yaml @@ -67,6 +67,12 @@ latent_freq_loss: lambda: 0.5 lr: 3.0e-05 +# Stage 2 uses the original objective. Enable known-far supervision only in +# config/train_sky_finetune.yaml for post-training from a completed Stage 2 model. +far_depth_supervision: + enabled: false + weight: 0.5 + lr_scheduler: name: IterExponential kwargs: @@ -97,4 +103,4 @@ eval: - delta2_acc - delta3_acc - i_rmse - - silog_rmse \ No newline at end of file + - silog_rmse diff --git a/config/train_sky_finetune.yaml b/config/train_sky_finetune.yaml new file mode 100644 index 0000000..01be5bd --- /dev/null +++ b/config/train_sky_finetune.yaml @@ -0,0 +1,21 @@ +# Optional post-training AFTER Stage 2 (including FFT refinement) is complete. +# Initialize from --init_checkpoint output/train_apdepth/checkpoint/iter_021000. +# The 6000 new steps use reconstruction + far supervision with fresh optimizer/LR. +# Use --resume_run only to resume this post-training run with its saved config. +base_config: +- config/train_apdepth.yaml + +lr: 5.0e-06 +max_iter: 6000 +lr_scheduler: + kwargs: + total_iter: 6000 + final_ratio: 0.1 + warmup_steps: 100 + +far_depth_supervision: + enabled: true + weight: 0.5 + +trainer: + backup_period: 1000 diff --git a/doc/main.jpg b/doc/main.jpg deleted file mode 100644 index 79efa3c..0000000 Binary files a/doc/main.jpg and /dev/null differ diff --git a/external_encoder/__init__.py b/external_encoder/__init__.py deleted file mode 100644 index 8b13789..0000000 --- a/external_encoder/__init__.py +++ /dev/null @@ -1 +0,0 @@ - diff --git a/external_encoder/dinov2/__init__.py b/external_encoder/dinov2/__init__.py deleted file mode 100644 index 8b13789..0000000 --- a/external_encoder/dinov2/__init__.py +++ /dev/null @@ -1 +0,0 @@ - diff --git a/external_encoder/dinov2/dinov2.py b/external_encoder/dinov2/dinov2.py deleted file mode 100644 index 83d2508..0000000 --- a/external_encoder/dinov2/dinov2.py +++ /dev/null @@ -1,415 +0,0 @@ -# Copyright (c) Meta Platforms, Inc. and affiliates. -# -# This source code is licensed under the Apache License, Version 2.0 -# found in the LICENSE file in the root directory of this source tree. - -# References: -# https://github.com/facebookresearch/dino/blob/main/vision_transformer.py -# https://github.com/rwightman/pytorch-image-models/tree/master/timm/models/vision_transformer.py - -from functools import partial -import math -import logging -from typing import Sequence, Tuple, Union, Callable - -import torch -import torch.nn as nn -import torch.utils.checkpoint -from torch.nn.init import trunc_normal_ - -from .dinov2_layers import Mlp, PatchEmbed, SwiGLUFFNFused, MemEffAttention, NestedTensorBlock as Block - - -logger = logging.getLogger("dinov2") - - -def named_apply(fn: Callable, module: nn.Module, name="", depth_first=True, include_root=False) -> nn.Module: - if not depth_first and include_root: - fn(module=module, name=name) - for child_name, child_module in module.named_children(): - child_name = ".".join((name, child_name)) if name else child_name - named_apply(fn=fn, module=child_module, name=child_name, depth_first=depth_first, include_root=True) - if depth_first and include_root: - fn(module=module, name=name) - return module - - -class BlockChunk(nn.ModuleList): - def forward(self, x): - for b in self: - x = b(x) - return x - - -class DinoVisionTransformer(nn.Module): - def __init__( - self, - img_size=224, - patch_size=16, - in_chans=3, - embed_dim=768, - depth=12, - num_heads=12, - mlp_ratio=4.0, - qkv_bias=True, - ffn_bias=True, - proj_bias=True, - drop_path_rate=0.0, - drop_path_uniform=False, - init_values=None, # for layerscale: None or 0 => no layerscale - embed_layer=PatchEmbed, - act_layer=nn.GELU, - block_fn=Block, - ffn_layer="mlp", - block_chunks=1, - num_register_tokens=0, - interpolate_antialias=False, - interpolate_offset=0.1, - ): - """ - Args: - img_size (int, tuple): input image size - patch_size (int, tuple): patch size - in_chans (int): number of input channels - embed_dim (int): embedding dimension - depth (int): depth of transformer - num_heads (int): number of attention heads - mlp_ratio (int): ratio of mlp hidden dim to embedding dim - qkv_bias (bool): enable bias for qkv if True - proj_bias (bool): enable bias for proj in attn if True - ffn_bias (bool): enable bias for ffn if True - drop_path_rate (float): stochastic depth rate - drop_path_uniform (bool): apply uniform drop rate across blocks - weight_init (str): weight init scheme - init_values (float): layer-scale init values - embed_layer (nn.Module): patch embedding layer - act_layer (nn.Module): MLP activation layer - block_fn (nn.Module): transformer block class - ffn_layer (str): "mlp", "swiglu", "swiglufused" or "identity" - block_chunks: (int) split block sequence into block_chunks units for FSDP wrap - num_register_tokens: (int) number of extra cls tokens (so-called "registers") - interpolate_antialias: (str) flag to apply anti-aliasing when interpolating positional embeddings - interpolate_offset: (float) work-around offset to apply when interpolating positional embeddings - """ - super().__init__() - norm_layer = partial(nn.LayerNorm, eps=1e-6) - - self.num_features = self.embed_dim = embed_dim # num_features for consistency with other models - self.num_tokens = 1 - self.n_blocks = depth - self.num_heads = num_heads - self.patch_size = patch_size - self.num_register_tokens = num_register_tokens - self.interpolate_antialias = interpolate_antialias - self.interpolate_offset = interpolate_offset - - self.patch_embed = embed_layer(img_size=img_size, patch_size=patch_size, in_chans=in_chans, embed_dim=embed_dim) - num_patches = self.patch_embed.num_patches - - self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim)) - self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + self.num_tokens, embed_dim)) - assert num_register_tokens >= 0 - self.register_tokens = ( - nn.Parameter(torch.zeros(1, num_register_tokens, embed_dim)) if num_register_tokens else None - ) - - if drop_path_uniform is True: - dpr = [drop_path_rate] * depth - else: - dpr = [x.item() for x in torch.linspace(0, drop_path_rate, depth)] # stochastic depth decay rule - - if ffn_layer == "mlp": - logger.info("using MLP layer as FFN") - ffn_layer = Mlp - elif ffn_layer == "swiglufused" or ffn_layer == "swiglu": - logger.info("using SwiGLU layer as FFN") - ffn_layer = SwiGLUFFNFused - elif ffn_layer == "identity": - logger.info("using Identity layer as FFN") - - def f(*args, **kwargs): - return nn.Identity() - - ffn_layer = f - else: - raise NotImplementedError - - blocks_list = [ - block_fn( - dim=embed_dim, - num_heads=num_heads, - mlp_ratio=mlp_ratio, - qkv_bias=qkv_bias, - proj_bias=proj_bias, - ffn_bias=ffn_bias, - drop_path=dpr[i], - norm_layer=norm_layer, - act_layer=act_layer, - ffn_layer=ffn_layer, - init_values=init_values, - ) - for i in range(depth) - ] - if block_chunks > 0: - self.chunked_blocks = True - chunked_blocks = [] - chunksize = depth // block_chunks - for i in range(0, depth, chunksize): - # this is to keep the block index consistent if we chunk the block list - chunked_blocks.append([nn.Identity()] * i + blocks_list[i : i + chunksize]) - self.blocks = nn.ModuleList([BlockChunk(p) for p in chunked_blocks]) - else: - self.chunked_blocks = False - self.blocks = nn.ModuleList(blocks_list) - - self.norm = norm_layer(embed_dim) - self.head = nn.Identity() - - self.mask_token = nn.Parameter(torch.zeros(1, embed_dim)) - - self.init_weights() - - def init_weights(self): - trunc_normal_(self.pos_embed, std=0.02) - nn.init.normal_(self.cls_token, std=1e-6) - if self.register_tokens is not None: - nn.init.normal_(self.register_tokens, std=1e-6) - named_apply(init_weights_vit_timm, self) - - def interpolate_pos_encoding(self, x, w, h): - previous_dtype = x.dtype - npatch = x.shape[1] - 1 - N = self.pos_embed.shape[1] - 1 - if npatch == N and w == h: - return self.pos_embed - pos_embed = self.pos_embed.float() - class_pos_embed = pos_embed[:, 0] - patch_pos_embed = pos_embed[:, 1:] - dim = x.shape[-1] - w0 = w // self.patch_size - h0 = h // self.patch_size - # we add a small number to avoid floating point error in the interpolation - # see discussion at https://github.com/facebookresearch/dino/issues/8 - # DINOv2 with register modify the interpolate_offset from 0.1 to 0.0 - w0, h0 = w0 + self.interpolate_offset, h0 + self.interpolate_offset - # w0, h0 = w0 + 0.1, h0 + 0.1 - - sqrt_N = math.sqrt(N) - sx, sy = float(w0) / sqrt_N, float(h0) / sqrt_N - patch_pos_embed = nn.functional.interpolate( - patch_pos_embed.reshape(1, int(sqrt_N), int(sqrt_N), dim).permute(0, 3, 1, 2), - scale_factor=(sx, sy), - # (int(w0), int(h0)), # to solve the upsampling shape issue - mode="bicubic", - antialias=self.interpolate_antialias - ) - - assert int(w0) == patch_pos_embed.shape[-2] - assert int(h0) == patch_pos_embed.shape[-1] - patch_pos_embed = patch_pos_embed.permute(0, 2, 3, 1).view(1, -1, dim) - return torch.cat((class_pos_embed.unsqueeze(0), patch_pos_embed), dim=1).to(previous_dtype) - - def prepare_tokens_with_masks(self, x, masks=None): - B, nc, w, h = x.shape - x = self.patch_embed(x) - if masks is not None: - x = torch.where(masks.unsqueeze(-1), self.mask_token.to(x.dtype).unsqueeze(0), x) - - x = torch.cat((self.cls_token.expand(x.shape[0], -1, -1), x), dim=1) - x = x + self.interpolate_pos_encoding(x, w, h) - - if self.register_tokens is not None: - x = torch.cat( - ( - x[:, :1], - self.register_tokens.expand(x.shape[0], -1, -1), - x[:, 1:], - ), - dim=1, - ) - - return x - - def forward_features_list(self, x_list, masks_list): - x = [self.prepare_tokens_with_masks(x, masks) for x, masks in zip(x_list, masks_list)] - for blk in self.blocks: - x = blk(x) - - all_x = x - output = [] - for x, masks in zip(all_x, masks_list): - x_norm = self.norm(x) - output.append( - { - "x_norm_clstoken": x_norm[:, 0], - "x_norm_regtokens": x_norm[:, 1 : self.num_register_tokens + 1], - "x_norm_patchtokens": x_norm[:, self.num_register_tokens + 1 :], - "x_prenorm": x, - "masks": masks, - } - ) - return output - - def forward_features(self, x, masks=None): - if isinstance(x, list): - return self.forward_features_list(x, masks) - - x = self.prepare_tokens_with_masks(x, masks) - - for blk in self.blocks: - x = blk(x) - - x_norm = self.norm(x) - return { - "x_norm_clstoken": x_norm[:, 0], - "x_norm_regtokens": x_norm[:, 1 : self.num_register_tokens + 1], - "x_norm_patchtokens": x_norm[:, self.num_register_tokens + 1 :], - "x_prenorm": x, - "masks": masks, - } - - def _get_intermediate_layers_not_chunked(self, x, n=1): - x = self.prepare_tokens_with_masks(x) - # If n is an int, take the n last blocks. If it's a list, take them - output, total_block_len = [], len(self.blocks) - blocks_to_take = range(total_block_len - n, total_block_len) if isinstance(n, int) else n - for i, blk in enumerate(self.blocks): - x = blk(x) - if i in blocks_to_take: - output.append(x) - assert len(output) == len(blocks_to_take), f"only {len(output)} / {len(blocks_to_take)} blocks found" - return output - - def _get_intermediate_layers_chunked(self, x, n=1): - x = self.prepare_tokens_with_masks(x) - output, i, total_block_len = [], 0, len(self.blocks[-1]) - # If n is an int, take the n last blocks. If it's a list, take them - blocks_to_take = range(total_block_len - n, total_block_len) if isinstance(n, int) else n - for block_chunk in self.blocks: - for blk in block_chunk[i:]: # Passing the nn.Identity() - x = blk(x) - if i in blocks_to_take: - output.append(x) - i += 1 - assert len(output) == len(blocks_to_take), f"only {len(output)} / {len(blocks_to_take)} blocks found" - return output - - def get_intermediate_layers( - self, - x: torch.Tensor, - n: Union[int, Sequence] = 1, # Layers or n last layers to take - reshape: bool = False, - return_class_token: bool = False, - norm=True - ) -> Tuple[Union[torch.Tensor, Tuple[torch.Tensor]]]: - if self.chunked_blocks: - outputs = self._get_intermediate_layers_chunked(x, n) - else: - outputs = self._get_intermediate_layers_not_chunked(x, n) - if norm: - outputs = [self.norm(out) for out in outputs] - class_tokens = [out[:, 0] for out in outputs] - outputs = [out[:, 1 + self.num_register_tokens:] for out in outputs] - if reshape: - B, _, w, h = x.shape - outputs = [ - out.reshape(B, w // self.patch_size, h // self.patch_size, -1).permute(0, 3, 1, 2).contiguous() - for out in outputs - ] - if return_class_token: - return tuple(zip(outputs, class_tokens)) - return tuple(outputs) - - def forward(self, *args, is_training=False, **kwargs): - ret = self.forward_features(*args, **kwargs) - if is_training: - return ret - else: - return self.head(ret["x_norm_clstoken"]) - - -def init_weights_vit_timm(module: nn.Module, name: str = ""): - """ViT weight initialization, original timm impl (for reproducibility)""" - if isinstance(module, nn.Linear): - trunc_normal_(module.weight, std=0.02) - if module.bias is not None: - nn.init.zeros_(module.bias) - - -def vit_small(patch_size=16, num_register_tokens=0, **kwargs): - model = DinoVisionTransformer( - patch_size=patch_size, - embed_dim=384, - depth=12, - num_heads=6, - mlp_ratio=4, - block_fn=partial(Block, attn_class=MemEffAttention), - num_register_tokens=num_register_tokens, - **kwargs, - ) - return model - - -def vit_base(patch_size=16, num_register_tokens=0, **kwargs): - model = DinoVisionTransformer( - patch_size=patch_size, - embed_dim=768, - depth=12, - num_heads=12, - mlp_ratio=4, - block_fn=partial(Block, attn_class=MemEffAttention), - num_register_tokens=num_register_tokens, - **kwargs, - ) - return model - - -def vit_large(patch_size=16, num_register_tokens=0, **kwargs): - model = DinoVisionTransformer( - patch_size=patch_size, - embed_dim=1024, - depth=24, - num_heads=16, - mlp_ratio=4, - block_fn=partial(Block, attn_class=MemEffAttention), - num_register_tokens=num_register_tokens, - **kwargs, - ) - return model - - -def vit_giant2(patch_size=16, num_register_tokens=0, **kwargs): - """ - Close to ViT-giant, with embed-dim 1536 and 24 heads => embed-dim per head 64 - """ - model = DinoVisionTransformer( - patch_size=patch_size, - embed_dim=1536, - depth=40, - num_heads=24, - mlp_ratio=4, - block_fn=partial(Block, attn_class=MemEffAttention), - num_register_tokens=num_register_tokens, - **kwargs, - ) - return model - - -def DINOv2(model_name): - model_zoo = { - "vits": vit_small, - "vitb": vit_base, - "vitl": vit_large, - "vitg": vit_giant2 - } - - return model_zoo[model_name]( - img_size=518, - patch_size=14, - init_values=1.0, - ffn_layer="mlp" if model_name != "vitg" else "swiglufused", - block_chunks=0, - num_register_tokens=0, - interpolate_antialias=False, - interpolate_offset=0.1 - ) diff --git a/external_encoder/dinov2/dinov2_layers/__init__.py b/external_encoder/dinov2/dinov2_layers/__init__.py deleted file mode 100644 index 8120f4b..0000000 --- a/external_encoder/dinov2/dinov2_layers/__init__.py +++ /dev/null @@ -1,11 +0,0 @@ -# Copyright (c) Meta Platforms, Inc. and affiliates. -# All rights reserved. -# -# This source code is licensed under the license found in the -# LICENSE file in the root directory of this source tree. - -from .mlp import Mlp -from .patch_embed import PatchEmbed -from .swiglu_ffn import SwiGLUFFN, SwiGLUFFNFused -from .block import NestedTensorBlock -from .attention import MemEffAttention diff --git a/external_encoder/dinov2/dinov2_layers/attention.py b/external_encoder/dinov2/dinov2_layers/attention.py deleted file mode 100644 index 815a2bf..0000000 --- a/external_encoder/dinov2/dinov2_layers/attention.py +++ /dev/null @@ -1,83 +0,0 @@ -# Copyright (c) Meta Platforms, Inc. and affiliates. -# All rights reserved. -# -# This source code is licensed under the license found in the -# LICENSE file in the root directory of this source tree. - -# References: -# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py -# https://github.com/rwightman/pytorch-image-models/tree/master/timm/models/vision_transformer.py - -import logging - -from torch import Tensor -from torch import nn - - -logger = logging.getLogger("dinov2") - - -try: - from xformers.ops import memory_efficient_attention, unbind, fmha - - XFORMERS_AVAILABLE = True -except ImportError: - logger.warning("xFormers not available") - XFORMERS_AVAILABLE = False - - -class Attention(nn.Module): - def __init__( - self, - dim: int, - num_heads: int = 8, - qkv_bias: bool = False, - proj_bias: bool = True, - attn_drop: float = 0.0, - proj_drop: float = 0.0, - ) -> None: - super().__init__() - self.num_heads = num_heads - head_dim = dim // num_heads - self.scale = head_dim**-0.5 - - self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) - self.attn_drop = nn.Dropout(attn_drop) - self.proj = nn.Linear(dim, dim, bias=proj_bias) - self.proj_drop = nn.Dropout(proj_drop) - - def forward(self, x: Tensor) -> Tensor: - B, N, C = x.shape - qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4) - - q, k, v = qkv[0] * self.scale, qkv[1], qkv[2] - attn = q @ k.transpose(-2, -1) - - attn = attn.softmax(dim=-1) - attn = self.attn_drop(attn) - - x = (attn @ v).transpose(1, 2).reshape(B, N, C) - x = self.proj(x) - x = self.proj_drop(x) - return x - - -class MemEffAttention(Attention): - def forward(self, x: Tensor, attn_bias=None) -> Tensor: - if not XFORMERS_AVAILABLE: - assert attn_bias is None, "xFormers is required for nested tensors usage" - return super().forward(x) - - B, N, C = x.shape - qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads) - - q, k, v = unbind(qkv, 2) - - x = memory_efficient_attention(q, k, v, attn_bias=attn_bias) - x = x.reshape([B, N, C]) - - x = self.proj(x) - x = self.proj_drop(x) - return x - - \ No newline at end of file diff --git a/external_encoder/dinov2/dinov2_layers/block.py b/external_encoder/dinov2/dinov2_layers/block.py deleted file mode 100644 index 25488f5..0000000 --- a/external_encoder/dinov2/dinov2_layers/block.py +++ /dev/null @@ -1,252 +0,0 @@ -# Copyright (c) Meta Platforms, Inc. and affiliates. -# All rights reserved. -# -# This source code is licensed under the license found in the -# LICENSE file in the root directory of this source tree. - -# References: -# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py -# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/patch_embed.py - -import logging -from typing import Callable, List, Any, Tuple, Dict - -import torch -from torch import nn, Tensor - -from .attention import Attention, MemEffAttention -from .drop_path import DropPath -from .layer_scale import LayerScale -from .mlp import Mlp - - -logger = logging.getLogger("dinov2") - - -try: - from xformers.ops import fmha - from xformers.ops import scaled_index_add, index_select_cat - - XFORMERS_AVAILABLE = True -except ImportError: - logger.warning("xFormers not available") - XFORMERS_AVAILABLE = False - - -class Block(nn.Module): - def __init__( - self, - dim: int, - num_heads: int, - mlp_ratio: float = 4.0, - qkv_bias: bool = False, - proj_bias: bool = True, - ffn_bias: bool = True, - drop: float = 0.0, - attn_drop: float = 0.0, - init_values=None, - drop_path: float = 0.0, - act_layer: Callable[..., nn.Module] = nn.GELU, - norm_layer: Callable[..., nn.Module] = nn.LayerNorm, - attn_class: Callable[..., nn.Module] = Attention, - ffn_layer: Callable[..., nn.Module] = Mlp, - ) -> None: - super().__init__() - # print(f"biases: qkv: {qkv_bias}, proj: {proj_bias}, ffn: {ffn_bias}") - self.norm1 = norm_layer(dim) - self.attn = attn_class( - dim, - num_heads=num_heads, - qkv_bias=qkv_bias, - proj_bias=proj_bias, - attn_drop=attn_drop, - proj_drop=drop, - ) - self.ls1 = LayerScale(dim, init_values=init_values) if init_values else nn.Identity() - self.drop_path1 = DropPath(drop_path) if drop_path > 0.0 else nn.Identity() - - self.norm2 = norm_layer(dim) - mlp_hidden_dim = int(dim * mlp_ratio) - self.mlp = ffn_layer( - in_features=dim, - hidden_features=mlp_hidden_dim, - act_layer=act_layer, - drop=drop, - bias=ffn_bias, - ) - self.ls2 = LayerScale(dim, init_values=init_values) if init_values else nn.Identity() - self.drop_path2 = DropPath(drop_path) if drop_path > 0.0 else nn.Identity() - - self.sample_drop_ratio = drop_path - - def forward(self, x: Tensor) -> Tensor: - def attn_residual_func(x: Tensor) -> Tensor: - return self.ls1(self.attn(self.norm1(x))) - - def ffn_residual_func(x: Tensor) -> Tensor: - return self.ls2(self.mlp(self.norm2(x))) - - if self.training and self.sample_drop_ratio > 0.1: - # the overhead is compensated only for a drop path rate larger than 0.1 - x = drop_add_residual_stochastic_depth( - x, - residual_func=attn_residual_func, - sample_drop_ratio=self.sample_drop_ratio, - ) - x = drop_add_residual_stochastic_depth( - x, - residual_func=ffn_residual_func, - sample_drop_ratio=self.sample_drop_ratio, - ) - elif self.training and self.sample_drop_ratio > 0.0: - x = x + self.drop_path1(attn_residual_func(x)) - x = x + self.drop_path1(ffn_residual_func(x)) # FIXME: drop_path2 - else: - x = x + attn_residual_func(x) - x = x + ffn_residual_func(x) - return x - - -def drop_add_residual_stochastic_depth( - x: Tensor, - residual_func: Callable[[Tensor], Tensor], - sample_drop_ratio: float = 0.0, -) -> Tensor: - # 1) extract subset using permutation - b, n, d = x.shape - sample_subset_size = max(int(b * (1 - sample_drop_ratio)), 1) - brange = (torch.randperm(b, device=x.device))[:sample_subset_size] - x_subset = x[brange] - - # 2) apply residual_func to get residual - residual = residual_func(x_subset) - - x_flat = x.flatten(1) - residual = residual.flatten(1) - - residual_scale_factor = b / sample_subset_size - - # 3) add the residual - x_plus_residual = torch.index_add(x_flat, 0, brange, residual.to(dtype=x.dtype), alpha=residual_scale_factor) - return x_plus_residual.view_as(x) - - -def get_branges_scales(x, sample_drop_ratio=0.0): - b, n, d = x.shape - sample_subset_size = max(int(b * (1 - sample_drop_ratio)), 1) - brange = (torch.randperm(b, device=x.device))[:sample_subset_size] - residual_scale_factor = b / sample_subset_size - return brange, residual_scale_factor - - -def add_residual(x, brange, residual, residual_scale_factor, scaling_vector=None): - if scaling_vector is None: - x_flat = x.flatten(1) - residual = residual.flatten(1) - x_plus_residual = torch.index_add(x_flat, 0, brange, residual.to(dtype=x.dtype), alpha=residual_scale_factor) - else: - x_plus_residual = scaled_index_add( - x, brange, residual.to(dtype=x.dtype), scaling=scaling_vector, alpha=residual_scale_factor - ) - return x_plus_residual - - -attn_bias_cache: Dict[Tuple, Any] = {} - - -def get_attn_bias_and_cat(x_list, branges=None): - """ - this will perform the index select, cat the tensors, and provide the attn_bias from cache - """ - batch_sizes = [b.shape[0] for b in branges] if branges is not None else [x.shape[0] for x in x_list] - all_shapes = tuple((b, x.shape[1]) for b, x in zip(batch_sizes, x_list)) - if all_shapes not in attn_bias_cache.keys(): - seqlens = [] - for b, x in zip(batch_sizes, x_list): - for _ in range(b): - seqlens.append(x.shape[1]) - attn_bias = fmha.BlockDiagonalMask.from_seqlens(seqlens) - attn_bias._batch_sizes = batch_sizes - attn_bias_cache[all_shapes] = attn_bias - - if branges is not None: - cat_tensors = index_select_cat([x.flatten(1) for x in x_list], branges).view(1, -1, x_list[0].shape[-1]) - else: - tensors_bs1 = tuple(x.reshape([1, -1, *x.shape[2:]]) for x in x_list) - cat_tensors = torch.cat(tensors_bs1, dim=1) - - return attn_bias_cache[all_shapes], cat_tensors - - -def drop_add_residual_stochastic_depth_list( - x_list: List[Tensor], - residual_func: Callable[[Tensor, Any], Tensor], - sample_drop_ratio: float = 0.0, - scaling_vector=None, -) -> Tensor: - # 1) generate random set of indices for dropping samples in the batch - branges_scales = [get_branges_scales(x, sample_drop_ratio=sample_drop_ratio) for x in x_list] - branges = [s[0] for s in branges_scales] - residual_scale_factors = [s[1] for s in branges_scales] - - # 2) get attention bias and index+concat the tensors - attn_bias, x_cat = get_attn_bias_and_cat(x_list, branges) - - # 3) apply residual_func to get residual, and split the result - residual_list = attn_bias.split(residual_func(x_cat, attn_bias=attn_bias)) # type: ignore - - outputs = [] - for x, brange, residual, residual_scale_factor in zip(x_list, branges, residual_list, residual_scale_factors): - outputs.append(add_residual(x, brange, residual, residual_scale_factor, scaling_vector).view_as(x)) - return outputs - - -class NestedTensorBlock(Block): - def forward_nested(self, x_list: List[Tensor]) -> List[Tensor]: - """ - x_list contains a list of tensors to nest together and run - """ - assert isinstance(self.attn, MemEffAttention) - - if self.training and self.sample_drop_ratio > 0.0: - - def attn_residual_func(x: Tensor, attn_bias=None) -> Tensor: - return self.attn(self.norm1(x), attn_bias=attn_bias) - - def ffn_residual_func(x: Tensor, attn_bias=None) -> Tensor: - return self.mlp(self.norm2(x)) - - x_list = drop_add_residual_stochastic_depth_list( - x_list, - residual_func=attn_residual_func, - sample_drop_ratio=self.sample_drop_ratio, - scaling_vector=self.ls1.gamma if isinstance(self.ls1, LayerScale) else None, - ) - x_list = drop_add_residual_stochastic_depth_list( - x_list, - residual_func=ffn_residual_func, - sample_drop_ratio=self.sample_drop_ratio, - scaling_vector=self.ls2.gamma if isinstance(self.ls1, LayerScale) else None, - ) - return x_list - else: - - def attn_residual_func(x: Tensor, attn_bias=None) -> Tensor: - return self.ls1(self.attn(self.norm1(x), attn_bias=attn_bias)) - - def ffn_residual_func(x: Tensor, attn_bias=None) -> Tensor: - return self.ls2(self.mlp(self.norm2(x))) - - attn_bias, x = get_attn_bias_and_cat(x_list) - x = x + attn_residual_func(x, attn_bias=attn_bias) - x = x + ffn_residual_func(x) - return attn_bias.split(x) - - def forward(self, x_or_x_list): - if isinstance(x_or_x_list, Tensor): - return super().forward(x_or_x_list) - elif isinstance(x_or_x_list, list): - assert XFORMERS_AVAILABLE, "Please install xFormers for nested tensors usage" - return self.forward_nested(x_or_x_list) - else: - raise AssertionError diff --git a/external_encoder/dinov2/dinov2_layers/drop_path.py b/external_encoder/dinov2/dinov2_layers/drop_path.py deleted file mode 100644 index af05625..0000000 --- a/external_encoder/dinov2/dinov2_layers/drop_path.py +++ /dev/null @@ -1,35 +0,0 @@ -# Copyright (c) Meta Platforms, Inc. and affiliates. -# All rights reserved. -# -# This source code is licensed under the license found in the -# LICENSE file in the root directory of this source tree. - -# References: -# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py -# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/drop.py - - -from torch import nn - - -def drop_path(x, drop_prob: float = 0.0, training: bool = False): - if drop_prob == 0.0 or not training: - return x - keep_prob = 1 - drop_prob - shape = (x.shape[0],) + (1,) * (x.ndim - 1) # work with diff dim tensors, not just 2D ConvNets - random_tensor = x.new_empty(shape).bernoulli_(keep_prob) - if keep_prob > 0.0: - random_tensor.div_(keep_prob) - output = x * random_tensor - return output - - -class DropPath(nn.Module): - """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).""" - - def __init__(self, drop_prob=None): - super(DropPath, self).__init__() - self.drop_prob = drop_prob - - def forward(self, x): - return drop_path(x, self.drop_prob, self.training) diff --git a/external_encoder/dinov2/dinov2_layers/layer_scale.py b/external_encoder/dinov2/dinov2_layers/layer_scale.py deleted file mode 100644 index ca5daa5..0000000 --- a/external_encoder/dinov2/dinov2_layers/layer_scale.py +++ /dev/null @@ -1,28 +0,0 @@ -# Copyright (c) Meta Platforms, Inc. and affiliates. -# All rights reserved. -# -# This source code is licensed under the license found in the -# LICENSE file in the root directory of this source tree. - -# Modified from: https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/vision_transformer.py#L103-L110 - -from typing import Union - -import torch -from torch import Tensor -from torch import nn - - -class LayerScale(nn.Module): - def __init__( - self, - dim: int, - init_values: Union[float, Tensor] = 1e-5, - inplace: bool = False, - ) -> None: - super().__init__() - self.inplace = inplace - self.gamma = nn.Parameter(init_values * torch.ones(dim)) - - def forward(self, x: Tensor) -> Tensor: - return x.mul_(self.gamma) if self.inplace else x * self.gamma diff --git a/external_encoder/dinov2/dinov2_layers/mlp.py b/external_encoder/dinov2/dinov2_layers/mlp.py deleted file mode 100644 index 5e4b315..0000000 --- a/external_encoder/dinov2/dinov2_layers/mlp.py +++ /dev/null @@ -1,41 +0,0 @@ -# Copyright (c) Meta Platforms, Inc. and affiliates. -# All rights reserved. -# -# This source code is licensed under the license found in the -# LICENSE file in the root directory of this source tree. - -# References: -# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py -# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/mlp.py - - -from typing import Callable, Optional - -from torch import Tensor, nn - - -class Mlp(nn.Module): - def __init__( - self, - in_features: int, - hidden_features: Optional[int] = None, - out_features: Optional[int] = None, - act_layer: Callable[..., nn.Module] = nn.GELU, - drop: float = 0.0, - bias: bool = True, - ) -> None: - super().__init__() - out_features = out_features or in_features - hidden_features = hidden_features or in_features - self.fc1 = nn.Linear(in_features, hidden_features, bias=bias) - self.act = act_layer() - self.fc2 = nn.Linear(hidden_features, out_features, bias=bias) - self.drop = nn.Dropout(drop) - - def forward(self, x: Tensor) -> Tensor: - x = self.fc1(x) - x = self.act(x) - x = self.drop(x) - x = self.fc2(x) - x = self.drop(x) - return x diff --git a/external_encoder/dinov2/dinov2_layers/patch_embed.py b/external_encoder/dinov2/dinov2_layers/patch_embed.py deleted file mode 100644 index 574abe4..0000000 --- a/external_encoder/dinov2/dinov2_layers/patch_embed.py +++ /dev/null @@ -1,89 +0,0 @@ -# Copyright (c) Meta Platforms, Inc. and affiliates. -# All rights reserved. -# -# This source code is licensed under the license found in the -# LICENSE file in the root directory of this source tree. - -# References: -# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py -# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/patch_embed.py - -from typing import Callable, Optional, Tuple, Union - -from torch import Tensor -import torch.nn as nn - - -def make_2tuple(x): - if isinstance(x, tuple): - assert len(x) == 2 - return x - - assert isinstance(x, int) - return (x, x) - - -class PatchEmbed(nn.Module): - """ - 2D image to patch embedding: (B,C,H,W) -> (B,N,D) - - Args: - img_size: Image size. - patch_size: Patch token size. - in_chans: Number of input image channels. - embed_dim: Number of linear projection output channels. - norm_layer: Normalization layer. - """ - - def __init__( - self, - img_size: Union[int, Tuple[int, int]] = 224, - patch_size: Union[int, Tuple[int, int]] = 16, - in_chans: int = 3, - embed_dim: int = 768, - norm_layer: Optional[Callable] = None, - flatten_embedding: bool = True, - ) -> None: - super().__init__() - - image_HW = make_2tuple(img_size) - patch_HW = make_2tuple(patch_size) - patch_grid_size = ( - image_HW[0] // patch_HW[0], - image_HW[1] // patch_HW[1], - ) - - self.img_size = image_HW - self.patch_size = patch_HW - self.patches_resolution = patch_grid_size - self.num_patches = patch_grid_size[0] * patch_grid_size[1] - - self.in_chans = in_chans - self.embed_dim = embed_dim - - self.flatten_embedding = flatten_embedding - - self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_HW, stride=patch_HW) - self.norm = norm_layer(embed_dim) if norm_layer else nn.Identity() - - def forward(self, x: Tensor) -> Tensor: - _, _, H, W = x.shape - patch_H, patch_W = self.patch_size - - assert H % patch_H == 0, f"Input image height {H} is not a multiple of patch height {patch_H}" - assert W % patch_W == 0, f"Input image width {W} is not a multiple of patch width: {patch_W}" - - x = self.proj(x) # B C H W - H, W = x.size(2), x.size(3) - x = x.flatten(2).transpose(1, 2) # B HW C - x = self.norm(x) - if not self.flatten_embedding: - x = x.reshape(-1, H, W, self.embed_dim) # B H W C - return x - - def flops(self) -> float: - Ho, Wo = self.patches_resolution - flops = Ho * Wo * self.embed_dim * self.in_chans * (self.patch_size[0] * self.patch_size[1]) - if self.norm is not None: - flops += Ho * Wo * self.embed_dim - return flops diff --git a/external_encoder/dinov2/dinov2_layers/swiglu_ffn.py b/external_encoder/dinov2/dinov2_layers/swiglu_ffn.py deleted file mode 100644 index b3324b2..0000000 --- a/external_encoder/dinov2/dinov2_layers/swiglu_ffn.py +++ /dev/null @@ -1,63 +0,0 @@ -# Copyright (c) Meta Platforms, Inc. and affiliates. -# All rights reserved. -# -# This source code is licensed under the license found in the -# LICENSE file in the root directory of this source tree. - -from typing import Callable, Optional - -from torch import Tensor, nn -import torch.nn.functional as F - - -class SwiGLUFFN(nn.Module): - def __init__( - self, - in_features: int, - hidden_features: Optional[int] = None, - out_features: Optional[int] = None, - act_layer: Callable[..., nn.Module] = None, - drop: float = 0.0, - bias: bool = True, - ) -> None: - super().__init__() - out_features = out_features or in_features - hidden_features = hidden_features or in_features - self.w12 = nn.Linear(in_features, 2 * hidden_features, bias=bias) - self.w3 = nn.Linear(hidden_features, out_features, bias=bias) - - def forward(self, x: Tensor) -> Tensor: - x12 = self.w12(x) - x1, x2 = x12.chunk(2, dim=-1) - hidden = F.silu(x1) * x2 - return self.w3(hidden) - - -try: - from xformers.ops import SwiGLU - - XFORMERS_AVAILABLE = True -except ImportError: - SwiGLU = SwiGLUFFN - XFORMERS_AVAILABLE = False - - -class SwiGLUFFNFused(SwiGLU): - def __init__( - self, - in_features: int, - hidden_features: Optional[int] = None, - out_features: Optional[int] = None, - act_layer: Callable[..., nn.Module] = None, - drop: float = 0.0, - bias: bool = True, - ) -> None: - out_features = out_features or in_features - hidden_features = hidden_features or in_features - hidden_features = (int(hidden_features * 2 / 3) + 7) // 8 * 8 - super().__init__( - in_features=in_features, - hidden_features=hidden_features, - out_features=out_features, - bias=bias, - ) diff --git a/external_encoder/dinov2/util/transform.py b/external_encoder/dinov2/util/transform.py deleted file mode 100644 index 1d2e589..0000000 --- a/external_encoder/dinov2/util/transform.py +++ /dev/null @@ -1,160 +0,0 @@ -import numpy as np -import cv2 -import torch.nn.functional as F - - -class Resize(object): - """Resize sample to given size (width, height). - """ - - def __init__( - self, - width, - height, - resize_target=True, - keep_aspect_ratio=False, - ensure_multiple_of=1, - resize_method="lower_bound", - image_interpolation_method=cv2.INTER_AREA, - ): - """Init. - - Args: - width (int): desired output width - height (int): desired output height - resize_target (bool, optional): - True: Resize the full sample (image, mask, target). - False: Resize image only. - Defaults to True. - keep_aspect_ratio (bool, optional): - True: Keep the aspect ratio of the input sample. - Output sample might not have the given width and height, and - resize behaviour depends on the parameter 'resize_method'. - Defaults to False. - ensure_multiple_of (int, optional): - Output width and height is constrained to be multiple of this parameter. - Defaults to 1. - resize_method (str, optional): - "lower_bound": Output will be at least as large as the given size. - "upper_bound": Output will be at max as large as the given size. (Output size might be smaller than given size.) - "minimal": Scale as least as possible. (Output size might be smaller than given size.) - Defaults to "lower_bound". - """ - self.__width = width - self.__height = height - - self.__resize_target = resize_target - self.__keep_aspect_ratio = keep_aspect_ratio - self.__multiple_of = ensure_multiple_of - self.__resize_method = resize_method - self.__image_interpolation_method = image_interpolation_method - - def constrain_to_multiple_of(self, x, min_val=0, max_val=None): - y = (np.round(x / self.__multiple_of) * self.__multiple_of).astype(int) - - if max_val is not None and y > max_val: - y = (np.floor(x / self.__multiple_of) * self.__multiple_of).astype(int) - - if y < min_val: - y = (np.ceil(x / self.__multiple_of) * self.__multiple_of).astype(int) - - return y - - def get_size(self, width, height): - # determine new height and width - scale_height = self.__height / height - scale_width = self.__width / width - - if self.__keep_aspect_ratio: - if self.__resize_method == "lower_bound": - # scale such that output size is lower bound - if scale_width > scale_height: - # fit width - scale_height = scale_width - else: - # fit height - scale_width = scale_height - elif self.__resize_method == "upper_bound": - # scale such that output size is upper bound - if scale_width < scale_height: - # fit width - scale_height = scale_width - else: - # fit height - scale_width = scale_height - elif self.__resize_method == "minimal": - # scale as least as possbile - if abs(1 - scale_width) < abs(1 - scale_height): - # fit width - scale_height = scale_width - else: - # fit height - scale_width = scale_height - else: - raise ValueError(f"resize_method {self.__resize_method} not implemented") - - if self.__resize_method == "lower_bound": - new_height = self.constrain_to_multiple_of(scale_height * height, min_val=self.__height) - new_width = self.constrain_to_multiple_of(scale_width * width, min_val=self.__width) - elif self.__resize_method == "upper_bound": - new_height = self.constrain_to_multiple_of(scale_height * height, max_val=self.__height) - new_width = self.constrain_to_multiple_of(scale_width * width, max_val=self.__width) - elif self.__resize_method == "minimal": - new_height = self.constrain_to_multiple_of(scale_height * height) - new_width = self.constrain_to_multiple_of(scale_width * width) - else: - raise ValueError(f"resize_method {self.__resize_method} not implemented") - - return (new_width, new_height) - - def __call__(self, sample): - width, height = self.get_size(sample["image"].shape[-1], sample["image"].shape[-2]) - - # resize sample - # sample["image"] = cv2.resize(sample["image"], dsize=(width, height), interpolation=cv2.INTER_NEAREST) - sample["image"] = F.interpolate(sample["image"], size=(height, width), mode='bilinear', align_corners=False) - - if self.__resize_target: - if "depth" in sample: - sample["depth"] = cv2.resize(sample["depth"], (width, height), interpolation=cv2.INTER_NEAREST) - - if "mask" in sample: - sample["mask"] = cv2.resize(sample["mask"].astype(np.float32), (width, height), interpolation=cv2.INTER_NEAREST) - - return sample, (height, width) - - -class NormalizeImage(object): - """Normlize image by given mean and std. - """ - - def __init__(self, mean, std): - self.__mean = mean - self.__std = std - - def __call__(self, sample): - sample["image"] = (sample["image"] - self.__mean) / self.__std - - return sample - - -class PrepareForNet(object): - """Prepare sample for usage as network input. - """ - - def __init__(self): - pass - - def __call__(self, sample): - image = np.transpose(sample["image"], (2, 0, 1)) - sample["image"] = np.ascontiguousarray(image).astype(np.float32) - - if "depth" in sample: - depth = sample["depth"].astype(np.float32) - sample["depth"] = np.ascontiguousarray(depth) - - if "mask" in sample: - sample["mask"] = sample["mask"].astype(np.float32) - sample["mask"] = np.ascontiguousarray(sample["mask"]) - - return sample \ No newline at end of file diff --git a/script/apdepth_train_s1.sh b/script/apdepth_train_s1.sh deleted file mode 100644 index bf85322..0000000 --- a/script/apdepth_train_s1.sh +++ /dev/null @@ -1,8 +0,0 @@ -BASE_DATA_DIR=${BASE_DATA_DIR:-"path/to/basedata"} -BASE_CKPT_DIR=${BASE_CKPT_DIR:-"path/to/pretrained_checkpoint"} - -python apdepth_train_s1.py --config config/apdepth_train_s1.yaml \ - --base_data_dir $BASE_DATA_DIR \ - --base_ckpt_dir $BASE_CKPT_DIR \ - --output_dir output/stage1 \ - --no_wandb diff --git a/script/audit_far_supervision.py b/script/audit_far_supervision.py new file mode 100644 index 0000000..8369cfd --- /dev/null +++ b/script/audit_far_supervision.py @@ -0,0 +1,64 @@ +"""Inspect real training targets without loading SD2 or DA2 checkpoints.""" + +import argparse +import json +from pathlib import Path + +import numpy as np +import torch +from PIL import Image + +from src.dataset import DatasetMode, get_dataset +from src.util.config_util import recursive_load_config +from src.util.depth_transform import get_depth_normalizer +from src.util.far_supervision import prepare_far_supervision + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--config", default="config/train_sky_finetune.yaml") + parser.add_argument("--base_data_dir", required=True) + parser.add_argument("--samples", type=int, default=20) + parser.add_argument("--output_dir", default="output/far_audit") + args = parser.parse_args() + if args.samples < 1: + parser.error("--samples must be positive") + cfg = recursive_load_config(args.config) + configs = cfg.dataset.train.dataset_list if cfg.dataset.train.name == "mixed" else [cfg.dataset.train] + vkitti = [c for c in configs if c.name == "vkitti"] + if not vkitti: + parser.error("No VKITTI dataset found in training config") + destination = Path(args.output_dir) + destination.mkdir(parents=True, exist_ok=True) + records = [] + for dataset_index, data_cfg in enumerate(vkitti): + dataset = get_dataset(data_cfg, args.base_data_dir, DatasetMode.TRAIN, + depth_transform=get_depth_normalizer(cfg.depth_normalization)) + if not len(dataset): + raise RuntimeError("VKITTI training split is empty") + indices = np.linspace(0, len(dataset) - 1, min(args.samples, len(dataset)), dtype=int) + for index in indices: + item = dataset[int(index)] + far = item["known_far_mask"] + target, _, added, _ = prepare_far_supervision( + item[cfg.gt_depth_type][None], item[cfg.gt_mask_type][None], far[None]) + record = { + "rgb": item["rgb_relative_path"], + "known_far_pixel_ratio": far.float().mean().item(), + "added_latent_ratio": added.float().mean().item(), + "saturated_pixel_ratio": (item["depth_raw_linear"] >= 655.3).float().mean().item(), + } + records.append(record) + prefix = destination / f"{dataset_index}_{index:06d}" + Image.fromarray(item["rgb_int"].permute(1, 2, 0).numpy().astype(np.uint8)).save(f"{prefix}_rgb.png") + Image.fromarray(far[0].numpy().astype(np.uint8) * 255).save(f"{prefix}_far_mask.png") + gray = ((target[0, 0].clamp(-1, 1) + 1) * 127.5).numpy().astype(np.uint8) + Image.fromarray(gray).save(f"{prefix}_target_far_white.png") + print(json.dumps(record)) + (destination / "coverage.json").write_text(json.dumps(records, indent=2), encoding="utf-8") + if not any(r["added_latent_ratio"] > 0 for r in records): + raise RuntimeError("No far training support found. Inspect native depth encoding before training.") + + +if __name__ == "__main__": + main() diff --git a/src/dataset/base_depth_dataset.py b/src/dataset/base_depth_dataset.py index 170b763..d1dff90 100755 --- a/src/dataset/base_depth_dataset.py +++ b/src/dataset/base_depth_dataset.py @@ -153,6 +153,10 @@ def _get_data_item(self, index): rasters["valid_mask_filled"] = self._get_valid_mask( rasters["depth_filled_linear"] ).clone() + # Measured far depth is separate from missing GT and evaluation masks. + rasters["known_far_mask"] = self._get_known_far_mask( + rasters["depth_raw_linear"] + ) other = {"index": index, "rgb_relative_path": rgb_rel_path} @@ -229,6 +233,10 @@ def _get_valid_mask(self, depth: torch.Tensor): ).bool() return valid_mask + def _get_known_far_mask(self, depth: torch.Tensor): + # Only datasets with a known native far-depth encoding opt in. + return torch.zeros_like(depth, dtype=torch.bool) + def _training_preprocess(self, rasters): # Augmentation if self.augm_args is not None: diff --git a/src/dataset/vkitti_dataset.py b/src/dataset/vkitti_dataset.py index ceb7903..eb4222a 100755 --- a/src/dataset/vkitti_dataset.py +++ b/src/dataset/vkitti_dataset.py @@ -58,6 +58,12 @@ def _read_depth_file(self, rel_path): depth_decoded = depth_in / 100.0 return depth_decoded + def _get_known_far_mask(self, depth: torch.Tensor): + # PNG16 centimeters: known >=80 m depth (up to 655.35 m) is excluded + # from metric evaluation, but supplies the clipped +1 training target. + # This includes far buildings as well as sky; zero is still unknown. + return torch.isfinite(depth) & (depth >= self.max_depth) & (depth <= 655.35) + def _load_rgb_data(self, rgb_rel_path): rgb_data = super()._load_rgb_data(rgb_rel_path) if self.kitti_bm_crop: diff --git a/src/trainer/__init__.py b/src/trainer/__init__.py index fe2a276..12bef3c 100755 --- a/src/trainer/__init__.py +++ b/src/trainer/__init__.py @@ -2,12 +2,10 @@ # Last modified: 2024-05-17 from .apdepth_trainer import ApDepthTrainer -from .apdepth_trainer_s1 import ApDepthTrainerS1 trainer_cls_name_dict = { "ApDepthTrainer": ApDepthTrainer, - "ApDepthTrainerS1": ApDepthTrainerS1, } diff --git a/src/trainer/apdepth_trainer.py b/src/trainer/apdepth_trainer.py index 88f1131..a84eba5 100644 --- a/src/trainer/apdepth_trainer.py +++ b/src/trainer/apdepth_trainer.py @@ -47,6 +47,7 @@ from src.util.data_loader import skip_first_batches from src.util.logging_util import tb_logger, eval_dic_to_text from src.util.loss import get_loss +from src.util.far_supervision import masked_loss, prepare_far_supervision from src.util.lr_scheduler import IterExponential from src.util.metric import MetricTracker from src.util.alignment import align_depth_least_square @@ -113,10 +114,23 @@ def __init__( self.loss = get_loss(loss_name=self.cfg.loss.name, **self.cfg.loss.kwargs) self.l1_loss = get_loss(loss_name=self.cfg.l1_loss.name, **self.cfg.l1_loss.kwargs) self.latent_freq_loss = get_loss(loss_name=self.cfg.latent_freq_loss.name, ** self.cfg.latent_freq_loss.kwargs) + + far_cfg = self.cfg.get("far_depth_supervision", {}) + self.far_enabled = bool(far_cfg.get("enabled", False)) + self.far_weight = float(far_cfg.get("weight", 0.5)) + if self.far_enabled: + if not np.isfinite(self.far_weight) or self.far_weight <= 0: + raise ValueError("far_depth_supervision.weight must be finite and positive") + norm = self.cfg.depth_normalization + if not (norm.type == "scale_shift_depth" and norm.clip + and norm.norm_min == -1.0 and norm.norm_max == 1.0): + raise ValueError("Far supervision requires clipped linear depth in [-1, 1]") # Eval metrics self.metric_funcs = [getattr(metric, _met) for _met in cfg.eval.eval_metrics] - self.train_metrics = MetricTracker(*["loss", "latent_freq_loss"]) + self.train_metrics = MetricTracker( + "loss", "latent_freq_loss", "far_loss", "far_pixel_ratio", "far_latent_ratio" + ) self.val_metrics = MetricTracker(*[m.__name__ for m in self.metric_funcs]) # main metric for best checkpoint saving @@ -196,11 +210,18 @@ def train(self, t_end=None): if self.gt_mask_type is not None: valid_mask_for_latent = batch[self.gt_mask_type].to(device) - invalid_mask = ~valid_mask_for_latent - valid_mask_down = ~torch.max_pool2d( - invalid_mask.float(), 8, 8 - ).bool() + if self.far_enabled: + far_mask = batch["known_far_mask"].to(device).bool() + far_mask = far_mask & ~valid_mask_for_latent + else: + far_mask = torch.zeros_like(valid_mask_for_latent, dtype=torch.bool) + depth_gt_for_latent, valid_mask_down, far_mask_down, _ = ( + prepare_far_supervision( + depth_gt_for_latent, valid_mask_for_latent, far_mask + ) + ) valid_mask_down = valid_mask_down.repeat((1, 4, 1, 1)) + far_mask_down = far_mask_down.repeat((1, 4, 1, 1)) else: raise NotImplementedError @@ -238,45 +259,15 @@ def train(self, t_end=None): torch.cuda.empty_cache() - loss = 0.0 - - if self.effective_iter <= 20000: - # Masked latent loss (MSE loss) - if self.gt_mask_type is not None: - latent_loss = self.loss( - depth_pred[valid_mask_down].float(), - gt_depth_latent[valid_mask_down].float(), - ) - pixel_loss = self.l1_loss( - depth[valid_mask_for_latent].float(), - depth_gt_for_loss[valid_mask_for_latent].float(), - ) - else: - latent_loss = self.loss(depth.float(), depth_gt_for_loss.float()) - pixel_loss = self.l1_loss(depth.float(), depth_gt_for_latent.float()) - - # update loss - loss = latent_loss.mean() + pixel_loss.mean() - - self.train_metrics.update("loss", loss.item()) - else: - # FFT latent loss (FFT loss) - if self.gt_mask_type is not None: - freq_loss = self.latent_freq_loss( - depth_pred.float(), - gt_depth_latent.float(), - valid_mask_down - ) - else: - freq_loss = self.latent_freq_loss( - depth_pred.float(), - gt_depth_latent.float(), - mask=None - ) - - self.train_metrics.update("latent_freq_loss", freq_loss.item()) - lambda_freq = self.cfg.latent_freq_loss.get('lambda', 1.0) - loss = lambda_freq * freq_loss + loss, freq_loss, far_loss = self._compute_depth_loss( + depth_pred, gt_depth_latent, depth, depth_gt_for_loss, + valid_mask_down, valid_mask_for_latent, far_mask_down, far_mask, + ) + self.train_metrics.update("loss", loss.item()) + self.train_metrics.update("latent_freq_loss", freq_loss.item()) + self.train_metrics.update("far_loss", far_loss.item()) + self.train_metrics.update("far_pixel_ratio", far_mask.float().mean().item()) + self.train_metrics.update("far_latent_ratio", far_mask_down.float().mean().item()) loss = loss / self.gradient_accumulation_steps loss.backward() @@ -299,7 +290,7 @@ def train(self, t_end=None): # Log to tensorboard accumulated_metrics = self.train_metrics.result() - loss_to_log = accumulated_metrics.get("latent_freq_loss", 0.0) if self.effective_iter > 20000 else accumulated_metrics.get("loss", 0.0) # Only ACMer would do this + loss_to_log = accumulated_metrics["loss"] tb_logger.log_dic( { @@ -346,6 +337,28 @@ def train(self, t_end=None): # Epoch end self.n_batch_in_epoch = 0 + def _compute_depth_loss( + self, pred_latent, target_latent, pred_depth, target_depth, + valid_latent, valid_pixel, far_latent, far_pixel, + ): + # Preserve the original valid-GT objective and its normalization. + freq_loss = pred_latent.new_zeros(()) + if self.effective_iter <= 20000: + loss = masked_loss(self.loss, pred_latent.float(), target_latent.float(), valid_latent) + loss = loss + masked_loss(self.l1_loss, pred_depth.float(), target_depth.float(), valid_pixel) + else: + freq_loss = self.latent_freq_loss(pred_latent.float(), target_latent.float(), valid_latent) + loss = self.cfg.latent_freq_loss.get("lambda", 1.0) * freq_loss + + far_loss = pred_latent.new_zeros(()) + if self.far_enabled: + # Maintain an absolute far-depth target even during FFT refinement: + # spectral magnitude alone cannot distinguish opposite-sign targets. + far_loss = masked_loss(self.loss, pred_latent.float(), target_latent.float(), far_latent) + far_loss = far_loss + masked_loss(self.l1_loss, pred_depth.float(), target_depth.float(), far_pixel) + loss = loss + self.far_weight * far_loss + return loss, freq_loss, far_loss + def encode_depth(self, depth_in): # stack depth into 3-channel stacked = self.stack_depth_images(depth_in) diff --git a/src/trainer/apdepth_trainer_s1.py b/src/trainer/apdepth_trainer_s1.py deleted file mode 100644 index 5f6549a..0000000 --- a/src/trainer/apdepth_trainer_s1.py +++ /dev/null @@ -1,335 +0,0 @@ -# Last modified: 2025-07-13 -# -# Copyright 2025 Jiawei Wang, SJZU. All rights reserved. -# -# This file has been modified from the original version. -# Original copyright (c) 2023 Bingxin Ke, ETH Zurich. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -------------------------------------------------------------------------- -# If you find this code useful, we kindly ask you to cite our paper in your work. -# Please find bibtex at: https://github.com/Haruko386/ApDepth#-citation -# More information about the method can be found at https://haruko386.github.io/apdepth -# -------------------------------------------------------------------------- - -import argparse -import logging -import os -import shutil -from datetime import datetime, timedelta -from typing import List - -import torch -from omegaconf import OmegaConf -from torch.utils.data import ConcatDataset, DataLoader -from tqdm import tqdm - - -from apdepth import ApDepthPipeline -from diffusers import UNet2DConditionModel -from src.dataset import BaseDepthDataset, DatasetMode, get_dataset -from src.dataset.mixed_sampler import MixedBatchSampler -from src.trainer import get_trainer_cls -from src.util.config_util import ( - find_value_in_omegaconf, - recursive_load_config, -) -from src.util.depth_transform import ( - DepthNormalizerBase, - get_depth_normalizer, -) -from src.util.logging_util import ( - config_logging, - init_wandb, - load_wandb_job_id, - log_slurm_job_id, - save_wandb_job_id, - tb_logger, -) -from src.util.slurm_util import get_local_scratch_dir, is_on_slurm - -if "__main__" == __name__: - t_start = datetime.now() - print(f"start at {t_start}") - - # -------------------- Arguments -------------------- - parser = argparse.ArgumentParser(description="Train your cute model!") - parser.add_argument( - "--config", - type=str, - default="config/apdepth_train_s1.yaml", - help="Path to config file.", - ) - parser.add_argument( - "--resume_run", - action="store", - default=None, - help="Path of checkpoint to be resumed. If given, will ignore --config, and checkpoint in the config", - ) - parser.add_argument( - "--output_dir", type=str, default=None, help="directory to save checkpoints" - ) - parser.add_argument("--no_cuda", action="store_true", help="Do not use cuda.") - parser.add_argument( - "--exit_after", - type=int, - default=-1, - help="Save checkpoint and exit after X minutes.", - ) - parser.add_argument("--no_wandb", action="store_true", help="run without wandb") - parser.add_argument( - "--do_not_copy_data", - action="store_true", - help="On Slurm cluster, do not copy data to local scratch", - ) - parser.add_argument( - "--base_data_dir", type=str, default="/root/Dataset", help="directory of training data" - ) - parser.add_argument( - "--base_ckpt_dir", - type=str, - default="/root/ApDepth/pretrained_checkpoint", - help="directory of pretrained checkpoint", - ) - parser.add_argument( - "--add_datetime_prefix", - action="store_true", - help="Add datetime to the output folder name", - ) - - args = parser.parse_args() - print("\n=== Arguments Summary ===") - max_len = max(len(arg) for arg in vars(args)) - for arg in vars(args): - print(f"{arg.ljust(max_len)} : {getattr(args, arg)}") - resume_run = args.resume_run - output_dir = args.output_dir - base_data_dir = ( - args.base_data_dir - if args.base_data_dir is not None - else os.environ["BASE_DATA_DIR"] - ) - base_ckpt_dir = ( - args.base_ckpt_dir - if args.base_ckpt_dir is not None - else os.environ["BASE_CKPT_DIR"] - ) - - # -------------------- Initialization -------------------- - # Resume previous run - if resume_run is not None: - print(f"Resume run: {resume_run}") - out_dir_run = os.path.dirname(os.path.dirname(resume_run)) - job_name = os.path.basename(out_dir_run) - # Resume config file - cfg = OmegaConf.load(os.path.join(out_dir_run, "config.yaml")) - else: - # Run from start - cfg = recursive_load_config(args.config) - - # Output dir - if output_dir is not None: - out_dir_run = output_dir - job_name = os.path.basename(args.config) - else: - job_name = os.path.basename(args.config).split(".")[0] - out_dir_run = os.path.join("./output", job_name) - os.makedirs(out_dir_run, exist_ok=False) - - cfg_data = cfg.dataset - - # Other directories - out_dir_ckpt = os.path.join(out_dir_run, "checkpoint") - if not os.path.exists(out_dir_ckpt): - os.makedirs(out_dir_ckpt) - out_dir_tb = os.path.join(out_dir_run, "tensorboard") - if not os.path.exists(out_dir_tb): - os.makedirs(out_dir_tb) - out_dir_eval = os.path.join(out_dir_run, "evaluation") - if not os.path.exists(out_dir_eval): - os.makedirs(out_dir_eval) - out_dir_vis = os.path.join(out_dir_run, "visualization") - if not os.path.exists(out_dir_vis): - os.makedirs(out_dir_vis) - - # -------------------- Logging settings -------------------- - config_logging(cfg.logging, out_dir=out_dir_run) - logging.debug(f"config: {cfg}") - - # Initialize wandb - # if not args.no_wandb: - # if resume_run is not None: - # wandb_id = load_wandb_job_id(out_dir_run) - # wandb_cfg_dic = { - # "id": wandb_id, - # "resume": "must", - # **cfg.wandb, - # } - # else: - # wandb_cfg_dic = { - # "config": dict(cfg), - # "name": job_name, - # "mode": "online", - # **cfg.wandb, - # } - # wandb_cfg_dic.update({"dir": out_dir_run}) - # wandb_run = init_wandb(enable=True, **wandb_cfg_dic) - # save_wandb_job_id(wandb_run, out_dir_run) - # else: - # init_wandb(enable=False) - - # Tensorboard (should be initialized after wandb) - tb_logger.set_dir(out_dir_tb) - - log_slurm_job_id(step=0) - - # -------------------- Device -------------------- - cuda_avail = torch.cuda.is_available() and not args.no_cuda - device = torch.device("cuda" if cuda_avail else "cpu") - logging.info(f"device = {device}") - - # -------------------- Snapshot of code and config -------------------- - if resume_run is None: - _output_path = os.path.join(out_dir_run, "config.yaml") - with open(_output_path, "w+") as f: - OmegaConf.save(config=cfg, f=f) - logging.info(f"Config saved to {_output_path}") - # Copy and tar code on the first run - _temp_code_dir = os.path.join(out_dir_run, "code_tar") - _code_snapshot_path = os.path.join(out_dir_run, "code_snapshot.tar") - os.system( - f"rsync --relative -arhvz --quiet --filter=':- .gitignore' --exclude '.git' . '{_temp_code_dir}'" - ) - os.system(f"tar -cf {_code_snapshot_path} {_temp_code_dir}") - os.system(f"rm -rf {_temp_code_dir}") - logging.info(f"Code snapshot saved to: {_code_snapshot_path}") - - # -------------------- Copy data to local scratch (Slurm) -------------------- - if is_on_slurm() and (not args.do_not_copy_data): - # local scratch dir - original_data_dir = base_data_dir - base_data_dir = os.path.join(get_local_scratch_dir(), "ApDepth_data") - # copy data - required_data_list = find_value_in_omegaconf("dir", cfg_data) - required_data_list = list(set(required_data_list)) - logging.info(f"Required_data_list: {required_data_list}") - for d in tqdm(required_data_list, desc="Copy data to local scratch"): - ori_dir = os.path.join(original_data_dir, d) - dst_dir = os.path.join(base_data_dir, d) - os.makedirs(os.path.dirname(dst_dir), exist_ok=True) - if os.path.isfile(ori_dir): - shutil.copyfile(ori_dir, dst_dir) - elif os.path.isdir(ori_dir): - shutil.copytree(ori_dir, dst_dir) - logging.info(f"Data copied to: {base_data_dir}") - - # -------------------- Gradient accumulation steps -------------------- - eff_bs = cfg.dataloader.effective_batch_size - accumulation_steps = eff_bs / cfg.dataloader.max_train_batch_size - assert int(accumulation_steps) == accumulation_steps - accumulation_steps = int(accumulation_steps) - - logging.info( - f"Effective batch size: {eff_bs}, accumulation steps: {accumulation_steps}" - ) - - # -------------------- Data -------------------- - loader_seed = cfg.dataloader.seed - if loader_seed is None: - loader_generator = None - else: - loader_generator = torch.Generator().manual_seed(loader_seed) - - # Training dataset - depth_transform: DepthNormalizerBase = get_depth_normalizer( - cfg_normalizer=cfg.depth_normalization - ) - train_dataset: BaseDepthDataset = get_dataset( - cfg_data.train, - base_data_dir=base_data_dir, - mode=DatasetMode.TRAIN, - augmentation_args=cfg.augmentation, - depth_transform=depth_transform, - ) - logging.debug("Augmentation: ", cfg.augmentation) - if "mixed" == cfg_data.train.name: - dataset_ls = train_dataset - assert len(cfg_data.train.prob_ls) == len( - dataset_ls - ), "Lengths don't match: `prob_ls` and `dataset_list`" - concat_dataset = ConcatDataset(dataset_ls) - mixed_sampler = MixedBatchSampler( - src_dataset_ls=dataset_ls, - batch_size=cfg.dataloader.max_train_batch_size, - drop_last=True, - prob=cfg_data.train.prob_ls, - shuffle=True, - generator=loader_generator, - ) - train_loader = DataLoader( - concat_dataset, - batch_sampler=mixed_sampler, - num_workers=cfg.dataloader.num_workers, - ) - else: - train_loader = DataLoader( - dataset=train_dataset, - batch_size=cfg.dataloader.max_train_batch_size, - num_workers=cfg.dataloader.num_workers, - shuffle=True, - generator=loader_generator, - ) - - # -------------------- Model -------------------- - _pipeline_kwargs = cfg.pipeline.kwargs if cfg.pipeline.kwargs is not None else {} - model = ApDepthPipeline.from_pretrained( - os.path.join(base_ckpt_dir, cfg.model.pretrained_path), **_pipeline_kwargs - ) - unet = UNet2DConditionModel.from_pretrained( - os.path.join(base_ckpt_dir, cfg.model.pretrained_path, "unet") - ) - model.unet = unet - - # -------------------- Trainer -------------------- - # Exit time - if args.exit_after > 0: - t_end = t_start + timedelta(minutes=args.exit_after) - logging.info(f"Will exit at {t_end}") - else: - t_end = None - - trainer_cls = get_trainer_cls(cfg.trainer.name) - logging.debug(f"Trainer: {trainer_cls}") - trainer = trainer_cls( - cfg=cfg, - model=model, - train_dataloader=train_loader, - device=device, - base_ckpt_dir=base_ckpt_dir, - out_dir_ckpt=out_dir_ckpt, - out_dir_eval=out_dir_eval, - out_dir_vis=out_dir_vis, - accumulation_steps=accumulation_steps, - ) - - # -------------------- Checkpoint -------------------- - if resume_run is not None: - trainer.load_checkpoint( - resume_run, load_trainer_state=True, resume_lr_scheduler=True - ) - - # -------------------- Training & Evaluation Loop -------------------- - try: - trainer.train(t_end=t_end) - except Exception as e: - logging.exception(e) diff --git a/src/util/build_mlp.py b/src/util/build_mlp.py deleted file mode 100644 index eba220c..0000000 --- a/src/util/build_mlp.py +++ /dev/null @@ -1,39 +0,0 @@ -import torch.nn as nn - - -class ConvFeatureAdapter(nn.Module): - def __init__(self, in_channels, out_channels, hidden_channels=None): - super().__init__() - if hidden_channels is None: - hidden_channels = out_channels - - self.net = nn.Sequential( - nn.Conv2d(in_channels, hidden_channels, kernel_size=1), - nn.GroupNorm(num_groups=32, num_channels=hidden_channels), - nn.SiLU(), - nn.Conv2d(hidden_channels, hidden_channels, kernel_size=3, padding=1), - nn.GroupNorm(num_groups=32, num_channels=hidden_channels), - nn.SiLU(), - nn.Conv2d(hidden_channels, out_channels, kernel_size=1), - ) - - def forward(self, x): - return self.net(x) - - -def build_conv_adapter(in_channels=1280, out_channels=1536, hidden_channels=1536): - return ConvFeatureAdapter( - in_channels=in_channels, - out_channels=out_channels, - hidden_channels=hidden_channels, - ) - - -def build_mlp_(hidden_size=640, projector_dim=1024, z_dim=768): - return nn.Sequential( - nn.Linear(hidden_size, projector_dim), - nn.SiLU(), - nn.Linear(projector_dim, projector_dim), - nn.SiLU(), - nn.Linear(projector_dim, z_dim), - ) diff --git a/src/util/far_supervision.py b/src/util/far_supervision.py new file mode 100644 index 0000000..a945806 --- /dev/null +++ b/src/util/far_supervision.py @@ -0,0 +1,35 @@ +"""Measured far-depth supervision, ported from ApDepth-G to single-step ApDepth.""" + +import torch +import torch.nn.functional as F + + +def prepare_far_supervision(depth, valid_mask, far_mask, far_value=1.0, scale=8): + """Complete targets before VAE encoding; keep metric GT masks unchanged. + + Far masks must come from native dataset depth, never missing GT or DA2. + Normalization quantiles must already have used only metric-valid GT. + A latent cell is supported only when every pixel is valid or known far, + including mixed foreground/far cells. Unknown pixels remain excluded. + """ + if depth.shape != valid_mask.shape or depth.shape != far_mask.shape: + raise ValueError("Depth, valid mask and far mask must have identical shapes") + valid_mask, far_mask = valid_mask.bool(), far_mask.bool() + far_mask = far_mask & ~valid_mask + target = torch.where(far_mask, depth.new_tensor(far_value), depth) + + def all_pixels(mask): + return ~F.max_pool2d((~mask).float(), scale, scale).bool() + + valid = all_pixels(valid_mask) + support = all_pixels(valid_mask | far_mask) + added = support & ~valid + return target, valid, added, support + + +def masked_loss(loss_fn, prediction, target, mask): + """Apply a reconstruction criterion with a differentiable zero if empty.""" + selected = prediction.masked_select(mask) + if selected.numel() == 0: + return selected.sum() + return loss_fn(selected, target.masked_select(mask)).mean() diff --git a/train.py b/train.py index 1a5d68f..7370e7d 100755 --- a/train.py +++ b/train.py @@ -63,7 +63,7 @@ parser.add_argument( "--config", type=str, - default="ApDepth/config/train_apdepth.yaml", + default="config/train_apdepth.yaml", help="Path to config file.", ) parser.add_argument( @@ -75,6 +75,10 @@ parser.add_argument( "--output_dir", type=str, default=None, help="directory to save checkpoints" ) + parser.add_argument( + "--init_checkpoint", type=str, default=None, + help="Initialize UNet from a safetensors checkpoint directory with new config, optimizer and steps.", + ) parser.add_argument("--no_cuda", action="store_true", help="Do not use cuda.") parser.add_argument( "--exit_after", @@ -89,12 +93,12 @@ help="On Slurm cluster, do not copy data to local scratch", ) parser.add_argument( - "--base_data_dir", type=str, default="/root/Dataset", help="directory of training data", + "--base_data_dir", type=str, default=None, help="directory of training data (or BASE_DATA_DIR)", ) parser.add_argument( "--base_ckpt_dir", type=str, - default="./pretrained_checkpoint", + default=None, help="directory of pretrained checkpoint", ) parser.add_argument( @@ -104,6 +108,17 @@ ) args = parser.parse_args() + if args.resume_run is not None and args.init_checkpoint is not None: + parser.error("--resume_run and --init_checkpoint are mutually exclusive") + if args.init_checkpoint is not None: + init_unet_path = os.path.join( + args.init_checkpoint, "unet", "diffusion_pytorch_model.safetensors" + ) + if not os.path.isfile(init_unet_path): + parser.error( + "--init_checkpoint requires a checkpoint directory containing " + "unet/diffusion_pytorch_model.safetensors; .bin checkpoints are not accepted." + ) print("\n=== Arguments Summary ===") max_len = max(len(arg) for arg in vars(args)) for arg in vars(args): @@ -118,7 +133,7 @@ base_ckpt_dir = ( args.base_ckpt_dir if args.base_ckpt_dir is not None - else os.environ["BASE_CKPT_DIR"] + else os.environ.get("BASE_CKPT_DIR", "./pretrained_checkpoint") ) # -------------------- Initialization -------------------- @@ -359,9 +374,12 @@ trainer.load_checkpoint( resume_run, load_trainer_state=True, resume_lr_scheduler=True ) + elif args.init_checkpoint is not None: + trainer.load_checkpoint(args.init_checkpoint, load_trainer_state=False) # -------------------- Training & Evaluation Loop -------------------- try: trainer.train(t_end=t_end) except Exception as e: logging.exception(e) + raise