diff --git a/README.md b/README.md index a0f6de6..0b392b7 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,24 @@ # XFusion Deep learning-based spatiotemporal fusion for high-fidelity ultra-high-speed x-ray radiography A model to reconstruct high quality x-ray images by combining the high spatial resolution of high-speed camera and high temporal resolution of ultra-high-speed camera image sequences. +Models and methods for spatial and temporal calibration are also included. ## Prerequisites -This implementation is based on the [BasicSR toolbox](https://github.com/XPixelGroup/BasicSR). Data for model pre-training are collected from the [REDS dataset](https://seungjunnah.github.io/Datasets/reds). +This implementation is based on the [BasicSR toolbox](https://github.com/XPixelGroup/BasicSR) and the [RAFT model](https://github.com/princeton-vl/RAFT). Data for model pre-training are collected from the [REDS dataset](https://seungjunnah.github.io/Datasets/reds). ## Usage ### Package description -Currently, xfusion supports 2 model familities for high-quality xray image sequence reconstruction- the EDVR and Swin vision transformer, respectively. +Currently, xfusion supports 2 model familities for high-quality xray image sequence reconstruction- the EDVR and Swin vision transformer, respectively. +In addition, xfusion is functional with x-ray image sequences from both the actual (data type: actual) and virtual (data type: virtual) experiments. +* Actual experiments utilize the dual-detector setup for data acquisition and virtual experiments synthetize dual-detector data from single-camera experiments. + +With the actual dual-detector experimental data, xfusion first conducts the spatial and temporal calibration of the two image sequences. +* Temporal calibration is specific to the choice of the Phantom TMX7510 (Vision Research Inc., USA) camera and Shimadzu HPV-X2 (Shimadzu Corp., Japan) camera. +* Spatial calibration is based on the use of an electroformed copper grid (G400, Gilder Grids Ltd., UK). + +Then at inference time, xfusion traverses the dual-camera image sequences in two ways: continuous and double-interval. +* Under the continuous mode, high-quality xray images corresponding to each Shimadzu camera image are reconstructed. +* Under the double-interval mode, independent high-quality xray images corresponding to each Phantom camera image are reconstructed to evaluate performance metrics. ### Package installation Navigate to the project root directory and then run @@ -19,7 +30,7 @@ to install the package to the selected virtual environment. ### Initialization Run
-xfusion init --model_type [EDVRModel or SwinIRModel]
+xfusion init --model_type [EDVRModel or SwinIRModel] --data_type [virtual or actual]
 
After initialization, a configuration file "xfusion.conf" will be generated in the home directory. This configuration file will be updated automatically within the workflow of the xfusion package. @@ -38,9 +49,27 @@ xfusion convert --dir-lo-convert [directory/to/low resolution/RGB/training im ### Training -Run +To train the model with data from the virtual experiments, run +
+xfusion train --dir-lo-train [directory/to/low resolution/gray-scale/training image] --dir-hi-train [directory/to/high resolution/gray-scale/training image] --dir-lo-val [directory/to/low resolution/gray-scale/validation image] --dir-hi-val [directory/ti/high resolution/gray-scale/validation image] --opt directory/to/training setting/yaml file --path-train-meta-info-file [directory/to/training image/meta data] --path-val-meta-info-file [directory/to/validation image/meta data] --pretrain_network_g [path/to/model weight/file/for/model initialization]
+
+To train the model with data from the actual experiments, run +
+xfusion train --dir-lo-train [directory/to/low resolution/gray-scale/paired training image] --dir-hi-train [directory/to/high resolution/gray-scale/paired training image] --dir-lo-val [directory/to/low resolution/gray-scale/paired validation image] --dir-hi-val [directory/ti/high resolution/gray-scale/paired validation image] --dataroot_context [directory/to/low resolution/gray-scale/training image full] --opt directory/to/training setting/yaml file --path-train-meta-info-file [directory/to/training image/meta data] --path-val-meta-info-file [directory/to/validation image/meta data] --meta_info_context [directory/to/paired/high/lo/resolution/training image/index map] --pretrain_network_g [path/to/model weight/file/for/model initialization]
+
+ +### Train calibration model +To train the calibration model with data from the virtual experiments (pretrain), run +
+xfusion train_calibration --image_root [root/directory/to/training images] --meta_info_file [directory/to/training image/meta data]
+--evaluator_weights [path/to/spatiotemporal/fusion/model weight/file/for/training loss]
+--flow_weights None --full_weights None
+
+To train the calibration model with data from the actual experiments (fine tune), run
-xfusion train --dir-lo-train [directory/to/low resolution/gray-scale/training image] --dir-hi-train [directory/to/high resolution/gray-scale/training image] --dir-lo-val [directory/to/low resolution/gray-scale/validation image] --dir-hi-val [directory/ti/high resolution/gray-scale/validation image] --opt directory/to/training setting/yaml file --path-train-meta-info-file [directory/to/training image/meta data] --path-val-meta-info-file [directory/to/validation image/meta data] --pretrain_network_g [directory/to/model weight/file/for/model initialization]
+xfusion train_calibration --image_root [root/directory/to/training images] --meta_info_file [directory/to/training image/meta data]
+--evaluator_weights [path/to/spatiotemporal/fusion/model weight/file/for/training loss]
+--full_weights [path/to/pretrained/calibration/model weight/file/for/model initialization]
 
### Test data @@ -49,9 +78,28 @@ To download test data, run xfusion download --dir-inf [tomobank/link/address/of/test/dataset] --out-dir-inf [directory/to/testing image] -### Inference +### Spatial calibration Run
-xfusion inference --opt directory/to/testing dataset/setting/yaml file --arch-opt directory/to/training setting/yaml file --model_file [path/to/model file] --machine tomo or polaris
+xfusion calibrate --cal_model_file [path/to/calibration/model file] --cal_dir [root/directory/to/calibration/experiment/data]
+--cal_id [calibration experiment subdirectory name]
+
+ +### Inference +For virtual experiment data, run +
+xfusion inference --opt directory/to/testing dataset/setting/yaml file --arch-opt directory/to/model architecture/yaml file --img-class dataset1 or dataset2 --model_file [path/to/model file] --machine tomo or polaris
 
-Currently to work for EDVRModel in the single-process mode and SwinIRModel in the multi-process mode. +For actual experiment data, run +
+xfusion inference --arch-opt directory/to/model architecture/yaml file --img-class name of high-speed experiment --model_file [path/to/model file] --traverse_mode double or continuous --tfm_file [path/to/rigid/body/spatial/transformation/file] --meta_file_scale [path/to/nonuniform/scaling/parameters] --input_dir [root/directory/to/high/speed/experiment/data] --case_list experiment subdirectory names
+
+ +### Suggested usage +* For virtual experiment data, currently xfusion works for EDVRModel in the single-process mode and SwinIRModel in the multi-process mode. +When working on the virtual experiment data, first initialize the software, and then do the inference. + +* For actual experiment data, xfusion works for both models in a unified way. +Do an independent initialization every time the model (EDVR-STF or SWIN-XVR) or the type of data (virtual or actual) changes. +When working on the actual experiment data, first initialize the software, then do the spatial calibration, and last do the inference. +Multiple experiments sharing the same spatial calibration can be grouped to run the model inference. diff --git a/xfusion/config.py b/xfusion/config.py index ea144f0..d571050 100644 --- a/xfusion/config.py +++ b/xfusion/config.py @@ -20,11 +20,19 @@ 'parse_known_args', 'write'] def list_of_ints(arg): + if ',' in arg: + return list(map(int,arg.split(','))) + else: + if arg.lower() != 'none': + return [int(arg)] + else: + return None + +def list_of_strs(arg): if ',' in arg: return list(arg.split(',')) else: return [arg] - CONFIG_FILE_NAME = os.path.join(str(pathlib.Path.home()), 'xfusion.conf') XFUSION_HOME = os.path.join(str(pathlib.Path.home()), 'xfusion') XFUSION_TRAIN_HOME = os.path.join(str(XFUSION_HOME), 'train') @@ -188,10 +196,49 @@ def list_of_ints(arg): 'help': 'Relative frame separation in the hr images after and before the augmentation' }, 'poisson_noise_b0_exp': { - 'default': [], + 'default': 'none', 'type': list_of_ints, 'help': 'upper and lower bounds of the blank scan factor in log scale' }, + # newly added ddp configs + 'iter_div_factor': { + 'default': 10, + 'type': int, + 'help': "Divisor of the total number of iterations" + }, + 'batch_size': { + 'default': 1, + 'type': int, + 'help': "Number of samples in a batch" + }, + 'num_workers': { + 'default': 3, + 'type': int, + 'help': "Number of workers for data loader"}, + 'warmup_iter': { + 'default': -1, + 'type': int, + 'help': "Number of iterations for training warm up"}, + 'dataset_enlarge_ratio': { + 'default': 10.0, + 'type': float, + 'help': "Multiplier of the original number of training samples"}, + 'sub_dataset': { + 'default': False, + 'type': lambda x: bool(strtobool(x)), + 'help': "When set the original training data are randomly sub-sampled for training"}, + 'sub_dataset_frac': { + 'default': 1.0, + 'type': float, + 'help': "Fraction of the original training data set size to sub-sample"}, + 'single_epoch_ok': { + 'default': False, + 'type': lambda x: bool(strtobool(x)), + 'help': "If to only train data for one epoch"}, + 'profiler_iter_num': { + 'default': 3, + 'type': int, + 'help': "Number of subsequent iterations to track computation performance for HTA"}, } MODEL_TYPES = {'train':OrderedDict(),'inference':OrderedDict()} @@ -225,50 +272,17 @@ def list_of_ints(arg): 'type': int, 'help': "Size of convolution window if used in the vit residual connection"}, # newly added ddp configs - 'iter_div_factor': { - 'default': 10, - 'type': int, - 'help': "Divisor of the total number of iterations"}, + 'num_frame': { 'default': 3, 'type': int, 'help': "Number of input low resolution frames to the model"}, - 'batch_size': { - 'default': 1, - 'type': int, - 'help': "Number of samples in a batch"}, + 'initial_lr': { - 'default': 2e-4, + 'default': 5e-6, 'type': float, 'help': "Initial learning rate"}, - 'num_workers': { - 'default': 3, - 'type': int, - 'help': "Number of workers for data loader"}, - 'warmup_iter': { - 'default': -1, - 'type': int, - 'help': "Number of iterations for training warm up"}, - 'dataset_enlarge_ratio': { - 'default': 10.0, - 'type': float, - 'help': "Multiplier of the original number of training samples"}, - 'sub_dataset': { - 'default': False, - 'type': lambda x: bool(strtobool(x)), - 'help': "When set the original training data are randomly sub-sampled for training"}, - 'sub_dataset_frac': { - 'default': 1.0, - 'type': float, - 'help': "Fraction of the original training data set size to sub-sample"}, - 'single_epoch_ok': { - 'default': False, - 'type': lambda x: bool(strtobool(x)), - 'help': "If to only train data for one epoch"}, - 'profiler_iter_num': { - 'default': 3, - 'type': int, - 'help': "Number of subsequent iterations to track computation performance for HTA"},} + } MODEL_TYPES['train']['EDVRModel'] = { 'initial_lr': { @@ -326,7 +340,7 @@ def list_of_ints(arg): 'default': 'none', 'type': Path, 'help': "When set the path to the model weights to initialize the model"}, - 'num_frame': { + 'num_frame_infer': { 'default': 3, 'type': int, 'help': "Number of input low resolution frames to the model"}, @@ -510,7 +524,7 @@ def list_of_ints(arg): 'help': 'when set the model has separate encoders for the hr/lr input image tensors' }, 'cal_train_poisson_b0_exponent': { - 'default': [], + 'default': 'none', 'type': list_of_ints, 'help': 'upper and lower bounds of the blank scan factor in log scale' }, @@ -688,7 +702,7 @@ def list_of_ints(arg): 'help': "Full path to the user experiment data" }, 'case_list': {'default': [], - 'type': list_of_ints, + 'type': list_of_strs, 'help': "Names of the individual experiment subdirectories" }, 'device': {'default': 0, diff --git a/xfusion/inference/infer_swin_ddp.py b/xfusion/inference/infer_swin_ddp.py index f9c6858..4f14786 100644 --- a/xfusion/inference/infer_swin_ddp.py +++ b/xfusion/inference/infer_swin_ddp.py @@ -59,10 +59,10 @@ def run_inference(ts, args): opt['network_g']['embed_dim'] = args.embed_dim opt['network_g']['num_feat_ext'] = args.embed_dim - opt['network_g']['num_frame'] = args.num_frame + opt['network_g']['num_frame'] = args.num_frame_infer opt['network_g']['depths'] = [d * args.scale_depth for d in opt['network_g']['depths']] - opt['network_g']['window_size'][0] += (args.num_frame - 3) + opt['network_g']['window_size'][0] += (args.num_frame_infer - 3) opt['network_g']['window_size'][1:] = [args.window_size_spatial] * len(opt['network_g']['window_size'][1:]) opt['network_g']['num_heads'] = [int(np.ceil(h * args.scale_mha)) for h in opt['network_g']['num_heads']] @@ -75,7 +75,7 @@ def run_inference(ts, args): dataroot = config.get_inf_data_dirs(img_class) inf_home_dir = Path(dataroot).parent - out_dir = Path(inf_home_dir / f'{test_set_name}_{mode}_align_{args.align_features_ok}_prepostx_fixed_lr_r_{lo_frame_sep}_hr_d_{hi_frame_sep*2}_b0_{b0}_em_{args.embed_dim}_nf_{args.num_frame}_dep_{args.scale_depth}_nh_{args.scale_mha}_win_{args.window_size_spatial}_model_{args.model_file.parent.stem}_ddp_swinVRHP_rem1_{ts}') + out_dir = Path(inf_home_dir / f'{test_set_name}_{mode}_align_{args.align_features_ok}_prepostx_fixed_lr_r_{lo_frame_sep}_hr_d_{hi_frame_sep*2}_b0_{b0}_em_{args.embed_dim}_nf_{args.num_frame_infer}_dep_{args.scale_depth}_nh_{args.scale_mha}_win_{args.window_size_spatial}_model_{args.model_file.parent.stem}_ddp_swinVRHP_rem1_{ts}') out_dir.mkdir(exist_ok=True,parents=True) log_file = os.path.join(out_dir, f"inference.log") @@ -129,7 +129,7 @@ def run_inference(ts, args): # continue dataset_opt['scale'] = 4 dataset_opt['gt_size'] = gt_size[:2] - dataset_opt['num_frame'] = args.num_frame + dataset_opt['num_frame'] = args.num_frame_infer #test_set = VideoTestDatasetSTF(dataset_opt) #test_set = build_dataset(dataset_opt) test_set = XrayVideoTestDatasetSTF(dataset_opt) diff --git a/xfusion/inference/infer_with_auto_calibration_main4_continuous2.py b/xfusion/inference/infer_with_auto_calibration_main4_continuous2.py index b5f2ec1..fa9ec71 100644 --- a/xfusion/inference/infer_with_auto_calibration_main4_continuous2.py +++ b/xfusion/inference/infer_with_auto_calibration_main4_continuous2.py @@ -79,7 +79,7 @@ def run_inference(args): exp_root = Path(config.get_inf_data_dirs(input_dir.name)) device = args.device - opt_path = args.opt + opt_path = args.arch_opt opt = yaml_load(opt_path) opt['manual_seed'] = 10 torch.manual_seed(opt['manual_seed']) diff --git a/xfusion/train/basicsr/metrics/metric_util.py b/xfusion/train/basicsr/metrics/metric_util.py index 06d81d4..79f4f6d 100644 --- a/xfusion/train/basicsr/metrics/metric_util.py +++ b/xfusion/train/basicsr/metrics/metric_util.py @@ -15,8 +15,6 @@ import numpy as np -from xfusion.train.basicsr.utils import bgr2ycbcr - def reorder_image(img, input_order='HWC'): """Reorder images to 'HWC' order. @@ -42,19 +40,3 @@ def reorder_image(img, input_order='HWC'): if input_order == 'CHW': img = img.transpose(1, 2, 0) return img - - -def to_y_channel(img): - """Change to Y channel of YCbCr. - - Args: - img (ndarray): Images with range [0, 255]. - - Returns: - (ndarray): Images with range [0, 255] (float type) without round. - """ - img = img.astype(np.float32) / 255. - if img.ndim == 3 and img.shape[2] == 3: - img = bgr2ycbcr(img, y_only=True) - img = img[..., None] - return img * 255. diff --git a/xfusion/train/basicsr/metrics/niqe.py b/xfusion/train/basicsr/metrics/niqe.py index 74a6237..6e8d5c3 100644 --- a/xfusion/train/basicsr/metrics/niqe.py +++ b/xfusion/train/basicsr/metrics/niqe.py @@ -20,7 +20,7 @@ from scipy.ndimage import convolve from scipy.special import gamma -from xfusion.train.basicsr.metrics.metric_util import reorder_image, to_y_channel +from xfusion.train.basicsr.metrics.metric_util import reorder_image from xfusion.train.basicsr.utils.matlab_functions import imresize from xfusion.train.basicsr.utils.registry import METRIC_REGISTRY @@ -198,7 +198,7 @@ def calculate_niqe(img, crop_border, input_order='HWC', convert_to='y', **kwargs if input_order != 'HW': img = reorder_image(img, input_order=input_order) if convert_to == 'y': - img = to_y_channel(img) + raise Exception('not implemented yet...') elif convert_to == 'gray': img = cv2.cvtColor(img / 255., cv2.COLOR_BGR2GRAY) * 255. img = np.squeeze(img) diff --git a/xfusion/train/basicsr/metrics/niqe_pris_params.npz b/xfusion/train/basicsr/metrics/niqe_pris_params.npz deleted file mode 100644 index 204ddce..0000000 Binary files a/xfusion/train/basicsr/metrics/niqe_pris_params.npz and /dev/null differ diff --git a/xfusion/train/basicsr/metrics/psnr_ssim.py b/xfusion/train/basicsr/metrics/psnr_ssim.py index 25f4f42..423f276 100644 --- a/xfusion/train/basicsr/metrics/psnr_ssim.py +++ b/xfusion/train/basicsr/metrics/psnr_ssim.py @@ -18,8 +18,7 @@ import torch import torch.nn.functional as F -from xfusion.train.basicsr.metrics.metric_util import reorder_image, to_y_channel -from xfusion.train.basicsr.utils.color_util import rgb2ycbcr_pt +from xfusion.train.basicsr.metrics.metric_util import reorder_image from xfusion.train.basicsr.utils.registry import METRIC_REGISTRY @@ -51,8 +50,7 @@ def calculate_psnr(img, img2, crop_border, input_order='HWC', test_y_channel=Fal img2 = img2[crop_border:-crop_border, crop_border:-crop_border, ...] if test_y_channel: - img = to_y_channel(img) - img2 = to_y_channel(img2) + raise Exception('not implemented yet...') img = img.astype(np.float64) img2 = img2.astype(np.float64) @@ -86,8 +84,7 @@ def calculate_psnr_pt(img, img2, crop_border, test_y_channel=False, **kwargs): img2 = img2[:, :, crop_border:-crop_border, crop_border:-crop_border] if test_y_channel: - img = rgb2ycbcr_pt(img, y_only=True) - img2 = rgb2ycbcr_pt(img2, y_only=True) + raise Exception('not implemented yet...') img = img.to(torch.float64) img2 = img2.to(torch.float64) @@ -131,8 +128,7 @@ def calculate_ssim(img, img2, crop_border, input_order='HWC', test_y_channel=Fal img2 = img2[crop_border:-crop_border, crop_border:-crop_border, ...] if test_y_channel: - img = to_y_channel(img) - img2 = to_y_channel(img2) + raise Exception('not implemented yet...') img = img.astype(np.float64) img2 = img2.astype(np.float64) @@ -172,8 +168,7 @@ def calculate_ssim_pt(img, img2, crop_border, test_y_channel=False, **kwargs): img2 = img2[:, :, crop_border:-crop_border, crop_border:-crop_border] if test_y_channel: - img = rgb2ycbcr_pt(img, y_only=True) - img2 = rgb2ycbcr_pt(img2, y_only=True) + raise Exception('not implemented yet...') img = img.to(torch.float64) img2 = img2.to(torch.float64) diff --git a/xfusion/train/train_EDVR_L_x4_STF_REDS_mono_prepost_xfull_plus_mult_poiss_u1v3.yml b/xfusion/train/train_EDVR_L_x4_STF_REDS_mono_prepost_xfull_plus_mult_poiss_u1v3.yml new file mode 100644 index 0000000..830ae5e --- /dev/null +++ b/xfusion/train/train_EDVR_L_x4_STF_REDS_mono_prepost_xfull_plus_mult_poiss_u1v3.yml @@ -0,0 +1,141 @@ +# Copyright 2018-2022 BasicSR Authors + +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at + +# http://www.apache.org/licenses/LICENSE-2.0 + +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Modifications Copyright 2025 xfusion authors + +# general settings +name: 104_EDVR_L_x4_STF_REDS_600k_B4G8_valREDS4_103pretrain_wandb_mono_prepost_xfull_plus_mult_poiss +model_type: EDVRModel +scale: 4 +num_gpu: 4 # set num_gpu: 0 for cpu mode +manual_seed: 10 + +# dataset and data loader settings +datasets: + train: + name: XRAY + type: XrayDatasetSTF + flag: grayscale + dataroot_gt: [/train/HR] + dataroot_lq: [/train/LR] + dataroot_flow: ~ + meta_info_file: [meta_info_xray_GT_u1_.txt] + val_partition: official #REDS4 # set to 'official' when use the official validation partition + io_backend: + type: disk + + num_frame: 3 + num_frame_hi: 2 + gt_size: 256 + interval_list: [1] + hi_interval_mult: 1 + random_reverse: false + use_hflip: true + use_rot: true + + # data loader + num_worker_per_gpu: 3 + batch_size_per_gpu: 1 + dataset_enlarge_ratio: 400 + prefetch_mode: ~ + + val: + name: XRAY + type: XrayVideoTestDatasetSTF + flag: grayscale + dataroot_gt: [/val/HR] + dataroot_lq: [/val/LR] + meta_info_file: [meta_info_ADDMA_test_GT.txt] #basicsr/data/meta_info/meta_info_REDS4_test_GT.txt + # change to 'meta_info_REDSofficial4_test_GT' when use the official validation partition + io_backend: + type: disk + + cache_data: false + num_frame: 3 + num_frame_hi: 2 + padding: reflection_circle + +# network structures +network_g: + #ema_decay: 0.999 + type: EDVRSTF + num_in_ch: 1 + num_out_ch: 1 + num_feat: 128 + num_frame: 3 + num_frame_hi: 2 + deformable_groups: 8 + num_extract_block: 5 + num_reconstruct_block: 40 + center_frame_idx: ~ + hr_in: false + with_predeblur: false + with_tsa: true + +# path +path: + pretrain_network_g: net_g_latest.pth + strict_load_g: true + resume_state: ~ + +# training settings +train: + optim_g: + type: Adam + lr: !!float 1e-4 + weight_decay: 0 + betas: [0.9, 0.99] + + scheduler: + type: CosineAnnealingRestartLR + periods: [300000] + restart_weights: [1] + eta_min: !!float 1e-7 + + total_iter: 300000 + warmup_iter: -1 # no warm up + tsa_iter: 0 + dcn_lr_mul: 1 + + # losses + pixel_opt: + type: CharbonnierLoss + loss_weight: 1.0 + reduction: sum + +# validation settings +val: + val_freq: !!float 5e3 + save_img: false + + metrics: + psnr: # metric name, can be arbitrary + type: calculate_psnr + crop_border: 0 + test_y_channel: false + +# logging settings +logger: + print_freq: 100 + save_checkpoint_freq: !!float 5e3 + use_tb_logger: true + wandb: + project: ~ + resume_id: ~ + +# dist training settings +dist_params: + backend: nccl + port: 29500 + +find_unused_parameters: true diff --git a/xfusion/train/train_SwinVR_VSTFx4_xray_scratch_remove1_u1v3_contextmap.yml b/xfusion/train/train_SwinVR_VSTFx4_xray_scratch_remove1_u1v3_contextmap.yml new file mode 100644 index 0000000..e06fd34 --- /dev/null +++ b/xfusion/train/train_SwinVR_VSTFx4_xray_scratch_remove1_u1v3_contextmap.yml @@ -0,0 +1,143 @@ +# Copyright 2018-2022 BasicSR Authors + +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at + +# http://www.apache.org/licenses/LICENSE-2.0 + +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Modifications Copyright 2025 xfusion authors + +# general settings +name: train_SwinVR_STFx4_scratch_P48W8_DIV2K_500k_B4G8_s18_x2_0.4 +model_type: SwinIRModel +scale: 4 +num_gpu: 4 +manual_seed: 0 + +# dataset and data loader settings +datasets: + train: + name: XRAY + type: XrayDatasetSTF + flag: grayscale + dataroot_gt: [/train/HR] + dataroot_lq: [/train/LR] + dataroot_context: [/train/LRC] + dataroot_flow: ~ + meta_info_file: [meta_info_xray_GT_u1.txt] + meta_info_context: [context_map.csv] + use_nearest_context: true + val_partition: official #REDS4 # set to 'official' when use the official validation partition + io_backend: + type: disk + + num_frame: 3 + num_frame_hi: 2 + gt_size: 96 + interval_list: [1] + hi_interval_mult: 1 + random_reverse: false + use_hflip: true + use_rot: true + use_local_renorm: false + poisson_noise_b0_exp: [1,7] + + # data loader + num_worker_per_gpu: 3 + batch_size_per_gpu: 1 + dataset_enlarge_ratio: 400 + prefetch_mode: ~ + + val: + name: XRAY + type: XrayVideoTestDatasetSTF + flag: grayscale + dataroot_gt: [/val/HR] + dataroot_lq: [/val/LR] + meta_info_file: [meta_info_ADDMA_test_GT.txt] #basicsr/data/meta_info/meta_info_REDS4_test_GT.txt + # change to 'meta_info_REDSofficial4_test_GT' when use the official validation partition + io_backend: + type: disk + + cache_data: false + num_frame: 3 + num_frame_hi: 2 + padding: reflection_circle + use_local_renorm: false + +# network structures +network_g: + type: PatchTransformerSTF + upscale: 4 + num_in_ch: 1 + num_frame: 3 + #img_size: 48 + window_size: [5,8,8] + #img_range: 1. + depths: [2,2,18,2] + embed_dim: 192 + num_heads: [3,3,3,3] + mlp_ratio: 2 + upsampler: 'pixelshuffle' + resi_connection: '1conv' + +# path +path: + pretrain_network_g: net_g_latest.pth + strict_load_g: true + resume_state: ~ + +# training settings +train: + ema_decay: 0.999 + optim_g: + type: Adam + lr: !!float 2e-4 + weight_decay: 0 + betas: [0.9, 0.99] + + scheduler: + type: CosineAnnealingRestartLR + periods: [300000] + restart_weights: [1] + eta_min: !!float 1e-7 + + total_iter: 300000 + warmup_iter: -1 # no warm up + + # losses + pixel_opt: + type: L1Loss + loss_weight: 1.0 + reduction: mean + +# validation settings +val: + val_freq: !!float 5e3 + save_img: false + + metrics: + psnr: # metric name, can be arbitrary + type: calculate_psnr + crop_border: 4 + test_y_channel: false + +# logging settings +logger: + print_freq: 100 + save_checkpoint_freq: !!float 5e3 + use_tb_logger: true + wandb: + project: ~ + resume_id: ~ + +# dist training settings +dist_params: + backend: nccl + port: 29500 diff --git a/xfusion/train/train_simple_gray4_VSWINIR_xray_multinode_ddp_new4.py b/xfusion/train/train_simple_gray4_VSWINIR_xray_multinode_ddp_new4.py index e7ee2ca..631013d 100644 --- a/xfusion/train/train_simple_gray4_VSWINIR_xray_multinode_ddp_new4.py +++ b/xfusion/train/train_simple_gray4_VSWINIR_xray_multinode_ddp_new4.py @@ -68,6 +68,8 @@ def parse_options_(root_path,args,opt): opt['datasets']['train']['interval_list'] = args.interval_list opt['datasets']['train']['hi_interval_mult'] = args.hi_interval_mult opt['datasets']['val']['num_frame'] = args.num_frame + if args.poisson_noise_b0_exp is not None: + opt['datasets']['train']['poisson_noise_b0_exp'] = args.poisson_noise_b0_exp opt['network_g']['num_frame'] = args.num_frame opt['network_g']['window_size'][0] += (args.num_frame - 3) opt['network_g']['window_size'][1:] = [args.window_size_spatial] * len(opt['network_g']['window_size'][1:]) @@ -257,91 +259,91 @@ def train_pipeline(args,opt): opt['root_path'] = root_path if type(args.dir_hi_train) is not list: - if str(args.dir_hi_train) != 'none': + if str(args.dir_hi_train) not in ['none','.']: if opt['datasets']['train']['type'] == 'XrayDatasetSTF': opt['datasets']['train']['dataroot_gt'] = [str(args.dir_hi_train)] else: opt['datasets']['train']['dataroot_gt'] = str(args.dir_hi_train) else: assert opt['datasets']['train']['type'] == 'XrayDatasetSTF' - if not any([True if str(dh) == 'none' else False for dh in args.dir_hi_train]): + if not any([True if str(dh) in ['none','.'] else False for dh in args.dir_hi_train]): opt['datasets']['train']['dataroot_gt'] = [str(dh) for dh in args.dir_hi_train] if type(args.dir_lo_train) is not list: - if str(args.dir_lo_train) != 'none': + if str(args.dir_lo_train) not in ['none','.']: if opt['datasets']['train']['type'] == 'XrayDatasetSTF': opt['datasets']['train']['dataroot_lq'] = [str(args.dir_lo_train)] else: opt['datasets']['train']['dataroot_lq'] = str(args.dir_lo_train) else: assert opt['datasets']['train']['type'] == 'XrayDatasetSTF' - if not any([True if str(dl) == 'none' else False for dl in args.dir_lo_train]): + if not any([True if str(dl) in ['none','.'] else False for dl in args.dir_lo_train]): opt['datasets']['train']['dataroot_lq'] = [str(dl) for dl in args.dir_lo_train] if type(args.dir_hi_val) is not list: - if str(args.dir_hi_val) != 'none': + if str(args.dir_hi_val) not in ['none','.']: if opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF': opt['datasets']['val']['dataroot_gt'] = [str(args.dir_hi_val)] else: opt['datasets']['val']['dataroot_gt'] = str(args.dir_hi_val) else: assert opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF' - if not any([True if str(dh) == 'none' else False for dh in args.dir_hi_val]): + if not any([True if str(dh) in ['none','.'] else False for dh in args.dir_hi_val]): opt['datasets']['val']['dataroot_gt'] = [str(dh) for dh in args.dir_hi_val] if type(args.dir_lo_val) is not list: - if str(args.dir_lo_val) != 'none': + if str(args.dir_lo_val) not in ['none','.']: if opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF': opt['datasets']['val']['dataroot_lq'] = [str(args.dir_lo_val)] else: opt['datasets']['val']['dataroot_lq'] = str(args.dir_lo_val) else: assert opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF' - if not any([True if str(dl) == 'none' else False for dl in args.dir_lo_val]): + if not any([True if str(dl) in ['none','.'] else False for dl in args.dir_lo_val]): opt['datasets']['val']['dataroot_lq'] = [str(dl) for dl in args.dir_lo_val] if type(args.path_train_meta_info_file) is not list: - if str(args.path_train_meta_info_file) != 'none': + if str(args.path_train_meta_info_file) not in ['none','.']: if opt['datasets']['train']['type'] == 'XrayDatasetSTF': opt['datasets']['train']['meta_info_file'] = [str(args.path_train_meta_info_file)] else: opt['datasets']['train']['meta_info_file'] = str(args.path_train_meta_info_file) else: assert opt['datasets']['train']['type'] == 'XrayDatasetSTF' - if not any([True if str(pt) == 'none' else False for pt in args.path_train_meta_info_file]): + if not any([True if str(pt) in ['none','.'] else False for pt in args.path_train_meta_info_file]): opt['datasets']['train']['meta_info_file'] = [str(pt) for pt in args.path_train_meta_info_file] if type(args.path_val_meta_info_file) is not list: - if str(args.path_val_meta_info_file) != 'none': + if str(args.path_val_meta_info_file) not in ['none','.']: if opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF': opt['datasets']['val']['meta_info_file'] = [str(args.path_val_meta_info_file)] else: opt['datasets']['val']['meta_info_file'] = str(args.path_val_meta_info_file) else: assert opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF' - if not any([True if str(pv) == 'none' else False for pv in args.path_val_meta_info_file]): + if not any([True if str(pv) in ['none','.'] else False for pv in args.path_val_meta_info_file]): opt['datasets']['val']['meta_info_file'] = [str(pv) for pv in args.path_val_meta_info_file] if type(args.dataroot_context) is not list: - if str(args.dataroot_context) != 'none': + if str(args.dataroot_context) not in ['none','.']: if opt['datasets']['train']['type'] == 'XrayDatasetSTF': opt['datasets']['train']['dataroot_context'] = [str(args.dataroot_context)] else: opt['datasets']['train']['dataroot_context'] = str(args.dataroot_context) else: assert opt['datasets']['train']['type'] == 'XrayDatasetSTF' - if not any([True if str(dc) == 'none' else False for dc in args.dataroot_context]): + if not any([True if str(dc) in ['none','.'] else False for dc in args.dataroot_context]): opt['datasets']['train']['dataroot_context'] = [str(dc) for dc in args.dataroot_context] if type(args.meta_info_context) is not list: - if str(args.meta_info_context) != 'none': + if str(args.meta_info_context) not in ['none','.']: if opt['datasets']['train']['type'] == 'XrayDatasetSTF': opt['datasets']['train']['meta_info_context'] = [str(args.meta_info_context)] else: opt['datasets']['train']['meta_info_context'] = str(args.meta_info_context) else: assert opt['datasets']['train']['type'] == 'XrayDatasetSTF' - if not any([True if str(mic) == 'none' else False for mic in args.meta_info_context]): + if not any([True if str(mic) in ['none','.'] else False for mic in args.meta_info_context]): opt['datasets']['train']['meta_info_context'] = [str(mic) for mic in args.meta_info_context] opt['datasets']['train']['use_nearest_context'] = args.use_nearest_context opt['datasets']['train']['num_worker_per_gpu'] = args.num_workers diff --git a/xfusion/train/train_simple_gray6_xfull_mult_poiss.py b/xfusion/train/train_simple_gray6_xfull_mult_poiss.py index d229ee8..899dccc 100644 --- a/xfusion/train/train_simple_gray6_xfull_mult_poiss.py +++ b/xfusion/train/train_simple_gray6_xfull_mult_poiss.py @@ -36,7 +36,8 @@ def parse_options_(root_path,args,opt): import random from xfusion.utils import _postprocess_yml_value - + if args.force_yml == 'none': + args.force_yml = None # distributed settings if args.launcher == 'none': print(f"debugging...{opt}") @@ -213,6 +214,94 @@ def train_pipeline(args,opt): opt['train']['total_iter'] /= args.iter_div_factor opt['train']['optim_g']['lr'] = args.initial_lr opt['val']['val_freq'] = args.val_freq + + if type(args.dir_hi_train) is not list: + if str(args.dir_hi_train) not in ['none','.']: + if opt['datasets']['train']['type'] == 'XrayDatasetSTF': + opt['datasets']['train']['dataroot_gt'] = [str(args.dir_hi_train)] + else: + opt['datasets']['train']['dataroot_gt'] = str(args.dir_hi_train) + else: + assert opt['datasets']['train']['type'] == 'XrayDatasetSTF' + if not any([True if str(dh) in ['none','.'] else False for dh in args.dir_hi_train]): + opt['datasets']['train']['dataroot_gt'] = [str(dh) for dh in args.dir_hi_train] + + if type(args.dir_lo_train) is not list: + if str(args.dir_lo_train) not in ['none','.']: + if opt['datasets']['train']['type'] == 'XrayDatasetSTF': + opt['datasets']['train']['dataroot_lq'] = [str(args.dir_lo_train)] + else: + opt['datasets']['train']['dataroot_lq'] = str(args.dir_lo_train) + else: + assert opt['datasets']['train']['type'] == 'XrayDatasetSTF' + if not any([True if str(dl) in ['none','.'] else False for dl in args.dir_lo_train]): + opt['datasets']['train']['dataroot_lq'] = [str(dl) for dl in args.dir_lo_train] + + if type(args.dir_hi_val) is not list: + if str(args.dir_hi_val) not in ['none','.']: + if opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF': + opt['datasets']['val']['dataroot_gt'] = [str(args.dir_hi_val)] + else: + opt['datasets']['val']['dataroot_gt'] = str(args.dir_hi_val) + else: + assert opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF' + if not any([True if str(dh) in ['none','.'] else False for dh in args.dir_hi_val]): + opt['datasets']['val']['dataroot_gt'] = [str(dh) for dh in args.dir_hi_val] + + if type(args.dir_lo_val) is not list: + if str(args.dir_lo_val) not in ['none','.']: + if opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF': + opt['datasets']['val']['dataroot_lq'] = [str(args.dir_lo_val)] + else: + opt['datasets']['val']['dataroot_lq'] = str(args.dir_lo_val) + else: + assert opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF' + if not any([True if str(dl) in ['none','.'] else False for dl in args.dir_lo_val]): + opt['datasets']['val']['dataroot_lq'] = [str(dl) for dl in args.dir_lo_val] + + if type(args.path_train_meta_info_file) is not list: + if str(args.path_train_meta_info_file) not in ['none','.']: + if opt['datasets']['train']['type'] == 'XrayDatasetSTF': + opt['datasets']['train']['meta_info_file'] = [str(args.path_train_meta_info_file)] + else: + opt['datasets']['train']['meta_info_file'] = str(args.path_train_meta_info_file) + else: + assert opt['datasets']['train']['type'] == 'XrayDatasetSTF' + if not any([True if str(pt) in ['none','.'] else False for pt in args.path_train_meta_info_file]): + opt['datasets']['train']['meta_info_file'] = [str(pt) for pt in args.path_train_meta_info_file] + + if type(args.path_val_meta_info_file) is not list: + if str(args.path_val_meta_info_file) not in ['none','.']: + if opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF': + opt['datasets']['val']['meta_info_file'] = [str(args.path_val_meta_info_file)] + else: + opt['datasets']['val']['meta_info_file'] = str(args.path_val_meta_info_file) + else: + assert opt['datasets']['val']['type'] == 'XrayVideoTestDatasetSTF' + if not any([True if str(pv) in ['none','.'] else False for pv in args.path_val_meta_info_file]): + opt['datasets']['val']['meta_info_file'] = [str(pv) for pv in args.path_val_meta_info_file] + + if type(args.dataroot_context) is not list: + if str(args.dataroot_context) not in ['none','.']: + if opt['datasets']['train']['type'] == 'XrayDatasetSTF': + opt['datasets']['train']['dataroot_context'] = [str(args.dataroot_context)] + else: + opt['datasets']['train']['dataroot_context'] = str(args.dataroot_context) + else: + assert opt['datasets']['train']['type'] == 'XrayDatasetSTF' + if not any([True if str(dc) in ['none','.'] else False for dc in args.dataroot_context]): + opt['datasets']['train']['dataroot_context'] = [str(dc) for dc in args.dataroot_context] + + if type(args.meta_info_context) is not list: + if str(args.meta_info_context) not in ['none','.']: + if opt['datasets']['train']['type'] == 'XrayDatasetSTF': + opt['datasets']['train']['meta_info_context'] = [str(args.meta_info_context)] + else: + opt['datasets']['train']['meta_info_context'] = str(args.meta_info_context) + else: + assert opt['datasets']['train']['type'] == 'XrayDatasetSTF' + if not any([True if str(mic) in ['none','.'] else False for mic in args.meta_info_context]): + opt['datasets']['train']['meta_info_context'] = [str(mic) for mic in args.meta_info_context] if args.poisson_noise_b0_exp is not None: opt['datasets']['train']['poisson_noise_b0_exp'] = args.poisson_noise_b0_exp torch.backends.cudnn.benchmark = True