Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
32666cb
initial code from Tal
michalozeryflato May 24, 2022
4b9f1bf
initial running example
michalozeryflato May 24, 2022
1f6ddcd
backup running example (3 first ops)
michalozeryflato May 25, 2022
cb11fd9
working on the third and forth operator
michalozeryflato May 25, 2022
f71b7c3
add initial version of duke dataset
michalozeryflato May 26, 2022
92e0b01
initial version of complete MRI processing pipeline
michalozeryflato May 31, 2022
5312017
backup changes
michalozeryflato Jun 1, 2022
a09a963
Merge branch 'fuse2' of github.com:IBM/fuse-med-ml into op_mri
michalozeryflato Jun 1, 2022
b79d2a5
fixing imports in duke_breast_cancer old code
michalozeryflato Jun 1, 2022
2b8beb9
backup changes - startting to work on adding labels + runner_duke
michalozeryflato Jun 2, 2022
75e7735
data_cuaration for duke
Jun 2, 2022
346b37a
Merge branch 'op_mri' of github.com:IBM/fuse-med-ml into op_mri
Jun 2, 2022
5329ac4
new version for data curation
Jun 6, 2022
e7ab739
add lesion properties operations
Jun 6, 2022
ee8e248
data curation file, contain pre-preprocessing functions
Jun 6, 2022
6f4c512
backup fixes. runner_duke throws an exception:
michalozeryflato Jun 6, 2022
7f46051
Merge branch 'op_mri' of github.com:IBM/fuse-med-ml into op_mri
michalozeryflato Jun 6, 2022
f83be44
add radiomics Op
Jun 7, 2022
c0875cc
add radiomics processing functions
Jun 7, 2022
5ede2f0
add radiomics
Jun 7, 2022
5c7e513
merge op
Jun 7, 2022
c58337c
''
Jun 7, 2022
a06bbac
''
Jun 7, 2022
dc9ad94
add radiomics op
Jun 7, 2022
3ae6a41
minor change
Jun 7, 2022
787a961
delete=false
Jun 7, 2022
007fab3
delete tmp file
Jun 7, 2022
344f688
backup first version of duke running example in Fuse2
michalozeryflato Jun 7, 2022
df1efc5
no global variables
michalozeryflato Jun 7, 2022
082797a
backup first version of duke running example in Fuse2
michalozeryflato Jun 8, 2022
769965f
Merge branch 'op_mri' of github.com:IBM/fuse-med-ml into op_mri
michalozeryflato Jun 8, 2022
a1da232
add DukeRadiomics to duke.py, add operations to covert from and to ST…
michalozeryflato Jun 14, 2022
f2ebd83
backup changes in duke, integrating general fuse ops
michalozeryflato Jun 15, 2022
e401e11
backup lesion properties generation
michalozeryflato Jun 16, 2022
e333b3f
add prostate_x example
Jun 19, 2022
f209de7
Merge branch 'op_mri' of github.com:IBM/fuse-med-ml into op_mri
michalozeryflato Jun 19, 2022
e01ed65
backup merge
michalozeryflato Jun 19, 2022
54f3a8f
backup lesion properties generation
michalozeryflato Jun 16, 2022
4908194
backup work on duke dataset (although it fails running), to check pro…
michalozeryflato Jun 19, 2022
45d8260
fix ground truth in duke to be torch.int64
michalozeryflato Jun 19, 2022
3e3e573
backup re-orgs in ops_mri
michalozeryflato Jun 23, 2022
a677218
backup changes in prostatex dataset: add augmentations
michalozeryflato Jun 25, 2022
4a29da2
backup changes in prostatex and duke datasets: add augmentations
michalozeryflato Jun 25, 2022
244d458
start re-org and cleaning of duke and prostate_x
michalozeryflato Jun 27, 2022
8ba69c4
clean prostate_x and duke code
michalozeryflato Jun 28, 2022
618866d
backup work on duke dataset (although it fails running), to check pro…
michalozeryflato Jun 19, 2022
98d450b
fix ground truth in duke to be torch.int64
michalozeryflato Jun 19, 2022
cf308c7
backup re-orgs in ops_mri
michalozeryflato Jun 23, 2022
0019736
backup changes in prostatex dataset: add augmentations
michalozeryflato Jun 25, 2022
0b9c120
backup changes in prostatex and duke datasets: add augmentations
michalozeryflato Jun 25, 2022
c08977b
start re-org and cleaning of duke and prostate_x
michalozeryflato Jun 27, 2022
e119b91
clean prostate_x and duke code
michalozeryflato Jun 28, 2022
24f0135
minor delete of a comment
michalozeryflato Jun 28, 2022
76da4e2
Merge remote-tracking branch 'origin/op_mri_v2' into op_mri_v2
michalozeryflato Jun 28, 2022
90da516
Merge pull request #120 from IBM/op_mri_v2
michalozeryflato Jun 28, 2022
1eb0289
ProstateX: remove dependency in Tal's processed file
michalozeryflato Jun 28, 2022
f39825b
Merge branch 'op_mri' into op_mri_v2
michalozeryflato Jun 28, 2022
1e877e0
Merge pull request #123 from IBM/op_mri_v2
michalozeryflato Jun 28, 2022
97f0975
fix the problem in prostate_x by removing 5 samples
michalozeryflato Jun 28, 2022
cb308da
Merge remote-tracking branch 'origin/fuse2' into op_mri
michalozeryflato Jun 29, 2022
4583832
create folds file keeping each patient's samples within the same fold.
michalozeryflato Jun 30, 2022
3d90670
handle pull request comments, such as using existing operations
michalozeryflato Jun 30, 2022
ea1522a
Merge branch 'fuse2' of github.com:IBM/fuse-med-ml into op_mri
michalozeryflato Jun 30, 2022
d98f01d
initial (non-working) version of runner_prostate_x with ManagerDefaul…
michalozeryflato Jul 3, 2022
beef102
adding test code for duke; fixing mask in duke to be integer.
michalozeryflato Jul 3, 2022
b5e313c
fix user_path_home to be optional in duke and prostatex examples
michalozeryflato Jul 4, 2022
aeecf25
Merge branch 'fuse2' of github.com:IBM/fuse-med-ml into op_mri
michalozeryflato Jul 4, 2022
60d8a6e
cancel @run_in_subprocess - to locate errors
michalozeryflato Jul 4, 2022
b754c0d
add test for prostatex
michalozeryflato Jul 4, 2022
4e9d35f
return sub-process in test
michalozeryflato Jul 4, 2022
68c8452
modify run_multiprocessed.py to get subprocess error
michalozeryflato Jul 5, 2022
8f4e0f5
run_multiprocessed.py: fixing printing of error
michalozeryflato Jul 5, 2022
a44c086
debegging
michalozeryflato Jul 5, 2022
6d7c8e8
run tests with prev run in subprocess
mosheraboh Jul 5, 2022
15991bf
replace exmaples.fuse_examples with fuse_examples
michalozeryflato Jul 7, 2022
68cc65d
Merge remote-tracking branch 'origin/op_mri' into op_mri
michalozeryflato Jul 7, 2022
40282eb
cosmetic organization of prostate_x static pipeline
michalozeryflato Jul 25, 2022
1188915
prostate runner is now running OK
Oct 9, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions examples/fuse_examples/fuse_examples_utils.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import os
def ask_user(yes_no_question):
res = ''
while res not in ['y', 'n']:
res = input(f'{yes_no_question}? [y/n]')
return res =='y'

def get_fuse_examples_user_dir():
if 'USER_HOME_PATH' in os.environ:
fuse_examples_dir = os.path.join(os.environ['USER_HOME_PATH'],'fuse_examples')
else:
fuse_examples_dir = './fuse_examples'
if not os.path.exists(fuse_examples_dir):
os.mkdir(fuse_examples_dir)
return fuse_examples_dir
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import os

from fuse_examples.fuse_examples_utils import get_fuse_examples_user_dir


def get_duke_user_dir():
return os.path.join(get_fuse_examples_user_dir(), 'duke')

def get_duke_radiomics_user_dir():
return os.path.join(get_fuse_examples_user_dir(), ' duke_radiomics')


This file was deleted.

Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
import gzip
import os
import pickle

import pandas as pd


def save_object(obj, filename):
open_func = gzip.open if filename.endswith(".gz") else open
filename_tmp = filename+ ".del"
if os.path.exists(filename_tmp):
os.remove(filename_tmp)
with open_func(filename_tmp, 'wb') as output:
pickle.dump(obj, output, pickle.HIGHEST_PROTOCOL)

os.rename(filename_tmp, filename)
return filename


def load_object(filename):
open_func = gzip.open if filename.endswith(".gz") else open

with open_func(filename, 'rb') as myinput:
try:
res = pickle.load(myinput)
except RuntimeError as e:
print("Failed to read", filename)
raise e
return res


DUKE_PROCESSED_FILE_DIR = '/projects/msieve_dev3/usr/common/duke_processed_files'


def get_duke_annotations_from_tal_df():
annotations_path = os.path.join(DUKE_PROCESSED_FILE_DIR, 'dataset_DUKE_folds_ver11102021TumorSize_seed1.pickle')
with open(annotations_path, 'rb') as infile:
fold_annotations_dict = pickle.load(infile)
annotations_df = pd.concat(
[fold_annotations_dict[f'data_fold{fold}'] for fold in range(len(fold_annotations_dict))])
return annotations_df


def get_col_mapping():
return {'MRI Findings:Skin/Nipple Invovlement': 'Skin Invovlement',
'US features:Tumor Size (cm)': 'Tumor Size US',
'Mammography Characteristics:Tumor Size (cm)': 'Tumor Size MG',
'MRI Technical Information:FOV Computed (Field of View) in cm': 'Field of View',
'MRI Technical Information:Contrast Bolus Volume (mL)': 'Contrast Bolus Volume',
'Demographics:Race and Ethnicity': 'Race',
'MRI Technical Information:Manufacturer Model Name': 'Manufacturer',
'MRI Technical Information:Slice Thickness': 'Slice Thickness',
'MRI Findings:Multicentric/Multifocal': 'Multicentric',
'Mammography Characteristics:Breast Density': 'Breast Density MG',
'Tumor Characteristics:PR': 'PR',
'Tumor Characteristics:HER2': 'HER2',
'Tumor Characteristics:ER': 'ER',
'Near Complete Response:Overall Near-complete Response: Stricter Definition': 'Near pCR Strict',
'Tumor Characteristics:Staging(Tumor Size)# [T]': 'Staging Tumor Size',
'Tumor Characteristics:Histologic type': 'Histologic type',
}
Loading