Skip to content

Fix dim Xenium morphology tiles and improve OME-TIFF loading #294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jaspreetishar wants to merge 23 commits into
base: main
Choose a base branch
from
Open

Fix dim Xenium morphology tiles and improve OME-TIFF loading #294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
96ae69e
fixed dim Xenium image tiles - added percentile-based outlier clippin…
jaspreetishar Mar 12, 2026
b2909ae
ruff
jaspreetishar Mar 12, 2026
c91e5c0
ruff suggested to remove "None" from trx_tile.py
jaspreetishar Mar 12, 2026
d06ca2e
pytest: chromium unit test fix
jaspreetishar Mar 12, 2026
a3a6d3d
ruff fix: remove None from `tile_dict.get((tile_i, tile_j))`
jaspreetishar Mar 12, 2026
a14e81f
including gamma and white_level
jaspreetishar Mar 13, 2026
991ccd6
cleaning up
jaspreetishar Mar 13, 2026
a7a6988
ruff
jaspreetishar Mar 13, 2026
edec366
notebook update
jaspreetishar Mar 13, 2026
b883603
added white_level argument in dega.pre.main(); added subsampling duri…
jaspreetishar Mar 17, 2026
a6aa9a3
ruff
jaspreetishar Mar 17, 2026
1c2bf1e
Potential fix for chromium unit test
jaspreetishar Mar 18, 2026
b1f4adb
adding copilot's suggestion of validating upper_percentile before use
jaspreetishar Mar 18, 2026
17f8479
validating white_level and gamma as well
jaspreetishar Mar 18, 2026
0a80f93
docstring for create_image_tiles
jaspreetishar Mar 18, 2026
3b93cec
docstring update for run_pre_processing
jaspreetishar Mar 18, 2026
c0f8f2b
ruff
jaspreetishar Mar 18, 2026
975bbc2
fixing chromium pytest + ruff
jaspreetishar Mar 18, 2026
58fa827
fixing _check_required_files for newer image filenames
jaspreetishar Apr 4, 2026
74f0471
ruff
jaspreetishar Apr 4, 2026
082c21a
cursor changes;
cornhundred Apr 13, 2026
3067a62
Merge branch 'img_intensity_fix' of github.com:broadinstitute/celldeg…
cornhundred Apr 13, 2026
5b3bb57
merge main
cornhundred Jun 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 70 additions & 51 deletions notebooks/Xenium_pre-process.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,18 @@
"text": [
"env: ANYWIDGET_HMR=1\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/jishar/Documents/celldega/dega/lib/python3.13/site-packages/dask/dataframe/__init__.py:31: FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning.\n",
" warnings.warn(\n",
"/Users/jishar/Documents/celldega/dega/lib/python3.13/site-packages/xarray_schema/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.\n",
" from pkg_resources import DistributionNotFound, get_distribution\n",
"/Users/jishar/Documents/celldega/dega/lib/python3.13/site-packages/anndata/__init__.py:70: FutureWarning: Importing read_text from `anndata` is deprecated. Import anndata.io.read_text instead.\n",
" return module_get_attr_redirect(attr_name, deprecated_mapping=_DEPRECATED)\n"
]
}
],
"source": [
Expand Down Expand Up @@ -49,13 +61,13 @@
"output_type": "stream",
"text": [
"Starting preprocessing for sample: Xenium_V1_human_Pancreas_FFPE_outs\n",
"Created directory: data/landscape_files/Xenium_V1_human_Pancreas_FFPE_outs_test\n",
"Created directory: data/landscape_files/Xenium_V1_human_Pancreas_FFPE_outs_test_img_intensity_fix\n",
"\n",
"========Unzip and extract Xenium-related files========\n",
"All files have been successfully extracted or skipped.\n",
"\n",
"========Write xenium transform file from the Zarr folder========\n",
"Transformation matrix saved to 'data/landscape_files/Xenium_V1_human_Pancreas_FFPE_outs_test/micron_to_image_transform.csv'.\n",
"Transformation matrix saved to 'data/landscape_files/Xenium_V1_human_Pancreas_FFPE_outs_test_img_intensity_fix/micron_to_image_transform.csv'.\n",
"\n",
"========Check if all required files or directories exist========\n",
"All required files or directories for technology 'Xenium' are present in 'data/xenium_data/Xenium_V1_human_Pancreas_FFPE_outs'.\n",
Expand All @@ -78,7 +90,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/jishar/Documents/celldega/src/celldega/pre/__init__.py:218: PerformanceWarning: Concatenating sparse arrays with multiple fill values: '[True, False]'. Picking the first and converting the rest.\n",
"/Users/jishar/Documents/celldega/src/celldega/pre/__init__.py:225: PerformanceWarning: Concatenating sparse arrays with multiple fill values: '[True, False]'. Picking the first and converting the rest.\n",
" df_sig = df_sig.dropna(axis=1, how=\"all\")\n"
]
},
Expand All @@ -93,11 +105,6 @@
"Calculating mean expression\n",
"Calculating variance\n",
"All meta gene files are succesfully saved.\n",
"data/landscape_files/Xenium_V1_human_Pancreas_FFPE_outs_test/cbg\n",
"cbg.index before mapping:\n",
"Index(['aaaadnje-1', 'aaacalai-1', 'aaacjgil-1', 'aaacpcil-1', 'aaadhocp-1'], dtype='object', name=0)\n",
"cbg.index after mapping:\n",
"Index([0, 1, 2, 3, 4], dtype='int64', name=0)\n",
"Processing gene 0: ABCC11\n",
"All gene-specific parquet files are succesfully saved.\n",
"\n",
Expand All @@ -121,24 +128,64 @@
"\n",
"========Generating image tiles========\n",
"------ xenium\n",
"generating dapi image tiles ...\n"
"Using morphology image: data/xenium_data/Xenium_V1_human_Pancreas_FFPE_outs/morphology_focus/morphology_focus_0000.ome.tif\n",
"OME shape: (4, 13770, 34155)\n",
"OME axes: CYX\n",
"OME dtype: uint16\n",
"Reading channel 'dapi' (index 0)\n",
"generating dapi image tiles ...\n",
"dapi: lo=0.00, p95=1596.00, gamma=1\n",
"Reading channel 'bound' (index 1)\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/jishar/Documents/celldega/dega/lib/python3.13/site-packages/skimage/_shared/utils.py:328: UserWarning: /Users/jishar/Documents/celldega/notebooks/data/landscape_files/Xenium_V1_human_Pancreas_FFPE_outs_test/dapi_output_regular.tif is a low contrast image\n",
" return func(*args, **kwargs)\n"
"/Users/jishar/Documents/celldega/dega/lib/python3.13/site-packages/tifffile/tifffile.py:9310: UserWarning: <tifffile.TiffPage 0 @16> reading array from closed file\n",
" warnings.warn(\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"generating bound image tiles ...\n",
"bound: lo=0.00, p95=2544.00, gamma=1\n",
"Reading channel 'rna' (index 2)\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/jishar/Documents/celldega/dega/lib/python3.13/site-packages/tifffile/tifffile.py:9310: UserWarning: <tifffile.TiffPage 0 @16> reading array from closed file\n",
" warnings.warn(\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"generating rna image tiles ...\n",
"rna: lo=0.00, p95=4301.00, gamma=1\n",
"Reading channel 'prot' (index 3)\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/jishar/Documents/celldega/dega/lib/python3.13/site-packages/tifffile/tifffile.py:9310: UserWarning: <tifffile.TiffPage 0 @16> reading array from closed file\n",
" warnings.warn(\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"generating prot image tiles ...\n",
"prot: lo=0.00, p95=1074.00, gamma=1\n",
"Image tiles created successfully.\n",
"\n",
"======== Transcript Tiles========\n"
Expand All @@ -148,8 +195,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"Processing chunks: 100%|███████████████████████| 81/81 [00:00<00:00, 900.17it/s]\n",
"Processing coarse tiles: 84tile [00:21, 3.97tile/s]\n"
"Processing chunks: 100%|███████████████████████| 81/81 [00:00<00:00, 611.97it/s]\n",
"Processing coarse tiles: 84tile [00:20, 4.11tile/s]\n"
]
},
{
Expand All @@ -161,45 +208,14 @@
"======== Cell Boundary Tiles ========\n",
"\n",
"========Create cell boundary spatial tiles========\n",
"technology Xenium\n",
" geometry_micron \\\n",
"cell_id \n",
"0 POLYGON ((445.613 1697.663, 444.763 1698.300, ... \n",
"1 POLYGON ((442.850 1730.812, 440.938 1731.875, ... \n",
"2 POLYGON ((470.688 1706.163, 470.475 1706.375, ... \n",
"3 POLYGON ((429.888 1703.400, 429.038 1704.038, ... \n",
"4 POLYGON ((478.125 1702.125, 476.213 1703.188, ... \n",
"\n",
" GEOMETRY \\\n",
"cell_id \n",
"0 [[[2096.9999288922727, 7988.9998603837885], [2... \n",
"1 [[[2083.9998724224242, 8144.999389125], [2074.... \n",
"2 [[[2214.999833875, 8028.999857383789], [2213.9... \n",
"3 [[[2022.9999057198486, 8015.999513689697], [20... \n",
"4 [[[2249.99983125, 8009.999399249999], [2240.99... \n",
"\n",
" geometry center_x \\\n",
"cell_id \n",
"0 POLYGON ((2097.000 7989.000, 2093.000 7992.000... 2099.898777 \n",
"1 POLYGON ((2084.000 8144.999, 2075.000 8149.999... 2076.246077 \n",
"2 POLYGON ((2215.000 8029.000, 2214.000 8029.999... 2192.645181 \n",
"3 POLYGON ((2023.000 8016.000, 2019.000 8019.000... 2027.103432 \n",
"4 POLYGON ((2250.000 8009.999, 2241.000 8014.999... 2240.146290 \n",
"\n",
" center_y \n",
"cell_id \n",
"0 8005.963877 \n",
"1 8168.366465 \n",
"2 8057.422615 \n",
"3 8034.700824 \n",
"4 8051.613206 \n"
"technology Xenium\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing coarse tiles: 100%|██████████████████| 14/14 [00:19<00:00, 1.41s/it]\n"
"Processing coarse tiles: 100%|██████████████████| 14/14 [00:17<00:00, 1.22s/it]\n"
]
},
{
Expand All @@ -217,7 +233,7 @@
"source": [
"sample = 'Xenium_V1_human_Pancreas_FFPE_outs'\n",
"data_dir = f'data/xenium_data/'\n",
"path_landscape_files=f'data/landscape_files/{sample}_test'\n",
"path_landscape_files=f'data/landscape_files/{sample}_test_img_intensity_fix'\n",
"\n",
"tile_size=250\n",
"\n",
Expand All @@ -227,6 +243,9 @@
" tile_size=tile_size,\n",
" path_landscape_files=path_landscape_files,\n",
" use_int_index=True,\n",
" # use_row_groups=True,\n",
" upper_percentile=95,\n",
" white_level=40\n",
" )"
]
},
Expand All @@ -240,22 +259,22 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 4,
"id": "4a1320eb",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "80f3435bda344cbf890fcbe3d56da16b",
"model_id": "156e2217606e41f4be8f5de155e04781",
"version_major": 2,
"version_minor": 1
},
"text/plain": [
"Landscape(base_url='http://localhost:53080/data/landscape_files/Xenium_V1_human_Pancreas_FFPE_outs_test', cell…"
"Landscape(base_url='http://localhost:64759/data/landscape_files/Xenium_V1_human_Pancreas_FFPE_outs_test_img_in…"
]
},
"execution_count": 3,
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
Expand Down
Loading
Loading