From 299371bb99d959fad2c93bd54cbcc0a60ef2f536 Mon Sep 17 00:00:00 2001 From: emiglietta Date: Wed, 1 May 2024 14:44:41 -0700 Subject: [PATCH 1/4] draft of cocumentation notes and tests for the index --- .gitignore | 1 + CPGvalidator_docs.ipynb | 872 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 873 insertions(+) create mode 100644 CPGvalidator_docs.ipynb diff --git a/.gitignore b/.gitignore index 48a89be..b2214b5 100644 --- a/.gitignore +++ b/.gitignore @@ -162,3 +162,4 @@ cython_debug/ .DS_Store scratch data +EM_notes.rtf diff --git a/CPGvalidator_docs.ipynb b/CPGvalidator_docs.ipynb new file mode 100644 index 0000000..b93bb63 --- /dev/null +++ b/CPGvalidator_docs.ipynb @@ -0,0 +1,872 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# GOALS\n", + ">DESCRIBE:\n", + "What is the input\n", + "What is the output\n", + "\n", + ">Overview—: to the readme\n", + "Include, what is a bucket! --> done!(ish)\n", + "\n", + ">Rules: \n", + "How to construct new rules\n", + "How to run them\n", + "Hoe to understand the output\n", + "\n", + ">In progress is a valid flag bc this work is being done\n", + "\n", + ">Inventory autogenerates ¿once a week? --> Yes! but the index is generated manually\n", + "\n", + ">Explain the example case for downloading files\n", + "\n", + ">README describing what it does\n", + "Tutorial and or examples" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Description\n", + "`cpgdata` is a CLI toolset for navigating and exploring the Cell Painting Gallery.\n", + "\n", + "\n", + "## Context\n", + "\n", + "The CellPainting Gallery (CPG) is hosted in a cloud object storage, AWS's S3.\n", + "\n", + "Within this system, all files within the CPG live in a single 'bucket'. Because it is an object storage, this bucket does not contain a folder structure, but rather all objects live together. In order to identify them, each object is assigned a unique 'key' that consists of a string of characters, akin to the directory path in a regular file structure with folders.\n", + "\n", + "For example, <'s3://cellpainting-gallery/cpg0016-jump/source_4/workspace/analysis/2021_06_21_Batch7/BR00125168/analysis/BR00125168-G17-5/Cells.csv'> is the key to a specific .csv file that lives within the CPG S3 bucket.\n", + "\n", + "The [**aws inventory**]( https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html) is a list of all the objects contained in a bucket and their associated metadata (i.e. object size, date of upload, last modified date, etc). This inventory is updated automatically on a weekly basis for the CPG.\n", + "\n", + "However, the format in which this inventory is structured is not friendly for exploring it. \n", + "\n", + "That is why, we use `cpgdata` to parse (using `cpgparser`, a Python library written in Rust) the inventory, retireving and organizing useful information about all objects in a dataframe called the **Index**. You can [manually browse the Index contents online using Quilt](https://open.quiltdata.com/b/cellpainting-gallery/tree/).\n", + "\n", + "The `cpgdata` package also provides tools to navigate and filter the Index file which can then be used to selectively download certain files from the CPG or explore its contents.\n", + "\n", + "Moreover, these same tools can also be used to create rules to validate the structure and completeness of new data before uploading it to the CPG bucket, to ensure that it complies with the CPG requirements.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The **Inventory** for the CPG lives in \n", + "\n", + "s3://cellpainting-gallery-inventory/\n", + " └── cellpainting-gallery/\n", + " └──index/\n", + " └── [all the index chunks in .parquet format]\n", + " └── whole_bucket/\n", + " ├──2024-03-31T01-00Z/\n", + " ├── 2024-04-07T01-00Z/\n", + " ├── data/\n", + " └──hive/" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The **Index** file lives in \n", + "\n", + "If you want to get an idea of the expected file structure of the CPG you do so [HERE](https://broadinstitute.github.io/cellpainting-gallery/data_structure.html).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### In its current version, `cpgdata` runs on Python 3.10 so the first step will be to create an environment to run it" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!conda create --name cpgdata python==3.10\n", + "!conda activate cpgdata" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Next, you need to install the and packages (only once)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install cpgparser\n", + "!pip install cpgdata" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Import the necessary libraries and packages" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from pathlib import Path\n", + "\n", + "# We use polars to read and explore the index \n", + "import polars as pl\n", + "from cpgdata.utils import parallel, download_s3_files\n", + "\n", + "#These were included in the example but not necessary for the code so far\n", + "# from typing import List\n", + "# from pathlib import Path\n", + "# from pprint import pprint\n", + "# import os\n", + "# from tqdm import tqdm" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### The generation of the **Index** file is a very time-consuming process, but you don't need to do it yourself! \n", + "### You can easily download it using the following command:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Select a local diretory in which to download the Index\n", + "\n", + "index_dir = Path(\"/Users/emigliet/Documents/CPGvalidator_docs\")\n", + "# index_dir = Path(\"Your/Local/Destination/Directory\")\n", + "\n", + "# Note that the Index file is fairly large (over 20gb) so it's divided into several .parquet files.\n", + "# !cpg sync index {index_dir}\n", + "\n", + "# Load the index using polars (pl)\n", + "index_files = [file for file in index_dir.glob(\"*.parquet\")]\n", + "index = pl.scan_parquet(index_files)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Columns included in the Index file:\n", + "\n", + "- `key` : object key identifier, useful for downloading files\n", + "- `root_dir`: ## if col(\"worskpace')==\"workspace, from 'workspace' to leaf node, \n", + "- `images_root_dir`: path after \"{dataset_id}/{source_id}/{batch_id}/images/\" to the object. Is 'null' is object is not within that path.\n", + "- `images_batch_root_dir`: path after \"{dataset_id}/{source_id}/{batch_id}/\" to the object. Is 'null' if object is no within that path.\n", + "- `images_illum_root_dir`: path after \"{dataset_id}/{source_id}/{batch_id}/images/illum/\" to the object. If \"is_dir\"==True, \"images_illum_root_dir\"==\"plate_id\". If \"is_dir\"==False, \"images_illum_root_dir\"==\"plate_id/illumFile\".\n", + "- `images_images_root_dir`: path after \"{dataset_id}/{source_id}/{batch_id}/images/{plate_id}/images/\" to the object. Is 'null' if object is no within that path. \n", + "- `images_images_aligned_root_dir`: \n", + "- `images_images_corrected_root_dir`: \n", + "- `images_images_corrected_cropped_root_dir`: \n", + "- `workspace_root_dir`: path after \"{dataset_id}/{source_id}/workspace/\" to the object. Is 'null' is object is not within that path.\n", + "- `analysis_root_dir`: \n", + "- `backend_root_dir`: \n", + "- `load_data_csv_root_dir`: path from \"load_data_csv\" to leaf node (?)\n", + "- `metadata_root_dir`: \n", + "- `profiles_root_dir`: \n", + "- `assaydev_root_dir`: \n", + "- `embeddings_root_dir`: \n", + "- `pipelines_root_dir`: \n", + "- `qc_root_dir`: \n", + "- `segmentation_root_dir`: \n", + "- `software_root_dir`: \n", + "- `workspace_dl_root_dir`: \n", + "- `collated_root_dir`: \n", + "- `consensus_root_dir`: \n", + "- `dl_embeddings_root_dir`: \n", + "- `dl_profiles_root_dir`: \n", + "- `sep`: \n", + "- `images`: is \"images\" if \"images\" is part of the key. Is \"null\" otherwise.\n", + "- `workspace`: is \"workspace\" if \"workspace\" is part of the key. Is \"null\" otherwise.\n", + "- `workspace_dl`: \n", + "- `dataset_id`: name of the dataset (e.g. \"cpg0016-jump\", \"cpg0021-periscope\", etc.)\n", + "- `source_id`: code for the source of the images (the institution who produced them)\n", + "- `batch_id`: batch number\n", + "- `plate_id`: unique plate identification code\n", + "- `well_id`: well position\n", + "- `site_id`: site number (sites are each of the fields of view imaged in a well)\n", + "- `well_site_id`: \n", + "- `plate_well_site_id`: \n", + "- `ml_model_id`: \n", + "- `leaf_node`: if the object is a file (\"is_dir\"==False), the name of the file. Otherwise, 'null'.\n", + "- `filename`: leaf node filename, without extension\n", + "- `extension`: leaf node extension\n", + "- `software_hash`: \n", + "- `software`: \n", + "- `hash`: \n", + "- `allowed_names`: \n", + "- `bucket`: \n", + "- `obj_key`: \n", + "- `size`: \n", + "- `last_modified_date`: \n", + "- `e_tag`: \n", + "- `storage_class`: \n", + "- `is_multipart_uploaded`: \n", + "- `replication_status`: \n", + "- `encryption_status`: \n", + "- `object_lock_retain_until_date`: \n", + "- `object_lock_mode`: \n", + "- `object_lock_legal_hold_status`: \n", + "- `intelligent_tiering_access_tier`: \n", + "- `bucket_key_status`: \n", + "- `checksum_algorithm`: \n", + "- `object_access_control_list`: \n", + "- `object_owner`: \n", + "- `is_parsing_error`: \n", + "- `errors`: \n", + "- `is_dir`: \n", + "- `key_parts`: \n", + "- `workspace_dir`: if within \"workspace\", which workspace dir is object related to ('profiles', 'load_data_csv', 'software', 'metadata', 'backend', 'quality_control', 'assaydev', 'analysis', 'pipelines'). Is 'None' if the object does not contain \"workspace\" in it's key or is in \n", + " \n", + "You can see all the Index column names and the type of data stored in each using `df.schema`.\n", + "\n", + "**Refer to CPG schema and use those same keys! check matching and come up with useful key maybe**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# dictionary of index col names and their respective data types\n", + "index.schema" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Example: use the index to download just a specific subset of files from the CPG" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will use **Polars** tools in the following examples to explore and filter the index.\n", + "You can find more info on the context and expressions in:\n", + "https://docs.pola.rs/user-guide/concepts/contexts/\n", + "https://docs.pola.rs/user-guide/concepts/expressions/" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Setting the maximum length of displayed strings to 500 helps to visualize the complete keys\n", + "pl.Config(fmt_str_lengths=500)\n", + "pl.Config.set_fmt_table_cell_list_len(500)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "#Pull the keys (file location within the bucket) of all 'Cells.csv' files from Source 4 in the JUMP dataset (cpg0016-jump)\n", + "\n", + "df = (\n", + " index\n", + " #Use filtering to get to the failing rows and not the other way around\n", + " .filter(pl.col(\"dataset_id\").eq(\"cpg0016-jump\")) \n", + " .filter(pl.col(\"source_id\").eq(\"source_4\"))\n", + " .filter(pl.col(\"leaf_node\").str.contains(\"Cells.csv\"))\n", + " \n", + " # Always add a `select` at the end of the chain and ONLY select for keys\n", + " .select(pl.col(\"key\",\"load_data_csv_root_dir\")) \n", + " \n", + " # Materialize this polars LazyFrame into a DataFrame.\n", + " .collect(streaming=True) # the streaming option prevents out of memory errors when loading big dataframes\n", + ")\n", + "\n", + "# print first 10 results\n", + "print(df.to_dicts()[0:10])\n", + "\n", + "# List the keys of the files to download\n", + "download_keys = list(df.to_dict()[\"key\"])\n", + "\n", + "# Choose a destination directory for the files\n", + "dest_dir = \"Path/To/Save/Your/files\"\n", + "\n", + "# Run a parallel command to dowload all files specified in the list of keys\n", + "parallel(download_keys, download_s3_files, [\"cellpainting-gallery\", Path(dest_dir)], jobs=20)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Further Tests and issues" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "## some useful structures for filtering and selecting\n", + "\n", + " # .filter(pl.col(\"images_images_root_dir\").is_in([\"2020_11_04_CPJUMP1\", \"2020_11_19_TimepointDay4\", \"2020_12_08_CPJUMP1_Bleaching\"])) \n", + " # .filter(pl.col(\"leaf_node\").str.contains(\"^.*(.tiff)\"))\n", + " # .filter(pl.col(\"well_id\").eq(\"E7\")) \n", + "\n", + " # .select(pl.col(\"well_id\").unique())\n", + " # .select(pl.col(\"*\").exclude([\"size\", \"is_multipart_uploaded\"]))\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "### There appear to be some issues when parsing well_id, particularly in the embedding.parquet files from sources 1, 2 and 7 of the JUMP dataset.\n", + "\n", + "df = (\n", + " index\n", + " .unique(subset=\"well_id\")\n", + " .filter(pl.col(\"is_parsing_error\").eq(False)) \n", + " .select(\"well_id\", \"key\", \"dataset_id\", \"source_id\", \"leaf_node\")\n", + " .unique(subset=[\"dataset_id\",\"leaf_node\",\"source_id\"])\n", + " .collect(streaming=True)\n", + " )\n", + "df" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge: write this validations:\n", + " - Is there a folder with illum files for every plate within raw images?\n", + " - Is there a load_data.csv for every plate, is there a load_data csv?" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Idea: compare total number of unique plates in raw images with number of unique load.csv files and illum folders\n", + "\n", + "Works fine fo just source 4 of JUMP but fails when applied bucket-wide" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# get total number of distinct plates within the raw images folder\n", + "df1 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(\"cpg0016-jump\"))\n", + " .filter(pl.col(\"source_id\").eq(\"source_4\"))\n", + " \n", + " .filter(pl.col(\"is_dir\").eq(True))\n", + " .filter(pl.col(\"images\").eq(\"images\"))\n", + " .filter(pl.col(\"images_images_root_dir\").is_not_null())\n", + " .filter(pl.col(\"dataset_id\").eq(\"jump\").not_()) #this dataset_id has parsing errors\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\"]) # gives me 1150 unique plates\n", + " # .unique(subset=[\"plate_id\"]) # gives me 1150 unique plates\n", + " .collect(streaming=True)\n", + " )\n", + "\n", + "\n", + "# get total number of load_data.csv files\n", + "df2 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(\"cpg0016-jump\"))\n", + " .filter(pl.col(\"source_id\").eq(\"source_4\"))\n", + " \n", + " .filter(pl.col(\"workspace\").eq(\"workspace\"))\n", + " .filter(pl.col(\"leaf_node\").eq(\"load_data.csv\"))\n", + " .filter(pl.col(\"dataset_id\").eq(\"jump\").not_()) #this dataset_id has parsing errors\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " # .unique(subset=[\"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\"]) #gives me 3478 unique combinations\n", + " .unique(subset=[\"plate_id\"]) # gives me 3652 unique plates if I don't filter the \"jump\" dataset_id because of parsing shenanigans\n", + " .collect(streaming=True)\n", + " )\n", + "\n", + "# get total number of illum/ folders\n", + "df3 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(\"cpg0016-jump\"))\n", + " .filter(pl.col(\"source_id\").eq(\"source_4\"))\n", + " \n", + " .filter(pl.col(\"is_dir\").eq(True))\n", + " .filter(pl.col(\"images\").eq(\"images\"))\n", + " .filter(pl.col(\"images_illum_root_dir\").is_not_null())\n", + " .filter(pl.col(\"dataset_id\").eq(\"jump\").not_()) #this dataset_id has parsing errors\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\"]) # gives me 2386 unique plates\n", + " # .unique(subset=[\"plate_id\"]) # gives me 2386 unique plates\n", + " .collect(streaming=True)\n", + " )\n", + "\n", + "print(f\"df1: {df1.shape}\")\n", + "print(f\"df2: {df2.shape}\")\n", + "print(f\"df3: {df3.shape}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 109, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "dev-cpg0016-jump, deflaux-workflow-test-BR00117012: number of plates and load_data.csv files MATCH\n", + "dev-cpg0016-jump, deflaux-workflow-test-BR00117012: number of plates and illum folders MATCH\n", + "cpg0011-lipocyteprofiler, broad: number of plates and load_data.csv files MATCH\n", + "cpg0011-lipocyteprofiler, broad: number of plates and illum folders MATCH\n", + "cpg0025-dactyloscopy, broad: number of plates and load_data.csv files MATCH\n", + "cpg0025-dactyloscopy, broad: number of plates and illum folders MATCH\n", + "dev-cpg0016-jump, deflaux-workflow-test2: number of plates and load_data.csv files MATCH\n", + "dev-cpg0016-jump, deflaux-workflow-test2: number of plates and illum folders MATCH\n", + "cpg0015-heterogeneity, broad: number of plates and load_data.csv files MATCH\n", + "cpg0015-heterogeneity, broad: number of plates and illum folders MATCH\n", + "cpg0016-jump-fixed, source_4: number of plates and load_data.csv files MATCH\n", + "cpg0016-jump-fixed, source_4: number of plates and illum folders MATCH\n", + "test-cpg0016-jump, source_4: number of plates and load_data.csv files MATCH\n", + "test-cpg0016-jump, source_4: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_5: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0016-jump, source_5: --ERROR-- number of plates and illum folders DON'T MATCH\n", + "cpg0023-mpi, mpi: number of plates and load_data.csv files MATCH\n", + "cpg0023-mpi, mpi: number of plates and illum folders MATCH\n", + "cpg0000-jump-pilot, source_4: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0000-jump-pilot, source_4: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_9: number of plates and load_data.csv files MATCH\n", + "cpg0016-jump, source_9: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_10: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0016-jump, source_10: --ERROR-- number of plates and illum folders DON'T MATCH\n", + "cpg0004-lincs, broad: number of plates and load_data.csv files MATCH\n", + "cpg0004-lincs, broad: number of plates and illum folders MATCH\n", + "cpg0031-caicedo-cmvip, broad: number of plates and load_data.csv files MATCH\n", + "cpg0031-caicedo-cmvip, broad: number of plates and illum folders MATCH\n", + "cpg0018-singh-seedseq, x: number of plates and load_data.csv files MATCH\n", + "cpg0018-singh-seedseq, x: number of plates and illum folders MATCH\n", + "cpg0003-rosetta, None: number of plates and load_data.csv files MATCH\n", + "cpg0003-rosetta, None: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_6: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0016-jump, source_6: --ERROR-- number of plates and illum folders DON'T MATCH\n", + "cpg0006-miami, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0006-miami, broad: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_1: number of plates and load_data.csv files MATCH\n", + "cpg0016-jump, source_1: --ERROR-- number of plates and illum folders DON'T MATCH\n", + "dev-cpg0016-jump, source_4: number of plates and load_data.csv files MATCH\n", + "dev-cpg0016-jump, source_4: number of plates and illum folders MATCH\n", + "cpg0016-jump-fixed, source_7: number of plates and load_data.csv files MATCH\n", + "cpg0016-jump-fixed, source_7: number of plates and illum folders MATCH\n", + "cpg0001-cellpainting-protocol, source_4: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0001-cellpainting-protocol, source_4: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_4: number of plates and load_data.csv files MATCH\n", + "cpg0016-jump, source_4: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_3: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0016-jump, source_3: --ERROR-- number of plates and illum folders DON'T MATCH\n", + "cpg0002-jump-scope, source_4: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0002-jump-scope, source_4: --ERROR-- number of plates and illum folders DON'T MATCH\n", + "cpg0016-jump, source_8: number of plates and load_data.csv files MATCH\n", + "cpg0016-jump, source_8: number of plates and illum folders MATCH\n", + "cpg0021-periscope, broad: number of plates and load_data.csv files MATCH\n", + "cpg0021-periscope, broad: number of plates and illum folders MATCH\n", + "cpg0028-kelley-resistance, broad: number of plates and load_data.csv files MATCH\n", + "cpg0028-kelley-resistance, broad: number of plates and illum folders MATCH\n", + "cpg0030-gustafsdottir-cellpainting, broad: number of plates and load_data.csv files MATCH\n", + "cpg0030-gustafsdottir-cellpainting, broad: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_2: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0016-jump, source_2: --ERROR-- number of plates and illum folders DON'T MATCH\n", + "cpg0003-rosetta, broad: number of plates and load_data.csv files MATCH\n", + "cpg0003-rosetta, broad: number of plates and illum folders MATCH\n", + "cpg0026-lacoste_haghighi-rare-diseases, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0026-lacoste_haghighi-rare-diseases, broad: --ERROR-- number of plates and illum folders DON'T MATCH\n", + "cpg0022-cmqtl, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0022-cmqtl, broad: number of plates and illum folders MATCH\n", + "cpg0017-rohban-pathways, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0017-rohban-pathways, broad: number of plates and illum folders MATCH\n", + "cpg0018-singh-seedseq, broad: number of plates and load_data.csv files MATCH\n", + "cpg0018-singh-seedseq, broad: number of plates and illum folders MATCH\n", + "cpg0020-varchamp, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0020-varchamp, broad: number of plates and illum folders MATCH\n", + "cpg0009-molglue, broad: number of plates and load_data.csv files MATCH\n", + "cpg0009-molglue, broad: number of plates and illum folders MATCH\n", + "cpg0024-bortezomib, source_4: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0024-bortezomib, source_4: number of plates and illum folders MATCH\n", + "dev-cpg0016-jump, deflaux-workflow-tests: number of plates and load_data.csv files MATCH\n", + "dev-cpg0016-jump, deflaux-workflow-tests: number of plates and illum folders MATCH\n", + "cpg0010-caie-drugresponse, broad-az: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0010-caie-drugresponse, broad-az: number of plates and illum folders MATCH\n", + "cpg0012-wawer-bioactivecompoundprofiling, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0012-wawer-bioactivecompoundprofiling, broad: number of plates and illum folders MATCH\n", + "cpg0016-jump-fixed, source_1: number of plates and load_data.csv files MATCH\n", + "cpg0016-jump-fixed, source_1: number of plates and illum folders MATCH\n", + "cpg0005-gerry-bioactivity, broad: number of plates and load_data.csv files MATCH\n", + "cpg0005-gerry-bioactivity, broad: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_11: number of plates and load_data.csv files MATCH\n", + "cpg0016-jump, source_11: number of plates and illum folders MATCH\n", + "cpg0019-moshkov-deepprofiler, broad: number of plates and load_data.csv files MATCH\n", + "cpg0019-moshkov-deepprofiler, broad: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_13: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0016-jump, source_13: --ERROR-- number of plates and illum folders DON'T MATCH\n", + "dev-cpg0016-jump, deflaux_test: number of plates and load_data.csv files MATCH\n", + "dev-cpg0016-jump, deflaux_test: number of plates and illum folders MATCH\n", + "jump, source_15: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "jump, source_15: number of plates and illum folders MATCH\n", + "cpg0014-jump-adipocyte, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0014-jump-adipocyte, broad: number of plates and illum folders MATCH\n", + "dev-cpg0016-jump, deflaux-workflow-test-BR00125638-B22: number of plates and load_data.csv files MATCH\n", + "dev-cpg0016-jump, deflaux-workflow-test-BR00125638-B22: number of plates and illum folders MATCH\n", + "cpg0016-jump, source_7: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", + "cpg0016-jump, source_7: --ERROR-- number of plates and illum folders DON'T MATCH\n" + ] + } + ], + "source": [ + "# Try to check within each dataset and source to find where there are missmatches between plates and load_data.csv or illum files\n", + "\n", + "df0 =(\n", + " index\n", + " .unique(subset=[\"dataset_id\",\"source_id\"])\n", + " .filter(pl.col(\"dataset_id\").is_not_null())\n", + " .select(pl.col(\"dataset_id\",\"source_id\"))\n", + " .collect(streaming=True)\n", + ")\n", + "\n", + "for dataset,source in zip(df0[\"dataset_id\"],df0[\"source_id\"]):\n", + " # get total number of distinct plates within the raw images folder\n", + " df1 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(dataset))\n", + " .filter(pl.col(\"source_id\").eq(source))\n", + " \n", + " .filter(pl.col(\"is_dir\").eq(True))\n", + " .filter(pl.col(\"images\").eq(\"images\"))\n", + " .filter(pl.col(\"images_images_root_dir\").is_not_null())\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"plate_id\"])\n", + " .collect(streaming=True)\n", + " )\n", + "\n", + "\n", + " # get total number of load_data.csv files\n", + " df2 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(dataset))\n", + " .filter(pl.col(\"source_id\").eq(source))\n", + " \n", + " .filter(pl.col(\"workspace\").eq(\"workspace\"))\n", + " .filter(pl.col(\"leaf_node\").eq(\"load_data.csv\"))\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"plate_id\"]) \n", + " .collect(streaming=True)\n", + " )\n", + "\n", + " # get total number of illum/ folders\n", + " df3 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(dataset))\n", + " .filter(pl.col(\"source_id\").eq(source))\n", + " \n", + " .filter(pl.col(\"is_dir\").eq(True))\n", + " .filter(pl.col(\"images\").eq(\"images\"))\n", + " .filter(pl.col(\"images_illum_root_dir\").is_not_null())\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"plate_id\"]) \n", + " .collect(streaming=True)\n", + " )\n", + "\n", + " if df1.shape[0]==df2.shape[0]:\n", + " print(f\"{dataset}, {source}: number of plates and load_data.csv files MATCH\")\n", + " else:\n", + " print(f\"{dataset}, {source}: --ERROR-- number of plates and load_data.csv files DON'T MATCH\")\n", + "\n", + " if df1.shape[0]==df3.shape[0]:\n", + " print(f\"{dataset}, {source}: number of plates and illum folders MATCH\")\n", + " else:\n", + " print(f\"{dataset}, {source}: --ERROR-- number of plates and illum folders DON'T MATCH\")" + ] + }, + { + "cell_type": "code", + "execution_count": 84, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "shape: (5, 5)
well_idkeydataset_idsource_idleaf_node
strstrstrstrstr
"UL001673""cpg0016-jump/source_1/workspace_dl/embeddings/efficientnet_v2_imagenet21k_s_feature_vector_2_0260bc96/Batch2_20221006/UL001673/UL001673/A02/embedding.parquet""cpg0016-jump""source_1""embedding.parquet"
"CP3-SC1-07""cpg0016-jump/source_7/workspace_dl/embeddings/efficientnet_v2_imagenet21k_s_feature_vector_2_0260bc96/20210727_Run3/CP3-SC1-07/CP3-SC1-07/A01/embedding.parquet""cpg0016-jump""source_7""embedding.parquet"
"M07""cpg0019-moshkov-deepprofiler/broad/workspace_dl/embeddings/105281_zenodo7114558/BBBC022/20585/M07/1/embedding.npz""cpg0019-moshkov-deepprofiler""broad""embedding.npz"
null"cpg0016-jump/source_8/""cpg0016-jump""source_8"null
"1086292259""cpg0016-jump/source_2/workspace_dl/embeddings/efficientnet_v2_imagenet21k_s_feature_vector_2_0260bc96/20210816_Batch_9/1086292259/1086292259/A01/embedding.parquet""cpg0016-jump""source_2""embedding.parquet"
" + ], + "text/plain": [ + "shape: (5, 5)\n", + "┌────────────┬──────────────────────────┬──────────────────────────┬───────────┬───────────────────┐\n", + "│ well_id ┆ key ┆ dataset_id ┆ source_id ┆ leaf_node │\n", + "│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", + "│ str ┆ str ┆ str ┆ str ┆ str │\n", + "╞════════════╪══════════════════════════╪══════════════════════════╪═══════════╪═══════════════════╡\n", + "│ UL001673 ┆ cpg0016-jump/source_1/wo ┆ cpg0016-jump ┆ source_1 ┆ embedding.parquet │\n", + "│ ┆ rkspace_dl/embeddings/ef ┆ ┆ ┆ │\n", + "│ ┆ ficientnet_v2_imagenet21 ┆ ┆ ┆ │\n", + "│ ┆ k_s_feature_vector_2_026 ┆ ┆ ┆ │\n", + "│ ┆ 0bc96/Batch2_20221006/UL ┆ ┆ ┆ │\n", + "│ ┆ 001673/UL001673/A02/embe ┆ ┆ ┆ │\n", + "│ ┆ dding.parquet ┆ ┆ ┆ │\n", + "│ CP3-SC1-07 ┆ cpg0016-jump/source_7/wo ┆ cpg0016-jump ┆ source_7 ┆ embedding.parquet │\n", + "│ ┆ rkspace_dl/embeddings/ef ┆ ┆ ┆ │\n", + "│ ┆ ficientnet_v2_imagenet21 ┆ ┆ ┆ │\n", + "│ ┆ k_s_feature_vector_2_026 ┆ ┆ ┆ │\n", + "│ ┆ 0bc96/20210727_Run3/CP3- ┆ ┆ ┆ │\n", + "│ ┆ SC1-07/CP3-SC1-07/A01/em ┆ ┆ ┆ │\n", + "│ ┆ bedding.parquet ┆ ┆ ┆ │\n", + "│ M07 ┆ cpg0019-moshkov-deepprof ┆ cpg0019-moshkov-deepprof ┆ broad ┆ embedding.npz │\n", + "│ ┆ iler/broad/workspace_dl/ ┆ iler ┆ ┆ │\n", + "│ ┆ embeddings/105281_zenodo ┆ ┆ ┆ │\n", + "│ ┆ 7114558/BBBC022/20585/M0 ┆ ┆ ┆ │\n", + "│ ┆ 7/1/embedding.npz ┆ ┆ ┆ │\n", + "│ null ┆ cpg0016-jump/source_8/ ┆ cpg0016-jump ┆ source_8 ┆ null │\n", + "│ 1086292259 ┆ cpg0016-jump/source_2/wo ┆ cpg0016-jump ┆ source_2 ┆ embedding.parquet │\n", + "│ ┆ rkspace_dl/embeddings/ef ┆ ┆ ┆ │\n", + "│ ┆ ficientnet_v2_imagenet21 ┆ ┆ ┆ │\n", + "│ ┆ k_s_feature_vector_2_026 ┆ ┆ ┆ │\n", + "│ ┆ 0bc96/20210816_Batch_9/1 ┆ ┆ ┆ │\n", + "│ ┆ 086292259/1086292259/A01 ┆ ┆ ┆ │\n", + "│ ┆ /embedding.parquet ┆ ┆ ┆ │\n", + "└────────────┴──────────────────────────┴──────────────────────────┴───────────┴───────────────────┘" + ] + }, + "execution_count": 84, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# There appear to be some issues when parsing well_id, particularly in the embedding.parquet files from sources 1, 2 and 7 of the JUMP dataset. \n", + "# The well_id listed in the index corresponds to the previous \"segment\" of the key.\n", + "\n", + "df = (\n", + " index\n", + " .unique(subset=\"well_id\")\n", + " .filter(pl.col(\"is_parsing_error\").eq(False)) \n", + " .select(\"well_id\", \"key\", \"dataset_id\", \"source_id\", \"leaf_node\")\n", + " .unique(subset=[\"dataset_id\",\"leaf_node\",\"source_id\"])\n", + " .collect(streaming=True)\n", + " )\n", + "df" + ] + }, + { + "cell_type": "code", + "execution_count": 110, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "shape: (0, 4)
dataset_idsource_idplate_idload_data_csv_root_dir
strstrstrstr
" + ], + "text/plain": [ + "shape: (0, 4)\n", + "┌────────────┬───────────┬──────────┬────────────────────────┐\n", + "│ dataset_id ┆ source_id ┆ plate_id ┆ load_data_csv_root_dir │\n", + "│ --- ┆ --- ┆ --- ┆ --- │\n", + "│ str ┆ str ┆ str ┆ str │\n", + "╞════════════╪═══════════╪══════════╪════════════════════════╡\n", + "└────────────┴───────────┴──────────┴────────────────────────┘" + ] + }, + "execution_count": 110, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "#24 plates of the source 15 in JUMP have a load_data.csv without having a load_data_csv_root_dir\n", + "# also, the dataset_id is 'jump' and not 'cpg0016-jump'\n", + "df = (index\n", + " .unique(subset=[\"dataset_id\",\"plate_id\"])\n", + " .filter(pl.col(\"leaf_node\").eq(\"load_data.csv\"))\n", + " .select(pl.col([\"dataset_id\",\"source_id\",\"plate_id\",\"load_data_csv_root_dir\"]))\n", + " .filter(pl.col(\"load_data_csv_root_dir\").is_null()) #https://docs.pola.rs/py-polars/html/reference/dataframe/api/polars.DataFrame.drop_nulls.html\n", + " .collect(streaming=True)\n", + " )\n", + "df" + ] + }, + { + "cell_type": "code", + "execution_count": 111, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "shape: (460, 1)
plate_id
str
"PEC00001815__2022-02-23T18_03_17-Measurement3"
"PEP00004102__2021-12-02T12_05_47-Measurement1"
"PEP00004137__2021-12-01T16_50_57-Measurement1"
"PEP00004139__2021-12-03T05_02_12-Measurement1"
"PEC00001790"
"PEP00004092"
"PEP00004102"
"PEC00001785__2021-12-07T01_54_26-Measurement1"
"PEP00004041__2021-12-16T04_45_56-Measurement1"
"PEC00001854__2021-12-17T13_35_15-Measurement1"
"PEP00004072"
"PEP00004065__2021-12-13T12_34_12-Measurement1"
…
"PEC00001842__2021-12-17T04_56_37-Measurement1"
"PEP00004049__2022-02-22T10_42_29-Measurement1"
"PEC00001837"
"PEP00004026__2021-12-15T18_17_27-Measurement1"
"PEP00004023__2022-02-23T16_16_10-Measurement2"
"PEC00001858__2021-12-17T10_07_06-Measurement1"
"PEC00001863__2022-01-18T15_18_04-Measurement1"
"PEC00001805__2022-02-24T19_35_58-Measurement2"
"PEP00004326__2022-03-01T06_12_25-Measurement1"
"PEC00001835"
null
"PEP00004136"
" + ], + "text/plain": [ + "shape: (460, 1)\n", + "┌───────────────────────────────────────────────┐\n", + "│ plate_id │\n", + "│ --- │\n", + "│ str │\n", + "╞═══════════════════════════════════════════════╡\n", + "│ PEC00001815__2022-02-23T18_03_17-Measurement3 │\n", + "│ PEP00004102__2021-12-02T12_05_47-Measurement1 │\n", + "│ PEP00004137__2021-12-01T16_50_57-Measurement1 │\n", + "│ PEP00004139__2021-12-03T05_02_12-Measurement1 │\n", + "│ … │\n", + "│ PEP00004326__2022-03-01T06_12_25-Measurement1 │\n", + "│ PEC00001835 │\n", + "│ null │\n", + "│ PEP00004136 │\n", + "└───────────────────────────────────────────────┘" + ] + }, + "execution_count": 111, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# There are 460 plate_ids for source 15 in JUMP, are there really 460 plates? \n", + "# also, plate_id varies in structure!\n", + "df = (index\n", + " .filter(pl.col(\"dataset_id\").eq(\"jump\"))\n", + " .filter(pl.col(\"source_id\").eq(\"source_15\"))\n", + " .unique(subset=[\"plate_id\"])\n", + " .select(pl.col([\"plate_id\"]))\n", + " .collect(streaming=True)\n", + " )\n", + "df" + ] + }, + { + "cell_type": "code", + "execution_count": 112, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "shape: (183, 1)
plate_id
str
"PEC00001782"
"PEC00001783"
"PEC00001784"
"PEC00001785"
"PEC00001786"
"PEC00001787"
"PEC00001788"
"PEC00001789"
"PEC00001790"
"PEC00001791"
"PEC00001792"
"PEC00001793"
…
"PEP00004335"
"PEP00004421"
"PEP00004422"
"PEP00004423"
"PEP00004425"
"PEP00004426"
"PEP00004427"
"PEP00004430"
"PEP00004431"
"PEP00004432"
"PEP00004457"
"PEP00004458"
" + ], + "text/plain": [ + "shape: (183, 1)\n", + "┌─────────────┐\n", + "│ plate_id │\n", + "│ --- │\n", + "│ str │\n", + "╞═════════════╡\n", + "│ PEC00001782 │\n", + "│ PEC00001783 │\n", + "│ PEC00001784 │\n", + "│ PEC00001785 │\n", + "│ … │\n", + "│ PEP00004431 │\n", + "│ PEP00004432 │\n", + "│ PEP00004457 │\n", + "│ PEP00004458 │\n", + "└─────────────┘" + ] + }, + "execution_count": 112, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# For source 15 in JUMP, are there really 460 plates? \n", + "# There are only 183 unique ones matching the regex for the plate name structure\n", + "df = (index\n", + " .filter(pl.col(\"dataset_id\").eq(\"jump\"))\n", + " .filter(pl.col(\"source_id\").eq(\"source_15\"))\n", + " # .unique(subset=[\"plate_id\"])\n", + " .filter(pl.col(\"plate_id\").str.contains(\"^PE(P|C)[0-9]{8}$\"))\n", + " .select(pl.col([\"plate_id\"]).unique().sort())\n", + " .collect(streaming=True)\n", + " )\n", + "df" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "jupy310", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From c4c3b8e8fdc9b77250cc00809d453e355f27032a Mon Sep 17 00:00:00 2001 From: emiglietta Date: Wed, 1 May 2024 16:03:42 -0700 Subject: [PATCH 2/4] update documentation and tests --- CPGvalidator_docs.ipynb | 912 ++++++++++++++++++++++++++++++++++------ 1 file changed, 793 insertions(+), 119 deletions(-) diff --git a/CPGvalidator_docs.ipynb b/CPGvalidator_docs.ipynb index b93bb63..c15aa3d 100644 --- a/CPGvalidator_docs.ipynb +++ b/CPGvalidator_docs.ipynb @@ -442,120 +442,14 @@ }, { "cell_type": "code", - "execution_count": 109, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "dev-cpg0016-jump, deflaux-workflow-test-BR00117012: number of plates and load_data.csv files MATCH\n", - "dev-cpg0016-jump, deflaux-workflow-test-BR00117012: number of plates and illum folders MATCH\n", - "cpg0011-lipocyteprofiler, broad: number of plates and load_data.csv files MATCH\n", - "cpg0011-lipocyteprofiler, broad: number of plates and illum folders MATCH\n", - "cpg0025-dactyloscopy, broad: number of plates and load_data.csv files MATCH\n", - "cpg0025-dactyloscopy, broad: number of plates and illum folders MATCH\n", - "dev-cpg0016-jump, deflaux-workflow-test2: number of plates and load_data.csv files MATCH\n", - "dev-cpg0016-jump, deflaux-workflow-test2: number of plates and illum folders MATCH\n", - "cpg0015-heterogeneity, broad: number of plates and load_data.csv files MATCH\n", - "cpg0015-heterogeneity, broad: number of plates and illum folders MATCH\n", - "cpg0016-jump-fixed, source_4: number of plates and load_data.csv files MATCH\n", - "cpg0016-jump-fixed, source_4: number of plates and illum folders MATCH\n", - "test-cpg0016-jump, source_4: number of plates and load_data.csv files MATCH\n", - "test-cpg0016-jump, source_4: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_5: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0016-jump, source_5: --ERROR-- number of plates and illum folders DON'T MATCH\n", - "cpg0023-mpi, mpi: number of plates and load_data.csv files MATCH\n", - "cpg0023-mpi, mpi: number of plates and illum folders MATCH\n", - "cpg0000-jump-pilot, source_4: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0000-jump-pilot, source_4: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_9: number of plates and load_data.csv files MATCH\n", - "cpg0016-jump, source_9: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_10: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0016-jump, source_10: --ERROR-- number of plates and illum folders DON'T MATCH\n", - "cpg0004-lincs, broad: number of plates and load_data.csv files MATCH\n", - "cpg0004-lincs, broad: number of plates and illum folders MATCH\n", - "cpg0031-caicedo-cmvip, broad: number of plates and load_data.csv files MATCH\n", - "cpg0031-caicedo-cmvip, broad: number of plates and illum folders MATCH\n", - "cpg0018-singh-seedseq, x: number of plates and load_data.csv files MATCH\n", - "cpg0018-singh-seedseq, x: number of plates and illum folders MATCH\n", - "cpg0003-rosetta, None: number of plates and load_data.csv files MATCH\n", - "cpg0003-rosetta, None: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_6: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0016-jump, source_6: --ERROR-- number of plates and illum folders DON'T MATCH\n", - "cpg0006-miami, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0006-miami, broad: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_1: number of plates and load_data.csv files MATCH\n", - "cpg0016-jump, source_1: --ERROR-- number of plates and illum folders DON'T MATCH\n", - "dev-cpg0016-jump, source_4: number of plates and load_data.csv files MATCH\n", - "dev-cpg0016-jump, source_4: number of plates and illum folders MATCH\n", - "cpg0016-jump-fixed, source_7: number of plates and load_data.csv files MATCH\n", - "cpg0016-jump-fixed, source_7: number of plates and illum folders MATCH\n", - "cpg0001-cellpainting-protocol, source_4: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0001-cellpainting-protocol, source_4: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_4: number of plates and load_data.csv files MATCH\n", - "cpg0016-jump, source_4: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_3: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0016-jump, source_3: --ERROR-- number of plates and illum folders DON'T MATCH\n", - "cpg0002-jump-scope, source_4: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0002-jump-scope, source_4: --ERROR-- number of plates and illum folders DON'T MATCH\n", - "cpg0016-jump, source_8: number of plates and load_data.csv files MATCH\n", - "cpg0016-jump, source_8: number of plates and illum folders MATCH\n", - "cpg0021-periscope, broad: number of plates and load_data.csv files MATCH\n", - "cpg0021-periscope, broad: number of plates and illum folders MATCH\n", - "cpg0028-kelley-resistance, broad: number of plates and load_data.csv files MATCH\n", - "cpg0028-kelley-resistance, broad: number of plates and illum folders MATCH\n", - "cpg0030-gustafsdottir-cellpainting, broad: number of plates and load_data.csv files MATCH\n", - "cpg0030-gustafsdottir-cellpainting, broad: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_2: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0016-jump, source_2: --ERROR-- number of plates and illum folders DON'T MATCH\n", - "cpg0003-rosetta, broad: number of plates and load_data.csv files MATCH\n", - "cpg0003-rosetta, broad: number of plates and illum folders MATCH\n", - "cpg0026-lacoste_haghighi-rare-diseases, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0026-lacoste_haghighi-rare-diseases, broad: --ERROR-- number of plates and illum folders DON'T MATCH\n", - "cpg0022-cmqtl, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0022-cmqtl, broad: number of plates and illum folders MATCH\n", - "cpg0017-rohban-pathways, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0017-rohban-pathways, broad: number of plates and illum folders MATCH\n", - "cpg0018-singh-seedseq, broad: number of plates and load_data.csv files MATCH\n", - "cpg0018-singh-seedseq, broad: number of plates and illum folders MATCH\n", - "cpg0020-varchamp, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0020-varchamp, broad: number of plates and illum folders MATCH\n", - "cpg0009-molglue, broad: number of plates and load_data.csv files MATCH\n", - "cpg0009-molglue, broad: number of plates and illum folders MATCH\n", - "cpg0024-bortezomib, source_4: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0024-bortezomib, source_4: number of plates and illum folders MATCH\n", - "dev-cpg0016-jump, deflaux-workflow-tests: number of plates and load_data.csv files MATCH\n", - "dev-cpg0016-jump, deflaux-workflow-tests: number of plates and illum folders MATCH\n", - "cpg0010-caie-drugresponse, broad-az: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0010-caie-drugresponse, broad-az: number of plates and illum folders MATCH\n", - "cpg0012-wawer-bioactivecompoundprofiling, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0012-wawer-bioactivecompoundprofiling, broad: number of plates and illum folders MATCH\n", - "cpg0016-jump-fixed, source_1: number of plates and load_data.csv files MATCH\n", - "cpg0016-jump-fixed, source_1: number of plates and illum folders MATCH\n", - "cpg0005-gerry-bioactivity, broad: number of plates and load_data.csv files MATCH\n", - "cpg0005-gerry-bioactivity, broad: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_11: number of plates and load_data.csv files MATCH\n", - "cpg0016-jump, source_11: number of plates and illum folders MATCH\n", - "cpg0019-moshkov-deepprofiler, broad: number of plates and load_data.csv files MATCH\n", - "cpg0019-moshkov-deepprofiler, broad: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_13: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0016-jump, source_13: --ERROR-- number of plates and illum folders DON'T MATCH\n", - "dev-cpg0016-jump, deflaux_test: number of plates and load_data.csv files MATCH\n", - "dev-cpg0016-jump, deflaux_test: number of plates and illum folders MATCH\n", - "jump, source_15: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "jump, source_15: number of plates and illum folders MATCH\n", - "cpg0014-jump-adipocyte, broad: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0014-jump-adipocyte, broad: number of plates and illum folders MATCH\n", - "dev-cpg0016-jump, deflaux-workflow-test-BR00125638-B22: number of plates and load_data.csv files MATCH\n", - "dev-cpg0016-jump, deflaux-workflow-test-BR00125638-B22: number of plates and illum folders MATCH\n", - "cpg0016-jump, source_7: --ERROR-- number of plates and load_data.csv files DON'T MATCH\n", - "cpg0016-jump, source_7: --ERROR-- number of plates and illum folders DON'T MATCH\n" - ] - } - ], + "outputs": [], "source": [ "# Try to check within each dataset and source to find where there are missmatches between plates and load_data.csv or illum files\n", + "import pandas as pd\n", + "\n", + "result = pd.DataFrame(columns=['dataset_id', 'source_id', '#plates','#load_data_csvs','#illum_folders','plates-and-load_data_csvs-match', 'plates-and-illum_folders-match'])\n", "\n", "df0 =(\n", " index\n", @@ -608,15 +502,795 @@ " .collect(streaming=True)\n", " )\n", "\n", - " if df1.shape[0]==df2.shape[0]:\n", - " print(f\"{dataset}, {source}: number of plates and load_data.csv files MATCH\")\n", - " else:\n", - " print(f\"{dataset}, {source}: --ERROR-- number of plates and load_data.csv files DON'T MATCH\")\n", + " # Evaluate if number of plates matches with number of load_data_csv and illum folders\n", + " matchLoadData= df1.shape[0]==df2.shape[0]\n", + " matchIllum = df1.shape[0]==df3.shape[0]\n", "\n", - " if df1.shape[0]==df3.shape[0]:\n", - " print(f\"{dataset}, {source}: number of plates and illum folders MATCH\")\n", - " else:\n", - " print(f\"{dataset}, {source}: --ERROR-- number of plates and illum folders DON'T MATCH\")" + " # Collect results in the results df\n", + " new_row_data = {'dataset_id': dataset, \n", + " 'source_id': source, \n", + " '#plates': len(df1),\n", + " '#load_data_csvs': len(df2),\n", + " '#illum_folders': len(df3),\n", + " 'plates-and-load_data_csvs-match': matchLoadData, \n", + " 'plates-and-illum_folders-match': matchIllum\n", + " }\n", + " result = pd.concat([result, pd.DataFrame([new_row_data])], ignore_index=True)\n", + "\n", + "result" + ] + }, + { + "cell_type": "code", + "execution_count": 127, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
dataset_idsource_id#plates#load_data_csvs#illum_foldersplates-and-load_data_csvs-matchplates-and-illum_folders-match
0cpg0015-heterogeneitybroad000TrueTrue
1cpg0016-jumpsource_9108108108TrueTrue
2cpg0004-lincsbroad000TrueTrue
3cpg0016-jumpsource_50260260FalseFalse
4cpg0000-jump-pilotsource_40750FalseTrue
5cpg0019-moshkov-deepprofilerbroad000TrueTrue
6cpg0031-caicedo-cmvipbroad000TrueTrue
7cpg0016-jump-fixedsource_1000TrueTrue
8cpg0011-lipocyteprofilerbroad000TrueTrue
9cpg0028-kelley-resistancebroad000TrueTrue
10cpg0002-jump-scopesource_41445FalseFalse
11cpg0025-dactyloscopybroad000TrueTrue
12cpg0022-cmqtlbroad0120FalseTrue
13dev-cpg0016-jumpdeflaux-workflow-test-BR00117012000TrueTrue
14dev-cpg0016-jumpdeflaux-workflow-test-BR00125638-B22000TrueTrue
15cpg0018-singh-seedseqbroad000TrueTrue
16cpg0018-singh-seedseqx000TrueTrue
17cpg0016-jumpsource_130154154FalseFalse
18cpg0024-bortezomibsource_4040FalseTrue
19cpg0003-rosettaNone000TrueTrue
20cpg0016-jump-fixedsource_7000TrueTrue
21cpg0016-jumpsource_100222222FalseFalse
22dev-cpg0016-jumpsource_4000TrueTrue
23cpg0021-periscopebroad000TrueTrue
24cpg0020-varchampbroad0570FalseTrue
25cpg0003-rosettabroad000TrueTrue
26cpg0016-jumpsource_11182182182TrueTrue
27cpg0010-caie-drugresponsebroad-az0550FalseTrue
28cpg0017-rohban-pathwaysbroad050FalseTrue
29cpg0016-jumpsource_8216216216TrueTrue
30cpg0012-wawer-bioactivecompoundprofilingbroad04050FalseTrue
31cpg0016-jump-fixedsource_4000TrueTrue
32dev-cpg0016-jumpdeflaux_test000TrueTrue
33cpg0016-jumpsource_60246246FalseFalse
34jumpsource_1501740FalseTrue
35cpg0001-cellpainting-protocolsource_401080FalseTrue
36dev-cpg0016-jumpdeflaux-workflow-test2000TrueTrue
37test-cpg0016-jumpsource_4000TrueTrue
38cpg0030-gustafsdottir-cellpaintingbroad000TrueTrue
39cpg0016-jumpsource_20229229FalseFalse
40cpg0014-jump-adipocytebroad01380FalseTrue
41cpg0016-jumpsource_4277277277TrueTrue
42cpg0006-miamibroad0100FalseTrue
43cpg0026-lacoste_haghighi-rare-diseasesbroad11850FalseFalse
44cpg0016-jumpsource_70128128FalseFalse
45cpg0009-molgluebroad000TrueTrue
46cpg0023-mpimpi000TrueTrue
47cpg0005-gerry-bioactivitybroad000TrueTrue
48dev-cpg0016-jumpdeflaux-workflow-tests000TrueTrue
49cpg0016-jumpsource_3310303303FalseFalse
50cpg0016-jumpsource_1555556TrueFalse
\n", + "
" + ], + "text/plain": [ + " dataset_id \\\n", + "0 cpg0015-heterogeneity \n", + "1 cpg0016-jump \n", + "2 cpg0004-lincs \n", + "3 cpg0016-jump \n", + "4 cpg0000-jump-pilot \n", + "5 cpg0019-moshkov-deepprofiler \n", + "6 cpg0031-caicedo-cmvip \n", + "7 cpg0016-jump-fixed \n", + "8 cpg0011-lipocyteprofiler \n", + "9 cpg0028-kelley-resistance \n", + "10 cpg0002-jump-scope \n", + "11 cpg0025-dactyloscopy \n", + "12 cpg0022-cmqtl \n", + "13 dev-cpg0016-jump \n", + "14 dev-cpg0016-jump \n", + "15 cpg0018-singh-seedseq \n", + "16 cpg0018-singh-seedseq \n", + "17 cpg0016-jump \n", + "18 cpg0024-bortezomib \n", + "19 cpg0003-rosetta \n", + "20 cpg0016-jump-fixed \n", + "21 cpg0016-jump \n", + "22 dev-cpg0016-jump \n", + "23 cpg0021-periscope \n", + "24 cpg0020-varchamp \n", + "25 cpg0003-rosetta \n", + "26 cpg0016-jump \n", + "27 cpg0010-caie-drugresponse \n", + "28 cpg0017-rohban-pathways \n", + "29 cpg0016-jump \n", + "30 cpg0012-wawer-bioactivecompoundprofiling \n", + "31 cpg0016-jump-fixed \n", + "32 dev-cpg0016-jump \n", + "33 cpg0016-jump \n", + "34 jump \n", + "35 cpg0001-cellpainting-protocol \n", + "36 dev-cpg0016-jump \n", + "37 test-cpg0016-jump \n", + "38 cpg0030-gustafsdottir-cellpainting \n", + "39 cpg0016-jump \n", + "40 cpg0014-jump-adipocyte \n", + "41 cpg0016-jump \n", + "42 cpg0006-miami \n", + "43 cpg0026-lacoste_haghighi-rare-diseases \n", + "44 cpg0016-jump \n", + "45 cpg0009-molglue \n", + "46 cpg0023-mpi \n", + "47 cpg0005-gerry-bioactivity \n", + "48 dev-cpg0016-jump \n", + "49 cpg0016-jump \n", + "50 cpg0016-jump \n", + "\n", + " source_id #plates #load_data_csvs \\\n", + "0 broad 0 0 \n", + "1 source_9 108 108 \n", + "2 broad 0 0 \n", + "3 source_5 0 260 \n", + "4 source_4 0 75 \n", + "5 broad 0 0 \n", + "6 broad 0 0 \n", + "7 source_1 0 0 \n", + "8 broad 0 0 \n", + "9 broad 0 0 \n", + "10 source_4 1 44 \n", + "11 broad 0 0 \n", + "12 broad 0 12 \n", + "13 deflaux-workflow-test-BR00117012 0 0 \n", + "14 deflaux-workflow-test-BR00125638-B22 0 0 \n", + "15 broad 0 0 \n", + "16 x 0 0 \n", + "17 source_13 0 154 \n", + "18 source_4 0 4 \n", + "19 None 0 0 \n", + "20 source_7 0 0 \n", + "21 source_10 0 222 \n", + "22 source_4 0 0 \n", + "23 broad 0 0 \n", + "24 broad 0 57 \n", + "25 broad 0 0 \n", + "26 source_11 182 182 \n", + "27 broad-az 0 55 \n", + "28 broad 0 5 \n", + "29 source_8 216 216 \n", + "30 broad 0 405 \n", + "31 source_4 0 0 \n", + "32 deflaux_test 0 0 \n", + "33 source_6 0 246 \n", + "34 source_15 0 174 \n", + "35 source_4 0 108 \n", + "36 deflaux-workflow-test2 0 0 \n", + "37 source_4 0 0 \n", + "38 broad 0 0 \n", + "39 source_2 0 229 \n", + "40 broad 0 138 \n", + "41 source_4 277 277 \n", + "42 broad 0 10 \n", + "43 broad 1 185 \n", + "44 source_7 0 128 \n", + "45 broad 0 0 \n", + "46 mpi 0 0 \n", + "47 broad 0 0 \n", + "48 deflaux-workflow-tests 0 0 \n", + "49 source_3 310 303 \n", + "50 source_1 55 55 \n", + "\n", + " #illum_folders plates-and-load_data_csvs-match \\\n", + "0 0 True \n", + "1 108 True \n", + "2 0 True \n", + "3 260 False \n", + "4 0 False \n", + "5 0 True \n", + "6 0 True \n", + "7 0 True \n", + "8 0 True \n", + "9 0 True \n", + "10 5 False \n", + "11 0 True \n", + "12 0 False \n", + "13 0 True \n", + "14 0 True \n", + "15 0 True \n", + "16 0 True \n", + "17 154 False \n", + "18 0 False \n", + "19 0 True \n", + "20 0 True \n", + "21 222 False \n", + "22 0 True \n", + "23 0 True \n", + "24 0 False \n", + "25 0 True \n", + "26 182 True \n", + "27 0 False \n", + "28 0 False \n", + "29 216 True \n", + "30 0 False \n", + "31 0 True \n", + "32 0 True \n", + "33 246 False \n", + "34 0 False \n", + "35 0 False \n", + "36 0 True \n", + "37 0 True \n", + "38 0 True \n", + "39 229 False \n", + "40 0 False \n", + "41 277 True \n", + "42 0 False \n", + "43 0 False \n", + "44 128 False \n", + "45 0 True \n", + "46 0 True \n", + "47 0 True \n", + "48 0 True \n", + "49 303 False \n", + "50 56 True \n", + "\n", + " plates-and-illum_folders-match \n", + "0 True \n", + "1 True \n", + "2 True \n", + "3 False \n", + "4 True \n", + "5 True \n", + "6 True \n", + "7 True \n", + "8 True \n", + "9 True \n", + "10 False \n", + "11 True \n", + "12 True \n", + "13 True \n", + "14 True \n", + "15 True \n", + "16 True \n", + "17 False \n", + "18 True \n", + "19 True \n", + "20 True \n", + "21 False \n", + "22 True \n", + "23 True \n", + "24 True \n", + "25 True \n", + "26 True \n", + "27 True \n", + "28 True \n", + "29 True \n", + "30 True \n", + "31 True \n", + "32 True \n", + "33 False \n", + "34 True \n", + "35 True \n", + "36 True \n", + "37 True \n", + "38 True \n", + "39 False \n", + "40 True \n", + "41 True \n", + "42 True \n", + "43 False \n", + "44 False \n", + "45 True \n", + "46 True \n", + "47 True \n", + "48 True \n", + "49 False \n", + "50 False " + ] + }, + "execution_count": 127, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result" ] }, { From b5bcb74c4d005e3c3e2b37a95719c7119145f90a Mon Sep 17 00:00:00 2001 From: emiglietta Date: Mon, 6 May 2024 13:22:59 -0700 Subject: [PATCH 3/4] Update CPGvalidator_docs.ipynb --- CPGvalidator_docs.ipynb | 882 ++++++++++++++++++++-------------------- 1 file changed, 442 insertions(+), 440 deletions(-) diff --git a/CPGvalidator_docs.ipynb b/CPGvalidator_docs.ipynb index c15aa3d..8947fc3 100644 --- a/CPGvalidator_docs.ipynb +++ b/CPGvalidator_docs.ipynb @@ -123,7 +123,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": {}, "outputs": [], "source": [ @@ -151,7 +151,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": {}, "outputs": [], "source": [ @@ -277,9 +277,20 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "polars.config.Config" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# Setting the maximum length of displayed strings to 500 helps to visualize the complete keys\n", "pl.Config(fmt_str_lengths=500)\n", @@ -442,87 +453,7 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Try to check within each dataset and source to find where there are missmatches between plates and load_data.csv or illum files\n", - "import pandas as pd\n", - "\n", - "result = pd.DataFrame(columns=['dataset_id', 'source_id', '#plates','#load_data_csvs','#illum_folders','plates-and-load_data_csvs-match', 'plates-and-illum_folders-match'])\n", - "\n", - "df0 =(\n", - " index\n", - " .unique(subset=[\"dataset_id\",\"source_id\"])\n", - " .filter(pl.col(\"dataset_id\").is_not_null())\n", - " .select(pl.col(\"dataset_id\",\"source_id\"))\n", - " .collect(streaming=True)\n", - ")\n", - "\n", - "for dataset,source in zip(df0[\"dataset_id\"],df0[\"source_id\"]):\n", - " # get total number of distinct plates within the raw images folder\n", - " df1 = (\n", - " index\n", - " .filter(pl.col(\"dataset_id\").eq(dataset))\n", - " .filter(pl.col(\"source_id\").eq(source))\n", - " \n", - " .filter(pl.col(\"is_dir\").eq(True))\n", - " .filter(pl.col(\"images\").eq(\"images\"))\n", - " .filter(pl.col(\"images_images_root_dir\").is_not_null())\n", - " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", - " .unique(subset=[\"plate_id\"])\n", - " .collect(streaming=True)\n", - " )\n", - "\n", - "\n", - " # get total number of load_data.csv files\n", - " df2 = (\n", - " index\n", - " .filter(pl.col(\"dataset_id\").eq(dataset))\n", - " .filter(pl.col(\"source_id\").eq(source))\n", - " \n", - " .filter(pl.col(\"workspace\").eq(\"workspace\"))\n", - " .filter(pl.col(\"leaf_node\").eq(\"load_data.csv\"))\n", - " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", - " .unique(subset=[\"plate_id\"]) \n", - " .collect(streaming=True)\n", - " )\n", - "\n", - " # get total number of illum/ folders\n", - " df3 = (\n", - " index\n", - " .filter(pl.col(\"dataset_id\").eq(dataset))\n", - " .filter(pl.col(\"source_id\").eq(source))\n", - " \n", - " .filter(pl.col(\"is_dir\").eq(True))\n", - " .filter(pl.col(\"images\").eq(\"images\"))\n", - " .filter(pl.col(\"images_illum_root_dir\").is_not_null())\n", - " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", - " .unique(subset=[\"plate_id\"]) \n", - " .collect(streaming=True)\n", - " )\n", - "\n", - " # Evaluate if number of plates matches with number of load_data_csv and illum folders\n", - " matchLoadData= df1.shape[0]==df2.shape[0]\n", - " matchIllum = df1.shape[0]==df3.shape[0]\n", - "\n", - " # Collect results in the results df\n", - " new_row_data = {'dataset_id': dataset, \n", - " 'source_id': source, \n", - " '#plates': len(df1),\n", - " '#load_data_csvs': len(df2),\n", - " '#illum_folders': len(df3),\n", - " 'plates-and-load_data_csvs-match': matchLoadData, \n", - " 'plates-and-illum_folders-match': matchIllum\n", - " }\n", - " result = pd.concat([result, pd.DataFrame([new_row_data])], ignore_index=True)\n", - "\n", - "result" - ] - }, - { - "cell_type": "code", - "execution_count": 127, + "execution_count": 4, "metadata": {}, "outputs": [ { @@ -558,8 +489,8 @@ " \n", " \n", " 0\n", - " cpg0015-heterogeneity\n", - " broad\n", + " cpg0003-rosetta\n", + " None\n", " 0\n", " 0\n", " 0\n", @@ -568,18 +499,18 @@ " \n", " \n", " 1\n", - " cpg0016-jump\n", - " source_9\n", - " 108\n", - " 108\n", - " 108\n", + " cpg0028-kelley-resistance\n", + " broad\n", + " 0\n", + " 0\n", + " 0\n", " True\n", " True\n", " \n", " \n", " 2\n", - " cpg0004-lincs\n", - " broad\n", + " dev-cpg0016-jump\n", + " deflaux-workflow-tests\n", " 0\n", " 0\n", " 0\n", @@ -589,36 +520,36 @@ " \n", " 3\n", " cpg0016-jump\n", - " source_5\n", - " 0\n", - " 260\n", - " 260\n", - " False\n", + " source_1\n", + " 55\n", + " 55\n", + " 56\n", + " True\n", " False\n", " \n", " \n", " 4\n", - " cpg0000-jump-pilot\n", - " source_4\n", + " dev-cpg0016-jump\n", + " deflaux-workflow-test-BR00117012\n", " 0\n", - " 75\n", " 0\n", - " False\n", + " 0\n", + " True\n", " True\n", " \n", " \n", " 5\n", - " cpg0019-moshkov-deepprofiler\n", - " broad\n", - " 0\n", - " 0\n", + " cpg0016-jump\n", + " source_7\n", " 0\n", - " True\n", - " True\n", + " 128\n", + " 128\n", + " False\n", + " False\n", " \n", " \n", " 6\n", - " cpg0031-caicedo-cmvip\n", + " cpg0018-singh-seedseq\n", " broad\n", " 0\n", " 0\n", @@ -629,7 +560,7 @@ " \n", " 7\n", " cpg0016-jump-fixed\n", - " source_1\n", + " source_4\n", " 0\n", " 0\n", " 0\n", @@ -638,17 +569,27 @@ " \n", " \n", " 8\n", - " cpg0011-lipocyteprofiler\n", - " broad\n", - " 0\n", - " 0\n", - " 0\n", + " cpg0002-jump-scope\n", + " source_4\n", + " 1\n", + " 44\n", + " 5\n", + " False\n", + " False\n", + " \n", + " \n", + " 9\n", + " cpg0016-jump\n", + " source_11\n", + " 182\n", + " 182\n", + " 182\n", " True\n", " True\n", " \n", " \n", - " 9\n", - " cpg0028-kelley-resistance\n", + " 10\n", + " cpg0009-molglue\n", " broad\n", " 0\n", " 0\n", @@ -657,18 +598,18 @@ " True\n", " \n", " \n", - " 10\n", - " cpg0002-jump-scope\n", - " source_4\n", - " 1\n", - " 44\n", - " 5\n", + " 11\n", + " cpg0016-jump\n", + " source_2\n", + " 0\n", + " 229\n", + " 229\n", " False\n", " False\n", " \n", " \n", - " 11\n", - " cpg0025-dactyloscopy\n", + " 12\n", + " cpg0019-moshkov-deepprofiler\n", " broad\n", " 0\n", " 0\n", @@ -677,38 +618,28 @@ " True\n", " \n", " \n", - " 12\n", - " cpg0022-cmqtl\n", + " 13\n", + " cpg0020-varchamp\n", " broad\n", " 0\n", - " 12\n", + " 57\n", " 0\n", " False\n", " True\n", " \n", " \n", - " 13\n", - " dev-cpg0016-jump\n", - " deflaux-workflow-test-BR00117012\n", - " 0\n", - " 0\n", - " 0\n", - " True\n", - " True\n", - " \n", - " \n", " 14\n", - " dev-cpg0016-jump\n", - " deflaux-workflow-test-BR00125638-B22\n", - " 0\n", - " 0\n", - " 0\n", + " cpg0016-jump\n", + " source_4\n", + " 277\n", + " 277\n", + " 277\n", " True\n", " True\n", " \n", " \n", " 15\n", - " cpg0018-singh-seedseq\n", + " cpg0031-caicedo-cmvip\n", " broad\n", " 0\n", " 0\n", @@ -718,16 +649,6 @@ " \n", " \n", " 16\n", - " cpg0018-singh-seedseq\n", - " x\n", - " 0\n", - " 0\n", - " 0\n", - " True\n", - " True\n", - " \n", - " \n", - " 17\n", " cpg0016-jump\n", " source_13\n", " 0\n", @@ -737,19 +658,19 @@ " False\n", " \n", " \n", - " 18\n", - " cpg0024-bortezomib\n", - " source_4\n", + " 17\n", + " cpg0011-lipocyteprofiler\n", + " broad\n", " 0\n", - " 4\n", " 0\n", - " False\n", + " 0\n", + " True\n", " True\n", " \n", " \n", - " 19\n", - " cpg0003-rosetta\n", - " None\n", + " 18\n", + " dev-cpg0016-jump\n", + " deflaux-workflow-test-BR00125638-B22\n", " 0\n", " 0\n", " 0\n", @@ -757,9 +678,19 @@ " True\n", " \n", " \n", + " 19\n", + " cpg0016-jump\n", + " source_8\n", + " 216\n", + " 216\n", + " 216\n", + " True\n", + " True\n", + " \n", + " \n", " 20\n", - " cpg0016-jump-fixed\n", - " source_7\n", + " dev-cpg0016-jump\n", + " deflaux_test\n", " 0\n", " 0\n", " 0\n", @@ -768,18 +699,8 @@ " \n", " \n", " 21\n", - " cpg0016-jump\n", - " source_10\n", - " 0\n", - " 222\n", - " 222\n", - " False\n", - " False\n", - " \n", - " \n", - " 22\n", - " dev-cpg0016-jump\n", - " source_4\n", + " cpg0016-jump-fixed\n", + " source_1\n", " 0\n", " 0\n", " 0\n", @@ -787,28 +708,38 @@ " True\n", " \n", " \n", - " 23\n", - " cpg0021-periscope\n", + " 22\n", + " cpg0022-cmqtl\n", " broad\n", " 0\n", + " 12\n", " 0\n", - " 0\n", + " False\n", + " True\n", + " \n", + " \n", + " 23\n", + " cpg0016-jump\n", + " source_9\n", + " 108\n", + " 108\n", + " 108\n", " True\n", " True\n", " \n", " \n", " 24\n", - " cpg0020-varchamp\n", - " broad\n", - " 0\n", - " 57\n", - " 0\n", + " cpg0016-jump\n", + " source_3\n", + " 310\n", + " 303\n", + " 303\n", + " False\n", " False\n", - " True\n", " \n", " \n", " 25\n", - " cpg0003-rosetta\n", + " cpg0005-gerry-bioactivity\n", " broad\n", " 0\n", " 0\n", @@ -818,83 +749,83 @@ " \n", " \n", " 26\n", - " cpg0016-jump\n", - " source_11\n", - " 182\n", - " 182\n", - " 182\n", + " dev-cpg0016-jump\n", + " deflaux-workflow-test2\n", + " 0\n", + " 0\n", + " 0\n", " True\n", " True\n", " \n", " \n", " 27\n", - " cpg0010-caie-drugresponse\n", - " broad-az\n", + " cpg0030-gustafsdottir-cellpainting\n", + " broad\n", " 0\n", - " 55\n", " 0\n", - " False\n", + " 0\n", + " True\n", " True\n", " \n", " \n", " 28\n", - " cpg0017-rohban-pathways\n", + " cpg0003-rosetta\n", " broad\n", " 0\n", - " 5\n", " 0\n", - " False\n", + " 0\n", + " True\n", " True\n", " \n", " \n", " 29\n", - " cpg0016-jump\n", - " source_8\n", - " 216\n", - " 216\n", - " 216\n", + " cpg0021-periscope\n", + " broad\n", + " 0\n", + " 0\n", + " 0\n", " True\n", " True\n", " \n", " \n", " 30\n", - " cpg0012-wawer-bioactivecompoundprofiling\n", + " cpg0026-lacoste_haghighi-rare-diseases\n", " broad\n", - " 0\n", - " 405\n", + " 1\n", + " 185\n", " 0\n", " False\n", - " True\n", + " False\n", " \n", " \n", " 31\n", - " cpg0016-jump-fixed\n", + " cpg0001-cellpainting-protocol\n", " source_4\n", " 0\n", + " 108\n", " 0\n", - " 0\n", - " True\n", + " False\n", " True\n", " \n", " \n", " 32\n", - " dev-cpg0016-jump\n", - " deflaux_test\n", - " 0\n", + " cpg0010-caie-drugresponse\n", + " broad-az\n", " 0\n", + " 55\n", " 0\n", - " True\n", + " False\n", " True\n", " \n", " \n", " 33\n", - " cpg0016-jump\n", - " source_6\n", + " cpg0024-bortezomib\n", + " source_4\n", + " 0\n", + " 4\n", " 0\n", - " 246\n", - " 246\n", - " False\n", " False\n", + " True\n", " \n", " \n", " 34\n", @@ -908,28 +839,28 @@ " \n", " \n", " 35\n", - " cpg0001-cellpainting-protocol\n", + " dev-cpg0016-jump\n", " source_4\n", " 0\n", - " 108\n", " 0\n", - " False\n", + " 0\n", + " True\n", " True\n", " \n", " \n", " 36\n", - " dev-cpg0016-jump\n", - " deflaux-workflow-test2\n", - " 0\n", + " cpg0017-rohban-pathways\n", + " broad\n", " 0\n", + " 5\n", " 0\n", - " True\n", + " False\n", " True\n", " \n", " \n", " 37\n", - " test-cpg0016-jump\n", - " source_4\n", + " cpg0015-heterogeneity\n", + " broad\n", " 0\n", " 0\n", " 0\n", @@ -938,8 +869,8 @@ " \n", " \n", " 38\n", - " cpg0030-gustafsdottir-cellpainting\n", - " broad\n", + " cpg0016-jump-fixed\n", + " source_7\n", " 0\n", " 0\n", " 0\n", @@ -948,68 +879,58 @@ " \n", " \n", " 39\n", - " cpg0016-jump\n", - " source_2\n", + " cpg0012-wawer-bioactivecompoundprofiling\n", + " broad\n", + " 0\n", + " 405\n", " 0\n", - " 229\n", - " 229\n", - " False\n", " False\n", + " True\n", " \n", " \n", " 40\n", - " cpg0014-jump-adipocyte\n", + " cpg0004-lincs\n", " broad\n", " 0\n", - " 138\n", " 0\n", - " False\n", + " 0\n", + " True\n", " True\n", " \n", " \n", " 41\n", - " cpg0016-jump\n", - " source_4\n", - " 277\n", - " 277\n", - " 277\n", + " cpg0023-mpi\n", + " mpi\n", + " 0\n", + " 0\n", + " 0\n", " True\n", " True\n", " \n", " \n", " 42\n", - " cpg0006-miami\n", - " broad\n", - " 0\n", - " 10\n", + " cpg0016-jump\n", + " source_10\n", " 0\n", + " 222\n", + " 222\n", + " False\n", " False\n", - " True\n", " \n", " \n", " 43\n", - " cpg0026-lacoste_haghighi-rare-diseases\n", - " broad\n", - " 1\n", - " 185\n", + " cpg0000-jump-pilot\n", + " source_4\n", " 0\n", - " False\n", - " False\n", - " \n", - " \n", - " 44\n", - " cpg0016-jump\n", - " source_7\n", + " 75\n", " 0\n", - " 128\n", - " 128\n", - " False\n", " False\n", + " True\n", " \n", " \n", - " 45\n", - " cpg0009-molglue\n", - " broad\n", + " 44\n", + " test-cpg0016-jump\n", + " source_4\n", " 0\n", " 0\n", " 0\n", @@ -1017,18 +938,28 @@ " True\n", " \n", " \n", - " 46\n", - " cpg0023-mpi\n", - " mpi\n", - " 0\n", + " 45\n", + " cpg0016-jump\n", + " source_5\n", " 0\n", + " 260\n", + " 260\n", + " False\n", + " False\n", + " \n", + " \n", + " 46\n", + " cpg0016-jump\n", + " source_6\n", " 0\n", - " True\n", - " True\n", + " 246\n", + " 246\n", + " False\n", + " False\n", " \n", " \n", " 47\n", - " cpg0005-gerry-bioactivity\n", + " cpg0025-dactyloscopy\n", " broad\n", " 0\n", " 0\n", @@ -1038,33 +969,33 @@ " \n", " \n", " 48\n", - " dev-cpg0016-jump\n", - " deflaux-workflow-tests\n", - " 0\n", + " cpg0014-jump-adipocyte\n", + " broad\n", " 0\n", + " 138\n", " 0\n", - " True\n", + " False\n", " True\n", " \n", " \n", " 49\n", - " cpg0016-jump\n", - " source_3\n", - " 310\n", - " 303\n", - " 303\n", - " False\n", + " cpg0006-miami\n", + " broad\n", + " 0\n", + " 10\n", + " 0\n", " False\n", + " True\n", " \n", " \n", " 50\n", - " cpg0016-jump\n", - " source_1\n", - " 55\n", - " 55\n", - " 56\n", + " cpg0018-singh-seedseq\n", + " x\n", + " 0\n", + " 0\n", + " 0\n", + " True\n", " True\n", - " False\n", " \n", " \n", "\n", @@ -1072,163 +1003,163 @@ ], "text/plain": [ " dataset_id \\\n", - "0 cpg0015-heterogeneity \n", - "1 cpg0016-jump \n", - "2 cpg0004-lincs \n", + "0 cpg0003-rosetta \n", + "1 cpg0028-kelley-resistance \n", + "2 dev-cpg0016-jump \n", "3 cpg0016-jump \n", - "4 cpg0000-jump-pilot \n", - "5 cpg0019-moshkov-deepprofiler \n", - "6 cpg0031-caicedo-cmvip \n", + "4 dev-cpg0016-jump \n", + "5 cpg0016-jump \n", + "6 cpg0018-singh-seedseq \n", "7 cpg0016-jump-fixed \n", - "8 cpg0011-lipocyteprofiler \n", - "9 cpg0028-kelley-resistance \n", - "10 cpg0002-jump-scope \n", - "11 cpg0025-dactyloscopy \n", - "12 cpg0022-cmqtl \n", - "13 dev-cpg0016-jump \n", - "14 dev-cpg0016-jump \n", - "15 cpg0018-singh-seedseq \n", - "16 cpg0018-singh-seedseq \n", - "17 cpg0016-jump \n", - "18 cpg0024-bortezomib \n", - "19 cpg0003-rosetta \n", - "20 cpg0016-jump-fixed \n", - "21 cpg0016-jump \n", - "22 dev-cpg0016-jump \n", - "23 cpg0021-periscope \n", - "24 cpg0020-varchamp \n", - "25 cpg0003-rosetta \n", - "26 cpg0016-jump \n", - "27 cpg0010-caie-drugresponse \n", - "28 cpg0017-rohban-pathways \n", - "29 cpg0016-jump \n", - "30 cpg0012-wawer-bioactivecompoundprofiling \n", - "31 cpg0016-jump-fixed \n", - "32 dev-cpg0016-jump \n", - "33 cpg0016-jump \n", + "8 cpg0002-jump-scope \n", + "9 cpg0016-jump \n", + "10 cpg0009-molglue \n", + "11 cpg0016-jump \n", + "12 cpg0019-moshkov-deepprofiler \n", + "13 cpg0020-varchamp \n", + "14 cpg0016-jump \n", + "15 cpg0031-caicedo-cmvip \n", + "16 cpg0016-jump \n", + "17 cpg0011-lipocyteprofiler \n", + "18 dev-cpg0016-jump \n", + "19 cpg0016-jump \n", + "20 dev-cpg0016-jump \n", + "21 cpg0016-jump-fixed \n", + "22 cpg0022-cmqtl \n", + "23 cpg0016-jump \n", + "24 cpg0016-jump \n", + "25 cpg0005-gerry-bioactivity \n", + "26 dev-cpg0016-jump \n", + "27 cpg0030-gustafsdottir-cellpainting \n", + "28 cpg0003-rosetta \n", + "29 cpg0021-periscope \n", + "30 cpg0026-lacoste_haghighi-rare-diseases \n", + "31 cpg0001-cellpainting-protocol \n", + "32 cpg0010-caie-drugresponse \n", + "33 cpg0024-bortezomib \n", "34 jump \n", - "35 cpg0001-cellpainting-protocol \n", - "36 dev-cpg0016-jump \n", - "37 test-cpg0016-jump \n", - "38 cpg0030-gustafsdottir-cellpainting \n", - "39 cpg0016-jump \n", - "40 cpg0014-jump-adipocyte \n", - "41 cpg0016-jump \n", - "42 cpg0006-miami \n", - "43 cpg0026-lacoste_haghighi-rare-diseases \n", - "44 cpg0016-jump \n", - "45 cpg0009-molglue \n", - "46 cpg0023-mpi \n", - "47 cpg0005-gerry-bioactivity \n", - "48 dev-cpg0016-jump \n", - "49 cpg0016-jump \n", - "50 cpg0016-jump \n", + "35 dev-cpg0016-jump \n", + "36 cpg0017-rohban-pathways \n", + "37 cpg0015-heterogeneity \n", + "38 cpg0016-jump-fixed \n", + "39 cpg0012-wawer-bioactivecompoundprofiling \n", + "40 cpg0004-lincs \n", + "41 cpg0023-mpi \n", + "42 cpg0016-jump \n", + "43 cpg0000-jump-pilot \n", + "44 test-cpg0016-jump \n", + "45 cpg0016-jump \n", + "46 cpg0016-jump \n", + "47 cpg0025-dactyloscopy \n", + "48 cpg0014-jump-adipocyte \n", + "49 cpg0006-miami \n", + "50 cpg0018-singh-seedseq \n", "\n", " source_id #plates #load_data_csvs \\\n", - "0 broad 0 0 \n", - "1 source_9 108 108 \n", - "2 broad 0 0 \n", - "3 source_5 0 260 \n", - "4 source_4 0 75 \n", - "5 broad 0 0 \n", + "0 None 0 0 \n", + "1 broad 0 0 \n", + "2 deflaux-workflow-tests 0 0 \n", + "3 source_1 55 55 \n", + "4 deflaux-workflow-test-BR00117012 0 0 \n", + "5 source_7 0 128 \n", "6 broad 0 0 \n", - "7 source_1 0 0 \n", - "8 broad 0 0 \n", - "9 broad 0 0 \n", - "10 source_4 1 44 \n", - "11 broad 0 0 \n", - "12 broad 0 12 \n", - "13 deflaux-workflow-test-BR00117012 0 0 \n", - "14 deflaux-workflow-test-BR00125638-B22 0 0 \n", + "7 source_4 0 0 \n", + "8 source_4 1 44 \n", + "9 source_11 182 182 \n", + "10 broad 0 0 \n", + "11 source_2 0 229 \n", + "12 broad 0 0 \n", + "13 broad 0 57 \n", + "14 source_4 277 277 \n", "15 broad 0 0 \n", - "16 x 0 0 \n", - "17 source_13 0 154 \n", - "18 source_4 0 4 \n", - "19 None 0 0 \n", - "20 source_7 0 0 \n", - "21 source_10 0 222 \n", - "22 source_4 0 0 \n", - "23 broad 0 0 \n", - "24 broad 0 57 \n", + "16 source_13 0 154 \n", + "17 broad 0 0 \n", + "18 deflaux-workflow-test-BR00125638-B22 0 0 \n", + "19 source_8 216 216 \n", + "20 deflaux_test 0 0 \n", + "21 source_1 0 0 \n", + "22 broad 0 12 \n", + "23 source_9 108 108 \n", + "24 source_3 310 303 \n", "25 broad 0 0 \n", - "26 source_11 182 182 \n", - "27 broad-az 0 55 \n", - "28 broad 0 5 \n", - "29 source_8 216 216 \n", - "30 broad 0 405 \n", - "31 source_4 0 0 \n", - "32 deflaux_test 0 0 \n", - "33 source_6 0 246 \n", + "26 deflaux-workflow-test2 0 0 \n", + "27 broad 0 0 \n", + "28 broad 0 0 \n", + "29 broad 0 0 \n", + "30 broad 1 185 \n", + "31 source_4 0 108 \n", + "32 broad-az 0 55 \n", + "33 source_4 0 4 \n", "34 source_15 0 174 \n", - "35 source_4 0 108 \n", - "36 deflaux-workflow-test2 0 0 \n", - "37 source_4 0 0 \n", - "38 broad 0 0 \n", - "39 source_2 0 229 \n", - "40 broad 0 138 \n", - "41 source_4 277 277 \n", - "42 broad 0 10 \n", - "43 broad 1 185 \n", - "44 source_7 0 128 \n", - "45 broad 0 0 \n", - "46 mpi 0 0 \n", + "35 source_4 0 0 \n", + "36 broad 0 5 \n", + "37 broad 0 0 \n", + "38 source_7 0 0 \n", + "39 broad 0 405 \n", + "40 broad 0 0 \n", + "41 mpi 0 0 \n", + "42 source_10 0 222 \n", + "43 source_4 0 75 \n", + "44 source_4 0 0 \n", + "45 source_5 0 260 \n", + "46 source_6 0 246 \n", "47 broad 0 0 \n", - "48 deflaux-workflow-tests 0 0 \n", - "49 source_3 310 303 \n", - "50 source_1 55 55 \n", + "48 broad 0 138 \n", + "49 broad 0 10 \n", + "50 x 0 0 \n", "\n", " #illum_folders plates-and-load_data_csvs-match \\\n", "0 0 True \n", - "1 108 True \n", + "1 0 True \n", "2 0 True \n", - "3 260 False \n", - "4 0 False \n", - "5 0 True \n", + "3 56 True \n", + "4 0 True \n", + "5 128 False \n", "6 0 True \n", "7 0 True \n", - "8 0 True \n", - "9 0 True \n", - "10 5 False \n", - "11 0 True \n", - "12 0 False \n", - "13 0 True \n", - "14 0 True \n", + "8 5 False \n", + "9 182 True \n", + "10 0 True \n", + "11 229 False \n", + "12 0 True \n", + "13 0 False \n", + "14 277 True \n", "15 0 True \n", - "16 0 True \n", - "17 154 False \n", - "18 0 False \n", - "19 0 True \n", + "16 154 False \n", + "17 0 True \n", + "18 0 True \n", + "19 216 True \n", "20 0 True \n", - "21 222 False \n", - "22 0 True \n", - "23 0 True \n", - "24 0 False \n", + "21 0 True \n", + "22 0 False \n", + "23 108 True \n", + "24 303 False \n", "25 0 True \n", - "26 182 True \n", - "27 0 False \n", - "28 0 False \n", - "29 216 True \n", + "26 0 True \n", + "27 0 True \n", + "28 0 True \n", + "29 0 True \n", "30 0 False \n", - "31 0 True \n", - "32 0 True \n", - "33 246 False \n", + "31 0 False \n", + "32 0 False \n", + "33 0 False \n", "34 0 False \n", - "35 0 False \n", - "36 0 True \n", + "35 0 True \n", + "36 0 False \n", "37 0 True \n", "38 0 True \n", - "39 229 False \n", - "40 0 False \n", - "41 277 True \n", - "42 0 False \n", + "39 0 False \n", + "40 0 True \n", + "41 0 True \n", + "42 222 False \n", "43 0 False \n", - "44 128 False \n", - "45 0 True \n", - "46 0 True \n", + "44 0 True \n", + "45 260 False \n", + "46 246 False \n", "47 0 True \n", - "48 0 True \n", - "49 303 False \n", - "50 56 True \n", + "48 0 False \n", + "49 0 False \n", + "50 0 True \n", "\n", " plates-and-illum_folders-match \n", "0 True \n", @@ -1236,60 +1167,131 @@ "2 True \n", "3 False \n", "4 True \n", - "5 True \n", + "5 False \n", "6 True \n", "7 True \n", - "8 True \n", + "8 False \n", "9 True \n", - "10 False \n", - "11 True \n", + "10 True \n", + "11 False \n", "12 True \n", "13 True \n", "14 True \n", "15 True \n", - "16 True \n", - "17 False \n", + "16 False \n", + "17 True \n", "18 True \n", "19 True \n", "20 True \n", - "21 False \n", + "21 True \n", "22 True \n", "23 True \n", - "24 True \n", + "24 False \n", "25 True \n", "26 True \n", "27 True \n", "28 True \n", "29 True \n", - "30 True \n", + "30 False \n", "31 True \n", "32 True \n", - "33 False \n", + "33 True \n", "34 True \n", "35 True \n", "36 True \n", "37 True \n", "38 True \n", - "39 False \n", + "39 True \n", "40 True \n", "41 True \n", - "42 True \n", - "43 False \n", - "44 False \n", - "45 True \n", - "46 True \n", + "42 False \n", + "43 True \n", + "44 True \n", + "45 False \n", + "46 False \n", "47 True \n", "48 True \n", - "49 False \n", - "50 False " + "49 True \n", + "50 True " ] }, - "execution_count": 127, + "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ + "# Try to check within each dataset and source to find where there are missmatches between plates and load_data.csv or illum files\n", + "import pandas as pd\n", + "\n", + "result = pd.DataFrame(columns=['dataset_id', 'source_id', '#plates','#load_data_csvs','#illum_folders','plates-and-load_data_csvs-match', 'plates-and-illum_folders-match'])\n", + "\n", + "df0 =(\n", + " index\n", + " .unique(subset=[\"dataset_id\",\"source_id\"])\n", + " .filter(pl.col(\"dataset_id\").is_not_null())\n", + " .select(pl.col(\"dataset_id\",\"source_id\"))\n", + " .collect(streaming=True)\n", + ")\n", + "\n", + "for dataset,source in zip(df0[\"dataset_id\"],df0[\"source_id\"]):\n", + " # get total number of distinct plates within the raw images folder\n", + " df1 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(dataset))\n", + " .filter(pl.col(\"source_id\").eq(source))\n", + " \n", + " .filter(pl.col(\"is_dir\").eq(True))\n", + " .filter(pl.col(\"images\").eq(\"images\"))\n", + " .filter(pl.col(\"images_images_root_dir\").is_not_null())\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"plate_id\"])\n", + " .collect(streaming=True)\n", + " )\n", + "\n", + "\n", + " # get total number of load_data.csv files\n", + " df2 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(dataset))\n", + " .filter(pl.col(\"source_id\").eq(source))\n", + " \n", + " .filter(pl.col(\"workspace\").eq(\"workspace\"))\n", + " .filter(pl.col(\"leaf_node\").eq(\"load_data.csv\"))\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"plate_id\"]) \n", + " .collect(streaming=True)\n", + " )\n", + "\n", + " # get total number of illum/ folders\n", + " df3 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(dataset))\n", + " .filter(pl.col(\"source_id\").eq(source))\n", + " \n", + " .filter(pl.col(\"is_dir\").eq(True))\n", + " .filter(pl.col(\"images\").eq(\"images\"))\n", + " .filter(pl.col(\"images_illum_root_dir\").is_not_null())\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"plate_id\"]) \n", + " .collect(streaming=True)\n", + " )\n", + "\n", + " # Evaluate if number of plates matches with number of load_data_csv and illum folders\n", + " matchLoadData= df1.shape[0]==df2.shape[0]\n", + " matchIllum = df1.shape[0]==df3.shape[0]\n", + "\n", + " # Collect results in the results df\n", + " new_row_data = {'dataset_id': dataset, \n", + " 'source_id': source, \n", + " '#plates': len(df1),\n", + " '#load_data_csvs': len(df2),\n", + " '#illum_folders': len(df3),\n", + " 'plates-and-load_data_csvs-match': matchLoadData, \n", + " 'plates-and-illum_folders-match': matchIllum\n", + " }\n", + " result = pd.concat([result, pd.DataFrame([new_row_data])], ignore_index=True)\n", + "\n", "result" ] }, From ff05fddac224ea53f6d43f465f8c180c6857b831 Mon Sep 17 00:00:00 2001 From: emiglietta Date: Fri, 8 Nov 2024 16:55:07 +0100 Subject: [PATCH 4/4] Update CPGvalidator_docs.ipynb --- CPGvalidator_docs.ipynb | 50 ++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 23 deletions(-) diff --git a/CPGvalidator_docs.ipynb b/CPGvalidator_docs.ipynb index 8947fc3..bda8341 100644 --- a/CPGvalidator_docs.ipynb +++ b/CPGvalidator_docs.ipynb @@ -5,26 +5,26 @@ "metadata": {}, "source": [ "# GOALS\n", - ">DESCRIBE:\n", - "What is the input\n", - "What is the output\n", + "- DESCRIBE:\n", + " - What is the input\n", + " - What is the output\n", "\n", - ">Overview—: to the readme\n", - "Include, what is a bucket! --> done!(ish)\n", + "- Overview—: to the readme\n", + " - Include, what is a bucket! --> done!(ish)\n", "\n", - ">Rules: \n", - "How to construct new rules\n", - "How to run them\n", - "Hoe to understand the output\n", + "- Rules: \n", + " - How to construct new rules\n", + " - How to run them\n", + " - How to understand the output\n", "\n", - ">In progress is a valid flag bc this work is being done\n", + "- In progress is a valid flag bc this work is being done\n", "\n", - ">Inventory autogenerates ¿once a week? --> Yes! but the index is generated manually\n", + "- Inventory autogenerates ¿once a week? --> Yes! but the index is generated manually\n", "\n", - ">Explain the example case for downloading files\n", + "- Explain the example case for downloading files\n", "\n", - ">README describing what it does\n", - "Tutorial and or examples" + "- README describing what it does\n", + " - Tutorial and or examples" ] }, { @@ -41,17 +41,19 @@ "\n", "Within this system, all files within the CPG live in a single 'bucket'. Because it is an object storage, this bucket does not contain a folder structure, but rather all objects live together. In order to identify them, each object is assigned a unique 'key' that consists of a string of characters, akin to the directory path in a regular file structure with folders.\n", "\n", - "For example, <'s3://cellpainting-gallery/cpg0016-jump/source_4/workspace/analysis/2021_06_21_Batch7/BR00125168/analysis/BR00125168-G17-5/Cells.csv'> is the key to a specific .csv file that lives within the CPG S3 bucket.\n", + "For example, `s3://cellpainting-gallery/cpg0016-jump/source_4/workspace/analysis/2021_06_21_Batch7/BR00125168/analysis/BR00125168-G17-5/Cells.csv` is the key to a specific .csv file that lives within the CPG S3 bucket.\n", "\n", "The [**aws inventory**]( https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html) is a list of all the objects contained in a bucket and their associated metadata (i.e. object size, date of upload, last modified date, etc). This inventory is updated automatically on a weekly basis for the CPG.\n", "\n", "However, the format in which this inventory is structured is not friendly for exploring it. \n", "\n", - "That is why, we use `cpgdata` to parse (using `cpgparser`, a Python library written in Rust) the inventory, retireving and organizing useful information about all objects in a dataframe called the **Index**. You can [manually browse the Index contents online using Quilt](https://open.quiltdata.com/b/cellpainting-gallery/tree/).\n", + "That is why, we use `cpgdata` to parse the inventory, retireving and organizing useful information about all objects in a dataframe called the **Index**. In order to do this, `cpgdata` uses `cpgparser`, a Python library written in Rust.\n", "\n", - "The `cpgdata` package also provides tools to navigate and filter the Index file which can then be used to selectively download certain files from the CPG or explore its contents.\n", + "The `cpgdata` package also provides tools to **navigate and filter the Index e** which can then be used to selectively download certain files from the CPG or explore its contents.\n", "\n", - "Moreover, these same tools can also be used to create rules to validate the structure and completeness of new data before uploading it to the CPG bucket, to ensure that it complies with the CPG requirements.\n" + "If you prefer, you can [manually browse the Index contents online using Quilt](https://open.quiltdata.com/b/cellpainting-gallery/tree/).\n", + "\n", + "Moreover, `cpgdata` tools can also be used to create rules to **validate the structure and completeness** of new data before uploading it to the CPG bucket, to ensure that it complies with the CPG requirements.\n" ] }, { @@ -59,7 +61,7 @@ "metadata": {}, "source": [ "The **Inventory** for the CPG lives in \n", - "\n", + "```\n", "s3://cellpainting-gallery-inventory/\n", " └── cellpainting-gallery/\n", " └──index/\n", @@ -68,7 +70,8 @@ " ├──2024-03-31T01-00Z/\n", " ├── 2024-04-07T01-00Z/\n", " ├── data/\n", - " └──hive/" + " └──hive/\n", + "```" ] }, { @@ -146,7 +149,7 @@ "metadata": {}, "source": [ "### The generation of the **Index** file is a very time-consuming process, but you don't need to do it yourself! \n", - "### You can easily download it using the following command:" + "### You can easily download a pre-generated **Index** file using the following command:" ] }, { @@ -157,7 +160,7 @@ "source": [ "# Select a local diretory in which to download the Index\n", "\n", - "index_dir = Path(\"/Users/emigliet/Documents/CPGvalidator_docs\")\n", + "index_dir = Path(\"/Users/emigliet/Documents/CPG-index\")\n", "# index_dir = Path(\"Your/Local/Destination/Directory\")\n", "\n", "# Note that the Index file is fairly large (over 20gb) so it's divided into several .parquet files.\n", @@ -172,7 +175,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Columns included in the Index file:\n", + "## THIS DESCRIPTION IS INCOMPLETE AND CATEGORIES ARE LIKELY TO HAVE CHANGED AFTER REESTRUCTURING OF THE TOOLS\n", + "## Columns included in the Index file:\n", "\n", "- `key` : object key identifier, useful for downloading files\n", "- `root_dir`: ## if col(\"worskpace')==\"workspace, from 'workspace' to leaf node, \n",