diff --git a/.gitignore b/.gitignore index 902ea72..a9a8fd1 100644 --- a/.gitignore +++ b/.gitignore @@ -164,4 +164,5 @@ cython_debug/ .direnv scratch data -!cpgweb/src/lib + +EM_notes.rtf diff --git a/CPGvalidator_docs.ipynb b/CPGvalidator_docs.ipynb new file mode 100644 index 0000000..bda8341 --- /dev/null +++ b/CPGvalidator_docs.ipynb @@ -0,0 +1,1552 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# GOALS\n", + "- DESCRIBE:\n", + " - What is the input\n", + " - What is the output\n", + "\n", + "- Overview—: to the readme\n", + " - Include, what is a bucket! --> done!(ish)\n", + "\n", + "- Rules: \n", + " - How to construct new rules\n", + " - How to run them\n", + " - How to understand the output\n", + "\n", + "- In progress is a valid flag bc this work is being done\n", + "\n", + "- Inventory autogenerates ¿once a week? --> Yes! but the index is generated manually\n", + "\n", + "- Explain the example case for downloading files\n", + "\n", + "- README describing what it does\n", + " - Tutorial and or examples" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Description\n", + "`cpgdata` is a CLI toolset for navigating and exploring the Cell Painting Gallery.\n", + "\n", + "\n", + "## Context\n", + "\n", + "The CellPainting Gallery (CPG) is hosted in a cloud object storage, AWS's S3.\n", + "\n", + "Within this system, all files within the CPG live in a single 'bucket'. Because it is an object storage, this bucket does not contain a folder structure, but rather all objects live together. In order to identify them, each object is assigned a unique 'key' that consists of a string of characters, akin to the directory path in a regular file structure with folders.\n", + "\n", + "For example, `s3://cellpainting-gallery/cpg0016-jump/source_4/workspace/analysis/2021_06_21_Batch7/BR00125168/analysis/BR00125168-G17-5/Cells.csv` is the key to a specific .csv file that lives within the CPG S3 bucket.\n", + "\n", + "The [**aws inventory**]( https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html) is a list of all the objects contained in a bucket and their associated metadata (i.e. object size, date of upload, last modified date, etc). This inventory is updated automatically on a weekly basis for the CPG.\n", + "\n", + "However, the format in which this inventory is structured is not friendly for exploring it. \n", + "\n", + "That is why, we use `cpgdata` to parse the inventory, retireving and organizing useful information about all objects in a dataframe called the **Index**. In order to do this, `cpgdata` uses `cpgparser`, a Python library written in Rust.\n", + "\n", + "The `cpgdata` package also provides tools to **navigate and filter the Index e** which can then be used to selectively download certain files from the CPG or explore its contents.\n", + "\n", + "If you prefer, you can [manually browse the Index contents online using Quilt](https://open.quiltdata.com/b/cellpainting-gallery/tree/).\n", + "\n", + "Moreover, `cpgdata` tools can also be used to create rules to **validate the structure and completeness** of new data before uploading it to the CPG bucket, to ensure that it complies with the CPG requirements.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The **Inventory** for the CPG lives in \n", + "```\n", + "s3://cellpainting-gallery-inventory/\n", + " └── cellpainting-gallery/\n", + " └──index/\n", + " └── [all the index chunks in .parquet format]\n", + " └── whole_bucket/\n", + " ├──2024-03-31T01-00Z/\n", + " ├── 2024-04-07T01-00Z/\n", + " ├── data/\n", + " └──hive/\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The **Index** file lives in \n", + "\n", + "If you want to get an idea of the expected file structure of the CPG you do so [HERE](https://broadinstitute.github.io/cellpainting-gallery/data_structure.html).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### In its current version, `cpgdata` runs on Python 3.10 so the first step will be to create an environment to run it" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!conda create --name cpgdata python==3.10\n", + "!conda activate cpgdata" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Next, you need to install the and packages (only once)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install cpgparser\n", + "!pip install cpgdata" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Import the necessary libraries and packages" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "from pathlib import Path\n", + "\n", + "# We use polars to read and explore the index \n", + "import polars as pl\n", + "from cpgdata.utils import parallel, download_s3_files\n", + "\n", + "#These were included in the example but not necessary for the code so far\n", + "# from typing import List\n", + "# from pathlib import Path\n", + "# from pprint import pprint\n", + "# import os\n", + "# from tqdm import tqdm" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### The generation of the **Index** file is a very time-consuming process, but you don't need to do it yourself! \n", + "### You can easily download a pre-generated **Index** file using the following command:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "# Select a local diretory in which to download the Index\n", + "\n", + "index_dir = Path(\"/Users/emigliet/Documents/CPG-index\")\n", + "# index_dir = Path(\"Your/Local/Destination/Directory\")\n", + "\n", + "# Note that the Index file is fairly large (over 20gb) so it's divided into several .parquet files.\n", + "# !cpg sync index {index_dir}\n", + "\n", + "# Load the index using polars (pl)\n", + "index_files = [file for file in index_dir.glob(\"*.parquet\")]\n", + "index = pl.scan_parquet(index_files)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## THIS DESCRIPTION IS INCOMPLETE AND CATEGORIES ARE LIKELY TO HAVE CHANGED AFTER REESTRUCTURING OF THE TOOLS\n", + "## Columns included in the Index file:\n", + "\n", + "- `key` : object key identifier, useful for downloading files\n", + "- `root_dir`: ## if col(\"worskpace')==\"workspace, from 'workspace' to leaf node, \n", + "- `images_root_dir`: path after \"{dataset_id}/{source_id}/{batch_id}/images/\" to the object. Is 'null' is object is not within that path.\n", + "- `images_batch_root_dir`: path after \"{dataset_id}/{source_id}/{batch_id}/\" to the object. Is 'null' if object is no within that path.\n", + "- `images_illum_root_dir`: path after \"{dataset_id}/{source_id}/{batch_id}/images/illum/\" to the object. If \"is_dir\"==True, \"images_illum_root_dir\"==\"plate_id\". If \"is_dir\"==False, \"images_illum_root_dir\"==\"plate_id/illumFile\".\n", + "- `images_images_root_dir`: path after \"{dataset_id}/{source_id}/{batch_id}/images/{plate_id}/images/\" to the object. Is 'null' if object is no within that path. \n", + "- `images_images_aligned_root_dir`: \n", + "- `images_images_corrected_root_dir`: \n", + "- `images_images_corrected_cropped_root_dir`: \n", + "- `workspace_root_dir`: path after \"{dataset_id}/{source_id}/workspace/\" to the object. Is 'null' is object is not within that path.\n", + "- `analysis_root_dir`: \n", + "- `backend_root_dir`: \n", + "- `load_data_csv_root_dir`: path from \"load_data_csv\" to leaf node (?)\n", + "- `metadata_root_dir`: \n", + "- `profiles_root_dir`: \n", + "- `assaydev_root_dir`: \n", + "- `embeddings_root_dir`: \n", + "- `pipelines_root_dir`: \n", + "- `qc_root_dir`: \n", + "- `segmentation_root_dir`: \n", + "- `software_root_dir`: \n", + "- `workspace_dl_root_dir`: \n", + "- `collated_root_dir`: \n", + "- `consensus_root_dir`: \n", + "- `dl_embeddings_root_dir`: \n", + "- `dl_profiles_root_dir`: \n", + "- `sep`: \n", + "- `images`: is \"images\" if \"images\" is part of the key. Is \"null\" otherwise.\n", + "- `workspace`: is \"workspace\" if \"workspace\" is part of the key. Is \"null\" otherwise.\n", + "- `workspace_dl`: \n", + "- `dataset_id`: name of the dataset (e.g. \"cpg0016-jump\", \"cpg0021-periscope\", etc.)\n", + "- `source_id`: code for the source of the images (the institution who produced them)\n", + "- `batch_id`: batch number\n", + "- `plate_id`: unique plate identification code\n", + "- `well_id`: well position\n", + "- `site_id`: site number (sites are each of the fields of view imaged in a well)\n", + "- `well_site_id`: \n", + "- `plate_well_site_id`: \n", + "- `ml_model_id`: \n", + "- `leaf_node`: if the object is a file (\"is_dir\"==False), the name of the file. Otherwise, 'null'.\n", + "- `filename`: leaf node filename, without extension\n", + "- `extension`: leaf node extension\n", + "- `software_hash`: \n", + "- `software`: \n", + "- `hash`: \n", + "- `allowed_names`: \n", + "- `bucket`: \n", + "- `obj_key`: \n", + "- `size`: \n", + "- `last_modified_date`: \n", + "- `e_tag`: \n", + "- `storage_class`: \n", + "- `is_multipart_uploaded`: \n", + "- `replication_status`: \n", + "- `encryption_status`: \n", + "- `object_lock_retain_until_date`: \n", + "- `object_lock_mode`: \n", + "- `object_lock_legal_hold_status`: \n", + "- `intelligent_tiering_access_tier`: \n", + "- `bucket_key_status`: \n", + "- `checksum_algorithm`: \n", + "- `object_access_control_list`: \n", + "- `object_owner`: \n", + "- `is_parsing_error`: \n", + "- `errors`: \n", + "- `is_dir`: \n", + "- `key_parts`: \n", + "- `workspace_dir`: if within \"workspace\", which workspace dir is object related to ('profiles', 'load_data_csv', 'software', 'metadata', 'backend', 'quality_control', 'assaydev', 'analysis', 'pipelines'). Is 'None' if the object does not contain \"workspace\" in it's key or is in \n", + " \n", + "You can see all the Index column names and the type of data stored in each using `df.schema`.\n", + "\n", + "**Refer to CPG schema and use those same keys! check matching and come up with useful key maybe**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# dictionary of index col names and their respective data types\n", + "index.schema" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Example: use the index to download just a specific subset of files from the CPG" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will use **Polars** tools in the following examples to explore and filter the index.\n", + "You can find more info on the context and expressions in:\n", + "https://docs.pola.rs/user-guide/concepts/contexts/\n", + "https://docs.pola.rs/user-guide/concepts/expressions/" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "polars.config.Config" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Setting the maximum length of displayed strings to 500 helps to visualize the complete keys\n", + "pl.Config(fmt_str_lengths=500)\n", + "pl.Config.set_fmt_table_cell_list_len(500)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "#Pull the keys (file location within the bucket) of all 'Cells.csv' files from Source 4 in the JUMP dataset (cpg0016-jump)\n", + "\n", + "df = (\n", + " index\n", + " #Use filtering to get to the failing rows and not the other way around\n", + " .filter(pl.col(\"dataset_id\").eq(\"cpg0016-jump\")) \n", + " .filter(pl.col(\"source_id\").eq(\"source_4\"))\n", + " .filter(pl.col(\"leaf_node\").str.contains(\"Cells.csv\"))\n", + " \n", + " # Always add a `select` at the end of the chain and ONLY select for keys\n", + " .select(pl.col(\"key\",\"load_data_csv_root_dir\")) \n", + " \n", + " # Materialize this polars LazyFrame into a DataFrame.\n", + " .collect(streaming=True) # the streaming option prevents out of memory errors when loading big dataframes\n", + ")\n", + "\n", + "# print first 10 results\n", + "print(df.to_dicts()[0:10])\n", + "\n", + "# List the keys of the files to download\n", + "download_keys = list(df.to_dict()[\"key\"])\n", + "\n", + "# Choose a destination directory for the files\n", + "dest_dir = \"Path/To/Save/Your/files\"\n", + "\n", + "# Run a parallel command to dowload all files specified in the list of keys\n", + "parallel(download_keys, download_s3_files, [\"cellpainting-gallery\", Path(dest_dir)], jobs=20)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Further Tests and issues" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "## some useful structures for filtering and selecting\n", + "\n", + " # .filter(pl.col(\"images_images_root_dir\").is_in([\"2020_11_04_CPJUMP1\", \"2020_11_19_TimepointDay4\", \"2020_12_08_CPJUMP1_Bleaching\"])) \n", + " # .filter(pl.col(\"leaf_node\").str.contains(\"^.*(.tiff)\"))\n", + " # .filter(pl.col(\"well_id\").eq(\"E7\")) \n", + "\n", + " # .select(pl.col(\"well_id\").unique())\n", + " # .select(pl.col(\"*\").exclude([\"size\", \"is_multipart_uploaded\"]))\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "### There appear to be some issues when parsing well_id, particularly in the embedding.parquet files from sources 1, 2 and 7 of the JUMP dataset.\n", + "\n", + "df = (\n", + " index\n", + " .unique(subset=\"well_id\")\n", + " .filter(pl.col(\"is_parsing_error\").eq(False)) \n", + " .select(\"well_id\", \"key\", \"dataset_id\", \"source_id\", \"leaf_node\")\n", + " .unique(subset=[\"dataset_id\",\"leaf_node\",\"source_id\"])\n", + " .collect(streaming=True)\n", + " )\n", + "df" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge: write this validations:\n", + " - Is there a folder with illum files for every plate within raw images?\n", + " - Is there a load_data.csv for every plate, is there a load_data csv?" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Idea: compare total number of unique plates in raw images with number of unique load.csv files and illum folders\n", + "\n", + "Works fine fo just source 4 of JUMP but fails when applied bucket-wide" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# get total number of distinct plates within the raw images folder\n", + "df1 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(\"cpg0016-jump\"))\n", + " .filter(pl.col(\"source_id\").eq(\"source_4\"))\n", + " \n", + " .filter(pl.col(\"is_dir\").eq(True))\n", + " .filter(pl.col(\"images\").eq(\"images\"))\n", + " .filter(pl.col(\"images_images_root_dir\").is_not_null())\n", + " .filter(pl.col(\"dataset_id\").eq(\"jump\").not_()) #this dataset_id has parsing errors\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\"]) # gives me 1150 unique plates\n", + " # .unique(subset=[\"plate_id\"]) # gives me 1150 unique plates\n", + " .collect(streaming=True)\n", + " )\n", + "\n", + "\n", + "# get total number of load_data.csv files\n", + "df2 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(\"cpg0016-jump\"))\n", + " .filter(pl.col(\"source_id\").eq(\"source_4\"))\n", + " \n", + " .filter(pl.col(\"workspace\").eq(\"workspace\"))\n", + " .filter(pl.col(\"leaf_node\").eq(\"load_data.csv\"))\n", + " .filter(pl.col(\"dataset_id\").eq(\"jump\").not_()) #this dataset_id has parsing errors\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " # .unique(subset=[\"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\"]) #gives me 3478 unique combinations\n", + " .unique(subset=[\"plate_id\"]) # gives me 3652 unique plates if I don't filter the \"jump\" dataset_id because of parsing shenanigans\n", + " .collect(streaming=True)\n", + " )\n", + "\n", + "# get total number of illum/ folders\n", + "df3 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(\"cpg0016-jump\"))\n", + " .filter(pl.col(\"source_id\").eq(\"source_4\"))\n", + " \n", + " .filter(pl.col(\"is_dir\").eq(True))\n", + " .filter(pl.col(\"images\").eq(\"images\"))\n", + " .filter(pl.col(\"images_illum_root_dir\").is_not_null())\n", + " .filter(pl.col(\"dataset_id\").eq(\"jump\").not_()) #this dataset_id has parsing errors\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\"]) # gives me 2386 unique plates\n", + " # .unique(subset=[\"plate_id\"]) # gives me 2386 unique plates\n", + " .collect(streaming=True)\n", + " )\n", + "\n", + "print(f\"df1: {df1.shape}\")\n", + "print(f\"df2: {df2.shape}\")\n", + "print(f\"df3: {df3.shape}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
dataset_idsource_id#plates#load_data_csvs#illum_foldersplates-and-load_data_csvs-matchplates-and-illum_folders-match
0cpg0003-rosettaNone000TrueTrue
1cpg0028-kelley-resistancebroad000TrueTrue
2dev-cpg0016-jumpdeflaux-workflow-tests000TrueTrue
3cpg0016-jumpsource_1555556TrueFalse
4dev-cpg0016-jumpdeflaux-workflow-test-BR00117012000TrueTrue
5cpg0016-jumpsource_70128128FalseFalse
6cpg0018-singh-seedseqbroad000TrueTrue
7cpg0016-jump-fixedsource_4000TrueTrue
8cpg0002-jump-scopesource_41445FalseFalse
9cpg0016-jumpsource_11182182182TrueTrue
10cpg0009-molgluebroad000TrueTrue
11cpg0016-jumpsource_20229229FalseFalse
12cpg0019-moshkov-deepprofilerbroad000TrueTrue
13cpg0020-varchampbroad0570FalseTrue
14cpg0016-jumpsource_4277277277TrueTrue
15cpg0031-caicedo-cmvipbroad000TrueTrue
16cpg0016-jumpsource_130154154FalseFalse
17cpg0011-lipocyteprofilerbroad000TrueTrue
18dev-cpg0016-jumpdeflaux-workflow-test-BR00125638-B22000TrueTrue
19cpg0016-jumpsource_8216216216TrueTrue
20dev-cpg0016-jumpdeflaux_test000TrueTrue
21cpg0016-jump-fixedsource_1000TrueTrue
22cpg0022-cmqtlbroad0120FalseTrue
23cpg0016-jumpsource_9108108108TrueTrue
24cpg0016-jumpsource_3310303303FalseFalse
25cpg0005-gerry-bioactivitybroad000TrueTrue
26dev-cpg0016-jumpdeflaux-workflow-test2000TrueTrue
27cpg0030-gustafsdottir-cellpaintingbroad000TrueTrue
28cpg0003-rosettabroad000TrueTrue
29cpg0021-periscopebroad000TrueTrue
30cpg0026-lacoste_haghighi-rare-diseasesbroad11850FalseFalse
31cpg0001-cellpainting-protocolsource_401080FalseTrue
32cpg0010-caie-drugresponsebroad-az0550FalseTrue
33cpg0024-bortezomibsource_4040FalseTrue
34jumpsource_1501740FalseTrue
35dev-cpg0016-jumpsource_4000TrueTrue
36cpg0017-rohban-pathwaysbroad050FalseTrue
37cpg0015-heterogeneitybroad000TrueTrue
38cpg0016-jump-fixedsource_7000TrueTrue
39cpg0012-wawer-bioactivecompoundprofilingbroad04050FalseTrue
40cpg0004-lincsbroad000TrueTrue
41cpg0023-mpimpi000TrueTrue
42cpg0016-jumpsource_100222222FalseFalse
43cpg0000-jump-pilotsource_40750FalseTrue
44test-cpg0016-jumpsource_4000TrueTrue
45cpg0016-jumpsource_50260260FalseFalse
46cpg0016-jumpsource_60246246FalseFalse
47cpg0025-dactyloscopybroad000TrueTrue
48cpg0014-jump-adipocytebroad01380FalseTrue
49cpg0006-miamibroad0100FalseTrue
50cpg0018-singh-seedseqx000TrueTrue
\n", + "
" + ], + "text/plain": [ + " dataset_id \\\n", + "0 cpg0003-rosetta \n", + "1 cpg0028-kelley-resistance \n", + "2 dev-cpg0016-jump \n", + "3 cpg0016-jump \n", + "4 dev-cpg0016-jump \n", + "5 cpg0016-jump \n", + "6 cpg0018-singh-seedseq \n", + "7 cpg0016-jump-fixed \n", + "8 cpg0002-jump-scope \n", + "9 cpg0016-jump \n", + "10 cpg0009-molglue \n", + "11 cpg0016-jump \n", + "12 cpg0019-moshkov-deepprofiler \n", + "13 cpg0020-varchamp \n", + "14 cpg0016-jump \n", + "15 cpg0031-caicedo-cmvip \n", + "16 cpg0016-jump \n", + "17 cpg0011-lipocyteprofiler \n", + "18 dev-cpg0016-jump \n", + "19 cpg0016-jump \n", + "20 dev-cpg0016-jump \n", + "21 cpg0016-jump-fixed \n", + "22 cpg0022-cmqtl \n", + "23 cpg0016-jump \n", + "24 cpg0016-jump \n", + "25 cpg0005-gerry-bioactivity \n", + "26 dev-cpg0016-jump \n", + "27 cpg0030-gustafsdottir-cellpainting \n", + "28 cpg0003-rosetta \n", + "29 cpg0021-periscope \n", + "30 cpg0026-lacoste_haghighi-rare-diseases \n", + "31 cpg0001-cellpainting-protocol \n", + "32 cpg0010-caie-drugresponse \n", + "33 cpg0024-bortezomib \n", + "34 jump \n", + "35 dev-cpg0016-jump \n", + "36 cpg0017-rohban-pathways \n", + "37 cpg0015-heterogeneity \n", + "38 cpg0016-jump-fixed \n", + "39 cpg0012-wawer-bioactivecompoundprofiling \n", + "40 cpg0004-lincs \n", + "41 cpg0023-mpi \n", + "42 cpg0016-jump \n", + "43 cpg0000-jump-pilot \n", + "44 test-cpg0016-jump \n", + "45 cpg0016-jump \n", + "46 cpg0016-jump \n", + "47 cpg0025-dactyloscopy \n", + "48 cpg0014-jump-adipocyte \n", + "49 cpg0006-miami \n", + "50 cpg0018-singh-seedseq \n", + "\n", + " source_id #plates #load_data_csvs \\\n", + "0 None 0 0 \n", + "1 broad 0 0 \n", + "2 deflaux-workflow-tests 0 0 \n", + "3 source_1 55 55 \n", + "4 deflaux-workflow-test-BR00117012 0 0 \n", + "5 source_7 0 128 \n", + "6 broad 0 0 \n", + "7 source_4 0 0 \n", + "8 source_4 1 44 \n", + "9 source_11 182 182 \n", + "10 broad 0 0 \n", + "11 source_2 0 229 \n", + "12 broad 0 0 \n", + "13 broad 0 57 \n", + "14 source_4 277 277 \n", + "15 broad 0 0 \n", + "16 source_13 0 154 \n", + "17 broad 0 0 \n", + "18 deflaux-workflow-test-BR00125638-B22 0 0 \n", + "19 source_8 216 216 \n", + "20 deflaux_test 0 0 \n", + "21 source_1 0 0 \n", + "22 broad 0 12 \n", + "23 source_9 108 108 \n", + "24 source_3 310 303 \n", + "25 broad 0 0 \n", + "26 deflaux-workflow-test2 0 0 \n", + "27 broad 0 0 \n", + "28 broad 0 0 \n", + "29 broad 0 0 \n", + "30 broad 1 185 \n", + "31 source_4 0 108 \n", + "32 broad-az 0 55 \n", + "33 source_4 0 4 \n", + "34 source_15 0 174 \n", + "35 source_4 0 0 \n", + "36 broad 0 5 \n", + "37 broad 0 0 \n", + "38 source_7 0 0 \n", + "39 broad 0 405 \n", + "40 broad 0 0 \n", + "41 mpi 0 0 \n", + "42 source_10 0 222 \n", + "43 source_4 0 75 \n", + "44 source_4 0 0 \n", + "45 source_5 0 260 \n", + "46 source_6 0 246 \n", + "47 broad 0 0 \n", + "48 broad 0 138 \n", + "49 broad 0 10 \n", + "50 x 0 0 \n", + "\n", + " #illum_folders plates-and-load_data_csvs-match \\\n", + "0 0 True \n", + "1 0 True \n", + "2 0 True \n", + "3 56 True \n", + "4 0 True \n", + "5 128 False \n", + "6 0 True \n", + "7 0 True \n", + "8 5 False \n", + "9 182 True \n", + "10 0 True \n", + "11 229 False \n", + "12 0 True \n", + "13 0 False \n", + "14 277 True \n", + "15 0 True \n", + "16 154 False \n", + "17 0 True \n", + "18 0 True \n", + "19 216 True \n", + "20 0 True \n", + "21 0 True \n", + "22 0 False \n", + "23 108 True \n", + "24 303 False \n", + "25 0 True \n", + "26 0 True \n", + "27 0 True \n", + "28 0 True \n", + "29 0 True \n", + "30 0 False \n", + "31 0 False \n", + "32 0 False \n", + "33 0 False \n", + "34 0 False \n", + "35 0 True \n", + "36 0 False \n", + "37 0 True \n", + "38 0 True \n", + "39 0 False \n", + "40 0 True \n", + "41 0 True \n", + "42 222 False \n", + "43 0 False \n", + "44 0 True \n", + "45 260 False \n", + "46 246 False \n", + "47 0 True \n", + "48 0 False \n", + "49 0 False \n", + "50 0 True \n", + "\n", + " plates-and-illum_folders-match \n", + "0 True \n", + "1 True \n", + "2 True \n", + "3 False \n", + "4 True \n", + "5 False \n", + "6 True \n", + "7 True \n", + "8 False \n", + "9 True \n", + "10 True \n", + "11 False \n", + "12 True \n", + "13 True \n", + "14 True \n", + "15 True \n", + "16 False \n", + "17 True \n", + "18 True \n", + "19 True \n", + "20 True \n", + "21 True \n", + "22 True \n", + "23 True \n", + "24 False \n", + "25 True \n", + "26 True \n", + "27 True \n", + "28 True \n", + "29 True \n", + "30 False \n", + "31 True \n", + "32 True \n", + "33 True \n", + "34 True \n", + "35 True \n", + "36 True \n", + "37 True \n", + "38 True \n", + "39 True \n", + "40 True \n", + "41 True \n", + "42 False \n", + "43 True \n", + "44 True \n", + "45 False \n", + "46 False \n", + "47 True \n", + "48 True \n", + "49 True \n", + "50 True " + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Try to check within each dataset and source to find where there are missmatches between plates and load_data.csv or illum files\n", + "import pandas as pd\n", + "\n", + "result = pd.DataFrame(columns=['dataset_id', 'source_id', '#plates','#load_data_csvs','#illum_folders','plates-and-load_data_csvs-match', 'plates-and-illum_folders-match'])\n", + "\n", + "df0 =(\n", + " index\n", + " .unique(subset=[\"dataset_id\",\"source_id\"])\n", + " .filter(pl.col(\"dataset_id\").is_not_null())\n", + " .select(pl.col(\"dataset_id\",\"source_id\"))\n", + " .collect(streaming=True)\n", + ")\n", + "\n", + "for dataset,source in zip(df0[\"dataset_id\"],df0[\"source_id\"]):\n", + " # get total number of distinct plates within the raw images folder\n", + " df1 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(dataset))\n", + " .filter(pl.col(\"source_id\").eq(source))\n", + " \n", + " .filter(pl.col(\"is_dir\").eq(True))\n", + " .filter(pl.col(\"images\").eq(\"images\"))\n", + " .filter(pl.col(\"images_images_root_dir\").is_not_null())\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"plate_id\"])\n", + " .collect(streaming=True)\n", + " )\n", + "\n", + "\n", + " # get total number of load_data.csv files\n", + " df2 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(dataset))\n", + " .filter(pl.col(\"source_id\").eq(source))\n", + " \n", + " .filter(pl.col(\"workspace\").eq(\"workspace\"))\n", + " .filter(pl.col(\"leaf_node\").eq(\"load_data.csv\"))\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"plate_id\"]) \n", + " .collect(streaming=True)\n", + " )\n", + "\n", + " # get total number of illum/ folders\n", + " df3 = (\n", + " index\n", + " .filter(pl.col(\"dataset_id\").eq(dataset))\n", + " .filter(pl.col(\"source_id\").eq(source))\n", + " \n", + " .filter(pl.col(\"is_dir\").eq(True))\n", + " .filter(pl.col(\"images\").eq(\"images\"))\n", + " .filter(pl.col(\"images_illum_root_dir\").is_not_null())\n", + " .select(\"key\", \"dataset_id\", \"source_id\", \"batch_id\", \"plate_id\")\n", + " .unique(subset=[\"plate_id\"]) \n", + " .collect(streaming=True)\n", + " )\n", + "\n", + " # Evaluate if number of plates matches with number of load_data_csv and illum folders\n", + " matchLoadData= df1.shape[0]==df2.shape[0]\n", + " matchIllum = df1.shape[0]==df3.shape[0]\n", + "\n", + " # Collect results in the results df\n", + " new_row_data = {'dataset_id': dataset, \n", + " 'source_id': source, \n", + " '#plates': len(df1),\n", + " '#load_data_csvs': len(df2),\n", + " '#illum_folders': len(df3),\n", + " 'plates-and-load_data_csvs-match': matchLoadData, \n", + " 'plates-and-illum_folders-match': matchIllum\n", + " }\n", + " result = pd.concat([result, pd.DataFrame([new_row_data])], ignore_index=True)\n", + "\n", + "result" + ] + }, + { + "cell_type": "code", + "execution_count": 84, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "shape: (5, 5)
well_idkeydataset_idsource_idleaf_node
strstrstrstrstr
"UL001673""cpg0016-jump/source_1/workspace_dl/embeddings/efficientnet_v2_imagenet21k_s_feature_vector_2_0260bc96/Batch2_20221006/UL001673/UL001673/A02/embedding.parquet""cpg0016-jump""source_1""embedding.parquet"
"CP3-SC1-07""cpg0016-jump/source_7/workspace_dl/embeddings/efficientnet_v2_imagenet21k_s_feature_vector_2_0260bc96/20210727_Run3/CP3-SC1-07/CP3-SC1-07/A01/embedding.parquet""cpg0016-jump""source_7""embedding.parquet"
"M07""cpg0019-moshkov-deepprofiler/broad/workspace_dl/embeddings/105281_zenodo7114558/BBBC022/20585/M07/1/embedding.npz""cpg0019-moshkov-deepprofiler""broad""embedding.npz"
null"cpg0016-jump/source_8/""cpg0016-jump""source_8"null
"1086292259""cpg0016-jump/source_2/workspace_dl/embeddings/efficientnet_v2_imagenet21k_s_feature_vector_2_0260bc96/20210816_Batch_9/1086292259/1086292259/A01/embedding.parquet""cpg0016-jump""source_2""embedding.parquet"
" + ], + "text/plain": [ + "shape: (5, 5)\n", + "┌────────────┬──────────────────────────┬──────────────────────────┬───────────┬───────────────────┐\n", + "│ well_id ┆ key ┆ dataset_id ┆ source_id ┆ leaf_node │\n", + "│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", + "│ str ┆ str ┆ str ┆ str ┆ str │\n", + "╞════════════╪══════════════════════════╪══════════════════════════╪═══════════╪═══════════════════╡\n", + "│ UL001673 ┆ cpg0016-jump/source_1/wo ┆ cpg0016-jump ┆ source_1 ┆ embedding.parquet │\n", + "│ ┆ rkspace_dl/embeddings/ef ┆ ┆ ┆ │\n", + "│ ┆ ficientnet_v2_imagenet21 ┆ ┆ ┆ │\n", + "│ ┆ k_s_feature_vector_2_026 ┆ ┆ ┆ │\n", + "│ ┆ 0bc96/Batch2_20221006/UL ┆ ┆ ┆ │\n", + "│ ┆ 001673/UL001673/A02/embe ┆ ┆ ┆ │\n", + "│ ┆ dding.parquet ┆ ┆ ┆ │\n", + "│ CP3-SC1-07 ┆ cpg0016-jump/source_7/wo ┆ cpg0016-jump ┆ source_7 ┆ embedding.parquet │\n", + "│ ┆ rkspace_dl/embeddings/ef ┆ ┆ ┆ │\n", + "│ ┆ ficientnet_v2_imagenet21 ┆ ┆ ┆ │\n", + "│ ┆ k_s_feature_vector_2_026 ┆ ┆ ┆ │\n", + "│ ┆ 0bc96/20210727_Run3/CP3- ┆ ┆ ┆ │\n", + "│ ┆ SC1-07/CP3-SC1-07/A01/em ┆ ┆ ┆ │\n", + "│ ┆ bedding.parquet ┆ ┆ ┆ │\n", + "│ M07 ┆ cpg0019-moshkov-deepprof ┆ cpg0019-moshkov-deepprof ┆ broad ┆ embedding.npz │\n", + "│ ┆ iler/broad/workspace_dl/ ┆ iler ┆ ┆ │\n", + "│ ┆ embeddings/105281_zenodo ┆ ┆ ┆ │\n", + "│ ┆ 7114558/BBBC022/20585/M0 ┆ ┆ ┆ │\n", + "│ ┆ 7/1/embedding.npz ┆ ┆ ┆ │\n", + "│ null ┆ cpg0016-jump/source_8/ ┆ cpg0016-jump ┆ source_8 ┆ null │\n", + "│ 1086292259 ┆ cpg0016-jump/source_2/wo ┆ cpg0016-jump ┆ source_2 ┆ embedding.parquet │\n", + "│ ┆ rkspace_dl/embeddings/ef ┆ ┆ ┆ │\n", + "│ ┆ ficientnet_v2_imagenet21 ┆ ┆ ┆ │\n", + "│ ┆ k_s_feature_vector_2_026 ┆ ┆ ┆ │\n", + "│ ┆ 0bc96/20210816_Batch_9/1 ┆ ┆ ┆ │\n", + "│ ┆ 086292259/1086292259/A01 ┆ ┆ ┆ │\n", + "│ ┆ /embedding.parquet ┆ ┆ ┆ │\n", + "└────────────┴──────────────────────────┴──────────────────────────┴───────────┴───────────────────┘" + ] + }, + "execution_count": 84, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# There appear to be some issues when parsing well_id, particularly in the embedding.parquet files from sources 1, 2 and 7 of the JUMP dataset. \n", + "# The well_id listed in the index corresponds to the previous \"segment\" of the key.\n", + "\n", + "df = (\n", + " index\n", + " .unique(subset=\"well_id\")\n", + " .filter(pl.col(\"is_parsing_error\").eq(False)) \n", + " .select(\"well_id\", \"key\", \"dataset_id\", \"source_id\", \"leaf_node\")\n", + " .unique(subset=[\"dataset_id\",\"leaf_node\",\"source_id\"])\n", + " .collect(streaming=True)\n", + " )\n", + "df" + ] + }, + { + "cell_type": "code", + "execution_count": 110, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "shape: (0, 4)
dataset_idsource_idplate_idload_data_csv_root_dir
strstrstrstr
" + ], + "text/plain": [ + "shape: (0, 4)\n", + "┌────────────┬───────────┬──────────┬────────────────────────┐\n", + "│ dataset_id ┆ source_id ┆ plate_id ┆ load_data_csv_root_dir │\n", + "│ --- ┆ --- ┆ --- ┆ --- │\n", + "│ str ┆ str ┆ str ┆ str │\n", + "╞════════════╪═══════════╪══════════╪════════════════════════╡\n", + "└────────────┴───────────┴──────────┴────────────────────────┘" + ] + }, + "execution_count": 110, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "#24 plates of the source 15 in JUMP have a load_data.csv without having a load_data_csv_root_dir\n", + "# also, the dataset_id is 'jump' and not 'cpg0016-jump'\n", + "df = (index\n", + " .unique(subset=[\"dataset_id\",\"plate_id\"])\n", + " .filter(pl.col(\"leaf_node\").eq(\"load_data.csv\"))\n", + " .select(pl.col([\"dataset_id\",\"source_id\",\"plate_id\",\"load_data_csv_root_dir\"]))\n", + " .filter(pl.col(\"load_data_csv_root_dir\").is_null()) #https://docs.pola.rs/py-polars/html/reference/dataframe/api/polars.DataFrame.drop_nulls.html\n", + " .collect(streaming=True)\n", + " )\n", + "df" + ] + }, + { + "cell_type": "code", + "execution_count": 111, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "shape: (460, 1)
plate_id
str
"PEC00001815__2022-02-23T18_03_17-Measurement3"
"PEP00004102__2021-12-02T12_05_47-Measurement1"
"PEP00004137__2021-12-01T16_50_57-Measurement1"
"PEP00004139__2021-12-03T05_02_12-Measurement1"
"PEC00001790"
"PEP00004092"
"PEP00004102"
"PEC00001785__2021-12-07T01_54_26-Measurement1"
"PEP00004041__2021-12-16T04_45_56-Measurement1"
"PEC00001854__2021-12-17T13_35_15-Measurement1"
"PEP00004072"
"PEP00004065__2021-12-13T12_34_12-Measurement1"
…
"PEC00001842__2021-12-17T04_56_37-Measurement1"
"PEP00004049__2022-02-22T10_42_29-Measurement1"
"PEC00001837"
"PEP00004026__2021-12-15T18_17_27-Measurement1"
"PEP00004023__2022-02-23T16_16_10-Measurement2"
"PEC00001858__2021-12-17T10_07_06-Measurement1"
"PEC00001863__2022-01-18T15_18_04-Measurement1"
"PEC00001805__2022-02-24T19_35_58-Measurement2"
"PEP00004326__2022-03-01T06_12_25-Measurement1"
"PEC00001835"
null
"PEP00004136"
" + ], + "text/plain": [ + "shape: (460, 1)\n", + "┌───────────────────────────────────────────────┐\n", + "│ plate_id │\n", + "│ --- │\n", + "│ str │\n", + "╞═══════════════════════════════════════════════╡\n", + "│ PEC00001815__2022-02-23T18_03_17-Measurement3 │\n", + "│ PEP00004102__2021-12-02T12_05_47-Measurement1 │\n", + "│ PEP00004137__2021-12-01T16_50_57-Measurement1 │\n", + "│ PEP00004139__2021-12-03T05_02_12-Measurement1 │\n", + "│ … │\n", + "│ PEP00004326__2022-03-01T06_12_25-Measurement1 │\n", + "│ PEC00001835 │\n", + "│ null │\n", + "│ PEP00004136 │\n", + "└───────────────────────────────────────────────┘" + ] + }, + "execution_count": 111, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# There are 460 plate_ids for source 15 in JUMP, are there really 460 plates? \n", + "# also, plate_id varies in structure!\n", + "df = (index\n", + " .filter(pl.col(\"dataset_id\").eq(\"jump\"))\n", + " .filter(pl.col(\"source_id\").eq(\"source_15\"))\n", + " .unique(subset=[\"plate_id\"])\n", + " .select(pl.col([\"plate_id\"]))\n", + " .collect(streaming=True)\n", + " )\n", + "df" + ] + }, + { + "cell_type": "code", + "execution_count": 112, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "shape: (183, 1)
plate_id
str
"PEC00001782"
"PEC00001783"
"PEC00001784"
"PEC00001785"
"PEC00001786"
"PEC00001787"
"PEC00001788"
"PEC00001789"
"PEC00001790"
"PEC00001791"
"PEC00001792"
"PEC00001793"
…
"PEP00004335"
"PEP00004421"
"PEP00004422"
"PEP00004423"
"PEP00004425"
"PEP00004426"
"PEP00004427"
"PEP00004430"
"PEP00004431"
"PEP00004432"
"PEP00004457"
"PEP00004458"
" + ], + "text/plain": [ + "shape: (183, 1)\n", + "┌─────────────┐\n", + "│ plate_id │\n", + "│ --- │\n", + "│ str │\n", + "╞═════════════╡\n", + "│ PEC00001782 │\n", + "│ PEC00001783 │\n", + "│ PEC00001784 │\n", + "│ PEC00001785 │\n", + "│ … │\n", + "│ PEP00004431 │\n", + "│ PEP00004432 │\n", + "│ PEP00004457 │\n", + "│ PEP00004458 │\n", + "└─────────────┘" + ] + }, + "execution_count": 112, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# For source 15 in JUMP, are there really 460 plates? \n", + "# There are only 183 unique ones matching the regex for the plate name structure\n", + "df = (index\n", + " .filter(pl.col(\"dataset_id\").eq(\"jump\"))\n", + " .filter(pl.col(\"source_id\").eq(\"source_15\"))\n", + " # .unique(subset=[\"plate_id\"])\n", + " .filter(pl.col(\"plate_id\").str.contains(\"^PE(P|C)[0-9]{8}$\"))\n", + " .select(pl.col([\"plate_id\"]).unique().sort())\n", + " .collect(streaming=True)\n", + " )\n", + "df" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "jupy310", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}