From 4eb77465d27eac9f4ac6606fb076fe9f153d7ad5 Mon Sep 17 00:00:00 2001 From: peterchang77 Date: Mon, 20 Nov 2023 13:25:43 -0800 Subject: [PATCH 1/2] Update README.md --- sessions/best-practices-training/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/sessions/best-practices-training/README.md b/sessions/best-practices-training/README.md index d2a5084..d979008 100644 --- a/sessions/best-practices-training/README.md +++ b/sessions/best-practices-training/README.md @@ -1,9 +1,10 @@ # Best Practices for Model Training: Architectures, Hyperparameters, & Optimization +In this tutorial we will explore best practices for building modern CNN models, including recommendations for baseline (simple but robust) design choices as well as advanced motifs including separable convolutions in combination with alternative normalization and activation functions. In addition we will examine strategies for tracking model performance across a variety of network architecture and hyperparameter configurations. + ## Presenters - Peter Chang -- TBD ## Session Date & Time Tuesday, November 29, 2022 -1:00 PM \ No newline at end of file +1:00 PM From 331e63a6153369060ea6c1672614d4063266c471 Mon Sep 17 00:00:00 2001 From: peterchang77 Date: Mon, 20 Nov 2023 13:28:27 -0800 Subject: [PATCH 2/2] Add files via upload --- sessions/best-practices-training/tuning.ipynb | 713 ++++++++++++++++++ 1 file changed, 713 insertions(+) create mode 100644 sessions/best-practices-training/tuning.ipynb diff --git a/sessions/best-practices-training/tuning.ipynb b/sessions/best-practices-training/tuning.ipynb new file mode 100644 index 0000000..b0ac8c3 --- /dev/null +++ b/sessions/best-practices-training/tuning.ipynb @@ -0,0 +1,713 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Overview\n", + "\n", + "In this tutorial we will explore best practices for building modern CNN models, including recommendations for baseline (simple but robust) design choices as well as advanced motifs including separable convolutions in combination with alternative normalization and activation functions. In addition we will examine strategies for tracking model performance across a variety of network architecture and hyperparameter configurations. As a representative use case, we will build various convolutional neural networks (CNNs) for classification of pneumonia (lung infection) from chest radiographs, the most common imaging modality used to screen for pulmonary disease. \n", + "\n", + "## Workshop Links\n", + "\n", + "This tutorial focuses on specific considerations related network architecture and hyperparameter tuning. For more detailed information on topics covered in this notebook, consider the following:\n", + "\n", + "* Introduction to TensorFlow 2 and Keras: https://bit.ly/2VSYaop\n", + "* CNN for pneumonia classification: https://bit.ly/2D9ZBrX\n", + "* CNN for pneumonia segmentation: https://bit.ly/2VQMWk9\n", + "\n", + "Other useful tutorials can be found at this link: https://github.com/peterchang77/dl_tutor/tree/master/workshops" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "56d3oMiMw8Wm" + }, + "source": [ + "# Environment\n", + "\n", + "The following lines of code will configure your Google Colab environment for this tutorial." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Enable GPU runtime\n", + "\n", + "Use the following instructions to switch the default Colab instance into a GPU-enabled runtime:\n", + "\n", + "```\n", + "Runtime > Change runtime type > Hardware accelerator > GPU\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Jarvis library\n", + "\n", + "In this notebook we will Jarvis, a custom Python package to facilitate data science and deep learning for healthcare. Among other things, this library will be used for low-level data management, stratification and visualization of high-dimensional medical data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Install Jarvis library\n", + "%pip install jarvis-md" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Imports\n", + "\n", + "Use the following lines to import any needed libraries:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np, pandas as pd\n", + "import tensorflow as tf\n", + "from tensorflow import losses, optimizers\n", + "from tensorflow.keras import Input, Model, models, layers, metrics\n", + "from jarvis.train import datasets, params\n", + "from jarvis.utils.display import imshow" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Data\n", + "\n", + "The data used in this tutorial will consist of (frontal projection) chest radiographs from a subset of the RSNA / Kaggle pneumonia challenge (https://www.kaggle.com/c/rsna-pneumonia-detection-challenge). From the complete cohort, a random subset of 1,000 exams will be used for training and evaluation.\n", + "\n", + "### Download\n", + "\n", + "The custom `datasets.download(...)` method can be used to download a local copy of the dataset. By default the dataset will be archived at `/data/raw/xr_pna`; as needed an alternate location may be specified using `datasets.download(name=..., path=...)`. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Download dataset\n", + "datasets.download(name='xr/pna-512')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Python generators\n", + "\n", + "Once the dataset is downloaded locally, Python generators to iterate through the dataset can be easily prepared using the `datasets.prepare(...)` method:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Prepare generators\n", + "gen_train, gen_valid, client = datasets.prepare(name='xr/pna-512', keyword='cls-512')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The created generators, `gen_train` and `gen_valid`, are designed to yield two variables per iteration: `xs` and `ys`. Both `xs` and `ys` each represent a dictionary of NumPy arrays containing model input(s) and output(s) for a single *batch* of training. The use of Python generators provides a generic interface for data input for a number of machine learning libraries including TensorFlow 2 / Keras.\n", + "\n", + "Note that any valid Python iterable method can be used to loop through the generators indefinitely. For example the Python built-in `next(...)` method will yield the next batch of data:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Yield one example\n", + "xs, ys = next(gen_train)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Data exploration\n", + "\n", + "To help facilitate algorithm design, each original chest radiograph has been resampled to a uniform `(512, 512)` matrix. Overall, the dataset comprises a total of `1,000` 2D images: a total of `500` negaative exams and `500` positive exams." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### `xs` dictionary\n", + "\n", + "The `xs` dictionary contains a single batch of model inputs:\n", + "\n", + "1. `dat`: input chest radiograph resampled to `(1, 512, 512, 1)` matrix shape" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Print keys \n", + "for key, arr in xs.items():\n", + " print('xs key: {} | shape = {}'.format(key.ljust(8), arr.shape))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### `ys` dictionary\n", + "\n", + "The `ys` dictionary contains a single batch of model outputs:\n", + "\n", + "1. `pna`: binary classification of pneumonia vs. not pneumonia chest radiographs\n", + "\n", + "* 0 = negative\n", + "* 1 = positive of pneumonia" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Print keys \n", + "for key, arr in ys.items():\n", + " print('ys key: {} | shape = {}'.format(key.ljust(8), arr.shape))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Visualization\n", + "\n", + "Use the following lines of code to visualize a single input image using the `imshow(...)` method:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Show labels\n", + "xs, ys = next(gen_train)\n", + "imshow(xs['dat'][0])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Use the following lines of code to visualize an N x N mosaic of all images in the current batch using the `imshow(...)` method:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Show \"montage\" of all images\n", + "imshow(xs['dat'], figsize=(12, 12))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Model inputs\n", + "\n", + "For every input in `xs`, a corresponding `Input(...)` variable can be created and returned in a `inputs` dictionary for ease of model development:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Create model inputs\n", + "inputs = client.get_inputs(Input)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this example, the equivalent Python code to generate `inputs` would be:\n", + "\n", + "```python\n", + "inputs = {}\n", + "inputs['dat'] = Input(shape=(1, 512, 512, 1))\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Hyperparameters\n", + "\n", + "In this tutorial, all model hyperparameters are maintained in a CSV file and organized such that each column represents a single hyperparameter and each row represents a unique combination of hyperparameter options. This strategy helps to record an archive of previous experiments and to improve the modularity and readibility of code. Note that in a realistic workflow, a CSV file may be created and manipulated directly (either in a Jupyter notebook or other editor) however in this tutorial, the CSV file will be generated programatically using Pandas. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def create_hyperparameters(csv='./hyper.csv'):\n", + " \"\"\"\n", + " Method to create CSV hyperparameter file\n", + " \n", + " \"\"\"\n", + " # --- Define hyperparameters\n", + " p = {\n", + " 'name': ['exp-01', 'exp-02', 'exp-03', 'exp-04'],\n", + " 'filters': [8, 8, 8, 16],\n", + " 'n_blocks': [3, 4, 5, 5]}\n", + "\n", + " # --- Create Pandas DataFrame\n", + " df = pd.DataFrame(p)\n", + "\n", + " # --- Create CSV file\n", + " df.to_csv(csv, index=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To create our CSV file:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Create hyperparameters\n", + "create_hyperparameters()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once prepared, the `params` module as part of the `jarvis-md` library will be used to read each row of hyperparameters into a Python dictionary which may referenced as part of the model building code in subsequent sections." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Load hyperparameters\n", + "p = params.load('./hyper.csv', row=0)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Creating Model\n", + "\n", + "In this section, we will define a template neural network architecture that dynamically references the hyperparameters defined in the `hyper.csv` file. Using this strategy, modifications to the network topology and training hyperparameters may be propogated through modification of the `hyper.csv` file only." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Standard CNN\n", + "\n", + "Based on historic best practices, the following design choices are recommended for a simple, baseline CNN approach:\n", + "\n", + "* 3x3 convolutional kernel size\n", + "* batch normalization (after convolution and before nonlinearity)\n", + "* ReLU (or leaky ReLU) activation function\n", + "* stride-2 convolutions for subsampling\n", + "\n", + "The following code block creates nested lambda function to quickly implement CNN models using these strategies:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def create_blocks():\n", + " \"\"\"\n", + " Method to define simple stride-1 and stride-2 convolutional blocks\n", + " \n", + " \"block\" = conv > norm > relu\n", + " \n", + " \"\"\"\n", + " # --- Define kwargs dictionary\n", + " kwargs = {\n", + " 'kernel_size': (1, 3, 3),\n", + " 'padding': 'same'}\n", + "\n", + " # --- Define lambda functions\n", + " conv = lambda x, filters, strides : layers.Conv3D(filters=filters, strides=strides, **kwargs)(x)\n", + " norm = lambda x : layers.BatchNormalization()(x)\n", + " relu = lambda x : layers.ReLU()(x)\n", + "\n", + " # --- Define stride-1, stride-2 blocks\n", + " conv1 = lambda filters, x : relu(norm(conv(x, filters, strides=1)))\n", + " conv2 = lambda filters, x : relu(norm(conv(x, filters, strides=(1, 2, 2))))\n", + " \n", + " return conv1, conv2" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Advanced CNN\n", + "\n", + "In the past few years, several key advances in design choice have become popular in most state-of-the-art designs. These include:\n", + "\n", + "* separable convolutions (depthwise and pointwise operations)\n", + "* layer normalization (or group normalization)\n", + "* Gaussian error linear unit (GeLU) activation\n", + "\n", + "Many additional recent design advances are summarized well in the ConvNeXt (2022) paper: https://arxiv.org/pdf/2201.03545.pdf.\n", + "\n", + "Note that while these design choices may yield small incremental gains, a proportional increase in dataset size is commonly needed to maximize more complex approaches. \n", + "\n", + "The following code block creates nested lambda function to quickly implement CNN models using these strategies:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def create_blocks():\n", + " \"\"\"\n", + " Method to define advanced stride-1 and stride-2 convolutional blocks\n", + " \n", + " \"block\" = conv > norm > gelu\n", + " \n", + " \"\"\"\n", + " # --- Define kwargs dictionary\n", + " kwargs_point = {\n", + " 'kernel_size': 1,\n", + " 'padding': 'same',\n", + " 'strides': 1}\n", + " \n", + " kwargs_depth = {\n", + " 'kernel_size': (1, 3, 3),\n", + " 'padding': 'same'}\n", + "\n", + " # --- Define separable conv as depthwise + pointwise convolutions\n", + " conv = lambda x, filters, strides : layers.Conv3D(filters=filters, **kwargs_point)(\n", + " layers.Conv3D(filters=x.shape[-1], strides=strides, groups=x.shape[-1], **kwargs_depth)(x))\n", + " \n", + " # --- Define lambda functions\n", + " norm = lambda x : layers.LayerNormalization()(x)\n", + " gelu = lambda x : tf.nn.gelu(x)\n", + "\n", + " # --- Define stride-1, stride-2 blocks\n", + " conv1 = lambda filters, x : gelu(norm(conv(x, filters, strides=1)))\n", + " conv2 = lambda filters, x : gelu(norm(conv(x, filters, strides=(1, 2, 2))))\n", + " \n", + " return conv1, conv2" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Creating Layers\n", + "\n", + "In this tutorial, we will explore the following key variations in network topology:\n", + "\n", + "* total number of feature maps (channels) for each convolutional operation\n", + "* total number of convolutional blocks\n", + "\n", + "To define the model, we will use a for-loop to create a series of stride-2 and stride-1 convolutional blocks spanning a total of `n_blocks` repeats. After each subsampling operation (stride-2 convolution), the total number of features is scaled linearly based on the `filters` variable. \n", + "\n", + "After a series of convolutional blocks, a flatten operation is used to convert high dimensional feature maps into a single dimensional feature vector (note that you may alternatively implement a global pooling operation here as well). At this time, a single hidden layer is defined using a dense matrix multiplication and ReLU nonlinearity. The final logit scores should be implemented using a two-element projection operation (non-activated matrix multiplication). \n", + "\n", + "The following code block will flexibly define a CNN model using the hyperparameters defined above:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def create_layers(x, p, hidden_size=64):\n", + " \"\"\"\n", + " Method to create model layers based on hyperparameters defined in p\n", + " \n", + " \"\"\"\n", + " # --- Create lambda functions for creating blocks\n", + " conv1, conv2 = create_blocks()\n", + " \n", + " # --- Create lambda function of extracting last layer\n", + " last = lambda blocks : list(blocks.values())[-1]\n", + " \n", + " # --- Create first conv layer\n", + " blocks = {}\n", + " blocks['l0'] = conv1(p['filters'], x)\n", + " \n", + " # --- Create remaining conv layers\n", + " for i in range(p['n_blocks']):\n", + " layer_key = 'l{}'.format(i + 1)\n", + " n_filters = p['filters'] * (i + 2)\n", + " blocks[layer_key] = conv1(n_filters, conv2(n_filters, last(blocks)))\n", + " \n", + " # --- Create hidden layer\n", + " blocks['f0'] = layers.Flatten()(last(blocks))\n", + " blocks['h0'] = layers.Dense(hidden_size, activation='relu')(blocks['f0'])\n", + " \n", + " # --- Create final logit scores\n", + " blocks['pna'] = layers.Dense(2, name='pna')(blocks['h0'])\n", + "\n", + " return blocks" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let us test the code block here:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "blocks = create_layers(x=inputs['dat'], p=p)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Model\n", + "\n", + "Putting everything together, use the following cell to create and compile a convolutional neural network corresponding the target `row` of hyperparameter values. By default, the following initial configurations are good baseline values for training hyperparameters:\n", + "\n", + "* Optimizer: Adam\n", + "* Loss: softmax cross-entropy\n", + "* Learning rate: 2e-4" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Load hyperparameters\n", + "p = params.load('./hyper.csv', row=0)\n", + "\n", + "# --- Define blocks\n", + "blocks = create_layers(x=inputs['dat'], p=p)\n", + "\n", + "# --- Create model\n", + "model = Model(inputs=inputs, outputs=blocks['pna'])\n", + "\n", + "# --- Compile model\n", + "model.compile(\n", + " optimizer=optimizers.Adam(learning_rate=2e-4), \n", + " loss={'pna': losses.SparseCategoricalCrossentropy(from_logits=True)}, \n", + " metrics={'pna': 'sparse_categorical_accuracy'})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To check the properties of the created model object, use the `model.summary()` method:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Print model summary\n", + "model.summary()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Model Training\n", + "\n", + "### In-Memory Data\n", + "\n", + "The following line of code will load all training data into RAM memory. This strategy can be effective for increasing speed of training for small to medium-sized datasets." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Load data into memory\n", + "client.load_data_in_memory()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Tensorboard\n", + "\n", + "To use Tensorboard, create the necessary Keras callbacks:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from tensorflow.keras import callbacks \n", + "tensorboard_callback = callbacks.TensorBoard('./logs/{}'.format(p['name']))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Training\n", + "\n", + "Once the model has been compiled and the data prepared (via a generator), training can be invoked using the `model.fit(...)` method. Ensure that both the training and validation data generators are used. In this particular example, we are defining arbitrary epochs of 50 steps each. Training will proceed for 20 epochs in total. Validation statistics will be assess every fifth epoch. As needed, tune these arugments as need." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "model.fit(\n", + " x=gen_train, \n", + " steps_per_epoch=50, \n", + " epochs=20,\n", + " validation_data=gen_valid,\n", + " validation_steps=50,\n", + " validation_freq=5,\n", + " callbacks=[tensorboard_callback]\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Launching Tensorboard\n", + "\n", + "After running several iterations, start Tensorboard using the following cells. After Tensorboard has registered the first several checkpoints, subsequent data will be updated automatically (asynchronously) and model training can be resumed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%load_ext tensorboard\n", + "%tensorboard --logdir logs" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Saving and Loading a Model\n", + "\n", + "After a model has been successfully trained, it can be saved and/or loaded by simply using the `model.save()` and `models.load_model()` methods. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Serialize a model\n", + "model.save('./model.hdf5')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- Load a serialized model\n", + "del model\n", + "model = models.load_model('./model.hdf5', compile=False)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.5" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}