{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tile Generation Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to the tile generation tutorial!\n",
    "\n",
    "As a whole slide image is too large for deep learning model training, a slide is often divded into a set of small tiles, and used for training. For tile-based whole slide image analysis, generating tiles and labels is an important and laborious step. With LUNA tiling CLIs and tutorials, you can easily generate tile labels and get your data ready for downstream analysis. In this notebook, we will see how to generate tiles and labels using LUNA tiling CLIs. Here are the main steps we will review:\n",
    "\n",
    "- Load slides\n",
    "- Generate tiles, labels\n",
    "- Collect tiles for model training\n",
    "\n",
    "Through out this notebook, we will use different method parameter files. Please refer to the example parameter files in the `configs` directory to follow these steps.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load slides"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step in generating tiles is to load slides in a data store, where our results will be generated. We will use **load_slide** CLI to prepare slides from a whole slide image (WSI) table to our analysis location. The slide is represented as a WholeSlideImage data type.\n",
    "\n",
    "All LUNA tiling CLIs offer a help option. To check the the CLI arguments, simply run your CLI with `--help` option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: load_slide [OPTIONS]\n",
      "\n",
      "Options:\n",
      "  -a, --app_config TEXT         application configuration yaml file. See\n",
      "                                config.yaml.template for details.  [required]\n",
      "\n",
      "  -s, --datastore_id TEXT       datastore name. usually a slide id.\n",
      "                                [required]\n",
      "\n",
      "  -m, --method_param_path TEXT  json parameter file with path to a WSI delta\n",
      "                                table.  [required]\n",
      "\n",
      "  --help                        Show this message and exit.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "load_slide --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import multiprocessing\n",
    "import subprocess\n",
    "\n",
    "slide_ids = ['2551571', '2551531', '2551028', '2551389', '2551129']\n",
    "\n",
    "# simple wrapper around the cli for multiple slides\n",
    "def pool_process(func, slides):\n",
    "    pool = multiprocessing.Pool(3)\n",
    "    pool.map(func, slides)\n",
    "    pool.close()\n",
    "    pool.join()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# call load_slide as subprocess\n",
    "def call_load_slide(slide):\n",
    "    subprocess.run(f\"load_slide -a configs/app_config.yaml -s {slide} -m configs/load_slides.yaml\", shell=True)\n",
    "    return slide\n",
    "\n",
    "pool_process(call_load_slide, slide_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once this step is done, the data store will be created at your `datastore_path` or `PRO_12-123/tiles` with the example method parameters.\n",
    "\n",
    "Let's take a look at the WholeSlideImage location for slide 2551571. We'll see that this process created a softlink pointing to the svs image path, along with a `metadata.json`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 1.0K\n",
      "lrwxrwxrwx 1 pashaa pashaa  104 Jul 13 13:29 data -> /gpfs/mskmindhdp_emc/user/shared_data_folder/pathology-tutorial/PRO_12-123/data/toy_data_set/2551571.svs\n",
      "-rwxrwxrwx 1 pashaa pashaa 3.1K Jul 13 13:29 metadata.json\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ls -lhtr PRO_12-123/tiles/2551571/ov_slides/WholeSlideImage/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate tiles and labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the main tiling step. The CLI generates tiles, populates otsu and purple scores along with the regional annotation label. An otsu score is calculated using the otsu foreground/background detection algorithm commonly used to filter out the background of the slide. Purple scores are calculated to provide additional guidance to H&E slide analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: generate_tiles [OPTIONS]\n",
      "\n",
      "Options:\n",
      "  -a, --app_config TEXT         application configuration yaml file. See\n",
      "                                config.yaml.template for details.  [required]\n",
      "\n",
      "  -s, --datastore_id TEXT       datastore name. usually a slide id.\n",
      "                                [required]\n",
      "\n",
      "  -m, --method_param_path TEXT  json file with method parameters for tile\n",
      "                                generation and filtering.  [required]\n",
      "\n",
      "  --help                        Show this message and exit.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "generate_tiles --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this method configuration, the tile size is set to 128, scale factor to 16 and slide magnification (from slide metadata) to 20. In this example, we label the tiles with the default labels provided by the regional annotations. Note that we keep only the tiles that have been annotated and have an otsu score above 0.5 for our analysis. Please refer to `configs/generate_tiles.yaml` for more details on the method parameters.\n",
    "\n",
    "Here we reserve 4 slides for model training, and 1 slide for testing. For training, we will only generate tiles for the areas that have been annotated by the pathologists, so the model will have ground-truth labels. For testing, we will generate tiles for the whole slide.\n",
    "\n",
    "We reserve the test slide, to be annotated by the model in the inference notebook. For this test slide, as mentioned before, we generate tiles for *all* tissue regions (otsu score > 0.5). Note here that we use a different config file `configs/generate_tiles_all_tissues.yaml` which excludes parameters `project_id`, `labelset`, `annotation_table_path` which pertains to the regional annotation.\n",
    "\n",
    "Depending on the size of the WSI and tiles, this step can take up to 10 minutes per slide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "slide_ids_train = ['2551571', '2551531', '2551028', '2551389']\n",
    "slide_ids_test = '2551129'\n",
    "\n",
    "# call generate_tiles as subprocess\n",
    "def call_generate_tiles(slide):\n",
    "    subprocess.run(f\"generate_tiles -a configs/app_config.yaml -s {slide} -m configs/generate_tiles.yaml\", shell=True)\n",
    "    return slide\n",
    "\n",
    "pool_process(call_generate_tiles, slide_ids_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "generate_tiles -a configs/app_config.yaml -s 2551129 -m configs/generate_tiles_all_tissues.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the step is done, you can find the tiles and score CSV for your slide, at your output location. For slide id 2551571, we have the tile image and metadata stored at `PRO_12-123/tiles/2551571/ov_default_labels/TileImages/data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 3.6G\n",
      "-rwxrwxrwx 1 pashaa pashaa 3.6G Jul 13 13:44 tiles.slice.pil\n",
      "-rwxrwxrwx 1 pashaa pashaa 207K Jul 13 13:45 address.slice.csv\n",
      "-rwxrwxrwx 1 pashaa pashaa  635 Jul 13 13:45 metadata.json\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ls -lhtr PRO_12-123/tiles/2551571/ov_default_labels/TileImages/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the tile metadata in the output CSV.\n",
    "\n",
    "The tile otsu_score, purple score and regional annotation labels are stored along tile metadata such as address, coordinates, size, and offset. From the log, we see that out of total 206830 tiles only a subset that meets the filter criteria has been kept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>address</th>\n",
       "      <th>coordinates</th>\n",
       "      <th>otsu_score</th>\n",
       "      <th>purple_score</th>\n",
       "      <th>regional_label</th>\n",
       "      <th>tile_image_offset</th>\n",
       "      <th>tile_image_length</th>\n",
       "      <th>tile_image_size_xy</th>\n",
       "      <th>tile_image_mode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>x107_y183_z20</td>\n",
       "      <td>(107, 183)</td>\n",
       "      <td>0.859375</td>\n",
       "      <td>0.984375</td>\n",
       "      <td>veins</td>\n",
       "      <td>2.845409e+08</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>x107_y184_z20</td>\n",
       "      <td>(107, 184)</td>\n",
       "      <td>0.890625</td>\n",
       "      <td>0.984375</td>\n",
       "      <td>veins</td>\n",
       "      <td>2.845901e+08</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>x107_y185_z20</td>\n",
       "      <td>(107, 185)</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>veins</td>\n",
       "      <td>2.846392e+08</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>x107_y192_z20</td>\n",
       "      <td>(107, 192)</td>\n",
       "      <td>0.593750</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>veins</td>\n",
       "      <td>2.849833e+08</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>x108_y183_z20</td>\n",
       "      <td>(108, 183)</td>\n",
       "      <td>0.875000</td>\n",
       "      <td>0.953125</td>\n",
       "      <td>veins</td>\n",
       "      <td>2.918154e+08</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2615</th>\n",
       "      <td>x453_y148_z20</td>\n",
       "      <td>(453, 148)</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>lympho_rich_tumor</td>\n",
       "      <td>3.706257e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2616</th>\n",
       "      <td>x454_y146_z20</td>\n",
       "      <td>(454, 146)</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>lympho_rich_tumor</td>\n",
       "      <td>3.713237e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2617</th>\n",
       "      <td>x454_y147_z20</td>\n",
       "      <td>(454, 147)</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>lympho_rich_tumor</td>\n",
       "      <td>3.713286e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2618</th>\n",
       "      <td>x454_y148_z20</td>\n",
       "      <td>(454, 148)</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>lympho_rich_tumor</td>\n",
       "      <td>3.713335e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2619</th>\n",
       "      <td>x455_y143_z20</td>\n",
       "      <td>(455, 143)</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>lympho_rich_stroma</td>\n",
       "      <td>3.719725e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2620 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            address coordinates  otsu_score  purple_score      regional_label  \\\n",
       "0     x107_y183_z20  (107, 183)    0.859375      0.984375               veins   \n",
       "1     x107_y184_z20  (107, 184)    0.890625      0.984375               veins   \n",
       "2     x107_y185_z20  (107, 185)    1.000000      1.000000               veins   \n",
       "3     x107_y192_z20  (107, 192)    0.593750      1.000000               veins   \n",
       "4     x108_y183_z20  (108, 183)    0.875000      0.953125               veins   \n",
       "...             ...         ...         ...           ...                 ...   \n",
       "2615  x453_y148_z20  (453, 148)    1.000000      1.000000   lympho_rich_tumor   \n",
       "2616  x454_y146_z20  (454, 146)    1.000000      1.000000   lympho_rich_tumor   \n",
       "2617  x454_y147_z20  (454, 147)    1.000000      1.000000   lympho_rich_tumor   \n",
       "2618  x454_y148_z20  (454, 148)    1.000000      1.000000   lympho_rich_tumor   \n",
       "2619  x455_y143_z20  (455, 143)    1.000000      1.000000  lympho_rich_stroma   \n",
       "\n",
       "      tile_image_offset  tile_image_length  tile_image_size_xy tile_image_mode  \n",
       "0          2.845409e+08            49152.0               128.0             RGB  \n",
       "1          2.845901e+08            49152.0               128.0             RGB  \n",
       "2          2.846392e+08            49152.0               128.0             RGB  \n",
       "3          2.849833e+08            49152.0               128.0             RGB  \n",
       "4          2.918154e+08            49152.0               128.0             RGB  \n",
       "...                 ...                ...                 ...             ...  \n",
       "2615       3.706257e+09            49152.0               128.0             RGB  \n",
       "2616       3.713237e+09            49152.0               128.0             RGB  \n",
       "2617       3.713286e+09            49152.0               128.0             RGB  \n",
       "2618       3.713335e+09            49152.0               128.0             RGB  \n",
       "2619       3.719725e+09            49152.0               128.0             RGB  \n",
       "\n",
       "[2620 rows x 9 columns]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# For a train slide, we have generated tiles for annotated regions, and populated regional_labels\n",
    "df = pd.read_csv(\"PRO_12-123/tiles/2551571/ov_default_labels/TileImages/data/address.slice.csv\")\n",
    "df"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=== LOGS ===\n",
    "2021-07-06 17:58:24,533 - INFO - data_processing.pathology.common.preprocess - Params = {'input_wsi_tag': 'ov_slides', 'job_tag': 'ov_default_labels', 'tile_size': 128, 'scale_factor': 16, 'magnification': 20, 'project_id': 'PRO_12-123', 'labelset': 'DEFAULT_LABELS', 'filter': {'otsu_score': 0.5}, 'root_path': '/gpfs/mskmindhdp_emc/user/shared_data_folder/pathology-tutorial/PRO_12-123/tiles', 'annotation_table_path': '/gpfs/mskmindhdp_emc/user/shared_data_folder/pathology-tutorial/PRO_12-123/tables/REGIONAL_METADATA_RESULTS'}\n",
    "2021-07-06 17:58:24,645 - INFO - data_processing.pathology.common.preprocess - Slide size = [71711,47602]\n",
    "2021-07-06 17:58:24,646 - INFO - data_processing.pathology.common.preprocess - Normalized magnification scale factor for 20x is 1, overall thumbnail scale factor is 16\n",
    "2021-07-06 17:58:24,646 - INFO - data_processing.pathology.common.preprocess - Requested tile size=128, tile size at full magnficiation=128, tile size at thumbnail=8\n",
    "2021-07-06 17:58:26,508 - INFO - data_processing.pathology.common.preprocess - tiles x 561, tiles y 372\n",
    "2021-07-06 17:58:27,282 - INFO - data_processing.pathology.common.preprocess - Number of tiles in raster: 206830\n",
    "2021-07-06 18:01:42,894 - INFO - data_processing.pathology.common.preprocess - Proccessing tiles [10000,78149]\n",
    "2021-07-06 18:02:41,410 - INFO - data_processing.pathology.common.preprocess - Proccessing tiles [20000,78149]\n",
    "2021-07-06 18:03:39,555 - INFO - data_processing.pathology.common.preprocess - Proccessing tiles [30000,78149]\n",
    "2021-07-06 18:04:36,638 - INFO - data_processing.pathology.common.preprocess - Proccessing tiles [40000,78149]\n",
    "2021-07-06 18:05:33,419 - INFO - data_processing.pathology.common.preprocess - Proccessing tiles [50000,78149]\n",
    "2021-07-06 18:06:30,591 - INFO - data_processing.pathology.common.preprocess - Proccessing tiles [60000,78149]\n",
    "2021-07-06 18:07:31,272 - INFO - data_processing.pathology.common.preprocess - Proccessing tiles [70000,78149]\n",
    "2021-07-06 18:08:21,267 - INFO - data_processing.pathology.common.preprocess - Saved tile scores and images at /gpfs/mskmindhdp_emc/user/shared_data_folder/pathology-tutorial/PRO_12-123/tiles/2551571/ov_default_labels/TileImages/data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the test slide, we keep all tissue regions, so we have far more tiles generated. Notice we don't have the regional labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>address</th>\n",
       "      <th>coordinates</th>\n",
       "      <th>otsu_score</th>\n",
       "      <th>purple_score</th>\n",
       "      <th>tile_image_offset</th>\n",
       "      <th>tile_image_length</th>\n",
       "      <th>tile_image_size_xy</th>\n",
       "      <th>tile_image_mode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>x1_y1_z20</td>\n",
       "      <td>(1, 1)</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>x1_y2_z20</td>\n",
       "      <td>(1, 2)</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.915200e+04</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>x2_y1_z20</td>\n",
       "      <td>(2, 1)</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.830400e+04</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>x2_y2_z20</td>\n",
       "      <td>(2, 2)</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.474560e+05</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>x3_y1_z20</td>\n",
       "      <td>(3, 1)</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.966080e+05</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28750</th>\n",
       "      <td>x636_y2_z20</td>\n",
       "      <td>(636, 2)</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.413120e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28751</th>\n",
       "      <td>x636_y3_z20</td>\n",
       "      <td>(636, 3)</td>\n",
       "      <td>0.8125</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.413169e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28752</th>\n",
       "      <td>x637_y1_z20</td>\n",
       "      <td>(637, 1)</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.413218e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28753</th>\n",
       "      <td>x637_y2_z20</td>\n",
       "      <td>(637, 2)</td>\n",
       "      <td>0.9375</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.413267e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28754</th>\n",
       "      <td>x637_y3_z20</td>\n",
       "      <td>(637, 3)</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.413317e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28755 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           address coordinates  otsu_score  purple_score  tile_image_offset  \\\n",
       "0        x1_y1_z20      (1, 1)      1.0000           0.0       0.000000e+00   \n",
       "1        x1_y2_z20      (1, 2)      1.0000           0.0       4.915200e+04   \n",
       "2        x2_y1_z20      (2, 1)      1.0000           0.0       9.830400e+04   \n",
       "3        x2_y2_z20      (2, 2)      1.0000           0.0       1.474560e+05   \n",
       "4        x3_y1_z20      (3, 1)      1.0000           0.0       1.966080e+05   \n",
       "...            ...         ...         ...           ...                ...   \n",
       "28750  x636_y2_z20    (636, 2)      0.8750           0.0       1.413120e+09   \n",
       "28751  x636_y3_z20    (636, 3)      0.8125           0.0       1.413169e+09   \n",
       "28752  x637_y1_z20    (637, 1)      1.0000           0.0       1.413218e+09   \n",
       "28753  x637_y2_z20    (637, 2)      0.9375           0.0       1.413267e+09   \n",
       "28754  x637_y3_z20    (637, 3)      0.8750           0.0       1.413317e+09   \n",
       "\n",
       "       tile_image_length  tile_image_size_xy tile_image_mode  \n",
       "0                49152.0               128.0             RGB  \n",
       "1                49152.0               128.0             RGB  \n",
       "2                49152.0               128.0             RGB  \n",
       "3                49152.0               128.0             RGB  \n",
       "4                49152.0               128.0             RGB  \n",
       "...                  ...                 ...             ...  \n",
       "28750            49152.0               128.0             RGB  \n",
       "28751            49152.0               128.0             RGB  \n",
       "28752            49152.0               128.0             RGB  \n",
       "28753            49152.0               128.0             RGB  \n",
       "28754            49152.0               128.0             RGB  \n",
       "\n",
       "[28755 rows x 8 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For the test slide, we have generated tiles for all tissue regions\n",
    "df = pd.read_csv(\"PRO_12-123/tiles/2551129/ov_default_labels/TileImages/data/address.slice.csv\")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collect tiles for model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have created tile labels, we can use **collect_tiles** CLI to collect the tile metadata as a set of parquet tables and save the outputs for multiple slide ids in the same dataset. This step is done to gather our dataset for model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: collect_tiles [OPTIONS]\n",
      "\n",
      "Options:\n",
      "  -a, --app_config TEXT         application configuration yaml file. See\n",
      "                                config.yaml.template for details.  [required]\n",
      "\n",
      "  -s, --datastore_id TEXT       datastore name. usually a slide id.\n",
      "                                [required]\n",
      "\n",
      "  -m, --method_param_path TEXT  json file with method parameters including\n",
      "                                input, output details.  [required]\n",
      "\n",
      "  --help                        Show this message and exit.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "collect_tiles --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, it is critical to note that our model will train on the 4 slides reserved for trainig. We have reserved one slide out of the model training step in order to use it for the inference step. \n",
    "\n",
    "We will call **collect_tiles** on the training slides to prepare a dataset for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "slide_ids_train = ['2551571', '2551531', '2551028', '2551389']\n",
    "\n",
    "# call collect_tiles as subprocess\n",
    "def call_collect_tiles(slide):\n",
    "    subprocess.run(f\"collect_tiles -a configs/app_config.yaml -s {slide} -m configs/collect_tiles.yaml\", shell=True)\n",
    "\n",
    "pool_process(call_collect_tiles, slide_ids_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the output. The collected parquet files can be loaded as a pyarrow ParquetDataset, and be converted to Pandas Dataframe.\n",
    "\n",
    "You'll notice the table is indexed by `patient_id`, `slide id` and `address`. The `data_path` points to the tile image file. The rest of the metadata stored in this table are similar to the output of **generate_tiles** CLI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>coordinates</th>\n",
       "      <th>otsu_score</th>\n",
       "      <th>purple_score</th>\n",
       "      <th>regional_label</th>\n",
       "      <th>tile_image_offset</th>\n",
       "      <th>tile_image_length</th>\n",
       "      <th>tile_image_size_xy</th>\n",
       "      <th>tile_image_mode</th>\n",
       "      <th>data_path</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>patient_id</th>\n",
       "      <th>id_slide_container</th>\n",
       "      <th>address</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">4</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">2551028</th>\n",
       "      <th>x128_y72_z20</th>\n",
       "      <td>(128, 72)</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>arteries</td>\n",
       "      <td>4.446781e+08</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "      <td>/gpfs/mskmindhdp_emc/user/shared_data_folder/p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x128_y73_z20</th>\n",
       "      <td>(128, 73)</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>arteries</td>\n",
       "      <td>4.447273e+08</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "      <td>/gpfs/mskmindhdp_emc/user/shared_data_folder/p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x128_y74_z20</th>\n",
       "      <td>(128, 74)</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>arteries</td>\n",
       "      <td>4.447764e+08</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "      <td>/gpfs/mskmindhdp_emc/user/shared_data_folder/p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x128_y75_z20</th>\n",
       "      <td>(128, 75)</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>arteries</td>\n",
       "      <td>4.448256e+08</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "      <td>/gpfs/mskmindhdp_emc/user/shared_data_folder/p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x128_y76_z20</th>\n",
       "      <td>(128, 76)</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>arteries</td>\n",
       "      <td>4.448748e+08</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "      <td>/gpfs/mskmindhdp_emc/user/shared_data_folder/p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">1</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">2551571</th>\n",
       "      <th>x453_y148_z20</th>\n",
       "      <td>(453, 148)</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>lympho_rich_tumor</td>\n",
       "      <td>3.706257e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "      <td>/gpfs/mskmindhdp_emc/user/shared_data_folder/p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x454_y146_z20</th>\n",
       "      <td>(454, 146)</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>lympho_rich_tumor</td>\n",
       "      <td>3.713237e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "      <td>/gpfs/mskmindhdp_emc/user/shared_data_folder/p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x454_y147_z20</th>\n",
       "      <td>(454, 147)</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>lympho_rich_tumor</td>\n",
       "      <td>3.713286e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "      <td>/gpfs/mskmindhdp_emc/user/shared_data_folder/p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x454_y148_z20</th>\n",
       "      <td>(454, 148)</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>lympho_rich_tumor</td>\n",
       "      <td>3.713335e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "      <td>/gpfs/mskmindhdp_emc/user/shared_data_folder/p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x455_y143_z20</th>\n",
       "      <td>(455, 143)</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>lympho_rich_stroma</td>\n",
       "      <td>3.719725e+09</td>\n",
       "      <td>49152.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>RGB</td>\n",
       "      <td>/gpfs/mskmindhdp_emc/user/shared_data_folder/p...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14696 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            coordinates  otsu_score  \\\n",
       "patient_id id_slide_container address                                 \n",
       "4          2551028            x128_y72_z20    (128, 72)         1.0   \n",
       "                              x128_y73_z20    (128, 73)         1.0   \n",
       "                              x128_y74_z20    (128, 74)         1.0   \n",
       "                              x128_y75_z20    (128, 75)         1.0   \n",
       "                              x128_y76_z20    (128, 76)         1.0   \n",
       "...                                                 ...         ...   \n",
       "1          2551571            x453_y148_z20  (453, 148)         1.0   \n",
       "                              x454_y146_z20  (454, 146)         1.0   \n",
       "                              x454_y147_z20  (454, 147)         1.0   \n",
       "                              x454_y148_z20  (454, 148)         1.0   \n",
       "                              x455_y143_z20  (455, 143)         1.0   \n",
       "\n",
       "                                             purple_score      regional_label  \\\n",
       "patient_id id_slide_container address                                           \n",
       "4          2551028            x128_y72_z20            1.0            arteries   \n",
       "                              x128_y73_z20            1.0            arteries   \n",
       "                              x128_y74_z20            1.0            arteries   \n",
       "                              x128_y75_z20            1.0            arteries   \n",
       "                              x128_y76_z20            1.0            arteries   \n",
       "...                                                   ...                 ...   \n",
       "1          2551571            x453_y148_z20           1.0   lympho_rich_tumor   \n",
       "                              x454_y146_z20           1.0   lympho_rich_tumor   \n",
       "                              x454_y147_z20           1.0   lympho_rich_tumor   \n",
       "                              x454_y148_z20           1.0   lympho_rich_tumor   \n",
       "                              x455_y143_z20           1.0  lympho_rich_stroma   \n",
       "\n",
       "                                             tile_image_offset  \\\n",
       "patient_id id_slide_container address                            \n",
       "4          2551028            x128_y72_z20        4.446781e+08   \n",
       "                              x128_y73_z20        4.447273e+08   \n",
       "                              x128_y74_z20        4.447764e+08   \n",
       "                              x128_y75_z20        4.448256e+08   \n",
       "                              x128_y76_z20        4.448748e+08   \n",
       "...                                                        ...   \n",
       "1          2551571            x453_y148_z20       3.706257e+09   \n",
       "                              x454_y146_z20       3.713237e+09   \n",
       "                              x454_y147_z20       3.713286e+09   \n",
       "                              x454_y148_z20       3.713335e+09   \n",
       "                              x455_y143_z20       3.719725e+09   \n",
       "\n",
       "                                             tile_image_length  \\\n",
       "patient_id id_slide_container address                            \n",
       "4          2551028            x128_y72_z20             49152.0   \n",
       "                              x128_y73_z20             49152.0   \n",
       "                              x128_y74_z20             49152.0   \n",
       "                              x128_y75_z20             49152.0   \n",
       "                              x128_y76_z20             49152.0   \n",
       "...                                                        ...   \n",
       "1          2551571            x453_y148_z20            49152.0   \n",
       "                              x454_y146_z20            49152.0   \n",
       "                              x454_y147_z20            49152.0   \n",
       "                              x454_y148_z20            49152.0   \n",
       "                              x455_y143_z20            49152.0   \n",
       "\n",
       "                                             tile_image_size_xy  \\\n",
       "patient_id id_slide_container address                             \n",
       "4          2551028            x128_y72_z20                128.0   \n",
       "                              x128_y73_z20                128.0   \n",
       "                              x128_y74_z20                128.0   \n",
       "                              x128_y75_z20                128.0   \n",
       "                              x128_y76_z20                128.0   \n",
       "...                                                         ...   \n",
       "1          2551571            x453_y148_z20               128.0   \n",
       "                              x454_y146_z20               128.0   \n",
       "                              x454_y147_z20               128.0   \n",
       "                              x454_y148_z20               128.0   \n",
       "                              x455_y143_z20               128.0   \n",
       "\n",
       "                                            tile_image_mode  \\\n",
       "patient_id id_slide_container address                         \n",
       "4          2551028            x128_y72_z20              RGB   \n",
       "                              x128_y73_z20              RGB   \n",
       "                              x128_y74_z20              RGB   \n",
       "                              x128_y75_z20              RGB   \n",
       "                              x128_y76_z20              RGB   \n",
       "...                                                     ...   \n",
       "1          2551571            x453_y148_z20             RGB   \n",
       "                              x454_y146_z20             RGB   \n",
       "                              x454_y147_z20             RGB   \n",
       "                              x454_y148_z20             RGB   \n",
       "                              x455_y143_z20             RGB   \n",
       "\n",
       "                                                                                     data_path  \n",
       "patient_id id_slide_container address                                                           \n",
       "4          2551028            x128_y72_z20   /gpfs/mskmindhdp_emc/user/shared_data_folder/p...  \n",
       "                              x128_y73_z20   /gpfs/mskmindhdp_emc/user/shared_data_folder/p...  \n",
       "                              x128_y74_z20   /gpfs/mskmindhdp_emc/user/shared_data_folder/p...  \n",
       "                              x128_y75_z20   /gpfs/mskmindhdp_emc/user/shared_data_folder/p...  \n",
       "                              x128_y76_z20   /gpfs/mskmindhdp_emc/user/shared_data_folder/p...  \n",
       "...                                                                                        ...  \n",
       "1          2551571            x453_y148_z20  /gpfs/mskmindhdp_emc/user/shared_data_folder/p...  \n",
       "                              x454_y146_z20  /gpfs/mskmindhdp_emc/user/shared_data_folder/p...  \n",
       "                              x454_y147_z20  /gpfs/mskmindhdp_emc/user/shared_data_folder/p...  \n",
       "                              x454_y148_z20  /gpfs/mskmindhdp_emc/user/shared_data_folder/p...  \n",
       "                              x455_y143_z20  /gpfs/mskmindhdp_emc/user/shared_data_folder/p...  \n",
       "\n",
       "[14696 rows x 9 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyarrow.parquet import ParquetDataset\n",
    "\n",
    "ds = ParquetDataset('PRO_12-123/tiles/ov_tileset').read().to_pandas()\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! Now you have the tiles images and labels ready to train your model."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}