Tile Generation Tutorial
Contents
Tile Generation Tutorial#
Welcome to the tile generation tutorial!
As a whole slide image is too large for deep learning model training, a slide is often divded into a set of small tiles, and used for training. For tile-based whole slide image analysis, generating tiles and labels is an important and laborious step. With LUNA tiling CLIs and tutorials, you can easily generate tile labels and get your data ready for downstream analysis. In this notebook, we will see how to generate tiles and labels using LUNA tiling CLIs. Here are the main steps we will review:
Load slides
Generate tiles, labels
Collect tiles for model training
Through out this notebook, we will use different method parameter files. Please refer to the example parameter files in the configs directory to follow these steps.
1. Load slides#
The first step in generating tiles is to load slides in a data store, where our results will be generated. We will use load_slide CLI to prepare slides from a whole slide image (WSI) table to our analysis location. The slide is represented as a WholeSlideImage data type.
All LUNA tiling CLIs offer a help option. To check the the CLI arguments, simply run your CLI with --help option.
[1]:
!load_slide --help
Usage: load_slide [OPTIONS]
Load a slide to the datastore from the whole slide image table.
app_config - application configuration yaml file. See config.yaml.template
for details.
datastore_id - datastore name. usually a slide id.
method_param_path - json parameter file with path to a WSI delta table.
- job_tag: job tag to use for loading the slide
- table_path: path to the whole slide image table
- datastore_path: path to store data
Options:
-a, --app_config TEXT application configuration yaml file. See
config.yaml.template for details. [required]
-s, --datastore_id TEXT datastore name. usually a slide id.
[required]
-m, --method_param_path TEXT json parameter file with path to a WSI delta
table. [required]
--help Show this message and exit.
[2]:
import multiprocessing
import subprocess
slide_ids = ['01OV002-bd8cdc70-3d46-40ae-99c4-90ef77', '01OV002-ed65cf94-8bc6-492b-9149-adc16f',
'01OV007-9b90eb78-2f50-4aeb-b010-d642f9', '01OV008-308ad404-7079-4ff8-8232-12ee2e',
'01OV008-7579323e-2fae-43a9-b00f-a15c28']
# simple wrapper around the cli for multiple slides
def pool_process(func, slides):
pool = multiprocessing.Pool(3)
pool.map(func, slides)
pool.close()
pool.join()
[12]:
# call load_slide as subprocess
def call_load_slide(slide):
subprocess.run(f"python3 -m luna.pathology.cli.load_slide -a ../conf/app_config.yaml -s {slide} -m ../conf/load_slides.yaml", shell=True)
return slide
pool_process(call_load_slide, slide_ids)
Once this step is done, the data store will be created at your datastore_path or PRO_12-123/tiles with the example method parameters.
Let’s take a look at the WholeSlideImage location for slide 2551571. We’ll see that this process created a softlink pointing to the svs image path, along with a metadata.json
[16]:
!ls -lhtr ~/vmount/PRO_12-123/tables/tiles/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77/ov_slides/WholeSlideImage/
total 4.0K
lrwxr-xr-x 1 rosed2 rosed2 91 Dec 20 21:06 data -> /home/rosed2/vmount/PRO_12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
-rw-r--r-- 1 rosed2 rosed2 3.1K Dec 20 21:06 metadata.json
2. Generate tiles and labels#
This is the main tiling step. The CLI generates tiles, populates otsu and purple scores along with the regional annotation label. An otsu score is calculated using the otsu foreground/background detection algorithm commonly used to filter out the background of the slide. Purple scores are calculated to provide additional guidance to H&E slide analysis.
[14]:
!generate_tiles --help
Usage: generate_tiles [OPTIONS]
Generate tile addresses, scores and optionally annotation labels.
app_config - application configuration yaml file. See config.yaml.template
for details.
datastore_id - datastore name. usually a slide id.
method_param_path - json file with method parameters for tile generation and
filtering.
- input_wsi_tag: job tag used to load slides
- job_tag: job tag for generating tile labels
- tile_size: size of patches
- scale_factor: desired downscale factor
- requested_magnification: desired magnification
- root_path: path to output data
- filter: optional filter map to select subset of the tiles e.g. {
"otsu_score": 0.5 }
- project_id: optional project id, if using regional annotations
- labelset: optional annotation labelset name, if using regional annotations
- annotation_table_path: optional path to the regional annotation table
Options:
-a, --app_config TEXT application configuration yaml file. See
config.yaml.template for details. [required]
-s, --datastore_id TEXT datastore name. usually a slide id.
[required]
-m, --method_param_path TEXT json file with method parameters for tile
generation and filtering. [required]
--help Show this message and exit.
With this method configuration, the tile size is set to 128, scale factor to 16 and slide magnification (from slide metadata) to 20. In this example, we label the tiles with the default labels provided by the regional annotations. Note that we keep only the tiles that have been annotated and have an otsu score above 0.5 for our analysis. Please refer to ~/luna/conf/generate_tiles.yaml for more details on the method parameters.
Here we reserve 4 slides for model training, and 1 slide for testing. For training, we will only generate tiles for the areas that have been annotated by the pathologists, so the model will have ground-truth labels. For testing, we will generate tiles for the whole slide.
We reserve the test slide, to be annotated by the model in the inference notebook. For this test slide, as mentioned before, we generate tiles for all tissue regions (otsu score > 0.5). Note here that we use a different config file ~/luna/conf/generate_tiles_all_tissues.yaml which excludes parameters project_id, labelset, annotation_table_path which pertains to the regional annotation.
Depending on the size of the WSI and tiles, this step can take up to 10 minutes per slide.
[17]:
slide_ids_train = ['01OV002-bd8cdc70-3d46-40ae-99c4-90ef77', '01OV002-ed65cf94-8bc6-492b-9149-adc16f',
'01OV007-9b90eb78-2f50-4aeb-b010-d642f9', '01OV008-308ad404-7079-4ff8-8232-12ee2e']
slide_ids_test = '01OV008-7579323e-2fae-43a9-b00f-a15c28'
# call generate_tiles as subprocess
def call_generate_tiles(slide):
subprocess.run(f"generate_tiles -a ../conf/app_config.yaml -s {slide} -m ../conf/generate_tiles.yaml", shell=True)
return slide
pool_process(call_generate_tiles, slide_ids_train)
!tree ../PRO_12-123/tables/tiles
../PRO_12-123/tables/tiles
├── 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77
│ ├── ov_default_labels
│ │ └── TileImages
│ │ └── data
│ │ ├── address.slice.csv
│ │ ├── metadata.json
│ │ └── tiles.slice.pil
│ └── ov_slides
│ └── WholeSlideImage
│ ├── data -> /home/rosed2/vmount/PRO_12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
│ └── metadata.json
├── 01OV002-ed65cf94-8bc6-492b-9149-adc16f
│ ├── ov_default_labels
│ │ └── TileImages
│ │ └── data
│ │ ├── address.slice.csv
│ │ ├── metadata.json
│ │ └── tiles.slice.pil
│ └── ov_slides
│ └── WholeSlideImage
│ ├── data -> /home/rosed2/vmount/PRO_12-123/data/toy_data_set/01OV002-ed65cf94-8bc6-492b-9149-adc16f.svs
│ └── metadata.json
├── 01OV007-9b90eb78-2f50-4aeb-b010-d642f9
│ ├── ov_default_labels
│ │ └── TileImages
│ │ └── data
│ │ ├── address.slice.csv
│ │ ├── metadata.json
│ │ └── tiles.slice.pil
│ └── ov_slides
│ └── WholeSlideImage
│ ├── data -> /home/rosed2/vmount/PRO_12-123/data/toy_data_set/01OV007-9b90eb78-2f50-4aeb-b010-d642f9.svs
│ └── metadata.json
├── 01OV008-308ad404-7079-4ff8-8232-12ee2e
│ ├── ov_default_labels
│ │ └── TileImages
│ │ └── data
│ │ ├── address.slice.csv
│ │ ├── metadata.json
│ │ └── tiles.slice.pil
│ └── ov_slides
│ └── WholeSlideImage
│ ├── data -> /home/rosed2/vmount/PRO_12-123/data/toy_data_set/01OV008-308ad404-7079-4ff8-8232-12ee2e.svs
│ └── metadata.json
└── 01OV008-7579323e-2fae-43a9-b00f-a15c28
└── ov_slides
└── WholeSlideImage
├── data -> /home/rosed2/vmount/PRO_12-123/data/toy_data_set/01OV008-7579323e-2fae-43a9-b00f-a15c28.svs
└── metadata.json
27 directories, 22 files
[20]:
!generate_tiles \
-a ../conf/app_config.yaml \
-s 01OV008-7579323e-2fae-43a9-b00f-a15c28 \
-m ../conf/generate_tiles_all_tissues.yaml
2021-12-20 21:23:17,430 - INFO - root - FYI: Initalized logger, log file at: data-processing.log with handlers: [<StreamHandler <stderr> (INFO)>, <RotatingFileHandler /home/rosed2/vmount/notebooks/data-processing.log (INFO)>]
2021-12-20 21:23:17,436 - INFO - luna.common.config - loading config file ../conf/app_config.yaml
2021-12-20 21:23:17,441 - INFO - luna.common.config - loading config file /home/rosed2/vmount/conf/datastore.cfg
2021-12-20 21:23:17,445 - INFO - luna.common.DataStore - Configured datastore with {'GRAPH_STORE_ENABLED': False, 'GRAPH_URI': 'neo4j://localhost:7687', 'GRAPH_USER': 'neo4j', 'GRAPH_PASSWORD': 'password', 'OBJECT_STORE_ENABLED': False, 'MINIO_URI': 'localhost:8001', 'MINIO_USER': 'minio', 'MINIO_PASSWORD': 'password', 'DOC_STORE_ENABLED': False, 'MONGODB_URI': 'mongodb://localhost:27017/'}
2021-12-20 21:23:17,447 - INFO - luna.common.DataStore - Datstore file backend= ../PRO_12-123/tables/tiles
2021-12-20 21:23:17,448 - INFO - [datastore=01OV008-7579323e-2fae-43a9-b00f-a15c28] - Whole slide image path: ../PRO_12-123/tables/tiles/01OV008-7579323e-2fae-43a9-b00f-a15c28/ov_slides/WholeSlideImage/data
2021-12-20 21:23:17,457 - INFO - [datastore=01OV008-7579323e-2fae-43a9-b00f-a15c28] - Writing to output dir: ../PRO_12-123/tables/tiles/01OV008-7579323e-2fae-43a9-b00f-a15c28/ov_default_labels/TileImages/data
2021-12-20 21:23:17,458 - INFO - luna.pathology.common.preprocess - Processing slide ../PRO_12-123/tables/tiles/01OV008-7579323e-2fae-43a9-b00f-a15c28/ov_slides/WholeSlideImage/data
2021-12-20 21:23:17,461 - INFO - luna.pathology.common.preprocess - Params = {'input_wsi_tag': 'ov_slides', 'job_tag': 'ov_default_labels', 'tile_size': 128, 'scale_factor': 16, 'requested_magnification': 20, 'filter': {'otsu_score': 0.5}, 'root_path': '../PRO_12-123/tables/tiles'}
2021-12-20 21:23:17,707 - INFO - luna.pathology.common.preprocess - Slide size = [42240,59209]
2021-12-20 21:23:17,708 - INFO - luna.pathology.common.preprocess - Normalized magnification scale factor for 20x is 2, overall thumbnail scale factor is 32
2021-12-20 21:23:17,709 - INFO - luna.pathology.common.preprocess - Requested tile size=128, tile size at full magnficiation=256, tile size at thumbnail=8
2021-12-20 21:23:18,073 - INFO - luna.pathology.common.preprocess - tiles x 165, tiles y 232
2021-12-20 21:23:18,180 - INFO - luna.pathology.common.preprocess - Number of tiles in raster: 37490
/home/rosed2/.local/lib/python3.6/site-packages/pandas/core/indexing.py:1596: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
self.obj[key] = _infer_fill_value(value)
/home/rosed2/.local/lib/python3.6/site-packages/pandas/core/indexing.py:1763: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
isetter(loc, value)
2021-12-20 21:24:55,830 - INFO - luna.pathology.common.preprocess - Proccessing tiles [10000,11342]
2021-12-20 21:25:08,371 - INFO - luna.pathology.common.preprocess - Saved tile scores and images at ../PRO_12-123/tables/tiles/01OV008-7579323e-2fae-43a9-b00f-a15c28/ov_default_labels/TileImages/data
Once the step is done, you can find the tiles and score CSV for your slide, at your output location. For slide id 01OV008-7579323e-2fae-43a9-b00f-a15c28, we have the tile image and metadata stored at ~/vmount/PRO_12-123/tables/tiles/01OV008-7579323e-2fae-43a9-b00f-a15c28/ov_default_labels/TileImages/data.
[21]:
!ls -lhtr ~/vmount/PRO_12-123/tables/tiles/01OV008-7579323e-2fae-43a9-b00f-a15c28/ov_default_labels/TileImages/data
total 545M
-rw-r--r-- 1 rosed2 rosed2 532M Dec 20 21:25 tiles.slice.pil
-rw-r--r-- 1 rosed2 rosed2 763K Dec 20 21:25 address.slice.csv
-rw-r--r-- 1 rosed2 rosed2 572 Dec 20 21:25 metadata.json
Let’s look at the tile metadata in the output CSV.
The tile otsu_score, purple score and regional annotation labels are stored along tile metadata such as address, coordinates, size, and offset.
For the training slide, we see that only the tiles that meet the filter criteria has been kept. For the test slide, we keep all tissue regions, so we have far more tiles generated. Notice we don’t have the regional labels.
[22]:
import pandas as pd
# For a train slide, we have generated tiles for annotated regions, and populated regional_labels
df = pd.read_csv("../PRO_12-123/tables/tiles/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77/ov_default_labels/TileImages/data/address.slice.csv")
df
[22]:
| address | coordinates | otsu_score | purple_score | regional_label | tile_image_offset | tile_image_length | tile_image_size_xy | tile_image_mode | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | x54_y114_z20 | (54, 114) | 0.843750 | 1.000000 | stroma | 199409664.0 | 49152.0 | 128.0 | RGB |
| 1 | x55_y113_z20 | (55, 113) | 0.640625 | 0.968750 | stroma | 203882496.0 | 49152.0 | 128.0 | RGB |
| 2 | x55_y114_z20 | (55, 114) | 0.734375 | 0.937500 | stroma | 203931648.0 | 49152.0 | 128.0 | RGB |
| 3 | x56_y113_z20 | (56, 113) | 0.546875 | 1.000000 | stroma | 208502784.0 | 49152.0 | 128.0 | RGB |
| 4 | x56_y114_z20 | (56, 114) | 0.656250 | 1.000000 | stroma | 208551936.0 | 49152.0 | 128.0 | RGB |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 144 | x110_y170_z20 | (110, 170) | 1.000000 | 0.968750 | tumor | 477806592.0 | 49152.0 | 128.0 | RGB |
| 145 | x110_y171_z20 | (110, 171) | 0.984375 | 1.000000 | tumor | 477855744.0 | 49152.0 | 128.0 | RGB |
| 146 | x110_y172_z20 | (110, 172) | 0.984375 | 1.000000 | tumor | 477904896.0 | 49152.0 | 128.0 | RGB |
| 147 | x113_y94_z20 | (113, 94) | 0.546875 | 0.703125 | fat | 493928448.0 | 49152.0 | 128.0 | RGB |
| 148 | x115_y94_z20 | (115, 94) | 0.656250 | 0.859375 | fat | 505921536.0 | 49152.0 | 128.0 | RGB |
149 rows × 9 columns
[26]:
# For the test slide, we have generated tiles for all tissue regions
df = pd.read_csv("../PRO_12-123/tables/tiles/01OV008-7579323e-2fae-43a9-b00f-a15c28/ov_default_labels/TileImages/data/address.slice.csv")
df
[26]:
| address | coordinates | otsu_score | purple_score | tile_image_offset | tile_image_length | tile_image_size_xy | tile_image_mode | |
|---|---|---|---|---|---|---|---|---|
| 0 | x7_y146_z20 | (7, 146) | 0.546875 | 0.593750 | 0.0 | 49152.0 | 128.0 | RGB |
| 1 | x8_y133_z20 | (8, 133) | 0.687500 | 0.796875 | 49152.0 | 49152.0 | 128.0 | RGB |
| 2 | x8_y134_z20 | (8, 134) | 0.890625 | 0.968750 | 98304.0 | 49152.0 | 128.0 | RGB |
| 3 | x8_y135_z20 | (8, 135) | 0.859375 | 1.000000 | 147456.0 | 49152.0 | 128.0 | RGB |
| 4 | x8_y136_z20 | (8, 136) | 0.968750 | 0.968750 | 196608.0 | 49152.0 | 128.0 | RGB |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 11337 | x126_y136_z20 | (126, 136) | 0.843750 | 0.843750 | 557236224.0 | 49152.0 | 128.0 | RGB |
| 11338 | x126_y137_z20 | (126, 137) | 0.875000 | 1.000000 | 557285376.0 | 49152.0 | 128.0 | RGB |
| 11339 | x126_y138_z20 | (126, 138) | 0.796875 | 1.000000 | 557334528.0 | 49152.0 | 128.0 | RGB |
| 11340 | x126_y139_z20 | (126, 139) | 0.796875 | 0.968750 | 557383680.0 | 49152.0 | 128.0 | RGB |
| 11341 | x126_y140_z20 | (126, 140) | 0.562500 | 0.812500 | 557432832.0 | 49152.0 | 128.0 | RGB |
11342 rows × 8 columns
3. Collect tiles for model training#
Now that we have created tile labels, we can use collect_tiles CLI to collect the tile metadata as a set of parquet tables and save the outputs for multiple slide ids in the same dataset. This step is done to gather our dataset for model training.
[27]:
!collect_tiles --help
Usage: collect_tiles [OPTIONS]
Save tiles as a parquet file, indexed by slide id, address, and optionally
patient_id.
app_config - application configuration yaml file. See config.yaml.template
for details.
datastore_id - datastore name. usually a slide id.
method_param_path - json file with method parameters including input, output
details.
- input_label_tag: job tag used for generating tile labels
- input_wsi_tag: job tag used for loading the slide
- output_datastore: job tag for collecting tiles
- root_path: path to output data
Options:
-a, --app_config TEXT application configuration yaml file. See
config.yaml.template for details. [required]
-s, --datastore_id TEXT datastore name. usually a slide id.
[required]
-m, --method_param_path TEXT json file with method parameters including
input, output details. [required]
--help Show this message and exit.
At this point, it is critical to note that our model will train on the 4 slides reserved for trainig. We have reserved one slide out of the model training step in order to use it for the inference step.
We will call collect_tiles on the training slides to prepare a dataset for training.
[28]:
slide_ids_train = ['01OV002-bd8cdc70-3d46-40ae-99c4-90ef77', '01OV002-ed65cf94-8bc6-492b-9149-adc16f',
'01OV007-9b90eb78-2f50-4aeb-b010-d642f9', '01OV008-308ad404-7079-4ff8-8232-12ee2e']
# call collect_tiles as subprocess
def call_collect_tiles(slide):
subprocess.run(f"collect_tiles -a ~/luna/conf/app_config.yaml -s {slide} -m ~/luna/conf/collect_tiles.yaml", shell=True)
pool_process(call_collect_tiles, slide_ids_train)
Let’s check the output. The collected parquet files can be loaded as a pyarrow ParquetDataset, and be converted to Pandas Dataframe.
You’ll notice the table is indexed by patient_id, slide id and address. The data_path points to the tile image file. The rest of the metadata stored in this table are similar to the output of generate_tiles CLI.
[29]:
from pyarrow.parquet import ParquetDataset
ds = ParquetDataset('../PRO_12-123/tables/tiles/ov_tileset').read().to_pandas()
ds
[29]:
| coordinates | otsu_score | purple_score | regional_label | tile_image_offset | tile_image_length | tile_image_size_xy | tile_image_mode | data_path | |||
|---|---|---|---|---|---|---|---|---|---|---|---|
| patient_id | id_slide_container | address | |||||||||
| 4 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | x54_y114_z20 | (54, 114) | 0.843750 | 1.000000 | stroma | 199409664.0 | 49152.0 | 128.0 | RGB | ../PRO_12-123/tables/tiles/01OV002-bd8cdc70-3d... |
| x55_y113_z20 | (55, 113) | 0.640625 | 0.968750 | stroma | 203882496.0 | 49152.0 | 128.0 | RGB | ../PRO_12-123/tables/tiles/01OV002-bd8cdc70-3d... | ||
| x55_y114_z20 | (55, 114) | 0.734375 | 0.937500 | stroma | 203931648.0 | 49152.0 | 128.0 | RGB | ../PRO_12-123/tables/tiles/01OV002-bd8cdc70-3d... | ||
| x56_y113_z20 | (56, 113) | 0.546875 | 1.000000 | stroma | 208502784.0 | 49152.0 | 128.0 | RGB | ../PRO_12-123/tables/tiles/01OV002-bd8cdc70-3d... | ||
| x56_y114_z20 | (56, 114) | 0.656250 | 1.000000 | stroma | 208551936.0 | 49152.0 | 128.0 | RGB | ../PRO_12-123/tables/tiles/01OV002-bd8cdc70-3d... | ||
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 5 | 01OV008-308ad404-7079-4ff8-8232-12ee2e | x146_y89_z20 | (146, 89) | 0.875000 | 0.953125 | stroma | 484786176.0 | 49152.0 | 128.0 | RGB | ../PRO_12-123/tables/tiles/01OV008-308ad404-70... |
| x146_y90_z20 | (146, 90) | 0.953125 | 0.984375 | stroma | 484835328.0 | 49152.0 | 128.0 | RGB | ../PRO_12-123/tables/tiles/01OV008-308ad404-70... | ||
| x147_y90_z20 | (147, 90) | 0.734375 | 0.937500 | stroma | 488816640.0 | 49152.0 | 128.0 | RGB | ../PRO_12-123/tables/tiles/01OV008-308ad404-70... | ||
| x147_y91_z20 | (147, 91) | 0.968750 | 1.000000 | stroma | 488865792.0 | 49152.0 | 128.0 | RGB | ../PRO_12-123/tables/tiles/01OV008-308ad404-70... | ||
| x148_y91_z20 | (148, 91) | 1.000000 | 1.000000 | stroma | 493191168.0 | 49152.0 | 128.0 | RGB | ../PRO_12-123/tables/tiles/01OV008-308ad404-70... |
1506 rows × 9 columns
Congratulations! Now you have the tiles images and labels ready to train your model.