Dataset Preparation Tutorial¶
Welcome to the dataset preparation tutorial! In this notebook, we will download the toy data set for the tutorial and prepare the necessary tables used for later analysis. Here are the steps we will review:
Check server connection
Create a new directory for your project
Download data
Set up project configuration files
Build the proxy table
Run regional annotation ETL
Please note that, for the remainder of the tutorial, we assume that you are on the LUNA servers. The following steps will not execute properly if this is not the case.
Check server connection¶
To check that that you are connected to the LUNA servers, make sure you can run the following without errors:
[1]:
import data_processing
data_processing.__path__
[1]:
['/gpfs/mskmindhdp_emc/user/shared_data_folder/pathology-tutorial/pathology-tutorial-sandbox/data-processing/data_processing']
If so, congratulations! It is as simple as that. You are ready to start making the project workspace and preparing the data!
Create a new directory for your project¶
Next, we will create a project space for your configurations, data, models, and outputs to go for this tutorial.
To do so, we first need to create a file called manifest.yaml and populate its contents with those of the template configuration file.
[6]:
%%bash
touch manifest.yaml
cp -v configs/manifest.yaml manifest.yaml
‘configs/manifest.yaml’ -> ‘manifest.yaml’
Next, we will use this file to create a new project space for this tutorial using a CLI from the repository.
[3]:
%%bash
python3 -m data_processing.project.generate --manifest_file /gpfs/mskmindhdp_emc/user/shared_data_folder/pathology-tutorial/manifest.yaml
You should now see a new folder called PRO_12-123 in this directory. This will be your project name!
Download data¶
The data that you will be using for this tutorial is a set of 5 whole slide images of ovarian cancer H&E slides, available in the svs file format. Whole slide imaging refers to the scanning of conventional glass slides for research purposes; in this case, these are slides that oncologists have used to inspecting cancer samples! We will download these images from Synapse, a data warehouse used for digital research.
We will now make a folder for your data and the toy data set in this new project workspace.
[10]:
%%bash
cd PRO_12-123
mkdir data && cd data
mkdir toy_data_set
You can find the pathology slides for your toy data set on Synapse. First, you must navigate to the Synapse website (https://www.synapse.org/) and create an account if you do not already have one. Once your account is created, open the site, search for the project ID (syn25946167) in the righthand corner, click the “Files” tab, and download the tar.gz file as a file (not as a package). This process may take a while, as you will be downloading a little under 5 GB of data onto your machine. Once downloaded, expand the tar file, and then relocate the five svs files into the toy_data_set folder.
Set up project configuration files¶
Next, you must set up your configuration files.
In your project workspace, make a new directory called my_conf and copy the contents of the configs/ file into it.
[8]:
%%bash
cp -R configs/ PRO_12-123/my_conf
Note: while you do not have to change the contents of my_conf/app_confi.yaml, you must fill out a few personal fields in my_conf/data_config.yaml and regional_annotation_config.yaml, namely: REQUESTOR, REQUESTOR_EMAIL, and DATE. Please take a moment to do so now, manually.
Build the proxy table¶
Now, we will run the Whole Slide Image (WSI) ETL to database the slides and build a proxy table. For reference, ETL stands for extract-transform-load; it is the method that often involves clearning data, transforming data types, and loading data into different systems. We will use to translate and obtain data hosted on LUNA servers into our project environment. Additionally, a proxy table is a local table that points to a remote object. The table that we will create will point to data from the LUNA servers that is relevant for our whole slide image slides that we just downloaded for the toy data set.
First, make sure that your environment variables are set to the right destinations:
[6]:
%%bash
export PYSPARK_PYTHON=/gpfs/mskmindhdp_emc/sw/env/bin/python3
export PYSPARK_DRIVER_PYTHON=/gpfs/mskmindhdp_emc/sw/env/bin/python3
echo $PYSPARK_PYTHON
echo $PYSPARK_DRIVER_PYTHON
echo $SPARK_HOME
/gpfs/mskmindhdp_emc/sw/env/bin/python3
/gpfs/mskmindhdp_emc/sw/env/bin/python3
/opt/spark-3.0.0-bin-hadoop3.2
Then, to run the ETL, run the following command:
[ ]:
%%bash
python3 -m data_processing.pathology.proxy_table.generate \
-d configs/wsi_config.yaml \
-a configs/app_config.yaml \
-p delta
This step may take a while. At the end, your proxy table should be generated!
Before we view the table, we must first update it to associate patient ID’s with the slides. This is necessary for correctly training and validating the machine learning model in the coming notebooks. Once the slides are divided into “tiles” in the next notebook, the tiles are split between the training and validation sets for the ML model. If the tiles do not have patient ID’s associated with them, then it is possible for tiles from one individual to appear in both the training and validation of the model; this would cause researchers to have an exaggerated interpretation of the model’s accuracy, since we would essentially be validating the model on information that is too near to what it has already seen.
Note that we will not be using patient IDs associated with MSK. Instead, we will be using spoof IDs that will suffice for this tutorial. When running this workflow with real data, make sure to include the IDs safely and securely. Run the following block of code to add a ‘patient_id’ column to the table and store it using Spark.
[33]:
from pyspark.sql import SparkSession
# setup spark session
spark = SparkSession.builder \
.appName("test") \
.master('local[*]') \
.config("spark.driver.host", "127.0.0.1") \
.config("spark.jars.packages", "io.delta:delta-core_2.12:0.7.0") \
.config("spark.delta.logStore.class", "org.apache.spark.sql.delta.storage.HDFSLogStore") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
.config("spark.hadoop.dfs.client.use.datanode.hostname", "true") \
.config("spark.driver.memory", "6g") \
.config("spark.executor.memory", "6g") \
.getOrCreate()
# read WSI delta table
wsi_table = spark.read.format("delta")
.load("file:////gpfs/mskmindhdp_emc/user/shared_data_folder/pathology-tutorial/PRO_12-123/tables/WSI_toy_data_set").toPandas()
# insert spoof patient ids
patient_id=[1,2,3,4,5]
wsi_table['patient_id']=patient_id
wsi_table
# convert back to a spark table (update table)
x = spark.createDataFrame(wsi_table)
x.write.format("delta").mode("overwrite").option("mergeSchema", "true").save("file:////gpfs/mskmindhdp_emc/user/shared_data_folder/pathology-tutorial/PRO_12-123/tables/WSI_toy_data_set")
Next, we may view the WSI table! This table should have the metadata associated with the WSI slides that you just collected, including the patient IDs.
[30]:
import data_processing
data_processing.__path__
[30]:
['/gpfs/mskmindhdp_emc/etl-runner/data-processing/data_processing']
[34]:
# read WSI delta table
wsi_table = spark.read.format("delta") \
.load("file:////gpfs/mskmindhdp_emc/user/shared_data_folder/pathology-tutorial/PRO_12-123/tables/WSI_toy_data_set").toPandas()
# view table
wsi_table
[34]:
| path | modificationTime | length | wsi_record_uuid | slide_id | metadata | patient_id | |
|---|---|---|---|---|---|---|---|
| 0 | file:/gpfs/mskmindhdp_emc/user/shared_data_fol... | 2021-07-06 11:42:52 | 584611357 | WSI-93ccfd50a210d0b8c7589352be9036ef5abf6b4f81... | 2551129 | {'aperio_User': 'd9286672-cd53-4139-87ba-d68c4... | 2 |
| 1 | file:/gpfs/mskmindhdp_emc/user/shared_data_fol... | 2021-07-06 11:53:51 | 1413574341 | WSI-03662b6be585f8bdb1a16a175a7cfda07c4057afe5... | 2551571 | {'aperio_User': 'd9286672-cd53-4139-87ba-d68c4... | 1 |
| 2 | file:/gpfs/mskmindhdp_emc/user/shared_data_fol... | 2021-07-06 11:46:00 | 520642043 | WSI-12677b7d98691d1eef8043727f27878eb9fda14b65... | 2551531 | {'aperio_User': 'd9286672-cd53-4139-87ba-d68c4... | 3 |
| 3 | file:/gpfs/mskmindhdp_emc/user/shared_data_fol... | 2021-07-06 11:43:07 | 1322921471 | WSI-1ba07f58166fc2073c854dd9b00a11eaca2203ff20... | 2551028 | {'aperio_User': 'd9286672-cd53-4139-87ba-d68c4... | 4 |
| 4 | file:/gpfs/mskmindhdp_emc/user/shared_data_fol... | 2021-07-06 14:18:26 | 966069709 | WSI-f3890775a7f36c982aae28ac58de43b1852652fc20... | 2551389 | {'aperio_User': 'd9286672-cd53-4139-87ba-d68c4... | 5 |
If the table is depicted above, congratulations, you have successfully run the Whole Slide Image (WSI) ETL to database the slides!
Run the regional annotation ETL¶
The whole slide images that you downloaded are images of ovarian cancer, but not every pixel on each slide is a tumor. In fact, the images show tumor cells, normal ovarian cells, necrosis (dead cells), fibrosis (scarred cells), and more. Pathologists at Memorial Sloan Kettering examined each slide and denoted these different features by hand, providing us with regional annotations. You may think of regional annotations as scientific highlighter marks over the different regions of the image.
What actually happens when the regional annotation ETL is run? First, annotation bitmaps are downloaded from SlideViewer, a repository which stores WSI images and their annotation data. These bitmaps are converted into numpy arrays, which are then converted into GeoJSON files and organized in the proxy table. The GeoJSON files store the annotation regions marked by pathologists as polygons, which makes the data simpler to store and analyze. Once the annotation files are loaded into QuPath- a software used for digital pathology- later in the pipeline, this data format becomes incredibly useful and easy to work with.
To run the regional annotation ETL, try:
[13]:
%%bash
python3 -m data_processing.pathology.refined_table.regional_annotation.dask_generate \
-d configs/regional_annotation_config.yaml \
-a configs/app_config.yaml
>>>>>>> Processing [2551389] <<<<<<<<
No label 1 found
Building contours for label 2
num_pixels with label 344474790
num_contours 2
[-1, 0]
No label 3 found
Building contours for label 4
num_pixels with label 62336170
num_contours 3
[-1, 0, 0]
No label 5 found
No label 6 found
Building contours for label 7
num_pixels with label 2720232
num_contours 2
[-1, -1]
No label 8 found
No label 9 found
No label 10 found
No label 11 found
No label 12 found
No label 13 found
No label 14 found
No label 15 found
>>>>>>> Processing [2551571] <<<<<<<<
Building contours for label 1
num_pixels with label 3612930
num_contours 3
[-1, -1, -1]
No label 2 found
Building contours for label 3
num_pixels with label 20257170
num_contours 3
[-1, -1, -1]
No label 4 found
Building contours for label 5
num_pixels with label 38403188
num_contours 2
[-1, -1]
Building contours for label 6
num_pixels with label 28809658
num_contours 29
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]
Building contours for label 7
num_pixels with label 4451030
num_contours 11
[-1, -1, -1, -1, -1, 2, 2, -1, -1, -1, -1]
Building contours for label 8
num_pixels with label 1248
num_contours 1
[-1]
No label 9 found
No label 10 found
No label 11 found
No label 12 found
No label 13 found
No label 14 found
No label 15 found
>>>>>>> Processing [2551028] <<<<<<<<
No label 1 found
Building contours for label 2
num_pixels with label 385750
num_contours 1
[-1]
Building contours for label 3
num_pixels with label 1205304
num_contours 5
[-1, -1, -1, -1, -1]
Building contours for label 4
num_pixels with label 6615826
num_contours 11
[-1, -1, 0, -1, -1, -1, -1, -1, -1, -1, -1]
No label 5 found
Building contours for label 6
num_pixels with label 51908714
num_contours 8
[-1, -1, -1, -1, -1, -1, 4, -1]
Building contours for label 7
num_pixels with label 3202104
num_contours 2
[-1, -1]
No label 8 found
No label 9 found
No label 10 found
No label 11 found
No label 12 found
No label 13 found
No label 14 found
No label 15 found
>>>>>>> Processing [2551129] <<<<<<<<
Building contours for label 1
num_pixels with label 394812
num_contours 3
[-1, -1, -1]
No label 2 found
Building contours for label 3
num_pixels with label 880568
num_contours 1
[-1]
Building contours for label 4
num_pixels with label 255604
num_contours 2
[-1, -1]
Building contours for label 5
num_pixels with label 3428922
num_contours 3
[-1, -1, -1]
No label 6 found
No label 7 found
No label 8 found
No label 9 found
Building contours for label 10
num_pixels with label 2587508
num_contours 8
[-1, -1, -1, -1, -1, -1, 5, 5]
No label 11 found
Building contours for label 12
num_pixels with label 11236
num_contours 1
[-1]
No label 13 found
No label 14 found
No label 15 found
>>>>>>> Processing [2551531] <<<<<<<<
Building contours for label 1
num_pixels with label 16475072
num_contours 5
[-1, -1, -1, 2, 2]
Building contours for label 2
num_pixels with label 3750146
num_contours 3
[-1, 0, 0]
Building contours for label 3
num_pixels with label 115784442
num_contours 11
[-1, -1, 0, 1, 1, 1, -1, 1, -1, -1, 9]
No label 4 found
Building contours for label 5
num_pixels with label 62971884
num_contours 27
[-1, 0, -1, -1, 0, -1, -1, 6, 0, -1, -1, 3, 3, 0, 0, 0, -1, 0, 0, 16, 16, 16, 16, 16, 16, 16, -1]
No label 6 found
No label 7 found
No label 8 found
No label 9 found
No label 10 found
No label 11 found
No label 12 found
No label 13 found
No label 14 found
No label 15 found
<Client: 'tcp://127.0.0.1:41273' processes=25 threads=25, memory=2.03 TB>
Table output directory = /gpfs/mskmindhdp_emc/user/shared_data_folder/pathology-tutorial/PRO_12-123/tables/REGIONAL_METADATA_RESULTS
sv_project_id ... labelset
0 134 ... DEFAULT_LABELS
1 134 ... PIXEL_CLASSIFIER_LABELS
2 134 ... OBJECT_CLASSIFIER_LABELS
3 134 ... SIMPLIFIED_PIXEL_CLASSIFIER_LABELS
4 134 ... DEFAULT_LABELS
5 134 ... PIXEL_CLASSIFIER_LABELS
6 134 ... OBJECT_CLASSIFIER_LABELS
7 134 ... SIMPLIFIED_PIXEL_CLASSIFIER_LABELS
[8 rows x 10 columns]
sv_project_id ... labelset
0 134 ... DEFAULT_LABELS
1 134 ... PIXEL_CLASSIFIER_LABELS
2 134 ... OBJECT_CLASSIFIER_LABELS
3 134 ... SIMPLIFIED_PIXEL_CLASSIFIER_LABELS
4 134 ... DEFAULT_LABELS
5 134 ... PIXEL_CLASSIFIER_LABELS
6 134 ... OBJECT_CLASSIFIER_LABELS
7 134 ... SIMPLIFIED_PIXEL_CLASSIFIER_LABELS
[8 rows x 10 columns]
sv_project_id ... labelset
0 134 ... DEFAULT_LABELS
1 134 ... PIXEL_CLASSIFIER_LABELS
2 134 ... OBJECT_CLASSIFIER_LABELS
3 134 ... SIMPLIFIED_PIXEL_CLASSIFIER_LABELS
4 134 ... DEFAULT_LABELS
5 134 ... PIXEL_CLASSIFIER_LABELS
6 134 ... OBJECT_CLASSIFIER_LABELS
7 134 ... SIMPLIFIED_PIXEL_CLASSIFIER_LABELS
[8 rows x 10 columns]
sv_project_id ... labelset
0 134 ... DEFAULT_LABELS
1 134 ... PIXEL_CLASSIFIER_LABELS
2 134 ... OBJECT_CLASSIFIER_LABELS
3 134 ... SIMPLIFIED_PIXEL_CLASSIFIER_LABELS
4 134 ... DEFAULT_LABELS
5 134 ... PIXEL_CLASSIFIER_LABELS
6 134 ... OBJECT_CLASSIFIER_LABELS
7 134 ... SIMPLIFIED_PIXEL_CLASSIFIER_LABELS
[8 rows x 10 columns]
sv_project_id ... labelset
0 134 ... DEFAULT_LABELS
1 134 ... PIXEL_CLASSIFIER_LABELS
2 134 ... OBJECT_CLASSIFIER_LABELS
3 134 ... SIMPLIFIED_PIXEL_CLASSIFIER_LABELS
4 134 ... DEFAULT_LABELS
5 134 ... PIXEL_CLASSIFIER_LABELS
6 134 ... OBJECT_CLASSIFIER_LABELS
7 134 ... SIMPLIFIED_PIXEL_CLASSIFIER_LABELS
[8 rows x 10 columns]
2021-07-21 15:16:11,351 - ERROR - asyncio - Task was destroyed but it is pending!
2021-07-21 15:16:11,351 - ERROR - asyncio - task: <Task pending coro=<RequestHandler._execute() running at /gpfs/mskmindhdp_emc/sw/env/lib64/python3.6/site-packages/tornado/web.py:1703> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f5a4b4b9828>()]> cb=[_HandlerDelegate.execute.<locals>.<lambda>() at /gpfs/mskmindhdp_emc/sw/env/lib64/python3.6/site-packages/tornado/web.py:2333]>
To check that the regional annotation ETL was correctly run, after the Jupyter cell finishes, you may load the regional annotations table! This table contains the metadata saved from running the ETL. It includes paths to the bitmap files, numpy files, and geoJSON files that were mentioned before. To load the table, run the following cell:
[17]:
from pyarrow.parquet import read_table
regional_annotation_table = read_table("PRO_12-123/tables/REGIONAL_METADATA_RESULTS",
filters = [('user', '!=', f'CONCAT')]).to_pandas()
regional_annotation_table
[17]:
| sv_project_id | slideviewer_path | slide_id | user | bmp_filepath | npy_filepath | geojson_path | date | labelset | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 134 | 2019;HobS19-409411851898;2551028.svs | 2551028 | ellensol | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810254 | DEFAULT_LABELS |
| 1 | 134 | 2019;HobS19-409411851898;2551028.svs | 2551028 | ellensol | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810254 | PIXEL_CLASSIFIER_LABELS |
| 2 | 134 | 2019;HobS19-409411851898;2551028.svs | 2551028 | ellensol | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810254 | OBJECT_CLASSIFIER_LABELS |
| 3 | 134 | 2019;HobS19-409411851898;2551028.svs | 2551028 | ellensol | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810254 | SIMPLIFIED_PIXEL_CLASSIFIER_LABELS |
| 4 | 134 | 2019;HobS19-159147602774;2551129.svs | 2551129 | ellensol | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810066 | DEFAULT_LABELS |
| 5 | 134 | 2019;HobS19-159147602774;2551129.svs | 2551129 | ellensol | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810066 | PIXEL_CLASSIFIER_LABELS |
| 6 | 134 | 2019;HobS19-159147602774;2551129.svs | 2551129 | ellensol | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810066 | OBJECT_CLASSIFIER_LABELS |
| 7 | 134 | 2019;HobS19-159147602774;2551129.svs | 2551129 | ellensol | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810066 | SIMPLIFIED_PIXEL_CLASSIFIER_LABELS |
| 8 | 134 | 2019;HobS19-475053909405;2551389.svs | 2551389 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810126 | DEFAULT_LABELS |
| 9 | 134 | 2019;HobS19-475053909405;2551389.svs | 2551389 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810126 | PIXEL_CLASSIFIER_LABELS |
| 10 | 134 | 2019;HobS19-475053909405;2551389.svs | 2551389 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810126 | OBJECT_CLASSIFIER_LABELS |
| 11 | 134 | 2019;HobS19-475053909405;2551389.svs | 2551389 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810126 | SIMPLIFIED_PIXEL_CLASSIFIER_LABELS |
| 12 | 134 | 2019;HobS19-176164505079;2551531.svs | 2551531 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810039 | DEFAULT_LABELS |
| 13 | 134 | 2019;HobS19-176164505079;2551531.svs | 2551531 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810039 | PIXEL_CLASSIFIER_LABELS |
| 14 | 134 | 2019;HobS19-176164505079;2551531.svs | 2551531 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810039 | OBJECT_CLASSIFIER_LABELS |
| 15 | 134 | 2019;HobS19-176164505079;2551531.svs | 2551531 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.810039 | SIMPLIFIED_PIXEL_CLASSIFIER_LABELS |
| 16 | 134 | 2019;HobS19-030513574376;2551571.svs | 2551571 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.809901 | DEFAULT_LABELS |
| 17 | 134 | 2019;HobS19-030513574376;2551571.svs | 2551571 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.809901 | PIXEL_CLASSIFIER_LABELS |
| 18 | 134 | 2019;HobS19-030513574376;2551571.svs | 2551571 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.809901 | OBJECT_CLASSIFIER_LABELS |
| 19 | 134 | 2019;HobS19-030513574376;2551571.svs | 2551571 | soslowr | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | /gpfs/mskmindhdp_emc/user/shared_data_folder/p... | 2021-07-06 14:02:04.809901 | SIMPLIFIED_PIXEL_CLASSIFIER_LABELS |
At this point, you have successfully set up your workspace, dowloaded the data, and run both the pathology and regional annotation ETLs to prepare your data. You are ready to move on to the tiling notebook!