diff --git a/README.md b/README.md index 4478f7d..0790818 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,9 @@ Notebooks in this folder focus on topics that will require understanding of the * [Direct access to tags/frames from GCS/AWS buckets](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/gcs_aws_direct_access.ipynb): learn how to access individual frames or tags of large DICOM files from the bucket without having to download the entire file (this notebook accompanies documentation article here: https://learn.canceridc.dev/data/downloading-data/direct-loading) * [Using DICOMweb to access IDC data](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/idc_dicomweb_access.ipynb): both metadata and pixel data can be access using DICOMweb, which is particularly important while working with digital pathology, as it enables granular access to the individual pyramid tiles (frames) * [IDC on AWS](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/idc_on_aws/idc-on-aws-tutorial.ipynb): learn how to work with IDC data using AWS services, including AWS HealthImaging. +* [Visualizing IDC data in 3D with trame-slicer](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/trame_slicer_visualization.ipynb): view IDC images and segmentations interactively in 3D inside a Colab cell using [trame-slicer](https://github.com/KitwareMedical/trame-slicer), which brings the 3D Slicer rendering engine to the browser (multi-planar reformat, volume rendering, segmentation overlay). +* [Interactive visualization of IDC images and segmentations with ipyniivue](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/image_visualization_with_ipyniivue.ipynb): render IDC images and DICOM Segmentation overlays interactively inside a Colab cell using [ipyniivue](https://github.com/niivue/ipyniivue), the Jupyter widget for the WebGL-based [NiiVue](https://github.com/niivue/niivue) viewer (multiplanar + 3D views, multi-organ overlays colored using the colors stored in the DICOM SEG, no desktop software needed). +* [Running the full 3D Slicer desktop in Colab with IDC data](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/image_visualization_with_slicer_desktop.ipynb): launch the complete [3D Slicer](https://www.slicer.org) desktop application headlessly and stream its live GUI into a Colab cell using the [desktopia](https://github.com/pieper/desktopia) project, loading an IDC DICOM image and DICOM SEG natively (every Slicer module available, no desktop install). ## [`viewers_deployment`](https://github.com/ImagingDataCommons/IDC-Tutorials/tree/master/notebooks/viewers_deployment) diff --git a/notebooks/advanced_topics/image_visualization_with_ipyniivue.ipynb b/notebooks/advanced_topics/image_visualization_with_ipyniivue.ipynb new file mode 100644 index 0000000..1bed1bf --- /dev/null +++ b/notebooks/advanced_topics/image_visualization_with_ipyniivue.ipynb @@ -0,0 +1,576 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Interactive 3D visualization of IDC images and segmentations with ipyniivue\n", + "\n", + "---\n", + "\n", + "## Summary\n", + "\n", + "This notebook shows how to use [**ipyniivue**](https://github.com/niivue/ipyniivue) \u2014 the Jupyter\n", + "widget wrapper around the WebGL-based [NiiVue](https://github.com/niivue/niivue) medical image\n", + "viewer \u2014 to interactively visualize radiology images and segmentation overlays from\n", + "[NCI Imaging Data Commons (IDC)](https://imaging.datacommons.cancer.gov), directly inside a notebook.\n", + "\n", + "Unlike a static screenshot, the embedded viewer lets you scroll through slices, pan/zoom, switch\n", + "between axial/coronal/sagittal/3D-render layouts, and toggle the segmentation overlay \u2014 all without\n", + "leaving the notebook and **without installing any desktop DICOM software**. Everything in this\n", + "notebook runs in [Google Colab](https://colab.research.google.com).\n", + "\n", + "As an example we visualize a chest CT from the [National Lung Screening Trial (NLST)](https://imaging.datacommons.cancer.gov/explore/?filters_for_load=collection_id_nlst)\n", + "collection together with a whole-body multi-organ segmentation produced by\n", + "[TotalSegmentator](https://github.com/wasserth/TotalSegmentator) (one of the AI analysis results\n", + "hosted by IDC).\n", + "\n", + "Upon completion of this tutorial, you will learn how to:\n", + "* find an image series in IDC that has an accompanying DICOM Segmentation (SEG) object\n", + "* download the image and segmentation with [`idc-index`](https://github.com/ImagingDataCommons/idc-index)\n", + "* convert a DICOM image series to NIfTI with [SimpleITK](https://simpleitk.org/), and a multi-segment\n", + " DICOM SEG to a spatially-aligned label map with [highdicom](https://github.com/ImagingDataCommons/highdicom)\n", + "* build a color lookup table from the **recommended display colors stored inside the DICOM SEG itself**\n", + "* render the image volume in a multiplanar + 3D view with ipyniivue, overlay the labeled segmentation,\n", + " and switch viewer layouts and overlay opacity\n", + "\n", + "> **Why convert to NIfTI?** NiiVue renders volumetric data and aligns overlays using the spatial\n", + "> metadata (affine matrix) stored in each volume. Converting the DICOM image and the DICOM SEG to\n", + "> NIfTI while preserving their patient-coordinate geometry guarantees that the segmentation lands\n", + "> exactly on top of the anatomy it describes.\n", + "\n", + "---\n", + "Initial version: Jun 2026\n", + "\n", + "Updated: Jun 2026" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Setup\n", + "\n", + "Install the packages used in this notebook. In Google Colab `idc-index`, `SimpleITK`, `highdicom`,\n", + "and `ipyniivue` are not pre-installed, so we add them here. This takes a minute or so." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "%%capture\n", + "%pip install --upgrade idc-index ipyniivue highdicom SimpleITK" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Enable interactive widgets in Colab\n", + "\n", + "`ipyniivue` is a custom Jupyter widget. Google Colab does not render third-party widgets unless the\n", + "*custom widget manager* is explicitly enabled, so we do that here. The same code is a harmless no-op\n", + "when the notebook runs in a local Jupyter/JupyterLab environment." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "try:\n", + " # Running in Google Colab: enable rendering of third-party (anywidget-based) widgets\n", + " from google.colab import output\n", + " output.enable_custom_widget_manager()\n", + " IN_COLAB = True\n", + " print(\"Running in Google Colab - custom widget manager enabled.\")\n", + "except ImportError:\n", + " IN_COLAB = False\n", + " print(\"Not running in Colab (e.g. local Jupyter) - no extra widget setup needed.\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "from pathlib import Path\n", + "\n", + "import numpy as np\n", + "import pandas as pd\n", + "import nibabel as nib\n", + "import SimpleITK as sitk\n", + "import highdicom as hd\n", + "\n", + "from idc_index import IDCClient\n", + "from ipyniivue import NiiVue, SliceType\n", + "\n", + "client = IDCClient()\n", + "print(\"idc-index is using IDC data version:\", client.get_idc_version())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Find an image series that has a segmentation\n", + "\n", + "IDC stores segmentations as standard **DICOM Segmentation (SEG)** objects. The `seg_index` table\n", + "records, for every SEG series, the `SeriesInstanceUID` of the original image series it was derived\n", + "from (`segmented_SeriesInstanceUID`). We join it back to the main `index` to find image+SEG pairs.\n", + "\n", + "For this tutorial we use a chest CT from the **NLST** collection (license: CC BY, commercial use\n", + "allowed) together with its **TotalSegmentator** whole-body organ segmentation, which is published in\n", + "IDC as the `TotalSegmentator-CT-Segmentations` analysis result. The query below lists candidate\n", + "pairs; we then pin one specific example by its UIDs so the notebook is fully reproducible." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "client.fetch_index(\"seg_index\")\n", + "\n", + "candidates = client.sql_query(\"\"\"\n", + " SELECT\n", + " seg.SeriesInstanceUID AS seg_series_uid,\n", + " seg.segmented_SeriesInstanceUID AS image_series_uid,\n", + " img.collection_id,\n", + " seg_meta.SeriesDescription AS seg_description,\n", + " ROUND(img.series_size_MB, 1) AS image_MB,\n", + " ROUND(seg_meta.series_size_MB, 1) AS seg_MB\n", + " FROM seg_index seg\n", + " JOIN index img ON seg.segmented_SeriesInstanceUID = img.SeriesInstanceUID\n", + " JOIN index seg_meta ON seg.SeriesInstanceUID = seg_meta.SeriesInstanceUID\n", + " WHERE seg_meta.analysis_result_id = 'TotalSegmentator-CT-Segmentations'\n", + " AND img.collection_id = 'nlst'\n", + " ORDER BY image_MB\n", + " LIMIT 5\n", + "\"\"\")\n", + "candidates" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "# Pin one specific example for reproducibility (a chest CT + its TotalSegmentator multi-organ SEG)\n", + "image_series_uid = \"1.2.840.113654.2.55.71041873368734986406977154644223539362\"\n", + "seg_series_uid = \"1.2.276.0.7230010.3.1.3.313263360.84.1706324554.366745\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Download the image and segmentation\n", + "\n", + "We download each series into its own folder. `dirTemplate=\"\"` puts the DICOM files directly in the\n", + "target directory (no nested `collection/patient/study/...` hierarchy), which keeps the paths simple\n", + "for the conversion step below." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "work_dir = Path(\"idc_niivue_demo\")\n", + "image_dir = work_dir / \"image_dicom\"\n", + "seg_dir = work_dir / \"seg_dicom\"\n", + "\n", + "client.download_dicom_series(seriesInstanceUID=image_series_uid,\n", + " downloadDir=str(image_dir), dirTemplate=\"\")\n", + "client.download_dicom_series(seriesInstanceUID=seg_series_uid,\n", + " downloadDir=str(seg_dir), dirTemplate=\"\")\n", + "\n", + "print(\"Image DICOM files:\", len(list(image_dir.glob(\"*.dcm\"))))\n", + "print(\"SEG DICOM files: \", len(list(seg_dir.glob(\"*.dcm\"))))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. Convert the DICOM image series to NIfTI\n", + "\n", + "A CT/MR series is a stack of single-slice DICOM files. `SimpleITK` reads the whole stack, sorts the\n", + "slices into a 3D volume, and writes a single NIfTI file. SimpleITK preserves the patient-coordinate\n", + "geometry (origin, voxel spacing, orientation) and handles the DICOM (LPS) to NIfTI (RAS) coordinate\n", + "convention for us." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "reader = sitk.ImageSeriesReader()\n", + "reader.SetFileNames(reader.GetGDCMSeriesFileNames(str(image_dir)))\n", + "image = reader.Execute()\n", + "\n", + "image_nii = work_dir / \"image.nii.gz\"\n", + "sitk.WriteImage(image, str(image_nii))\n", + "\n", + "print(\"Volume size (cols, rows, slices):\", image.GetSize())\n", + "print(\"Voxel spacing (mm):\", tuple(round(s, 3) for s in image.GetSpacing()))\n", + "print(\"Saved:\", image_nii)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Convert the DICOM SEG to an aligned NIfTI label map\n", + "\n", + "`highdicom` reads the DICOM SEG and reconstructs the segmentation as a 3D volume that carries the\n", + "same frame-of-reference geometry as the source image. `get_volume(combine_segments=True)` produces a\n", + "single **label map** where background = 0 and each organ gets its own integer label.\n", + "\n", + "We save it as NIfTI, converting highdicom's index→LPS affine to the NIfTI index→RAS\n", + "convention by flipping the first two axes (`diag(-1, -1, 1, 1)`). Because both the image and the\n", + "label map are written in the same RAS world coordinates, NiiVue overlays them in perfect alignment." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "seg = hd.seg.segread(next(seg_dir.glob(\"*.dcm\")))\n", + "\n", + "# Map each segment number (the integer voxel value in the label map) to its label\n", + "segment_labels = {n: seg.get_segment_description(n).segment_label for n in seg.segment_numbers}\n", + "print(f\"This SEG object contains {len(segment_labels)} segments, e.g.:\")\n", + "for n in list(segment_labels)[:8]:\n", + " print(f\" {n:>3}: {segment_labels[n]}\")\n", + "print(\" ...\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "# Combined multi-label map: 0 = background, 1..N = organs. Some TotalSegmentator structures can\n", + "# touch/overlap, so we skip the overlap check (higher-numbered label wins on any overlap).\n", + "vol = seg.get_volume(combine_segments=True, skip_overlap_checks=True)\n", + "\n", + "# LPS (DICOM) -> RAS (NIfTI): flip the x and y axes\n", + "LPS_TO_RAS = np.diag([-1.0, -1.0, 1.0, 1.0])\n", + "\n", + "labelmap_nii = work_dir / \"segmentation_labelmap.nii.gz\"\n", + "nib.save(nib.Nifti1Image(vol.array.astype(np.uint8), LPS_TO_RAS @ vol.affine), str(labelmap_nii))\n", + "print(\"Label map shape:\", vol.array.shape, \"| labels present:\", int(vol.array.max()))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Use the colors stored inside the DICOM SEG\n", + "\n", + "A DICOM SEG can carry a **Recommended Display CIELab Value** for each segment \u2014 the color the data\n", + "producer intended for that structure. Rather than inventing our own palette, we read these colors\n", + "straight from the SEG and build a NiiVue *label colormap* (a lookup table mapping each integer label\n", + "to an RGBA color and a name).\n", + "\n", + "The recommended color is stored in the DICOM CIELab encoding (PS3.3 C.10.7.1.1), so we decode it to\n", + "CIELab and then convert to sRGB." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "def dicom_cielab_to_rgb(cielab):\n", + " \"\"\"Convert a DICOM-encoded Recommended Display CIELab Value (3 uint16) to an sRGB [0-255] triplet.\"\"\"\n", + " L = cielab[0] / 65535.0 * 100.0 # L* in [0, 100]\n", + " a = cielab[1] / 65535.0 * 255.0 - 128.0 # a* in [-128, 127]\n", + " b = cielab[2] / 65535.0 * 255.0 - 128.0 # b* in [-128, 127]\n", + "\n", + " # CIELab -> XYZ (D65 white point)\n", + " fy = (L + 16) / 116.0\n", + " fx = fy + a / 500.0\n", + " fz = fy - b / 200.0\n", + " finv = lambda t: t ** 3 if t ** 3 > 0.008856 else (t - 16 / 116.0) / 7.787\n", + " X = 95.047 * finv(fx) / 100.0\n", + " Y = 100.0 * finv(fy) / 100.0\n", + " Z = 108.883 * finv(fz) / 100.0\n", + "\n", + " # XYZ -> linear sRGB -> gamma-corrected sRGB\n", + " r = 3.2406 * X - 1.5372 * Y - 0.4986 * Z\n", + " g = -0.9689 * X + 1.8758 * Y + 0.0415 * Z\n", + " bl = 0.0557 * X - 0.2040 * Y + 1.0570 * Z\n", + "\n", + " def gamma(c):\n", + " c = min(max(c, 0.0), 1.0)\n", + " return 1.055 * c ** (1 / 2.4) - 0.055 if c > 0.0031308 else 12.92 * c\n", + "\n", + " return [int(round(gamma(c) * 255)) for c in (r, g, bl)]\n", + "\n", + "\n", + "# Build a NiiVue label colormap. Entry 0 is background (fully transparent); entries 1..N use the\n", + "# SEG's own recommended display color for each organ.\n", + "R, G, B, A, I, labels = [0], [0], [0], [0], [0], [\"background\"]\n", + "for n in seg.segment_numbers:\n", + " desc = seg.get_segment_description(n)\n", + " rgb = dicom_cielab_to_rgb(desc.RecommendedDisplayCIELabValue)\n", + " R.append(rgb[0]); G.append(rgb[1]); B.append(rgb[2]); A.append(255)\n", + " I.append(int(n)); labels.append(desc.segment_label)\n", + "\n", + "label_colormap = {\"R\": R, \"G\": G, \"B\": B, \"A\": A, \"I\": I, \"labels\": labels}\n", + "print(f\"Built a label colormap with {len(R) - 1} organ colors taken from the DICOM SEG.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's preview a few of the organ colors that came from the SEG:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "legend = pd.DataFrame({\n", + " \"label\": labels[1:],\n", + " \"hex\": [f\"#{r:02x}{g:02x}{b:02x}\" for r, g, b in zip(R[1:], G[1:], B[1:])],\n", + "})\n", + "legend.head(12).style.apply(\n", + " lambda row: [f\"background-color: {row['hex']}; color: white\"] * len(row), axis=1\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 6. Visualize the image with ipyniivue\n", + "\n", + "We start with the image volume alone. Creating a `NiiVue` widget and calling `load_volumes` with a\n", + "list of `{\"path\": ...}` dictionaries renders the volume. By default NiiVue shows a **multiplanar**\n", + "layout (axial, coronal, sagittal, plus a 3D render).\n", + "\n", + "The `cal_min`/`cal_max` values set the display window in the image's native intensity units. For CT\n", + "these are Hounsfield Units; `-1000` to `400` gives a good lung/soft-tissue window.\n", + "\n", + "> Scroll with the mouse wheel to move through slices, drag to pan, and use the render panel to rotate\n", + "> the 3D view. If you don't see the viewer in Colab, make sure the \"Enable interactive widgets\" cell\n", + "> above ran successfully." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "nv = NiiVue(height=500)\n", + "nv.load_volumes([\n", + " {\"path\": str(image_nii), \"colormap\": \"gray\", \"cal_min\": -1000, \"cal_max\": 400},\n", + "])\n", + "nv" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 7. Overlay the multi-organ segmentation\n", + "\n", + "Now we load the label map on top of the image and apply the label colormap we built from the SEG.\n", + "The image is layer 0 and the segmentation is layer 1; we set the overlay opacity in `load_volumes`\n", + "and then attach the discrete label colormap with `set_colormap_label`, so each organ is drawn in its\n", + "own DICOM-recommended color. Hovering over the overlay in the viewer shows the organ name." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "nv_overlay = NiiVue(height=500)\n", + "nv_overlay.load_volumes([\n", + " {\"path\": str(image_nii), \"colormap\": \"gray\", \"cal_min\": -1000, \"cal_max\": 400},\n", + " {\"path\": str(labelmap_nii), \"opacity\": 0.5},\n", + "])\n", + "\n", + "# Apply the per-label colors taken from the DICOM SEG to the overlay (layer index 1)\n", + "nv_overlay.volumes[1].set_colormap_label(label_colormap)\n", + "nv_overlay" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 8. Customize the view\n", + "\n", + "The viewer is fully scriptable from Python. A few common adjustments:\n", + "\n", + "* **`set_slice_type`** switches the layout: `SliceType.AXIAL`, `CORONAL`, `SAGITTAL`, `RENDER` (3D\n", + " only), or `SliceType.MULTIPLANAR` (the default 2x2 grid).\n", + "* **`set_opacity(index, value)`** changes the opacity of the volume at a given layer index\n", + " (0 = base image, 1 = the segmentation overlay), letting you fade the segmentation in and out.\n", + "\n", + "Run the cell below and watch the viewer above update live." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "# Show only the axial plane, and make the segmentation overlay more transparent\n", + "nv_overlay.set_slice_type(SliceType.AXIAL)\n", + "nv_overlay.set_opacity(1, 0.3) # layer 1 == the segmentation overlay" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Switch back to the multiplanar + 3D layout and a more opaque overlay:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "nv_overlay.set_slice_type(SliceType.MULTIPLANAR)\n", + "nv_overlay.set_opacity(1, 0.6)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 9. Cite the data you used\n", + "\n", + "Most IDC data (including this collection) is covered by a CC BY license, which permits commercial use\n", + "**with attribution**. `idc-index` can generate ready-to-use citations for any selection of data \u2014\n", + "here both the NLST images and the TotalSegmentator analysis result." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "citations = client.citations_from_selection(seriesInstanceUID=[image_series_uid, seg_series_uid])\n", + "for c in citations:\n", + " print(c, \"\\n\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 10. Clean up (optional)\n", + "\n", + "Remove the downloaded DICOM and converted NIfTI files to free disk space." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "import shutil\n", + "# shutil.rmtree(work_dir, ignore_errors=True) # uncomment to delete the downloaded/converted files\n", + "print(\"To clean up, uncomment the line above and re-run this cell.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Next steps\n", + "\n", + "* Try a different image \u2014 the query in Section 2 works for any `collection_id` with SEG objects;\n", + " drop the `analysis_result_id` filter to also include expert/manual segmentations. Browse collections\n", + " in the [IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/).\n", + "* ipyniivue can also display **surface meshes** and **multi-frame 4D** data, draw/edit segmentations\n", + " in-browser, and export screenshots \u2014 see the [ipyniivue documentation](https://niivue.github.io/ipyniivue/).\n", + "* For a zero-setup browser viewer (no conversion needed), `client.get_viewer_URL(seriesInstanceUID=...)`\n", + " returns an OHIF/Slim viewer link for any IDC series.\n", + "* More tutorials are available in the\n", + " [IDC-Tutorials repository](https://github.com/ImagingDataCommons/IDC-Tutorials)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Support\n", + "\n", + "If you have any questions about this notebook, please post your question on the\n", + "[IDC User Forum](https://discourse.canceridc.dev) or\n", + "[open an issue](https://github.com/ImagingDataCommons/IDC-Tutorials/issues/new) in the\n", + "[IDC Tutorials repository](https://github.com/ImagingDataCommons/IDC-Tutorials)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Acknowledgments\n", + "\n", + "Imaging Data Commons has been funded in whole or in part with Federal funds from the National Cancer\n", + "Institute, National Institutes of Health, under Task Order No. HHSN26110071 under Contract No.\n", + "HHSN261201500003I.\n", + "\n", + "If you use IDC in your research, please cite the following publication:\n", + "\n", + "> Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S. D., Gibbs, D. L., Bridge, C., Herrmann, M. D., Homeyer, A., Lewis, R., Aerts, H. J. W., Krishnaswamy, D., Thiriveedhi, V. K., Ciausu, C., Schacherer, D. P., Bontempi, D., Pihl, T., Wagner, U., Farahani, K., Kim, E. & Kikinis, R. _National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence_. RadioGraphics (2023). https://doi.org/10.1148/rg.230180" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python" + }, + "colab": { + "provenance": [] + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/notebooks/advanced_topics/image_visualization_with_slicer_desktop.ipynb b/notebooks/advanced_topics/image_visualization_with_slicer_desktop.ipynb new file mode 100644 index 0000000..c73cfd0 --- /dev/null +++ b/notebooks/advanced_topics/image_visualization_with_slicer_desktop.ipynb @@ -0,0 +1,115 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": "\"Open" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "# Run the full 3D Slicer desktop application in Colab with IDC data\n\n---\n\n## Summary\n\nThis notebook shows how to run the **complete [3D Slicer](https://www.slicer.org) desktop\napplication** \u2014 the real, full graphical program, not just its rendering engine \u2014 **inside a Google\nColab cell**, and use it to interactively explore a radiology image and its segmentation downloaded\nfrom [NCI Imaging Data Commons (IDC)](https://imaging.datacommons.cancer.gov).\n\n3D Slicer is normally a desktop app. Here we launch the genuine application *headlessly* on the Colab\nvirtual machine (software OpenGL rendering into a virtual X display) and **stream its live GUI into\nthe notebook** over a WebSocket video connection using the [**desktopia**](https://github.com/pieper/desktopia)\nproject. The result is the actual Slicer interface \u2014 menus, module panel, slice views, 3D view,\nmouse interaction \u2014 embedded in an output cell, with **no desktop install** on your machine.\n\nAs the example dataset we use the **same CT + segmentation** as the other IDC 3D-viewer tutorials\n([ipyniivue](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/image_visualization_with_ipyniivue.ipynb)\nand [trame-slicer](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/trame_slicer_visualization.ipynb)):\na low-dose chest CT from the [National Lung Screening Trial (NLST)](https://imaging.datacommons.cancer.gov/explore/?filters_for_load=collection_id_nlst)\ntogether with its whole-body multi-organ [TotalSegmentator](https://github.com/wasserth/TotalSegmentator)\nsegmentation, published in IDC as the `TotalSegmentator-CT-Segmentations` analysis result.\n\n**How this differs from the other two viewer tutorials**\n* **ipyniivue** embeds a WebGL viewer (NiiVue) and needs the data converted to NIfTI first.\n* **trame-slicer** embeds the 3D Slicer *rendering engine* through a custom trame web UI.\n* **This notebook** streams the *entire 3D Slicer desktop application itself* \u2014 you get every Slicer\n module and the exact desktop UI, and Slicer loads the **DICOM image and DICOM SEG natively** (no\n conversion needed).\n\nUpon completion of this tutorial you will learn how to:\n* set up a headless Slicer environment in Colab and install the [QuantitativeReporting](https://github.com/QIICR/QuantitativeReporting) extension (which lets Slicer read DICOM SEG)\n* download a DICOM image series and its DICOM Segmentation from IDC with [`idc-index`](https://github.com/ImagingDataCommons/idc-index)\n* launch the real 3D Slicer application headlessly, load the DICOM data, and build 3D organ surfaces\n* stream and interact with the live Slicer desktop directly inside a Colab cell\n\n> **Runtime note.** A standard **CPU** Colab runtime is sufficient \u2014 rendering uses software OpenGL\n> (Mesa/llvmpipe). The first two cells download ~400 MB (3D Slicer) and install system libraries, so\n> the initial setup takes a few minutes.\n\n> **Credit.** The headless-Slicer-in-Colab streaming setup used here is from Steve Pieper's\n> [desktopia](https://github.com/pieper/desktopia) project; this notebook adapts its\n> `desktopia_tcia_kidney` example to use IDC as the data source.\n\n---\nInitial version: Jun 2026\n\nUpdated: Jun 2026" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 1. Install system dependencies\n\n3D Slicer is a Qt/VTK desktop application, so to run it headlessly we need a virtual X display\n(`xvfb`), a software OpenGL stack (Mesa), a lightweight window manager, and the GStreamer + XCB\nlibraries that desktopia uses to capture and stream the GUI. This cell installs them." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "%%bash\nset -e\ncd /content\nexport DEBIAN_FRONTEND=noninteractive\napt-get update -qq\napt-get install -y -qq --no-install-recommends \\\n xvfb xclip matchbox-window-manager fonts-dejavu-core libgl1-mesa-dri libglu1-mesa \\\n gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \\\n python3-gi gir1.2-gstreamer-1.0 gir1.2-gst-plugins-base-1.0 python3-xlib \\\n libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-shape0 \\\n libxcb-sync1 libxcb-xfixes0 libxcb-xinerama0 libxcb-xkb1 libxkbcommon-x11-0 libxcb-cursor0 libxcb-util1 \\\n libodbc2 libpq5 libpulse-mainloop-glib0 libpcre2-16-0 \\\n libxcomposite1 libxdamage1 libxtst6 libhwloc15 libnspr4 libnss3 >/dev/null\napt-get install -y -qq --no-install-recommends libasound2 libcups2 >/dev/null 2>&1 \\\n || apt-get install -y -qq --no-install-recommends libasound2t64 libcups2t64 >/dev/null\npip install -q websockets aioquic\necho 'deps installed' " + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 2. Install 3D Slicer and the QuantitativeReporting extension\n\nThis cell clones the desktopia streaming server, downloads the latest 3D Slicer release (~400 MB)\ninto `/opt`, and installs the [**QuantitativeReporting**](https://github.com/QIICR/QuantitativeReporting)\nextension. QuantitativeReporting provides the DICOM plugin that lets Slicer read **DICOM Segmentation\n(SEG)** objects like the one we will load from IDC. The install runs Slicer once, headlessly, just to\nadd the extension." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "%%bash\nset -e\ncd /content\nREPO=${DESKTOPIA_REPO:-https://github.com/pieper/desktopia}\nBRANCH=${DESKTOPIA_BRANCH:-software-render}\nrm -rf /content/desktopia\ngit clone -q --branch \"$BRANCH\" \"$REPO\" /content/desktopia || git clone -q \"$REPO\" /content/desktopia\nif ! ls -d /opt/Slicer-*/ >/dev/null 2>&1; then\n echo 'downloading 3D Slicer (~400 MB)...'\n curl -L --retry 3 'https://download.slicer.org/download?os=linux&stability=release' | tar -xz -C /opt\nfi\nSDIR=$(ls -d /opt/Slicer-*/ | head -1); echo \"Slicer: $SDIR\"\ncat > /tmp/install_qr.py <<'PY'\nimport slicer\nemm = slicer.app.extensionsManagerModel()\nemm.interactive = False\ntry: emm.updateExtensionsMetadataFromServer(True, True)\nexcept Exception as e: print('metadata:', e, flush=True)\nok = False\ntry:\n emm.downloadAndInstallExtensionByName('QuantitativeReporting', True, True)\n ok = emm.isExtensionInstalled('QuantitativeReporting')\nexcept Exception as e:\n print('install error:', e, flush=True)\nprint('QR_INSTALLED=', ok, flush=True)\nslicer.util.exit(0 if ok else 1)\nPY\necho 'installing QuantitativeReporting...'\nLIBGL_ALWAYS_SOFTWARE=1 GALLIUM_DRIVER=llvmpipe HOME=/root \\\n xvfb-run -a \"$SDIR/Slicer\" --no-splash --no-main-window --ignore-slicerrc \\\n --python-script /tmp/install_qr.py || echo 'QR install returned nonzero (see log above)' " + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 3. Download a CT and its segmentation from IDC\n\nWe use [`idc-index`](https://github.com/ImagingDataCommons/idc-index) to download the data directly\nfrom IDC \u2014 no TCIA / `tcia_utils` and no Google Cloud authentication required. For a fully\nreproducible demo we pin the **same** image + segmentation pair used in the ipyniivue and trame-slicer\ntutorials: a small NLST low-dose chest CT and its whole-body TotalSegmentator organ segmentation\n(both licensed CC BY 4.0).\n\nBoth series are downloaded with a *flat* layout (`dirTemplate=\"\"`) into one folder so that Slicer's\nDICOM importer can ingest the image and the SEG together." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "%pip install -q --upgrade idc-index\n\nimport glob\nfrom idc_index import IDCClient\n\nclient = IDCClient()\nprint(\"idc-index is using IDC data version:\", client.get_idc_version())\n\n# Same pinned pair as the other IDC 3D-viewer tutorials, for consistency:\nct_series = \"1.2.840.113654.2.55.71041873368734986406977154644223539362\" # NLST low-dose chest CT\nseg_series = \"1.2.276.0.7230010.3.1.3.313263360.84.1706324554.366745\" # TotalSegmentator SEG\n\nclient.download_from_selection(\n downloadDir=\"/content/idcDownload\",\n seriesInstanceUID=[ct_series, seg_series],\n dirTemplate=\"\", # flat layout: all .dcm files directly under idcDownload/\n)\n\nprint(\"DICOM files downloaded:\", len(glob.glob(\"/content/idcDownload/*.dcm\")))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 4. Launch 3D Slicer and load the data\n\nThis cell starts the real Slicer application on a virtual X display and runs a small startup script\n*inside* Slicer that:\n\n1. imports everything under `/content/idcDownload` into a temporary DICOM database and loads it \u2014\n Slicer reads the CT series **and** the DICOM SEG natively (the SEG becomes a segmentation node);\n2. builds a **3D closed-surface representation** for every segment so the organs appear in the 3D view\n (the TotalSegmentator SEG has dozens of structures, so this takes a few seconds);\n3. sets the four-up layout and frames the slice and 3D views.\n\ndesktopia then captures the GUI from the virtual display and serves it over a WebSocket video stream." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "import os, time, glob, pathlib, subprocess\nos.chdir('/content'); WORK = '/content/desktopia'\nWIDTH, HEIGHT, FPS, BITRATE = 1280, 720, 15, 4000\n\n# STARTUP runs inside Slicer: load /content/idcDownload, overlay the SEG, build 3D surfaces, frame views.\nSTARTUP = r\"\"\"\nimport slicer\nfrom DICOMLib import DICOMUtils\nloaded = []\nwith DICOMUtils.TemporaryDICOMDatabase() as db:\n DICOMUtils.importDicom('/content/idcDownload', db)\n for p in db.patients():\n loaded += DICOMUtils.loadPatientByUID(p)\nprint('loaded nodes:', loaded, flush=True)\nfor seg in slicer.util.getNodesByClass('vtkMRMLSegmentationNode'):\n seg.CreateClosedSurfaceRepresentation() # 3D surface of the segments\n dn = seg.GetDisplayNode()\n if dn:\n dn.SetVisibility(True); dn.SetVisibility3D(True)\nslicer.app.layoutManager().setLayout(slicer.vtkMRMLLayoutNode.SlicerLayoutFourUpView)\nvols = slicer.util.getNodesByClass('vtkMRMLScalarVolumeNode')\nif vols:\n slicer.util.setSliceViewerLayers(background=vols[0], fit=True)\nslicer.util.resetSliceViews()\ntry:\n tdv = slicer.app.layoutManager().threeDWidget(0).threeDView()\n tdv.resetFocalPoint(); tdv.resetCamera()\nexcept Exception as e:\n print('3d reset:', e, flush=True)\n\"\"\"\n\nenv = dict(os.environ, DISPLAY=':2', LIBGL_ALWAYS_SOFTWARE='1', GALLIUM_DRIVER='llvmpipe', HOME='/root')\nfor pat in ('server.py', 'SlicerApp-real'):\n subprocess.run(['pkill', '-f', pat], check=False)\nfor proc in ('Xvfb', 'matchbox-window-manager'):\n subprocess.run(['pkill', '-x', proc], check=False)\ntime.sleep(1)\n\nsubprocess.run('openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 '\n '-keyout /tmp/k.pem -out /tmp/c.pem -days 1 -nodes -subj /CN=desktopia',\n shell=True, check=True, stderr=subprocess.DEVNULL)\nsubprocess.Popen(f'Xvfb :2 -screen 0 {WIDTH}x{HEIGHT}x24 +extension GLX +render -noreset',\n shell=True, env=env, stdout=open('/tmp/xvfb.log','w'), stderr=subprocess.STDOUT)\nfor _ in range(80):\n if os.path.exists('/tmp/.X11-unix/X2'): break\n time.sleep(0.25)\nsubprocess.Popen('matchbox-window-manager -use_titlebar no', shell=True, env=env,\n stdout=open('/tmp/wm.log','w'), stderr=subprocess.STDOUT)\ntime.sleep(1)\n\nSDIR = sorted(glob.glob('/opt/Slicer-*/'))[0]\npathlib.Path('/tmp/slicer_startup.py').write_text(STARTUP)\nsubprocess.Popen(f'{SDIR}/Slicer --no-splash --python-script /tmp/slicer_startup.py',\n shell=True, env=env, stdout=open('/tmp/slicer.log','w'), stderr=subprocess.STDOUT)\n\npathlib.Path(f'{WORK}/client/status.json').write_text('{\"ready\":true,\"transport\":\"websocket\"}')\nsubprocess.Popen('python3 server.py --cert /tmp/c.pem --key /tmp/k.pem '\n f'--source xvfb --width {WIDTH} --height {HEIGHT} --fps {FPS} --bitrate {BITRATE} '\n '--ws-plain --serve-dir client',\n shell=True, env=env, cwd=WORK,\n stdout=open('/tmp/server.log','w'), stderr=subprocess.STDOUT)\ntime.sleep(8)\nprint('--- server.log ---'); print(open('/tmp/server.log').read()[-1200:])\nprint('Slicer is loading the CT+SEG; the view appears below in a few seconds.')" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 5. View the live 3D Slicer desktop\n\nRun the cell below to embed the streamed Slicer GUI in the notebook. Give it a few seconds to connect\nand for the CT + segmentation to finish loading, then interact directly:\n\n* **scroll** in a slice view to page through slices;\n* **left-drag** in the 3D view to rotate, **scroll** to zoom;\n* use the **module panel** on the left exactly as in desktop Slicer \u2014 every Slicer module is available\n (try *Segmentations*, *Volume Rendering*, or *Data*).\n\n> If the view is blank, re-run this cell (the stream may still have been starting), or re-run the\n> previous cell to relaunch Slicer." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "from google.colab import output\noutput.serve_kernel_port_as_iframe(4434, path='/index.html', height=600, cache_in_notebook=False)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 6. Cite the data you used\n\nMost IDC data (including this collection) is licensed CC BY 4.0, which permits commercial use **with\nattribution**. `idc-index` generates ready-to-use citations for any selection \u2014 here both the NLST\nimages and the TotalSegmentator analysis result." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "for citation in client.citations_from_selection(seriesInstanceUID=[ct_series, seg_series]):\n print(citation, \"\\n\")" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## Next steps\n\n* **Use the full Slicer app.** Because this is the complete 3D Slicer desktop, you can do anything\n Slicer supports \u2014 edit the segmentation in *Segment Editor*, tune *Volume Rendering*, take\n measurements, or run other extensions.\n* **Swap in different data.** The download in Section 3 works for any IDC series; change `ct_series`\n / `seg_series` to any image (CT, MR, PT, ...) and any DICOM SEG. Browse collections in the\n [IDC Portal](https://portal.imaging.datacommons.cancer.gov/explore/). Drop the segmentation to view\n an image alone.\n* **Prefer a lighter-weight embedded viewer?** See the companion tutorials that render inside the cell\n without streaming a full desktop:\n [ipyniivue](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/image_visualization_with_ipyniivue.ipynb)\n (WebGL widget) and\n [trame-slicer](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/trame_slicer_visualization.ipynb)\n (Slicer rendering engine via trame).\n* More tutorials are in the [IDC-Tutorials repository](https://github.com/ImagingDataCommons/IDC-Tutorials)." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## Support\n\nIf you have any questions about this notebook, please post your question on the\n[IDC User Forum](https://discourse.canceridc.dev) or\n[open an issue](https://github.com/ImagingDataCommons/IDC-Tutorials/issues/new) in the\n[IDC Tutorials repository](https://github.com/ImagingDataCommons/IDC-Tutorials).\n\nFor the Slicer-in-Colab streaming mechanism itself, see the\n[desktopia repository](https://github.com/pieper/desktopia)." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## Acknowledgments\n\nImaging Data Commons has been funded in whole or in part with Federal funds from the National Cancer\nInstitute, National Institutes of Health, under Task Order No. HHSN26110071 under Contract No.\nHHSN261201500003I.\n\nThis notebook runs [3D Slicer](https://www.slicer.org) headlessly in Colab using Steve Pieper's\n[desktopia](https://github.com/pieper/desktopia) project, with the\n[QuantitativeReporting](https://github.com/QIICR/QuantitativeReporting) extension for DICOM SEG\nsupport. The segmentation shown is from the IDC analysis result *TotalSegmentator-CT-Segmentations*\n([doi:10.5281/zenodo.8347011](https://doi.org/10.5281/zenodo.8347011)), computed on the NLST collection.\n\nIf you use IDC in your research, please cite the following publication:\n\n> Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S. D., Gibbs, D. L., Bridge, C., Herrmann, M. D., Homeyer, A., Lewis, R., Aerts, H. J. W., Krishnaswamy, D., Thiriveedhi, V. K., Ciausu, C., Schacherer, D. P., Bontempi, D., Pihl, T., Wagner, U., Farahani, K., Kim, E. & Kikinis, R. _National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence_. RadioGraphics (2023). https://doi.org/10.1148/rg.230180" + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/notebooks/advanced_topics/trame_slicer_visualization.ipynb b/notebooks/advanced_topics/trame_slicer_visualization.ipynb new file mode 100644 index 0000000..03e5196 --- /dev/null +++ b/notebooks/advanced_topics/trame_slicer_visualization.ipynb @@ -0,0 +1,467 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "5363e994", + "metadata": {}, + "source": [ + "# Visualizing IDC data in 3D with trame-slicer\n", + "\n", + "\"Open\n", + "\n", + "Updated: Jun 2026\n", + "\n", + "This notebook shows how to view [NCI Imaging Data Commons (IDC)](https://imaging.datacommons.cancer.gov) images **interactively, in 3D, inside a Colab cell** using [**trame-slicer**](https://github.com/KitwareMedical/trame-slicer).\n", + "\n", + "`trame-slicer` brings the [3D Slicer](https://www.slicer.org) rendering engine (MRML scene, multi‑planar reformat, volume rendering, segmentation display) to the browser through [trame](https://kitware.github.io/trame/). Rendering happens server‑side on the Colab VM and is streamed to the notebook output, so you get a real medical‑image viewer that understands orientation, windowing, spacing, and can overlay segmentations — without installing any desktop software.\n", + "\n", + "> **Background.** This tutorial was put together in response to [trame-slicer issue #81](https://github.com/KitwareMedical/trame-slicer/issues/81), where the IDC team described the need for a robust, notebook‑native 3D viewer for medical images to replace fragile alternatives such as `itkwidgets`. It also addresses the request in that thread to demonstrate viewing a DICOM image together with a DICOM segmentation.\n", + "\n", + "**What you will do**\n", + "1. Set up a headless OpenGL environment and install the packages.\n", + "2. Find a CT scan and a matching segmentation in IDC with `idc-index`.\n", + "3. Download the DICOM data.\n", + "4. Build a small, reusable trame-slicer viewer.\n", + "5. **Demo 1** — explore a CT volume with a 4‑up (Axial / Coronal / Sagittal / 3D) layout and interactive volume rendering.\n", + "6. **Demo 2** — overlay a whole‑body segmentation on the CT.\n", + "\n", + "**Runtime note.** A standard **CPU** runtime is sufficient (rendering uses software OpenGL). The first run installs the 3D Slicer Python core, which can take a few minutes." + ] + }, + { + "cell_type": "markdown", + "id": "8d022059", + "metadata": {}, + "source": [ + "## 1. Set up the environment\n", + "\n", + "The cell below installs the system OpenGL libraries that let VTK render **off‑screen** on a headless Colab VM (Mesa + OSMesa software rendering — the same setup `trame-slicer` uses in its own CI), then installs the Python packages:\n", + "\n", + "- `trame-slicer[standalone]` — the viewer plus the bundled 3D Slicer Python core (`slicer-core`);\n", + "- `idc-index` — querying and downloading IDC data;\n", + "- `dcmqi` — converting a DICOM Segmentation object to a labelmap that the viewer can load." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fe1bad26", + "metadata": {}, + "outputs": [], + "source": [ + "%%capture --no-stderr\n", + "# System OpenGL / off-screen rendering libraries (headless Colab VM).\n", + "!apt-get -qq update\n", + "!apt-get -qq install -y libgl1-mesa-glx libglu1-mesa libosmesa6 > /dev/null\n", + "\n", + "# Python packages. trame-slicer[standalone] also installs the 3D Slicer core.\n", + "!pip install -q \"trame-slicer[standalone]==1.11.0\" \"idc-index>=0.12.2\" dcmqi" + ] + }, + { + "cell_type": "markdown", + "id": "f1752499", + "metadata": {}, + "source": [ + "> If a later cell fails with an import or version error right after installation, use **Runtime → Restart session**, then run the notebook again **skipping the install cell above**. This is occasionally needed because the install upgrades packages already loaded by Colab." + ] + }, + { + "cell_type": "markdown", + "id": "d38df44b", + "metadata": {}, + "source": [ + "## 2. Find a CT scan and its segmentation in IDC\n", + "\n", + "We use `idc-index` to join the main `index` (one row per DICOM series) with `seg_index` (one row per DICOM Segmentation), matching each segmentation to the CT series it was computed on via `segmented_SeriesInstanceUID`. Here we look for small low‑dose chest CTs from the **NLST** collection that have a segmentation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "95dbdd4e", + "metadata": {}, + "outputs": [], + "source": [ + "from idc_index import IDCClient\n", + "\n", + "client = IDCClient()\n", + "print(\"IDC data version:\", client.get_idc_version())\n", + "\n", + "client.fetch_index(\"seg_index\")\n", + "candidates = client.sql_query(\"\"\"\n", + " SELECT i.collection_id,\n", + " i.PatientID,\n", + " i.SeriesInstanceUID AS ct_series,\n", + " i.instanceCount AS ct_slices,\n", + " ROUND(i.series_size_MB, 1) AS ct_MB,\n", + " s.SeriesInstanceUID AS seg_series,\n", + " i.license_short_name AS license\n", + " FROM seg_index s\n", + " JOIN index i ON s.segmented_SeriesInstanceUID = i.SeriesInstanceUID\n", + " WHERE i.Modality = 'CT'\n", + " AND i.collection_id = 'nlst'\n", + " AND i.instanceCount BETWEEN 80 AND 200\n", + " ORDER BY i.series_size_MB\n", + " LIMIT 5\n", + "\"\"\")\n", + "candidates" + ] + }, + { + "cell_type": "markdown", + "id": "1c01434d", + "metadata": {}, + "source": [ + "For a reproducible demo we pin to one specific pair from this collection. The CT is a single low‑dose chest acquisition (~80 slices, ~42 MB); the segmentation is a whole‑body organ segmentation from the IDC analysis result [**TotalSegmentator-CT-Segmentations**](https://doi.org/10.5281/zenodo.8347011) (71 structures). Both are licensed CC BY 4.0." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "69d75da6", + "metadata": {}, + "outputs": [], + "source": [ + "# Pinned series for this tutorial (feel free to swap in a pair from `candidates` above).\n", + "ct_series = \"1.2.840.113654.2.55.71041873368734986406977154644223539362\"\n", + "seg_series = \"1.2.276.0.7230010.3.1.3.313263360.84.1706324554.366745\"" + ] + }, + { + "cell_type": "markdown", + "id": "ab38c3d9", + "metadata": {}, + "source": [ + "## 3. Download the data\n", + "\n", + "We download each series into its own folder with a *flat* layout (`dirTemplate=\"\"`) so the DICOM files are easy to list." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4734bd87", + "metadata": {}, + "outputs": [], + "source": [ + "import glob\n", + "from pathlib import Path\n", + "\n", + "ct_dir, seg_dir = \"data/ct\", \"data/seg\"\n", + "client.download_from_selection(downloadDir=ct_dir, seriesInstanceUID=[ct_series], dirTemplate=\"\")\n", + "client.download_from_selection(downloadDir=seg_dir, seriesInstanceUID=[seg_series], dirTemplate=\"\")\n", + "\n", + "ct_files = sorted(glob.glob(f\"{ct_dir}/*.dcm\"))\n", + "seg_dcm = glob.glob(f\"{seg_dir}/*.dcm\")[0]\n", + "print(f\"Downloaded {len(ct_files)} CT slices\")\n", + "print(f\"Segmentation object: {seg_dcm}\")" + ] + }, + { + "cell_type": "markdown", + "id": "9a9e6348", + "metadata": {}, + "source": [ + "## 4. A reusable trame-slicer viewer\n", + "\n", + "The class below wraps the boilerplate from the `trame-slicer` examples into a small, reusable viewer:\n", + "\n", + "- a `SlicerApp` (the MRML scene, view manager, IO, display and volume‑rendering helpers);\n", + "- `register_rca_factories(...)` to stream the server‑side renders to the browser (the *remote‑controlled area*);\n", + "- a `LayoutManager` set to the **Quad View** (Axial / Coronal / Sagittal / 3D);\n", + "- methods to `load_volume(...)` and `load_segmentation(...)`, plus a *VR shift* slider that brightens/darkens the volume rendering.\n", + "\n", + "`show_in_colab(...)` starts the trame server in the background and embeds it in the cell. In Colab it routes the server port through `google.colab.kernel.proxyPort`; elsewhere (e.g. local Jupyter) it falls back to `localhost`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d6cf72cd", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "os.environ[\"TRAME_IPYWIDGETS_DISABLE\"] = \"1\" # use a plain iframe, not the ipywidgets backend\n", + "\n", + "import socket\n", + "from IPython.display import IFrame\n", + "\n", + "from trame.app import TrameApp, get_server\n", + "from trame_vuetify.ui.vuetify3 import SinglePageLayout\n", + "from trame_vuetify.widgets.vuetify3 import VSlider\n", + "\n", + "from trame_slicer.core import LayoutManager, SlicerApp\n", + "from trame_slicer.rca_view import register_rca_factories\n", + "\n", + "\n", + "class IDCSlicerViewer(TrameApp):\n", + " \"\"\"Minimal 4-up (Axial / Coronal / Sagittal / 3D) Slicer viewer for a notebook.\"\"\"\n", + "\n", + " _instances = 0\n", + "\n", + " def __init__(self, title=\"IDC trame-slicer viewer\"):\n", + " IDCSlicerViewer._instances += 1\n", + " # A uniquely named server per instance so multiple viewers can coexist in one notebook.\n", + " super().__init__(server=get_server(f\"idc_slicer_{IDCSlicerViewer._instances}\"))\n", + "\n", + " self._title = title\n", + " self._volume_node = None\n", + " self._preset_name = None\n", + "\n", + " self._slicer_app = SlicerApp()\n", + " register_rca_factories(self._slicer_app.view_manager, self.server)\n", + "\n", + " self._layout = LayoutManager(\n", + " self._slicer_app.scene,\n", + " self._slicer_app.view_manager,\n", + " self.server,\n", + " )\n", + " self._layout.register_layout_dict(LayoutManager.default_grid_configuration())\n", + " self._layout.set_layout(\"Quad View\")\n", + "\n", + " self.state.change(\"vr_shift\")(self._on_vr_shift)\n", + " self._build_ui()\n", + "\n", + " # ---- data loading -------------------------------------------------------\n", + " def load_volume(self, dicom_files, vr_preset=\"CT-Chest-Contrast-Enhanced\",\n", + " show_volume_rendering=True):\n", + " app = self._slicer_app\n", + " nodes = app.io_manager.load_volumes([str(f) for f in dicom_files])\n", + " if not nodes:\n", + " raise RuntimeError(\"Could not load the volume from the provided DICOM files.\")\n", + " self._volume_node = nodes[0]\n", + "\n", + " # Fall back gracefully if the requested rendering preset is unavailable.\n", + " presets = app.volume_rendering.preset_names()\n", + " if vr_preset not in presets:\n", + " vr_preset = next((p for p in presets if \"Chest\" in p),\n", + " presets[0] if presets else \"\")\n", + " self._preset_name = vr_preset\n", + "\n", + " app.display_manager.show_volume(self._volume_node, vr_preset=vr_preset,\n", + " do_reset_views=True)\n", + " if not show_volume_rendering:\n", + " vr_display = app.volume_rendering.get_vr_display_node(self._volume_node)\n", + " if vr_display:\n", + " vr_display.SetVisibility(False)\n", + "\n", + " # Configure the VR-shift slider range from the preset.\n", + " if self._preset_name:\n", + " lo, hi = app.volume_rendering.get_preset_vr_shift_range(self._preset_name)\n", + " self.state.vr_shift_min, self.state.vr_shift_max = lo / 10, hi / 10\n", + " else:\n", + " self.state.vr_shift_min, self.state.vr_shift_max = -125.0, 125.0\n", + " self.state.vr_shift = 0.0\n", + " return self._volume_node\n", + "\n", + " def load_segmentation(self, segmentation_file, show_3d=True):\n", + " app = self._slicer_app\n", + " seg = app.io_manager.load_segmentation(str(segmentation_file))\n", + " if seg is None:\n", + " raise RuntimeError(\"Could not load the segmentation file.\")\n", + " app.segmentation_editor.set_active_segmentation(seg, self._volume_node)\n", + " app.segmentation_editor.set_surface_representation_enabled(show_3d)\n", + " app.display_manager.reset_views()\n", + " return seg\n", + "\n", + " # ---- interactivity ------------------------------------------------------\n", + " def _on_vr_shift(self, vr_shift=0.0, **kwargs):\n", + " if not self._volume_node or not self._preset_name:\n", + " return\n", + " self._slicer_app.volume_rendering.set_absolute_vr_shift_from_preset(\n", + " self._volume_node, self._preset_name, float(vr_shift)\n", + " )\n", + "\n", + " # ---- UI -----------------------------------------------------------------\n", + " def _build_ui(self):\n", + " with SinglePageLayout(self.server) as self.ui:\n", + " self.ui.root.theme = \"dark\"\n", + " with self.ui.toolbar:\n", + " self.ui.toolbar.clear()\n", + " self.ui.title.set_text(self._title)\n", + " VSlider(\n", + " v_model=(\"vr_shift\", 0),\n", + " min=(\"vr_shift_min\", -125),\n", + " max=(\"vr_shift_max\", 125),\n", + " step=1, label=\"VR shift\", hide_details=True, density=\"compact\",\n", + " style=\"max-width: 320px; margin: auto 16px;\",\n", + " )\n", + " with self.ui.content:\n", + " self._layout.initialize_layout_grid(self.ui)\n", + "\n", + "\n", + "def _free_port():\n", + " s = socket.socket()\n", + " s.bind((\"\", 0))\n", + " port = s.getsockname()[1]\n", + " s.close()\n", + " return port\n", + "\n", + "\n", + "async def show_in_colab(app, height=720):\n", + " \"\"\"Start the trame server and embed the viewer in the current cell.\"\"\"\n", + " port = _free_port()\n", + " app.server.start(port=port, exec_mode=\"task\",\n", + " open_browser=False, show_connection_info=False)\n", + " await app.server.ready\n", + " try:\n", + " from google.colab.output import eval_js\n", + " url = eval_js(f\"google.colab.kernel.proxyPort({port})\")\n", + " except ModuleNotFoundError:\n", + " url = f\"http://localhost:{port}/\"\n", + " return IFrame(src=url, width=\"100%\", height=height)" + ] + }, + { + "cell_type": "markdown", + "id": "084e27ad", + "metadata": {}, + "source": [ + "## 5. Demo 1 — explore the CT volume\n", + "\n", + "Run the cell, wait a few seconds for the viewer to connect, and interact directly in the embedded view:\n", + "\n", + "- **Left‑drag** in the 3D view to rotate; **scroll** to zoom.\n", + "- **Scroll** in any slice view to page through slices.\n", + "- Drag the **VR shift** slider in the toolbar to brighten or darken the volume rendering." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4469b791", + "metadata": {}, + "outputs": [], + "source": [ + "viewer = IDCSlicerViewer(title=\"NLST low-dose chest CT\")\n", + "viewer.load_volume(ct_files, vr_preset=\"CT-Chest-Contrast-Enhanced\")\n", + "await show_in_colab(viewer)" + ] + }, + { + "cell_type": "markdown", + "id": "c2c9bcb5", + "metadata": {}, + "source": [ + "## 6. Demo 2 — overlay the segmentation\n", + "\n", + "A DICOM Segmentation object is not a scalar image, so we first convert it to a labelmap NRRD with `dcmqi`'s `segimage2itkimage`. The `--mergeSegments` flag writes all (non‑overlapping) structures into a single multi‑label volume — exactly what the viewer's `load_segmentation` expects." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b50d7adf", + "metadata": {}, + "outputs": [], + "source": [ + "import subprocess\n", + "\n", + "seg_out = \"data/seg_nrrd\"\n", + "Path(seg_out).mkdir(parents=True, exist_ok=True)\n", + "subprocess.run(\n", + " [\"segimage2itkimage\",\n", + " \"--inputDICOM\", seg_dcm,\n", + " \"--outputType\", \"nrrd\",\n", + " \"--prefix\", \"segment\",\n", + " \"--outputDirectory\", seg_out,\n", + " \"--mergeSegments\"],\n", + " check=True,\n", + ")\n", + "seg_nrrd = sorted(glob.glob(f\"{seg_out}/*.nrrd\"))[0]\n", + "print(\"Merged labelmap:\", seg_nrrd)" + ] + }, + { + "cell_type": "markdown", + "id": "37bec28b", + "metadata": {}, + "source": [ + "Now load the CT and the segmentation into a fresh viewer. We turn the CT volume rendering off so the colored organ surfaces are clearly visible in the 3D view, while the segmentation is also overlaid on the slice views." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aa803c95", + "metadata": {}, + "outputs": [], + "source": [ + "viewer2 = IDCSlicerViewer(title=\"NLST CT + TotalSegmentator\")\n", + "viewer2.load_volume(ct_files, vr_preset=\"CT-Chest-Contrast-Enhanced\",\n", + " show_volume_rendering=False)\n", + "viewer2.load_segmentation(seg_nrrd, show_3d=True)\n", + "await show_in_colab(viewer2)" + ] + }, + { + "cell_type": "markdown", + "id": "fcab5e3e", + "metadata": {}, + "source": [ + "## 7. Recap and next steps\n", + "\n", + "You loaded a CT scan and a segmentation straight from IDC and explored them in an interactive, server‑rendered 3D Slicer viewer running entirely inside Colab — no desktop install, no `itkwidgets`.\n", + "\n", + "From here you can:\n", + "- Swap in any IDC series UID (the same `load_volume` path also reads MR, PT, and other modalities, and a list of NRRD/NIfTI files).\n", + "- Load expert or AI segmentations from other IDC analysis results the same way.\n", + "- Build a richer UI: trame-slicer also ships ready‑made `MedicalViewerApp` and `SegmentationApp` (segment editing) applications under `trame_slicer.app`.\n", + "\n", + "**Attribution.** IDC data carries per‑collection licenses; always check them and cite the original sources. The cell below generates citations for the data used here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "035a1783", + "metadata": {}, + "outputs": [], + "source": [ + "for citation in client.citations_from_selection(seriesInstanceUID=[ct_series, seg_series]):\n", + " print(citation, \"\\n\")" + ] + }, + { + "cell_type": "markdown", + "id": "be6bb69e", + "metadata": {}, + "source": [ + "## Support\n", + "\n", + "If you have questions about IDC or this tutorial:\n", + "- Ask on the [IDC user forum](https://discourse.canceridc.dev/).\n", + "- Browse the documentation at [learn.canceridc.dev](https://learn.canceridc.dev/).\n", + "- For trame-slicer itself, see the [trame-slicer repository](https://github.com/KitwareMedical/trame-slicer).\n", + "\n", + "You can find more tutorials in the IDC-Tutorials repository: https://github.com/ImagingDataCommons/IDC-Tutorials\n", + "\n", + "## Acknowledgments\n", + "\n", + "Imaging Data Commons is a node within the broader NCI [Cancer Research Data Commons (CRDC)](https://datacommons.cancer.gov/) and has been funded in whole or in part with Federal funds from the NCI, NIH, under task order no. HHSN26110071 under contract no. HHSN261201500003l.\n", + "\n", + "This notebook uses [`trame-slicer`](https://github.com/KitwareMedical/trame-slicer) by Kitware, built on [3D Slicer](https://www.slicer.org) and [trame](https://kitware.github.io/trame/). The segmentation shown is from the IDC analysis result *TotalSegmentator-CT-Segmentations* ([doi:10.5281/zenodo.8347011](https://doi.org/10.5281/zenodo.8347011)), computed on the NLST collection.\n", + "\n", + "If you use IDC in your work, please cite:\n", + "\n", + "> Fedorov, A., et al. \"National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence.\" *RadioGraphics* 43.12 (2023). https://doi.org/10.1148/rg.230180" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}