Add 3D visualization notebooks (trame-slicer, ipyniivue, Slicer desktop)

[fedorov]

Add three advanced-topics notebooks for interactive visualization of IDC images and DICOM SEG overlays in Colab, and link them from the README:

• trame-slicer in-browser 3D rendering
• ipyniivue WebGL viewer widget
• full 3D Slicer desktop streamed via desktopia

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[fedorov]

Visualization approach analysis

How this was generated. This analysis was produced by Claude Code (Anthropic, model claude-sonnet-4-6) in response to a prompt from @fedorov asking for a deep-dive review of the three notebooks added in this PR. The agent read all three notebooks in full, fetched the README and release history of each upstream repository (trame-slicer, ipyniivue/NiiVue, desktopia) from GitHub, and synthesized the findings into the report below. The analysis focuses on the rendering architecture; DICOM conversion pipeline differences are out of scope (the assumption is that conversion steps can be added to any notebook as needed).

---

Three rendering approaches

Notebook	Tool	Rendering model
image_visualization_with_ipyniivue.ipynb	ipyniivue / NiiVue	WebGL in the browser — client-side
trame_slicer_visualization.ipynb	trame-slicer	VTK on the server, streamed as compressed frames
image_visualization_with_slicer_desktop.ipynb	desktopia	Full Slicer desktop, headless, H.264 video stream

---

1. ipyniivue — WebGL rendering in the browser

How it works

ipyniivue delivers the NiiVue JavaScript library to the browser as an anywidget bundle. All rendering — slice compositing, multi-planar reformat, volume ray-casting, colormap application — runs in WebGL inside the user's browser. The Python side handles configuration and state sync; once the files are loaded, the Python process is not in the render path at all.

Rendering performance

Excellent for the operations NiiVue is designed for: slice scrolling, pan/zoom, opacity changes, and layout switching respond at display-frame rate because no server round-trip is involved. GPU-accelerated volume ray-casting runs in the browser — the Colab VM's hardware does not matter because the GPU doing the work is the user's own machine.

Interactivity latency

Near-zero. All interaction is processed locally in the browser. This is the most responsive of the three approaches for typical viewing operations.

Browser and platform compatibility

Works in any browser with WebGL 2 support — Chrome, Firefox, Safari, Edge — on any operating system. No server-side GPU required. No OS-level native libraries. The anywidget delivery mechanism works in JupyterLab 3+, classic Notebook, VS Code notebooks, and Google Colab (one output.enable_custom_widget_manager() call required, guarded in a try/except). This is the most portable rendering approach of the three.

Rendering capability ceiling

NiiVue is a capable viewer but not a VTK-class rendering engine. Its volume rendering, while good, does not match the fidelity of VTK-based transfer functions or the multi-resolution streaming of large datasets. There is no concept of a MRML scene, no support for editing, no module system. For pure viewing tasks on data up to a few hundred MB, the ceiling is unlikely to matter.

Robustness to Colab environment changes

High. The rendering is entirely in the browser — the Colab VM only serves the widget bundle and the data files. The only Colab-specific hook is output.enable_custom_widget_manager(), which is a stable, documented Google API. Changes to system-level packages or the Colab Python runtime have essentially no effect on the rendering path.

Tool maturity and maintenance

ipyniivue is 5 years old (created 2021), currently at v2.4.4, with 12 contributors and an active release history. The primary maintenance dependency is keeping the bundled NiiVue JS version current as NiiVue adds features; recent history shows this tracking is done manually but consistently (v0.67.0 bundled as of Feb 2026).

Anticipated maintenance effort

Low. No server processes, no native binaries, no platform-specific wheels.

---

2. trame-slicer — VTK rendering on the server, frame-streamed to browser

How it works

trame-slicer embeds the 3D Slicer rendering engine — the full VTK/ITK pipeline, MRML scene graph, volume rendering, segmentation display — as a Python wheel (slicer-core). Rendering runs entirely server-side on the Colab VM using Mesa/OSMesa software OpenGL. trame's Remote Controlled Area (RCA) mechanism captures rendered frames, compresses them as JPEG, and streams them to an <iframe> in the browser. Mouse and keyboard events are forwarded back to the server, which re-renders and sends the next frame.

Rendering performance

VTK-grade: accurate MPR resampling, physically-based volume rendering with proper transfer functions, precise segmentation overlay compositing — identical quality to 3D Slicer on the desktop. The tradeoff is that rendering happens on a CPU-only VM (Mesa/llvmpipe on Colab's CPU runtime), so complex volume renders take longer to produce than on a GPU workstation, but output quality is authoritative.

Interactivity latency

Moderate. Every user interaction triggers a server render and a JPEG frame delivery. On a good Colab network connection this round-trip is typically under 500 ms — usable but perceptible. Slice scrolling is noticeably laggier than ipyniivue. Latency scales with network conditions between the user and the Colab datacenter.

Browser and platform compatibility

No browser restriction — the RCA stream is standard HTTP/WebSocket JPEG frames.

Significant platform restriction on the server side: slicer-core is available only for Linux x86-64 with Python 3.10–3.13. Colab is currently compatible. Local Jupyter on macOS or Windows cannot install this package. If Colab increments its Python minor version to one not yet covered by a slicer-core wheel build, the notebook breaks until a new wheel is published.

Robustness to Colab environment changes

Medium. The rendering stack depends on Mesa OpenGL system libraries installed via apt. The slicer-core platform wheel is the highest single-point risk: a Python version increment on Colab can make the wheel unavailable until the trame-slicer team publishes a new build. The trame server runs as a background task in the Python kernel — if the session goes idle and the kernel is recycled, the server process dies and the iframe goes blank (re-running the cell recovers it).

The notebook sets TRAME_IPYWIDGETS_DISABLE=1 before imports to force a plain iframe in Colab. This is a fragile compatibility shim that may need updating across trame versions.

Tool maturity and maintenance

v1.11.0, 29 releases in 13 months, 9 contributors, monthly cadence, CI with coverage, commercially backed by Kitware. However: the README explicitly states the API has not been reviewed by 3D Slicer core developers. The API is evolving, and the notebook's pinned version will need updating over time.

Anticipated maintenance effort

Medium. Periodic re-testing and version-pin updates as trame-slicer releases. Platform wheel availability requires monitoring.

---

3. desktopia — Full Slicer desktop, H.264 video stream

How it works

The notebook launches the complete 3D Slicer desktop application inside a headless virtual X display (Xvfb) on the Colab VM. Mesa/llvmpipe provides software OpenGL. A GStreamer pipeline captures the X display, encodes it as H.264 (software x264 — Colab blocks NVENC hardware encoding), and streams it to the browser via WebSocket using desktopia's custom server. The user sees and interacts with the complete Slicer GUI — menus, module panel, all views — as a video stream.

Rendering performance

Same VTK-based Slicer engine as trame-slicer on the same software OpenGL stack, with an additional layer of H.264 video compression artifacts — the image the user sees is a lossy-compressed version of what Slicer renders. Streaming is capped at 15 fps.

Interactivity latency

High. The interaction chain is: user input → WebSocket → VM → Slicer → VTK render → GStreamer H.264 encode → WebSocket → browser H.264 decode → canvas. Software encoding adds significant latency. This is a distinctly remote-desktop-like experience.

Browser and platform compatibility

The primary desktopia architecture (QUIC/WebTransport + WebCodecs) requires Chromium. The Colab notebook uses a software-only WebSocket path that bypasses this — so in Colab any browser works. But the notebook is entirely dependent on Colab's internal port-proxying mechanism (output.serve_kernel_port_as_iframe). Outside Colab, the notebook does not function.

Robustness to Colab environment changes

Very low. The failure surface is large:

1. Runtime git clone of an unversioned branch. The notebook fetches desktopia at runtime by cloning the software-render branch. Any push to that branch changes what every future run of the notebook executes. If the branch is renamed or deleted, the notebook fails immediately. The repository is 9 days old with no releases and one contributor.
2. ~20 apt packages. Colab's apt environment changes; version conflicts can silently break rendering or streaming.
3. Process orchestration. Xvfb, matchbox-window-manager, Slicer, and the streaming server are started as independent background subprocesses with time.sleep() synchronization — inherently fragile.
4. Colab port proxy. output.serve_kernel_port_as_iframe is a Google-internal API not documented as stable.
5. Slicer download URL. The notebook fetches https://download.slicer.org/download?os=linux&stability=release — always the current latest release, so the Slicer version can change silently between runs.

Tool maturity and maintenance

desktopia was created on 2026-05-31 — 9 days before this PR was filed. Zero formal releases, no semantic versioning, no CI, one contributor (Steve Pieper, a respected Slicer core developer). It is explicitly an experimental proof-of-concept.

Anticipated maintenance effort

Very high. This notebook will require active testing and fixes after essentially every Colab infrastructure change, every desktopia commit, and every new Slicer release. There is no practical CI path for catching regressions before users encounter them.

---

Comparative summary

Dimension	ipyniivue	trame-slicer	desktopia
Rendering location	Browser (WebGL)	Server (VTK/Mesa)	Server (VTK/Mesa)
Render quality	Good (WebGL)	Excellent (VTK)	Excellent (VTK), degraded by H.264
Interaction latency	Near-zero	Moderate (JPEG round-trip)	High (H.264 encode+decode round-trip)
Rendering FPS	Display-rate (browser)	On-demand frame delivery	Capped at 15 fps
Browser compatibility	All browsers	All browsers	All in Colab; Chromium-only otherwise
Platform (server side)	Any OS/arch	Linux x86-64 only	Linux only; Colab-specific delivery
Local Jupyter support	Excellent	Linux only	Not supported
Colab robustness	High	Medium	Very low
Colab idle-session resilience	Stateless (no server)	Server dies; re-run cell	All processes die; complex restart
Tool age / maturity	5 years, v2.4.4, stable API	17 months, v1.11, API unstable	9 days, no releases, 1 contributor
Maintenance burden	Low	Medium	Very high
Feature ceiling	Viewer only	Full Slicer rendering pipeline	Complete Slicer application

---

Recommendation

ipyniivue is the right default for IDC tutorials. It is the most portable, most robust, lowest-latency, and lowest-maintenance option. Its rendering quality is sufficient for the core IDC use case of exploring volumetric images and segmentation overlays. It should be the primary recommended notebook for users who want interactive 3D viewing in Colab.

trame-slicer is a valuable complementary notebook for users who need VTK-quality rendering or who want to work with the Slicer MRML pipeline. It should be clearly scoped as a Linux/Colab-only tool, and the IDC team should establish a periodic testing cadence against new trame-slicer releases to keep the pinned version current.

desktopia should not be published as a supported IDC tutorial in its current state. Depending at runtime on a 9-day-old, single-contributor, unversioned experimental repository that is cloned from a branch is too fragile for a public tutorial the IDC team will be asked to support. As a developer demo or community showcase — clearly labeled experimental, not linked from the main README — it could be appropriate. It should be reconsidered for promotion to a supported tutorial once desktopia publishes versioned releases and has demonstrated stability over several months.