enable interactive exploration of examples using jupyterlite-sphinx

[keewis]

We're already using jupyterlite in the main website, but we could also make all the docstring examples executable using jupyterlite-sphinx.

This doesn't quite work at the moment because:

• try_examples fails if the examples sections is last jupyterlite/jupyterlite-sphinx#291 causes the build to fail
• after fixing that locally it requires running a local http server (python -m http.server -b 127.0.0.1, for example) such that the browser does not block http requests because of the CORS headers

---

• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[agriyakhetarpal]

Thank you for reporting the bug and playing around with it upstream, @keewis! It should be fixed with a 0.20.1 release, but please feel free to test it from my fork if you'd like to, just in case. I confirm that it resolves the original reproducer, by the way!

[agriyakhetarpal]

• after fixing that locally it requires running a local http server (python -m http.server -b 127.0.0.1, for example) such that the browser does not block http requests because of the CORS headers

I'm slightly curious about this point, though – I imagine one would need to start a server to view the Sphinx docs anyway?

[keewis]

you can use the file protocol to access static html files locally, which is what I had been doing to check the results of builds on my computer.

[agriyakhetarpal]

Ah, yes, that wouldn't work with JupyterLite indeed.

[keewis]

@agriyakhetarpal, do you know if it's possible to globally add a preamble cell to each example notebook? The purpose would be to import common modules and set up the environment, like we do for pytest's doctest runner

[agriyakhetarpal]

@agriyakhetarpal, do you know if it's possible to globally add a preamble cell to each example notebook? The purpose would be to import common modules and set up the environment, like we do for pytest's doctest runner

Yes, this is currently not implemented, but it should be possible to include (I was working on a feature exactly like this some time back, but there's no PR yet). One thing I have to note here is that such code will be displayed to the user and would be included in a cell at the top of each example's generated notebook; it won't be hidden/silent, unfortunately – jupyterlite/jupyterlite#461 has some more details around this.

Making some code executed in a hidden has been a long-requested feature in JupyterLite in some ways (especially for installing packages at runtime with the Pyodide kernel but not showing the cell in which is done, for example, see jupyterlite/pyodide-kernel#60).

However, if you mean that each example can do simple things like import numpy, import matplotlib, set the random seed, etc. and such code doesn't necessarily mean to be hidden, that simplifies things :)

[keewis]

I don't mind this being visible, so just inserting a code cell before / after the warning would be fine with me. I just played around with this by modifying my local installation of jupyterlite-sphinx, and this seems to be very easy to do.

[agriyakhetarpal]

don't mind this being visible, so just inserting a code cell before / after the warning would be fine with me.

Thanks for the clarification, that works!

I just played around with this by modifying my local installation of jupyterlite-sphinx, and this seems to be very easy to do.

Nice! Just to ensure that our codes don't conflict – could you please put together a PR when you have a chance? I've just pushed my (a little incomplete and possible conflicting with yours) changes to my branch at https://github.com/agriyakhetarpal/jupyterlite-sphinx/tree/feat/generic-notebook-modifications – I last worked on this a little more than a month ago.

Note that the original discussion around my changes is here: sphinx-gallery/sphinx-gallery#1427, which is where we should be settling down on the implementation after everything is done.

In short, the idea was that Sphinx-Gallery offers "notebook modification functions" for its JupyterLite notebook galleries (like the way you suggest) and it also uses jupyterlite-sphinx as an optional dependency, but I believed that such functionality belongs in jupyterlite-sphinx itself, and Sphinx-Gallery can simply reuse it from us if we provide APIs to do so.

I could go forward with my changes, but I don't intend to block yours at all, so please go for it or let me know what can work best! :)

[agriyakhetarpal]

In the meantime, the 0.20.1 release is out: https://github.com/jupyterlite/jupyterlite-sphinx/releases/tag/v0.20.1, on both PyPI and conda-forge.

[keewis]

I've just pushed my (a little incomplete and possible conflicting with yours) changes to my branch

yours is a much bigger change, but I don't think it will conflict: I modified the return value of examples_to_notebook by calling nb.cells.insert(1, new_code_cell(preamble)), where preamble is read from a (hard-coded) file called try_examples_preamble.py (could also be a setting in conf.py, but that way we get autoformatting / linting). This change is thus confined to TryExamplesDirective.run, whereas yours appears to also apply to the other directives?

I'll open a draft PR to show you want I mean. Edit: see jupyterlite/jupyterlite-sphinx#293

[keewis]

I can confirm that the examples here work in the RTD preview if I manually fill in the preamble we discussed above. If anyone's interested, it might be nice to style the buttons in a way that is in line with the sphinx theme we use.

I did notice that retrying with a different example appears to re-install everything, which is a bit wasteful (but not sure how easy that would be to fix)

[keewis]

with the most recent version jupyterlite-sphinx is working locally (the missing _contents directory is still weird, not sure how to fix that).

However, there are a few rough edges remaining: right now, we use rattler-build directly, but this could also make use of the pixi rattler build backend. Also, the declared version is always a dev version, it could make use of setuptools_scm and modify the returned version.

[keewis]

Looks like ea4a483 fixed the build!

Now all we need to do is to clean up the pixi tasks (cc @VeckoTheGecko, maybe we can figure out how to depend on pixi build from a task) and the creation of the environment file, but I guess this can be done in a later PR.

[VeckoTheGecko]

re. naming, perhaps ci/rattler/recipe.yaml and ci/rattler/prepare_recipe.py is clearer ?

[keewis]

I was hoping to at some point get rid of the recipe in favor of using the output of pixi build with the python backend, we'd just need to get the version to work properly (I copied this from healpix-geo, where I had to use rattler directly because I need to compile rust code to emscripten-wasm32).

Edit: in the meantime the rename sounds fine to me.

[VeckoTheGecko]

AFAIK Rattler already has quite good templating ability, and can even read from environment variables. Perhaps we can have a pre-build step here that just sets these environment variables before being picked up by rattler (then we don't have to work outside of Rattler's conceptual framework here)

[keewis]

there's two templates right now: the version in the recipe, and the local path of the built conda package in the jupyterlite-xeus environment file. I agree, we can probably use a environment variable to set the version in the rattler recipe, but the environment file may need to have the external templating step? Unless you're proposing to use a rattler build step for that, as well?