Switch doc building CI provider?

[larsoner]

Thanks! OT, but I don't like a core service failing because a contributor must first configure something or is not logged in (as a side note, I've never been able to restart a job even though I am logged in, but I seem to have two accounts connected to my email, which is also very confusing). I suggest that we try to fix this on our side, or consider moving to a different service with better GitHub integration.

Originally posted by @cbrnr in #13680 (comment)

[larsoner]

Then @drammock mentioned:

happy to entertain alternatives if someone can do the research. IIRC, netlify is awful and GHA has no way of serving the built artifact for inspection, so RTD is the only one I know of that might be worth exploring. But maybe there are others I haven't heard of.

and @cbrnr said:

Yes, I would also avoid Netlify.

GHA has no way of serving the built artifact for inspection

It might be possible with a gh-pages branch, but I'm not sure. Something like https://github.com/mmkal/artifact.ci maybe, although probably not exactly this because of potential costs that the developer might charge, but we can always roll our own solution.

so RTD is the only one I know of that might be worth exploring

I use RTD for my projects and I'm very happy. Building and serving pretty much works out of the box, and I've never had any problems. Their build infrastructure is probably not as powerful, so we'd have to check how long the build process actually takes on their site.

[larsoner]

Indeed someone would need to look into it. Right now I see two main disadvantages of CircleCI:

1. Login issues. There are some login problems for new contributors, and they can be tricky to resolve. The workaround is typically for us maintainers to manually click a button on CircleCI saying to build it anyway, or to merge main into the branch, or to manually push an empty commit.
2. Cost. We go over the CircleCI free tier usage and pay for it. (Actually most of the cost comes from MNE-BIDS-Pipeline, but even before it started eating credits, we were still over and probably would have had to pay at some point.)

Other than these two issues, CircleCI has been flexible for years for us, and we have a lot of knowledge built around how to get it to do the things we want. Many (most?) other MNE-* packages also use CircleCI as well. So I am not super motivated to look into a change.

To flesh out a bit more why we use CircleCI, here are the advantages / features we make use of for CircleCI that we'd ideally want to keep at a new provider, as all of these things were added to meet a need / improve contribution and/or maintenance experience:

• Container specification and large(r than default) compute. We need a larger than default runner for CircleCI at least (memory+CPU), have had to change it in the past, and might need to change it in the future if memory or CPU requirements of examples go up.
• Long runtime support. Our main job takes ~1h with parallelization across examples, so the doc build has to tolerate long runtimes.
• Running multiple jobs. We currently do both make html and make linkcheck (on different schedules).
• Deployment to gh-pages with commit control. Eventually we will probably want to automate the squash step so we can remove it from the release process).
• Triaging doc build / Sphinx options based on commit contents and message. For PRs we run just modified examples, unless there's a [circle full] in there, then we run all of them. For main we build all examples, but only on a schedule. This is all possible because we have full control over what gets run when and with which options.
• Rurun-with-SSH. I've used this dozens (maybe hundreds across projects!) of times to debug CI issues for MNE in hours that would have taken days by multiple pushes and/or couldn't be adequately tested locally.
• Artifact redirection control. circleci-artifact-redirector-action allows us to specify which artifact to point to, not just index.html.

There could be other reasons, too -- these are just the ones that come to mind.

Moreover, if we switch, we'd want to be aware of any new disadvantages that we'd take on. For just one example, RTD at one point injected ads into built docs, which seems like a pretty big negative. Running our own self-hosted GH runner would come with new, un-fun / onerous maintenance overhead, etc.