Example names, pyEXP and documentation integration

[The9Cat]

Thinking about a brief push to clean up the integration of pyEXP-examples, EXP-examples, and EXP-docs. In short:

• A good fraction of the pyEXP tutorials depend on or refer to DiskHaloA and DiskHaloB. These names were intended to be placeholders, I think. But we should name them for real. Or just decide to keep the current names.
• Do we want a reference run for either of these that can be used for comparison? We used to have one, I think, at least for DiskHaloA. This can be used by a pyEXP user who doesn't want to run EXP for the express purpose of going through the tutorial.
• Then, after we settle on the names, there are places in the readthedocs docs that might benefit from references to the some of the pyEXP tutorials and how-tos.

The goal is to provide a more seamless experience between the docs, the code base and the example repos.

[michael-petersen]

• Yes, DiskHaloA and DiskHaloB were placeholders. I was thinking the intent behind two examples would be to have one fully included with the software such that users could test pyEXP-examples without needing to run EXP. Some outputs from DiskHaloA can ship with the code. Then DiskHaloB would be a 'run-it-yourself' example for EXP, that could also serve as the benchmark.
• Given the push towards CI, can we have one of the examples actually run after the code is compiled, to serve as a formal reference check? We'd burn through GitHub hours, but we'd also catch any changes sooner.
• Yes, agreed that pyEXP tutorials and documentation will need to be edited once we pick a strategy.

[The9Cat]

• I'm fully in favor of your first point. Let's try to finalize the names so I/we can pick away at the docs. I'm leaning towards just leaving them as DiskHaloA and DiskHaloB because I don't have a better idea. At the very least, we should develop a sanity check based on DiskHaloA. And have that run do dual purpose for pyEXP examples. The only downside is that DiskHaloA is pretty noisy. So the current tutorial uses DiskHaloB coefficients because they look better. No other reason. Maybe that's an irrelevant argument?
• I like the idea of some sort of automatic check. I worry that it's a lot for the GitHub CI. I wonder if a make test on EXP is a better solution at least to start? I have experimented with DejaGnu (https://www.gnu.org/software/dejagnu) in the past (for BIE). A quick google does not suggest a more modern replacement for that. Someone would have to take on the task of implementing that.

[The9Cat]

CMake has module called ctest which relies on some code that runs and checks results through its return code. That code, could be Python, of course. So maybe that's the way to go?

See https://coderefinery.github.io/cmake-workshop/testing/

[The9Cat]

With the make test generated by ctest by CMake, GitHub describes how to put that into CI here.

So nice to have, and worth putting on a ToDo list, but I'm not making at a priority for me.

[michael-petersen]

I like that plan. Let's leave this open, and circle back to ctest at some point.

As for the nomenclature of DiskHaloA and DiskHaloB, what about DemoDiskHaloRun and DemoNBodyScript? The latter loses the specificity of noting that this is a disc+halo, but these are closer to explaining what we intend with the demos.

[The9Cat]

Hm, maybe DemoNbodyRun instead of DemoNbodyScript? Since the current the DiskHaloB config differs from the first one only in being more similar to what we might do for production? But I like these better than the A & B for sure.

[The9Cat]

Bumping this.

1. Let's decide on the directory names. They are included in various places in both pyEXP-examples and readthedocs.
2. Also: do we want to include/restore the data directory with some fiducial output? Such as OUTLOG.run0 and outcoef.*.run0, and perhaps OUT.run0.*? The argument for outcoef.*.run0 is for pyEXP notebooks and there's a tutorial that uses OUT.run0.* to make coefficients as well. OUTLOG.run0 is optional. Although perhaps we want a notebook that illustrates OUTLOG.* as well?

[The9Cat]

Tweaking the names a bit more, I'm thinking:

• DiskHaloA-->DemoDiskHalo
• DiskHaloB-->DemoNbody

I'm also in favor of restoring the data directory so that people using pyEXP-examples/Tutorials have a full data set to use without having to run a simulation themselves.

[The9Cat]

Also wondering if the Demo prefix is superfluous? What does that add? Perhaps we want:

• DiskHaloA-->DiskHalo
• DiskHaloB-->Nbody
  ??

[The9Cat]

The naming convention is now addressed by PR #5. Nothing included about ctest yet.

[michael-petersen]

I like these names a lot. Thinking SimpleHalo --> Halo as well? I can make this change.

What kinds of tests would we want? An actual small N-body run? Or just the python side?

[The9Cat]

Yes, SimpleHalo-->Halo also makes good sense; 'simple' isn't distinguishing here.

I've updated bits in the readthedocs in a local branch to reflect these changes. Haven't pushed that yet.

The Nbody/data now contains the snapshots, coef files and outlog from the Nbody run. I'm thinking that a Python script that compares some aspects of these files with a users local run would be useful. E.g. perhaps some difference within some tolerance of the coefficient files and/or outlog file? Maybe selected coefficient indices (lower order ones?) and some summary outlog fields (a few energy ones), but not all? This script could be the scaffolding for ctest integration later on.