Can the location of the disk cache be customised, and does CodeQL gracefully handle missing cache?

[gsutherland-trailofbits]

My key questions here are:

• Can the location of the disk cache (the one whose size is set by the --max-disk-cache flag) be set to a custom path?
• What happens if the cache directory disappears (e.g. due to a disk unmount) in the middle of an analysis run? Does CodeQL recover gracefully from this?

I tried looking for a flag to set the cache path but I couldn't find one that looked correct. I saw --compilation-cache and --common-caches but it didn't seem like they control the path of the intermediate result cache.

My motivation here is that I've got an extremely large codebase that I'm trying to run CodeQL analysis on, but the memory requirements make this analysis infeasible on any local machines I have access to - even with 100GiB of RAM and a 200GiB NVMe disk cache allocated to CodeQL, the queries haven't completed after 36h. The process stats make it clear that this is a memory bottleneck issue.

I would like to run the analysis on a cloud host with 1024 GiB of RAM. Understandably this is quite expensive, but since this is an occasional one-off job and I don't need guaranteed uptime I can save a lot of money by using spot instances on AWS. Spot instances can be much cheaper per hour (I'm looking at 70-80% savings per hour), but they come with no continuous uptime guarantees - at any point the instance can be automatically stopped (hibernated), migrated to other hardware, then resumed. This is mostly transparent to the instance.

On AWS the persistent storage attached to the instance is Elastic Block Storage (EBS), which is backed by a SAN. This means that all EBS IO goes over the network. While the link is reasonably fast (around 30Gbps) it is nowhere near as performant as local storage, especially for high IOPS tasks like disk cache. AWS also offers instances that come with local NVMe storage, which are much more suited for this type of cache. The downside is that this NVMe storage is ephemeral and will not persist if the spot instance is migrated.

I would like to be able to place the CodeQL disk cache on this NVMe storage since it offers a significant performance increase. However, I can't place the CodeQL database itself on the local NVMe storage because it will not be persisted across migrations. This is why I would like to specify the disk cache path. I can probably hack around this by running an analysis pass with a fast dummy query to generate a valid cache directory, copying the cache data to the NVMe, symlinking the cache directory to the NVMe mount path, then running analysis with the actual queries I want to run, but it would be nice to not have to do all that.

However, this still leaves the question of what happens if a migration occurs mid-query. Since the system is hibernated, moved to a new system, then resumed, it will be as if the contents of the cache directory were deleted mid-run. If CodeQL can't handle this gracefully then using the local NVMe for cache is probably a non-starter.

Is this use-case currently workable within CodeQL? If not, would it be feasible to implement?