check_mod_item_types query times regressed bigly

[tmandry]

There is a crate inside of Fuchsia whose compile times increased from ~500ms to ~120s in the last month of Rust compiler updates. It hasn't been easy so far to provide a minimal example, but I will keep working at it.

Some bisecting revealed that it actually consisted of two performance regressions, both affecting the time spent in check_mod_item_types:

Commit hash	Item	Self time	% of total time	Item count	Cache hits	Blocked time	Incremental load time
87b0c90	check_mod_item_types	34.92ms	7.120	6	0	0.00ns	0.00ns
34e82a7	check_mod_item_types	21.92s	97.786	6	0	0.00ns	0.00ns
66bf391	check_mod_item_types	21.76s	97.838	6	0	0.00ns	0.00ns
0221e26	check_mod_item_types	133.53s	99.526	6	0	0.00ns	0.00ns

The first regression occurred in the rollup #64354, and the second occurred sometime in the two weeks between changes 66bf391..0221e26.

There is heavy use of async/await in the code, and I've identified a couple places where taking out .await makes the compile finish right away. This makes me suspect #64292 for the first regression, but I'm not sure.

I'll update the issue as I have more information.

[tmandry]

There is heavy use of async/await in the code, and I've identified a couple places where taking out .await makes the compile finish right away.

Interestingly, removing .await from either of these places makes the compile much faster. If I leave both in, it's slow, but taking one or both out makes it faster. Maybe there's some kind of cycle being created when both are left in?

In fact, switching on RUSTC_LOG=debug actually causes a cycle error to be reported when there wasn't one before. This is really weird to me, but it may be a red herring. Removing one or both of the .await calls (or calls to the methods that create the futures) doesn't resolve the cycle error, even though it resolves the slowdown. Here's a sample of the error, for completeness.

[tmandry]

I don't have a lot more information to put here, except that the compile time seems to increase with the size of a particular call graph of async fns (it's not binary slow/fast, but different gradations of slow). This continues to make it hard to minimize.

I did some profiling, and noticed a lot of time being spent collecting items in SmallVec::iter. Unfortunately I couldn't see what was calling that function. There was some speculation that it could be from this code (cc @Mark-Simulacrum who suggested this).

See this chat thread for more details, and some flame graphs of the profiles I ran.