Distribute libc.a as a separate library instead of bundling it into liblibc.rlib on musl and wasi

[petrochenkov]

Right now the libc crate links native libc with #[link(kind = "static")] for two targets - musl and wasi.

kind = "static" means that object files from native libc.a are bundled into liblibc.rlib and distributed with rustc toolchain in this form.
Then if user builds something with +crt-static then the bundled object files are used, and if user builds with -crt-static then they are "unbundled" and dynamic linking with libc.so available at user site happens.

Experience with windows-gnu targets (#67429) and porting rust to musl-native targets (#40113) showed that it's strongly preferable to use the native toolchain available at user site to using anything bundled with rustc toolchain, and that includes libc.a as well.

At the same time rustc historically provides more "self-contained" experience for selected targets (windows-gnu, musl, wasm), so we must continue shipping libc.a in some form in case the user doesn't have a native toolchain available.

That we means we need to fall back from using the native toolchain if it's available to using bundled libc.a (and other libraries) if it's not.
Such scheme is already used relatively successfully for windows-gnu, but not for musl and wasi.

So, the plan of the work should be roughly like this:

• Link libc as kind = "static-nobundle" instead of kind = "static" in the libc crate (Don't bundle static libc. libc#1327).
• Modify rustbuild to distribute libc.a for musl (and whatever is its equivalent for wasi) similarly to how we distribute crt1.o objects for the same targets.
• Organize library search directories during linking in such way that they find the native toolchain first and then fall back to the rust toolchain's sysroot where the bundled libc.a resides (should be similar to windows-gnu: prefer system crt libraries if they are available #67429).
• Eliminate the "unbundling" logic from the compiler.

Once this is done #65760 should become fixed as well.

UPD: libunwind (https://github.com/rust-lang/rust/tree/master/src/libunwind) on musl also bundles libunwind.a, it should probably have the same treatment.

cc @hvenev @gnzlbg @alexcrichton @mati865 @smaeul

[retep998]

If only we could also eliminate kind = "static" with fire...

[alexcrichton]

FWIW It's hard for me personally at least to read comments like "Eliminate the unbundling logic from the compiler (with fire)." and "If only we could also eliminate kind = static with fire..." as semi-attacks on me personally since I originally implemented all of this. I really do feel that much ire about these features is directed personally at me since I've tried to review patches and provide feedback on other designs, and often times things don't work out. I did the best I could with the resources and knowledge I had at the time. I'm sorry if things didn't work out.

The general idea here is "things just work". I don't think it really matters how they work, so long as they work. The "thing" here for targets like musl and wasi is "you don't need to install a native toolchain". I think anyone has much of a preference how things works so long as it works.

I'm gonna unsubscribe from this issue because I don't think I have much to offer and I'm likely to just get stressed out staying on this. Feel free to cc me if there's a specific question (e.g. about wasi), however.

[petrochenkov]

@alexcrichton
Apologies if some of this sounded like an attack.
This post is written with 5 years of hindsight available, but when this was originally implemented nobody could predict the future.

[retep998]

I'd also like to apologize for my comment. You've done a lot of fantastic work on Rust over the years, and I don't want any of my comments on specific features to feel like they're directed at you!
We've all done stuff in Rust that later turned out to not be the greatest (myself included), but such is hindsight.

kind=static has some unfortunate consequences such as LLVM mangling certain sections it doesn't understand (like Microsoft's LTCG stuff) and not being able to link to static libraries provided by the system. Unless they need to distribute rlibs with C libraries bundled in (which is rare outside of the rust toolchain) there's very little reason for users to not use static-nobundle over static. That it has taken the easy to remember name of static while static-nobundle has been left to languish in unstable land for years is somewhat frustrating, and I wish the situation could improve.

[petrochenkov]

while static-nobundle has been left to languish in unstable land for years

FWIW, I want to write an RFC for general syntax for linking modifiers for native libraries that would cover things like --whole-archive and --as-needed, and could cover bundling as well.
Perhaps it could be less controversial to stabilize it as a modifier to static rather than a separate linking kind.
(This will hopefully happen this or next month, otherwise I think static-nobundle can be stabilized as is.)

UPD: Done in rust-lang/rfcs#2951.

[mati865]

That would also fix #58163