[idea] Add an option put the metadata in a separate file

[bjorn3]

Detailed design

When -Zsplit-metadata is passed to rustc, it will put the metadata normally inside a rlib or dylib in a separate rmeta file. The rlib/dylib will contain the SVH to prevent accidentally mixing the rmeta and rlib/dylib of different compilations.

Advantages

This would make the increase in size of #56987 easily fixable by deleting the rlib files, while keeping the rmeta files. Those are not necessary anyway, because rustc_driver.so already contains the code.

This will decrease staticlib, dylib and cdylib compilation time, because there is no need to copy all object files to a new archive to remove the rust.metadata.bin file from the rlib.

rust/src/librustc_codegen_llvm/back/link.rs

Lines 1340 to 1342 in 6d9640b

	if f.ends_with(RLIB_BYTECODE_EXTENSION) || f == METADATA_FILENAME {
	archive.remove_file(&f);
	continue

Disadvantages

Rustc and cargo get a bit more complex.

[Xanewok]

If I understand this correctly, an another advantage is that we'd get a cargo check done for free when we do cargo {build, run, test}, which might speed up some workflows (no need to generate separate .rmeta for deps).

[bjorn3]

@Xanewok If I read rust-lang/cargo#3501 correctly, it is technically possible to do so already, but cargo doesn't support it.

[bjorn3]

If this is accepted rust-lang/cargo#4831 could be done without invoking rustc twice. Just let rustc print a message when the metadata file is emitted and allow it to read .rmeta files before the .rlib file is ready. When a crate is built which needs to run the linker (dylib or bin) either cargo or rustc could wait till all necessary .rlib files are generated.

[steveklabnik]

Triage: no real changes; sort of a duplicate of the cargo ticket, kinda.