FreeBSD segfault when capturing backtrace with RUST_BACKTRACE=1

[asomers]

Problem

With recent nightly toolchains, cargo segfaults on FreeBSD x86_64. The 2024-10-17 toolchain worked fine, but the bug was present in the 2024-10-24 toolchain . The segfault is dependent on the RUST_BACKTRACE=1 environment variable being set. This bug is causing rust-lang/libc's CI to fail.

An example CI failure:
https://github.com/rust-lang/libc/runs/32002285710

The stack trace of the segfault:

Program received signal SIGSEGV, Segmentation fault.
Address not mapped to object.
libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseCIE (addressSpace=..., cie=18446744073501904251, cieInfo=0x7fffffffacb0)
    at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/DwarfParser.hpp:332
332	  pint_t cieLength = (pint_t)addressSpace.get32(p);
(gdb) bt
#0  libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseCIE (addressSpace=..., cie=18446744073501904251, cieInfo=0x7fffffffacb0)
    at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/DwarfParser.hpp:332
rust-lang/cargo#1  0x0000000802b4ea0c in libunwind::CFI_Parser<libunwind::LocalAddressSpace>::findFDE (addressSpace=..., pc=pc@entry=19184556, ehSectionStart=41958640, sectionLength=<optimized out>, 
    fdeHint=fdeHint@entry=0, fdeInfo=fdeInfo@entry=0x7ffffffface8, cieInfo=0x7fffffffacb0) at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/DwarfParser.hpp:264
rust-lang/cargo#2  0x0000000802b4e6a2 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getInfoFromDwarfSection (this=this@entry=0x7fffffffb330, pc=pc@entry=19184556, 
    sects=..., fdeSectionOffsetHint=fdeSectionOffsetHint@entry=0) at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/UnwindCursor.hpp:1689
rust-lang/cargo#3  0x0000000802b4bb60 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister (this=0x7fffffffb330, isReturnAddress=true)
    at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/UnwindCursor.hpp:2634
rust-lang/cargo#4  0x0000000802b4b9b2 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0x7fffffffb330, stage2=<optimized out>)
    at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/UnwindCursor.hpp:2946
rust-lang/cargo#5  0x0000000802b48ef9 in _Unwind_Backtrace (callback=0x25720d0 <std::backtrace_rs::backtrace::libunwind::trace::trace_fn>, ref=0x7fffffffb4d8)
    at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/UnwindLevel1-gcc-ext.c:156
rust-lang/cargo#6  0x0000000002558fd5 in std::backtrace_rs::backtrace::libunwind::trace () at std/src/../../backtrace/src/backtrace/libunwind.rs:116
rust-lang/cargo#7  std::backtrace_rs::backtrace::trace_unsynchronized<std::backtrace::{impl#4}::create::{closure_env#0}> () at std/src/../../backtrace/src/backtrace/mod.rs:66
rust-lang/cargo#8  std::backtrace::Backtrace::create () at std/src/backtrace.rs:331
rust-lang/cargo#9  0x0000000002558e40 in std::backtrace::Backtrace::capture () at std/src/backtrace.rs:296
rust-lang/cargo#10 0x00000000011d5c08 in <anyhow::Error as core::convert::From<clap_builder::error::Error>>::from ()
rust-lang/cargo#11 0x00000000018a1f26 in <cargo::util::errors::CliError as core::convert::From<clap_builder::error::Error>>::from ()
rust-lang/cargo#12 0x00000000012de941 in cargo::cli::main ()
rust-lang/cargo#13 0x00000000012d14e4 in cargo::main ()
rust-lang/cargo#14 0x00000000012c98e3 in std::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()> ()
rust-lang/cargo#15 0x00000000012eebf9 in std::rt::lang_start::<()>::{closure#0} ()
rust-lang/cargo#16 0x0000000002558047 in core::ops::function::impls::{impl#2}::call_once<(), (dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)>
    () at core/src/ops/function.rs:284
rust-lang/cargo#17 std::panicking::try::do_call<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> () at std/src/panicking.rs:557
rust-lang/cargo#18 std::panicking::try<i32, &(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> () at std/src/panicking.rs:520
rust-lang/cargo#19 std::panic::catch_unwind<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> () at std/src/panic.rs:358
rust-lang/cargo#20 std::rt::lang_start_internal::{closure#1} () at std/src/rt.rs:174
rust-lang/cargo#21 std::panicking::try::do_call<std::rt::lang_start_internal::{closure_env#1}, isize> () at std/src/panicking.rs:557
rust-lang/cargo#22 std::panicking::try<isize, std::rt::lang_start_internal::{closure_env#1}> () at std/src/panicking.rs:520
rust-lang/cargo#23 std::panic::catch_unwind<std::rt::lang_start_internal::{closure_env#1}, isize> () at std/src/panic.rs:358
rust-lang/cargo#24 std::rt::lang_start_internal () at std/src/rt.rs:174
rust-lang/cargo#25 0x00000000012d58d5 in main ()

Steps

In a FreeBSD x86_64 environment with the latest nightly toolchain installed:

1. git clone git@github.com:rust-lang/libc.git
2. cd libc
3. env RUST_BACKTRACE=1 cargo test --features extra_traits --manifest-path libc-test/Cargo.toml --target x86_64-unknown-freebsd

Possible Solution(s)

No response

Notes

No response

Version

> cargo version --verbose
cargo 1.84.0-nightly (cf53cc54b 2024-10-18)
release: 1.84.0-nightly
commit-hash: cf53cc54bb593b5ec3dc2be4b1702f50c36d24d5
commit-date: 2024-10-18
host: x86_64-unknown-freebsd
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: FreeBSD 15.0-CURRENT [64-bit]

[weihanglo]

Thanks for the report. rust-lang/cargo#14712 might be related. Do you happen to know the RAM size of the CI machine?

[asomers]

Thanks for the report. #14712 might be related. Do you happen to know the RAM size of the CI machine?

I don't. But I can also reproduce the problem locally with 64 GB of RAM.

[weihanglo]

Cargo on nightly hasn't changed since nightly-2024-10-20. It is on the commit cf53cc54bb593b5ec3dc2be4b1702f50c36d24d5.

The last submodule update is #131910.

I can't tell which PR is relevant to this backtrace 😞.

rust-lang/cargo#10 0x00000000011d5c08 in <anyhow::Error as core::convert::From<clap_builder::error::Error>>::from ()
rust-lang/cargo#11 0x00000000018a1f26 in <cargo::util::errors::CliError as core::convert::From<clap_builder::error::Error>>::from ()
rust-lang/cargo#12 0x00000000012de941 in cargo::cli::main ()
rust-lang/cargo#13 0x00000000012d14e4 in cargo::main ()

[weihanglo]

Thanks for the report. #14712 might be related. Do you happen to know the RAM size of the CI machine?

I don't. But I can also reproduce the problem locally with 64 GB of RAM.

If you are able to reproduce locally, would you mind building rust-lang/cargo@8c30ce5 cargo and running with rustc from 2024-10-24? Setting RUSTC=~/.rustup/toolchains/x86_64-unknown-freebsd/bin/rustc env var should help choose the correct compiler.

[asomers]

I've managed to bisect the problem. It was introduced by 6d88158 . I don't know why yet.

[linyihai]

I reproduced it when I ran cargo test on the Cargo repository.

Cargo repository commt sha: e75214ea4936d2f2c909a71a1237042cc0e14b07

About 43 freshness_checksum::* tests failed, I ran cargo test again, it ouputs core dumped

failures:
aaa_trigger_cross_compile_disabled_check
freshness_checksum::binary_depinfo_correctly_encoded
freshness_checksum::bust_patched_dep
freshness_checksum::change_panic_mode
freshness_checksum::changing_bin_features_caches_targets
freshness_checksum::changing_bin_paths_common_target_features_caches_targets
freshness_checksum::changing_lib_features_caches_targets
freshness_checksum::changing_linker
freshness_checksum::changing_profiles_caches_targets
freshness_checksum::changing_rustflags_is_cached
freshness_checksum::channel_shares_filenames
freshness_checksum::checksum_actually_uses_checksum
freshness_checksum::checksum_build_compatible_with_mtime_build
freshness_checksum::dirty_both_lib_and_test
freshness_checksum::dont_rebuild_based_on_plugins
freshness_checksum::edition_change_invalidates
freshness_checksum::env_build_script_no_rebuild
freshness_checksum::env_in_code_causes_rebuild
freshness_checksum::linking_interrupted
freshness_checksum::metadata_change_invalidates
freshness_checksum::modifying_and_moving
freshness_checksum::move_target_directory_with_path_deps
freshness_checksum::no_rebuild_if_build_artifacts_move_backwards_in_time
freshness_checksum::no_rebuild_transitive_target_deps
freshness_checksum::no_rebuild_when_rename_dir
freshness_checksum::path_dev_dep_registry_updates
freshness_checksum::rebuild_if_environment_changes
freshness_checksum::rebuild_on_mid_build_file_modification
freshness_checksum::rebuild_sub_package_then_while_package
freshness_checksum::rebuild_tests_if_lib_changes
freshness_checksum::rename_with_path_deps
freshness_checksum::rerun_if_changed_in_dep
freshness_checksum::rerun_if_changes
freshness_checksum::reuse_panic_build_dep_test
freshness_checksum::reuse_panic_pm
freshness_checksum::reuse_shared_build_dep
freshness_checksum::reuse_workspace_lib
freshness_checksum::same_build_dir_cached_packages
freshness_checksum::same_size_different_content
freshness_checksum::script_fails_stay_dirty
freshness_checksum::skip_checksum_check_in_selected_cargo_home_subdirs
freshness_checksum::unused_optional_dep
freshness_checksum::use_checksum_cache_in_cargo_home
freshness_checksum::verify_source_before_recompile

test result: FAILED. 3473 passed; 44 failed; 22 ignored; 0 measured; 0 filtered out; finished in 138.74s

error: test failed, to rerun pass --test testsuite
➜ cargo git:(master) ✗
➜ cargo git:(master) ✗ cargo test
memory allocation of 63063237072 bytes failed
[1] 496663 IOT instruction (core dumped) cargo test

My toolchains

cargo -vV
cargo 1.83.0-nightly (15fbd2f60 2024-10-08)
release: 1.83.0-nightly
commit-hash: 15fbd2f607d4defc87053b8b76bf5038f2483cf4
commit-date: 2024-10-08
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Ubuntu 22.4.0 (jammy) [64-bit]


➜  cargo git:(master) ✗ rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /root/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu
nightly-2024-03-05-x86_64-unknown-linux-gnu
nightly-2024-05-13-x86_64-unknown-linux-gnu
nightly-2024-06-30-x86_64-unknown-linux-gnu
nightly-2024-08-11-x86_64-unknown-linux-gnu
nightly-2024-09-05-x86_64-unknown-linux-gnu
nightly-2024-09-23-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu (default)
1.75.0-x86_64-unknown-linux-gnu

installed targets for active toolchain
--------------------------------------

wasm32-wasi
x86_64-unknown-linux-gnu

active toolchain
----------------

nightly-x86_64-unknown-linux-gnu (default)
rustc 1.83.0-nightly (52fd99839 2024-10-10)

BTW, I remembered I had came across the core dump issues this month, my solutions is update my toolchain and run cargo clean.

Hope my experence can be helpful.

[asomers]

@linyihai I think you actually ran into the other bug: rust-lang/cargo#14712 , judging by the "memory allocation ... bytes failed" message.