lint / ImproperCTypes: better handling of indirections, take 2

[niacdoial]

This PR tries to re-add the changes in #131669 (which were reverted in #134064 after one (1) nightly),
and adds better coverage of ty_kinds:

• in the take-1-added TypeSizedness enum and its construction
• in a new test file

The changes in the original PR aim to make ImproperCTypes/ImproperCTypesDefinitions produce better warnings when dealing with indirections (Box, &T, *T), especially for those to DSTs.

r? workingjubilee (because you reviewed the first attempt)

[niacdoial]

(
hi Jubilee, I'm back at this again!
I know this is not the best time of year to add PRs, so I'm fine with postponing this if you don't feel like tackling it these upcoming weeks.
In any case, have some nice end-of-year festivities, if you celebrate any!
)

[niacdoial]

ah, and before I forget: a small part of the new test file is commented out because it hits ICE #134587, but there should be more than decent coverage anyway

[workingjubilee]

unfortunately the lint needs to be gutted and rewritten.

[workingjubilee]

Also while I was possibly having a mild case of get-there-itis and thus mostly tried to just make sure things were coherent, I would prefer all new code for the lint be in compiler/rustc_lint/src/types/improper_ctypes.rs.

[workingjubilee]

The version cut happened so there will be less time pressure now.

[niacdoial]

I have a first version for compiler/rustc_lint/src/types/improper_ctypes.rs if you want.
it's less of a from-the-ground-up rewrite as it is scrapping the original for parts, if the analogy makes sense.

you probably have things to say about its architecture, even if the whole thing still have a bunch of TODO comments
the progress so far looks like this:

• completely separate the type-checking and reporting systems
• (part of the way there) remove some of the special cases and integrate them to the "main logic"
  • check_for_opaque_types is still a "special case" part of the checking logic
  • Cstr and Cstring are also somewhat special-cased because the advice for them depends on the type around them, if any
  • the unit type is handled in multiple places, see if this can be fixed
• (almost complete) compile, pass existing tests
  • only failed tests are for Cstring, due to different error messages
  • one unrelated test had to have a second "#[allow(improper_ctypes)]" added, but it makes more sense for it to need that anyway
• better separation of the different checks in different visit_* methods of ImproperCTypesVisitor
• better tracking of how the currently-checked type is used (static, function argument, function return's inner type, etc...)
  • raises questions about the separation of improper_ctypes and improper_ctypes_definitions versus declared/defined functions, especially when FnPtr:s are involved
• allow single argument check to emit multiple errors (for fnptr:s, structures with multiple FFI-unsafe fields, etc)
• review what is considered FFI-safe or not (once everything else is complete)

If you want to take a look in this state, should I just commit it here? (possibly put the PR in draft mode while I'm at it?)
or send you the files in a different way?

[bors]

☔ The latest upstream changes (presumably #135525) made this pull request unmergeable. Please resolve the merge conflicts.

[niacdoial]

aaaand I think I have something that's "first draft" material! (should I put this pull in the "draft" state?)
There's still a bunch of TODOs (...well, they were changed to FIXMEs to be pushed here) for questions I didn't manage to answer (and a bunch of failing tests because I don't know if the error should be here), but yeah.

here's a list of some of my remaining questions and concerns:

• visit_numeric seems too x86_64-specific

• should we revisit the distinction between ImproperCTypes and ImproperCTypesDefinitions?

  • part 1: the output ("external fn" or vs "external block" vs other possibilities)
  • part 2: handling opaque types (there's a high correlation between ImproperCTypesDefinitions and places where we allow FFI-opaque types to be fully specified. do we want this correlation to be 1?)

• more on FFI-opaque types: how do we handle that in the context of the "context switch" between functions and possible FnPtr arguments? The answer that seems correct currently prevents a stage1 compiler from being built

  • should we introduce a std::ffi::FfiOpaquePtr type? (which would be a *const c_void and some phantomdata, on first approximation)

• for indirections whose values may be supplied by non-rust code: do we only allow pointers (and Optionstd::ptr::NonNull), or do we also allow Option<&T> and Option<Box<T>>?

• not sure if the new error messages are intelligible in all cases (especially if there's a type param like Self or <Self as ::std::ops::Add<Self>>::Output that gets resolved in the error message).

• it feels like the current handling of CStr/Cstring and Option-like enums uses special casing, since those are tested for in multiple places.

• if we deny references and boxes in defined functions, what of &self in methods? We don't allow *const Self, last I checked.

[bors]

☔ The latest upstream changes (presumably #135921) made this pull request unmergeable. Please resolve the merge conflicts.

[workingjubilee]

hmm.

[workingjubilee]

aaaand I think I have something that's "first draft" material! (should I put this pull in the "draft" state?)

Yes, it's a good marker for "I don't want this merged yet, even if it looks done".

[workingjubilee]

visit_numeric seems too x86_64-specific

It probably is.

should we revisit the distinction between ImproperCTypes and ImproperCTypesDefinitions?

Yes, but in particular, not to just repartition them between: I think breaking them into as many conceptually-smaller lints as possible is good, as long as each one is a distinct idea (no splitting just for the sake of splitting!).

more on FFI-opaque types: how do we handle that in the context of the "context switch" between functions and possible FnPtr arguments? The answer that seems correct currently prevents a stage1 compiler from being built

I'm not sure what you mean?

for indirections whose values may be supplied by non-rust code: do we only allow pointers (and Option<std::ptr::NonNull>), or do we also allow Option<&T> and Option<Box<T>>?

We must allow Rust code to declare a pointer in a C signature to be Option<&T> or a number of things about our FFI story fall apart.

it feels like the current handling of CStr/CString and Option-like enums uses special casing, since those are tested for in multiple places.

Yes, probably.

[workingjubilee]

some initial nits on a first pass

[workingjubilee]

1. Because unsized function parameters are something we may want to support.
2. The code may not be well-formed: as you may have noticed at some point, you get warnings even if you get errors (usually), and this is because we lint even on "bad" code. This is because rustc didn't use to, once upon a time, and it was a bad debugging experience.

[niacdoial]

alright, sorry for taking a while!

I'm currently planning what changes I'll do in terms of splitting the lint(s)
my current idea is to separate based on the nature of the thing being (presumably) propped up against a FFI boundary

• improper_ctypes: what's definitely an interface to an outside library (extern statics, extern function declarations)
• improper_ctypes_fn_definitions: functions written in rust intended to be exported
• improper_ctypes_callbacks: FnPtr arguments, no matter in what function they are being used
• improper_ctypes_ty_definitions: repr(C) structs/enums(/unions)?

more on FFI-opaque types: how do we handle that in the context of the "context switch" between functions and possible FnPtr arguments? The answer that seems correct currently prevents a stage1 compiler from being built

I'm not sure what you mean?

well, this is more or less answered in what I said before that, but my question was about how to deal with "switching" from checking arguments for, say, a function definition, to checking the arguments of a FnPtr argument?

1. should the nature of the lint change?
   (temptative answer: yes)
2. how should FFI-Safe-pointers-to-FFI-Unsafe-pointees work in FnPtr arguments? Should it be the rules for extern fn declarations? (throw the lint because one should use *const c_void, an extern type declaration, etc...) or the rules for extern fn definitions? (allow that, the function's body needs the full type even if it's opaque to the other side of the FFI boundary)
   (temptative answer: it should be the former, but parts of the rustc codebase doesn't follow this rule, so I can't get a stage1 compiler if I make that the rule)

---

// you would think that int-range pattern types that exclude 0 would have Option layout optimisation
// they don't (see tests/ui/type/pattern_types/range_patterns.stderr)
// so there's no need to allow Option<pattern_type!(u32 in 1..)>.

oh, I should fix that probably

I... maybe? I can't for the life of me find the link to that again but I think I saw a discussion about that and type covariance/contravariance,
where i32 is 1.. is a subtype of i32 (well that was under consideration), meaning fn(Option<i32>) is a type of fn(Option<i32 is 1..>) and it might have impacts on whether there should be an optimisation because of transmutation?

Though you'll definitely know more than me on all the moving parts.
Especially assuming you might have looked at this more in the past week.

---

As for the rest of your advice, I already took all this in!
thanks for shedding light on my code, one nit at a time!

[niacdoial]

it has been. two months.
er... here, it's what I currently have. it still needs to be rebased on top of the main branch, but if for some reason you wanted to review it early (or if you wanted proof that I didn't just give up), here it is.

I think I made quite a few decisions, but here are the most important ones I remember:

• repr(C) on a struct/union/enum is now seen as enough of a signal for the lint to look closer
• the "this type's value can come from the other side of the FFI boundary and might break the nonzero assumption" part is deactivated when linting on FnPtrs (I'm banking on the warning showing up in another place anyway, eventhough there's a specific case where it might not)
• uninhabited types are now linted against in argument position and static variables, and only ! and empty enums are allowed as uninhabited return types.
• &self in methods will be considered just as unsafe as other un-Option'd references, eventhough last I checked self: Option<&Self> is not something that compiles

[niacdoial]

alright, now the latest commit does (locally) pass tests and build stage2.
(it also did yesterday before that impromptu rebase)
I'll check the previous commits weren't degraded tomorrow, and then I'll make a summary of what is done and what still needs to be done

[rust-log-analyzer]

The job x86_64-gnu-llvm-19 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

#18 exporting to docker image format
#18 sending tarball 20.4s done
#18 DONE 28.6s
##[endgroup]
Setting extra environment values for docker:  --env ENABLE_GCC_CODEGEN=1 --env GCC_EXEC_PREFIX=/usr/lib/gcc/
[CI_JOB_NAME=x86_64-gnu-llvm-19]
[CI_JOB_NAME=x86_64-gnu-llvm-19]
debug: `DISABLE_CI_RUSTC_IF_INCOMPATIBLE` configured.
---
sccache: Listening on address 127.0.0.1:4226
##[group]Configure the build
configure: processing command line
configure: 
configure: build.configure-args := ['--build=x86_64-unknown-linux-gnu', '--llvm-root=/usr/lib/llvm-19', '--enable-llvm-link-shared', '--set', 'rust.randomize-layout=true', '--set', 'rust.thin-lto-import-instr-limit=10', '--set', 'build.print-step-timings', '--enable-verbose-tests', '--set', 'build.metrics', '--enable-verbose-configure', '--enable-sccache', '--disable-manage-submodules', '--enable-locked-deps', '--enable-cargo-native-static', '--set', 'rust.codegen-units-std=1', '--set', 'dist.compression-profile=balanced', '--dist-compression-formats=xz', '--set', 'rust.lld=false', '--disable-dist-src', '--release-channel=nightly', '--enable-debug-assertions', '--enable-overflow-checks', '--enable-llvm-assertions', '--set', 'rust.verify-llvm-ir', '--set', 'rust.codegen-backends=llvm,cranelift,gcc', '--set', 'llvm.static-libstdcpp', '--set', 'gcc.download-ci-gcc=true', '--enable-new-symbol-mangling']
configure: build.build          := x86_64-unknown-linux-gnu
configure: target.x86_64-unknown-linux-gnu.llvm-config := /usr/lib/llvm-19/bin/llvm-config
configure: llvm.link-shared     := True
configure: rust.randomize-layout := True
configure: rust.thin-lto-import-instr-limit := 10
---
test [crashes] tests/crashes/130346.rs ... ok
test [crashes] tests/crashes/130395.rs ... ok
test [crashes] tests/crashes/130524.rs ... ok
test [crashes] tests/crashes/130411.rs ... ok
2025-05-22T21:40:13.124989Z ERROR compiletest::runtest: fatal error, panic: "crashtest no longer crashes/triggers ICE, hooray! Please give it a meaningful name, add a doc-comment to the start of the test explaining why it exists and move it to tests/ui or wherever you see fit. Adding 'Fixes #<issueNr>' to your PR description ensures that the corresponding ticket is auto-closed upon merge. If you want to see verbose output, set `COMPILETEST_VERBOSE_CRASHES=1`."
test [crashes] tests/crashes/130310.rs ... FAILED
test [crashes] tests/crashes/130797.rs ... ok
test [crashes] tests/crashes/130970.rs ... ok
test [crashes] tests/crashes/131292.rs ... ok
test [crashes] tests/crashes/131048.rs ... ok
---
failures:

---- [crashes] tests/crashes/130310.rs stdout ----

error: crashtest no longer crashes/triggers ICE, hooray! Please give it a meaningful name, add a doc-comment to the start of the test explaining why it exists and move it to tests/ui or wherever you see fit. Adding 'Fixes #<issueNr>' to your PR description ensures that the corresponding ticket is auto-closed upon merge. If you want to see verbose output, set `COMPILETEST_VERBOSE_CRASHES=1`.

thread '[crashes] tests/crashes/130310.rs' panicked at src/tools/compiletest/src/runtest/crashes.rs:17:18:
fatal error
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace