Run the UI test suite with multiple editions

[Veykril]

Proposal

Right now the whole UI test suite is executed under edition 2015, unless a test opts into another one.
While in general this is fine (most code doesn't change behavior between editions), it prevents testing that the test suite passes on all editions.
This is something we need to do in Ferrocene, and it might also be desirable for the project to test it too in the future.

The good news is, compiletest already allows changing the default edition:

./x test --test-args=--edition=2024 tests/ui

Unfortunately, some test files are not compatible with newer editions.
For example, running the UI test suite with --edition 2021 causes 740 tests to fail in a recent-ish clone of the master branch.
This MCP proposes we update the test suite so that it's compatible with all editions.

I didn't manually check all the failures, but by grouping them by failure reason (and spot checking some of them) none of the failures seem to indicate actual bugs.
Rather, a lot of tests depend on the 2015 module system, omit the dyn keyword, try to run rustfix without specifying an edition, or similar cases.

This MCP thus makes four proposals.

Fix tests that fail on 2018 and newer

We should go through all the test failures and make sure all tests work on editions 2018, 2021 and 2024.
While this will have to be a manual effort, most of the failures have the same underlying cause.
So it will be possible to send PRs like "add dyn to every test" instead of huge PRs with miscellaneous fixes in them.

Add a CI job running the test suite in all supported editions

We should add a CI job running ./x test tests/ui --test-args=--edition=YYYY for all supported editions.
This job can run on a single platform, as its goal would be to ensure we don't accidentally add new tests that only work in a few editions.

Running it on a single platform won't catch all failures (if there are //@only-windows tests or similar), but it would provide enough coverage without costing too many resources.

Add range syntax to //@ edition

While a small subset of tests can run only on a single edition (for example rustfix tests converting from 2018 to 2021), most of the uses of //@ edition mean "this test requires at least this edition".

We should add a way to specify a range of supported editions, and update most tests to use it.
The syntax is bikesheddable, a possibility would be to implement ranges in the field:

//@ edition: 2018
//@ edition: 2018..
//@ edition: ..2018
//@ edition: 2018..2024

When a range is specified, compiletest will pick the default version if it fits within the range, or otherwise the lowest version in the range.

Add a compiletest flag to only test the default edition

It would be helpful to add a compiletest flag to automatically ignore tests meant for an edition different than the one specified in --edition.
This would be helpful for the CI job mentioned above, and for Ferrocene use.
It would not be enabled by default.

Mentors or Reviewers

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

[rustbot]

Important

This issue is not meant to be used for technical discussion. There is a Zulip stream for that.
Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections can formally be registered here by adding a comment.

@rfcbot concern reason-for-concern
<description of the concern>

Concerns can be lifted with:

@rfcbot resolve reason-for-concern

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler

[Zalathar]

@rfcbot concern edition-directive-meaning

Zulip: This adds a lot of complexity to the current meaning and behaviour of the edition directive, and it's unclear how contributors/maintainers should understand and use the edition directive in the future.