Implement forced keywords (k#)

[fmease]

View all comments

Part of #153839. CC @dianne

Introduces a new token kind to Rust >=2021 that looks like k#ident and that is called forced keyword (identifier). This is gated behind a new unstable feature called forced_keywords. This is backed by compiler MCP 945.

The ident in k#ident must be a keyword (from any edition) or a weak/contextual keyword for the token to be valid.

---

What won't be done in this PR:

1. implementing forced keyword lifetimes ('k#static)
2. migrating away from built-in syntax builtin # $ident($($tt)*) / introducing k#-exclusive (weak) keywords
   • that'll be done in PR Replace builtin # with k# #162232 instead
3. allowing the user to force edition-dependent & context-dependent keywords to be keywords
   • re. edition-dependent: forcing them to be keywords even in editions where they're usually not a (strong) keyword (today, that only affects gen anyway IINM)
   • re. context-dependent: E.g., union is only "active" if it's followed by a non-reserved identifier and under this PR the same rules apply to k#union even though that's not necessary; changing it would result in better diagnostics (e.g., for k#union struct {}: error: expected item, found `k#union` => expected identifier, found keyword `struct`) and maybe also allow for disambiguation for other context-dependent keywords
4. extending the proc_macro::Ident API to allow users to programmatically create forced keywords (might never be added)

_{(No LLM was or will be used by me during the entire creation process of this PR)}

[fmease]

@bors try @rust-timer queue

[fmease]

I've intentionally not marked it internal even though this feature is only backed by a T-compiler MCP and not by a T-lang RFC or in-tree experiment:

I would find it a bit weird if we (in a hypothetical future) told users to "please try out" "k#only bounds" or "&k#own types & exprs" (features that would obviously be non-internal) but they'd have to enable an internal feature in tandem (could be perceived as off-putting).

However, I'm okay with switching it to incomplete or even internal if requested.

View changes since the review

[fmease]

We can't test the roundtripping in the current form of this PR because I haven't added any k#-exclusive weak keywords where it'd be observable if we accidentally dropped k# during encoding&decoding.

I have however tested locally that it works by introducing a new dummy keyword. A hypothetical UI test would have a setup like

use proc_macro::TokenStream;

#[proc_macro]
pub fn perform(_: TokenStream) -> TokenStream {
    let stream: TokenStream = "const _: Option<k#never> = None;").parse().unwrap(); // positive
    //let stream: TokenStream = "fn k#branded() {}").parse().unwrap(); // negative

    stream.into_iter().collect() // forces encoding+decoding
}

View changes since the review

[rust-bors]

☀️ Try build successful (CI)
Build commit: 4eb8eac (4eb8eac3e01e4dbc9d58546df979f51a2ac2e5d5)
Base parent: b751e7a (b751e7a48501be14dcd57b4a430b73a0e51a0c52)

[rust-timer]

Finished benchmarking commit (4eb8eac): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking means the PR may be perf-sensitive. Consider adding rollup=never if this change is not fit for rolling up.

@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.2%	[-0.3%, -0.2%]	2
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results (primary -3.0%, secondary 1.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.8%	[2.5%, 3.1%]	2
Improvements ✅ (primary)	-3.0%	[-4.1%, -1.9%]	3
Improvements ✅ (secondary)	-2.2%	[-2.2%, -2.2%]	1
All ❌✅ (primary)	-3.0%	[-4.1%, -1.9%]	3

Cycles

Results (primary 2.1%, secondary -1.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.1%	[2.1%, 2.1%]	1
Regressions ❌ (secondary)	2.2%	[2.2%, 2.2%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.2%	[-4.1%, -2.3%]	2
All ❌✅ (primary)	2.1%	[2.1%, 2.1%]	1

Binary size

Results (primary 0.0%, secondary -0.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.0%, 0.1%]	4
Regressions ❌ (secondary)	0.0%	[0.0%, 0.0%]	1
Improvements ✅ (primary)	-0.0%	[-0.1%, -0.0%]	4
Improvements ✅ (secondary)	-0.0%	[-0.1%, -0.0%]	2
All ❌✅ (primary)	0.0%	[-0.1%, 0.1%]	8

Bootstrap: 477.576s -> 469.017s (-1.79%)
Artifact size: 400.35 MiB -> 400.38 MiB (0.01%)

[fmease]

Ready for review.

Caution

Only review the commits created after ------------------------- BRANCH SEPARATOR -------------------------.
That's because this PR is stacked on top of PR #161792. Hence, it's marked S-blocked Status: Blocked on something else such as an RFC or other implementation work. .

Best reviewed commit by commit.

[rustbot]

Some changes occurred in compiler/rustc_builtin_macros/src/autodiff.rs

cc @ZuseZ4

rust-analyzer is developed in its own repository. If possible, consider making this change to rust-lang/rust-analyzer instead.

cc @rust-lang/rust-analyzer

[fmease]

Either in this PR or in a follow-up PR I'll do something less hazardous than blindly using is_weak wrt. stability:

Let's assume forced_keywords was stable, then we certainly wouldn't want to insta-stabilize k#WEAK if somebody introduced an unstable keyword WEAK but that's what would happen in the current implementation as we'd immediately start accepting #[cfg(false)] M!(k#WEAK);.

(On main, declaring a weak keyword in symbol.rs doesn't have any semantic consequences, it just defines a new interned symbol for which is_weak (only used in diagnostics & tools atm) returns true. It's more of a convention, so you can write things like .is_keyword(kw::Weak) instead of .is_keyword(sym::weak) in the parser.)

Anyways, I plan on splitting splitting weak keywords into two categories, "stable" and "non-committal" (…). For the latter category, I'm either gonna issue a feature gate in lexer/mod.rs if forced with k# or I'm just not gonna allow using k# until stabilization. Emitting a feature gate for the token, too, means we'd start emitting two feature gates for unstable syntax (one in the lexer, one in the parser) which most likely wouldn't get deduplicated by our diagnostic infra since the spans will differ. There might be some tricks for avoiding it though. Will think about that some other time.

View changes since the review

[petrochenkov]

Could you move all the commits here except fffba8a to a separate PR?
They don't require any language process unlike the primary change, and I could review and merge them quickly, and it would make this PR much smaller.
@rustbot author

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[rust-bors]

☔ The latest upstream changes (presumably #163043) made this pull request unmergeable. Please resolve the merge conflicts by rebasing.