Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: dtolnay/proc-macro2
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 1.0.52
Choose a base ref
...
head repository: dtolnay/proc-macro2
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref

Commits on Mar 22, 2023

  1. Copy the full SHA
    67c6cba View commit details
  2. Call site has no source text

    dtolnay committed Mar 22, 2023
    Copy the full SHA
    efeb5ec View commit details
  3. Merge pull request #350 from dtolnay/sourcetext

    Expose proc_macro's source_text() on Span
    dtolnay authored Mar 22, 2023
    Copy the full SHA
    b88dc25 View commit details
  4. Release 1.0.53

    dtolnay committed Mar 22, 2023
    Copy the full SHA
    98645fd View commit details

Commits on Mar 26, 2023

  1. Copy the full SHA
    990ddbb View commit details
  2. Copy the full SHA
    6eda714 View commit details
  3. Merge pull request #373 from dtolnay/filepath

    Lazily construct PathBuf for sourcemap entries
    dtolnay authored Mar 26, 2023
    Copy the full SHA
    5c05f4b View commit details
  4. Copy the full SHA
    16c2eda View commit details
  5. Copy the full SHA
    eb49335 View commit details
  6. Copy the full SHA
    da3949d View commit details
  7. Merge pull request #374 from dtolnay/cfgfuzzing

    Disable sourcemap thread_local during fuzzing
    dtolnay authored Mar 26, 2023
    Copy the full SHA
    35cda6d View commit details
  8. Release 1.0.54

    dtolnay committed Mar 26, 2023
    Copy the full SHA
    9f9328b View commit details

Commits on Mar 27, 2023

  1. Copy the full SHA
    f7c0435 View commit details
  2. Copy the full SHA
    8310b84 View commit details
  3. Copy the full SHA
    055a793 View commit details
  4. Work around cargo-afl checking LLVM runtime

    In GitHub Actions it was failing with:
    
        AFL LLVM runtime is not built with Rust rustc-1.70.0-nightly-0c61c7a,
        run `cargo install --force afl` to build it.
    dtolnay committed Mar 27, 2023
    Copy the full SHA
    93f8ee7 View commit details
  5. Merge pull request #375 from dtolnay/afl

    Add AFL++ support to fuzz target
    dtolnay authored Mar 27, 2023
    Copy the full SHA
    e9e5eb8 View commit details
  6. Copy the full SHA
    ba7ad43 View commit details
  7. Copy the full SHA
    75d9042 View commit details
  8. Merge pull request #376 from dtolnay/honggfuzz

    Add honggfuzz support to fuzz target
    dtolnay authored Mar 27, 2023
    Copy the full SHA
    455ca39 View commit details
  9. Copy the full SHA
    b539d5a View commit details

Commits on Apr 1, 2023

  1. Copy the full SHA
    287979f View commit details
  2. Add test of Literal span

    dtolnay committed Apr 1, 2023
    Copy the full SHA
    606ea5e View commit details
  3. Copy the full SHA
    da4c83d View commit details
  4. Merge pull request #378 from dtolnay/literalspan

    Create meaningful span for Literal in FromStr
    dtolnay authored Apr 1, 2023
    Copy the full SHA
    e163e79 View commit details
  5. Add test of Literal::subspan

    dtolnay committed Apr 1, 2023
    Copy the full SHA
    c1cf0f8 View commit details
  6. Implement Literal::subspan

    dtolnay committed Apr 1, 2023
    Copy the full SHA
    daccddb View commit details
  7. Merge pull request #379 from dtolnay/subspan

    Implement Literal::subspan
    dtolnay authored Apr 1, 2023
    Copy the full SHA
    3310787 View commit details
  8. Release 1.0.55

    dtolnay committed Apr 1, 2023
    Copy the full SHA
    c4a3e19 View commit details

Commits on Apr 3, 2023

  1. Copy the full SHA
    d827973 View commit details
  2. Copy the full SHA
    57b4db1 View commit details
  3. Ignore octal_escapes clippy lint in test

        error: octal-looking escape in string literal
           --> tests/test.rs:118:25
            |
        118 |         Literal::string("a\00b\07c\08d\0e\0").to_string(),
            |                         ^^^^^^^^^^^^^^^^^^^^
            |
            = help: octal escapes are not supported, `\0` is always a null character
            = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#octal_escapes
            = note: `-D clippy::octal-escapes` implied by `-D clippy::all`
        help: if an octal escape was intended, use the hexadecimal representation instead
            |
        118 |         Literal::string("a\x00b\x07c\08d\0e\0").to_string(),
            |                         ~~~~~~~~~~~~~~~~~~~~~~
        help: if the null character is intended, disambiguate using
            |
        118 |         Literal::string("a\x000b\x007c\08d\0e\0").to_string(),
            |                         ~~~~~~~~~~~~~~~~~~~~~~~~
    
        error: octal-looking escape in byte string literal
           --> tests/test.rs:155:30
            |
        155 |         Literal::byte_string(b"a\00b\07c\08d\0e\0").to_string(),
            |                              ^^^^^^^^^^^^^^^^^^^^^
            |
            = help: octal escapes are not supported, `\0` is always a null byte
            = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#octal_escapes
        help: if an octal escape was intended, use the hexadecimal representation instead
            |
        155 |         Literal::byte_string(b"a\x00b\x07c\08d\0e\0").to_string(),
            |                              ~~~~~~~~~~~~~~~~~~~~~~~
        help: if the null byte is intended, disambiguate using
            |
        155 |         Literal::byte_string(b"a\x000b\x007c\08d\0e\0").to_string(),
            |                              ~~~~~~~~~~~~~~~~~~~~~~~~~
    dtolnay committed Apr 3, 2023
    Copy the full SHA
    5d3e58b View commit details
  4. Copy the full SHA
    6c21627 View commit details
  5. Merge pull request #380 from dtolnay/octalescape

    Circumvent clippy::octal_escapes lint in generated literals
    dtolnay authored Apr 3, 2023
    Copy the full SHA
    9c092a3 View commit details
  6. Copy the full SHA
    7efc1c4 View commit details
  7. Merge pull request #381 from dtolnay/escape0

    Implement consistent behavior for Literal::string on all versions of Rust
    dtolnay authored Apr 3, 2023
    Copy the full SHA
    e82b074 View commit details
  8. Ensure data structures are RefUnwindSafe on all versions of Rust

    Previously it was not the case on versions from 1.32 through 1.57.
    
        error[E0277]: the type `UnsafeCell<usize>` may contain interior mutability and a reference may not be safely transferrable across a catch_unwind boundary
          --> tests/marker.rs:13:13
           |
        13 |               assert_implemented::<$ty>();
           |               ^^^^^^^^^^^^^^^^^^^^^^^^^ `UnsafeCell<usize>` may contain interior mutability and a reference may not be safely transferrable across a catch_unwind boundary
        ...
        82 | /     assert_unwind_safe! {
        83 | |         Delimiter
        84 | |         Group
        85 | |         Ident
        ...  |
        92 | |         TokenTree
        93 | |     }
           | |_____- in this macro invocation
           |
           = help: within `proc_macro2::Group`, the trait `RefUnwindSafe` is not implemented for `UnsafeCell<usize>`
           = note: required because it appears within the type `Cell<usize>`
           = note: required because it appears within the type `rc::RcBox<Vec<TokenTree>>`
           = note: required because it appears within the type `PhantomData<rc::RcBox<Vec<TokenTree>>>`
           = note: required because it appears within the type `Rc<Vec<TokenTree>>`
           = note: required because it appears within the type `proc_macro2::rcvec::RcVec<TokenTree>`
           = note: required because it appears within the type `fallback::TokenStream`
           = note: required because it appears within the type `fallback::Group`
           = note: required because it appears within the type `proc_macro2::imp::Group`
           = note: required because it appears within the type `proc_macro2::Group`
        note: required by a bound in `unwind_safe::Group::assert_implemented`
          --> tests/marker.rs:12:40
           |
        12 |               fn assert_implemented<T: $($marker +)+>() {}
           |                                          ^^^^^^^ required by this bound in `unwind_safe::Group::assert_implemented`
        ...
        82 | /     assert_unwind_safe! {
        83 | |         Delimiter
        84 | |         Group
        85 | |         Ident
        ...  |
        92 | |         TokenTree
        93 | |     }
           | |_____- in this macro invocation
           = note: this error originates in the macro `assert_impl` (in Nightly builds, run with -Z macro-backtrace for more info)
    
        error[E0277]: the type `UnsafeCell<usize>` may contain interior mutability and a reference may not be safely transferrable across a catch_unwind boundary
          --> tests/marker.rs:13:13
           |
        13 |               assert_implemented::<$ty>();
           |               ^^^^^^^^^^^^^^^^^^^^^^^^^ `UnsafeCell<usize>` may contain interior mutability and a reference may not be safely transferrable across a catch_unwind boundary
        ...
        82 | /     assert_unwind_safe! {
        83 | |         Delimiter
        84 | |         Group
        85 | |         Ident
        ...  |
        92 | |         TokenTree
        93 | |     }
           | |_____- in this macro invocation
           |
           = help: within `TokenStream`, the trait `RefUnwindSafe` is not implemented for `UnsafeCell<usize>`
           = note: required because it appears within the type `Cell<usize>`
           = note: required because it appears within the type `rc::RcBox<Vec<TokenTree>>`
           = note: required because it appears within the type `PhantomData<rc::RcBox<Vec<TokenTree>>>`
           = note: required because it appears within the type `Rc<Vec<TokenTree>>`
           = note: required because it appears within the type `proc_macro2::rcvec::RcVec<TokenTree>`
           = note: required because it appears within the type `fallback::TokenStream`
           = note: required because it appears within the type `proc_macro2::imp::TokenStream`
           = note: required because it appears within the type `TokenStream`
        note: required by a bound in `unwind_safe::TokenStream::assert_implemented`
          --> tests/marker.rs:12:40
           |
        12 |               fn assert_implemented<T: $($marker +)+>() {}
           |                                          ^^^^^^^ required by this bound in `unwind_safe::TokenStream::assert_implemented`
        ...
        82 | /     assert_unwind_safe! {
        83 | |         Delimiter
        84 | |         Group
        85 | |         Ident
        ...  |
        92 | |         TokenTree
        93 | |     }
           | |_____- in this macro invocation
           = note: this error originates in the macro `assert_impl` (in Nightly builds, run with -Z macro-backtrace for more info)
    
        error[E0277]: the type `UnsafeCell<usize>` may contain interior mutability and a reference may not be safely transferrable across a catch_unwind boundary
          --> tests/marker.rs:13:13
           |
        13 |               assert_implemented::<$ty>();
           |               ^^^^^^^^^^^^^^^^^^^^^^^^^ `UnsafeCell<usize>` may contain interior mutability and a reference may not be safely transferrable across a catch_unwind boundary
        ...
        82 | /     assert_unwind_safe! {
        83 | |         Delimiter
        84 | |         Group
        85 | |         Ident
        ...  |
        92 | |         TokenTree
        93 | |     }
           | |_____- in this macro invocation
           |
           = help: within `TokenTree`, the trait `RefUnwindSafe` is not implemented for `UnsafeCell<usize>`
           = note: required because it appears within the type `Cell<usize>`
           = note: required because it appears within the type `rc::RcBox<Vec<TokenTree>>`
           = note: required because it appears within the type `PhantomData<rc::RcBox<Vec<TokenTree>>>`
           = note: required because it appears within the type `Rc<Vec<TokenTree>>`
           = note: required because it appears within the type `proc_macro2::rcvec::RcVec<TokenTree>`
           = note: required because it appears within the type `fallback::TokenStream`
           = note: required because it appears within the type `fallback::Group`
           = note: required because it appears within the type `proc_macro2::imp::Group`
           = note: required because it appears within the type `proc_macro2::Group`
           = note: required because it appears within the type `TokenTree`
        note: required by a bound in `unwind_safe::TokenTree::assert_implemented`
          --> tests/marker.rs:12:40
           |
        12 |               fn assert_implemented<T: $($marker +)+>() {}
           |                                          ^^^^^^^ required by this bound in `unwind_safe::TokenTree::assert_implemented`
        ...
        82 | /     assert_unwind_safe! {
        83 | |         Delimiter
        84 | |         Group
        85 | |         Ident
        ...  |
        92 | |         TokenTree
        93 | |     }
           | |_____- in this macro invocation
           = note: this error originates in the macro `assert_impl` (in Nightly builds, run with -Z macro-backtrace for more info)
    dtolnay committed Apr 3, 2023
    Copy the full SHA
    cda93c5 View commit details
  9. Merge pull request #382 from dtolnay/refunwindsafe

    Ensure data structures are RefUnwindSafe on all versions of Rust
    dtolnay authored Apr 3, 2023
    Copy the full SHA
    ddf7692 View commit details
  10. Release 1.0.56

    dtolnay committed Apr 3, 2023
    Copy the full SHA
    72ee0b3 View commit details

Commits on May 5, 2023

  1. Copy the full SHA
    2b25bd9 View commit details
  2. Find out whether rust-src rustup component fixes ui test

    `cargo test --manifest-path tests/ui/Cargo.toml` is passing locally but
    fails in GitHub Actions.
    dtolnay committed May 5, 2023
    Copy the full SHA
    37706ca View commit details
  3. Copy the full SHA
    c282649 View commit details
  4. Copy the full SHA
    2d3959c View commit details

Commits on May 7, 2023

  1. Ignore new_without_default clippy lint

        error: you should consider adding a `Default` implementation for `TokenStream`
          --> src/fallback.rs:55:5
           |
        55 | /     pub fn new() -> Self {
        56 | |         TokenStream {
        57 | |             inner: RcVecBuilder::new().build(),
        58 | |         }
        59 | |     }
           | |_____^
           |
           = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#new_without_default
           = note: `-D clippy::new-without-default` implied by `-D clippy::all`
        help: try adding this
           |
        54 + impl Default for TokenStream {
        55 +     fn default() -> Self {
        56 +         Self::new()
        57 +     }
        58 + }
           |
    
        error: you should consider adding a `Default` implementation for `TokenStream`
          --> src/wrapper.rs:74:5
           |
        74 | /     pub fn new() -> Self {
        75 | |         if inside_proc_macro() {
        76 | |             TokenStream::Compiler(DeferredTokenStream::new(proc_macro::TokenStream::new()))
        77 | |         } else {
        78 | |             TokenStream::Fallback(fallback::TokenStream::new())
        79 | |         }
        80 | |     }
           | |_____^
           |
           = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#new_without_default
        help: try adding this
           |
        73 + impl Default for TokenStream {
        74 +     fn default() -> Self {
        75 +         Self::new()
        76 +     }
        77 + }
           |
    dtolnay committed May 7, 2023
    Copy the full SHA
    7c637e7 View commit details
  2. Revert "Temporarily disable miri CI"

    This reverts commit 2d3959c.
    
    Fixed in nightly-2023-05-06.
    dtolnay committed May 7, 2023
    Copy the full SHA
    2130305 View commit details

Commits on May 14, 2023

  1. Copy the full SHA
    a597b3c View commit details

Commits on May 15, 2023

  1. Copy the full SHA
    5f9d3fe View commit details
  2. Merge pull request #383 from dtolnay/procmacrospan

    Disable start()/end() on Span::Compiler
    dtolnay authored May 15, 2023
    Copy the full SHA
    8bbbd60 View commit details
  3. Release 1.0.57

    dtolnay committed May 15, 2023
    Copy the full SHA
    2c1b102 View commit details

Commits on May 17, 2023

  1. Copy the full SHA
    a8db10f View commit details
Showing with 2,044 additions and 1,052 deletions.
  1. +0 −1 .clippy.toml
  2. +108 −12 .github/workflows/ci.yml
  3. +2 −3 .gitignore
  4. +16 −9 Cargo.toml
  5. +2 −2 README.md
  6. +180 −128 build.rs
  7. +41 −0 build/probe.rs
  8. +9 −3 fuzz/.gitignore
  9. +11 −2 fuzz/Cargo.toml
  10. +63 −8 fuzz/fuzz_targets/parse_token_stream.rs
  11. +2 −0 rust-toolchain.toml
  12. +73 −22 src/extra.rs
  13. +495 −242 src/fallback.rs
  14. +167 −139 src/lib.rs
  15. +1 −1 src/location.rs
  16. +9 −10 src/marker.rs
  17. +224 −108 src/parse.rs
  18. +25 −21 src/rcvec.rs
  19. +222 −234 src/wrapper.rs
  20. +2 −0 tests/features.rs
  21. +4 −7 tests/marker.rs
  22. +307 −78 tests/test.rs
  23. +1 −1 tests/test_fmt.rs
  24. +53 −14 tests/test_size.rs
  25. +2 −2 tests/ui/compiletest.rs
  26. +25 −5 tests/ui/test-not-send.stderr
1 change: 0 additions & 1 deletion .clippy.toml

This file was deleted.

120 changes: 108 additions & 12 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -24,13 +24,14 @@ jobs:
strategy:
fail-fast: false
matrix:
rust: [1.31.0, stable, beta]
rust: [1.67.0, stable, beta]
timeout-minutes: 45
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@master
with:
toolchain: ${{matrix.rust}}
components: rust-src
- run: cargo test
- run: cargo test --no-default-features
- run: cargo test --features span-locations
@@ -50,10 +51,15 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
with:
components: rust-src
- name: Enable type layout randomization
run: echo RUSTFLAGS=${RUSTFLAGS}\ -Zrandomize-layout >> $GITHUB_ENV
run: echo RUSTFLAGS=${RUSTFLAGS}\ -Zrandomize-layout\ --cfg=randomize_layout >> $GITHUB_ENV
- run: cargo check
env:
RUSTFLAGS: --cfg procmacro2_nightly_testing ${{env.RUSTFLAGS}}
- run: cargo test
- run: cargo test --no-default-features
- run: cargo test --no-default-features --test features -- --ignored make_sure_no_proc_macro # run the ignored test to make sure the `proc-macro` feature is disabled
@@ -70,8 +76,64 @@ jobs:
- name: RUSTFLAGS='-Z allow-features=' cargo test
run: cargo test
env:
RUSTFLAGS: -Z allow-features= ${{env.RUSTFLAGS}}
- run: cargo update -Z minimal-versions && cargo build
RUSTFLAGS: -Z allow-features= --cfg procmacro2_backtrace ${{env.RUSTFLAGS}}
- uses: actions/upload-artifact@v4
if: always()
with:
name: Cargo.lock
path: Cargo.lock
continue-on-error: true

layout:
name: Layout
needs: pre_ci
if: needs.pre_ci.outputs.continue
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
with:
components: rust-src
- run: cargo test --test test_size
- run: cargo test --test test_size --features span-locations
- run: cargo test --test test_size --no-default-features
- run: cargo test --test test_size --no-default-features --features span-locations

msrv:
name: Rust 1.56.0
needs: pre_ci
if: needs.pre_ci.outputs.continue
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@1.56.0
with:
components: rust-src
- run: cargo check
- run: cargo check --no-default-features
- run: cargo check --features span-locations
- name: RUSTFLAGS='--cfg procmacro2_semver_exempt' cargo check
run: cargo check
env:
RUSTFLAGS: --cfg procmacro2_semver_exempt ${{env.RUSTFLAGS}}
- name: RUSTFLAGS='--cfg procmacro2_semver_exempt' cargo check --no-default-features
run: cargo check --no-default-features
env:
RUSTFLAGS: --cfg procmacro2_semver_exempt ${{env.RUSTFLAGS}}

minimal:
name: Minimal versions
needs: pre_ci
if: needs.pre_ci.outputs.continue
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
- run: cargo generate-lockfile -Z minimal-versions
- run: cargo check --locked

webassembly:
name: WebAssembly
@@ -80,10 +142,13 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
with:
target: wasm32-unknown-unknown
components: rust-src
- name: Ignore WebAssembly linker warning
run: echo RUSTFLAGS=${RUSTFLAGS}\ -Alinker_messages >> $GITHUB_ENV
- run: cargo test --target wasm32-unknown-unknown --no-run

fuzz:
@@ -93,19 +158,46 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
with:
components: rust-src
- uses: dtolnay/install@cargo-fuzz
- run: cargo fuzz check
- run: cargo check --no-default-features --features afl
working-directory: fuzz
- uses: dtolnay/install@honggfuzz
- run: sudo apt-get update # https://github.com/actions/runner-images/issues/8953
- run: sudo apt-get install binutils-dev libunwind-dev
- run: cargo hfuzz build --no-default-features --features honggfuzz
working-directory: fuzz

doc:
name: Documentation
needs: pre_ci
if: needs.pre_ci.outputs.continue
runs-on: ubuntu-latest
timeout-minutes: 45
env:
RUSTDOCFLAGS: -Dwarnings
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
with:
components: rust-src
- uses: dtolnay/install@cargo-docs-rs
- run: cargo docs-rs

clippy:
name: Clippy
runs-on: ubuntu-latest
if: github.event_name != 'pull_request'
timeout-minutes: 45
steps:
- uses: actions/checkout@v3
- uses: dtolnay/rust-toolchain@clippy
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
with:
components: clippy, rust-src
- run: cargo clippy --tests -- -Dclippy::all -Dclippy::pedantic
- run: cargo clippy --tests --all-features -- -Dclippy::all -Dclippy::pedantic

@@ -116,8 +208,11 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@miri
with:
toolchain: nightly-2025-05-16 # https://github.com/rust-lang/miri/issues/4323
- run: cargo miri setup
- run: cargo miri test
env:
MIRIFLAGS: -Zmiri-strict-provenance
@@ -128,7 +223,8 @@ jobs:
if: github.event_name != 'pull_request'
timeout-minutes: 45
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: dtolnay/install@cargo-outdated
- run: cargo outdated --workspace --exit-code 1
- run: cargo outdated --manifest-path fuzz/Cargo.toml --exit-code 1
5 changes: 2 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
/target
**/*.rs.bk
Cargo.lock
/target/
/Cargo.lock
25 changes: 16 additions & 9 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,21 +1,28 @@
[package]
name = "proc-macro2"
version = "1.0.52" # remember to update html_root_url
version = "1.0.95"
authors = ["David Tolnay <dtolnay@gmail.com>", "Alex Crichton <alex@alexcrichton.com>"]
autobenches = false
categories = ["development-tools::procedural-macro-helpers"]
description = "A substitute implementation of the compiler's `proc_macro` API to decouple token-based libraries from the procedural macro use case."
documentation = "https://docs.rs/proc-macro2"
edition = "2018"
edition = "2021"
keywords = ["macros", "syn"]
license = "MIT OR Apache-2.0"
repository = "https://github.com/dtolnay/proc-macro2"
rust-version = "1.31"
rust-version = "1.56"

[package.metadata.docs.rs]
rustc-args = ["--cfg", "procmacro2_semver_exempt"]
rustdoc-args = ["--cfg", "procmacro2_semver_exempt", "--cfg", "doc_cfg"]
rustc-args = ["--cfg=procmacro2_semver_exempt"]
targets = ["x86_64-unknown-linux-gnu"]
rustdoc-args = [
"--cfg=procmacro2_semver_exempt",
"--generate-link-to-definition",
"--extern-html-root-url=core=https://doc.rust-lang.org",
"--extern-html-root-url=alloc=https://doc.rust-lang.org",
"--extern-html-root-url=std=https://doc.rust-lang.org",
"--extern-html-root-url=proc_macro=https://doc.rust-lang.org",
]

[package.metadata.playground]
features = ["span-locations"]
@@ -24,8 +31,11 @@ features = ["span-locations"]
unicode-ident = "1.0"

[dev-dependencies]
quote = { version = "1.0", default_features = false }
flate2 = "1.0"
quote = { version = "1.0", default-features = false }
rayon = "1.0"
rustversion = "1"
tar = "0.4"

[features]
proc-macro = []
@@ -38,9 +48,6 @@ span-locations = []
# This feature no longer means anything.
nightly = []

[lib]
doc-scrape-examples = false

[workspace]
members = ["benches/bench-libproc-macro", "tests/ui"]

4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -52,7 +52,7 @@ pub fn my_derive(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
If parsing with [Syn], you'll use [`parse_macro_input!`] instead to propagate
parse errors correctly back to the compiler when parsing fails.

[`parse_macro_input!`]: https://docs.rs/syn/1.0/syn/macro.parse_macro_input.html
[`parse_macro_input!`]: https://docs.rs/syn/2.0/syn/macro.parse_macro_input.html

## Unstable features

@@ -62,7 +62,7 @@ proc-macro2 by default.

To opt into the additional APIs available in the most recent nightly compiler,
the `procmacro2_semver_exempt` config flag must be passed to rustc. We will
polyfill those nightly-only APIs back to Rust 1.31.0. As these are unstable APIs
polyfill those nightly-only APIs back to Rust 1.56.0. As these are unstable APIs
that track the nightly compiler, minor versions of proc-macro2 may make breaking
changes to them at any time.

308 changes: 180 additions & 128 deletions build.rs
Original file line number Diff line number Diff line change
@@ -1,60 +1,32 @@
// rustc-cfg emitted by the build script:
//
// "use_proc_macro"
// Link to extern crate proc_macro. Available on any compiler and any target
// except wasm32. Requires "proc-macro" Cargo cfg to be enabled (default is
// enabled). On wasm32 we never link to proc_macro even if "proc-macro" cfg
// is enabled.
//
// "wrap_proc_macro"
// Wrap types from libproc_macro rather than polyfilling the whole API.
// Enabled on rustc 1.29+ as long as procmacro2_semver_exempt is not set,
// because we can't emulate the unstable API without emulating everything
// else. Also enabled unconditionally on nightly, in which case the
// procmacro2_semver_exempt surface area is implemented by using the
// nightly-only proc_macro API.
//
// "hygiene"
// Enable Span::mixed_site() and non-dummy behavior of Span::resolved_at
// and Span::located_at. Enabled on Rust 1.45+.
//
// "proc_macro_span"
// Enable non-dummy behavior of Span::start and Span::end methods which
// requires an unstable compiler feature. Enabled when building with
// nightly, unless `-Z allow-feature` in RUSTFLAGS disallows unstable
// features.
//
// "super_unstable"
// Implement the semver exempt API in terms of the nightly-only proc_macro
// API. Enabled when using procmacro2_semver_exempt on a nightly compiler.
//
// "span_locations"
// Provide methods Span::start and Span::end which give the line/column
// location of a token. Enabled by procmacro2_semver_exempt or the
// "span-locations" Cargo cfg. This is behind a cfg because tracking
// location inside spans is a performance hit.
//
// "is_available"
// Use proc_macro::is_available() to detect if the proc macro API is
// available or needs to be polyfilled instead of trying to use the proc
// macro API and catching a panic if it isn't available. Enabled on Rust
// 1.57+.
#![allow(unknown_lints)]
#![allow(unexpected_cfgs)]

use std::env;
use std::process::{self, Command};
use std::ffi::OsString;
use std::fs;
use std::io::ErrorKind;
use std::iter;
use std::path::Path;
use std::process::{self, Command, Stdio};
use std::str;

fn main() {
println!("cargo:rerun-if-changed=build.rs");

let version = match rustc_version() {
Some(version) => version,
None => return,
};

if version.minor < 31 {
eprintln!("Minimum supported rustc version is 1.31");
process::exit(1);
let rustc = rustc_minor_version().unwrap_or(u32::MAX);

if rustc >= 80 {
println!("cargo:rustc-check-cfg=cfg(fuzzing)");
println!("cargo:rustc-check-cfg=cfg(no_is_available)");
println!("cargo:rustc-check-cfg=cfg(no_literal_byte_character)");
println!("cargo:rustc-check-cfg=cfg(no_literal_c_string)");
println!("cargo:rustc-check-cfg=cfg(no_source_text)");
println!("cargo:rustc-check-cfg=cfg(proc_macro_span)");
println!("cargo:rustc-check-cfg=cfg(procmacro2_backtrace)");
println!("cargo:rustc-check-cfg=cfg(procmacro2_nightly_testing)");
println!("cargo:rustc-check-cfg=cfg(procmacro2_semver_exempt)");
println!("cargo:rustc-check-cfg=cfg(randomize_layout)");
println!("cargo:rustc-check-cfg=cfg(span_locations)");
println!("cargo:rustc-check-cfg=cfg(super_unstable)");
println!("cargo:rustc-check-cfg=cfg(wrap_proc_macro)");
}

let docs_rs = env::var_os("DOCS_RS").is_some();
@@ -65,119 +37,199 @@ fn main() {
}

if semver_exempt || cfg!(feature = "span-locations") {
// Provide methods Span::start and Span::end which give the line/column
// location of a token. This is behind a cfg because tracking location
// inside spans is a performance hit.
println!("cargo:rustc-cfg=span_locations");
}

if version.minor < 32 {
println!("cargo:rustc-cfg=no_libprocmacro_unwind_safe");
if rustc < 57 {
// Do not use proc_macro::is_available() to detect whether the proc
// macro API is available vs needs to be polyfilled. Instead, use the
// proc macro API unconditionally and catch the panic that occurs if it
// isn't available.
println!("cargo:rustc-cfg=no_is_available");
}

if version.minor < 39 {
println!("cargo:rustc-cfg=no_bind_by_move_pattern_guard");
if rustc < 66 {
// Do not call libproc_macro's Span::source_text. Always return None.
println!("cargo:rustc-cfg=no_source_text");
}

if version.minor < 44 {
println!("cargo:rustc-cfg=no_lexerror_display");
if rustc < 79 {
// Do not call Literal::byte_character nor Literal::c_string. They can
// be emulated by way of Literal::from_str.
println!("cargo:rustc-cfg=no_literal_byte_character");
println!("cargo:rustc-cfg=no_literal_c_string");
}

if version.minor < 45 {
println!("cargo:rustc-cfg=no_hygiene");
if !cfg!(feature = "proc-macro") {
println!("cargo:rerun-if-changed=build.rs");
return;
}

if version.minor < 47 {
println!("cargo:rustc-cfg=no_ident_new_raw");
println!("cargo:rerun-if-changed=build/probe.rs");

let proc_macro_span;
let consider_rustc_bootstrap;
if compile_probe(false) {
// This is a nightly or dev compiler, so it supports unstable features
// regardless of RUSTC_BOOTSTRAP. No need to rerun build script if
// RUSTC_BOOTSTRAP is changed.
proc_macro_span = true;
consider_rustc_bootstrap = false;
} else if let Some(rustc_bootstrap) = env::var_os("RUSTC_BOOTSTRAP") {
if compile_probe(true) {
// This is a stable or beta compiler for which the user has set
// RUSTC_BOOTSTRAP to turn on unstable features. Rerun build script
// if they change it.
proc_macro_span = true;
consider_rustc_bootstrap = true;
} else if rustc_bootstrap == "1" {
// This compiler does not support the proc macro Span API in the
// form that proc-macro2 expects. No need to pay attention to
// RUSTC_BOOTSTRAP.
proc_macro_span = false;
consider_rustc_bootstrap = false;
} else {
// This is a stable or beta compiler for which RUSTC_BOOTSTRAP is
// set to restrict the use of unstable features by this crate.
proc_macro_span = false;
consider_rustc_bootstrap = true;
}
} else {
// Without RUSTC_BOOTSTRAP, this compiler does not support the proc
// macro Span API in the form that proc-macro2 expects, but try again if
// the user turns on unstable features.
proc_macro_span = false;
consider_rustc_bootstrap = true;
}

if proc_macro_span || !semver_exempt {
// Wrap types from libproc_macro rather than polyfilling the whole API.
// Enabled as long as procmacro2_semver_exempt is not set, because we
// can't emulate the unstable API without emulating everything else.
// Also enabled unconditionally on nightly, in which case the
// procmacro2_semver_exempt surface area is implemented by using the
// nightly-only proc_macro API.
println!("cargo:rustc-cfg=wrap_proc_macro");
}

if version.minor < 54 {
println!("cargo:rustc-cfg=no_literal_from_str");
if proc_macro_span {
// Enable non-dummy behavior of Span::start and Span::end methods which
// requires an unstable compiler feature. Enabled when building with
// nightly, unless `-Z allow-feature` in RUSTFLAGS disallows unstable
// features.
println!("cargo:rustc-cfg=proc_macro_span");
}

if version.minor < 55 {
println!("cargo:rustc-cfg=no_group_open_close");
if semver_exempt && proc_macro_span {
// Implement the semver exempt API in terms of the nightly-only
// proc_macro API.
println!("cargo:rustc-cfg=super_unstable");
}

if version.minor < 57 {
println!("cargo:rustc-cfg=no_is_available");
if consider_rustc_bootstrap {
println!("cargo:rerun-if-env-changed=RUSTC_BOOTSTRAP");
}
}

let target = env::var("TARGET").unwrap();
if !enable_use_proc_macro(&target) {
return;
fn compile_probe(rustc_bootstrap: bool) -> bool {
if env::var_os("RUSTC_STAGE").is_some() {
// We are running inside rustc bootstrap. This is a highly non-standard
// environment with issues such as:
//
// https://github.com/rust-lang/cargo/issues/11138
// https://github.com/rust-lang/rust/issues/114839
//
// Let's just not use nightly features here.
return false;
}

println!("cargo:rustc-cfg=use_proc_macro");
let rustc = cargo_env_var("RUSTC");
let out_dir = cargo_env_var("OUT_DIR");
let out_subdir = Path::new(&out_dir).join("probe");
let probefile = Path::new("build").join("probe.rs");

if version.nightly || !semver_exempt {
println!("cargo:rustc-cfg=wrap_proc_macro");
if let Err(err) = fs::create_dir(&out_subdir) {
if err.kind() != ErrorKind::AlreadyExists {
eprintln!("Failed to create {}: {}", out_subdir.display(), err);
process::exit(1);
}
}

if version.nightly
&& feature_allowed("proc_macro_span")
&& feature_allowed("proc_macro_span_shrink")
{
println!("cargo:rustc-cfg=proc_macro_span");
let rustc_wrapper = env::var_os("RUSTC_WRAPPER").filter(|wrapper| !wrapper.is_empty());
let rustc_workspace_wrapper =
env::var_os("RUSTC_WORKSPACE_WRAPPER").filter(|wrapper| !wrapper.is_empty());
let mut rustc = rustc_wrapper
.into_iter()
.chain(rustc_workspace_wrapper)
.chain(iter::once(rustc));
let mut cmd = Command::new(rustc.next().unwrap());
cmd.args(rustc);

if !rustc_bootstrap {
cmd.env_remove("RUSTC_BOOTSTRAP");
}

cmd.stderr(Stdio::null())
.arg("--edition=2021")
.arg("--crate-name=proc_macro2")
.arg("--crate-type=lib")
.arg("--cap-lints=allow")
.arg("--emit=dep-info,metadata")
.arg("--out-dir")
.arg(&out_subdir)
.arg(probefile);

if let Some(target) = env::var_os("TARGET") {
cmd.arg("--target").arg(target);
}

// If Cargo wants to set RUSTFLAGS, use that.
if let Ok(rustflags) = env::var("CARGO_ENCODED_RUSTFLAGS") {
if !rustflags.is_empty() {
for arg in rustflags.split('\x1f') {
cmd.arg(arg);
}
}
}

if semver_exempt && version.nightly {
println!("cargo:rustc-cfg=super_unstable");
}
}
let success = match cmd.status() {
Ok(status) => status.success(),
Err(_) => false,
};

fn enable_use_proc_macro(target: &str) -> bool {
// wasm targets don't have the `proc_macro` crate, disable this feature.
if target.contains("wasm32") {
return false;
// Clean up to avoid leaving nondeterministic absolute paths in the dep-info
// file in OUT_DIR, which causes nonreproducible builds in build systems
// that treat the entire OUT_DIR as an artifact.
if let Err(err) = fs::remove_dir_all(&out_subdir) {
if err.kind() != ErrorKind::NotFound {
eprintln!("Failed to clean up {}: {}", out_subdir.display(), err);
process::exit(1);
}
}

// Otherwise, only enable it if our feature is actually enabled.
cfg!(feature = "proc-macro")
}

struct RustcVersion {
minor: u32,
nightly: bool,
success
}

fn rustc_version() -> Option<RustcVersion> {
let rustc = env::var_os("RUSTC")?;
fn rustc_minor_version() -> Option<u32> {
let rustc = cargo_env_var("RUSTC");
let output = Command::new(rustc).arg("--version").output().ok()?;
let version = str::from_utf8(&output.stdout).ok()?;
let nightly = version.contains("nightly") || version.contains("dev");
let mut pieces = version.split('.');
if pieces.next() != Some("rustc 1") {
return None;
}
let minor = pieces.next()?.parse().ok()?;
Some(RustcVersion { minor, nightly })
pieces.next()?.parse().ok()
}

fn feature_allowed(feature: &str) -> bool {
// Recognized formats:
//
// -Z allow-features=feature1,feature2
//
// -Zallow-features=feature1,feature2

let flags_var;
let flags_var_string;
let flags = if let Some(encoded_rustflags) = env::var_os("CARGO_ENCODED_RUSTFLAGS") {
flags_var = encoded_rustflags;
flags_var_string = flags_var.to_string_lossy();
flags_var_string.split('\x1f')
} else {
return true;
};

for mut flag in flags {
if flag.starts_with("-Z") {
flag = &flag["-Z".len()..];
}
if flag.starts_with("allow-features=") {
flag = &flag["allow-features=".len()..];
return flag.split(',').any(|allowed| allowed == feature);
}
}

// No allow-features= flag, allowed by default.
true
fn cargo_env_var(key: &str) -> OsString {
env::var_os(key).unwrap_or_else(|| {
eprintln!(
"Environment variable ${} is not set during execution of build script",
key,
);
process::exit(1);
})
}
41 changes: 41 additions & 0 deletions build/probe.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// This code exercises the surface area that we expect of Span's unstable API.
// If the current toolchain is able to compile it, then proc-macro2 is able to
// offer these APIs too.

#![feature(proc_macro_span)]

extern crate proc_macro;

use core::ops::{Range, RangeBounds};
use proc_macro::{Literal, Span};

pub fn byte_range(this: &Span) -> Range<usize> {
this.byte_range()
}

pub fn start(this: &Span) -> Span {
this.start()
}

pub fn end(this: &Span) -> Span {
this.end()
}

pub fn line(this: &Span) -> usize {
this.line()
}

pub fn column(this: &Span) -> usize {
this.column()
}

pub fn join(this: &Span, other: Span) -> Option<Span> {
this.join(other)
}

pub fn subspan<R: RangeBounds<usize>>(this: &Literal, range: R) -> Option<Span> {
this.subspan(range)
}

// Include in sccache cache key.
const _: Option<&str> = option_env!("RUSTC_BOOTSTRAP");
12 changes: 9 additions & 3 deletions fuzz/.gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
artifacts/
corpus/
target/
/artifacts/
/corpus/
/coverage/
/hfuzz_target/
/hfuzz_workspace/
/in/
/out/
/target/
/Cargo.lock
13 changes: 11 additions & 2 deletions fuzz/Cargo.toml
Original file line number Diff line number Diff line change
@@ -9,8 +9,17 @@ publish = false
cargo-fuzz = true

[dependencies]
libfuzzer-sys = "0.4"
proc-macro2 = { path = ".." }
afl = { version = "0.15", optional = true }
honggfuzz = { version = "0.5", optional = true }
libfuzzer-sys = { version = "0.4.7", optional = true }
proc-macro2 = { path = "..", default-features = false }

[features]
default = ["libfuzzer"]
afl = ["dep:afl"]
honggfuzz = ["dep:honggfuzz"]
libfuzzer = ["dep:libfuzzer-sys"]
span-locations = ["proc-macro2/span-locations"]

[[bin]]
name = "parse_token_stream"
71 changes: 63 additions & 8 deletions fuzz/fuzz_targets/parse_token_stream.rs
Original file line number Diff line number Diff line change
@@ -1,12 +1,67 @@
#![no_main]
// libfuzzer:
//
// cargo install cargo-fuzz
// cargo fuzz run parse_token_stream -j $(nproc) -- -max_len=200 -timeout=1
//
// afl++:
//
// cargo install cargo-afl
// cargo afl build --no-default-features --features afl --release
// cargo afl fuzz -i in -o out target/release/parse_token_stream
//
// honggfuzz:
//
// cargo install honggfuzz
// cargo hfuzz build --no-default-features --features honggfuzz
// HFUZZ_RUN_ARGS="--threads $(nproc) --max_file_size 200 --timeout 1" cargo hfuzz run parse_token_stream

#![cfg_attr(feature = "libfuzzer", no_main)]

use libfuzzer_sys::fuzz_target;
use std::str;

fuzz_target!(|bytes: &[u8]| {
if bytes.len() < 200 {
if let Ok(string) = str::from_utf8(bytes) {
_ = string.parse::<proc_macro2::TokenStream>();
}
#[cfg(not(any(
all(
feature = "libfuzzer",
not(feature = "afl"),
not(feature = "honggfuzz")
),
all(
not(feature = "libfuzzer"),
feature = "afl",
not(feature = "honggfuzz")
),
all(
not(feature = "libfuzzer"),
not(feature = "afl"),
feature = "honggfuzz"
),
)))]
fn main() {
compile_error! {
r#"exactly one of feature="libfuzzer" or feature="afl" or feature="honggfuzz" must be enabled"#
}
}

#[cfg(feature = "libfuzzer")]
libfuzzer_sys::fuzz_target!(|bytes: &[u8]| do_fuzz(bytes));

#[cfg(feature = "afl")]
fn main() {
let hook = true; // turn panic into crashes
afl::fuzz(hook, do_fuzz);
}

#[cfg(feature = "honggfuzz")]
fn main() {
loop {
honggfuzz::fuzz(do_fuzz);
}
});
}

fn do_fuzz(bytes: &[u8]) {
let ..=199 = bytes.len() else { return };
let Ok(string) = str::from_utf8(bytes) else {
return;
};
let _ = string.parse::<proc_macro2::TokenStream>();
}
2 changes: 2 additions & 0 deletions rust-toolchain.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[toolchain]
components = ["rust-src"]
95 changes: 73 additions & 22 deletions src/extra.rs
Original file line number Diff line number Diff line change
@@ -3,28 +3,93 @@
use crate::fallback;
use crate::imp;
use crate::marker::Marker;
use crate::marker::{ProcMacroAutoTraits, MARKER};
use crate::Span;
use core::fmt::{self, Debug};

/// Invalidate any `proc_macro2::Span` that exist on the current thread.
///
/// The implementation of `Span` uses thread-local data structures and this
/// function clears them. Calling any method on a `Span` on the current thread
/// created prior to the invalidation will return incorrect values or crash.
///
/// This function is useful for programs that process more than 2<sup>32</sup>
/// bytes of Rust source code on the same thread. Just like rustc, proc-macro2
/// uses 32-bit source locations, and these wrap around when the total source
/// code processed by the same thread exceeds 2<sup>32</sup> bytes (4
/// gigabytes). After a wraparound, `Span` methods such as `source_text()` can
/// return wrong data.
///
/// # Example
///
/// As of late 2023, there is 200 GB of Rust code published on crates.io.
/// Looking at just the newest version of every crate, it is 16 GB of code. So a
/// workload that involves parsing it all would overflow a 32-bit source
/// location unless spans are being invalidated.
///
/// ```
/// use flate2::read::GzDecoder;
/// use std::ffi::OsStr;
/// use std::io::{BufReader, Read};
/// use std::str::FromStr;
/// use tar::Archive;
///
/// rayon::scope(|s| {
/// for krate in every_version_of_every_crate() {
/// s.spawn(move |_| {
/// proc_macro2::extra::invalidate_current_thread_spans();
///
/// let reader = BufReader::new(krate);
/// let tar = GzDecoder::new(reader);
/// let mut archive = Archive::new(tar);
/// for entry in archive.entries().unwrap() {
/// let mut entry = entry.unwrap();
/// let path = entry.path().unwrap();
/// if path.extension() != Some(OsStr::new("rs")) {
/// continue;
/// }
/// let mut content = String::new();
/// entry.read_to_string(&mut content).unwrap();
/// match proc_macro2::TokenStream::from_str(&content) {
/// Ok(tokens) => {/* ... */},
/// Err(_) => continue,
/// }
/// }
/// });
/// }
/// });
/// #
/// # fn every_version_of_every_crate() -> Vec<std::fs::File> {
/// # Vec::new()
/// # }
/// ```
///
/// # Panics
///
/// This function is not applicable to and will panic if called from a
/// procedural macro.
#[cfg(span_locations)]
#[cfg_attr(docsrs, doc(cfg(feature = "span-locations")))]
pub fn invalidate_current_thread_spans() {
crate::imp::invalidate_current_thread_spans();
}

/// An object that holds a [`Group`]'s `span_open()` and `span_close()` together
/// (in a more compact representation than holding those 2 spans individually.
/// in a more compact representation than holding those 2 spans individually.
///
/// [`Group`]: crate::Group
#[derive(Copy, Clone)]
pub struct DelimSpan {
inner: DelimSpanEnum,
_marker: Marker,
_marker: ProcMacroAutoTraits,
}

#[derive(Copy, Clone)]
enum DelimSpanEnum {
#[cfg(wrap_proc_macro)]
Compiler {
join: proc_macro::Span,
#[cfg(not(no_group_open_close))]
open: proc_macro::Span,
#[cfg(not(no_group_open_close))]
close: proc_macro::Span,
},
Fallback(fallback::Span),
@@ -36,9 +101,7 @@ impl DelimSpan {
let inner = match group {
imp::Group::Compiler(group) => DelimSpanEnum::Compiler {
join: group.span(),
#[cfg(not(no_group_open_close))]
open: group.span_open(),
#[cfg(not(no_group_open_close))]
close: group.span_close(),
},
imp::Group::Fallback(group) => DelimSpanEnum::Fallback(group.span()),
@@ -49,7 +112,7 @@ impl DelimSpan {

DelimSpan {
inner,
_marker: Marker,
_marker: MARKER,
}
}

@@ -66,13 +129,7 @@ impl DelimSpan {
pub fn open(&self) -> Span {
match &self.inner {
#[cfg(wrap_proc_macro)]
DelimSpanEnum::Compiler {
#[cfg(not(no_group_open_close))]
open,
#[cfg(no_group_open_close)]
join: open,
..
} => Span::_new(imp::Span::Compiler(*open)),
DelimSpanEnum::Compiler { open, .. } => Span::_new(imp::Span::Compiler(*open)),
DelimSpanEnum::Fallback(span) => Span::_new_fallback(span.first_byte()),
}
}
@@ -81,13 +138,7 @@ impl DelimSpan {
pub fn close(&self) -> Span {
match &self.inner {
#[cfg(wrap_proc_macro)]
DelimSpanEnum::Compiler {
#[cfg(not(no_group_open_close))]
close,
#[cfg(no_group_open_close)]
join: close,
..
} => Span::_new(imp::Span::Compiler(*close)),
DelimSpanEnum::Compiler { close, .. } => Span::_new(imp::Span::Compiler(*close)),
DelimSpanEnum::Fallback(span) => Span::_new_fallback(span.last_byte()),
}
}
737 changes: 495 additions & 242 deletions src/fallback.rs

Large diffs are not rendered by default.

306 changes: 167 additions & 139 deletions src/lib.rs

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion src/location.rs
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@ use core::cmp::Ordering;
/// A line-column pair representing the start or end of a `Span`.
///
/// This type is semver exempt and not exposed by default.
#[cfg_attr(doc_cfg, doc(cfg(feature = "span-locations")))]
#[cfg_attr(docsrs, doc(cfg(feature = "span-locations")))]
#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash)]
pub struct LineColumn {
/// The 1-indexed line in the source file on which the span starts or ends
19 changes: 9 additions & 10 deletions src/marker.rs
Original file line number Diff line number Diff line change
@@ -1,18 +1,17 @@
use alloc::rc::Rc;
use core::marker::PhantomData;
use std::panic::{RefUnwindSafe, UnwindSafe};
use std::rc::Rc;
use core::panic::{RefUnwindSafe, UnwindSafe};

// Zero sized marker with the correct set of autotrait impls we want all proc
// macro types to have.
pub(crate) type Marker = PhantomData<ProcMacroAutoTraits>;
#[derive(Copy, Clone)]
#[cfg_attr(
all(procmacro2_semver_exempt, any(not(wrap_proc_macro), super_unstable)),
derive(PartialEq, Eq)
)]
pub(crate) struct ProcMacroAutoTraits(PhantomData<Rc<()>>);

pub(crate) use self::value::*;

mod value {
pub(crate) use core::marker::PhantomData as Marker;
}

pub(crate) struct ProcMacroAutoTraits(Rc<()>);
pub(crate) const MARKER: ProcMacroAutoTraits = ProcMacroAutoTraits(PhantomData);

impl UnwindSafe for ProcMacroAutoTraits {}
impl RefUnwindSafe for ProcMacroAutoTraits {}
332 changes: 224 additions & 108 deletions src/parse.rs

Large diffs are not rendered by default.

46 changes: 25 additions & 21 deletions src/rcvec.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
use alloc::rc::Rc;
use alloc::vec;
use core::mem;
use core::panic::RefUnwindSafe;
use core::slice;
use std::rc::Rc;
use std::vec;

pub(crate) struct RcVec<T> {
inner: Rc<Vec<T>>,
@@ -21,19 +22,19 @@ pub(crate) struct RcVecIntoIter<T> {
}

impl<T> RcVec<T> {
pub fn is_empty(&self) -> bool {
pub(crate) fn is_empty(&self) -> bool {
self.inner.is_empty()
}

pub fn len(&self) -> usize {
pub(crate) fn len(&self) -> usize {
self.inner.len()
}

pub fn iter(&self) -> slice::Iter<T> {
pub(crate) fn iter(&self) -> slice::Iter<T> {
self.inner.iter()
}

pub fn make_mut(&mut self) -> RcVecMut<T>
pub(crate) fn make_mut(&mut self) -> RcVecMut<T>
where
T: Clone,
{
@@ -42,17 +43,17 @@ impl<T> RcVec<T> {
}
}

pub fn get_mut(&mut self) -> Option<RcVecMut<T>> {
pub(crate) fn get_mut(&mut self) -> Option<RcVecMut<T>> {
let inner = Rc::get_mut(&mut self.inner)?;
Some(RcVecMut { inner })
}

pub fn make_owned(mut self) -> RcVecBuilder<T>
pub(crate) fn make_owned(mut self) -> RcVecBuilder<T>
where
T: Clone,
{
let vec = if let Some(owned) = Rc::get_mut(&mut self.inner) {
mem::replace(owned, Vec::new())
mem::take(owned)
} else {
Vec::clone(&self.inner)
};
@@ -61,52 +62,53 @@ impl<T> RcVec<T> {
}

impl<T> RcVecBuilder<T> {
pub fn new() -> Self {
pub(crate) fn new() -> Self {
RcVecBuilder { inner: Vec::new() }
}

pub fn with_capacity(cap: usize) -> Self {
pub(crate) fn with_capacity(cap: usize) -> Self {
RcVecBuilder {
inner: Vec::with_capacity(cap),
}
}

pub fn push(&mut self, element: T) {
pub(crate) fn push(&mut self, element: T) {
self.inner.push(element);
}

pub fn extend(&mut self, iter: impl IntoIterator<Item = T>) {
pub(crate) fn extend(&mut self, iter: impl IntoIterator<Item = T>) {
self.inner.extend(iter);
}

pub fn as_mut(&mut self) -> RcVecMut<T> {
pub(crate) fn as_mut(&mut self) -> RcVecMut<T> {
RcVecMut {
inner: &mut self.inner,
}
}

pub fn build(self) -> RcVec<T> {
pub(crate) fn build(self) -> RcVec<T> {
RcVec {
inner: Rc::new(self.inner),
}
}
}

impl<'a, T> RcVecMut<'a, T> {
pub fn push(&mut self, element: T) {
pub(crate) fn push(&mut self, element: T) {
self.inner.push(element);
}

pub fn extend(&mut self, iter: impl IntoIterator<Item = T>) {
pub(crate) fn extend(&mut self, iter: impl IntoIterator<Item = T>) {
self.inner.extend(iter);
}

pub fn pop(&mut self) -> Option<T> {
self.inner.pop()
pub(crate) fn as_mut(&mut self) -> RcVecMut<T> {
RcVecMut { inner: self.inner }
}

pub fn as_mut(&mut self) -> RcVecMut<T> {
RcVecMut { inner: self.inner }
pub(crate) fn take(self) -> RcVecBuilder<T> {
let vec = mem::take(self.inner);
RcVecBuilder { inner: vec }
}
}

@@ -140,3 +142,5 @@ impl<T> Iterator for RcVecIntoIter<T> {
self.inner.size_hint()
}
}

impl<T> RefUnwindSafe for RcVec<T> where T: RefUnwindSafe {}
456 changes: 222 additions & 234 deletions src/wrapper.rs

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions tests/features.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
#![allow(clippy::ignore_without_reason)]

#[test]
#[ignore]
fn make_sure_no_proc_macro() {
11 changes: 4 additions & 7 deletions tests/marker.rs
Original file line number Diff line number Diff line change
@@ -21,6 +21,7 @@ macro_rules! assert_impl {
$(
{
// Implemented for types that implement $marker.
#[allow(dead_code)]
trait IsNotImplemented {
fn assert_not_implemented() {}
}
@@ -55,20 +56,17 @@ assert_impl!(TokenTree is not Send or Sync);

#[cfg(procmacro2_semver_exempt)]
mod semver_exempt {
use proc_macro2::{LineColumn, SourceFile};
use proc_macro2::LineColumn;

assert_impl!(LineColumn is Send and Sync);

assert_impl!(SourceFile is not Send or Sync);
}

#[cfg(not(no_libprocmacro_unwind_safe))]
mod unwind_safe {
#[cfg(procmacro2_semver_exempt)]
use proc_macro2::LineColumn;
use proc_macro2::{
Delimiter, Group, Ident, LexError, Literal, Punct, Spacing, Span, TokenStream, TokenTree,
};
#[cfg(procmacro2_semver_exempt)]
use proc_macro2::{LineColumn, SourceFile};
use std::panic::{RefUnwindSafe, UnwindSafe};

macro_rules! assert_unwind_safe {
@@ -95,6 +93,5 @@ mod unwind_safe {
#[cfg(procmacro2_semver_exempt)]
assert_unwind_safe! {
LineColumn
SourceFile
}
}
385 changes: 307 additions & 78 deletions tests/test.rs

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion tests/test_fmt.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#![allow(clippy::from_iter_instead_of_collect)]

use proc_macro2::{Delimiter, Group, Ident, Span, TokenStream, TokenTree};
use std::iter::{self, FromIterator};
use std::iter;

#[test]
fn test_fmt_group() {
67 changes: 53 additions & 14 deletions tests/test_size.rs
Original file line number Diff line number Diff line change
@@ -1,42 +1,81 @@
#![allow(unused_attributes)]

extern crate proc_macro;

use std::mem;

#[rustversion::attr(before(1.32), ignore)]
#[rustversion::attr(before(1.64), ignore = "requires Rust 1.64+")]
#[cfg_attr(not(target_pointer_width = "64"), ignore = "only applicable to 64-bit")]
#[cfg_attr(randomize_layout, ignore = "disabled due to randomized layout")]
#[test]
fn test_proc_macro_span_size() {
fn test_proc_macro_size() {
assert_eq!(mem::size_of::<proc_macro::Span>(), 4);
assert_eq!(mem::size_of::<Option<proc_macro::Span>>(), 4);
assert_eq!(mem::size_of::<proc_macro::Group>(), 20);
assert_eq!(mem::size_of::<proc_macro::Ident>(), 12);
assert_eq!(mem::size_of::<proc_macro::Punct>(), 8);
assert_eq!(mem::size_of::<proc_macro::Literal>(), 16);
assert_eq!(mem::size_of::<proc_macro::TokenStream>(), 4);
}

#[cfg_attr(not(all(not(wrap_proc_macro), not(span_locations))), ignore)]
#[cfg_attr(not(target_pointer_width = "64"), ignore = "only applicable to 64-bit")]
#[cfg_attr(randomize_layout, ignore = "disabled due to randomized layout")]
#[cfg_attr(wrap_proc_macro, ignore = "wrapper mode")]
#[cfg_attr(span_locations, ignore = "span locations are on")]
#[test]
fn test_proc_macro2_fallback_span_size_without_locations() {
fn test_proc_macro2_fallback_size_without_locations() {
assert_eq!(mem::size_of::<proc_macro2::Span>(), 0);
assert_eq!(mem::size_of::<Option<proc_macro2::Span>>(), 1);
assert_eq!(mem::size_of::<proc_macro2::Group>(), 16);
assert_eq!(mem::size_of::<proc_macro2::Ident>(), 24);
assert_eq!(mem::size_of::<proc_macro2::Punct>(), 8);
assert_eq!(mem::size_of::<proc_macro2::Literal>(), 24);
assert_eq!(mem::size_of::<proc_macro2::TokenStream>(), 8);
}

#[cfg_attr(not(all(not(wrap_proc_macro), span_locations)), ignore)]
#[cfg_attr(not(target_pointer_width = "64"), ignore = "only applicable to 64-bit")]
#[cfg_attr(randomize_layout, ignore = "disabled due to randomized layout")]
#[cfg_attr(wrap_proc_macro, ignore = "wrapper mode")]
#[cfg_attr(not(span_locations), ignore = "span locations are off")]
#[test]
fn test_proc_macro2_fallback_span_size_with_locations() {
fn test_proc_macro2_fallback_size_with_locations() {
assert_eq!(mem::size_of::<proc_macro2::Span>(), 8);
assert_eq!(mem::size_of::<Option<proc_macro2::Span>>(), 12);
assert_eq!(mem::size_of::<proc_macro2::Group>(), 24);
assert_eq!(mem::size_of::<proc_macro2::Ident>(), 32);
assert_eq!(mem::size_of::<proc_macro2::Punct>(), 16);
assert_eq!(mem::size_of::<proc_macro2::Literal>(), 32);
assert_eq!(mem::size_of::<proc_macro2::TokenStream>(), 8);
}

#[rustversion::attr(before(1.32), ignore)]
#[rustversion::attr(
since(1.32),
cfg_attr(not(all(wrap_proc_macro, not(span_locations))), ignore)
)]
#[rustversion::attr(before(1.71), ignore = "requires Rust 1.71+")]
#[cfg_attr(not(target_pointer_width = "64"), ignore = "only applicable to 64-bit")]
#[cfg_attr(randomize_layout, ignore = "disabled due to randomized layout")]
#[cfg_attr(not(wrap_proc_macro), ignore = "fallback mode")]
#[cfg_attr(span_locations, ignore = "span locations are on")]
#[test]
fn test_proc_macro2_wrapper_span_size_without_locations() {
fn test_proc_macro2_wrapper_size_without_locations() {
assert_eq!(mem::size_of::<proc_macro2::Span>(), 4);
assert_eq!(mem::size_of::<Option<proc_macro2::Span>>(), 8);
assert_eq!(mem::size_of::<proc_macro2::Group>(), 24);
assert_eq!(mem::size_of::<proc_macro2::Ident>(), 24);
assert_eq!(mem::size_of::<proc_macro2::Punct>(), 12);
assert_eq!(mem::size_of::<proc_macro2::Literal>(), 24);
assert_eq!(mem::size_of::<proc_macro2::TokenStream>(), 32);
}

#[cfg_attr(not(all(wrap_proc_macro, span_locations)), ignore)]
#[rustversion::attr(before(1.65), ignore = "requires Rust 1.65+")]
#[cfg_attr(not(target_pointer_width = "64"), ignore = "only applicable to 64-bit")]
#[cfg_attr(randomize_layout, ignore = "disabled due to randomized layout")]
#[cfg_attr(not(wrap_proc_macro), ignore = "fallback mode")]
#[cfg_attr(not(span_locations), ignore = "span locations are off")]
#[test]
fn test_proc_macro2_wrapper_span_size_with_locations() {
fn test_proc_macro2_wrapper_size_with_locations() {
assert_eq!(mem::size_of::<proc_macro2::Span>(), 12);
assert_eq!(mem::size_of::<Option<proc_macro2::Span>>(), 12);
assert_eq!(mem::size_of::<proc_macro2::Group>(), 32);
assert_eq!(mem::size_of::<proc_macro2::Ident>(), 32);
assert_eq!(mem::size_of::<proc_macro2::Punct>(), 20);
assert_eq!(mem::size_of::<proc_macro2::Literal>(), 32);
assert_eq!(mem::size_of::<proc_macro2::TokenStream>(), 32);
}
4 changes: 2 additions & 2 deletions tests/ui/compiletest.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#[rustversion::attr(not(nightly), ignore)]
#[cfg_attr(miri, ignore)]
#[rustversion::attr(not(nightly), ignore = "requires nightly")]
#[rustversion::attr(nightly, cfg_attr(miri, ignore = "incompatible with miri"))]
#[test]
fn ui() {
let t = trybuild::TestCases::new();
30 changes: 25 additions & 5 deletions tests/ui/test-not-send.stderr
Original file line number Diff line number Diff line change
@@ -5,8 +5,16 @@ error[E0277]: `proc_macro::Span` cannot be sent between threads safely
| ^^^^ `proc_macro::Span` cannot be sent between threads safely
|
= help: within `Span`, the trait `Send` is not implemented for `proc_macro::Span`
= note: required because it appears within the type `Span`
= note: required because it appears within the type `Span`
note: required because it appears within the type `proc_macro2::imp::Span`
--> $WORKSPACE/src/wrapper.rs
|
| pub(crate) enum Span {
| ^^^^
note: required because it appears within the type `Span`
--> $WORKSPACE/src/lib.rs
|
| pub struct Span {
| ^^^^
note: required by a bound in `requires_send`
--> test-not-send.rs:4:25
|
@@ -20,9 +28,21 @@ error[E0277]: `Rc<()>` cannot be sent between threads safely
| ^^^^ `Rc<()>` cannot be sent between threads safely
|
= help: within `Span`, the trait `Send` is not implemented for `Rc<()>`
= note: required because it appears within the type `ProcMacroAutoTraits`
= note: required because it appears within the type `PhantomData<ProcMacroAutoTraits>`
= note: required because it appears within the type `Span`
note: required because it appears within the type `PhantomData<Rc<()>>`
--> $RUST/core/src/marker.rs
|
| pub struct PhantomData<T: PointeeSized>;
| ^^^^^^^^^^^
note: required because it appears within the type `proc_macro2::marker::ProcMacroAutoTraits`
--> $WORKSPACE/src/marker.rs
|
| pub(crate) struct ProcMacroAutoTraits(PhantomData<Rc<()>>);
| ^^^^^^^^^^^^^^^^^^^
note: required because it appears within the type `Span`
--> $WORKSPACE/src/lib.rs
|
| pub struct Span {
| ^^^^
note: required by a bound in `requires_send`
--> test-not-send.rs:4:25
|