I couldn't get installation with the pinned versions of tokenizers and transformers in place, using the latest nightly rustc.
Will my fast-bert installation be incorrect without those version pins? Is there a quick correctness check I could run? I simply removed the pins like so to get installation:
Then I was able to install from source.
$ pip install fast-bert
Collecting fast-bert
Using cached fast_bert-1.9.15-py3-none-any.whl (99 kB)
Collecting spacy
Using cached spacy-3.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.1 MB)
Collecting pandas
Using cached pandas-1.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
Collecting seqeval
Using cached seqeval-1.2.2-py3-none-any.whl
Collecting fastprogress
Using cached fastprogress-1.0.0-py3-none-any.whl (12 kB)
Collecting more-itertools
Using cached more_itertools-8.12.0-py3-none-any.whl (54 kB)
Collecting tokenizers==0.8.1.rc1
Using cached tokenizers-0.8.1rc1.tar.gz (97 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting tensorboardX
Using cached tensorboardX-2.4.1-py2.py3-none-any.whl (124 kB)
Collecting sklearn
Using cached sklearn-0.0-py2.py3-none-any.whl
Collecting matplotlib
Using cached matplotlib-3.5.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.2 MB)
Collecting transformers==3.0.2
Using cached transformers-3.0.2-py3-none-any.whl (769 kB)
Collecting python-box
Using cached python_box-5.4.1-py3-none-any.whl (21 kB)
Collecting pytorch-lamb
Using cached pytorch_lamb-1.0.0-py3-none-any.whl (4.4 kB)
Requirement already satisfied: sentencepiece!=0.1.92 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from transformers==3.0.2->fast-bert) (0.1.96)
Requirement already satisfied: numpy in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from transformers==3.0.2->fast-bert) (1.21.2)
Requirement already satisfied: packaging in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from transformers==3.0.2->fast-bert) (21.3)
Requirement already satisfied: filelock in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from transformers==3.0.2->fast-bert) (3.4.2)
Requirement already satisfied: regex!=2019.12.17 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from transformers==3.0.2->fast-bert) (2022.1.18)
Requirement already satisfied: requests in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from transformers==3.0.2->fast-bert) (2.27.1)
Requirement already satisfied: tqdm>=4.27 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from transformers==3.0.2->fast-bert) (4.62.3)
Requirement already satisfied: sacremoses in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from transformers==3.0.2->fast-bert) (0.0.47)
Requirement already satisfied: pillow>=6.2.0 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from matplotlib->fast-bert) (8.4.0)
Collecting fonttools>=4.22.0
Using cached fonttools-4.29.1-py3-none-any.whl (895 kB)
Collecting kiwisolver>=1.0.1
Using cached kiwisolver-1.3.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
Requirement already satisfied: pyparsing>=2.2.1 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from matplotlib->fast-bert) (3.0.7)
Requirement already satisfied: python-dateutil>=2.7 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from matplotlib->fast-bert) (2.8.2)
Collecting cycler>=0.10
Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting pytz>=2020.1
Using cached pytz-2021.3-py2.py3-none-any.whl (503 kB)
Requirement already satisfied: torch>=0.4.1 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from pytorch-lamb->fast-bert) (1.10.2)
Requirement already satisfied: torchvision in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from pytorch-lamb->fast-bert) (0.11.3)
Collecting scikit-learn>=0.21.3
Using cached scikit_learn-1.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.4 MB)
Collecting preshed<3.1.0,>=3.0.2
Using cached preshed-3.0.6-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (128 kB)
Requirement already satisfied: setuptools in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from spacy->fast-bert) (58.0.4)
Collecting pathy>=0.3.5
Using cached pathy-0.6.1-py3-none-any.whl (42 kB)
Collecting spacy-legacy<3.1.0,>=3.0.8
Using cached spacy_legacy-3.0.8-py2.py3-none-any.whl (14 kB)
Collecting spacy-loggers<2.0.0,>=1.0.0
Using cached spacy_loggers-1.0.1-py3-none-any.whl (7.0 kB)
Collecting cymem<2.1.0,>=2.0.2
Using cached cymem-2.0.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (35 kB)
Collecting blis<0.8.0,>=0.4.0
Using cached blis-0.7.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.9 MB)
Collecting typer<0.5.0,>=0.3.0
Using cached typer-0.4.0-py3-none-any.whl (27 kB)
Collecting srsly<3.0.0,>=2.4.1
Using cached srsly-2.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (452 kB)
Collecting pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4
Using cached pydantic-1.8.2-cp39-cp39-manylinux2014_x86_64.whl (11.3 MB)
Collecting catalogue<2.1.0,>=2.0.6
Using cached catalogue-2.0.6-py3-none-any.whl (17 kB)
Collecting langcodes<4.0.0,>=3.2.0
Using cached langcodes-3.3.0-py3-none-any.whl (181 kB)
Collecting wasabi<1.1.0,>=0.8.1
Using cached wasabi-0.9.0-py3-none-any.whl (25 kB)
Collecting thinc<8.1.0,>=8.0.12
Using cached thinc-8.0.13-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (635 kB)
Collecting jinja2
Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting murmurhash<1.1.0,>=0.28.0
Using cached murmurhash-1.0.6-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (21 kB)
Collecting protobuf>=3.8.0
Using cached protobuf-3.19.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting smart-open<6.0.0,>=5.0.0
Using cached smart_open-5.2.1-py3-none-any.whl (58 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy->fast-bert) (3.10.0.2)
Requirement already satisfied: six>=1.5 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->fast-bert) (1.16.0)
Requirement already satisfied: certifi>=2017.4.17 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from requests->transformers==3.0.2->fast-bert) (2021.10.8)
Requirement already satisfied: idna<4,>=2.5 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from requests->transformers==3.0.2->fast-bert) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from requests->transformers==3.0.2->fast-bert) (2.0.11)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from requests->transformers==3.0.2->fast-bert) (1.26.8)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Requirement already satisfied: joblib>=0.11 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from scikit-learn>=0.21.3->seqeval->fast-bert) (1.1.0)
Collecting scipy>=1.1.0
Using cached scipy-1.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.1 MB)
Requirement already satisfied: click<9.0.0,>=7.1.1 in ./anaconda3/envs/jasonenv/lib/python3.9/site-packages (from typer<0.5.0,>=0.3.0->spacy->fast-bert) (8.0.3)
Collecting MarkupSafe>=2.0
Using cached MarkupSafe-2.0.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (30 kB)
Building wheels for collected packages: tokenizers
Building wheel for tokenizers (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for tokenizers (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [203 lines of output]
/tmp/pip-build-env-pneje7pk/overlay/lib/python3.9/site-packages/setuptools/dist.py:502: UserWarning: Normalizing '0.8.1.rc1' to '0.8.1rc1'
warnings.warn(tmpl.format(**locals()))
running bdist_wheel
running build
running build_py
running build_ext
running build_rust
Updating crates.io index
Updating git repository `https://github.com/n1t0/rayon-cond`
cargo rustc --lib --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib
warning: unused manifest key: target.x86_64-apple-darwin.rustflags
Compiling proc-macro2 v1.0.36
Compiling unicode-xid v0.2.2
Compiling syn v1.0.86
Compiling libc v0.2.117
Compiling autocfg v1.1.0
Compiling cfg-if v1.0.0
Compiling lazy_static v1.4.0
Compiling serde_derive v1.0.136
Compiling memchr v2.4.1
Compiling scopeguard v1.1.0
Compiling crossbeam-utils v0.8.7
Compiling serde v1.0.136
Compiling proc-macro-hack v0.5.19
Compiling regex-syntax v0.6.25
Compiling crossbeam-epoch v0.9.7
Compiling serde_json v1.0.78
Compiling rayon-core v1.9.1
Compiling getrandom v0.1.16
Compiling itoa v1.0.1
Compiling unicode-width v0.1.9
Compiling smallvec v1.8.0
Compiling cc v1.0.72
Compiling pkg-config v0.3.24
Compiling ryu v1.0.9
Compiling either v1.6.1
Compiling once_cell v1.9.0
Compiling erased-serde v0.3.18
Compiling ppv-lite86 v0.2.16
Compiling unindent v0.1.7
Compiling version_check v0.9.4
Compiling inventory v0.1.11
Compiling cfg-if v0.1.10
Compiling bitflags v1.3.2
Compiling number_prefix v0.3.0
Compiling strsim v0.8.0
Compiling vec_map v0.8.2
Compiling ansi_term v0.12.1
Compiling unicode_categories v0.1.1
Running `rustc --crate-name unicode_xid /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-xid-0.2.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=5169222e9a9224f7 -C extra-filename=-5169222e9a9224f7 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name lazy_static /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.4.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=79cb8478fcd4b466 -C extra-filename=-79cb8478fcd4b466 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name cfg_if --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=2deae1b316fdaf6f -C extra-filename=-2deae1b316fdaf6f --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.36/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=127f0f35c6e17df7 -C extra-filename=-127f0f35c6e17df7 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/proc-macro2-127f0f35c6e17df7 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name scopeguard /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/scopeguard-1.1.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=3c7ae9dd20140962 -C extra-filename=-3c7ae9dd20140962 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_derive-1.0.136/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=22ed2124e3dc3757 -C extra-filename=-22ed2124e3dc3757 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/serde_derive-22ed2124e3dc3757 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.117/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=0b981d156e8a7171 -C extra-filename=-0b981d156e8a7171 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/libc-0b981d156e8a7171 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name autocfg /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.1.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=061a37f812b47518 -C extra-filename=-061a37f812b47518 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.8.7/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=6559af75350ec7e6 -C extra-filename=-6559af75350ec7e6 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/crossbeam-utils-6559af75350ec7e6 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.4.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=c28282e528ea8998 -C extra-filename=-c28282e528ea8998 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/memchr-c28282e528ea8998 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.136/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=1f3db4bc67db0301 -C extra-filename=-1f3db4bc67db0301 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/serde-1f3db4bc67db0301 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.86/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' -C metadata=7cccbbc3608bd282 -C extra-filename=-7cccbbc3608bd282 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/syn-7cccbbc3608bd282 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.19/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=235c58df4bfc2d26 -C extra-filename=-235c58df4bfc2d26 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/proc-macro-hack-235c58df4bfc2d26 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name regex_syntax --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-syntax-0.6.25/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=79e1cae865a51a95 -C extra-filename=-79e1cae865a51a95 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.78/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=0aca0db3ea81e27c -C extra-filename=-0aca0db3ea81e27c --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/serde_json-0aca0db3ea81e27c -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.9.7/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=1b53a95c38fac831 -C extra-filename=-1b53a95c38fac831 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/crossbeam-epoch-1b53a95c38fac831 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=d2283c246bafe80c -C extra-filename=-d2283c246bafe80c --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/rayon-core-d2283c246bafe80c -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.16/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="std"' -C metadata=f7f1f89d3879f12a -C extra-filename=-f7f1f89d3879f12a --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/getrandom-f7f1f89d3879f12a -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name itoa --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/itoa-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=34da94ddc3c6a630 -C extra-filename=-34da94ddc3c6a630 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name unicode_width /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-width-0.1.9/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=bfedf134330d4bf0 -C extra-filename=-bfedf134330d4bf0 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name smallvec --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.8.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=a659c1158aa05c54 -C extra-filename=-a659c1158aa05c54 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name pkg_config /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/pkg-config-0.3.24/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=92694de5e874cd28 -C extra-filename=-92694de5e874cd28 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name cc --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.72/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=46bea57c09b258af -C extra-filename=-46bea57c09b258af --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name ryu --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.9/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=8fa7abc9dff34c50 -C extra-filename=-8fa7abc9dff34c50 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name either /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/either-1.6.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=7a8a5a271dc7cb51 -C extra-filename=-7a8a5a271dc7cb51 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name once_cell --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="race"' --cfg 'feature="std"' -C metadata=a203cc7f4a593743 -C extra-filename=-a203cc7f4a593743 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/erased-serde-0.3.18/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=f7c32cc485c2d89f -C extra-filename=-f7c32cc485c2d89f --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/erased-serde-f7c32cc485c2d89f -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name ppv_lite86 --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.16/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=f41c402efdf936bd -C extra-filename=-f41c402efdf936bd --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name unindent --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/unindent-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=ffdf58ea3b9e347d -C extra-filename=-ffdf58ea3b9e347d --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/inventory-0.1.11/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=e086f8ea4f266166 -C extra-filename=-e086f8ea4f266166 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/inventory-e086f8ea4f266166 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name cfg_if --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=33adbc7c23a934c7 -C extra-filename=-33adbc7c23a934c7 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name version_check /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/version_check-0.9.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=2be5c2cc4c1c8e71 -C extra-filename=-2be5c2cc4c1c8e71 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name bitflags --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.3.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=07e4b438a47db117 -C extra-filename=-07e4b438a47db117 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name strsim /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/strsim-0.8.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=0b9af79a9eabd5cc -C extra-filename=-0b9af79a9eabd5cc --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name vec_map /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/vec_map-0.8.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=99222fb3abba5a69 -C extra-filename=-99222fb3abba5a69 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name number_prefix /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/number_prefix-0.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=fa210d0967aff1ff -C extra-filename=-fa210d0967aff1ff --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name ansi_term /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/ansi_term-0.12.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=299089b8a23df6cc -C extra-filename=-299089b8a23df6cc --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Running `rustc --crate-name unicode_categories /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode_categories-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=6796d54f4a15ffb4 -C extra-filename=-6796d54f4a15ffb4 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow`
Compiling lock_api v0.3.4
Running `rustc --crate-name lock_api --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="nightly"' -C metadata=0e8ba3e8e8af44f0 -C extra-filename=-0e8ba3e8e8af44f0 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern scopeguard=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libscopeguard-3c7ae9dd20140962.rmeta --cap-lints allow`
Compiling textwrap v0.11.0
Running `rustc --crate-name textwrap /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/textwrap-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=1e5358ff7f49d80c -C extra-filename=-1e5358ff7f49d80c --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern unicode_width=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libunicode_width-bfedf134330d4bf0.rmeta --cap-lints allow`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/rayon-core-d2283c246bafe80c/build-script-build`
Compiling itertools v0.8.2
Running `rustc --crate-name itertools /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/itertools-0.8.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=ea6969d200b46984 -C extra-filename=-ea6969d200b46984 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern either=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libeither-7a8a5a271dc7cb51.rmeta --cap-lints allow`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/crossbeam-utils-6559af75350ec7e6/build-script-build`
Running `rustc --crate-name crossbeam_utils --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.8.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=e1fe7783182401eb -C extra-filename=-e1fe7783182401eb --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern cfg_if=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern lazy_static=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/liblazy_static-79cb8478fcd4b466.rmeta --cap-lints allow`
Compiling unicode-normalization-alignments v0.1.12
Running `rustc --crate-name unicode_normalization_alignments /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-normalization-alignments-0.1.12/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=af70868c129dc73e -C extra-filename=-af70868c129dc73e --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern smallvec=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libsmallvec-a659c1158aa05c54.rmeta --cap-lints allow`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/serde_derive-22ed2124e3dc3757/build-script-build`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/crossbeam-epoch-1b53a95c38fac831/build-script-build`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/memchr-c28282e528ea8998/build-script-build`
Running `rustc --crate-name memchr --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.4.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=671cb2e32aa232f7 -C extra-filename=-671cb2e32aa232f7 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow --cfg memchr_runtime_simd --cfg memchr_runtime_sse2 --cfg memchr_runtime_sse42 --cfg memchr_runtime_avx`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/getrandom-f7f1f89d3879f12a/build-script-build`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/libc-0b981d156e8a7171/build-script-build`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/serde-1f3db4bc67db0301/build-script-build`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/proc-macro-hack-235c58df4bfc2d26/build-script-build`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/inventory-e086f8ea4f266166/build-script-build`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/serde_json-0aca0db3ea81e27c/build-script-build`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/syn-7cccbbc3608bd282/build-script-build`
Running `rustc --crate-name libc /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.117/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=3deba52fa91f444f -C extra-filename=-3deba52fa91f444f --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN --cfg libc_cfg_target_vendor --cfg libc_non_exhaustive`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/erased-serde-f7c32cc485c2d89f/build-script-build`
Running `rustc --crate-name proc_macro_hack --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.19/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -Cembed-bitcode=no -C metadata=940fbea956d42b7d -C extra-filename=-940fbea956d42b7d --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern proc_macro --cap-lints allow`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/proc-macro2-127f0f35c6e17df7/build-script-build`
Compiling crossbeam-channel v0.5.2
Running `rustc --crate-name crossbeam_channel --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-channel-0.5.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="crossbeam-utils"' --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5d5b0d1f4a915704 -C extra-filename=-5d5b0d1f4a915704 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern cfg_if=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern crossbeam_utils=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcrossbeam_utils-e1fe7783182401eb.rmeta --cap-lints allow`
Running `rustc --crate-name proc_macro2 --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.36/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=4cc4c99b8d0fab22 -C extra-filename=-4cc4c99b8d0fab22 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern unicode_xid=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libunicode_xid-5169222e9a9224f7.rmeta --cap-lints allow --cfg no_literal_from_str --cfg no_group_open_close --cfg no_is_available --cfg use_proc_macro --cfg wrap_proc_macro --cfg proc_macro_span`
Compiling memoffset v0.6.5
Compiling rayon v1.5.1
Compiling num-traits v0.2.14
Running `rustc --crate-name build_script_build /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.6.5/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=a8a139922711247a -C extra-filename=-a8a139922711247a --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/memoffset-a8a139922711247a -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern autocfg=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libautocfg-061a37f812b47518.rlib --cap-lints allow`
Running `rustc --crate-name build_script_build --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=2cbfc7fb53d9ab52 -C extra-filename=-2cbfc7fb53d9ab52 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/rayon-2cbfc7fb53d9ab52 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern autocfg=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libautocfg-061a37f812b47518.rlib --cap-lints allow`
Running `rustc --crate-name build_script_build /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.14/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=0f54341be5041910 -C extra-filename=-0f54341be5041910 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/num-traits-0f54341be5041910 -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern autocfg=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libautocfg-061a37f812b47518.rlib --cap-lints allow`
Compiling aho-corasick v0.7.18
Running `rustc --crate-name aho_corasick --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.18/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=157f3d95428c1664 -C extra-filename=-157f3d95428c1664 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern memchr=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libmemchr-671cb2e32aa232f7.rmeta --cap-lints allow`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/memoffset-a8a139922711247a/build-script-build`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/num-traits-0f54341be5041910/build-script-build`
Running `/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/build/rayon-2cbfc7fb53d9ab52/build-script-build`
Compiling quote v1.0.15
Running `rustc --crate-name quote --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/quote-1.0.15/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=37cabfa38d900970 -C extra-filename=-37cabfa38d900970 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern proc_macro2=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libproc_macro2-4cc4c99b8d0fab22.rmeta --cap-lints allow`
Running `rustc --crate-name memoffset /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.6.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=2bcd7283847f3dbc -C extra-filename=-2bcd7283847f3dbc --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow --cfg tuple_ty --cfg allow_clippy --cfg maybe_uninit --cfg doctests`
Running `rustc --crate-name crossbeam_epoch --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.9.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=5a5aa8866fece43c -C extra-filename=-5a5aa8866fece43c --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern cfg_if=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern crossbeam_utils=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcrossbeam_utils-e1fe7783182401eb.rmeta --extern lazy_static=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/liblazy_static-79cb8478fcd4b466.rmeta --extern memoffset=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libmemoffset-2bcd7283847f3dbc.rmeta --extern scopeguard=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libscopeguard-3c7ae9dd20140962.rmeta --cap-lints allow`
Compiling num_cpus v1.13.1
Compiling terminal_size v0.1.17
Compiling parking_lot_core v0.7.2
Compiling atty v0.2.14
Running `rustc --crate-name getrandom --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.16/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="std"' -C metadata=02d36cc9cb79e5aa -C extra-filename=-02d36cc9cb79e5aa --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern cfg_if=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern libc=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/liblibc-3deba52fa91f444f.rmeta --cap-lints allow`
Running `rustc --crate-name num_cpus /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/num_cpus-1.13.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=b9f33399c948b917 -C extra-filename=-b9f33399c948b917 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern libc=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/liblibc-3deba52fa91f444f.rmeta --cap-lints allow`
Running `rustc --crate-name terminal_size --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/terminal_size-0.1.17/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=1fc20f7f9da277df -C extra-filename=-1fc20f7f9da277df --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern libc=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/liblibc-3deba52fa91f444f.rmeta --cap-lints allow`
Running `rustc --crate-name parking_lot_core --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.7.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="nightly"' -C metadata=1b1c1dba5e1ae5d7 -C extra-filename=-1b1c1dba5e1ae5d7 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern cfg_if=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcfg_if-33adbc7c23a934c7.rmeta --extern libc=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/liblibc-3deba52fa91f444f.rmeta --extern smallvec=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libsmallvec-a659c1158aa05c54.rmeta --cap-lints allow`
Running `rustc --crate-name atty /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/atty-0.2.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=96bd073f4656261d -C extra-filename=-96bd073f4656261d --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern libc=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/liblibc-3deba52fa91f444f.rmeta --cap-lints allow`
Compiling clap v2.34.0
Running `rustc --crate-name clap --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="ansi_term"' --cfg 'feature="atty"' --cfg 'feature="color"' --cfg 'feature="default"' --cfg 'feature="strsim"' --cfg 'feature="suggestions"' --cfg 'feature="vec_map"' -C metadata=2d260623f0c0c640 -C extra-filename=-2d260623f0c0c640 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern ansi_term=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libansi_term-299089b8a23df6cc.rmeta --extern atty=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libatty-96bd073f4656261d.rmeta --extern bitflags=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libbitflags-07e4b438a47db117.rmeta --extern strsim=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libstrsim-0b9af79a9eabd5cc.rmeta --extern textwrap=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libtextwrap-1e5358ff7f49d80c.rmeta --extern unicode_width=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libunicode_width-bfedf134330d4bf0.rmeta --extern vec_map=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libvec_map-99222fb3abba5a69.rmeta --cap-lints allow`
Running `rustc --crate-name syn --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.86/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' -C metadata=3e33e9c139dfd7d6 -C extra-filename=-3e33e9c139dfd7d6 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern proc_macro2=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libproc_macro2-4cc4c99b8d0fab22.rmeta --extern quote=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libquote-37cabfa38d900970.rmeta --extern unicode_xid=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libunicode_xid-5169222e9a9224f7.rmeta --cap-lints allow --cfg syn_no_negative_literal_parse`
Running `rustc --crate-name num_traits /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=f850401c6cabd015 -C extra-filename=-f850401c6cabd015 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --cap-lints allow --cfg has_i128 --cfg has_to_int_unchecked`
Compiling rand_core v0.5.1
Running `rustc --crate-name rand_core --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_core-0.5.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="getrandom"' --cfg 'feature="std"' -C metadata=3ce944ff9b15e65f -C extra-filename=-3ce944ff9b15e65f --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern getrandom=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libgetrandom-02d36cc9cb79e5aa.rmeta --cap-lints allow`
Compiling parking_lot v0.10.2
Running `rustc --crate-name parking_lot --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.10.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="nightly"' -C metadata=2e9b927e64e2fd23 -C extra-filename=-2e9b927e64e2fd23 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern lock_api=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/liblock_api-0e8ba3e8e8af44f0.rmeta --extern parking_lot_core=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libparking_lot_core-1b1c1dba5e1ae5d7.rmeta --cap-lints allow`
Compiling crossbeam-deque v0.8.1
Running `rustc --crate-name crossbeam_deque --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.8.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="crossbeam-epoch"' --cfg 'feature="crossbeam-utils"' --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=aa7eb6221c37ab49 -C extra-filename=-aa7eb6221c37ab49 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern cfg_if=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern crossbeam_epoch=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcrossbeam_epoch-5a5aa8866fece43c.rmeta --extern crossbeam_utils=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcrossbeam_utils-e1fe7783182401eb.rmeta --cap-lints allow`
error[E0658]: `if` is not allowed in a `const fn`
--> /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/app/settings.rs:7:1
|
7 | / bitflags! {
8 | | struct Flags: u64 {
9 | | const SC_NEGATE_REQS = 1;
10 | | const SC_REQUIRED = 1 << 1;
... |
51 | | }
52 | | }
| |_^
|
= note: see issue #49146 <https://github.com/rust-lang/rust/issues/49146> for more information
= help: add `#![feature(const_if_match)]` to the crate attributes to enable
= note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
Compiling rand_chacha v0.2.2
Running `rustc --crate-name rand_chacha --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_chacha-0.2.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="std"' -C metadata=78432008163ec0a9 -C extra-filename=-78432008163ec0a9 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern ppv_lite86=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libppv_lite86-f41c402efdf936bd.rmeta --extern rand_core=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/librand_core-3ce944ff9b15e65f.rmeta --cap-lints allow`
error[E0658]: `if` is not allowed in a `const fn`
--> /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/args/settings.rs:6:1
|
6 | / bitflags! {
7 | | struct Flags: u32 {
8 | | const REQUIRED = 1;
9 | | const MULTIPLE = 1 << 1;
... |
28 | | }
29 | | }
| |_^
|
= note: see issue #49146 <https://github.com/rust-lang/rust/issues/49146> for more information
= help: add `#![feature(const_if_match)]` to the crate attributes to enable
= note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
Running `rustc --crate-name rayon_core --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=268a2b41a39bead0 -C extra-filename=-268a2b41a39bead0 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern crossbeam_channel=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcrossbeam_channel-5d5b0d1f4a915704.rmeta --extern crossbeam_deque=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcrossbeam_deque-aa7eb6221c37ab49.rmeta --extern crossbeam_utils=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcrossbeam_utils-e1fe7783182401eb.rmeta --extern lazy_static=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/liblazy_static-79cb8478fcd4b466.rmeta --extern num_cpus=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libnum_cpus-b9f33399c948b917.rmeta --cap-lints allow`
Compiling paste-impl v0.1.18
Running `rustc --crate-name paste_impl --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-impl-0.1.18/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -Cembed-bitcode=no -C metadata=7047fcfa649dd5ba -C extra-filename=-7047fcfa649dd5ba --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern proc_macro_hack=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libproc_macro_hack-940fbea956d42b7d.so --extern proc_macro --cap-lints allow`
Compiling rand v0.7.3
Running `rustc --crate-name rand --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="getrandom"' --cfg 'feature="getrandom_package"' --cfg 'feature="libc"' --cfg 'feature="std"' -C metadata=795b7e2f5f181642 -C extra-filename=-795b7e2f5f181642 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern getrandom_package=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libgetrandom-02d36cc9cb79e5aa.rmeta --extern libc=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/liblibc-3deba52fa91f444f.rmeta --extern rand_chacha=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/librand_chacha-78432008163ec0a9.rmeta --extern rand_core=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/librand_core-3ce944ff9b15e65f.rmeta --cap-lints allow`
Running `rustc --crate-name rayon --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=9ba1226349fb0a01 -C extra-filename=-9ba1226349fb0a01 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern crossbeam_deque=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libcrossbeam_deque-aa7eb6221c37ab49.rmeta --extern either=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libeither-7a8a5a271dc7cb51.rmeta --extern rayon_core=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/librayon_core-268a2b41a39bead0.rmeta --cap-lints allow --cfg step_by`
Compiling regex v1.5.4
Running `rustc --crate-name regex --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-1.5.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="aho-corasick"' --cfg 'feature="default"' --cfg 'feature="memchr"' --cfg 'feature="perf"' --cfg 'feature="perf-cache"' --cfg 'feature="perf-dfa"' --cfg 'feature="perf-inline"' --cfg 'feature="perf-literal"' --cfg 'feature="std"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=1e333a3fef530f25 -C extra-filename=-1e333a3fef530f25 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern aho_corasick=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libaho_corasick-157f3d95428c1664.rmeta --extern memchr=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libmemchr-671cb2e32aa232f7.rmeta --extern regex_syntax=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libregex_syntax-79e1cae865a51a95.rmeta --cap-lints allow`
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0658`.
error: could not compile `clap`.
Caused by:
process didn't exit successfully: `rustc --crate-name clap --edition=2018 /home/jaten/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="ansi_term"' --cfg 'feature="atty"' --cfg 'feature="color"' --cfg 'feature="default"' --cfg 'feature="strsim"' --cfg 'feature="suggestions"' --cfg 'feature="vec_map"' -C metadata=2d260623f0c0c640 -C extra-filename=-2d260623f0c0c640 --out-dir /tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps -L dependency=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps --extern ansi_term=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libansi_term-299089b8a23df6cc.rmeta --extern atty=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libatty-96bd073f4656261d.rmeta --extern bitflags=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libbitflags-07e4b438a47db117.rmeta --extern strsim=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libstrsim-0b9af79a9eabd5cc.rmeta --extern textwrap=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libtextwrap-1e5358ff7f49d80c.rmeta --extern unicode_width=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libunicode_width-bfedf134330d4bf0.rmeta --extern vec_map=/tmp/pip-install-tednfh26/tokenizers_3e8135338e264710b8ac175053db5595/target/release/deps/libvec_map-99222fb3abba5a69.rmeta --cap-lints allow` (exit code: 1)
warning: build failed, waiting for other jobs to finish...
error: build failed
error: cargo failed with code: 101
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
$
$ rustc --version
rustc 1.60.0-nightly (e7aca8959 2022-02-09)
I couldn't get installation with the pinned versions of tokenizers and transformers in place, using the latest nightly rustc.
Will my fast-bert installation be incorrect without those version pins? Is there a quick correctness check I could run? I simply removed the pins like so to get installation:
Then I was able to install from source.
With the version pins in place I get this failure to install using pip install: