Instructions for getting started don't work on macOS

[foolip]

I'm trying to follow these instructions:

https://web-platform-tests.org/running-tests/from-local-system.html
https://web-platform-tests.org/running-tests/safari.html

The command I'm running is ./wpt run --channel=preview safari dom/historical.html fullscreen/api/historical.html.

That fails in pip3 install, the equivalent of invoking ./_venv3/bin/pip3 install --prefer-binary -r tools/webtransport/requirements.txt. Running that directly produces this error:

Collecting aioquic==0.9.20
  Using cached aioquic-0.9.20.tar.gz (153 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: certifi in ./_venv3/lib/python3.11/site-packages (from aioquic==0.9.20->-r tools/webtransport/requirements.txt (line 1)) (2023.5.7)
Requirement already satisfied: cryptography>=3.1 in ./_venv3/lib/python3.11/site-packages (from aioquic==0.9.20->-r tools/webtransport/requirements.txt (line 1)) (40.0.2)
Requirement already satisfied: pylsqpack<0.4.0,>=0.3.3 in ./_venv3/lib/python3.11/site-packages (from aioquic==0.9.20->-r tools/webtransport/requirements.txt (line 1)) (0.3.17)
Requirement already satisfied: pyopenssl>=20 in ./_venv3/lib/python3.11/site-packages (from aioquic==0.9.20->-r tools/webtransport/requirements.txt (line 1)) (23.1.1)
Requirement already satisfied: cffi>=1.12 in ./_venv3/lib/python3.11/site-packages (from cryptography>=3.1->aioquic==0.9.20->-r tools/webtransport/requirements.txt (line 1)) (1.15.1)
Requirement already satisfied: pycparser in ./_venv3/lib/python3.11/site-packages (from cffi>=1.12->cryptography>=3.1->aioquic==0.9.20->-r tools/webtransport/requirements.txt (line 1)) (2.21)
Building wheels for collected packages: aioquic
  Building wheel for aioquic (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [62 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-13-arm64-cpython-311
      creating build/lib.macosx-13-arm64-cpython-311/aioquic
      copying src/aioquic/__init__.py -> build/lib.macosx-13-arm64-cpython-311/aioquic
      copying src/aioquic/about.py -> build/lib.macosx-13-arm64-cpython-311/aioquic
      copying src/aioquic/tls.py -> build/lib.macosx-13-arm64-cpython-311/aioquic
      copying src/aioquic/buffer.py -> build/lib.macosx-13-arm64-cpython-311/aioquic
      creating build/lib.macosx-13-arm64-cpython-311/aioquic/asyncio
      copying src/aioquic/asyncio/server.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/asyncio
      copying src/aioquic/asyncio/protocol.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/asyncio
      copying src/aioquic/asyncio/client.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/asyncio
      copying src/aioquic/asyncio/__init__.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/asyncio
      creating build/lib.macosx-13-arm64-cpython-311/aioquic/h0
      copying src/aioquic/h0/__init__.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h0
      copying src/aioquic/h0/connection.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h0
      creating build/lib.macosx-13-arm64-cpython-311/aioquic/h3
      copying src/aioquic/h3/events.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h3
      copying src/aioquic/h3/__init__.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h3
      copying src/aioquic/h3/connection.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h3
      copying src/aioquic/h3/exceptions.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h3
      creating build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/configuration.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/recovery.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/packet_builder.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/events.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/__init__.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/logger.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/stream.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/retry.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/connection.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/crypto.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/packet.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/rangeset.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      running egg_info
      writing src/aioquic.egg-info/PKG-INFO
      writing dependency_links to src/aioquic.egg-info/dependency_links.txt
      writing requirements to src/aioquic.egg-info/requires.txt
      writing top-level names to src/aioquic.egg-info/top_level.txt
      reading manifest file 'src/aioquic.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file 'src/aioquic.egg-info/SOURCES.txt'
      copying src/aioquic/_buffer.c -> build/lib.macosx-13-arm64-cpython-311/aioquic
      copying src/aioquic/_crypto.c -> build/lib.macosx-13-arm64-cpython-311/aioquic
      running build_ext
      building 'aioquic._buffer' extension
      creating build/temp.macosx-13-arm64-cpython-311
      creating build/temp.macosx-13-arm64-cpython-311/src
      creating build/temp.macosx-13-arm64-cpython-311/src/aioquic
      clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/foolip/wpt/wpt/_venv3/include -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c src/aioquic/_buffer.c -o build/temp.macosx-13-arm64-cpython-311/src/aioquic/_buffer.o -std=c99
      clang -bundle -undefined dynamic_lookup build/temp.macosx-13-arm64-cpython-311/src/aioquic/_buffer.o -o build/lib.macosx-13-arm64-cpython-311/aioquic/_buffer.cpython-311-darwin.so
      ld: warning: -undefined dynamic_lookup may not work with chained fixups
      building 'aioquic._crypto' extension
      clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/foolip/wpt/wpt/_venv3/include -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c src/aioquic/_crypto.c -o build/temp.macosx-13-arm64-cpython-311/src/aioquic/_crypto.o -std=c99
      src/aioquic/_crypto.c:4:10: fatal error: 'openssl/err.h' file not found
      #include <openssl/err.h>
               ^~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for aioquic
  Running setup.py clean for aioquic
Failed to build aioquic
Installing collected packages: aioquic
  Running setup.py install for aioquic ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for aioquic did not run successfully.
  │ exit code: 1
  ╰─> [64 lines of output]
      running install
      /Users/foolip/wpt/wpt/_venv3/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-13-arm64-cpython-311
      creating build/lib.macosx-13-arm64-cpython-311/aioquic
      copying src/aioquic/__init__.py -> build/lib.macosx-13-arm64-cpython-311/aioquic
      copying src/aioquic/about.py -> build/lib.macosx-13-arm64-cpython-311/aioquic
      copying src/aioquic/tls.py -> build/lib.macosx-13-arm64-cpython-311/aioquic
      copying src/aioquic/buffer.py -> build/lib.macosx-13-arm64-cpython-311/aioquic
      creating build/lib.macosx-13-arm64-cpython-311/aioquic/asyncio
      copying src/aioquic/asyncio/server.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/asyncio
      copying src/aioquic/asyncio/protocol.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/asyncio
      copying src/aioquic/asyncio/client.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/asyncio
      copying src/aioquic/asyncio/__init__.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/asyncio
      creating build/lib.macosx-13-arm64-cpython-311/aioquic/h0
      copying src/aioquic/h0/__init__.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h0
      copying src/aioquic/h0/connection.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h0
      creating build/lib.macosx-13-arm64-cpython-311/aioquic/h3
      copying src/aioquic/h3/events.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h3
      copying src/aioquic/h3/__init__.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h3
      copying src/aioquic/h3/connection.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h3
      copying src/aioquic/h3/exceptions.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/h3
      creating build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/configuration.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/recovery.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/packet_builder.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/events.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/__init__.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/logger.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/stream.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/retry.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/connection.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/crypto.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/packet.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      copying src/aioquic/quic/rangeset.py -> build/lib.macosx-13-arm64-cpython-311/aioquic/quic
      running egg_info
      writing src/aioquic.egg-info/PKG-INFO
      writing dependency_links to src/aioquic.egg-info/dependency_links.txt
      writing requirements to src/aioquic.egg-info/requires.txt
      writing top-level names to src/aioquic.egg-info/top_level.txt
      reading manifest file 'src/aioquic.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file 'src/aioquic.egg-info/SOURCES.txt'
      copying src/aioquic/_buffer.c -> build/lib.macosx-13-arm64-cpython-311/aioquic
      copying src/aioquic/_crypto.c -> build/lib.macosx-13-arm64-cpython-311/aioquic
      running build_ext
      building 'aioquic._buffer' extension
      creating build/temp.macosx-13-arm64-cpython-311
      creating build/temp.macosx-13-arm64-cpython-311/src
      creating build/temp.macosx-13-arm64-cpython-311/src/aioquic
      clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/foolip/wpt/wpt/_venv3/include -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c src/aioquic/_buffer.c -o build/temp.macosx-13-arm64-cpython-311/src/aioquic/_buffer.o -std=c99
      clang -bundle -undefined dynamic_lookup build/temp.macosx-13-arm64-cpython-311/src/aioquic/_buffer.o -o build/lib.macosx-13-arm64-cpython-311/aioquic/_buffer.cpython-311-darwin.so
      ld: warning: -undefined dynamic_lookup may not work with chained fixups
      building 'aioquic._crypto' extension
      clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/foolip/wpt/wpt/_venv3/include -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c src/aioquic/_crypto.c -o build/temp.macosx-13-arm64-cpython-311/src/aioquic/_crypto.o -std=c99
      src/aioquic/_crypto.c:4:10: fatal error: 'openssl/err.h' file not found
      #include <openssl/err.h>
               ^~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> aioquic

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: /Users/foolip/wpt/wpt/_venv3/bin/python -m pip install --upgrade pip

I don't know what to install to get openssl/err.h.

cc @past

[foolip]

FWIW, running ./_venv3/bin/python -m pip install --upgrade pip first doesn't fix the problem, so the pip warning can be ignored.

[foolip]

aiortc/aioquic#56 is about this, and links to aiortc/aioquic#372 filed by none other than @jgraham. I indeed have Python 3.11 locally.

[foolip]

The problem seems to be Python 3.11. I have some Python distributions via Homebrew, and running brew unlink python3 gets that out of the way.

But it still fails:

Traceback (most recent call last):
  File "/Users/foolip/wpt/wpt/./wpt", line 10, in <module>
    wpt.main()
  File "/Users/foolip/wpt/wpt/tools/wpt/wpt.py", line 211, in main
    script, parser = import_command(prog, command, props)
  File "/Users/foolip/wpt/wpt/tools/wpt/wpt.py", line 102, in import_command
    mod = __import__(mod_name)
  File "/Users/foolip/wpt/wpt/tools/wpt/run.py", line 13, in <module>
    from . import browser, install, testfiles
  File "/Users/foolip/wpt/wpt/tools/wpt/browser.py", line 15, in <module>
    import requests
  File "/Users/foolip/wpt/wpt/_venv3/lib/python3.9/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/Users/foolip/wpt/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py", line 38, in <module>
    raise ImportError(
ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with LibreSSL 2.8.3. See: https://github.com/urllib3/urllib3/issues/2168

[foolip]

I think the root cause of this is that we didn't upgrade to support Python 3.11 fast enough, so users can end up trying to run WPT with Python 3.11 and see broken things.

I've sent #40192 for what the CI changes would be, let's see what breaks. #39081

[cookiecrook]

But it still fails:

ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+ …

That's a recent change that's causing wider breakage, including some AWS and other deployments, too.
urllib3/urllib3#2168

[sahilshah]

This is blocking us too. Is there a workaround for this?

[foolip]

@sahilshah I don't have a workaround I'm afraid. It probably still works on Linux so if I was in a hurry perhaps I'd run WPT in a Linux VM. May I ask what you're using WPT for, are you contributing tests or running them locally?

[Crazy467]

I want to add SD and pin wipe protection to my Trezor model T.

I have followed the steps to install treozrctl on my mac and have confirmed the trezorctl version
 
/Users//Library/Python/3.9/bin/trezorctl

but when I run the commands to set a wipe pin or enable sd-protect I get this error

Traceback (most recent call last):
  File "/Users//Library/Python/3.9/bin/trezorctl", line 5, in
    from trezorlib.cli.trezorctl import cli
  File "/Users//Library/Python/3.9/lib/python/site-packages/trezorlib/cli/trezorctl.py", line 31, in
    from . import (
  File "/Users//Library/Python/3.9/lib/python/site-packages/trezorlib/cli/ethereum.py", line 37, in
    from .. import definitions, ethereum, tools
  File "/Users//Library/Python/3.9/lib/python/site-packages/trezorlib/definitions.py", line 7, in
    import requests
  File "/Users//Library/Python/3.9/lib/python/site-packages/requests/init.py", line 43, in
    import urllib3
  File "/Users//Library/Python/3.9/lib/python/site-packages/urllib3/init.py", line 38, in
    raise ImportError(
ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with LibreSSL 2.8.3. See: urllib3/urllib3#2168

What am I missing?

[gsnedders]

But it still fails:

ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+ …

That's a recent change that's causing wider breakage, including some AWS and other deployments, too. urllib3/urllib3#2168

urllib3/urllib3#3024 relaxed the check (and shipped in urllib3 2.0.3), so we now get a warning with LibreSSL instead of an error. So at this point the Xcode-provided Python 3 should work, I believe. (Our dependency got updated in #40434.)

[Crazy467]

I did finally get the sd protect and pin wipe functions to work with python 3. So happy!