Skip to content

Potential method for avoiding late breaking packages updates #741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
karlhigley wants to merge 3 commits into from
Closed

Potential method for avoiding late breaking packages updates #741

karlhigley wants to merge 3 commits into from

Conversation

@karlhigley
Copy link
Contributor

@karlhigley karlhigley commented Nov 22, 2022

This is the lowest effort way I can think of to reduce the surface area of Python packages that can break our release builds by publishing new versions. It works like this:

  • We condense the list of Python packages to install into a requirements file (instead of keeping that directly in the Dockerfile)
  • We move the base image Dockerfile into its own directory so that the requirements file can be provided in the build context
  • In our normal (non-release) builds, everything continues to work basically the same as it has, with the packages auto-updating when new versions are released
  • When we want to cut release branches, we take the latest build of the base container, run pip freeze in it, overwrite the requirements file with the resulting list of packages, and commit that to the release branch only
  • The pip freeze commits never get added to the main branch, we just create a new freeze commit in each release branch
  • No big deal if we forget to add the freeze commit, then we're just back where we started with potential breaking changes sneaking in late

@karlhigley
Potential method for avoiding late breaking packages updates
a03b4ed 
This is the lowest effort way I can think of to reduce the surface area of Python packages that can break our release builds by publishing new versions. It works like this:
* We condense the list of Python packages to install into a requirements file (instead of keeping that directly in the Dockerfile)
* We move the base image Dockerfile into its own directory so that the requirements file can be provided in the build context
* In our normal (non-release) builds, everything continues to work basically the same as it has, with the packages auto-updating when new versions are released
* When we want to cut release branches, we take the latest build of the base container, run `pip freeze` in it, overwrite the requirements file with the resulting list of packages, and commit that to the release branch *only*
* The `pip freeze` commits never get added to the `main` branch, we just create a new `freeze` commit in each release branch
* No big deal if we forget to add the `freeze` commit, then we're just back where we started with potential breaking changes sneaking in late
@karlhigley karlhigley added chore Infrastructure update ci labels Nov 22, 2022
@karlhigley karlhigley added this to the Merlin 22.12 milestone Nov 22, 2022
@karlhigley karlhigley self-assigned this Nov 22, 2022
@github-actions
Copy link

github-actions bot commented Nov 22, 2022

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-741

@nvidia-merlin-bot
Copy link
Contributor

nvidia-merlin-bot commented Nov 22, 2022

Click to view CI Results
GitHub pull request #741 of commit a03b4ed7f39a098233adf21449545c9cd9d17c70, no merge conflicts.
Running as SYSTEM
Setting status of a03b4ed7f39a098233adf21449545c9cd9d17c70 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/642/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/741/*:refs/remotes/origin/pr/741/* # timeout=10
 > git rev-parse a03b4ed7f39a098233adf21449545c9cd9d17c70^{commit} # timeout=10
Checking out Revision a03b4ed7f39a098233adf21449545c9cd9d17c70 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a03b4ed7f39a098233adf21449545c9cd9d17c70 # timeout=10
Commit message: "Potential method for avoiding late breaking packages updates"
 > git rev-list --no-walk 0996d12353665ee01d1f28cb0b8db4c79a77c0b7 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins13039084959698467625.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.14,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.14,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='45539446'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-um30chou
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-um30chou
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit c08966972ddb60547099740a1e18f886dda922e8
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+19.gc089669) (2.22.0)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.gc089669) (0.3.0+12.g78ecddd)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.gc089669) (1.1.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (21.3)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.2.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (0.12.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (3.1.2)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 52.5 MB/s eta 0:00:00
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.gc089669) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.gc089669-py3-none-any.whl size=94042 sha256=a2f75343457023bb070fb897ee3191e24f7f56489e690337e870748908824f2d
  Stored in directory: /tmp/pip-ephem-wheel-cache-m9eqqtar/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+19.gc089669
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-haluw01p
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-haluw01p
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 634edfd9f0d12e509be5dcce8e2bdd0b2cb717aa
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+52.g634edfd9) (0.3.0+12.g78ecddd)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (21.3)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.2.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (0.12.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (3.1.2)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+52.g634edfd9) (6.0.1)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+52.g634edfd9-py3-none-any.whl size=366028 sha256=e0a9835c2df6a15d5b4826b386965020def3e761e3fd8bbced48546b5d227692
  Stored in directory: /tmp/pip-ephem-wheel-cache-1c0j4sjj/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+52.g634edfd9
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-3wxz0td2
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-3wxz0td2
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit dec7bacd4c215093e2d27b8abcb566dc0e9e1359
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+7.gdec7bacd) (1.8.1)
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.2.tar.gz (44 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 1.3 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+7.gdec7bacd) (0.3.0+12.g78ecddd)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (21.3)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.2.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (0.55.1)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+7.gdec7bacd) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (0.12.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (3.1.2)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+7.gdec7bacd) (6.0.1)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+7.gdec7bacd-cp38-cp38-linux_x86_64.whl size=257588 sha256=9e294f8d0263bc113411eba260beb48e4266a4b796d867b7b33984c2a91ef153
  Stored in directory: /tmp/pip-ephem-wheel-cache-ur1y81sw/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2-py3-none-any.whl size=29203 sha256=e119d4db7e53e5cec73ff1cb262784cc1aa4f41cfba12713ceede43f6ba59b7f
  Stored in directory: /tmp/pip-ephem-wheel-cache-ur1y81sw/wheels/76/ef/ed/cb880e3ef5192ec5940e26fd9442247b569fb0cf8602f97137
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.2 nvtabular-1.6.0+7.gdec7bacd
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-ue7bkptu
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-ue7bkptu
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 025f10da5f96c9ae470ee31133801da05fabfbf9
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+12.g025f10d) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+12.g025f10d) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+12.g025f10d) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+12.g025f10d) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+12.g025f10d) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+12.g025f10d) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+12.g025f10d) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+12.g025f10d) (3.19.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+12.g025f10d) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+12.g025f10d) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+12.g025f10d) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+12.g025f10d) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+12.g025f10d) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (0.12.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (3.1.2)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+12.g025f10d) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+12.g025f10d) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+12.g025f10d) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+12.g025f10d) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+12.g025f10d) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+12.g025f10d) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+12.g025f10d) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+12.g025f10d) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+12.g025f10d) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+12.g025f10d) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+12.g025f10d) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+12.g025f10d) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+12.g025f10d) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+12.g025f10d) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+12.g025f10d-py3-none-any.whl size=118882 sha256=e42bdf238e534d80847d48c2a71e8f39880aac3a316b276ce38df5fd6266bc90
  Stored in directory: /tmp/pip-ephem-wheel-cache-z73hbzin/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+12.g025f10d
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py .      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    215    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    146    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      4    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     27    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    161    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    344    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     175     95    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 242    108    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                106     71    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                60     36    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               756    340    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31     10    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     33    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       229     59    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     49    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      4    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               24      8    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77     27    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     46    61%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79     32    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     22    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             65     19    71%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           54      5    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 140     47    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143     74    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     14    81%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%


TOTAL                                                                                                 14924   6988    53%


======================== 3 passed in 344.96s (0:05:44) =========================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins6553711687508337987.sh

oliverholworthy
oliverholworthy reviewed Nov 23, 2022
View reviewed changes
docker/base/dockerfile.merlin
RUN pip install numba
RUN pip install "cuda-python>=11.5,<12.0"
RUN pip install fsspec==2022.5.0 llvmlite
RUN pip install pynvml dask==${DASK_VER} distributed==${DASK_VER}
Copy link
Contributor

@oliverholworthy oliverholworthy Nov 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we intend to remove the pinned dask version in the requirement file?

Copy link
Contributor Author

@karlhigley karlhigley Nov 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're actually installed twice, so it seems safe to remove them here? @jperez999, is there a reason to install these libraries here and then again later?

Copy link
Contributor Author

@karlhigley karlhigley Nov 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not safe because packages that are installed in between change the versions

@karlhigley
Add dask, distributed, and numpy installs back
d941a85 
These were there to override the (incorrect) versions installed by other packages, so it's safer if we keep them.
@nvidia-merlin-bot
Copy link
Contributor

nvidia-merlin-bot commented Nov 29, 2022

Click to view CI Results
GitHub pull request #741 of commit d941a85143db2972ec08a5969eaa44ab1baba985, no merge conflicts.
Running as SYSTEM
Setting status of d941a85143db2972ec08a5969eaa44ab1baba985 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/659/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/741/*:refs/remotes/origin/pr/741/* # timeout=10
 > git rev-parse d941a85143db2972ec08a5969eaa44ab1baba985^{commit} # timeout=10
Checking out Revision d941a85143db2972ec08a5969eaa44ab1baba985 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d941a85143db2972ec08a5969eaa44ab1baba985 # timeout=10
Commit message: "Add `dask`, `distributed`, and `numpy` installs back"
 > git rev-list --no-walk 8864907b18c0d4d8db9dd154d84bfc5f8a63d0cc # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins7140811434378580052.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.18,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.18,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='165177437'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-d8lq2s0y
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-d8lq2s0y
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit c73e81898c4c3ba10df019b9f36fa409a39d2626
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting treelite-runtime==2.4.0
  Downloading treelite_runtime-2.4.0-py3-none-manylinux2014_x86_64.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 3.1 MB/s eta 0:00:00
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (0.3.0+12.g78ecddd)
Collecting treelite==2.4.0
  Downloading treelite-2.4.0-py3-none-manylinux2014_x86_64.whl (852 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.7/852.7 kB 8.4 MB/s eta 0:00:00
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+21.gc73e818) (2.22.0)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.8.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.20.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.19.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.10.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.12.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.8.0)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 27.4 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+21.gc73e818-py3-none-any.whl size=98087 sha256=b51dcd0e83db1d41fea14164d3a0a7248453688acf0d03b345a0a43e66983028
  Stored in directory: /tmp/pip-ephem-wheel-cache-6cuipzhr/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, treelite-runtime, treelite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: treelite-runtime
    Found existing installation: treelite-runtime 2.3.0
    Not uninstalling treelite-runtime at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite-runtime'. No files were found to uninstall.
  Attempting uninstall: treelite
    Found existing installation: treelite 2.3.0
    Not uninstalling treelite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+21.gc73e818 treelite-2.4.0 treelite-runtime-2.4.0
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-yrmkoe_j
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-yrmkoe_j
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 3124b03033723de762d90d47085bac3191f37ff5
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+59.g3124b030) (0.3.0+12.g78ecddd)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.19.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.10.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.12.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.8.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.1)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+59.g3124b030-py3-none-any.whl size=366427 sha256=9a6557e9061e1a5d5cbbd604cc9658f91caa056968cd58bf4d9f745348b6c9c3
  Stored in directory: /tmp/pip-ephem-wheel-cache-3tljv_5s/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+59.g3124b030
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-nly3xl49
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-nly3xl49
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ee3b7440eb22fad71ad50a6df978e1fa526f4817
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.4 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+11.gee3b7440) (1.8.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+11.gee3b7440) (0.3.0+12.g78ecddd)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.19.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.10.0)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+11.gee3b7440) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.12.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.8.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.1)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+11.gee3b7440-cp38-cp38-linux_x86_64.whl size=257603 sha256=f65edd8a3ef93304a4bbfa7aa2a30d4e6e3e8e329cfb0df9afebbc39072a9a9c
  Stored in directory: /tmp/pip-ephem-wheel-cache-l3x6sv97/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=cbee12253b8b2ff615bac82a1fbebb62edb03fbf42e034476e75589c6da4aff6
  Stored in directory: /tmp/pip-ephem-wheel-cache-l3x6sv97/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+11.gee3b7440
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-zzb3j_q6
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-zzb3j_q6
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=4996243212666e41b204306308a207e467fa7a403e50562cee94d4fe0072c6c4
  Stored in directory: /tmp/pip-ephem-wheel-cache-e71lrauv/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py .      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    213    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    136    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     93    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    157    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    344    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     20    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     78    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     175     95    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 242    108    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                106     71    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                64     39    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               758    342    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     33    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       229     57    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     49    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               32      8    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77      3    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     19    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/fil.py                                 211    117    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     18    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             65     19    71%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/fil.py                  95     66    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           53      4    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 145     47    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143     74    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     13    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%


TOTAL                                                                                                 15293   7057    54%


======================== 3 passed in 327.02s (0:05:27) =========================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins5162650678089752415.sh

@nvidia-merlin-bot
Copy link
Contributor

nvidia-merlin-bot commented Nov 29, 2022

Click to view CI Results
GitHub pull request #741 of commit 1d942c800f5b48634fc75794d48aa2010c32d5e8, no merge conflicts.
Running as SYSTEM
Setting status of 1d942c800f5b48634fc75794d48aa2010c32d5e8 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/660/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/741/*:refs/remotes/origin/pr/741/* # timeout=10
 > git rev-parse 1d942c800f5b48634fc75794d48aa2010c32d5e8^{commit} # timeout=10
Checking out Revision 1d942c800f5b48634fc75794d48aa2010c32d5e8 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1d942c800f5b48634fc75794d48aa2010c32d5e8 # timeout=10
Commit message: "Bump `treelite` to 2.4.0"
 > git rev-list --no-walk d941a85143db2972ec08a5969eaa44ab1baba985 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins8702696895652706541.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.18,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.18,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3951540861'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-oy2qj_8y
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-oy2qj_8y
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit c73e81898c4c3ba10df019b9f36fa409a39d2626
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting treelite==2.4.0
  Downloading treelite-2.4.0-py3-none-manylinux2014_x86_64.whl (852 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.7/852.7 kB 8.4 MB/s eta 0:00:00
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+21.gc73e818) (2.22.0)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (0.3.0+12.g78ecddd)
Collecting treelite-runtime==2.4.0
  Downloading treelite_runtime-2.4.0-py3-none-manylinux2014_x86_64.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 149.3 MB/s eta 0:00:00
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.20.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.8.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.0.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.7.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.1.2)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (65.5.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 73.7 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+21.gc73e818-py3-none-any.whl size=98087 sha256=a1cbfc4d21a39e6c64563f64d6a73416d47180aa6a1d57b8b38cd93ac869ba87
  Stored in directory: /tmp/pip-ephem-wheel-cache-m9txp25q/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, treelite-runtime, treelite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: treelite-runtime
    Found existing installation: treelite-runtime 2.3.0
    Not uninstalling treelite-runtime at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite-runtime'. No files were found to uninstall.
  Attempting uninstall: treelite
    Found existing installation: treelite 2.3.0
    Not uninstalling treelite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+21.gc73e818 treelite-2.4.0 treelite-runtime-2.4.0
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-w63pbxkm
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-w63pbxkm
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 3124b03033723de762d90d47085bac3191f37ff5
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+59.g3124b030) (0.3.0+12.g78ecddd)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.0.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.7.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.1.2)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.1)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+59.g3124b030-py3-none-any.whl size=366427 sha256=a087cc201ddc468b6ecd38cbbf76acd251b7df1cac2c788db67417c7ccb441cf
  Stored in directory: /tmp/pip-ephem-wheel-cache-azyv75mx/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+59.g3124b030
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-4fg644hj
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-4fg644hj
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ee3b7440eb22fad71ad50a6df978e1fa526f4817
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.9 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+11.gee3b7440) (0.3.0+12.g78ecddd)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+11.gee3b7440) (1.8.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.55.1)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+11.gee3b7440) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.0.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.7.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.1.2)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.1)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+11.gee3b7440-cp38-cp38-linux_x86_64.whl size=257603 sha256=1d4937856f93fbebd5b89166d3c40f5fc6cc0bb2e17b67e94fb36baa91bbb673
  Stored in directory: /tmp/pip-ephem-wheel-cache-gf0wipj8/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=0b378adee5667540b6c7b3fd965b8380a1dc93ee5ebf607bfcbbd3732da5bb36
  Stored in directory: /tmp/pip-ephem-wheel-cache-gf0wipj8/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+11.gee3b7440
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-39bix46o
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-39bix46o
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=d0cbf691e3e76eb067ecbef35c84c739c7ba527a315a031ddaacd8d598ccb56b
  Stored in directory: /tmp/pip-ephem-wheel-cache-mrk03sxn/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]


=================================== FAILURES ===================================

__________________________________ test_func ___________________________________


self = <testbook.client.TestbookNotebookClient object at 0x7eff9a876d60>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:





          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)





../../../.local/lib/python3.8/site-packages/testbook/client.py:133:




args = (<testbook.client.TestbookNotebookClient object at 0x7eff9a876d60>, {'id': '7a25c2d3', 'cell_type': 'code', 'metadata'...erenceServerException\x1b[0m: [StatusCode.NOT_FOUND] Request for unknown model: 'ensemble_model' is not found"]}]}, 55)

kwargs = {}


def wrapped(*args, **kwargs):





  return just_run(coro(*args, **kwargs))





../../../.local/lib/python3.8/site-packages/nbclient/util.py:84:




coro = <coroutine object NotebookClient.async_execute_cell at 0x7efebc021ec0>


def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    # original from vaex/asyncio.py
    loop = asyncio._get_running_loop()
    if loop is None:
        had_running_loop = False
        try:
            loop = asyncio.get_event_loop()
        except RuntimeError:
            # we can still get 'There is no current event loop in ...'
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()





  return loop.run_until_complete(coro)





../../../.local/lib/python3.8/site-packages/nbclient/util.py:62:




self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-377' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/.local/lib...found\nInferenceServerException: [StatusCode.NOT_FOUND] Request for unknown model: 'ensemble_model' is not found\n')>


def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')





  return future.result()





/usr/lib/python3.8/asyncio/base_events.py:616:




self = <testbook.client.TestbookNotebookClient object at 0x7eff9a876d60>

cell = {'id': '7a25c2d3', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-29T23:47:48.075434Z',...1mInferenceServerException\x1b[0m: [StatusCode.NOT_FOUND] Request for unknown model: 'ensemble_model' is not found"]}]}

cell_index = 55, execution_count = None, store_history = True


async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count





  await self._check_raise_for_error(cell, cell_index, exec_reply)





../../../.local/lib/python3.8/site-packages/nbclient/client.py:965:




self = <testbook.client.TestbookNotebookClient object at 0x7eff9a876d60>

cell = {'id': '7a25c2d3', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-29T23:47:48.075434Z',...1mInferenceServerException\x1b[0m: [StatusCode.NOT_FOUND] Request for unknown model: 'ensemble_model' is not found"]}]}

cell_index = 55

exec_reply = {'buffers': [], 'content': {'ename': 'InferenceServerException', 'engine_info': {'engine_id': -1, 'engine_uuid': 'fdff...e, 'engine': 'fdff20a9-a45a-4340-859c-02ec1a4c2d05', 'started': '2022-11-29T23:47:48.075725Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E

E           import shutil

E           from merlin.core.dispatch import get_lib

E           from merlin.models.loader.tf_utils import configure_tensorflow

E           configure_tensorflow()

E           df_lib = get_lib()

E           batch = df_lib.read_parquet(

E               os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E               num_rows=1,

E               columns=["user_id_raw"],

E           )

E           from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E           response = run_ensemble_on_tritonserver(

E               "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E           )

E           response = [x.tolist()[0] for x in response["ordered_ids"]]

E           shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mInferenceServerException�[0m                  Traceback (most recent call last)

E           Cell �[0;32mIn [32], line 12�[0m

E           �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E           �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E           �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E           �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E           �[1;32m     10�[0m )

E           �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E           �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E           �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E           �[1;32m     14�[0m �[43m)�[49m

E           �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E           �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:139�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E           �[1;32m    137�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[1;32m    138�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E           �[0;32m--> 139�[0m     response �[38;5;241m=�[39m �[43msend_triton_request�[49m�[43m(�[49m

E           �[1;32m    140�[0m �[43m        �[49m�[43mschema�[49m�[43m,�[49m�[43m �[49m�[43mdf�[49m�[43m,�[49m�[43m �[49m�[43moutput_columns�[49m�[43m,�[49m�[43m �[49m�[43mclient�[49m�[38;5;241;43m=�[39;49m�[43mclient�[49m�[43m,�[49m�[43m �[49m�[43mtriton_model�[49m�[38;5;241;43m=�[39;49m�[43mmodel_name�[49m

E           �[1;32m    141�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m    143�[0m �[38;5;28;01mreturn�[39;00m response

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:192�[0m, in �[0;36msend_triton_request�[0;34m(schema, df, outputs_list, client, endpoint, request_id, triton_model)�[0m

E           �[1;32m    190�[0m outputs �[38;5;241m=�[39m [grpcclient�[38;5;241m.�[39mInferRequestedOutput(col) �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list]

E           �[1;32m    191�[0m �[38;5;28;01mwith�[39;00m client:

E           �[0;32m--> 192�[0m     response �[38;5;241m=�[39m �[43mclient�[49m�[38;5;241;43m.�[39;49m�[43minfer�[49m�[43m(�[49m�[43mtriton_model�[49m�[43m,�[49m�[43m �[49m�[43minputs�[49m�[43m,�[49m�[43m �[49m�[43mrequest_id�[49m�[38;5;241;43m=�[39;49m�[43mrequest_id�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[38;5;241;43m=�[39;49m�[43moutputs�[49m�[43m)�[49m

E           �[1;32m    194�[0m results �[38;5;241m=�[39m {}

E           �[1;32m    195�[0m �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list:

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:1322�[0m, in �[0;36mInferenceServerClient.infer�[0;34m(self, model_name, inputs, model_version, outputs, request_id, sequence_id, sequence_start, sequence_end, priority, timeout, client_timeout, headers, compression_algorithm)�[0m

E           �[1;32m   1320�[0m     �[38;5;28;01mreturn�[39;00m result

E           �[1;32m   1321�[0m �[38;5;28;01mexcept�[39;00m grpc�[38;5;241m.�[39mRpcError �[38;5;28;01mas�[39;00m rpc_error:

E           �[0;32m-> 1322�[0m     �[43mraise_error_grpc�[49m�[43m(�[49m�[43mrpc_error�[49m�[43m)�[49m

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:62�[0m, in �[0;36mraise_error_grpc�[0;34m(rpc_error)�[0m

E           �[1;32m     61�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mraise_error_grpc�[39m(rpc_error):

E           �[0;32m---> 62�[0m     �[38;5;28;01mraise�[39;00m get_error_grpc(rpc_error) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E           �[0;31mInferenceServerException�[0m: [StatusCode.NOT_FOUND] Request for unknown model: 'ensemble_model' is not found

E           InferenceServerException: [StatusCode.NOT_FOUND] Request for unknown model: 'ensemble_model' is not found


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError


During handling of the above exception, another exception occurred:


def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=180,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"





      tb2.inject(





            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )



tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:




../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)




self = <testbook.client.TestbookNotebookClient object at 0x7eff9a876d60>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:





          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))





E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E

E               import shutil

E               from merlin.core.dispatch import get_lib

E               from merlin.models.loader.tf_utils import configure_tensorflow

E               configure_tensorflow()

E               df_lib = get_lib()

E               batch = df_lib.read_parquet(

E                   os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E                   num_rows=1,

E                   columns=["user_id_raw"],

E               )

E               from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E               response = run_ensemble_on_tritonserver(

E                   "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E               )

E               response = [x.tolist()[0] for x in response["ordered_ids"]]

E               shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mInferenceServerException�[0m                  Traceback (most recent call last)

E               Cell �[0;32mIn [32], line 12�[0m

E               �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E               �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E               �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E               �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E               �[1;32m     10�[0m )

E               �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E               �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E               �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E               �[1;32m     14�[0m �[43m)�[49m

E               �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E               �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:139�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E               �[1;32m    137�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E               �[1;32m    138�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E               �[0;32m--> 139�[0m     response �[38;5;241m=�[39m �[43msend_triton_request�[49m�[43m(�[49m

E               �[1;32m    140�[0m �[43m        �[49m�[43mschema�[49m�[43m,�[49m�[43m �[49m�[43mdf�[49m�[43m,�[49m�[43m �[49m�[43moutput_columns�[49m�[43m,�[49m�[43m �[49m�[43mclient�[49m�[38;5;241;43m=�[39;49m�[43mclient�[49m�[43m,�[49m�[43m �[49m�[43mtriton_model�[49m�[38;5;241;43m=�[39;49m�[43mmodel_name�[49m

E               �[1;32m    141�[0m �[43m    �[49m�[43m)�[49m

E               �[1;32m    143�[0m �[38;5;28;01mreturn�[39;00m response

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:192�[0m, in �[0;36msend_triton_request�[0;34m(schema, df, outputs_list, client, endpoint, request_id, triton_model)�[0m

E               �[1;32m    190�[0m outputs �[38;5;241m=�[39m [grpcclient�[38;5;241m.�[39mInferRequestedOutput(col) �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list]

E               �[1;32m    191�[0m �[38;5;28;01mwith�[39;00m client:

E               �[0;32m--> 192�[0m     response �[38;5;241m=�[39m �[43mclient�[49m�[38;5;241;43m.�[39;49m�[43minfer�[49m�[43m(�[49m�[43mtriton_model�[49m�[43m,�[49m�[43m �[49m�[43minputs�[49m�[43m,�[49m�[43m �[49m�[43mrequest_id�[49m�[38;5;241;43m=�[39;49m�[43mrequest_id�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[38;5;241;43m=�[39;49m�[43moutputs�[49m�[43m)�[49m

E               �[1;32m    194�[0m results �[38;5;241m=�[39m {}

E               �[1;32m    195�[0m �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list:

E

E               File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:1322�[0m, in �[0;36mInferenceServerClient.infer�[0;34m(self, model_name, inputs, model_version, outputs, request_id, sequence_id, sequence_start, sequence_end, priority, timeout, client_timeout, headers, compression_algorithm)�[0m

E               �[1;32m   1320�[0m     �[38;5;28;01mreturn�[39;00m result

E               �[1;32m   1321�[0m �[38;5;28;01mexcept�[39;00m grpc�[38;5;241m.�[39mRpcError �[38;5;28;01mas�[39;00m rpc_error:

E               �[0;32m-> 1322�[0m     �[43mraise_error_grpc�[49m�[43m(�[49m�[43mrpc_error�[49m�[43m)�[49m

E

E               File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:62�[0m, in �[0;36mraise_error_grpc�[0;34m(rpc_error)�[0m

E               �[1;32m     61�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mraise_error_grpc�[39m(rpc_error):

E               �[0;32m---> 62�[0m     �[38;5;28;01mraise�[39;00m get_error_grpc(rpc_error) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E               �[0;31mInferenceServerException�[0m: [StatusCode.NOT_FOUND] Request for unknown model: 'ensemble_model' is not found

E               InferenceServerException: [StatusCode.NOT_FOUND] Request for unknown model: 'ensemble_model' is not found


../../../.local/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError

----------------------------- Captured stderr call -----------------------------

2022-11-29 23:46:31.905632: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-29 23:46:36.062047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-29 23:46:36.062770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-29 23:46:36.063425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-29 23:46:36.064069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

2022-11-29 23:46:36.071678: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 7.95G (8534360064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2022-11-29 23:46:36.074030: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 7.15G (7680923648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2022-11-29 23:46:36.075147: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 6.44G (6912830976 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2022-11-29 23:46:36.076314: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 5.79G (6221547520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

WARNING clustering 461 points to 32 centroids: please provide at least 1248 training points

2022-11-29 23:47:41.153754: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-29 23:47:45.364432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-29 23:47:45.365332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-29 23:47:45.367130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-29 23:47:45.368332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

__________________________________ test_func ___________________________________


def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )





      tb1.execute()





tests/unit/examples/test_scaling_criteo_merlin_models.py:33:




../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)




self = <testbook.client.TestbookNotebookClient object at 0x7eff9a5ac2e0>

cell = {'cell_type': 'code', 'execution_count': 7, 'metadata': {'jupyter': {'outputs_hidden': False}, 'execution': {'iopub.st...pool_size=(device_pool_size // 256) * 256\n    )\n\n# Create the distributed client\nclient = Client(cluster)\nclient'}

cell_index = 12

exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': '51e742ce-df92-4da...e, 'engine': '51e742ce-df92-4dab-b252-17fdd20aea6e', 'started': '2022-11-29T23:47:57.149523Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           # Dask dashboard

E           dashboard_port = "8787"

E

E           # Deploy a Single-Machine Multi-GPU Cluster

E           protocol = "tcp"  # "tcp" or "ucx"

E           if numba.cuda.is_available():

E               NUM_GPUS = list(range(len(numba.cuda.gpus)))

E           else:

E               NUM_GPUS = []

E           visible_devices = ",".join([str(n) for n in NUM_GPUS])  # Select devices to place workers

E           device_limit_frac = 0.7  # Spill GPU-Worker memory to host at this limit.

E           device_pool_frac = 0.8

E           part_mem_frac = 0.15

E

E           # Use total device size to calculate args.device_limit_frac

E           device_size = device_mem_size(kind="total")

E           device_limit = int(device_limit_frac * device_size)

E           device_pool_size = int(device_pool_frac * device_size)

E           part_size = int(part_mem_frac * device_size)

E

E           # Check if any device memory is already occupied

E           for dev in visible_devices.split(","):

E               fmem = pynvml_mem_size(kind="free", index=int(dev))

E               used = (device_size - fmem) / 1e9

E               if used > 1.0:

E                   warnings.warn(f"BEWARE - {used} GB is already occupied on device {int(dev)}!")

E

E           cluster = None  # (Optional) Specify existing scheduler port

E           if cluster is None:

E               cluster = LocalCUDACluster(

E                   protocol=protocol,

E                   n_workers=len(visible_devices.split(",")),

E                   CUDA_VISIBLE_DEVICES=visible_devices,

E                   device_memory_limit=device_limit,

E                   local_directory=dask_workdir,

E                   dashboard_address=":" + dashboard_port,

E                   rmm_pool_size=(device_pool_size // 256) * 256

E               )

E

E           # Create the distributed client

E           client = Client(cluster)

E           client

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mMemoryError�[0m                               Traceback (most recent call last)

E           Cell �[0;32mIn [7], line 30�[0m

E           �[1;32m     28�[0m cluster �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m  �[38;5;66;03m# (Optional) Specify existing scheduler port�[39;00m

E           �[1;32m     29�[0m �[38;5;28;01mif�[39;00m cluster �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m:

E           �[0;32m---> 30�[0m     cluster �[38;5;241m=�[39m �[43mLocalCUDACluster�[49m�[43m(�[49m

E           �[1;32m     31�[0m �[43m        �[49m�[43mprotocol�[49m�[38;5;241;43m=�[39;49m�[43mprotocol�[49m�[43m,�[49m

E           �[1;32m     32�[0m �[43m        �[49m�[43mn_workers�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mlen�[39;49m�[43m(�[49m�[43mvisible_devices�[49m�[38;5;241;43m.�[39;49m�[43msplit�[49m�[43m(�[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m,�[39;49m�[38;5;124;43m"�[39;49m�[43m)�[49m�[43m)�[49m�[43m,�[49m

E           �[1;32m     33�[0m �[43m        �[49m�[43mCUDA_VISIBLE_DEVICES�[49m�[38;5;241;43m=�[39;49m�[43mvisible_devices�[49m�[43m,�[49m

E           �[1;32m     34�[0m �[43m        �[49m�[43mdevice_memory_limit�[49m�[38;5;241;43m=�[39;49m�[43mdevice_limit�[49m�[43m,�[49m

E           �[1;32m     35�[0m �[43m        �[49m�[43mlocal_directory�[49m�[38;5;241;43m=�[39;49m�[43mdask_workdir�[49m�[43m,�[49m

E           �[1;32m     36�[0m �[43m        �[49m�[43mdashboard_address�[49m�[38;5;241;43m=�[39;49m�[38;5;124;43m"�[39;49m�[38;5;124;43m:�[39;49m�[38;5;124;43m"�[39;49m�[43m �[49m�[38;5;241;43m+�[39;49m�[43m �[49m�[43mdashboard_port�[49m�[43m,�[49m

E           �[1;32m     37�[0m �[43m        �[49m�[43mrmm_pool_size�[49m�[38;5;241;43m=�[39;49m�[43m(�[49m�[43mdevice_pool_size�[49m�[43m �[49m�[38;5;241;43m/�[39;49m�[38;5;241;43m/�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m�[43m)�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m

E           �[1;32m     38�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m     40�[0m �[38;5;66;03m# Create the distributed client�[39;00m

E           �[1;32m     41�[0m client �[38;5;241m=�[39m Client(cluster)

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/local_cuda_cluster.py:367�[0m, in �[0;36mLocalCUDACluster.__init__�[0;34m(self, CUDA_VISIBLE_DEVICES, n_workers, threads_per_worker, memory_limit, device_memory_limit, data, local_directory, shared_filesystem, protocol, enable_tcp_over_ucx, enable_infiniband, enable_nvlink, enable_rdmacm, rmm_pool_size, rmm_maximum_pool_size, rmm_managed_memory, rmm_async, rmm_log_directory, rmm_track_allocations, jit_unspill, log_spilling, worker_class, pre_import, **kwargs)�[0m

E           �[1;32m    365�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mcuda_visible_devices �[38;5;241m=�[39m CUDA_VISIBLE_DEVICES

E           �[1;32m    366�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mscale(n_workers)

E           �[0;32m--> 367�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43msync�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_correct_state�[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:309�[0m, in �[0;36mSyncMethodMixin.sync�[0;34m(self, func, asynchronous, callback_timeout, args, **kwargs)�[0m

E           �[1;32m    307�[0m     �[38;5;28;01mreturn�[39;00m future

E           �[1;32m    308�[0m �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 309�[0m     �[38;5;28;01mreturn�[39;00m �[43msync�[49m�[43m(�[49m

E           �[1;32m    310�[0m �[43m        �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mloop�[49m�[43m,�[49m�[43m �[49m�[43mfunc�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[43mcallback_timeout�[49m�[38;5;241;43m=�[39;49m�[43mcallback_timeout�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m*�[39;49m�[43mkwargs�[49m

E           �[1;32m    311�[0m �[43m    �[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:376�[0m, in �[0;36msync�[0;34m(loop, func, callback_timeout, *args, **kwargs)�[0m

E           �[1;32m    374�[0m �[38;5;28;01mif�[39;00m error:

E           �[1;32m    375�[0m     typ, exc, tb �[38;5;241m=�[39m error

E           �[0;32m--> 376�[0m     �[38;5;28;01mraise�[39;00m exc�[38;5;241m.�[39mwith_traceback(tb)

E           �[1;32m    377�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    378�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:349�[0m, in �[0;36msync..f�[0;34m()�[0m

E           �[1;32m    347�[0m         future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mwait_for(future, callback_timeout)

E           �[1;32m    348�[0m     future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mensure_future(future)

E           �[0;32m--> 349�[0m     result �[38;5;241m=�[39m �[38;5;28;01myield�[39;00m future

E           �[1;32m    350�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    351�[0m     error �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762�[0m, in �[0;36mRunner.run�[0;34m(self)�[0m

E           �[1;32m    759�[0m exc_info �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[1;32m    761�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 762�[0m     value �[38;5;241m=�[39m �[43mfuture�[49m�[38;5;241;43m.�[39;49m�[43mresult�[49m�[43m(�[49m�[43m)�[49m

E           �[1;32m    763�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    764�[0m     exc_info �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/deploy/spec.py:352�[0m, in �[0;36mSpecCluster._correct_state_internal�[0;34m(self)�[0m

E           �[1;32m    350�[0m     �[38;5;28;01mfor�[39;00m w �[38;5;129;01min�[39;00m workers:

E           �[1;32m    351�[0m         w�[38;5;241m.�[39m_cluster �[38;5;241m=�[39m weakref�[38;5;241m.�[39mref(�[38;5;28mself�[39m)

E           �[0;32m--> 352�[0m         �[38;5;28;01mawait�[39;00m w  �[38;5;66;03m# for tornado gen.coroutine support�[39;00m

E           �[1;32m    353�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mworkers�[38;5;241m.�[39mupdate(�[38;5;28mdict�[39m(�[38;5;28mzip�[39m(to_open, workers)))

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36mServer.await.._�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:347�[0m, in �[0;36mNanny.start�[0;34m(self)�[0m

E           �[1;32m    344�[0m     �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mplugin_add(plugin�[38;5;241m=�[39mplugin, name�[38;5;241m=�[39mname)

E           �[1;32m    346�[0m logger�[38;5;241m.�[39minfo(�[38;5;124m"�[39m�[38;5;124m        Start Nanny at: �[39m�[38;5;132;01m%r�[39;00m�[38;5;124m"�[39m, �[38;5;28mself�[39m�[38;5;241m.�[39maddress)

E           �[0;32m--> 347�[0m response �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minstantiate()

E           �[1;32m    348�[0m �[38;5;28;01mif�[39;00m response �[38;5;241m==�[39m Status�[38;5;241m.�[39mrunning:

E           �[1;32m    349�[0m     �[38;5;28;01massert�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mworker_address

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:430�[0m, in �[0;36mNanny.instantiate�[0;34m(self)�[0m

E           �[1;32m    428�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    429�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 430�[0m         result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mprocess�[38;5;241m.�[39mstart()

E           �[1;32m    431�[0m     �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    432�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mclose()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:685�[0m, in �[0;36mWorkerProcess.start�[0;34m(self)�[0m

E           �[1;32m    683�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus

E           �[1;32m    684�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 685�[0m     msg �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_wait_until_connected(uid)

E           �[1;32m    686�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    687�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mfailed

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:803�[0m, in �[0;36mWorkerProcess._wait_until_connected�[0;34m(self, uid)�[0m

E           �[1;32m    799�[0m �[38;5;28;01mif�[39;00m �[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m �[38;5;129;01min�[39;00m msg:

E           �[1;32m    800�[0m     logger�[38;5;241m.�[39merror(

E           �[1;32m    801�[0m         �[38;5;124m"�[39m�[38;5;124mFailed while trying to start worker process: �[39m�[38;5;132;01m%s�[39;00m�[38;5;124m"�[39m, msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    802�[0m     )

E           �[0;32m--> 803�[0m     �[38;5;28;01mraise�[39;00m msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    804�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    805�[0m     �[38;5;28;01mreturn�[39;00m msg

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:869�[0m, in �[0;36mrun�[0;34m()�[0m

E           �[1;32m    865�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    866�[0m �[38;5;124;03mTry to start worker and inform parent of outcome.�[39;00m

E           �[1;32m    867�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    868�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 869�[0m     �[38;5;28;01mawait�[39;00m worker

E           �[1;32m    870�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m �[38;5;28;01mas�[39;00m e:

E           �[1;32m    871�[0m     logger�[38;5;241m.�[39mexception(�[38;5;124m"�[39m�[38;5;124mFailed to start worker�[39m�[38;5;124m"�[39m)

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36m_�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:1372�[0m, in �[0;36mstart�[0;34m()�[0m

E           �[1;32m   1370�[0m         �[38;5;28;01mfor�[39;00m exc �[38;5;129;01min�[39;00m plugins_exceptions:

E           �[1;32m   1371�[0m             logger�[38;5;241m.�[39merror(�[38;5;28mrepr�[39m(exc))

E           �[0;32m-> 1372�[0m     �[38;5;28;01mraise�[39;00m plugins_exceptions[�[38;5;241m0�[39m]

E           �[1;32m   1374�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_pending_plugins �[38;5;241m=�[39m ()

E           �[1;32m   1376�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_register_with_scheduler()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:3248�[0m, in �[0;36mplugin_add�[0;34m()�[0m

E           �[1;32m   3246�[0m �[38;5;28;01mif�[39;00m �[38;5;28mhasattr�[39m(plugin, �[38;5;124m"�[39m�[38;5;124msetup�[39m�[38;5;124m"�[39m):

E           �[1;32m   3247�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m-> 3248�[0m         result �[38;5;241m=�[39m plugin�[38;5;241m.�[39msetup(worker�[38;5;241m=�[39m�[38;5;28mself�[39m)

E           �[1;32m   3249�[0m         �[38;5;28;01mif�[39;00m isawaitable(result):

E           �[1;32m   3250�[0m             result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py:78�[0m, in �[0;36msetup�[0;34m()�[0m

E           �[1;32m     74�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E           �[1;32m     76�[0m     pool_allocator �[38;5;241m=�[39m �[38;5;28;01mFalse�[39;00m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m �[38;5;28;01melse�[39;00m �[38;5;28;01mTrue�[39;00m

E           �[0;32m---> 78�[0m     rmm�[38;5;241m.�[39mreinitialize(

E           �[1;32m     79�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     80�[0m         managed_memory�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmanaged_memory,

E           �[1;32m     81�[0m         initial_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size,

E           �[1;32m     82�[0m         maximum_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmaximum_pool_size,

E           �[1;32m     83�[0m         logging�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mlogging,

E           �[1;32m     84�[0m         log_file_name�[38;5;241m=�[39mget_rmm_log_file_name(

E           �[1;32m     85�[0m             worker, �[38;5;28mself�[39m�[38;5;241m.�[39mlogging, �[38;5;28mself�[39m�[38;5;241m.�[39mlog_directory

E           �[1;32m     86�[0m         ),

E           �[1;32m     87�[0m     )

E           �[1;32m     88�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mrmm_track_allocations:

E           �[1;32m     89�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/rmm/rmm.py:85�[0m, in �[0;36mreinitialize�[0;34m()�[0m

E           �[1;32m     32�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mreinitialize�[39m(

E           �[1;32m     33�[0m     pool_allocator�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[1;32m     34�[0m     managed_memory�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[0;32m   (...)�[0m

E           �[1;32m     39�[0m     log_file_name�[38;5;241m=�[39m�[38;5;28;01mNone�[39;00m,

E           �[1;32m     40�[0m ):

E           �[1;32m     41�[0m     �[38;5;124;03m"""�[39;00m

E           �[1;32m     42�[0m �[38;5;124;03m    Finalizes and then initializes RMM using the options passed. Using memory�[39;00m

E           �[1;32m     43�[0m �[38;5;124;03m    from a previous initialization of RMM is undefined behavior and should be�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m     83�[0m �[38;5;124;03m    corresponding to each device.�[39;00m

E           �[1;32m     84�[0m �[38;5;124;03m    """�[39;00m

E           �[0;32m---> 85�[0m     rmm�[38;5;241m.�[39mmr�[38;5;241m.�[39m_initialize(

E           �[1;32m     86�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     87�[0m         managed_memory�[38;5;241m=�[39mmanaged_memory,

E           �[1;32m     88�[0m         initial_pool_size�[38;5;241m=�[39minitial_pool_size,

E           �[1;32m     89�[0m         maximum_pool_size�[38;5;241m=�[39mmaximum_pool_size,

E           �[1;32m     90�[0m         devices�[38;5;241m=�[39mdevices,

E           �[1;32m     91�[0m         logging�[38;5;241m=�[39mlogging,

E           �[1;32m     92�[0m         log_file_name�[38;5;241m=�[39mlog_file_name,

E           �[1;32m     93�[0m     )

E

E           File �[0;32mmemory_resource.pyx:823�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:883�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:342�[0m, in �[0;36mrmm._lib.memory_resource.PoolMemoryResource.__cinit__�[0;34m()�[0m

E

E           �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

E           MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-11-29 23:48:00,125 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 23:48:00,139 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 23:48:00,198 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 23:48:00,203 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 23:48:00,610 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:00,611 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:03,405 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-1ae3ejfz', purging

2022-11-29 23:48:03,406 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 23:48:03,677 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:03,678 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm.lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:06,524 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-89j_bbz', purging

2022-11-29 23:48:06,525 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 23:48:06,550 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 23:48:06,585 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 23:48:06,604 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 23:48:06,946 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:06,947 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:07,087 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:07,088 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:07,209 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:07,210 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:07,210 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:07,211 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:09,755 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-00ul01je', purging

2022-11-29 23:48:09,755 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-cllfykoy', purging

2022-11-29 23:48:09,755 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-yvwfh5re', purging

2022-11-29 23:48:09,755 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-m15n2xei', purging

2022-11-29 23:48:09,756 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 23:48:10,020 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-29 23:48:10,021 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 57 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    213    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     17    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     41    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    163    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    164    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    345    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     17    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    114    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49     15    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     175     95    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    111    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 242    114    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    161    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                106     71    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     84    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                64     39    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39     16    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    162    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               758    357    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    110    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     24    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     33    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     40    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       229     58    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     74    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     21    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               32      8    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77     23    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     54    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/fil.py                                 211    117    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51     17    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     20    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50     15    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/fil.py                  95     66    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           53      5    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 145     47    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     17    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143    120    16%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    234    13%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     23    68%


TOTAL                                                                                                 15014   7334    51%


=================== 2 failed, 1 passed in 117.49s (0:01:57) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins14197453914791870574.sh

@karlhigley karlhigley closed this Apr 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jperez999 jperez999 jperez999 approved these changes

+1 more reviewer

@oliverholworthy oliverholworthy oliverholworthy left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@karlhigley karlhigley

Labels

chore Infrastructure update ci

Projects

None yet

Milestone

Merlin 22.12

Development

Successfully merging this pull request may close these issues.

5 participants

@karlhigley @nvidia-merlin-bot @oliverholworthy @jperez999