[python-package] Completely different training result using exact same data and parameters

[dameiqinghai]

Description

I have 2 server, and I got completely differnt results when training using exact same data and parameters. The lgbm version are both v4.5.0. I found "Stopped training because there are no more leaves that meet the split requirements" in server 2. Which i believe is the reason for the huge difference, i set num_trees = 1000, and in server 2 dumped model we got n_trees is ~ 400(results in server 1 looks just fine).

Reproducible example

Environment info

LightGBM version or commit hash:

v4.5.0

Command(s) you used to install LightGBM

pip install lightgbm

One is python 3.12 and another is 3.11

Additional Comments

[jameslamb]

Thanks for using LightGBM.

We'll need a lot more information before anyone can help you.

If you provide the following, we'll try to help you:

• operating system on each server?
• versions of dependencies?
  • run pip freeze
• exact code you are running?
  • (made as minimal as possible, see "How to create a Minimal, Reproducible Example"
  • ideally using a publicly-accessible dataset, like those from sklearn.datasets... see for example deterministic and related flags don't guarantee same result on different machines #6683 (comment)

[dameiqinghai]

• operating system on each server?
  Both rhel 8.10
• versions of dependencies?

pip_freeze_server2.log

pip_freeze_server1.log

• exact code you are running?
  Sorry i can't share the source code, the training part is enssentially:
  from lightgbm import LGBMRegressor a = LGBMRegressor() a.fit()
  and non-default params:
  {n_estimators: 1000, max_depth: 8, num_leaves: 120, learning_rate: 0.04, subsample: 0.46, colsample_bytree: 0.3, min_child_samples: 8400, n_jobs: 10, max_bin: 15, device_type: 'cpu', force_row_wise: true, lambda_l2: 193, min_sum_hessian_in_leaf: 0.002, lambda_l1: 96}

[jameslamb]

Thanks.

Ok, if you're not willing to put together a minimal, reproducible example we can try to work with that information.

I put those logs through diff to see the difference:

diff -u pip_freeze_server1.log pip_freeze_server2.log

diff output (click me)

--- pip_freeze_server1.log	2025-01-19 21:42:56
+++ pip_freeze_server2.log	2025-01-19 21:40:42
@@ -1,304 +1,430 @@
-aiohttp @ file:///home/conda/feedstock_root/build_artifacts/aiohttp_1710511549044/work
-aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work
-aiosqlite @ file:///home/conda/feedstock_root/build_artifacts/aiosqlite_1682491975081/work
-alembic @ file:///home/conda/feedstock_root/build_artifacts/alembic_1705179948871/work
-amqp @ file:///home/conda/feedstock_root/build_artifacts/amqp_1703885206855/work
-aniso8601 @ file:///home/conda/feedstock_root/build_artifacts/aniso8601_1618789466884/work
-annotated-types @ file:///home/conda/feedstock_root/build_artifacts/annotated-types_1696634205638/work
-ansiwrap==0.8.4
-anyio==4.4.0
-apprise==1.9.0
-argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1692818318753/work
-argon2-cffi-bindings @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi-bindings_1695386553988/work
-arrow @ file:///home/conda/feedstock_root/build_artifacts/arrow_1696128962909/work
-asgi-lifespan @ file:///home/conda/feedstock_root/build_artifacts/asgi-lifespan_1680033598692/work
-astor==0.8.1
-asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1698341106958/work
-astunparse @ file:///home/conda/feedstock_root/build_artifacts/astunparse_1610696312422/work
-async-exit-stack @ file:///home/conda/feedstock_root/build_artifacts/async-exit-stack_1617751130790/work
-async-lru @ file:///home/conda/feedstock_root/build_artifacts/async-lru_1690563019058/work
-async-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1691763562544/work
-asyncpg @ file:///home/conda/feedstock_root/build_artifacts/asyncpg_1700296476802/work
-attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1704011227531/work
-Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1702422572539/work
-bcrypt @ file:///home/conda/feedstock_root/build_artifacts/bcrypt_1702663837749/work
-beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1705564648255/work
-billiard @ file:///home/conda/feedstock_root/build_artifacts/billiard_1703820048184/work
-black @ file:///home/conda/feedstock_root/build_artifacts/black-recipe_1708248203050/work
-bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1696630167146/work
-blessed==1.20.0
-blinker @ file:///home/conda/feedstock_root/build_artifacts/blinker_1698890160476/work
-boto3 @ file:///home/conda/feedstock_root/build_artifacts/boto3_1710538318103/work
-botocore @ file:///home/conda/feedstock_root/build_artifacts/botocore_1710535860374/work
-Brotli @ file:///home/conda/feedstock_root/build_artifacts/brotli-split_1695989787169/work
-build==1.2.1
-cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
-cachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1708987703938/work
-catboost==1.2.3
-category-encoders @ file:///home/conda/feedstock_root/build_artifacts/category_encoders_1698628756523/work/dist
-celery @ file:///home/conda/feedstock_root/build_artifacts/celery_1703905883217/work
-certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1725278078093/work/certifi
-cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1696001724357/work
-chardet @ file:///home/conda/feedstock_root/build_artifacts/chardet_1695468614576/work
-charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1698833585322/work
-click @ file:///home/conda/feedstock_root/build_artifacts/click_1692311806742/work
-click-didyoumean @ file:///home/conda/feedstock_root/build_artifacts/click-didyoumean_1642673782881/work
-click-plugins==1.1.1
-click-repl @ file:///home/conda/feedstock_root/build_artifacts/click-repl_1694959444233/work
-cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1674202310934/work
-colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1666700638685/work
-comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1710320294760/work
-contextlib2==21.6.0
-contourpy @ file:///home/conda/feedstock_root/build_artifacts/contourpy_1699041375599/work
-coolname @ file:///home/conda/feedstock_root/build_artifacts/coolname_1673305982850/work
-croniter @ file:///home/conda/feedstock_root/build_artifacts/croniter_1686929181238/work
-cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography-split_1708780250343/work
-cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1696677705766/work
-dash @ file:///home/conda/feedstock_root/build_artifacts/dash_1710075327199/work
-dash-bootstrap-components @ file:///home/conda/feedstock_root/build_artifacts/dash-bootstrap-components_1694086999250/work
-dash-core-components @ file:///home/conda/feedstock_root/build_artifacts/dash-core-components_1645162191927/work
-dash-daq @ file:///home/conda/feedstock_root/build_artifacts/dash-daq_1596869611219/work
-dash-html-components @ file:///home/conda/feedstock_root/build_artifacts/dash-html-components_1645162180756/work
-dash-renderer @ file:///home/conda/feedstock_root/build_artifacts/dash-renderer_1604054245287/work
-dash-table @ file:///home/conda/feedstock_root/build_artifacts/dash-table_1645162199872/work
-dataclasses @ file:///home/conda/feedstock_root/build_artifacts/dataclasses_1628958434797/work
-dateparser @ file:///home/conda/feedstock_root/build_artifacts/dateparser_1700222000722/work
-debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1707444446407/work
-decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
-defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
-diff-match-patch==20230430
-dnspython==2.6.1
-docker @ file:///home/conda/feedstock_root/build_artifacts/docker-py_1688805481281/work
-docopt==0.6.2
-email_validator==2.1.1
-entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work
+aiobotocore @ file:///croot/aiobotocore_1714464387507/work
+aiohappyeyeballs @ file:///croot/aiohappyeyeballs_1725434011349/work
+aiohttp @ file:///croot/aiohttp_1725527756643/work
+aiohttp-cors==0.7.0
+aioitertools @ file:///tmp/build/80754af9/aioitertools_1607109665762/work
+aiosignal @ file:///tmp/build/80754af9/aiosignal_1637843061372/work
+aiosqlite==0.20.0
+alabaster @ file:///croot/alabaster_1718201490751/work
+alembic==1.14.0
+altair @ file:///work/perseverance-python-buildout/croot/altair_1698866618831/work
+anaconda-catalogs @ file:///work/perseverance-python-buildout/croot/anaconda-catalogs_1701755653155/work
+annotated-types @ file:///croot/annotated-types_1709542908624/work
+anyio==4.8.0
+appdirs==1.4.4
+apprise==1.9.2
+archspec @ file:///croot/archspec_1709217642129/work
+argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work
+argon2-cffi-bindings @ file:///work/perseverance-python-buildout/croot/argon2-cffi-bindings_1698877183685/work
+arrow @ file:///work/perseverance-python-buildout/croot/arrow_1698877225835/work
+asgi-lifespan==2.1.0
+astroid @ file:///work/perseverance-python-buildout/croot/astroid_1698851155125/work
+astropy @ file:///croot/astropy_1726174598094/work
+astropy-iers-data @ file:///croot/astropy-iers-data_1726000528288/work
+asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work
+async-lru @ file:///work/perseverance-python-buildout/croot/async-lru_1701732681408/work
+asyncpg==0.30.0
+atomicwrites==1.4.0
+attrs @ file:///work/perseverance-python-buildout/croot/attrs_1698845858439/work
+Automat @ file:///tmp/build/80754af9/automat_1600298431173/work
+autopep8 @ file:///croot/autopep8_1708962882016/work
+Babel @ file:///work/perseverance-python-buildout/croot/babel_1698863996602/work
+bayesian-optimization==1.4.1
+bcrypt @ file:///work/perseverance-python-buildout/croot/bcrypt_1698864043363/work
+beautifulsoup4 @ file:///croot/beautifulsoup4-split_1718029820055/work
+binaryornot @ file:///tmp/build/80754af9/binaryornot_1617751525010/work
+black @ file:///croot/black_1725573853246/work
+bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work
+blinker @ file:///work/perseverance-python-buildout/croot/blinker_1698846408677/work
+bokeh @ file:///croot/bokeh_1727914487135/work
+boltons @ file:///work/perseverance-python-buildout/croot/boltons_1698851177130/work
+botocore @ file:///croot/botocore_1714460531862/work
+Bottleneck @ file:///croot/bottleneck_1709069899917/work
+Brotli @ file:///croot/brotli-split_1714483155106/work
+cachetools @ file:///croot/cachetools_1713977085444/work
+catboost==1.2.7
+certifi @ file:///croot/certifi_1725551672989/work/certifi
+cffi @ file:///croot/cffi_1726856441404/work
+chardet @ file:///work/perseverance-python-buildout/croot/chardet_1698867756032/work
+charset-normalizer @ file:///croot/charset-normalizer_1721748349566/work
+click @ file:///work/perseverance-python-buildout/croot/click_1698845879718/work
+cloudpickle @ file:///croot/cloudpickle_1721657346512/work
+colorama @ file:///work/perseverance-python-buildout/croot/colorama_1698847155966/work
+colorcet @ file:///croot/colorcet_1709758348067/work
+colorful==0.5.6
+colorlog==6.9.0
+comm @ file:///croot/comm_1709322850197/work
+conda-content-trust @ file:///croot/conda-content-trust_1714483159009/work
+conda-pack @ file:///croot/conda-pack_1710258022278/work
+conda-package-handling @ file:///croot/conda-package-handling_1718138267740/work
+conda-repo-cli @ file:///croot/conda-repo-cli_1727366904598/work
+conda_package_streaming @ file:///croot/conda-package-streaming_1718136078615/work
+constantly @ file:///croot/constantly_1703165600746/work
+contourpy @ file:///work/perseverance-python-buildout/croot/contourpy_1701756524386/work
+cookiecutter @ file:///croot/cookiecutter_1711059812130/work
+coolname==2.2.0
+croniter==5.0.1
+cryptography @ file:///croot/cryptography_1724940562255/work
+cssselect @ file:///croot/cssselect_1707339882883/work
+cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
+cytoolz @ file:///croot/cytoolz_1701723583781/work
+dask @ file:///croot/dask-core_1725461349163/work
+dask-expr @ file:///croot/dask-expr_1725522981173/work
+datashader @ file:///croot/datashader_1720533861326/work
+dateparser==1.2.0
+debugpy @ file:///work/perseverance-python-buildout/croot/debugpy_1698884710808/work
+decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
+defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
+Deprecated==1.2.15
+diff-match-patch @ file:///Users/ktietz/demo/mc3/conda-bld/diff-match-patch_1630511840874/work
+dill @ file:///croot/dill_1715094664823/work
+distlib==0.3.9
+distributed @ file:///croot/distributed_1725523003979/work
+distro @ file:///croot/distro_1714488253808/work
+docker==7.1.0
+docstring-to-markdown @ file:///work/perseverance-python-buildout/croot/docstring-to-markdown_1698864372211/work
+docutils @ file:///work/perseverance-python-buildout/croot/docutils_1698846509640/work
+et-xmlfile @ file:///work/perseverance-python-buildout/croot/et_xmlfile_1698868193149/work
 exceptiongroup==1.2.2
-executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1698579936712/work
-fastapi==0.111.0
-fastapi-cli==0.0.4
-fastjsonschema @ file:///home/conda/feedstock_root/build_artifacts/python-fastjsonschema_1703780968325/work/dist
-filelock==3.11.0
+executing @ file:///opt/conda/conda-bld/executing_1646925071911/work
+fastapi==0.115.6
+fastjsonschema @ file:///work/perseverance-python-buildout/croot/python-fastjsonschema_1698846742882/work
+filelock @ file:///work/perseverance-python-buildout/croot/filelock_1701733993137/work
 first==2.0.2
-Flask @ file:///home/conda/feedstock_root/build_artifacts/flask_1690881361613/work
-fonttools @ file:///home/conda/feedstock_root/build_artifacts/fonttools_1708049097969/work
-fqdn @ file:///home/conda/feedstock_root/build_artifacts/fqdn_1638810296540/work/dist
-freetype-py @ file:///home/conda/feedstock_root/build_artifacts/freetype-py_1650983368720/work
-frozenlist @ file:///home/conda/feedstock_root/build_artifacts/frozenlist_1702645450877/work
-fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1710607797444/work
-gitdb @ file:///home/conda/feedstock_root/build_artifacts/gitdb_1697791558612/work
-GitPython @ file:///home/conda/feedstock_root/build_artifacts/gitpython_1708069240306/work
-google-auth @ file:///home/conda/feedstock_root/build_artifacts/google-auth_1709994919355/work
-gpustat==1.1.1
-graphene @ file:///home/conda/feedstock_root/build_artifacts/graphene_1690379572063/work
-graphql-core @ file:///home/conda/feedstock_root/build_artifacts/graphql-core_1663938321907/work
-graphql-relay @ file:///home/conda/feedstock_root/build_artifacts/graphql-relay_1650134628625/work
-graphviz @ file:///home/conda/feedstock_root/build_artifacts/python-graphviz_1658658635601/work
-greenlet @ file:///home/conda/feedstock_root/build_artifacts/greenlet_1703201578921/work
-griffe==1.2.0
-gunicorn @ file:///home/conda/feedstock_root/build_artifacts/gunicorn_1695414783979/work
-h11 @ file:///home/conda/feedstock_root/build_artifacts/h11_1664132893548/work
-h2 @ file:///home/conda/feedstock_root/build_artifacts/h2_1634280454336/work
+flake8 @ file:///croot/flake8_1708965256870/work
+Flask @ file:///croot/flask_1716545870149/work
+fonttools @ file:///croot/fonttools_1713551344105/work
+frozendict @ file:///croot/frozendict_1713194832637/work
+frozenlist @ file:///work/perseverance-python-buildout/croot/frozenlist_1698886842990/work
+fsspec @ file:///croot/fsspec_1724855583730/work
+gensim @ file:///croot/gensim_1725057941774/work
+gitdb @ file:///tmp/build/80754af9/gitdb_1617117951232/work
+GitPython @ file:///croot/gitpython_1720455034730/work
+google-api-core==2.24.0
+google-auth==2.37.0
+googleapis-common-protos==1.66.0
+graphviz==0.20.3
+greenlet @ file:///croot/greenlet_1702059959679/work
+griffe==1.5.4
+grpcio==1.69.0
+h11 @ file:///croot/h11_1706652277403/work
+h2==4.1.0
+h5py @ file:///croot/h5py_1715094721489/work
+HeapDict @ file:///Users/ktietz/demo/mc3/conda-bld/heapdict_1630598515714/work
+holoviews @ file:///croot/holoviews_1720533861358/work
 hpack==4.0.0
-httpcore @ file:///home/conda/feedstock_root/build_artifacts/httpcore_1711596990900/work
-httptools==0.6.1
-httpx @ file:///home/conda/feedstock_root/build_artifacts/httpx_1708530890843/work
-humanize @ file:///home/conda/feedstock_root/build_artifacts/humanize_1700607927158/work
-hyperframe @ file:///home/conda/feedstock_root/build_artifacts/hyperframe_1619110129307/work
-idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1701026962277/work
-importlib_metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1709821103657/work
-importlib_resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1709821111795/work
-ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1708996548741/work
-ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1709559745751/work
-ipywidgets==8.1.2
-isoduration @ file:///home/conda/feedstock_root/build_artifacts/isoduration_1638811571363/work/dist
-itsdangerous @ file:///home/conda/feedstock_root/build_artifacts/itsdangerous_1648147185463/work
-jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work
-Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1704966972576/work
-jinja2_humanize_extension @ file:///home/conda/feedstock_root/build_artifacts/jinja2-humanize-extension_1714124780176/work
-jmespath @ file:///home/conda/feedstock_root/build_artifacts/jmespath_1655568249366/work
-joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1691577114857/work
-json-tricks==3.17.3
-json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1710632238570/work
-jsonpatch @ file:///home/conda/feedstock_root/build_artifacts/jsonpatch_1695536281965/work
-jsonpickle @ file:///home/conda/feedstock_root/build_artifacts/jsonpickle_1697667537447/work
-jsonpointer @ file:///home/conda/feedstock_root/build_artifacts/jsonpointer_1695397236330/work
-jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1705707496704/work
-jsonschema-specifications @ file:///tmp/tmpkv1z7p57/src
-jupyter-events @ file:///home/conda/feedstock_root/build_artifacts/jupyter_events_1710262592257/work
-jupyter-lsp @ file:///home/conda/feedstock_root/build_artifacts/jupyter-lsp-meta_1709672523643/work/jupyter-lsp
-jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1710255804825/work
-jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1710257359434/work
-jupyter_server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1709563276146/work
-jupyter_server_terminals @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_terminals_1710262634903/work
-jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1710450352071/work
-jupyterlab_pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1707149102966/work
-jupyterlab_server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server-split_1710189681706/work
-jupyterlab_widgets==3.0.10
-kiwisolver @ file:///home/conda/feedstock_root/build_artifacts/kiwisolver_1695379920604/work
-kombu @ file:///home/conda/feedstock_root/build_artifacts/kombu_1705103119146/work
-kubernetes @ file:///home/conda/feedstock_root/build_artifacts/python-kubernetes_1689680899878/work
+httpcore==1.0.7
+httpx @ file:///croot/httpx_1723474802858/work
+humanize==4.11.0
+hvplot @ file:///croot/hvplot_1727775570677/work
+hyperframe==6.0.1
+hyperlink @ file:///tmp/build/80754af9/hyperlink_1610130746837/work
+idna @ file:///croot/idna_1714398848350/work
+imagecodecs @ file:///work/perseverance-python-buildout/croot/imagecodecs_1698851963060/work
+imageio @ file:///croot/imageio_1707247282708/work
+imagesize @ file:///work/perseverance-python-buildout/croot/imagesize_1698864514488/work
+imbalanced-learn @ file:///croot/imbalanced-learn_1718132237102/work
+importlib-metadata @ file:///croot/importlib_metadata-suite_1704813515092/work
+incremental @ file:///croot/incremental_1708639938299/work
+inflection @ file:///work/perseverance-python-buildout/croot/inflection_1698869084799/work
+iniconfig @ file:///home/linux1/recipes/ci/iniconfig_1610983019677/work
+intake @ file:///croot/intake_1726109551815/work
+intervaltree @ file:///Users/ktietz/demo/mc3/conda-bld/intervaltree_1630511889664/work
+ipykernel @ file:///croot/ipykernel_1705933831282/work
+ipython @ file:///croot/ipython_1726064227985/work
+ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
+ipywidgets @ file:///croot/ipywidgets_1710961502038/work
+isort @ file:///croot/isort_1718289883491/work
+itemadapter @ file:///tmp/build/80754af9/itemadapter_1626442940632/work
+itemloaders @ file:///croot/itemloaders_1708639918324/work
+itsdangerous @ file:///croot/itsdangerous_1716533337253/work
+jaraco.classes @ file:///tmp/build/80754af9/jaraco.classes_1620983179379/work
+jedi @ file:///croot/jedi_1721058342488/work
+jeepney @ file:///tmp/build/80754af9/jeepney_1627537048313/work
+jellyfish @ file:///work/perseverance-python-buildout/croot/jellyfish_1698888203179/work
+Jinja2 @ file:///croot/jinja2_1716993405101/work
+jinja2-humanize-extension==0.4.0
+jmespath @ file:///work/perseverance-python-buildout/croot/jmespath_1701734034266/work
+joblib @ file:///croot/joblib_1718217211762/work
+json5 @ file:///tmp/build/80754af9/json5_1624432770122/work
+jsonpatch @ file:///croot/jsonpatch_1714483231291/work
+jsonpointer==2.1
+jsonschema @ file:///croot/jsonschema_1728486696720/work
+jsonschema-specifications @ file:///work/perseverance-python-buildout/croot/jsonschema-specifications_1701731664072/work
+jupyter @ file:///croot/jupyter_1709837193779/work
+jupyter-console @ file:///work/perseverance-python-buildout/croot/jupyter_console_1707343324117/work
+jupyter-events @ file:///croot/jupyter_events_1718738097486/work
+jupyter-lsp @ file:///work/perseverance-python-buildout/croot/jupyter-lsp-meta_1707343107456/work
+jupyter_client @ file:///work/perseverance-python-buildout/croot/jupyter_client_1701731788627/work
+jupyter_core @ file:///croot/jupyter_core_1718818295206/work
+jupyter_server @ file:///croot/jupyter_server_1718827083372/work
+jupyter_server_terminals @ file:///work/perseverance-python-buildout/croot/jupyter_server_terminals_1701732095362/work
+jupyterlab @ file:///croot/jupyterlab_1725895214311/work
+jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
+jupyterlab-widgets @ file:///tmp/build/80754af9/jupyterlab_widgets_1609884341231/work
+jupyterlab_server @ file:///croot/jupyterlab_server_1725865349919/work
+keyring @ file:///croot/keyring_1709632513808/work
+kiwisolver @ file:///work/perseverance-python-buildout/croot/kiwisolver_1698847502605/work
+lazy-object-proxy @ file:///croot/lazy-object-proxy_1712908712693/work
+lazy_loader @ file:///croot/lazy_loader_1718176737906/work
+lckr_jupyterlab_variableinspector @ file:///work/perseverance-python-buildout/croot/jupyterlab-variableinspector_1709066217947/work
+libarchive-c @ file:///croot/python-libarchive-c_1726069797193/work
+libmambapy @ file:///croot/mamba-split_1714483352891/work/libmambapy
 lightgbm==4.5.0
-llvmlite==0.42.0
-Mako @ file:///home/conda/feedstock_root/build_artifacts/mako_1706629259024/work
-Markdown @ file:///home/conda/feedstock_root/build_artifacts/markdown_1710435156458/work
-markdown-it-py @ file:///home/conda/feedstock_root/build_artifacts/markdown-it-py_1686175045316/work
-MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1706899926732/work
-matplotlib @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-suite_1708026439111/work
-matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work
-mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1704317613764/work
-mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1698947099619/work
-mlflow @ file:///home/conda/feedstock_root/build_artifacts/mlflow-split_1709861614743/work
-mlflow-skinny @ file:///home/conda/feedstock_root/build_artifacts/mlflow-split_1709861614743/work
-msgpack==1.0.8
-multidict @ file:///home/conda/feedstock_root/build_artifacts/multidict_1707040702345/work
-munch @ file:///home/conda/feedstock_root/build_artifacts/munch_1688318326844/work
-munkres==1.1.4
-mypy-extensions @ file:///home/conda/feedstock_root/build_artifacts/mypy_extensions_1675543315189/work
-nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1710317608672/work
-nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert-meta_1709581262419/work
-nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1710502215509/work
-nest_asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1705850609492/work
-nni==3.0
-notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1710504300620/work
-notebook_shim @ file:///home/conda/feedstock_root/build_artifacts/notebook-shim_1707957777232/work
-numba @ file:///home/conda/feedstock_root/build_artifacts/numba_1707024788644/work
-numpy==1.26.4
-nvidia-ml-py==12.535.133
-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1666056362788/work
-orjson @ file:///home/conda/feedstock_root/build_artifacts/orjson_1708717146318/work/target/wheels/orjson-3.9.15-cp311-cp311-linux_x86_64.whl#sha256=53f481983770be697dc7941513677cccc8e8f7704b2f873e1f7120b424dc5708
-overrides @ file:///home/conda/feedstock_root/build_artifacts/overrides_1706394519472/work
-packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1696202382185/work
-pandas @ file:///home/conda/feedstock_root/build_artifacts/pandas_1708708634263/work
-pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
-papermill @ file:///home/conda/feedstock_root/build_artifacts/papermill_1709166650826/work
-paramiko @ file:///home/conda/feedstock_root/build_artifacts/paramiko_1703015906107/work
-parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
-pathspec @ file:///home/conda/feedstock_root/build_artifacts/pathspec_1702249949303/work
-patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1704469236901/work
-pendulum==3.0.0
-pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1706113125309/work
-phik @ file:///home/conda/feedstock_root/build_artifacts/phik_1697266224495/work
-pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
-pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1704252032614/work
-pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1694617248815/work
-platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1706713388748/work
-plotly @ file:///home/conda/feedstock_root/build_artifacts/plotly_1708020413888/work
-plotly-express==0.4.1
-prefect==3.0.1
-prettytable==3.10.0
-prometheus-flask-exporter @ file:///home/conda/feedstock_root/build_artifacts/prometheus_flask_exporter_1698385049585/work
-prometheus_client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1707932675456/work
-prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1702399386289/work
+linkify-it-py @ file:///work/perseverance-python-buildout/croot/linkify-it-py_1698889355052/work
+llvmlite @ file:///croot/llvmlite_1720455102465/work
+lmdb @ file:///work/perseverance-python-buildout/croot/python-lmdb_1698865572354/work
+locket @ file:///work/perseverance-python-buildout/croot/locket_1698847555568/work
+lxml @ file:///croot/lxml_1715289006750/work
+lz4 @ file:///work/perseverance-python-buildout/croot/lz4_1698895161487/work
+Mako==1.3.8
+Markdown @ file:///work/perseverance-python-buildout/croot/markdown_1698870813381/work
+markdown-it-py @ file:///work/perseverance-python-buildout/croot/markdown-it-py_1698846045803/work
+MarkupSafe @ file:///work/perseverance-python-buildout/croot/markupsafe_1707342994941/work
+matplotlib==3.9.2
+matplotlib-inline @ file:///work/perseverance-python-buildout/croot/matplotlib-inline_1698864771271/work
+mccabe @ file:///opt/conda/conda-bld/mccabe_1644221741721/work
+mdit-py-plugins @ file:///work/perseverance-python-buildout/croot/mdit-py-plugins_1698895393342/work
+mdurl @ file:///work/perseverance-python-buildout/croot/mdurl_1698845653285/work
+memray==1.15.0
+menuinst @ file:///croot/menuinst_1723567589013/work
+mistune @ file:///work/perseverance-python-buildout/croot/mistune_1698895588147/work
+mkl-service==2.4.0
+mkl_fft @ file:///croot/mkl_fft_1725370245198/work
+mkl_random @ file:///croot/mkl_random_1725370241878/work
+more-itertools @ file:///croot/more-itertools_1727185441804/work
+mpmath @ file:///work/perseverance-python-buildout/croot/mpmath_1698864994882/work
+msgpack @ file:///work/perseverance-python-buildout/croot/msgpack-python_1698846122235/work
+multidict @ file:///croot/multidict_1701096859099/work
+multipledispatch @ file:///work/perseverance-python-buildout/croot/multipledispatch_1698846147141/work
+mypy @ file:///croot/mypy-split_1725573876658/work
+mypy-extensions @ file:///work/perseverance-python-buildout/croot/mypy_extensions_1698863276135/work
+nbclient @ file:///work/perseverance-python-buildout/croot/nbclient_1701731912506/work
+nbconvert @ file:///croot/nbconvert_1728049414448/work
+nbformat @ file:///croot/nbformat_1728049424075/work
+nest-asyncio @ file:///croot/nest-asyncio_1708532673751/work
+networkx @ file:///croot/networkx_1720002482208/work
+nltk @ file:///croot/nltk_1724427688180/work
+notebook @ file:///croot/notebook_1727197365392/work
+notebook_shim @ file:///work/perseverance-python-buildout/croot/notebook-shim_1707343174853/work
+numba @ file:///croot/numba_1720540961817/work
+numexpr @ file:///work/perseverance-python-buildout/croot/numexpr_1698871031164/work
+numpy @ file:///croot/numpy_and_numpy_base_1708638617955/work/dist/numpy-1.26.4-cp312-cp312-linux_x86_64.whl#sha256=1d700f51d8b4fa684d858c9e3b56b1656bc5c82b6b79ff08d4e3b491c430059f
+numpydoc @ file:///croot/numpydoc_1718279157024/work
+nvidia-cublas-cu12==12.4.5.8
+nvidia-cuda-cupti-cu12==12.4.127
+nvidia-cuda-nvrtc-cu12==12.4.127
+nvidia-cuda-runtime-cu12==12.4.127
+nvidia-cudnn-cu12==9.1.0.70
+nvidia-cufft-cu12==11.2.1.3
+nvidia-curand-cu12==10.3.5.147
+nvidia-cusolver-cu12==11.6.1.9
+nvidia-cusparse-cu12==12.3.1.170
+nvidia-nccl-cu12==2.21.5
+nvidia-nvjitlink-cu12==12.4.127
+nvidia-nvtx-cu12==12.4.127
+oauthlib==3.2.2
+opencensus==0.11.4
+opencensus-context==0.1.3
+openpyxl @ file:///croot/openpyxl_1721752957391/work
+opentelemetry-api==1.29.0
+optuna==4.1.0
+orjson==3.10.14
+overrides @ file:///work/perseverance-python-buildout/croot/overrides_1701732220415/work
+packaging @ file:///croot/packaging_1720101850331/work
+pandas @ file:///croot/pandas_1718308974269/work/dist/pandas-2.2.2-cp312-cp312-linux_x86_64.whl#sha256=92c518f7e09edd50b5caa5862636c51d6a29391803f3ada62f68aa52f27d8f92
+pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work
+panel @ file:///croot/panel_1728066367874/work
+param @ file:///croot/param_1719347913870/work
+parsel @ file:///croot/parsel_1707503445438/work
+parso @ file:///opt/conda/conda-bld/parso_1641458642106/work
+partd @ file:///work/perseverance-python-buildout/croot/partd_1698863295708/work
+pathspec @ file:///work/perseverance-python-buildout/croot/pathspec_1698805478393/work
+patsy @ file:///croot/patsy_1718378176128/work
+pendulum==3.0.0
+pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
+pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
+pillow @ file:///croot/pillow_1721059439630/work
+pkce @ file:///work/perseverance-python-buildout/croot/pkce_1698863336124/work
+pkginfo @ file:///croot/pkginfo_1715695984887/work
+platformdirs @ file:///work/perseverance-python-buildout/croot/platformdirs_1701732573265/work
+plotly @ file:///croot/plotly_1726245554316/work
+pluggy @ file:///work/perseverance-python-buildout/croot/pluggy_1698805497733/work
+ply @ file:///work/perseverance-python-buildout/croot/ply_1698846188635/work
+prefect==3.1.11
+prometheus_client==0.21.1
+prompt-toolkit @ file:///croot/prompt-toolkit_1704404351921/work
+Protego @ file:///tmp/build/80754af9/protego_1598657180827/work
+proto-plus==1.25.0
 protobuf==4.25.3
-psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1705722403006/work
-psycopg2==2.9.9
-ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
-pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work
-py-cpuinfo @ file:///home/conda/feedstock_root/build_artifacts/py-cpuinfo_1666774466606/work
-pyarrow==15.0.1
-pyasn1 @ file:///home/conda/feedstock_root/build_artifacts/pyasn1_1701287008248/work
-pyasn1-modules @ file:///home/conda/feedstock_root/build_artifacts/pyasn1-modules_1695107857548/work
-pycairo==1.26.0
-pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
-pydantic==2.7.3
-pydantic-extra-types==2.9.0
-pydantic-settings==2.3.1
-pydantic_core==2.18.4
-Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1700607939962/work
-PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1706895065046/work
-PyNaCl @ file:///home/conda/feedstock_root/build_artifacts/pynacl_1695544874308/work
-pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1706660063483/work
-pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1709721012883/work
-pyproject_hooks==1.0.0
-PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
-python-dateutil==2.9.0.post0
-python-dotenv==1.0.1
-python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1677079630776/work
-python-multipart @ file:///home/conda/feedstock_root/build_artifacts/python-multipart_1707760088566/work
-python-slugify @ file:///home/conda/feedstock_root/build_artifacts/python-slugify-split_1707425621764/work
-PythonWebHDFS==0.2.3
-pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1706549378554/work
-pytzdata @ file:///home/conda/feedstock_root/build_artifacts/pytzdata_1594644346367/work
-pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work
-PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1695373611984/work
-pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1701783162530/work
-querystring-parser==1.2.4
-ray==2.35.0
-readchar @ file:///home/conda/feedstock_root/build_artifacts/readchar_1679045163536/work
-redis==5.0.4
-referencing @ file:///home/conda/feedstock_root/build_artifacts/referencing_1706711412823/work
-regex @ file:///home/conda/feedstock_root/build_artifacts/regex_1703393490389/work
-reportlab @ file:///home/conda/feedstock_root/build_artifacts/reportlab_1707446951177/work
-requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1684774241324/work
-requests-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/requests-oauthlib_1710149579371/work
-responses==0.25.0
-retrying==1.3.3
-rfc3339-validator @ file:///home/conda/feedstock_root/build_artifacts/rfc3339-validator_1638811747357/work
-rfc3986-validator @ file:///home/conda/feedstock_root/build_artifacts/rfc3986-validator_1598024191506/work
-rich @ file:///home/conda/feedstock_root/build_artifacts/rich-split_1709150387247/work/dist
-rlPyCairo @ file:///home/conda/feedstock_root/build_artifacts/rlpycairo_1687519531733/work
-rpds-py @ file:///home/conda/feedstock_root/build_artifacts/rpds-py_1707922729351/work
-rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1658328885051/work
-ruamel.yaml @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml_1707298093865/work
-ruamel.yaml.clib @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml.clib_1707314491256/work
-s3transfer @ file:///home/conda/feedstock_root/build_artifacts/s3transfer_1710497251148/work
-sacred @ file:///home/conda/feedstock_root/build_artifacts/sacred_1701182981738/work
-schema==0.7.5
-scikit-learn==1.4.1.post1
-scipy==1.12.0
-seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1706340836595/work
-Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1682601222253/work
-shap @ file:///home/conda/feedstock_root/build_artifacts/shap_1710256440821/work
-shapash @ file:///home/conda/feedstock_root/build_artifacts/shapash_1710254338922/work
-shellingham @ file:///home/conda/feedstock_root/build_artifacts/shellingham_1698144360966/work
-simplejson==3.19.2
-six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
-slicer @ file:///home/conda/feedstock_root/build_artifacts/slicer_1608146800664/work
-smmap @ file:///home/conda/feedstock_root/build_artifacts/smmap_1634310307496/work
-sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1708952932303/work
-soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1693929250441/work
-SQLAlchemy @ file:///home/conda/feedstock_root/build_artifacts/sqlalchemy_1709646244405/work
-sqlparse @ file:///home/conda/feedstock_root/build_artifacts/sqlparse_1681817562700/work
-stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
-starlette==0.37.2
-statsmodels @ file:///home/conda/feedstock_root/build_artifacts/statsmodels_1702575375433/work
-tenacity @ file:///home/conda/feedstock_root/build_artifacts/tenacity_1692026804430/work
+psutil @ file:///work/perseverance-python-buildout/croot/psutil_1698863411559/work
+ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
+pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work
+py-cpuinfo @ file:///work/perseverance-python-buildout/croot/py-cpuinfo_1698865397566/work
+py-spy==0.4.0
+pyarrow @ file:///croot/pyarrow_1721664224170/work/python
+pyasn1 @ file:///Users/ktietz/demo/mc3/conda-bld/pyasn1_1629708007385/work
+pyasn1-modules==0.2.8
+pycodestyle @ file:///croot/pycodestyle_1701910164456/work
+pycosat @ file:///croot/pycosat_1714510623388/work
+pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
+pyct @ file:///work/perseverance-python-buildout/croot/pyct_1698871492339/work
+pycurl @ file:///croot/pycurl_1725370229412/work
+pydantic @ file:///croot/pydantic_1725040526370/work
+pydantic-extra-types==2.10.1
+pydantic-settings==2.7.1
+pydantic_core @ file:///croot/pydantic-core_1724790363792/work
+pydeck @ file:///croot/pydeck_1706194064552/work
+PyDispatcher @ file:///work/perseverance-python-buildout/croot/pydispatcher_1698871512960/work
+pydocstyle @ file:///work/perseverance-python-buildout/croot/pydocstyle_1698865417470/work
+pyerfa @ file:///croot/pyerfa_1717700762709/work
+pyflakes @ file:///croot/pyflakes_1708962956225/work
+Pygments @ file:///work/perseverance-python-buildout/croot/pygments_1698846270603/work
+PyJWT @ file:///croot/pyjwt_1715094751094/work
+pylint @ file:///work/perseverance-python-buildout/croot/pylint_1698865458204/work
+pylint-venv @ file:///croot/pylint-venv_1709837612550/work
+pyls-spyder==0.4.0
+pyodbc @ file:///croot/pyodbc_1725560222485/work
+pyOpenSSL @ file:///croot/pyopenssl_1723651579422/work
+pyparsing @ file:///croot/pyparsing_1725041624202/work
+PyQt5==5.15.10
+PyQt5-sip @ file:///work/perseverance-python-buildout/croot/pyqt-split_1698847927472/work/pyqt_sip
+PyQtWebEngine==5.15.6
+PySocks @ file:///work/perseverance-python-buildout/croot/pysocks_1698845478203/work
+pytest @ file:///croot/pytest_1717793244625/work
+python-dateutil @ file:///croot/python-dateutil_1716495738603/work
+python-dotenv @ file:///work/perseverance-python-buildout/croot/python-dotenv_1698846722824/work
+python-json-logger @ file:///work/perseverance-python-buildout/croot/python-json-logger_1698873656334/work
+python-lsp-black @ file:///croot/python-lsp-black_1709232897954/work
+python-lsp-jsonrpc @ file:///croot/python-lsp-jsonrpc_1708962872556/work
+python-lsp-server @ file:///croot/python-lsp-server_1708971736441/work
+python-slugify @ file:///tmp/build/80754af9/python-slugify_1620405669636/work
+python-socks==2.6.1
+pytoolconfig @ file:///croot/pytoolconfig_1701728692402/work
+pytz @ file:///croot/pytz_1713974312559/work
+pyviz_comms @ file:///croot/pyviz_comms_1711136832766/work
+PyWavelets @ file:///croot/pywavelets_1725657945951/work
+pyxdg @ file:///tmp/build/80754af9/pyxdg_1603822279816/work
+PyYAML @ file:///work/perseverance-python-buildout/croot/pyyaml_1698849903511/work
+pyzmq @ file:///croot/pyzmq_1705605076900/work
+QDarkStyle @ file:///croot/qdarkstyle_1709231003551/work
+qstylizer @ file:///work/perseverance-python-buildout/croot/qstylizer_1698900556042/work/dist/qstylizer-0.2.2-py2.py3-none-any.whl#sha256=6d40f32007acd9673a44b9d879388ca3e09a68f0342daa31cab082bc596303c8
+QtAwesome @ file:///croot/qtawesome_1726169351455/work
+qtconsole @ file:///croot/qtconsole_1709231153903/work
+QtPy @ file:///work/perseverance-python-buildout/croot/qtpy_1701733558435/work
+queuelib @ file:///work/perseverance-python-buildout/croot/queuelib_1698873899800/work
+ray==2.40.0
+readchar==4.2.1
+referencing @ file:///work/perseverance-python-buildout/croot/referencing_1701731622327/work
+regex @ file:///croot/regex_1726669635272/work
+reportlab==4.2.5
+requests @ file:///croot/requests_1721410876868/work
+requests-file @ file:///Users/ktietz/demo/mc3/conda-bld/requests-file_1629455781986/work
+requests-oauthlib==2.0.0
+requests-toolbelt @ file:///work/perseverance-python-buildout/croot/requests-toolbelt_1698846872000/work
+rfc3339-validator @ file:///work/perseverance-python-buildout/croot/rfc3339-validator_1698873973408/work
+rfc3986-validator @ file:///work/perseverance-python-buildout/croot/rfc3986-validator_1698874010100/work
+rich @ file:///croot/rich_1720637495510/work
+rope @ file:///croot/rope_1708963153974/work
+rpds-py @ file:///work/perseverance-python-buildout/croot/rpds-py_1698982686240/work
+rsa==4.9
+Rtree @ file:///work/perseverance-python-buildout/croot/rtree_1698900851071/work
+ruamel-yaml-conda @ file:///work/perseverance-python-buildout/croot/ruamel_yaml_1698874047131/work
+ruamel.yaml @ file:///croot/ruamel.yaml_1727980156079/work
+ruamel.yaml.clib @ file:///croot/ruamel.yaml.clib_1727769819935/work
+s3fs @ file:///croot/s3fs_1724924097918/work
+scikit-image @ file:///croot/scikit-image_1726737398091/work
+scikit-learn @ file:///croot/scikit-learn_1721921875708/work
+scipy @ file:///croot/scipy_1717521478074/work/dist/scipy-1.13.1-cp312-cp312-linux_x86_64.whl#sha256=a60c1bdf983ee297b25e41830d00cc2af50f00bcb47e55578703fef1594ec56e
+Scrapy @ file:///croot/scrapy_1708714682090/work
+seaborn @ file:///croot/seaborn_1718302919398/work
+SecretStorage @ file:///work/perseverance-python-buildout/croot/secretstorage_1698846935813/work
+semver @ file:///croot/semver_1709243621175/work
+Send2Trash @ file:///work/perseverance-python-buildout/croot/send2trash_1701732325573/work
+service-identity @ file:///Users/ktietz/demo/mc3/conda-bld/service_identity_1629460757137/work
+setuptools==75.1.0
+shellingham==1.5.4
+sip @ file:///work/perseverance-python-buildout/croot/sip_1698846977033/work
+six @ file:///tmp/build/80754af9/six_1644875935023/work
+smart-open @ file:///work/perseverance-python-buildout/croot/smart_open_1698863627550/work
+smmap @ file:///tmp/build/80754af9/smmap_1611694433573/work
+sniffio @ file:///croot/sniffio_1705431295498/work
+snowballstemmer @ file:///tmp/build/80754af9/snowballstemmer_1637937080595/work
+sortedcontainers @ file:///tmp/build/80754af9/sortedcontainers_1623949099177/work
+soupsieve @ file:///work/perseverance-python-buildout/croot/soupsieve_1698866207280/work
+Sphinx @ file:///croot/sphinx_1718275386998/work
+sphinxcontrib-applehelp @ file:///home/ktietz/src/ci/sphinxcontrib-applehelp_1611920841464/work
+sphinxcontrib-devhelp @ file:///home/ktietz/src/ci/sphinxcontrib-devhelp_1611920923094/work
+sphinxcontrib-htmlhelp @ file:///tmp/build/80754af9/sphinxcontrib-htmlhelp_1623945626792/work
+sphinxcontrib-jsmath @ file:///home/ktietz/src/ci/sphinxcontrib-jsmath_1611920942228/work
+sphinxcontrib-qthelp @ file:///home/ktietz/src/ci/sphinxcontrib-qthelp_1611921055322/work
+sphinxcontrib-serializinghtml @ file:///croot/sphinxcontrib-serializinghtml_1718201486943/work
+spyder @ file:///croot/spyder_1727197262862/work
+spyder-kernels @ file:///croot/spyder-kernels_1707937713588/work
+SQLAlchemy @ file:///croot/sqlalchemy_1725885050993/work
+stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work
+starlette==0.41.3
+statsmodels @ file:///croot/statsmodels_1718381181899/work
+streamlit @ file:///croot/streamlit_1724335162935/work
+sympy==1.13.1
+tables @ file:///croot/pytables_1725380772899/work
+tabulate @ file:///work/perseverance-python-buildout/croot/tabulate_1701748610754/work
+tblib @ file:///Users/ktietz/demo/mc3/conda-bld/tblib_1629402031467/work
+tenacity @ file:///croot/tenacity_1721222508356/work
 tensorboardX==2.6.2.2
-terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1710262609923/work
-text-unidecode @ file:///home/conda/feedstock_root/build_artifacts/text-unidecode_1694707102786/work
-textwrap3==0.9.2
-threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1707930541534/work
-time-machine==2.14.1
-tinycss2 @ file:///home/conda/feedstock_root/build_artifacts/tinycss2_1666100256010/work
-toml @ file:///home/conda/feedstock_root/build_artifacts/toml_1604308577558/work
-tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1644342247877/work
-tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1708363099148/work
-tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1707598593068/work
-traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1710254411456/work
+terminado @ file:///work/perseverance-python-buildout/croot/terminado_1698874934826/work
+text-unidecode @ file:///Users/ktietz/demo/mc3/conda-bld/text-unidecode_1629401354553/work
+textdistance @ file:///tmp/build/80754af9/textdistance_1612461398012/work
+textual==1.0.0
+threadpoolctl @ file:///croot/threadpoolctl_1719407800858/work
+three-merge @ file:///tmp/build/80754af9/three-merge_1607553261110/work
+tifffile @ file:///work/perseverance-python-buildout/croot/tifffile_1698866341575/work
+time-machine==2.16.0
+tinycss2 @ file:///work/perseverance-python-buildout/croot/tinycss2_1698875051986/work
+tldextract @ file:///croot/tldextract_1723064386918/work
+toml @ file:///tmp/build/80754af9/toml_1616166611790/work
+tomli @ file:///work/perseverance-python-buildout/croot/tomli_1698805555176/work
+tomlkit @ file:///work/perseverance-python-buildout/croot/tomlkit_1698847005123/work
+toolz @ file:///work/perseverance-python-buildout/croot/toolz_1698846367152/work
+torch==2.5.1
+torchaudio==2.5.1
+torchvision==0.20.1
+tornado @ file:///croot/tornado_1718740109488/work
+tqdm @ file:///croot/tqdm_1724853939799/work
+traitlets @ file:///croot/traitlets_1718227057033/work
 treelite==3.9.1
 treelite-runtime==3.9.1
-typeguard==4.1.2
-typer==0.12.3
-typer-slim==0.12.3
-types-python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/types-python-dateutil_1710589910274/work
-typing-utils @ file:///home/conda/feedstock_root/build_artifacts/typing_utils_1622899189314/work
-typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1708904622550/work
-tzdata @ file:///home/conda/feedstock_root/build_artifacts/python-tzdata_1707747584337/work
-tzlocal @ file:///home/conda/feedstock_root/build_artifacts/tzlocal_1698012935096/work
-ujson @ file:///home/conda/feedstock_root/build_artifacts/ujson_1702256681939/work
-uri-template @ file:///home/conda/feedstock_root/build_artifacts/uri-template_1688655812972/work/dist
-urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1708239446578/work
-uvicorn @ file:///home/conda/feedstock_root/build_artifacts/uvicorn-split_1710145975107/work
-uvloop==0.19.0
-vine @ file:///home/conda/feedstock_root/build_artifacts/vine_1699264759755/work
-watchfiles==0.22.0
-wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1704731205417/work
-webcolors @ file:///home/conda/feedstock_root/build_artifacts/webcolors_1679900785843/work
-webencodings @ file:///home/conda/feedstock_root/build_artifacts/webencodings_1694681268211/work
-websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1701630677416/work
-websockets==12.0
-Werkzeug @ file:///home/conda/feedstock_root/build_artifacts/werkzeug_1676411946679/work
-widgetsnbextension==4.0.10
-wrapt @ file:///home/conda/feedstock_root/build_artifacts/wrapt_1699532786512/work
-yarl @ file:///home/conda/feedstock_root/build_artifacts/yarl_1705508295175/work
-zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1695255097490/work
+triton==3.1.0
+truststore @ file:///work/perseverance-python-buildout/croot/truststore_1701735771625/work
+Twisted @ file:///croot/twisted_1708702809815/work
+typer==0.13.1
+typing_extensions @ file:///croot/typing_extensions_1715268824938/work
+tzdata @ file:///croot/python-tzdata_1690578112552/work
+tzlocal==5.2
+uc-micro-py @ file:///work/perseverance-python-buildout/croot/uc-micro-py_1698875458503/work
+ujson @ file:///croot/ujson_1717597529825/work
+unicodedata2 @ file:///croot/unicodedata2_1713212950228/work
+Unidecode @ file:///croot/unidecode_1724790039185/work
+urllib3 @ file:///croot/urllib3_1727769808118/work
+uvicorn==0.34.0
+virtualenv==20.28.1
+w3lib @ file:///Users/ktietz/demo/mc3/conda-bld/w3lib_1629359764703/work
+watchdog @ file:///croot/watchdog_1717166512516/work
+wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work
+webencodings @ file:///work/perseverance-python-buildout/croot/webencodings_1698866454420/work
+websocket-client @ file:///croot/websocket-client_1715878298792/work
+websockets==13.1
+Werkzeug @ file:///croot/werkzeug_1716533313107/work
+whatthepatch @ file:///work/perseverance-python-buildout/croot/whatthepatch_1698866493803/work
+wheel==0.44.0
+widgetsnbextension @ file:///croot/widgetsnbextension_1710960037596/work
+wrapt @ file:///work/perseverance-python-buildout/croot/wrapt_1698851040578/work
+wurlitzer @ file:///work/perseverance-python-buildout/croot/wurlitzer_1698906481008/work
+xarray @ file:///work/perseverance-python-buildout/croot/xarray_1698851063735/work
+xyzservices @ file:///work/perseverance-python-buildout/croot/xyzservices_1698866513675/work
+yapf @ file:///croot/yapf_1708964320665/work
+yarl @ file:///croot/yarl_1725976495189/work
+zict @ file:///work/perseverance-python-buildout/croot/zict_1698876364831/work
+zipp @ file:///work/perseverance-python-buildout/croot/zipp_1707367835224/work
+zope.interface @ file:///work/perseverance-python-buildout/croot/zope.interface_1698866533743/work
+zstandard @ file:///croot/zstandard_1728569189425/work

It looks like these are VERY different environments, with very different sets of packages installed. If you want reproducibility, I strongly recommend create trying to make those environments identical, e.g. by creating a virtual env (venv docs), using a tool like uv run that does that for you (uv run) or something similar.

I also cannot really tell what versions of some things you have, because it looks like in one of the servers you are either building some projects in a custom way or maybe using a tool other than pip. For example:

-scikit-learn==1.4.1.post1
+scikit-learn @ file:///croot/scikit-learn_1721921875708/work

After controlling Python dependency versions, also add the following to your params:

{
   # setting 'deterministic' causes LightGBM to choose slower but deterministic versions of some operations
   "deterministic": True,
   # set 'seed' to a non-0 number so random sampling is identical in each run
   "seed": 708,
   # multi-threading can lead to non-determinism... turn it off
   "n_jobs": 1
}

These will ensure that lightgbm makes the same choices in both training runs. Right now, you are for example asking LightGBM to randomly choose 30% of the features for each tree (colsample_bytree: 0.3)... those random choices will be different on different training runs if you do not set seed to a non-0 value.

[dameiqinghai]

Hi, thanks for your help!

It looks like these are VERY different environments, with very different sets of packages installed. If you want reproducibility, I strongly recommend create trying to make those environments identical, e.g. by creating a virtual env (venv docs), using a tool like uv run that does that for you (uv run) or something similar.

Since i am using conda. i using my old server env config to create a new conda env
conda env export --no-builds > environment_test.yml
conda env create -f environment_test.yml

{
"deterministic": True,
"seed": 708,
"n_jobs": 1
}

And i did set up above params.
However i still got the problem, on server 2 the training service stopped too early
Stopped training because there are no more leaves that meet the split requirements
Thus the model is underfitted which causes huge result difference.
Any idea?

[jameslamb]

I am using conda.

Ok. If you are using conda to manage other dependencies, you really should use conda to install lightgbm as well.

# remove the pip-installed 'lightgbm'
pip uninstall --yes lightgbm

# conda install
conda install -c conda-forge --yes 'lightgbm>=4.5.0'

conda env create -f environment_test.yml

Did you activate the environment before running your training code?

conda env create --name lgb-test -f ./environment_test.yml
source activate lgb-test

I did set up above params ... huge result difference

Are you absolutely sure that you're using identical datasets? Same column order, same row order, same data types, same representation of missing values?

Run the example code from #6683 (comment) ... do you see the same results on both servers?

[dameiqinghai]

Hi, thanks for your help. I got exact the same results in running the example you provided.
But i found that the difference comes from the data. When i compress and transfer it. There are some errors when i tried to get a better compression ratio. when I use the decompressed data, i did get the same results in both server.