-
Notifications
You must be signed in to change notification settings - Fork 33
Open
Description
bug briefly description:
The PythonInterpreter will initialize with exception "ModuleNotFoundError: No module named 'find_libpython' " when the PYTHON_HOME is a pure python interpreter (not include find_libpython) and the PYTHON_PATH include 'find_libpython' lib. It look likes that the PYTHON_PATH has no effects when initialize PythonInterpreter
This is a Flink TaskManager Error log when enable pemja mode
2026-03-06 10:50:26,740 WARN [org.apache.flink.runtime.taskmanager.Task.transitionState(Task.java:1138)] [] - Source: source[1] -> PythonCalc[2] -> Sink: Unnamed (1/1)#0 (42b42f3ddcc7d2bd8f0fe59d3bf83f7c_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from INITIALIZING to FAILED with failure cause:
java.lang.RuntimeException: Failed to find libpython
at pemja.utils.CommonUtils.getPythonLibrary(CommonUtils.java:161) ~[flink-python-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at pemja.utils.CommonUtils.loadPython(CommonUtils.java:44) ~[flink-python-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at pemja.core.PythonInterpreter$MainInterpreter.initialize(PythonInterpreter.java:365) ~[flink-python-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at pemja.core.PythonInterpreter.initialize(PythonInterpreter.java:144) ~[flink-python-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at pemja.core.PythonInterpreter.<init>(PythonInterpreter.java:45) ~[flink-python-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.streaming.api.operators.python.embedded.AbstractEmbeddedPythonFunctionOperator.open(AbstractEmbeddedPythonFunctionOperator.java:72) ~[flink-python-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.table.runtime.operators.python.AbstractEmbeddedStatelessFunctionOperator.open(AbstractEmbeddedStatelessFunctionOperator.java:84) ~[flink-python-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.table.runtime.operators.python.scalar.EmbeddedPythonScalarFunctionOperator.open(EmbeddedPythonScalarFunctionOperator.java:89) ~[flink-python-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:107) ~[flink-dist-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreStateAndGates(StreamTask.java:858) ~[flink-dist-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$restoreInternal$5(StreamTask.java:812) ~[flink-dist-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100) ~[flink-dist-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:812) ~[flink-dist-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771) ~[flink-dist-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:979) ~[flink-dist-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:948) ~[flink-dist-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:772) ~[flink-dist-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584) ~[flink-dist-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.io.IOException: Failed to execute the command: /tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-archives/cpython39-venv.tar.gz/venv/bin/python -c from find_libpython import find_libpython;print(find_libpython())
output: Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'find_libpython'
at pemja.utils.CommonUtils.execute(CommonUtils.java:180) ~[flink-python-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
at pemja.utils.CommonUtils.getPythonLibrary(CommonUtils.java:157) ~[flink-python-release-TDFLINK-131811631-v1.jar:release-TDFLINK-131811631-v1]
... 18 more
Additional, I print the python interpreter config before initialize Python interpreter, the log show like this:
2026-03-06 10:50:25,330 INFO [org.apache.flink.python.env.AbstractPythonEnvironmentManager.constructEnvironmentVariables(AbstractPythonEnvironmentManager.java:182)] [] - PYTHONPATH of python worker: /tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-files/101856c7-35c0-4ca4-a89e-6784fab28218_toolz_test.py:/tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-files/90576691-06db-4be9-9d84-3fd32654a041_tdflink-lib.zip/tdflink-lib:/tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-files/ad745872-8f94-4087-8162-963d732d8929_user-python-dependency-32227-1.zip/user-python-dependency-32227-1
2026-03-06 10:50:25,330 INFO [org.apache.flink.python.env.AbstractPythonEnvironmentManager.constructArchivesDirectory(AbstractPythonEnvironmentManager.java:377)] [] - Python working dir of python worker: /tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-archives
2026-03-06 10:50:26,095 INFO [org.apache.flink.python.env.AbstractPythonEnvironmentManager.constructEnvironmentVariables(AbstractPythonEnvironmentManager.java:201)] [] - Python interpreter path: cpython39-venv.tar.gz/venv/bin/python
2026-03-06 10:50:26,097 INFO [org.apache.flink.python.env.embedded.EmbeddedPythonEnvironmentManager.createEnvironment(EmbeddedPythonEnvironmentManager.java:111)] [] - Print pemja config, pythonHome: /tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-archives/cpython39-venv.tar.gz/venv/, pythonExec: /tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-archives/cpython39-venv.tar.gz/venv/bin/python, pythonPath: [/tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-files/101856c7-35c0-4ca4-a89e-6784fab28218_toolz_test.py, /tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-files/90576691-06db-4be9-9d84-3fd32654a041_tdflink-lib.zip/tdflink-lib, /tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-files/ad745872-8f94-4087-8162-963d732d8929_user-python-dependency-32227-1.zip/user-python-dependency-32227-1], execType: MULTI_THREAD
The 'find_libpython' module exists in the python path:
/tmp/python-dist-1103fa2b-e5c8-4c33-86ba-c19993909bbc/python-files/90576691-06db-4be9-9d84-3fd32654a041_tdflink-lib.zip/tdflink-lib
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels