asr whisper model in torchserve

[navaneethakarli]

🐛 Describe the bug

I am trying to surve ASR whisper model in torchserve but i am facing worker issue

Error logs

main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin...
2023-09-08T11:47:27,120 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: torchserve_stt/model_store/speechtotext.mar
2023-09-08T11:47:44,718 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model speechtotext
2023-09-08T11:47:44,718 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model speechtotext
2023-09-08T11:47:44,718 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model speechtotext loaded.
2023-09-08T11:47:44,719 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: speechtotext, count: 1
2023-09-08T11:47:44,727 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/batman/navaneetha/my_name/bin/python, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-09-08T11:47:44,729 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2023-09-08T11:47:44,836 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8080
2023-09-08T11:47:44,836 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2023-09-08T11:47:44,837 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8081
2023-09-08T11:47:44,837 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.
2023-09-08T11:47:44,838 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082
Model server started.
2023-09-08T11:47:45,044 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.
2023-09-08T11:47:45,631 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:black,timestamp:1694153865
2023-09-08T11:47:45,633 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:64.30083847045898|#Level:Host|#hostname:black,timestamp:1694153865
2023-09-08T11:47:45,633 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:1803.2123374938965|#Level:Host|#hostname:black,timestamp:1694153865
2023-09-08T11:47:45,633 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:96.6|#Level:Host|#hostname:black,timestamp:1694153865
2023-09-08T11:47:45,634 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.02629618267081562|#Level:Host,DeviceId:0|#hostname:black,timestamp:1694153865
2023-09-08T11:47:45,634 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:12.0|#Level:Host,DeviceId:0|#hostname:black,timestamp:1694153865
2023-09-08T11:47:45,634 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0.0|#Level:Host,DeviceId:0|#hostname:black,timestamp:1694153865
2023-09-08T11:47:45,634 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:53097.89453125|#Level:Host|#hostname:black,timestamp:1694153865
2023-09-08T11:47:45,634 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:10246.71875|#Level:Host|#hostname:black,timestamp:1694153865
2023-09-08T11:47:45,635 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:17.4|#Level:Host|#hostname:black,timestamp:1694153865
2023-09-08T11:47:46,079 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=127192
2023-09-08T11:47:46,084 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2023-09-08T11:47:46,087 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Successfully loaded /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-09-08T11:47:46,087 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - [PID]127192
2023-09-08T11:47:46,087 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Torch worker started.
2023-09-08T11:47:46,088 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Python runtime: 3.8.10
2023-09-08T11:47:46,088 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change null -> WORKER_STARTED
2023-09-08T11:47:46,093 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2023-09-08T11:47:46,100 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2023-09-08T11:47:46,103 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1694153866103
2023-09-08T11:47:46,135 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - model_name: speechtotext, batchSize: 1
2023-09-08T11:47:46,682 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Backend worker process died.
2023-09-08T11:47:46,683 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2023-09-08T11:47:46,683 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py", line 253, in
2023-09-08T11:47:46,683 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - worker.run_server()
2023-09-08T11:47:46,683 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py", line 221, in run_server
2023-09-08T11:47:46,683 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2023-09-08T11:47:46,684 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py", line 184, in handle_connection
2023-09-08T11:47:46,684 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2023-09-08T11:47:46,684 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py", line 131, in load_model
2023-09-08T11:47:46,684 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - service = model_loader.load(
2023-09-08T11:47:46,684 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_loader.py", line 135, in load
2023-09-08T11:47:46,685 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - initialize_fn(service.context)
2023-09-08T11:47:46,685 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/tmp/models/47afe52012fe4966ae64cb2af806c171/handler.py", line 85, in initialize
2023-09-08T11:47:46,685 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - self.processor = WhisperProcessor.from_pretrained(model_dir)
2023-09-08T11:47:46,685 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/home/batman/navaneetha/my_name/lib/python3.8/site-packages/transformers/processing_utils.py", line 215, in from_pretrained
2023-09-08T11:47:46,685 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
2023-09-08T11:47:46,685 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/home/batman/navaneetha/my_name/lib/python3.8/site-packages/transformers/processing_utils.py", line 259, in _get_arguments_from_pretrained
2023-09-08T11:47:46,686 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
2023-09-08T11:47:46,686 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/home/batman/navaneetha/my_name/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1841, in from_pretrained
2023-09-08T11:47:46,686 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - return cls._from_pretrained(
2023-09-08T11:47:46,686 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2023-09-08T11:47:46,686 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/home/batman/navaneetha/my_name/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2004, in _from_pretrained
2023-09-08T11:47:46,686 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - tokenizer = cls(*init_inputs, **init_kwargs)
2023-09-08T11:47:46,686 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - File "/home/batman/navaneetha/my_name/lib/python3.8/site-packages/transformers/models/whisper/tokenization_whisper.py", line 294, in init
2023-09-08T11:47:46,686 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2023-09-08T11:47:46,687 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - with open(merges_file, encoding="utf-8") as merges_handle:
2023-09-08T11:47:46,687 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - TypeError: expected str, bytes or os.PathLike object, not NoneType
2023-09-08T11:47:46,687 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:213) [model-server.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-09-08T11:47:46,698 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: speechtotext, error: Worker died.
2023-09-08T11:47:46,698 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2023-09-08T11:47:46,698 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery start timestamp: 1694153866698
2023-09-08T11:47:46,698 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stderr
2023-09-08T11:47:46,699 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stdout
2023-09-08T11:47:46,699 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds.
2023-09-08T11:47:46,713 [INFO ] W-9000-speechtotext_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stdout
2023-09-08T11:47:46,713 [INFO ] W-9000-speechtotext_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stderr
2023-09-08T11:47:47,700 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/batman/navaneetha/my_name/bin/python, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-09-08T11:47:48,978 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=127281
2023-09-08T11:47:48,979 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2023-09-08T11:47:48,987 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Successfully loaded /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-09-08T11:47:48,987 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - [PID]127281
2023-09-08T11:47:48,987 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Torch worker started.
2023-09-08T11:47:48,987 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Python runtime: 3.8.10
2023-09-08T11:47:48,987 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2023-09-08T11:47:48,988 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2023-09-08T11:47:48,989 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2023-09-08T11:47:48,989 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2023-09-08T11:47:48,990 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2023-09-08T11:47:48,990 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:513) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:675) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-09-08T11:47:48,991 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2023-09-08T11:47:48,991 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2023-09-08T11:47:48,991 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stderr
2023-09-08T11:47:48,991 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stdout
2023-09-08T11:47:48,991 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds.
2023-09-08T11:47:49,003 [INFO ] W-9000-speechtotext_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stdout
2023-09-08T11:47:49,003 [INFO ] W-9000-speechtotext_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stderr
2023-09-08T11:47:49,992 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/batman/navaneetha/my_name/bin/python, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-09-08T11:47:51,274 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=127305
2023-09-08T11:47:51,275 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2023-09-08T11:47:51,283 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Successfully loaded /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-09-08T11:47:51,283 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - [PID]127305
2023-09-08T11:47:51,283 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Torch worker started.
2023-09-08T11:47:51,283 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2023-09-08T11:47:51,283 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Python runtime: 3.8.10
2023-09-08T11:47:51,284 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2023-09-08T11:47:51,285 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2023-09-08T11:47:51,285 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2023-09-08T11:47:51,285 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2023-09-08T11:47:51,286 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:513) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:675) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-09-08T11:47:51,286 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2023-09-08T11:47:51,286 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2023-09-08T11:47:51,286 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stderr
2023-09-08T11:47:51,286 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stdout
2023-09-08T11:47:51,287 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds.
2023-09-08T11:47:51,300 [INFO ] W-9000-speechtotext_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stdout
2023-09-08T11:47:51,300 [INFO ] W-9000-speechtotext_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stderr
2023-09-08T11:47:53,287 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/batman/navaneetha/my_name/bin/python, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-09-08T11:47:54,567 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=127341
2023-09-08T11:47:54,567 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2023-09-08T11:47:54,576 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Successfully loaded /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-09-08T11:47:54,576 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - [PID]127341
2023-09-08T11:47:54,576 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Torch worker started.
2023-09-08T11:47:54,576 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Python runtime: 3.8.10
2023-09-08T11:47:54,576 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2023-09-08T11:47:54,576 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2023-09-08T11:47:54,578 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2023-09-08T11:47:54,578 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2023-09-08T11:47:54,578 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2023-09-08T11:47:54,578 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:513) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:675) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-09-08T11:47:54,579 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2023-09-08T11:47:54,579 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2023-09-08T11:47:54,580 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stderr
2023-09-08T11:47:54,580 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stdout
2023-09-08T11:47:54,580 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 3 seconds.
2023-09-08T11:47:54,593 [INFO ] W-9000-speechtotext_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stdout
2023-09-08T11:47:54,593 [INFO ] W-9000-speechtotext_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stderr
2023-09-08T11:47:57,581 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/batman/navaneetha/my_name/bin/python, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-09-08T11:47:58,848 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=127382
2023-09-08T11:47:58,849 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2023-09-08T11:47:58,857 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Successfully loaded /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-09-08T11:47:58,857 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - [PID]127382
2023-09-08T11:47:58,857 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Torch worker started.
2023-09-08T11:47:58,857 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Python runtime: 3.8.10
2023-09-08T11:47:58,857 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2023-09-08T11:47:58,858 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2023-09-08T11:47:58,859 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2023-09-08T11:47:58,859 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2023-09-08T11:47:58,861 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2023-09-08T11:47:58,861 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:513) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:675) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-09-08T11:47:58,862 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2023-09-08T11:47:58,862 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2023-09-08T11:47:58,862 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stderr
2023-09-08T11:47:58,862 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stdout
2023-09-08T11:47:58,862 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 5 seconds.
2023-09-08T11:47:58,873 [INFO ] W-9000-speechtotext_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stdout
2023-09-08T11:47:58,873 [INFO ] W-9000-speechtotext_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stderr
2023-09-08T11:48:03,863 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/batman/navaneetha/my_name/bin/python, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-09-08T11:48:05,216 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=127418
2023-09-08T11:48:05,216 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2023-09-08T11:48:05,224 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Successfully loaded /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-09-08T11:48:05,225 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - [PID]127418
2023-09-08T11:48:05,225 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Torch worker started.
2023-09-08T11:48:05,225 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Python runtime: 3.8.10
2023-09-08T11:48:05,225 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2023-09-08T11:48:05,225 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2023-09-08T11:48:05,227 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2023-09-08T11:48:05,227 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2023-09-08T11:48:05,227 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2023-09-08T11:48:05,227 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:513) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:675) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-09-08T11:48:05,228 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2023-09-08T11:48:05,228 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2023-09-08T11:48:05,228 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stderr
2023-09-08T11:48:05,228 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stdout
2023-09-08T11:48:05,229 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 8 seconds.
2023-09-08T11:48:05,240 [INFO ] W-9000-speechtotext_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stdout
2023-09-08T11:48:05,240 [INFO ] W-9000-speechtotext_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stderr
2023-09-08T11:48:13,229 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/batman/navaneetha/my_name/bin/python, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-09-08T11:48:14,608 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=127471
2023-09-08T11:48:14,609 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2023-09-08T11:48:14,617 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Successfully loaded /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-09-08T11:48:14,617 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - [PID]127471
2023-09-08T11:48:14,617 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Torch worker started.
2023-09-08T11:48:14,617 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Python runtime: 3.8.10
2023-09-08T11:48:14,617 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2023-09-08T11:48:14,617 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2023-09-08T11:48:14,619 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2023-09-08T11:48:14,619 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2023-09-08T11:48:14,619 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2023-09-08T11:48:14,619 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:513) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:675) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-09-08T11:48:14,620 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2023-09-08T11:48:14,620 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2023-09-08T11:48:14,620 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stderr
2023-09-08T11:48:14,620 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stdout
2023-09-08T11:48:14,620 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 13 seconds.
2023-09-08T11:48:14,631 [INFO ] W-9000-speechtotext_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stdout
2023-09-08T11:48:14,631 [INFO ] W-9000-speechtotext_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stderr
2023-09-08T11:48:27,621 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/batman/navaneetha/my_name/bin/python, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-09-08T11:48:28,901 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=127510
2023-09-08T11:48:28,901 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2023-09-08T11:48:28,909 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Successfully loaded /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-09-08T11:48:28,910 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - [PID]127510
2023-09-08T11:48:28,910 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Torch worker started.
2023-09-08T11:48:28,910 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Python runtime: 3.8.10
2023-09-08T11:48:28,910 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2023-09-08T11:48:28,910 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2023-09-08T11:48:28,913 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2023-09-08T11:48:28,913 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2023-09-08T11:48:28,913 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2023-09-08T11:48:28,913 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:513) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:675) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-09-08T11:48:28,913 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2023-09-08T11:48:28,913 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2023-09-08T11:48:28,914 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stderr
2023-09-08T11:48:28,914 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stdout
2023-09-08T11:48:28,914 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 21 seconds.
2023-09-08T11:48:28,924 [INFO ] W-9000-speechtotext_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stdout
2023-09-08T11:48:28,924 [INFO ] W-9000-speechtotext_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stderr
2023-09-08T11:48:45,629 [INFO ] pool-3-thread-2 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:black,timestamp:1694153925
2023-09-08T11:48:45,630 [INFO ] pool-3-thread-2 TS_METRICS - DiskAvailable.Gigabytes:64.30076217651367|#Level:Host|#hostname:black,timestamp:1694153925
2023-09-08T11:48:45,630 [INFO ] pool-3-thread-2 TS_METRICS - DiskUsage.Gigabytes:1803.2124137878418|#Level:Host|#hostname:black,timestamp:1694153925
2023-09-08T11:48:45,630 [INFO ] pool-3-thread-2 TS_METRICS - DiskUtilization.Percent:96.6|#Level:Host|#hostname:black,timestamp:1694153925
2023-09-08T11:48:45,630 [INFO ] pool-3-thread-2 TS_METRICS - GPUMemoryUtilization.Percent:0.02629618267081562|#Level:Host,DeviceId:0|#hostname:black,timestamp:1694153925
2023-09-08T11:48:45,631 [INFO ] pool-3-thread-2 TS_METRICS - GPUMemoryUsed.Megabytes:12.0|#Level:Host,DeviceId:0|#hostname:black,timestamp:1694153925
2023-09-08T11:48:45,631 [INFO ] pool-3-thread-2 TS_METRICS - GPUUtilization.Percent:0.0|#Level:Host,DeviceId:0|#hostname:black,timestamp:1694153925
2023-09-08T11:48:45,631 [INFO ] pool-3-thread-2 TS_METRICS - MemoryAvailable.Megabytes:53042.8046875|#Level:Host|#hostname:black,timestamp:1694153925
2023-09-08T11:48:45,631 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUsed.Megabytes:10301.76953125|#Level:Host|#hostname:black,timestamp:1694153925
2023-09-08T11:48:45,631 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUtilization.Percent:17.5|#Level:Host|#hostname:black,timestamp:1694153925
2023-09-08T11:48:49,914 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/batman/navaneetha/my_name/bin/python, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-09-08T11:48:51,243 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=127607
2023-09-08T11:48:51,243 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2023-09-08T11:48:51,252 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Successfully loaded /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-09-08T11:48:51,252 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - [PID]127607
2023-09-08T11:48:51,252 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Torch worker started.
2023-09-08T11:48:51,252 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Python runtime: 3.8.10
2023-09-08T11:48:51,252 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2023-09-08T11:48:51,253 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2023-09-08T11:48:51,256 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2023-09-08T11:48:51,256 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2023-09-08T11:48:51,256 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2023-09-08T11:48:51,256 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:513) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:675) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-09-08T11:48:51,256 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2023-09-08T11:48:51,257 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2023-09-08T11:48:51,257 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stderr
2023-09-08T11:48:51,257 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stdout
2023-09-08T11:48:51,257 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 34 seconds.
2023-09-08T11:48:51,270 [INFO ] W-9000-speechtotext_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stdout
2023-09-08T11:48:51,270 [INFO ] W-9000-speechtotext_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stderr
2023-09-08T11:49:25,258 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/batman/navaneetha/my_name/bin/python, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-09-08T11:49:26,828 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=127660
2023-09-08T11:49:26,829 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2023-09-08T11:49:26,837 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Successfully loaded /home/batman/navaneetha/my_name/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-09-08T11:49:26,838 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - [PID]127660
2023-09-08T11:49:26,838 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Torch worker started.
2023-09-08T11:49:26,838 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Python runtime: 3.8.10
2023-09-08T11:49:26,838 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2023-09-08T11:49:26,838 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2023-09-08T11:49:26,841 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2023-09-08T11:49:26,841 [INFO ] W-9000-speechtotext_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2023-09-08T11:49:26,841 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2023-09-08T11:49:26,841 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:513) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:675) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-09-08T11:49:26,842 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-speechtotext_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2023-09-08T11:49:26,842 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2023-09-08T11:49:26,842 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stderr
2023-09-08T11:49:26,842 [WARN ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-speechtotext_1.0-stdout
2023-09-08T11:49:26,842 [INFO ] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 55 seconds.
2023-09-08T11:49:26,853 [INFO ] W-9000-speechtotext_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stdout
2023-09-08T11:49:26,853 [INFO ] W-9000-speechtotext_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-speechtotext_1.0-stderr
2023-09-08T11:50:21,844 [DEBUG] W-9000-speechtotext_1.0 org.pytorch.serve.wlm.WorkerLi

Installation instructions

i am not using Docker i am using server

Model Packaing

#!/bin/bash
set -euo pipefail

mkdir -p model_store

Extra files add all files necessary for processor

torch-model-archiver --model-name speechtotext --version 1.0 --serialized-file model/pytorch_model.bin --handler ./handler.py --extra-files "model/config.json,model/special_tokens_map.json,model/tokenizer_config.json,model/vocab.json,model/preprocessor_config.json" -f
mv speechtotext.mar model_store

config.properties

No response

Versions

absl-py==1.4.0
accelerate==0.22.0
aiohttp==3.8.5
aiosignal==1.3.1
aniso8601==9.0.1
ansi2html==1.8.0
arrow==1.2.3
asttokens==2.2.1
async-timeout==4.0.2
attrs==23.1.0
backcall==0.2.0
blinker==1.6.2
certifi==2023.7.22
cffi==1.15.1
charset-normalizer==3.2.0
click==8.1.6
cmake==3.27.0
colorama==0.4.6
coloredlogs==15.0.1
contourpy==1.1.0
cycler==0.11.0
datasets==2.14.2
decorator==5.1.1
dill==0.3.7
enum-compat==0.0.3
evaluate==0.4.0
exceptiongroup==1.1.3
executing==1.2.0
ffmpeg==1.4
ffmpeg-python==0.2.0
filelock==3.12.2
Flask==2.3.2
Flask-RESTful==0.3.10
flatbuffers==23.5.26
fonttools==4.42.1
frozenlist==1.4.0
fsspec==2023.6.0
future==0.18.3
huggingface-hub==0.16.4
humanfriendly==10.0
idna==3.4
importlib-metadata==6.8.0
importlib-resources==6.0.1
iniconfig==2.0.0
install==1.3.5
ipython==8.12.2
itsdangerous==2.1.2
jedi==0.19.0
Jinja2==3.1.2
joblib==1.3.1
kiwisolver==1.4.5
lit==16.0.6
lxml==4.9.3
MarkupSafe==2.1.3
matplotlib==3.7.2
matplotlib-inline==0.1.6
mpmath==1.3.0
multidict==6.0.4
multiprocess==0.70.15
networkx==3.1
nltk==3.8.1
numpy==1.24.4
nvgpu==0.9.0
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-cupti-cu11==11.7.101
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.2.10.91
nvidia-cusolver-cu11==11.4.0.1
nvidia-cusparse-cu11==11.7.4.91
nvidia-nccl-cu11==2.14.3
nvidia-nvtx-cu11==11.7.91
onnx==1.14.0
onnxruntime==1.15.1
optimum==1.10.1
packaging==23.1
pandas==2.0.3
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==10.0.0
pluggy==1.2.0
portalocker==2.7.0
prompt-toolkit==3.0.39
protobuf==4.23.4
psutil==5.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==12.0.1
pycparser==2.21
Pygments==2.16.1
pynvml==11.4.1
pyparsing==3.0.9
pyreadline3==3.4.1
pytest==7.4.0
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0.1
regex==2023.6.3
requests==2.31.0
responses==0.18.0
rouge-score==0.1.2
sacrebleu==2.3.1
sacremoses==0.0.53
safetensors==0.3.1
scikit-learn==1.3.0
scipy==1.10.1
sentencepiece==0.1.99
six==1.16.0
sklearn==0.0.post7
soundfile==0.12.1
stack-data==0.6.2
sympy==1.12
tabulate==0.9.0
termcolor==2.3.0
threadpoolctl==3.2.0
tokenizers==0.13.3
tomli==2.0.1
torch==2.0.0
torch-model-archiver==0.8.1
torchaudio==2.0.1
torchdata==0.6.1
torchserve==0.8.1
torchtext==0.15.2
torchvision==0.15.1
tqdm==4.65.0
traitlets==5.9.0
transformers==4.31.0
triton==2.0.0
typing-extensions==4.7.1
tzdata==2023.3
urllib3==2.0.4
wcwidth==0.2.6
Werkzeug==2.3.6
whisper==1.1.10
xxhash==3.3.0
yarl==1.9.2
zipp==3.16.2

Repro instructions

#!/bin/bash
set -euo pipefail

mkdir -p model_store

Extra files add all files necessary for processor

torch-model-archiver --model-name speechtotext --version 1.0 --serialized-file model/pytorch_model.bin --handler ./handler.py --extra-files "model/config.json,model/special_tokens_map.json,model/tokenizer_config.json,model/vocab.json,model/preprocessor_config.json" -f
mv speechtotext.mar model_store

Possible Solution

please give me any solution for this

[agunapal]

Can you please share a link to the example/model you are trying and what does your handler look like. You could try to first run the handler standalone to make sure it works

[navaneethakarli]

import torch
import torchaudio
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import io

class SpeechToTextHandler(object):
def init(self):
self._context = None
self.initialized = False
self.model = None
self.processor = None
self.device = None
# Sampling rate for the Whisper model must be 16k
self.expected_sampling_rate = 16_000

def initialize(self, context):
    """Initialize properties and load model"""
    self._context = context
    self.initialized = True
    properties = context.system_properties

    # See https://pytorch.org/serve/custom_service.html#handling-model-execution-on-multiple-gpus
    self.device = torch.device("cuda:" + str(properties.get("gpu_id")) if torch.cuda.is_available() else "cpu")

    model_dir = properties.get("model_dir")
    self.processor = WhisperProcessor.from_pretrained(model_dir)
    self.model = WhisperForConditionalGeneration.from_pretrained(model_dir)
    self.forced_decoder_ids = self.processor.get_decoder_prompt_ids(language="sinhala", task="transcribe")

def handle(self, data, context):
    """Transform input to tensor, resample, run model, and return transcribed text."""
    input = data[0].get("data")
    if input is None:
        input = data[0].get("body")
        input = input.decode('utf-8')
    
    # Load the audio using torchaudio and resample if needed
    waveform, sample_rate = torchaudio.load(io.BytesIO(input), format="wav")
    
    # Ensure the sampling rate is the same as the trained model
    if sample_rate != self.expected_sampling_rate:
        resampler = torchaudio.transforms.Resample(orig_freq=sample_rate, new_freq=self.expected_sampling_rate)
        waveform = resampler(waveform)
    
    print("Input Shape Before Preprocessing:", waveform.shape)  # Debugging line
    
    # Process the waveform using the processor
    input_features = self.processor(waveform.numpy(), sampling_rate=self.expected_sampling_rate, return_tensors="pt").input_features
    predicted_ids = self.model.generate(input_features, forced_decoder_ids=self.forced_decoder_ids)
    transcription = self.processor.batch_decode(predicted_ids, skip_special_tokens=True)

    return [transcription]

Instantiate and initialize the handler

_service = SpeechToTextHandler()

def handle(data, context):
"""
Entry point for SpeechToTextHandler handler
"""
try:
if not _service.initialized:
_service.initialize(context)

    if data is None:
        return None

    return _service.handle(data, context)
except Exception as e:
    raise Exception("Unable to process input data. " + str(e))

i am using this code for asr whisper
and also i want text_to_speech of microsoft/speecht5_tts i am not getting any reference.please give me suggestions

[navaneethakarli]

i am susing fine-tune models

[agunapal]

Hi @navaneethakarli You can debug your handler code using this example.

Please let me know if this helps

[navaneethakarli]

is it possible using of serialized file as .bin file i am using .bin file for all models . i am getting this error for my models
2023-10-03T12:54:01,537 [DEBUG] W-9000-spp_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.