An error occurred during the speech processing process

[jiTest921]

(m3_agent_env_311) (base) ubuntu@VM-0-13-ubuntu:~/m3-agent-master$ python m3_agent/memorization_intermediate_outputs.py \

--data_file data/data.jsonl
/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/albumentations/check_version.py:147: UserWarning: Error fetching version info The read operation timed out
data = fetch_version_info()
/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:121: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider'
warnings.warn(
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /home/ubuntu/.insightface/models/buffalo_l/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /home/ubuntu/.insightface/models/buffalo_l/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /home/ubuntu/.insightface/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /home/ubuntu/.insightface/models/buffalo_l/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /home/ubuntu/.insightface/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)
/home/ubuntu/m3-agent-master/mmagent/voice_processing.py:35: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
pretrained_state = torch.load("models/pretrained_eres2netv2.ckpt", map_location='cpu')
{'video_found': True, 'audio_found': True, 'metadata': {'major_brand': 'isom', 'minor_version': '512', 'compatible_brands': 'isomiso2mp41', 'encoder': 'Lavf58.29.100'}, 'inputs': [{'streams': [{'input_number': 0, 'stream_number': 0, 'stream_type': 'video', 'language': None, 'default': True, 'size': [1578, 720], 'bitrate': 7249, 'fps': 30.0, 'codec_name': 'hevc', 'profile': '(Main)', 'metadata': {'Metadata': '', 'handler_name': 'VideoHandler', 'vendor_id': '[0][0][0][0]'}}, {'input_number': 0, 'stream_number': 1, 'stream_type': 'audio', 'language': None, 'default': True, 'fps': 44100, 'bitrate': 129, 'metadata': {'Metadata': '', 'handler_name': 'SoundHandler', 'vendor_id': '[0][0][0][0]'}}], 'input_number': 0}], 'duration': 30.17, 'bitrate': 7397, 'start': 0.0, 'default_video_input_number': 0, 'default_video_stream_number': 0, 'video_codec_name': 'hevc', 'video_profile': '(Main)', 'video_size': [1578, 720], 'video_bitrate': 7249, 'video_fps': 30.0, 'default_audio_input_number': 0, 'default_audio_stream_number': 1, 'audio_fps': 44100, 'audio_bitrate': 129, 'video_duration': 30.17, 'video_n_frames': 905}
/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/imageio_ffmpeg/binaries/ffmpeg-linux-x86_64-v7.0.2 -i data/clips/robot/bedroom_01/48.mp4 -loglevel error -f image2pipe -vf scale=1578:720 -sws_flags bicubic -pix_fmt rgb24 -vcodec rawvideo -
MoviePy - Writing audio in /tmp/tmp_1wfsi4j.wav
MoviePy - Done.
Unrecognized keys in rope_scaling for 'rope_type'='default': {'mrope_section'}
You are attempting to use Flash Attention 2 without specifying a torch dtype. This might lead to unexpected behaviour
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:03<00:00, 1.27it/s]
The image processor of type Qwen2VLImageProcessor is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with use_fast=False. Note that this behavior will be extended to all models in a future release.
You have video processor config saved in preprocessor.json file which is deprecated. Video processor configs should be saved in their own video_preprocessor.json file. You can rename the file or load and save the processor back which renames it automatically. Loading from preprocessor.json will be removed in v5.0.
Traceback (most recent call last):
File "/home/ubuntu/m3-agent-master/mmagent/voice_processing.py", line 234, in process_voices
with open(save_path, "r") as f:
^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'data/intermediate_outputs/robot/bedroom_01/clip_8_voices.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/m3-agent-master/m3_agent/memorization_intermediate_outputs.py", line 93, in
streaming_process_video(json.loads(line))
File "/home/ubuntu/m3-agent-master/m3_agent/memorization_intermediate_outputs.py", line 77, in streaming_process_video
process_segment(
File "/home/ubuntu/m3-agent-master/m3_agent/memorization_intermediate_outputs.py", line 40, in process_segment
process_voices(
File "/home/ubuntu/m3-agent-master/mmagent/voice_processing.py", line 239, in process_voices
asrs = diarize_audio(base64_video, filter=filter_duration_based)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/m3-agent-master/mmagent/voice_processing.py", line 158, in diarize_audio
response, _ = qwen_get_response(messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/m3-agent-master/mmagent/utils/chat_qwen.py", line 48, in get_response
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py", line 343, in apply_chat_template
or conversation[0]["content"][0]["text"]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
TypeError: string indices must be integers, not 'str'
(m3_agent_env_311) (base) ubuntu@VM-0-13-ubuntu:~/m3-agent-master$

[jiTest921]

(m3_agent_env_311) (base) ubuntu@VM-0-13-ubuntu:/m3-agent-master$ python m3_agent/memorization_memory_graphs.py --data_file data/data.jsonl
/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:121: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider'
warnings.warn(
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /home/ubuntu/.insightface/models/buffalo_l/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /home/ubuntu/.insightface/models/buffalo_l/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /home/ubuntu/.insightface/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /home/ubuntu/.insightface/models/buffalo_l/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /home/ubuntu/.insightface/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)
/home/ubuntu/m3-agent-master/mmagent/voice_processing.py:35: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
pretrained_state = torch.load("models/pretrained_eres2netv2.ckpt", map_location='cpu')
{'video_found': True, 'audio_found': True, 'metadata': {'major_brand': 'isom', 'minor_version': '512', 'compatible_brands': 'isomiso2mp41', 'encoder': 'Lavf58.29.100'}, 'inputs': [{'streams': [{'input_number': 0, 'stream_number': 0, 'stream_type': 'video', 'language': None, 'default': True, 'size': [1578, 720], 'bitrate': 7334, 'fps': 30.0, 'codec_name': 'hevc', 'profile': '(Main)', 'metadata': {'Metadata': '', 'handler_name': 'VideoHandler', 'vendor_id': '[0][0][0][0]'}}, {'input_number': 0, 'stream_number': 1, 'stream_type': 'audio', 'language': None, 'default': True, 'fps': 44100, 'bitrate': 129, 'metadata': {'Metadata': '', 'handler_name': 'SoundHandler', 'vendor_id': '[0][0][0][0]'}}], 'input_number': 0}], 'duration': 30.17, 'bitrate': 7646, 'start': 0.0, 'default_video_input_number': 0, 'default_video_stream_number': 0, 'video_codec_name': 'hevc', 'video_profile': '(Main)', 'video_size': [1578, 720], 'video_bitrate': 7334, 'video_fps': 30.0, 'default_audio_input_number': 0, 'default_audio_stream_number': 1, 'audio_fps': 44100, 'audio_bitrate': 129, 'video_duration': 30.17, 'video_n_frames': 905}
/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/imageio_ffmpeg/binaries/ffmpeg-linux-x86_64-v7.0.2 -i data/clips/robot/bedroom_01/69.mp4 -loglevel error -f image2pipe -vf scale=1578:720 -sws_flags bicubic -pix_fmt rgb24 -vcodec rawvideo -
MoviePy - Writing audio in /tmp/tmpj4pfl7bl.wav
MoviePy - Done.
Unrecognized keys in rope_scaling for 'rope_type'='default': {'mrope_section'}
You are attempting to use Flash Attention 2 without specifying a torch dtype. This might lead to unexpected behaviour
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:08<00:00, 2.10s/it]
The image processor of type Qwen2VLImageProcessor is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with use_fast=False. Note that this behavior will be extended to all models in a future release.
You have video processor config saved in preprocessor.json file which is deprecated. Video processor configs should be saved in their own video_preprocessor.json file. You can rename the file or load and save the processor back which renames it automatically. Loading from preprocessor.json will be removed in v5.0.
/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/qwen_omni_utils/v2_5/audio_process.py:57: UserWarning: PySoundFile failed. Trying audioread instead.
audios.append(librosa.load(path, sr=16000)[0])
/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/librosa/core/audio.py:184: FutureWarning: librosa.core.audio.__audioread_load
Deprecated as of librosa version 0.10.0.
It will be removed in librosa version 1.0.
y, sr_native = __audioread_load(path, offset, duration, dtype)
qwen-vl-utils using torchvision to read video.
Keyword argument audios is not a valid argument for this processor and will be ignored.
Traceback (most recent call last):
File "/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py", line 113, in
streaming_process_video(video_graph, sample)
File "/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py", line 88, in streaming_process_video
process_segment(
File "/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py", line 59, in process_segment
episodic_memories, semantic_memories = generate_memories(
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/m3-agent-master/mmagent/memory_processing_qwen.py", line 180, in generate_memories
episodic_memories, semantic_memories = generate_all_memories(video_context, model_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/m3-agent-master/mmagent/memory_processing_qwen.py", line 158, in generate_all_memories
memories_string = get_response(messages)[0]
^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/m3-agent-master/mmagent/utils/chat_qwen.py", line 53, in get_response
inputs = processor(text=text, audios=audios, images=images, videos=videos, return_tensors="pt", padding=True, use_audio_in_video=USE_AUDIO_IN_VIDEO)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py", line 198, in call
text = self.replace_multimodal_special_tokens(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py", line 250, in replace_multimodal_special_tokens
audio_token_indices = np.arange(next(audio_lengths))
^^^^^^^^^^^^^^^^^^^
StopIteration
(m3_agent_env_311) (base) ubuntu@VM-0-13-ubuntu:/m3-agent-master$

[xungeer29]

(m3_agent_env_311) (base) ubuntu@VM-0-13-ubuntu:/m3-agent-master$ python m3_agent/memorization_memory_graphs.py --data_file data/data.jsonl /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:121: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider' warnings.warn( Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /home/ubuntu/.insightface/models/buffalo_l/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /home/ubuntu/.insightface/models/buffalo_l/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /home/ubuntu/.insightface/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /home/ubuntu/.insightface/models/buffalo_l/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /home/ubuntu/.insightface/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5 set det-size: (640, 640) /home/ubuntu/m3-agent-master/mmagent/voice_processing.py:35: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. pretrained_state = torch.load("models/pretrained_eres2netv2.ckpt", map_location='cpu') {'video_found': True, 'audio_found': True, 'metadata': {'major_brand': 'isom', 'minor_version': '512', 'compatible_brands': 'isomiso2mp41', 'encoder': 'Lavf58.29.100'}, 'inputs': [{'streams': [{'input_number': 0, 'stream_number': 0, 'stream_type': 'video', 'language': None, 'default': True, 'size': [1578, 720], 'bitrate': 7334, 'fps': 30.0, 'codec_name': 'hevc', 'profile': '(Main)', 'metadata': {'Metadata': '', 'handler_name': 'VideoHandler', 'vendor_id': '[0][0][0][0]'}}, {'input_number': 0, 'stream_number': 1, 'stream_type': 'audio', 'language': None, 'default': True, 'fps': 44100, 'bitrate': 129, 'metadata': {'Metadata': '', 'handler_name': 'SoundHandler', 'vendor_id': '[0][0][0][0]'}}], 'input_number': 0}], 'duration': 30.17, 'bitrate': 7646, 'start': 0.0, 'default_video_input_number': 0, 'default_video_stream_number': 0, 'video_codec_name': 'hevc', 'video_profile': '(Main)', 'video_size': [1578, 720], 'video_bitrate': 7334, 'video_fps': 30.0, 'default_audio_input_number': 0, 'default_audio_stream_number': 1, 'audio_fps': 44100, 'audio_bitrate': 129, 'video_duration': 30.17, 'video_n_frames': 905} /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/imageio_ffmpeg/binaries/ffmpeg-linux-x86_64-v7.0.2 -i data/clips/robot/bedroom_01/69.mp4 -loglevel error -f image2pipe -vf scale=1578:720 -sws_flags bicubic -pix_fmt rgb24 -vcodec rawvideo - MoviePy - Writing audio in /tmp/tmpj4pfl7bl.wav MoviePy - Done. Unrecognized keys in rope_scaling for 'rope_type'='default': {'mrope_section'} You are attempting to use Flash Attention 2 without specifying a torch dtype. This might lead to unexpected behaviour Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:08<00:00, 2.10s/it] The image processor of type Qwen2VLImageProcessor is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with use_fast=False. Note that this behavior will be extended to all models in a future release. You have video processor config saved in preprocessor.json file which is deprecated. Video processor configs should be saved in their own video_preprocessor.json file. You can rename the file or load and save the processor back which renames it automatically. Loading from preprocessor.json will be removed in v5.0. /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/qwen_omni_utils/v2_5/audio_process.py:57: UserWarning: PySoundFile failed. Trying audioread instead. audios.append(librosa.load(path, sr=16000)[0]) /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/librosa/core/audio.py:184: FutureWarning: librosa.core.audio.__audioread_load Deprecated as of librosa version 0.10.0. It will be removed in librosa version 1.0. y, sr_native = __audioread_load(path, offset, duration, dtype) qwen-vl-utils using torchvision to read video. Keyword argument audios is not a valid argument for this processor and will be ignored. Traceback (most recent call last): File "/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py", line 113, in streaming_process_video(video_graph, sample) File "/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py", line 88, in streaming_process_video process_segment( File "/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py", line 59, in process_segment episodic_memories, semantic_memories = generate_memories( ^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/m3-agent-master/mmagent/memory_processing_qwen.py", line 180, in generate_memories episodic_memories, semantic_memories = generate_all_memories(video_context, model_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/m3-agent-master/mmagent/memory_processing_qwen.py", line 158, in generate_all_memories memories_string = get_response(messages)[0] ^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/m3-agent-master/mmagent/utils/chat_qwen.py", line 53, in get_response inputs = processor(text=text, audios=audios, images=images, videos=videos, return_tensors="pt", padding=True, use_audio_in_video=USE_AUDIO_IN_VIDEO) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py", line 198, in call text = self.replace_multimodal_special_tokens( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py", line 250, in replace_multimodal_special_tokens audio_token_indices = np.arange(next(audio_lengths)) ^^^^^^^^^^^^^^^^^^^ StopIteration (m3_agent_env_311) (base) ubuntu@VM-0-13-ubuntu:/m3-agent-master$

这个StopIteration的问题是怎么解决的？

[jiTest921]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。

[xungeer29]

(m3_agent_env_311) (base) ubuntu@VM-0-13-ubuntu:/m3-agent-master$ python m3_agent/memorization_memory_graphs.py --data_file data/data.jsonl /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:121: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider' warnings.warn( Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /home/ubuntu/.insightface/models/buffalo_l/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /home/ubuntu/.insightface/models/buffalo_l/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /home/ubuntu/.insightface/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /home/ubuntu/.insightface/models/buffalo_l/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /home/ubuntu/.insightface/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5 set det-size: (640, 640) /home/ubuntu/m3-agent-master/mmagent/voice_processing.py:35: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. pretrained_state = torch.load("models/pretrained_eres2netv2.ckpt", map_location='cpu') {'video_found': True, 'audio_found': True, 'metadata': {'major_brand': 'isom', 'minor_version': '512', 'compatible_brands': 'isomiso2mp41', 'encoder': 'Lavf58.29.100'}, 'inputs': [{'streams': [{'input_number': 0, 'stream_number': 0, 'stream_type': 'video', 'language': None, 'default': True, 'size': [1578, 720], 'bitrate': 7334, 'fps': 30.0, 'codec_name': 'hevc', 'profile': '(Main)', 'metadata': {'Metadata': '', 'handler_name': 'VideoHandler', 'vendor_id': '[0][0][0][0]'}}, {'input_number': 0, 'stream_number': 1, 'stream_type': 'audio', 'language': None, 'default': True, 'fps': 44100, 'bitrate': 129, 'metadata': {'Metadata': '', 'handler_name': 'SoundHandler', 'vendor_id': '[0][0][0][0]'}}], 'input_number': 0}], 'duration': 30.17, 'bitrate': 7646, 'start': 0.0, 'default_video_input_number': 0, 'default_video_stream_number': 0, 'video_codec_name': 'hevc', 'video_profile': '(Main)', 'video_size': [1578, 720], 'video_bitrate': 7334, 'video_fps': 30.0, 'default_audio_input_number': 0, 'default_audio_stream_number': 1, 'audio_fps': 44100, 'audio_bitrate': 129, 'video_duration': 30.17, 'video_n_frames': 905} /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/imageio_ffmpeg/binaries/ffmpeg-linux-x86_64-v7.0.2 -i data/clips/robot/bedroom_01/69.mp4 -loglevel error -f image2pipe -vf scale=1578:720 -sws_flags bicubic -pix_fmt rgb24 -vcodec rawvideo - MoviePy - Writing audio in /tmp/tmpj4pfl7bl.wav MoviePy - Done. Unrecognized keys in rope_scaling for 'rope_type'='default': {'mrope_section'} You are attempting to use Flash Attention 2 without specifying a torch dtype. This might lead to unexpected behaviour Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:08<00:00, 2.10s/it] The image processor of type Qwen2VLImageProcessor is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with use_fast=False. Note that this behavior will be extended to all models in a future release. You have video processor config saved in preprocessor.json file which is deprecated. Video processor configs should be saved in their own video_preprocessor.json file. You can rename the file or load and save the processor back which renames it automatically. Loading from preprocessor.json will be removed in v5.0. /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/qwen_omni_utils/v2_5/audio_process.py:57: UserWarning: PySoundFile failed. Trying audioread instead. audios.append(librosa.load(path, sr=16000)[0]) /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/librosa/core/audio.py:184: FutureWarning: librosa.core.audio.__audioread_load Deprecated as of librosa version 0.10.0. It will be removed in librosa version 1.0. y, sr_native = __audioread_load(path, offset, duration, dtype) qwen-vl-utils using torchvision to read video. Keyword argument audios is not a valid argument for this processor and will be ignored. Traceback (most recent call last): File "/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py", line 113, in streaming_process_video(video_graph, sample) File "/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py", line 88, in streaming_process_video process_segment( File "/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py", line 59, in process_segment episodic_memories, semantic_memories = generate_memories( ^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/m3-agent-master/mmagent/memory_processing_qwen.py", line 180, in generate_memories episodic_memories, semantic_memories = generate_all_memories(video_context, model_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/m3-agent-master/mmagent/memory_processing_qwen.py", line 158, in generate_all_memories memories_string = get_response(messages)[0] ^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/m3-agent-master/mmagent/utils/chat_qwen.py", line 53, in get_response inputs = processor(text=text, audios=audios, images=images, videos=videos, return_tensors="pt", padding=True, use_audio_in_video=USE_AUDIO_IN_VIDEO) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py", line 198, in call text = self.replace_multimodal_special_tokens( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py", line 250, in replace_multimodal_special_tokens audio_token_indices = np.arange(next(audio_lengths)) ^^^^^^^^^^^^^^^^^^^ StopIteration (m3_agent_env_311) (base) ubuntu@VM-0-13-ubuntu:/m3-agent-master$

这个StopIteration的问题是怎么解决的？

找到问题了，chat_qwen.py中
inputs = processor(text=text, audios=audios, images=images, videos=videos, return_tensors="pt", padding=True, use_audio_in_video=USE_AUDIO_IN_VIDEO)
audios=audios 改成 audio=audios

[Celina-love-sweet]

（m3_agent_env_311）（基础）ubuntu@VM-0-13-ubuntu：/m3-agent-master$ python m3_agent/memorization_memory_graphs.py --data_file data/data.jsonl /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:121: UserWarning: 指定的提供程序“CUDAExecutionProvider”不在可用提供程序名称中。可用提供程序：“AzureExecutionProvider”、“CPUExecutionProvider”。warnings.warn( 已应用提供程序：['CPUExecutionProvider']，选项：{'CPUExecutionProvider': {}} 查找模型：/home/ubuntu/.insightface/models/buffalo_l/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0 应用提供程序：['CPUExecutionProvider']，选项：{'CPUExecutionProvider': {}} 查找模型：/home/ubuntu/.insightface/models/buffalo_l/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0 应用提供程序：['CPUExecutionProvider']，选项：{'CPUExecutionProvider': {}} 查找模型：/home/ubuntu/.insightface/models/buffalo_l/det_10g.onnx 检测 [1, 3, '?', '?'] 127.5 128.0 应用提供程序：['CPUExecutionProvider']，选项：{'CPUExecutionProvider': {}} 查找模型： /home/ubuntu/.insightface/models/buffalo_l/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0 应用提供程序：['CPUExecutionProvider']，选项：{'CPUExecutionProvider': {}} 查找模型：/home/ubuntu/.insightface/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5 设置检测大小：(640, 640) /home/ubuntu/m3-agent-master/mmagent/voice_processing.py:35: FutureWarning: 您正在使用torch.load（weights_only=False当前默认值），它隐式地使用默认的 pickle 模块。有可能构造恶意 pickle 数据，该数据将在反序列化期间执行任意代码（参见更多详情请参阅https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models）。在未来的版本中，默认值`weights_only`将更改为 false True。这将限制反序列化期间可以执行的函数。除非用户通过 allowlist 显式地将任意对象添加到允许列表中，否则将不再允许通过此模式加载它们torch.serialization.add_safe_globals。我们建议您开始设置 false。weights_only=True适用于任何您无法完全控制已加载文件的情况。如果您遇到与此实验性功能相关的任何问题，请在 GitHub 上提交 issue。 pretrained_state = torch.load("models/pretrained_eres2netv2.ckpt", map_location='cpu') {'video_found': True, 'audio_found': True, 'metadata': {'major_brand': 'isom', 'minor_version': '512', 'compatible_brands': 'isomiso2mp41', 'encoder': 'Lavf58.29.100'}, 'inputs': [{'streams': [{'input_number': 0, 'stream_number': 0, 'stream_type': 'video', 'language': None, 'default': True, 'size': [1578, 720], 'bitrate': 7334, 'fps': 30.0, 'codec_name': 'hevc', 'profile': '(Main)', 'metadata': {'Metadata': '', 'handler_name': 'VideoHandler', 'vendor_id': '[0][0][0][0]'}}, {'input_number': 0, 'stream_number': 1, 'stream_type': 'audio', 'language': None, 'default': True, 'fps': 44100, 'bitrate': 129, 'metadata': {'Metadata': '', 'handler_name': 'SoundHandler', 'vendor_id': '[0][0][0][0]'}}], 'input_number': 0}], 'duration': 30.17, 'bitrate': 7646, 'start': 0.0, 'default_video_input_number': 0, 'default_video_stream_number': 0, 'video_codec_name': 'hevc', 'video_profile': '(Main)', 'video_size': [1578, 720], 'video_bitrate': 7334, 'video_fps': 30.0, 'default_audio_input_number': 0, 'default_audio_stream_number': 1, 'audio_fps': 44100, 'audio_bitrate': 129, 'video_duration': 30.17, 'video_n_frames': 905} /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/imageio_ffmpeg/binaries/ffmpeg-linux-x86_64-v7.0.2 -i data/clips/robot/bedroom_01/69.mp4 -loglevel error -f image2pipe -vf scale=1578:720 -sws_flags bicubic -pix_fmt rgb24 -vcodec rawvideo - MoviePy - 正在将音频写入 /tmp/tmpj4pfl7bl.wav MoviePy - 完成。无法识别的键为rope_scaling'rope_type'='default'：{'mrope_section'} 您尝试使用 Flash Attention 2 时未指定 torch dtype。这可能会导致意外行为。正在加载检查点分片：100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:08<00:00, 2.10s/it]Qwen2VLImageProcessor现在，即使模型检查点是用慢速处理器保存的，该类型的图像处理器现在默认加载为快速处理器。这是一个重大更改，可能会产生略有不同的输出。要继续使用慢速处理器，请使用以下方式实例化此类use_fast=False：.请注意，此行为将在未来的版本中扩展到所有型号。您当前保存的视频处理器配置preprocessor.json文件已弃用。视频处理器配置应保存在单独的video_preprocessor.json文件中。您可以重命名该文件，或者重新加载并保存处理器配置（系统会自动重命名）。preprocessor.json将在 v5.0 版本中移除。/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/qwen_omni_utils/v2_5/audio_process.py:57: UserWarning: PySoundFile 失败。尝试使用 audioread 代替。audios.append(librosa.load(path, sr=16000)[0]) /home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/librosa/core/audio.py:184: FutureWarning: librosa.core.audio.__audioread_load 自 librosa 版本 0.10.0 起已弃用。将在 librosa 版本 1.0 中移除。 y, sr_native = __audioread_load(path, offset, duration, dtype) qwen-vl-utils 使用 torchvision 读取视频。关键字参数audios对于此处理器无效，将被忽略。回溯（最近一次调用）：文件“/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py”，第 113 行，在 streaming_process_video(video_graph, sample) 中；文件“/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py”，第 88 行，在 streaming_process_video 中；process_segment(；文件“/home/ubuntu/m3-agent-master/m3_agent/memorization_memory_graphs.py”，第 59 行，在 process_segment 中；episodic_memories, semantic_memories = generate_memories(；^^^^^^^^^^^^^^^^^^^；文件“/home/ubuntu/m3-agent-master/mmagent/memory_processing_qwen.py”，第 180 行，在 generate_memories 中episodic_memories, semantic_memories = generate_all_memories(video_context, model_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 文件 "/home/ubuntu/m3-agent-master/mmagent/memory_processing_qwen.py", 第 158 行，在 generate_all_memories 中 memories_string = get_response(messages)[0] ^^^^^^^^^^^^^^^^^^^^^^ 文件 "/home/ubuntu/m3-agent-master/mmagent/utils/chat_qwen.py", 第 53 行，在 get_response 中 inputs = processor(text=text, audios=audios, images=images, videos=videos, return_tensors="pt", padding=True, use_audio_in_video=USE_AUDIO_IN_VIDEO) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 文件“/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py”，第 198 行，在call text 中= self.replace_multimodal_special_tokens( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 文件“/home/ubuntu/m3-agent-master/m3_agent_env_311/lib/python3.11/site-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py”，第 250 行，在 replace_multimodal_special_tokens 中 audio_token_indices = np.arange(next(audio_lengths)) ^^^^^^^^^^^^^^^^^^^ StopIteration (m3_agent_env_311) (base) ubuntu@VM-0-13-ubuntu:/m3-agent-master$

这个StopIteration的问题是怎么解决的？

找到问题了，chat_qwen.py中 inputs = process(text=text,audios=audios,images=images,videos=videos,return_tensors="pt",padding=True,use_audio_in_video=USE_AUDIO_IN_VIDEO) audios=audios改成audio=audios

Could you please let me know how long it takes to generate a memory graph for one video? It's quite slow on my end.

[jiTest921]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。