-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathslurm_8864602.out
More file actions
86 lines (78 loc) · 6.39 KB
/
Copy pathslurm_8864602.out
File metadata and controls
86 lines (78 loc) · 6.39 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
Unsloth: Failed to patch Gemma3ForConditionalGeneration.
🦥 Unsloth Zoo will now patch everything to make training faster!
INFO 04-19 17:19:11 [__init__.py:239] Automatically detected platform cuda.
Running GRPO script
Sat Apr 19 17:19:14 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A6000 Off | 00000000:17:00.0 Off | Off |
| 30% 34C P2 77W / 300W | 267MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1345193 C python 260MiB |
+-----------------------------------------------------------------------------------------+
Traceback (most recent call last):
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/unsloth_zoo/compiler.py", line 391, in create_new_function
new_module, old_path = import_module(compile_folder, name)
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/unsloth_zoo/compiler.py", line 386, in import_module
new_module = importlib.import_module(name)
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 879, in exec_module
File "<frozen importlib._bootstrap_external>", line 1017, in get_code
File "<frozen importlib._bootstrap_external>", line 947, in source_to_code
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/users/ap794/final_project_distillLLM/aleGRPO/unsloth_compiled_cache/unsloth_compiled_module_gpt2.py", line 635
= loss_fct(shift_logits, shift_labels)
IndentationError: unexpected indent
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/unsloth_zoo/compiler.py", line 416, in create_new_function
spec.loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 879, in exec_module
File "<frozen importlib._bootstrap_external>", line 1017, in get_code
File "<frozen importlib._bootstrap_external>", line 947, in source_to_code
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/tmp/unsloth_compiled_cache/unsloth_compiled_module_gpt2.py", line 635
= loss_fct(shift_logits, shift_labels)
IndentationError: unexpected indent
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/unsloth_zoo/compiler.py", line 2033, in unsloth_compile_transformers
combined_module = create_new_function(
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/unsloth_zoo/compiler.py", line 418, in create_new_function
raise RuntimeError(f"Direct module loading failed for {name}: {e}")
RuntimeError: Direct module loading failed for unsloth_compiled_module_gpt2: unexpected indent (unsloth_compiled_module_gpt2.py, line 635)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/users/ap794/final_project_distillLLM/aleGRPO/src/main.py", line 233, in <module>
main()
File "/home/users/ap794/final_project_distillLLM/aleGRPO/src/main.py", line 110, in main
model, tokenizer = FastLanguageModel.from_pretrained(
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/unsloth/models/loader.py", line 308, in from_pretrained
return FastModel.from_pretrained(
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/unsloth/models/loader.py", line 666, in from_pretrained
model_types, supports_sdpa = unsloth_compile_transformers(
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/unsloth/models/_utils.py", line 1184, in unsloth_compile_transformers
_unsloth_compile_transformers(
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/unsloth_zoo/compiler.py", line 2045, in unsloth_compile_transformers
raise RuntimeError(exception)
RuntimeError: Direct module loading failed for unsloth_compiled_module_gpt2: unexpected indent (unsloth_compiled_module_gpt2.py, line 635)
srun: error: compsci-cluster-fitz-05: task 0: Exited with exit code 1