-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathslurm_8866711.out
More file actions
316 lines (305 loc) · 26.1 KB
/
Copy pathslurm_8866711.out
File metadata and controls
316 lines (305 loc) · 26.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: fineGrained).
The token `llama3` has been saved to /dev/shm/hf-home/stored_tokens
Your token has been saved to /dev/shm/hf-home/token
Login successful.
The current active token is: `llama3`
/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
Unsloth: Failed to patch Gemma3ForConditionalGeneration.
🦥 Unsloth Zoo will now patch everything to make training faster!
INFO 04-20 12:10:00 [__init__.py:239] Automatically detected platform cuda.
[WARNING|logging.py:328] 2025-04-20 12:17:07,685 >> Unsloth 2025.3.19 patched 28 layers with 28 QKV layers, 28 O layers and 28 MLP layers.
Running GRPO script
Sun Apr 20 12:10:04 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P100-PCIE-12GB Off | 00000000:02:00.0 Off | 0 |
| N/A 36C P0 31W / 250W | 257MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1638626 C python 254MiB |
+-----------------------------------------------------------------------------------------+
Unsloth: vLLM does not work on older GPUs - will switch to Unsloth inference!
==((====))== Unsloth 2025.3.19: Fast Qwen2 patching. Transformers: 4.51.3. vLLM: 0.8.2.
\\ /| Tesla P100-PCIE-12GB. Num GPUs = 1. Max memory: 11.901 GB. Platform: Linux.
O^O/ \_/ \ Torch: 2.6.0+cu124. CUDA: 6.0. CUDA Toolkit: 12.4. Triton: 3.2.0
\ / Bfloat16 = FALSE. FA [Xformers = 0.0.29.post2. FA2 = False]
"-____-" Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
/home/users/ap794/final_project_distillLLM/minillm/results/qwen2.5/train/sft/qwen2.5-1.5B-Instruct/e10-bs1-lr1e-05-G2-N4-NN1/8000 does not have a padding token! Will use pad_token = <|vision_pad|>.
Sun Apr 20 12:17:05 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P100-PCIE-12GB Off | 00000000:02:00.0 Off | 0 |
| N/A 36C P0 33W / 250W | 1481MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1638626 C python 1478MiB |
+-----------------------------------------------------------------------------------------+
Sun Apr 20 12:17:09 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P100-PCIE-12GB Off | 00000000:02:00.0 Off | 0 |
| N/A 36C P0 32W / 250W | 1587MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1638626 C python 1584MiB |
+-----------------------------------------------------------------------------------------+
Unsloth: We now expect `per_device_train_batch_size` to be a multiple of `num_generations`.
We will change the batch size of 1 to the `num_generations` of 6
Generating train split: 0%| | 0/47780 [00:00<?, ? examples/s]Generating train split: 2%|▏ | 1000/47780 [00:00<00:37, 1233.08 examples/s]Generating train split: 4%|▍ | 2000/47780 [00:01<00:22, 2020.49 examples/s]Generating train split: 6%|▋ | 3000/47780 [00:01<00:19, 2285.01 examples/s]Generating train split: 8%|▊ | 4000/47780 [00:01<00:15, 2899.24 examples/s]Generating train split: 12%|█▏ | 5778/47780 [00:02<00:14, 2851.25 examples/s]Generating train split: 14%|█▍ | 6778/47780 [00:02<00:13, 2975.48 examples/s]Generating train split: 16%|█▋ | 7778/47780 [00:02<00:12, 3160.73 examples/s]Generating train split: 18%|█▊ | 8778/47780 [00:03<00:10, 3580.61 examples/s]Generating train split: 22%|██▏ | 10556/47780 [00:03<00:11, 3268.94 examples/s]Generating train split: 24%|██▍ | 11556/47780 [00:03<00:10, 3478.11 examples/s]Generating train split: 26%|██▋ | 12556/47780 [00:04<00:10, 3335.62 examples/s]Generating train split: 28%|██▊ | 13556/47780 [00:04<00:09, 3715.33 examples/s]Generating train split: 32%|███▏ | 15334/47780 [00:05<00:09, 3333.65 examples/s]Generating train split: 34%|███▍ | 16334/47780 [00:05<00:09, 3337.70 examples/s]Generating train split: 36%|███▋ | 17334/47780 [00:05<00:09, 3345.21 examples/s]Generating train split: 38%|███▊ | 18334/47780 [00:05<00:07, 4035.62 examples/s]Generating train split: 42%|████▏ | 20112/47780 [00:06<00:08, 3433.14 examples/s]Generating train split: 44%|████▍ | 21112/47780 [00:06<00:07, 3353.89 examples/s]Generating train split: 46%|████▋ | 22112/47780 [00:07<00:07, 3245.99 examples/s]Generating train split: 52%|█████▏ | 24890/47780 [00:07<00:06, 3450.53 examples/s]Generating train split: 54%|█████▍ | 25890/47780 [00:08<00:06, 3458.06 examples/s]Generating train split: 56%|█████▋ | 26890/47780 [00:08<00:06, 3440.02 examples/s]Generating train split: 58%|█████▊ | 27890/47780 [00:08<00:05, 3942.85 examples/s]Generating train split: 62%|██████▏ | 29668/47780 [00:09<00:05, 3515.76 examples/s]Generating train split: 64%|██████▍ | 30668/47780 [00:09<00:04, 3611.89 examples/s]Generating train split: 66%|██████▋ | 31668/47780 [00:09<00:04, 3643.23 examples/s]Generating train split: 68%|██████▊ | 32668/47780 [00:09<00:03, 4090.16 examples/s]Generating train split: 72%|███████▏ | 34446/47780 [00:10<00:04, 3279.75 examples/s]Generating train split: 74%|███████▍ | 35446/47780 [00:10<00:03, 3277.28 examples/s]Generating train split: 76%|███████▋ | 36446/47780 [00:11<00:03, 3519.08 examples/s]Generating train split: 78%|███████▊ | 37446/47780 [00:11<00:02, 4173.20 examples/s]Generating train split: 82%|████████▏ | 39224/47780 [00:11<00:02, 3293.43 examples/s]Generating train split: 84%|████████▍ | 40224/47780 [00:12<00:02, 3508.06 examples/s]Generating train split: 86%|████████▋ | 41224/47780 [00:12<00:01, 3341.35 examples/s]Generating train split: 88%|████████▊ | 42224/47780 [00:12<00:01, 3786.48 examples/s]Generating train split: 92%|█████████▏| 44002/47780 [00:13<00:01, 3412.63 examples/s]Generating train split: 94%|█████████▍| 45002/47780 [00:13<00:00, 3285.84 examples/s]Generating train split: 96%|█████████▋| 46002/47780 [00:13<00:00, 2994.45 examples/s]Generating train split: 98%|█████████▊| 47002/47780 [00:14<00:00, 3639.98 examples/s]Generating train split: 100%|██████████| 47780/47780 [00:14<00:00, 3395.18 examples/s]
Map: 0%| | 0/47780 [00:00<?, ? examples/s]Map: 2%|▏ | 724/47780 [00:00<00:06, 7143.09 examples/s]Map: 4%|▎ | 1708/47780 [00:00<00:10, 4370.58 examples/s]Map: 5%|▍ | 2322/47780 [00:00<00:13, 3352.21 examples/s]Map: 6%|▌ | 2959/47780 [00:00<00:11, 4026.29 examples/s]Map: 8%|▊ | 3608/47780 [00:00<00:13, 3304.42 examples/s]Map: 9%|▉ | 4340/47780 [00:01<00:16, 2637.94 examples/s]Map: 10%|█ | 5000/47780 [00:01<00:15, 2732.58 examples/s]Map: 12%|█▏ | 5673/47780 [00:01<00:12, 3356.77 examples/s]Map: 13%|█▎ | 6311/47780 [00:01<00:12, 3198.85 examples/s]Map: 14%|█▍ | 6865/47780 [00:02<00:11, 3609.25 examples/s]Map: 16%|█▌ | 7624/47780 [00:02<00:13, 2992.03 examples/s]Map: 17%|█▋ | 8000/47780 [00:02<00:15, 2538.27 examples/s]Map: 18%|█▊ | 8608/47780 [00:02<00:12, 3101.17 examples/s]Map: 19%|█▉ | 9313/47780 [00:02<00:12, 3016.45 examples/s]Map: 21%|██ | 10000/47780 [00:03<00:13, 2841.30 examples/s]Map: 22%|██▏ | 10628/47780 [00:03<00:10, 3389.61 examples/s]Map: 24%|██▎ | 11333/47780 [00:03<00:12, 2926.90 examples/s]Map: 25%|██▌ | 12000/47780 [00:03<00:12, 2935.15 examples/s]Map: 26%|██▋ | 12623/47780 [00:03<00:10, 3461.60 examples/s]Map: 28%|██▊ | 13320/47780 [00:04<00:10, 3251.21 examples/s]Map: 29%|██▉ | 13970/47780 [00:04<00:08, 3814.43 examples/s]Map: 31%|███ | 14647/47780 [00:04<00:09, 3450.23 examples/s]Map: 32%|███▏ | 15314/47780 [00:04<00:10, 3173.30 examples/s]Map: 33%|███▎ | 15970/47780 [00:04<00:08, 3747.81 examples/s]Map: 35%|███▍ | 16616/47780 [00:05<00:09, 3322.37 examples/s]Map: 36%|███▋ | 17352/47780 [00:05<00:10, 2947.60 examples/s]Map: 38%|███▊ | 18000/47780 [00:05<00:10, 2907.86 examples/s]Map: 39%|███▉ | 18675/47780 [00:05<00:08, 3508.71 examples/s]Map: 40%|████ | 19347/47780 [00:06<00:09, 3133.75 examples/s]Map: 42%|████▏ | 20000/47780 [00:06<00:09, 3059.37 examples/s]Map: 43%|████▎ | 20653/47780 [00:06<00:07, 3628.74 examples/s]Map: 45%|████▍ | 21331/47780 [00:06<00:07, 3426.96 examples/s]Map: 46%|████▌ | 21942/47780 [00:06<00:06, 3910.06 examples/s]Map: 47%|████▋ | 22647/47780 [00:06<00:07, 3481.60 examples/s]Map: 49%|████▉ | 23319/47780 [00:07<00:07, 3081.16 examples/s]Map: 50%|█████ | 24000/47780 [00:07<00:08, 2902.67 examples/s]Map: 51%|█████▏ | 24573/47780 [00:07<00:06, 3343.34 examples/s]Map: 52%|█████▏ | 25000/47780 [00:07<00:08, 2637.35 examples/s]Map: 54%|█████▎ | 25626/47780 [00:07<00:06, 3228.68 examples/s]Map: 55%|█████▌ | 26321/47780 [00:08<00:07, 2999.11 examples/s]Map: 56%|█████▋ | 26898/47780 [00:08<00:06, 3472.12 examples/s]Map: 58%|█████▊ | 27605/47780 [00:08<00:06, 2897.01 examples/s]Map: 59%|█████▊ | 28000/47780 [00:08<00:07, 2599.16 examples/s]Map: 60%|█████▉ | 28639/47780 [00:08<00:05, 3221.51 examples/s]Map: 61%|██████▏ | 29314/47780 [00:09<00:06, 2826.12 examples/s]Map: 63%|██████▎ | 30000/47780 [00:09<00:06, 2742.60 examples/s]Map: 64%|██████▍ | 30681/47780 [00:09<00:05, 3376.73 examples/s]Map: 66%|██████▌ | 31350/47780 [00:09<00:05, 3211.73 examples/s]Map: 67%|██████▋ | 31957/47780 [00:09<00:04, 3707.50 examples/s]Map: 68%|██████▊ | 32637/47780 [00:10<00:04, 3354.54 examples/s]Map: 70%|██████▉ | 33310/47780 [00:10<00:04, 2947.39 examples/s]Map: 71%|███████ | 33893/47780 [00:10<00:04, 3410.70 examples/s]Map: 72%|███████▏ | 34628/47780 [00:10<00:04, 3123.71 examples/s]Map: 74%|███████▍ | 35311/47780 [00:11<00:04, 2802.30 examples/s]Map: 75%|███████▌ | 35942/47780 [00:11<00:03, 3333.56 examples/s]Map: 77%|███████▋ | 36654/47780 [00:11<00:03, 2994.14 examples/s]Map: 78%|███████▊ | 37335/47780 [00:11<00:03, 3022.44 examples/s]Map: 79%|███████▉ | 37985/47780 [00:11<00:02, 3580.95 examples/s]Map: 81%|████████ | 38602/47780 [00:12<00:02, 3231.82 examples/s]Map: 82%|████████▏ | 39000/47780 [00:12<00:03, 2651.08 examples/s]Map: 83%|████████▎ | 39680/47780 [00:12<00:02, 3333.76 examples/s]Map: 84%|████████▍ | 40340/47780 [00:12<00:02, 2949.04 examples/s]Map: 86%|████████▌ | 40981/47780 [00:12<00:01, 3531.22 examples/s]Map: 87%|████████▋ | 41626/47780 [00:13<00:02, 3028.86 examples/s]Map: 89%|████████▊ | 42296/47780 [00:13<00:02, 2673.08 examples/s]Map: 90%|████████▉ | 42976/47780 [00:13<00:01, 3294.65 examples/s]Map: 91%|█████████▏| 43687/47780 [00:13<00:01, 2981.44 examples/s]Map: 93%|█████████▎| 44341/47780 [00:14<00:01, 2870.37 examples/s]Map: 94%|█████████▍| 45000/47780 [00:14<00:01, 2736.49 examples/s]Map: 95%|█████████▌| 45617/47780 [00:14<00:00, 3251.10 examples/s]Map: 97%|█████████▋| 46258/47780 [00:14<00:00, 2739.41 examples/s]Map: 98%|█████████▊| 46800/47780 [00:14<00:00, 3150.36 examples/s]Map: 99%|█████████▉| 47371/47780 [00:15<00:00, 2543.69 examples/s]Map: 100%|██████████| 47780/47780 [00:15<00:00, 2555.13 examples/s]Map: 100%|██████████| 47780/47780 [00:15<00:00, 3110.04 examples/s]
[WARNING|<string>:173] 2025-04-20 12:18:06,981 >> ==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1
\\ /| Num examples = 47,780 | Num Epochs = 1 | Total steps = 250
O^O/ \_/ \ Batch size per device = 6 | Gradient accumulation steps = 1
\ / Data Parallel GPUs = 1 | Total batch size (6 x 1 x 1) = 6
"-____-" Trainable parameters = 36,929,536/5,000,000,000 (0.74% trained)
wandb: WARNING The `run_name` is currently set to the same value as `TrainingArguments.output_dir`. If this was not intended, please specify a different run name by setting the `TrainingArguments.run_name` parameter.
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: alejandro-paredeslatorre to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
[{'content': '\n Respond in the following format:\n <think>\n ...\n </think>\n <answer>\n ...\n </answer>\n ', 'role': 'system'}, {'content': 'You will be given a competitive programming problem. Please reason step by step about the solution, then provide a complete implementation in C++17.\n\nYour solution must read input from standard input (cin), write output to standard output (cout).\nDo not include any debug prints or additional output.\n\nPut your final solution within a single code block:\n```cpp\n<your code here>\n```\n\n# Problem\n\nYou are given an array $$$a$$$ of $$$n$$$ integers, where $$$n$$$ is odd.\n\nIn one operation, you will remove two adjacent elements from the array $$$a$$$, and then concatenate the remaining parts of the array. For example, given the array $$$[4,7,4,2,9]$$$, we can obtain the arrays $$$[4,2,9]$$$ and $$$[4,7,9]$$$ by the operations $$$[\\underline{4,7}, 4,2,9] \\to [4,2,9]$$$ and $$$[4,7,\\underline{4,2},9] \\to [4,7,9]$$$ respectively. However, we cannot obtain the array $$$[7,2,9]$$$ as it requires deleting non-adjacent elements $$$[\\underline{4},7,\\underline{4},2,9]$$$.\n\nYou will repeatedly perform this operation until exact', 'role': 'user'}]
```cpp
#include <bits/stdc++.h>
using namespace std;
int main() {
int t;
cin >> t;
while (t--) {
int n;
cin >> n;
vector<int> a(n);
for (int i = 0; i < n; ++i) {
cin >> a[i];
}
int max_val = a[0];
for (int i = 2; i < n; i += 2) {
if (a[i] > max_val) {
max_val = a[i];
}
}
cout << max_val << '\n';
}
return 0;
}
```
Sun Apr 20 12:18:06 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P100-PCIE-12GB Off | 00000000:02:00.0 Off | 0 |
| N/A 36C P0 32W / 250W | 1587MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1638626 C python 1584MiB |
+-----------------------------------------------------------------------------------------+
wandb: ERROR failed to upsert bucket: returned error 403: {"data":{"upsertBucket":null},"errors":[{"message":"permission denied","path":["upsertBucket"],"extensions":{"code":"PERMISSION_ERROR"}}]}
Traceback (most recent call last):
File "/home/users/ap794/final_project_distillLLM/aleGRPO/src/main.py", line 236, in <module>
main()
File "/home/users/ap794/final_project_distillLLM/aleGRPO/src/main.py", line 188, in main
trainer.train()
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/trainer.py", line 2245, in train
return inner_training_loop(
File "<string>", line 223, in _fast_inner_training_loop
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/trainer_callback.py", line 506, in on_train_begin
return self.call_event("on_train_begin", args, state, control)
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/trainer_callback.py", line 556, in call_event
result = getattr(callback, event)(
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/integrations/integration_utils.py", line 930, in on_train_begin
self.setup(args, state, model, **kwargs)
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/integrations/integration_utils.py", line 857, in setup
self._wandb.init(
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1544, in init
wandb._sentry.reraise(e)
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/wandb/analytics/sentry.py", line 156, in reraise
raise exc.with_traceback(sys.exc_info()[2])
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1530, in init
return wi.init(run_settings, run_config)
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 987, in init
raise error
wandb.errors.errors.CommError: failed to upsert bucket: returned error 403: {"data":{"upsertBucket":null},"errors":[{"message":"permission denied","path":["upsertBucket"],"extensions":{"code":"PERMISSION_ERROR"}}]}
Traceback (most recent call last):
File "/home/users/ap794/final_project_distillLLM/aleGRPO/src/main.py", line 236, in <module>
main()
File "/home/users/ap794/final_project_distillLLM/aleGRPO/src/main.py", line 188, in main
trainer.train()
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/trainer.py", line 2245, in train
return inner_training_loop(
File "<string>", line 223, in _fast_inner_training_loop
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/trainer_callback.py", line 506, in on_train_begin
return self.call_event("on_train_begin", args, state, control)
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/trainer_callback.py", line 556, in call_event
result = getattr(callback, event)(
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/integrations/integration_utils.py", line 930, in on_train_begin
self.setup(args, state, model, **kwargs)
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/integrations/integration_utils.py", line 857, in setup
self._wandb.init(
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1544, in init
wandb._sentry.reraise(e)
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/wandb/analytics/sentry.py", line 156, in reraise
raise exc.with_traceback(sys.exc_info()[2])
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1530, in init
return wi.init(run_settings, run_config)
File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 987, in init
raise error
wandb.errors.errors.CommError: failed to upsert bucket: returned error 403: {"data":{"upsertBucket":null},"errors":[{"message":"permission denied","path":["upsertBucket"],"extensions":{"code":"PERMISSION_ERROR"}}]}
[1;34mwandb[0m:
[1;34mwandb[0m: 🚀 View run [33moutputs[0m at: [34mhttps://wandb.ai/alejandro-paredeslatorre/llama-cot-training/runs/hl9v6cyy[0m
[1;34mwandb[0m: Find logs at: [1;35mwandb/run-20250420_121808-hl9v6cyy/logs[0m
srun: error: linux41: task 0: Exited with exit code 1