slurm_8867684.out

The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: fineGrained).
The token `llama3` has been saved to /dev/shm/hf-home/stored_tokens
Your token has been saved to /dev/shm/hf-home/token
Login successful.
The current active token is: `llama3`
wandb: Appending key for api.wandb.ai to your netrc file: /home/users/ap794/.netrc
wandb: W&B API key is configured. Use `wandb login --relogin` to force relogin
/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
Unsloth: Failed to patch Gemma3ForConditionalGeneration.
🦥 Unsloth Zoo will now patch everything to make training faster!
INFO 04-20 21:53:30 [__init__.py:239] Automatically detected platform cuda.
[WARNING|logging.py:328] 2025-04-20 21:55:49,011 >> Unsloth 2025.3.19 patched 24 layers with 24 QKV layers, 24 O layers and 24 MLP layers.
Running GRPO script
Sun Apr 20 21:53:38 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P100-PCIE-12GB           Off |   00000000:02:00.0 Off |                    0 |
| N/A   41C    P0             31W /  250W |     257MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1750017      C   python                                        254MiB |
+-----------------------------------------------------------------------------------------+

Unsloth: vLLM does not work on older GPUs - will switch to Unsloth inference!
==((====))==  Unsloth 2025.3.19: Fast Qwen2 patching. Transformers: 4.51.3. vLLM: 0.8.2.
   \\   /|    Tesla P100-PCIE-12GB. Num GPUs = 1. Max memory: 11.901 GB. Platform: Linux.
O^O/ \_/ \    Torch: 2.6.0+cu124. CUDA: 6.0. CUDA Toolkit: 12.4. Triton: 3.2.0
\        /    Bfloat16 = FALSE. FA [Xformers = 0.0.29.post2. FA2 = False]
 "-____-"     Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
/home/users/ap794/final_project_distillLLM/minillm/results/qwen2.5/train/kd/Qwen2.5-0.5B-to-Qwen2.5-1.5B-sft/e10-bs8-lr1e-05-G1-N2-NN1-kd0.5/4000 does not have a padding token! Will use pad_token = <|vision_pad|>.
Sun Apr 20 21:55:47 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P100-PCIE-12GB           Off |   00000000:02:00.0 Off |                    0 |
| N/A   42C    P0             32W /  250W |     795MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1750017      C   python                                        792MiB |
+-----------------------------------------------------------------------------------------+

Sun Apr 20 21:55:50 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P100-PCIE-12GB           Off |   00000000:02:00.0 Off |                    0 |
| N/A   42C    P0             32W /  250W |     861MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1750017      C   python                                        858MiB |
+-----------------------------------------------------------------------------------------+

Unsloth: We now expect `per_device_train_batch_size` to be a multiple of `num_generations`.
We will change the batch size of 1 to the `num_generations` of 6

Map:   0%|          | 0/47780 [00:00<?, ? examples/s]
Map:   1%|▏         | 670/47780 [00:00<00:07, 6598.52 examples/s]
Map:   3%|▎         | 1336/47780 [00:00<00:12, 3736.91 examples/s]
Map:   4%|▍         | 2000/47780 [00:00<00:15, 2952.19 examples/s]
Map:   6%|▌         | 2634/47780 [00:00<00:12, 3711.17 examples/s]
Map:   7%|▋         | 3285/47780 [00:00<00:14, 3038.61 examples/s]
Map:   8%|▊         | 3941/47780 [00:01<00:11, 3718.83 examples/s]
Map:  10%|▉         | 4671/47780 [00:01<00:15, 2768.78 examples/s]
Map:  11%|█         | 5299/47780 [00:01<00:15, 2746.56 examples/s]
Map:  13%|█▎        | 6000/47780 [00:01<00:14, 2848.63 examples/s]
Map:  14%|█▍        | 6586/47780 [00:02<00:12, 3325.09 examples/s]
Map:  15%|█▍        | 7000/47780 [00:02<00:15, 2580.95 examples/s]
Map:  16%|█▌        | 7612/47780 [00:02<00:12, 3160.35 examples/s]
Map:  17%|█▋        | 8291/47780 [00:02<00:14, 2803.08 examples/s]
Map:  19%|█▊        | 8933/47780 [00:02<00:11, 3397.77 examples/s]
Map:  20%|██        | 9637/47780 [00:03<00:12, 3174.92 examples/s]
Map:  22%|██▏       | 10277/47780 [00:03<00:13, 2856.36 examples/s]
Map:  23%|██▎       | 10941/47780 [00:03<00:10, 3459.44 examples/s]
Map:  24%|██▍       | 11639/47780 [00:03<00:12, 2973.66 examples/s]
Map:  26%|██▌       | 12310/47780 [00:03<00:12, 2951.29 examples/s]
Map:  27%|██▋       | 12937/47780 [00:04<00:10, 3480.00 examples/s]
Map:  29%|██▊       | 13634/47780 [00:04<00:10, 3123.17 examples/s]
Map:  30%|██▉       | 14317/47780 [00:04<00:11, 3022.72 examples/s]
Map:  31%|███▏      | 14948/47780 [00:04<00:09, 3552.32 examples/s]
Map:  33%|███▎      | 15651/47780 [00:04<00:09, 3246.79 examples/s]
Map:  34%|███▍      | 16299/47780 [00:05<00:10, 2921.29 examples/s]
Map:  35%|███▌      | 16913/47780 [00:05<00:08, 3430.34 examples/s]
Map:  36%|███▋      | 17344/47780 [00:05<00:11, 2756.76 examples/s]
Map:  38%|███▊      | 17999/47780 [00:05<00:08, 3392.99 examples/s]
Map:  39%|███▉      | 18657/47780 [00:05<00:09, 3194.30 examples/s]
Map:  40%|████      | 19338/47780 [00:06<00:09, 2939.92 examples/s]
Map:  42%|████▏     | 20000/47780 [00:06<00:09, 2935.85 examples/s]
Map:  43%|████▎     | 20639/47780 [00:06<00:07, 3496.74 examples/s]
Map:  45%|████▍     | 21322/47780 [00:06<00:08, 3277.98 examples/s]
Map:  46%|████▌     | 21924/47780 [00:06<00:06, 3760.35 examples/s]
Map:  47%|████▋     | 22631/47780 [00:07<00:07, 3261.38 examples/s]
Map:  49%|████▉     | 23296/47780 [00:07<00:08, 2907.60 examples/s]
Map:  50%|█████     | 23962/47780 [00:07<00:06, 3501.40 examples/s]
Map:  51%|█████▏    | 24570/47780 [00:07<00:07, 3004.81 examples/s]
Map:  52%|█████▏    | 25000/47780 [00:08<00:09, 2415.13 examples/s]
Map:  54%|█████▎    | 25609/47780 [00:08<00:07, 2969.57 examples/s]
Map:  55%|█████▌    | 26314/47780 [00:08<00:07, 2856.69 examples/s]
Map:  56%|█████▋    | 26882/47780 [00:08<00:06, 3318.98 examples/s]
Map:  57%|█████▋    | 27304/47780 [00:08<00:07, 2650.77 examples/s]
Map:  58%|█████▊    | 27915/47780 [00:08<00:06, 3239.94 examples/s]
Map:  60%|█████▉    | 28627/47780 [00:09<00:06, 3018.18 examples/s]
Map:  61%|██████▏   | 29308/47780 [00:09<00:06, 2711.91 examples/s]
Map:  63%|██████▎   | 29995/47780 [00:09<00:05, 3356.38 examples/s]
Map:  64%|██████▍   | 30668/47780 [00:09<00:05, 3021.54 examples/s]
Map:  66%|██████▌   | 31331/47780 [00:10<00:05, 2973.90 examples/s]
Map:  67%|██████▋   | 31932/47780 [00:10<00:04, 3465.94 examples/s]
Map:  68%|██████▊   | 32614/47780 [00:10<00:04, 3194.23 examples/s]
Map:  69%|██████▉   | 33000/47780 [00:10<00:05, 2634.70 examples/s]
Map:  70%|███████   | 33586/47780 [00:10<00:04, 3163.83 examples/s]
Map:  71%|███████   | 34000/47780 [00:11<00:05, 2678.86 examples/s]
Map:  72%|███████▏  | 34607/47780 [00:11<00:04, 3283.18 examples/s]
Map:  74%|███████▍  | 35306/47780 [00:11<00:04, 2790.34 examples/s]
Map:  75%|███████▌  | 35924/47780 [00:11<00:03, 3355.59 examples/s]
Map:  77%|███████▋  | 36645/47780 [00:11<00:03, 2957.47 examples/s]
Map:  78%|███████▊  | 37327/47780 [00:12<00:03, 2985.66 examples/s]
Map:  79%|███████▉  | 37964/47780 [00:12<00:02, 3540.27 examples/s]
Map:  81%|████████  | 38589/47780 [00:12<00:02, 3172.23 examples/s]
Map:  82%|████████▏ | 39000/47780 [00:12<00:03, 2612.07 examples/s]
Map:  83%|████████▎ | 39662/47780 [00:12<00:02, 3268.05 examples/s]
Map:  84%|████████▍ | 40334/47780 [00:13<00:02, 2961.47 examples/s]
Map:  86%|████████▌ | 40975/47780 [00:13<00:01, 3545.52 examples/s]
Map:  87%|████████▋ | 41617/47780 [00:13<00:02, 3035.07 examples/s]
Map:  88%|████████▊ | 42284/47780 [00:13<00:02, 2640.31 examples/s]
Map:  90%|████████▉ | 42952/47780 [00:13<00:01, 3244.96 examples/s]
Map:  91%|█████████▏| 43654/47780 [00:14<00:01, 2894.01 examples/s]
Map:  93%|█████████▎| 44331/47780 [00:14<00:01, 2891.57 examples/s]
Map:  94%|█████████▍| 45000/47780 [00:14<00:01, 2758.40 examples/s]
Map:  95%|█████████▌| 45605/47780 [00:14<00:00, 3251.50 examples/s]
Map:  97%|█████████▋| 46254/47780 [00:15<00:00, 2757.69 examples/s]
Map:  98%|█████████▊| 46785/47780 [00:15<00:00, 3149.51 examples/s]
Map:  99%|█████████▉| 47361/47780 [00:15<00:00, 2489.85 examples/s]
Map: 100%|██████████| 47780/47780 [00:15<00:00, 2511.55 examples/s]
Map: 100%|██████████| 47780/47780 [00:15<00:00, 3038.72 examples/s]
[WARNING|<string>:173] 2025-04-20 21:56:08,278 >> ==((====))==  Unsloth - 2x faster free finetuning | Num GPUs used = 1
   \\   /|    Num examples = 47,780 | Num Epochs = 1 | Total steps = 5,000
O^O/ \_/ \    Batch size per device = 6 | Gradient accumulation steps = 1
\        /    Data Parallel GPUs = 1 | Total batch size (6 x 1 x 1) = 6
 "-____-"     Trainable parameters = 17,596,416/5,000,000,000 (0.35% trained)
wandb: WARNING The `run_name` is currently set to the same value as `TrainingArguments.output_dir`. If this was not intended, please specify a different run name by setting the `TrainingArguments.run_name` parameter.
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: alejandro-paredeslatorre (alejandro-paredeslatorre-duke-university) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.19.9
wandb: Run data is saved locally in /home/users/ap794/final_project_distillLLM/aleGRPO/wandb/run-20250420_215610-ve9s0q53
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run outputs
wandb: ⭐️ View project at https://wandb.ai/alejandro-paredeslatorre-duke-university/qwen-cot-training-qwen2.5-0.5B-v2
wandb: 🚀 View run at https://wandb.ai/alejandro-paredeslatorre-duke-university/qwen-cot-training-qwen2.5-0.5B-v2/runs/ve9s0q53
[{'content': '\n    Respond in the following format:\n    <think>\n    ...\n    </think>\n    <answer>\n    ...\n    </answer>\n    ', 'role': 'system'}, {'content': 'You will be given a competitive programming problem. Please reason step by step about the solution, then provide a complete implementation in C++17.\n\nYour solution must read input from standard input (cin), write output to standard output (cout).\nDo not include any debug prints or additional output.\n\nPut your final solution within a single code block:\n```cpp\n<your code here>\n```\n\n# Problem\n\nYou are given an array $$$a$$$ of $$$n$$$ integers, where $$$n$$$ is odd.\n\nIn one operation, you will remove two adjacent elements from the array $$$a$$$, and then concatenate the remaining parts of the array. For example, given the array $$$[4,7,4,2,9]$$$, we can obtain the arrays $$$[4,2,9]$$$ and $$$[4,7,9]$$$ by the operations $$$[\\underline{4,7}, 4,2,9] \\to [4,2,9]$$$ and $$$[4,7,\\underline{4,2},9] \\to [4,7,9]$$$ respectively. However, we cannot obtain the array $$$[7,2,9]$$$ as it requires deleting non-adjacent elements $$$[\\underline{4},7,\\underline{4},2,9]$$$.\n\nYou will repeatedly perform this operation until exact', 'role': 'user'}]
```cpp
#include <bits/stdc++.h>
using namespace std;

int main() {
    int t;
    cin >> t;
    while (t--) {
        int n;
        cin >> n;
        vector<int> a(n);
        for (int i = 0; i < n; ++i) {
            cin >> a[i];
        }
        int max_val = a[0];
        for (int i = 2; i < n; i += 2) {
            if (a[i] > max_val) {
                max_val = a[i];
            }
        }
        cout << max_val << '\n';
    }
    return 0;
}
```
Sun Apr 20 21:56:07 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P100-PCIE-12GB           Off |   00000000:02:00.0 Off |                    0 |
| N/A   42C    P0             32W /  250W |     861MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1750017      C   python                                        858MiB |
+-----------------------------------------------------------------------------------------+


  0%|          | 0/5000 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/src/main.py", line 235, in <module>
    main()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/src/main.py", line 187, in main
    trainer.train()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/trainer.py", line 2245, in train
    return inner_training_loop(
  File "<string>", line 314, in _fast_inner_training_loop
  File "<string>", line 31, in _unsloth_training_step
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/unsloth_compiled_cache/UnslothGRPOTrainer.py", line 1132, in compute_loss
    loss, completion_length, mean_kl = grpo_accumulated_loss(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/unsloth_compiled_cache/UnslothGRPOTrainer.py", line 199, in grpo_accumulated_loss
    loss, completion_length, mean_kl = UnslothEfficientGRPO.apply(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/unsloth_compiled_cache/UnslothGRPOTrainer.py", line 148, in forward
    accumulate_chunk(new_hidden_states_j, old_hidden_states_j, input_ids_j, mask_j, advantages_j, scaling)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 574, in _fn
    return fn(*args, **kwargs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1380, in __call__
    return self._torchdynamo_orig_callable(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 547, in __call__
    return _compile(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 986, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 715, in compile_inner
    return _compile_inner(code, one_graph, hooks, transform)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_utils_internal.py", line 95, in wrapper_function
    return function(*args, **kwargs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 750, in _compile_inner
    out_code = transform_code_object(code, transform)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1361, in transform_code_object
    transformations(instructions, code_options)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 231, in _fn
    return fn(*args, **kwargs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 662, in transform
    tracer.run()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2868, in run
    super().run()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
    while self.step():
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3048, in RETURN_VALUE
    self._return(inst)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3033, in _return
    self.output.compile_subgraph(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1101, in compile_subgraph
    self.compile_and_call_fx_graph(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1382, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1432, in call_user_compiler
    return self._call_user_compiler(gm)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1483, in _call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1462, in _call_user_compiler
    compiled_fn = compiler_fn(gm, self.example_inputs())
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 130, in __call__
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 130, in __call__
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/__init__.py", line 2340, in __call__
    return compile_fx(model_, inputs_, config_patches=self.config)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1552, in compile_fx
    return compile_fx(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1863, in compile_fx
    return aot_autograd(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 83, in __call__
    cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1155, in aot_module_simplified
    compiled_fn = dispatch_and_compile()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1131, in dispatch_and_compile
    compiled_fn, _ = create_aot_dispatcher_function(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 580, in create_aot_dispatcher_function
    return _create_aot_dispatcher_function(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 830, in _create_aot_dispatcher_function
    compiled_fn, fw_metadata = compiler_fn(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 203, in aot_dispatch_base
    compiled_fw = compiler(fw_module, updated_flat_args)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 489, in __call__
    return self.compiler_fn(gm, example_inputs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1741, in fw_compiler_base
    return inner_compile(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 569, in compile_fx_inner
    return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 102, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 685, in _compile_fx_inner
    mb_compiled_graph = fx_codegen_and_compile(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1129, in fx_codegen_and_compile
    return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1044, in codegen_and_compile
    compiled_fn = graph.compile_to_module().call
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2027, in compile_to_module
    return self._compile_to_module()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2033, in _compile_to_module
    self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1964, in codegen
    self.scheduler = Scheduler(self.operations)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 1798, in __init__
    self._init(nodes)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 1816, in _init
    self.nodes = [self.create_scheduler_node(n) for n in nodes]
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 1816, in <listcomp>
    self.nodes = [self.create_scheduler_node(n) for n in nodes]
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 1947, in create_scheduler_node
    return SchedulerNode(self, node)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 893, in __init__
    self._compute_attrs()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 907, in _compute_attrs
    group_fn = self.scheduler.get_backend(device).group_fn
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 3441, in get_backend
    self.backends[device] = self.create_backend(device)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 3428, in create_backend
    raise RuntimeError(
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
RuntimeError: Found Tesla P100-PCIE-12GB which is too old to be supported by the triton GPU compiler, which is used as the backend. Triton only supports devices of CUDA Capability >= 7.0, but your device is of CUDA capability 6.0

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

Traceback (most recent call last):
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/src/main.py", line 235, in <module>
    main()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/src/main.py", line 187, in main
    trainer.train()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/transformers/trainer.py", line 2245, in train
    return inner_training_loop(
  File "<string>", line 314, in _fast_inner_training_loop
  File "<string>", line 31, in _unsloth_training_step
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/unsloth_compiled_cache/UnslothGRPOTrainer.py", line 1132, in compute_loss
    loss, completion_length, mean_kl = grpo_accumulated_loss(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/unsloth_compiled_cache/UnslothGRPOTrainer.py", line 199, in grpo_accumulated_loss
    loss, completion_length, mean_kl = UnslothEfficientGRPO.apply(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/unsloth_compiled_cache/UnslothGRPOTrainer.py", line 148, in forward
    accumulate_chunk(new_hidden_states_j, old_hidden_states_j, input_ids_j, mask_j, advantages_j, scaling)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 574, in _fn
    return fn(*args, **kwargs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1380, in __call__
    return self._torchdynamo_orig_callable(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 547, in __call__
    return _compile(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 986, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 715, in compile_inner
    return _compile_inner(code, one_graph, hooks, transform)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_utils_internal.py", line 95, in wrapper_function
    return function(*args, **kwargs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 750, in _compile_inner
    out_code = transform_code_object(code, transform)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1361, in transform_code_object
    transformations(instructions, code_options)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 231, in _fn
    return fn(*args, **kwargs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 662, in transform
    tracer.run()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2868, in run
    super().run()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
    while self.step():
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3048, in RETURN_VALUE
    self._return(inst)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3033, in _return
    self.output.compile_subgraph(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1101, in compile_subgraph
    self.compile_and_call_fx_graph(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1382, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1432, in call_user_compiler
    return self._call_user_compiler(gm)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1483, in _call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1462, in _call_user_compiler
    compiled_fn = compiler_fn(gm, self.example_inputs())
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 130, in __call__
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 130, in __call__
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/__init__.py", line 2340, in __call__
    return compile_fx(model_, inputs_, config_patches=self.config)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1552, in compile_fx
    return compile_fx(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1863, in compile_fx
    return aot_autograd(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 83, in __call__
    cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1155, in aot_module_simplified
    compiled_fn = dispatch_and_compile()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1131, in dispatch_and_compile
    compiled_fn, _ = create_aot_dispatcher_function(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 580, in create_aot_dispatcher_function
    return _create_aot_dispatcher_function(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 830, in _create_aot_dispatcher_function
    compiled_fn, fw_metadata = compiler_fn(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 203, in aot_dispatch_base
    compiled_fw = compiler(fw_module, updated_flat_args)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 489, in __call__
    return self.compiler_fn(gm, example_inputs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1741, in fw_compiler_base
    return inner_compile(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 569, in compile_fx_inner
    return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 102, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 685, in _compile_fx_inner
    mb_compiled_graph = fx_codegen_and_compile(
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1129, in fx_codegen_and_compile
    return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1044, in codegen_and_compile
    compiled_fn = graph.compile_to_module().call
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2027, in compile_to_module
    return self._compile_to_module()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2033, in _compile_to_module
    self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1964, in codegen
    self.scheduler = Scheduler(self.operations)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 1798, in __init__
    self._init(nodes)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 1816, in _init
    self.nodes = [self.create_scheduler_node(n) for n in nodes]
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 1816, in <listcomp>
    self.nodes = [self.create_scheduler_node(n) for n in nodes]
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 1947, in create_scheduler_node
    return SchedulerNode(self, node)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 893, in __init__
    self._compute_attrs()
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 907, in _compute_attrs
    group_fn = self.scheduler.get_backend(device).group_fn
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 3441, in get_backend
    self.backends[device] = self.create_backend(device)
  File "/home/users/ap794/final_project_distillLLM/aleGRPO/grpo_venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 3428, in create_backend
    raise RuntimeError(
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
RuntimeError: Found Tesla P100-PCIE-12GB which is too old to be supported by the triton GPU compiler, which is used as the backend. Triton only supports devices of CUDA Capability >= 7.0, but your device is of CUDA capability 6.0

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

wandb: 
wandb: 🚀 View run outputs at: https://wandb.ai/alejandro-paredeslatorre-duke-university/qwen-cot-training-qwen2.5-0.5B-v2/runs/ve9s0q53
wandb: Find logs at: wandb/run-20250420_215610-ve9s0q53/logs
srun: error: linux45: task 0: Exited with exit code 1