Reworking tune ls

[SalmanMohammadi]

Currently, tune ls is a bit unweildy. Can we make it better?

@joecummings

[felipemello1]

@SalmanMohammadi , do you have some example of what you would like to change?

[SalmanMohammadi]

Good question. This is the current output:

(venv) combuter:torchtune salmanmohammadi$ tune ls
W0805 18:37:08.061000 8424225472 torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
RECIPE                                   CONFIG                                  
full_finetune_single_device              llama2/7B_full_low_memory               
                                         code_llama2/7B_full_low_memory          
                                         llama3/8B_full_single_device            
                                         llama3_1/8B_full_single_device          
                                         mistral/7B_full_low_memory              
                                         phi3/mini_full_low_memory               
                                         qwen2/7B_full_low_memory                
full_finetune_distributed                llama2/7B_full                          
                                         llama2/13B_full                         
                                         llama3/8B_full                          
                                         llama3_1/8B_full                        
                                         llama3/70B_full                         
                                         llama3_1/70B_full                       
                                         mistral/7B_full                         
                                         gemma/2B_full                           
                                         gemma/7B_full                           
                                         phi3/mini_full                          
                                         qwen2/7B_full                           
lora_finetune_single_device              llama2/7B_lora_single_device            
                                         llama2/7B_qlora_single_device           
                                         code_llama2/7B_lora_single_device       
                                         code_llama2/7B_qlora_single_device      
                                         llama3/8B_lora_single_device            
                                         llama3_1/8B_lora_single_device          
                                         llama3/8B_qlora_single_device           
                                         llama3_1/8B_qlora_single_device         
                                         llama2/13B_qlora_single_device          
                                         mistral/7B_lora_single_device           
                                         mistral/7B_qlora_single_device          
                                         gemma/2B_lora_single_device             
                                         gemma/2B_qlora_single_device            
                                         gemma/7B_lora_single_device             
                                         gemma/7B_qlora_single_device            
                                         phi3/mini_lora_single_device            
                                         phi3/mini_qlora_single_device           
                                         qwen2/7B_lora_single_device             
lora_dpo_single_device                   llama2/7B_lora_dpo_single_device        
lora_dpo_distributed                     llama2/7B_lora_dpo                      
lora_finetune_distributed                llama2/7B_lora                          
                                         llama2/13B_lora                         
                                         llama2/70B_lora                         
                                         llama3/70B_lora                         
                                         llama3_1/70B_lora                       
                                         llama3/8B_lora                          
                                         llama3_1/8B_lora                        
                                         mistral/7B_lora                         
                                         gemma/2B_lora                           
                                         gemma/7B_lora                           
                                         phi3/mini_lora                          
                                         qwen2/7B_lora                           
lora_finetune_fsdp2                      llama2/7B_lora                          
                                         llama2/13B_lora                         
                                         llama2/70B_lora                         
                                         llama2/7B_qlora                         
                                         llama2/70B_qlora                        
generate                                 generation                              
eleuther_eval                            eleuther_evaluation                     
quantize                                 quantization                            
qat_distributed                          llama2/7B_qat_full                      
                                         llama3/8B_qat_full    

Personally, I find this almost impossible to parse - my eyes go all blurry trying to scan through this to find a config I might be interested in.

This is scaling quickly with new models and new recipes and I'd like to open up discussion on how we can improve UX here. Will add an idea I have soon.

[felipemello1]

this is another discussion, but i wonder if we should delete qlora configs, and just make it a flag in lora configs, so we can reduce the amount of configs

[RdoubleA]

I recall @joecummings mentioning we can call tune ls <recipe> instead of listing all recipes. We can also split into a "models" column and a "config" column so it's slightly easier to read

[SalmanMohammadi]

this is another discussion, but i wonder if we should delete qlora configs, and just make it a flag in lora configs, so we can reduce the amount of configs

This is a really interesting point. My very radical suggestion here was going to be storing only template configs, and forcing users to always specify config parameters which aren't common across models (tokenizer, model, et cetera). This is crazy though.

We're balancing keeping things clean and easily discoverable/accessible, with offering sufficient abstractions to allow the lowest-common-denominator technical user to use our features. Right now, the surface area of no-code pathways for our features is pretty good, so I'd suggest we think about when we're reducing that and asking the user to figure things out by themselves.

I recall @joecummings mentioning we can call tune ls instead of listing all recipes. We can also split into a "models" column and a "config" column so it's slightly easier to read

I was thinking along similar lines. I'd looove to see some fuzzy matching here, so we might have:

• tune ls by default requires some entity to query over, either a config or model, exempli gratia; tune ls llama2, tune ls lora, tune ls llama2_7b, tune ls single_device et cetera.
• tune ls as is is replaced by tune ls --all and formatted nicer

[SalmanMohammadi]

bumping #1163 as relevant here. subsequently ccing @pbontrager for thoughts