Skip to content

Hot-swap LoRA with updated llama.cpp #212

Closed
@ltoniazzi

Description

@ltoniazzi

Describe the feature

Feature

Since the PRs linked below in llama.cpp, it is possible to how-swap lora adapters. This allows to personalise NPCs and other GenAi game features by fine-tuning adapters that then can be quickly swapped on the same base model in memory.

Since llama.cpp has is being updated in #209. It would be nice to check how easily one can integrate this feature.

Todo list

  • bin files for adapters are now deprecated in favour of new gguf files, which should be ameneded in the documentation, if appearing.
  • Add in documentation link to how to convert adapters to gguf.
  • Add example on performing hot-swap
  • Test on using the new gguf formats should be run (I suspect the new adapter will automatically be used in hot-swapping mode)
  • Test using multiple adapters and hot swapping them, add code if needed, and add an example in the examples folder.

Related links

Hot lora PRs in llama.cpp:

Discord threads:

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions