[FEAT] LLM server with CPU and GPU Support #15
Description
Objective
As a Neon Hub user, I want to have access to the latest in private, local artificial intelligence technology. LLMs were left out of the initial work effort for the Neon Hub but should be included in one of the next ones. In order to keep the scope manageable, it should be divided into phases.
Initial Implementation Requirements
- Identify the preferred LLM server experience
- Implement LLM server support for CPU inference
- Implement LLM server support for NVIDIA GPU inference
- Implement LLM server support for AMD GPU inference
- Implement LLM server support for Metal (Apple Silicon) inference
Other Considerations
No response