OfflineOnDevice LLM Android

Run large language models fully on-device. No internet required after download. No API keys. No data leaves your phone.

Built with llama.cpp + Jetpack Compose + Clean Architecture.

---

Demo

Models Screen	Chat Screen
Download, pause, resume, delete	Streaming tokens, multi-turn memory

---

Features

• Fully offline inference — model runs on CPU/GPU(best possible hardware), no network needed after download
• In-app model download — browse and download GGUF models directly from the app
• Resumable downloads — pause and continue downloads, even after app restart
• Streaming output — tokens appear as they generate, not all at once
• Persistent chat history — conversations saved locally via Room
• Multi-turn memory — KV cache preserved across turns in the same session
• Clean Architecture — domain / data / presentation layers, fully separated
• ARM optimised — NEON, DOTPROD, I8MM flags enabled for modern Snapdragon and Tensor chips

---

Architecture

Presentation (Compose + ViewModel)
        ↓
   Domain (UseCases + Interfaces)
        ↓
  Data (Repositories + Room + JNI)
        ↓
  llama_bridge.cpp (C++ JNI)
        ↓
     llama.cpp (GGUF inference)

Rule: each layer only talks to the one below it. UI never touches the database. Domain never imports Android.

---

Tech Stack

Layer	Library
UI	Jetpack Compose + Material 3
State	ViewModel + StateFlow
DI	Hilt
DB	Room
Native	llama.cpp via JNI (C++17)
Build	CMake 3.22 + NDK r27
Min SDK	Android 8.0 (API 26)

---

Getting Started

Prerequisites

• Android Studio Hedgehog or newer
• NDK r27 installed (SDK Manager → SDK Tools → NDK)
• CMake 3.22+ installed (SDK Manager → SDK Tools → CMake)
• 4 GB free disk space for build + models

1. Clone

git clone https://github.com/MENT0Z/Offline-on-Device-LLM-Android
cd OfflineonDeviceLLMAndroid

2. Clone llama.cpp

mkdir third_party
git clone https://github.com/ggerganov/llama.cpp third_party/llama.cpp
cd third_party/llama.cpp
git checkout b3729        # pinned stable release
cd ../..

3. Build and Run

Android Studio → Sync Project → Run
(Note->Run the release version for the fast inference , build version is slow to run)

The app seeds a model catalogue on first launch. Go to the Models tab, tap Download on any model, wait for it to finish, tap Load, then switch to Chat.

---

Supported Models

Any GGUF model works. These are pre-configured in the catalogue:

Model	Size	RAM needed	Best for
TinyLlama 1.1B Q2_K	270 MB	~500 MB	Testing, low-end devices
TinyLlama 1.1B Q4_K_M	670 MB	~900 MB	Low-end devices
Phi-2 2.7B Q4_K_M	1.7 GB	~2.2 GB	Mid-range devices
Mistral 7B Instruct Q4_K_M	4.4 GB	~6 GB	Flagship phones only

Add a custom model

Edit ModelCatalogue.kt and add an entry:

LLMModel(
    id           = "your-model-id",
    name         = "Your Model Name",
    fileName     = "your-model.gguf",
    downloadUrl  = "https://huggingface.co/.../your-model.gguf",
    fileSizeBytes = 1_700_000_000L,
    quantization  = "Q4_K_M",
    parameterCount = "3B",
    contextLength  = 4096,
    description   = "Your description here.",
)

---

Performance

Tested on Pixel 8 (Tensor G3):

Model	Time to first token	Tokens/sec
TinyLlama 1.1B Q2_K	~1s	10–15 t/s
TinyLlama 1.1B Q4_K_M	~1s	10–12 t/s
Phi-2 2.7B Q4_K_M	~(4-5)s	3–5 t/s

Performance varies by device. ARM chips with DOTPROD and I8MM support (Snapdragon 8 Gen 2+, Tensor G3+) see the best results.

---

Project Structure

app/src/main/
├── cpp/                        C++ JNI bridge
│   ├── CMakeLists.txt
│   ├── llama_bridge.h
│   └── llama_bridge.cpp
├── java/com/mruraza/.../
│   ├── domain/
│   │   ├── model/              Pure Kotlin data classes
│   │   ├── repository/         Interfaces (ports)
│   │   └── usecase/            One class, one job
│   ├── data/
│   │   ├── local/              Room DB, DAOs, JNI wrapper
│   │   └── repository/         Interface implementations
│   ├── di/                     Hilt modules
│   └── presentation/
│       ├── vm/                 ViewModels
│       └── ui/                 Compose screens
└── third_party/
    └── llama.cpp/              Git submodule (not committed)

---

How It Works

1. Download — ModelRepositoryImpl streams the .gguf file over HTTP with Range header support for resume. Progress emitted as a Flow<DownloadState>.

2. Load — LlamaLocalDataSource calls nativeLoadModel() via JNI, which calls llama_model_load_from_file() and llama_init_from_model() in C++. Returns a Long handle (pointer to a LlamaSession struct).

3. Inference — nativeGenerate() tokenises the prompt, decodes in batches of 512 tokens, then auto-regressively samples new tokens using a llama_sampler_chain. Each token is passed back to Kotlin via the TokenCallback interface, emitted into a callbackFlow, and collected by the ViewModel as streaming text.

4. Memory — KV cache is preserved between turns. Calling nativeClearContext() resets it for a new conversation without reloading model weights.

---

Configuration

Tune inference defaults in InferenceConfig.kt:

---

Acknowledgements

• llama.cpp by Georgi Gerganov — the entire inference engine
• TheBloke on HuggingFace — GGUF model quantisations