Skip to content

Latest commit

 

History

History
101 lines (73 loc) · 3.67 KB

README.md

File metadata and controls

101 lines (73 loc) · 3.67 KB

LLM-D Inference Scheduler with preprocessing support

Overview

This repository contains a custom fork of the llm-d-inference-scheduler with modifications to add chat completions preprocessing functionality.

Repository Information

  • Original Repository: https://github.com/llm-d/llm-d-inference-scheduler.git
  • Fork Location: /Users/guygirmonsky/llm-d-build/llm-d-inference-scheduler
  • Custom Image: ghcr.io/guygir/llm-d-inference-scheduler:latest
  • Base Commit: 82f0cf2 (Makefile fixes #322)

Quick Start

Prerequisites

  • Docker/Podman
  • Go 1.25+
  • Python 3.12 development headers

Building the Image

cd /Users/guygirmonsky/llm-d-build/llm-d-inference-scheduler
TARGETARCH=amd64 TARGETOS=linux make image-build

Changes Made

1. Repository Structure Changes

Added Local Repository Clones

  • gateway-api-inference-extension/ - Local clone of the Gateway API Inference Extension
  • llm-d-kv-cache-manager/ - Local clone of the KV Cache Manager with preprocessing capabilities

Why Local Clones Were Necessary: The upstream repositories had structural differences, missing functionality, and version compatibility issues that prevented successful builds:

  • Package Structure Mismatch: Upstream had pkg/epp/config/loader but code expected pkg/epp/common/config/loader
  • API Version Issues: Code expected api/v1alpha2 but upstream had apix/v1alpha2
  • Missing Preprocessing Code: Chat completions preprocessing code wasn't available in upstream
  • Version Compatibility: Upstream v0.5.1 didn't have required functionality

Go Module Configuration (go.mod)

// Added replace directives to point to local clones:
replace github.com/llm-d/llm-d-kv-cache-manager => ./llm-d-kv-cache-manager
replace sigs.k8s.io/gateway-api-inference-extension => ./gateway-api-inference-extension

2. Dockerfile Modifications

Builder Stage Changes

  • Added Python 3.12 development tools and runtime
  • Integrated Python dependencies for chat completions preprocessing
  • Added CGO environment variables for Python integration
  • Modified build process to include Python libraries

Runtime Stage Changes

  • Installed Python 3.12 runtime in final image
  • Copied Python wrapper files and dependencies
  • Set proper environment variables for Python library discovery

3. Go Code Modifications

Main Application Entry Point (cmd/epp/main.go)

  • Fixed plugin registration to use custom plugins
  • Removed call to non-existent runner.RegisterAllPlugins()

Precise Prefix Cache Scorer (pkg/plugins/scorer/precise_prefix_cache.go)

  • Added chat completions preprocessing import
  • Modified Score function to preprocess requests before cache lookup
  • Added preprocessRequest function for chat completion handling

Profile Handler (pkg/plugins/profile/pd_profile_handler.go)

  • Added same preprocessing functionality as the scorer
  • Modified Pick function to use preprocessed prompts for calculations
  • Ensures consistent preprocessing across all components

Troubleshooting

Common Issues

  1. Python.h not found during build

    • Ensure Python 3.12 development headers are installed
    • Check that python3.12-devel package is available

  2. Import errors during Go build

    • Verify local repository clones are present
    • Check that go.mod replace directives are correct
    • Ensure all required files are copied to the build context

  3. Runtime Python errors

    • Verify Python 3.12 runtime is installed in the final image
    • Check that PYTHONPATH environment variable is set correctly
    • Ensure all Python dependencies are properly installed