Skip to content

feat: mixed precision quantization #159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

feat: mixed precision quantization #159

wants to merge 1 commit into from
+163 −0

Conversation

@rebel-jaehwang
Copy link
Contributor

@rebel-jaehwang rebel-jaehwang commented Nov 18, 2025

🚀 Summary of Changes

  • Add MPLinearKernel for RBLN, which supports compressed-tensors models with w4a16 channel/group quant and w8a16 channel quant. The compiler support will be in included in the next release.
  • Monkey-patch the MPLinearKernel selector as the upstream version doesn't support OOT kernel registration.

✅ Type of Change

  • ✨ Feature (feature)
  • 🧠 Model support (model)
  • 🧬 Core engine changes (core)
  • 🛠 Bug fix (bug-fix)
  • ⚙️ Performance improvement (perf)
  • 🔁 Refactor or code cleanup (refactor)
  • 📄 Documentation (docs)
  • ❓ Other (other): please describe

🧪 How to Test

Run examples/experimental/offline_inference_basic.py with model changed to quantized models such as RedHatAI/phi-4-quantized.w4a16.

📋 Checklist

  • PR title follows Conventional Commits format
  • This PR is linked to an existing issue
  • The test method is described, and the expected result is clearly stated
  • Relevant documentation has been updated (if applicable)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rebel-jaehwang