Skip to content

Q40 fp32 int4 experiments + save model #3524

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 11 commits into
base: main
Choose a base branch
from
Draft

Q40 fp32 int4 experiments + save model #3524

wants to merge 11 commits into from

Conversation

@mwlasiuk
Copy link
Contributor

@mwlasiuk mwlasiuk commented Oct 14, 2025 •
edited
Loading

  1. Run application with Q4_0 weights model
  2. During run model_basic.bin file will be created in CWD
  3. model_basic.bin can be used as input file after selecting QINT4 as weight type

NOTE: this is not proper model exporter / converted - this is only hack

gkisalapl and others added 10 commits October 6, 2025 12:45
@gkisalapl
INT4 quantization fixes
b1ecb4f 
Fix problem of group size incosistency

Signed-off-by: Grzegorz Kisala/Neural Computing (AIS) /SRPOL/Senior Professional/Samsung Electronics <[email protected]>
@gkisalapl
Move int4 logic to separate file and fix scale order
04ee578 
Signed-off-by: Grzegorz Kisala/Neural Computing (AIS) /SRPOL/Senior Professional/Samsung Electronics <[email protected]>
@gkisalapl
Fix linux build
9e5b7c1 
Signed-off-by: Grzegorz Kisala/Neural Computing (AIS) /SRPOL/Senior Professional/Samsung Electronics <[email protected]>
@gkisalapl
Add INT4 dequantization
634fd5a 
Signed-off-by: Grzegorz Kisala/Neural Computing (AIS) /SRPOL/Senior Professional/Samsung Electronics <[email protected]>
@gkisalapl
Add Q4_0 dequantization
ca3a51f 
Signed-off-by: Grzegorz Kisala/Neural Computing (AIS) /SRPOL/Senior Professional/Samsung Electronics <[email protected]>
@gkisalapl
Calculate mse for gemm
c3811ac 
Signed-off-by: Grzegorz Kisala/Neural Computing (AIS) /SRPOL/Senior Professional/Samsung Electronics <[email protected]>
@mnalewaj
[gemm cl] Fix for reading scales in int4 gemm cl kernel
df56f41 
**Self evaluation:**
1. Build test:     [X]Passed [ ]Failed [ ]Skipped
2. Run test:     [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Maciej Nalewaj <[email protected]>
@gkisalapl
Q4_0 -> Fp32 -> INT4 experiments
061d8a0 
Signed-off-by: Grzegorz Kisala/Neural Computing (AIS) /SRPOL/Senior Professional/Samsung Electronics <[email protected]>
@gkisalapl
Replace int4 gemvs wirh gemm
18f2c55 
Signed-off-by: Grzegorz Kisala/Neural Computing (AIS) /SRPOL/Senior Professional/Samsung Electronics <[email protected]>
Save model repacking inflight
e21a59a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@myungjoo myungjoo Awaiting requested review from myungjoo myungjoo will be requested when the pull request is marked ready for review myungjoo is a code owner

@jijoongmoon jijoongmoon Awaiting requested review from jijoongmoon jijoongmoon will be requested when the pull request is marked ready for review jijoongmoon is a code owner

@again4you again4you Awaiting requested review from again4you again4you will be requested when the pull request is marked ready for review again4you is a code owner

@jaeyun-jung jaeyun-jung Awaiting requested review from jaeyun-jung jaeyun-jung will be requested when the pull request is marked ready for review jaeyun-jung is a code owner

@leemgs leemgs Awaiting requested review from leemgs leemgs will be requested when the pull request is marked ready for review leemgs is a code owner

@wooksong wooksong Awaiting requested review from wooksong wooksong will be requested when the pull request is marked ready for review wooksong is a code owner

@gichan-jang gichan-jang Awaiting requested review from gichan-jang gichan-jang will be requested when the pull request is marked ready for review gichan-jang is a code owner

@anyj0527 anyj0527 Awaiting requested review from anyj0527 anyj0527 will be requested when the pull request is marked ready for review anyj0527 is a code owner

@lhs8928 lhs8928 Awaiting requested review from lhs8928 lhs8928 will be requested when the pull request is marked ready for review lhs8928 is a code owner

@songgot songgot Awaiting requested review from songgot songgot will be requested when the pull request is marked ready for review songgot is a code owner

@jihochu jihochu Awaiting requested review from jihochu jihochu will be requested when the pull request is marked ready for review jihochu is a code owner

@DonghakPark DonghakPark Awaiting requested review from DonghakPark DonghakPark will be requested when the pull request is marked ready for review DonghakPark is a code owner

@SeoHyungjun SeoHyungjun Awaiting requested review from SeoHyungjun SeoHyungjun will be requested when the pull request is marked ready for review SeoHyungjun is a code owner

@baek2sm baek2sm Awaiting requested review from baek2sm baek2sm will be requested when the pull request is marked ready for review baek2sm is a code owner

@skykongkong8 skykongkong8 Awaiting requested review from skykongkong8 skykongkong8 will be requested when the pull request is marked ready for review skykongkong8 is a code owner

@djeong20 djeong20 Awaiting requested review from djeong20 djeong20 will be requested when the pull request is marked ready for review djeong20 is a code owner

@EunjuYang EunjuYang Awaiting requested review from EunjuYang EunjuYang will be requested when the pull request is marked ready for review EunjuYang is a code owner

@dkjung dkjung Awaiting requested review from dkjung dkjung will be requested when the pull request is marked ready for review dkjung is a code owner

@haehun haehun Awaiting requested review from haehun haehun will be requested when the pull request is marked ready for review haehun is a code owner

At least 3 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

Need Review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mwlasiuk @gkisalapl @mnalewaj