[Not Mergable Yet] Docker Image for SGLang Integration of Apertus 1.5 - #190
Open
Anunay-Yadav wants to merge 26 commits into
Open
[Not Mergable Yet] Docker Image for SGLang Integration of Apertus 1.5#190Anunay-Yadav wants to merge 26 commits into
Anunay-Yadav wants to merge 26 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new Docker image definition to run the Apertus v1.5 models via an sglang fork, aligning with the PR description’s intended sglang serve usage (FlashInfer backend, CUDA 13 base).
Changes:
- Introduces
images/sglang_apertus_1.5/Dockerfileto build an Ubuntu 24.04 + CUDA 13 + Python 3.12 environment. - Pins and installs
sglang-apertusand a forkedtransformersat specific commits, then installs matching FlashInfer artifacts. - Installs CUDA 13 PyTorch wheels (torch/torchvision/torchaudio) from the PyTorch index for
cu130.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| bzip2 \ | ||
| && rm -rf /var/lib/apt/lists/* | ||
|
|
||
| ADD "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-${TARGET_PLATFORM}.sh" /root/miniforge.sh |
AryanAhadinia
requested changes
Jul 31, 2026
AryanAhadinia
left a comment
AryanAhadinia
left a comment
Member
There was a problem hiding this comment.
As the CI is not passing, I would not be able to test the changes.
AryanAhadinia
changed the title
AryanAhadinia
added
model-support
AryanAhadinia
left a comment
AryanAhadinia
left a comment
Member
There was a problem hiding this comment.
(ignore this review; this is for testing)
AryanAhadinia
self-requested a review
July 31, 2026 23:38
AryanAhadinia
requested changes
Aug 1, 2026
AryanAhadinia
left a comment
AryanAhadinia
left a comment
Member
There was a problem hiding this comment.
🚨 SGL is not doing really well (M9-M12).
(quick-model-tests) ahadinia@clariden-ln003:~/repositories/quick-model-tests> bash run.sh --local --detail \
> --spec dev \
> --model swiss-ai/Apertus-v1.5-8B-vllm-ahadinia \
> --model swiss-ai/Apertus-v1.5-70B-vllm-ahadinia \
> --model swiss-ai/Apertus-v1.5-8B-thinking-vllm-ahadinia \
> --model swiss-ai/Apertus-v1.5-70B-thinking-vllm-ahadinia \
> --model swiss-ai/Apertus-v1.5-8B-vllm-ahadinia-test \
> --model swiss-ai/Apertus-v1.5-70B-vllm-ahadinia-test \
> --model swiss-ai/Apertus-v1.5-8B-thinking-vllm-ahadinia-test \
> --model swiss-ai/Apertus-v1.5-70B-thinking-vllm-ahadinia-test \
> --model swiss-ai/Apertus-v1.5-8B-sgl-ahadinia \
> --model swiss-ai/Apertus-v1.5-70B-sgl-ahadinia \
> --model swiss-ai/Apertus-v1.5-8B-thinking-sgl-ahadinia \
> --model swiss-ai/Apertus-v1.5-70B-thinking-sgl-ahadinia \
> --record-responses ./recordings-full
Setting up test environment...
Running quick-model-tests...
Capability comparison (https://api.swissai.svc.cscs.ch/v1)
| Check | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 |
|--------------------------------------|----|----|----|----|----|----|----|----|----|-----|-----|-----|
| core_health | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| core_system | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| core_maxtokens | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| core_stop | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| core_usage | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| core_template_no_leak | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| core_tokenizer_roundtrip | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| core_tokenizer_unicode | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| core_no_degeneration | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| core_no_degeneration_hard | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| core_multi_system | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| core_assistant_prefill | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| core_determinism | ✔ | ✔ | ✗ | ✔ | ✔ | ✔ | ✗ | ✔ | ✔ | ✔ | ✗ | ✔ |
| mm_image_small | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| mm_image_large | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| mm_image_multi | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| mm_audio_small | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| mm_audio_large | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| mm_interleaved | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| mm_no_degeneration_hard | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| mm_no_degeneration_hard_audio | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| mt_context | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| mt_roles | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| reason_parser_wired | – | – | ✔ | ✔ | – | – | ✔ | ✔ | – | – | ✔ | – |
| reason_produced | – | – | ✔ | ✔ | – | – | ✔ | ✔ | – | ⚠ | ✔ | ⚠ |
| reason_separation | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| reason_clean_channel | – | – | ✔ | ✔ | – | – | ✔ | ✔ | – | ⚠ | ✔ | ⚠ |
| reason_answer | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| reason_stream | – | – | ✔ | ✔ | – | – | ✔ | ✔ | – | ⚠ | ✔ | ⚠ |
| reason_tools | – | – | – | – | – | – | – | – | – | ⚠ | – | ⚠ |
| reason_nothink_no_inner_leak | ✔ | ✔ | – | – | ✔ | ✔ | – | – | ✔ | – | ✔ | – |
| reason_nothink_no_inner_leak_sampled | ✔ | ✔ | – | – | ✔ | ✔ | – | – | ✔ | – | ✔ | – |
| reason_disabled | – | – | ✔ | ✔ | – | – | ✔ | ✔ | – | ⚠ | ✔ | ⚠ |
| robust_specialtokens | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| robust_consecutive_role | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| robust_unicode | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| robust_empty | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✗ | ✔ | ✗ |
| robust_errors | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✗ | ✔ | ✗ |
| bos_single_in_chat | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| bos_single_in_mm_chat | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| bos_single_in_mm_chat_audio | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| bos_single_in_raw_tokenize | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| bos_single_in_completions | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| bos_rendered_prompt_stops | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| bos_single_token | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| bos_consistent_identity | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| bos_generation_matches_tokenize | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| eos | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| eos_not_appended_to_prompt | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | – | – | – | – |
| stream_basic | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✗ | ✔ | ✗ |
| stream_finish | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| stream_stop | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| stream_equiv | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ⚠ | ✔ | ⚠ |
| tools_single | ✔ | ✔ | ⚠ | ⚠ | ✔ | ✔ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_choice_required | ✔ | ✗ | ⚠ | ⚠ | ✔ | ✗ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_choice_named | ✔ | ✔ | ⚠ | ⚠ | ✔ | ✔ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_stream | ✔ | ✔ | ⚠ | ⚠ | ✔ | ✔ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_none | ✔ | ✔ | ⚠ | ⚠ | ✔ | ✔ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_parallel | ✗ | ✗ | ⚠ | ⚠ | ✗ | ✗ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_multiturn | ✔ | ✔ | ⚠ | ⚠ | ✔ | ✔ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_followup | ✔ | ✔ | ⚠ | ⚠ | ✔ | ✔ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_no_content_leak | ✗ | ✗ | ⚠ | ⚠ | ✗ | ✗ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_arg_schema | ✔ | ✔ | ⚠ | ⚠ | ✔ | ✔ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_empty_args | ✔ | ✔ | ⚠ | ⚠ | ✔ | ✔ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
| tools_phantom | ✔ | ✔ | ⚠ | ⚠ | ✔ | ✔ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ | ⚠ |
|======================================|====|====|====|====|====|====|====|====|====|=====|=====|=====|
| passed | 56 | 55 | 48 | 49 | 56 | 55 | 48 | 49 | 34 | 11 | 38 | 11 |
| failed/broken | 3 | 4 | 14 | 13 | 3 | 4 | 14 | 13 | 13 | 39 | 14 | 39 |
| skipped | 6 | 6 | 3 | 3 | 6 | 6 | 3 | 3 | 18 | 15 | 13 | 15 |
Legend: ✔ pass · ✗ fail · ⚠ broken · – skip
M1 = swiss-ai/Apertus-v1.5-8B-vllm-ahadinia
M2 = swiss-ai/Apertus-v1.5-70B-vllm-ahadinia
M3 = swiss-ai/Apertus-v1.5-8B-thinking-vllm-ahadinia
M4 = swiss-ai/Apertus-v1.5-70B-thinking-vllm-ahadinia
M5 = swiss-ai/Apertus-v1.5-8B-vllm-ahadinia-test
M6 = swiss-ai/Apertus-v1.5-70B-vllm-ahadinia-test
M7 = swiss-ai/Apertus-v1.5-8B-thinking-vllm-ahadinia-test
M8 = swiss-ai/Apertus-v1.5-70B-thinking-vllm-ahadinia-test
M9 = swiss-ai/Apertus-v1.5-8B-sgl-ahadinia
M10 = swiss-ai/Apertus-v1.5-70B-sgl-ahadinia
M11 = swiss-ai/Apertus-v1.5-8B-thinking-sgl-ahadinia
M12 = swiss-ai/Apertus-v1.5-70B-thinking-sgl-ahadinia
Contributor
Author
M9 and M11 has the same number of failures as others hence they are doing well. |
Member
Not really, the tool calling tests are failing there. Also, for BoS & EoS, we have to find another way to test. I would not interpret these results as good results. |
Contributor
Author
Sure if you think that i dont agree with this. Anyway, Can you please point me to the codebase of QMT tests i will run them myself and i also want to see if the tests are properly made. |
AryanAhadinia
requested changes
Aug 4, 2026
AryanAhadinia
left a comment
AryanAhadinia
left a comment
Member
There was a problem hiding this comment.
I believe we should invest more time in making FA3 work so that can drop --attention-backend flashinfer from the launch command and fallback to SGL default, unless we have a valid justification for keeping that. A few other minor changes would be required.
| --framework-args "--model-path /capstor/store/cscs/swissai/infra01/hf_models/models/swiss-ai/Apertus-v1.5-70B \ | ||
| --served-model-name swiss-ai/Apertus-v1.5-70B-sglang-$(whoami) \ | ||
| --tp-size 4 \ | ||
| --attention-backend flashinfer \ |
Member
There was a problem hiding this comment.
Although I added this file myself, but this is based the sglang commands you dropped in the description of the PR.
I am a bit concerned about how this flag and bypassing FA3 would affect performance of the model.
- Do you know what will be the performance implications of this flag?
- I know that FA3 is a headache, but I didn't dug into it that much. How severe is the problem? As we are going to release this image as official image of the model, I would prefer to have the optimal performance with SGL defaults.
Nevertheless, I am open with having flashinfer instead if we have a valid justification for it.
Contributor
Author
There was a problem hiding this comment.
i tried to build it from source but it seems to be turning off compilation for arm architecture. i saw this and did not debug further. will look into it more.
| build-essential pkgconf \ | ||
| libibverbs-dev libnuma1 libnuma-dev \ | ||
| python3 python3-dev python3-venv \ | ||
| python3 python3-dev python3-venv shellcheck \ |
Member
There was a problem hiding this comment.
Why do we need spellcheck as a dependency in Docker Image? IMO, it is a dev dependency, not for runtime.
Contributor
Author
There was a problem hiding this comment.
Yeah it is not needed
| @@ -0,0 +1,35 @@ | |||
| image = "/capstor/store/cscs/swissai/infra01/container-images/ci/pr-190/sglang_apertus_1.5-{arch}.sqsh" | |||
Member
There was a problem hiding this comment.
Suggested change
| image = "/capstor/store/cscs/swissai/infra01/container-images/ci/pr-190/sglang_apertus_1.5-{arch}.sqsh" | |
| image = "/capstor/store/cscs/swissai/infra01/container-images/ci/sglang_apertus_1.5-{arch}.sqsh" |
This comment is just a reminder that we should change the directory before merging with main.
| ENV PATH=/opt/conda/bin:${PATH} \ | ||
| SGLANG_BUILD_RUST_EXTS=none | ||
|
|
||
| # NOTE: these pins go stale by design. The Ubuntu archive only serves the |
Member
There was a problem hiding this comment.
Claude is too verbose in comments. Some of them are really good but most of them can be dropped. Would be great if you could do so. We are moving forward publishing SML for community and we want to move forward cleaning up repo.
I would like to emphasize some of the comments are good and must be kept, no need to delete all of them.
AryanAhadinia
requested changes
Aug 5, 2026
AryanAhadinia
left a comment
AryanAhadinia
left a comment
Member
There was a problem hiding this comment.
Added a few other comments. Nothing special, just pre-merge reminders.
Member
There was a problem hiding this comment.
Also, we should add _release suffix to be consistent with the released vLLM image.
Comment on lines
+63
to
+64
| ARG SGLANG_APERTUS_BRANCH=tool-call-fix | ||
| ARG SGLANG_APERTUS_COMMIT=abedd9f0cb55a57528e7f27f3459589f96cf3072 |
Member
There was a problem hiding this comment.
Before merging, we should change the refs to upstream main/latest release.
Comment on lines
+9
to
+12
| ARG SWISS_AI_VLLM_BRANCH=apertus-1-5 | ||
| ARG SWISS_AI_VLLM_COMMIT=b65fae6065decc18108d2324fb47723638d2b2c9 | ||
| ARG TRANSFORMERS_BRANCH=add-apertus1p5 | ||
| ARG TRANSFORMERS_COMMIT=aa9e3c1a3a80ccbf12577885c9b30f494687d5e4 |
Member
There was a problem hiding this comment.
Before merging, we should change the refs to upstream main/latest release.
AryanAhadinia
force-pushed
the
sglang_apertus
branch
from
August 6, 2026 21:19
8e3b3fd to
aa5e075
Compare
Member
|
forced push because of https://swissai-initiative.slack.com/archives/C07DHJFRE2X/p1786016678962569 |
AryanAhadinia
force-pushed
the
sglang_apertus
branch
2 times, most recently
from
August 12, 2026 23:33
72272ee to
2d0a76b
Compare
Member
|
DON'T MERGE THIS PR YET. Merging will overwrite published images and we should do so once we are integrated with upstream repos. |
AryanAhadinia
changed the title
…niforge installation with architecture detection
AryanAhadinia
force-pushed
the
sglang_apertus
branch
from
August 13, 2026 09:38
e3fa089 to
b3e1c04
Compare
formers Refs
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attention
DON'T MERGE THIS PR YET. Merging will overwrite published images and we should do so once we are integrated with upstream repos. Contact @AryanAhadinia for more details.
Attention backend flashinfer is needed i could not make FA3 work. FA3 installation of sglang-kernel disables by default for aarch/arm architecture. i tried to build it from source by forcefully enabling it but faced issues.