Add Prefill–Decode Separation with Batched Prompt Ingestion and Logits Skipping by orionpapadakis · Pull Request #102 · beehive-lab/GPULlama3.java

[refactor] Move QuantizedLayerPlanner to layerplanner package root-level

orionpapadakis

committed

[prf/dec] Add CLI options for batched prefill and prefill batch size configuration

orionpapadakis

committed

[prf/dec] Add CPU path for prefill/decode. Separates inference path with InferenceCoreWithPrefillDecode and InferenceEngineWithPrefillDecode

orionpapadakis

committed

[prf/dec] Add GPU path for prefill/decode with TornadoVM integration. Implements `InferenceEngineWithPrefillDecode` and `TornadoVMMasterPlanWithPrefillDecode` for batched token generation. Refactor…

orionpapadakis

committed

[prf/dec] Batch prompt tokens during prefill phase in CPU path

orionpapadakis

committed

[prf/dec][wip] Add GPU-based prefill-decode with batched prefill (working state, with cuda graphs only)

orionpapadakis

committed

[prf/dec][refactor] Restructure prefill-decode ExecutionPlan components to dedicated classes and packages

orionpapadakis

committed

[prf/dec][dbg] Guard CUDA Graphs enable/disable behind `--no-cuda-graphs` option to ease debugging

orionpapadakis

committed

[prf/dec][refactor] Rename `LlamaFP16FFNLayersForUnifiedDecode` to `LlamaFP16FFNLayersDecode`

orionpapadakis

committed

[prf/dec][refactor] Rename `LlamaFP16BatchPrefillLayers` to `LlamaFP16LayersBatchPrefill`

orionpapadakis

committed

[prf/dec][refactor] Rename `TornadoVMMasterPlanBatchPrefill` to `TornadoVMMasterPlanWithBatchPrefillDecode`

orionpapadakis

committed

[prf/dec] Fix KV-cache propagation bug from prefill to decode path and refactor task graph consumption logic

orionpapadakis

committed

[prf/dec] Provide distinct support for `standard`, `prefill-decode` and `batched-prefill-decode` execution paths for both CPU and GPU

orionpapadakis

committed

[prf/dec] Refactor TornadoVM execution plans to unify GPU paths for standard, prefill-decode, and batched-prefill-decode setups.

orionpapadakis

committed

[prf/dec][cleanup] Remove unused debug logs and commented-out code from TornadoVM execution paths.

orionpapadakis

committed

[prf/dec][refactor] Standardize task graph and grid scheduler naming for prefill and decode paths in TornadoVM

orionpapadakis

committed

[prf/dec][doc] Update javadoc to reflect unified batched prefill-decode plan

orionpapadakis

committed

[prf/dec][fix] Restructure and fix issues in `TornadoVMMasterPlanWithPrefillDecode`

orionpapadakis

committed

[prf/dec] Separate inference paths (`InferenceEngine`, `InferenceCore`, CPU/GPU) for standard, prefill-decode and prefill-decode with batching

orionpapadakis

committed

[prf/dec] Add unsupported operation exceptions for CPU/GPU prefill-decode and batched-prefill-decode paths in `Mistral`, `Phi3`, `Qwen2`, and `Qwen3` models.

orionpapadakis

committed

[prf/dec][refactor] Add unsupported exceptions for Q8_0 weights in GPU prefill-decode and batched-prefill-decode paths

orionpapadakis

committed

[prf/dec][ci] Extend CI workflow to include GPU `prefill-decode` and `batched-prefill-decode` test cases for Llama 3.2 1B FP16

orionpapadakis

committed

[ci][prf/dec] Enforce `--ptx` usage in GPU prefill-decode tests

orionpapadakis

committed

[ci][prf/dec] Add conditional checks for PTX backend in GPU prefill-decode tests

orionpapadakis

committed

[ci] Limit CI workflow to serial execution by setting max-parallel to 1

orionpapadakis

committed

[cuda-graphs] Add support for `--cuda-graphs` argument and update default behavior for CUDA Graphs

orionpapadakis

committed

[ci][prf/dec] Simplify CI workflow by unifying backend-specific prefill-decode and batch-prefill-decode steps

orionpapadakis

committed

[prf/dec] Fix predecessor graph naming and device data consumption for non-CUDA Graphs

orionpapadakis

committed

[prf/dec] Update GPU execution plans to clarify prefill/decode structure and KV cache handling

orionpapadakis

committed

[ci] Revert max-parallel limit from CI workflow matrix

orionpapadakis

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Prefill–Decode Separation with Batched Prompt Ingestion and Logits Skipping #102

Add Prefill–Decode Separation with Batched Prompt Ingestion and Logits Skipping #102
mikepapadim merged 30 commits intobeehive-lab:mainfrom
orionpapadakis:feat/prefill-decode

Commits on May 4, 2026

[refactor] Move QuantizedLayerPlanner to layerplanner package root-level

[prf/dec] Add CLI options for batched prefill and prefill batch size configuration

[prf/dec] Add CPU path for prefill/decode. Separates inference path with InferenceCoreWithPrefillDecode and InferenceEngineWithPrefillDecode

[prf/dec] Add GPU path for prefill/decode with TornadoVM integration. Implements `InferenceEngineWithPrefillDecode` and `TornadoVMMasterPlanWithPrefillDecode` for batched token generation. Refactor…

[prf/dec] Batch prompt tokens during prefill phase in CPU path

[prf/dec][wip] Add GPU-based prefill-decode with batched prefill (working state, with cuda graphs only)

[prf/dec][refactor] Restructure prefill-decode ExecutionPlan components to dedicated classes and packages

[prf/dec][dbg] Guard CUDA Graphs enable/disable behind `--no-cuda-graphs` option to ease debugging

[prf/dec][refactor] Rename `LlamaFP16FFNLayersForUnifiedDecode` to `LlamaFP16FFNLayersDecode`

[prf/dec][refactor] Rename `LlamaFP16BatchPrefillLayers` to `LlamaFP16LayersBatchPrefill`

[prf/dec][refactor] Rename `TornadoVMMasterPlanBatchPrefill` to `TornadoVMMasterPlanWithBatchPrefillDecode`

[prf/dec] Fix KV-cache propagation bug from prefill to decode path and refactor task graph consumption logic

[prf/dec] Provide distinct support for `standard`, `prefill-decode` and `batched-prefill-decode` execution paths for both CPU and GPU

[prf/dec] Refactor TornadoVM execution plans to unify GPU paths for standard, prefill-decode, and batched-prefill-decode setups.

[prf/dec][cleanup] Remove unused debug logs and commented-out code from TornadoVM execution paths.

[prf/dec][refactor] Standardize task graph and grid scheduler naming for prefill and decode paths in TornadoVM

[prf/dec][doc] Update javadoc to reflect unified batched prefill-decode plan

[prf/dec][fix] Restructure and fix issues in `TornadoVMMasterPlanWithPrefillDecode`

[prf/dec] Separate inference paths (`InferenceEngine`, `InferenceCore`, CPU/GPU) for standard, prefill-decode and prefill-decode with batching

[prf/dec] Add unsupported operation exceptions for CPU/GPU prefill-decode and batched-prefill-decode paths in `Mistral`, `Phi3`, `Qwen2`, and `Qwen3` models.

[prf/dec][refactor] Add unsupported exceptions for Q8_0 weights in GPU prefill-decode and batched-prefill-decode paths

[prf/dec][ci] Extend CI workflow to include GPU `prefill-decode` and `batched-prefill-decode` test cases for Llama 3.2 1B FP16

[ci][prf/dec] Enforce `--ptx` usage in GPU prefill-decode tests

[ci][prf/dec] Add conditional checks for PTX backend in GPU prefill-decode tests

[ci] Limit CI workflow to serial execution by setting max-parallel to 1

[cuda-graphs] Add support for `--cuda-graphs` argument and update default behavior for CUDA Graphs

[ci][prf/dec] Simplify CI workflow by unifying backend-specific prefill-decode and batch-prefill-decode steps

[prf/dec] Fix predecessor graph naming and device data consumption for non-CUDA Graphs

[prf/dec] Update GPU execution plans to clarify prefill/decode structure and KV cache handling

[ci] Revert max-parallel limit from CI workflow matrix