docs: refine speculative decoding proposal design

Signed-off-by: Juntao Zhang <juntaozhang22@m.fudan.edu.cn>

Author: zjteee

docs/proposals/algorithms/joint-inference/cloud-edge-benchmark-for-llm-speculative-decoding.md

@@ -8,12 +8,10 @@
     - [Overall Architecture](#overall-architecture)
     - [Changes in Sedna](#changes-in-sedna)
     - [Changes in Ianvs](#changes-in-ianvs)
-    - [Integration Options](#integration-options)
-    - [High-Level Paradigm Concept](#high-level-paradigm-concept)
+    - [Speculative-Decoding Modules](#speculative-decoding-modules)
     - [Benchmark Construction](#benchmark-construction)
     - [Algorithm Exploration](#algorithm-exploration)
   - [Roadmap](#roadmap)
-  - [References](#references)
 
 # Cloud-Edge Speculative Decoding Benchmark for LLM based on KubeEdge-Ianvs
 
@@ -34,7 +32,7 @@ However, the effectiveness of speculative decoding in cloud-edge scenarios canno
 
 We propose that KubeEdge-Ianvs adopt a cloud-edge speculative decoding benchmark implemented on top of Sedna `JointInference`, with the goal of evaluating and improving LLM inference efficiency in cloud-edge environments.
 
-This proposal adopts Ianvs `jointinference` as the primary integration paradigm and extends it with speculative-decoding support. Under this design, the edge model is responsible for drafting candidate tokens, while the cloud model performs token-level verification and correction.
+This proposal adopts Ianvs `jointinference` as the primary integration paradigm and extends it with speculative-decoding support. Under this design, the example provides dedicated `draft` and `verify` modules, while the paradigm layer implements the speculative-decoding workflow and organizes how these modules collaborate for one sample.
 
 At the current stage, the proposal focuses on building and validating the speculative-decoding benchmark within the existing in-process execution model. On top of this stable foundation, the subsequent step is to explore more effective cloud-edge speculative-decoding strategies, such as improved draft-window design, acceptance-aware collaboration, and network-sensitive policy optimization.
 
@@ -50,19 +48,23 @@ At the current stage, the proposal focuses on building and validating the specul
 This proposal differs from existing cloud-edge collaborative inference examples in Ianvs in the following respects:
 
 - **Paradigm-level workflow extension**: the proposal implements the speculative-decoding workflow in the `JointInference` paradigm layer, so that the framework can organize repeated collaboration rounds for one sample.
-- **Stable model-side interface**: the proposal does not introduce new required model APIs for draft and verify operations. Both sides continue to use `inference()` as the common model-side entry.
-- **Example-level algorithm freedom**: the framework controls the workflow, while the example retains ownership of how request payloads, metadata, draft tokens, and verification logic are actually implemented.
+- **Dedicated speculative-decoding modules**: the proposal introduces dedicated `draft` and `verify` modules, instead of overloading the existing `cloud` and `edge` modules with speculative-decoding-specific semantics.
+- **Example-level algorithm freedom**: the framework controls the workflow, while the example retains ownership of how the `draft` and `verify` modules actually invoke models and exchange metadata.
 
 ### Overall Architecture
 
-This proposal adopts Ianvs `jointinference` as the integration entry because speculative decoding in cloud-edge scenarios remains a joint-inference problem: one sample enters the controller, the paradigm layer organizes collaboration, and the example layer implements the concrete model behavior.
+This proposal adopts Ianvs `jointinference` as the integration entry because speculative decoding in cloud-edge scenarios remains a joint-inference problem: one sample enters the controller, the paradigm layer organizes collaboration, and the example layer implements the concrete module behavior.
 
 The design has two clear boundaries:
 
 - **Change of Sedna**: implement `JointInference` as a workflow controller that can support repeated collaboration rounds for one sample.
 - **Change of Ianvs**: keep Ianvs `joint_inference` as the controller-side wrapper that loads data, passes modules and configuration to Sedna, and collects benchmark outputs.
 
-Under this design, the paradigm layer owns the workflow, while the example layer owns the actual inference behavior of edge and cloud models.
+Under this design, the paradigm layer owns the workflow, while the example layer owns the actual behavior of the `draft` and `verify` modules.
+
+The current architecture is illustrated below:
+
+<img src="./images/SD.png" alt="New Architecture" width="80%">
 
 ### Changes in Sedna
 
@@ -76,20 +78,21 @@ The main Sedna change is to implement the **speculative-decoding workflow** in t
 
 This design gives the paradigm genuine multi-round inference capability.
 
-At the same time, the proposal keeps the model-side interface stable. It does **not** introduce new required framework APIs such as dedicated `draft()` or `verify()` methods. Both sides continue to use `inference()` as the common entry. The distinction between drafting and verification is carried through example-defined request payloads, metadata, and `kwargs`, while the exact semantics remain under example control.
+At the same time, the proposal keeps the framework boundary clear. The paradigm layer is responsible for coordinating repeated rounds through the speculative-decoding workflow, while the concrete model invocation remains inside the example-defined `draft` and `verify` modules. In this design, Sedna does not need to reinterpret the original `cloud` and `edge` modules for speculative decoding. Instead, the new modules make the speculative-decoding semantics explicit at the example layer.
 
 This design has two advantages:
 
-- it avoids creating compatibility pressure on existing examples;
-- it keeps the framework generic instead of binding it to one fixed speculative-decoding API shape.
+- it keeps speculative-decoding-specific behavior separate from the existing cloud-edge module semantics;
+- it makes the implementation easier for users to understand, because drafting and verification are represented by dedicated modules, while workflow control is clearly placed in the paradigm layer.
 
 ### Changes in Ianvs
 
 On the Ianvs side, `joint_inference` continues to play the role of the controller-side wrapper. Ianvs loads the dataset, applies the optional dataset processor, builds the Sedna `JointInference` job, iterates over samples, and collects benchmark results. In this proposal, the Ianvs core flow can be reused as it is and does not require dedicated core-code modification for speculative decoding.
 
 Accordingly, the Ianvs-side work is mainly concentrated in the example layer:
 
-- implement the edge and cloud model behavior required by the speculative-decoding benchmark;
+- implement the `draft` module that performs speculative draft generation;
+- implement the `verify` module that performs cloud-side verification and correction;
 - provide dataset processing that maps benchmark samples into the request format expected by the example;
 - provide benchmark metrics and result parsing for latency, throughput, acceptance rate, and related outputs.
 
@@ -103,9 +106,9 @@ examples/cloud-edge-speculative-decoding-benchmark
 ├── README.md
 ├── testalgorithms
 │   └── speculative-decoding
-│       ├── cloud_model.py
 │       ├── data_processor.py
-│       ├── edge_model.py
+│       ├── draft.py
+│       ├── verify.py
 │       └── test_speculative_decoding.yaml
 └── testenv
     ├── acceptance_rate.py
@@ -117,80 +120,24 @@ examples/cloud-edge-speculative-decoding-benchmark
     └── time_to_first_token.py
 ```
 
-### Integration Options
-
-Based on the above design principles, this proposal keeps two integration options.
-
-#### Option 1: Explicit SD Mode in `JointInference`
-
-The first option is to keep a dedicated SD-oriented mode in the paradigm layer. In this option, Sedna `JointInference` explicitly recognizes speculative decoding and enters the paradigm-level workflow. The workflow belongs to the paradigm, but each round still invokes the example-defined edge model and cloud model through the common `inference()` entry.
-
-<p align="center">
-  <img src="./images/SD-1.png" alt="Plan A" width="80%">
-</p>
-
-This option is attractive for the following reasons:
-
-- the workflow is explicit at the paradigm layer;
-- the benchmark configuration can expose speculative decoding directly;
-- the framework loop is visible without forcing new model-side interfaces;
-- the example still retains full control over how drafting and verification are encoded;
-- the SD path is kept separate from the original HEM-oriented flow, which makes the semantics of this mode more direct and easier to reason about.
-
-For this option, the module boundary can be summarized as follows:
-
-- **Original modules**: Ianvs `TestEnvManager`, `TestCaseController`, and `StoryManager`; the existing Sedna `JointInference` control skeleton; the existing Ianvs controller-side module loading and benchmark execution flow.
-- **New or extended modules**: an explicit SD-oriented mode in the paradigm layer; paradigm-level speculative-decoding workflow control in Sedna `JointInference`; example-defined edge and cloud modules that interpret `inference()` requests for drafting and verification.
-- **Impact on other examples**: other examples are affected only if they explicitly select the SD-oriented mode. Examples that continue to use the original modes and their current `inference()` behavior remain compatible. The shared-code impact is concentrated in Sedna `JointInference` mode dispatch and result handling.
-
-#### Option 2: Reusing `mining-then-inference`
-
-The second option is to reuse the existing `mining-then-inference` branch and let that branch enter the speculative-decoding workflow when routing requires cloud-side collaboration. Under this design, the paradigm owns the workflow, and the SD loop is entered from the route-first control path already provided by the current code structure.
-
-<p align="center">
-  <img src="./images/SD-2.png" alt="Plan B" width="80%">
-</p>
-
-This option is attractive for the following reasons:
-
-- it preserves more of the current `JointInference` control structure;
-- the original route-first semantics remain meaningful;
-- speculative decoding is activated only when the routing result requires it;
-- the model-side interface still remains `inference()` only;
-- it follows the flow shape already exposed by the current code, which can reduce the amount of structural change required in the paradigm layer.
-
-For this option, the module boundary can be summarized as follows:
-
-- **Original modules**: Ianvs controller-side execution flow; the existing Sedna `mining-then-inference` branch; the existing HEM-based route-first structure; existing edge and cloud module registration in Ianvs.
-- **New or extended modules**: a speculative-decoding branch within the reused route-first path; paradigm-level multi-round workflow control once the branch enters speculative decoding; example-defined edge and cloud `inference()` behavior for draft and verify semantics.
-- **Impact on other examples**: this option touches the reused `mining-then-inference` branch and therefore requires more attention to branch-level compatibility. Its configuration constraints are also stronger, because speculative decoding must be enabled together with `mining-then-inference`. Examples that rely on the original route-first semantics should still keep their behavior as long as the speculative-decoding branch is not enabled.
-
-
-### High-Level Paradigm Concept
+### Speculative-Decoding Modules
 
-Beyond the two retained options above, there is a more general joint-inference abstraction worth recording. The idea is to extract a router-driven multi-round paradigm:
+Based on the current design, the proposal introduces two dedicated example-side modules for speculative decoding, while the workflow itself is implemented in the paradigm layer.
 
-- one sample enters the controller;
-- a routing module decides whether to execute on edge, cloud, or exit;
-- the selected side performs inference;
-- the router examines the returned output and decides whether another round is needed or whether execution should terminate.
+#### draft
 
-In that more general design, routing becomes the generic abstraction, and HEM can be moved into the example layer as one possible router implementation. This would make the paradigm less tied to any specific example and more reusable across different collaboration strategies. Under this abstraction, many existing examples can be described in a unified way: a HEM-based example becomes a custom router that performs one-shot binary routing and then exits, while a speculative-decoding example becomes a custom router that alternates between edge-side drafting and cloud-side verification across multiple rounds. This is why the high-level direction is attractive: it can, in principle, subsume the current examples under one common joint-inference framework.
+The `draft` module is responsible for speculative draft generation. It encapsulates how the drafting model is loaded, how inputs are interpreted, and how draft tokens and related metadata are produced for the workflow. This keeps drafting logic separate from the semantics of the original `edge` module.
 
-This high-level direction is illustrated below:
+#### verify
 
-<p align="center">
-  <img src="./images/SD-3.png" alt="High-Level Design" width="80%">
-</p>
+The `verify` module is responsible for verification and correction. It encapsulates how the verification model is invoked, how draft tokens are checked, and how accepted or corrected outputs are returned to the workflow. This keeps verification logic separate from the semantics of the original `cloud` module.
 
-However, this direction would require a broader redesign of the current joint-inference abstraction. For that reason, it is recorded here as a longer-term paradigm direction rather than the immediate implementation target.
+From a module-boundary perspective, the current design can be summarized as follows:
 
-From an implementation perspective, the current proposal keeps the scope clear:
+- **Original modules**: Ianvs `TestEnvManager`, `TestCaseController`, and `StoryManager`; the existing Ianvs controller-side benchmark flow; the existing Sedna `JointInference` skeleton.
+- **New or extended modules**: paradigm-level speculative-decoding workflow support in Sedna `JointInference`; example-defined `draft` and `verify` modules.
+- **Impact on other examples**: the original `cloud` and `edge` modules do not need to be reinterpreted for speculative decoding. Existing examples can continue to use their original modules and semantics. The speculative-decoding benchmark is isolated through the new modules, which reduces confusion for users and limits compatibility impact on unrelated examples.
 
-- Sedna changes focus on implementing multi-round workflow capability in `JointInference`;
-- Ianvs changes focus on passing configuration and collecting benchmark results;
-- model-side algorithm details remain in the example layer;
-- compatibility is protected by continuing to use `inference()` as the common model-side entry.
 
 ### Benchmark Construction