Skip to content

Commit 54c8225

Browse files
mise42mise42
authored
Introduce PVQ (#126)
Looking for feedback. ## Summary This proposal introduces PVQ (PolkaVM Query), which aims to serve as an intermediary layer between different chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries. ## Related Discussions https://forum.polkadot.network/t/wasm-view-functions/1045 ## PoC implementations https://github.com/open-web3-stack/XCQ
1 parent 7f673a4 commit 54c8225

File tree

1 file changed

+363
-0
lines changed

1 file changed

+363
-0
lines changed

‎text/0126-introduce-pvq.md‎

Lines changed: 363 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,363 @@
1+
# RFC-0126: Introduce PVQ (PolkaVM Query)
2+
3+
| | |
4+
| --------------- | ------------------------------------------------------------------------------------------- |
5+
| **Start Date** | Oct 25, 2024 |
6+
| **Description** | Introduce PVQ (PolkaVM Query) |
7+
| **Authors** | Bryan Chen, Jiyuan Zheng |
8+
9+
## Summary
10+
11+
This proposal introduces PVQ (PolkaVM Query), a unified query interface that bridges different chain runtime implementations and client tools/UIs. PVQ provides an extension-based system where runtime developers can expose chain-specific functionality through standardized interfaces, while allowing client-side developers to perform custom computations on the data through PolkaVM programs. By abstracting away concrete implementations across chains and supporting both off-chain and cross-chain scenarios, PVQ aims to reduce code duplication and development complexity while maintaining flexibility for custom use cases.
12+
13+
## Motivation
14+
15+
In Substrate, runtime APIs facilitate off-chain clients in reading the state of the consensus system.
16+
However, the APIs defined and implemented by individual chains often fall short of meeting the diverse requirements of client-side developers.
17+
For example, client-side developers may want some aggregated data from multiple pallets, or apply various custom transformations on the raw data.
18+
Additionally, chains often implement different APIs and data types (such as `AccountId`) for similar functionality, which increases complexity and development effort on the client side.
19+
As a result, client-side developers frequently resort to directly accessing storage (which is susceptible to breaking changes) and reimplementing custom computations on the raw data. This leads to code duplication between the Rust runtime logic and UI JavaScript/TypeScript logic, increasing development effort and introducing potential for bugs.
20+
21+
Moreover, the diversity also extends to cross-chain queries.
22+
23+
Therefore, a system that serves as an intermediary layer between runtime implementations and client-side implementations with a unified but flexible interface will be beneficial for both sides. It should be able to:
24+
25+
- Allow runtime developers to provide query APIs which may include data across multiple pallets but aggregate these APIs through a unified interface
26+
- Allow client-side developers to query data from this unified interface across different chains while maintaining the flexibility to perform custom transformations on the raw data
27+
- Support cross-chain queries through XCM integration
28+
29+
Use cases that will benefit from such a system:
30+
31+
- XCM bridge UI:
32+
- Query asset balances
33+
- Query XCM weight and fee from hop and dest chains
34+
- Wallets:
35+
- Query asset balances
36+
- Query weights and fees for operations across chains
37+
- Universal dApp that supports all the parachains:
38+
- Perform Feature discovery
39+
- Query pallet-specific features
40+
- Construct extrinsics by querying pallet index, call index, etc
41+
42+
## Stakeholders
43+
44+
- Runtime Developers
45+
- Tools/UI Developers
46+
47+
## Explanation
48+
49+
The core idea of PVQ is to have a unified interface that meets the aforementioned requirements.
50+
51+
On the runtime side, an extension-based system is introduced to serve as a standardization layer across different chains.
52+
Each extension specification defines a set of cohesive APIs.
53+
Runtime developers can freely select which extensions they want to implement, and have full control over how the data is sourced - whether from single or multiple pallet functions, with optional data transformations applied.
54+
The runtime aggregates all implemented extensions into a single unified interface that takes a query program and corresponding arguments, and returns the query result.
55+
This interface will be exposed in two ways: as a Substrate RuntimeAPI for off-chain queries, and as an XCM instruction for cross-chain queries.
56+
57+
On the client side or in XCM use cases, the client can easily detect which extensions are supported by a runtime through the metadata API. This allows for runtime feature discovery and enables clients to adapt their behavior based on available functionality.
58+
Client-side developers can encode their desired custom computation logic into the query program and its arguments, while the actual data access happens through runtime-implemented extensions.
59+
60+
In conclusion, PVQ involves three components:
61+
62+
- PVQ Extension System: Standardize the functionality across different chains.
63+
- PVQ Executor: Aggregates the extensions and perform the query from off-chain or cross-chain.
64+
- RuntimeAPI/XCM Integration: Support off-chain and cross-chain scenarios.
65+
66+
### PVQ Extension System
67+
68+
The PVQ extension system has the following features:
69+
70+
- Defines an extension as a Rust trait with optional associated types.
71+
72+
**Example Design**:
73+
74+
The following code declares two extensions: `extension_core` and `extension_fungibles` with some associated types.
75+
76+
```rust
77+
#[extension_decl]
78+
pub mod extension_core {
79+
#[extension_decl::extension]
80+
pub trait ExtensionCore {
81+
type ExtensionId;
82+
fn has_extension(id: Self::ExtensionId) -> bool;
83+
}
84+
}
85+
86+
#[extension_decl]
87+
pub mod extension_fungibles {
88+
#[extension_decl::extension]
89+
pub trait ExtensionFungibles {
90+
type AssetId;
91+
type Balance;
92+
type AccountId;
93+
fn total_supply(asset: Self::AssetId) -> Self::Balance;
94+
fn balance(asset: Self::AssetId, who: Self::AccountId) -> Self::Balance;
95+
}
96+
}
97+
```
98+
99+
The following code implements the extensions, amalgamates them and generates the corresponding metadata.
100+
101+
```rust
102+
#[extensions_impl]
103+
pub mod extensions {
104+
#[extensions_impl::impl_struct]
105+
pub struct ExtensionsImpl;
106+
107+
#[extensions_impl::extension]
108+
impl pvq_extension_core::extension::ExtensionCore for ExtensionsImpl {
109+
type ExtensionId = u64;
110+
fn has_extension(id: Self::ExtensionId) -> bool {
111+
matches!(id, 0 | 1)
112+
}
113+
}
114+
115+
#[extensions_impl::extension]
116+
impl pvq_extension_fungibles::extension::ExtensionFungibles for ExtensionsImpl {
117+
type AssetId = u32;
118+
type AccountId = [u8; 32];
119+
type Balance = u64;
120+
fn total_supply(_asset: Self::AssetId) -> Self::Balance {
121+
100
122+
}
123+
fn balance(_asset: Self::AssetId, _who: Self::AccountId) -> Self::Balance {
124+
100
125+
}
126+
}
127+
}
128+
```
129+
130+
- Hash-based extension id generation mechanism
131+
132+
Extensions are uniquely identified by a hash value computed from their name and method names. This means that modifying either the extension name or its method names results in a new extension. This design allows new functionality to be added independently of the PVQ core version, enabling a permissionless extension system while keeping the core implementation minimal.
133+
134+
The extension ID generation can be expressed mathematically as:
135+
136+
$ExtID = twox64(P \parallel E \parallel M_1 \parallel M_2 \parallel ... \parallel M_n)$
137+
138+
Where:
139+
140+
- $P$ is the prefix string constant, `pvq-ext`
141+
- $E$ is the extension name
142+
- $M_1...M_n$ are the method names in lexicographical order
143+
- $\parallel$ represents string concatenation with a separator `@` to avoid collision
144+
- $twox64()$ is the 64-bit xxHash function
145+
146+
- A permission control system allows filtering extension method invocations based on their origin (Runtime, Extrinsics, RuntimeAPI, or XCM). This enables runtime developers to restrict certain functions from being called through specific interfaces, such as preventing access via XCM when desired.
147+
148+
### PVQ Executor
149+
150+
The PVQ Executor provides a unified interface that only takes query programs with corresponding arguments and returns results. It can be formulated as a PVM program-argument invocation, as detailed in [Appendix A.8 in the JAM Gray Paper](https://graypaper.com/). Specifically, we call it PVQ invocation.
151+
152+
#### Program initialization and Results Return
153+
154+
- PVQ program size limit:
155+
While the standard PVM code format contains instructions, jump table, and initial RAM state information, PVQ programs can be significantly trimmed down. This is because PVQ separates computation logic from state access: the computation happens in the program while state access is handled through host functions. This makes the program stateless, allowing us to eliminate the initial read-write (heap) data and stack sections along with their length encodings in the PVQ program binary. The read-only data section can be minimized to only contain essential utility data like host function extension IDs, keeping it within a reasonable size limit.
156+
157+
- Entrypoint:
158+
PVQ programs have a single static entrypoint that begins at instruction 0, since all PVQ computation can be expressed through a single entry point.
159+
160+
- Argument passing:
161+
Query data is encoded as invocation arguments. Specifically, it includes the view function index and its arguments. As discussed in [Equation A.36 in the Gray Paper](https://graypaper.com/), arguments start at `0xfeff0000` which is stored in `a0`(7th register), and the length is specified at `a1`(8th register).
162+
163+
- Return results
164+
As discussed in [Equation A.39 in the Gray Paper](https://graypaper.com/), the invocation returns its output through register `a0` (7th register), which contains a pointer to the output buffer. Register `a1`(8th register) contains the length of the output buffer. The output buffer must be allocated within the program's memory space and contain the SCALE-encoded return value.
165+
166+
#### Host Functions
167+
168+
The following host functions are available to PVQ invocations. The index numbers shown below correspond to the values used in the `ecalli` instruction.
169+
170+
1. `extension_call`: A unified entry point that routes queries to various extensions. It accepts two parameters:
171+
172+
- `extension_id`
173+
A `u64` value for selecting which extension to query, split across two 32-bit registers: lower 32 bits in `a0` and upper 32 bits in `a1`
174+
- `query_data`
175+
SCALE-encoded value including the view function index and its arguments, pointer in `a2` and length in `a3`.
176+
177+
The returned results are stored in registers `a0` (pointer) and `a1` (length). The output buffer contains the SCALE-encoded return value.
178+
179+
All host functions must properly account for and deduct gas based on their computational costs.
180+
181+
**Example Rust Implementation using [PolkaVM SDK](https://github.com/paritytech/polkavm)**:
182+
183+
```rust
184+
#[polkavm_derive::polkavm_import]
185+
extern "C" {
186+
fn extension_call(extension_id:u64, call_ptr:u32, call_len: u32) -> (u32, u32);
187+
}
188+
```
189+
190+
#### PVQ Executor Implementation
191+
192+
Practically, the executor has a core method `execute` to initialize the program and perform argument invocation, which takes:
193+
194+
- `program`: PVQ main binary.
195+
- `args`: PVQ query data.
196+
- `gas_limit`: Maximum PVM gas limit for the query. If not provided, the executor works at no gas metering mode.
197+
198+
**Example Rust Implementation**:
199+
200+
```rust
201+
pub fn execute(
202+
&mut self,
203+
program: &[u8],
204+
args: &[u8],
205+
gas_limit: Option<i64>,
206+
) -> Result<Vec<u8>, PvqExecutorError> {...}
207+
208+
pub enum PvqExecutorError<UserError> {
209+
InvalidProgramFormat,
210+
MemoryAccessError(polkavm::MemoryAccessError),
211+
Trap,
212+
NotEnoughGas,
213+
User(UserError),
214+
OtherPvmError(polkavm::Error),
215+
// Implementors can define additional error variants to differentiate specific panic reasons for internal debugging purposes.
216+
}
217+
```
218+
219+
Additionally, it provides an initialization method that sets up the PVM execution environment and external interfaces by pre-registering the required host functions:
220+
221+
**Example Rust Implementation**:
222+
223+
```rust
224+
pub fn new(context: PvqContext) -> Self
225+
```
226+
227+
### RuntimeAPI Integration
228+
229+
The RuntimeAPI for off-chain query usage includes two methods:
230+
231+
- `execute_query`: Executes the query and returns the result. It takes:
232+
- `program`: PVQ binary.
233+
- `args`: Query arguments that is SCALE-encoded.
234+
- `gas_limit`: Optional gas limit. If not provided, we have the gas limit which corresponds to 2 seconds as maximum execution time the query.
235+
236+
- `metadata`: Returns information about available extensions, including their IDs, supported methods, gas costs, etc. This provides feature discovery capabilities. The metadata is encoded using `scale-info`, following a similar approach to [`frame-metadata`](https://github.com/paritytech/frame-metadata/).
237+
238+
**Example PVQ Runtime API**:
239+
240+
```rust
241+
decl_runtime_apis! {
242+
pub trait PvqApi {
243+
fn execute_query(program: Vec<u8>, args: Vec<u8>, gas_limit: Option<i64>) -> PvqResult;
244+
fn metadata() -> Vec<u8>;
245+
}
246+
}
247+
type PvqResult = Result<PvqResponse, PvqError>;
248+
type PvqResponse = Vec<u8>;
249+
enum PvqError {
250+
InvalidProgramFormat,
251+
Timeout,
252+
Panic(String),
253+
}
254+
```
255+
256+
**Example Metadata**:
257+
258+
```rust
259+
pub struct Metadata {
260+
pub types: PortableRegistry,
261+
pub extensions: Vec<ExtensionMetadata<PortableForm>>,
262+
}
263+
```
264+
265+
### XCM integration
266+
267+
The integration of PVQ into XCM is achieved by adding a new instruction to XCM, as well as a new variant of the `Response` type in the `QueryResponse` message.
268+
269+
- A new `ReportQuery` instruction: report to a given destination the results of a PVQ. After query, a `QueryResponse` message of type `PvqResult` will be sent to the described destination.
270+
271+
Operands:
272+
273+
- `query: BoundedVec<u8, MaxPvqSize>`: Encoded bytes of the tuple `(program, args)`. `MaxPvqSize` is the generic parameter type size limit (i.e. 2MB).
274+
275+
- `max_weight: Weight`: Maximum weight that the query should take.
276+
- `info: QueryResponseInfo`: Information for making the response.
277+
278+
```rust
279+
ReportQuery {
280+
query: BoundedVec<u8, MaxPvqSize>,
281+
max_weight: Weight,
282+
info: QueryResponseInfo,
283+
}
284+
```
285+
286+
- A new variant to the `Response` type in `QueryResponse`
287+
- `PvqResult = 6 (BoundedVec<u8, MaxPvqResult>)`
288+
289+
`PvqResult` is a variant type:
290+
291+
- `Ok(Vec<u8>)`: Successful query result
292+
- `Err(PvqError)`: The query panics, the specific panic reason is encoded in the bytes.
293+
294+
#### Errors
295+
296+
- `FailedToDecode`: Invalid PVQ program format
297+
- `MaxWeightInvalid`: Query exceeds the weight limit
298+
- `Overflow`: Query result is too large to fit into the bounded vec
299+
- `BadOrigin`
300+
- `ReanchorFailed`
301+
- `NotHoldingFees`
302+
- `Unroutable`
303+
- `DestinationUnsupported`
304+
- `ExceedsMaxMessageSize`
305+
- `Transport`
306+
307+
## Drawbacks
308+
309+
### Performance issues
310+
311+
- PVQ Program Size: The size of a complicated PVQ program may be too large to be suitable for efficient storage and transmission via XCMP/HRMP.
312+
313+
## Testing, Security, and Privacy
314+
315+
- Testing:
316+
- A comprehensive test suite should be developed to cover various scenarios:
317+
- Positive test cases:
318+
- Basic queries with various extensions, data types, return values, custom computations, etc.
319+
- Accurate conversion between given weight limit and the gas limit of PolkaVM for both off-chain and cross-chain queries
320+
- Negative test cases:
321+
- Queries with invalid input data
322+
- Queries exceeding weight limits
323+
- Queries that panic including (no permission, host function error, etc.)
324+
- End-to-end integration testing to verify seamless interaction both off-chain and cross-chain scenarios, validating all use cases outlined in the **Motivation** section above.
325+
326+
- Security:
327+
- The PVQ extension implementors must enforce a strict read-only policy for all extension methods.
328+
- The implementation of the PVM engine must be secure and robust, refer to the discussion in [Gray Paper](https://graypaper.com/) for more details.
329+
330+
- Privacy:
331+
N/A
332+
333+
## Performance, Ergonomics, and Compatibility
334+
335+
### Performance
336+
337+
As a newly introduced feature, PVQ operates independently and does not impact or degrade the performance of existing runtime implementations.
338+
339+
### Ergonomics
340+
341+
From the perspective of off-chain tooling, this proposal streamlines development by unifying multiple chain-specific RuntimeAPIs under a single consistent interface.
342+
This significantly benefits wallet and dApp developers by eliminating the need to handle individual implementations for similar operations across different chains. The proposal also enhances development flexibility by allowing custom computations to be modularly encapsulated as PolkaVM programs that interact with the exposed APIs.
343+
344+
### Compatibility
345+
346+
For RuntimeAPI integration, the proposal defines new APIs, which do not break compatibility with existing interfaces.
347+
For XCM Integration, the proposal does not modify the existing XCM message format, which is backwards compatible.
348+
349+
## Prior Art and References
350+
351+
There are several discussions related to the proposal, including:
352+
353+
- [Original discussion](https://forum.polkadot.network/t/wasm-view-functions/1045) about having a mechanism to avoid code duplications between the runtime and front-ends/wallets. In the original design, the custom computations are compiled as a wasm function.
354+
- [View functions](https://github.com/paritytech/polkadot-sdk/pull/4722) aims to provide view-only functions at the pallet level. Additionally, [Facade Project](https://github.com/paritytech/polkadot-sdk/pull/4722) aims to gather and return commonly wanted information in runtime level.
355+
PVQ does not conflict with them, and it can take advantage of these Pallet View Functions / Runtime APIs and allow people to build arbitrary PVQ programs to obtain more custom/complex data that is not otherwise expressed by these two proposals.
356+
357+
## Unresolved Questions
358+
359+
- The specific conversion between gas and weight has not been finalized and will likely require development of a suitable benchmarking methodology.
360+
361+
## Future Directions and Related Material
362+
363+
Once PVQ and the aforementioned Facade Project are ready, there are opportunities to consolidate overlapping functionality between the two systems. For example, the metadata APIs could potentially be unified to provide a more cohesive interface for runtime information. This would help reduce duplication and improve maintainability while preserving the distinct benefits of each approach.

0 commit comments

Comments
 (0)