Skip to content

Commit 20eecd5

Browse files
cdamianmikiquantumniklabh
authored
Add events and extrinsics parsing logic based on metadata (#326)
* events: Add initial metadata parsing logic * events: Add override decoding funcs, add support for primitive arrays * events: Extract decode type def * events: Add event parsing test script, provide more details in error messages * events: Add variant support for vectors * milestone 1: Add type registry and test (#327) * milestone: Add type registry and test * milestone1: Add metadatas from multiple chains and docker file for tests * registry: Add methods to create call and error registries * dockerfile: Fix test Dockerfile * registry: Add more tests * Events parsing v2 milestone 2 (#338) * milestone: Add type registry and test * milestone1: Add metadatas from multiple chains and docker file for tests * events: Add decoders for each event field type * registry: Add decoders * registry: Add retryable executor and use it in parser * registry: Add mocks and more tests * registry: Use BitVec when parsing bit sequences * test: Add more mocks and tests * registry: Update field name retrieval * registry-test: Get headers instead of blocks * Add generic chain RPC, more tests, and Dockerfiles for the 2nd milestone * make: Add container name * chain: Add constructor for default chain * registry: Change field separator to * parser: Add DefaultExtrinsicParser and DefaultExtrinsic * retriever: Add DefaultExtrinsicRetriever and adjust tests * rpc: Add more comments to default entities * retriever: Enable all live tests * Add registry readme (#351) * Fix docker run for w3f milestone 2 tests (#353) * Address review milestone 2 (#355) * add configurable event threshold * Add natural language desc on event parsing * make: Remove grant related targets --------- Co-authored-by: Miguel Hervas <miguel.hervas.lazaro@gmail.com> Co-authored-by: Nikhil Ranjan <niklabh811@gmail.com>
1 parent fa39166 commit 20eecd5

45 files changed

Lines changed: 8140 additions & 2 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ This package is feature complete, but it is relatively new and might still conta
1717

1818
Please refer to https://godoc.org/github.com/centrifuge/go-substrate-rpc-client
1919

20+
### Usage test examples of Dynamic Parsing of events & extrinsics
21+
[Registry docs](registry/REGISTRY.md)
2022
## Contributing
2123

2224
1. Install dependencies by running `make`

error/error.go

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
package error
2+
3+
import (
4+
"fmt"
5+
"strings"
6+
)
7+
8+
type Error string
9+
10+
func (e Error) Error() string {
11+
return string(e)
12+
}
13+
14+
func (e Error) Is(err error) bool {
15+
return strings.Contains(string(e), err.Error())
16+
}
17+
18+
func (e Error) Wrap(err error) Error {
19+
return Error(fmt.Errorf("%s: %w", e, err).Error())
20+
}
21+
22+
func (e Error) WithMsg(msgFormat string, formatArgs ...any) Error {
23+
msg := fmt.Sprintf(msgFormat, formatArgs...)
24+
25+
return Error(fmt.Sprintf("%s: %s", e, msg))
26+
}

error/error_test.go

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
package error
2+
3+
import (
4+
"errors"
5+
"fmt"
6+
"testing"
7+
8+
"github.com/stretchr/testify/assert"
9+
)
10+
11+
const (
12+
testErr = Error("test error")
13+
)
14+
15+
func TestError(t *testing.T) {
16+
newStdErr := errors.New("new std error")
17+
wrappedErr := testErr.Wrap(newStdErr)
18+
19+
assert.True(t, errors.Is(wrappedErr, testErr))
20+
assert.True(t, errors.Is(wrappedErr, newStdErr))
21+
assert.Equal(t, fmt.Sprintf("%s: %s", testErr.Error(), newStdErr.Error()), wrappedErr.Error())
22+
23+
newErr := Error("new error")
24+
newWrappedErr := newErr.Wrap(wrappedErr)
25+
26+
assert.True(t, errors.Is(newWrappedErr, newErr))
27+
assert.True(t, errors.Is(newWrappedErr, testErr))
28+
assert.True(t, errors.Is(newWrappedErr, newStdErr))
29+
assert.Equal(t, fmt.Sprintf("%s: %s", newErr.Error(), wrappedErr.Error()), newWrappedErr.Error())
30+
31+
err := testErr.WithMsg("%d", 1)
32+
assert.Equal(t, fmt.Sprintf("%s: 1", testErr), err.Error())
33+
34+
err = testErr.WithMsg("test msg")
35+
assert.Equal(t, fmt.Sprintf("%s: test msg", testErr), err.Error())
36+
}

registry/REGISTRY.md

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# GSRPC Registry
2+
The GSRPC Registry can parse target metadata information into an in-memory registry of complex structures.
3+
4+
By leveraging the on-chain metadata, GSRPC is more robust to changes on types, allowing clients to only keep updated the types that are relevant to their business operation.
5+
6+
This registry can be used afterwards to decode data read from live chains (events & extrinsics).
7+
8+
## How to parse events and its types
9+
First we instantiate the API with the client node and open a connection:
10+
```go
11+
testURL := "wss://fullnode.parachain.centrifuge.io" // Your endpoint
12+
api, err := gsrpc.NewSubstrateAPI(testURL)
13+
14+
if err != nil {
15+
log.Printf("Couldn't connect to '%s': %s\n", testURL, err)
16+
return
17+
}
18+
```
19+
Then we instantiate the Event Retriever logic which internally creates a new EventRegistry reading from the target metadata of the connected chain. We pass as well the state RPC so the storage API is available:
20+
```go
21+
retriever, err := NewDefaultEventRetriever(state.NewEventProvider(api.RPC.State), api.RPC.State)
22+
23+
if err != nil {
24+
log.Printf("Couldn't create event retriever: %s", err)
25+
return
26+
}
27+
```
28+
At this point what we need is a block hash to read the events within. In this example we get the latest block header and the correspondent block hash out of the block number:
29+
```go
30+
header, err := api.RPC.Chain.GetHeaderLatest()
31+
32+
if err != nil {
33+
log.Printf("Couldn't get latest header for '%s': %s\n", testURL, err)
34+
return
35+
}
36+
37+
blockHash, err := api.RPC.Chain.GetBlockHash(uint64(header.Number))
38+
39+
if err != nil {
40+
log.Printf("Couldn't retrieve blockHash for '%s', block number %d: %s\n", testURL, header.Number, err)
41+
return
42+
}
43+
```
44+
Finally, we just use the retriever function to read all the events in that block based on the chain metadata loaded in the event registry:
45+
```go
46+
events, err := retriever.GetEvents(blockHash)
47+
48+
if err != nil {
49+
log.Printf("Couldn't retrieve events for '%s', block number %d: %s\n", testURL, header.Number, err)
50+
return
51+
}
52+
53+
log.Printf("Found %d events for '%s', at block number %d.\n", len(events), testURL, header.Number)
54+
55+
// Example of the events returned structure
56+
for _, event := range events {
57+
log.Printf("Event ID: %x \n", event.EventID)
58+
log.Printf("Event Name: %s \n", event.Name)
59+
log.Printf("Event Fields Count: %d \n", len(event.Fields))
60+
for k, v := range event.Fields {
61+
log.Printf("Field Name: %s \n", k)
62+
log.Printf("Field Type: %v \n", reflect.TypeOf(v))
63+
log.Printf("Field Value: %v \n", v)
64+
}
65+
}
66+
67+
```
68+
69+
## Extended Usage
70+
Since docs get outdated fairly quick, here are links to tests that will always be up-to-date.
71+
### Populate Call, Error & Events Registries
72+
[Browse me](registry_test.go)
73+
74+
### Event retriever
75+
[TestLive_EventRetriever_GetEvents](retriever/event_retriever_live_test.go)
76+
### Extrinsic retriever
77+
Since chain runtimes can be customized, modifying core types such as Accounts, Signature payloads or Payment payloads, the code supports a customizable way of passing those custom types to the extrinsic retriever.
78+
79+
On the other hand, since a great majority of chains do not need to change these types, the tool provides a default for the most common used ones.
80+
#### Using Chain Defaults
81+
[TestExtrinsicRetriever_NewDefault](retriever/extrinsic_retriever_test.go#L179)
82+
#### Using Custom core types
83+
[TestLive_ExtrinsicRetriever_GetExtrinsics](retriever/extrinsic_retriever_live_test.go)

registry/error.go

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
package registry
2+
3+
import libErr "github.com/centrifuge/go-substrate-rpc-client/v4/error"
4+
5+
const (
6+
ErrRecursiveDecodersResolving = libErr.Error("recursive decoders resolving")
7+
ErrErrorsTypeNotFound = libErr.Error("errors type not found")
8+
ErrErrorsTypeNotVariant = libErr.Error("errors type not a variant")
9+
ErrErrorFieldsRetrieval = libErr.Error("error fields retrieval")
10+
ErrCallsTypeNotFound = libErr.Error("calls type not found")
11+
ErrCallsTypeNotVariant = libErr.Error("calls type not a variant")
12+
ErrCallFieldsRetrieval = libErr.Error("call fields retrieval")
13+
ErrEventsTypeNotFound = libErr.Error("events type not found")
14+
ErrEventsTypeNotVariant = libErr.Error("events type not a variant")
15+
ErrEventFieldsRetrieval = libErr.Error("event fields retrieval")
16+
ErrFieldDecoderForRecursiveFieldNotFound = libErr.Error("field decoder for recursive field not found")
17+
ErrRecursiveFieldResolving = libErr.Error("recursive field resolving")
18+
ErrFieldTypeNotFound = libErr.Error("field type not found")
19+
ErrFieldDecoderRetrieval = libErr.Error("field decoder retrieval")
20+
ErrCompactFieldTypeNotFound = libErr.Error("compact field type not found")
21+
ErrCompositeTypeFieldsRetrieval = libErr.Error("composite type fields retrieval")
22+
ErrArrayFieldTypeNotFound = libErr.Error("array field type not found")
23+
ErrVectorFieldTypeNotFound = libErr.Error("vector field type not found")
24+
ErrFieldTypeDefinitionNotSupported = libErr.Error("field type definition not supported")
25+
ErrVariantTypeFieldsRetrieval = libErr.Error("variant type fields decoding")
26+
ErrCompactTupleItemTypeNotFound = libErr.Error("compact tuple item type not found")
27+
ErrCompactTupleItemFieldDecoderRetrieval = libErr.Error("compact tuple item field decoder retrieval")
28+
ErrCompactCompositeFieldTypeNotFound = libErr.Error("compact composite field type not found")
29+
ErrCompactCompositeFieldDecoderRetrieval = libErr.Error("compact composite field decoder retrieval")
30+
ErrArrayItemFieldDecoderRetrieval = libErr.Error("array item field decoder retrieval")
31+
ErrSliceItemFieldDecoderRetrieval = libErr.Error("slice item field decoder retrieval")
32+
ErrTupleItemTypeNotFound = libErr.Error("tuple item type not found")
33+
ErrTupleItemFieldDecoderRetrieval = libErr.Error("tuple item field decoder retrieval")
34+
ErrBitStoreTypeNotFound = libErr.Error("bit store type not found")
35+
ErrBitStoreTypeNotSupported = libErr.Error("bit store type not supported")
36+
ErrBitOrderTypeNotFound = libErr.Error("bit order type not found")
37+
ErrBitOrderCreation = libErr.Error("bit order creation")
38+
ErrPrimitiveTypeNotSupported = libErr.Error("primitive type not supported")
39+
ErrTypeFieldDecoding = libErr.Error("type field decoding")
40+
ErrVariantByteDecoding = libErr.Error("variant byte decoding")
41+
ErrVariantFieldDecoderNotFound = libErr.Error("variant field decoder not found")
42+
ErrArrayItemDecoderNotFound = libErr.Error("array item decoder not found")
43+
ErrArrayItemDecoding = libErr.Error("array item decoding")
44+
ErrSliceItemDecoderNotFound = libErr.Error("slice item decoder not found")
45+
ErrSliceLengthDecoding = libErr.Error("slice length decoding")
46+
ErrSliceItemDecoding = libErr.Error("slice item decoding")
47+
ErrCompositeFieldDecoding = libErr.Error("composite field decoding")
48+
ErrValueDecoding = libErr.Error("value decoding")
49+
ErrRecursiveFieldDecoderNotFound = libErr.Error("recursive field decoder not found")
50+
ErrBitVecDecoding = libErr.Error("bit vec decoding")
51+
)

registry/exec/exec.go

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
package exec
2+
3+
import (
4+
"errors"
5+
"fmt"
6+
"strings"
7+
"time"
8+
)
9+
10+
//go:generate mockery --name RetryableExecutor --structname RetryableExecutorMock --filename exec_mock.go --inpackage
11+
12+
// RetryableExecutor is the interface used for executing a closure and its fallback if the initial execution fails.
13+
//
14+
// The interface is generic over type T which represents the return value of the closure.
15+
type RetryableExecutor[T any] interface {
16+
ExecWithFallback(execFn func() (T, error), fallbackFn func() error) (T, error)
17+
}
18+
19+
// retryableExecutor implements RetryableExecutor.
20+
//
21+
// It can be configured via the provided OptsFn(s).
22+
type retryableExecutor[T any] struct {
23+
opts *Opts
24+
}
25+
26+
// NewRetryableExecutor creates a new RetryableExecutor.
27+
func NewRetryableExecutor[T any](opts ...OptsFn) RetryableExecutor[T] {
28+
execOpts := NewDefaultExecOpts()
29+
30+
for _, opt := range opts {
31+
opt(execOpts)
32+
}
33+
34+
return &retryableExecutor[T]{
35+
execOpts,
36+
}
37+
}
38+
39+
// ExecWithFallback will attempt to execute the provided execFn and, in the case of failure, it will execute
40+
// the fallbackFn and retry execution of execFn.
41+
func (r *retryableExecutor[T]) ExecWithFallback(execFn func() (T, error), fallbackFn func() error) (res T, err error) {
42+
if execFn == nil {
43+
return res, ErrMissingExecFn
44+
}
45+
46+
if fallbackFn == nil {
47+
return res, ErrMissingFallbackFn
48+
}
49+
50+
execErr := &Error{}
51+
52+
retryCount := uint(0)
53+
54+
for {
55+
res, err = execFn()
56+
57+
if err == nil {
58+
return res, nil
59+
}
60+
61+
execErr.AddErr(fmt.Errorf("exec function error: %w", err))
62+
63+
if retryCount == r.opts.maxRetryCount {
64+
return res, execErr
65+
}
66+
67+
if err = fallbackFn(); err != nil && !r.opts.retryOnFallbackError {
68+
execErr.AddErr(fmt.Errorf("fallback function error: %w", err))
69+
70+
return res, execErr
71+
}
72+
73+
retryCount++
74+
75+
time.Sleep(r.opts.retryTimeout)
76+
}
77+
}
78+
79+
var (
80+
ErrMissingExecFn = errors.New("no exec function provided")
81+
ErrMissingFallbackFn = errors.New("no fallback function provided")
82+
)
83+
84+
const (
85+
defaultMaxRetryCount = 3
86+
defaultErrTimeout = 0 * time.Second
87+
defaultRetryOnFallbackError = true
88+
)
89+
90+
// Opts holds the configurable options for a RetryableExecutor.
91+
type Opts struct {
92+
// maxRetryCount holds maximum number of retries in the case of failure.
93+
maxRetryCount uint
94+
95+
// retryTimeout holds the timeout between retries.
96+
retryTimeout time.Duration
97+
98+
// retryOnFallbackError specifies whether a retry will be done in the case of
99+
// failure of the fallback function.
100+
retryOnFallbackError bool
101+
}
102+
103+
// NewDefaultExecOpts creates the default Opts.
104+
func NewDefaultExecOpts() *Opts {
105+
return &Opts{
106+
maxRetryCount: defaultMaxRetryCount,
107+
retryTimeout: defaultErrTimeout,
108+
retryOnFallbackError: defaultRetryOnFallbackError,
109+
}
110+
}
111+
112+
// OptsFn is function that operate on Opts.
113+
type OptsFn func(opts *Opts)
114+
115+
// WithMaxRetryCount sets the max retry count.
116+
//
117+
// Note that a default value is provided if the provided count is 0.
118+
func WithMaxRetryCount(maxRetryCount uint) OptsFn {
119+
return func(opts *Opts) {
120+
if maxRetryCount == 0 {
121+
maxRetryCount = defaultMaxRetryCount
122+
}
123+
124+
opts.maxRetryCount = maxRetryCount
125+
}
126+
}
127+
128+
// WithRetryTimeout sets the retry timeout.
129+
func WithRetryTimeout(retryTimeout time.Duration) OptsFn {
130+
return func(opts *Opts) {
131+
opts.retryTimeout = retryTimeout
132+
}
133+
}
134+
135+
// WithRetryOnFallBackError sets the retryOnFallbackError flag.
136+
func WithRetryOnFallBackError(retryOnFallbackError bool) OptsFn {
137+
return func(opts *Opts) {
138+
opts.retryOnFallbackError = retryOnFallbackError
139+
}
140+
}
141+
142+
// Error holds none or multiple errors that can happen during execution.
143+
type Error struct {
144+
errs []error
145+
}
146+
147+
// AddErr appends an error to the error slice of Error.
148+
func (e *Error) AddErr(err error) {
149+
e.errs = append(e.errs, err)
150+
}
151+
152+
// Error implements the standard error interface.
153+
func (e *Error) Error() string {
154+
sb := strings.Builder{}
155+
156+
for i, err := range e.errs {
157+
sb.WriteString(fmt.Sprintf("error %d: %s\n", i, err))
158+
}
159+
160+
return sb.String()
161+
}

0 commit comments

Comments
 (0)