Merge pull request #25 from kaleido-io/issue-24

Add delete functionality with policy engine involvement

Author: peterbroadhurst

README.md

@@ -2,9 +2,10 @@
 
 # Hyperledger FireFly Transaction Manager
 
-Plugable microservice component of Hyperledger FireFly, responsible for:
+The core component of the FireFly Connector Framework for Blockchains, responsible for:
 
 - Submission of transactions to blockchains of all types
+  - Nonce management - idempotent submission of transactions, and assignment of nonces 
   - Protocol connectivity decoupled with additional lightweight API connector
   - Easy to add additional protocols that conform to normal patterns of TX submission / events
 
@@ -16,12 +17,96 @@ Plugable microservice component of Hyperledger FireFly, responsible for:
   - Extensible policy engine
   - Gas station API integration
 
-- Event streaming [* work in progress to extract from previous location in ethconnect]
+- Event streaming
   - Protocol agnostic event polling/streaming support
   - Reliable checkpoint restart
   - At least once delivery API
 
-![Hyperledger FireFly Transaction Manager](./images/firefly_transaction_manager.jpg)
+## Architecture
+
+The core architecture of the FireFly Connector Framework is as follows:
+
+[![Hyperledger FireFly Transaction Manager](./images/firefly_connector_framework_architecture.jpg)](./images/firefly_connector_framework_architecture.jpg)
+
+This re-usable codebase contains as much as possible of the re-usable heavy lifting code needed across any blockchain.
+
+The framework is currently constrained to blockchains that adhere to certain basic principals:
+
+1. Has transactions
+  - That are signed
+  - That can optionally have gas semantics (limits and prices, expressed in a blockchain specific way)
+2. Has events (or "logs")
+  - That are emitted as a deterministic outcome of transactions
+3. Has blocks
+  - Containing zero or more transactions, with their associated events
+  - With a sequential numeric order
+  - With a hash
+  - With a parent hash
+4. Has finality for transactions & events that can be expressed as a level of confidence over time
+  - Confirmations: A number of sequential blocks in the canonical chain that contain the transaction
+
+## Nonce management
+
+The nonces for transactions is assigned as early as possible in the flow:
+- Before the REST API for submission of the transaction occurs
+- After the FFCAPI blockchain connector verifies the transaction can be encoded successfully to the chain
+- With protection against multiple parallel API requests for the same signing address
+- With stateful persistence meaning the connector knows about all nonces it previously allocated, to avoids duplicates
+
+This "at source" allocation of nonces provides the strictest assurance of order of transactions possible,
+because the order is locked in with the coordination of the business logic of the application submitting the transaction.
+
+As well as protecting against loss of transactions, this protects against duplication of transactions - even in crash
+recovery scenarios with a sufficiently reliable persistence layer.
+
+### Avoid multiple nonce management systems against the same signing key
+
+FFTM is optimized for cases where all transactions for a given signing address flow through the
+same FireFly connector. If you have signing and nonce allocation happening elsewhere, not going through the
+FireFly blockchain connector, then it is possible that the same nonce will be allocated in two places.
+
+> Be careful that the signing keys for transactions you stream through the Nonce Management of the FireFly
+> blockchain connector are not used elsewhere.
+
+If you must have multiple systems performing nonce management against the same keys you use with FireFly nonce management,
+you can set the `transactions.nonceStateTimeout` to `0` (or a low threshold like `100ms`) to cause the nonce management
+to query the pending transaction pool of the node every time a nonce is allocated.
+
+This reduces the window for concurrent nonce allocation to be small (basically the same as if you had
+multiple simple web/mobile wallets used against the same key), but it does not eliminate it completely it.
+
+### Why "at source" nonce management was chosen vs. "at target"
+
+The "at source" approach to ordering used in FFTM could be compared with the "at target" allocation of nonces used in
+[EthConnect](https://github.com/hyperledger/firefly-ethconnect)).
+
+The "at target" approach optimizes for throughput and ability to send new transactions to the chain,
+with an at-least-once delivery assurance to the applications.
+
+An "at target" algorithm as used in EthConnect could resume transaction delivery automatically without operator intervention
+from almost all scenarios, including where nonces have been double allocated.
+
+However, "at target" comes with two compromises that mean FFTM chose the "at source" approach was chosen for FFTM:
+
+- Individual transactions might fail in certain scenarios, and subsequent transactions will still be streamed to the chain.
+  While desirable for automation and throughput, this reduces the ordering guarantee for high value transactions.
+
+- In crash recovery scenarios the assurance is at-least-once delivery for "at target" ordering (rather than "exactly once"),
+  although the window can be made very small through various optimizations included in the EthConnect codebase.
+
+## Policy Manager
+
+> TODO: Add more detail to describe the pluggability of the Policy Manager component, to perform transaction gas price
+>       estimation, advanced monitoring of transactions, submission and re-submission of the transactions with updated
+>       parameters (such as gas price) etc.
+
+## Event streaming
+
+One of the most sophisticated parts of the FireFly Connector Framework is the handling of event streams.
+
+> TODO: More detail to back up this diagram.
+
+[![Event Streams](./images/fftm_event_streams_architecture.jpg)](./images/fftm_event_streams_architecture.jpg)
 
 # Configuration
 

images/fftm_event_streams_architecture.jpg

@@ -36,6 +36,7 @@ var (
 	APIEndpointGetEventStreams              = ffm("api.endpoints.get.eventstreams", "List event streams")
 	APIEndpointGetEventStream               = ffm("api.endpoints.get.eventstream", "Get an event stream with status")
 	APIEndpointDeleteEventStream            = ffm("api.endpoints.delete.eventstream", "Delete an event stream")
+	APIEndpointDeleteTransaction            = ffm("api.endpoints.delete.transaction", "Request transaction deletion by the policy engine. Result could be immediate (200), asynchronous (202), or rejected with an error")
 	APIEndpointGetSubscriptions             = ffm("api.endpoints.get.subscriptions", "Get listeners - route deprecated in favor of /eventstreams/{streamId}/listeners")
 	APIEndpointGetSubscription              = ffm("api.endpoints.get.subscription", "Get listener - route deprecated in favor of /eventstreams/{streamId}/listeners/{listenerId}")
 	APIEndpointPostSubscriptions            = ffm("api.endpoints.post.subscriptions", "Create new listener - route deprecated in favor of /eventstreams/{streamId}/listeners")

images/firefly_connector_framework_architecture.jpg

@@ -82,4 +82,7 @@ var (
 	MsgInvalidSortDirection          = ffe("FF21064", "Sort direction must be 'asc'/'ascending' or 'desc'/'descending': '%s'", http.StatusBadRequest)
 	MsgDuplicateID                   = ffe("FF21065", "ID '%s' is not unique", http.StatusConflict)
 	MsgTransactionFailed             = ffe("FF21066", "Transaction execution failed")
+	MsgTransactionNotFound           = ffe("FF21067", "Transaction '%s' not found", http.StatusNotFound)
+	MsgPolicyEngineRequestTimeout    = ffe("FF21068", "The policy engine did not acknowledge the request after %.2fs", 408)
+	MsgPolicyEngineRequestInvalid    = ffe("FF21069", "Invalid policy engine request type '%d'")
 )

images/firefly_transaction_manager.jpg

@@ -61,6 +61,7 @@ type ManagedTX struct {
 	Created            *fftypes.FFTime                    `json:"created"`
 	Updated            *fftypes.FFTime                    `json:"updated"`
 	Status             TxStatus                           `json:"status"`
+	DeleteRequested    *fftypes.FFTime                    `json:"deleteRequested,omitempty"`
 	SequenceID         *fftypes.UUID                      `json:"sequenceId"`
 	Nonce              *fftypes.FFBigInt                  `json:"nonce"`
 	Gas                *fftypes.FFBigInt                  `json:"gas"`

internal/tmmsgs/en_api_descriptions.go

@@ -44,6 +44,26 @@ type Manager interface {
 	Close()
 }
 
+type policyEngineAPIRequestType int
+
+const (
+	policyEngineAPIRequestTypeDelete policyEngineAPIRequestType = iota
+)
+
+// policyEngineAPIRequest requests are queued to the policy engine thread for processing against a given Transaction
+type policyEngineAPIRequest struct {
+	requestType policyEngineAPIRequestType
+	txID        string
+	startTime   time.Time
+	response    chan policyEngineAPIResponse
+}
+
+type policyEngineAPIResponse struct {
+	tx     *apitypes.ManagedTX
+	err    error
+	status int // http status code (200 Ok vs. 202 Accepted) - only set for success cases
+}
+
 type manager struct {
 	ctx            context.Context
 	cancelCtx      func()
@@ -58,14 +78,15 @@ type manager struct {
 	inflightUpdate chan bool
 	inflight       []*pendingState
 
-	mux               sync.Mutex
-	lockedNonces      map[string]*lockedNonce
-	eventStreams      map[fftypes.UUID]events.Stream
-	streamsByName     map[string]*fftypes.UUID
-	policyLoopDone    chan struct{}
-	blockListenerDone chan struct{}
-	started           bool
-	apiServerDone     chan error
+	mux                     sync.Mutex
+	policyEngineAPIRequests []*policyEngineAPIRequest
+	lockedNonces            map[string]*lockedNonce
+	eventStreams            map[fftypes.UUID]events.Stream
+	streamsByName           map[string]*fftypes.UUID
+	policyLoopDone          chan struct{}
+	blockListenerDone       chan struct{}
+	started                 bool
+	apiServerDone           chan error
 
 	policyLoopInterval time.Duration
 	nonceStateTimeout  time.Duration