[docs] Update sidecar and verifier docs

docs/sidecar.md

@@ -13,9 +13,11 @@ SPDX-License-Identifier: Apache-2.0
    - [Task 2. Relaying Blocks to the Coordinator](#task-2-relaying-blocks-to-the-coordinator)
    - [Task 3. Persisting Committed Block in the File System](#task-3-persisting-committed-block-in-the-file-system)
 5. [Delivering Committed Block to Registered Clients](#5-delivering-committed-block-to-registered-clients)
-6. [Failure and Recovery](#6-failure-and-recovery)
+6. [Notification Service](#6-notification-service)
+7. [Failure and Recovery](#7-failure-and-recovery)
    - [A. State Persistence](#a-state-persistence)
    - [B. Restart and Recovery Procedure](#b-restart-and-recovery-procedure)
+   - [Notification Service Recovery](#notification-service-recovery)
 
 
 ## 1. Overview
@@ -29,18 +31,20 @@ delivered to downstream clients.
 The Sidecar performs four main tasks:
 
 1.  **Fetch Blocks:** Retrieves blocks sequentially from the Ordering Service.
-2.  **Relay and Validate:** Forwards fetched blocks to the Coordinator and receives feedback on transaction statuses
+2.  **Translate and Validate:** Decode transactions to a committer transactions (`protoblocktx.Tx`), 
+    and filter malformed transactions.
+3. **Relay and Collect:** Forwards fetched blocks to the Coordinator and receives feedback on transaction statuses
     within those blocks.
-3.  **Persist Committed Blocks:** Stores blocks confirmed as committed by the Coordinator in a local, append-only file
+4. **Persist Committed Blocks:** Stores blocks confirmed as committed by the Coordinator in a local, append-only file
     store for durability and auditability.
-4.  **Deliver to Clients:** Delivers the committed blocks to registered client applications.
+5. **Deliver to Clients:** Delivers the committed blocks to registered client applications.
+6. **Notification:** Notifies subscribers when transactions are committed or aborted, on a per-transaction ID basis.
 
 Note that the fourth task is executed only when users/clients creates a stream with the sidecar.
 
 ## 3. Configuration
 
-The Sidecar requires the following configuration settings, provided in a Yaml configuration
-file:
+The Sidecar requires the following configuration settings, provided in a Yaml configuration file:
 
  - *Ordering Service Endpoints*: Address(es) of the ordering service node(s).
  - *Coordinator Endpoint*: Address of the coordinator service.
@@ -220,7 +224,38 @@ type Block struct {
 	TxsNum []uint32           
 }
 ```
-This conversion facilitates easier processing and decouples the Sidecar's internal logic from complex Fabric block structures.
+
+This conversion facilitates easier processing and decouples the committer's internal logic from the nested Fabric block structures.
+
+This internal block will only contain well-formed committer transactions.
+I.e., the sidecar filters transactions that cannot be processed correctly.
+Following are a list of statuses that the sidecar filters by:
+```protobuf
+enum Status {
+   // ...
+
+   // Cannot be stored in the state database because the TX ID cannot be extracted,
+   // or the TX ID entry is already occupied.
+   REJECTED_DUPLICATE_TX_ID = 100;                     // Transaction with the same ID has already been processed.
+   MALFORMED_BAD_ENVELOPE = 101;                       // Cannot unmarshal envelope.
+   MALFORMED_MISSING_TX_ID = 102;                      // Envelope is missing transaction ID.
+
+   // Stored in the state database. Prevents submitting a transaction with the same ID.
+   MALFORMED_UNSUPPORTED_ENVELOPE_PAYLOAD = 103;       // Unsupported envelope payload type.
+   MALFORMED_BAD_ENVELOPE_PAYLOAD = 104;               // Cannot unmarshal envelope's payload.
+   MALFORMED_TX_ID_CONFLICT = 105;                     // Envelope's TX ID does not match the payload's TX ID.
+   MALFORMED_EMPTY_NAMESPACES = 106;                   // No namespaces provided.
+   MALFORMED_DUPLICATE_NAMESPACE = 107;                // Duplicate namespace detected.
+   MALFORMED_NAMESPACE_ID_INVALID = 108;               // Invalid namespace identifier.
+   MALFORMED_BLIND_WRITES_NOT_ALLOWED = 109;           // Blind writes are not allowed in a namespace transaction.
+   MALFORMED_NO_WRITES = 110;                          // No write operations in the transaction.
+   MALFORMED_EMPTY_KEY = 111;                          // Unset key detected.
+   MALFORMED_DUPLICATE_KEY_IN_READ_WRITE_SET = 112;    // Duplicate key in the read-write set.
+   MALFORMED_MISSING_SIGNATURE = 113;                  // Number of signatures does not match the number of namespaces.
+   MALFORMED_NAMESPACE_POLICY_INVALID = 114;           // Invalid namespace policy.
+   MALFORMED_CONFIG_TX_INVALID = 115;                  // Invalid configuration transaction.
+}
+```
 
 **c. In-Flight Duplicate Transaction Detection**: 
 
@@ -378,7 +413,28 @@ The Sidecar sends back `DeliverResponse` messages, each containing a committed b
 If the client requests a future block (one not yet committed), the `Recv()` call will 
 block until that block becomes available for delivery from the Sidecar.
 
-## 6. Failure and Recovery
+## 6. Notification Service
+
+The sidecar exposes an API that allows clients to subscribe to transaction status updates and 
+receive notifications when transactions are either committed or aborted.
+
+From [api/protonotify/notify.proto](/api/protonotify/notify.proto)
+```protobuf
+// The notifier service provides API to subscribe to ledger events and receive asynchronous notifications.
+service Notifier {
+    rpc OpenNotificationStream (stream NotificationRequest) returns (stream NotificationResponse);
+}
+```
+
+Once a stream is established, the client can send a `NotificationRequest` containing a list of transaction IDs 
+and a timeout value. The sidecar responds with a `NotificationResponse`, which includes batches of transactions
+and their corresponding commit statuses for the subscribed transaction IDs. If the timeout expires before some 
+of the transactions complete, the response will include the IDs of the transactions that timed out.
+
+Note that if a transaction has already completed before the notification request is submitted, 
+no notification will be sent for it.
+
+## 7. Failure and Recovery
 
 The Sidecar is designed for resilience, allowing it to recover from failures that might
 occur at any point during block processing. This recovery capability relies on persistently
@@ -419,3 +475,8 @@ When the Sidecar restarts after a failure, it follows these steps:
   **g. Resume Operation**: Once the local block store is synchronized (missing blocks fetched)
    and the transaction statuses for the recovered range are confirmed, the Sidecar seamlessly
    resumes its standard block processing routines from block N+1.
+
+### Notification Service Recovery
+
+When a sidecar fails and restarts, the client should assume that no state is persisted at the sidecar and,
+therefore, needs to resubmit the transaction IDs of interest for which it has not yet received a status update.

docs/verification-service.md

@@ -82,11 +82,10 @@ The service uses a lock-free design with an atomic pointer to update the policy
 
 The verification process follows these steps:
 
-1. **Validate Format**: Check that the transaction is well-formed.
-2. **Policy Lookup**: For each namespace in a transaction, the service looks up the corresponding policy.
-3. **Signature Validation**: The service validates the transaction signatures against the namespace policy.
-4. **Result Determination**: If it is well-formed, and all signatures are valid, 
-the transaction is marked as `COMMITTED`. Otherwise, it's marked as `ABORTED_<reason>`. 
+1. **Policy Lookup**: For each namespace in a transaction, the service looks up the corresponding policy.
+2. **Signature Validation**: The service validates the transaction signatures against the namespace policy.
+3. **Result Determination**: If all signatures are valid, the transaction is marked as `COMMITTED`.
+   Otherwise, it's marked as `ABORTED_SIGNATURE_INVALID`.
 
 ## 4. Parallel Execution
 
@@ -454,15 +453,8 @@ func (v *verifier) verifyRequest(request *protosigverifierservice.Request) *prot
         TxId:     request.Tx.Id,
         BlockNum: request.BlockNum,
         TxNum:    request.TxNum,
-        Status:   v.verifyTX(request.Tx),
+        Status:   protoblocktx.Status_COMMITTED,
     }
-}
-
-func (v *verifier) verifyTX(tx *protoblocktx.Tx) protoblocktx.Status {
-    if status := verifyTxForm(tx); status != retValid {
-        return status
-    }
-
     // The verifiers might temporarily retain the old map while updatePolicies has already set a new one.
     // This is acceptable, provided the coordinator sends the validation status to the dependency graph
     // after updating the policies in the verifier.
@@ -478,7 +470,8 @@ func (v *verifier) verifyTX(tx *protoblocktx.Tx) protoblocktx.Status {
         nsVerifier, ok := verifiers[ns.NsId]
         if !ok {
             logger.Debugf("No verifier for namespace: '%v'", ns.NsId)
-            return protoblocktx.Status_ABORTED_SIGNATURE_INVALID
+            response.Status = protoblocktx.Status_ABORTED_SIGNATURE_INVALID
+            return response
         }
 
         // NOTE: We do not compare the namespace version in the transaction
@@ -492,15 +485,14 @@ func (v *verifier) verifyTX(tx *protoblocktx.Tx) protoblocktx.Status {
         //       namespace version, which would reflect the correct validation status.
         if err := nsVerifier.VerifyNs(request.Tx, nsIndex); err != nil {
             logger.Debugf("Invalid signature found: '%v', namespace id: '%v'", request.Tx.Id, ns.NsId)
-            return protoblocktx.Status_ABORTED_SIGNATURE_INVALID
+            response.Status = protoblocktx.Status_ABORTED_SIGNATURE_INVALID
+            return response
         }
     }
-
-    return retValid
+    return response
 }
 ```
 
-The `verifyTxForm()` verifies that a transaction is well-formed. 
 The `VerifyNs()` method in the `signature.NsVerifier` verifies signatures for a specific namespace:
 
 1. It extracts the signatures for the namespace from the transaction.