feat(backend): add middleware that handles STREAM KYC/additional data frames by sanducb · Pull Request #3706 · interledger/rafiki

sanducb · 2025-10-16T07:56:36Z

Changes proposed in this pull request

This PR introduces support for sending additional data over ILP prepare packets. This allows a receiving ASE to reject/fulfill a payment packet based on the validity of the additional data.

Context

Closes #RAF-1185

Checklist

Related issues linked using fixes #number
Tests added/updated
Make sure that all checks pass
Bruno collection updated (if necessary)
Documentation issue created with user-docs label (if necessary)
OpenAPI specs updated (if necessary)

netlify · 2025-10-16T07:56:42Z

❌ Deploy Preview for brilliant-pasca-3e80ec failed. Why did it fail? →

Name	Link
🔨 Latest commit	`8b77d8d`
🔍 Latest deploy log	https://app.netlify.com/projects/brilliant-pasca-3e80ec/deploys/69b011bf4024f50008dec8b6

mkurapov

Only focused on the kyc decision middleware here

mkurapov · 2026-01-19T21:23:32Z

+
+    const readDecision = async (): Promise<string | undefined> => {
+      try {
+        const value = await redis.get(cacheKey)


We have a poll utility function for this in utils.ts

mkurapov · 2026-01-19T21:27:33Z

+
+    const incomingPaymentId = ctx.state.streamDestination
+    const connectionId = ctx.state.connectionId ?? 'unknown'
+    const cacheKey = `kyc_decision:${incomingPaymentId}:${connectionId}`


I think we should incorporate the polling inside the existing incoming payment service that will handle the logic for this. That is where we can also put the code to actually confirm or reject the partial incoming payment.

I think the incoming payment service is fine for this one.

mkurapov · 2026-01-19T21:27:46Z

+    }
+
+    const incomingPaymentId = ctx.state.streamDestination
+    const connectionId = ctx.state.connectionId ?? 'unknown'


Let's generate our own UUID partialIncomingPaymentId

mkurapov · 2026-01-19T21:28:34Z

+import { ILPContext, ILPMiddleware } from '../rafiki'
+import { StreamState } from './stream-address'
+
+export function createKycDecisionMiddleware(): ILPMiddleware {


I think we should not have any references to "kyc" in this code change, but simply partialPaymentDecisionMiddleware or something

mkurapov

Few comments

mkurapov · 2026-01-20T12:02:42Z

      createIncomingErrorHandlerMiddleware(ilpAddress),
      createStreamAddressMiddleware(),
-
+      createKycDecisionMiddleware(),


should we move this to be after the createBalanceMiddleware ? That way, we will only publish the webhook & start polling for the decision after we were able to created a pending transfer between the peer account to the incoming payment account, confirming we have enough funds to make the transfer anyway.

mkurapov · 2026-01-26T14:22:06Z

-  dataToTransmit?: string
-): Promise<IncomingPayment | IncomingPaymentError> {
-  const { config, knex } = deps
+  options?: {


Suggested change

options?: {

options: {

(since we would only call this function when it is defined)

mkurapov · 2026-01-26T14:25:01Z

+        request: async (): Promise<string> => {
+          try {
+            const value = await redis.get(cacheKey)
+            return value ?? ''


I think we should expect the value to be stringified { approved: boolean, reason?: string }), and we can then just check for approved boolean (see #3687)

mkurapov · 2026-02-09T19:09:49Z

      createIncomingErrorHandlerMiddleware(ilpAddress),
      createStreamAddressMiddleware(),
-
+      createPartialPaymentDecisionMiddleware(),


should we move this to be after the createBalanceMiddleware ? That way, we will only publish the webhook & start polling for the decision after we were able to created a pending transfer between the peer account to the incoming payment account, confirming we have enough funds to make the transfer anyway.

Thoughts on this ^?

mkurapov · 2026-02-09T19:10:26Z

+      plugin,
+      destination,
+      quote,
+      appData: Buffer.from('hello kyc')


We can use the outgoing payments' dataToTransmit here now I think

mkurapov · 2026-02-09T19:10:31Z

+      plugin,
+      destination,
+      quote,
+      appData: Buffer.from('hello kyc')


We can use the outgoing payments' dataToTransmit here now I think

mkurapov

Few comments

mkurapov

Few comments

… payloads - wip

…unused env vars

…anges

…eiver updates

…lockfile

…artial payments to mock ASE

github-actions · 2026-03-24T14:25:31Z

🚀 Performance Test Results

Test Configuration:

VUs: 4
Duration: 1m0s

Test Metrics:

Requests/s: 40.69
Iterations/s: 13.59
Failed Requests: 0.00% (0 of 2449)

📜 Logs


> performance@1.0.0 run-tests:testenv /home/runner/work/rafiki/rafiki/test/performance
> ./scripts/run-tests.sh -e test "-k" "-q" "--vus" "4" "--duration" "1m"

Cloud Nine GraphQL API is up: http://localhost:3101/graphql
Cloud Nine Wallet Address is up: http://localhost:3100/
Happy Life Bank Address is up: http://localhost:4100/
cloud-nine-wallet-test-backend already set
cloud-nine-wallet-test-auth already set
happy-life-bank-test-backend already set
happy-life-bank-test-auth already set
     data_received..................: 884 kB 15 kB/s
     data_sent......................: 1.9 MB 31 kB/s
     http_req_blocked...............: avg=7.37µs   min=2.42µs   med=5.31µs   max=2.46ms   p(90)=6.43µs   p(95)=6.97µs  
     http_req_connecting............: avg=417ns    min=0s       med=0s       max=414.68µs p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=97.65ms  min=7.31ms   med=77.8ms   max=634.29ms p(90)=173.04ms p(95)=194.64ms
       { expected_response:true }...: avg=97.65ms  min=7.31ms   med=77.8ms   max=634.29ms p(90)=173.04ms p(95)=194.64ms
     http_req_failed................: 0.00%  ✓ 0         ✗ 2449
     http_req_receiving.............: avg=92.97µs  min=26.91µs  med=81.75µs  max=3.17ms   p(90)=119.8µs  p(95)=150.59µs
     http_req_sending...............: avg=35.35µs  min=11.04µs  med=28.35µs  max=1.38ms   p(90)=41.21µs  p(95)=53.29µs 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=97.52ms  min=7.14ms   med=77.68ms  max=634.11ms p(90)=172.88ms p(95)=194.54ms
     http_reqs......................: 2449   40.6944/s
     iteration_duration.............: avg=293.98ms min=173.88ms med=284.32ms max=1.11s    p(90)=350.45ms p(95)=377.14ms
     iterations.....................: 818    13.592495/s
     vus............................: 4      min=4       max=4 
     vus_max........................: 4      min=4       max=4

mkurapov · 2026-03-24T15:51:38Z

+      "@interledger/stream-receiver": "0.3.3-alpha.4",
+      "@interledger/pay": "0.4.0-alpha.10",


Let's put these just in the backend package (unless I'm mistaken and they are used elsewhere too)

mkurapov · 2026-03-24T15:53:43Z

+      destination,
+      quote,
+      appData: outgoingPayment.dataToTransmit
+        ? Buffer.from(outgoingPayment.dataToTransmit, 'utf8')


if you use outgoingPayment.getDataToTransmit() it will be decrypted, such that the reciever won't have to do that. It is already encrypted via STREAM

mkurapov · 2026-03-24T15:53:54Z

      createIncomingErrorHandlerMiddleware(ilpAddress),
      createStreamAddressMiddleware(),
-
+      createPartialPaymentDecisionMiddleware(),


Thoughts on this ^?

mkurapov · 2026-03-24T15:56:03Z

+      message ?? 'Error processing partial payment',
+      'utf8'
+    )
+    ctx.response.reply = replyOrMoney.finalDecline(errorData)


What are the consequences here if we do not throw (like before)?

When setting the reply by calling finalDecline we respond to the sender with a final F99 error (along with the error message coming from the ASE/Rafiki) and closing frame on the reject. Throwing was not correct IMO since purposefully rejecting a payment shouldn't be treated as an unexpected error i.e. handled by the error middleware.

mkurapov · 2026-03-24T15:58:03Z

+    )
+    return false
+  }
  const cacheKey = `${PARTIAL_PAYMENT_DECISION_PREFIX}:${options.id}:${options.partialPaymentId}`


Maybe let's have a separate small function for this so it's consistent in both methods

mkurapov · 2026-03-24T16:00:21Z

    'SEND_TENANT_WEBHOOKS_TO_OPERATOR',
    false
  ),
+  partialPaymentDecisionMaxWaitMs: envInt(


Maybe we should have the decision behaviour (send webhook & poll) false by default (even if there are additional data frames on the packet). What do you think? The ASE could enable it smoothly.

mkurapov · 2026-03-31T13:43:26Z

 | `ENABLE_AUTO_PEERING`                                 | `backend.enable.autoPeering`                             | `false`                                                                     | When `true`, auto-peering is enabled.                                                                                                                                                                                                       |
 | `ENABLE_MANUAL_MIGRATIONS`                            | `backend.enableManualMigrations`                         | `false`                                                                     | When `true`, you must run the database manually with the command `npm run knex – migrate:latest –env production`                                                                                                                            |
 | `ENABLE_SPSP_PAYMENT_POINTERS`                        | `backend.enable.spspPaymentPointers`                     | `true`                                                                      | When `true`, the SPSP route is enabled.                                                                                                                                                                                                     |
+| `DB_ENCRYPTION_SECRET`                                | _undefined_                                              | _undefined_                                                                 | Base64-encoded 32-byte secret used to encrypt/decrypt transmitted payment data (`dataToTransmit`) stored on incoming/outgoing payment records and events.                                                                                   |


We don't need to include it in the v1-beta docs

mkurapov · 2026-03-31T17:00:01Z

+      const { cipherText, tag, iv } = JSON.parse(rawDataToTransmit) as {
+        cipherText: string
+        tag: string
+        iv: string
+      }
+
+      const decipher = createDecipheriv(
+        'aes-256-gcm',
+        Uint8Array.from(Buffer.from(dbEncryptionSecret, 'base64')),
+        iv
+      )
+      decipher.setAuthTag(Uint8Array.from(Buffer.from(tag, 'base64')))
+      decipher.update(cipherText, 'base64', 'utf8')
+      decipher.final('utf8')


Do you think we should decrypt this data before publishing the webhook?

mkurapov · 2026-03-31T17:01:37Z

  walletAddressService: WalletAddressService
  assetService: AssetService
  config: IAppConfig
+  redis?: Redis


What are the consequences of making this required?

None, I made it required 👍

mkurapov · 2026-03-31T17:04:25Z

Do we need to change the specs here? or was it leftover from a merge?

Leftover from the rebase, fixed it in the latest commit

…-exchange

mkurapov

Very minor suggestions

mkurapov · 2026-04-02T19:40:14Z

+          partialIncomingPaymentId: uuid(),
+          expiresAt: prepare.expiresAt
+        }
+      )) as PartialPaymentDecision


Suggested change

)) as PartialPaymentDecision

))

mkurapov · 2026-04-02T19:40:33Z

+    options?: {
+      dataToTransmit?: string
+      partialIncomingPaymentId?: string
+      expiresAt?: Date
+    }
+  ): Promise<PartialPaymentDecision>


Suggested change

options?: {

dataToTransmit?: string

partialIncomingPaymentId?: string

expiresAt?: Date

}

): Promise<PartialPaymentDecision>

options: {

dataToTransmit: string

partialIncomingPaymentId: string

expiresAt: Date

}

): Promise<PartialPaymentDecision>

… frames (#3706) * feat(backend): add middleware that handles STREAM KYC/additional data payloads - wip * feat(backend): add wip comment * feat: support arbitrary data on prepare packets - WIP * feat(pay): add controller to handle additional data in STREAM packets * feat: add STREAM connection id in cache key for KYC response, remove unused env vars * chore: address PR comments * chore: add tests for payment decision middleware and interledgerjs changes * fix: update app data controller ti return F99 * feat: update additional data middleware logic, add pay and stream-receiver updates * chore(backend): use updated pay and stream-receiver from npm, update lockfile * feat(backend): refactor partial payment middleware, add handler for partial payments to mock ASE * fix(backend): fix build errors in CI * fix(backend): formatting * fix(backend): formatting * fix(backend): partial payment decision middleware tests * fix(backend): formatting * feat(backend): address review comments * feat(backend): address PR comments * fix(backend): fix failing CI tests * chore: align open-payments-specifications with feature/encrypted-data-exchange * fix: webhooks tests * fix: delete flaky test * fix: backend build * chore: address review comments

* feat(backend): update sender data in incoming partial payment webhook (#3810) * feat: update sender data in incoming partial payment webhook * feat(backend): update partial payment payload in webhook * feat: uuid to id, use args object * feat(backend): add confirmPartialIncomingPayment resolver (#3819) * feat: update sender data in incoming partial payment webhook * feat(backend): update partial payment payload in webhook * feat: uuid to id, use args object * feat(backend): add confirmPartialIncomingPayment resolver * feat: move return, revert jest env * chore: rename kyc decision prefix * feat(backend): reject partial payment gql api (#3823) * feat(backend): add confirmPartialIncomingPayment resolver * feat: move return, revert jest env * feat(backend): reject partial payment gql api * fix: use correct redis key * fix: use const * feat: move redis call out of resolver * feat(backend): add middleware that handles STREAM KYC/additional data frames (#3706) * feat(backend): add middleware that handles STREAM KYC/additional data payloads - wip * feat(backend): add wip comment * feat: support arbitrary data on prepare packets - WIP * feat(pay): add controller to handle additional data in STREAM packets * feat: add STREAM connection id in cache key for KYC response, remove unused env vars * chore: address PR comments * chore: add tests for payment decision middleware and interledgerjs changes * fix: update app data controller ti return F99 * feat: update additional data middleware logic, add pay and stream-receiver updates * chore(backend): use updated pay and stream-receiver from npm, update lockfile * feat(backend): refactor partial payment middleware, add handler for partial payments to mock ASE * fix(backend): fix build errors in CI * fix(backend): formatting * fix(backend): formatting * fix(backend): partial payment decision middleware tests * fix(backend): formatting * feat(backend): address review comments * feat(backend): address PR comments * fix(backend): fix failing CI tests * chore: align open-payments-specifications with feature/encrypted-data-exchange * fix: webhooks tests * fix: delete flaky test * fix: backend build * chore: address review comments * fix: cherrypick 4917c14 * chore: format * fix: remove duplicate property * fix: backend tests * chore: update lock file * chore: format * chore: update lock file * fix: attempt to fix CI builds * fix: address PR comments * feat: add encrypted data exchange docker compose variant and manual test doc (#3905) * feat: add partial payment rejection description on outgoing_payment.failed webhook * fix: pin testcontainers to 10.16.0 for Tigerbeetle tests * fix: attempt to fix CI backend build * fix: revert testcontainers version changes and update lockfile * chore: format * fix: missing error message in outgoing payment tests * chore: minor updates for encrypted data exchange feature (#3911) * doc: update encrypted data exchange doc * chore: rename dataToTransmit to dataFromSender on recipient side * chore: rename updatePartialPaymentDecision args * test: add test for updatePartialPaymentDecision and for processPartialPayment * chore: make the partial payment decision success case more explicit * test: rename dataToTransmit to dataFromSender * test(backend): updates partial payment decision middleware tests * chore: update pay library version * chore: update pay library to alpha 12 * fix: use dataFromSender instead dataToTransmit when decrypting webhook data * chore: update localenv doc * chore: update RejectPartialIncomingPaymentInput reason description Co-authored-by: Max Kurapov <max@interledger.org> * chore: regenerate graphql types --------- Co-authored-by: Nathan Lie <lie4nathan@gmail.com> Co-authored-by: Blair Currey <12960453+BlairCurrey@users.noreply.github.com> Co-authored-by: Max Kurapov <max@interledger.org>

github-actions Bot added pkg: backend Changes in the backend package. type: source Changes business logic labels Oct 16, 2025

sanducb commented Oct 16, 2025

View reviewed changes

Comment thread packages/backend/src/payment-method/ilp/connector/core/middleware/kyc-decision.ts Outdated

sanducb force-pushed the bs/raf-1185 branch from fa9a81c to 329a6f2 Compare December 9, 2025 14:57

mkurapov reviewed Jan 19, 2026

View reviewed changes

github-actions Bot added the type: tests Testing related label Jan 26, 2026

sanducb requested a review from mkurapov January 26, 2026 10:56

sanducb mentioned this pull request Feb 3, 2026

feat: add support for sending arbitrary data over STREAM interledgerjs/interledgerjs#492

Merged

mkurapov reviewed Feb 9, 2026

View reviewed changes

sanducb added 10 commits March 5, 2026 12:15

feat(backend): add middleware that handles STREAM KYC/additional data…

0619e99

… payloads - wip

feat(backend): add wip comment

bb26b18

feat: support arbitrary data on prepare packets - WIP

d689088

feat(pay): add controller to handle additional data in STREAM packets

a82ef91

feat: add STREAM connection id in cache key for KYC response, remove …

448dfe0

…unused env vars

chore: address PR comments

3bd976b

chore: add tests for payment decision middleware and interledgerjs ch…

704955c

…anges

fix: update app data controller ti return F99

b4aac72

feat: update additional data middleware logic, add pay and stream-rec…

16f9ef1

…eiver updates

chore(backend): use updated pay and stream-receiver from npm, update …

8b77d8d

…lockfile

sanducb force-pushed the bs/raf-1185 branch from 6cced90 to 8b77d8d Compare March 10, 2026 12:42

github-actions Bot added pkg: frontend Changes in the frontend package. pkg: mock-ase pkg: mock-account-service-lib labels Mar 10, 2026

sanducb changed the base branch from main to feature/encrypted-data-exchange March 10, 2026 12:43

feat(backend): refactor partial payment middleware, add handler for p…

42460ea

…artial payments to mock ASE

github-actions Bot removed the pkg: frontend Changes in the frontend package. label Mar 24, 2026

fix(backend): fix build errors in CI

2717155

sanducb added 2 commits March 24, 2026 16:08

fix(backend): formatting

eb0c370

fix(backend): formatting

9dfeaff

sanducb requested a review from mkurapov March 24, 2026 14:19

sanducb marked this pull request as ready for review March 24, 2026 14:20

sanducb added 2 commits March 24, 2026 16:54

fix(backend): partial payment decision middleware tests

4d0c678

fix(backend): formatting

ccf014c

mkurapov reviewed Mar 24, 2026

View reviewed changes

feat(backend): address review comments

773dd8a

github-actions Bot added the pkg: documentation Changes in the documentation package. label Mar 26, 2026

sanducb requested a review from mkurapov March 27, 2026 14:59

mkurapov reviewed Mar 31, 2026

View reviewed changes

sanducb added 6 commits April 2, 2026 00:20

feat(backend): address PR comments

b48d3bf

fix(backend): fix failing CI tests

f5408ba

chore: align open-payments-specifications with feature/encrypted-data…

62e17c4

…-exchange

fix: webhooks tests

7d628ee

fix: delete flaky test

6acc11d

fix: backend build

82006fe

sanducb requested review from mkurapov and njlie April 2, 2026 09:16

mkurapov reviewed Apr 2, 2026

View reviewed changes

chore: address review comments

6dcd12a

sanducb requested a review from mkurapov April 6, 2026 11:04

mkurapov approved these changes Apr 6, 2026

View reviewed changes

sanducb merged commit e8c8822 into feature/encrypted-data-exchange Apr 6, 2026
32 of 52 checks passed

sanducb deleted the bs/raf-1185 branch April 6, 2026 14:41

		"@interledger/stream-receiver": "0.3.3-alpha.4",
		"@interledger/pay": "0.4.0-alpha.10",

Conversation

sanducb commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this pull request

Context

Checklist

Uh oh!

netlify Bot commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Deploy Preview for brilliant-pasca-3e80ec failed. Why did it fail? →

Uh oh!

Uh oh!

mkurapov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkurapov Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkurapov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkurapov left a comment

Choose a reason for hiding this comment

Uh oh!

mkurapov left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Performance Test Results

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanducb Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

sanducb commented Oct 16, 2025 •

edited

Loading

netlify Bot commented Oct 16, 2025 •

edited

Loading

mkurapov Jan 19, 2026 •

edited

Loading

github-actions Bot commented Mar 24, 2026 •

edited

Loading

sanducb Mar 26, 2026 •

edited

Loading