Bottlenecks caused by sequential processing of RPC queries

[jinoosss]

Description

RPC simulate and query requests are processed sequentially, causing severe performance degradation in concurrent request environments.

Problem Situation

1. Sequential RPC Request Processing

• While the first request is being processed, the remaining 49 requests accumulate in the waiting queue
• Each request takes an average of 1 second to process, accumulating wait time for subsequent requests
• The 50th request starts processing after waiting 49+ seconds

2. Timeout Cascade Effect

• Subsequent requests timeout due to accumulated wait time
• Requests exceeding the timeout setting (typically 10 seconds) fail
• 80% failure rate is confirmed to be mostly due to timeouts

3. CPU Resource Exhaustion

• CPU usage increases as computational load accumulates
• CPU intensive operations affect other RPC requests (block queries, transaction broadcast, etc.)

---

Test Steps

Test Environment

• Branch: gnoswap-labs/test/rpc-performance-7337ab5e72 (https://github.com/gnoswap-labs/gno/tree/test/rpc-performance-7337ab5e72)
• Commit: 7337ab5e72

Test Contract

• Package: gno.land/r/.../tnft
• Functions Tested:
  • BulkMint(cur, to, bulkSize)
  • BulkGenerateTokenURI(bulkSize)

Test Script

cd test-scripts && ./run_batch_query_bulk_generate_image_100.sh
cd test-scripts && ./run_batch_simulate_bulk_mint_100.sh

// Computationally intensive function calls
BulkMint(cur, to, 100)           // Simulate minting 100 NFTs
BulkGenerateTokenURI(100)        // Query to generate 100 TokenURIs

The problem occurs when multiple clients simultaneously call simulate or query on computationally intensive functions as shown above.

Test Results (50 concurrent requests)

Response Summary:
  - Successful Requests (200): 11
  - Failed Requests: 39
  - Total Requests: 50
  - Success Rate: 22.00%

Response Time Statistics:
  - Min Response Time: 1.071566s
  - Max Response Time: 49.636059s
  - Average Response Time: 25.367s

Performance Metrics:
  - Requests per Second: 1.00
  - Total Test Duration: 49.769917000s

[jinoosss]

What are your thoughts?
cc @zivkovicmilos @thehowl @piux2

[thehowl]

IIRC, there is basically a global lock on the blockchain state, because transactionality doesn't really work well.

So yeah, I'm aware but also think trying to solve it is a major rewrite of the tm2 codebase.

[omarsy]

Just for reference I tried to do a fix #3746 for this