Skip to content

Commit ae21fc1

Browse files
authored
chore: reduce custom API max payload size to 2 MB (#1221)
## Problem Large payloads may cause performance issues on the frontend. ## Solution Reduce payload size to 2 MB, which is a reasonable size for a reasonably large payload from various sources such as data.gov.sg ## Tests - [ ] Custom API calls should still work as per normal (most custom API actions receive less than 2 MB) - [ ] Test with large payload (e.g., `https://api-production.data.gov.sg/v2/public/api/datasets/d_3f960c10fed6145404ca7b821f263b87/list-rows?limit=10000`, should get `Payload too large` error) ## Screenshots <img width="896" height="449" alt="Screenshot 2025-09-19 at 9 45 47 AM" src="https://github.com/user-attachments/assets/6b90b89c-dc99-4b62-a7da-f6d7674c2cd4" />
1 parent e934acd commit ae21fc1

File tree

2 files changed

+8
-4
lines changed

2 files changed

+8
-4
lines changed

packages/backend/src/apps/custom-api/__tests__/common/size-monitor.test.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,9 +50,9 @@ describe('createSizeMonitor', () => {
5050
expect(mocks.warn).not.toHaveBeenCalled()
5151
})
5252

53-
it('errors when total size exceeds 20MB', async () => {
53+
it('errors when total size exceeds 2 MB', async () => {
5454
const monitor = createSizeMonitor()
55-
const overLimit = Buffer.alloc(20 * 1024 * 1024 + 1)
55+
const overLimit = Buffer.alloc(2 * 1024 * 1024 + 1)
5656
await expect(writeBuffers(monitor, [overLimit])).rejects.toMatchObject({
5757
name: 'AxiosError',
5858
isAxiosError: true,

packages/backend/src/apps/custom-api/common/size-monitor.ts

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,13 @@ import { Transform } from 'stream'
22

33
import logger from '@/helpers/logger'
44

5-
const MAX_SIZE_IN_MB = 20
5+
/**
6+
* NOTE: we set the limit to 2 MB to prevent abuse and protect frontend performance
7+
* for large JSON payloads
8+
*/
9+
const MAX_SIZE_IN_MB = 2
610
const MB = 1024 * 1024
7-
const MAX_CONTENT_LENGTH = MAX_SIZE_IN_MB * MB // 20MB
11+
const MAX_CONTENT_LENGTH = MAX_SIZE_IN_MB * MB
812
const MAX_COMPRESSION_RATIO = 100 // Maximum compression ratio to prevent gzip bombs
913

1014
const ERROR_RESPONSE = {

0 commit comments

Comments
 (0)