Bun.file(path).json() is missing a max length check for UTF-16 strings

[capaj]

What version of Bun is running?

1.2.11-canary.53+83a2a6496

What platform is your computer?

Linux 6.8.0-58-generic x86_64 x86_6

What steps can reproduce the bug?

have a 4GB json file and run

const file = await Bun.file('4GB.json').json()

What is the expected behavior?

Either the file gets opened or

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (4240424690) is greater than 2 GiB
    at FSReqCallback.readFileAfterStat [as oncomplete] (node:fs:322:11) {
  code: 'ERR_FS_FILE_TOO_LARGE'
}

should get thrown like in node.js

What do you see instead?

it throws:

[1]    3602275 IOT instruction (core dumped)  bun monster/parse.ts

Additional information

I know I can use streaming, but I have a huge invalid json, so parsing is problematic too. I have plenty of memory(128 GB) so loading up 4 GB JSON would ideally work

[Jarred-Sumner]

sad, we used to have a test for this but it made the CI machine run out of disk space

[capaj]

AI slop warning:

// large_json_test.ts
import { test, expect } from "bun:test";
import fs from "node:fs"; // For sync deletion on success

test("Generate, open, delete ~3GB JSON", async () => {
  // --- Configuration ---
  const filePath = `/tmp/bun_test_3gb_${Date.now()}.json`; // Use /tmp or OS temp dir
  const targetSize = 3 * 1024 * 1024 * 1024; // 3 GB
  const chunkSize = 1 * 1024 * 1024; // 1MB write chunk size
  const characterCode = 'a'.charCodeAt(0); // Byte value for 'a'
  // Pre-allocate buffer chunk for repeated writing (minimal memory trick)
  const chunkBuffer = new Uint8Array(chunkSize).fill(characterCode);

  // --- File Generation ---
  let writer;
  let fileExistsBeforeCleanup = false; // Track if file was created for finally block
  console.log(`Attempting to generate ~3GB JSON: ${filePath}`);
  const startTime = performance.now();

  try {
    writer = Bun.file(filePath).writer();

    // Write JSON structure start: {"data":"
    const prefix = '{"data":"';
    await writer.write(prefix);
    let writtenBytes = prefix.length;

    // Write the large data payload in chunks
    while (writtenBytes < targetSize - 2) { // Leave space for trailing '"}
      const remaining = targetSize - 2 - writtenBytes;
      const bytesToWrite = Math.min(chunkSize, remaining);

      if (bytesToWrite === chunkSize) {
        // Write the full pre-allocated chunk (fastest)
        await writer.write(chunkBuffer);
        writtenBytes += chunkSize;
      } else if (bytesToWrite > 0) {
        // Write the remaining partial chunk
        await writer.write(chunkBuffer.slice(0, bytesToWrite));
        writtenBytes += bytesToWrite;
      } else {
        break; // Safety break
      }
    }

    // Write JSON structure end: "}
    const suffix = '"}';
    await writer.write(suffix);
    writtenBytes += suffix.length;

    // Flush and close the writer
    await writer.end();
    writer = null; // Prevent finally block from trying to close again
    fileExistsBeforeCleanup = true; // Mark file as successfully created

    const endTime = performance.now();
    console.log(`File generation took: ${(endTime - startTime).toFixed(2)} ms`);
    console.log(`Target size: ${targetSize} bytes`);
    console.log(`Approx written: ${writtenBytes} bytes (${(writtenBytes / (1024**3)).toFixed(2)} GB)`);


    // --- Verification ("tries to open it") ---
    console.log("Verifying file existence and size...");
    const file = Bun.file(filePath);
    const exists = await file.exists();
    expect(exists).toBe(true); // Check it was actually created

    const actualSize = file.size;
    console.log(`Actual file size: ${actualSize} bytes (${(actualSize / (1024**3)).toFixed(2)} GB)`);
    // Check if the size is reasonably close (within ~5% as chunking isn't exact)
    expect(actualSize).toBeGreaterThan(targetSize * 0.95);
    expect(actualSize).toBeLessThan(targetSize * 1.05 + prefix.length + suffix.length);

    // Minimal check passed - file exists and has expected size range

    // --- Cleanup (on pass) ---
    console.log("Test passed. Deleting file...");
    fs.unlinkSync(filePath); // Delete the file synchronously since test passed
    fileExistsBeforeCleanup = false; // Mark as deleted
    console.log(`File deleted: ${filePath}`);

  } catch (error) {
    console.error("Test failed during execution:", error);
    // Re-throw the error so Bun marks the test as failed
    throw error;
  } finally {
    // Safety cleanup: Ensure writer is closed if error occurred mid-write
    if (writer && !writer.closed) {
      try {
        await writer.end();
        console.warn("Writer closed in finally block due to earlier error.");
      } catch (closeError) {
        console.error("Error closing writer in finally block:", closeError);
      }
    }
    // If the file exists *here*, it means the test failed *before* the cleanup step
    if (fileExistsBeforeCleanup) {
      console.warn(`Test failed. Leaving file for inspection: ${filePath}`);
      // Optional: uncomment to delete even on failure
      // try { fs.unlinkSync(filePath); console.log("Cleaned up file on failure."); } catch(e) {}
    }
  }
}, {
  timeout: 300_000 // 300 seconds (5 minutes) timeout - essential for large I/O
});

wouldn't something like this work?

[190n]

We now can use FUSE in CI, so we could set up a virtual filesystem that appears to have a huge JSON file but doesn't occupy any space. It could only be tested on Linux though :/

wouldn't something like this work?

That looks to write a big JSON file to disk, so how would it prevent CI from running out of disk space?

[capaj]

The default GitHub CI runners have 25-29 GB of disk space so seems like allocating 3GB mid test and deleting it right after should work ok without running out of disk space.