How to keep mocked API routes consistent with actual API routes in full-stack projects?

[bard]

This is more of a high-level strategy question.

Context: a NextJS project based containing API routes and the frontend relying on them. (For simplicity, let's assume that said routes are called from the client side only, no SSR.)

Scenario:

1. Backend dev Alice makes changes to src/pages/api/items.ts and notifies frontend dev Bob
2. Frontend dev Bob makes equivalent changes to src/mocks/msw-mocks.ts, edits src/pages/items.tsx until tests pass, commits and push
3. CI tests, builds, and deploys successfully
4. In production, someone visits the /items page, only to be greeted by a white page and a stack trace in the console.

What happened?

At step 2), Bob's changes weren't so equivalent after all. Whether due to wrong understanding or wrong implementation, it doesn't matter, what matters is that correctness of the system as a whole ultimately depended on someone eyeballing two pieces of code and deciding whether one was representative of the other. No automated tests verified consistency across the wire (the tests at 2) ran against the mocks).

(Note that this still a relatively optimistic scenario: at 1), Alice might have forgotten to notify Bob, yet tests would still not have broken.)

I'd be interested in hearing your strategies to deal with this fairly common scenario. Ideas that come to mind:

1. Use tooling to generate typed frontend SDKs from backend information (e.g. graphql-codegen, openapi-generator); changes in the backend cause rebuilds of the SDK, and if the change was incompatible, the build breaks due to failed typechecks, alerting Bob. I've done this before; it's certainly useful, but correct types alone aren't always sufficient.
2. Write end-to-end tests with Playwright or similar. Expensive and, I'd argue, misapplied if aiming for good coverage.
3. For each frontend test file, start an actual server exposing real API routes, but backed by an in-memory database, and with calls to 3rd party APIs mocked through msw/node.

I find 3) attractive because you can quickly stub new routes just like you do with msw, but the stubs evolve into actual routes that don't run the risk of getting out of sync with mocks (because there wouldn't be any). On the flip side, running a server for each test file might (haven't tried yet) slow things down, and slow tests are run less often.

Thoughts?

[bard]

Update: I gave 3) a go and it's working nicely so far, and with no difference in speed.

Here's my prototype in case anyone else wants to try:

App.test.tsx:

/**
 * @jest-environment jsdom
 */

import React from "react";
import { App } from "./App";
import { setupTestApi } from "./test-api";
import { createDbConnection as mockCreateDbConnection } from "src/data/db.fixture";

jest.mock("src/data/db", () => ({
  createDbConnection: jest.fn(() => mockCreateDbConnection()),
}));

const testApi = setupTestApi();
let apiBaseUrl: string;
beforeAll(async () => {
  apiBaseUrl = await testApi.start();
});
afterAll(() => testApi.stop());

describe('App', () => {
  // ... test as usual, making sure that requests go apiBaseUrl ...
})

test-api.ts:

import { AddressInfo } from "net";
import express from "express";
import * as http from "http";
import { apiResolver } from "next/dist/server/api-utils/node";
import itemsRoute from "src/pages/api/items";

export const setupTestApi = (): {
  start: () => Promise<string>;
  stop: () => void;
} => {
  const app = express();

  const apiContext = {
    previewModeEncryptionKey: "",
    previewModeId: "",
    previewModeSigningKey: "",
  };

  app.all("/api/items", (req, res) =>
    apiResolver(req, res, req.query, itemsRoute, apiContext, false)
  );

  app.use((req, _res, next) => {
    console.warn("Unhandled request " + req.url);
    next();
  });

  let server: http.Server | null = null;

  return {
    start: async () => {
      server = await new Promise<http.Server>((resolve) => {
        const server = app.listen(0, () => resolve(server));
      });

      return (
        "http://localhost:" + (server.address() as AddressInfo).port + "/api"
      );
    },
    stop: () => {
      if (!server) throw new Error("Server not started");
      server.close();
    },
  };
};

jest.setup.js:

// Required to make TextEncoder accessible in jsdom environment, in turn
// required to run test server
import { TextEncoder, TextDecoder } from "util";
global.TextEncoder = TextEncoder;
global.TextDecoder = TextDecoder;

There's still a risk that tests and actual API get out of sync when routes are added or renamed, but that's far easier to catch than mismatches in implementation.

Also, I don't have routes of the form src/pages/api/items/[id].ts, and those might require extra steps.

[kettanaito]

Hey, @bard. That's a great question!

In any strategy you chose, I'd be careful in trusting backend as the source of truth. I'd rather encourage to have a strictly typed contract between backend and frontend, like OpenAPI spec or generated types from a GraphQL schema, just as you've mentioned. The reason for that is that implementations may be faulty, and taking backend runtime as a source of truth opens up for a brand new list of problems. The worst part is that reasoning about a fixed state of the system (frontend in this case) becomes extremely hard.

This is where I like explicit mocks because they depict API state at a fixed point of time. But I understand that in larger projects it may be unsustainable to maintain manually written mocks that will inevitably get out of sync from whichever runtime your application is actually hitting in production.

If I had to tackle this issue, I'd go for the static spec route as my first choice. If, for some reason, I can't, I'd do the following:

1. Run my integration/E2E tests against the actual backend.
2. Record the traffic I get into an HAR file.
3. Generate request handlers for MSW from that HAR file in my tests.
4. Automate the process into a shell script or a CI job, if necessary.

This would make that network snapshot a source of truth fixed in time, which is always what I want.

I've been working on an in-house solution to generate handlers out of HAR files, and I dare say it's in a good shape right now but it's not open sourced and, frankly, I don't know if it ever be. Given my recent burnout, I don't see how throwing a new tool into the furnace would make my life any more manageable. Perhaps this solution will surface as a paid library in time. I honestly don't know.

So, for now, you can use https://github.com/Tapico/tapico-msw-webarchive to generate handlers from your HAR files.

[kettanaito]

You can also design your own network recorder by using MSW! Since it can intercept all requests via rest.all('*', resolver), you can catch everything that's going on in your tests and record it in some shape or form. You wouldn't be able to produce a fully compliant HAR file since MSW doesn't expose certain network data like request timings for it to be complete. But the bright side is that you don't really have to go for a full HAR.

[bard]

Thanks for chiming in, @kettanaito !

In any strategy you chose, I'd be careful in trusting backend as the source of truth. I'd rather encourage to have a strictly typed contract between backend and frontend, like OpenAPI spec or generated types from a GraphQL schema, just as you've mentioned. The reason for that is that implementations may be faulty, and taking backend runtime as a source of truth opens up for a brand new list of problems. The worst part is that reasoning about a fixed state of the system (frontend in this case) becomes extremely hard.

However, we wouldn't be taking the backend runtime at the source of truth, rather we'd be taking the backend's source (which sits in the same repo as the frontend's) as the source of truth. And that's the ultimate source of truth, no?

To expand on why I'm not keen on SDKs as a way to address this, imagine you have:

1. POST { "type": "widget" } /api/items => 201 Location: /api/items?id=123
2. GET /api/items?id=123 => 200 { "type": "widget" "}

Implement that in mocks; tests run fine; implement it in the route handler, but forget setting the location header; production breaks. The tests running against the mocks in this case were a form of false positive. A typed SDK wouldn't prevent that.

This is where I like explicit mocks because they depict API state at a fixed point of time. But I understand that in larger projects it may be unsustainable to maintain manually written mocks that will inevitably get out of sync from whichever runtime your application is actually hitting in production.

Well, I started feeling the pain at four API routes, but given it's a one-person side-project, that probably still counts as "larger project". :)

I understand the need to test against a stable target, but in a NextJS project, or in any monorepo where frontend and backend sit in the same repository, API state is expressed by the commit hash.

I can see, however, that when the backend is a separate project, developed in a separate repository, API mocks provide that stability, and HAR dumps can be a way to track API evolution in a controlled way.