Approaches to executing e2e tests for affected parts only

[parky128]

Currently we have a single app (Angular) consuming a bunch of feature libs which in turn consume component libraries we have written.

When we initially created the app project with Nx we got an e2e app auto generated for us, and in there we have been adding our test suites (Cypress). These suites cover all the various feature libs that the main app depends on.

With nx affected commands, we can of course ensure we only rebuild and unit test parts of the code base affected by changes being introduced.

Rather than having a single e2e app with all of our feature test suites in, is it practical to split these out into separate feature e2e apps with implicit dependencies on the feature libs? That way when we run nx affected with a target of e2e we run the minimum set of test suites needed to help scale out the development workflow. We could still keep a general e2e app for more involved user work flows, e.g. multi feature operations.

Has this kind of thing been asked before? Are there any dangers/pitfalls in implementing this kind of approach you can foresee?

[rfprod]

If you mean smth like below, I think it should work fine. Some overhead with restarting cypress several times for each e2e-test app, but it should pay off with e2e tests granularity. If you manage to run these e2e tests in parallel on different runners without side effects, it can be faster than running a single monolithic e2e application.

In addition to e2e applications, you will have to implement at least one more shared library, e.g. e2e-testing, to keep your code dry, because you will most likely have a number of duplicate commands (such as user authentication) in every e2e application.

{
   ...
   "app": {
      "$schema": "../../node_modules/nx/schemas/project-schema.json",
      "architect": {
        ...
      },
      "prefix": "app",
      "projectType": "application",
      "root": "apps/app",
      "sourceRoot": "apps/app/src",
      "tags": ["scope:app", "type:application"]
    },
    "app-e2e": {
      "$schema": "../../node_modules/nx/schemas/project-schema.json",
      "architect": {
        "e2e": {
          "builder": "@nrwl/cypress:cypress",
          "configurations": {
            "production": {
              "devServerTarget": "app:serve:production"
            }
          },
          "options": {
            "cypressConfig": "apps/app-e2e/cypress.json",
            "devServerTarget": "app:serve",
            "tsConfig": "apps/app-e2e/tsconfig.e2e.json"
          }
        },
        ...
      },
      "implicitDependencies": ["app"],
      "projectType": "application",
      "root": "apps/app-e2e",
      "sourceRoot": "apps/app-e2e/src",
      "tags": ["scope:app-e2e", "type:e2e"]
    },
    ...
    "feature-one": {
      "$schema": "../../node_modules/nx/schemas/project-schema.json",
      "architect": {
        ...
      },
      "prefix": "app",
      "projectType": "library",
      "root": "libs/feature-one",
      "sourceRoot": "libs/feature-one/src",
      "tags": ["scope:feature-one", "type:feature"]
    },
    "feature-one-e2e": {
      "$schema": "../../node_modules/nx/schemas/project-schema.json",
      "architect": {
        "e2e": {
          "builder": "@nrwl/cypress:cypress",
          "configurations": {
            "production": {
              "devServerTarget": "app:serve:production"
            }
          },
          "options": {
            "cypressConfig": "apps/feature-one-e2e/cypress.json",
            "devServerTarget": "app:serve",
            "tsConfig": "apps/feature-one-e2e/tsconfig.e2e.json"
          }
        },
        ...
      },
      "implicitDependencies": ["feature-one"],
      "projectType": "application",
      "root": "apps/feature-one-e2e",
      "sourceRoot": "apps/feature-one-e2e/src",
      "tags": ["scope:feature-one-e2e", "type:e2e"]
   },
   ...
   "feature-two": {
      "$schema": "../../node_modules/nx/schemas/project-schema.json",
      "architect": {
        ...
      },
      "prefix": "app",
      "projectType": "library",
      "root": "libs/feature-two",
      "sourceRoot": "libs/feature-two/src",
      "tags": ["scope:feature-two", "type:feature"]
    },
    "feature-two-e2e": {
      "$schema": "../../node_modules/nx/schemas/project-schema.json",
      "architect": {
        "e2e": {
          "builder": "@nrwl/cypress:cypress",
          "configurations": {
            "production": {
              "devServerTarget": "app:serve:production"
            }
          },
          "options": {
            "cypressConfig": "apps/feature-two-e2e/cypress.json",
            "devServerTarget": "app:serve",
            "tsConfig": "apps/feature-two-e2e/tsconfig.e2e.json"
          }
        },
        ...
      },
      "implicitDependencies": ["feature-two"],
      "projectType": "application",
      "root": "apps/feature-two-e2e",
      "sourceRoot": "apps/feature-two-e2e/src",
      "tags": ["scope:feature-two-e2e", "type:e2e"]
   },
}

[parky128]

Thanks @rfprod for the reply and the advice. This is reassuring to know that my thought's on how to tackle this are sound.

[work933k]

Nowadays it is possible to put feature-specific e2e-tests in your feature-library. You can use the generator for this.

[VladZapodeanuBe]

Hi there,

We're currently using Nx with an Angular(18.2.13) app that consumes multiple feature libraries, each of which depends on shared component libraries. Our E2E tests (written in Cypress 13.17.0) currently live in a single E2E app, covering all feature libraries.

We’d like to optimize our CI pipeline by running only the E2E tests relevant to the affected modules when changes are introduced. Specifically, if a developer modifies a particular feature module, we want to detect that and trigger only the automated E2E tests for that module instead of running the full suite.

Could you provide guidance on:

How to configure our Nx workspace to run Cypress E2E tests for affected modules only?
Whether there are any recommended patterns or best practices for structuring E2E tests per feature module?
Any potential pitfalls or limitations we should be aware of with this approach?
Thanks in advance for your help!

[work933k]

We started with doing everything in E2E-testing of the affected apps. Quite quickly this resulted in the CI-pipeline to take a very long time.
To get around this, we went the following route:

• If possible we do component-testing (Storybook interaction-testing) to test the interaction of components. Without consideration you run into the risk that the components don't actually work in your application. Therefore it's important to only do component-testing of (standalone) functionality which is contained in the component and doesn't depend on outside interaction (thus: no state/store dependency, no parent-component dependency). As we try to follow "smart components" structure, this makes if more clear for us that we test the "dumb components" as this structure enforces to don't have any dependencies. So we can test things like: input-validation (edge case input and such), expand/collapse functionality, etc.
• We have implemented Storybook per project/library. This allows us to test components of affected libraries. We also use Chromatic per library which allows us to also visually check components.
• We try to reduce E2E-tests as much as possible as they don't "scale" that well. The E2E-tests of apps will "always" be affected and thus triggered. We try to do happy-flow testing with E2E.
• We nowadays also have E2E-tests per feature. Just like the component-testing, it's important to be choose correctly what can be tested in E2E-feature-testing. With the E2E-feature-testing we start/serve the app and then test the route of the feature. If the feature should impact outside the route, than this means that it might not be suitable to use this technique.
• With NX it's important to have files be part of the library/project for the affected to not get complex.

e2e-target in the project.json of a library:

 "e2e": {
      "executor": "@nx/cypress:cypress",
      "options": {
        "testingType": "e2e",
        "devServerTarget": "dummy-app:serve"
      },
    },

.eslintrc.json of the library with e2e-target:

{
  "extends": ["plugin:cypress/recommended", "../../../../../.eslintrc.json"],
  "ignorePatterns": ["!**/*"],
  "overrides": [
    {
      "files": ["*.ts"],
      "extends": [
        "plugin:@nx/angular",
        "plugin:@angular-eslint/template/process-inline-templates",
        "plugin:@typescript-eslint/strict"
      ],
      "parserOptions": {
        "project": [
          "libs/dummy/libs/e2e/feature1-e2e/tsconfig.*?.json",
          "libs/dummy/libs/e2e/feature1-e2e/cypress/tsconfig.json"
        ]
      },
  ...

tsconfig.json of the library with the e2e-target:

{
  "extends": "../../../../../tsconfig.base.json",
  "compilerOptions": {
      ...
  },
  "files": [],
  "include": [],
  "references": [
    {
      "path": "./cypress/tsconfig.json"
    }
  ],
  "angularCompilerOptions": {
    ...
  }
}

cypress.config.ts of the library with the e2e-target:

import { customE2EPreset } from '@our-nx/cypress-preset';
import { defineConfig } from 'cypress';

export default defineConfig({
  e2e: customE2EPreset(__dirname, { cypressDir: 'cypress' })
});

And then in the same library you will need the folder cypress containing these files:

• cypress/tsconfig.json
• cypress/e2e/test-feature.cy
• cypress/e2e/support/e2e.cy

The above ensures:

• that a change in the e2e-test-scripts (*.cy.ts) will make the e2e-task be triggered by "affected".
• that a dependency of the library will trigger the e2e-test-script by being "affected".

Hope this helps..

[VladZapodeanuBe]

Hi again,

Thanks for the detailed response! Your approach makes a lot of sense, especially with the combination of Storybook interaction testing, Chromatic, and minimizing full E2E runs.

I have a follow-up question: How do you mark or determine which E2E tests should be executed when a specific module is modified?

Do you use tags in Cypress (e.g., environment variables, spec filtering, or test metadata)?
Do you rely solely on Nx’s implicit dependencies to detect and trigger the relevant tests?
Are you using a custom script or mechanism to further refine which feature E2E tests are executed in CI?
For now, we want to focus on running only the existing E2E tests that we have already designed, without adding new component or Storybook-based tests. Our goal is to ensure that when a PR modifies a specific feature module, only the relevant E2E tests run in CI.

If you have any best practices or examples of how to achieve this efficiently, we'd love to hear them!

Thanks again for your help!

[astanb]

I'm also very interested in this, did you manage to come up with a good solution?

[work933k]

You should always trust and use "NX affected" which the NX-toolset provides. A well-know saying is "don't fight the framework".
NX will find out what tasks need to be triggered. If you only have E2E-tests for your single app, then this will almost always result in your whole E2E-set being run.
The first step for you would be to add an "E2E" task to your library and move the related E2E-test-scripts from your app over to your library. If you would continue on doing this, you should have a small number of E2E tests at the level of you app. The most E2E-tests will be part of libraries.

We don't need to make use of tags for detect and/or triggering of tasks. The NX-framework does provide support for adding "tags" (in the project.json files), but this is (mostly) for ensuring correct workspace architecture. For instance "data-access"-libraries should (probably) not depend on ui/component-libraries. NX has eslint-rules to enforce this: @nx/enforce-module-boundaries.

You can choose to have a subset of E2E-tests to run in the CI-pipeline for each app. And to use the complete set of E2E-tests to run at another time. Although i don't make use of that. One thing which is important to think about that this needs to be selected by a command line parameter, for instance: npx nx affected --target=e2e --configuration=subset-small. This ensures that the correct output will be fetched from NX-cache as the command-line parameters are part of the "cache key".

If you would have i18n translation json files, then ideally each library will have it's own set of translation json files. If you would change a translation, this would make your library affected and trigger the library-specific E2E tests.

Some examples:

• If you make a small change in your app, for instance a build-setting. Then this should make your app "affected" and thus trigger: lint, build and also e2e to do a high-level happy-flow check of functionality.
• If you make a small change in feature1, for instance a translation. Then this should make your app "affected" and thus trigger: lint (fetched from cache), build and also e2e to do a high-level happy-flow check of functionality. But the "feature1"-library will also be "affected" and thus trigger: lint and also e2e to test the feature functionality. In this case 2 seperate E2E-tests-sets will run.