RFC: Additional Project Directories

[FrozenPandaz]

RFC: Additional Project Directories

Problem

Currently, all projects must reside within the workspaceRoot. This is the only directory that Nx walks and it is assumed that all files reside in the workspace root. For some organizations, Nx is only used for a portion of the repository. The repository may also be organized in way where the portion of the workspace where Nx is used is not all within a single directory but multiple directories. This makes it difficult to use Nx within the multiple directories.

Example Layout

In the example layout below, there are multiple apps which eaech have their own nx.json files. All of these apps should be able to use the packages within packages. Having a single nx.json at the root of the workspace is not a viable solution in this case because it would change the intended architecture to have 1 single Nx workspace as opposed to 3 separate workspaces which share the same package. While this isn't ideal for Nx, Nx should be able to support cases like this.

.
└── src/
    ├── apps/
    │   ├── mobile
    │   │   └── nx.json
    │   ├── admin
    │   │   └── nx.json
    │   └── backend
    │   │   └── nx.json
    └── packages/
        ├── javascript
        |    └──package1
        |    |   └──package.json
        ├── java
        ├── go
        └── ruby

Proposed Solutions

Raised Workspace Root disconnected from nx.json

Nx could allow overriding the workspace root via a property in nx.json, this would allow raising the workspace root to a parent directory containing all of the portions where Nx is needed. This may encapsulate too much however including areas not within the areas where Nx would be used. Also, this would lead to things like workspace filesets to be specified oddly as relative from a root different from the nx.json directory. This would be easy to implement but it has many footguns which users would have to be wary of.

Multiple Workspace Roots

Nx could allow for multiple directories to act as workspace roots. Those directories would all be synonymous the workspace root where nx.json resides. Any of the workspace roots could contain projects but also contain workspace inputs. There are many concepts which currently assume that there is a single workspace root and going this direction would involve changing many of those concepts. This would be more confusing and difficult way for Nx to support this.

Additional Project Directories

Nx could still have a single workspace root but could allow for additional project directories to contain project files. When walking for files, Nx would also walk those additional project directories to discover projects. Projects in the main workspace can depend on the additional projects and vice versa.

Workspace inputs must remain within the workspace root. To add inputs from outside of the workspace root, they must belong to a project. Thus, they can be added as project inputs.

Outputs would be able to exist outside of the workspace root.

Generators, in theory could work outside of the workspace root but it should be done in a way where a generator can modify the one tree it is passed. The one tree would be able to make modifications outside of the workspace root. This would be nice to support but would not be a blocker for the rest of the features.

This is the preferred solution and it would be easier to implement, less disruptive, and less confusing for users.

Detailed Implementation

In nx.json, users can specify additionalProjectDirectories where Nx will walk and locate project files. These project files will join the ones in the workspaceRoot so that they are analyzed to create nodes, dependencies, and metadata.

Example Repo

The Nx team will work on preparing an example PR with a PR release version of Nx to show how it would work. With the PR release, those who are looking to experiment with this feature can also validate it within their workspaces.

Requested Feedback

The areas which we are looking for the most feedback is if this concept is easy to understand. Once we have a PR release out, we will also be looking for people to test it to see if the implementation works for your use case.

As always, any feedback is appreciated.

[colinta]

This is relevant to Shopify because we have two projects that are using Nx and those are both being merged into one repository... AND they share a packages folder... AND neither is suitable to be moved to the top-level. So having this support for >1 nx.json files (and searching outside the workspace root for project folders) would allow both projects to continue forward.

Since this is outside the usual use case I can see the argument for keeping things simple, but I will defend this folder structure. One immediate benefit is that this could make it possible to include an existing repo (like an open source project) as if it lives inside the workspace.

I was looking at the nx.json options and nothing jumped out at me as a flag in terms of "it would be bad if this wasn't kept in sync", but that's one concern we have in this setup (there are a few tools that expect one "workspace" file and we have multiple due to how we have organized our folders).