[Feature] Enable to filter the Node.js/Python/Java/R environments to prepare

[tschaffter]

What projects is this feature for?

No response

Description

Currently the tools that maintain the Node.js/Python/Java/R environments are applied to all the projects. However, doing so may be a waste of time and resources (mainly storage space) for projects that a developer is not contributing too.

For example, developers who work on Schematic shouldn't ideally have to prepare the OC projects. On the other hand, schematic-api comes with many dependencies that take about two minutes to install (see logs below).

There are different strategies we may explore to save resources:

Approach 1

The first method is to rely on Nx to run the preparation tasks for the affected projects. One edge case is that when cloning the repo, there are no affected projects and so nothing could be built without preparing some project first.

A potential solution may be to initialize the workspace by preparing all the projects (potentially in combination with solution 2), and from then only prepare again the projects that are affected.

Another solution is to make "build", "test", "lint", etc. tasks depends on the preparation tasks. This approach would not be worth it if running the preparation task on project that have already been prepare takes time.

Approach 2

Here the idea would be to allow the developers to define the list of products they want to work with. E.g. "openchallenges", "schematic", "synapse", etc.

Approach 3

This approach consist in evaluating whether running the preparation task on a project that has already been prepared and has not changed is fast or if it takes time. If preparing the environment again is super fast, then developers will not waste time. However, storage space would still be wasted.

Notes

• There are files in the workspace that Nx will consider as affecting all the projects if they are changed. Examples include package.json and some configuration files. When that happens, all the project will have their environments prepared if we rely on the list of affected projects.
• We may be able to tell Nx that among the projects it consider as affected, ignore some.

cc: @andrewelamb @linglp @milen-sage

Anything else?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[tschaffter]

Storage Usage

Type of EC2 instance used: t3.2xlarge (see docs)

Starting a fresh dev container + removing node_modules:

vscode@108c376e44d7:/workspaces/sage-monorepo$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          80G   38G   43G  48% /
tmpfs            64M     0   64M   0% /dev
tmpfs            16G     0   16G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/nvme0n1p1   80G   38G   43G  48% /vscode
none             16G  160K   16G   1% /tmp

Preparing all the environments. This operation needs to be done only once!

$ time workspace-install
...
➤ YN0000: Done with warnings in 1m 7s
 
    ✔  nx run openchallenges-keycloak:prepare (390ms)
    ✔  nx run openchallenges-service-registry:prepare (407ms)
    ✔  nx run openchallenges-api-gateway:prepare (433ms)
    ✔  nx run openchallenges-mariadb:prepare (385ms)
    ✔  nx run openchallenges-challenge-to-elasticsearch-service:prepare (355ms)
    ✔  nx run schematic-api:prepare (397ms)
    ✔  nx run openchallenges-kaggle-to-kafka-service:prepare (309ms)
    ✔  nx run openchallenges-organization-service:prepare (340ms)
    ✔  nx run openchallenges-challenge-service:prepare (374ms)
    ✔  nx run openchallenges-mysqld-exporter:prepare (366ms)
    ✔  nx run openchallenges-image-service:prepare (374ms)
    ✔  nx run openchallenges-config-server:prepare (373ms)
    ✔  nx run openchallenges-elasticsearch:prepare (382ms)
    ✔  nx run openchallenges-auth-service:prepare (427ms)
    ✔  nx run openchallenges-core-service:prepare (390ms)
    ✔  nx run openchallenges-user-service:prepare (403ms)
    ✔  nx run openchallenges-api-docs:prepare (409ms)
    ✔  nx run openchallenges-prometheus:prepare (432ms)
    ✔  nx run openchallenges-postgres:prepare (386ms)
    ✔  nx run openchallenges-notebook:prepare (438ms)
    ✔  nx run openchallenges-grafana:prepare (464ms)
    ✔  nx run openchallenges-rstudio:prepare (434ms)
    ✔  nx run openchallenges-thumbor:prepare (364ms)
    ✔  nx run openchallenges-zipkin:prepare (399ms)
    ✔  nx run openchallenges-mongo:prepare (362ms)
    ✔  nx run openchallenges-vault:prepare (358ms)
    ✔  nx run openchallenges-minio:prepare (369ms)
    ✔  nx run openchallenges-kafka:prepare (391ms)
    ✔  nx run openchallenges-apex:prepare (372ms)
    ✔  nx run schematic-api-docs:prepare (416ms)
    ✔  nx run openchallenges-app:prepare (425ms)
    ✔  nx run schematic-notebook:prepare (444ms)
    ✔  nx run synapse-rstudio:prepare (455ms)

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target prepare for 33 projects (5s)
 
   View logs and investigate cache misses at https://cloud.nx.app/runs/MSTcuhdHQX

 
    ✔  nx run openchallenges-app-config-data:prepare-java (4s)
    ✔  nx run shared-java-util:prepare-java (820ms)
    ✔  nx run openchallenges-kafka-admin:prepare-java (954ms)
    ✔  nx run openchallenges-kafka-model:prepare-java (1s)
    ✔  nx run openchallenges-service-registry:prepare-java (1s)
    ✔  nx run openchallenges-api-gateway:prepare-java (1s)
    ✔  nx run openchallenges-kafka-consumer:prepare-java (920ms)
    ✔  nx run openchallenges-kafka-producer:prepare-java (1s)
    ✔  nx run openchallenges-challenge-to-elasticsearch-service:prepare-java (1s)
    ✔  nx run openchallenges-kaggle-to-kafka-service:prepare-java (934ms)
    ✔  nx run openchallenges-organization-service:prepare-java (978ms)
    ✔  nx run openchallenges-challenge-service:prepare-java (941ms)
    ✔  nx run openchallenges-config-server:prepare-java (1s)
    ✔  nx run openchallenges-image-service:prepare-java (997ms)
    ✔  nx run openchallenges-auth-service:prepare-java (796ms)
    ✔  nx run openchallenges-user-service:prepare-java (863ms)
    ✔  nx run openchallenges-core-service:prepare-java (850ms)

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target prepare-java for 17 projects (20s)
 
   View logs and investigate cache misses at https://cloud.nx.app/runs/5j7bZtTDjC

 
    ✔  nx run schematic-api-client-python:prepare-python (6s)
    ✔  nx run openchallenges-notebook:prepare-python (11s)
    ✔  nx run schematic-notebook:prepare-python (10s)
    ✔  nx run schematic-api:prepare-python (2m)

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target prepare-python for 4 projects (2m)
 
   View logs and investigate cache misses at https://cloud.nx.app/runs/IH7LFwfyVx

 
    ✔  nx run openchallenges-api-client-r:prepare-r (3m)

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target prepare-r for project openchallenges-api-client-r (3m)
 
   View logs and investigate cache misses at https://cloud.nx.app/runs/9uaJqVf5pk


real    6m18.141s
user    10m19.576s
sys     2m38.068s

Running the same command again. Nothing has changed but it still takes some time to the project to verify that they don't need to install anything. The time varies depending on the programming language used.

➤ YN0000: └ Completed in 0s 976ms
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed in 1s 810ms
➤ YN0000: ┌ Link step
➤ YN0000: └ Completed in 1s 495ms
➤ YN0000: Done with warnings in 5s 181ms
 
    ✔  nx run openchallenges-keycloak:prepare (353ms)
    ✔  nx run openchallenges-api-gateway:prepare (383ms)
    ✔  nx run openchallenges-service-registry:prepare (457ms)
    ✔  nx run openchallenges-mariadb:prepare (343ms)
    ✔  nx run openchallenges-challenge-to-elasticsearch-service:prepare (348ms)
    ✔  nx run schematic-api:prepare (435ms)
    ✔  nx run openchallenges-kaggle-to-kafka-service:prepare (289ms)
    ✔  nx run openchallenges-challenge-service:prepare (316ms)
    ✔  nx run openchallenges-organization-service:prepare (433ms)
    ✔  nx run openchallenges-mysqld-exporter:prepare (448ms)
    ✔  nx run openchallenges-elasticsearch:prepare (419ms)
    ✔  nx run openchallenges-image-service:prepare (373ms)
    ✔  nx run openchallenges-config-server:prepare (342ms)
    ✔  nx run openchallenges-core-service:prepare (305ms)
    ✔  nx run openchallenges-user-service:prepare (329ms)
    ✔  nx run openchallenges-auth-service:prepare (323ms)
    ✔  nx run openchallenges-notebook:prepare (318ms)
    ✔  nx run openchallenges-prometheus:prepare (412ms)
    ✔  nx run openchallenges-api-docs:prepare (364ms)
    ✔  nx run openchallenges-postgres:prepare (392ms)
    ✔  nx run openchallenges-thumbor:prepare (395ms)
    ✔  nx run openchallenges-grafana:prepare (371ms)
    ✔  nx run openchallenges-zipkin:prepare (345ms)
    ✔  nx run openchallenges-rstudio:prepare (391ms)
    ✔  nx run openchallenges-minio:prepare (347ms)
    ✔  nx run openchallenges-vault:prepare (361ms)
    ✔  nx run openchallenges-mongo:prepare (396ms)
    ✔  nx run openchallenges-kafka:prepare (340ms)
    ✔  nx run openchallenges-apex:prepare (409ms)
    ✔  nx run openchallenges-app:prepare (358ms)
    ✔  nx run schematic-api-docs:prepare (383ms)
    ✔  nx run synapse-rstudio:prepare (328ms)
    ✔  nx run schematic-notebook:prepare (364ms)

 ———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target prepare for 33 projects (4s)
 
   View logs and investigate cache misses at https://cloud.nx.app/runs/aTFiBqQoRW

 
    ✔  nx run openchallenges-app-config-data:prepare-java (792ms)
    ✔  nx run shared-java-util:prepare-java (939ms)
    ✔  nx run openchallenges-kafka-model:prepare-java (854ms)
    ✔  nx run openchallenges-kafka-admin:prepare-java (831ms)
    ✔  nx run openchallenges-service-registry:prepare-java (912ms)
    ✔  nx run openchallenges-api-gateway:prepare-java (815ms)
    ✔  nx run openchallenges-kafka-consumer:prepare-java (934ms)
    ✔  nx run openchallenges-kafka-producer:prepare-java (877ms)
    ✔  nx run openchallenges-challenge-to-elasticsearch-service:prepare-java (740ms)
    ✔  nx run openchallenges-kaggle-to-kafka-service:prepare-java (960ms)
    ✔  nx run openchallenges-organization-service:prepare-java (1s)
    ✔  nx run openchallenges-challenge-service:prepare-java (1s)
    ✔  nx run openchallenges-image-service:prepare-java (1s)
    ✔  nx run openchallenges-config-server:prepare-java (1s)
    ✔  nx run openchallenges-auth-service:prepare-java (890ms)
    ✔  nx run openchallenges-core-service:prepare-java (941ms)
    ✔  nx run openchallenges-user-service:prepare-java (847ms)

 ———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target prepare-java for 17 projects (16s)
 
   View logs and investigate cache misses at https://cloud.nx.app/runs/Ul872VD3pc

 
    ✔  nx run schematic-api-client-python:prepare-python (5s)
    ✔  nx run openchallenges-notebook:prepare-python (9s)
    ✔  nx run schematic-api:prepare-python (11s)
    ✔  nx run schematic-notebook:prepare-python (9s)

 ———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target prepare-python for 4 projects (14s)
 
   View logs and investigate cache misses at https://cloud.nx.app/runs/3A3qdoyABt

 
    ✔  nx run openchallenges-api-client-r:prepare-r (3s)

 ———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target prepare-r for project openchallenges-api-client-r (3s)
 
   View logs and investigate cache misses at https://cloud.nx.app/runs/292emRi0Gm


real    0m46.165s
user    1m15.760s
sys     0m9.433s

Storage after preparing all the projects

vscode@108c376e44d7:/workspaces/sage-monorepo$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          80G   42G   39G  52% /
tmpfs            64M     0   64M   0% /dev
tmpfs            16G     0   16G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/nvme0n1p1   80G   42G   39G  52% /vscode
none             16G  2.2M   16G   1% /tmp

Conclusions

TODO: Check how much time installing Node.js dependencies take

Here are the projects that take time to install:

• Installing the dependencies of openchallenges-api-client-r takes 3 minutes
• Installing the dependencies schematic-api takes 2 minutes

The total size of the dependencies for the existing projects is 4 GB.

Preparing 17 Java projects takes 20 seconds, which is really fast. Note that Java projects must be prepared sequentially because of a concurrency issue.

[tschaffter]

Building all the projects

$ nx run-many --target=build
 
    ✔  nx run schematic-api-description:build (2s)
    ✔  nx run openchallenges-api-description:build-individuals (3s)
    ✔  nx run openchallenges-api-description:build (990ms)
    ✔  nx run openchallenges-api-client-angular-deprecated:build:production (6s)
    ✔  nx run openchallenges-api-client-angular:build:production (6s)
    ✔  nx run openchallenges-app-config-data:build (35s)
    ✔  nx run openchallenges-app-config-data:install (5s)
    ✔  nx run shared-java-util:build (40s)
    ✔  nx run openchallenges-kafka-model:build (39s)
    ✔  nx run shared-java-util:install (4s)
    ✔  nx run openchallenges-kafka-model:install (5s)
    ✔  nx run schematic-api-client-angular:build:production (5s)
    ✔  nx run openchallenges-kafka-admin:build (14s)
    ✔  nx run openchallenges-kafka-admin:install (15s)
    ✔  nx run openchallenges-service-registry:build (26s)
    ✔  nx run openchallenges-api-gateway:build (26s)
    ✔  nx run synapse-api-description:build (2s)
    ✔  nx run openchallenges-kafka-consumer:build (12s)
    ✔  nx run openchallenges-db-cli:build (3s)
    ✔  nx run openchallenges-kafka-producer:build (6s)
    ✔  nx run openchallenges-kafka-consumer:install (6s)
    ✔  nx run schematic-api-docs:build (2s)
    ✔  nx run openchallenges-kafka-producer:install (3s)
    ✔  nx run openchallenges-api-docs:build (2s)
    ✔  nx run openchallenges-infra:build (10s)
    ✔  nx run openchallenges-config-server:build (5s)
    ✔  nx run openchallenges-app:build:production (1m)
    ✔  nx run openchallenges-organization-service:build (1m)
    ✔  nx run openchallenges-challenge-service:build (34s)
    ✔  nx run openchallenges-api-client-r:build (2m)
    ✔  nx run openchallenges-auth-service:build (7s)
    ✔  nx run openchallenges-user-service:build (11s)
    ✔  nx run openchallenges-image-service:build (23s)
    ✔  nx run openchallenges-challenge-to-elasticsearch-service:build (16s)
    ✔  nx run openchallenges-core-service:build (33s)
    ✔  nx run openchallenges-kaggle-to-kafka-service:build (5s)
    ✔  nx run schematic-app:build:production (31s)

 ———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target build for 30 projects and 7 tasks they depend on (4m)