Skip to content

Commit 0525426

Browse files
AgentEnderAgentEnder
authored
feat(core): add support for http based caches (#30593)
Implements http based remote caches per the RFC here: #30548
1 parent 494f150 commit 0525426

File tree

13 files changed

+1581
-42
lines changed

13 files changed

+1581
-42
lines changed

‎Cargo.lock

Lines changed: 877 additions & 26 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/shared/recipes/running-tasks/self-hosted-caching.md

Lines changed: 154 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,160 @@ To learn more about migrating from custom task runners, [please refer to this de
4444

4545
## Build Your Own Caching Server
4646

47-
We have [published a new RFC](https://github.com/nrwl/nx/discussions/30548) detailing a custom self-hosted cache based on an OpenAPI specification. This will be available before Nx 21, ensuring a smooth migration path for those who are looking for full control.
47+
Starting in Nx version 20.8, you can build your own caching server using the OpenAPI specification provided below. This allows you to create a custom remote cache server that fits your specific needs. The server is responsible for managing all aspects of the remote cache, including storage, retrieval, and authentication.
48+
49+
Implementation is left up to you, but the server must adhere to the OpenAPI specification provided below to ensure compatibility with Nx's caching mechanism. The endpoints described below involve the transfer of tar archives which are sent as binary data. It is important to note that the underlying format of that data is subject to change in future versions of Nx, but the OpenAPI specification should remain stable.
50+
51+
As long as your server adheres to the OpenAPI spec, you can implement it in any programming language or framework of your choice.
52+
53+
### Open API Specification
54+
55+
```json {% fileName="Nx 20.8+" %}
56+
{
57+
"openapi": "3.0.0",
58+
"info": {
59+
"title": "Nx custom remote cache specification.",
60+
"description": "Nx is a build system, optimized for monorepos, with AI-powered architectural awareness and advanced CI capabilities.",
61+
"version": "1.0.0"
62+
},
63+
"paths": {
64+
"/v1/cache/{hash}": {
65+
"put": {
66+
"description": "Upload a task output",
67+
"operationId": "put",
68+
"security": [
69+
{
70+
"bearerToken": []
71+
}
72+
],
73+
"responses": {
74+
"202": {
75+
"description": "Successfully uploaded the output"
76+
},
77+
"401": {
78+
"description": "Missing or invalid authentication token.",
79+
"content": {
80+
"text/plain": {
81+
"schema": {
82+
"type": "string",
83+
"description": "Error message provided to the Nx CLI user"
84+
}
85+
}
86+
}
87+
},
88+
"403": {
89+
"description": "Access forbidden. (e.g. read-only token used to write)",
90+
"content": {
91+
"text/plain": {
92+
"schema": {
93+
"type": "string",
94+
"description": "Error message provided to the Nx CLI user"
95+
}
96+
}
97+
}
98+
},
99+
"409": {
100+
"description": "Cannot override an existing record"
101+
}
102+
},
103+
"parameters": [
104+
{
105+
"in": "header",
106+
"description": "The file size in bytes",
107+
"required": true,
108+
"schema": {
109+
"type": "number"
110+
},
111+
"name": "Content-Length"
112+
},
113+
{
114+
"name": "hash",
115+
"description": "The task hash corresponding to the uploaded task output",
116+
"in": "path",
117+
"required": true,
118+
"schema": {
119+
"type": "string"
120+
}
121+
}
122+
],
123+
"requestBody": {
124+
"content": {
125+
"application/octet-stream": {
126+
"schema": {
127+
"type": "string",
128+
"format": "binary"
129+
}
130+
}
131+
}
132+
}
133+
},
134+
"get": {
135+
"description": "Download a task output",
136+
"operationId": "get",
137+
"security": [
138+
{
139+
"bearerToken": []
140+
}
141+
],
142+
"responses": {
143+
"200": {
144+
"description": "Successfully retrieved cache artifact",
145+
"content": {
146+
"application/octet-stream": {
147+
"schema": {
148+
"type": "string",
149+
"format": "binary",
150+
"description": "An octet stream with the content."
151+
}
152+
}
153+
}
154+
},
155+
"403": {
156+
"description": "Access forbidden",
157+
"content": {
158+
"text/plain": {
159+
"schema": {
160+
"type": "string",
161+
"description": "Error message provided to the Nx CLI user"
162+
}
163+
}
164+
}
165+
},
166+
"404": {
167+
"description": "The record was not found"
168+
}
169+
},
170+
"parameters": [
171+
{
172+
"name": "hash",
173+
"in": "path",
174+
"required": true,
175+
"schema": {
176+
"type": "string"
177+
}
178+
}
179+
]
180+
}
181+
}
182+
},
183+
"components": {
184+
"securitySchemes": {
185+
"bearerToken": {
186+
"type": "http",
187+
"description": "Auth mechanism",
188+
"scheme": "bearer"
189+
}
190+
}
191+
}
192+
}
193+
```
194+
195+
### Usage Notes
196+
197+
To use your custom caching server, you must set the `NX_SELF_HOSTED_REMOTE_CACHE_SERVER` environment variable. Additionally, the following environment variables also affect the behavior:
198+
199+
- `NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN`: The authentication token to access the cache server.
200+
- `NODE_TLS_REJECT_UNAUTHORIZED`: Set to `0` to disable TLS certificate validation.
48201

49202
## Why Switch to Nx Cloud
50203

‎e2e/nx/src/__fixtures__/remote-cache.js

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
//@ts-check
2+
3+
const http = require('http');
4+
5+
const inMemoryCache = {};
6+
7+
// IMPORTANT: This implementation serves only as a test fixture
8+
// and is not intended for production use. It is a simple in-memory cache server.
9+
// If one was to wish to use something like this in production, the following
10+
// items should be considered:
11+
// - Persistence: Use a database or a file system for storage
12+
// - Security:
13+
// - Implement proper authentication and authorization
14+
// - Ensure existing data is not overwritten without checks
15+
const server = http.createServer((req, res) => {
16+
const url = req.url;
17+
const parts = url?.split('/');
18+
const hash = parts?.[parts.length - 1];
19+
20+
const auth = req.headers.authorization;
21+
if (auth !== 'Bearer test-token') {
22+
res.statusCode = 401;
23+
res.setHeader('Content-Type', 'text/plain');
24+
res.end(
25+
'Unauthorized: Missing or invalid token. Set NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN to proceed.'
26+
);
27+
return;
28+
}
29+
30+
if (req.method === 'GET') {
31+
console.log('Checking for hash:', hash);
32+
console.log('In memory cache:', !!inMemoryCache[hash]);
33+
if (inMemoryCache[hash]) {
34+
res.statusCode = 200;
35+
res.setHeader('Content-Type', 'application/octet-stream');
36+
res.end(inMemoryCache[hash]);
37+
return;
38+
}
39+
console.log('Not found:', hash);
40+
res.statusCode = 404;
41+
res.end('Not found');
42+
return;
43+
}
44+
45+
if (req.method === 'PUT') {
46+
req.on('data', (chunk) => {
47+
if (!inMemoryCache[hash]) {
48+
inMemoryCache[hash] = new ArrayBuffer(); // initialize if not present
49+
}
50+
// Append the chunk to the existing buffer
51+
const newBuffer = Buffer.concat([
52+
Buffer.from(inMemoryCache[hash]),
53+
chunk,
54+
]);
55+
inMemoryCache[hash] = newBuffer;
56+
});
57+
req.on('end', () => {
58+
console.log('Stored in memory cache:', hash);
59+
res.statusCode = 200;
60+
res.end('OK');
61+
});
62+
return;
63+
}
64+
});
65+
66+
const PORT = 3000;
67+
server.listen(PORT, () => {
68+
console.log(`Server running at http://localhost:${PORT}/`);
69+
});

‎e2e/nx/src/cache.test.ts

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,19 @@
11
import {
22
cleanupProject,
33
directoryExists,
4+
fileExists,
45
listFiles,
56
newProject,
67
readFile,
8+
removeFile,
79
rmDist,
810
runCLI,
911
tmpProjPath,
1012
uniq,
1113
updateFile,
1214
updateJson,
1315
} from '@nx/e2e/utils';
16+
import { fork } from 'child_process';
1417

1518
import { readdir, stat } from 'fs/promises';
1619

@@ -424,6 +427,111 @@ describe('cache', () => {
424427
expect(cacheEntriesSize).toBeLessThanOrEqual(500 * 1024);
425428
});
426429

430+
describe('http remote cache', () => {
431+
let cacheServer: any;
432+
beforeAll(() => {
433+
cacheServer = fork(join(__dirname, '__fixtures__', 'remote-cache.js'), {
434+
stdio: 'inherit',
435+
});
436+
});
437+
438+
afterAll(() => {
439+
cacheServer.kill();
440+
});
441+
442+
it('should PUT and GET cache from remote cache', async () => {
443+
const projectName = uniq('myapp');
444+
const outputFilePath = `dist/${projectName}/output.txt`;
445+
updateFile(
446+
`projects/${projectName}/project.json`,
447+
JSON.stringify({
448+
name: projectName,
449+
targets: {
450+
build: {
451+
command: `node -e 'const {mkdirSync, writeFileSync} = require("fs"); mkdirSync("dist/${projectName}", {recursive: true}); writeFileSync("${outputFilePath}", "Hello World")'`,
452+
outputs: ['{workspaceRoot}/dist/{projectName}'],
453+
cache: true,
454+
},
455+
},
456+
})
457+
);
458+
runCLI(`build ${projectName}`, {
459+
env: {
460+
NX_SELF_HOSTED_REMOTE_CACHE_SERVER: 'http://localhost:3000',
461+
NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN: 'test-token',
462+
},
463+
});
464+
// removing the file should not affect the cache retrieval,
465+
// but we can check that the file exists to ensure the cache is
466+
// being used.
467+
removeFile(outputFilePath);
468+
runCLI(`reset`);
469+
const output = runCLI(`build ${projectName}`, {
470+
env: {
471+
NX_SELF_HOSTED_REMOTE_CACHE_SERVER: 'http://localhost:3000',
472+
NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN: 'test-token',
473+
},
474+
});
475+
expectProjectMatchTaskCacheStatus(output, [projectName], 'remote cache');
476+
expect(fileExists(tmpProjPath(outputFilePath))).toBe(true);
477+
});
478+
479+
it('should handle 401 without ACCESS_TOKEN appropriately', async () => {
480+
const projectName = uniq('myapp');
481+
const outputFilePath = `dist/${projectName}/output.txt`;
482+
updateFile(
483+
`projects/${projectName}/project.json`,
484+
JSON.stringify({
485+
name: projectName,
486+
targets: {
487+
build: {
488+
command: `node -e 'const {mkdirSync, writeFileSync} = require("fs"); mkdirSync("dist/${projectName}", {recursive: true}); writeFileSync("${outputFilePath}", "Hello World")'`,
489+
outputs: ['{workspaceRoot}/dist/{projectName}'],
490+
cache: true,
491+
},
492+
},
493+
})
494+
);
495+
const output = runCLI(`build ${projectName}`, {
496+
env: {
497+
NX_SELF_HOSTED_REMOTE_CACHE_SERVER: 'http://localhost:3000',
498+
},
499+
silenceError: true,
500+
});
501+
502+
expect(output).toContain(
503+
'Unauthorized: Missing or invalid token. Set NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN to proceed.'
504+
);
505+
});
506+
507+
it('should error if server is not running', async () => {
508+
const projectName = uniq('myapp');
509+
const outputFilePath = `dist/${projectName}/output.txt`;
510+
updateFile(
511+
`projects/${projectName}/project.json`,
512+
JSON.stringify({
513+
name: projectName,
514+
targets: {
515+
build: {
516+
command: `node -e 'const {mkdirSync, writeFileSync} = require("fs"); mkdirSync("dist/${projectName}", {recursive: true}); writeFileSync("${outputFilePath}", "Hello World")'`,
517+
outputs: ['{workspaceRoot}/dist/{projectName}'],
518+
cache: true,
519+
},
520+
},
521+
})
522+
);
523+
const output = runCLI(`build ${projectName}`, {
524+
env: {
525+
NX_SELF_HOSTED_REMOTE_CACHE_SERVER: 'http://localhost:3001',
526+
NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN: 'test-token',
527+
},
528+
silenceError: true,
529+
});
530+
531+
expect(output).toContain('http://localhost:3001');
532+
});
533+
});
534+
427535
function expectCached(
428536
actualOutput: string,
429537
expectedCachedProjects: string[]

‎packages/nx/Cargo.toml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ colored = "2"
1717
crossbeam-channel = '0.5'
1818
dashmap = { version = "5.5.3", features = ["rayon"] }
1919
dunce = "1"
20+
flate2 = "1.1.1"
2021
fs_extra = "1.3.0"
2122
globset = "0.4.10"
2223
hashbrown = { version = "0.14.5", features = ["rayon", "rkyv"] }
@@ -34,6 +35,7 @@ nom = '7.1.3'
3435
regex = "1.9.1"
3536
rayon = "1.7.0"
3637
rkyv = { version = "0.7", features = ["validation"] }
38+
tar = "0.4.44"
3739
thiserror = "1.0.40"
3840
tracing = "0.1.37"
3941
tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }
@@ -57,6 +59,7 @@ portable-pty = { git = "https://github.com/cammisuli/wezterm", rev = "b538ee29e1
5759
crossterm = "0.27.0"
5860
ignore-files = "2.1.0"
5961
fs4 = "0.12.0"
62+
reqwest = "0.12.15"
6063
rusqlite = { version = "0.32.1", features = ["bundled", "array", "vtab"] }
6164
watchexec = "3.0.1"
6265
watchexec-events = "2.0.1"

0 commit comments

Comments
 (0)