Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -56,34 +56,29 @@ export function buildExplicitTypeScriptDependencies(
'.mjs',
'.cjs',
'.cts',
'.vue',
];

// TODO: This can be removed when vue is stable
if (isVuePluginInstalled()) {
moduleExtensions.push('.vue');
}

for (const [project, fileData] of Object.entries(
ctx.filesToProcess.projectFileMap
)) {
Object.keys(ctx.filesToProcess.projectFileMap).forEach((project) => {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this actually yield a performance benefit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I don't have the numbers anymore, but it shaved off 100+ms IIRC.

filesToProcess[project] ??= [];
for (const { file } of fileData) {
if (moduleExtensions.some((ext) => file.endsWith(ext))) {
filesToProcess[project].push(join(workspaceRoot, file));
ctx.filesToProcess.projectFileMap[project].forEach((fileData) => {
if (moduleExtensions.some((ext) => fileData.file.endsWith(ext))) {
filesToProcess[project].push(join(workspaceRoot, fileData.file));
}
}
}
});
});

const { findImports } = require('../../../../native');

// TODO(jon): check why this function is very slow and whether we it can be optimized
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be better for @FrozenPandaz

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All files get processed in parallel... it goes as fast as the largest file which is the slowest to process. IF this is taking long it likely mans that there is a considerably large typescript file. You can see which one it is with NATIVE_FILE_LOGGING tbh, I don't think the TODO comment is necessary.

const imports = findImports(filesToProcess);

for (const {
sourceProject,
file,
staticImportExpressions,
dynamicImportExpressions,
} of imports) {
for (let i = 0; i < imports.length; i++) {
const {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this actually yield any performance benefit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does, however, in this location it's not large impact so if you don't like this old-school syntax we can revert this block.

sourceProject,
file,
staticImportExpressions,
dynamicImportExpressions,
} = imports[i];
const normalizedFilePath = normalizePath(relative(workspaceRoot, file));

for (const importExpr of staticImportExpressions) {
Expand Down Expand Up @@ -131,13 +126,3 @@ export function buildExplicitTypeScriptDependencies(

return res;
}

function isVuePluginInstalled() {
try {
// nx-ignore-next-line
require.resolve('@nx/vue');
return true;
} catch {
return false;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,15 @@ import {
getRootTsConfigFileName,
resolveModuleByImport,
} from '../../utils/typescript';
import { existsSync } from 'node:fs';

/**
* The key is a combination of the package name and the workspace relative directory
* containing the file importing it e.g. `lodash__packages/my-lib`, the value is the
* resolved external node name from the project graph.
*/
type NpmResolutionCache = Map<string, string | null>;
type PackageJsonResolutionCache = Map<string, PackageJson | null>;

type PathPattern = {
pattern: string;
Expand All @@ -44,6 +46,7 @@ type ParsedPatterns = {
* Use a shared cache to avoid repeated npm package resolution work within the TargetProjectLocator.
*/
const defaultNpmResolutionCache: NpmResolutionCache = new Map();
const defaultPackageJsonResolutionCache: PackageJsonResolutionCache = new Map();

const experimentalNodeModules = new Set(['node:sqlite']);

Expand Down Expand Up @@ -71,7 +74,8 @@ export class TargetProjectLocator {
string,
ProjectGraphExternalNode
> = {},
private readonly npmResolutionCache: NpmResolutionCache = defaultNpmResolutionCache
private readonly npmResolutionCache: NpmResolutionCache = defaultNpmResolutionCache,
private readonly packageJsonResolutionCache: PackageJsonResolutionCache = defaultPackageJsonResolutionCache
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not review too carefully.. but just to sanity check me...

This is OK for the daemon because everytime we calculate dependencies, we create a brand new project locator with no cache right? So this cache will not remain between different graph calculations?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, exactly. The cache is reused only during the single CLI command/daemon OP lifetime.

) {
/**
* Only the npm external nodes should be included.
Expand Down Expand Up @@ -211,7 +215,8 @@ export class TargetProjectLocator {
// package.json refers to an external package, we do not match against the version found in there, we instead try and resolve the relevant package how node would
const externalPackageJson = this.readPackageJson(
packageName,
fullDirPath
fullDirPath,
workspaceRoot
);
// The external package.json path might be not be resolvable, e.g. if a reference has been added to a project package.json, but the install command has not been run yet.
if (!externalPackageJson) {
Expand Down Expand Up @@ -533,18 +538,26 @@ export class TargetProjectLocator {
*/
private readPackageJson(
packageName: string,
relativeToDir: string
relativeToDir: string,
workspaceRoot: string
): PackageJson | null {
// The package.json is directly resolvable
const packageJsonPath = resolveRelativeToDir(
join(packageName, 'package.json'),
relativeToDir
);
if (packageJsonPath) {
if (this.packageJsonResolutionCache.has(packageJsonPath)) {
return this.packageJsonResolutionCache.get(packageJsonPath);
}
const parsedPackageJson = readJsonFile(packageJsonPath);

if (parsedPackageJson.name && parsedPackageJson.version) {
this.packageJsonResolutionCache.set(packageJsonPath, parsedPackageJson);
return parsedPackageJson;
} else {
this.packageJsonResolutionCache.set(packageJsonPath, null);
return null;
}
}

Expand All @@ -556,14 +569,29 @@ export class TargetProjectLocator {

while (dir !== dirname(dir)) {
const packageJsonPath = join(dir, 'package.json');
try {
const parsedPackageJson = readJsonFile(packageJsonPath);
// Ensure the package.json contains the "name" and "version" fields
if (parsedPackageJson.name && parsedPackageJson.version) {
return parsedPackageJson;
if (this.packageJsonResolutionCache.has(packageJsonPath)) {
return this.packageJsonResolutionCache.get(packageJsonPath);
}
if (existsSync(packageJsonPath)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should get rid of this existsSync.. it will throw an error if it doesn't exist which is already handled

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more common for the file to not exist (falling through down to the root package.json). The existSync saves time as readFileSync + try/catch is noticeably slower at scale.

try {
const parsedPackageJson = readJsonFile(packageJsonPath);
// Ensure the package.json contains the "name" and "version" fields
if (parsedPackageJson.name && parsedPackageJson.version) {
this.packageJsonResolutionCache.set(
packageJsonPath,
parsedPackageJson
);
return parsedPackageJson;
} else {
this.packageJsonResolutionCache.set(packageJsonPath, null);
return null;
}
} catch {
// Package.json is invalid, keep traversing
}
} catch {
// Package.json doesn't exist, keep traversing
}
if (dir === workspaceRoot) {
return null;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So before.. we were able to.. resolve outside of the workspace root?

Have we considered if this actually helps when the workspaceRoot is not the same as the node_modules root? WHat if someone has like..

repo-root/package.json
repo-root/node_modules
repo-root/nx-workspace-root/nx.json

If this doesn't yield a major perf behefit, I would not make htis change for now

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can test on the customer repo if reverting this change impacts performance.
I was more worried about correctness. Omitting the version is root package.json would cause traversal down to machine root, hoping no other package.json is found it the way. Also, this would be repeated for every single package name lookup.

}
dir = dirname(dir);
}
Expand Down
Loading