Skip to content

Performance: Parallelize recursive directory copying #2409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ivan-ottinger wants to merge 4 commits into
base: trunk
Choose a base branch
from
Open

Performance: Parallelize recursive directory copying #2409

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
479c9f1
Parallelize recursive directory copying with Promise.all
ivan-ottinger Jan 16, 2026
b52bc8f
Limit concurrent file operations
ivan-ottinger Jan 20, 2026
a528e7f
Replace custom recursive copy implementation with fs-extra
ivan-ottinger Jan 21, 2026
d13094b
Remove recursiveCopyDirectory wrapper in favor of direct fs-extra usage
ivan-ottinger Jan 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 3 additions & 8 deletions cli/commands/site/create.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,7 @@ import {
validateBlueprintData,
} from 'common/lib/blueprint-validation';
import { getDomainNameValidationError } from 'common/lib/domains';
import {
arePathsEqual,
isEmptyDir,
isWordPressDirectory,
pathExists,
recursiveCopyDirectory,
} from 'common/lib/fs-utils';
import { arePathsEqual, isEmptyDir, isWordPressDirectory, pathExists } from 'common/lib/fs-utils';
import { DEFAULT_LOCALE } from 'common/lib/locale';
import { isOnline } from 'common/lib/network-utils';
import { createPassword } from 'common/lib/passwords';
Expand All @@ -33,6 +27,7 @@ import {
isWordPressVersionAtLeast,
} from 'common/lib/wordpress-version-utils';
import { SiteCommandLoggerAction as LoggerAction } from 'common/logger-actions';
import fse from 'fs-extra';
import {
lockAppdata,
readAppdata,
Expand Down Expand Up @@ -161,7 +156,7 @@ export async function runCommand(
}

logger.reportStart( LoggerAction.SETUP_WORDPRESS, __( 'Copying bundled WordPress…' ) );
await recursiveCopyDirectory( bundledWPPath, sitePath );
await fse.copy( bundledWPPath, sitePath );
logger.reportSuccess( __( 'WordPress files copied' ) );
} else if ( ! isOnlineStatus ) {
throw new LoggerError(
Expand Down
13 changes: 5 additions & 8 deletions cli/commands/site/tests/create.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,7 @@ import {
filterUnsupportedBlueprintFeatures,
validateBlueprintData,
} from 'common/lib/blueprint-validation';
import {
isEmptyDir,
isWordPressDirectory,
pathExists,
arePathsEqual,
recursiveCopyDirectory,
} from 'common/lib/fs-utils';
import { isEmptyDir, isWordPressDirectory, pathExists, arePathsEqual } from 'common/lib/fs-utils';
import { isOnline } from 'common/lib/network-utils';
import { portFinder } from 'common/lib/port-finder';
import {
Expand Down Expand Up @@ -40,6 +34,10 @@ jest.mock( 'common/lib/port-finder', () => ( {
jest.mock( 'common/lib/passwords', () => ( {
createPassword: jest.fn().mockReturnValue( 'generated-password-123' ),
} ) );
jest.mock( 'fs-extra', () => ( {
...jest.requireActual( 'fs-extra' ),
copy: jest.fn().mockResolvedValue( undefined ),
} ) );
jest.mock( 'common/lib/blueprint-validation' );
jest.mock( 'cli/lib/appdata', () => ( {
...jest.requireActual( 'cli/lib/appdata' ),
Expand Down Expand Up @@ -132,7 +130,6 @@ describe( 'CLI: studio site create', () => {
( isEmptyDir as jest.Mock ).mockResolvedValue( true );
( isWordPressDirectory as jest.Mock ).mockReturnValue( false );
( arePathsEqual as jest.Mock ).mockImplementation( ( a, b ) => a === b );
( recursiveCopyDirectory as jest.Mock ).mockResolvedValue( undefined );
( portFinder.getOpenPort as jest.Mock ).mockResolvedValue( mockPort );
( readAppdata as jest.Mock ).mockResolvedValue( {
sites: [ ...mockAppdata.sites ],
Expand Down
20 changes: 0 additions & 20 deletions common/lib/fs-utils.ts
View file
Open in desktop
Copy link
Contributor

@fredrikekelund fredrikekelund Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to suggest we do something like my suggestion below, but apparently, there are several issues with this solution (see bottom):

const recursiveCopyLimiter = pLimit( 100 );

export async function recursiveCopyDirectory(
	source: string,
	destination: string
): Promise< void > {
	await fsPromises.mkdir( destination, { recursive: true } );
	const entries = await fsPromises.readdir( source, { withFileTypes: true } );

	await recursiveCopyLimiter.map( entries, async ( entry ) => {
		const sourcePath = path.join( source, entry.name );
		const destinationPath = path.join( destination, entry.name );

		if ( entry.isDirectory() ) {
			await recursiveCopyDirectory( sourcePath, destinationPath );
		} else if ( entry.isFile() ) {
			await fsPromises.copyFile( sourcePath, destinationPath );
		}
	} );
}

The solution to these problems, as I see it, is to leave the code as is and add p-limit@^3.1.0 to our package.json file.

Original file line number Diff line number Diff line change
Expand Up @@ -77,26 +77,6 @@ export async function pathExists( path: string ): Promise< boolean > {
}
}

export async function recursiveCopyDirectory(
source: string,
destination: string
): Promise< void > {
await fsPromises.mkdir( destination, { recursive: true } );

const entries = await fsPromises.readdir( source, { withFileTypes: true } );

for ( const entry of entries ) {
const sourcePath = path.join( source, entry.name );
const destinationPath = path.join( destination, entry.name );

if ( entry.isDirectory() ) {
await recursiveCopyDirectory( sourcePath, destinationPath );
} else if ( entry.isFile() ) {
await fsPromises.copyFile( sourcePath, destinationPath );
}
}
}

export async function isEmptyDir( directory: string ): Promise< boolean > {
const stats = await fsPromises.stat( directory );
if ( ! stats.isDirectory() ) {
Expand Down
5 changes: 3 additions & 2 deletions src/lib/wordpress-setup.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
*/

import nodePath from 'path';
import { recursiveCopyDirectory, pathExists } from 'common/lib/fs-utils';
import fs from 'fs-extra';
import { pathExists } from 'common/lib/fs-utils';
import { getResourcesPath } from 'src/storage/paths';

/**
Expand All @@ -18,5 +19,5 @@ export async function setupWordPressFilesOnly( path: string ): Promise< void > {
throw new Error( 'Bundled WordPress files not found. Please reinstall WordPress Studio.' );
}

await recursiveCopyDirectory( bundledWPPath, path );
await fs.copy( bundledWPPath, path );
}
6 changes: 1 addition & 5 deletions src/lib/wp-versions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import path from 'path';
import fs from 'fs-extra';
import semver from 'semver';
import { recursiveCopyDirectory } from 'common/lib/fs-utils';
import { downloadWordPress } from 'src/lib/download-utils';
import { getWordPressVersionPath } from 'src/lib/server-files-paths';

Expand Down Expand Up @@ -68,10 +67,7 @@ export async function updateLatestWordPressVersion() {
const latestVersion = await getLatestWordPressVersion();
if ( installedVersion && latestVersion !== 'latest' && installedVersion !== latestVersion ) {
// We keep a copy of the latest installed version instead of removing it.
await recursiveCopyDirectory(
latestVersionPath,
getWordPressVersionPath( installedVersion )
);
await fs.copy( latestVersionPath, getWordPressVersionPath( installedVersion ) );
shouldOverwrite = true;
}
}
Expand Down
5 changes: 3 additions & 2 deletions src/migrations/migrate-from-wp-now-folder.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { app } from 'electron';
import path from 'path';
import { pathExists, recursiveCopyDirectory } from 'common/lib/fs-utils';
import fs from 'fs-extra';
import { pathExists } from 'common/lib/fs-utils';
import { getServerFilesPath } from 'src/storage/paths';
import { loadUserData } from 'src/storage/user-data';

Expand All @@ -26,5 +27,5 @@ export async function needsToMigrateFromWpNowFolder() {
}

export async function migrateFromWpNowFolder() {
await recursiveCopyDirectory( wpNowPath, getServerFilesPath() );
await fs.copy( wpNowPath, getServerFilesPath() );
}
7 changes: 3 additions & 4 deletions src/setup-wp-server-files.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import path from 'path';
import fs from 'fs-extra';
import semver from 'semver';
import { recursiveCopyDirectory } from 'common/lib/fs-utils';
import { updateLatestWPCliVersion } from 'src/lib/download-utils';
import { getWordPressVersionPath, getSqlitePath, getWpCliPath } from 'src/lib/server-files-paths';
import {
Expand Down Expand Up @@ -42,7 +41,7 @@ export async function copyBundledLatestWPVersion() {
} );
}
console.log( `Copying bundled WP version ${ bundledWPVersion } as 'latest' version…` );
await recursiveCopyDirectory( bundledWPVersionPath, latestWPVersionPath );
await fs.copy( bundledWPVersionPath, latestWPVersionPath );
}
}

Expand All @@ -69,7 +68,7 @@ async function copyBundledSqlite() {
installedSqliteVersion && semver.gt( bundledSqliteVersion, installedSqliteVersion );
if ( ! isSqliteInstalled || isBundledVersionNewer ) {
console.log( `Copying bundled SQLite version ${ bundledSqliteVersion }…` );
await recursiveCopyDirectory( bundledSqlitePath, getSqlitePath() );
await fs.copy( bundledSqlitePath, getSqlitePath() );
}
}

Expand Down Expand Up @@ -97,7 +96,7 @@ async function copyBundledSQLiteCommand() {
semver.gt( bundledSqliteCommandVersion, installedSqliteCommandVersion );
if ( ! isSqliteCommandInstalled || isBundledVersionNewer ) {
console.log( `Copying bundled SQLite command version ${ bundledSqliteCommandVersion }…` );
await recursiveCopyDirectory( bundledSqliteCommandPath, installedSqliteCommandPath );
await fs.copy( bundledSqliteCommandPath, installedSqliteCommandPath );
}
}

Expand Down