Skip to content

config: support checkout_jobs #9942

Open
@JohnAtl

Description

@JohnAtl

Bug Report

checkout: slow checkouts

Description

Checkout copies all files in parallel, leading to disk saturation, and excessive checkout times. E.g. At this time, lsof for the dvc process shows 331 files open.

Reproduce

dvc pull

Expected

Parallelization in moderation, respecting the jobs: parameter in .dvc/config, or some similar parameter.

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 3.11.1 (pip)
-------------------------
Platform: Python 3.10.10 on Linux-6.1.0-11-amd64-x86_64-with-glibc2.36
Subprojects:
	dvc_data = 2.10.1
	dvc_objects = 0.24.1
	dvc_render = 0.5.3
	dvc_task = 0.3.0
	scmrepo = 1.1.0
Supports:
	http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	ssh (sshfs = 2023.7.0)
Config:
	Global: /home/john/.config/dvc
	System: /etc/xdg/dvc

Additional Information (if any):

https://discuss.dvc.org/t/is-jobs-n-ignored-on-local-stores/1768

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions