directory download and uploads are slow

[iesahin]

Bug Report

• This affects multiple commands, get/fetch/pull/push... so I didn't put a tag.

Description

We have added a directory containing 70,000 small images to the Dataset Registry. There is also a tar.gz version of the dataset which is downloaded quickly:

time dvc get https://github.com/iterative/dataset-registry mnist/images.tar.gz
dvc get https://github.com/iterative/dataset-registry mnist/images.tar.gz  3.41s user 1.36s system 45% cpu 10.411 total

When I issue:

dvc get https://github.com/iterative/dataset-registry mnist/images

I get ~16 hours ETA for 70.000 downloads in my VPS.

This is reduced to ~3 hours on my faster local machine.

I didn't wait to finish these, so the real times may be different but you get the idea.

For -j 10 it doesn't differ much:

dvc pull is better, it's takes about 20-25 minutes.

(At this point, while writing a new version released and the rest of the report is in 2.4.1 😄 )

dvc pull -j 100 seems to reduce the ETA to 10 minutes.

(I waited for dvc pull -j 100 to finish and it took ~15 minutes.)

I also had this issue while uploading the data in iterative/dataset-registry#18 and we have a discussion there.

Reproduce

git clone https://github.com/iterative/dataset-registry
cd dataset-registry
dvc pull mnist/images.dvc

or

dvc get https://github.com/iterative/dataset-registry mnist/images

Expected

We will use this dataset (and fashion-mnist similar to this) in example repositories, we would like to have some acceptable time (<2 minutes) for the whole directory to download.

Environment information

Output of dvc doctor:

Some of this report is with 2.3.0 but currently:

$ dvc doctor
DVC version: 2.4.1 (pip)
---------------------------------
Platform: Python 3.8.5 on Linux-5.4.0-74-generic-x86_64-with-glibc2.29
Supports: azure, gdrive, gs, hdfs, webhdfs, http, https, s3, ssh, oss

Discussion

DVC uses new requests.Session objects in connection and this requires new HTTP(S) connection for each file. Although the files are small, establishing a new connection for each file takes time.

There is a mechanism in HTTP/1.1 to use the same connection. but requests doesn't support it..

Note that increasing the number of jobs doesn't make much difference, because servers usually limit the number of connections per IP. Even if you have 100 threads/processes to download, it's probably a small number (~4-8) of these can be connected at a time. (I'm banned from AWS once while testing the commands with large -j.)

There may be 2 solutions for this:

• DVC can consider directories as implicit tar archives. Instead of a directory containing many files, it works with a single tar file per directory in the cache and expands them in checkout. tar and gzip are supported in Python standard library. This probably requires all Repo class to be updated though.

• Instead of requests, DVC can use a custom solution or another library like dugong that supports HTTP pipelining. I didn't test any HTTP pipelining solution in Python, so I can't vouch for any of them but this may be better for all asynchronous operations using HTTP(S).

[pmrowla]

DVC push/pull performance for many small files is a known limitation. But a part of the issue is probably also specific to the HTTP remote type, it's possible you would get better performance pulling from the registry using S3 rather than HTTP (due to remote fs implementation differences)

For the purpose of the example projects, it may be better to just handle compressing/extracting images as an archive within the example pipelines.

---

• DVC can consider directories as implicit tar archives. Instead of a directory containing many files, it works with a single tar file per directory in the cache and expands them in checkout.

This does not really work as a general solution for DVC, as we would lose the ability to de-duplicate files between directory versions.

[pmrowla]

This is also an issue that can be addressed by future cache structure changes (#829). Ideally with those kinds of cache changes we will be pushing and pulling a small # of large packed objects rather than a large # of small loose objects (so the end result is similar to the difference you see now between downloading a single tar vs 70k individual images)

[iesahin]

Thank you @pmrowla

But a part of the issue is probably also specific to the HTTP remote type, it's possible you would get better performance pulling from the registry using S3 rather than HTTP (due to remote fs implementation differences)

Sure. I researched a bit but couldn't find a definite answer: Does boto3 reuse HTTP connections? In my tests, dvc pull with S3 URL is faster but not that faster.

This does not really work as a general solution for DVC, as we would lose the ability to de-duplicate files between directory versions.

Having an option to track immutable data directories as single tar files might have some benefit. I agree though this benefit may not cover the required architecture changes.

This multiple small objects in a directory issue needs to be resolved for #829 as well. Otherwise the cache itself will be blobs of large objects and that will probably bring some other problems.

[shcheklein]

@pmrowla what about @iesahin 's suggestions :

• use more thread by default - why don't we do this?
• check if we indeed recreate connections (we don't use connections pool?)
• HTTP pipelining - not that familiar with this one

WDYT?

---

My experience with S3 and small objects and DVC. It seems it can be fast (s5cmd) even on small objects, boto3 seems to be quite suboptimal.

Not saying that we should not be migrating to the new cache structure, but it feels that we could significantly improve it even before that. And that migration might take a long time.

---

For the purpose of the example projects, it may be better to just handle compressing/extracting images as an archive within the example pipelines.

My take on this - by doing this, we pretty much in the get started saying that we don't handle even small datasets. Compressing all images looks quite artificial and people will be asking why are we doing this. Also it complicates the document.

[pmrowla]

use more thread by default - why don't we do this?

When we used the old "# of cores x2" method (without a hard cap on the default) for setting the number of threads, it caused problems for users running on clusters that have very high #'s of cores.

check if we indeed recreate connections (we don't use connections pool?)

We only use our own connection pooling for specific remote types (HDFS and SSH), since pyarrow and paramiko don't provide their own.

DVC uses new requests.Session objects in connection and this requires new HTTP(S) connection for each file. Although the files are small, establishing a new connection for each file takes time.

DVC does not create new session objects for each file being downloaded.

dvc/dvc/fs/http.py

Line 72 in 4e792ae

def _session(self):

_session is a cached property, and is only created once per filesystem instance

HTTP pipelining - not that familiar with this one

Regarding HTTP pipelining, do the S3 webservers even support it properly? HTTP/1.1 pipelining isn't supported in most modern browsers due to the general lack of adoption and/or incorrect implementations in servers.

This is very old: https://stackoverflow.com/questions/7752802/does-s3-support-http-pipelining but I can't find anything newer, which seems to indicate that the answer is probably still "no"

---

In general, my understanding is that w/the move to using async instead of python thread pools (and aiobotocore) we should be able to start getting throughputs that are much faster than native aws-cli/boto, @isidentical should have some more insight on that

[isidentical]

In general, my understanding is that w/the move to using async instead of python thread pools (and aiobotocore) we should be able to start getting throughputs that are much faster than native aws-cli/boto, @isidentical should have some more insight on that

For the other async operations, yes that is true. But the problem with a lot of small files vs a single big file is that, the relative significance of the processing stage (e.g parsing the API response for each object) to the actual data transport is much higher on small files compared to a single big file, and those steps are blocking for async. One possible (and kind of tricky) route would be (which I believe there are still some places in DVC and fsspec itself that are not compatible) creating a process-pool and attaching each process it's own event loop, which would scale these blocking operations to the number of cores and when used with async should maximize the throughput. (btw we still use regular boto for the push/pull).

[skshetry]

Regarding pipelining, as @pmrowla said, it's not very well supported, and users will run into issues with proxy as well.

buggy proxies, marginal improvements, HOL blocking ...

It's considered a design mistake and has been superseded by multiplexing in http/2. Though we won't find much benefits with this unless we go async + http/2, which we should (HTTPFileSystem uses aiohttp which does not support http/2).

[iesahin]

Is there any data regarding the adoption rate of HTTP/2 vs HTTP/1.1 pipelining?

[pmrowla]

S3 does not directly support HTTP/2 unless you are serving it behind CloudFront (or your own proxy server).

---

Also, just to be clear, HTTP/1.1 pipelining is not the same thing as re-using HTTP connections via the Keep-alive Header. requests.Session instances (and DVC) already use connection pools and Keep-Alive for connections in those pools by default.

[dberenbaum]

Are we planning to migrate http to fsspec and/or optimize it?

It's considered a design mistake and has been superseded by multiplexing in http/2. Though we won't find much benefits with this unless we go async + http/2, which we should (HTTPFileSystem uses aiohttp which does not support http/2).

Do you mean fsspec HTTPFileSystem? I don't see it in the current dvc implementation unless I'm missing something.

It seems like we should be able to fairly easily migrate to fsspec/aiohttp at this point? Can we see if it's an improvement before worrying about pipelines or multiplexing yet?

[iesahin]

I can test HTTP pipelining with S3 with a script, if you'd like. If I can come up with faster download with a single thread, we can talk about implementing it in the core.

[skshetry]

Are we planning to migrate http to fsspec and/or optimize it?

I think moving to fsspec's HTTPFileSystem is the plan. Right now, I think it's not possible to upload and there are some performance concerns which we could contribute to.

It seems like we should be able to fairly easily migrate to fsspec/aiohttp at this point? Can we see if it's an improvement before worrying about pipelines or multiplexing yet?

I don't think it'll make much difference unless we go async.