Improve Performance on LocalFileSystem.ls() if detail=False

[FrankEssenberger]

Relates to #1788

Only get the infos on a file if they are requested by the user - otherwise return just the path as usual.

[martindurant]

The failures are not your fault (also seen in #1790 ), I'll fix early next week.

[martindurant]

The windows failure I think is genuine - we are getting windows path separators inside an otherwise posix path in the ls() code path.

[FrankEssenberger]

removed the hard coded / in the tests, I hope it is ok now - I see a different error in the pipeline which could be unrelated I guess.

[martindurant]

I think using pathlib is the wrong choice for two reasons:

• it is considerably slower than string manipulation
• you will get windows paths (in the test), but actually we want the real code to return posix paths always, that's the contract fsspec promises. So instead of fixing the test, we need to fix the code.

Hah, yeah, instead of removing the one errant "\", it made all the separators like that:

  -     'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_ls_on_files0\\file_2.json',
  ?        ^^     ^^           ^^       ^^     ^^    ^^                     ^^        ^^                 ^^
  +     'C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_ls_on_files0/file_2.json',

[FrankEssenberger]

ah I was not aware of the we want the real code to return posix paths always paradime - who uses windows anyway :-). No jokes aside there is the make_path_posix this should do the trick if I use it properly.

[martindurant]

Can you do a speed test, please, and then I'll merge this. I ask because make_path_posix isn't free either, but should still be faster than info().

[FrankEssenberger]

Can you do a speed test, please, and then I'll merge this. I ask because make_path_posix isn't free either, but should still be faster than info().

sure. Since the info calls the make_path_posix as well it must be faster. I would assume the path.stat() is the costly thing to do in the info and if I add a very deep structure with a lot of data the aggregation in the test it will take time. What do you think would be a fair setup for such a test? Is there some test creating a representative data layout?

[martindurant]

I meant a quick and dirty test, perhaps on your home directory - no need to go crazy or to add code.

[martindurant]

Running the following locally:

In [1]: import fsspec
In [2]: fs = fsspec.filesystem("file")
In [3]: %timeit fs.ls("", detail=False)

I get 85us on this branch and 171us on master. So 👍 .

[FrankEssenberger]

Thanks for taking care of the review.