api.rst

API Reference

.. currentmodule:: fsspec

User Functions

.. autosummary::
   fsspec.available_compressions
   fsspec.available_protocols
   fsspec.filesystem
   fsspec.fuse.run
   fsspec.generic.rsync
   fsspec.get_filesystem_class
   fsspec.get_mapper
   fsspec.gui.FileSelector
   fsspec.open
   fsspec.open_files
   fsspec.open_local

.. autofunction:: fsspec.available_compressions

.. autofunction:: fsspec.available_protocols

.. autofunction:: fsspec.filesystem

.. autofunction:: fsspec.fuse.run

.. autofunction:: fsspec.generic.rsync

.. autofunction:: fsspec.get_filesystem_class

.. autofunction:: fsspec.get_mapper

.. autoclass:: fsspec.gui.FileSelector
   :members:

.. autofunction:: fsspec.open

.. autofunction:: fsspec.open_files

.. autofunction:: fsspec.open_local

Base Classes

.. autosummary::
   fsspec.archive.AbstractArchiveFileSystem
   fsspec.asyn.AsyncFileSystem
   fsspec.callbacks.Callback
   fsspec.callbacks.DotPrinterCallback
   fsspec.callbacks.NoOpCallback
   fsspec.callbacks.TqdmCallback
   fsspec.core.BaseCache
   fsspec.core.OpenFile
   fsspec.core.OpenFiles
   fsspec.core.get_fs_token_paths
   fsspec.core.url_to_fs
   fsspec.dircache.DirCache
   fsspec.FSMap
   fsspec.generic.GenericFileSystem
   fsspec.registry.register_implementation
   fsspec.spec.AbstractBufferedFile
   fsspec.spec.AbstractFileSystem
   fsspec.spec.Transaction

.. autoclass:: fsspec.archive.AbstractArchiveFileSystem
   :members:

.. autoclass:: fsspec.callbacks.Callback
   :members:

.. autoclass:: fsspec.callbacks.DotPrinterCallback
   :members:

.. autoclass:: fsspec.callbacks.NoOpCallback
   :members:

.. autoclass:: fsspec.callbacks.TqdmCallback
   :members:

.. autoclass:: fsspec.core.BaseCache
   :members:

.. autoclass:: fsspec.core.OpenFile
   :members:

.. autoclass:: fsspec.core.OpenFiles

.. autofunction:: fsspec.core.get_fs_token_paths

.. autofunction:: fsspec.core.url_to_fs

.. autoclass:: fsspec.dircache.DirCache
   :members: __init__

.. autoclass:: fsspec.FSMap
   :members:

.. autoclass:: fsspec.generic.GenericFileSystem

.. autofunction:: fsspec.registry.register_implementation

.. autoclass:: fsspec.spec.AbstractBufferedFile
   :members:

.. autoclass:: fsspec.spec.AbstractFileSystem
   :members:

.. autoclass:: fsspec.spec.Transaction
   :members:

Built-in Implementations

.. autosummary::
   fsspec.implementations.arrow.ArrowFSWrapper
   fsspec.implementations.arrow.HadoopFileSystem
   fsspec.implementations.cached.CachingFileSystem
   fsspec.implementations.cached.SimpleCacheFileSystem
   fsspec.implementations.cached.WholeFileCacheFileSystem
   fsspec.implementations.dask.DaskWorkerFileSystem
   fsspec.implementations.data.DataFileSystem
   fsspec.implementations.dbfs.DatabricksFileSystem
   fsspec.implementations.dirfs.DirFileSystem
   fsspec.implementations.ftp.FTPFileSystem
   fsspec.implementations.gist.GistFileSystem
   fsspec.implementations.git.GitFileSystem
   fsspec.implementations.github.GithubFileSystem
   fsspec.implementations.http.HTTPFileSystem
   fsspec.implementations.jupyter.JupyterFileSystem
   fsspec.implementations.libarchive.LibArchiveFileSystem
   fsspec.implementations.local.LocalFileSystem
   fsspec.implementations.memory.MemoryFileSystem
   fsspec.implementations.reference.ReferenceFileSystem
   fsspec.implementations.reference.LazyReferenceMapper
   fsspec.implementations.sftp.SFTPFileSystem
   fsspec.implementations.smb.SMBFileSystem
   fsspec.implementations.tar.TarFileSystem
   fsspec.implementations.webhdfs.WebHDFS
   fsspec.implementations.zip.ZipFileSystem

.. autoclass:: fsspec.implementations.arrow.ArrowFSWrapper
   :members: __init__

.. autoclass:: fsspec.implementations.arrow.HadoopFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.cached.CachingFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.cached.SimpleCacheFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.cached.WholeFileCacheFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.dask.DaskWorkerFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.data.DataFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.dbfs.DatabricksFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.dirfs.DirFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.ftp.FTPFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.gist.GistFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.git.GitFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.github.GithubFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.http.HTTPFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.jupyter.JupyterFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.libarchive.LibArchiveFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.local.LocalFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.memory.MemoryFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.reference.ReferenceFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.reference.LazyReferenceMapper
   :members: __init__

.. autoclass:: fsspec.implementations.sftp.SFTPFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.smb.SMBFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.tar.TarFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.webhdfs.WebHDFS
   :members: __init__

.. autoclass:: fsspec.implementations.zip.ZipFileSystem
   :members: __init__

Other Known Implementations

Note that most of these projects are hosted outside of the fsspec organisation. Please read their documentation carefully before using any particular package.

• abfs for Azure Blob service, with protocol "abfs://"
• adl for Azure DataLake storage, with protocol "adl://"
• alluxiofs to access fsspec implemented filesystem with Alluxio distributed cache
• boxfs for access to Box file storage, with protocol "box://"
• csvbase for access to csvbase.com hosted CSV files, with protocol "csvbase://"
• dropbox for access to dropbox shares, with protocol "dropbox://"
• dvc to access DVC/Git repository as a filesystem
• fsspec-encrypted for transparent encryption on top of other fsspec filesystems.
• gcsfs for Google Cloud Storage, with protocol "gcs://"
• gdrive to access Google Drive and shares (experimental)
• git to access Git repositories
• huggingface_hub to access the Hugging Face Hub filesystem, with protocol "hf://"
• hdfs-native to access Hadoop filesystem, with protocol "hdfs://"
• httpfs-sync to access HTTP(s) files in a synchronous manner to offer an alternative to the aiohttp-based implementation.
• ipfsspec for the InterPlanetary File System (IPFS), with protocol "ipfs://"
• irods for access to iRODS servers, with protocol "irods://"
• lakefs for lakeFS data lakes, with protocol "lakefs://"
• morefs for OverlayFileSystem, DictFileSystem, and others
• ocifs for access to Oracle Cloud Object Storage, with protocol "oci://"
• ocilake for OCI Data Lake storage
• ossfs for Alibaba Cloud (Aliyun) Object Storage System (OSS)
• p9fs for 9P (Plan 9 Filesystem Protocol) servers
• PyAthena for S3 access to Amazon Athena, with protocol "s3://" or "s3a://"
• PyDrive2 for Google Drive access
• s3fs for Amazon S3 and other compatible stores, with protocol "s3://"
• sshfs for access to SSH servers, with protocol "ssh://" or "sftp://"
• swiftspec for OpenStack SWIFT, with protocol "swift://"
• tosfs for ByteDance volcano engine Tinder Object Storage (TOS)
• wandbfs to access Wandb run data (experimental)
• wandbfsspec to access Weights & Biases (experimental)
• webdav4 for WebDAV, with protocol "webdav://" or "dav://"
• xrootd for xrootd, with protocol "root://"

Read Buffering

.. autosummary::
   fsspec.caching.BlockCache
   fsspec.caching.BytesCache
   fsspec.caching.MMapCache
   fsspec.caching.ReadAheadCache
   fsspec.caching.FirstChunkCache
   fsspec.caching.BackgroundBlockCache

.. autoclass:: fsspec.caching.BlockCache
   :members:

.. autoclass:: fsspec.caching.BytesCache
   :members:

.. autoclass:: fsspec.caching.MMapCache
   :members:

.. autoclass:: fsspec.caching.ReadAheadCache
   :members:

.. autoclass:: fsspec.caching.FirstChunkCache
   :members:

.. autoclass:: fsspec.caching.BackgroundBlockCache
   :members:

Utilities

.. autosummary::

   fsspec.utils.read_block

.. autofunction:: fsspec.utils.read_block