feat: add remote storage loaders and writers (S3, GCS, Azure Blob, SQL)

[CoronRing]

Summary

• Adds S3Loader / GCSLoader / AzureBlobLoader / SQLLoader
• All providers use optional extras (railtracks[aws], railtracks[gcp], railtracks[azure-blob], railtracks[sql]) so the core package stays lean
• All loaders expose sync and async interfaces (load/aload)
• SQL classes include a context-manager (with SQLLoader(...) as l) and explicit close() for engine lifecycle management
• SQL identifier arguments validated against a strict allowlist at construction time to prevent injection
• Full unit test coverage across all classes
• Comprehensive developer docs
• Writer are not included in the PR, it will be in a separate module

Security hardening

• SQL table/column names validated at __init__ time — raises ValueError on any metacharacter ([A-Za-z_][A-Za-z0-9_$]* allowlist, supports schema.table)
• Helpful ValueError when content_column is missing from query results (was a bare KeyError)
• __repr__ on all classes exposes only non-sensitive fields (bucket/container name); credentials never appear in repr
• All ImportError messages include both pip install and uv add forms

Limitations documented

• CTE (WITH …) queries not supported as table_or_query; workaround shown in docs
• aload are thread-backed (asyncio.to_thread) not true-async; noted in docs with guidance for high-concurrency cases

Test plan

• 127 unit tests passing across all 4 providers × loader + writer
• SQL tests use real in-memory SQLite (no mocks for correctness)
• Cloud tests (S3/GCS/Azure) use provider SDK mocks
• Async variants covered for all classes

Additional file

Tutorial file are added to ipynb file on colab, not included per requirements for single-location consistency.

[Pooria90]

Good work @CoronRing

I went over the code. A few substantial changes needs to be applied:

1. We need to move loaders and writers modules to packages/railtracks/src/railtracks/retrieval/loaders as they are being built on top of Amir's work on loaders. I recommend having all classes for a specific provider under one single module: for example having loaders/s3.py and writers/s3.py all under one s3.py.
   Or we can also have them under a loaders/cloud folder. We can discuss the structure.
2. The modules are returning Chunk which is located in our old vector_stores module. That module is deprecated now and will be removed from the framework. The new type that we use for our retrieval module is called Document which is located in packages/railtracks/src/railtracks/retrieval/models.py. Please refer to Amir's loaders for examples.
3. Please adjust the Base classes, data models, and the docs accordingly.

[Copilot]

Pull request overview

This PR extends Railtracks’ retrieval subsystem with new cloud/database document loaders (S3, GCS, Azure Blob, SQL), improves the sync load() wrapper to work inside an active event loop, and adds documentation/navigation for the new storage integrations.

Changes:

• Added new cloud/database loaders under railtracks.retrieval.loaders.cloud (S3/GCS/Azure Blob/SQL) and re-exported them from railtracks.retrieval.loaders.
• Updated BaseDocumentLoader.load() to work when called from within a running event loop (e.g., Jupyter/async contexts).
• Added storage integration docs + tutorial pages and updated MkDocs navigation; added unit tests for the new loaders.

Reviewed changes

Copilot reviewed 30 out of 32 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pyproject.toml	Reformats root deps list; narrows uv workspace members to packages/railtracks.
packages/railtracks/tests/unit_tests/retrieval/loaders/test_text_loader.py	Adds a regression test for recursive directory loading.
packages/railtracks/tests/unit_tests/retrieval/loaders/test_base.py	Adds async-context test to ensure load() works with a running event loop.
packages/railtracks/tests/unit_tests/retrieval/loaders/cloud/test_sql.py	Adds SQLLoader tests using in-memory SQLite.
packages/railtracks/tests/unit_tests/retrieval/loaders/cloud/test_s3.py	Adds S3Loader tests (mocked boto3 client).
packages/railtracks/tests/unit_tests/retrieval/loaders/cloud/test_gcs.py	Adds GCSLoader tests (mocked google storage client).
packages/railtracks/tests/unit_tests/retrieval/loaders/cloud/test_azure_blob.py	Adds AzureBlobLoader tests (mocked Azure SDK).
packages/railtracks/tests/unit_tests/retrieval/loaders/cloud/conftest.py	Shared cloud-loader test helpers/mocks + SQLite engine helper.
packages/railtracks/tests/unit_tests/retrieval/loaders/cloud/init.py	Declares the cloud loader test package.
packages/railtracks/tests/unit_tests/integrations/writers/conftest.py	Adds writer-test fixtures/mocks (though writer tests/impls aren’t present here).
packages/railtracks/tests/unit_tests/integrations/writers/init.py	Declares the writers test package.
packages/railtracks/src/railtracks/retrieval/models.py	Adds richer Document documentation describing fields/semantics.
packages/railtracks/src/railtracks/retrieval/loaders/cloud/sql.py	Implements SQLLoader (SQLAlchemy-backed, thread-backed async).
packages/railtracks/src/railtracks/retrieval/loaders/cloud/s3.py	Implements S3Loader (boto3-backed, thread-backed async).
packages/railtracks/src/railtracks/retrieval/loaders/cloud/gcs.py	Implements GCSLoader (google-cloud-storage-backed, thread-backed async).
packages/railtracks/src/railtracks/retrieval/loaders/cloud/azure_blob.py	Implements AzureBlobLoader (azure-storage-blob-backed, thread-backed async).
packages/railtracks/src/railtracks/retrieval/loaders/cloud/_common.py	Adds shared infer_document_type() helper for cloud keys.
packages/railtracks/src/railtracks/retrieval/loaders/cloud/init.py	Exposes cloud loaders via a package __init__.
packages/railtracks/src/railtracks/retrieval/loaders/base.py	Makes load() safe to call from inside an already-running event loop.
packages/railtracks/src/railtracks/retrieval/loaders/init.py	Re-exports new cloud loaders from retrieval loaders package.
packages/railtracks/src/railtracks/integrations/writers/storage_writers.py	Adds writer “examples” module (currently problematic as shipped code).
packages/railtracks/src/railtracks/init.py	Exposes the retrieval submodule at the package top level.
packages/railtracks/pyproject.toml	Adds new optional extras for aws/gcp/azure-blob/sql and expands integrations/all.
mkdocs.yml	Adds nav entries for storage integrations + new tutorial/notebook pages.
docs/tutorials/walkthroughs/storage_loaders_tutorial.md	New tutorial: loading from cloud storage into a RAG workflow.
docs/tutorials/notebooks/file_embedding.md	New notebook landing page linking to Colab notebooks.
docs/scripts/storage_loaders.py	Adds snippet source file for storage loader docs (--8<-- includes).
docs/integrations/storage/sql.md	New SQL loader integration documentation.
docs/integrations/storage/s3.md	New S3 loader integration documentation.
docs/integrations/storage/overview.md	New overview page for supported storage/database loaders.
docs/integrations/storage/gcs.md	New GCS loader integration documentation.
docs/integrations/storage/azure_blob.md	New Azure Blob loader integration documentation.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[CoronRing]

?

[CoronRing]

closes #1090