Do we have better names for opendal python's Layer?

[Xuanwo]

The term Layer is commonly used in Rust but not as much in Python. What is the equivalent term in Python? Should we use Middleware instead?

inviting @Zheaoli @messense @asukaminato0721 @yihong0618

[asukaminato0721]

Did some research, in python's ecosystem, Middleware is more commonly used.

• https://docs.djangoproject.com/en/5.2/topics/http/middleware/
• https://flask.palletsprojects.com/en/stable/lifecycle/#middleware
• https://fastapi.tiangolo.com/tutorial/middleware/

layer is mostly used in

• https://www.tensorflow.org/api_docs/python/tf/keras/layers
• https://pytorch.org/docs/stable/nn.html#convolution-layers

so I think Middleware is better name.

[Xuanwo]

Thank you @asukaminato0721 for your research. Would you be interested in conducting a more in-depth study on how python users use middleware? This could help us develop a better API than what we currently offer.

[asukaminato0721]

a more in-depth study on how python users use middleware

Basically define a class -> define special method, usually __call__ or dispatch -> put them into a list/builder pattern api

1. Django Middleware

django_mw_project/
├── myapp/
│   ├── __init__.py
│   ├── middleware.py  # <---
│   ├── views.py
│   └── urls.py
├── django_mw_project/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── manage.py

myapp/middleware.py:

import time

class TimingMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response
    def __call__(self, request):
        ...
        response = self.get_response(request)
        ...
        return response

class LoggingMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response
    def __call__(self, request):
        ...
        response = self.get_response(request)
        ...
        return response

MIDDLEWARE = [
    ...
    # require seq: LoggingMiddleware -> TimingMiddleware -> ... -> View
    # respond seq: View -> ... -> TimingMiddleware -> LoggingMiddleware
    'myapp.middleware.LoggingMiddleware',
    'myapp.middleware.TimingMiddleware',
]

---

2. Flask Middleware (use WSGI Middleware)

flask_app.py:

from flask import Flask, Response
import time

class TimingMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        response_iterable = self.wsgi_app(environ, custom_start_response)
        return response_iterable

class LoggingMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        def custom_start_response(status, headers, exc_info=None):
            ...
        response_iterable = self.wsgi_app(environ, custom_start_response)
        return response_iterable

app = Flask(__name__)

@app.route('/')
def hello():
    time.sleep(0.1)
    return Response("Hello from Flask with WSGI Middleware!", mimetype='text/plain')

# req seq: Logging -> Timing -> Flask App
# res seq: Flask App -> Timing -> Logging
app.wsgi_app = LoggingMiddleware(app.wsgi_app)
app.wsgi_app = TimingMiddleware(app.wsgi_app) # Timing in Logging

if __name__ == '__main__':
    app.run(debug=True, port=5000)

---

3. FastAPI Middleware

fastapi_app.py:

from fastapi import FastAPI, Request, Response
import time
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.types import ASGIApp, Receive, Scope, Send

class TimingMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
        response = await call_next(request)
        return response

class LoggingMiddleware(BaseHTTPMiddleware):
     async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
        response = await call_next(request)
        return response

app = FastAPI()

# use add_middleware 
# req seq: Logging -> Timing -> Route
# res seq: Route -> Timing -> Logging
app.add_middleware(LoggingMiddleware)         # most outside
app.add_middleware(TimingMiddleware)         # inner side

@app.get("/")
async def read_root():
    await asyncio.sleep(0.1)
    return {"message": "Hello from FastAPI with Middleware!"}

---

[Xuanwo]

cc @erickguan who may aslo interested in this.

[erickguan]

Thanks for the invite. @asukaminato0721's investigation is solid.

Popular web frameworks use middlewares. They made that choice for obvious reasons — each middleware can add additional features, short-circuit executions, and more.
In contrast, deep learning frameworks use layers because the term layer originates from multi-layer perceptrons, which are made up of stacked layers. OpenDAL has nothing to do with neural network layers, so that analogy isn't applicable.

For further reference, scrapy also uses middleware.

Some other libraries, such as requests, mainly offer additional configurations through extra arguments rather than through configurable middlewares. Since OpenDAL has many layers, such a design (relying only on arguments) would be less desirable.

Other names, such as Handlers (e.g., logging.StreamHandler), mostly act as adapters. They don't align well with OpenDAL's concept of layers either.

While "layers" make sense technically — especially since OpenDAL is storage-focused — using "middlewares" would align better with Python ecosystems that are common in web applications and CLIs (which interact heavily with remote services).
I imagine most users who are interested in the Python bindings will want to use OpenDAL to share a unified interface across web services and data applications, leveraging its broad support for various storage backends.
In other words, most users would expect the Python binding to serve as a convenience layer for building applications, not for building a storage/database system itself.

Therefore, I concur with using middlewares — adopting the middleware concept in the Python bindings will make OpenDAL more ergonomic and intuitive for users.