April 20, 2025: This Week in OpenDAL

[github-actions[bot]]

This weekly update is generated by LLMs. You're welcome to join our Discord for in-depth discussions.

Overall Activity Summary

The apache/opendal repository has been bustling with activity over the past week, featuring contributions across feature development, bug fixes, refactoring, and documentation improvements. A total of 38 pull requests were created or updated, and several discussions have been active, focusing on roadmap planning and technical challenges. Key themes include performance optimization, service refactoring, and expanding bindings support.

Key Ongoing Projects

• Refactoring Services to Core/Backend Convention: Progress continues on Tracking issue: Refactor services to use the backend.rs and core.rs convention #5702, with contributions like feat(services/hdfs): refactor HdfsBackend to use HdfsCore for client operations #6033 by @uruemu refactoring HDFS services and refactor: Introduce IpfsCore for improved service structure #5950 by @miroim restructuring IPFS services.
• Caching Layer Implementation: Work on feat: Initial foyer layer implementation #5906 by @jorgehermo9 is advancing, aiming to provide a robust caching layer for OpenDAL.
• Write Returns Metadata: Enhancements like feat(services/gcs): implement write returns metadata #5933 by @meteorgan for GCS and feat(services/azblob): implement write returns metadata #5958 by @meteorgan for Azure Blob Storage are adding metadata extraction during write operations.
• High-Performance OpenDAL Documentation: Contributions such as docs(core): Add performance guide for concurrent write #6011 by @Xuanwo are adding guides for concurrent writes and HTTP optimizations (docs: Add performance guide http optimization #6020).
• Bindings Expansion: Efforts to improve bindings continue, including behavior tests for Go (ci(bindings/go): include go binding into behavior tests #6018 by @yuchanns) and enhancements to the C++ binding (feat(bindings/cpp) remove lib.rs.h from opendal.hpp #6041 by @deadlinefen).

Priority Items

• Open PRs Requiring Immediate Attention:

  • feat: Initial foyer layer implementation #5906 by @jorgehermo9: Initial caching layer implementation. Feedback on scope and use cases is needed.
  • new feature: Add start-after support in python binding #6042 by @asukaminato0721: Adding start-after support in Python binding. Requires review and testing.
  • feat(services/hdfs): refactor HdfsBackend to use HdfsCore for client operations #6033 by @uruemu: Refactoring HDFS services to align with the core/backend convention.

• Critical Issues:

  • sync check in opendal need work around #5995: Sync check workaround in bindings needs investigation into the root cause.
  • opendal python could hang while using with multiprocess #5330: Python binding hangs when used with multiprocessing. Requires debugging and resolution.

Notable Discussions

• Native Pagination Support for File Listing in OpenDAL #6040: Native Pagination Support for File Listing: @cbisht31 raised questions about efficient iteration over large datasets, leading to discussions on implementing pagination.
• Topic: Optimizing the C++ Binding #6035: Optimizing the C++ Binding: @deadlinefen proposed a roadmap for improving the C++ binding, including cleaner implementation and idiomatic APIs.
• [VOTE] Release Apache OpenDAL v0.53.1 - Vote Round 1 #6029: [VOTE] Release Apache OpenDAL v0.53.1: Community voting is underway for the latest release, with several contributors actively participating.

Emerging Trends

• Refactoring for Maintainability: The repository is increasingly adopting a standardized structure with core.rs and backend.rs modules for services, improving readability and maintainability.
• Focus on Performance: Contributions like docs(core): Add performance guide for concurrent write #6011 and feat(core): Use Buffer as http_body::Body directly #6026 aim to enhance performance through concurrent operations and reduced overhead in HTTP requests.
• Documentation Expansion: Multiple PRs are focused on improving documentation, making the project more accessible to new contributors.

Good First Issues

• new feature: Add start-after support in python binding #6042: Add start-after support in Python binding: This issue introduces a straightforward feature to Python bindings, making it a great entry point for contributors familiar with Python.
• Tracking issue: Refactor services to use the backend.rs and core.rs convention #5702: Refactor services to use the backend.rs and core.rs convention: Tasks involve restructuring service implementations, providing a clear and manageable scope for new contributors.
• Implement Write Returns Metadata for all services #5693: Implement Write Returns Metadata for all services: Contributors can focus on extracting metadata for specific services, offering a well-defined task with impactful results.
• Tracking issues of RFC-5479: Context #5480: Context Implementation Tasks: Smaller tasks related to RFC-5479, ideal for contributors looking to familiarize themselves with OpenDAL's core concepts.

We encourage new contributors to explore these "Good First Issues" and join the community discussions to get started!