May 04, 2025: This Week in OpenDAL

[github-actions[bot]]

This weekly update is generated by LLMs. You're welcome to join our Discord for in-depth discussions.

---

Overall Activity Summary

The past week has seen significant activity in the Apache OpenDAL repository, with 33 pull requests, 20 issues, and 5 discussions. Contributions spanned across multiple areas, including feature enhancements, bug fixes, and refactoring efforts. Notable progress was made in improving bindings (Python, C, Java), adding context to HTTP-based services, and addressing long-standing issues like metadata handling and API consistency.

---

Key Ongoing Projects

• Python Binding Enhancements:

  • feat(bindings/python): Enhance Reader and Writer #6086 by @chitralverma introduces streaming AsyncReader and AsyncWriter interfaces for Python bindings, enabling efficient handling of large datasets.

• C Binding Improvements:

  • feat(bindings/c)!: Make features configurable via CMakeLists #6143 by @asukaminato0721 adds support for configurable features in C bindings, addressing issues Support configure features for C binding #4313 and doc(bindings): add docs about how to enable features #4318.
  • feat(bindings/c): Add seek support for c-binding  #6119 by @taoseng adds seek support for C bindings, closing Need range reader or seek in c binding #6083.

• HTTP Context in Services:

  • Several PRs (e.g., feat(services/azdls): add request context #6122, feat(services/azblob): add request context #6121, feat(services/onedrive): Add operation in http context #6111) added request context to HTTP-based services like Azure Data Lake, Azure Blob, and OneDrive, contributing to the broader initiative tracked in Add http context in for all http based services #5774.

• Refactoring for Maintainability:

  • feat(services/hdfs): refactor HdfsBackend to use HdfsCore for client operations #6033 by @uruemu refactors the HDFS backend to separate core logic, aligning with the convention outlined in Tracking issue: Refactor services to use the backend.rs and core.rs convention #5702.

---

Priority Items

• Open Pull Requests:

  • feat(bindings/python): Enhance Reader and Writer #6086 by @chitralverma: Introduces critical Python binding enhancements. Needs review and testing.
  • feat(services/hdfs): refactor HdfsBackend to use HdfsCore for client operations #6033 by @uruemu: Refactors HDFS backend. Requires feedback to finalize.

• Issues Needing Attention:

  • bindings/c: write may fail, but the error is silently ignored. #6125 by @taoseng: Highlights a critical issue where write errors in C bindings are silently ignored.
  • bug: object_store path will be percent-encoded twice #6129 by @Colerar: Fixes double percent-encoding in the object-store integration.

---

Notable Discussions

• How can I verify my modification on OpenDAL? #6144: @Here-is-wen-yi inquired about best practices for verifying local modifications in OpenDAL.
• Remove blocking API entirely from `raw::Access` #6126: @Xuanwo proposed removing blocking APIs from raw::Access to reduce maintenance overhead.
• Concurrent support for opendal list #6115: @Xuanwo initiated a discussion on adding concurrent support for the list operation, which could significantly improve performance for large datasets.

---

Emerging Trends

• Focus on Bindings: Significant improvements are being made to Python, C, and Java bindings, reflecting a trend toward enhancing cross-language support.
• Refactoring for Clarity: The repository is moving toward a more modular and maintainable structure, as seen in efforts like Tracking issue: Refactor services to use the backend.rs and core.rs convention #5702 and feat(services/hdfs): refactor HdfsBackend to use HdfsCore for client operations #6033.
• HTTP Context Standardization: Adding context to HTTP-based services (Add http context in for all http based services #5774) is becoming a standard practice, improving debugging and observability.

---

Good First Issues

• Tracking issue: Refactor services to use the backend.rs and core.rs convention #5702: Refactor services to use the backend.rs and core.rs convention. This is a great way to get familiar with OpenDAL's architecture.
• Implement Write Returns Metadata for all services #5693: Implement "Write Returns Metadata" for additional services. This involves extending existing functionality and is well-documented.
• Tracking issues of RFC-5479: Context #5480: Add context support for operations. This is a straightforward task with clear examples in existing PRs.
• Tracking Issues of RFC-5556: Write Returns Metadata #5557: Extend metadata handling for write operations. A good opportunity to contribute to a high-impact feature.

---

Thank you to all contributors for your hard work this week! 🚀