Make some optimizations to the Document module

[wzh1994]

📌 PR 内容 / PR Description

1. 让SchemaExtractor继承自ModuleBase，以便模型的关联启动；同时在多个Document复用DocListManager时，让SchemaExtractor可以单独设置

✅ 变更类型 / Type of Change

• 修复 Bug / Bug fix (non-breaking change that fixes an issue)
• 新功能 / New feature (non-breaking change that adds functionality)
• 重构 / Refactor (no functionality change, code structure optimized)
• 重大变更 / Breaking change (fix or feature that would cause existing functionality to change)
• 文档更新 / Documentation update (changes to docs only)
• 性能优化 / Performance optimization

🧪 如何测试 / How Has This Been Tested?

1. 单元测试

[gemini-code-assist]

Summary of Changes

Hello @wzh1994, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refines the Document module's architecture by integrating the SchemaExtractor more deeply into the lazyllm module system. This change enhances the manageability and associated startup of schema extraction processes within the RAG framework, providing greater flexibility in how document schemas are handled. Additionally, it introduces a new RichDocNode type and optimizes metadata handling for better code clarity and potential performance gains.

Highlights

• SchemaExtractor Module Integration: The SchemaExtractor class now inherits from ModuleBase, allowing it to be managed and integrated more seamlessly within the lazyllm module system, enabling associated startup and lifecycle management.
• Method Renaming for ModuleBase Compliance: The primary execution method in SchemaExtractor has been renamed from __call__ to forward, aligning with the standard interface for ModuleBase subclasses.
• RichDocNode Introduction: A new document node type, RichDocNode, has been introduced and made available for use within the RAG module.
• Enhanced Schema Extractor Management in Document Module: The Document module now registers schema_extractor instances as submodules and ensures they are properly passed down to DocImpl when adding new knowledge base groups, improving the flexibility and configurability of schema extraction.
• Optimized Metadata Handling: The metadata initialization logic in parsing_service/impl.py has been refactored for improved conciseness and efficiency, using itertools.repeat for default metadata generation.

Changelog

• lazyllm/tools/rag/init.py
  • Imported RichDocNode from .doc_node.
  • Added RichDocNode to the module's __all__ export list.
• lazyllm/tools/rag/doc_to_db/extractor.py
  • Imported ModuleBase from lazyllm.module.
  • Modified SchemaExtractor to inherit from ModuleBase.
  • Renamed the __call__ method to forward within SchemaExtractor.
• lazyllm/tools/rag/document.py
  • Added _schema_extractor as a registered submodule in the Document._Manager constructor.
  • Introduced a new private helper method _register_submodules to centralize the registration of ModuleBase instances.
  • Updated add_kb_group to accept and pass a schema_extractor argument to the DocImpl constructor.
  • Removed the direct assignment of _schema_extractor in the Document class's __init__ method, now passing it through the _Manager constructor.
• lazyllm/tools/rag/parsing_service/impl.py
  • Imported repeat from itertools.
  • Refactored the metadata initialization logic in the add_doc method to use itertools.repeat and dictionary merging for more efficient and cleaner code.

Activity

• The author indicates that unit tests were performed to verify the changes.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request successfully integrates SchemaExtractor as a ModuleBase, which is a positive step towards better module management and lifecycle control within the lazyllm framework. The refactoring of submodule registration into a dedicated _register_submodules method enhances code reusability and maintains consistency. The changes also include a more concise approach to metadata handling in the parsing service, with an identified improvement for handling potential None values in metadata merging. Overall, these changes contribute to a more robust and maintainable codebase.

[gemini-code-assist]

The current metadata merging logic might raise a TypeError if individual items within the metadatas list are None. For example, if metadatas is [None, {'user_key': 'user_value'}], the dictionary unpacking **(metadata) will fail for the None item. It's safer to ensure that metadata is always a dictionary before unpacking it.

Suggested change

	metadatas = [{**temp, **(metadata)} for metadata, temp in zip(metadatas or repeat({}), temp_metas)]
	metadatas = [{**temp, **(metadata or {})} for metadata, temp in zip(metadatas or repeat({}), temp_metas)]