data_process.md

Data Processing Operators

Base Operators

::: lazyllm.tools.data.LazyLLMDataBase members: exclude-members:

Demo Operators

::: lazyllm.tools.data.operators.demo_ops members: exclude-members:

Preference Operators

::: lazyllm.tools.data.operators.preference_ops members: exclude-members:

Tool-Use Operators

::: lazyllm.tools.data.operators.tool_use_ops members: exclude-members:

Text2SQL Operators

::: lazyllm.tools.data.operators.text2sql_ops members: exclude-members:

PT Operators

::: lazyllm.tools.data.operators.pt_op members: exclude-members:

Refine Operators

::: lazyllm.tools.data.operators.refine_op members: exclude-members:

Filter Operators

::: lazyllm.tools.data.operators.filter_op members: exclude-members:

Token Chunker

::: lazyllm.tools.data.operators.token_chunker members: exclude-members:

Code Generation Operators

::: lazyllm.tools.data.operators.codegen_ops members: exclude-members:

Text to QA pairs Operators

::: lazyllm.tools.data.operators.text2qa_ops members: exclude-members:

CoT QA Operators

::: lazyllm.tools.data.operators.cot_ops members: exclude-members:

Enhanced QA Operators

::: lazyllm.tools.data.operators.enQa_ops members: exclude-members:

Math QA Operators

::: lazyllm.tools.data.operators.math_ops members: exclude-members:

Pdf QA Operators

::: lazyllm.tools.data.operators.pdf_ops members: exclude-members:

Agentic rag

::: lazyllm.tools.data.operators.agentic_rag.agenticrag_atomic_task_generator members: exclude-members:

::: lazyllm.tools.data.operators.agentic_rag.agenticrag_depth_qa_generator members: exclude-members:

::: lazyllm.tools.data.operators.agentic_rag.agenticrag_qaf1_sample_evaluator members: exclude-members:

::: lazyllm.tools.data.operators.agentic_rag.agenticrag_width_qa_generator members: exclude-members:

Embedding synthesis

::: lazyllm.tools.data.operators.embedding_synthesis members: exclude-members:

Knowledge clean

::: lazyllm.tools.data.operators.knowledge_cleaning.file_or_url_to_markdown_converter_api members: exclude-members:

::: lazyllm.tools.data.operators.knowledge_cleaning.kbc_chunk_generator_batch members: exclude-members:

::: lazyllm.tools.data.operators.knowledge_cleaning.kbc_chunk_generator members: exclude-members:

::: lazyllm.tools.data.operators.knowledge_cleaning.kbc_multihop_qa_generator_batch members: exclude-members:

::: lazyllm.tools.data.operators.knowledge_cleaning.kbc_text_cleaner_batch members: exclude-members:

::: lazyllm.tools.data.operators.knowledge_cleaning.kbc_text_cleaner members: exclude-members:

::: lazyllm.tools.data.operators.knowledge_cleaning.qa_extract members: exclude-members:

Reranker synthesis

::: lazyllm.tools.data.operators.reranker_synthesis members: exclude-members:

LLM JSON Operators

::: lazyllm.tools.data.operators.llm_base_ops members: exclude-members:

::: lazyllm.tools.data.operators.llm_json_ops members: exclude-members:

Data Processing Pipeline

Demo Pipeline

::: lazyllm.tools.data.pipelines.demo_pipelines members: exclude-members:

Text to QA Pipeline

::: lazyllm.tools.data.pipelines.text_pipelines members: exclude-members:

QA Enhancement Pipeline

::: lazyllm.tools.data.pipelines.enhance_pipelines members: exclude-members:

Chain-of-Thought Pipeline

::: lazyllm.tools.data.pipelines.cot_pipelines members: exclude-members:

Image Pipeline

::: lazyllm.tools.data.pipelines.img_pipelines members: exclude-members:

Math Pipeline

::: lazyllm.tools.data.pipelines.math_pipelines members: exclude-members:

PDF Pipeline

::: lazyllm.tools.data.pipelines.pdf_pipelines members: exclude-members: