task executor issues

[concertdictate]

What problem does this PR solve?

Fixes #8706 - InfinityException: TOO_MANY_CONNECTIONS when running multiple task executor workers

Problem Description

When running RAGFlow with 8-16 task executor workers, most workers fail to start properly. Checking logs revealed that workers were stuck/hanging during Infinity connection initialization - only 1-2 workers would successfully register in Redis while the rest remained blocked.

Root Cause

The Infinity SDK ConnectionPool pre-allocates all connections in __init__. With the default max_size=32 and multiple workers (e.g., 16), this creates 16×32=512 connections immediately on startup, exceeding Infinity's default 128 connection limit. Workers hang while waiting for connections that can never be established.

Changes

1. Prevent Infinity connection storm (rag/utils/infinity_conn.py, rag/svr/task_executor.py)

   • Reduced ConnectionPool max_size from 32 to 4 (sufficient since operations are synchronous)
   • Added staggered startup delay (2s per worker) to spread connection initialization

2. Handle None children_delimiter (rag/app/naive.py)

   • Use or "" to handle explicitly set None values from parser config

3. MinerU parser robustness (deepdoc/parser/mineru_parser.py)

   • Use .get() for optional output fields that may be missing
   • Fix DISCARDED block handling: change pass to continue to skip discarded blocks entirely

Why max_size=4 is sufficient

Workers	Pool Size	Total Connections	Infinity Limit
16	32	512	128 ❌
16	4	64	128 ✅
32	4	128	128 ✅

• All RAGFlow operations are synchronous: get_conn() → operation → release_conn()
• No parallel docStoreConn operations in the codebase
• Maximum 1-2 concurrent connections needed per worker; 4 provides safety margin

MinerU DISCARDED block bug

When MinerU returns blocks with type: "discarded" (headers, footers, watermarks, page numbers, artifacts), the previous code used pass which left the section variable undefined, causing:

• UnboundLocalError if DISCARDED is the first block
• Duplicate content if DISCARDED follows another block (stale value from previous iteration)

Root cause confirmed via MinerU source code:

From mineru/utils/enum_class.py:

class BlockType:
    DISCARDED = 'discarded'
    # VLM 2.5+ also has: HEADER, FOOTER, PAGE_NUMBER, ASIDE_TEXT, PAGE_FOOTNOTE

Per MinerU documentation, discarded blocks contain content that should be filtered out for clean text extraction.

Fix: Changed pass to continue to skip discarded blocks entirely.

Testing

• Verified all 16 workers now register successfully in Redis
• All workers heartbeating correctly
• Document parsing works as expected
• MinerU parsing with DISCARDED blocks no longer crashes

Type of change

• Bug Fix (non-breaking change which fixes an issue)

[yuzhichang]

LGTM