Skip to content

struct_builder.py并行处理chunks可能导致的负优化问题 #59

@Eloise010

Description

@Eloise010

graphrag_agent\graph\structure\struct_builder.py
parallel_process_chunks()函数:
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
future_to_batch = {
executor.submit(process_chunk_batch, batch, i * batch_size): i
for i, batch in enumerate(chunk_batches)
}
process_chunk_batch()应该是纯CPU操作,开的线程池因为GIL有可能是负优化。优化是否可以考虑写入数据库时加线程池?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions