Add warning for small n_clusters in SemanticDeduplicationWorkflow

[KunalSachdev2005]

Description

Warn users when n_clusters < 1000 as this may cause OOM errors for large datasets since each cluster must fit in memory.
Fixes: #1224

Usage

# Add snippet demonstrating usage

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

Added a warning when n_clusters < 1000 to prevent out-of-memory errors on large datasets, addressing issue #1224.

Changes made:

• Added MIN_RECOMMENDED_N_CLUSTERS = 1000 constant at the module level
• Added validation check in _validate_config() method that logs a warning when n_clusters is below the threshold
• Used logger.warning (consistent with project patterns) to inform users about potential OOM risks
• Warning message clearly explains the issue and recommends using n_clusters >= 1000 for large datasets

Note: A previous reviewer suggested adding a test to verify the warning is raised. This would be a good addition to prevent regression, but the core functionality is correct as-is.

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The change is a simple, non-breaking addition of a warning message that follows the existing codebase patterns. It uses logger.warning consistently with other warnings in the project, adds a well-documented constant, and addresses the issue requirements. The warning logic is straightforward with no edge cases or potential bugs.
• No files require special attention

Important Files Changed

Filename	Overview
nemo_curator/stages/deduplication/semantic/workflow.py	Added warning when n_clusters < 1000 to prevent OOM errors on large datasets - implementation is correct and follows project patterns

Sequence Diagram

sequenceDiagram
    participant User
    participant Workflow as SemanticDeduplicationWorkflow
    participant Logger
    
    User->>Workflow: __init__(n_clusters=500, ...)
    Workflow->>Workflow: Store configuration parameters
    Workflow->>Workflow: _validate_config()
    
    alt n_clusters < MIN_RECOMMENDED_N_CLUSTERS (1000)
        Workflow->>Logger: logger.warning("n_clusters=500 is less than 1000...")
        Logger-->>User: Display warning message
    end
    
    Workflow->>Workflow: Continue validation (distance_metric, etc.)
    Workflow-->>User: Workflow initialized

Loading

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

style: Consider adding a test to verify this warning is raised when n_clusters < 1000. This ensures the feature works as expected and prevents regression.

# In test_workflow.py
def test_warning_for_small_n_clusters():
    with pytest.warns(UserWarning, match="n_clusters=.*is less than 1000"):
        SemanticDeduplicationWorkflow(
            input_path="/tmp/test",
            output_path="/tmp/output", 
            n_clusters=500
        )

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[sarahyurick]

Thanks @KunalSachdev2005 !

[greptile-apps]

Greptile found no issues!

From now on, if a review finishes and we haven't found any issues, we will not post anything, but you can confirm that we reviewed your changes in the status check section.

_{This feature can be toggled off in your Code Review Settings by deselecting "Create a status check for each PR".}

[sarahyurick]

/ok to test aa6cab8

[praateekmahajan]

Let's use logger.warning instead of warnings.warn...
wdyt @sarahyurick / @ayushdg