Validation Dataset Creator - Specification

Version: 1.0 Date: 2025-10-01 Status: Approved

1. Overview

1.1 Purpose

Transform validation pipeline results into HuggingFace training datasets suitable for embedding model training and retrieval tasks.

1.2 Motivation

The validation pipeline (explode-questions → batch-candidates → validate-batched-requests) produces detailed validation results with LLM judgments on which chunks can answer each question. This data needs to be transformed into a training-ready format that separates positive examples (chunks that can answer) from negative examples (chunks that cannot).

1.3 Key Requirements

Memory Efficient: Handle large datasets (100K+ questions) without OOM
Batch Merging: Combine multi-batch validation results per question
Quality Filtering: Handle inconsistent LLM responses gracefully
HuggingFace Compatible: Output format loadable with load_dataset("parquet", ...)
Metrics Tracking: Provide statistics for pipeline monitoring

2. Data Flow

Input: validation_results.parquet
│
├─ Multiple rows per question (one per batch)
├─ Fields: question_id, question, answer, positive_chunk_id
├─ Parallel arrays: candidate_chunk_ids[], is_answerable[], quality_scores[]
│
↓ [Transform]
│
├─ Group by question_id
├─ Merge all batches
├─ Separate positive/negative candidates
├─ Lookup chunk content
├─ Deduplicate
│
↓
Output: HuggingFace Dataset (parquet format)
│
└─ One row per question
   ├─ Question: str
   ├─ Positive_Chunks: List[str]
   ├─ Negative_Chunks: List[str]
   └─ Answer: str

3. Output Format Specification

3.1 Dataset Schema

{
  "Question": "What is 5G network slicing?",
  "Positive_Chunks": [
    "5G network slicing allows operators to create multiple virtual networks on a shared physical infrastructure...",
    "Network slicing is a key feature of 5G that enables customized network services..."
  ],
  "Negative_Chunks": [
    "eSIM technology enables programmable SIM cards...",
    "WiFi 6 provides faster wireless connectivity...",
    "LTE Cat-M1 is optimized for IoT devices..."
  ],
  "Answer": "5G network slicing allows operators to create multiple virtual networks on a shared physical infrastructure, each optimized for specific use cases."
}

3.2 Field Descriptions

Field	Type	Description
`Question`	`str`	The input question from Q&A generation
`Positive_Chunks`	`List[str]`	Chunk contents that can answer the question (is_answerable=True)
`Negative_Chunks`	`List[str]`	Chunk contents that cannot answer the question (is_answerable=False)
`Answer`	`str`	The expected answer (from original Q&A generation)

3.3 Data Types (HuggingFace)

from datasets import Features, Value, Sequence

features = Features({
    "Question": Value("string"),
    "Positive_Chunks": Sequence(Value("string")),
    "Negative_Chunks": Sequence(Value("string")),
    "Answer": Value("string")
})

4. Data Transformation Rules

4.1 Core Rules

Positive Chunk Always Included
- The positive_chunk_id (original source chunk) MUST always be included in Positive_Chunks
- It is never validated by the LLM (already known to be correct)
- Lookup content from enriched_chunks.parquet
Candidate Classification
- Candidate with is_answerable == True → Add to Positive_Chunks
- Candidate with is_answerable == False → Add to Negative_Chunks
Quality Filtering
- If is_answerable == True AND answer_quality_score == 0.0:
  - Skip this candidate (inconsistent validation result)
  - Increment metrics.candidates_skipped_inconsistent
  - Log warning
Multi-Batch Merging
- Group all batches by question_id
- Collect candidates from all batches
- Process as single question
Deduplication
- If same chunk_id appears in multiple batches:
  - Keep first occurrence only
  - Track by chunk_id, not content
- Apply separately to Positive and Negative lists
Content Lookup
- Load enriched_chunks.parquet once at start
- Create {chunk_id: chunk_content} lookup dict
- Use for all content retrievals
Missing Content Handling
- If chunk_id not found in enriched_chunks:
  - Log warning: "Chunk {chunk_id} not found in enriched_chunks"
  - Skip this chunk (exclude from output)
  - Increment metrics.chunks_skipped_missing_content
Empty Lists Allowed
- Positive_Chunks can have only 1 item (original positive)
- Negative_Chunks can be empty (all candidates positive)
- Both scenarios are valid, include in dataset
Ordering Preserved
- Keep original order from validation results
- Do NOT sort by quality score
- Maintains validation batch structure
One Record Per Question
- Output has exactly one row per unique question_id
- All batches merged into single record

5. Processing Algorithm

5.1 High-Level Steps

1. Load enriched_chunks.parquet → create chunk_id:content lookup dict
2. Stream validation_results.parquet in batches (--batch-size parameter)
3. Group records by question_id
4. For each question:
   a. Extract question, answer, positive_chunk_id
   b. Collect all candidate_chunk_ids across batches
   c. Separate by is_answerable flag
   d. Filter out inconsistent candidates (is_answerable=True, quality=0)
   e. Deduplicate chunk IDs
   f. Lookup chunk contents from enriched_chunks
   g. Handle missing content (skip, log warning)
   h. Always include positive_chunk_id in Positive_Chunks
   i. Create ValidationDatasetExample
   j. Update metrics
5. Yield records to IterableDataset.from_generator()
6. Save dataset to parquet
7. Write metrics JSON

5.2 Detailed Transform Function

def transform_question_to_example(
    question_data: list[dict],  # All batches for one question
    chunk_lookup: dict[str, str],  # chunk_id -> content
    metrics: DatasetMetrics
) -> Optional[ValidationDatasetExample]:
    """Transform validation results for one question."""

    # Extract from first batch (same across all batches)
    first_batch = question_data[0]
    question = first_batch["question"]
    answer = first_batch["answer"]
    positive_chunk_id = first_batch["positive_chunk_id"]

    # Collect all candidates across batches
    seen_positive_ids = set()
    seen_negative_ids = set()
    positive_ids = []
    negative_ids = []

    for batch in question_data:
        candidates = json.loads(batch["candidate_chunk_ids"])
        is_answerable = json.loads(batch["is_answerable"])
        quality_scores = json.loads(batch["answer_quality_scores"])

        for chunk_id, answerable, quality in zip(candidates, is_answerable, quality_scores):
            # Skip inconsistent results
            if answerable and quality == 0.0:
                metrics.candidates_skipped_inconsistent += 1
                logger.warning(f"Skipping {chunk_id}: is_answerable=True but quality=0")
                continue

            # Classify and deduplicate
            if answerable:
                if chunk_id not in seen_positive_ids:
                    positive_ids.append(chunk_id)
                    seen_positive_ids.add(chunk_id)
            else:
                if chunk_id not in seen_negative_ids:
                    negative_ids.append(chunk_id)
                    seen_negative_ids.add(chunk_id)

    # Always include positive chunk (if not already in candidates)
    if positive_chunk_id not in seen_positive_ids:
        positive_ids.insert(0, positive_chunk_id)

    # Lookup content
    positive_chunks = []
    for chunk_id in positive_ids:
        content = chunk_lookup.get(chunk_id)
        if content is None:
            logger.warning(f"Chunk {chunk_id} not found in enriched_chunks")
            metrics.chunks_skipped_missing_content += 1
            continue
        positive_chunks.append(content)

    negative_chunks = []
    for chunk_id in negative_ids:
        content = chunk_lookup.get(chunk_id)
        if content is None:
            logger.warning(f"Chunk {chunk_id} not found in enriched_chunks")
            metrics.chunks_skipped_missing_content += 1
            continue
        negative_chunks.append(content)

    # Update metrics
    metrics.total_questions += 1
    metrics.total_positive_chunks += len(positive_chunks)
    metrics.total_negative_chunks += len(negative_chunks)
    if len(positive_chunks) == 1:
        metrics.questions_with_no_additional_positives += 1

    return ValidationDatasetExample(
        Question=question,
        Positive_Chunks=positive_chunks,
        Negative_Chunks=negative_chunks,
        Answer=answer
    )

6. Edge Cases

6.1 Handled Scenarios

Scenario	Behavior
All candidates negative	Include question with only original positive chunk
All candidates positive	Include question with empty Negative_Chunks
Duplicate chunk across batches	Keep first occurrence only
Missing chunk content	Skip chunk, log warning, continue
is_answerable=True, quality=0	Skip candidate, log warning
positive_chunk_id missing content	Log ERROR, skip entire question
Empty validation results	Create empty dataset, log warning
Single batch per question	Process normally (common case)

6.2 Error Conditions

Fatal Errors (stop processing):

enriched_chunks.parquet not found
validation_results.parquet not found
Output directory not writable

Recoverable Errors (log and continue):

Missing chunk content → skip chunk
Inconsistent validation → skip candidate
Malformed JSON in parallel arrays → skip batch

7. CLI Command Specification

7.1 Command

gsma datasets create-from-validation [OPTIONS]

7.2 Parameters

Parameter	Type	Required	Default	Description
`--input`	Path	✅	-	Path to validation_results.parquet
`--enriched-chunks`	Path	✅	-	Path to enriched_chunks.parquet
`--output`	Path	✅	-	Output parquet path
`--batch-size`	int	❌	1000	Batch size for streaming processing
`--max-positives`	int	❌	None	Maximum positive chunks per question (top quality selection)
`--max-negatives`	int	❌	None	Maximum negative chunks per question (top quality selection)
`--metrics-output`	Path	❌	None	Optional metrics JSON output
`--logger-level`	str	❌	INFO	Logging level (DEBUG/INFO/WARNING/ERROR)

7.3 Example Usage

# Basic usage
uv run gsma datasets create-from-validation \
  --input data/validation/validation_results.parquet \
  --enriched-chunks data/enriched_chunks.parquet \
  --output data/validation/validation_dataset.parquet

# With metrics, filtering, and larger batch size
uv run gsma datasets create-from-validation \
  --input data/validation/validation_results.parquet \
  --enriched-chunks data/enriched_chunks.parquet \
  --output data/validation/validation_dataset.parquet \
  --batch-size 5000 \
  --max-positives 10 \
  --max-negatives 20 \
  --metrics-output metrics/validation_dataset.json \
  --logger-level DEBUG

7.4 Exit Codes

Code	Meaning
0	Success
1	Fatal error (missing input files, etc.)
2	Invalid parameters

8. Metrics Specification

8.1 Metrics Schema

{
  "total_questions": 12543,
  "total_positive_chunks": 18829,
  "total_negative_chunks": 105387,
  "avg_positives_per_question": 1.50,
  "avg_negatives_per_question": 8.40,
  "questions_with_no_additional_positives": 8201,
  "questions_skipped_missing_content": 0,
  "chunks_skipped_missing_content": 23,
  "candidates_skipped_inconsistent": 47,

  "positive_chunks_histogram": {
    "1": 8201,
    "2": 2847,
    "3": 1203,
    "4": 292
  },
  "negative_chunks_histogram": {
    "0": 142,
    "5": 1847,
    "10": 8203,
    "15": 2351
  },
  "positive_quality_score_histogram": {
    "0.0-0.1": 0,
    "0.1-0.2": 0,
    "0.2-0.3": 0,
    "0.3-0.4": 142,
    "0.4-0.5": 1203,
    "0.5-0.6": 3847,
    "0.6-0.7": 5201,
    "0.7-0.8": 4892,
    "0.8-0.9": 2341,
    "0.9-1.0": 1203
  },
  "negative_quality_score_histogram": {
    "0.0-0.1": 105387,
    "0.1-0.2": 0,
    "0.2-0.3": 0
  },
  "positive_similarity_score_histogram": {
    "0.3-0.4": 142,
    "0.4-0.5": 892,
    "0.5-0.6": 2341,
    "0.6-0.7": 4203,
    "0.7-0.8": 6201,
    "0.8-0.9": 3847,
    "0.9-1.0": 1203
  },
  "negative_similarity_score_histogram": {
    "0.0-0.1": 0,
    "0.1-0.2": 203,
    "0.2-0.3": 1847,
    "0.3-0.4": 15203,
    "0.4-0.5": 35201,
    "0.5-0.6": 28203,
    "0.6-0.7": 18201,
    "0.7-0.8": 5203,
    "0.8-0.9": 1203,
    "0.9-1.0": 123
  }
}

8.2 Metric Descriptions

Summary Metrics

Metric	Description
`total_questions`	Number of unique questions in output dataset
`total_positive_chunks`	Total positive chunk instances across all questions
`total_negative_chunks`	Total negative chunk instances across all questions
`avg_positives_per_question`	Mean number of positive chunks per question
`avg_negatives_per_question`	Mean number of negative chunks per question
`questions_with_no_additional_positives`	Questions where only original positive chunk exists
`questions_skipped_missing_content`	Questions skipped due to missing positive chunk content
`chunks_skipped_missing_content`	Individual chunks skipped due to missing content
`candidates_skipped_inconsistent`	Candidates skipped due to inconsistent validation (is_answerable=True, quality=0)

Histogram Metrics

Metric	Description
`positive_chunks_histogram`	Distribution of number of positive chunks per question (count → frequency)
`negative_chunks_histogram`	Distribution of number of negative chunks per question (count → frequency)
`positive_quality_score_histogram`	Distribution of positive chunk quality scores (binned by 0.1, e.g., "0.8-0.9" → count)
`negative_quality_score_histogram`	Distribution of negative chunk quality scores (binned by 0.1)
`positive_similarity_score_histogram`	Distribution of positive chunk similarity scores (binned by 0.1)
`negative_similarity_score_histogram`	Distribution of negative chunk similarity scores (binned by 0.1)

Note: Histogram data provides insights into dataset quality and distribution:

Chunk count histograms show how many positive/negative examples per question
Quality score histograms reveal LLM confidence in classifications
Similarity score histograms indicate how semantically related chunks are to questions

9. DVC Pipeline Integration

9.1 Pipeline Position

explode_questions
  ↓
batch_candidates
  ↓
validate_batched_requests
  ↓
create_validation_dataset  ← NEW STAGE

9.2 DVC Stage Configuration

create_validation_dataset:
  wdir: ../..
  cmd: >-
    uv run gsma datasets create-from-validation
    --input data/validation/validation_results.parquet
    --enriched-chunks data/enriched_chunks.parquet
    --output data/validation/validation_dataset.parquet
    --batch-size 1000
    --metrics-output metrics/validation_dataset.json
    --logger-level INFO
  deps:
    - data/validation/validation_results.parquet
    - data/enriched_chunks.parquet
    - gsma_dataset_creation/datasets_cli.py
    - gsma_dataset_creation/datasets/validation_dataset_creator.py
  outs:
    - data/validation/validation_dataset.parquet
  metrics:
    - metrics/validation_dataset.json:
        cache: false
  desc: "Create HuggingFace dataset from validation results"

9.3 Running the Pipeline

# Run only this stage
dvc repro create_validation_dataset

# Run full validation pipeline
cd pipelines/validation && dvc repro

10. Testing Requirements

10.1 Unit Tests

Basic Transformation
- Input: Single question, 1 batch, 3 candidates (2 positive, 1 negative)
- Expected: Positive_Chunks has 3 items (original + 2 candidates), Negative_Chunks has 1
Multi-Batch Merging
- Input: Question with 3 batches (25 total candidates)
- Expected: All candidates collected and classified correctly
Positive Chunk Always Included
- Input: positive_chunk_id NOT in candidate list
- Expected: Positive_Chunks contains positive chunk content
Inconsistent Validation Handling
- Input: Candidate with is_answerable=True, quality_score=0.0
- Expected: Candidate excluded, metrics.candidates_skipped_inconsistent == 1
No Additional Positives
- Input: All candidates have is_answerable=False
- Expected: Positive_Chunks = [original], Negative_Chunks = [all candidates]
Deduplication
- Input: Chunk "C" appears in batch 0 and batch 2 (both marked answerable)
- Expected: Positive_Chunks contains chunk C content exactly once
Missing Chunk Content
- Input: Candidate chunk_id not in enriched_chunks
- Expected: Chunk excluded, warning logged, metrics updated
Metrics Calculation
- Input: Process 5 questions with varied positive/negative counts
- Expected: Metrics JSON has correct totals and averages

10.2 Test Fixtures

Create synthetic test data in tests/datasets/fixtures/:

validation_results_sample.parquet (10 questions, various scenarios)
enriched_chunks_sample.parquet (chunk content lookup)

10.3 Test Execution

# Run dataset tests only
uv run pytest tests/datasets/ -v

# Run with coverage
uv run pytest tests/datasets/ --cov=gsma_dataset_creation.datasets

11. Performance Considerations

11.1 Memory Usage

Chunk Lookup Dict: ~100MB for 100K chunks (assuming 1KB/chunk)
Streaming Processing: Processes --batch-size questions at a time
IterableDataset: Does not load entire dataset into memory

11.2 Scalability

Dataset Size	Batch Size	Est. Memory	Est. Time
10K questions	1000	~150MB	2 minutes
100K questions	1000	~200MB	15 minutes
500K questions	5000	~500MB	60 minutes

11.3 Optimization Tips

Increase --batch-size for more memory, faster processing
Use SSDs for input/output files (I/O bound)
Pre-filter validation_results if only subset needed

12. Example Output

12.1 Sample Record

{
  "Question": "What are the key components of the MDSCert Scheme?",
  "Positive_Chunks": [
    "The MDSCert Scheme comprises several key components: the Evaluation Report, the Certification Body (CB), the Manufacturer, and the MSTL (Mobile Security Testing Laboratory). Each component plays a specific role in the certification process.",
    "MDSCert Scheme components include the certification process itself, the evaluation criteria based on industry standards, and the ongoing maintenance requirements for certified devices."
  ],
  "Negative_Chunks": [
    "eSIM technology enables remote SIM provisioning for mobile devices, allowing users to switch carriers without physical SIM cards.",
    "5G network slicing provides isolated virtual networks for different use cases, improving service quality and security.",
    "LTE Cat-M1 is optimized for low-power IoT applications with extended battery life and deep indoor coverage."
  ],
  "Answer": "The MDSCert Scheme includes the Evaluation Report, Certification Body (CB), Manufacturer, and MSTL as key components, each with specific roles in device security certification."
}

12.2 Loading the Dataset

from datasets import load_dataset

# Load from parquet
dataset = load_dataset("parquet", data_files="data/validation/validation_dataset.parquet")

# Access records
print(dataset["train"][0])
# {'Question': '...', 'Positive_Chunks': [...], 'Negative_Chunks': [...], 'Answer': '...'}

# Get statistics
print(f"Total examples: {len(dataset['train'])}")
print(f"Average positive chunks: {sum(len(ex['Positive_Chunks']) for ex in dataset['train']) / len(dataset['train'])}")

13. Future Enhancements

13.1 Potential Features (Not in Scope)

Quality score thresholding (--min-quality-score)
Max negatives per question (--max-negatives)
Stratified sampling by question_type
Direct HuggingFace Hub upload
Train/val/test split generation
Dataset filtering by document_source or working_group

13.2 Extension Points

The implementation should be modular enough to add:

Alternative output formats (JSON, CSV)
Custom transformation functions
Additional metadata fields
Dataset augmentation strategies

End of Specification

Uh oh!