Build Unified Distilled Knowledge Graph with Human/AI Source Fusion

[durapensa]

Overview

Build a sophisticated knowledge fusion system that creates a deduplicated, weighted graph blending human and AI insights with transparent provenance tracking.

Architecture

Storage Layers

Raw Layer (append-only):
- events/hot.jsonl         # Human thoughts (weight: 1.0)
- derived/approved.jsonl   # AI findings (weight: varies)
- derived/rejected.jsonl   # Learning data

Fusion Layer (periodic rebuild):
- distilled/concepts.jsonl       # Unified concepts with weights
- distilled/relations.jsonl      # Connections between concepts
- distilled/provenance.jsonl     # Source tracking

Example Unified Concept

{
  "id": "evolutionary-ethics",
  "canonical_form": "Evolutionary Ethics",
  "description": "Morality emerging from evolutionary dynamics",
  "weight": 0.85,
  "sources": {
    "human": {
      "count": 5,
      "events": ["2025-06-10T17:03:28Z", "2025-06-10T17:01:36Z"],
      "weight_contribution": 0.6
    },
    "ai": {
      "count": 2,
      "findings": ["2025-06-10T18:02:26Z-pattern-analysis"],
      "weight_contribution": 0.4
    }
  },
  "confidence": 0.92,
  "last_updated": "2025-06-10T18:30:00Z"
}

Key Features

1. Dual-Write System

• Summary in hot.jsonl for discoverability
• Full finding in derived/approved.jsonl with complete metadata
• Bidirectional linking via derived_ref

2. Relative Contribution Tracking

• Human contributions weighted as ground truth (1.0)
• AI contributions weighted by approval confidence and recency
• Transparent source attribution showing percentage from each source

3. Deduplication Strategy

• Concept clustering using embeddings/similarity
• Canonical form selection (prefer human-originated terms)
• Alias tracking for all variations

4. Rejection Learning

• Track rejected findings in derived/rejected.jsonl
• Include rejection reasons and improvement suggestions
• Use for ML training and analysis improvement

Implementation Steps

Phase 1: Foundation (Immediate)

• Update review-findings to implement dual-write
• Add enriched metadata tracking (source events, analysis parameters)
• Implement rejected findings storage

Phase 2: Distillation Process

• Build concept extraction and clustering
• Implement weight calculation algorithm
• Create periodic distillation background job

Phase 3: Query Integration

• Update query tools to search unified graph
• Add provenance display options
• Implement confidence-based filtering

Phase 4: Evolution Tracking

• Track concept strength over time
• Visualize knowledge graph growth
• Identify emerging themes

Benefits

1. Trust Indicators - Clear visibility of human vs AI contributions
2. Feedback Loop - Rejected findings improve future analysis
3. Clean Queries - Single deduplicated concept instead of fragments
4. Knowledge Evolution - Watch insights strengthen and connect over time
5. Transparent Provenance - Always know where knowledge originated

Related Issues

• Depends on completion of Fix review-findings jq errors and reorganize workflow tools #17 (review-findings fixes)
• Enhances Build derived knowledge persistence pipeline #10 (derived knowledge pipeline)
• Supports long-term knowledge curation goals

[durapensa]

Implementation sequence for Knowledge Graph milestone:

1. Implement knowledge distillation tool (kd) #9 - Knowledge distillation tool (kd) - Extract key insights from raw events
2. Build derived knowledge persistence pipeline #10 - Derived knowledge persistence - Store distilled insights
3. Build Unified Distilled Knowledge Graph with Human/AI Source Fusion #18 - Unified KG design (THIS ISSUE) - Architecture and schema
4. Implement SQLite-based distilled knowledge graph with minimal schema #19 - SQLite implementation - Build the actual knowledge graph
5. Claude-Claude dialogue experiment: Discovering conceptual attractors #20 - Claude dialogue experiments - Advanced KG applications

Dependencies:

• Implement SQLite-based distilled knowledge graph with minimal schema #19 depends on Build Unified Distilled Knowledge Graph with Human/AI Source Fusion #18 (need design before implementation)
• Claude-Claude dialogue experiment: Discovering conceptual attractors #20 can use infrastructure from Create conversation test harness for automated testing #1 (conversation harness)
• Distillation tools from Implement SQLite-based distilled knowledge graph with minimal schema #19 fulfill the "missing tools" from closed Find and implement missing background analysis tools (deep connections, etc.) #16

[durapensa]

Note: This issue represents the evolution of earlier concepts from:

• Implement knowledge distillation tool (kd) #9 (knowledge distillation tool) - Early file-based distillation approach
• Build derived knowledge persistence pipeline #10 (derived knowledge persistence) - File-based concept/connection storage

Key ideas adapted from those issues:

• Time-based filtering for analysis
• Concept extraction as core functionality
• Incremental updates to avoid full regeneration
• Integration with existing query tools
• Connection/relationship mapping between concepts

This unified knowledge graph design provides a more sophisticated implementation using SQLite instead of flat files.

[durapensa]

Status Update: Unified Knowledge Graph Partially Complete

Significant progress has been made on the unified knowledge graph system. Here's what's currently implemented:

✅ Completed Components

• Dual-Write System - review-findings writes to both summary (approved.jsonl) and detailed storage
• Source Attribution - Human vs AI contribution tracking with metadata
• Rejection Learning - Rejected findings stored in derived/rejected.jsonl with reasons
• Basic Distillation - Concept extraction and weight calculation working
• Provenance Tracking - Full source event attribution in approved findings
• Query Integration - Search across both raw events and distilled concepts

📋 Remaining Work

• Advanced Concept Deduplication - Clustering similar concepts with embeddings
• Sophisticated Weight Fusion - Complex algorithms for human/AI weight blending
• Temporal Evolution Tracking - Concept strength changes over time
• Cross-Reference Optimization - Bidirectional linking between events and concepts
• Confidence Calibration - Better confidence scoring for AI contributions

🔄 Implementation Strategy

The foundation is solid with working dual-write, source attribution, and basic fusion. Next phase should focus on:

1. Enhanced Deduplication using concept similarity algorithms
2. Advanced Weight Calculation with temporal decay and confidence factors
3. Query Optimization for unified concept/event searches

🎯 Current Capabilities

• Human insights and AI findings are properly attributed and weighted
• Rejected findings provide feedback for analysis improvement
• Basic concept distillation creates unified knowledge representation
• Provenance is maintained throughout the pipeline

The system provides a working foundation for unified knowledge management with clear areas for enhancement.