Skip to content

QUICKSTART.md

Latest commit

 

History

History
236 lines (178 loc) · 6.39 KB

QUICKSTART.md

File metadata and controls

236 lines (178 loc) · 6.39 KB
Raw

ITC Quick Start Guide

Welcome to Information Transform Compression! Get started in 5 minutes.

Installation

# Clone and install
git clone https://github.com/makangachristopher/Information-Transform-Compression.git
cd Information-Transform-Compression
pip install -e .

# Verify installation
python -c "import torch; import transformers; print('✓ Dependencies OK')"

Your First Compression

Compress a Transformer (BERT/GPT-2)

# Using the CLI - Simplest method
python itc.py compress bert-base-uncased --preset balanced

# Output: bert-base-uncased-compressed.pt (4-7x smaller)

What happened?

  • Downloaded BERT from HuggingFace
  • Analyzed attention head importance
  • Pruned 25% of redundant heads
  • Pruned 25% of FFN neurons
  • Applied INT8 quantization
  • Saved compressed model

Compress a CNN

# Analyze a CNN with information-theoretic methods
python itc.py analyze examples.simple_model.SimpleModel \
    --input-shape "1,3,32,32" \
    --target 8.0 \
    --output compressed/

# Output: 8x compressed model with IDM analysis report

What happened?

  • Computed entropy, sensitivity, redundancy for each layer
  • Calculated Information Density Metric (IDM)
  • Allocated precision: FP16/INT8/INT4/Binary per layer
  • Applied mixed-precision quantization
  • Exported to ONNX and TorchScript

Python API

Transformer Compression

from compression_pipeline import compress_model

# One-function compression
model, stats = compress_model(
    'bert-base-uncased',
    head_prune_ratio=0.25,
    ffn_prune_ratio=0.25,
    quantize=True
)

print(f"Compression: {stats['total_compression']:.2f}x")
print(f"Time: {stats['compression_time']:.2f}s")
# Output: 4.96x compression in ~2.5s

CNN Compression

import torch
from passes.information_analysis.entropy import EntropyAnalyzer
from passes.information_analysis.sensitivity import SensitivityAnalyzer
from passes.information_analysis.redundancy import RedundancyDetector
from passes.information_analysis.density import InformationDensityMetric
from passes.adaptive_precision import AdaptivePrecisionAllocator
from passes.mixed_precision import MixedPrecisionQuantizer

model = YourModel()

# 1. Analyze information density
entropy_analyzer = EntropyAnalyzer()
entropy_map = entropy_analyzer.analyze_weight_entropy(model)

sensitivity_analyzer = SensitivityAnalyzer()
test_inputs = torch.randn(5, 3, 224, 224)
sensitivity_map = sensitivity_analyzer.profile_all_layers(model, test_inputs)

redundancy_detector = RedundancyDetector()
redundancy_map = redundancy_detector.compute_redundancy_score(model)

idm_analyzer = InformationDensityMetric()
idm_results = idm_analyzer.analyze_model(
    model, entropy_map, sensitivity_map, redundancy_map
)

# 2. Allocate precision based on IDM
allocator = AdaptivePrecisionAllocator(compression_target=8.0)
precision_map = allocator.allocate(model, idm_results)

# 3. Apply mixed-precision quantization
quantizer = MixedPrecisionQuantizer()
compressed_model = quantizer.quantize_model(model, precision_map)

# Result: 5-15x compression with <3% accuracy loss

Understanding the Output

Transformer Compression Statistics

Original parameters: 109,482,240
Compressed parameters: 88,232,448
Head pruning: 1.07x
FFN pruning: 1.15x
Quantization: 4.0x
Total compression: 4.96x
Processing time: 2.5s

CNN Analysis Report

Layer: conv1
  IDM: 0.000643 (Critical)
  Entropy: 7.5 bits
  Sensitivity: 0.9
  Redundancy: 0.05
  Allocated: 16-bit float16

Layer: fc3
  IDM: 0.000014 (Compressible)
  Entropy: 2.0 bits
  Sensitivity: 0.1
  Redundancy: 0.4
  Allocated: 4-bit int4

Common Use Cases

1. Deploy to Mobile

# Compress and export to ONNX for mobile
python itc.py compress bert-base-uncased --preset aggressive
python itc.py export bert-base-uncased-compressed.pt \
    --format onnx --output bert_mobile.onnx

2. Batch Compression

from compression_pipeline import compress_model

models = ['bert-base-uncased', 'roberta-base', 'distilbert-base-uncased']

for model_name in models:
    model, stats = compress_model(
        model_name,
        output_path=f"{model_name}-compressed.pt",
        verbose=False
    )
    print(f"{model_name}: {stats['total_compression']:.2f}x")

3. Custom Compression

# Fine-tune compression ratios
python itc.py compress gpt2 \
    --heads 0.3 \
    --ffn 0.4 \
    --no-quantize \
    --output custom_gpt2.pt

Next Steps

  1. Read the Documentation

  2. Explore Examples

    • examples/run_pipeline.py - Complete pipeline example
    • examples/mixed_precision_example.py - Mixed-precision demo
    • examples/simple_model.py - Example CNN model

  3. Run Tests

    pytest tests/ -v

  4. Join the Community

FAQ

Q: Which compression method should I use?

  • Transformers (BERT/GPT-2/RoBERTa): Use itc compress (structural pruning)
  • CNNs/ResNets: Use itc analyze (information-theoretic)

Q: How much accuracy will I lose?

  • Conservative preset: <1% accuracy loss
  • Balanced preset: <2% accuracy loss
  • Aggressive preset: <3% accuracy loss

Q: Can I compress custom models? Yes! ITC works with any PyTorch model. For transformers, use compress. For CNNs, use analyze.

Q: Do I need to fine-tune after compression? Not required, but fine-tuning for a few epochs can recover most accuracy loss.

Q: What's the Information Density Metric (IDM)? IDM = (Entropy × Sensitivity) / (Size × (1 + Redundancy))

It quantifies how much unique, critical information each layer contains, enabling intelligent precision allocation.

Getting Help

Happy compressing! 🎉