Release Date: April 30, 2026
Dataset: 405 open-source repositories across 30+ programming languages
DOI: [Pending Zenodo assignment]
GitHub: https://github.com/manojmallick/sigmap-benchmark-suite
🚀 Announcing: SigMap Benchmark Suite Extended Dataset — 405 Repositories
We're excited to release the extended version of our large-scale AI context extraction benchmark, expanding from 240 to 405 open-source repositories.
What is SigMap? A comprehensive empirical study evaluating how effectively context extraction works across diverse programming languages and project types. Critical for optimizing LLM token usage in software engineering.
The Numbers: 📊 405 repositories | 30+ languages | 2,025+ benchmark operations 💾 1,664,157 source files | 108M+ lines of code 🎯 96.2% average token reduction (consistent with published version) ✅ 99.6% execution success rate | 100% data completeness
What's New in Extended (405 repos): ✅ 165 additional repositories for better generalization ✅ More comprehensive language & framework coverage ✅ Enhanced monorepo analysis (45 identified) ✅ Validates that published findings hold at scale ✅ Identical methodology & hardware (c2-standard-8)
Key Findings:
- Language matters more than size: code organization drives compression
- 45 monorepos identified (18.8%) → 2-3% improvement with specialized handling
- Framework choices significantly impact extraction effectiveness
- Results consistent across 240-repo (published) and 405-repo versions
What's Included: 📦 Dataset exports (CSV, JSON, JSONL, SQL) 📖 Complete research paper + methodology documentation 🔧 All reproducibility scripts (clone, benchmark, finalize) 📋 Step-by-step reproduction guide (~2 hours execution time) 📜 CC-BY-4.0 license — free for research & commercial use
Impact: This dataset enables AI/ML researchers to:
- Develop improved context extraction strategies
- Understand language-specific compression patterns
- Optimize for enterprise monorepos
- Benchmark against real-world codebases
- Build better AI-assisted software engineering tools
Related: Primary dataset (240 repos): https://doi.org/10.5281/zenodo.19898842 Extended version (405 repos): Ready for Zenodo submission
All materials available on GitHub with complete documentation.
📥 Download: https://github.com/manojmallick/sigmap-benchmark-suite
📚 Documentation: See zenodo_submission_extended_405/ folder
🤝 License: CC-BY-4.0 — cite and share freely
#AI #LLM #SoftwareEngineering #OpenSource #Research #Benchmark #ContextExtraction #Programming #DataScience
🧵 Excited to announce the SigMap Benchmark Suite Extended Dataset (405 repos)!
We just released 165 additional open-source repositories to our AI context extraction benchmark, taking it from 240 to 405 diverse projects across 30+ programming languages.
What started as a research question → became a comprehensive empirical study.
🧵/1
Token Reduction Across Languages:
- Python: 96.2% ± 1.8% (most consistent)
- Go: 95.2% ± 2.1%
- Rust: 94.8% ± 2.4%
- Java: 94.5% ± 2.6%
- JavaScript: 92.1% ± 4.2% (highest variability)
The data is crystal clear: language & code organization matter more than project size.
🧵/2
Key Finding: We identified 45 monorepos (18.8%) in the dataset. With specialized handling, you can improve context extraction by 2-3%.
This is actionable intelligence for teams building AI-assisted dev tools.
🧵/3
Dataset Size: 📊 405 repositories 💾 1.6M+ source files 📈 108M+ lines of code 🎯 2,025+ benchmark operations ⚡ Executed in 1h 20m on standard hardware
Reproducible, documented, ready to use.
🧵/4
What's Included: ✅ Dataset (CSV, JSON, JSONL, SQL) ✅ Research papers ✅ Methodology docs ✅ Reproducibility scripts ✅ CC-BY-4.0 license
Everything you need to benchmark your own context extraction tool.
GitHub: https://github.com/manojmallick/sigmap-benchmark-suite
🧵/5
Most Important Finding: Extended dataset (405 repos) shows identical 96.2% avg token reduction as published (240 repos).
This validates that our methodology is robust & findings generalize across larger samples.
Science! 🔬
🧵/6
For tool builders: This dataset tells you exactly where to optimize context extraction for maximum impact.
Monorepos are the leverage point. Framework choices matter. Language idioms shape compression.
Use this to build better tools.
🧵/1
Why This Matters: LLM token limits are a constraint. Context extraction is the solution.
96.2% token reduction = you can analyze ~25x more code with the same token budget.
That's the difference between "useless" and "enterprise-ready" AI tooling.
🧵/2
Reproducible & Open:
- All scripts included
- Hardware specs documented (c2-standard-8)
- Expected variance < 2%
- CC-BY-4.0 license
- Free for research + commercial use
Run it yourself. Verify the results. Extend it.
🧵/3
Who should use this? ✅ AI/ML researchers ✅ LLM engineers ✅ Dev tool builders ✅ Software engineering researchers ✅ Academic teams studying code compression
If you work on code understanding + LLMs, this is your benchmark dataset.
🧵/1
Next Steps:
- Download from GitHub
- Read the research papers
- Run reproduction scripts
- Adapt for your own tools
- Cite in your work
Everything documented. No gatekeeping.
GitHub: https://github.com/manojmallick/sigmap-benchmark-suite
🧵/2
Help Us Spread It: If you find this useful, please:
- ⭐ Star the repo
- 🔗 Share with colleagues
- 📚 Cite in your work
- 💬 Tell us how you're using it
This is open science. The more people benefit, the better.
🧵/3
Title: [P] SigMap Benchmark Suite Extended: 405-Repository AI Context Extraction Dataset
Post:
We're announcing the extended version of the SigMap Benchmark Suite — a comprehensive evaluation of context extraction strategies across 405 open-source repositories spanning 30+ programming languages.
Dataset Overview:
- 405 diverse repositories (240 published + 165 new)
- 2,025+ benchmark operations
- 1.6M+ source files
- 108M+ lines of code
- 99.6% success rate
- 100% data completeness
Key Findings:
- Language Consistency: 96.2% average token reduction across all 405 repos (validates 240-repo published findings)
- Monorepo Leverage: 45 monorepos identified (18.8%) → 2-3% improvement with specialized handling
- Framework Impact: Code organization and idioms matter more than project size
- Language Specifics:
- Python: Most consistent (96.2% ± 1.8%)
- JavaScript: Highest variability (92.1% ± 4.2%)
- Go/Rust: Strong performers (95%+ range)
What's Included:
- Complete datasets (CSV, JSON, JSONL, SQL)
- Research papers (primary + extended)
- Full methodology documentation
- Reproducibility package with scripts
- CC-BY-4.0 license
Resources:
- GitHub: https://github.com/manojmallick/sigmap-benchmark-suite
- Primary Dataset DOI: https://doi.org/10.5281/zenodo.19898842
- Extended Version: Ready for Zenodo submission
Reproducibility:
- Hardware specs documented (Google Cloud c2-standard-8)
- Expected execution: ~1h 20m
- Expected variance: < 2%
- All parameters included
This dataset enables researchers to develop improved context extraction strategies and provides an empirical foundation for AI-assisted software engineering research.
Questions? Check the extensive documentation or reach out!
Title: SigMap Benchmark: 405 Open-Source Python Projects Analyzed for AI Context Extraction
Post:
We analyzed 405 open-source projects (including 45 Python projects) to understand how effectively context extraction works for AI models.
TL;DR:
- Python achieves 96.2% token reduction (most consistent across languages)
- With smart context extraction, you can analyze ~25x more Python code with same token budget
- Complete dataset, scripts, and methodology are open-source (CC-BY-4.0)
Why It Matters for Python Devs:
- AI Tool Development: If you're building IDE plugins, code analysis tools, or AI-assisted development platforms, this tells you exactly what compression rates to expect
- LLM Integration: Token limits are a real constraint. This data shows you what's possible
- Python-Specific Patterns: Django, FastAPI, NumPy patterns analyzed alongside 40+ other Python projects
What You Get: ✅ Python-specific metrics and patterns ✅ Comparison with other languages ✅ Scripts to run benchmark on your own code ✅ Documentation on methodology ✅ Real-world data from 405 active projects
Links:
- GitHub: https://github.com/manojmallick/sigmap-benchmark-suite
- Extended Dataset: zenodo_submission_extended_405/
- License: CC-BY-4.0 (free for any use)
Would love to hear how you might use this data!
Title: 405-Repository Empirical Study: How Context Extraction Works Across Programming Languages
Post:
We conducted a large-scale empirical study evaluating context extraction (the ability to compress code while maintaining semantic meaning) across 405 diverse open-source projects.
Study Design:
- 405 repositories (30+ languages)
- 2,025+ benchmark operations
- Identical methodology across all languages
- Reproducible (all scripts included)
Key Insights for Engineering Teams:
- Monorepos are the leverage point — 45 identified, 2-3% improvement possible with specialized handling
- Framework choice impacts compression — React vs Vue patterns, Django vs FastAPI patterns, etc.
- Language consistency — Results hold across 240→405 repos, suggesting robust methodology
- Real-world scale — 1.6M files, 108M LOC of actual production code
Enterprise Relevance: If your team is building:
- IDE plugins with AI assistance
- Code review automation
- Documentation generation
- Knowledge base systems
- AI-powered debugging tools
...this dataset tells you what's achievable and where to optimize.
What's Available:
- Complete reproducible study (scripts included)
- Dataset exports in multiple formats
- Methodology documentation
- Hardware specs (easy to replicate)
- Open license (CC-BY-4.0)
Useful for anyone thinking about LLM integration in software development.
GitHub: https://github.com/manojmallick/sigmap-benchmark-suite
Title: SigMap Extended Dataset (405 repos) Now Available — Feedback & Use Cases
Post:
👋 Hi everyone!
We're excited to share the SigMap Benchmark Suite Extended Dataset — a comprehensive evaluation of AI context extraction across 405 open-source repositories.
What is it? A large-scale empirical study measuring how effectively you can compress code for LLM consumption while preserving semantic meaning. Critical for building scalable AI-assisted development tools.
Dataset Stats:
- 405 repositories (30+ languages)
- 2,025+ benchmark operations
- 1.6M+ source files
- 99.6% success rate
- Reproducible (all scripts included)
Key Findings:
- 96.2% average token reduction (consistent across dataset size)
- Monorepos: 45 identified, 2-3% improvement opportunity
- Language matters more than project size
- Results validate published findings at scale
What's Included:
- ✅ Dataset (CSV, JSON, JSONL, SQL)
- ✅ Research papers
- ✅ Reproducibility scripts
- ✅ Complete documentation
- ✅ CC-BY-4.0 license
Files:
zenodo_submission_extended_405/— Complete extended dataset packagezenodo_submission/— Published version (240 repos)- README.md — Overview
- All scripts in
/scripts/folder
Questions & Feedback:
- How are you planning to use this dataset?
- What additional metrics would be helpful?
- Issues with reproducibility?
- Ideas for extensions?
Let's discuss! 🚀
Title: SigMap Benchmark Suite: 405-Repository Analysis of AI Context Extraction [pdf link if available]
Comments to Prepare For: (Draft responses to anticipated questions)
Q: Why does this matter? A: LLM token limits are a hard constraint. AI-assisted dev tools need to extract maximum meaning from minimum tokens. This dataset quantifies what's achievable (96.2% reduction on average) and identifies leverage points (monorepos, language patterns).
Q: How reproducible is this? A: Fully reproducible. All scripts included, hardware specs documented (c2-standard-8), expected variance < 2%. Takes ~1h 20m to run benchmark on 405 repos.
Q: Is this only for academic use? A: No. CC-BY-4.0 license — free for any use (commercial included). Just cite the work.
Q: What about other languages? A: Covers 30+ languages. Python (45 repos), JavaScript (40), Java (20), Go (15), Rust (15), C/C++ (15), and many others.
Q: Can I use this for my own tool? A: Yes. The scripts and methodology are included. You can run the benchmark on your own codebases or adapt it for different tools.
Title: SigMap Benchmark: How Much Code Can AI Tools Really Compress?
Introduction:
LLM token limits are a constraint. If you're building AI-assisted development tools,
you've faced this problem: how much code context can you actually fit into your model?
We analyzed 405 open-source repositories to find out. Here's what we learned.Body Sections:
- The Problem — Token limits force hard choices
- The Study — 405 repos, 30+ languages, reproducible
- The Findings — 96.2% token reduction, monorepo patterns
- The Data — What's included, how to use it
- Practical Applications — Build better tools with this data
- Next Steps — Run it yourself, extend it
Closing: "This dataset is open. Use it. Build on it. Share your results."
Subject: SigMap Benchmark Suite Extended (405 repos) — Now Available for Research & Development
To: Research contacts, tool developers, AI/ML engineers
Body:
Dear [Researcher/Developer/Team],
We're announcing the SigMap Benchmark Suite Extended Dataset — a comprehensive evaluation of AI context extraction across 405 open-source repositories.
If you work on AI-assisted software engineering, code understanding, or LLM integration, this dataset is relevant to you.
Quick Facts:
- 405 repositories (30+ languages)
- 2,025+ benchmark operations
- 96.2% token reduction (validated at scale)
- Complete reproducibility package
- CC-BY-4.0 license (free for research & commercial use)
Download: GitHub: https://github.com/manojmallick/sigmap-benchmark-suite
Primary Dataset (Published): DOI: https://doi.org/10.5281/zenodo.19898842
Extended Dataset: Ready for Zenodo submission
What's Included: ✅ Complete datasets (CSV, JSON, JSONL, SQL) ✅ Research papers & methodology ✅ Reproducibility scripts (run it yourself) ✅ Step-by-step documentation ✅ Hardware specs for replication
Key Findings:
- Language organization matters more than project size
- Monorepos offer 2-3% improvement opportunity
- Results consistent across 240→405 repos
- Actionable intelligence for tool optimization
How to Use:
- Download dataset
- Read research papers
- Run reproduction scripts
- Adapt for your own tools
- Cite in your work
Questions? Check the documentation or reach out.
Best regards, [Your Name]
🎉 **NEW: SigMap Benchmark Suite Extended Dataset (405 repos)**
We just released the extended version of our AI context extraction benchmark!
📊 **By The Numbers:**
- 405 open-source repositories
- 30+ programming languages
- 2,025+ benchmark operations
- 1.6M source files analyzed
- 96.2% token reduction (consistent!)
🔬 **Key Finding:**
Monorepos offer 2-3% improvement potential. Language patterns matter more than size.
📦 **Everything Included:**
✅ Datasets (4 formats)
✅ Research papers
✅ Reproducibility scripts
✅ Full documentation
✅ CC-BY-4.0 license
🔗 **Download:** https://github.com/manojmallick/sigmap-benchmark-suite
Questions? Thread 👇
Research Announcement: SigMap Benchmark Study Extends to 405 Repositories
We've expanded our empirical study of context extraction from 240 to 405 repositories.
Extended dataset validates that published findings are robust and generalizable.
Results:
- 96.2% avg token reduction (same as 240-repo version)
- 45 monorepos identified (18.8%)
- Language-specific patterns quantified
- Reproducible methodology documented
Useful for:
- LLM context optimization research
- Tool development benchmarking
- Software engineering empirical studies
- AI-assisted development tool evaluation
All materials open-source (CC-BY-4.0)
Dataset Entry:
Title: SigMap Benchmark Suite — Large-Scale AI Context Extraction Dataset (405 Repositories)
Abstract:
Comprehensive evaluation of context extraction effectiveness across 405 diverse
open-source repositories spanning 30+ programming languages. 2,025+ benchmark operations
with 99.6% success rate. Includes complete reproducibility package.
Key Metrics:
- 405 repositories
- Average token reduction: 96.2%
- Success rate: 99.6%
- Data completeness: 100%
- Languages: 30+
Files:
- Datasets (CSV, JSON, JSONL, SQL)
- Research papers
- Methodology documentation
- Reproducibility scripts
- License: CC-BY-4.0
GitHub: https://github.com/manojmallick/sigmap-benchmark-suite
Subject: [ANNOUNCE] SigMap Benchmark Suite — 405-Repository Dataset for AI Context Extraction Research
Appropriate Lists:
- ACM SIGSOFT (software engineering)
- IEEE Software Engineering
- ArXiv (cs.SE, cs.AI)
- SigPlan (programming languages)
- Research institution mailing lists
- LinkedIn — Professional, detailed version
- Twitter/X — Thread 1 (announcement), Thread 2 (technical), Thread 3 (CTA)
- Reddit — r/MachineLearning, r/Python, r/SoftwareEngineering
- GitHub Discussions — Community feedback request
- HackerNews — Technical announcement
- Dev.to — Article format
- Email — Researchers & collaborators
- Slack/Discord — Community announcements
- Papers with Code — Dataset entry
- Academic Mailing Lists — ACM SIGSOFT, IEEE, etc.
Posting Timeline:
- Day 1: GitHub, Twitter Thread 1
- Day 2: LinkedIn, Reddit, Dev.to
- Day 3: HackerNews, Slack/Discord
- Day 4: Academic mailing lists
- Ongoing: GitHub discussions, Papers with Code
Key Message to Emphasize:
"405 repositories validated the findings from our 240-repo study. This is robust, reproducible science — open for everyone to use."