GSoC 2025: Enhanced SwitchMap-NG Scalability #323
abbi4code
started this conversation in
GSoC Project Reports
Replies: 11 comments 1 reply
-
Progress Summary - Week 1-2Prototype Repository: SNMP PollerWorking VideoScreencast.from.2025-06-17.09-16-08.webmThings I have already implemented
Technical Architecture OverviewThe system follows a modular design with clear separation of concerns:
DocumentationI have added the complete project documentation in the prototype repo:
Current Implementation StatusCompleted: ✅
In Progress/Planned:
For detailed setup instructions, see README.md |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
also I will attach a full functional video of the prototype here asap |
Beta Was this translation helpful? Give feedback.
-
Upcoming Work - Week 3-4Planned Development Goals
|
Beta Was this translation helpful? Give feedback.
0 replies
-
Progress Summary - Week 3-4Prototype Repository: SNMP PollerKey Implementations CompletedEnhanced Data Persistence & Reliability (Prototype)
Started Migration from
|
Beta Was this translation helpful? Give feedback.
0 replies
-
Progress Summary - Weeks 5-7: Async SNMP MigrationComplete Async SNMP Infrastructure ImplementationPR: #328Async SNMP Manager (
Async Polling Infrastructure
Partial MIB Layer Conversion✅ System Layer (Active)
Current Migration Status: ~40% Complete✅ Fully Async Components:
🔄 In Progress:
⏳ Pending:
|
Beta Was this translation helpful? Give feedback.
0 replies
-
Weeks 8-9: Async MIB Conversion & Polling ImprovementsAsynchronous MIB MigrationSuccessfully converted all interface-related MIB modules from synchronous to asynchronous operations: Generic MIBs:
Cisco Vendor MIBs:
Juniper Vendor MIBs:
Critical Bug FixesFixed multiple polling stability issues that were causing freezes and errors:
Performance Optimization
|
Beta Was this translation helpful? Give feedback.
0 replies
-
Weeks 10-11: Architecture Modernization & Async RefactorLegacy Code RemovalComplete cleanup of synchronous SNMP architecture: PR: #334
Multiprocessing to Async MigrationReplaced the old multiprocessor-based polling architecture with asyncio to work efficiently with our curr architecture: poll.py Transformation:
Key Changes: async def devices(max_concurrent_devices=None):
device_semaphore = asyncio.Semaphore(max_concurrent_devices)
async with aiohttp.ClientSession() as session:
tasks = [device(meta, device_semaphore, session) for meta in arguments]
await asyncio.gather(*tasks, return_exceptions=True)SNMP Poller Integrationpoller.py, snmp_info.py & snmp_manager.py:
Database Integration ImprovementsMAC Address Decoding:
OUI Database Updates:
Minor Improvements
|
Beta Was this translation helpful? Give feedback.
0 replies
-
Weeks 12-13: Added system resource monitoring for network devices:PR: #344New SystemStat Entity:
Cisco Monitoring (CISCO-PROCESS-MIB):
Juniper Monitoring (JUNIPER-MIB):
Database Schema UpdatesSystemStat Table Operations:
GraphQL API ExpansionNew Query Support:
Ingest Pipeline:
Data Normalization & Resolution
Code Quality Improvements
|
Beta Was this translation helpful? Give feedback.
0 replies
-
Weeks 14-15: Interface Metrics & GraphQL EnhancementsGraphQL Query for Device LookupImplemented hostname-based device querying: PR: #347New Query Function:
Implementation: def resolve_device_by_hostname(self, info, hostname=None, **kwargs):
if not hostname:
return Device.query.filter(Device.enabled == 1)
hostname_bytes = hostname.encode("utf-8")
query = Device.query.filter(
Device.hostname == hostname_bytes,
Device.enabled == 1
).order_by(Device.ts_created.desc())
return queryInterface Statistics ExpansionExtended interface monitoring with comprehensive packet and byte counters: New Metrics Added to L1Interface:
Existing Metrics:
Database Schema Updatesmodels.py Changes:
Named Tuples Updated:
SNMP Polling ExtensionInterface Data Collection:
Data Pipeline: IL1Interface(
ifin_octets=interface.get("ifInOctets"),
ifout_octets=interface.get("ifOutOctets"),
ifin_nucast_pkts=interface.get("ifInNUcastPkts"),
ifout_nucast_pkts=interface.get("ifOutNUcastPkts"),
ifout_errors=interface.get("ifOutErrors"),
ifout_discards=interface.get("ifOutDiscards"),
)GraphQL Schema UpdatesL1InterfaceAttribute Enhancements:
|
Beta Was this translation helpful? Give feedback.
0 replies
-
Weeks 16-17: Bug Fixes & College mid semOUI Database Handling ImprovementsFixed critical bugs in OUI (Organizationally Unique Identifier) update logic: One of the Previous Issue: # Old code - redundant insert attempts
inserts.append({
"oui": (None if bool(row.oui) is False else row.oui.encode()),
"organization": (None if bool(row.organization) is False
else row.organization.encode()),
"enabled": int(bool(row.enabled) is True),
})Fixed Implementation: # New code - conditional field inclusion
insert_dict = {"enabled": int(bool(row.enabled) is True)}
# Only add oui if it's not None
if bool(row.oui) is True:
insert_dict["oui"] = row.oui.encode()
# Only add organization if it's not None
if bool(row.organization) is True:
insert_dict["organization"] = row.organization.encode()
inserts.append(insert_dict)Bug Fix Benefits
Academic Commitments
|
Beta Was this translation helpful? Give feedback.
0 replies
-
Weeks 18: Test Coverage & ValidationTest Coverage Increased
PR: #350SNMP Manager Testing (
Async Poller Testing (
SNMP Info Testing (
LLDP MIB Testing (
Additional MIB Test Coverage:
Async Testing PatternsMock Object Usage: mock_snmp_object = MagicMock()
mock_snmp_object.swalk = AsyncMock(return_value=mock_data)
async def run_test():
result = await query.method()
self.assertIsInstance(result, dict)
mock_snmp_object.swalk.assert_called_once()
asyncio.run(run_test())Validation ImprovementsStricter OID Validation: |
Beta Was this translation helpful? Give feedback.
0 replies
-
Weeks 19 & 20: Automated setup script for complete Switchmap deploymentGoals achievedPR: #360Screen.Recording.2025-10-24.at.5.32.08.PM.movScreen.Recording.2025-10-24.at.5.34.25.PM.mov
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi everyone,
As part of GSoC 2025, I'm working on 'Enhanced SwitchMap-NG Scalability' . This project aims to significantly improve SwitchMap-NG's performance and reliability, especially in distributed environments.
The core problem is the current system's struggles with data loss and performance bottlenecks when many pollers connect directly to the API server.
My proposed solution introduces a hierarchical architecture with regional aggregators. These aggregators will act as intermediaries, receiving data from pollers via
ZeroMQ, storing it resiliently inSQLite, and then forwarding it to the central API server. This will reduce server load and prevent data loss.Key project areas include:
Google Summer of Code 2025 Final Report
Project: Enhanced SwitchMap-NG Scalability
Contributor: Abhishek Raj
Mentors: Aashima Wadhwa, Dominic Mills
Organization: The Palisadoes Foundation
Duration: June–October 2025
Project Goals
When I began this project, Switchmap-NG faced several critical challenges:
Project recap
My GSoC project focused on modernizing Switchmap-NG through four key initiatives:
1. Legacy Synchronous Architecture:
EasySNMPwithPySNMP'sasync API for non-blocking operations, enabling efficient polling of thousands of devices.2. Removed resource intensive multiprocessing:
3. Feature Enhancements
4. Developer Experience
Completed Work
poll.pyto use asyncio with semaphore-based concurrency controlmac_utils.pymoduletest_mib_ipv6.py,test_mib_lldp.py,test_mib_junipervlan.py,test_poller.py,test_snmp_info.py,test_snmp_manager.pyandtest_async_poll.py.Impact: Reduced setup time from 30-50 minutes to 3-5 minutes, eliminated 90% of setup-related issues, and dramatically improved contributor onboarding experience.
Current State
Production-Ready Features
Async SNMP Infrastructure
Multi-Vendor Support
System Monitoring
Interface Statistics
GraphQL API
Developer Experience
Performance Metrics
Code Quality
What Remains
While the project has achieved its primary goals and delivered a production-ready async monitoring system, there are still exciting opportunities for future enhancements that will extend Switchmap-NG's capabilities:
Distributed Architecture with Store-and-Forward: The current architecture works well for centralized deployments, but could be enhanced for distributed scenarios where pollers may have intermittent connectivity to the central API server.
**100% Test Coverage: **: Test coverage has been significantly improved from around 60% to 85% with extensive tests added across major SNMP modules. However, achieving full coverage remains challenging since some core SNMP polling components involve complex asynchronous flows and low-level network interactions. Ongoing work focuses on building reliable test harnesses and mocks for these components to reach the 100% goal.
Challenges and Lessons Learned
Lesson: Never assume external data is clean. Validate everything at the boundaries, it’s far easier to fail early than debug downstream corruption.
From Multiprocessing to Async Concurrency
The legacy multiprocessing model didn’t translate directly to async patterns. Replacing process pools with semaphores and refactoring shared state into async-safe structures took significant design effort. Adopting async context managers and using aiohttp for non-blocking HTTP calls ultimately simplified resource management.
Lesson: Asyncio and multiprocessing solve different problems understanding when and how to use each is crucial for scalability and stability.
Complexity of Async Testing
Lesson: Async systems demand a new mindset for testing. Once the right patterns are in place, async tests can be just as reliable and even faster to execute.
Conclusion
Acknowledgements
I'm excited to collaborate and welcome your feedback and questions on these enhancements.
Best regards,
Abhishek Raj
GSoC'25 Participant
Beta Was this translation helpful? Give feedback.
All reactions