nm: various additions to ADMs and some fixes - #104
Conversation
Introduce a new "sys" Application Data Model for the NM agent/manager that reports host operating-system resource statistics: CPU count and utilization, load averages, physical/swap memory, disk capacity, open/ max file descriptors, process count, and uptime. Register the ADM via dtn_sys_init() in adm_init.c and list it in nm/doc/adms/index.json. Wire the new sources into both the autotools build (Makefile.am) and the development build (Makefile.dev). Also adds nm-unit coverage (tests/nm-unit/dotest.c) exercising every sys collector for non-NULL typed values plus sanity bounds on Linux hosts.
Expose six new externally-defined data items on the dtn/ion/ionadmin
ADM so a manager can monitor a node's memory pressure:
working_memory_{total,free,used} - ionwm partition (psm_usage)
heap_{total,free,used} - SDR heap (sdr_usage)
Free space is reported as small-pool + large-pool + never-allocated
regions, matching ionadmin's own accounting. All values are UVAST
bytes. Heap figures are read inside a read-only SDR transaction;
both getters return zero when the partition/SDR is unavailable.
agent->rpts and agent->tbls use auto-growing vectors, so vec_push() never reported "full" and the old "discard on full" path was dead: retained reports/tables accumulated without bound and leaked ION working memory. Add vec_pop_front() (insertion-order FIFO removal, permitted on stack-mode vectors) and use it in rx_data_rpt/rx_data_tbl to evict the oldest entry once AGENT_DEF_NUM_RPTS/_TBLS is reached. Also fix the rx_data_tbl unknown-sender warning to print senderEid.name rather than the eid_t struct.
Add scalar EDDs to the DTN/bp_agent ADM for the node's lifetime bundle-flow counters, each as both a bundle count and a byte count: sourced, forwarded, transmitted, received, delivered and expired (EDDs 0x1d-0x28). ION already tracks these but only via the by-priority map EDDs (0x09-0x0c), The new collectors mirror libbpP.c reportStateStats(): sourceStats/xmitStats /recvStats are summed across the three priority tallies, forwarded/expired read the BP_DB_FWD_OKAY/BP_DB_EXPIRED dbStats tallies, and delivered is aggregated over every endpoint of every scheme.
sysinfo(2)'s freeram counts only wholly-unused pages and ignores the page cache and other reclaimable memory, badly underestimating free RAM (and overestimating used RAM). Read the kernel's MemAvailable estimate from /proc/meminfo instead, falling back to freeram on kernels too old to publish it.
All byte-valued EDDs (mem/swap/disk total/free/used) now report bytes instead of KiB, so the ADM no longer mixes orders of magnitude. The EDD ids, getters, manager metadata, JSON and unit test are renamed from the _kb suffix to _bytes accordingly.
zebastian
changed the title
|
see also prepared integration to the munin stats monitoring tool in a timeshifted way here: |
|
…restart Resume the rotation sequence across restarts: agent_next_log_num() scans the log directory and continues after the highest existing file number instead of reopening "0.log" in append mode and clobbering the prior run's output. Add a per-agent retention limit on rotated report/table log files so a long-running Manager doesn't accumulate them unbounded. New -N / --log-max-files option (default 100, 0 = unlimited) sets the cap; on each rotation agent_prune_logs() deletes any of the agent's files whose rotation number is more than max_files behind the newest. Document -N in nm_mgr.pod and the usage text.
|
Hi @zebastian , Will need to take some time to delve into this. Its certainly desirable information so I imagine that it has been considered. Will need to consider this after some research. Also.. am curious about your testing methodology. |
|
thanks @SkyDeBaun regarding host level metrics defined in IETF: the only non-outdated draft i could find with specific edds is: draft-birrane-dtn-adm-agent draft-birrane-dtn-adm-agent-05 which are the ones the currently present adms implement, but those are all ion-dtn internal metrics. Maybe Mr Birrane can give some insight on this, or even rate the suggestions in this pull request. regarding test setup (used for the screenshot):
|
|
@zebastian - follows is some relevant information you may find useful. APL has pivoted once more from NM/AMP to ANMS. The IETF standards have changed (i.e. ANMS v1.0 & ION are not currently in alignment with those changes. Now.. ANMS v1.0 (a containerized NM management "interface" - supporting a GUI and API tightly coupled with ION) could issue commands and request reports to/from remote ION nodes (to include the current 4.2.0). ANMS v2.0 has moved to become implementation agnostic. But.. thats not necessarily problematic because it has its own ARI processing library (and each implementation is expected to have their own implementation-specific ADM. However, the required C bindings needed to interface with ION (via ANMS' CACE and distinct ION separated agent and manager services) remains to be.. developed. In any case see the DTNMA Tools repo here (has links to updated draft IETF docs): And for your further elucidation: I'll try to make time soon to perform an investigation into your proposed updates here. |
|
@zebastian Glad to see this kind of integration! My suggestion is to create a host-os resources ADM. I think this is needed and like your approach (to the extend I have reviewed to date). I would encourage you to join the next DTNWG meeting in July (it will be taking place in Vienna, but you can also join remotely) and to register and put comments in the IETF DTNWG mailing list, which is open to all. It is important to note that - as part of the IETF standards process - we are making syntactic updates to the YANG encoding of ADMs at the request and direction of the NETMOD and NETCONF WG's. @SkyDeBaun Just to be clear APL did not pivot from NM/AMP to ANMS. Standards related to DTN network management are being advanced in the IETF and as part of the process it was determined that DTN network management should be branded as "DTNMA" the architecture for which is published in RFC 9675: https://datatracker.ietf.org/doc/rfc9675/. Within that ecosystem, AMP is a bundle protocol binding of ARIs. ANMS is an open-source, NASA-supplied reference implementation of these standards. |
|
@edbirrane Appreciate the precise architectural breakdown and the RFC link! Glad to keep the terminology straight—things are moving fast out here. @zebastian Thanks for detailing the KVM and Munin setup—that bridge approach is a really slick way to visualize node telemetry. |
|
@SkyDeBaun @edbirrane |
|
@zebastian I still need to find some time to delve into this PR. I think the memory monitoring features alone are quite nice.. and as you say, such capabilities provide much insight (vs raw data). |
3 participants
sys:
Introduce a new "sys" Application Data Model for the NM agent/manager that reports host operating-system resource statistics: CPU count and utilization, load averages, physical/swap memory, disk capacity, open/ max file descriptors, process count, and uptime.
sdr heap and wm:
add working-memory and SDR heap usage EDDs to ionadmin ADM
fix memory leak for continious metric collection:
bound retained reports/tables with FIFO eviction
add more bundle metrics:
expose BP bundle-flow totals (count + bytes) as DTN/bp_agent EDDs
sys refinements:
gather stats always in byte (not in kb)
better mem usage estimation