Expand UTScapy dissection corpus field coverage for tshark vs Scapy on existing pcaps (including enum-string validation)

[Copilot]

This PR introduces a reusable dissection-verification corpus for UTScapy that compares tshark field output against Scapy dissection on real pcaps. It establishes a mapping-driven framework so new protocol checks can be added by data, not custom test logic.

• What was added

  • New test corpus file: test/scapy/layers/dissection_corpus.uts
  • Generic comparison pipeline that:
    • loads packets from pcap with Scapy
    • extracts per-packet fields from tshark (-T fields)
    • compares fields via a mapping dictionary (tshark_field -> scapy layer.field)

• Mapping/comparison model

  • Supports direct field equality for scalar fields.
  • Supports per-field custom comparators for non-trivial representations.
  • Supports a list of compare functions per mapping entry (OR semantics), so a field can validate across multiple representations.
  • Adds reusable integer normalization/comparison for numeric fields across tshark/Scapy representations (including decimal/hex text forms).
  • Uses dedicated numeric comparison for tcp.flags.
  • Adds enum-string validation support by normalizing and comparing display-form enum values when applicable.

• Corpus entries (using existing Scapy pcaps)

  • test/pcaps/http_content_length.pcap
  • test/pcaps/netflowv9.pcap
  • test/pcaps/http_compressed.pcap
  • test/pcaps/ssh_ed25519.pcap
  • test/pcaps/ipfix.pcap

• Field coverage expanded to compare many more protocol fields

  • IP: ip.src, ip.dst, ip.version, ip.dsfield, ip.len, ip.id, ip.flags, ip.frag_offset, ip.ttl, ip.proto, ip.checksum
  • TCP (where applicable): tcp.srcport, tcp.dstport, tcp.seq, tcp.ack, tcp.window_size_value, tcp.checksum, tcp.urgent_pointer, tcp.flags
  • UDP (where applicable): udp.srcport, udp.dstport, udp.length, udp.checksum

• Hardening in helper logic

  • Mapping format validation (Layer.field shape checks).
  • Layer whitelist for controlled layer resolution.
  • Clear assertion errors for unsupported layer/field mappings.
  • Robust comparator behavior for empty/non-integer values and normalized numeric comparisons.
  • Comparator-list validation with clearer mapping-context error messages.

# corpus entry shape
{
    "pcap": "test/pcaps/http_content_length.pcap",
    "mapping": {
        "ip.src": {"scapy": "IP.src"},
        "ip.version": {"scapy": "IP.version", "compare": _compare_int_field},
        "ip.proto": {
            "scapy": "IP.proto",
            "compare": [_compare_int_field, _compare_enum_field],
        },
        "tcp.flags": {
            "scapy": "TCP.flags",
            "compare": [_compare_tcp_flags, _compare_enum_field],
        },
    },
}

[polybassa]

@copilot add a few example tests using existing pcap files

[Copilot]

@copilot add a few example tests using existing pcap files

Implemented in commit e47a9d0. I added three more corpus examples using existing pcaps: http_compressed.pcap, ssh_ed25519.pcap, and ipfix.pcap.

[polybassa]

@copilot compare as much fields as possible

[Copilot]

@copilot compare as much fields as possible

Implemented in commit c93fa4a. I expanded each corpus entry to compare many more fields: broad IP coverage (version, dsfield, len, id, flags, frag_offset, ttl, proto, checksum) plus TCP (seq, ack, window_size_value, checksum, urgent_pointer, flags) or UDP (length, checksum) where applicable.

[polybassa]

@copilot If fields contain enums as strings, validate those as well. Maybe allow a list of compare functions in the map.

[Copilot]

@copilot If fields contain enums as strings, validate those as well. Maybe allow a list of compare functions in the map.

Implemented in commit 472d912. I added enum-string validation support and updated the mapping model to accept a list of compare functions per field (OR semantics), then applied this to enum-prone fields like ip.flags, ip.proto, and tcp.flags across the corpus.

[polybassa]

@copilot move all helper functions in a python file inside test. Apply proper typing and flake8.

[Copilot]

@copilot move all helper functions in a python file inside test. Apply proper typing and flake8.

Done in commit c8d5b09. I moved the dissection corpus helper logic into test/scapy/layers/dissection_corpus_helpers.py, added type annotations, kept the .uts file focused on corpus data/imports, and ensured the new helper module is flake8-clean.

[polybassa]

@copilot fix the AI trailer reminder check

[Copilot]

@copilot fix the AI trailer reminder check

Fixed in d44b566. I updated .config/ci/check_commits.sh so commits authored by the Copilot bot without an AI-Assisted trailer are treated as reminder-only (non-failing), while non-bot commits still fail when the trailer is missing.

[polybassa]

@copilot one unit test failed the dissection test. Analyse the issue

[Copilot]

@copilot one unit test failed the dissection test. Analyse the issue

I analyzed the failing run: the dissection corpus test mismatched on tcp.seq in http_content_length.pcap (tshark='0' vs Scapy absolute sequence), caused by tshark using relative TCP sequence display in that environment. I fixed this in commit c7e4168 by switching the corpus mappings from tcp.seq/tcp.ack to tcp.seq_raw/tcp.ack_raw (still compared against TCP.seq/TCP.ack) for the TCP pcap entries.