Improved Custom Block version with, for example, a payload length?

[winking324]

Hi,

I wrote a simple ntar reader in C++, and how to directly get the length of custom data when parsing the Custom Block?

My repo: ntar-reader and this part of the parsing code is in ntar-reader/src/block_custom.cpp.

[guyharris]

If a block supports options, you cannot have the length of the non-options part of the block contents be derived solely from the block total length, as that length includes the length of the options - you must have some other mechanism that allows the length of the non-options part of the block to be determined.

Thus, the format for a given PEN's custom block must either:

1. not support any options, including OPT_COMMENT, so that the length of the Custom Data field is (block total length - 16) or
2. be in a format that contains data allowing, for all blocks for that PEN, the length of the Custom Data field to be computed.

This means that readers will not be able to process arbitrary custom blocks unless they provide a plugin mechanism that allows, for an arbitrary PEN value, a plugin to be provided that understands the custom block formats for that PEN. That also means that if such a plugin is provided, it must be updated whenever the format of its PEN's custom block is extended to, for example, add a new subtype of custom block.

(And, yes this means that, given that Wireshark does not provide such a plugin mechanism, Wireshark, "out of the box", cannot process arbitrary custom blocks; it can only process custom blocks for which code in wiretap/pcapng.c includes support code.

And, yes, that needs to be fixed.)

[mcr]

Guy Harris ***@***.***> wrote: If a block supports options, you cannot have the length of the non-options part of the block contents be derived solely from the block total length, as that length includes the length of the options - you must have some other mechanism that allows the length of the non-options part of the block to be determined. Ick. Can we deprecate custom blocks with options, and create something new that doesn't suck?

[guyharris]

Ick.

That was my reaction when I realized this. I wish I'd realized it when it was first proposed.

Can we deprecate custom blocks with options, and create something new that
doesn't suck?

Or, at least, that has fewer problems.

If we insist that "Custom Blocks Done Right Blocks" pick a standard byte order for multi-byte integral and floating-point fields, and go with IEEE 754 for floating-point fields (VAX is dead, Data General is dead, 36-bit and 48-bit mainframes have other problems in a power-of-2-word-size world, and IBM mainframes now do IEEE 754), that should, I think, make it possible for code that knows nothing about the innards of a given Custom Blocks Done Right Block pass it through even if it's running on a machine with the opposite byte order from the machine that last wrote the file.

[mcr]

Guy Harris ***@***.***> wrote: If we insist that "Custom Blocks Done Right Blocks" pick a standard byte order for multi-byte integral and floating-point fields, and go Sounds good to me.

[guyharris]

If we insist that "Custom Blocks Done Right Blocks" pick a standard byte order for multi-byte integral and floating-point fields,

Section 5.4 "Endianness Issues" of the current version of the I-D says:

Therefore, the vendor should either encode the Custom Data of their Custom Blocks and Custom Options, the Option Type and Option Length fields of options in Custom Blocks, and the PEN field of Custom Options in Custom Blocks in a consistent manner, such as always in big-endian or always in little-endian format, regardless of the host platform's endianness, or should encode some flag in the Custom Data payload to indicate in which endianness the rest of the payload is written.

This is currently just a recommendation, not a requirement, but it's a recommendation that any particular custom block either pick a standard byte order or include a byte-order magic number or something such as that, which is sufficient to allow a program ignorant of a particular Custom Block's format to blindly pass that block through, even if that program is running on a host with a byte order different from the byte order of the file that it's reading, and have it readable by a program that does understand that Custom Block's format.

Perhaps it should be made into a requirement by replacing "should" with "MUST".

[guyharris]

As for the length of the Custom Data field, if, for a given PEN, the Custom Block with that PEN can contain options, the Custom Data field needs to contain information from which can be derived the total length of that field. This is not necessarily a "length of the Custom Data" field; see, for example, the Enhanced Packet Block, which has a Captured Packet Length field. The value of that field, plus 20 for the fields that precede the Packet Data subfield (and then rounded up to a multiple of 4), is the length of the block payload that precedes the option.

I.e., that doesn't require a "length of Custom Data" field, just something from which that length can be derived.

The question is whether, for some piece of pcapng-reading code, that code needs to be able to parse the options of a Custom Block the format of whose Custom Data it doesn't understand.

In the case of Wireshark, which is a single project containing both a capture-file-reading library and a capture-file-record-dissecting application (two of them, in fact, Wireshark and the command-line TShark), it's sufficient that the project as a whole - or for a third-party plugin for that project - understand the Custom Block format for a given PEN in order to do anything useful with those Custom Blocks. The Wireshark project working on a way to allow a plugin (or a set of related plugins) to fully handle Custom Blocks for a given PEN.

If, instead, you have a pcapng-reading library, atop which pcapng-reading programs can be built, there's now a question of how the work of handling Custom Blocks should be split up (well, that issue is even in Wireshark, but it's not split up between projects).

For example, in a future libpcap that supports pcapng, and a future tcpdump that supports the libpcap APIs that support pcapng, how much of the work should libpcap do, and how should it do that?

If tcpdump didn't support, say, some particular organization's Custom Block, any parsing libpcap does of that Custom Block, even if it's only whatever option parsing it can do without knowledge of the structure of any Custom Options in that block, wouldn't be of much use.

If it did support it, it might want to provide its own code to parse the block, in which case it would know how to find the options (so that it doesn't parse past the end of the Custom Data), and could call a libpcap routine to parse those options, providing to libpcap a routine or routines to handle any Custom Options it cares about.

In such a scheme, libpcap itself wouldn't have to know how to find the options for an arbitrary Custom Block. Even if libpcap were to support plugins for handling Custom Blocks and Options, the Custom Block handler, not the core libpcap code, would be responsible for finding the options in a Custom Block.

[guyharris]

As noted in this Wireshark issue comment, this reply, and this reply to that reply, it could be useful to allow a pcapng reader that doesn't understand the format of some particular Custom Block to at least parse the options of that block to report comments, so, if we had a payload length field after the PEN field, that would be possible.

It might also be useful to specify a 4-byte "subtype" field, in cases where more than one type of block is used by the organization who owns the PEN.