v1.11.2

Bug fixes

• GH-1860: Fix parsing for vectors of literals.

  This was broken in two ways:

  1. with the (LITERAL)[] syntax, the parser would not recognize literals using type constructors
  2. with the syntax LITERAL[], we'd try to store the parsed value into a vector

• GH-1847: Fix resynchronization issue with trimmed input.

  When input had been trimmed, View::advanceToNextData could end up returning a view starting ahead of the valid area.

• GH-1852: Fix skip with units.

  For unit parsing with skip, we would create a temporary instance but wouldn't properly initialize it, meaning for example that parameters weren't available. We now generally fully initialize any destination, even if temporary.

v1.11.1

Bug fixes

• GH-1831: Fix optimizer regression.

  We were no longer marking types as used that are referenced through a type name.

• GH-1823: Don't qualify magic linker symbols with C++ namespace.

  We need them at their original values because that's what the runtime lbirary is hard-coded to expect.

• Fix use of move'd from variable.

  Function parameters still shadown members in C++. This is a fixup of c3abbbe.

• Fix undefined shifts of 32bit integer in toInt().

  1U is 32bit on a 64bit system and shifting it by more than 31 bits is undefined. The following does currently produce -4294967296 instead of -1:

  b"\xff\xff\xff\xff".to_int(spicy::ByteOrder::Big)

• Fix to_uint(ByteOrder) for empty byte ranges.

  to_uint() and to_int() for empty byte ranges throw when attempting to convert printable decimals to integers. Do the same for the byte order versions. The assumption is that it is really an error when the user calls to_int() or to_uint() on an empty byte range.

• GH-1817: Prevent null ptr dereference when looking on nodes without Scope.

• GH-1815: Disallow expanding limited Views again with limit.

  The documented semantics of View::limit are that it creates a new view with equal or smaller size. In contrast to that we would still have allowed to expand views with more calls limit again as well.

  This patch changes the implementation of View::limit so it can only ever make a View smaller.

  We also tweak the implementation of the check for consumed &size when used together with &eod: if the &size was already nested in a limited view a larger &size value could previously extend the view so the &eod effectively was ignored. Since we now do not extend the View anymore we need to only activate the check for consumed &size if &eod was not specified since in this case the user communicated that they are fine with consuming less data.

• GH-1810: Fix nested look-ahead switches.

• Remember normalized paths when checking for duplicate files in driver.

  While we ignore duplicate files it was still possible to erroneously add the same file multiple times to a compilation. Catch this trivial case.

• GH-1462: Remember files processed by the driver.

  We did this previously but stopped doing it with #1462.

• Remove a few value copies.

• GH-1813: Fix equality implementation of module UID.

  We already computed a unique ID value for each module to allow declaring the same ID name multiple times; we however did not consistently use that value in the implementation of module::UID equality and hash operators which is addressed by this patch.

v1.11.0

New Functionality

• GH-3779: Add %sync_advance hook.

  This adds support for a new unit hook:

  on %sync_advance(offset: uint64) {
      ...
  }
  

  This hook is called regularly during error recovery when synchronization skips over data or gaps while searching for a valid synchronization point. It can be used to check in on the synchronization to, e.g., abort further processing if it just keeps failing. offset is the current position inside the input stream that synchronization just skipped to.

  By default, "called regularly" means that it's called every 4KB of input skipped over while searching for a synchronization point. That value can be changed by setting a unit property %sync-advance-block-size = <number of bytes>.

  As an additional minor tweak, this also changes the name of what used to be the __gap__ profiler to now be called __sync_advance because it's profiling the time spent in skipping data, not just gaps.

• Add unit method stream() to access current input stream, and stream method statistics() to retrieve input statistics.

  This returns a struct of the following type, reflecting the input seen so far:

  type StreamStatistics = struct {
      num_data_bytes: uint64;     ## number of data bytes processed
      num_data_chunks: uint64;    ## number of data chunks processed, excluding empty chunks
      num_gap_bytes: uint64;      ## number of gap bytes processed
      num_gap_chunks: uint64;     ## number of gap chunks processed, excluding empty chunks
  };
  

• GH-1750: Add to_real method to bytes. This interprets the data as representing an ASCII-encoded floating point number and converts that into a real. The data can be in either decimal or hexadecimal format. If it cannot be parsed as either, throws an InvalidValue exception.

• GH-1608: Add get_optional method to maps.
  This returns an optional value either containing the map's element for the given key if that entry exists, or an unset optional if it does not.

• GH-90/GH-1733: Add result and spicy::Error types to Spicy to facilitate error handling.

Changed Functionality

• The Spicy compiler has become a bit more strict and is now rejecting some ill-defined code constructs that previous versions ended up letting through. Specifically, the following cases will need updating in existing code:
  • Identifiers from the (internal) hilti:: namespace are no longer accessible. Usually you can just scope them with spicy:: instead.
  • Previous versions did not always enforce constness as it should have. In particular, function parameters could end up being mutable even when they weren't declared as inout. Now inout is required for supporting any mutable operations on a parameter, so make sure to add it where needed.
  • When using unit parameters, the type of any inout parameters now must be unit itself. To pass other types into a unit so that they can be modified by the unit, use reference instead of inout. For example, use type Foo = unit(s: sink&) instead of type Foo = unit(inout: sink). See https://docs.zeek.org/projects/spicy/en/latest/programming/parsing.html#unit-parameters for more.
• The Spicy compiler new uses a more streamlined storage and access scheme to represent source code. This speeds up work up util C++ source translation (e.g., faster time to first error message during development).
• spicyc options -c and -l no longer support compiling multiple Spicy source files to C++ code individually to then build them all together. This was a rarely used feature and actually already broken in some situations. Instead, use spicyc -x to produce the C++ code for all needed Spicy source files at once. -c and -l remain available for debugging purposes.
• The spicyc option -P now requires a prefix argument that sets the C++ namespace, just like -x <prefix> does. This is so that the prototypes match the actual code generated by -x. To get the same identifiers as before, use an empty prefix (-P "").
• GH-1763: Restrict initialization of const values to literals. This means that e.g., const values cannot be initialized from other const values or function calls anymore.
• result and network are now keywords and cannot be used anymore as user-specified indentifiers.
• GH-1661: Deprecate usage of &convert with &chunked.
• GH-1657: Reduce data copying when passing data to the driver.
• GH-1501: Improve some error messages for runtime parse errors.
• GH-1655: Reject joint usage of filters and look-ahead.
• GH-1675: Extend runtime profiling to measure parser input volume.
• GH-1624: Enable optimizations when running spicy-build.

Bug fixes

• GH-1759: Fix if-condition with switch parsing.
• Fix Spicy's support for network type.
• GH-1598: Enforce that the argument new is either a type or a ctor.
• GH-1742, GH-1760: Unroll constructors of big containers in generated code. We previously would generate code which would be expensive to compiler for some compilers. We now generate more friendly code.
• GH-1745: Fix C++ initialization of global constants through global functions.
• GH-1743: Use a checked cast for map's in operator.
• GH-1664: Fix &convert typing issue with bit ranges.
• GH-1724: Fix skipping in size-constrained units. We previously could skip too much data if skip was used in a unit with a global &size.
• Fix incremental skipping. We previously would incorrectly compute the amount of data to skip which could have potentially lead to the parser consuming more data than available.
• GH-1586: Make skip productions behave like the production they are wrapping.
• GH-1711: Fix forwarding of a reference unit parameter to a non-reference parameter.
• GH-1599: Fix integer increment/decrement operators require mutable arguments.
• GH-1493: Support/fix public type aliases to units.

Documentation

• Add new section with guidelines and best practices. This focuses on performance for now, but may be extended with other areas alter. Much of the content was contributed by Corelight Labs.
• Fix documented type mapping for integers.
• Document generic operators.

v1.10.1

• Update CI setups.

• Fix repeated evaluations of &parse-at expression.

v1.9.1

• Drop ; after #pragma.

• Update CI setups.

• Fix repeated evaluations of &parse-at expression.

• Fix stray Python escape sequence.

• Drop freebsd-12 from CI.

• GH-1617: Fix handling of %synchronize-* attributes for units in lists.

  We previously would not detect %synchronize-at or %synchronize-from attributes if the unit was not directly in a field, i.e., we mishandled the common case of synchronizing on a unit in a list.

  With this patch we now handle these attributes, regardless of how the unit appears.

v1.8.4

• Drop ; after #pragma.

• Update CI setups.

• Fix repeated evaluations of &parse-at expression.

• Fix stray Python escape sequence.

• Fix skipping of literal fields with condition.

• Fix type of generated code for string::size.

  While we defined string's size operator to return an uint64 and documented that it returns the length in codepoints, not bytes, we still generated C++ code which worked on the underlying bytes (i.e., it directly invoked std::string::size instead of using hilti::rt::string::size).

v1.10.0

Changed Functionality

• Numerous improvements to improve throughput of generated parsers.

  For this release we have revisited the code typically generated for parsers and the runtime libraries they use with the goal of improving throughput of parsers at runtime. Coarsely summarized this work was centered around

  • reduction of allocations during parsing
  • reduction of data copies during parsing
  • use of dedicated, hand-check implementations for automatically generated code to avoid overhead from safety checks in the runtime libraries

  With these changes we see throughput improvements of some parsers in the range of 20-30%. This work consisted of numerous incremental changes, see CHANGES for the full list of changes.

• GH-1667: Always advance input before attempting resynchronization.

  When we enter resynchronization after hitting a parse error we previously would have left the input alone, even though we know it fails to parse. We then relied fully on resynchronization to advance the input.

  With this patch we always forcibly advance the input to the next non-gap position. This has no effect for synchronization on literals, but allows it to happen earlier for regular expressions.

• GH-1659: Lift requirement that bytes forwarded from filter be mutable.

• GH-1489: Deprecate &bit-order on bit ranges.

  This had no effect and allowing it may be confusing to users. Deprecate it with the idea of eventual removal.

• Extend location printing to include single-line ranges.

  For a location of, e.g., "line 1, column 5 to 10", we now print 1:5-1:10, whereas we used to print it as only 1:5, hence dropping information.

• GH-1500: Add += operator for string.

  This allows appending to a string without having to allocate a new string. This might perform better most of the time.

• GH-1640: Implement skipping for any field with known size.

  This patch adds skip support for fields with &size attribute or of builtin type with known size. If a unit has a known size and it is specified in a &size attribute this also allows to skip over unit fields.

Bug fixes

• GH-1605: Allow for unresolved types for set in operator.

• GH-1617: Fix handling of %synchronize-* attributes for units in lists.

  We previously would not detect %synchronize-at or %synchronize-from attributes if the unit was not directly in a field, i.e., we mishandled the common case of synchronizing on a unit in a list.

  We now handle these attributes, regardless of how the unit appears.

• GH-1585: Put closing of unit sinks behind feature guard.

  This code gets emitted, regardless of whether a sink was actually connected or not. Put it behind a feature guard so it does not enable the feature on its own.

• GH-1652: Fix filters consuming too much data.

  We would previously assume that a filter would consume all available data. This only holds if the filter is attached to a top-level unit, but in general not if some sub-unit uses a filter. With this patch we explicitly compute how much data is consumed.

• GH-1668: Fix incorrect data consumption for &max-size.

  We would previously handle &size and &max-size almost identical with the only difference that &max-size sets up a slightly larger view to accommodate a sentinel. In particular, we also used identical code to set up the position where parsing should resume after such a field.

  This was incorrect as it is in general impossible to tell where parsing continues after a field with &max-size since it does not signify a fixed view like &size. We now compute the next position for a &max-size field by inspecting the limited view to detect how much data was extracted.

• GH-1522: Drop overzealous validator.

  A validator was intended to reject a pattern of incorrect parsing of vectors, but instead ending up rejecting all vector parsing if the vector elements itself produced vectors. We dropped this validation.

• GH-1632: Fix regex processing using {n,m} repeat syntax being off by one

• GH-1648: Provide meaningful unit __begin value when parsing starts.

  We previously would not provide __begin when starting the initial parse. This meant that e.g., offset() was not usable if nothing ever got parsed.

  We now provide a meaningful value.

• Fix skipping of literal fields with condition.

• GH-1645: Fix &size check.

  The current parsing offset could legitimately end up just beyond the &size amount.

• GH-1634: Fix infinite loop in regular expression parsing.

Documentation

• Update documentation of offset().

• Fix docs namespace for symbols from filter module.

  We previously would document these symbols to be in spicy even though they are in filter.

• Add bitfield examples.

v1.8.3

• GH-1645: Fix &size check.

  The current parsing offset could legitimately end up just beyond the &size amount.

• GH-1617: Fix handling of %synchronize-* attributes for units in lists.

  We previously would not detect %synchronize-at or %synchronize-from attributes if the unit was not directly in a field, i.e., we mishandled the common case of synchronizing on a unit in a list.

  With this patch we now handle these attributes, regardless of how the unit appears.

v1.9.0

New Functionality

• GH-1468: Allow to directly access members of anonymous bitfields.

  We now automatically map fields of anonymous bitfields into their containing unit.

  type Foo = unit {
      : bitfield(8) {
          x: 0..3;
          y: 4..7;
      };
  
      on %done {
          print self.x, self.y;
      }
  };

• GH-1467: Support bitfield constants in Spicy for parsing.

  One can now define bitfield "constants" for parsing by providing integer expressions with fields:

  type Foo = unit {
    x: bitfield(8) {
      a: 0..3 = 2;
      b: 4..7;
      c: 7 = 1;
    };

  This will first parse the bitfield as usual and then enforce that the two bit ranges that are coming with expressions (i.e., a and c) indeed containing the expected values. If they don't, that's a parse error.

  We also support using such bitfield constants for look-ahead parsing:

  type Foo = unit {
    x: uint8[];
    y: bitfield(8) {
      a: 0..3 = 4;
      b: 4..7;
    };
  };

  This will parse uint8s until a value is discovered that has its bits set as defined by the bitfield constant.

  (We use the term "constant" loosely here: only the bits with values are actually enforced to be constant, all others are parsed as usual.)

• GH-1089, GH-1421: Make offset() independent of random access functionality.

  We now store the value returned by offset() directly in the unit instead of computing it on the fly when requested from cur - begin. With that offset() can be used without enabling random access functionality on the unit.

• Add support for passing arbitrary C++ compiler flags.

  This adds a magic environment variable HILTI_CXX_FLAGS which if set specifies compiler flags which should be passed during C++ compilation after implicit flags. This could be used to e.g., set defines, or set low-level compiler flags.

  Even with this flag, for passing include directories one should still use HILTI_CXX_INCLUDE_DIRS since they are searched before any implicitly added paths.

• GH-1435: Add bitwise operators &, |, and ^ for booleans.

• GH-1465: Support skipping explicit %done in external hooks.

  Assuming Foo::X is a unit type, these two are now equivalent:

  on Foo::X::%done   { }
  on Foo::X          { }

Changed Functionality

• GH-1567: Speed up runtime calls to start profilers.

• GH-1565: Disable capturing backtraces with HILTI exceptions in non-debug builds.

• GH-1343: Include condition in &requires failure message.

• GH-1466: Reject uses of self in unit &size and &max-size attribute.

  Values in self are only available after parsing has started while &size and &max-size are consumed before that. This means that any use of self and its members in these contexts would only ever see unset members, so it should not be the intended use.

• GH-1485: Add validator rejecting unsupported multiple uses of attributes.

• GH-1465: Produce better error message when hooks are used on a unit field.

• GH-1503: Handle anonymous bitfields inside switch statements.

  We now map items of anonymous bitfields inside a switch cases into the unit namespace, just like we already do for top-level fields. We also catch if two anonymous bitfields inside those cases carry the same name, which would make accesses ambiguous.

  So the following works now:

  switch (self.n) {
      0 -> : bitfield(8) {
          A: 0..7;
      };
      * -> : bitfield(8) {
          B: 0..7;
      };
  };

  Whereas this does not work:

  switch (self.n) {
      0 -> : bitfield(8) {
          A: 0..7;
      };
      * -> : bitfield(8) {
          A: 0..7;
      };
  };

• GH-1571: Remove trimming inside individual chunks.

  Trimming a Chunk (always from the left) causes a lot of internal work with only limited benefit since we manage visibility with a stream::View on top of a Chunk anyway.

  We now trimming only removes a Chunk from a Chain, but does not internally change individual the Chunk anymore. This should benefit performance but might lead to slightly increased memory use, but callers usually have that data in memory anyway.

• Use find_package(Python) with version.

  Zeek's configure sets Python_EXECUTABLE has hint, but Spicy is using find_package(Python3) and would only use Python3_EXECUTABLE as hint. This results in Spicy finding a different (the default) Python executable when configuring Zeek with --with-python=/opt/custom/bin/python3.

  Switch Spicy over to use find_package(Python) and add the minimum version so it knows to look for Python3.

Bug fixes

• GH-1520: Fix handling of spicy-dump --enable-print.
• Fix spicy-build to correctly infer library directory.
• GH-1446: Initialize generated struct members in constructor body.
• GH-1464: Add special handling for potential advance failure in trial mode.
• GH-1275: Add missing lowering of Spicy unit ctor to HILTI struct ctor.
• Fix rendering in validation of %byte-order attribute.
• GH-1384: Fix stringification of DecodeErrorStrategy.
• Fix handling of --show-backtraces flag.
• GH-1032: Allow using using bitfields with type declarations.
• GH-1484: Fix using of &convert on bitfields.
• GH-1508: Fix returned value for <unit>.position().
• GH-1504: Use user-inaccessible chars for encoding :: in feature variables.
• GH-1550: Replace recursive deletion with explicit loop to avoid stack overflow.
• GH-1549: Add feature guards to accesses of a unit's __position.

Documentation

• Move Zeek-specific documentation into Zeek documentation.
• Clarify error handling docs.
• Mention unit switch statements in conditional parsing docs.

v1.8.2

• GH-1571: Remove trimming inside individual chunks.

  Trimming Chunks (always from the left) causes a lot of internal work with only limited benefit since we manage visibility with stream::Views on top of Chunks anyway.

  This patch removes trimming inside Chunks so now any trimming only removes Chunks from Chains, but does not internally change individual Chunks anymore. This might lead to slightly increased memory use, but callers usually have that data in memory anyway.

• GH-1549: GH-1554: Fix potential infinite loop when trimming data before stream.

  Previously we would trigger an infinite loop if one tried to trim before the head chunk of a stream. In praxis this seem to have been no issue due to #1549 and us emitting way less calls to trim than possible.

  This patch adds an explicit check whether we need to trim anything, and exits the low-level function early for such cases.

• GH-1550: Replace recursive deletion with explicit loop to avoid stack overflow.

• GH-1549: Add feature guards to accesses of a unit's __position.

  Access of __position triggers a random access functionality. In order to distinguish our internal uses from accesses due to user code, most access in our generated code should be guarded with a feature constant (if or ternary).

  In this patch add proper guards for a couple instances where we did not do that correctly. That mishap caused all units with containers to be random access (even the root unit) which in turn could have lead to e.g., unbounded memory growth, or runtime overhead due to generation and execution of unneeded code, or expensive cleanup on very large untrimmed inputs.

• Artificially limit the number of open files.

  This works around a silent failure in reproc where it would refuse to run on systems which huge rlimits for the number of open files. We have seen this hit on huge production boxes.

• GH-1478: Add regression test for #1478.

• Add begin to parser state.

  This patch adds the current begin position to the parser state, and makes the corresponding changes to generated parser functions so it is passed down.

  We already modelled the semantic beginning of the input in the unit, but had no reliable way to keep this up-to-date across non-unit contexts like &parse-from. This would then for certain setups lead to generated code where input and position would point to different inputs which in turn caused offset (modelled as position - input) to be incorrect.

• Expand validator error message.

• Disable a few newer clang-tidy categories.

  The options disabled here and triggered in newer versions of clang-tidy.

• Drop -noall_load linker option.

  We added this linker option on macos. This option was already obsolete, e.g., in the ld manpage:

  -noall_load
          This is the default.  This option is obsolete.
  

  Newer versions of xcode do not know this option anymore and instead generate a hard error.

• Declare Spicy pygments extension as parallel-safe.

  We previously would not declare that the Spicy pygments highlighter is safe to execute in parallel (reading or writing of sources). Sphinx then assumed that the extension was not safe to run in parallel and instead ran jobs sequentially.

  This patch declares the extension as able to execute in parallel. Since the extension does not manage any external state this is safe.

• Use find_package(Python) with version.

  Zeek's configure sets Python_EXECUTABLE has hint, but Spicy is using find_package(Python3) and would only use Python3_EXECUTABLE as hint. This results in Spicy finding a different (the default) Python executable when configuring Zeek with --with-python=/opt/custom/bin/python3.

  Switch Spicy over to use find_package(Python) and add the minimum version so it knows to look for Python3.