feat: v4.0.0, [email protected] (#119)

This upgrades go-libddwaf to v4.0.0, introducing libddwaf 1.24.1 and the
new `Builder` pattern for maintaing WAF configuration and vending Handle
instances.

This gets rid of older, deprecated APIs while introducing new ones. This
is a breaking upgrade from v3.

Author: RomainMuller

.github/workflows/_test_bare_metal.yml

@@ -33,3 +33,6 @@ jobs:
         run: ./.github/workflows/ci.sh
         env:
           DD_APPSEC_WAF_LOG_LEVEL: ${{ matrix.waf-log-level }}
+          # We need a GITHUB_TOKEN in order to access the latest release of the AppSec rules from
+          # the DataDog/appsec-rules repository.
+          GITHUB_TOKEN: ${{ secrets.ACCESS_RULES_GITHUB_TOKEN }}

.github/workflows/test.yml

@@ -16,6 +16,9 @@ jobs:
   bare-metal:
     name: GitHub Runner
     uses: ./.github/workflows/_test_bare_metal.yml
+    # Needs secret access so it can access a GITHUB_TOKEN to verify the builder works with the
+    # latest AppSec rules package.
+    secrets: inherit
   containerized:
     name: Containerized
     uses: ./.github/workflows/_test_containerized.yml

README.md

@@ -6,7 +6,7 @@ It consists of 2 separate entities: the bindings for the calls to libddwaf, and
 An example usage would be:
 
 ```go
-import waf "github.com/DataDog/go-libddwaf/v3"
+import waf "github.com/DataDog/go-libddwaf/v4"
 
 //go:embed
 var ruleset []byte
@@ -18,11 +18,19 @@ func main() {
         panic(err)
     }
 
-    wafHandle, err := waf.NewHandle(parsedRuleset, "", "")
+    builder, err := waf.NewBuilder("", "")
+    if err != nil {
+        panic(err)
+    }
+    _, err := builder.AddOrUpdateConfig(parsedRuleset)
     if err != nil {
         panic(err)
     }
 
+    wafHandle := builder.Build()
+    if wafHandle == nil {
+        panic("WAF handle is nil")
+    }
     defer wafHandle.Close()
 
     wafCtx := wafHandle.NewContext()
@@ -36,7 +44,7 @@ func main() {
 }
 ```
 
-The API documentation details can be found on [pkg.go.dev](https://pkg.go.dev/github.com/DataDog/go-libddwaf/v3).
+The API documentation details can be found on [pkg.go.dev](https://pkg.go.dev/github.com/DataDog/go-libddwaf/v4).
 
 Originally this project was only here to provide CGO Wrappers to the calls to libddwaf.
 But with the appearance of `ddwaf_object` tree like structure,
@@ -65,17 +73,18 @@ Note that:
 
 The WAF bindings have multiple moving parts that are necessary to understand:
 
-- Handle: a object wrapper over the pointer to the C WAF Handle
-- Context: a object wrapper over a pointer to the C WAF Context
+- `Builder`: an object wrapper over the pointer to the C WAF Builder
+- `Handle`: an object wrapper over the pointer to the C WAF Handle
+- `Context`: an object wrapper over a pointer to the C WAF Context
 - Encoder: its goal is to construct a tree of Waf Objects to send to the WAF
-- CGORefPool: Does all allocation operations for the construction of Waf Objects and keeps track of the equivalent go pointers
 - Decoder: Transforms Waf Objects returned from the WAF to usual go objects (e.g. maps, arrays, ...)
 - Library: The low-level go bindings to the C library, providing improved typing
 
 ```mermaid
 flowchart LR
+    START:::hidden -->|NewBuilder| Builder -->|Build| Handle
 
-    START:::hidden -->|NewHandle| Handle -->|NewContext| Context
+    Handle -->|NewContext| Context
 
     Context -->|Encode Inputs| Encoder
 
@@ -86,38 +95,23 @@ flowchart LR
     Handle -->|Decode Init Errors| Decoder
 
     Context -->|Run| Library
-    Context -->|Store Go References| CGORefPool
-
-    Encoder -->|Allocate Waf Objects| TempCGORefPool
-
-    TempCGORefPool -->|Copy after each encoding| CGORefPool
+    Encoder -->|Allocate Waf Objects| runtime.Pinner
 
     Library -->|Call C code| libddwaf
 
     classDef hidden display: none;
 ```
 
-### CGO Reference Pool
-
-The cgoRefPool type is a pure Go pointer pool of `ddwaf_object` C values on the Go memory heap.
-the `cgoRefPool` go type is a way to make sure we can safely send Go allocated data to the C side of the WAF
-The main issue is the following: the `WafObject` uses a C union to store the tree structure of the full object,
-union equivalent in go are interfaces and they are not compatible with C unions. The only way to be 100% sure
-that the Go `WafObject` struct has the same layout as the C one is to only use primitive types. So the only way to
-store a raw pointer is to use the `uintptr` type. But since `uintptr` do not have pointer semantics (and are just
-basically integers), we need another method to store the value as Go pointer because the GC will delete our data if it
-is not referenced by Go pointers.
-
-That's where the `cgoRefPool` object comes into play: all new `WafObject` elements are created via this API which is especially
-built to make sure there is no gap for the Garbage Collector to exploit. From there, since underlying values of the
-`wafObject` are either arrays of WafObjects (for maps, structs and arrays) or string (for all ints, booleans and strings),
-we can store 2 slices of arrays and use `unsafe.KeepAlive` in each code path to protect them from the GC.
+### `runtime.Pinner`
 
-All these objects stored in the reference pool need to live throughout the use of the associated Waf Context.
+When passing Go values to the WAF, it is necessary to make sure that memory remains valid and does
+not move until the WAF no longer has any pointers to it. We do this by using a `runtime.Pinner`.
+Persistent address data is added to a `Context`-associated `runtime.Pinner`; while ephemeral address
+data is managed by a transient `runtime.Pinner` that only exists for the duration of the call.
 
 ### Typical call to Run()
 
-Here is an example of the flow of operations on a simple call to Run():
+Here is an example of the flow of operations on a simple call to `Run()`:
 
 - Encode input data into WAF Objects and store references in the temporary pool
 - Lock the context mutex until the end of the call

alignement_test.go

@@ -6,13 +6,13 @@
 // Purego only works on linux/macOS with amd64 and arm64 from now
 //go:build (linux || darwin) && (amd64 || arm64) && !go1.25 && !datadog.no_waf && (cgo || appsec)
 
-package waf
+package libddwaf
 
 import (
 	"testing"
 
-	"github.com/DataDog/go-libddwaf/v3/internal/bindings"
-	"github.com/DataDog/go-libddwaf/v3/internal/unsafe"
+	"github.com/DataDog/go-libddwaf/v4/internal/bindings"
+	"github.com/DataDog/go-libddwaf/v4/internal/unsafe"
 
 	"github.com/ebitengine/purego"
 	"github.com/stretchr/testify/require"
@@ -31,52 +31,52 @@ func TestWafObject(t *testing.T) {
 	lib := wafLib.Handle()
 
 	t.Run("invalid", func(t *testing.T) {
-		var actual bindings.WafObject
-		expected := bindings.WafObject{
-			Type: bindings.WafInvalidType,
+		var actual bindings.WAFObject
+		expected := bindings.WAFObject{
+			Type: bindings.WAFInvalidType,
 		}
 		r1, _, _ := purego.SyscallN(getSymbol(t, lib, "ddwaf_object_invalid"), unsafe.PtrToUintptr(&actual))
 		require.NotEqualValues(t, 0, r1)
 		require.Equal(t, expected, actual)
 	})
 
 	t.Run("int", func(t *testing.T) {
-		var actual bindings.WafObject
+		var actual bindings.WAFObject
 		r1, _, _ := purego.SyscallN(getSymbol(t, lib, "ddwaf_object_signed"), unsafe.PtrToUintptr(&actual), 42)
 		require.NotEqualValues(t, 0, r1)
 		require.EqualValues(t, 42, actual.Value)
-		require.EqualValues(t, bindings.WafIntType, actual.Type)
+		require.EqualValues(t, bindings.WAFIntType, actual.Type)
 	})
 
 	t.Run("uint", func(t *testing.T) {
-		var actual bindings.WafObject
+		var actual bindings.WAFObject
 		r1, _, _ := purego.SyscallN(getSymbol(t, lib, "ddwaf_object_unsigned"), unsafe.PtrToUintptr(&actual), 42)
 		require.NotEqualValues(t, 0, r1)
 		require.EqualValues(t, 42, actual.Value)
-		require.EqualValues(t, bindings.WafUintType, actual.Type)
+		require.EqualValues(t, bindings.WAFUintType, actual.Type)
 	})
 
 	t.Run("string", func(t *testing.T) {
-		var actual bindings.WafObject
-		r1, _, _ := purego.SyscallN(getSymbol(t, lib, "ddwaf_object_string"), unsafe.PtrToUintptr[bindings.WafObject](&actual), unsafe.PtrToUintptr[byte](unsafe.Cstring("toto")))
+		var actual bindings.WAFObject
+		r1, _, _ := purego.SyscallN(getSymbol(t, lib, "ddwaf_object_string"), unsafe.PtrToUintptr(&actual), unsafe.PtrToUintptr(unsafe.Cstring("toto")))
 		require.NotEqualValues(t, 0, r1)
 		require.Equal(t, "toto", unsafe.Gostring(unsafe.Cast[byte](actual.Value)))
-		require.EqualValues(t, bindings.WafStringType, actual.Type)
+		require.EqualValues(t, bindings.WAFStringType, actual.Type)
 	})
 
 	t.Run("padding", func(t *testing.T) {
-		var actual [3]bindings.WafObject
+		var actual [3]bindings.WAFObject
 		r1, _, _ := purego.SyscallN(getSymbol(t, lib, "ddwaf_object_string"), unsafe.PtrToUintptr(&actual[0]), unsafe.PtrToUintptr(unsafe.Cstring("toto1")))
 		require.NotEqualValues(t, 0, r1)
 		require.Equal(t, "toto1", unsafe.Gostring(unsafe.Cast[byte](actual[0].Value)))
-		require.EqualValues(t, bindings.WafStringType, actual[0].Type)
+		require.EqualValues(t, bindings.WAFStringType, actual[0].Type)
 		r1, _, _ = purego.SyscallN(getSymbol(t, lib, "ddwaf_object_string"), unsafe.PtrToUintptr(&actual[1]), unsafe.PtrToUintptr(unsafe.Cstring("toto2")))
 		require.NotEqualValues(t, 0, r1)
 		require.Equal(t, "toto2", unsafe.Gostring(unsafe.Cast[byte](actual[1].Value)))
-		require.EqualValues(t, bindings.WafStringType, actual[1].Type)
+		require.EqualValues(t, bindings.WAFStringType, actual[1].Type)
 		r1, _, _ = purego.SyscallN(getSymbol(t, lib, "ddwaf_object_string"), unsafe.PtrToUintptr(&actual[2]), unsafe.PtrToUintptr(unsafe.Cstring("toto3")))
 		require.NotEqualValues(t, 0, r1)
 		require.Equal(t, "toto3", unsafe.Gostring(unsafe.Cast[byte](actual[2].Value)))
-		require.EqualValues(t, bindings.WafStringType, actual[2].Type)
+		require.EqualValues(t, bindings.WAFStringType, actual[2].Type)
 	})
 }

builder.go

@@ -0,0 +1,136 @@
+// Unless explicitly stated otherwise all files in this repository are licensed
+// under the Apache License Version 2.0.
+// This product includes software developed at Datadog (https://www.datadoghq.com/).
+// Copyright 2016-present Datadog, Inc.
+
+package libddwaf
+
+import (
+	"errors"
+	"fmt"
+	"runtime"
+
+	"github.com/DataDog/go-libddwaf/v4/internal/bindings"
+)
+
+// Builder manages an evolving WAF configuration over time. Its lifecycle is
+// typically tied to that of a remote configuration client, as its purpose is to
+// keep an up-to-date view of the current coniguration with low overhead. This
+// type is not safe for concurrent use, and users should protect it with a mutex
+// or similar when sharing it across multiple goroutines. All methods of this
+// type are safe to call with a nil receiver.
+type Builder struct {
+	handle bindings.WAFBuilder
+}
+
+// NewBuilder creates a new [Builder] instance. Its lifecycle is typically tied
+// to that of a remote configuration client, as its purpose is to keep an
+// up-to-date view of the current coniguration with low overhead. Returns nil if
+// an error occurs when initializing the builder. The caller is responsible for
+// calling [Builder.Close] when the builder is no longer needed.
+func NewBuilder(keyObfuscatorRegex string, valueObfuscatorRegex string) (*Builder, error) {
+	if ok, err := Load(); !ok {
+		return nil, err
+	}
+
+	var pinner runtime.Pinner
+	defer pinner.Unpin()
+	hdl := wafLib.BuilderInit(newConfig(&pinner, keyObfuscatorRegex, valueObfuscatorRegex))
+
+	if hdl == 0 {
+		return nil, errors.New("failed to initialize the WAF builder")
+	}
+
+	return &Builder{handle: hdl}, nil
+}
+
+// Close releases all resources associated with this builder.
+func (b *Builder) Close() {
+	if b == nil || b.handle == 0 {
+		return
+	}
+	wafLib.BuilderDestroy(b.handle)
+	b.handle = 0
+}
+
+var (
+	errUpdateFailed  = errors.New("failed to update WAF Builder instance")
+	errBuilderClosed = errors.New("builder has already been closed")
+)
+
+// AddOrUpdateConfig adds or updates a configuration fragment to this [Builder].
+// Returns the [Diagnostics] produced by adding or updating this configuration.
+func (b *Builder) AddOrUpdateConfig(path string, fragment any) (Diagnostics, error) {
+	if b == nil || b.handle == 0 {
+		return Diagnostics{}, errBuilderClosed
+	}
+
+	if path == "" {
+		return Diagnostics{}, errors.New("path cannot be blank")
+	}
+
+	var pinner runtime.Pinner
+	defer pinner.Unpin()
+
+	encoder := newMaxEncoder(&pinner)
+	frag, err := encoder.Encode(fragment)
+	if err != nil {
+		return Diagnostics{}, fmt.Errorf("could not encode the config fragment into a WAF object; %w", err)
+	}
+
+	var diagnosticsWafObj bindings.WAFObject
+	defer wafLib.ObjectFree(&diagnosticsWafObj)
+
+	res := wafLib.BuilderAddOrUpdateConfig(b.handle, path, frag, &diagnosticsWafObj)
+
+	var diags Diagnostics
+	if !diagnosticsWafObj.IsInvalid() {
+		// The Diagnostics object will be invalid if the config was completely
+		// rejected.
+		diags, err = decodeDiagnostics(&diagnosticsWafObj)
+		if err != nil {
+			return diags, fmt.Errorf("failed to decode WAF diagnostics: %w", err)
+		}
+	}
+
+	if !res {
+		return diags, errUpdateFailed
+	}
+	return diags, nil
+}
+
+// RemoveConfig removes the configuration associated with the given path from
+// this [Builder]. Returns true if the removal was successful.
+func (b *Builder) RemoveConfig(path string) bool {
+	if b == nil || b.handle == 0 {
+		return false
+	}
+
+	return wafLib.BuilderRemoveConfig(b.handle, path)
+}
+
+// ConfigPaths returns the list of currently loaded configuration paths.
+func (b *Builder) ConfigPaths(filter string) []string {
+	if b == nil || b.handle == 0 {
+		return nil
+	}
+
+	return wafLib.BuilderGetConfigPaths(b.handle, filter)
+}
+
+// Build creates a new [Handle] instance that uses the current configuration.
+// Returns nil if an error occurs when building the handle. The caller is
+// responsible for calling [Handle.Close] when the handle is no longer needed.
+// This function may return nil.
+func (b *Builder) Build() *Handle {
+	if b == nil || b.handle == 0 {
+		return nil
+	}
+
+	hdl := wafLib.BuilderBuildInstance(b.handle)
+	if hdl == 0 {
+		return nil
+	}
+
+	return wrapHandle(hdl)
+}