fatal error: concurrent map writes when using sheriff.Marshal

[lucasoares]

Issue: fatal error: concurrent map writes when using sheriff.Marshal

Description

When calling sheriff.Marshal with certain configuration structs, the application panics with a fatal error: concurrent map writes.

In this particular case, my system doesn't call marshal concurrently on the same object. But even if it did, I don't understand how marshal could fail under concurrent calls, since the map being written appears to be newly created within marshal itself.

Full Stack Trace

fatal error: concurrent map writes

goroutine 5078 [running]:
internal/runtime/maps.fatal({0x15be215?, 0x17c?})
    runtime/panic.go:1046 +0x18
github.com/liip/sheriff.Marshal(0x2b3f040, {0x1433720?, 0xc000f217c0?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:124 +0x5fb
github.com/liip/sheriff.marshalValue(0x2b3f040, {0x135b240?, 0xc001793418?, 0x1389738?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:234 +0x1ea
github.com/liip/sheriff.marshalValue(0x2b3f040, {0x12bbea0?, 0xc0017c1f60?, 0x5?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:240 +0x4c5
github.com/liip/sheriff.Marshal(0x2b3f040, {0x1433e00?, 0xc0017c1f60?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:178 +0x84c
github.com/liip/sheriff.marshalValue(0x2b3f040, {0x1433e00?, 0xc001490160?, 0x5?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:234 +0x1ea
github.com/liip/sheriff.Marshal(0x2b3f040, {0x1469f00?, 0xc0013cb890?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:178 +0x84c
github.com/liip/sheriff.marshalValue(0x2b3f040, {0x1294e80?, 0xc000124d98?, 0x13919e0?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:234 +0x1ea
github.com/liip/sheriff.marshalValue(0x2b3f040, {0x12bab80?, 0xc0000e4d00?, 0x5?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:240 +0x4c5
github.com/liip/sheriff.Marshal(0x2b3f040, {0x149fce0?, 0xc0000e4d00?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:178 +0x84c
github.com/liip/sheriff.marshalValue(0x2b3f040, {0x149fce0?, 0xc001571b40?, 0x5?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:234 +0x1ea
github.com/liip/sheriff.Marshal(0x2b3f040, {0x1469fc0?, 0xc0017c1f40?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:178 +0x84c
github.com/liip/sheriff.marshalValue(0x2b3f040, {0x1469fc0?, 0xc001490140?, 0x5?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:234 +0x1ea
github.com/liip/sheriff.Marshal(0x2b3f040, {0x135b480?, 0xc001410d20?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:178 +0x84c
github.com/liip/sheriff.marshalValue(0x2b3f040, {0x1346a20?, 0xc00156c650?, 0x5f44205ea467d39f?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:234 +0x1ea
github.com/liip/sheriff.Marshal(0x2b3f040, {0x1299b80?, 0xc00156c650?})
    github.com/liip/sheriff@v0.11.1/sheriff.go:82 +0x25e
github.com/stilingue-inteligencia-artificial/firehose-sindicator-manager/internal/config.MarshalAll({0x135b480, 0xc001410d20})

Code Snippet

config.go:49

func MarshalAll(config any) (map[string]any, error) {
	data, err := sheriff.Marshal(allOptions, &config)

	if err != nil {
		return nil, fmt.Errorf("parsing all config data: %w", err)
	}

	if data == nil {
		return nil, nil
	}

	if dataMap, ok := data.(map[string]any); ok {
		return dataMap, nil
	}

	return nil, fmt.Errorf("invalid all config data format: %w", err)
}

Options used:

var allOptions = &sheriff.Options{
	Groups:          []string{"public", "secret"},
	IncludeEmptyTag: true,
}

Example Config Struct

type Config struct {
	Id              string            `json:"id,omitempty" bson:"id,omitempty"`
	Name            *string           `json:"name,omitempty" bson:"name"`
	Description     *string           `json:"description,omitempty" bson:"description"`
	Input           *Input            `json:"input,omitempty" bson:"input,omitempty" groups:"public,secret" validate:"required_without=Output"`
	Output          *Output           `json:"output,omitempty" bson:"output,omitempty" groups:"public,secret" validate:"required_without=Input"`
	Filter          *model.FilterData `json:"filter,omitempty" bson:"filter,omitempty" validate:"excluded_with=Input"`
	Index           []*model.Index    `json:"index,omitempty" bson:"index,omitempty" validate:"excluded_with=Output,dive"`
	Transformer     *Transformer      `json:"transformer,omitempty" bson:"transformer,omitempty" groups:"public,secret"`
	Processor       *Processor        `json:"processor,omitempty" bson:"processor,omitempty" groups:"public,secret"`
	Status          string            `json:"status,omitempty" bson:"status,omitempty" validate:"oneof=ACTIVE INACTIVE|isdefault" enums:"ACTIVE,INACTIVE"`
	Groups          []string          `json:"groups,omitempty" bson:"groups,omitempty" validate:"max=1"`
	SharingGroups   []string          `json:"sharing_groups,omitempty" bson:"sharing_groups,omitempty" validate:"unique"`
	VariableUsage   []string          `json:"variable_usage,omitempty" bson:"variable_usage,omitempty"`
	SecretUsage     []string          `json:"secret_usage,omitempty" bson:"secret_usage,omitempty"`
	UpdatedAtMillis int64             `json:"-" bson:"-"`
}

Notes

• It might be related to sheriff.Marshal modifying internal maps without proper synchronization.

Environment

• Go version: 1.23
• Sheriff version: v0.11.1
• OS: Linux

ps.: I tried to update sheriff to the latest version to check if it would fix it but then I find that it have many breaking changes like #62

[mweibel]

@alecsammon do you have time to look into this?