Skip to content

Ng启动失败 (Ng startup failure) ,提示 Arena too small, toWrite:126 newTotal:xxx limit:xxx #182

@yinyuncan

Description

@yinyuncan

手动杀死Prometheus和ng后,ng重启失败 (Description ng fails to restart after prometheus and ng are manually killed)

tidb版本:6.5
ng版本: Git Commit Hash"=f1c05e221155c2c95d391957971defbcbbf56832

错误日志位于:/tidb-deploy/prometheus-9090/log/tsdb.log

2023-03-07T06:36:53.570Z        info    /go/pkg/mod/github.com/!victoria!metrics/!victoria!metrics@v1.65.0/app/vmselect/promql/rollup_result_cache.go:106       loading rollupResult cache from "/tidb-data/prometheus-9090/tsdb/cache/rollupResult"...
2023-03-07T06:36:53.571Z        info    /go/pkg/mod/github.com/!victoria!metrics/!victoria!metrics@v1.65.0/app/vmselect/promql/rollup_result_cache.go:132       loaded rollupResult cache from "/tidb-data/prometheus-9090/tsdb/cache/rollupResult" in 0.001 seconds; entriesCount: 0, sizeBytes: 0
2023-03-07T06:36:53.571Z        info    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/database/timeseries/vm.go:45 started VictoriaMetrics in 0.094 seconds
2023/03/07 14:36:53 Arena too small, toWrite:126 newTotal:21810362 limit:21810342
github.com/dgraph-io/badger/v3/y.AssertTruef
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/y/error.go:62
github.com/dgraph-io/badger/v3/skl.(*Arena).putVal
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/skl/arena.go:83
github.com/dgraph-io/badger/v3/skl.newNode
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/skl/skl.go:114
github.com/dgraph-io/badger/v3/skl.(*Skiplist).Put
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/skl/skl.go:306
github.com/dgraph-io/badger/v3.(*memTable).replayFunction.func1
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:258
github.com/dgraph-io/badger/v3.(*logFile).iterate
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:521
github.com/dgraph-io/badger/v3.(*memTable).UpdateSkipList
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:219
github.com/dgraph-io/badger/v3.(*DB).openMemTable
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:146
github.com/dgraph-io/badger/v3.(*DB).openMemTables
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:88
github.com/dgraph-io/badger/v3.Open
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/db.go:333
github.com/genjidb/genji/engine/badgerengine.NewEngine
        /go/pkg/mod/github.com/crazycs520/genji/engine/badgerengine@v0.12.1-0.20220328082424-727a2d089bde/engine.go:25
github.com/pingcap/ng-monitoring/database/document.Init
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/database/document/document.go:34
github.com/pingcap/ng-monitoring/database.Init
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/database/database.go:14
main.main
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/main.go:68
runtime.main
        /usr/local/go/src/runtime/proc.go:250
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1594

ng.log 无报错
最后的日志如下


[2023/03/07 14:41:44.272 +08:00] [INFO] [printer.go:25] ["Welcome to ng-monitoring."] ["Git Commit Hash"=f1c05e221155c2c95d391957971defbcbbf56832] ["Git Branch"=heads/refs/tags/v6.5.0] ["UTC Build Time"="2022-12-16 08:18:47"] [GoVersion=go1.19.3]
[2023/03/07 14:41:44.273 +08:00] [INFO] [main.go:64] [config] [config="{\"address\":\"0.0.0.0:12020\",\"advertise_address\":\"10.0.0.80:12020\",\"pd\":{\"endpoints\":[\"10.0.0.80:2379\",\"10.0.0.76:2379\",\"10.0.0.74:2379\"]},\"log\":{\"path\":\"/tidb-deploy/prometheus-9090/log\",\"level\":\"INFO\"},\"storage\":{\"path\":\"/tidb-data/prometheus-9090\"},\"continuous_profiling\":{\"enable\":false,\"profile_seconds\":10,\"interval_seconds\":60,\"timeout_seconds\":120,\"data_retention_seconds\":259200},\"security\":{\"ca_path\":\"\",\"cert_path\":\"\",\"key_path\":\"\"}}"]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions