FoundationDB Trace Metrics Definitions

FoundationDB outputs various messages to /var/log/foundationdb. While some of these may be self-explanatory, others are hard to decipher. This page is a full listing of all metrics (as of v6.2.15 20566f2ff06a7e822b30e8cfd91090fbd863a393) and their definitions.

Initially these will be blank. If you come here to find a metric definition and do not find it -- please update the page when you do!

Common Metrics

Metric Name	Description
Type
Time
Severity
Machine
LogGroup
ID
Roles

"AsyncFileKAIOOpen"

Metric Name	Example	Description
Severity	10
Time	1588967689.669706
Type	AsyncFileKAIOOpen
ID	0000000000000000
Filename	/mnt/fdb/transaction/4519/logqueue-V_3_LS_2-ebfd05340047e0023ebfd5976921797d-0.fdq
Flags	983046
Mode	1408
Fd	76
Machine	10.180.50.179:4519
LogGroup	default
Roles	TL

"BgDDMountainChopper"

Metric Name	Example	Description
Severity	10
Time	1588947778.846806
Type	BgDDMountainChopper
ID	47cedbda4d178447
SourceBytes	4853627018
DestBytes	3153236645
ShardBytes	24260500
AverageShardBytes	17832010
SourceTeam	Size 3; 10.180.50.80:4505 266809676c603f74 10.180.50.50:4504 54a823d4ae7529d3, 10.180.50.179:4502 5ce25c2920b67979
DestTeam	Size 3; 10.180.50.80:4500 1ddcb14a4183c131 10.180.50.179:4501 4c48cfbd569beedc, 10.180.50.59:4502 9cd1b1e104c9b375
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD

"BgDDValleyFiller"

Metric Name	Example	Description
Severity	10
Time	1588947773.943168
Type	BgDDValleyFiller
ID	47cedbda4d178447
SourceBytes	3209342253
DestBytes	2784814454
ShardBytes	30515983
AverageShardBytes	17832010
SourceTeam	Size 3; 10.180.50.133:4501 0d0549ca42863d3d 10.180.50.77:4505 16c1de13480928f6, 10.180.50.59:4500 ba09eca3913bf7ea
DestTeam	Size 3; 10.180.50.179:4500 739f9922184049d5 10.180.50.59:4501 7b4181ea6025fff2, 10.180.50.77:4502 e8181de673760f68
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD

"ClusterControllerMetrics"

Metric Name	Example	Description
Severity	10
Time	1589032655.510371
OriginalTime	1589032652.301211
Type	ClusterControllerMetrics
ID	71f062b5aa2e75b9
Elapsed	5.00001
OpenDatabaseRequests	1.2 5.11374 7659081
RegisterWorkerRequests	0 0 1201
GetWorkersRequests	0 0 0
GetClientWorkersRequests	0 0 0
RegisterMasterRequests	0 0 149
GetServerDBInfoRequests	0.399999 1.52986 1845456
StatusRequests	0 0 425054
FailureMonitoringRequests	719.198 4.12408 5493283854
ServersFailed	0 0 271
ServersUnfailed	0 0 343
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD
TrackLatestType	Rolled

"ClusterControllerWorkerAlreadyRegistered"

Metric Name	Example	Description
Severity	10
Time	1589306127.517835
Type	ClusterControllerWorkerAlreadyRegistered
ID	71f062b5aa2e75b9
SuppressedEventCount	0
WorkerId	6e9e2d06650b7805
ProcessId	a907b89762a1511a65776582eb9c6add
ZoneId	d5e9f92c0f8335f87d059e9f83e731b3
DataHall	3
PClass	stateless
Workers	72
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"ClusterGetStatus"

Metric Name	Example	Description
Severity	10
Time	1589032656.245691
Type	ClusterGetStatus
ID	0000000000000000
Duration	0.0272791
StatusSize	85872
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"CodeCoverage"

Metric Name	Example	Description
Severity	10
Time	1589229311.314538
Type	CodeCoverage
ID	0000000000000000
File	fdbserver/storageserver.actor.cpp
Line	3432
Condition	true
Machine	10.180.50.179:4501
LogGroup	default
Roles	SS

"CommitAttachID"

Metric Name	Example	Description
Severity	10
Time	1588620806.473401
Type	CommitAttachID
ID	0345df4c2682866e
To	8af7049501fab007
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"CommitBatchesMemoryLimit"

Metric Name	Example	Description
Severity	10
Time	1588620806.404859
Type	CommitBatchesMemoryLimit
ID	0000000000000000
BytesLimit	429496729
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"CommitDebug"

Metric Name	Example	Description
Severity	10
Time	1588620806.472888
Type	CommitDebug
ID	0345df4c2682866e
Location	MasterProxyServer.batcher
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"CommitDummyTransaction"

Metric Name	Example	Description
Severity	10
Time	1589306127.517460
Type	CommitDummyTransaction
ID	0000000000000000
Key	\xff/SC/p\xcb\xd8\xb2t\\xb3;1v\xa4\xcf\xf9\x8c\xf7
Retries	0
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"CommitDummyTransactionError"

Metric Name	Example	Description
Severity	10
Time	1588624446.751595
Type	CommitDummyTransactionError
ID	0000000000000000
Error	operation_cancelled
ErrorDescription	Asynchronous operation cancelled
ErrorCode	1101
Key	\xff/logs
Retries	0
Machine	10.180.50.77:4518
LogGroup	default
Roles	MS

"ConnectedOutgoing"

Metric Name	Example	Description
Severity	10
Time	1589440609.340105
Type	ConnectedOutgoing
ID	0000000000000000
SuppressedEventCount	0
PeerAddr	10.180.50.50:4516
Machine	10.180.50.179:4503
LogGroup	default
Roles	SS

"ConnectingTo"

Metric Name	Example	Description
Severity	10
Time	1589440609.338790
Type	ConnectingTo
ID	0000000000000000
SuppressedEventCount	0
PeerAddr	10.180.50.50:4516
Machine	10.180.50.179:4503
LogGroup	default
Roles	SS

"ConnectionClosed"

Metric Name	Example	Description
Severity	10
Time	1588950096.127790
Type	ConnectionClosed
ID	2ca7bfdaeffe987a
Error	connection_failed
ErrorDescription	Network connection failed
ErrorCode	1026
SuppressedEventCount	0
PeerAddr	10.180.50.77:4518
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"ConnectionEstablished"

Metric Name	Example	Description
Severity	10
Time	1588953503.058843
Type	ConnectionEstablished
ID	3c385b767912e1d8
SuppressedEventCount	0
Peer	10.180.50.59:34208
ConnectionId	0
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"ConnectionExchangingConnectPacket"

Metric Name	Example	Description
Severity	10
Time	1589440609.339243
Type	ConnectionExchangingConnectPacket
ID	cbe8783874023296
SuppressedEventCount	0
PeerAddr	10.180.50.50:4516
Machine	10.180.50.179:4503
LogGroup	default
Roles	SS

"ConnectionFrom"

Metric Name	Example	Description
Severity	10
Time	1588953503.058843
Type	ConnectionFrom
ID	3c385b767912e1d8
SuppressedEventCount	0
FromAddress	10.180.50.59:34208
ListenAddress	10.180.50.179:4500
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"ConnectionTimedOut"

Metric Name	Example	Description
Severity	10
Time	1588620913.463151
Type	ConnectionTimedOut
ID	0000000000000000
SuppressedEventCount	0
PeerAddr	10.180.50.80:4516
Machine	10.180.50.179:4514
LogGroup	default

"ConnectionTimeout"

Metric Name	Example	Description
Severity	10
Time	1588620801.277431
Type	ConnectionTimeout
ID	0000000000000000
SuppressedEventCount	1
WithAddr	10.180.50.80:4501
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"CoordinatedStateSet"

Metric Name	Example	Description
Severity	10
Time	1588624381.224096
Type	CoordinatedStateSet
ID	0000000000000000
Gen	72
Wgen	72
Genu	4132d40fe4b7295c
Wgenu	4132d40fe4b7295c
Cgen	72
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"CoordinationPing"

Metric Name	Example	Description
Severity	10
Time	1588949856.367059
Type	CoordinationPing
ID	99ae80c2014db2b1
CCID	71f062b5aa2e75b9
TimeStep	126270
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"CoordinationPingSent"

Metric Name	Example	Description
Severity	10
Time	1589032656.518897
Type	CoordinationPingSent
ID	71f062b5aa2e75b9
TimeStep	127650
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"DDInitGotInitialDD"

Metric Name	Example	Description
Severity	10
Time	1588935347.800644
OriginalTime	1584124303.900535
Type	DDInitGotInitialDD
ID	47cedbda4d178447
B	\x15(\x02\x02\x05\x0d\x00\x00\x00\x00\x00\x07h\xd2
E	\xff\xff
Src	014e19075c662a8f202b8a450ad604d42ad341d7b4caeb5e80b8975b6ef98d69,3821fc63e0d45b3238b4159ad69e2618
Dest	[no items]
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD
TrackLatestType	Rolled

"DDRecruiting"

Metric Name	Example	Description
Severity	10
Time	1588970360.114821
Type	DDRecruiting
ID	0000000000000000
State	Sending request to CC
Exclusions	36
Critical	0
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD

"DDTrackerStarting"

Metric Name	Example	Description
Severity	10
Time	1588935347.800644
OriginalTime	1584124339.015333
Type	DDTrackerStarting
ID	47cedbda4d178447
State	Active
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD
TrackLatestType	Rolled

"DDTrackerStats"

Metric Name	Example	Description
Severity	10
Time	1588935347.800644
OriginalTime	1588935343.580889
Type	DDTrackerStats
ID	47cedbda4d178447
Shards	6787
TotalSizeBytes	245391561136
SystemSizeBytes	0
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD
TrackLatestType	Rolled

"DiskMetrics"

Metric Name	Example	Description
Severity	10
Time	1588949825.432928
Type	DiskMetrics
ID	739f9922184049d5
ReadOps	0
WriteOps	15
ReadQueue	0
WriteQueue	0
GlobalSQLiteMemoryHighWater	0
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"DiskQueueReplaceFile"

Metric Name	Example	Description
Severity	10
Time	1588967689.669706
Type	DiskQueueReplaceFile
ID	ebfd05340047e002
Filename	/mnt/fdb/transaction/4519/logqueue-V_3_LS_2-ebfd05340047e0023ebfd5976921797d-0.fdq
OldFileSize	27211550720
ElidedTruncateSize	21211549696
Machine	10.180.50.179:4519
LogGroup	default
Roles	TL

"DiskQueueReplaceTruncateEnded"

Metric Name	Example	Description
Severity	10
Time	1588967744.860605
Type	DiskQueueReplaceTruncateEnded
ID	0000000000000000
Filename	/mnt/fdb/transaction/4519/logqueue-V_3_LS_2-ebfd05340047e0023ebfd5976921797d-0.fdq
Machine	10.180.50.179:4519
LogGroup	default
Roles	TL

"DumpToken"

Metric Name	Example	Description
Severity	10
Time	1588620806.382418
Type	DumpToken
ID	ef3357ccedce9d99
Name	recruited.commit
Token	ef3357ccedce9d99
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"EndpointNotFound"

Metric Name	Example	Description
Severity	10
Time	1588620888.228939
Type	EndpointNotFound
ID	0000000000000000
SuppressedEventCount	0
Address	10.180.50.77:4516
Token	0cc6c04fad8cdccf
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"FKBlockFail"

Metric Name	Example	Description
Severity	10
Time	1588951097.176744
Type	FKBlockFail
ID	739f9922184049d5
Error	operation_cancelled
ErrorDescription	Asynchronous operation cancelled
ErrorCode	1101
SuppressedEventCount	0
FKID	fa9327589855f12d
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"FSyncParentDir"

Metric Name	Example	Description
Severity	10
Time	1588967689.672385
Type	FSyncParentDir
ID	0000000000000000
Folder	/mnt/fdb/transaction/4519/
File	/mnt/fdb/transaction/4519/logqueue-V_3_LS_2-ebfd05340047e0023ebfd5976921797d-0.fdq
Machine	10.180.50.179:4519
LogGroup	default
Roles	TL

"FailureMonitorClientSlow"

Metric Name	Example	Description
Severity	20
Time	1589440609.332510
Type	FailureMonitorClientSlow
ID	0000000000000000
Elapsed	1.33938
Expected	0.1
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"FailureMonitoringServerDown"

Metric Name	Example	Description
Severity	20
Time	1588621263.584732
Type	FailureMonitoringServerDown
ID	0000000000000000
OldServerID	71f062b5aa2e75b9
Machine	10.180.50.80:4514
LogGroup	default

"FailureMonitoringServerUp"

Metric Name	Example	Description
Severity	10
Time	1588621263.886816
Type	FailureMonitoringServerUp
ID	0000000000000000
OldServer	71f062b5aa2e75b9
Machine	10.180.50.80:4514
LogGroup	default

"FetchKeysTooLong"

Metric Name	Example	Description
Severity	30
Time	1588970499.320355
Type	FetchKeysTooLong
ID	0000000000000000
Duration	600.003
Phase	1
Begin	mako065404742xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\x00
End	mako0654306
Machine	10.180.50.59:4501
LogGroup	default
Roles	SS

"FetchPast"

Metric Name	Example	Description
Severity	20
Time	1589227564.994779
Type	FetchPast
ID	4c48cfbd569beedc
TotalAttempts	1
FKID	ede17487d7eb7ec2
V	7856448270943
N	7856448270943
E	7856448270943
Machine	10.180.50.179:4501
LogGroup	default
Roles	SS

"FinishMoveKeysTooLong"

Metric Name	Example	Description
Severity	30
Time	1588951754.826600
Type	FinishMoveKeysTooLong
ID	0000000000000000
Duration	600
Servers	673a4fea98c33d79bd4a92cfcf3ff0eca89027238077abaed423b976c88806d5,b0037425027a811f4ab8ac1b8e92510f
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD

"FullSnapshotEnd"

Metric Name	Example	Description
Severity	10
Time	1588620806.359413
Type	FullSnapshotEnd
ID	d233af26b0ac14a9
PreviousSnapshotEndLoc	0.595359
SnapshotSize	1390
SnapshotElements	1
Machine	10.180.50.179:4516
LogGroup	default
Roles	CD

"GenerationRegReadReply"

Metric Name	Example	Description
Severity	10
Time	1588620806.358701
Type	GenerationRegReadReply
ID	0000000000000000
RVSize	1351
VWG	56
Machine	10.180.50.179:4516
LogGroup	default
Roles	CD

"GenerationRegReadRequest"

Metric Name	Example	Description
Severity	10
Time	1588620806.358701
Type	GenerationRegReadRequest
ID	0000000000000000
From	10.180.50.133:4518
K	fdb_chaos_testing:8CYRO6j9
Machine	10.180.50.179:4516
LogGroup	default
Roles	CD

"GenerationRegWrote"

Metric Name	Example	Description
Severity	10
Time	1588620806.362043
Type	GenerationRegWrote
ID	0000000000000000
From	10.180.50.133:4518
Key	fdb_chaos_testing:8CYRO6j9
ReqGen	57
Returning	57
Machine	10.180.50.179:4516
LogGroup	default
Roles	CD

"GetDurableResult"

Metric Name	Example	Description
Severity	10
Time	1588624381.223495
Type	GetDurableResult
ID	610cf5205c59d595
Required	2
Present	1
ServerState	fafff
RecoveryVersion	-1
EndVersion	7252781666373
SafeBegin	0
SafeEnd	0
NewSafeBegin	0
KnownCommittedVersion	7252779663229
EpochEnd	9223372036854775807
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"GetDurableResultWaiting"

Metric Name	Example	Description
Severity	10
Time	1588624381.222850
Type	GetDurableResultWaiting
ID	610cf5205c59d595
Required	2
Present	0
ServerState	fffff
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"GetLeaderReply"

Metric Name	Example	Description
Severity	10
Time	1588486349.726138
Type	GetLeaderReply
ID	0000000000000000
SuppressedEventCount	0
Coordinator	10.180.50.80:4516
Nominee	622c813b77d23c79
ClusterKey	fdb_chaos_testing:8CYRO6j9
Machine	10.180.50.179:4514
LogGroup	default

"GetMagazineSample"

Metric Name	Example	Description
Severity	10
Time	1588952423.949376
Type	GetMagazineSample
ID	0000000000000000
Size	128
Backtrace	addr2line -e fdbserver.debug -p -C -f -i 0x199945c 0x195f588 0x195f7d1 0x18bada4 0x18b2e32 0x18b5bb2 0x711564 0x746640 0x756378 0x7564e6 0x7565db 0x75668c 0xb5badf 0xb5d4fa 0xb5e10b 0xa38856 0xa353e9 0x189da86 0x7fbe95498140
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"GetTLogTeamDone"

Metric Name	Example	Description
Severity	10
Time	1589306128.017844
Type	GetTLogTeamDone
ID	0000000000000000
Completed	1
Policy	data_hall^2 x zoneid^2 x 1
Results	6
Processes	6
Workers	72
Required	4
Desired	6
RatingTests	200
PolicyGenerations	100
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"GetValueAttachID"

Metric Name	Example	Description
Severity	10
Time	1589032664.326002
Type	GetValueAttachID
ID	8cd1b5993774ae29
To	c8343088d8219a7e
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"GetValueDebug"

Metric Name	Example	Description
Severity	10
Time	1588974947.417118
Type	GetValueDebug
ID	42e97f74dbc76a76
Location	storageServer.received
Machine	10.180.50.179:4503
LogGroup	default
Roles	SS

"GotServerDBInfoChange"

Metric Name	Example	Description
Severity	10
Time	1588950044.938079
Type	GotServerDBInfoChange
ID	0000000000000000
ChangeID	300f5203b4b3ba9f
MasterID	1bb55f8b45a73d07
RatekeeperID	3ebe1a9f1f09a897
DataDistributorID	47cedbda4d178447
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"HugeArenaSample"

Metric Name	Example	Description
Severity	10
Time	1588951071.557801
Type	HugeArenaSample
ID	0000000000000000
Count	1
Size	411480
Backtrace	addr2line -e fdbserver.debug -p -C -f -i 0x1957f87 0x69c0ec 0x18deb12 0x19d8d88 0x663015 0x7fbe95472505
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"IncomingConnection"

Metric Name	Example	Description
Severity	10
Time	1588953503.058843
Type	IncomingConnection
ID	3c385b767912e1d8
SuppressedEventCount	0
FromAddr	10.180.50.59:34208
CanonicalAddr	10.180.50.59:34208
IsPublic	0
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"IncomingConnectionError"

Metric Name	Example	Description
Severity	10
Time	1588620845.918082
Type	IncomingConnectionError
ID	b13613d198545c6e
Error	connection_failed
ErrorDescription	Network connection failed
ErrorCode	1026
SuppressedEventCount	0
FromAddress	10.180.50.80:34600
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"KVCommit10sSample"

Metric Name	Example	Description
Severity	10
Time	1588951198.673726
Type	KVCommit10sSample
ID	739f9922184049d5
Queued	0.0461066
Commit	0.0101311
Checkpoint	0.168605
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"KVSMemRecovered"

Metric Name	Example	Description
Severity	10
Time	1588620806.404859
Type	KVSMemRecovered
ID	ef3357ccedce9d99
SnapshotItems	0
SnapshotEnd	0
Mutations	0
Commits	0
TimeTaken	0
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"KVSMemRecoveryComplete"

Metric Name	Example	Description
Severity	10
Time	1588620806.404859
Type	KVSMemRecoveryComplete
ID	ef3357ccedce9d99
Reason	Non-header sized data read
DataSize	0
ZeroFillSize	0
SnapshotEndLocation	0.1
NextReadLoc	0.1
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"KVSMemRecoveryStarted"

Metric Name	Example	Description
Severity	10
Time	1588620806.404859
Type	KVSMemRecoveryStarted
ID	ef3357ccedce9d99
SnapshotEndLocation	0.1
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"KVSMemStartingSnapshot"

Metric Name	Example	Description
Severity	10
Time	1588620806.404859
Type	KVSMemStartingSnapshot
ID	ef3357ccedce9d99
StartKey	``
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"KVSMemSwitchingToLargeTransactionMode"

Metric Name	Example	Description
Severity	10
Time	1588620806.359413
Type	KVSMemSwitchingToLargeTransactionMode
ID	d233af26b0ac14a9
TransactionSize	1417
DataSize	2120
Machine	10.180.50.179:4516
LogGroup	default
Roles	CD

"LayerStatusError"

Metric Name	Example	Description
Severity	20
Time	1589227424.258954
Type	LayerStatusError
ID	0000000000000000
Error	transaction_timed_out
ErrorDescription	Operation aborted because the transaction timed out
ErrorCode	1031
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"LayerStatusFetcher"

Metric Name	Example	Description
Severity	10
Time	1589032656.242950
Type	LayerStatusFetcher
ID	0000000000000000
Duration	0.00563216
StatusSize	3022
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"LeaderChanged"

Metric Name	Example	Description
Severity	10
Time	1588620801.624767
Type	LeaderChanged
ID	62734ed33127b4f7
ToID	002c813b77d23c79
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"MachineLoadDetail"

Metric Name	Example	Description
Severity	10
Time	1588949829.583142
Type	MachineLoadDetail
ID	0000000000000000
User	57406479
Nice	2418390
System	39988960
Idle	11992251032
IOWait	2063309
IRQ	0
SoftIRQ	8985462
Steal	37523
Guest	0
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"MachineMetrics"

Metric Name	Example	Description
Severity	10
Time	1588949824.773677
OriginalTime	1588949824.583129
Type	MachineMetrics
ID	0000000000000000
Elapsed	5.00001
MbpsSent	2.72133
MbpsReceived	2.14191
OutSegs	14067
RetransSegs	0
CPUSeconds	0.0450846
TotalMemory	133652574208
CommittedMemory	52253396992
AvailableMemory	81399177216
ZoneID	60062d710b037cb0bf737e2d7278790d
MachineID	60062d710b037cb0bf737e2d7278790d
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS
TrackLatestType	Rolled

"MasterCommittedTLogs"

Metric Name	Example	Description
Severity	10
Time	1588621409.821659
Type	MasterCommittedTLogs
ID	880566f6d4ddee15
TLogs	0: adcaec5a089596b096b5a18b206c9841 95d40059bdb8bfd0cba435415e532921, 70552e191c5685331d3016ddfff999e9, 4366e563e34a058e2b99e6246bf4e7a3, b3f6da478d8a9cf40e90bec614a71c68, c26b9bf32aac5c9ba76f3069a5b42b7f
RecoveryCount	45
RecoveryTransactionVersion	7249738881113
Machine	10.180.50.179:4518
LogGroup	default
Roles	MS

"MasterLifetime"

Metric Name	Example	Description
Severity	10
Time	1588624381.219712
Type	MasterLifetime
ID	610cf5205c59d595
LifetimeToken	71f062b5aa2e75b9#22
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"MasterProxyTerminated"

Metric Name	Example	Description
Severity	10
Time	1588620809.679012
Type	MasterProxyTerminated
ID	ef3357ccedce9d99
Error	tlog_stopped
ErrorDescription	TLog stopped
ErrorCode	1011
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"MasterRecoveredConfig"

Metric Name	Example	Description
Severity	10
Time	1588606950.607957
OriginalTime	1586587214.484365
Type	MasterRecoveredConfig
ID	c93a9d5aa846ac9b
Conf	`{\log_spill:2\logs:6,
edundancy_mode: hree_data_hall,\storage_engine:\ssd-2,\usable_regions:1}`
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS
TrackLatestType	Rolled

"MasterRecovering"

Metric Name	Example	Description
Severity	10
Time	1588624381.235072
Type	MasterRecovering
ID	610cf5205c59d595
LastEpochEnd	7252781666373
RecoveryTransactionVersion	7252881666373
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"MasterRecovery"

Metric Name	Example	Description
Severity	10
Time	1588624381.219712
Type	MasterRecovery
ID	610cf5205c59d595
BeginPair	0c5e5d4863f3ba9a
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"MasterRecoveryCommit"

Metric Name	Example	Description
Severity	10
Time	1588621409.775470
Type	MasterRecoveryCommit
ID	880566f6d4ddee15
Machine	10.180.50.179:4518
LogGroup	default
Roles	MS

"MasterRecoveryDuration"

Metric Name	Example	Description
Severity	10
Time	1588606950.607957
OriginalTime	1586587214.555439
Type	MasterRecoveryDuration
ID	c93a9d5aa846ac9b
RecoveryDuration	0.0903919
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS
TrackLatestType	Rolled

"MasterRecoveryState"

Metric Name	Example	Description
Severity	10
Time	1588606950.607957
OriginalTime	1586587225.075300
Type	MasterRecoveryState
ID	c93a9d5aa846ac9b
StatusCode	14
Status	fully_recovered
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS
TrackLatestType	Rolled

"MasterTerminated"

Metric Name	Example	Description
Severity	10
Time	1588620806.355769
Type	MasterTerminated
ID	c93a9d5aa846ac9b
Error	master_tlog_failed
ErrorDescription	Master terminating because a TLog failed
ErrorCode	1205
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"MasterUpdateRegistration"

Metric Name	Example	Description
Severity	10
Time	1588624381.226471
Type	MasterUpdateRegistration
ID	610cf5205c59d595
RecoveryCount	48
Logs	anti: 0 replication: 4 local: 1 routers: 0 tLogs: 701adf5893b590716c046170438ae7704b24114eba5b6c6c161121f96c764bfa,5af8954c2e85d21d50cb9472a392c008,f77128b5c4256d72d9968c7d8d523c38,d4d6e33e212b440a214c51d4feda4319 locality: -1
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"MemoryMetrics"

Metric Name	Example	Description
Severity	10
Time	1588949829.583142
Type	MemoryMetrics
ID	0000000000000000
TotalMemory16	262144
ApproximateUnusedMemory16	0
ActiveThreads16	1
TotalMemory32	7208960
ApproximateUnusedMemory32	4325376
ActiveThreads32	1
TotalMemory64	216268800
ApproximateUnusedMemory64	166985728
ActiveThreads64	3
TotalMemory96	284880960
ApproximateUnusedMemory96	49533120
ActiveThreads96	1
TotalMemory128	219676672
ApproximateUnusedMemory128	114163712
ActiveThreads128	1
TotalMemory256	9961472
ApproximateUnusedMemory256	9306112
ActiveThreads256	1
TotalMemory512	7864320
ApproximateUnusedMemory512	7602176
ActiveThreads512	1
TotalMemory1024	7602176
ApproximateUnusedMemory1024	7340032
ActiveThreads1024	1
TotalMemory2048	9568256
ApproximateUnusedMemory2048	9306112
ActiveThreads2048	1
TotalMemory4096	5375131648
ApproximateUnusedMemory4096	49938432
ActiveThreads4096	1
TotalMemory8192	323354624
ApproximateUnusedMemory8192	322174976
ActiveThreads8192	1
HugeArenaMemory	103468
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"MonitorLeaderChange"

Metric Name	Example	Description
Severity	10
Time	1588620801.624710
Type	MonitorLeaderChange
ID	0000000000000000
NewLeader	002c813b77d23c79
Machine	10.180.50.179:4514
LogGroup	default

"MonitorLeaderForProxiesChange"

Metric Name	Example	Description
Severity	10
Time	1588620801.624767
Type	MonitorLeaderForProxiesChange
ID	0000000000000000
NewLeader	002c813b77d23c79
Key	fdb_chaos_testing:8CYRO6j9
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"MonitorLeaderForProxiesGotClientInfo"

Metric Name	Example	Description
Severity	10
Time	1588606950.634456
Type	MonitorLeaderForProxiesGotClientInfo
ID	71f062b5aa2e75b9
Proxy0	85655d71658d880b
ClientID	a6b7ed2b5da6bee6
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"MovingData"

Metric Name	Example	Description
Severity	10
Time	1588935347.800644
OriginalTime	1588935347.593266
Type	MovingData
ID	47cedbda4d178447
InFlight	0
InQueue	0
AverageShardSize	44982850
UnhealthyRelocations	0
HighestPriority	0
BytesWritten	1854082563511
PriorityRecoverMove	0
PriorityRebalanceUnderutilizedTeam	0
PriorityRebalanceOverutilizedTeam	0
PriorityTeamHealthy	0
PriorityTeamContainsUndesiredServer	0
PriorityTeamRedundant	0
PriorityMergeShard	0
PriorityTeamUnhealthy	0
PriorityTeam2Left	0
PriorityTeam1Left	0
PriorityTeam0Left	0
PrioritySplitShard	0
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD
TrackLatestType	Rolled

"N2_ConnectError"

Metric Name	Example	Description
Severity	20
Time	1588620913.463151
Type	N2_ConnectError
ID	50b3dd1421b256ae
SuppressedEventCount	0
Message	125
Machine	10.180.50.179:4514
LogGroup	default

"N2_ReadError"

Metric Name	Example	Description
Severity	20
Time	1588950096.127790
Type	N2_ReadError
ID	2ca7bfdaeffe987a
SuppressedEventCount	0
Message	2
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"N2_ReadProbeError"

Metric Name	Example	Description
Severity	20
Time	1588950095.007774
Type	N2_ReadProbeError
ID	562454f4e1afb8be
SuppressedEventCount	0
Message	125
Machine	10.180.50.179:4501
LogGroup	default
Roles	SS

"Net2SlowTaskTrace"

Metric Name	Example	Description
Severity	20
Time	1589222264.244907
Type	Net2SlowTaskTrace
ID	0000000000000000
TraceTime	1589222264.383268
Trace	addr2line -e fdbserver.debug -p -C -f -i 0x7fbe95d375f0 0x69c3d0 0xf648ca 0xfa933e 0xfaa7f7 0xf6b111 0xf69469 0xfad5ec 0xf69a19 0xf6a2ab 0xfada46 0xfb6cac 0xf8602e 0xf865eb 0xf871bb 0xf87697 0x69f938 0xf64ef3 0xf650f1 0xf72e8b 0xf72ffc 0x69f938 0xdd1a79 0xb1be48 0xf9fb54 0xb52a51 0x19d8d88 0x663015 0x7fbe95472505 0x68f2f5
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"NetworkMetrics"

Metric Name	Example	Description
Severity	10
Time	1588949824.773677
OriginalTime	1588949824.583129
Type	NetworkMetrics
ID	0000000000000000
Elapsed	5.00001
CantSleep	0
WontSleep	0
Yields	5
YieldCalls	1434
YieldCallsTrue	0
SlowTaskSignals	0
YieldBigStack	0
RunLoopIterations	2590
TimersExecuted	2920
TasksExecuted	3479
ASIOEventsProcessed	3888
ReadCalls	1426
WriteCalls	712
ReadProbes	713
WriteProbes	0
PacketsRead	713
PacketsGenerated	712
WouldBlock	713
LaunchTime	0.0011158
ReactTime	0.00257254
SlowTask1M	1
SlowTask2M	1
PriorityBusy1	0.0246766
PriorityMaxBusy1	0.000935555
PriorityBusy2050	0.0246766
PriorityMaxBusy2050	0.000935555
PriorityBusy3050	0.0205729
PriorityMaxBusy3050	0.000935555
PriorityBusy4050	0.0205729
PriorityMaxBusy4050	0.000935555
PriorityBusy4950	0.0205729
PriorityMaxBusy4950	0.000935555
PriorityBusy5050	0.0189734
PriorityMaxBusy5050	0.000935555
PriorityBusy7050	0.014112
PriorityMaxBusy7050	0.000935555
PriorityBusy8050	0.014112
PriorityMaxBusy8050	0.000935555
PriorityBusy10050	0.00535178
PriorityMaxBusy10050	0.000935555
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS
TrackLatestType	Rolled

"NewEpochStartVersion"

Metric Name	Example	Description
Severity	10
Time	1588624381.236639
Type	NewEpochStartVersion
ID	610cf5205c59d595
StartVersion	7252779663230
EpochEnd	7252779663230
Locality	-1
OldLogRouterTags	0
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"NotifyAddressFailed"

Metric Name	Example	Description
Severity	10
Time	1588620806.335458
Type	NotifyAddressFailed
ID	0000000000000000
SuppressedEventCount	0
Address	10.180.50.80:4519
Machine	10.180.50.179:4514
LogGroup	default

"NotifyAddressHealthy"

Metric Name	Example	Description
Severity	10
Time	1588620809.176447
Type	NotifyAddressHealthy
ID	0000000000000000
SuppressedEventCount	0
Address	10.180.50.80:4519
Machine	10.180.50.179:4514
LogGroup	default

"PeekNextEnd"

Metric Name	Example	Description
Severity	10
Time	1588624381.235072
Type	PeekNextEnd
ID	0000000000000000
Queue	0
Bytes	12
Loc	7252781666374
End	7252781666374
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"PeekNextGetMore"

Metric Name	Example	Description
Severity	10
Time	1588624381.230646
Type	PeekNextGetMore
ID	0000000000000000
Total	0
Queue	0
Bytes	12
Loc	7252777665578
End	7252781666374
HasMessage	1
Version	7252777665578
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"PeerDestroy"

Metric Name	Example	Description
Severity	10
Time	1588950096.127790
Type	PeerDestroy
ID	0000000000000000
Error	connection_failed
ErrorDescription	Network connection failed
ErrorCode	1026
SuppressedEventCount	0
PeerAddr	10.180.50.77:4518
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"ProcessMetrics"

Metric Name	Example	Description
Severity	10
Time	1588949824.773677
OriginalTime	1588949824.583129
Type	ProcessMetrics
ID	0000000000000000
Elapsed	5.00001
CPUSeconds	0.045374
MainThreadCPUSeconds	0.044557
UptimeSeconds	7.57623e+06
Memory	7174557696
ResidentMemory	6983024640
UnusedAllocatedMemory	741200064
MbpsSent	0.103321
MbpsReceived	0.0856574
DiskTotalBytes	1056755048448
DiskFreeBytes	979928698880
DiskQueueDepth	0
DiskIdleSeconds	4.93201
DiskReads	107
DiskWrites	29
DiskReadsCount	45720243
DiskWritesCount	112056016
DiskWriteSectors	1520
DiskReadSectors	856
FileWrites	200
FileReads	107
CacheReadBytes	81920
CacheFinds	678
CacheWritesBlocked	0
CacheReadsBlocked	107
CachePageReadsMerged	0
CacheWrites	381
CacheReads	127
CacheHits	562
CacheMisses	107
CacheEvictions	0
ZoneID	60062d710b037cb0bf737e2d7278790d
MachineID	60062d710b037cb0bf737e2d7278790d
AIOSubmitCount	118
AIOCollectCount	126
AIOSubmitLag	4.35922e-09
AIODiskStall	1.19493e-05
CurrentConnections	58
ConnectionsEstablished	0
ConnectionsClosed	0
ConnectionErrors	0
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS
TrackLatestType	Rolled

"ProcessTimeOffset"

Metric Name	Example	Description
Severity	10
Time	1588949862.540402
Type	ProcessTimeOffset
ID	0000000000000000
ProcessTime	1588949862.540403
SystemTime	1588949862.540403
OffsetFromSystemTime	0.000000
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"ProgramStart"

Metric Name	Example	Description
Severity	10
Time	1588949824.773677
OriginalTime	1581373597.325596
Type	ProgramStart
ID	0000000000000000
RandomSeed	2847564695
SourceVersion	20566f2ff06a7e822b30e8cfd91090fbd863a393
Version	6.2.15
PackageName	6.2
FileSystem	``
DataFolder	/mnt/fdb/storage/4500
WorkingDirectory	/
ClusterFile	/etc/foundationdb/fdb.cluster
ConnectionString	fdb_chaos_testing:[email protected]:451610.180.50.59:4516,10.180.50.77:4516,10.180.50.80:4516,10.180.50.179:4516
ActualTime	1581373597
CommandLine	/usr/sbin/fdbserver --cache_memory=5GiB --cluster_file=/etc/foundationdb/fdb.cluster --datadir=/mnt/fdb/storage/4500 --listen_address=public --locality_data_hall=3 --logdir=/var/log/foundationdb --logsize=50MiB --maxlogssize=800MiB --memory=13GiB --public_address=auto:4500 --trace_format=json
BuggifyEnabled	0
MemoryLimit	13958643712
Machine	10.180.50.179:4500
LogGroup	default
TrackLatestType	Rolled

"ProxyMetrics"

Metric Name	Example	Description
Severity	10
Time	1588606950.607957
OriginalTime	1586587210.082944
Type	ProxyMetrics
ID	d8ea53f2da943a23
Elapsed	5.00001
TxnStartIn	1.2 3.05246 282
TxnStartOut	1.2 3.05327 282
TxnStartBatch	5.39999 8.84786 1084
TxnSystemPriorityStartIn	0.399999 2.07089 61
TxnSystemPriorityStartOut	0.399999 2.07146 61
TxnBatchPriorityStartIn	0 0 2
TxnBatchPriorityStartOut	0 0 2
TxnDefaultPriorityStartIn	0.799999 2.49219 219
TxnDefaultPriorityStartOut	0.799999 2.49272 219
TxnCommitIn	0 0 10
TxnCommitVersionAssigned	0 0 10
TxnCommitResolving	0 0 10
TxnCommitResolved	0 0 10
TxnCommitOut	0 0 10
TxnCommitOutSuccess	0 0 10
TxnConflicts	0 0 0
CommitBatchIn	0.2 0.209116 52
CommitBatchOut	0.2 0.209867 52
MutationBytes	0 0 7164
Mutations	0 0 16
ConflictRanges	0 0 38
KeyServerLocationRequests	0.399999 1.93754 60
LastAssignedCommitVersion	5214997671825
Version	5214997671825
CommittedVersion	5214997671825
CommitBatchesMemBytesCount	0
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP
TrackLatestType	Rolled

"ProxyRatekeeperChanged"

Metric Name	Example	Description
Severity	10
Time	1588620806.459838
Type	ProxyRatekeeperChanged
ID	ef3357ccedce9d99
RKID	3ebe1a9f1f09a897
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"ProxyReadyForReads"

Metric Name	Example	Description
Severity	10
Time	1588620806.462527
Type	ProxyReadyForReads
ID	ef3357ccedce9d99
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"ProxyReadyForTxnStarts"

Metric Name	Example	Description
Severity	10
Time	1588620806.459838
Type	ProxyReadyForTxnStarts
ID	ef3357ccedce9d99
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"ProxyReplies"

Metric Name	Example	Description
Severity	10
Time	1588624381.236639
Type	ProxyReplies
ID	610cf5205c59d595
WorkerID	8931592061f8bc6b
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"RTSSComplete"

Metric Name	Example	Description
Severity	10
Time	1588624381.235072
Type	RTSSComplete
ID	610cf5205c59d595
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"RecSnapshotEnd"

Metric Name	Example	Description
Severity	10
Time	1588624381.230646
Type	RecSnapshotEnd
ID	610cf5205c59d595
NextKey	\xff/tagLocalityList/\x00\x00
Nextlocation	0.7252777665578
IsSnapshotEnd	1
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"RecruitStorageNotAvailable"

Metric Name	Example	Description
Severity	20
Time	1589306128.020088
Type	RecruitStorageNotAvailable
ID	71f062b5aa2e75b9
Error	no_more_servers
ErrorDescription	Not enough physical servers available
ErrorCode	1008
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"RecruitedMasterWorker"

Metric Name	Example	Description
Severity	10
Time	1589032655.510371
OriginalTime	1588624446.749546
Type	RecruitedMasterWorker
ID	71f062b5aa2e75b9
Address	10.180.50.77:4516
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD
TrackLatestType	Rolled

"RedundantConnection"

Metric Name	Example	Description
Severity	10
Time	1588620801.775621
Type	RedundantConnection
ID	e78534b52842eb94
SuppressedEventCount	1
FromAddr	10.180.50.80:33774
CanonicalAddr	10.180.50.80:4517
LocalAddr	10.180.50.179:4516
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"RelocateShard"

Metric Name	Example	Description
Severity	10
Time	1588947771.083477
Type	RelocateShard
ID	47cedbda4d178447
BeginPair	2e60eacc0d8324b0
KeyBegin	\x15\x01\x02\x02\x05\x0d\x00\x00\x00\x00\x00\x00\x00+
KeyEnd	\x15\x01\x02\x02\x05\x0d\x00\x00\x00\x00\x00\x00\x01J
Priority	950
RelocationID	2e60eacc0d8324b0
SuppressedEventCount	0
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD

"RelocateShardHasDestination"

Metric Name	Example	Description
Severity	10
Time	1588947771.102311
Type	RelocateShardHasDestination
ID	47cedbda4d178447
PairId	2e60eacc0d8324b0
DestinationTeam	3f07c3c89688db6849c4e6e98ac35bc5753172b1ec946a9016a993c2270a8c91,ec81cb9448fdee9c84611a1757a148c0
ExtraIds	[no items]
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD

"RelocateShardMergeMetrics"

Metric Name	Example	Description
Severity	10
Time	1588948041.077504
Type	RelocateShardMergeMetrics
ID	47cedbda4d178447
OldKeys	mako005282 - mako0054021
NewKeys	mako005282 - mako0123825
EndingSize	0
BatchedMerges	79
LastLowBandwidthStartTime	1.58889e+09
ShardCount	79
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD

"RelocateShardStartSplitx100"

Metric Name	Example	Description
Severity	10
Time	1588947771.086890
Type	RelocateShardStartSplitx100
ID	47cedbda4d178447
Begin	\x15\x06\x02\x02\x05\x0d\x00\x00\x00\x00\x00\x00\xfd\x99
End	\x15\x06\x02\x02\x05\x0d\x00\x00\x00\x00\x00\x01\x00M
MaxBytes	35664020
MetricsBytes	35927694
Bandwidth	Low
BytesPerKSec	0
NumShards	2
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD

"RelocateShardTooLong"

Metric Name	Example	Description
Severity	30
Time	1588952239.804462
Type	RelocateShardTooLong
ID	0000000000000000
Duration	606.35
Dest	1ddcb14a4183c13170c7fc8b0ce1f8be673a4fea98c33d79bd4a92cfcf3ff0ec,9e20cee080b698117a03822b46bc4bdb
Src	1ddcb14a4183c13170c7fc8b0ce1f8be669c1758985b5952094370cdae7fcbaf,e05471afd96a7c806a3bcd7d9173f08c
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD

"RelocateShard_FinishMoveKeysRetrying"

Metric Name	Example	Description
Severity	20
Time	1588951378.208062
Type	RelocateShard_FinishMoveKeysRetrying
ID	bb8989e8b32b1696
Error	transaction_too_old
ErrorDescription	Transaction is too old to perform reads or be committed
ErrorCode	1007
KeyBegin	mako1604058
KeyEnd	mako1611403
IterationBegin	mako1604058
IterationEnd	mako1611403
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD

"ResolverInit"

Metric Name	Example	Description
Severity	10
Time	1588624414.664221
Type	ResolverInit
ID	eb714dfc38eaa791
RecoveryCount	56
Machine	10.180.50.179:4517
LogGroup	default
Roles	RV

"ResolverReplies"

Metric Name	Example	Description
Severity	10
Time	1588624381.236639
Type	ResolverReplies
ID	610cf5205c59d595
WorkerID	4e8dade3aca5e1cf
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"ResolverTerminated"

Metric Name	Example	Description
Severity	10
Time	1588624428.691057
Type	ResolverTerminated
ID	eb714dfc38eaa791
Error	worker_removed
ErrorDescription	Normal worker shut down
ErrorCode	1202
Machine	10.180.50.179:4517
LogGroup	default
Roles	RV

"RkUpdate"

Metric Name	Example	Description
Severity	10
Time	1589177956.122590
OriginalTime	1589177956.048386
Type	RkUpdate
ID	0000000000000000
TPSLimit	6.7099e+07
Reason	2
ReasonServerID	07e2bd4103ad63a4
ReleasedTPS	7.23594
ReleasedBatchTPS	0.222651
TPSBasis	7.23594
StorageServers	36
Proxies	3
TLogs	6
WorstFreeSpaceStorageServer	907272409498
WorstFreeSpaceTLog	886403637658
WorstStorageServerQueue	5995
LimitingStorageServerQueue	5904
WorstTLogQueue	1755
TotalDiskUsageBytes	1537087093944
WorstStorageServerVersionLag	0
LimitingStorageServerVersionLag	0
WorstStorageServerDurabilityLag	13965172
LimitingStorageServerDurabilityLag	13951194
Machine	10.180.50.80:4518
LogGroup	default
Roles	RK
TrackLatestType	Rolled

"RkUpdateBatch"

Metric Name	Example	Description
Severity	10
Time	1589177956.122590
OriginalTime	1589177956.048386
Type	RkUpdateBatch
ID	0000000000000000
TPSLimit	4.84604e+07
Reason	2
ReasonServerID	07e2bd4103ad63a4
ReleasedTPS	7.23594
ReleasedBatchTPS	0.222651
TPSBasis	7.23594
StorageServers	36
Proxies	3
TLogs	6
WorstFreeSpaceStorageServer	907272409498
WorstFreeSpaceTLog	886403637658
WorstStorageServerQueue	5995
LimitingStorageServerQueue	5904
WorstTLogQueue	1755
TotalDiskUsageBytes	1537087093944
WorstStorageServerVersionLag	0
LimitingStorageServerVersionLag	0
WorstStorageServerDurabilityLag	13965172
LimitingStorageServerDurabilityLag	13951194
Machine	10.180.50.80:4518
LogGroup	default
Roles	RK
TrackLatestType	Rolled

"Role"

Metric Name	Example	Description
Severity	10
Time	1588967689.669706
Type	AsyncFileKAIOOpen
ID	0000000000000000
Filename	/mnt/fdb/transaction/4519/logqueue-V_3_LS_2-ebfd05340047e0023ebfd5976921797d-0.fdq
Flags	983046
Mode	1408
Fd	76
Machine	10.180.50.179:4519
LogGroup	default
Roles	TL

"SecondCommitBatch"

Metric Name	Example	Description
Severity	10
Time	1588624438.608650
Type	SecondCommitBatch
ID	b2ca1c090d12d855
DebugID	c005e9c6479e71db
Machine	10.180.50.179:4517
LogGroup	default
Roles	MP

"ShardServerFutureVersion1000x"

Metric Name	Example	Description
Severity	20
Time	1588948086.506063
Type	ShardServerFutureVersion1000x
ID	15e62ed9c0f0e5df
Version	7576974507598
MyVersion	7576974506620
ServerID	15e62ed9c0f0e5df
Machine	10.180.50.50:4503
LogGroup	default
Roles	SS

"SlowKAIOLaunch"

Metric Name	Example	Description
Severity	10
Time	1588951241.348125
Type	SlowKAIOLaunch
ID	0000000000000000
IOSubmitTime	0.016397
TruncateTime	5.00679e-06
TruncateCount	0
TruncateBytes	0
LargestTruncate	0
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"SlowKAIOTruncate"

Metric Name	Example	Description
Severity	10
Time	1588970316.811875
Type	SlowKAIOTruncate
ID	0000000000000000
TruncateTime	0.0778663
TruncateBytes	14774272
Machine	10.180.50.179:4519
LogGroup	default
Roles	TL

"SlowSSLoopx100"

Metric Name	Example	Description
Severity	20
Time	1588949830.494709
Type	SlowSSLoopx100
ID	739f9922184049d5
Elapsed	0.0617683
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"SlowTask"

Metric Name	Example	Description
Severity	20
Time	1589222264.244907
Type	Net2SlowTaskTrace
ID	0000000000000000
TraceTime	1589222264.383268
Trace	addr2line -e fdbserver.debug -p -C -f -i 0x7fbe95d375f0 0x69c3d0 0xf648ca 0xfa933e 0xfaa7f7 0xf6b111 0xf69469 0xfad5ec 0xf69a19 0xf6a2ab 0xfada46 0xfb6cac 0xf8602e 0xf865eb 0xf871bb 0xf87697 0x69f938 0xf64ef3 0xf650f1 0xf72e8b 0xf72ffc 0x69f938 0xdd1a79 0xb1be48 0xf9fb54 0xb52a51 0x19d8d88 0x663015 0x7fbe95472505 0x68f2f5
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"SomewhatSlowRunLoopBottom"

Metric Name	Example	Description
Severity	10
Time	1589244944.324415
Type	SomewhatSlowRunLoopBottom
ID	0000000000000000
Elapsed	0.0427301
Machine	10.180.50.179:4502
LogGroup	default
Roles	SS

"SomewhatSlowRunLoopTop"

Metric Name	Example	Description
Severity	10
Time	1588950070.505679
Type	SomewhatSlowRunLoopTop
ID	0000000000000000
Elapsed	0.0206292
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"SpringCleaningMetrics"

Metric Name	Example	Description
Severity	10
Time	1588949825.432928
Type	SpringCleaningMetrics
ID	739f9922184049d5
SpringCleaningCount	7601254
LazyDeletePages	10386661
VacuumedPages	23618478
SpringCleaningTime	8966.72
LazyDeleteTime	4754.18
VacuumTime	4199.48
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"StorageMetrics"

Metric Name	Example	Description
Severity	10
Time	1588949824.773677
OriginalTime	1588949822.486929
Type	StorageMetrics
ID	739f9922184049d5
Elapsed	5.00001
QueryQueue	0 0 136693665
GetKeyQueries	0 0 0
GetValueQueries	0 0 124184553
GetRangeQueries	0 0 12509112
FinishedQueries	0 0 136693665
RowsQueried	0 0 1635435284
BytesQueried	0 0 979447889012
WatchQueries	0 0 21
EmptyQueries	0 0 11247314
BytesInput	0 0 262719843608
BytesDurable	0 0 262719843608
BytesFetched	0 0 170977964670
MutationBytes	0 0 132895506551
SampledBytesCleared	0 0 203599898330
Mutations	0 0 127402905
SetMutations	0 0 88396221
ClearRangeMutations	0 0 39006684
AtomicMutations	0 0 0
UpdateBatches	0.599999 1.90249 50512542
UpdateVersions	0 0 10420748
Loops	10.6 2.0473 264484588
FetchWaitingMS	0 0 100233602
FetchWaitingCount	0 0 91888
FetchExecutingMS	0 0 574783160
FetchExecutingCount	0 0 89939
ReadsRejected	0 0 0
LastTLogVersion	7578710932602
Version	7578710932602
StorageVersion	7578705932602
DurableVersion	7578705932602
DesiredOldestVersion	7578705932602
VersionLag	633531
LocalRate	100
FetchKeysFetchActive	0
FetchKeysWaiting	0
QueryQueueMax	0
BytesStored	3092723975
ActiveWatches	0
WatchBytes	0
KvstoreBytesUsed	23042601712
KvstoreBytesFree	979928698880
KvstoreBytesAvailable	999224475648
KvstoreBytesTotal	1056755048448
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS
TrackLatestType	Rolled

"StorageServerNoLongerBehind"

Metric Name	Example	Description
Severity	10
Time	1589227661.943201
Type	StorageServerNoLongerBehind
ID	4c48cfbd569beedc
CursorVersion	7856550574648
TLogVersion	7856550574647
Machine	10.180.50.179:4501
LogGroup	default
Roles	SS

"StorageServerRecruitment"

Metric Name	Example	Description
Severity	10
Time	1588935347.800644
OriginalTime	1585946133.285731
Type	StorageServerRecruitment
ID	47cedbda4d178447
State	Idle
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD
TrackLatestType	Rolled

"StorageServerUpdateLag"

Metric Name	Example	Description
Severity	20
Time	1589227527.665466
Type	StorageServerUpdateLag
ID	4c48cfbd569beedc
Version	7856416307986
DurableVersion	7856335382578
Machine	10.180.50.179:4501
LogGroup	default
Roles	SS

"TLogJoinedMe"

Metric Name	Example	Description
Severity	10
Time	1588624381.223162
Type	TLogJoinedMe
ID	610cf5205c59d595
TLog	70552e191c568533
Address	10.180.50.179:4519
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"TLogJoinedMeUnknown"

Metric Name	Example	Description
Severity	10
Time	1588624381.237857
Type	TLogJoinedMeUnknown
ID	610cf5205c59d595
TLog	42ea2e110cb30ee0
Address	10.180.50.179:4519
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"TLogLockStarted"

Metric Name	Example	Description
Severity	10
Time	1588624381.222850
Type	TLogLockStarted
ID	610cf5205c59d595
TLog	701adf5893b59071
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"TLogLocked"

Metric Name	Example	Description
Severity	10
Time	1588624381.223495
Type	TLogLocked
ID	610cf5205c59d595
TLog	4b24114eba5b6c6c
End	7252781666373
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"TLogMetrics"

Metric Name	Example	Description
Severity	10
Time	1588964390.566463
OriginalTime	1588964387.589247
Type	TLogMetrics
ID	d168134a9dfed8e7
Elapsed	5.00001
BytesInput	199.999 367.144 358402462588
BytesDurable	784.598 172.76 358402461588
Version	7593276200217
QueueCommittedVersion	7593276200217
PersistentDataVersion	7593273726639
PersistentDataDurableVersion	7593273726639
KnownCommittedVersion	7593274051719
QueuePoppedVersion	7593265945322
MinPoppedTagVersion	7593265945322
MinPoppedTagLocality	0
MinPoppedTagId	1
SharedBytesInput	1179255431786
SharedBytesDurable	1179255430786
SharedOverheadBytesInput	54142495985
SharedOverheadBytesDurable	54142453570
KvstoreBytesUsed	106118352
KvstoreBytesFree	952354414592
KvstoreBytesAvailable	952354414592
KvstoreBytesTotal	1056755048448
QueueDiskBytesUsed	50510909440	bytes used on disk for the files used by a particular instance
QueueDiskBytesFree	952354414592	free space on disk (e.g. as reported by the OS)
QueueDiskBytesAvailable	952354414592	amount available for storage engine use, to include free space on disk and any internal space within the storage engine’s files that’s not currently used but is reusable
QueueDiskBytesTotal	1056755048448	total size of the disk
PeekMemoryReserved	0
PeekMemoryRequestsStalled	0
Machine	10.180.50.179:4519
LogGroup	default
Roles	TL
TrackLatestType	Rolled

"TLogPeekAllAddingCurrent"

Metric Name	Example	Description
Severity	10
Time	1588624420.962159
Type	TLogPeekAllAddingCurrent
ID	0000000000000000
Tag	-1:1
Begin	7252987965976
End	7253101514298
BestLogs	6562755249e03c5948296a5fb89a1da0 fd9ec644fe1a887e92f37eacca89e1e0, fcc74bd989146f9fec729f2986fa8591, 8e0b2719c40e23c8b080eb0f43aeea5c, 4e875de1e062f8b5c11510f911a2adc6
Machine	10.180.50.179:4518
LogGroup	default
Roles	MS

"TLogPeekAllAddingOld"

Metric Name	Example	Description
Severity	10
Time	1588624420.962159
Type	TLogPeekAllAddingOld
ID	0000000000000000
Tag	-1:1
Begin	7252987965976
End	7253101514298
BestLogs	60f915066a11b332df1bf98ae3f29b3d 33a3221be249eb338ef4540aed763ce7, c1d924e2f8406ff83301a25a1586bcb8, adf74fe090a23a9e2df7ea184f5fbf0f, db3ff73dcb39cfbe80903214d3ef896d
LastBegin	7252993510677
ThisBegin	7252987965976
Machine	10.180.50.179:4518
LogGroup	default
Roles	MS

"TLogPeekAllCurrentOnly"

Metric Name	Example	Description
Severity	10
Time	1588624381.225630
Type	TLogPeekAllCurrentOnly
ID	0000000000000000
Tag	-1:1
Begin	7252777665578
End	7252781666374
BestLogs	701adf5893b590716c046170438ae770 4b24114eba5b6c6c161121f96c764bfa, 5af8954c2e85d21d50cb9472a392c008, f77128b5c4256d72d9968c7d8d523c38, d4d6e33e212b440a214c51d4feda4319
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"TLogPeekTxs"

Metric Name	Example	Description
Severity	10
Time	1588624381.225630
Type	TLogPeekTxs
ID	0000000000000000
Begin	7252777665578
End	7252781666374
LocalEnd	7252779663229
PeekLocality	-99
CanDiscardPopped	1
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"TLogQueueCommitSlow"

Metric Name	Example	Description
Severity	20
Time	1588970191.282965
Type	TLogQueueCommitSlow
ID	ebfd05340047e002
LateProcessCount	30
LoggingDelay	1
Machine	10.180.50.179:4519
LogGroup	default
Roles	TL

"TeamCollectionInfo"

Metric Name	Example	Description
Severity	10
Time	1588935347.800644
OriginalTime	1585946140.458718
Type	TeamCollectionInfo
ID	47cedbda4d178447
Primary	1
AddedTeams	0
TeamsToBuild	0
CurrentTeams	180
DesiredTeams	180
MaxTeams	900
StorageTeamSize	3
CurrentMachineTeams	8
CurrentHealthyMachineTeams	8
DesiredMachineTeams	30
MaxMachineTeams	150
TotalHealthyMachines	6
MinTeamsOnServer	10
MaxTeamsOnServer	17
MinMachineTeamsOnMachine	4
MaxMachineTeamsOnMachine	4
DoBuildTeams	0
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD
TrackLatestType	Rolled

"TimeKeeperCommit"

Metric Name	Example	Description
Severity	10
Time	1589032664.325156
Type	TimeKeeperCommit
ID	8cd1b5993774ae29
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"TotalDataInFlight"

Metric Name	Example	Description
Severity	10
Time	1588935347.800644
OriginalTime	1588935347.593266
Type	TotalDataInFlight
ID	47cedbda4d178447
Primary	1
TotalBytes	0
UnhealthyServers	0
ServerCount	36
StorageTeamSize	3
HighestPriority	140
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD
TrackLatestType	Rolled

"TransactionAttachID"

Metric Name	Example	Description
Severity	10
Time	1588620842.930019
Type	TransactionAttachID
ID	4ac2144aefd496c6
To	edd8cd7acb37f07f
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"TransactionCommit"

Metric Name	Example	Description
Severity	10
Time	1589032664.326260
Type	TransactionCommit
ID	0000000000000000
BeginPair	95aa1ffaebc10be3
Parent	8cd1b5993774ae29
Machine	10.180.50.59:4516
LogGroup	default
Roles	CCCD

"TransactionDebug"

Metric Name	Example	Description
Severity	10
Time	1588620806.471218
Type	TransactionDebug
ID	9d42d60dd934fe08
Location	MasterProxyServer.masterProxyServerCore.GetRawCommittedVersion
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMP

"TransactionMetrics"

Metric Name	Example	Description
Severity	10
Time	1588949825.792440
Type	TransactionMetrics
ID	189fd7818f8615af
Elapsed	5.00001
Cluster	``
Internal	1
ReadVersions	0 0 0
LogicalUncachedReads	0 0 2133637
PhysicalReadRequests	0 0 2134174
CommittedMutations	0 0 1
CommittedMutationBytes	0 0 567
CommitStarted	0 0 1
CommitCompleted	0 0 1
TooOld	0 0 0
FutureVersions	0 0 0
NotCommitted	0 0 0
MaybeCommitted	0 0 0
ResourceConstrained	0 0 0
ProcessBehind	0 0 0
MeanLatency	0
MedianLatency	0
Latency90	0
Latency98	0
MaxLatency	0
MeanRowReadLatency	0
MedianRowReadLatency	0
MaxRowReadLatency	0
MeanGRVLatency	0
MedianGRVLatency	0
MaxGRVLatency	0
MeanCommitLatency	0
MedianCommitLatency	0
MaxCommitLatency	0
MeanMutationsPerCommit	0
MedianMutationsPerCommit	0
MaxMutationsPerCommit	0
MeanBytesPerCommit	0
MedianBytesPerCommit	0
MaxBytesPerCommit	0
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"UpdatedLocalityForDcId"

Metric Name	Example	Description
Severity	10
Time	1588624381.225630
Type	UpdatedLocalityForDcId
ID	0000000000000000
DcId	[not set]
Locality0	-99
Locality1	-99
Version	7252779663229
Machine	10.180.50.179:4516
LogGroup	default
Roles	CDMS

"WaitForVersion1000x"

Metric Name	Example	Description
Severity	10
Time	1588951217.218532
Type	WaitForVersion1000x
ID	0000000000000000
Machine	10.180.50.179:4500
LogGroup	default
Roles	SS

"WaitStorageMetricsPenalty"

Metric Name	Example	Description
Severity	20
Time	1588948118.311610
Type	WaitStorageMetricsPenalty
ID	0000000000000000
Keys	mako09685116 - mako1626817
Limit	100
JitteredSecondsOfPenitence	15
Machine	10.180.50.133:4516
LogGroup	default
Roles	DD