New timestamp field in all structures by miiu96 · Pull Request #345 · multiversx/mx-chain-es-indexer-go

miiu96 · 2025-06-06T12:04:50Z

Added a new field timestampMs in all structures

# Conflicts: # go.mod # go.sum

ssd04 · 2025-06-11T11:30:00Z

process/elasticproc/block/blockProcessor.go

 		Size:                  int64(blockSizeInBytes),
 		SizeTxs:               int64(sizeTxs),
 		Timestamp:             time.Duration(obh.Header.GetTimeStamp()),
+		TimestampMs:           time.Duration(obh.OutportBlock.BlockData.TimestampMs),


Suggested change

TimestampMs: time.Duration(obh.OutportBlock.BlockData.TimestampMs),

TimestampMs: time.Duration(obh.OutportBlock.BlockData.GetTimestampMs()),

?

ssd04 · 2025-06-11T11:34:37Z

process/elasticproc/interface.go

 	SerializeSCRs(scrs []*data.ScResult, buffSlice *data.BufferSlice, index string, shardID uint32) error
 }
+
+// TemplatesAndPoliciesHandler  defines the actions that a templates and policies handler should do


Suggested change

// TemplatesAndPoliciesHandler defines the actions that a templates and policies handler should do

// TemplatesAndPoliciesHandler defines the actions that a templates and policies handler should do

ssd04 · 2025-06-11T11:40:26Z

templates/indices/timestampMs.go

+	"properties": Object{
+		"timestampMs": Object{
+			"type":   "date",
+			"format": "epoch_millis",


what epoch_millis means here? could have been header_millis?

epoch_millis in Elasticsearch means:
The date is stored and interpreted as the number of milliseconds since the Unix epoch (i.e., since 1970-01-01T00:00:00Z). (it's an Elasticsearch format for date )

ssd04 · 2025-06-11T11:40:49Z

templates/indices/timestampMs.go

+
+var TokensTimestampMs = Object{


add comments to the other fields in this file

ssd04 · 2025-06-11T11:46:52Z

process/elasticproc/accounts/accountsProcessor.go


 // PrepareRegularAccountsMap will prepare a map of regular accounts
-func (ap *accountsProcessor) PrepareRegularAccountsMap(timestamp uint64, accounts []*data.Account, shardID uint32) map[string]*data.AccountInfo {
+func (ap *accountsProcessor) PrepareRegularAccountsMap(timestamp uint64, accounts []*data.Account, shardID uint32, timestampMs uint64) map[string]*data.AccountInfo {


i'm thinking if it might work to pass only timestampMs and get old timestamp as timestampMs/1000 when needed, since we care about ms granularity now and we can get seconds granularity from milliseconds

in order to avoid passing kind of duplicated variables

and this can be applied to all other places if suitable

ssd04 · 2025-06-11T11:50:46Z

process/elasticproc/templatesAndPolicies/reader.go

+// GetTimestampMsMappings will return the timestampMs field mappings for all indices
+func (tr *templatesAndPolicyReader) GetTimestampMsMappings() ([]templates.ExtraMapping, error) {


why is this mapping needed?

this function is needed to add the new timestampMS mappings for already existing indices.

ssd04 · 2025-06-11T11:51:10Z

process/elasticproc/templatesAndPolicies/reader.go

+			Index:    indexer.EventsIndex,
+			Mappings: indices.TimestampMs.ToBuffer(),
+		},
+


Suggested change

delete empty line

sstanculeanu

no other comment

Copilot

Pull Request Overview

This PR introduces a new timestampMs field (milliseconds) alongside existing timestamp fields across all data structures and updates related tests and fixtures to use these values.

Added timestampMs to JSON test fixtures and data models, replacing time.Duration with uint64 for timestamp
Updated integration tests to pass millisecond timestamps to new methods
Bumped mx-chain-core-go dependency version

Reviewed Changes

Copilot reviewed 160 out of 160 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
integrationtests/testdata/accountsBalanceWithLowerTimestamp/account-balance-esdt-second-update.json	Add `timestampMs` to fixture
integrationtests/testdata/accountsBalanceWithLowerTimestamp/account-balance-esdt-first-update.json	Add `timestampMs` to fixture
integrationtests/testdata/accountsBalanceNftTransfer/balance-nft-after-transfer.json	Add `timestampMs` to fixture
integrationtests/testdata/accountsBalanceNftTransfer/balance-nft-after-create.json	Add `timestampMs` to fixture
integrationtests/miniblocks_test.go	Pass millisecond timestamp to `SaveMiniblocks`
integrationtests/logsCrossShard_test.go	Pass millisecond timestamp to `RemoveTransactions`
integrationtests/delegators_test.go	Pass millisecond timestamp to `RemoveTransactions`
integrationtests/accountsESDTRollback_test.go	Update `RemoveAccountsESDT` call with ms timestamp
integrationtests/accountsBalanceNftTransfer_test.go	Set `TimestampMs` in block creation helper
go.mod	Bump `mx-chain-core-go` version
data/transaction.go	Change `Timestamp` type; add `TimestampMs`
data/tokens.go	Change `Timestamp` type; add `TimestampMs`
data/scresult.go	Change `Timestamp` type; add `TimestampMs`
data/scDeploy.go	Add `TimestampMs` to `ScDeployInfo` and `Upgrade`
data/logs.go	Change `Timestamp` type; add `TimestampMs`
data/event.go	Change `Timestamp` type; add `TimestampMs`
data/delegators.go	Change `Timestamp` type; add `TimestampMs`
data/data.go	Change `Timestamp` type; add `TimestampMs`
data/block.go	Change `Timestamp` type; add `TimestampMs`
data/account.go	Change `Timestamp` type; add `TimestampMs`

Comments suppressed due to low confidence (1)

integrationtests/accountsESDTRollback_test.go:94

It looks like the arguments to RemoveAccountsESDT have been swapped compared to the original call (timestamp, shardID). Please confirm the new signature and correct the argument order.

err = esProc.RemoveAccountsESDT(2, 5040000)

Copilot · 2025-06-13T08:06:00Z

data/scDeploy.go

+	TxHash      string `json:"upgradeTxHash"`
+	Upgrader    string `json:"upgrader"`
+	Timestamp   uint64 `json:"timestamp"`
+	TimestampMs uint64 `json:"timestampMs"`


[nitpick] The TimestampMs field in Upgrade lacks the omitempty JSON tag, which causes zero values to always be serialized. Consider adding omitempty for consistency with other timestamp fields.

Suggested change

TimestampMs uint64 `json:"timestampMs"`

TimestampMs uint64 `json:"timestampMs,omitempty"`

Copilot · 2025-06-13T08:06:00Z

integrationtests/miniblocks_test.go

 		},
 	}
-	err = esProc.SaveMiniblocks(header, miniBlocks)
+	err = esProc.SaveMiniblocks(header, miniBlocks, 1234000)


[nitpick] This test uses a hardcoded millisecond value (1234000). To prevent mismatches if header.TimeStamp changes, consider computing it with header.TimeStamp * 1000.

Suggested change

err = esProc.SaveMiniblocks(header, miniBlocks, 1234000)

err = esProc.SaveMiniblocks(header, miniBlocks, header.TimeStamp*1000)

miiu96 added 9 commits June 6, 2025 15:03

timestamp ms all structures

1e96aa7

fix build

b367efd

fix unit tests

eda0fe2

fix integration tests 1

0637870

fixes

69c7173

Merge branch 'rc/barnard' into timestamp-ms-all-structuress

3d1707e

# Conflicts: # go.mod # go.sum

fix integration and unit test

ebeda3c

fixes timestamp ms

8f7c1bf

fixes and refactoring

37f5294

ssd04 self-requested a review June 11, 2025 11:23

ssd04 reviewed Jun 11, 2025

View reviewed changes

sstanculeanu reviewed Jun 11, 2025

View reviewed changes

miiu96 added 2 commits June 12, 2025 13:17

fixes after review

1889119

fix

f4cb56b

sstanculeanu approved these changes Jun 12, 2025

View reviewed changes

ssd04 approved these changes Jun 12, 2025

View reviewed changes

danidrasovean approved these changes Jun 13, 2025

View reviewed changes

andreibancioiu requested a review from Copilot June 13, 2025 08:04

Copilot AI reviewed Jun 13, 2025

View reviewed changes

sstanculeanu merged commit 50fb80e into rc/barnard Jun 18, 2025
8 checks passed

sstanculeanu deleted the timestamp-ms-all-structuress branch June 18, 2025 15:22

danidrasovean mentioned this pull request Jul 25, 2025

Add timestampMs column to all tables multiversx/multiversx-etl#52

Merged

	TimestampMs: time.Duration(obh.OutportBlock.BlockData.TimestampMs),
	TimestampMs: time.Duration(obh.OutportBlock.BlockData.GetTimestampMs()),

	// TemplatesAndPoliciesHandler defines the actions that a templates and policies handler should do
	// TemplatesAndPoliciesHandler defines the actions that a templates and policies handler should do

		// GetTimestampMsMappings will return the timestampMs field mappings for all indices
		func (tr *templatesAndPolicyReader) GetTimestampMsMappings() ([]templates.ExtraMapping, error) {

	TimestampMs uint64 `json:"timestampMs"`
	TimestampMs uint64 `json:"timestampMs,omitempty"`

	err = esProc.SaveMiniblocks(header, miniBlocks, 1234000)
	err = esProc.SaveMiniblocks(header, miniBlocks, header.TimeStamp*1000)

Conversation

miiu96 commented Jun 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sstanculeanu left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants