Make parsing log filenames future-proof #4518

europaul · 2025-01-10T18:21:14Z

Main commit is a8725c7. Instead of relying on a certain structure of log filenames, we should just use regexp to look for timestamps or UUIDs - this way the filename structure can change in the future, but it will still be backwards compatible with our filename parsing logic.

This PR contains the changes for pillar. In a later PR those changes will also be applied to newlog and edge-view.

europaul · 2025-01-10T18:35:57Z

the yetus warning can probably be ignored since we do the same in other files that import gomega

yetus: pkg/pillar/types/newlogtypes_test.go#L10
revive: should not use dot imports https://revive.run/r#dot-imports

rene · 2025-01-13T10:00:16Z

the yetus warning can probably be ignored since we do the same in other files that import gomega

yetus: pkg/pillar/types/newlogtypes_test.go#L10 revive: should not use dot imports https://revive.run/r#dot-imports

Actually not in all places, like here gomega it's not dot imported: https://github.com/lf-edge/eve/blob/master/pkg/pillar/types/patchenvelopestypes_test.go#L10

I would avoid as much as we can to introduce new Yetus errors and in this case it's really not hard to fix it...

rene · 2025-01-13T10:10:24Z

@europaul , LGTM and I liked the flexibility added by using regex. However, I would fix the Yetus error in order to avoid YAYEE (Yet Another Yetus Error on EVE 😃 )

OhmSpectator

I think we must address the same functions in the EdgeView package; otherwise, the next time we upgrade the Pillar package used by EdgeView, it will be broken.

pkg/pillar/types/newlogtypes.go

OhmSpectator · 2025-01-13T13:59:07Z

pkg/pillar/types/newlogtypes.go

@@ -85,21 +86,43 @@ type NewlogMetrics struct {
 	AppMetrics logfileMetrics // App metrics
 }

-// GetTimestampFromGzipName - get timestamp from gzip file name
-func GetTimestampFromGzipName(fName string) (time.Time, error) {


As far as I see, we still use this func in another package, edgeveiew:

eve/pkg/edgeview/src/log-search.go

Line 152 in 2edd016

timestamp, err := types.GetTimestampFromGzipName(f.Name())

and

eve/pkg/edgeview/src/system.go

Line 163 in 2edd016

timestamp, err := types.GetTimestampFromGzipName(l.Name())

So, we should be careful here...

europaul · 2025-01-13T19:31:09Z

I think we must address the same functions in the EdgeView package; otherwise, the next time we upgrade the Pillar package used by EdgeView, it will be broken.

The new functions are (mostly) backward compatible with the old ones. The only problem that I see is with logs that get created before the devices sets the correct time (probably through NTP), so the initial log files look like dev.log.123.gz instead of dev.log.1731491904032.gz. The new function will not pick up a timestamp from them since it doesn't fit into \b\d{13,16}\b regexp.

europaul · 2025-01-13T19:32:08Z

As far as I see, we still use this func in another package, edgeveiew:

@OhmSpectator yes, I will change them once this is merged and I can update the pillar dependency.

rene · 2025-01-14T08:55:09Z

I think we must address the same functions in the EdgeView package; otherwise, the next time we upgrade the Pillar package used by EdgeView, it will be broken.

The new functions are (mostly) backward compatible with the old ones. The only problem that I see is with logs that get created before the devices sets the correct time (probably through NTP), so the initial log files look like dev.log.123.gz instead of dev.log.1731491904032.gz. The new function will not pick up a timestamp from them since it doesn't fit into \b\d{13,16}\b regexp.

@europaul , can we affirm that the timestamp will always precede the file extension, like this <any file name>.<TIMESTAMP>.<extension (.gz, etc)> ?

If so, the following regex could be used \b(\d{3,16}).\w{2,3}, then the first occurrence will be the timestamp and it will cover the dev.log.123.gz case. If you want to be more picky with the timestamps, another options is to use this regex: \b(\d{13,16}|\d{3}).\w{2,3}, which will cover only valid timestamps and the case for 3 digits....

OhmSpectator · 2025-01-14T12:25:34Z

As far as I see, we still use this func in another package, edgeveiew:

@OhmSpectator yes, I will change them once this is merged and I can update the pillar dependency.

So, we are expecting a new PR very after this one is merged... Have you tested the changes in edge view somehow?

OhmSpectator

Run the tests

europaul · 2025-01-15T05:29:37Z

If so, the following regex could be used \b(\d{3,16}).\w{2,3}, then the first occurrence will be the timestamp and it will cover the dev.log.123.gz case. If you want to be more picky with the timestamps, another options is to use this regex: \b(\d{13,16}|\d{3}).\w{2,3}, which will cover only valid timestamps and the case for 3 digits....

it's actually not just 3 digits - the timestamp can be anything between 0 and now microseconds.

europaul · 2025-01-15T05:31:02Z

As far as I see, we still use this func in another package, edgeveiew:

@OhmSpectator yes, I will change them once this is merged and I can update the pillar dependency.

So, we are expecting a new PR very after this one is merged... Have you tested the changes in edge view somehow?

ok, let's postpone this merge until we can execute it really in sync with the PR for edge-view and newlog. I'll convert it to draft for now

eriknordmark · 2025-01-15T18:19:25Z

pkg/pillar/cmd/loguploader/loguploader.go

-		appUUID = fStr[0]
+	appUUID, err := types.GetUUIDFromFileName(fName)
+	if err != nil {
+		log.Errorf("buildAppUUIDMap: cannot parse app file name %s: %v", fName, err)


Could this generate/log an error when it is run on a device log filename?

it would, because a device log filename doesn't contain a UUID. however this function is only run on app log files

eve/pkg/pillar/cmd/loguploader/loguploader.go

Lines 653 to 655 in b703eb2

if isApp {

buildAppUUIDMap(f.Name())

}

eriknordmark · 2025-01-15T18:21:34Z

pkg/pillar/types/newlogtypes.go

+// GetTimestampFromFileName extracts a microsecond timestamp from a filename
+func GetTimestampFromFileName(filename string) (time.Time, error) {
+	// Regular expression to match a microsecond timestamp (sequence of 13-16 digits)
+	timestampRegex := regexp.MustCompile(`\b\d{13,16}\b`)


Isn't it better to declare this as a variable in the file so that it is compiled at init time and not for each Get call?

that absolutely makes sense!

eriknordmark · 2025-01-15T18:23:00Z

pkg/pillar/types/newlogtypes.go

-		return time.Time{}, fmt.Errorf("getTimestampFromGzipName: invalid log file name %s", fName)
+// GetTimestampFromFileName extracts a microsecond timestamp from a filename
+func GetTimestampFromFileName(filename string) (time.Time, error) {
+	// Regular expression to match a microsecond timestamp (sequence of 13-16 digits)


Comment below says millisecond and here you said microsecond. Which one is it?

thanks for noticing! it's always milliseconds

eriknordmark · 2025-01-15T18:23:32Z

pkg/pillar/types/newlogtypes.go

+// GetUUIDFromFileName extracts a UUID from a filename with dot-delimited parts
+func GetUUIDFromFileName(filename string) (string, error) {
+	// UUID regex pattern (supports v4 UUIDs like "123e4567-e89b-12d3-a456-426614174000")
+	uuidRegex := regexp.MustCompile(`[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}`)


Define as a local variable in the file?

eriknordmark · 2025-01-15T18:24:05Z

pkg/pillar/types/newlogtypes.go

+
+	// Check each part for a UUID match
+	for _, part := range parts {
+		if uuidRegex.MatchString(part) {


Or call the UUID parser function we use elsewhere?

I think we mostly use uuid.FromString, which returns a UUID type and I'm fine with string

Instead of parsing filenames according to a certain schema, we should use regexp to look for timestamps or UUIDs. This way the file names can change in the future without affecting backward compatibility with the existing code for parsing filenames. Signed-off-by: Paul Gaiduk <[email protected]>

Since not all logs in persist are to be sent, we rename LogRemainToSendMBytes to MaxGzipLogMBytesInPersist. This parameter represents the maxinum size of all gzipped logs in persist. Signed-off-by: Paul Gaiduk <[email protected]>

europaul · 2025-02-04T19:39:53Z

The only problem that I see is with logs that get created before the devices sets the correct time (probably through NTP), so the initial log files look like dev.log.123.gz instead of dev.log.1731491904032.gz. The new function will not pick up a timestamp from them since it doesn't fit into \b\d{13,16}\b regexp.

I fixed this problem, by using the first digits-only part that we find between dots as a timestamp

europaul · 2025-02-04T19:41:14Z

So, we are expecting a new PR very after this one is merged... Have you tested the changes in edge view somehow?

@OhmSpectator should I prepare another PR with changes to edge-view and newlog that would be merged right after this one before we merge this one?

OhmSpectator · 2025-02-04T22:53:29Z

@OhmSpectator should I prepare another PR with changes to edge-view and newlog that would be merged right after this one before we merge this one?

I don’t actually remember how we update Pillar as a library. Do we first need to get some kind of hash and add it to the go.mod file of another component? If it’s possible to do within the same PR (I’m not sure), I would prefer to do it that way. I always ask @christoph-zededa or @deitch when dealing with such things.

europaul · 2025-02-05T08:22:55Z

Do we first need to get some kind of hash and add it to the go.mod file of another component?

Yes, that's how we usually do it, we need to first publish pillar before we can update it as a dependency in other modules. Afaik, it's not possible to do both in one PR.

europaul requested review from OhmSpectator, deitch, rouming and eriknordmark as code owners January 10, 2025 18:21

europaul force-pushed the newlog-filenames branch 2 times, most recently from 02876f6 to 58d6ee9 Compare January 10, 2025 18:31

OhmSpectator requested changes Jan 13, 2025

View reviewed changes

europaul force-pushed the newlog-filenames branch from 58d6ee9 to b703eb2 Compare January 13, 2025 19:33

github-actions bot requested a review from OhmSpectator January 13, 2025 19:34

OhmSpectator approved these changes Jan 14, 2025

View reviewed changes

europaul marked this pull request as draft January 15, 2025 05:31

eriknordmark requested a review from naiming-zededa January 15, 2025 18:18

eriknordmark reviewed Jan 15, 2025

View reviewed changes

europaul added 2 commits February 4, 2025 20:36

global params: rename LogRemainToSendMBytes

82178e5

Since not all logs in persist are to be sent, we rename LogRemainToSendMBytes to MaxGzipLogMBytesInPersist. This parameter represents the maxinum size of all gzipped logs in persist. Signed-off-by: Paul Gaiduk <[email protected]>

europaul force-pushed the newlog-filenames branch from b703eb2 to 82178e5 Compare February 4, 2025 19:37

github-actions bot requested a review from eriknordmark February 4, 2025 19:37

github-actions bot requested a review from OhmSpectator February 4, 2025 19:37

Make parsing log filenames future-proof #4518

Are you sure you want to change the base?

Make parsing log filenames future-proof #4518

Conversation

europaul commented Jan 10, 2025

Uh oh!

europaul commented Jan 10, 2025

Uh oh!

rene commented Jan 13, 2025

Uh oh!

rene commented Jan 13, 2025

Uh oh!

OhmSpectator left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

europaul commented Jan 13, 2025

Uh oh!

europaul commented Jan 13, 2025

Uh oh!

rene commented Jan 14, 2025

Uh oh!

OhmSpectator commented Jan 14, 2025

Uh oh!

OhmSpectator left a comment

Choose a reason for hiding this comment

Uh oh!

europaul commented Jan 15, 2025

Uh oh!

europaul commented Jan 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

europaul Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

europaul commented Feb 4, 2025

Uh oh!

europaul commented Feb 4, 2025

Uh oh!

OhmSpectator commented Feb 4, 2025

Uh oh!

europaul commented Feb 5, 2025

Uh oh!

Uh oh!

OhmSpectator left a comment •

edited

Loading

europaul Feb 4, 2025 •

edited

Loading