need to json.unmarshal data_base64 #1129

duglin · 2025-03-12T23:00:26Z

ivan-penchev · 2025-03-13T00:13:10Z

@duglin I reckon you should add a test

https://github.com/cloudevents/sdk-go/blob/main/v2/event/event_unmarshal_test.go

Maybe a test case like this:

		"base64 json encoded data that has not escaped characters v1.0": {
			body: mustJsonMarshal(t, map[string]interface{}{
				"specversion":     "1.0",
				"datacontenttype": "application/octet-stream",
				"data_base64":     "\\u002B\\u002B\\u002B\\u002B",
				"id":              "ABC-123",
				"time":            now.Format(time.RFC3339Nano),
				"type":            "com.example.test",
				"dataschema":      "http://example.com/schema",
				"source":          "http://example.com/source",
			}),
			want: &event.Event{
				Context: event.EventContextV1{
					Type:            "com.example.test",
					Source:          *sourceV1,
					DataSchema:      schemaV1,
					ID:              "ABC-123",
					Time:            &now,
					DataContentType: event.StringOfTextPlain(),
				}.AsV1(),
				DataEncoded: []byte("++++"),
				DataBase64:  true,
			},
		},

duglin · 2025-03-13T00:20:05Z

@ivan-penchev yup - was working on a testcase we you were testing...

ivan-penchev · 2025-03-13T06:17:08Z

Sweet! This looks like exactly what I need for the fix.
Do you have in mind, who else needs to review it to be merged in main @duglin ?

P.S. What about making a new release? Whom do I need to talk with / tag? I can see it has been a while/year :D, since the last release was created.

duglin · 2025-03-13T11:42:24Z

@embano1 for a review

embano1 · 2025-03-13T12:00:08Z

v2/event/event_unmarshal.go

@@ -385,9 +386,13 @@ func consumeData(e *Event, isBase64 bool, iter *jsoniter.Iterator) error {
 		e.DataBase64 = true

 		// Allocate payload byte buffer
-		base64Encoded := iter.ReadStringAsSlice()
+		base64Encoded := iter.ReadString()
+		err := json.Unmarshal([]byte(`"`+base64Encoded+`"`), &base64Encoded)


could there be issues here if base64Encoded already contains surrounding "" ? For example, if it's a JSON-encoded string e.g., "{\"hello\":\"world\"}"?

nit: I don't prefer the in-place mutation of the base64Encoded variable, better use dedicated variables in case the code is later extended and another developer relies on the original base64Encoded without noticing the overwrite.

re: extra quotes... I think we're ok because any " in the string would need to be escaped, as you've done it. The quotes around your json object/string thingy are removed by the iter.ReadString() call, so just {\"hello\":\"world\"} is passed into Unmarshal (after being re-wrapped with quotes).

I added another variable for the Unmarshal per your suggestion.

Also, this is only for base64 encoded stuff, not normal json strings.

Also, this is only for base64 encoded stuff, not normal json strings.

In layman terms, since I am new to this package, and I need to explain it to myself:

this only gets triggered when data_base64 property is populated, and not when data property is populated.

the json.Unmarshal is used not to unmarshal the object into json, but to unescape any escaped string, hence the base64 string.

Hope this is helpful @embano1

embano1 · 2025-03-14T14:41:21Z

v2/event/event_unmarshal.go

+			return err
+		}
+		e.DataEncoded = make([]byte, base64.StdEncoding.DecodedLen(len(base64DeJSON)))
+		length, err := base64.StdEncoding.Decode(e.DataEncoded, []byte(base64DeJSON))


btw: as I'm trying to understand this code (existing, I know), having a hard time following what it really does without comments. For example why is e.DataEncoded[0:length] needed if we have e.DataEncoded = make([]byte, base64.StdEncoding.DecodedLen(len(base64DeJSON)))

I have no idea. I paused on those lines of code too but decided to not risk breaking anything by trying to clean it up, and instead just focus on the bug I was trying to fix.

embano1 · 2025-03-14T14:50:32Z

v2/event/event_unmarshal_test.go

+			body: []byte(`{
+				"specversion":     "1.0",
+				"datacontenttype": "text/plain",
+				"data_base64": "\\u002B\\u002B\\u002B\\u002B",


quick question: why are we double-escaping here? none of my base64 decoders provide valid output for this, so just wanted to understand the test fixtures better.

Also, I tried to update the test with a base64-encoded JSON string IntcImhlbGxvXCI6XCJ3b3JsZFwifSIK (which is "{\"hello\":\"world\"}") to see if we break existing users here somehow (with the new " change). For some reason, I can't get the test to pass when I provide

"data_base64": "IntcImhlbGxvXCI6XCJ3b3JsZFwifSIK" and expect DataEncoded: []byte(`"{\"hello\":\"world\"}"`). I'm just too stupid I guess - can you please tell me what's going wrong here?

for your reference, my modified test case (ignore the comment)

"base64 json encoded data v1.0 - escaped json string": { body: []byte(`{ "specversion": "1.0", "datacontenttype": "text/plain", "data_base64": "IntcImhlbGxvXCI6XCJ3b3JsZFwifSIK", "id": "ABC-123", "time": "` + now.Format(time.RFC3339Nano) + `", "type": "com.example.test", "dataschema": "http://example.com/schema", "source": "http://example.com/source" }`), want: &event.Event{ Context: event.EventContextV1{ Type: "com.example.test", Source: *sourceV1, DataSchema: schemaV1, ID: "ABC-123", Time: &now, DataContentType: event.StringOfTextPlain(), }.AsV1(), // base64 decode of "++++" DataEncoded: []byte(`"{\"hello\":\"world\"}"`), DataBase64: true, }, }

Hi @embano1

It doesn't matter if we do \\u002B\\u002B\\u002B\\u002B or \u002B\u002B\u002B\u002B. The test would pass either way, I prefer (and I believe @duglin just copied) the double \ as it gives you extra safety for some parsers, which would remove the \ if it is only one.

Regarding your test case, I think your data_base64 is somehow incorrect?

If I take your test case, and I run it with eyJoZWxsbyI6IndvcmxkIn0= as a value for data_base64, which I generated via Go, this way:

package main import ( "encoding/base64" "encoding/json" "fmt" ) func main() { // Define the JSON object jsonObj := map[string]string{"hello": "world"} // Marshal the JSON object to a byte slice jsonData, err := json.Marshal(jsonObj) if err != nil { fmt.Println("Error marshaling JSON:", err) return } // Encode the JSON byte slice to Base64 base64Data := base64.StdEncoding.EncodeToString(jsonData) // Print the Base64 encoded string fmt.Println("Base64 Encoded JSON:", base64Data) }

Your test case passes without issue

Sorry I can't be more of an help, I am a bit clueless how different base64 are generated/

Hm, strange. I quickly wanted to try it, so I used the OSX CLI to generate the test string and wasn't expecting differences in encoding :

echo IntcImhlbGxvXCI6XCJ3b3JsZFwifSIK | base64 -d "{\"hello\":\"world\"}"

Will try using the Go encoder and verify.

I can't explain why \\ and \ yield the same results.
Notice that I use back-tick and not double-quotes around the entire thing, so you'd think some parser would not be happy about the double-\.
But yes, I just copy-n-pasted it.

I would point out this though:

$ echo -e '\u002B\u002B\u002B\u002B' | base64 -d �� (which is correct) $ echo -e '\\u002B\\u002B\\u002B\\u002B' | base64 -d base64: invalid input

Which is actually closer to what I would expect.
I can change it to a single \ if you (@embano1 ) want.

I wonder if it's a golang thing to be more permissive???

If it was me @duglin , I would change it to a \ (single) one if that is passing the base64 test.
Maybe go is a bit more permissive :)

ok made them single.... testing now.
Once CI is happy I'll merge

embano1

LGTM

ivan-penchev · 2025-03-17T17:41:47Z

@duglin would you mind merging and possibly releasing? 🥇 Thank you in advance.

Signed-off-by: Doug Davis <[email protected]>

duglin requested a review from a team as a code owner March 12, 2025 23:00

duglin mentioned this pull request Mar 12, 2025

Unable to deserialize data_base64 non escaped string #1128

Closed

duglin force-pushed the issue1128 branch from 464769a to d3b6bdf Compare March 12, 2025 23:02

duglin force-pushed the issue1128 branch 2 times, most recently from 7d6dded to ddbe284 Compare March 13, 2025 00:18

embano1 reviewed Mar 13, 2025

View reviewed changes

duglin force-pushed the issue1128 branch from ddbe284 to 0922135 Compare March 13, 2025 12:33

embano1 reviewed Mar 14, 2025

View reviewed changes

embano1 previously approved these changes Mar 16, 2025

View reviewed changes

duglin dismissed embano1’s stale review via cd2770a March 18, 2025 18:39

duglin force-pushed the issue1128 branch from 0922135 to cd2770a Compare March 18, 2025 18:39

need to json.unmarshal data_base64

a5522a4

Signed-off-by: Doug Davis <[email protected]>

duglin force-pushed the issue1128 branch from cd2770a to a5522a4 Compare March 18, 2025 20:07

ivan-penchev approved these changes Mar 18, 2025

View reviewed changes

embano1 approved these changes Mar 19, 2025

View reviewed changes

embano1 merged commit b627455 into cloudevents:main Mar 19, 2025
9 checks passed

duglin mentioned this pull request Mar 21, 2025

Release changes from main #1132

Closed

need to json.unmarshal data_base64 #1129

need to json.unmarshal data_base64 #1129

Uh oh!

Conversation

duglin commented Mar 12, 2025

Uh oh!

ivan-penchev commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

duglin commented Mar 13, 2025

Uh oh!

ivan-penchev commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

duglin commented Mar 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivan-penchev Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivan-penchev Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

embano1 left a comment

Choose a reason for hiding this comment

Uh oh!

ivan-penchev commented Mar 17, 2025

Uh oh!

Uh oh!

Uh oh!

ivan-penchev commented Mar 13, 2025 •

edited

Loading

ivan-penchev commented Mar 13, 2025 •

edited

Loading

ivan-penchev Mar 13, 2025 •

edited

Loading

ivan-penchev Mar 14, 2025 •

edited

Loading