Skip to content

Commit 59c8744

Browse files
YOU54FYOU54F
authored
Merge pull request #99 from tienvx/define-matching-rules-for-form-urlencoded-body
rfc: Define matching rules for form-urlencoded body
2 parents a38cdc9 + b81ffbb commit 59c8744

1 file changed

Lines changed: 333 additions & 0 deletions

File tree

‎rfc/0000-define-matching-rules-for-form-urlencoded-body.md‎

Lines changed: 333 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,333 @@
1+
---
2+
name: define-matching-rules-for-form-urlencoded-body
3+
started: 2024-09-06
4+
pr: pact-foundation/roadmap#0000
5+
tracking_issue: pact-foundation/roadmap#0000
6+
---
7+
## Summary
8+
9+
This RFC proposes defining matching rules for bodies of type `application/x-www-form-urlencoded`.
10+
11+
## Motivation
12+
13+
Currently we have the method [`match_form_urlencoded`](https://github.com/pact-foundation/pact-reference/blob/master/rust/pact_matching/src/form_urlencoded.rs#L12) to match the `application/x-www-form-urlencoded` request's body using matching rules.
14+
But there is no way to define matching rules for the body to fully take advantage of this feature.
15+
16+
The only a way to define body for a `application/x-www-form-urlencoded` request is to define it as-is.
17+
18+
For example, we can define the request's body as `fullname=My+Full+Name&email=test%40example.com&password=abc%40123`.
19+
Then the consumer **must** send the request with body exactly like this: `fullname=My+Full+Name&email=test%40example.com&password=abc%40123`, or
20+
otherwise consumer's Pact test will failed, and the pact file will not be written.
21+
22+
That's because we can't define matching rules with this raw syntax. With the ability to define matching rules with the alternative syntax,
23+
consumer will be able to send different values.
24+
25+
This example is focusing on request's body, but it also apply to response's body.
26+
27+
## Guide-level explanation
28+
29+
### With matching rules
30+
31+
When the RFC is implemented, the example usage would be:
32+
33+
```rust
34+
let json = json!({
35+
"number": {
36+
"pact:matcher:type": "number",
37+
"value": 23.45
38+
},
39+
"string": {
40+
"pact:matcher:type": "type",
41+
"value": "example text"
42+
},
43+
"array": {
44+
"pact:matcher:type": "eachValue(matching(regex, 'value1|value2|value3|value4', 'value2'))",
45+
"value": ["value1", "value4"]
46+
}
47+
});
48+
pactffi_with_body(interaction, InteractionPart::Request, "application/x-www-form-urlencoded", json);
49+
```
50+
51+
After the matching rules are extracted, the example body will be returned:
52+
53+
```query
54+
number=23.45&string=example+text&array=value1&array=value4
55+
```
56+
57+
This example body (along with matchers) will be written into pact file.
58+
59+
### Without matching rules
60+
61+
We can also define body without matching rules:
62+
63+
```rust
64+
let json = json!({
65+
"number": 123,
66+
"string": "example value",
67+
"array": [null, -123.45, "inner text"],
68+
});
69+
pactffi_with_body(interaction, InteractionPart::Request, "application/x-www-form-urlencoded", json);
70+
```
71+
72+
The example body will be extracted and look like this:
73+
74+
```
75+
number=123&string=example+value&array=-123.45&array=inner+text
76+
```
77+
78+
It will be exactly the same as we define the body using raw syntax:
79+
80+
```rust
81+
let raw = "number=123&string=example+value&array=-123.45&array=inner+text";
82+
pactffi_with_body(interaction, InteractionPart::Request, "application/x-www-form-urlencoded", raw);
83+
```
84+
85+
### Generators
86+
87+
Generators can be defined like this:
88+
89+
```rust
90+
let json = json!({
91+
"id": {
92+
"pact:matcher:type": "regex",
93+
"regex": "^[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}$",
94+
"pact:generator:type": "Uuid"
95+
},
96+
"age": {
97+
"pact:matcher:type": "integer",
98+
"value": 0,
99+
"max": 130,
100+
"min": 0,
101+
"pact:generator:type": "RandomInt"
102+
},
103+
"name": [
104+
{
105+
"pact:matcher:type": "regex",
106+
"regex": "Mr\\.|Mrs\\.|Miss|Ms\\.",
107+
"value": "",
108+
"pact:generator:type": "Regex"
109+
},
110+
{
111+
"pact:matcher:type": "type",
112+
"value": "",
113+
"size": 10,
114+
"pact:generator:type": "RandomString"
115+
}
116+
]
117+
});
118+
pactffi_with_body(interaction, InteractionPart::Response, "application/x-www-form-urlencoded", json);
119+
```
120+
121+
Consumer will receive response's body like this from mock server:
122+
123+
```
124+
id=46441a68-1d3d-40f2-bbec-6e9bd26a4047&age=65&name=Mr.&name=Ea8XIXh7NQ
125+
```
126+
127+
### Unsupported syntax
128+
129+
These values are not supported:
130+
131+
- Null: There is no way to represent null in query string. User may want to define empty string instead.
132+
- Boolean (true/false): There is no standard way to represent boolean in query string. It can be 1/0, true/false or t/f. User need to define them explicitly.
133+
- Object: There is no standard way to represent object in query string.
134+
- Array of Arrays: There is no standard way to represent array of arrays in query string.
135+
- Array of Objects: There is no standard way to represent array of objects in query string.
136+
137+
138+
#### With matching rules:
139+
140+
```rust
141+
let json = json!({
142+
"null": {
143+
"pact:matcher:type": "null"
144+
},
145+
"true": {
146+
"pact:matcher:type": "boolean",
147+
"value": true
148+
},
149+
"false": {
150+
"pact:matcher:type": "boolean",
151+
"value": false
152+
},
153+
"object": {
154+
"pact:matcher:type": "type",
155+
"value": {
156+
"key" => {
157+
"pact:matcher:type": "type",
158+
"value": "value"
159+
}
160+
}
161+
},
162+
"array_of_arrays": {
163+
"pact:matcher:type": "type",
164+
"value": [["value1", "value2"]]
165+
},
166+
"array_of_objects": {
167+
"pact:matcher:type": "type",
168+
"value": [{"key": "value"}]
169+
}
170+
});
171+
pactffi_with_body(interaction, InteractionPart::Request, "application/x-www-form-urlencoded", json);
172+
```
173+
174+
These matchers and values are ignored. Unsupported error messages are logged. The extracted example body will be empty.
175+
176+
#### Without matching rules:
177+
178+
```rust
179+
let json = json!({
180+
"null": null,
181+
"true": true,
182+
"false": false,
183+
"object": {
184+
"key": "value"
185+
},
186+
"array_of_arrays": [["value1", "value2"]],
187+
"array_of_objects": [{"key": "value"}]
188+
});
189+
pactffi_with_body(interaction, InteractionPart::Request, "application/x-www-form-urlencoded", json);
190+
```
191+
192+
The values are ignored. Unsupported error messages are logged. The extracted example body will be empty.
193+
194+
### Special cases
195+
196+
- keys with no values (`a=&b=&c=`): can be achieved by define empty string value
197+
- values with no keys (`=a&=b&=c`)
198+
- can be defined by json
199+
- can be defined by raw syntax
200+
- ordering does matter:
201+
- `=c&=a&=b` does not match `=a&=b&=c`
202+
- repeated keys with different ordering: (`a=1&a=2` vs `a=2&a=1`)
203+
- can be defined by json
204+
- can be defined by raw syntax
205+
- ordering does matter:
206+
- `a=1&a=2` does not match `a=2&a=1`
207+
- `a=2&a=1` does not match `a=1&a=2`
208+
- special characters (in key and value)
209+
- `/`: will be encoded to `%2F`
210+
- `&`: will be encoded to `%26`
211+
- `?`: will be encoded to `%3F`
212+
- `=`: will be encoded to `%3D`
213+
- only ampersands (`&&&`)
214+
- can't be defined by json
215+
- can be defined by raw syntax
216+
- these query strings can match it:
217+
- `&`
218+
- `&&`
219+
- `&&&`
220+
- ...
221+
- empty string can't match it
222+
- only equals signs (`===`)
223+
- can't be defined by json because all of these cases are detected as `text/plain` by pact:
224+
- `{"==":""}`
225+
- `{"=":"="}`
226+
- `{"":"=="}`
227+
- can be defined by raw syntax
228+
- these query strings can match it:
229+
- `===`
230+
- `=%3D%3D`
231+
- empty string can't match it
232+
- no key and value (`=&=&=`)
233+
- can be defined by json
234+
- can be defined by raw syntax
235+
- these query strings can't match it:
236+
- ``
237+
- `=&`
238+
- `=&=`
239+
- `=&=&`
240+
- `=&=&=&=`
241+
- `=&=&=&=&`
242+
- ...
243+
- these query strings can match it:
244+
- `=&=&=`
245+
- `=&=&=&`
246+
- repeated keys with empty string value and different ordering: (`a=&a=1` vs `a=1&a=` vs `a=1` vs `a=`)
247+
- can be defined by json
248+
- can be defined by raw syntax
249+
- ordering and size does matter:
250+
- `a=&a=1` does not match `a=1&a=`
251+
- `a=&a=1` does not match `a=1`
252+
- `a=&a=1` does not match `a=`
253+
- ...
254+
255+
## Reference-level explanation
256+
257+
Here is the flow we need to implement in Rust core (pact-reference project):
258+
259+
- Pact implementation call FFI method `pactffi_with_body` with:
260+
- Content type: `application/x-www-form-urlencoded`
261+
- Body: Integration JSON format , which may include:
262+
- Matching rules
263+
- Example values
264+
- Generators
265+
- Rust core will process the body:
266+
- If content type hint is `application/x-www-form-urlencoded` AND detected content type from body is `application/json`
267+
- Extract matching rules
268+
- Extract generators
269+
- Return example JSON body with example values e.g. `{ "key": "example value" }`
270+
- Example JSON body will be converted to example Form UrlEncoded body e.g. `key=example+value`
271+
- These example values (including matchers and generators) will be ignored:
272+
- Null
273+
- Boolean (true/false)
274+
- Object
275+
- Array of Arrays
276+
- Array of Objects
277+
- Register the interaction as normal with:
278+
- Extracted matching rules
279+
- Extracted generators
280+
- Example Form UrlEncoded body
281+
- Content type: `application/x-www-form-urlencoded`
282+
283+
## Drawbacks
284+
285+
We need to modify Rust core (pact-reference project)
286+
287+
## Rationale and alternatives
288+
289+
Alternative approaches:
290+
291+
### Hybrid syntax
292+
293+
```rust
294+
let text = "number=matching(number, 123)&string=matching(regex, '\w\d', 'a1')&array=matching(eachValue(matching(regex, '\d{3}', '221')))"
295+
pactffi_with_body(interaction, InteractionPart::Request, "application/x-www-form-urlencoded", text);
296+
```
297+
298+
This is just a pseudocode. Parsing this syntax will be harder than Integration JSON format because we need to implement new code for it.
299+
300+
### Plugin
301+
302+
We can create a custom plugin to provide matching rules for `application/x-www-form-urlencoded` request.
303+
304+
I already create a POC for it at https://github.com/tienvx/pact-form-urlencoded-plugin
305+
306+
The problem with it are:
307+
308+
* We need to move matchers from Rust core to this plugin.
309+
* The performance will be bad compare to implementing inside Rust core (pact-reference project)
310+
311+
## Unresolved questions
312+
313+
- There will be some bugs in Rust core need to be fix through the implementation of this RFC
314+
- Postel's law need to be followed outside of the implementation of this RFC
315+
- Due to lack of standards, some of the syntax will not be supported
316+
317+
## Future possibilities
318+
319+
### Support arrays and objects
320+
321+
Probably not a good idea, because there are no standard syntax, and each language handle it differently
322+
323+
Reference: https://blog.shalvah.me/posts/fun-stuff-representing-arrays-and-objects-in-query-strings
324+
325+
### Follow Postel's law
326+
327+
Following Postel's law, the provider may return fields that the consumer will just ignore. But if provider send extra param in body, we will got this mismatch:
328+
329+
```
330+
Unexpected form post parameter 'extra' received
331+
```
332+
333+
I believe this is a bug, and it happen independently with this RFC, we need to fix it in the future PRs.

0 commit comments

Comments
 (0)