README: "Excluding other parameters" section: "using up" combinations

[dan-tripp]

There something I don't understand in the README.

It's under "Excluding other parameters" - the idea of "using up" combinations. eg.:

1. In these cases we do not want to 'use up' any testable values for other parameters.
2. as it would ... use up P2/P3 combinations
3. no P2 and P3 combinations are used up for the -1 test

As far as I can tell, "using up" means that a certain P2/P3 combination appears in a certain test X, along with P1 == -1, and that P2/P3 combination appears in no other tests. For brevity, I'll call this downside "masking". (Elsewhere, like in the "Pairwise Testing in Real World" paper, "masking" means something more specific: when "one invalid input prevents another invalid input from being tested" (emphasis mine). I hope it's okay if I ignore that for now.)

The "dummy value technique" (as compared to the "~" operator technique), has no advantage in terms of masking. That is: both techniques solve the masking problem. Here's where I get confused. Why does the section on the "dummy value technique" repeatedly claim that one of it's benefits is that it solves the masking problem? I would understand if it said that the "dummy value technique" is another way of solving the masking problem - i.e. it's an alternative to the "~" operator technique. But it doesn't say that.

There's much I could be missing here. I hope someone can shed light on it.

I'll be glad to submit a PR for the documentation. (I'm free to do so because unlike #96 I'm working on my own time now, not my employer's time)

Love the tool. I'm giving a talk on PICT and combinatorial testing in general at a conference a few weeks from now.

[jaccz]

Dan, a good question. These two things are unrelated and the docs should make it clearer. Currently “Excluding other parameters” is a section under “Negative values” where it should have been its own separate thing. “Negative testing” tries to handle that case where e.g. error handling is applied to each input separately: first check if a >= 0, then check if b >= 0 as in the SumSquareRoots example. We explicitly don’t want both a and b to be set to NULL in one test as the check for b could never get executed. “Excluding entire parameters” is a separate thing coming from cases where I saw users wanting to express the following logic: “if param P is set to this specific value, I don’t want other parameters to be set to anything at all. E.g. don’t set them to any values I provided in the model because I’d prefer to see them in other tests”. So don’t “use up” my good values for a case where software doesn’t care about their setting. PICT doesn’t have good handling for that case so that section of the doc is trying to say (clumsily): here’s a way to express that logic via constraints. Am I helping or muddying the picture even more? 😊

…

--Jacek From: Dan Tripp ***@***.***> Date: Wednesday, April 19, 2023 at 3:57 PM To: microsoft/pict ***@***.***> Cc: Subscribed ***@***.***> Subject: [microsoft/pict] README: "Excluding other parameters" section: "using up" combinations (Issue #101) There something I don't understand in the README. It's under "Excluding other parameters" - the idea of "using up" combinations. eg.: 1. In these cases we do not want to 'use up' any testable values for other parameters. 2. as it would ... use up P2/P3 combinations 3. no P2 and P3 combinations are used up for the -1 test As far as I can tell, "using up" means that a certain P2/P3 combination appears in a certain test X, along with P1 == -1, and that P2/P3 combination appears in no other tests. For brevity, I'll call this downside "masking". (Elsewhere, like in the "Pairwise Testing in Real World"<https://raw.githubusercontent.com/microsoft/pict/main/doc/Pairwise%20Testing%20in%20Real%20World.pdf> paper, "masking" means something more specific: when "one invalid input prevents another invalid input from being tested" (emphasis mine). I hope it's okay if I ignore that for now.) The "dummy value technique" (as compared to the "~" operator technique), has no advantage in terms of masking. That is: both techniques solve the masking problem. Here's where I get confused. Why does the section on the "dummy value technique" repeatedly claim that one of it's benefits is that it solves the masking problem? I would understand if it said that the "dummy value technique" is another way of solving the masking problem - i.e. it's an alternative to the "~" operator technique. But it doesn't say that. There's much I could be missing here. I hope someone can shed light on it. I'll be glad to submit a PR for the documentation. (I'm free to do so because unlike #96<#96> I'm working on my own time now, not my employer's time) Love the tool. I'm giving a talk on PICT and combinatorial testing in general at a conference a few weeks from now. — Reply to this email directly, view it on GitHub<#101> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACFRMV3JAF3LO7LTBORA47DXCBUVRBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJLJONZXKZNENZQW2ZNLORUHEZLBMRPXI6LQMWBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTLDTOVRGUZLDORPXI6LQMWSUS43TOVS2M5DPOBUWG44SQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKBUGQZTSMZSGMZIFJDUPFYGLJLJONZXKZNFOZQWY5LFVIYTMNZVG42DONBUGKTXI4TJM5TWK4VGMNZGKYLUMU>. You are receiving this email because you are subscribed to this thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[dan-tripp]

You're helping, no question! I will read this a few more times and respond here again.

[dan-tripp]

It seems that the dummy value technique has two benefits:

Benefit 1: "don’t set them [i.e. the valid params] to any values I provided in the model"
Benefit 2: "don’t “use up” my good values for a case where software doesn’t care about their setting"

Here's where I don't see it your way yet: it seems to me that negative testing does not have benefit 1 and does have benefit 2. So (it seems to me) that the dummy value technique is related to negative testing.

I hope I'm correct in saying that negative testing has benefit 2. I say that because this model:

P1: -1, 0, 1
P2:  A, B, C
P3:  X, Y, Z

... produces this output:

P1  P2  P3
0   A   Y
0   B   Z
0   C   X
1   A   X
1   A   Z
1   B   Y
1   C   Y
-1  A   Y
-1  B   X
-1  C   Z

... and in that output, the combinations "B X" and "C Z" are "used up".

This model:

P1: ~-1, 0, 1
P2:  A, B, C
P3:  X, Y, Z

... produces this output:

P1   P2  P3
~-1  A   Y
~-1  B   X
~-1  C   Z
0    A   Y
0    B   Z
0    C   X
0    C   Z
1    A   X
1    A   Z
1    B   X
1    B   Y
1    C   Y

... and no combinations are "used up" in that output.

If I'm right so far: it seems to me that these two techniques are closely related. And the techniques can be closely related, even though the kinds of users and models that motivate using one technique over the other are quite different.

I would even go so far as to say that the 'dummy value technique' can be seen as an extension of 'negative testing'. That is: negative testing will get you half-way there, in terms of doing what you want to do with that "NA" model. It will give you benefit 2 (but not benefit 1), and then when you're going through your test cases you just have to ignore P2 and P3. You can treat them like "don't care" values (to borrow a term from ACTS).

What do you think?

[jaccz]

I see how this can be confusing but you got the main point right, which that in practice, you may see both techniques as related to testing for scenarios that aren’t the success / happy path. Eliminating parameters has a purpose beyond negative testing. Let me try to go at it with an example. Assume you need to test rendering of some website on multiple mobile platforms. The main point is that on iOS there are no 3rd party rendering engines so 3rdPartyRenderingEngine is not relevant. # Service side Cloud: AWS, Azure, GCP ServiceVersion: 1.0, 2.0 # Client side MobileOS: Android, iOS, Custom Browser: Native, 3rdParty 3rdPartyRenderingEngine: Chrome, Edge IF [MobileOS] = "iOS" THEN [Browser] = "Native"; If [3rdPartyRenderingEngine] = "Edge" THEN [Browser] = "3rdParty"; You could go with the above but you will see tests like this: AWS 2.0 iOS Native Chrome You could theoretically ignore those values in your test runner (at runtime) for iOS but as far as test design, you just used “Native Chrome” pair in a test and it may not show up anywhere else. Better to “eliminate” the entire parameter by adding a “NA” value and a couple of constraints. 3rdPartyRenderingEngine: Chrome, Edge, NA IF [MobileOS] = "iOS" THEN [3rdPartyRenderingEngine] = "NA"; IF [MobileOS] <> "iOS" THEN [3rdPartyRenderingEngine] <> "NA"; Admittedly, this example is very artificial but the idea is occasionally useful.

…

--Jacek From: Dan Tripp ***@***.***> Date: Friday, April 21, 2023 at 3:39 PM To: microsoft/pict ***@***.***> Cc: Comment ***@***.***>, Subscribed ***@***.***> Subject: Re: [microsoft/pict] README: "Excluding other parameters" section: "using up" combinations (Issue #101) It seems that the dummy value technique has two benefits: Benefit 1: "don’t set them [i.e. the valid params] to any values I provided in the model" Benefit 2: "don’t “use up” my good values for a case where software doesn’t care about their setting" Here's where I don't see it your way yet: it seems to me that negative testing does not have benefit 1 and does have benefit 2. So (it seems to me) that the dummy value technique is related to negative testing. I hope I'm correct in saying that negative testing has benefit 2. I say that because this model: P1: -1, 0, 1 P2: A, B, C P3: X, Y, Z ... produces this output: P1 P2 P3 0 A Y 0 B Z 0 C X 1 A X 1 A Z 1 B Y 1 C Y

-1 A Y

-1 B X

-1 C Z ... and in that output, the combinations "B X" and "C Z" are "used up". This model: P1: ~-1, 0, 1 P2: A, B, C P3: X, Y, Z ... produces this output: P1 P2 P3 ~-1 A Y ~-1 B X ~-1 C Z 0 A Y 0 B Z 0 C X 0 C Z 1 A X 1 A Z 1 B X 1 B Y 1 C Y ... and no combinations are "used up" in that output. If I'm right so far: it seems to me that these two techniques are closely related. And the techniques can be closely related, even though the kinds of users and models that motivate using one technique over the other are quite different. I would even go so far as to say that the 'dummy value technique' can be seen as an extension of 'negative testing'. That is: negative testing will get you half-way there, in terms of doing what you want to do with that "NA" model. It will give you benefit 2 (but not benefit 1), and then when you're going through your test cases you just have to ignore P2 and P3. You can treat them like "don't care" values (to borrow a term from ACTS). What do you think? — Reply to this email directly, view it on GitHub<#101 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACFRMV45CGGESVCIEOIMNH3XCMECVBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVA2DIMZZGMZDGMUCUR2HS4DFUVUXG43VMWSXMYLMOVS2UMJWG42TONBXGQ2DFJ3UOJUWOZ3FOKTGG4TFMF2GK>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jaccz]

It is worth doing. I’m not sure this is the best example but it was one that came to mind today. Maybe you see a better one somewhere but for now, if you think this one is clearer, I’d love a doc revision.

…

--Jacek From: Dan Tripp ***@***.***> Date: Monday, April 24, 2023 at 3:55 PM To: microsoft/pict ***@***.***> Cc: Comment ***@***.***>, Subscribed ***@***.***> Subject: Re: [microsoft/pict] README: "Excluding other parameters" section: "using up" combinations (Issue #101) I find this example (with parameter names like MobileOS, Browser, and so on) to be less artificial than the current example in the PICT docs (which has the parameter names P1, P2, P3). So maybe I could distill this MobileOS example into a PR for the PICT docs, to replace the "P1, P2, P3" example. I'll try to do that - unless, of course, you think it's not worth doing. I defer to your expertise. — Reply to this email directly, view it on GitHub<#101 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACFRMV6JETNVKYA6T4EDTZLXC4AGHBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVA2DIMZZGMZDGMUCUR2HS4DFUVUXG43VMWSXMYLMOVS2UMJWG42TONBXGQ2DFJ3UOJUWOZ3FOKTGG4TFMF2GK>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[dan-tripp]

(Sorry, deleted my comment. Here it is again:)

I find this example (with parameter names like MobileOS, Browser, and so on) to be less artificial than the current example in the PICT docs (which has the parameter names P1, P2, P3). So maybe I could distill this MobileOS example into a PR for the PICT docs, to replace the "P1, P2, P3" example.

I'll try to do that - unless, of course, you think it's not worth doing. I defer to your expertise.

[dan-tripp]

Excellent. Thank you for your time on this. Not sure when, but I'll try to deliver something of value on this.

[dan-tripp]

I made a PR. I had a hard time understanding the "3rdPartyRenderingEngine" part of your model, so I borrowed a model from Introduction to Combinatorial Testing by Kuhn et. al (page 48) with "DotNetVersion". Let me know what you think.