Proposal for JSON Based Demo Results

[bernhste]

Hi all

I'm the developer of the MAS Reference App (https://github.com/Redguard/mas-reference-app) and I would like to contribute the demos to MAS. My idea is to use a script which extracts the MAS Reference App demos which can be used as MAS Demos. Then, I would be able to one by one create merge requests which then could be integrated into MASTG.

So I took a look at the MAS Test Apps and I would like to discuss a change:

Right now, the Demos return a simple String. According to the Documentation (https://docs.google.com/document/d/1EMsVdfrDBAu0gmjWAUEs60q-fWaOmDB5oecY9d9pOlg/edit?tab=t.0#heading=h.y7p2nsbtajht) there are FAIL and PASS states.

My proposal is to use a JSON array for the demo results. An example:

[
  {
    "status": "PASS",
    "testId": "[MASTG-TEST-0000]",
    "message": "The app implemented a demo which passed the test with the following value: 'Hello from the OWASP MASTG Test app.'"
  },
  {
    "status": "FAIL",
    "testId": "[MASTG-TEST-0000]",
    "message": "The app implemented a demo which failed the test."
  },
  {
    "status": "ERROR",
    "testId": "[MASTG-TEST-0000]",
    "message": "java.lang.Exception: Something went wrong during the demo."
  }
]

I forked https://github.com/cpholguera/MASTestApp-Android to implement a small PoC. For the result I created a class DemoResults which takes care of the results.

So when you write a new demo, you would implement the demo in the following way

fun mastgTest(): String {
    val r = DemoResults("0000")

    try {
        val sensitiveString = "Hello from the OWASP MASTG Test app."
        Log.d("MASTG-TEST", sensitiveString)

        // case 1: Demo implements a case which passes a test
        r.add(Status.PASS, "The app implemented a demo which passed the test with the following value: '$sensitiveString'" )

        // case 2: Demo implements a case which fails a test
        r.add(Status.FAIL, "The app implemented a demo which failed the test." )

        throw Exception("Something went wrong during the demo.")
    }
    catch (e: Exception){
        // case 3: Demo fails due to an exception.
        r.add(Status.ERROR, e.toString())
    }
    return r.toJson()
}

The test app could then use it to show the results accordingly:

In my option, this would have the following benefits:

• The structured demo result could lead to better demo quality as the result state must match the status specified in the documentation.
• It provides flexibility within a demo, as one demo can cover different cases (PASS,FAIL,ERROR due to exception) easily.
• We could use the DemoResult class to log the different results in one place and make it easier to automatially assess the results of the different demos during runtime.
• I would be able to migrate the MAS Reference App demos more easily as their return values are similar.

The PoC is compatible to the existing demos, meaning, it does not break if mastgTest() does not return a JSON string.

To illustrate the new demo result, I used MASTG-DEMO-0007 and migrated it to the new MAS Test App:

Current Code:

fun mastgTest(): String {

    // FAIL: [android-insecure-random-use] The app insecurely uses random numbers for generating authentication tokens.
    val random1 = Random().nextDouble()

    // FAIL: [android-insecure-random-use] The title of the function indicates that it generates a random number, but it is unclear how it is actually used in the rest of the app. Review any calls to this function to ensure that the random number is not used in a security-relevant context.
    val random2 = 1  +  Math.random()

    val length = 16
    val characters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
    val random = Random()
    val password = StringBuilder(length)

    for (i in 0 until length)  {
        // FAIL: [android-insecure-random-use] The app insecurely uses random numbers for generating passwords, which is a security-relevant context.
        password.append(characters[random.nextInt(characters.length)])
    }

    val random3 = password.toString()

    // PASS: [android-insecure-random-use] The app uses a secure random number generator.

    val random4 = SecureRandom().nextInt(21)

    return "Generated random numbers:\n$random1 \n$random2 \n$random3 \n$random4"
}

Current output:

Proposed code:

fun mastgTest(): String {
    val r = DemoResults("0007")

    val random1 = Random().nextDouble()
    r.add(Status.FAIL, "The app insecurely uses random numbers for generating authentication tokens. Generated number: $random1")

    val random2 = 1  +  Math.random()
    r.add(Status.FAIL, "The title of the function indicates that it generates a random number, but it is unclear how it is actually used in the rest of the app. Review any calls to this function to ensure that the random number is not used in a security-relevant context. Generated number: $random2")

    val length = 16
    val characters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
    val random = Random()
    val password = StringBuilder(length)
    for (i in 0 until length)  {
        password.append(characters[random.nextInt(characters.length)])
    }
    val random3 = password.toString()
    r.add(Status.FAIL, "The app insecurely uses random numbers for generating passwords, which is a security-relevant context. Generated number: $random3")

    val random4 = SecureRandom().nextInt(21)
    r.add(Status.PASS, "The app uses a secure random number generator. Generated number: $random4")

    return r.toJson()
}

Proposed output:

And finally, we have a log entry per test which could be used to automatically assess the result:

Please let me know if this is something which works for you in general. If so, I'm happy to provide the merge requests for Android and iOS.