Android screenshot tests (#2809)

* Make the screenshot tests run in an off frame buffer and correct gamma issues in screenshots

* Make the definition of "the same" a bit more flexible to avoid instability in the pipelines

* Restore gamma correction

* Set the colourspace for the off renderTexture

* Set the Srgb for the off renderTexture

* Rebake minor visual changes for off frame buffer

* Rearrange for submodule screenshot structure

* Add in dependencies for android

* Bring it closer to the android initialiser project

* Temporarily use non local android jme

* Simple screenshot test

* Wait for emulator

* Dont save a snapshot, use mesa

* Make the emulator logs less likely to hang

* Bit closer to the android-emulator-runner docs

* Add logging in test

* Try a more modern pixel copy

* Capture logcat

* Only get my applications logcat

* Pull logs inside app step

* Prevent snapshots (which are erroring)

* Actually grab the logcat

* Use the actual location of the file

* Save file to public directory

* The whole logcat isn't too long

* Pull image from quoted location

* Whole log cat actually a bit much to put in github logs

* Use latch to wait explicitly for write to complete

* Clean up the test code

* Go back to scoped storage as permissions do not allow downloads folder

* Pull file from within test

* Use explicit file location

* Revert "Use explicit file location"

This reverts commit b0ed44d.

* Revert "Pull file from within test"

This reverts commit b9d850c.

* Avoid auto unistall (which means I can't get the file)

* Try to get JME to render into the test

* Fresh emulator every time

* Take the screenshot from the glSurfaceView

* Move the screenshot framework to a shared environment

* Extract out one of the tests ready for reuse

* Start introducing shared android screenshot functionality

* Allow for test scenarios to be booted to both android and desktop

* Start splitting out android specific elements

* Introduce androidx-storage to save the report persistently

* Change from BufferedImage to JME Image for android compatibility

* Work towards allowing android to write a proto report

* Hook in workflow for android screenshots

* Hook in workflow for android screenshots

* Clean up ExtentReportExtensionJunit4

* Avoid reuse of AssetManager (as we are registering random folders)

* Give gradle more heap for build (specifically for the android build)

* Remove redundant request to write to external storage

* Run the adb pulls even if the gradle step fails

* Add the upgrade step for the report

* Use more modern android for jackson support

* Use x86_64 android

* Use x86_64 android

* Even more modern android

* Try 35 as docs only go up to 35

* Remove the matrix as we only run one

* Install test services before running test

* Debug error codes

* Create report directory before pulling to it

* Format the android screenshot tests readme

* Error if upgrade of report fails

* Correct upgrade code to access the files with absolute paths

* Try to trap the error code better

* One giant line of doom

* One giant line of doom 2

* Just say -1 for a fail

* Ensure report has a title

* Reinstate deserialisation constructor

* Migrate all tests to be android & desktop tests

* Ensure that we get a single test report

* Run the whole thing in a single shell

* Include logcat

* Increase ram for emulator

* Allow app to have large heap

* Even larger heap

* Remove some of the more memory intensive tests

* Remove some more of the more memory intensive tests

* Correct the calculateImageFilePrefix so it is flexible to thread depth which android can mess with)

* Remove more memory intensive tests

* Try for a wider device

* Try to find available profiles

* Try to find available profiles

* Initialise the reference images

* Load references images from resources (to make android happy)

* Correct colours are ints, not bytes (as unsigned)

* Use consistent pixel comparison

* Log pixel errors

* Fix inverted "very similar" test

* Set android tests to only run on push

* Restore android harness as it was

* Self review

* Single subshell to avoid EOF issues

* Avoid richard actually being contacted by the message

* Wrap whole shell in a >

* Natives moved out of project

* More recent java version

* Update location of desktop screenshot tests

* Fix AndroidTestHarness for change in AndroidHarnessFragment

* Ensure file load still works for screenshot tests

* Move reference files within normal resources

* All desktop screenshot tests with their correct base

* Use same layout as in the android example

* Simpler layout

* Correct javadoc

* Correct attached image save location

* Gen the generated images back from the android run if they are different sizes

* .png suffix on changeed images

* Restore accidentally changed ScenarioPostWater values

* Rebake images at new size

* Rebake images at new size

* Add missing copywright notices

* Add missing reference images

* Don't prematurely fail test, allow all images to be collected

* Add missing images

* Replace ImagePixelWrapper with the Raster for more cross format pixel manipulation

Author: richardTingle

.github/workflows/android-screenshot.yml

@@ -0,0 +1,84 @@
+name: Android Screenshot Test
+
+on:
+  push:
+    branches:
+      - master
+      - v3.7
+      - v3.6
+      - v3.5
+      - v3.4
+      - v3.3
+      - ios-2024_2
+  pull_request:
+
+jobs:
+  screenshot-test-android:
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    steps:
+      - name: checkout
+        uses: actions/checkout@v6
+      - name: Setup the java environment
+        uses: actions/setup-java@v5
+        with:
+          distribution: 'temurin'
+          java-version: '25'
+      - name: Enable KVM
+        run: |
+          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
+          sudo udevadm control --reload-rules
+          sudo udevadm trigger --name-match=kvm
+
+      - name: Gradle cache
+        uses: gradle/actions/setup-gradle@v5
+
+      - name: AVD cache
+        uses: actions/cache@v5
+        id: avd-cache
+        with:
+          path: |
+            ~/.android/avd/*
+            ~/.android/adb*
+          key: avd-35
+      - name: Run Android Screenshot Test
+        uses: reactivecircus/android-emulator-runner@v2
+        with:
+          api-level: 35
+          avd-name: test_avd
+          target: google_apis
+          arch: x86_64
+          force-avd-creation: true
+          ram-size: 4096M
+          heap-size: 2048M
+          profile: pixel_tablet
+          emulator-options: -no-window -gpu mesa -noaudio -no-boot-anim -camera-back none -no-snapshot -no-snapshot-save -no-snapshot-load
+          disable-animations: false
+          script: >
+            curl -L https://maven.google.com/androidx/test/services/test-services/1.6.0/test-services-1.6.0.apk --output test-services-1.6.0.apk;
+            adb install -r -g test-services-1.6.0.apk;
+            adb uninstall org.jmonkeyengine.screenshottests.android || true;
+            ./gradlew :jme3-screenshot-tests:jme3-screenshot-tests-android:connectedDebugAndroidTest;
+            exit_code=$?;
+            mkdir -p logcat;
+            adb logcat -d > logcat/logcat_full.txt || true;
+            adb logcat -d | grep org.jmonkeyengine.screenshottests.android > logcat/logcat.txt || true;
+            mkdir -p report;
+            adb pull /storage/emulated/0/googletest/test_outputfiles/report report/protoReport || true;
+            adb pull /storage/emulated/0/googletest/test_outputfiles/changed-images report/changed-images || true;
+            ./gradlew :jme3-screenshot-tests:jme3-screenshot-tests-proto-report:upgradeProtoReport --args="$(pwd)/report/protoReport $(pwd)/report/extentReport" || true;
+            echo "GRADLE_EXIT_CODE=$exit_code" >> $GITHUB_ENV;
+            exit $exit_code
+      - name: Upload logcat
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: android-logcat
+          path: logcat
+      - name: Upload Screenshot
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: android-screenshot-report
+          path: report
+          if-no-files-found: error

.github/workflows/main.yml

@@ -162,7 +162,7 @@ jobs:
         uses: gradle/actions/wrapper-validation@v6.1.0
       - name: Test with Gradle Wrapper
         run: |
-          ./gradlew :jme3-screenshot-test:screenshotTest
+          ./gradlew :jme3-screenshot-tests:jme3-screenshot-tests-desktop:screenshotTest
       - name: Upload Test Reports
         uses: actions/upload-artifact@v7.0.1
         if: always()

.github/workflows/screenshot-test-comment.yml

@@ -133,6 +133,6 @@ jobs:
 
             See https://github.com/jMonkeyEngine/jmonkeyengine/blob/master/jme3-screenshot-tests/README.md for more information
             
-            Contact @richardTingle (aka richtea) for guidance if required
+            Contact %40richardTingle (aka richtea) for guidance if required
           edit-mode: replace
           comment-id: ${{ steps.existingCommentId.outputs.comment-id }}

.gitignore

@@ -47,3 +47,8 @@ javadoc_deploy.pub
 !.vscode/JME_style.xml
 !.vscode/extensions.json
 joysticks-*.txt
+/jme3-screenshot-tests/jme3-screenshot-tests-desktop/build/
+/jme3-screenshot-tests/jme3-screenshot-tests-android/build/
+/jme3-screenshot-tests/jme3-screenshot-tests-shared/build/
+
+/jme3-screenshot-tests/jme3-screenshot-tests-proto-report/build/

build.gradle

@@ -51,10 +51,10 @@ allprojects {
 
 // This is applied to all sub projects
 subprojects {
-    if(!project.name.equals('jme3-android-examples')) {
-        apply from: rootProject.file('common.gradle')
-    } else {
+    if(project.name.equals('jme3-android-examples') || project.name.equals('jme3-screenshot-tests-android')) {
         apply from: rootProject.file('common-android-app.gradle')
+    } else {
+        apply from: rootProject.file('common.gradle')
     }
 
     def isAndroidApp = project.plugins.hasPlugin('com.android.application')

gradle.properties

@@ -33,3 +33,6 @@ POM_LICENSE_URL=http://opensource.org/licenses/BSD-3-Clause
 POM_LICENSE_DISTRIBUTION=repo
 POM_INCEPTION_YEAR=2009
 
+systemProp.org.gradle.unsafe.repositories=true
+
+org.gradle.jvmargs=-Xmx2048m

gradle/libs.versions.toml

@@ -11,6 +11,8 @@ saferalloc = "0.0.10"
 nifty = "1.4.3"
 spotbugs = "4.9.8"
 jmeAndroidNatives = "3.10.0-xt16kb-alloc"
+googleMaterial = "1.4.0"
+androidx-fragment-testing = "1.8.9"
 
 [libraries]
 
@@ -19,9 +21,21 @@ androidx-fragment = "androidx.fragment:fragment:1.8.9"
 androidx-lifecycle-common = "androidx.lifecycle:lifecycle-common:2.7.0"
 android-build-gradle = "com.android.tools.build:gradle:9.1.0"
 android-support-appcompat = "com.android.support:appcompat-v7:28.0.0"
+androidx-test-services = { module = "androidx.test.services:test-services", version = "1.6.0" }
+androidx-storage = { module = "androidx.test.services:storage" }  # used for peristent android test result storage, file version from androidx-fragment-testing-manifest
+androidxAppcompat = { module = "androidx.appcompat:appcompat", version = "1.4.0" }
+androidxConstraintlayout = { module = "androidx.constraintlayout:constraintlayout", version = "2.1.1" }
+androidx-test-core = "androidx.test:core:1.6.1"
+androidx-test-rules = "androidx.test:rules:1.6.1"
+androidx-test-ext-junit = "androidx.test.ext:junit:1.2.1"
+androidx-test-espresso-core = "androidx.test.espresso:espresso-core:3.6.1"
+androidx-fragment-testing = { module ="androidx.fragment:fragment-testing", version.ref = "androidx-fragment-testing"}
+androidx-fragment-testing-manifest = { module = "androidx.fragment:fragment-testing-manifest", version.ref = "androidx-fragment-testing"}
+gradle-git = "org.ajoberstar:gradle-git:1.2.0"
 androidx-test-runner = "androidx.test:runner:1.7.0"
 groovy-test = "org.apache.groovy:groovy-test:4.0.31"
 gson = "com.google.code.gson:gson:2.14.0"
+googleMaterial = { module = "com.google.android.material:material", version.ref = "googleMaterial" }
 j-ogg-vorbis = "com.github.stephengold:j-ogg-vorbis:1.0.6"
 jme3-android-natives = { module = "org.jmonkeyengine:jme3-android-native", version.ref = "jmeAndroidNatives" }
 jbullet = "com.github.stephengold:jbullet:1.0.3"
@@ -75,8 +89,13 @@ saferalloc-natives-macos-aarch64 = { module = "org.ngengine:saferalloc-natives-m
 saferalloc-natives-android = { module = "org.ngengine:saferalloc-natives-android", version.ref = "saferalloc" }
 saferalloc-natives-ios = { module = "org.ngengine:saferalloc-natives-ios", version.ref = "saferalloc" }
 
+extent-reports = { module = "com.aventstack:extentreports", version = "5.1.2"}
+
+jackson = {module = "tools.jackson.core:jackson-databind", version ="3.1.3"}
+
 [bundles]
 saferalloc = ["saferalloc", "saferalloc-natives-linux-x8664", "saferalloc-natives-linux-aarch64", "saferalloc-natives-windows-x8664", "saferalloc-natives-windows-aarch64", "saferalloc-natives-macos-x8664", "saferalloc-natives-macos-aarch64", "saferalloc-natives-android", "saferalloc-natives-ios"]
 
 [plugins]
 jacoco = { id = "jacoco", version.ref = "jacoco" }
+

jme3-screenshot-tests/build.gradle

@@ -5,46 +5,3 @@ plugins {
 repositories {
     mavenCentral()
 }
-
-dependencies {
-    implementation project(':jme3-desktop')
-    implementation project(':jme3-core')
-    implementation project(':jme3-effects')
-    implementation project(':jme3-terrain')
-    implementation project(':jme3-lwjgl3')
-    implementation project(':jme3-plugins')
-    implementation project(':jme3-saferallocator')
-
-    implementation 'com.aventstack:extentreports:5.1.2'
-    implementation platform('org.junit:junit-bom:5.9.1')
-    implementation 'org.junit.jupiter:junit-jupiter'
-    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
-    testRuntimeOnly project(':jme3-testdata')
-}
-
-testing {
-    suites {
-        test {
-            useJUnitJupiter('5.9.1')
-        }
-    }
-}
-
-tasks.register("screenshotTest", Test) {
-    testClassesDirs = sourceSets.test.output.classesDirs
-    classpath = sourceSets.test.runtimeClasspath
-    useJUnitPlatform{
-        filter{
-            includeTags 'integration'
-        }
-    }
-}
-
-
-test {
-    useJUnitPlatform{
-        filter{
-            excludeTags 'integration'
-        }
-    }
-}

jme3-screenshot-tests/jme3-screenshot-tests-android/build.gradle

@@ -0,0 +1,68 @@
+plugins {
+    id "io.github.0ffz.github-packages" version "1.2.1" // Plugin for anonymous inclusion of artifacts hosted in github package registry
+}
+
+apply plugin: 'com.android.application'
+
+android {
+    namespace "org.jmonkeyengine.screenshottests.android"
+    compileSdk 36
+
+    defaultConfig {
+        applicationId "org.jmonkeyengine.screenshottests.android"
+        minSdk 28
+        targetSdk 36
+        versionCode 1
+        versionName "1.0"
+
+        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
+    }
+
+    buildTypes {
+        release {
+            minifyEnabled false
+            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
+        }
+    }
+    configurations.all {
+        exclude group:"org.jmonkeyengine",module:"jme3-desktop"
+    }
+
+    packagingOptions {
+        jniLibs {
+            useLegacyPackaging = true
+        }
+        resources {
+            merges += "META-INF/LICENSE.md"
+            merges += "META-INF/LICENSE-notice.md"
+        }
+    }
+}
+
+dependencies {
+    implementation libs.androidxAppcompat
+    implementation libs.googleMaterial
+    implementation libs.androidxConstraintlayout
+
+    implementation project(":jme3-android")
+    implementation libs.jme3.android.natives
+
+    implementation libs.androidx.test.core
+    implementation libs.androidx.test.ext.junit
+    implementation libs.androidx.test.rules
+    implementation libs.androidx.test.espresso.core
+    implementation libs.androidx.fragment.testing
+    implementation libs.jackson
+    debugImplementation libs.androidx.fragment.testing.manifest
+    androidTestImplementation libs.androidx.fragment.testing.manifest
+    debugImplementation libs.androidx.storage
+
+    implementation project(':jme3-screenshot-tests:jme3-screenshot-tests-shared')
+    implementation project(':jme3-screenshot-tests:jme3-screenshot-tests-proto-report')
+
+    implementation project(':jme3-testdata')
+
+    // Allows the test to communicate with the orchestrator storage
+    androidTestImplementation libs.androidx.test.services
+    debugImplementation libs.androidx.test.services
+}

jme3-screenshot-tests/jme3-screenshot-tests-android/proguard-rules.pro

@@ -0,0 +1,21 @@
+# Add project specific ProGuard rules here.
+# You can control the set of applied configuration files using the
+# proguardFiles setting in build.gradle.
+#
+# For more details, see
+#   http://developer.android.com/guide/developing/tools/proguard.html
+
+# If your project uses WebView with JS, uncomment the following
+# and specify the fully qualified class name to the JavaScript interface
+# class:
+#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
+#   public *;
+#}
+
+# Uncomment this to preserve the line number information for
+# debugging stack traces.
+#-keepattributes SourceFile,LineNumberTable
+
+# If you keep the line number information, uncomment this to
+# hide the original source file name.
+#-renamesourcefileattribute SourceFile