[Identity] Project Orion

[Pearapps]

Progress on #13048

Android counterpart to stripe/stripe-ios#6523.

This started as a pass on the selfie capture UX but has gotten a lot bigger since the original version

It now includes most of the actual Orion / 3D selfie capture flow on Android, including the updated warmup and camera UI, MediaPipe face pose detection, ordered left and right capture, server driven gating, uploads, and the additional API data

• updates the selfie warmup and overall capture flow

  • updates the warmup illustration and copy
  • displays the server-provided training consent text when it is present
  • adds separate Allow and Decline actions for training consent
  • saves that choice and sends it with the selfie data
  • keeps the normal continue button when the server does not provide training consent copy
  • keeps the user in the camera experience while the captured images are uploaded and checked
  • freezes, mirrors, blurs, and slightly dims the final captured image
  • shows Great! Checking your images.... with an activity indicator over that image
  • announces the checking state for accessibility
  • keeps Having Trouble? available during capture and preserves the fallback URL behavior
  • keeps the existing legacy selfie flow when 3D is not enabled

• adds the new oval capture guide from the mockups

  • draws 68 individual tick marks around the face
  • uses the centered feathered shadow around the guide
  • fades the shadow in once a valid face has been found
  • turns the ticks green and makes them longer and thicker as captures are accepted
  • fills the complete oval when the front selfie is accepted
  • uses MediaPipe yaw to progressively fill only the requested half during a side pose
  • shows no green side progress when yaw is at zero, missing, or not usable
  • dims the opposite half while the user is turning
  • keeps that opposite half dim while the requested side is being accepted
  • adds the animated left and right directional arrows
  • adds the animated confirmation checkmark after each accepted capture
  • fades the status copy in and out instead of abruptly replacing it

• updates all of the live capture states and copy

  • Place your face within the frame
  • Hold still...
  • Captured front
  • Turn head right
  • Captured right
  • Turn head left
  • Captured left
  • Great! Checking your images....
  • only includes the strings that are actually used by the flow
  • blurs the live camera during the initial side-pose instruction
  • clears that blur before reading the user's side-pose progress
  • blurs again briefly when a side capture is accepted

• adds the full front and side-pose capture sequence

  • always captures the normal front samples first
  • continues using the API-provided num_samples and sample_interval for front capture
  • preserves the existing face centering, coverage, edge, confidence, and motion-blur checks
  • selects the front first, best, and last frames
  • rejects frames when motion blur is explicitly detected
  • scores best frames using face confidence, centering, coverage, and stability
  • waits on a short blurred instruction before starting each side pose
  • uses a 15 degree yaw target, represented internally in radians
  • maps MediaPipe yaw to the mirrored left and right camera directions
  • resets side progress when MediaPipe does not return a pose or the face is not usable
  • collects side candidates over a one-second best-frame window
  • selects the highest-scoring frame from that window for the side upload
  • holds the accepted front and side states long enough for the confirmation UI to be visible

• follows the server-provided pose order

  • reads pose_sequence from the selfie static content
  • always keeps front capture first
  • follows valid left and right entries in exactly the order returned by the API
  • removes duplicate side entries
  • supports sessions that request only one side
  • defaults to right then left when there is no usable side sequence

• adds the capture sound and haptic feedback

  • uses the same selfie_capture_confirmed.wav asset as iOS
  • plays it once when the front or either side is accepted
  • triggers Android's confirmation haptic on supported versions
  • uses the existing virtual-key haptic as the older Android fallback
  • ties both effects to the accepted capture state so they follow the actual pose sequence

• adds the MediaPipe face scanner

  • uses Google MediaPipe Tasks Vision with the bundled face_landmarker.task
  • runs one-face landmark detection with blendshapes and facial transformation matrices enabled
  • calculates the face bounding box from the returned landmarks
  • calculates yaw, pitch, and roll from the facial transformation matrix
  • keeps pose values in radians
  • encodes the landmarks, blendshapes, and transformation matrix for optional frame metadata
  • uses MediaPipe for the face bounding box and face score in place of the legacy detector output

• keeps the 3D and legacy model paths mutually exclusive

  • uses only MediaPipe when 3D capture is enabled
  • does not download, initialize, or run the remote legacy face detector in that mode
  • checks that MediaPipe can initialize as part of face-model loading
  • reports initialization failures with model_type=face and stage=media_pipe_detector
  • treats MediaPipe initialization failure as a model-loading failure instead of silently falling back
  • leaves the existing remote legacy detector behavior unchanged for non-3D sessions
  • legacy detector results continue to have no pose or landmark result

• makes 3D enablement server driven

  • decodes the top-level VerificationPage experiments array
  • recognizes the idprod_3d_face_capture_mobile experiment
  • enables capture and submission when the selfie pose_sequence contains a side pose or that experiment is present
  • uses the same combined signal for the scanner path, capture flow, uploads, and submitted data
  • leaves the local override set to null, so it is currently off

• adds the preloaded experiment exposure behavior

  • checks experiments when analytic events are actually emitted
  • requires the experiment event_name to match the analytic event
  • requires every experiment event_metadata value to match the analytic metadata
  • logs the exposure when the selfie screen_presented event matches
  • uses the experiment name provided by the server
  • does not log exposure just because the VerificationPage GET response was decoded

• updates the selfie uploads

  • continues uploading first, best, and last high-resolution images
  • continues uploading first, best, and last low-resolution images
  • uploads one selected full-frame image for each requested side pose
  • sends those tokens as left_high_res_image and right_high_res_image
  • only includes side image fields when 3D data should be submitted
  • does not upload every sampled front or side frame
  • keeps num_frames as the total number of sampled front frames

• adds the additional frame data expected by the API

  • sends best_frame_data, first_frame_data, and last_frame_data
  • sends left_frame_data and right_frame_data for captured 3D sides
  • includes face score and face-score variance
  • includes blur score fields where Android has values available
  • includes yaw, pitch, and roll
  • converts the normalized face box into pixel bbox values
  • includes the input image size
  • includes the capture timestamp and chronological capture order
  • includes the encoded camera lens information
  • continues sending the existing best-frame camera metadata and training consent

• keeps the landmark payload within the server limit

  • starts with the MediaPipe landmark, blendshape, and matrix result
  • compacts the submitted value down to the blendshape category names and rounded scores
  • base64 encodes the compact result
  • only includes it when the final encoded value is 5,000 characters or less
  • omits it instead of failing the whole data request when it is still too large

• includes the Android-specific plumbing needed for this

  • adds com.google.mediapipe:tasks-vision
  • bundles the MediaPipe task model in the Identity SDK
  • raises the Identity SDK and example app minimum SDK to API 24
  • uses CameraX's compatible texture-backed preview for the selfie camera
  • applies the live blur directly to that preview on supported Android versions
  • leaves the document camera preview behavior unchanged
  • adds optional brandColor configuration for the native flow's primary actions

• adds and updates coverage around the new behavior

  • VerificationPage experiment decoding and experiment-name gating
  • analytic exposure event and metadata matching
  • default and server-provided pose ordering
  • left and right yaw progress
  • non-3D front-only capture
  • motion-blur rejection and best-frame selection
  • side image and frame-data serialization
  • landmark compaction and oversized-value omission
  • selfie warmup consent states and final checking UI

[github-actions]

Diffuse output:

OLD: identity-example-release-base.apk (signature: V1, V2)
NEW: identity-example-release-pr.apk (signature: V2)

          │             compressed             │            uncompressed            
          ├───────────┬───────────┬────────────┼───────────┬───────────┬────────────
 APK      │ old       │ new       │ diff       │ old       │ new       │ diff       
──────────┼───────────┼───────────┼────────────┼───────────┼───────────┼────────────
      dex │   2.2 MiB │   2.7 MiB │ +475.1 KiB │   4.6 MiB │   6.2 MiB │   +1.5 MiB 
     arsc │   1.2 MiB │   1.2 MiB │   -7.9 KiB │   1.2 MiB │   1.2 MiB │   -7.9 KiB 
 manifest │   2.3 KiB │   2.5 KiB │     +257 B │     8 KiB │   9.2 KiB │   +1.2 KiB 
      res │   298 KiB │ 684.9 KiB │ +386.9 KiB │ 447.6 KiB │ 832.8 KiB │ +385.1 KiB 
   native │  18.6 MiB │  62.4 MiB │  +43.8 MiB │  18.6 MiB │  62.4 MiB │  +43.8 MiB 
    asset │   7.9 KiB │   3.2 MiB │   +3.2 MiB │   7.7 KiB │   3.6 MiB │   +3.6 MiB 
    other │ 110.1 KiB │  77.2 KiB │  -32.9 KiB │ 235.1 KiB │ 155.5 KiB │  -79.5 KiB 
──────────┼───────────┼───────────┼────────────┼───────────┼───────────┼────────────
    total │  22.4 MiB │  70.2 MiB │  +47.8 MiB │    25 MiB │  74.3 MiB │  +49.2 MiB 

 DEX     │ old   │ new   │ diff                   
─────────┼───────┼───────┼────────────────────────
   files │     1 │     1 │      0                 
 strings │ 22253 │ 29843 │  +7590 (+11338 -3748)  
   types │  6630 │  7957 │  +1327 (+4825 -3498)   
 classes │  5353 │  6557 │  +1204 (+4397 -3193)   
 methods │ 32418 │ 50740 │ +18322 (+43122 -24800) 
  fields │ 18773 │ 22431 │  +3658 (+21248 -17590) 

 ARSC    │ old  │ new  │ diff        
─────────┼──────┼──────┼─────────────
 configs │  158 │  158 │  0 (+1 -1)  
 entries │ 3982 │ 3990 │ +8 (+15 -7)

APK

       compressed       │      uncompressed      │                                                              
───────────┬────────────┼───────────┬────────────┤                                                              
 size      │ diff       │ size      │ diff       │ path                                                         
───────────┼────────────┼───────────┼────────────┼──────────────────────────────────────────────────────────────
  14.3 MiB │  +14.3 MiB │  14.3 MiB │  +14.3 MiB │ + lib/x86/libmediapipe_tasks_jni.so                          
  12.4 MiB │  +12.4 MiB │  12.4 MiB │  +12.4 MiB │ + lib/x86_64/libmediapipe_tasks_jni.so                       
    10 MiB │    +10 MiB │    10 MiB │    +10 MiB │ + lib/arm64-v8a/libmediapipe_tasks_jni.so                    
   7.1 MiB │   +7.1 MiB │   7.1 MiB │   +7.1 MiB │ + lib/armeabi-v7a/libmediapipe_tasks_jni.so                  
   3.2 MiB │   +3.2 MiB │   3.6 MiB │   +3.6 MiB │ + assets/face_landmarker.task                                
   2.7 MiB │ +475.1 KiB │   6.2 MiB │   +1.5 MiB │ ∆ classes.dex                                                
 295.6 KiB │ +295.6 KiB │ 295.5 KiB │ +295.5 KiB │ + res/7S.wav                                                 
  44.5 KiB │  +44.5 KiB │  44.4 KiB │  +44.4 KiB │ + res/GI.png                                                 
           │  -29.3 KiB │           │    -65 KiB │ - META-INF/CERT.SF                                           
           │  -25.8 KiB │           │  -64.9 KiB │ - META-INF/MANIFEST.MF                                       
  25.7 KiB │  +25.7 KiB │  25.6 KiB │  +25.6 KiB │ + res/Sg.png                                                 
  14.8 KiB │  +14.8 KiB │  14.7 KiB │  +14.7 KiB │ + res/UV.png                                                 
   7.9 KiB │   +7.9 KiB │   7.8 KiB │   +7.8 KiB │ + res/Y2.png                                                 
   1.2 MiB │   -7.9 KiB │   1.2 MiB │   -7.9 KiB │ ∆ resources.arsc                                             
   3.2 KiB │   +3.2 KiB │     8 KiB │     +8 KiB │ + google/protobuf/field_mask.proto                           
     3 KiB │     +3 KiB │   7.5 KiB │   +7.5 KiB │ + google/protobuf/api.proto                                  
   2.9 KiB │   +2.9 KiB │   6.3 KiB │   +6.3 KiB │ + google/protobuf/timestamp.proto                            
   2.7 KiB │   +2.7 KiB │     6 KiB │     +6 KiB │ + google/protobuf/any.proto                                  
   2.4 KiB │   +2.4 KiB │   6.2 KiB │   +6.2 KiB │ + google/protobuf/type.proto                                 
   2.1 KiB │   +2.1 KiB │   4.8 KiB │   +4.8 KiB │ + google/protobuf/duration.proto                             
   1.8 KiB │   +1.8 KiB │   3.7 KiB │   +3.7 KiB │ + google/protobuf/struct.proto                               
   1.6 KiB │   +1.6 KiB │   3.9 KiB │   +3.9 KiB │ + google/protobuf/wrappers.proto                             
   1.3 KiB │   +1.3 KiB │   2.3 KiB │   +2.3 KiB │ + google/protobuf/source_context.proto                       
   1.3 KiB │   +1.3 KiB │   2.3 KiB │   +2.3 KiB │ + google/protobuf/empty.proto                                
           │   -1.2 KiB │           │   -1.2 KiB │ - META-INF/CERT.RSA                                          
           │     -833 B │           │   -1.9 KiB │ - res/f-.xml                                                 
           │     -433 B │           │     -500 B │ - res/color-v23/abc_btn_colored_borderless_text_material.xml 
           │     -412 B │           │     -500 B │ - res/color-v23/abc_btn_colored_text_material.xml            
   6.7 KiB │     -263 B │   6.6 KiB │     -263 B │ ∆ assets/dexopt/baseline.prof                                
   2.5 KiB │     +257 B │   9.2 KiB │   +1.2 KiB │ ∆ AndroidManifest.xml                                        
     198 B │     +198 B │      84 B │      +84 B │ + firebase-encoders-proto.properties                         
     195 B │     +195 B │      82 B │      +82 B │ + firebase-encoders-json.properties                          
     192 B │     +192 B │      78 B │      +78 B │ + transport-backend-cct.properties                           
     180 B │     +180 B │      72 B │      +72 B │ + firebase-encoders.properties                               
     180 B │     +180 B │      70 B │      +70 B │ + transport-runtime.properties                               
     168 B │     +168 B │      62 B │      +62 B │ + transport-api.properties                
...✂

MANIFEST

@@ -11,3 +11,3 @@
   <uses-sdk
-      android:minSdkVersion="23"
+      android:minSdkVersion="24"
       android:targetSdkVersion="36"
@@ -143,2 +143,20 @@
     </receiver>
+    <service
+        android:exported="false"
+        android:name="com.google.android.datatransport.runtime.backends.TransportBackendDiscovery"
+        >
+      <meta-data
+          android:name="backend:com.google.android.datatransport.cct.CctBackendFactory"
+          android:value="cct"
+          />
+    </service>
+    <service
+        android:exported="false"
+        android:name="com.google.android.datatransport.runtime.scheduling.jobscheduling.JobInfoSchedulerService"
+        android:permission="android.permission.BIND_JOB_SERVICE"
+        />
+    <receiver
+        android:exported="false"
+        android:name="com.google.android.datatransport.runtime.scheduling.jobscheduling.AlarmManagerSchedulerBroadcastReceiver"
+        />
   </application>

DEX

STRINGS:

   old   │ new   │ diff                 
  ───────┼───────┼──────────────────────
   22253 │ 29843 │ +7590 (+11338 -3748) 
  
  +

[fedefrappi]

This may not be 100% consistent with how the buttons appear on iOS because the text is always white there while here we keep and rely on the main theme onPrimary I think.

This is actually a broader question for both platforms: is it ok that we are customizing just the main background colors with the risk that the texts may not be accessible/readable because of the lack of contrast? e.g white text on yellow bg

[fedefrappi]

iOS only encodes categories, I don't think the other two fields are read anywhere else here so we can probably save some encoding compute

[fedefrappi]

I believe this is needed for supporting mediapipe, will be a breaking change to mention?

[fedefrappi]

this is the counterpart of the iOS comment where the enabling logic is not matching

[fedefrappi]

MEDIA_PIPE_FACE_SCORE ends up being hardcoded to 1, just making sure this is expected