How to tap on the iOS simulator numeric keypad?

[jzhang015]

Description

Hi,
My react-native mobile app includes some iOS native numeric keyboard in the payment process, I would like to know how can I tap on the numeric keypad?

I tried to locate the key by by.text() or by.label(), for example, locate key 1 by

await element(by.text('1')).tap(); // nothing happened

or

await element(by.label('1')).tap(); // Interestingly, when using by.label('4').tap(); it actually tapped 7, and when using by.label('1').tap(), nothing tapped.

I can't find the reference of the input field either.

Could you kindly help to advise how can I handle this?

Your environment

Detox version: 20.33.0
React Native version: 0.76.8
Node version: 20
Test-runner (select one): jest

[MadSandwich]

@jzhang015 You can simply fill 'Card number' field with data you need. But if you need this for any specific reason I would definitely suggest you to do 2 actions:

1. Print all view with debug level to see difference in elements
2. Use combined locator await element(by.label('1').and(by.type('AXButton'))).tap() - It's not an accurate locator, but just the suggestion. (I was able to do such inside my application) - Documentation reference: https://wix.github.io/Detox/docs/api/matchers#bytypeclassname

[jzhang015]

Thanks @MadSandwich for your reply!

Print all view with debug level to see difference in elements

This is an awesome suggestion, I was able to work out a solution based on the view hierarchy printed out:

<_UIKBCompatInputView: 0x163039770; frame = (0 0; 393 233); layer = <CALayer: 0x6000008464a0>> | | | | | <UIKeyboardAutomatic: 0x16216a010; frame = (0 0; 393 233); layer = <CALayer: 0x600000846920>> | | | | | | <UIKeyboardImpl: 0x106aa3400; frame = (0 0; 393 233); layer = <CALayer: 0x600000850f80>> | | | | | | | <UIKeyboardLayoutStar: 0x110904a00; frame = (0 0; 393 233); layer = <CALayer: 0x600000822580>> | | | | | | | | <UIView: 0x1054fdd10; frame = (0 0; 393 233); layer = <CALayer: 0x600000858b20>> | | | | | | | | <UIKBKeyplaneView: 0x3752bcd00; frame = (0 0; 393 233); layer = <CALayer: 0x600000ae33e0>> | | | | | | | | | <UIKBKeyView: 0x3752c2890; frame = (263 162; 130 54); layer = <CALayer: 0x600000ae37e0>> | | | | | | | | | <UIKBKeyView: 0x3752c2d90; frame = (129 162; 135 54); layer = <CALayer: 0x600000ae1700>> | | | | | | | | | <UIKBKeyView: 0x3752db410; frame = (-2 162; 132.048 54); layer = <CALayer: 0x600000ae2200>> | | | | | | | | | <UIKBKeyView: 0x3752c0e00; frame = (263 108; 130 54); layer = <CALayer: 0x600000ae0cc0>> | | | | | | | | | <UIKBKeyView: 0x3752c1c10; frame = (129 108; 135 54); layer = <CALayer: 0x600000ae0840>> | | | | | | | | | <UIKBKeyView: 0x3752c1300; frame = (0 108; 130 54); layer = <CALayer: 0x600000ae2500>> | | | | | | | | | <UIKBKeyView: 0x3752d9b10; frame = (263 54; 130 54); layer = <CALayer: 0x600000ae08e0>> | | | | | | | | | <UIKBKeyView: 0x3752da010; frame = (129 54; 135 54); layer = <CALayer: 0x600000ae2860>> | | | | | | | | | <UIKBKeyView: 0x3752c2110; frame = (0 54; 130 54); layer = <CALayer: 0x600000ae1da0>> | | | | | | | | | <UIKBKeyView: 0x3752161b0; frame = (263 0; 130 54); layer = <CALayer: 0x600000ae1520>> | | | | | | | | | <UIKBKeyView: 0x3752e1df0; frame = (129 0; 135 54); layer = <CALayer: 0x600000ae0e40>> | | | | | | | | | <UIKBKeyView: 0x3752c5090; frame = (0 0; 130 54); layer = <CALayer: 0x600000ae2ac0>> | | | | | | | | | <UIKBSplitImageView: 0x3752d8c10; frame = (0 0; 393 233); layer = <CALayer: 0x600000ae3500>> | | | | | | | | | | <UIImageView: 0x3752a4500; frame = (0 0; 393 233); layer = <CALayer: 0x600000ae0e00>> | | | | | | | | | <UIKBSplitImageView: 0x3752d8de0; frame = (0 0; 393 233); layer = <CALayer: 0x600000ae07a0>> | | | | | | | | | | <UIImageView: 0x3752d5380; frame = (0 0; 393 233); layer = <CALayer: 0x600000ae35c0>> | | | | | | | | | <UIKBSplitImageView: 0x3752d5d80; frame = (0 0; 393 233); layer = <CALayer: 0x600000ae1e40>> | | | | | | | | | | <UIImageView: 0x3752a1ee0; frame = (0 0; 393 233); layer = <CALayer: 0x600000ae0500>> | | | | | | | | <UIAccessibilityElementKBKey: 0x6000028225e0; ax.frame = (262.856 162; 130 54); ax.label = "Delete"> | | | | | | | | <UIAccessibilityElementKBKey: 0x600002822820; ax.frame = (128.904 162; 135 54); ax.label = "0"> | | | | | | | | <UIAccessibilityElementKBKey: 0x600002821020; ax.frame = (-2 162; 132.048 54)> | | | | | | | | <UIAccessibilityElementKBKey: 0x600002821080; ax.frame = (262.904 108; 130 54); ax.label = "9"> | | | | | | | | <UIAccessibilityElementKBKey: 0x600002821140; ax.frame = (128.952 108; 135 54); ax.label = "8"> | | | | | | | | <UIAccessibilityElementKBKey: 0x600002821e00; ax.frame = (0 108; 130 54); ax.label = "7"> | | | | | | | | <UIAccessibilityElementKBKey: 0x600002820ea0; ax.frame = (262.904 54; 130 54); ax.label = "6"> | | | | | | | | <UIAccessibilityElementKBKey: 0x600002820480; ax.frame = (128.952 54; 135 54); ax.label = "5"> | | | | | | | | <UIAccessibilityElementKBKey: 0x6000028215c0; ax.frame = (0 54; 130 54); ax.label = "4"> | | | | | | | | <UIAccessibilityElementKBKey: 0x6000028227c0; ax.frame = (262.904 0; 130 54); ax.label = "3"> | | | | | | | | <UIAccessibilityElementKBKey: 0x600002822700; ax.frame = (128.952 0; 135 54); ax.label = "2"> | | | | | | | | <UIAccessibilityElementKBKey: 0x600002823ae0; ax.frame = (0 0; 130 54); ax.label = "1">

I first tried to use await element(by.label('1').and(by.type('UIAccessibilityElementKBKey'))).tap() but weirdly and interestingly I got inconsistent tapping behavior (some keys work, some tap the wrong key, some fail with "Test Failed: View is not hittable at its visible point. "), which I think indicates there's a coordinate mapping issue, so basically I created map to map the coordinates based on the frames from the view hierarchy printed, something like this:

await element(by.type("_UIKBCompatInputView")).tap({
    x: digitFrames[digit].x,
    y: digitFrames[digit].y,
  });

Sharing it here, hope this can help others who encounter similar issues, or if someone could advise a better/simpler solution.
Thanks again @MadSandwich for your suggestions!

[MadSandwich]

@jzhang015 could you please share a file with full view hierarchy printed out?

[MDG-JHowley]

FYI for ease of debugging this sort of thing you could also create a hook to generate the view hierarchy automatically e.g.

afterEach(async () => {
  const xml = await device.generateViewHierarchyXml();
  fs.writeFileSync('artifacts/viewHierarchy.xml', xml);
});

It would be nice if Detox supported this as an option in the artifacts configuration.