Inconsistent mappings in-game vs menus

[glebm]

As of #5464 the buttons in the main menu and in-game menus are now consistent.

However, they are both inconsistent with the buttons in-game, e.g. inventory management.
The "cancel" button is used as the primary action, the "confirm" button is used to bring up the spell menu, close inventory, etc. Should be the opposite for consistency

[evbo]

D2R uses the A button for all interaction with game/menus. Nice consistency and a lot of people who play D1 probably have heard of D2 before. Just sayin...

[glebm]

This is now fixed, I'm guessing the remapping PR fixed this somehow

[StephenCWills]

@EvilToasterDBU mentioned this is still an issue in #5506 for PlayStation/Xbox controller layouts. I figured I'd reopen this issue rather than wait for another write-up.

[StephenCWills]

Honestly, this issue is deceptively complicated. The madness stems from the options SDL gives us for identifying controller buttons via the SDL_HINT_GAMECONTROLLER_USE_BUTTON_LABELS hint. We have little choice but to set this option to false because of the on-screen hints we display when using the "Gamepad hotspell menu" and "Gamepad menu navigator". There would be no way to determine which position each button is in relative to each other if we let SDL abstract this for us. However, this is also convenient to maintain consistency between SDL1 and SDL2.

When not using button labels, SDL assumes Xbox layout. In other words, SDL_CONTROLLER_BUTTON_A is always the bottom button, not the A button. In order to make sure that nothing changes for the handhelds that use Nintendo layout, we introduced the following function to swap A/B and X/Y based on the GamepadLayout enum, and we use this in the menus to keep the confirm/cancel actions consistent with user expectations.

https://github.com/diasurgical/devilutionX/blob/c71eb0ff54d358f77851fe95d495ec736aa8bf1d/Source/controls/game_controls.cpp#L304

Note that the section which was recently added to the wiki talks about what's most intuitive for PlayStation users, but there actually is no consistency on PlayStation because Sony adopted Nintendo layout in Japan and Xbox layout in the US. It seems they recently made the decision to adopt the Xbox layout in Japan on the PS5 menus, but games apparently still use Nintendo layout. Also, some PS1 games released in the US use the Nintendo layout.

However, because the padmapper is entirely configurable by the user, it uses the positional button mapping defined by SDL. So even if the game detects that you are using a Nintendo layout, the bottom button will register as the A button when mapping your gamepad's buttons to actions in the settings menu. This has the advantage of being positionally consistent even if the user changes the type of gamepad they're using. But it has the disadvantage of making the mapping, and by extension the defaults, completely independent of the layout of the gamepad. Fundamentally, the inconsistency occurs because the menus are layout-aware and the padmapper is not.

Astute observers would be quick to notice that I mentioned an advantage as well as a disadvantage to the current approach. Unfortunately, this means that changing the behavior would fix one problem and introduce another. Since the padmapper is configurable, I'm inclined to leave the behavior as-is, but I can't deny that the inconsistency in the default mapping is kind of frustrating to deal with.

At this point, I'm open to suggestions.

[EvilToasterDBU]

I think that my proposal will be much more difficult to implement, but I think that there should be a PS1-like control layout, or rather a cross between the one that already exists and the one that is offered in the PS1 version of the game, because such a layout has a logical sequence (after all, it is logical that the main control buttons are closer to the finger and it is illogical to perform a more important action by reaching for the next button) + those who have played diablo on PS1 will not be confused in the management.
*And yes, by "there should be a control layout" I mean as an optional control layout.

Regarding the problem of determining the physical layout of the buttons, as a last resort, you can probably come up with some kind of crutch. For example, if it is determined that the gamepad does not correspond to the "*known reference", then the user will be asked to manually identify the ABXY buttons in some setting.
*By "known reference" I mean the most popular basic gamepads, such as xbox one controller, dualshock 4, dualsense and nintendo pro controller.

(Sorry for the poor-quality language, I'm not a native English speaker)

[StephenCWills]

I think that my proposal will be much more difficult to implement, but I think that there should be a PS1-like control layout, or rather a cross between the one that already exists and the one that is offered in the PS1 version of the game, because such a layout has a logical sequence (after all, it is logical that the main control buttons are closer to the finger and it is illogical to perform a more important action by reaching for the next button) + those who have played diablo on PS1 will not be confused in the management. *And yes, by "there should be a control layout" I mean as an optional control layout.

Have you tried using the latest test builds? Padmapper lets you change the control layout in the Settings menu. I'm unfamiliar with the controls from the PS1 version of Diablo so if there's some reason you aren't able to set it up the way you like, it would be helpful if you could elaborate on the reasons why.

Regarding the problem of determining the physical layout of the buttons, as a last resort, you can probably come up with some kind of crutch. For example, if it is determined that the gamepad does not correspond to the "*known reference", then the user will be asked to manually identify the ABXY buttons in some setting. *By "known reference" I mean the most popular basic gamepads, such as xbox one controller, dualshock 4, dualsense and nintendo pro controller.

(Sorry for the poor-quality language, I'm not a native English speaker)

Currently, we are automatically detecting the layout using SDL_GameControllerTypeForIndex(), or just hardcoding it for some SDL1 platforms.

https://github.com/diasurgical/devilutionX/blob/c71eb0ff54d358f77851fe95d495ec736aa8bf1d/Source/controls/devices/game_controller.cpp#L237-L279

However, I had considered making GamepadLayout configurable for the user, and I may still do that. I mentioned in my last comment that PlayStation has no consistency in this space. Currently we're assuming Xbox layout for PlayStation controllers, but it's conceivable that a user of a PlayStation controller might still prefer to use the Nintendo layout. It's also possible that we'll get something like SDL_CONTROLLER_TYPE_VIRTUAL or SDL_CONTROLLER_TYPE_UNKNOWN in which case there's no way to know which layout should be preferred.

[AJenbo]

Yeah probably wouldn't hurt to have it auto/nintendo/xbox

[rtrzebinski]

Would that be possible to select between most popular layouts (ps/nintendo/xbox) + have the option to customize these, or even provide a custom mapping from scratch?

[StephenCWills]

TBH, that sounds like an overly complicated setup for a single game. It might be nice to have profiles for couch co-op, but we don't have couch co-op.

More realistically, for 1.5.0 we will have remappable controls that are saved in the INI file. Mappings can be modified through the Settings menu, and different control schemes can be saved/restored by copy/pasting the [Padmapping] section of the INI file. Specifying the layout is only really useful for swapping the A/B and X/Y buttons. Other than that, most gamepads are pretty similar.