Develop SDL_ActionSet

[flibitijibibo]

Probably the most popular new feature in SDL 2.0 is the GameController subsystem. It was a major step forward over what used to be traditional joystick management, where you would open mysterious handles and blindly try to map random buttons, axes, and hats to something that roughly resembled what you thought might be the intended input layout (and then an axis would turn out to be always min/max or something and everything would break horribly). GameController brought all the miserable work into a simple format with a common, standard layout that effectively solved the hardware side of controller input once and for all (even with all the weird crap console vendors keep adding to their hardware).

In the decade(!) since, however, the input problem has evolved from a hardware issue to a software issue - yes, SDL now manages a billion different devices, but games do a pretty miserable job of using them. For the PC in particular, you have all these ways of getting input:

• Keyboard
• Mouse
• Controller
• Touch
• Sensor
• Combinations of all the above!

You also have various outputs as well:

• Traditional motors
• Linear actuators (Steam Controller, Taptic, HD Rumble)
• Trigger rumble
• Light bars
• Whatever it is that's in DualSense controllers

In addition to reading/writing to devices, you also have to display them as well, via device-specific glyphs! We don't even get near this very ugly problem, with reports like #4203 showing up on a regular basis.

Engines traditionally have to map and display all these types of devices into what's now commonly known as Action Sets, a term firmly established by the debut of Steam Input, a library designed to solve this problem:

https://partner.steamgames.com/doc/features/steam_controller

This was (and still is) only for controllers, however. A recent competitor to this is Apple's new virtual controller API, as demonstrated in this video (hi Nat!):

https://developer.apple.com/videos/play/wwdc2021/10081/

This implementation is far more complete, but of course it's Apple-only. Steam is no better, being tied to the Steam client as proprietary software (and I'm officially infamous for my opinion of its quality).

This is an API that is generally agreed upon as very useful, but today's solutions just aren't appropriate for multiplatform software. SDL is in a unique position to fulfill this need for a LOT of developers, and I think we have the infrastructure to do it very cleanly and have it appeal to more than just platform exclusives.

We should implement an ActionSet API that supports platform-specific libraries (Steam's and Apple's for example) while also providing a fully multiplatform backend that makes use of SDL's existing events system. Similar to how GameController events hook to Joystick events, it would be very easy for us to read in our already working input events and map them to an action set provided by app configurations (be it a file or programmatic configuration), and we already have pretty good examples of how action set APIs should look.

The difficult part is the configuration side... in addition to the platform-specific formats, we would have to figure out our own format, which of course risks further fragmentation in an already fragmented ecosystem. I'm garbage at config formats, so this part I'm intentionally leaving very open-ended.

Glyphs would probably suck too, but I'd be okay with there being a pass-through API for RGBA bitmaps if the backend supports it, with arbitrary strings as a fallback (giving applications enough context to display something accurate but without having to commit to making art that looks any good). I'm slightly more confident about this one given my experience supporting dynamic glyphs in FNA games, but again this is pretty open-ended at this point.

In any case this seems like a logical next step for SDL's input subsystem, and I have zero doubt that this would be a widely used feature given how popular the GameController system alone already is.

[icculus]

Obviously this isn't going to land in 2.0.16 or anything, but this isn't a bad idea.

[leo60228]

I might be missing something, but to me it seems like Apple's controller API is more like SDL_GameController except with multiple supported standard layouts than Steam Input.

[flibitijibibo]

I pushed the first draft of what will eventually be the spec here.

This is a big ol mess that's wildly unfinished, but the main point is that I'm balancing "writing a spec" with "actually writing the use case first," since I (and many others reading this) have written action set implementations before, but since they invariably have holes I'm trying to document this as much as possible from the start so that, when I write something stupid, it'll show up as stupid on the page and force me to actually address the problem early on (see: the entire Configuration section right now).

[flibitijibibo]

Cleared up all the TODOs, so this is pretty much the first draft done. There's a lot to rewrite and a few things to fix (mainly support for multiple simultaneous action bindings), but it's now possible to see a front-to-back explanation of what SDL_ActionSet is aiming to be.

[flibitijibibo]

Added a quick note to describe a possible string format for the input label, since that will be important for app-provided input bitmaps.

One thing I hadn't considered yet is input ranges - for example, a virtual touch controller button might be a 2-component absolute float, but for it to be useful it would need something like either a radius or... whatever a radius could be called for indicating a hitbox (sorry, I'm a failed musician doing math, please be gentle!). Definitely check out Chapter 2 Section B if you have any ideas!

[JimmyLefevre]

The overall design of the API looks good to me.

I am not sure I completely understand how to map game actions to your ActionTypes.

If I want a button-like action, I can use Boolean. Fair enough. However, if I'm making a game where the user is expected to mash a button, or if I want mouse wheels/turbo controllers (do those even exist nowadays?) to ever work as buttons, I'll also want to get the number of key presses that happened this frame, just to be sure I don't miss any.

If I want mouse-like movement, say to control a 3D camera, is the idea that I should use one of the Relative types, because we only care about deltas, or should I use one of the Absolute types to allow for very large, sudden movements, and do the delta myself?

What is the difference between an Integer and an UnsignedInteger action? Is Integer supposed to mean that the resting point of the input is in the middle, ie. a joystick-like source, whereas for UnsignedInteger it is at the minimum value, ie. an analog trigger-like source? Or is it supposed to mean something else?

Having SDL normalize inputs automatically with the Float types sounds nice, but, as you say, given a virtual touch controller, or even just a mouse pointer, I will definitely want to know the aspect ratio of the input surface at the very least. At that point, it makes more sense to just send integer bounds for the input source, along with integer coordinates, and either send normalized float coordinates along as a courtesy, or let me do the normalization myself if I really want to, because, to reuse rendering terms, I will know whether I want my input data to be letterboxed or stretched to the input source's bounds.
So we'd get something like:

typedef struct SDL_ActionSetEvent
{
	SDL_EventType type; /* SDL_ACTIONSET */
	Uint64 set;
	Uint64 action;
	Uint64 playerIndex;

	float normalized[SDL_ACTIONSET_MAX_VECTOR_SIZE]; // Just there if we want to be nice... Probably remove this?

	union
	{
		SDL_bool b[SDL_ACTIONSET_MAX_VECTOR_SIZE];
		struct
		{
			Int32 i[SDL_ACTIONSET_MAX_VECTOR_SIZE];
			Int32 imin[SDL_ACTIONSET_MAX_VECTOR_SIZE];
			Int32 imax[SDL_ACTIONSET_MAX_VECTOR_SIZE];
		};
		struct
		{
			Uint32 ui[SDL_ACTIONSET_MAX_VECTOR_SIZE];
			Uint32 umin[SDL_ACTIONSET_MAX_VECTOR_SIZE];
			Uint32 umax[SDL_ACTIONSET_MAX_VECTOR_SIZE];
		};
		Uint8 padding[64]; /* Please nobody ask for a float[16]... */
	} value;
} SDL_ActionSetEvent;

[flibitijibibo]

Before I start: Sorry for the delayed reply, been helping the GPU team this week 😅

If I want a button-like action, I can use Boolean. Fair enough. However, if I'm making a game where the user is expected to mash a button, or if I want mouse wheels/turbo controllers (do those even exist nowadays?) to ever work as buttons, I'll also want to get the number of key presses that happened this frame, just to be sure I don't miss any.

For the moment the plan is to have it done as events, so when two presses happen in a single frame you should receive two press events from a single PollEvent loop.

If I want mouse-like movement, say to control a 3D camera, is the idea that I should use one of the Relative types, because we only care about deltas, or should I use one of the Absolute types to allow for very large, sudden movements, and do the delta myself?

Ideally this would be relative motion, since you're less concerned about where the cursor is exactly and more about which direction the cursor went and the magnitude of the direction.

What is the difference between an Integer and an UnsignedInteger action? Is Integer supposed to mean that the resting point of the input is in the middle, ie. a joystick-like source, whereas for UnsignedInteger it is at the minimum value, ie. an analog trigger-like source? Or is it supposed to mean something else?

That's what I had in mind, yeah - the basic example is as you suggested: a control stick axis would be signed while a trigger axis would be unsigned.

Having SDL normalize inputs automatically with the Float types sounds nice, but, as you say, given a virtual touch controller, or even just a mouse pointer, I will definitely want to know the aspect ratio of the input surface at the very least. At that point, it makes more sense to just send integer bounds for the input source, along with integer coordinates, and either send normalized float coordinates along as a courtesy, or let me do the normalization myself if I really want to, because, to reuse rendering terms, I will know whether I want my input data to be letterboxed or stretched to the input source's bounds. So we'd get something like:

typedef struct SDL_ActionSetEvent
{
	SDL_EventType type; /* SDL_ACTIONSET */
	Uint64 set;
	Uint64 action;
	Uint64 playerIndex;

	float normalized[SDL_ACTIONSET_MAX_VECTOR_SIZE]; // Just there if we want to be nice... Probably remove this?

	union
	{
		SDL_bool b[SDL_ACTIONSET_MAX_VECTOR_SIZE];
		struct
		{
			Int32 i[SDL_ACTIONSET_MAX_VECTOR_SIZE];
			Int32 imin[SDL_ACTIONSET_MAX_VECTOR_SIZE];
			Int32 imax[SDL_ACTIONSET_MAX_VECTOR_SIZE];
		};
		struct
		{
			Uint32 ui[SDL_ACTIONSET_MAX_VECTOR_SIZE];
			Uint32 umin[SDL_ACTIONSET_MAX_VECTOR_SIZE];
			Uint32 umax[SDL_ACTIONSET_MAX_VECTOR_SIZE];
		};
		Uint8 padding[64]; /* Please nobody ask for a float[16]... */
	} value;
} SDL_ActionSetEvent;

Yeah, that makes sense to me - having (for example), numerator/denominator values for sensitive numbers like refresh rate is useful to have, so it would make sense for input to be the same way. I'll try and integrate this into the next draft!