Dedicate cores for emulation only

[System64fumo]

Hello, I've got a very weird question to ask and i highly doubt it's possibility but is it possible to pin FEX emulation to specific cores while keeping games running on other cores?

I'm using FEX on my Radxa Orion O6 and sort of recently found out that if i start a game in steam with taskset -c 0-7 %command% (Cores 0-3 being big, 4-7 being medium, 8-11 being small)
Then run taskset -cp 4-7 PIDOFGAME, I get a noticable performance increase in practically all games.

I have 2 theories for this:

1. Games/FEX create 8 threads on startup and after i run the taskset command something notices core count was lowered and thus moves the threads to the other cores?
2. FEX is pinned to the first core it's allowed to run the game from and by moving the game to other cores FEX is effectively utilizing some cores for emulation only and the other cores for running the actual game code uninterrupted.

I really want to believe theory 2 is true because if it is then that unlocks a way to better utilize such hardware by pinning emulation to specific cores.

[System64fumo]

@Anti-Ultimate iirc you mentioned you were getting an orion from a previous issue i've opened here, Sorry for bothering but do you mind testing this? 😅 (You will need the custom CIX unlocked UEFI for this as it fixes the core layout)

I would love to test this on another big.LITTLE machine to ensure this isn't some weird quirk my board has but unfortunately the only other one i got is an rk3588 one and FEX doesn't work there last i checked..

[Sonicadvance1]

The fundamental issue here is that application worker threads end up on the A520 cores, which are dramatically worse than the A720. To the point that the cpu_capacity number for scheduling work should really be quite a bit lower on those cores. The same problems occur on Intel x86 platforms with three cluster core arrangements as well.

The kernel just schedules work on those cores that would be better handled if the affinity was only on the set on the A720 cores from the start.

Affinity setting is something that needs to be controlled outside of FEX, there's no reason for FEX to handle this.

[System64fumo]

A520?
Nothing here is running on the A520 cores?

For me with the unlocked cix uefi (And some overclocking)
Cores 0-3 are A720@2.7GHz
Cores 4-7 are A720@2.6GHz
Cores 8-11 are A520@1.8GHz

What's really odd from my testing is:
Running games with cores 0-3 runs Good
Running games with cores 4-7 runs Ok
Running games with cores 8-11 runs poorly (Expected)
Running games with cores 0-7 runs Really well
Running games with cores 0-7 then tasksetting the game to 4-7 runs Exceptionally well (Like actual night and day difference in pretty much all games)
Running games with cores 0-7 then tasksetting the game to 0-3 runs about the same as just starting out with 0-3.

If necessary i can provide actual benchmarks and actual numbers.

[System64fumo]

Alright here's some actual data, Please take this with a grain of salt.

Setup

Radxa Orion O6
64GB RAM
RX 7600 XT
Linux 6.18.13 (Mainline, Old because of a bug in recent kernels)
FEX-2604-97-ga04b024
Custom overclocked UEFI, Cores: 0-3 at 2.7GHz, 4-7 at 2.6GHz, 8-11 at 1.8GHz

Perf table

FPS	Starting cores	Ending cores
129	0-3	0-3
140	0-3	4-7
151	4-7	4-7
147	4-7	0-3
34	8-11	8-11
60	8-11	0-3
62	8-11	4-7
164	0-7	0-7
156	0-7	0-3
181	0-7	4-7

Notes

Odd how cores 0-3 perform worse than 4-7 despite having a higher clock speed and scoring higher in sysbench's CPU benchmark:
0-3 = 12092.05 Events per second
4-7 = 11612.68 Events per second

If anyone with a big.LITTLE system can run some benchmarks that might give a better idea of my theory is right or wrong.

@Sonicadvance1 Any ideas?

[Sonicadvance1]

Pretty sure core 0 gets hit pretty hard in IRQs on this platform right? I've seen USB networking saturate core 0. Could just be getting CPU time stolen?

[System64fumo]

Not necessarily, Or not in my case.
I have a script to properly switch all other IRQs to cores 8-11 so the main cores aren't interrupted.

With/out changing the IRQs the results are pretty consistent.

[Anti-Ultimate]

Cannot reproduce currently...

Will spend more time on this if/when I have time as this week im on vacation at a festival 😆

Been a while since I played around with it though. Iirc the cause for the PCIe Bug was found and the Intel Xe driver now boots on ARM by default too...

[Anti-Ultimate]

It would be helpful to know which games were tested

[System64fumo]

War thunder
Space engineers (Massive improvement here)
And a few other games but saw marginal performance improvement since they're single threaded heavy.

[Anti-Ultimate]

I am not using the Neol00 UEFI, however i had a manual look at the core topology and figured my way from there.
Doesn't seem like it is reproduceable for me (dont have War Thunder but I have Space Engineers).

Alas I am not a great testing candidate because im using mainline with no patches to the kernel from CIX (currently waiting for them to rebase patches onto 7.1...)

[System64fumo]

Well i am using Neol00's custom UEFI but this should in theory work for unpatched firmwares.
That being said i am also using pure unpatched mainline linux rather than CIX's custom linux so that also shouldn't make a difference.

During my testing i saw a noticable performance stability/increase in war thunder and MAJOR sim time improvement in space engineers.
Can't stress enough how big of a deal the space engineers improvement is, Goes from playing the game at 0.3-0.5x sim speed to a stable 1.0x sim speed (Real time)