Adding support for SMP on AArch32

[ayankuma]

Hi,

I am trying to add support for SMP for AArch32 (specifically 'fvp_baser_aemv8r_aarch32' platform).

I see that AArch32 lacks support for context switching.
I am trying to follow the context switching code for Arm64, and z_arm64_context_switch() caught my attention.

Do we know any Arm specification that states how context switching is to be done ? For eg I see that we are saving and restoring registers x19 ... x29, but not the previous registers. Do we know the reason why ?

Is there any list of registers (specified in any Arm docs) which are to be saved and restored.

@carlocaione :- I see that you are author, can you provide some pointers, please ?

[carlocaione]

I see that AArch32 lacks support for context switching.

That's obviously not true, without context switching you would not have preemption nor multitasking.

Do we know any Arm specification that states how context switching is to be done ?

There is no spec for that. Every OS or architecture does that differently.

I see that we are saving and restoring registers x19 ... x29, but not the previous registers. Do we know the reason why ?

The caller-saved registers are saved on exception entry: in

zephyr/arch/arm64/core/vector_table.S

Line 27 in 44488f1

.macro z_arm64_enter_exc xreg0, xreg1

and restored on exception exit in:

zephyr/arch/arm64/core/vector_table.S

Line 213 in 44488f1

GTEXT(z_arm64_exit_exc)

The callee-saved registers are lazily saved / restored only when a context switch does happen in:

zephyr/arch/arm64/core/switch.S

Line 33 in 44488f1

SECTION_FUNC(TEXT, z_arm64_context_switch)

Is there any list of registers (specified in any Arm docs) which are to be saved and restored.

Again, it depends (i.e. MMU vs no-MMU, FPU vs no-FPU, ASID vs no-ASID, etc...) you can have a possible comprehensive list at https://developer.arm.com/documentation/den0024/a/The-Memory-Management-Unit/Context-switching

[ayankuma]

That's obviously not true, without context switching you would not have preemption nor multitasking.

Yes, you are correct. What I meant was that arch_switch() is not implemented for AArch32. Instead arch_swap() is used.

While trying to understand the difference, I came across this line

zephyr/kernel/include/kernel_arch_interface.h

Line 97 in 44488f1

* is saved, and must include whatever memory barriers or cache

This store must be done AFTER all relevant state
 * is saved, and must include whatever memory barriers or cache
 * management code is required to be sure another CPU will see the
 * result correctly.

However, looking at z_arm64_context_switch(), I don't see any barriers being used. IIUC, we would need a dmb at the following

zephyr/arch/arm64/core/switch.S

Line 53 in 44488f1

The reason being we would want to ensure that the stores are visible to all the other cores (ie they aren't cached or buffered before a new context is retrieved).

I am trying to implement this for arm32 and thus want to make sure I understand it correctly.

[carlocaione]

I am trying to implement this for arm32 and thus want to make sure I understand it correctly.

Yes, this is missing. You can add it to the switch.S file but that's half of the story, since you also would need that in:

zephyr/kernel/include/kswap.h

Lines 33 to 51 in c9225e4

	/* There is an unavoidable SMP race when threads swap -- their thread
	* record is in the queue (and visible to other CPUs) before
	* arch_switch() finishes saving state. We must spin for the switch
	* handle before entering a new thread. See docs on arch_switch().
	*
	* Note: future SMP architectures may need a fence/barrier or cache
	* invalidation here. Current ones don't, and sadly Zephyr doesn't
	* have a framework for that yet.
	*/
	static inline void wait_for_switch(struct k_thread *thread)
	{
	#ifdef CONFIG_SMP
	volatile void **shp = (void *)&thread->switch_handle;
	while (*shp == NULL) {
	k_busy_wait(1);
	}
	#endif
	}

Anyway, this is proved not to a be an issue (so far) on emulation and real hardware.