arch: arm64: smp: secondary cpu cannot boot if cluster id !=0

[vkorenblit]

Platform: ARM Fast Models 2xCortex-A55 + GIC400
Zephyr release 3.3.0
App: samples/hello_world

Config:
SMP=y
SMP_BOOT_DELAY=n
CPU_MPIDR_LSB=8
CPU_MPIDR_WIDTH=8
CONFIG_MP_MAX_NUM_CPUS=2

Hello,

I have a custom platform with 2 clusters of 2xCortex-A55 each, the first with cluster id = 0 and the 2nd one with cluster id = 1.

This gives the following MPIDR_EL1 for each core:
0x0000000081000000 Cluster 0 core 0
0x0000000081000100 Cluster 0 core 1

0x0000000081010000 Cluster 1 core 0
0x0000000081010100 Cluster 1 core 1

If I run Zephyr on cluster 0 with the following cpu nodes in dts everything works as expected:
cpu@0 {
device_type = "cpu";
compatible = "arm,cortex-a55";
reg = <0x0>;
};
cpu@1 {
device_type = "cpu";
compatible = "arm,cortex-a55";
reg = <0x1>;
};

Console logs:
Secondary CPU core 1 (MPID:0x1) is up
Hello World! Custom board

On the other hand, if cluster id != 0 I see the following problem: secondary cpu will wait until it sees its cpu_id (aff1) in BOOT_PARAM_MPID_OFFSET. However, primary cpu does not store cpu_id but cpu_mpid which comes from dts (arch_start_cpu). My point is that we are not comparing the same thing.

The only ugly workaround I have for now is declaring the 2 cpu nodes as follows:

	cpu@0 {
		device_type = "cpu";
		compatible = "arm,cortex-a55";
		reg = <0x10000>;  //this will make https://github.com/zephyrproject-rtos/zephyr/blob/71dfaff583b00c721ec2bbb451d7bf2a099a0a4f/arch/arm64/core/smp.c#L71 true
	};

	cpu@1 {
		device_type = "cpu";
		compatible = "arm,cortex-a55";
		reg = <0x1>;
	};

I think the main issue is that on one side we wait for our cpu number (only aff 1 in my opinion) to be stored but the primary cpu stores the value in reg property which "should correspond" to MPIDR_TO_CORE which has a mask for aff3 aff2 aff1 and aff0.

Am I missing something somewhere?

Best regards,

Valentin

[vkorenblit]

In dts bindings I didn't find a specification about what we should write in cpu reg property for arm64

[jgl-meta]

@carlocaione

[vkorenblit]

Actually my workaround is enough to boot but breaks the scheduling because broadcast_ipi from core 1 won't work as we'll send an SGI to target_mpidr 0x1 because it will think in the device tree that this core is not itself. This results in reading 1024 in GICC_IAR, which is actually not true.

[carlocaione]

Uhm, I totally forgot about this. Can you open an issue instead of an Q&A?

[vkorenblit]

I created Issue 57140 but it was closed. Anyway, I think will try the easiest solution which should be setting the real MPIDR_EL1 only with affinity bits in dts and secondary cpus should wait for this value (but read from their own register) instead of cpu_id in reset.S

[carlocaione]

that was a mistake. Please, open an issue and I'll check that or I'll assign it to the right dev.

[vkorenblit]

Done: #57585

[joakim-tjernlund]

Question: Can one run this HW configuration as one SMP OS with 4 cores?