Move thread_info into task_struct for LoongArch#8042
Open
kernel-patches-daemon-bpf-rc[bot] wants to merge 3 commits into
Open
Move thread_info into task_struct for LoongArch#8042kernel-patches-daemon-bpf-rc[bot] wants to merge 3 commits into
kernel-patches-daemon-bpf-rc[bot] wants to merge 3 commits into
Conversation
Like other architectures such as x86, arm64, riscv, powerpc and s390, select THREAD_INFO_IN_TASK for LoongArch to move thread_info off the stack into task_struct. This follows modern kernel standards and also makes the system more secure. With this patch, thread_info is included in task_struct at an offset of 0 instead of being placed at the bottom of the kernel stack. Thus, the $tp register points to both thread_info and task_struct. To support this, introduce a per-CPU variable cpu_tasks to store the pointer to the current task_struct. This decouples the recovery of the $tp register from the stack pointer during exception entry. Then initialize cpu_tasks for the primary and secondary CPUs during arch-specific setup and SMP boot paths. To eliminate the dangerous windows during the early initialization where the cpu_tasks remains uninitialized, set_current() is invoked as early as possible in both setup_arch() (right after unwind_init) and start_secondary() (right after cpu_probe). This ensures the $tp recovery barrier is armed in case any early boot exceptions or kernel panics occur. Modify SAVE_SOME and handle_syscall to restore the $tp register from cpu_tasks, and also use the la_abs absolute addressing for cpu_tasks access in assembly to bypass the relocation limits within exception handling sections. By advancing the preservation of u0 in SAVE_SOME, reuse the PERCPU_BASE_KS value in u0 for the cpu_tasks calculation, effectively eliminating a duplicate csrrd instruction execution on SMP platforms. Update <asm/switch_to.h> and <kernel/switch.S> to fully support the CONFIG_THREAD_INFO_IN_TASK feature. Remove the obsolete next_ti argument from __switch_to(), which shifts the remaining arguments ahead in the calling convention (sched_ra from a3 to a2, and sched_cfa from a4 to a3). Under the new configuration, __switch_to() now directly derives the thread pointer ($tp) from the next task_struct pointer in a1. To preserve the optimal and clean "move tp, a1" path for 64-bit kernels, the thread pointer ($tp) is assigned directly from a1 in the core path. For 32-bit kernels, where a1 carries a 2000-byte structural pointer bias at entry, an explicit adjustment "PTR_ADDI tp, tp, -TASK_STRUCT_OFFSET" is introduced at the function exit. Since __switch_to() executes under the protection of Direct Window Mapping with local interrupts disabled, the core path is immune to any exceptions, making this delayed un-biasing sequence strictly safe before returning to the scheduler. Additionally, evaluate the stack lookup as a single load instruction "LONG_LPTR t0, a1, (TASK_STACK - TASK_STRUCT_OFFSET)", this perfectly satisfies both 32-bit and 64-bit kernels. Using the "next" pointer in a1 as the base register, rather than $tp, effectively unchains the data dependency (RAW hazard) from the preceding move instruction, maximizing the instruction-level parallelism and superscalar execution efficiency while naturally adapting the structural shift. At last, safeguard the task stack access in show_backtrace() by wrapping the unwinding loop with try_get_task_stack() and put_task_stack(). This prevents the potential use-after-free or severe memory anomalies if the target task exits and its stack is freed concurrently during a backtrace. Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
The pointer to task_struct is always available in the $tp register,
the calls to bpf_get_current_task() and bpf_get_current_task_btf()
can be inlined into a single move instruction.
(1) Here is the sample test.c:
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
SEC("raw_tp/sys_enter")
long test_task(void *ctx)
{
return (long)bpf_get_current_task();
}
char _license[] SEC("license") = "GPL";
(2) Here are the test steps:
sudo yum install libbpf-devel kernel-devel bpftool
clang -target bpf -O2 -c test.c -o test.o
sudo sysctl -w net.core.bpf_jit_enable=1
sudo bpftool prog show name test_task
sudo rm -f /sys/fs/bpf/test_task
sudo bpftool prog load test.o /sys/fs/bpf/test_task
ID=$(sudo bpftool prog show pinned /sys/fs/bpf/test_task | grep -oE '^[0-9]+')
sudo bpftool prog dump jited id $ID
(3) Here are the test results:
Before: 6 instructions
...
64: lu12i.w $t1, 1093
68: ori $t1, $t1, 3320
6c: lu32i.d $t1, 0
70: lu52i.d $t1, $t1, -1792
74: jirl $ra, $t1, 0
78: move $a5, $a0
...
After: 1 instruction
...
64: move $a5, $tp
...
This is similar with commit 2bb138c ("bpf, arm64: Inline
bpf_get_current_task/_btf() helpers").
Additionally, a safety check for bpf_jit_enable is introduced in
bpf_jit_inlines_helper_call(). If CONFIG_BPF_JIT_ALWAYS_ON is not
set and JIT is disabled at runtime, the function returns false to
safely accommodate the fallback path to the BPF interpreter.
Without this check, bpf_jit_inlines_helper_call() unconditionally
returns true. As a result, the verifier would skip fixing up the
call offset, leaving insn->imm as the raw helper ID rather than
the expected "insn->imm = fn->func - __bpf_call_base".
When the fallback interpreter executes (__bpf_call_base + insn->imm)
with this raw ID, it jumps into an unaligned invalid address space,
triggering a fatal instruction alignment fault (ADEF) kernel panic.
(1) Here is the sample test_panic.c:
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
SEC("kprobe/sys_getpid")
int test_panic_vulner(void *ctx)
{
struct task_struct *task;
task = (struct task_struct *)bpf_get_current_task();
if (task)
bpf_printk("Task address: %p\n", task);
return 0;
}
char LICENSE[] SEC("license") = "GPL";
(2) Kernel panic reproduction steps without the bpf_jit_enable check:
clang -target bpf -O2 -g -c test_panic.c -o test_panic.o
sudo sysctl -w net.core.bpf_jit_enable=0
sudo bpftool prog load test_panic.o /sys/fs/bpf/test_panic autoattach
(3) Kernel panic information without the bpf_jit_enable check:
Kernel ade access[#1]:
...
ra: 9000000000486e50 ___bpf_prog_run+0x1370/0x36b0
ERA: 9000000000485383 __bpf_prog_ret0_warn+0x13/0x20
...
ESTAT: 00080000 [ADEF] (IS= ECode=8 EsubCode=0)
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
The pointer to thread_info is always available in the $tp register,
the call to bpf_get_smp_processor_id() can be inlined into a single
load instruction.
(1) Here is the sample test.c:
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
SEC("raw_tp/sys_enter")
int test_cpuid(void *ctx)
{
return bpf_get_smp_processor_id();
}
char _license[] SEC("license") = "GPL";
(2) Here are the test steps:
sudo yum install libbpf-devel kernel-devel bpftool
clang -target bpf -O2 -c test.c -o test.o
sudo sysctl -w net.core.bpf_jit_enable=1
sudo bpftool prog show name test_cpuid
sudo rm -f /sys/fs/bpf/test_cpuid
sudo bpftool prog load test.o /sys/fs/bpf/test_cpuid
ID=$(sudo bpftool prog show pinned /sys/fs/bpf/test_cpuid | grep -oE '^[0-9]+')
sudo bpftool prog dump jited id $ID
(3) Here are the test results:
Before: 6 instructions
...
64: lu12i.w $t1, 1213
68: ori $t1, $t1, 1680
6c: lu32i.d $t1, 0
70: lu52i.d $t1, $t1, -1792
74: jirl $ra, $t1, 0
78: move $a5, $a0
...
After: 1 instruction
...
64: ld.wu $a5, $tp, 16
...
This is similar with commit 2ddec2c ("riscv, bpf: inline
bpf_get_smp_processor_id()").
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Author
|
Upstream branch: 30dee2c |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request for series with
subject: Move thread_info into task_struct for LoongArch
version: 2
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=1109608