[linux-nvidia-6.17] Backport support for FEAT_LS64 #292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

nvmochs wants to merge 10 commits into NVIDIA:24.04_linux-nvidia-6.17-next from nvmochs:617_feat_ls64

+706 −12

Documentation/arch/arm64/booting.rst

-Original file line number
+Diff line change
@@ Expand Up @@
        - MDCR_EL3.TPM (bit 6) must be initialized to 0b0
+      For CPUs with support for 64-byte loads and stores without status (FEAT_LS64):
+      - If the kernel is entered at EL1 and EL2 is present:
+        - HCRX_EL2.EnALS (bit 1) must be initialised to 0b1.
+      For CPUs with support for 64-byte stores with status (FEAT_LS64_V):
+      - If the kernel is entered at EL1 and EL2 is present:
+        - HCRX_EL2.EnASR (bit 2) must be initialised to 0b1.
     The requirements described above for CPU mode, caches, MMUs, architected
     timers, coherency and system registers apply to all CPUs.  All CPUs must
     enter the kernel in the same exception level.  Where the values documented
@@ Expand Down @@

Documentation/arch/arm64/elf_hwcaps.rst

-Original file line number
+Diff line change
@@ Expand Up / @@ -441,6 +441,14 @@ HWCAP3_MTE_FAR @@
     HWCAP3_MTE_STORE_ONLY
         Functionality implied by ID_AA64PFR2_EL1.MTESTOREONLY == 0b0001.
+    HWCAP3_LS64
+        Functionality implied by ID_AA64ISAR1_EL1.LS64 == 0b0001. Note that
+        the function of instruction ld64b/st64b requires support by CPU, system
+        and target (device) memory location and HWCAP3_LS64 implies the support
+        of CPU. User should only use ld64b/st64b on supported target (device)
+        memory location, otherwise fallback to the non-atomic alternatives.
 . Unused AT_HWCAP bits
     -----------------------
@@ Expand Down @@

Documentation/virt/kvm/api.rst

-Original file line number
+Diff line change
@@ Expand Up @@
     information or because there is no device mapped at the accessed IPA, then
     userspace can ask the kernel to inject an external abort using the address
     from the exiting fault on the VCPU. It is a programming error to set
-    ext_dabt_pending after an exit which was not either KVM_EXIT_MMIO or
-    KVM_EXIT_ARM_NISV. This feature is only available if the system supports
-    KVM_CAP_ARM_INJECT_EXT_DABT. This is a helper which provides commonality in
-    how userspace reports accesses for the above cases to guests, across different
-    userspace implementations. Nevertheless, userspace can still emulate all Arm
-    exceptions by manipulating individual registers using the KVM_SET_ONE_REG API.
+    ext_dabt_pending after an exit which was not either KVM_EXIT_MMIO,
+    KVM_EXIT_ARM_NISV, or KVM_EXIT_ARM_LDST64B. This feature is only available if
+    the system supports KVM_CAP_ARM_INJECT_EXT_DABT. This is a helper which
+    provides commonality in how userspace reports accesses for the above cases to
+    guests, across different userspace implementations. Nevertheless, userspace
+    can still emulate all Arm exceptions by manipulating individual registers
+    using the KVM_SET_ONE_REG API.
     See KVM_GET_VCPU_EVENTS for the data structure.
@@ Expand Down Expand Up / @@ -7009,12 +7010,14 @@ in send_page or recv a buffer to recv_page). @@
     ::
-    		/* KVM_EXIT_ARM_NISV */
+    		/* KVM_EXIT_ARM_NISV / KVM_EXIT_ARM_LDST64B */
     		struct {
     			__u64 esr_iss;
     			__u64 fault_ipa;
     		} arm_nisv;
+    - KVM_EXIT_ARM_NISV:
     Used on arm64 systems. If a guest accesses memory not in a memslot,
     KVM will typically return to userspace and ask it to do MMIO emulation on its
     behalf. However, for certain classes of instructions, no instruction decode
@@ Expand Down Expand Up @@
     queried outside of a protected VM context, the feature will not be
     exposed if queried on a protected VM file descriptor.
+    - KVM_EXIT_ARM_LDST64B:
+    Used on arm64 systems. When a guest using a LD64B, ST64B, ST64BV, ST64BV0,
+    outside of a memslot, KVM will return to userspace with KVM_EXIT_ARM_LDST64B,
+    exposing the relevant ESR_EL2 information and faulting IPA, similarly to
+    KVM_EXIT_ARM_NISV.
+    Userspace is supposed to fully emulate the instructions, which includes:
+    	- fetch of the operands for a store, including ACCDATA_EL1 in the case
+    	  of a ST64BV0 instruction
+    	- deal with the endianness if the guest is big-endian
+    	- emulate the access, including the delivery of an exception if the
+    	  access didn't succeed
+    	- provide a return value in the case of ST64BV/ST64BV0
+    	- return the data in the case of a load
+    	- increment PC if the instruction was successfully executed
+    Note that there is no expectation of performance for this emulation, as it
+    involves a large number of interaction with the guest state. It is, however,
+    expected that the instruction's semantics are preserved, specially the
+    single-copy atomicity property of the 64 byte access.
+    This exit reason must be handled if userspace sets ID_AA64ISAR1_EL1.LS64 to a
+    non-zero value, indicating that FEAT_LS64* is enabled.
     ::
     		/* KVM_EXIT_X86_RDMSR / KVM_EXIT_X86_WRMSR */
@@ Expand Down Expand Up @@
     it will enter with output fields already valid; in the common case, the
     ``unknown.ret`` field of the union will be ``TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED``.
     Userspace need not do anything if it does not wish to support a TDVMCALL.
+    ::
+    		/* KVM_EXIT_ARM_SEA */
+    		struct {
+      #define KVM_EXIT_ARM_SEA_FLAG_GPA_VALID   (1ULL << 0)
+    			__u64 flags;
+    			__u64 esr;
+    			__u64 gva;
+    			__u64 gpa;
+    		} arm_sea;
+    Used on arm64 systems. When the VM capability ``KVM_CAP_ARM_SEA_TO_USER`` is
+    enabled, a KVM exits to userspace if a guest access causes a synchronous
+    external abort (SEA) and the host APEI fails to handle the SEA.
+    ``esr`` is set to a sanitized value of ESR_EL2 from the exception taken to KVM,
+    consisting of the following fields:
+     - ``ESR_EL2.EC``
+     - ``ESR_EL2.IL``
+     - ``ESR_EL2.FnV``
+     - ``ESR_EL2.EA``
+     - ``ESR_EL2.CM``
+     - ``ESR_EL2.WNR``
+     - ``ESR_EL2.FSC``
+     - ``ESR_EL2.SET`` (when FEAT_RAS is implemented for the VM)
+    ``gva`` is set to the value of FAR_EL2 from the exception taken to KVM when
+    ``ESR_EL2.FnV == 0``. Otherwise, the value of ``gva`` is unknown.
+    ``gpa`` is set to the faulting IPA from the exception taken to KVM when
+    the ``KVM_EXIT_ARM_SEA_FLAG_GPA_VALID`` flag is set. Otherwise, the value of
+    ``gpa`` is unknown.
     ::
     		/* Fix the size of the union. */
@@ Expand Down Expand Up @@
     can be safely mapped as cacheable. This relies on the presence of
     force write back (FWB) feature support on the hardware.
+.45 KVM_CAP_ARM_SEA_TO_USER
+    ----------------------------
+    :Architecture: arm64
+    :Target: VM
+    :Parameters: none
+    :Returns: 0 on success, -EINVAL if unsupported.
+    When this capability is enabled, KVM may exit to userspace for SEAs taken to
+    EL2 resulting from a guest access. See ``KVM_EXIT_ARM_SEA`` for more
+    information.
 . Other capabilities.
     ======================
@@ Expand Down @@

arch/arm64/include/asm/el2_setup.h

-Original file line number
+Diff line change
@@ Expand Up / @@ -57,9 +57,19 @@ @@
             /* Enable GCS if supported */
     	mrs_s	x1, SYS_ID_AA64PFR1_EL1
     	ubfx	x1, x1, #ID_AA64PFR1_EL1_GCS_SHIFT, #4
-    	cbz	x1, .Lset_hcrx_\@
+    	cbz	x1, .Lskip_gcs_hcrx_\@
     	orr	x0, x0, #HCRX_EL2_GCSEn
+    .Lskip_gcs_hcrx_\@:
+    	/* Enable LS64, LS64_V if supported */
+    	mrs_s	x1, SYS_ID_AA64ISAR1_EL1
+    	ubfx	x1, x1, #ID_AA64ISAR1_EL1_LS64_SHIFT, #4
+    	cbz	x1, .Lset_hcrx_\@
+    	orr	x0, x0, #HCRX_EL2_EnALS
+    	cmp	x1, #ID_AA64ISAR1_EL1_LS64_LS64_V
+    	b.lt	.Lset_hcrx_\@
+    	orr	x0, x0, #HCRX_EL2_EnASR
     .Lset_hcrx_\@:
     	msr_s	SYS_HCRX_EL2, x0
     .Lskip_hcrx_\@:
@@ Expand Down @@

arch/arm64/include/asm/esr.h

-Original file line number
+Diff line change
@@ Expand Up / @@ -124,6 +124,7 @@ @@
     #define ESR_ELx_FSC_SEA_TTW(n)	(0x14 + (n))
     #define ESR_ELx_FSC_SECC	(0x18)
     #define ESR_ELx_FSC_SECC_TTW(n)	(0x1c + (n))
+    #define ESR_ELx_FSC_EXCL_ATOMIC	(0x35)
     #define ESR_ELx_FSC_ADDRSZ	(0x00)
     /*
@@ Expand Down Expand Up @@
     	       (esr == ESR_ELx_FSC_ACCESS_L(0));
     }
+    static inline bool esr_fsc_is_excl_atomic_fault(unsigned long esr)
+    {
+    	esr = esr & ESR_ELx_FSC;
+    	return esr == ESR_ELx_FSC_EXCL_ATOMIC;
+    }
     static inline bool esr_fsc_is_addr_sz_fault(unsigned long esr)
     {
     	esr &= ESR_ELx_FSC;
@@ Expand Down @@

arch/arm64/include/asm/hwcap.h

-Original file line number
+Diff line change
@@ Expand Up / @@ -178,6 +178,7 @@ @@
     #define __khwcap3_feature(x)		(const_ilog2(HWCAP3_ ## x) + 128)
     #define KERNEL_HWCAP_MTE_FAR		__khwcap3_feature(MTE_FAR)
     #define KERNEL_HWCAP_MTE_STORE_ONLY	__khwcap3_feature(MTE_STORE_ONLY)
+    #define KERNEL_HWCAP_LS64		__khwcap3_feature(LS64)
     /*
      * This yields a mask that user programs can use to figure out what
@@ Expand Down @@

arch/arm64/include/asm/kvm_emulate.h

-Original file line number
+Diff line change
@@ Expand Up / @@ -47,6 +47,7 @@ void kvm_skip_instr32(struct kvm_vcpu *vcpu); @@
     void kvm_inject_undefined(struct kvm_vcpu *vcpu);
     int kvm_inject_serror_esr(struct kvm_vcpu *vcpu, u64 esr);
     int kvm_inject_sea(struct kvm_vcpu *vcpu, bool iabt, u64 addr);
+    int kvm_inject_dabt_excl_atomic(struct kvm_vcpu *vcpu, u64 addr);
     void kvm_inject_size_fault(struct kvm_vcpu *vcpu);
     static inline int kvm_inject_sea_dabt(struct kvm_vcpu *vcpu, u64 addr)
@@ Expand Down Expand Up / @@ -672,6 +673,12 @@ static inline void vcpu_set_hcrx(struct kvm_vcpu *vcpu) @@
     		if (kvm_has_sctlr2(kvm))
     			vcpu->arch.hcrx_el2 |= HCRX_EL2_SCTLR2En;
+    		if (kvm_has_feat(kvm, ID_AA64ISAR1_EL1, LS64, LS64))
+    			vcpu->arch.hcrx_el2 |= HCRX_EL2_EnALS;
+    		if (kvm_has_feat(kvm, ID_AA64ISAR1_EL1, LS64, LS64_V))
+    			vcpu->arch.hcrx_el2 |= HCRX_EL2_EnASR;
     	}
     }
     #endif /* __ARM64_KVM_EMULATE_H__ */

arch/arm64/include/asm/kvm_host.h

-Original file line number
+Diff line change
@@ Expand Up / @@ -349,6 +349,8 @@ struct kvm_arch { @@
     #define KVM_ARCH_FLAG_GUEST_HAS_SVE			9
     	/* MIDR_EL1, REVIDR_EL1, and AIDR_EL1 are writable from userspace */
     #define KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS		10
+    	/* Unhandled SEAs are taken to userspace */
+    #define KVM_ARCH_FLAG_EXIT_SEA				11
     	unsigned long flags;
     	/* VM-wide vCPU feature set */
@@ Expand Down @@

arch/arm64/include/uapi/asm/hwcap.h

-Original file line number
+Diff line change
@@ Expand Up / @@ -145,5 +145,6 @@ @@
      */
     #define HWCAP3_MTE_FAR		(1UL << 0)
     #define HWCAP3_MTE_STORE_ONLY		(1UL << 1)
+    #define HWCAP3_LS64		(1UL << 3)
     #endif /* _UAPI__ASM_HWCAP_H */

arch/arm64/kernel/cpufeature.c

-Original file line number
+Diff line change
@@ Expand Up / @@ -240,6 +240,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar0[] = { @@
     };
     static const struct arm64_ftr_bits ftr_id_aa64isar1[] = {
+    	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_EL1_LS64_SHIFT, 4, 0),
     	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_EL1_XS_SHIFT, 4, 0),
     	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_EL1_I8MM_SHIFT, 4, 0),
     	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_EL1_DGH_SHIFT, 4, 0),
@@ Expand Down Expand Up @@
     }
     #endif /* CONFIG_ARM64_E0PD */
+    static void cpu_enable_ls64(struct arm64_cpu_capabilities const *cap)
+    {
+    	sysreg_clear_set(sctlr_el1, SCTLR_EL1_EnALS, SCTLR_EL1_EnALS);
+    }
+    static void cpu_enable_ls64_v(struct arm64_cpu_capabilities const *cap)
+    {
+    	sysreg_clear_set(sctlr_el1, SCTLR_EL1_EnASR, 0);
+    }
     #ifdef CONFIG_ARM64_PSEUDO_NMI
     static bool can_use_gic_priorities(const struct arm64_cpu_capabilities *entry,
     				   int scope)
@@ Expand Down Expand Up @@
     		.matches = has_cpuid_feature,
     		ARM64_CPUID_FIELDS(ID_AA64PFR2_EL1, GCIE, IMP)
     	},
+    	{
+    		.desc = "LS64",
+    		.capability = ARM64_HAS_LS64,
+    		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+    		.matches = has_cpuid_feature,
+    		.cpu_enable = cpu_enable_ls64,
+    		ARM64_CPUID_FIELDS(ID_AA64ISAR1_EL1, LS64, LS64)
+    	},
+    	{
+    		.desc = "LS64_V",
+    		.capability = ARM64_HAS_LS64_V,
+    		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+    		.matches = has_cpuid_feature,
+    		.cpu_enable = cpu_enable_ls64_v,
+    		ARM64_CPUID_FIELDS(ID_AA64ISAR1_EL1, LS64, LS64_V)
+    	},
     	{},
     };
@@ Expand Down Expand Up @@
     	HWCAP_CAP(ID_AA64ISAR1_EL1, BF16, EBF16, CAP_HWCAP, KERNEL_HWCAP_EBF16),
     	HWCAP_CAP(ID_AA64ISAR1_EL1, DGH, IMP, CAP_HWCAP, KERNEL_HWCAP_DGH),
     	HWCAP_CAP(ID_AA64ISAR1_EL1, I8MM, IMP, CAP_HWCAP, KERNEL_HWCAP_I8MM),
+    	HWCAP_CAP(ID_AA64ISAR1_EL1, LS64, LS64, CAP_HWCAP, KERNEL_HWCAP_LS64),
     	HWCAP_CAP(ID_AA64ISAR2_EL1, LUT, IMP, CAP_HWCAP, KERNEL_HWCAP_LUT),
     	HWCAP_CAP(ID_AA64ISAR3_EL1, FAMINMAX, IMP, CAP_HWCAP, KERNEL_HWCAP_FAMINMAX),
     	HWCAP_CAP(ID_AA64MMFR2_EL1, AT, IMP, CAP_HWCAP, KERNEL_HWCAP_USCAT),
@@ Expand Down @@

arch/arm64/kernel/cpuinfo.c

-Original file line number
+Diff line change
@@ Expand Up / @@ -81,6 +81,7 @@ static const char *const hwcap_str[] = { @@
     	[KERNEL_HWCAP_PACA]		= "paca",
     	[KERNEL_HWCAP_PACG]		= "pacg",
     	[KERNEL_HWCAP_GCS]		= "gcs",
+    	[KERNEL_HWCAP_LS64]		= "ls64",
     	[KERNEL_HWCAP_DCPODP]		= "dcpodp",
     	[KERNEL_HWCAP_SVE2]		= "sve2",
     	[KERNEL_HWCAP_SVEAES]		= "sveaes",
@@ Expand Down @@

arch/arm64/kvm/arm.c

-Original file line number
+Diff line change
@@ Expand Up / @@ -133,6 +133,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, @@
     		}
     		mutex_unlock(&kvm->lock);
     		break;
+    	case KVM_CAP_ARM_SEA_TO_USER:
+    		r = 0;
+    		set_bit(KVM_ARCH_FLAG_EXIT_SEA, &kvm->arch.flags);
+    		break;
     	default:
     		break;
     	}
@@ Expand Down Expand Up / @@ -322,6 +326,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) @@
     	case KVM_CAP_IRQFD_RESAMPLE:
     	case KVM_CAP_COUNTER_OFFSET:
     	case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS:
+    	case KVM_CAP_ARM_SEA_TO_USER:
     		r = 1;
     		break;
     	case KVM_CAP_SET_GUEST_DEBUG2:
@@ Expand Down @@

arch/arm64/kvm/inject_fault.c

-Original file line number
+Diff line change
@@ Expand Up @@
     	return 1;
     }
+    static int kvm_inject_nested_excl_atomic(struct kvm_vcpu *vcpu, u64 addr)
+    {
+    	u64 esr = FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_DABT_LOW) |
+    		  FIELD_PREP(ESR_ELx_FSC, ESR_ELx_FSC_EXCL_ATOMIC) |
+    		  ESR_ELx_IL;
+    	vcpu_write_sys_reg(vcpu, addr, FAR_EL2);
+    	return kvm_inject_nested_sync(vcpu, esr);
+    }
+    /**
+     * kvm_inject_dabt_excl_atomic - inject a data abort for unsupported exclusive
+     *				 or atomic access
+     * @vcpu: The VCPU to receive the data abort
+     * @addr: The address to report in the DFAR
+     *
+     * It is assumed that this code is called from the VCPU thread and that the
+     * VCPU therefore is not currently executing guest code.
+     */
+    int kvm_inject_dabt_excl_atomic(struct kvm_vcpu *vcpu, u64 addr)
+    {
+    	u64 esr;
+    	if (is_nested_ctxt(vcpu) && (vcpu_read_sys_reg(vcpu, HCR_EL2) & HCR_VM))
+    		return kvm_inject_nested_excl_atomic(vcpu, addr);
+    	__kvm_inject_sea(vcpu, false, addr);
+    	esr = vcpu_read_sys_reg(vcpu, exception_esr_elx(vcpu));
+    	esr &= ~ESR_ELx_FSC;
+    	esr |= ESR_ELx_FSC_EXCL_ATOMIC;
+    	vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu));
+    	return 1;
+    }
     void kvm_inject_size_fault(struct kvm_vcpu *vcpu)
     {
     	unsigned long addr, esr;
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[linux-nvidia-6.17] Backport support for FEAT_LS64 #292

Diff view

Diff view

There are no files selected for viewing

Uh oh!

[linux-nvidia-6.17] Backport support for FEAT_LS64 #292

Are you sure you want to change the base?

[linux-nvidia-6.17] Backport support for FEAT_LS64 #292

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!