mirror of
https://github.com/torvalds/linux.git
synced 2026-04-18 06:44:00 -04:00
01f492e181
Pull kvm updates from Paolo Bonzini:
"Arm:
- Add support for tracing in the standalone EL2 hypervisor code,
which should help both debugging and performance analysis. This
uses the new infrastructure for 'remote' trace buffers that can be
exposed by non-kernel entities such as firmware, and which came
through the tracing tree
- Add support for GICv5 Per Processor Interrupts (PPIs), as the
starting point for supporting the new GIC architecture in KVM
- Finally add support for pKVM protected guests, where pages are
unmapped from the host as they are faulted into the guest and can
be shared back from the guest using pKVM hypercalls. Protected
guests are created using a new machine type identifier. As the
elusive guestmem has not yet delivered on its promises, anonymous
memory is also supported
This is only a first step towards full isolation from the host; for
example, the CPU register state and DMA accesses are not yet
isolated. Because this does not really yet bring fully what it
promises, it is hidden behind CONFIG_ARM_PKVM_GUEST +
'kvm-arm.mode=protected', and also triggers TAINT_USER when a VM is
created. Caveat emptor
- Rework the dreaded user_mem_abort() function to make it more
maintainable, reducing the amount of state being exposed to the
various helpers and rendering a substantial amount of state
immutable
- Expand the Stage-2 page table dumper to support NV shadow page
tables on a per-VM basis
- Tidy up the pKVM PSCI proxy code to be slightly less hard to
follow
- Fix both SPE and TRBE in non-VHE configurations so that they do not
generate spurious, out of context table walks that ultimately lead
to very bad HW lockups
- A small set of patches fixing the Stage-2 MMU freeing in error
cases
- Tighten-up accepted SMC immediate value to be only #0 for host
SMCCC calls
- The usual cleanups and other selftest churn
LoongArch:
- Use CSR_CRMD_PLV for kvm_arch_vcpu_in_kernel()
- Add DMSINTC irqchip in kernel support
RISC-V:
- Fix steal time shared memory alignment checks
- Fix vector context allocation leak
- Fix array out-of-bounds in pmu_ctr_read() and pmu_fw_ctr_read_hi()
- Fix double-free of sdata in kvm_pmu_clear_snapshot_area()
- Fix integer overflow in kvm_pmu_validate_counter_mask()
- Fix shift-out-of-bounds in make_xfence_request()
- Fix lost write protection on huge pages during dirty logging
- Split huge pages during fault handling for dirty logging
- Skip CSR restore if VCPU is reloaded on the same core
- Implement kvm_arch_has_default_irqchip() for KVM selftests
- Factored-out ISA checks into separate sources
- Added hideleg to struct kvm_vcpu_config
- Factored-out VCPU config into separate sources
- Support configuration of per-VM HGATP mode from KVM user space
s390:
- Support for ESA (31-bit) guests inside nested hypervisors
- Remove restriction on memslot alignment, which is not needed
anymore with the new gmap code
- Fix LPSW/E to update the bear (which of course is the breaking
event address register)
x86:
- Shut up various UBSAN warnings on reading module parameter before
they were initialized
- Don't zero-allocate page tables that are used for splitting
hugepages in the TDP MMU, as KVM is guaranteed to set all SPTEs in
the page table and thus write all bytes
- As an optimization, bail early when trying to unsync 4KiB mappings
if the target gfn can just be mapped with a 2MiB hugepage
x86 generic:
- Copy single-chunk MMIO write values into struct kvm_vcpu (more
precisely struct kvm_mmio_fragment) to fix use-after-free stack
bugs where KVM would dereference stack pointer after an exit to
userspace
- Clean up and comment the emulated MMIO code to try to make it
easier to maintain (not necessarily "easy", but "easier")
- Move VMXON+VMXOFF and EFER.SVME toggling out of KVM (not *all* of
VMX and SVM enabling) as it is needed for trusted I/O
- Advertise support for AVX512 Bit Matrix Multiply (BMM) instructions
- Immediately fail the build if a required #define is missing in one
of KVM's headers that is included multiple times
- Reject SET_GUEST_DEBUG with -EBUSY if there's an already injected
exception, mostly to prevent syzkaller from abusing the uAPI to
trigger WARNs, but also because it can help prevent userspace from
unintentionally crashing the VM
- Exempt SMM from CPUID faulting on Intel, as per the spec
- Misc hardening and cleanup changes
x86 (AMD):
- Fix and optimize IRQ window inhibit handling for AVIC; make it
per-vCPU so that KVM doesn't prematurely re-enable AVIC if multiple
vCPUs have to-be-injected IRQs
- Clean up and optimize the OSVW handling, avoiding a bug in which
KVM would overwrite state when enabling virtualization on multiple
CPUs in parallel. This should not be a problem because OSVW should
usually be the same for all CPUs
- Drop a WARN in KVM_MEMORY_ENCRYPT_REG_REGION where KVM complains
about a "too large" size based purely on user input
- Clean up and harden the pinning code for KVM_MEMORY_ENCRYPT_REG_REGION
- Disallow synchronizing a VMSA of an already-launched/encrypted
vCPU, as doing so for an SNP guest will crash the host due to an
RMP violation page fault
- Overhaul KVM's APIs for detecting SEV+ guests so that VM-scoped
queries are required to hold kvm->lock, and enforce it by lockdep.
Fix various bugs where sev_guest() was not ensured to be stable for
the whole duration of a function or ioctl
- Convert a pile of kvm->lock SEV code to guard()
- Play nicer with userspace that does not enable
KVM_CAP_EXCEPTION_PAYLOAD, for which KVM needs to set CR2 and DR6
as a response to ioctls such as KVM_GET_VCPU_EVENTS (even if the
payload would end up in EXITINFO2 rather than CR2, for example).
Only set CR2 and DR6 when consumption of the payload is imminent,
but on the other hand force delivery of the payload in all paths
where userspace retrieves CR2 or DR6
- Use vcpu->arch.cr2 when updating vmcb12's CR2 on nested #VMEXIT
instead of vmcb02->save.cr2. The value is out of sync after a
save/restore or after a #PF is injected into L2
- Fix a class of nSVM bugs where some fields written by the CPU are
not synchronized from vmcb02 to cached vmcb12 after VMRUN, and so
are not up-to-date when saved by KVM_GET_NESTED_STATE
- Fix a class of bugs where the ordering between KVM_SET_NESTED_STATE
and KVM_SET_{S}REGS could cause vmcb02 to be incorrectly
initialized after save+restore
- Add a variety of missing nSVM consistency checks
- Fix several bugs where KVM failed to correctly update VMCB fields
on nested #VMEXIT
- Fix several bugs where KVM failed to correctly synthesize #UD or
#GP for SVM-related instructions
- Add support for save+restore of virtualized LBRs (on SVM)
- Refactor various helpers and macros to improve clarity and
(hopefully) make the code easier to maintain
- Aggressively sanitize fields when copying from vmcb12, to guard
against unintentionally allowing L1 to utilize yet-to-be-defined
features
- Fix several bugs where KVM botched rAX legality checks when
emulating SVM instructions. There are remaining issues in that KVM
doesn't handle size prefix overrides for 64-bit guests
- Fail emulation of VMRUN/VMLOAD/VMSAVE if mapping vmcb12 fails
instead of somewhat arbitrarily synthesizing #GP (i.e. don't double
down on AMD's architectural but sketchy behavior of generating #GP
for "unsupported" addresses)
- Cache all used vmcb12 fields to further harden against TOCTOU bugs
x86 (Intel):
- Drop obsolete branch hint prefixes from the VMX instruction macros
- Use ASM_INPUT_RM() in __vmcs_writel() to coerce clang into using a
register input when appropriate
- Code cleanups
guest_memfd:
- Don't mark guest_memfd folios as accessed, as guest_memfd doesn't
support reclaim, the memory is unevictable, and there is no storage
to write back to
LoongArch selftests:
- Add KVM PMU test cases
s390 selftests:
- Enable more memory selftests
x86 selftests:
- Add support for Hygon CPUs in KVM selftests
- Fix a bug in the MSR test where it would get false failures on
AMD/Hygon CPUs with exactly one of RDPID or RDTSCP
- Add an MADV_COLLAPSE testcase for guest_memfd as a regression test
for a bug where the kernel would attempt to collapse guest_memfd
folios against KVM's will"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (373 commits)
KVM: x86: use inlines instead of macros for is_sev_*guest
x86/virt: Treat SVM as unsupported when running as an SEV+ guest
KVM: SEV: Goto an existing error label if charging misc_cg for an ASID fails
KVM: SVM: Move lock-protected allocation of SEV ASID into a separate helper
KVM: SEV: use mutex guard in snp_handle_guest_req()
KVM: SEV: use mutex guard in sev_mem_enc_unregister_region()
KVM: SEV: use mutex guard in sev_mem_enc_ioctl()
KVM: SEV: use mutex guard in snp_launch_update()
KVM: SEV: Assert that kvm->lock is held when querying SEV+ support
KVM: SEV: Document that checking for SEV+ guests when reclaiming memory is "safe"
KVM: SEV: Hide "struct kvm_sev_info" behind CONFIG_KVM_AMD_SEV=y
KVM: SEV: WARN on unhandled VM type when initializing VM
KVM: LoongArch: selftests: Add PMU overflow interrupt test
KVM: LoongArch: selftests: Add basic PMU event counting test
KVM: LoongArch: selftests: Add cpucfg read/write helpers
LoongArch: KVM: Add DMSINTC inject msi to vCPU
LoongArch: KVM: Add DMSINTC device support
LoongArch: KVM: Make vcpu_is_preempted() as a macro rather than function
LoongArch: KVM: Move host CSR_GSTAT save and restore in context switch
LoongArch: KVM: Move host CSR_EENTRY save and restore in context switch
...
298 lines
6.1 KiB
ArmAsm
298 lines
6.1 KiB
ArmAsm
/* SPDX-License-Identifier: GPL-2.0-only */
|
|
/*
|
|
* Copyright (C) 2012,2013 - ARM Ltd
|
|
* Author: Marc Zyngier <marc.zyngier@arm.com>
|
|
*/
|
|
|
|
#include <linux/arm-smccc.h>
|
|
#include <linux/cfi_types.h>
|
|
#include <linux/linkage.h>
|
|
|
|
#include <asm/alternative.h>
|
|
#include <asm/assembler.h>
|
|
#include <asm/el2_setup.h>
|
|
#include <asm/kvm_arm.h>
|
|
#include <asm/kvm_asm.h>
|
|
#include <asm/kvm_mmu.h>
|
|
#include <asm/pgtable-hwdef.h>
|
|
#include <asm/sysreg.h>
|
|
#include <asm/virt.h>
|
|
|
|
.text
|
|
.pushsection .idmap.text, "ax"
|
|
|
|
.align 11
|
|
|
|
SYM_CODE_START(__kvm_hyp_init)
|
|
ventry . // Synchronous EL2t
|
|
ventry . // IRQ EL2t
|
|
ventry . // FIQ EL2t
|
|
ventry . // Error EL2t
|
|
|
|
ventry . // Synchronous EL2h
|
|
ventry . // IRQ EL2h
|
|
ventry . // FIQ EL2h
|
|
ventry . // Error EL2h
|
|
|
|
ventry __do_hyp_init // Synchronous 64-bit EL1
|
|
ventry . // IRQ 64-bit EL1
|
|
ventry . // FIQ 64-bit EL1
|
|
ventry . // Error 64-bit EL1
|
|
|
|
ventry . // Synchronous 32-bit EL1
|
|
ventry . // IRQ 32-bit EL1
|
|
ventry . // FIQ 32-bit EL1
|
|
ventry . // Error 32-bit EL1
|
|
|
|
/*
|
|
* Only uses x0..x3 so as to not clobber callee-saved SMCCC registers.
|
|
*
|
|
* x0: SMCCC function ID
|
|
* x1: struct kvm_nvhe_init_params PA
|
|
*/
|
|
__do_hyp_init:
|
|
/* Check for a stub HVC call */
|
|
cmp x0, #HVC_STUB_HCALL_NR
|
|
b.lo __kvm_handle_stub_hvc
|
|
|
|
bic x0, x0, #ARM_SMCCC_CALL_HINTS
|
|
mov x3, #KVM_HOST_SMCCC_FUNC(__kvm_hyp_init)
|
|
cmp x0, x3
|
|
b.eq 1f
|
|
|
|
mov x0, #SMCCC_RET_NOT_SUPPORTED
|
|
eret
|
|
|
|
1: mov x0, x1
|
|
mov x3, lr
|
|
bl ___kvm_hyp_init // Clobbers x0..x2
|
|
mov lr, x3
|
|
|
|
/* Hello, World! */
|
|
mov x0, #SMCCC_RET_SUCCESS
|
|
eret
|
|
SYM_CODE_END(__kvm_hyp_init)
|
|
|
|
/*
|
|
* Initialize EL2 CPU state to sane values.
|
|
*
|
|
* HCR_EL2.E2H must have been initialized already.
|
|
*/
|
|
SYM_CODE_START_LOCAL(__kvm_init_el2_state)
|
|
init_el2_state // Clobbers x0..x2
|
|
finalise_el2_state
|
|
ret
|
|
SYM_CODE_END(__kvm_init_el2_state)
|
|
|
|
/*
|
|
* Initialize the hypervisor in EL2.
|
|
*
|
|
* Only uses x0..x2 so as to not clobber callee-saved SMCCC registers
|
|
* and leave x3 for the caller.
|
|
*
|
|
* x0: struct kvm_nvhe_init_params PA
|
|
*/
|
|
SYM_CODE_START_LOCAL(___kvm_hyp_init)
|
|
ldr x1, [x0, #NVHE_INIT_STACK_HYP_VA]
|
|
mov sp, x1
|
|
|
|
ldr x1, [x0, #NVHE_INIT_MAIR_EL2]
|
|
msr mair_el2, x1
|
|
|
|
ldr x1, [x0, #NVHE_INIT_HCR_EL2]
|
|
msr_hcr_el2 x1
|
|
|
|
mov x2, #HCR_E2H
|
|
and x2, x1, x2
|
|
cbz x2, 1f
|
|
|
|
// hVHE: Replay the EL2 setup to account for the E2H bit
|
|
// TPIDR_EL2 is used to preserve x0 across the macro maze...
|
|
isb
|
|
msr tpidr_el2, x0
|
|
str lr, [x0, #NVHE_INIT_TMP]
|
|
|
|
bl __kvm_init_el2_state
|
|
|
|
mrs x0, tpidr_el2
|
|
ldr lr, [x0, #NVHE_INIT_TMP]
|
|
|
|
1:
|
|
ldr x1, [x0, #NVHE_INIT_TPIDR_EL2]
|
|
msr tpidr_el2, x1
|
|
|
|
ldr x1, [x0, #NVHE_INIT_VTTBR]
|
|
msr vttbr_el2, x1
|
|
|
|
ldr x1, [x0, #NVHE_INIT_VTCR]
|
|
msr vtcr_el2, x1
|
|
|
|
ldr x1, [x0, #NVHE_INIT_PGD_PA]
|
|
phys_to_ttbr x2, x1
|
|
alternative_if ARM64_HAS_CNP
|
|
orr x2, x2, #TTBRx_EL1_CnP
|
|
alternative_else_nop_endif
|
|
msr ttbr0_el2, x2
|
|
|
|
ldr x0, [x0, #NVHE_INIT_TCR_EL2]
|
|
msr tcr_el2, x0
|
|
|
|
isb
|
|
|
|
/* Invalidate the stale TLBs from Bootloader */
|
|
tlbi alle2
|
|
tlbi alle1
|
|
dsb sy
|
|
|
|
mov_q x0, INIT_SCTLR_EL2_MMU_ON
|
|
alternative_if ARM64_HAS_ADDRESS_AUTH
|
|
mov_q x1, (SCTLR_ELx_ENIA | SCTLR_ELx_ENIB | \
|
|
SCTLR_ELx_ENDA | SCTLR_ELx_ENDB)
|
|
orr x0, x0, x1
|
|
alternative_else_nop_endif
|
|
|
|
#ifdef CONFIG_ARM64_BTI_KERNEL
|
|
alternative_if ARM64_BTI
|
|
orr x0, x0, #SCTLR_EL2_BT
|
|
alternative_else_nop_endif
|
|
#endif /* CONFIG_ARM64_BTI_KERNEL */
|
|
|
|
msr sctlr_el2, x0
|
|
isb
|
|
|
|
/* Set the host vector */
|
|
ldr x0, =__kvm_hyp_host_vector
|
|
msr vbar_el2, x0
|
|
|
|
ret
|
|
SYM_CODE_END(___kvm_hyp_init)
|
|
|
|
/*
|
|
* PSCI CPU_ON entry point
|
|
*
|
|
* x0: struct kvm_nvhe_init_params PA
|
|
*/
|
|
SYM_CODE_START(kvm_hyp_cpu_entry)
|
|
ldr x29, =__kvm_host_psci_cpu_on_entry
|
|
b __kvm_hyp_init_cpu
|
|
|
|
/*
|
|
* PSCI CPU_SUSPEND / SYSTEM_SUSPEND entry point
|
|
*
|
|
* x0: struct kvm_nvhe_init_params PA
|
|
*/
|
|
SYM_CODE_START(kvm_hyp_cpu_resume)
|
|
ldr x29, =__kvm_host_psci_cpu_resume_entry
|
|
|
|
SYM_INNER_LABEL(__kvm_hyp_init_cpu, SYM_L_LOCAL)
|
|
mov x28, x0 // Stash arguments
|
|
|
|
/* Check that the core was booted in EL2. */
|
|
mrs x0, CurrentEL
|
|
cmp x0, #CurrentEL_EL2
|
|
b.ne 1f
|
|
|
|
msr SPsel, #1 // We want to use SP_EL2
|
|
|
|
init_el2_hcr 0
|
|
|
|
bl __kvm_init_el2_state
|
|
|
|
/* Enable MMU, set vectors and stack. */
|
|
mov x0, x28
|
|
bl ___kvm_hyp_init // Clobbers x0..x2
|
|
|
|
/* Leave idmap -- using BLR is OK, LR is restored from host context */
|
|
blr x29
|
|
|
|
// The core booted in EL1, or the C code unexpectedly returned.
|
|
// Either way, KVM cannot be initialized on it.
|
|
1: wfe
|
|
wfi
|
|
b 1b
|
|
SYM_CODE_END(kvm_hyp_cpu_resume)
|
|
SYM_CODE_END(kvm_hyp_cpu_entry)
|
|
|
|
SYM_CODE_START(__kvm_handle_stub_hvc)
|
|
/*
|
|
* __kvm_handle_stub_hvc called from __host_hvc through branch instruction(br) so
|
|
* we need bti j at beginning.
|
|
*/
|
|
bti j
|
|
cmp x0, #HVC_SOFT_RESTART
|
|
b.ne 1f
|
|
|
|
/* This is where we're about to jump, staying at EL2 */
|
|
msr elr_el2, x1
|
|
mov x0, #(PSR_F_BIT | PSR_I_BIT | PSR_A_BIT | PSR_D_BIT | PSR_MODE_EL2h)
|
|
msr spsr_el2, x0
|
|
|
|
/* Shuffle the arguments, and don't come back */
|
|
mov x0, x2
|
|
mov x1, x3
|
|
mov x2, x4
|
|
b reset
|
|
|
|
1: cmp x0, #HVC_RESET_VECTORS
|
|
b.ne 1f
|
|
|
|
/*
|
|
* Set the HVC_RESET_VECTORS return code before entering the common
|
|
* path so that we do not clobber x0-x2 in case we are coming via
|
|
* HVC_SOFT_RESTART.
|
|
*/
|
|
mov x0, xzr
|
|
reset:
|
|
/* Reset kvm back to the hyp stub. */
|
|
mov_q x5, INIT_SCTLR_EL2_MMU_OFF
|
|
pre_disable_mmu_workaround
|
|
msr sctlr_el2, x5
|
|
isb
|
|
|
|
/* Install stub vectors */
|
|
adr_l x5, __hyp_stub_vectors
|
|
msr vbar_el2, x5
|
|
eret
|
|
|
|
1: /* Bad stub call */
|
|
mov_q x0, HVC_STUB_ERR
|
|
eret
|
|
|
|
SYM_CODE_END(__kvm_handle_stub_hvc)
|
|
|
|
/*
|
|
* void __pkvm_init_switch_pgd(phys_addr_t pgd, unsigned long sp,
|
|
* void (*fn)(void));
|
|
*
|
|
* SYM_TYPED_FUNC_START() allows C to call this ID-mapped function indirectly
|
|
* using a physical pointer without triggering a kCFI failure.
|
|
*/
|
|
SYM_TYPED_FUNC_START(__pkvm_init_switch_pgd)
|
|
/* Turn the MMU off */
|
|
pre_disable_mmu_workaround
|
|
mrs x3, sctlr_el2
|
|
bic x4, x3, #SCTLR_ELx_M
|
|
msr sctlr_el2, x4
|
|
isb
|
|
|
|
tlbi alle2
|
|
|
|
/* Install the new pgtables */
|
|
phys_to_ttbr x5, x0
|
|
alternative_if ARM64_HAS_CNP
|
|
orr x5, x5, #TTBRx_EL1_CnP
|
|
alternative_else_nop_endif
|
|
msr ttbr0_el2, x5
|
|
|
|
/* Set the new stack pointer */
|
|
mov sp, x1
|
|
|
|
/* And turn the MMU back on! */
|
|
dsb nsh
|
|
isb
|
|
set_sctlr_el2 x3
|
|
ret x2
|
|
SYM_FUNC_END(__pkvm_init_switch_pgd)
|
|
|
|
.popsection
|