Mathieu Desnoyers fe90f3967b sched: Add missing memory barrier in switch_mm_cid ... Many architectures' switch_mm() (e.g. arm64) do not have an smp_mb() which the core scheduler code has depended upon since commit: commit 223baf9d17 ("sched: Fix performance regression introduced by mm_cid") If switch_mm() doesn't call smp_mb(), sched_mm_cid_remote_clear() can unset the actively used cid when it fails to observe active task after it sets lazy_put. There *is* a memory barrier between storing to rq->curr and _return to userspace_ (as required by membarrier), but the rseq mm_cid has stricter requirements: the barrier needs to be issued between store to rq->curr and switch_mm_cid(), which happens earlier than: - spin_unlock(), - switch_to(). So it's fine when the architecture switch_mm() happens to have that barrier already, but less so when the architecture only provides the full barrier in switch_to() or spin_unlock(). It is a bug in the rseq switch_mm_cid() implementation. All architectures that don't have memory barriers in switch_mm(), but rather have the full barrier either in finish_lock_switch() or switch_to() have them too late for the needs of switch_mm_cid(). Introduce a new smp_mb__after_switch_mm(), defined as smp_mb() in the generic barrier.h header, and use it in switch_mm_cid() for scheduler transitions where switch_mm() is expected to provide a memory barrier. Architectures can override smp_mb__after_switch_mm() if their switch_mm() implementation provides an implicit memory barrier. Override it with a no-op on x86 which implicitly provide this memory barrier by writing to CR3. Fixes: 223baf9d17 ("sched: Fix performance regression introduced by mm_cid") Reported-by: levi.yun <yeoreum.yun@arm.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> # for arm64 Acked-by: Dave Hansen <dave.hansen@linux.intel.com> # for x86 Cc: <stable@vger.kernel.org> # 6.4.x Cc: Linus Torvalds <torvalds@linux-foundation.org> Link: https://lore.kernel.org/r/20240415152114.59122-2-mathieu.desnoyers@efficios.com		2024-04-16 13:59:45 +02:00
..
Makefile	…
autogroup.c	…
autogroup.h	…
build_policy.c	…
build_utility.c	…
clock.c	…
completion.c	…
core.c	RISC-V Patches for the 6.9 Merge Window	2024-03-22 10:41:13 -07:00
core_sched.c	…
cpuacct.c	…
cpudeadline.c	…
cpudeadline.h	…
cpufreq.c	…
cpufreq_schedutil.c	sched/fair: Fix frequency selection for non-invariant case	2024-01-16 10:41:25 +01:00
cpupri.c	…
cpupri.h	…
cputime.c	…
deadline.c	sched/deadline: Introduce deadline servers	2023-11-15 09:57:51 +01:00
debug.c	sched/fair: Simplify util_est	2023-12-23 15:59:58 +01:00
fair.c	- Sumanth Korikkar has taught s390 to allocate hotplug-time page frames	2024-03-14 17:43:30 -07:00
features.h	sched/fair: Remove SCHED_FEAT(UTIL_EST_FASTUP, true)	2023-12-23 15:59:56 +01:00
idle.c	Core x86 changes for v6.9:	2024-03-11 19:53:15 -07:00
isolation.c	…
loadavg.c	…
membarrier.c	RISC-V Patches for the 6.9 Merge Window	2024-03-22 10:41:13 -07:00
pelt.c	…
pelt.h	sched/fair: Simplify util_est	2023-12-23 15:59:58 +01:00
psi.c	…
rt.c	sched: Unify runtime accounting across classes	2023-11-15 09:57:48 +01:00
sched-pelt.h	…
sched.h	sched: Add missing memory barrier in switch_mm_cid	2024-04-16 13:59:45 +02:00
smp.h	…
stats.c	…
stats.h	…
stop_task.c	sched: Unify runtime accounting across classes	2023-11-15 09:57:48 +01:00
swait.c	…
topology.c	sched/topology: Rename SD_SHARE_PKG_RESOURCES to SD_SHARE_LLC	2024-02-28 15:43:17 +01:00
wait.c	…
wait_bit.c	…