Gh0st
/
linux
mirror of https://github.com/torvalds/linux.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Significant patch series in this merge are as follows: 

- The 10 patch series "__vmalloc()/kvmalloc() and no-block support" from
   Uladzislau Rezki reworks the vmalloc() code to support non-blocking
   allocations (GFP_ATOIC, GFP_NOWAIT).
 
 - The 2 patch series "ksm: fix exec/fork inheritance" from xu xin fixes
   a rare case where the KSM MMF_VM_MERGE_ANY prctl state is not inherited
   across fork/exec.
 
 - The 4 patch series "mm/zswap: misc cleanup of code and documentations"
   from SeongJae Park does some light maintenance work on the zswap code.
 
 - The 5 patch series "mm/page_owner: add debugfs files 'show_handles'
   and 'show_stacks_handles'" from Mauricio Faria de Oliveira enhances the
   /sys/kernel/debug/page_owner debug feature.  It adds unique identifiers
   to differentiate the various stack traces so that userspace monitoring
   tools can better match stack traces over time.
 
 - The 2 patch series "mm/page_alloc: pcp->batch cleanups" from Joshua
   Hahn makes some minor alterations to the page allocator's per-cpu-pages
   feature.
 
 - The 2 patch series "Improve UFFDIO_MOVE scalability by removing
   anon_vma lock" from Lokesh Gidra addresses a scalability issue in
   userfaultfd's UFFDIO_MOVE operation.
 
 - The 2 patch series "kasan: cleanups for kasan_enabled() checks" from
   Sabyrzhan Tasbolatov performs some cleanup in the KASAN code.
 
 - The 2 patch series "drivers/base/node: fold node register and
   unregister functions" from Donet Tom cleans up the NUMA node handling
   code a little.
 
 - The 4 patch series "mm: some optimizations for prot numa" from Kefeng
   Wang provides some cleanups and small optimizations to the NUMA
   allocation hinting code.
 
 - The 5 patch series "mm/page_alloc: Batch callers of
   free_pcppages_bulk" from Joshua Hahn addresses long lock hold times at
   boot on large machines.  These were causing (harmless) softlockup
   warnings.
 
 - The 2 patch series "optimize the logic for handling dirty file folios
   during reclaim" from Baolin Wang removes some now-unnecessary work from
   page reclaim.
 
 - The 10 patch series "mm/damon: allow DAMOS auto-tuned for per-memcg
   per-node memory usage" from SeongJae Park enhances the DAMOS auto-tuning
   feature.
 
 - The 2 patch series "mm/damon: fixes for address alignment issues in
   DAMON_LRU_SORT and DAMON_RECLAIM" from Quanmin Yan fixes DAMON_LRU_SORT
   and DAMON_RECLAIM with certain userspace configuration.
 
 - The 15 patch series "expand mmap_prepare functionality, port more
   users" from Lorenzo Stoakes enhances the new(ish)
   file_operations.mmap_prepare() method and ports additional callsites
   from the old ->mmap() over to ->mmap_prepare().
 
 - The 8 patch series "Fix stale IOTLB entries for kernel address space"
   from Lu Baolu fixes a bug (and possible security issue on non-x86) in
   the IOMMU code.  In some situations the IOMMU could be left hanging onto
   a stale kernel pagetable entry.
 
 - The 4 patch series "mm/huge_memory: cleanup __split_unmapped_folio()"
   from Wei Yang cleans up and optimizes the folio splitting code.
 
 - The 5 patch series "mm, swap: misc cleanup and bugfix" from Kairui
   Song implements some cleanups and a minor fix in the swap discard code.
 
 - The 8 patch series "mm/damon: misc documentation fixups" from SeongJae
   Park does as advertised.
 
 - The 9 patch series "mm/damon: support pin-point targets removal" from
   SeongJae Park permits userspace to remove a specific monitoring target
   in the middle of the current targets list.
 
 - The 2 patch series "mm: MISC follow-up patches for linux/pgalloc.h"
   from Harry Yoo implements a couple of cleanups related to mm header file
   inclusion.
 
 - The 2 patch series "mm/swapfile.c: select swap devices of default
   priority round robin" from Baoquan He improves the selection of swap
   devices for NUMA machines.
 
 - The 3 patch series "mm: Convert memory block states (MEM_*) macros to
   enums" from Israel Batista changes the memory block labels from macros
   to enums so they will appear in kernel debug info.
 
 - The 3 patch series "ksm: perform a range-walk to jump over holes in
   break_ksm" from Pedro Demarchi Gomes addresses an inefficiency when KSM
   unmerges an address range.
 
 - The 22 patch series "mm/damon/tests: fix memory bugs in kunit tests"
   from SeongJae Park fixes leaks and unhandled malloc() failures in DAMON
   userspace unit tests.
 
 - The 2 patch series "some cleanups for pageout()" from Baolin Wang
   cleans up a couple of minor things in the page scanner's
   writeback-for-eviction code.
 
 - The 2 patch series "mm/hugetlb: refactor sysfs/sysctl interfaces" from
   Hui Zhu moves hugetlb's sysfs/sysctl handling code into a new file.
 
 - The 9 patch series "introduce VM_MAYBE_GUARD and make it sticky" from
   Lorenzo Stoakes makes the VMA guard regions available in /proc/pid/smaps
   and improves the mergeability of guarded VMAs.
 
 - The 2 patch series "mm: perform guard region install/remove under VMA
   lock" from Lorenzo Stoakes reduces mmap lock contention for callers
   performing VMA guard region operations.
 
 - The 2 patch series "vma_start_write_killable" from Matthew Wilcox
   starts work in permitting applications to be killed when they are
   waiting on a read_lock on the VMA lock.
 
 - The 11 patch series "mm/damon/tests: add more tests for online
   parameters commit" from SeongJae Park adds additional userspace testing
   of DAMON's "commit" feature.
 
 - The 9 patch series "mm/damon: misc cleanups" from SeongJae Park does
   that.
 
 - The 2 patch series "make VM_SOFTDIRTY a sticky VMA flag" from Lorenzo
   Stoakes addresses the possible loss of a VMA's VM_SOFTDIRTY flag when
   that VMA is merged with another.
 
 - The 16 patch series "mm: support device-private THP" from Balbir Singh
   introduces support for Transparent Huge Page (THP) migration in zone
   device-private memory.
 
 - The 3 patch series "Optimize folio split in memory failure" from Zi
   Yan optimizes folio split operations in the memory failure code.
 
 - The 2 patch series "mm/huge_memory: Define split_type and consolidate
   split support checks" from Wei Yang provides some more cleanups in the
   folio splitting code.
 
 - The 16 patch series "mm: remove is_swap_[pte, pmd]() + non-swap
   entries, introduce leaf entries" from Lorenzo Stoakes cleans up our
   handling of pagetable leaf entries by introducing the concept of
   'software leaf entries', of type softleaf_t.
 
 - The 4 patch series "reparent the THP split queue" from Muchun Song
   reparents the THP split queue to its parent memcg.  This is in
   preparation for addressing the long-standing "dying memcg" problem,
   wherein dead memcg's linger for too long, consuming memory resources.
 
 - The 3 patch series "unify PMD scan results and remove redundant
   cleanup" from Wei Yang does a little cleanup in the hugepage collapse
   code.
 
 - The 6 patch series "zram: introduce writeback bio batching" from
   Sergey Senozhatsky improves zram writeback efficiency by introducing
   batched bio writeback support.
 
 - The 4 patch series "memcg: cleanup the memcg stats interfaces" from
   Shakeel Butt cleans up our handling of the interrupt safety of some
   memcg stats.
 
 - The 4 patch series "make vmalloc gfp flags usage more apparent" from
   Vishal Moola cleans up vmalloc's handling of incoming GFP flags.
 
 - The 6 patch series "mm: Add soft-dirty and uffd-wp support for RISC-V"
   from Chunyan Zhang teches soft dirty and userfaultfd write protect
   tracking to use RISC-V's Svrsw60t59b extension.
 
 - The 5 patch series "mm: swap: small fixes and comment cleanups" from
   Youngjun Park fixes a small bug and cleans up some of the swap code.
 
 - The 4 patch series "initial work on making VMA flags a bitmap" from
   Lorenzo Stoakes starts work on converting the vma struct's flags to a
   bitmap, so we stop running out of them, especially on 32-bit.
 
 - The 2 patch series "mm/swapfile: fix and cleanup swap list iterations"
   from Youngjun Park addresses a possible bug in the swap discard code and
   cleans things up a little.
 -----BEGIN PGP SIGNATURE-----
 
 iHUEABYKAB0WIQTTMBEPP41GrTpTJgfdBJ7gKXxAjgUCaTEb0wAKCRDdBJ7gKXxA
 jjfIAP94W4EkCCwNOupnChoG+YWw/JW21anXt5NN+i5svn1yugEAwzvv6A+cAFng
 o+ug/fyrfPZG7PLp2R8WFyGIP0YoBA4=
 =IUzS
 -----END PGP SIGNATURE-----

Merge tag 'mm-stable-2025-12-03-21-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Pull MM updates from Andrew Morton:

  "__vmalloc()/kvmalloc() and no-block support" (Uladzislau Rezki)
     Rework the vmalloc() code to support non-blocking allocations
     (GFP_ATOIC, GFP_NOWAIT)

  "ksm: fix exec/fork inheritance" (xu xin)
     Fix a rare case where the KSM MMF_VM_MERGE_ANY prctl state is not
     inherited across fork/exec

  "mm/zswap: misc cleanup of code and documentations" (SeongJae Park)
     Some light maintenance work on the zswap code

  "mm/page_owner: add debugfs files 'show_handles' and 'show_stacks_handles'" (Mauricio Faria de Oliveira)
     Enhance the /sys/kernel/debug/page_owner debug feature by adding
     unique identifiers to differentiate the various stack traces so
     that userspace monitoring tools can better match stack traces over
     time

  "mm/page_alloc: pcp->batch cleanups" (Joshua Hahn)
     Minor alterations to the page allocator's per-cpu-pages feature

  "Improve UFFDIO_MOVE scalability by removing anon_vma lock" (Lokesh Gidra)
     Address a scalability issue in userfaultfd's UFFDIO_MOVE operation

  "kasan: cleanups for kasan_enabled() checks" (Sabyrzhan Tasbolatov)

  "drivers/base/node: fold node register and unregister functions" (Donet Tom)
     Clean up the NUMA node handling code a little

  "mm: some optimizations for prot numa" (Kefeng Wang)
     Cleanups and small optimizations to the NUMA allocation hinting
     code

  "mm/page_alloc: Batch callers of free_pcppages_bulk" (Joshua Hahn)
     Address long lock hold times at boot on large machines. These were
     causing (harmless) softlockup warnings

  "optimize the logic for handling dirty file folios during reclaim" (Baolin Wang)
     Remove some now-unnecessary work from page reclaim

  "mm/damon: allow DAMOS auto-tuned for per-memcg per-node memory usage" (SeongJae Park)
     Enhance the DAMOS auto-tuning feature

  "mm/damon: fixes for address alignment issues in DAMON_LRU_SORT and DAMON_RECLAIM" (Quanmin Yan)
     Fix DAMON_LRU_SORT and DAMON_RECLAIM with certain userspace
     configuration

  "expand mmap_prepare functionality, port more users" (Lorenzo Stoakes)
     Enhance the new(ish) file_operations.mmap_prepare() method and port
     additional callsites from the old ->mmap() over to ->mmap_prepare()

  "Fix stale IOTLB entries for kernel address space" (Lu Baolu)
     Fix a bug (and possible security issue on non-x86) in the IOMMU
     code. In some situations the IOMMU could be left hanging onto a
     stale kernel pagetable entry

  "mm/huge_memory: cleanup __split_unmapped_folio()" (Wei Yang)
     Clean up and optimize the folio splitting code

  "mm, swap: misc cleanup and bugfix" (Kairui Song)
     Some cleanups and a minor fix in the swap discard code

  "mm/damon: misc documentation fixups" (SeongJae Park)

  "mm/damon: support pin-point targets removal" (SeongJae Park)
     Permit userspace to remove a specific monitoring target in the
     middle of the current targets list

  "mm: MISC follow-up patches for linux/pgalloc.h" (Harry Yoo)
     A couple of cleanups related to mm header file inclusion

  "mm/swapfile.c: select swap devices of default priority round robin" (Baoquan He)
     improve the selection of swap devices for NUMA machines

  "mm: Convert memory block states (MEM_*) macros to enums" (Israel Batista)
     Change the memory block labels from macros to enums so they will
     appear in kernel debug info

  "ksm: perform a range-walk to jump over holes in break_ksm" (Pedro Demarchi Gomes)
     Address an inefficiency when KSM unmerges an address range

  "mm/damon/tests: fix memory bugs in kunit tests" (SeongJae Park)
     Fix leaks and unhandled malloc() failures in DAMON userspace unit
     tests

  "some cleanups for pageout()" (Baolin Wang)
     Clean up a couple of minor things in the page scanner's
     writeback-for-eviction code

  "mm/hugetlb: refactor sysfs/sysctl interfaces" (Hui Zhu)
     Move hugetlb's sysfs/sysctl handling code into a new file

  "introduce VM_MAYBE_GUARD and make it sticky" (Lorenzo Stoakes)
     Make the VMA guard regions available in /proc/pid/smaps and
     improves the mergeability of guarded VMAs

  "mm: perform guard region install/remove under VMA lock" (Lorenzo Stoakes)
     Reduce mmap lock contention for callers performing VMA guard region
     operations

  "vma_start_write_killable" (Matthew Wilcox)
     Start work on permitting applications to be killed when they are
     waiting on a read_lock on the VMA lock

  "mm/damon/tests: add more tests for online parameters commit" (SeongJae Park)
     Add additional userspace testing of DAMON's "commit" feature

  "mm/damon: misc cleanups" (SeongJae Park)

  "make VM_SOFTDIRTY a sticky VMA flag" (Lorenzo Stoakes)
     Address the possible loss of a VMA's VM_SOFTDIRTY flag when that
     VMA is merged with another

  "mm: support device-private THP" (Balbir Singh)
     Introduce support for Transparent Huge Page (THP) migration in zone
     device-private memory

  "Optimize folio split in memory failure" (Zi Yan)

  "mm/huge_memory: Define split_type and consolidate split support checks" (Wei Yang)
     Some more cleanups in the folio splitting code

  "mm: remove is_swap_[pte, pmd]() + non-swap entries, introduce leaf entries" (Lorenzo Stoakes)
     Clean up our handling of pagetable leaf entries by introducing the
     concept of 'software leaf entries', of type softleaf_t

  "reparent the THP split queue" (Muchun Song)
     Reparent the THP split queue to its parent memcg. This is in
     preparation for addressing the long-standing "dying memcg" problem,
     wherein dead memcg's linger for too long, consuming memory
     resources

  "unify PMD scan results and remove redundant cleanup" (Wei Yang)
     A little cleanup in the hugepage collapse code

  "zram: introduce writeback bio batching" (Sergey Senozhatsky)
     Improve zram writeback efficiency by introducing batched bio
     writeback support

  "memcg: cleanup the memcg stats interfaces" (Shakeel Butt)
     Clean up our handling of the interrupt safety of some memcg stats

  "make vmalloc gfp flags usage more apparent" (Vishal Moola)
     Clean up vmalloc's handling of incoming GFP flags

  "mm: Add soft-dirty and uffd-wp support for RISC-V" (Chunyan Zhang)
     Teach soft dirty and userfaultfd write protect tracking to use
     RISC-V's Svrsw60t59b extension

  "mm: swap: small fixes and comment cleanups" (Youngjun Park)
     Fix a small bug and clean up some of the swap code

  "initial work on making VMA flags a bitmap" (Lorenzo Stoakes)
     Start work on converting the vma struct's flags to a bitmap, so we
     stop running out of them, especially on 32-bit

  "mm/swapfile: fix and cleanup swap list iterations" (Youngjun Park)
     Address a possible bug in the swap discard code and clean things
     up a little

[ This merge also reverts commit ebb9aeb980 ("vfio/nvgrace-gpu:
  register device memory for poison handling") because it looks
  broken to me, I've asked for clarification   - Linus ]

* tag 'mm-stable-2025-12-03-21-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (321 commits)
  mm: fix vma_start_write_killable() signal handling
  mm/swapfile: use plist_for_each_entry in __folio_throttle_swaprate
  mm/swapfile: fix list iteration when next node is removed during discard
  fs/proc/task_mmu.c: fix make_uffd_wp_huge_pte() huge pte handling
  mm/kfence: add reboot notifier to disable KFENCE on shutdown
  memcg: remove inc/dec_lruvec_kmem_state helpers
  selftests/mm/uffd: initialize char variable to Null
  mm: fix DEBUG_RODATA_TEST indentation in Kconfig
  mm: introduce VMA flags bitmap type
  tools/testing/vma: eliminate dependency on vma->__vm_flags
  mm: simplify and rename mm flags function for clarity
  mm: declare VMA flags by bit
  zram: fix a spelling mistake
  mm/page_alloc: optimize lowmem_reserve max lookup using its semantic monotonicity
  mm/vmscan: skip increasing kswapd_failures when reclaim was boosted
  pagemap: update BUDDY flag documentation
  mm: swap: remove scan_swap_map_slots() references from comments
  mm: swap: change swap_alloc_slow() to void
  mm, swap: remove redundant comment for read_swap_cache_async
  mm, swap: use SWP_SOLIDSTATE to determine if swap is rotational
  ...
This commit is contained in:
Linus Torvalds 2025-12-05 13:52:43 -08:00
parent ac20755937 faf3c92352
commit 7203ca412f
228 changed files with 11636 additions and 5135 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.clang-format
View File

@ -140,8 +140,8 @@ ForEachMacros:
- 'damon_for_each_scheme_safe'
- 'damon_for_each_target'
- 'damon_for_each_target_safe'
- 'damos_for_each_filter'
- 'damos_for_each_filter_safe'
- 'damos_for_each_core_filter'
- 'damos_for_each_core_filter_safe'
- 'damos_for_each_ops_filter'
- 'damos_for_each_ops_filter_safe'
- 'damos_for_each_quota_goal'

13
Documentation/ABI/testing/sysfs-kernel-mm-damon
View File

@ -164,6 +164,13 @@ Description: Writing to and reading from this file sets and gets the pid of
the target process if the context is for virtual address spaces
monitoring, respectively.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/targets/<T>/obsolete_target
Date: Oct 2025
Contact: SeongJae Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
obsoleteness of the matching parameters commit destination
target.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/targets/<T>/regions/nr_regions
Date: Mar 2022
Contact: SeongJae Park <sj@kernel.org>
@ -303,6 +310,12 @@ Contact: SeongJae Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the nid
parameter of the goal.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/path
Date: Oct 2025
Contact: SeongJae Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the path
parameter of the goal.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/weights/sz_permil
Date: Mar 2022
Contact: SeongJae Park <sj@kernel.org>

4
Documentation/admin-guide/cgroup-v2.rst
View File

@ -1513,6 +1513,10 @@ The following nested keys are defined.
oom_group_kill
The number of times a group OOM has occurred.
sock_throttled
The number of times network sockets associated with
this cgroup are throttled.
memory.events.local
Similar to memory.events but the fields in the file are local
to the cgroup i.e. not hierarchical. The file modified event

22
Documentation/admin-guide/mm/damon/lru_sort.rst
View File

@ -211,6 +211,28 @@ End of target memory region in physical address.
The end physical address of memory region that DAMON_LRU_SORT will do work
against. By default, biggest System RAM is used as the region.
addr_unit
---------
A scale factor for memory addresses and bytes.
This parameter is for setting and getting the :ref:`address unit
<damon_design_addr_unit>` parameter of the DAMON instance for DAMON_RECLAIM.
``monitor_region_start`` and ``monitor_region_end`` should be provided in this
unit. For example, let's suppose ``addr_unit``, ``monitor_region_start`` and
``monitor_region_end`` are set as ``1024``, ``0`` and ``10``, respectively.
Then DAMON_LRU_SORT will work for 10 KiB length of physical address range that
starts from address zero (``[0 * 1024, 10 * 1024)`` in bytes).
Stat parameters having ``bytes_`` prefix are also in this unit. For example,
let's suppose values of ``addr_unit``, ``bytes_lru_sort_tried_hot_regions`` and
``bytes_lru_sorted_hot_regions`` are ``1024``, ``42``, and ``32``,
respectively. Then it means DAMON_LRU_SORT tried to LRU-sort 42 KiB of hot
memory and successfully LRU-sorted 32 KiB of the memory in total.
If unsure, use only the default value (``1``) and forget about this.
kdamond_pid
-----------

22
Documentation/admin-guide/mm/damon/reclaim.rst
View File

@ -232,6 +232,28 @@ The end physical address of memory region that DAMON_RECLAIM will do work
against. That is, DAMON_RECLAIM will find cold memory regions in this region
and reclaims. By default, biggest System RAM is used as the region.
addr_unit
---------
A scale factor for memory addresses and bytes.
This parameter is for setting and getting the :ref:`address unit
<damon_design_addr_unit>` parameter of the DAMON instance for DAMON_RECLAIM.
``monitor_region_start`` and ``monitor_region_end`` should be provided in this
unit. For example, let's suppose ``addr_unit``, ``monitor_region_start`` and
``monitor_region_end`` are set as ``1024``, ``0`` and ``10``, respectively.
Then DAMON_RECLAIM will work for 10 KiB length of physical address range that
starts from address zero (``[0 * 1024, 10 * 1024)`` in bytes).
``bytes_reclaim_tried_regions`` and ``bytes_reclaimed_regions`` are also in
this unit. For example, let's suppose values of ``addr_unit``,
``bytes_reclaim_tried_regions`` and ``bytes_reclaimed_regions`` are ``1024``,
``42``, and ``32``, respectively. Then it means DAMON_RECLAIM tried to reclaim
42 KiB memory and successfully reclaimed 32 KiB memory in total.
If unsure, use only the default value (``1``) and forget about this.
skip_anon
---------

35
Documentation/admin-guide/mm/damon/stat.rst
View File

@ -10,6 +10,8 @@ on the system's entire physical memory using DAMON, and provides simplified
access monitoring results statistics, namely idle time percentiles and
estimated memory bandwidth.
.. _damon_stat_monitoring_accuracy_overhead:
Monitoring Accuracy and Overhead
================================
@ -17,9 +19,11 @@ DAMON_STAT uses monitoring intervals :ref:`auto-tuning
<damon_design_monitoring_intervals_autotuning>` to make its accuracy high and
overhead minimum. It auto-tunes the intervals aiming 4 % of observable access
events to be captured in each snapshot, while limiting the resulting sampling
events to be 5 milliseconds in minimum and 10 seconds in maximum. On a few
interval to be 5 milliseconds in minimum and 10 seconds in maximum. On a few
production server systems, it resulted in consuming only 0.x % single CPU time,
while capturing reasonable quality of access patterns.
while capturing reasonable quality of access patterns. The tuning-resulting
intervals can be retrieved via ``aggr_interval_us`` :ref:`parameter
<damon_stat_aggr_interval_us>`.
Interface: Module Parameters
============================
@ -41,6 +45,18 @@ You can enable DAMON_STAT by setting the value of this parameter as ``Y``.
Setting it as ``N`` disables DAMON_STAT. The default value is set by
``CONFIG_DAMON_STAT_ENABLED_DEFAULT`` build config option.
.. _damon_stat_aggr_interval_us:
aggr_interval_us
----------------
Auto-tuned aggregation time interval in microseconds.
Users can read the aggregation interval of DAMON that is being used by the
DAMON instance for DAMON_STAT. It is :ref:`auto-tuned
<damon_stat_monitoring_accuracy_overhead>` and therefore the value is
dynamically changed.
estimated_memory_bandwidth
--------------------------
@ -58,12 +74,13 @@ memory_idle_ms_percentiles
Per-byte idle time (milliseconds) percentiles of the system.
DAMON_STAT calculates how long each byte of the memory was not accessed until
now (idle time), based on the current DAMON results snapshot. If DAMON found a
region of access frequency (nr_accesses) larger than zero, every byte of the
region gets zero idle time. If a region has zero access frequency
(nr_accesses), how long the region was keeping the zero access frequency (age)
becomes the idle time of every byte of the region. Then, DAMON_STAT exposes
the percentiles of the idle time values via this read-only parameter. Reading
the parameter returns 101 idle time values in milliseconds, separated by comma.
now (idle time), based on the current DAMON results snapshot. For regions
having access frequency (nr_accesses) larger than zero, how long the current
access frequency level was kept multiplied by ``-1`` becomes the idlee time of
every byte of the region. If a region has zero access frequency (nr_accesses),
how long the region was keeping the zero access frequency (age) becomes the
idle time of every byte of the region. Then, DAMON_STAT exposes the
percentiles of the idle time values via this read-only parameter. Reading the
parameter returns 101 idle time values in milliseconds, separated by comma.
Each value represents 0-th, 1st, 2nd, 3rd, ..., 99th and 100th percentile idle
times.

29
Documentation/admin-guide/mm/damon/usage.rst
View File

@ -67,7 +67,7 @@ comma (",").
│ │ │ │ │ │ │ intervals_goal/access_bp,aggrs,min_sample_us,max_sample_us
│ │ │ │ │ │ nr_regions/min,max
│ │ │ │ │ :ref:`targets <sysfs_targets>`/nr_targets
│ │ │ │ │ │ :ref:`0 <sysfs_target>`/pid_target
│ │ │ │ │ │ :ref:`0 <sysfs_target>`/pid_target,obsolete_target
│ │ │ │ │ │ │ :ref:`regions <sysfs_regions>`/nr_regions
│ │ │ │ │ │ │ │ :ref:`0 <sysfs_region>`/start,end
│ │ │ │ │ │ │ │ ...
@ -81,7 +81,7 @@ comma (",").
│ │ │ │ │ │ │ :ref:`quotas <sysfs_quotas>`/ms,bytes,reset_interval_ms,effective_bytes
│ │ │ │ │ │ │ │ weights/sz_permil,nr_accesses_permil,age_permil
│ │ │ │ │ │ │ │ :ref:`goals <sysfs_schemes_quota_goals>`/nr_goals
│ │ │ │ │ │ │ │ │ 0/target_metric,target_value,current_value,nid
│ │ │ │ │ │ │ │ │ 0/target_metric,target_value,current_value,nid,path
│ │ │ │ │ │ │ :ref:`watermarks <sysfs_watermarks>`/metric,interval_us,high,mid,low
│ │ │ │ │ │ │ :ref:`{core_,ops_,}filters <sysfs_filters>`/nr_filters
│ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,target_idx,min,max
@ -134,7 +134,8 @@ Users can write below commands for the kdamond to the ``state`` file.
- ``on``: Start running.
- ``off``: Stop running.
- ``commit``: Read the user inputs in the sysfs files except ``state`` file
again.
again. Monitoring :ref:`target region <sysfs_regions>` inputs are also be
ignored if no target region is specified.
- ``update_tuned_intervals``: Update the contents of ``sample_us`` and
``aggr_us`` files of the kdamond with the auto-tuning applied ``sampling
interval`` and ``aggregation interval`` for the files. Please refer to
@ -264,13 +265,20 @@ to ``N-1``. Each directory represents each monitoring target.
targets/<N>/
------------
In each target directory, one file (``pid_target``) and one directory
(``regions``) exist.
In each target directory, two files (``pid_target`` and ``obsolete_target``)
and one directory (``regions``) exist.
If you wrote ``vaddr`` to the ``contexts/<N>/operations``, each target should
be a process. You can specify the process to DAMON by writing the pid of the
process to the ``pid_target`` file.
Users can selectively remove targets in the middle of the targets array by
writing non-zero value to ``obsolete_target`` file and committing it (writing
``commit`` to ``state`` file). DAMON will remove the matching targets from its
internal targets array. Users are responsible to construct target directories
again, so that those correctly represent the changed internal targets array.
.. _sysfs_regions:
targets/<N>/regions
@ -289,6 +297,11 @@ In the beginning, this directory has only one file, ``nr_regions``. Writing a
number (``N``) to the file creates the number of child directories named ``0``
to ``N-1``. Each directory represents each initial monitoring target region.
If ``nr_regions`` is zero when committing new DAMON parameters online (writing
``commit`` to ``state`` file of :ref:`kdamond <sysfs_kdamond>`), the commit
logic ignores the target regions. In other words, the current monitoring
results for the target are preserved.
.. _sysfs_region:
regions/<N>/
@ -402,9 +415,9 @@ number (``N``) to the file creates the number of child directories named ``0``
to ``N-1``. Each directory represents each goal and current achievement.
Among the multiple feedback, the best one is used.
Each goal directory contains four files, namely ``target_metric``,
``target_value``, ``current_value`` and ``nid``. Users can set and get the
four parameters for the quota auto-tuning goals that specified on the
Each goal directory contains five files, namely ``target_metric``,
``target_value``, ``current_value`` ``nid`` and ``path``. Users can set and
get the five parameters for the quota auto-tuning goals that specified on the
:ref:`design doc <damon_design_damos_quotas_auto_tuning>` by writing to and
reading from each of the files. Note that users should further write
``commit_schemes_quota_goals`` to the ``state`` file of the :ref:`kdamond

1
Documentation/admin-guide/mm/index.rst
View File

@ -39,7 +39,6 @@ the Linux memory management.
shrinker_debugfs
slab
soft-dirty
swap_numa
transhuge
userfaultfd
zswap

3
Documentation/admin-guide/mm/pagemap.rst
View File

@ -115,7 +115,8 @@ Short descriptions to the page flags
A free memory block managed by the buddy system allocator.
The buddy system organizes free memory in blocks of various orders.
An order N block has 2^N physically contiguous pages, with the BUDDY flag
set for and _only_ for the first page.
set for all pages.
Before 4.6 only the first page of the block had the flag set.
15 - COMPOUND_HEAD
A compound page with order N consists of 2^N physically contiguous pages.
A compound page with order 2 takes the form of "HTTT", where H donates its

78
Documentation/admin-guide/mm/swap_numa.rst
View File

@ -1,78 +0,0 @@
===========================================
Automatically bind swap device to numa node
===========================================
If the system has more than one swap device and swap device has the node
information, we can make use of this information to decide which swap
device to use in get_swap_pages() to get better performance.
How to use this feature
=======================
Swap device has priority and that decides the order of it to be used. To make
use of automatically binding, there is no need to manipulate priority settings
for swap devices. e.g. on a 2 node machine, assume 2 swap devices swapA and
swapB, with swapA attached to node 0 and swapB attached to node 1, are going
to be swapped on. Simply swapping them on by doing::
# swapon /dev/swapA
# swapon /dev/swapB
Then node 0 will use the two swap devices in the order of swapA then swapB and
node 1 will use the two swap devices in the order of swapB then swapA. Note
that the order of them being swapped on doesn't matter.
A more complex example on a 4 node machine. Assume 6 swap devices are going to
be swapped on: swapA and swapB are attached to node 0, swapC is attached to
node 1, swapD and swapE are attached to node 2 and swapF is attached to node3.
The way to swap them on is the same as above::
# swapon /dev/swapA
# swapon /dev/swapB
# swapon /dev/swapC
# swapon /dev/swapD
# swapon /dev/swapE
# swapon /dev/swapF
Then node 0 will use them in the order of::
swapA/swapB -> swapC -> swapD -> swapE -> swapF
swapA and swapB will be used in a round robin mode before any other swap device.
node 1 will use them in the order of::
swapC -> swapA -> swapB -> swapD -> swapE -> swapF
node 2 will use them in the order of::
swapD/swapE -> swapA -> swapB -> swapC -> swapF
Similaly, swapD and swapE will be used in a round robin mode before any
other swap devices.
node 3 will use them in the order of::
swapF -> swapA -> swapB -> swapC -> swapD -> swapE
Implementation details
======================
The current code uses a priority based list, swap_avail_list, to decide
which swap device to use and if multiple swap devices share the same
priority, they are used round robin. This change here replaces the single
global swap_avail_list with a per-numa-node list, i.e. for each numa node,
it sees its own priority based list of available swap devices. Swap
device's priority can be promoted on its matching node's swap_avail_list.
The current swap device's priority is set as: user can set a >=0 value,
or the system will pick one starting from -1 then downwards. The priority
value in the swap_avail_list is the negated value of the swap device's
due to plist being sorted from low to high. The new policy doesn't change
the semantics for priority >=0 cases, the previous starting from -1 then
downwards now becomes starting from -2 then downwards and -1 is reserved
as the promoted value. So if multiple swap devices are attached to the same
node, they will all be promoted to priority -1 on that node's plist and will
be used round robin before any other swap devices.

5
Documentation/admin-guide/mm/transhuge.rst
View File

@ -381,6 +381,11 @@ hugepage allocation policy for the tmpfs mount by using the kernel parameter
four valid policies for tmpfs (``always``, ``within_size``, ``advise``,
``never``). The tmpfs mount default policy is ``never``.
Additionally, Kconfig options are available to set the default hugepage
policies for shmem (``CONFIG_TRANSPARENT_HUGEPAGE_SHMEM_HUGE_*``) and tmpfs
(``CONFIG_TRANSPARENT_HUGEPAGE_TMPFS_HUGE_*``) at build time. Refer to the
Kconfig help for more details.
In the same manner as ``thp_anon`` controls each supported anonymous THP
size, ``thp_shmem`` controls each supported shmem THP size. ``thp_shmem``
has the same format as ``thp_anon``, but also supports the policy

10
Documentation/admin-guide/mm/zswap.rst
View File

@ -59,11 +59,11 @@ returned by the allocation routine and that handle must be mapped before being
accessed. The compressed memory pool grows on demand and shrinks as compressed
pages are freed. The pool is not preallocated.
When a swap page is passed from swapout to zswap, zswap maintains a mapping
of the swap entry, a combination of the swap type and swap offset, to the
zsmalloc handle that references that compressed swap page. This mapping is
achieved with a red-black tree per swap type. The swap offset is the search
key for the tree nodes.
When a swap page is passed from swapout to zswap, zswap maintains a mapping of
the swap entry, a combination of the swap type and swap offset, to the zsmalloc
handle that references that compressed swap page. This mapping is achieved
with an xarray per swap type. The swap offset is the search key for the xarray
nodes.
During a page fault on a PTE that is a swap entry, the swapin code calls the
zswap load function to decompress the page into the page allocated by the page

6
Documentation/devicetree/bindings/riscv/extensions.yaml
View File

@ -217,6 +217,12 @@ properties:
memory types as ratified in the 20191213 version of the privileged
ISA specification.
- const: svrsw60t59b
description:
The Svrsw60t59b extension for providing two more bits[60:59] to
PTE/PMD entry as ratified at commit 28bde925e7a7 ("PTE Reserved
for SW bits 60:59") of riscv-non-isa/riscv-iommu.
- const: svvptc
description:
The standard Svvptc supervisor-level extension for

5
Documentation/filesystems/porting.rst
View File

@ -1286,6 +1286,11 @@ The vm_area_desc provides the minimum required information for a filesystem
to initialise state upon memory mapping of a file-backed region, and output
parameters for the file system to set this state.
In nearly all cases, this is all that is required for a filesystem. However, if
a filesystem needs to perform an operation such a pre-population of page tables,
then that action can be specified in the vm_area_desc->action field, which can
be configured using the mmap_action_*() helpers.
---
**mandatory**

5
Documentation/filesystems/proc.rst
View File

@ -553,7 +553,7 @@ otherwise.
kernel flags associated with the particular virtual memory area in two letter
encoded manner. The codes are the following:
== =======================================
== =============================================================
rd readable
wr writeable
ex executable
@ -591,7 +591,8 @@ encoded manner. The codes are the following:
sl sealed
lf lock on fault pages
dp always lazily freeable mapping
== =======================================
gu maybe contains guard regions (if not set, definitely doesn't)
== =============================================================
Note that there is no guarantee that every flag and associated mnemonic will
be present in all further kernel releases. Things get changed, the flags may

4
Documentation/filesystems/vfs.rst
View File

@ -1213,6 +1213,10 @@ otherwise noted.
file-backed memory mapping, most notably establishing relevant
private state and VMA callbacks.
If further action such as pre-population of page tables is required,
this can be specified by the vm_area_desc->action field and related
parameters.
Note that the file operations are implemented by the specific
filesystem in which the inode resides. When opening a device node
(character or block special) most filesystems will call special

23
Documentation/mm/damon/design.rst
View File

@ -381,8 +381,8 @@ That is, assumes 4% (20% of 20%) DAMON-observed access events ratio (source)
to capture 64% (80% multipled by 80%) real access events (outcomes).
To know how user-space can use this feature via :ref:`DAMON sysfs interface
<sysfs_interface>`, refer to :ref:`intervals_goal <sysfs_scheme>` part of
the documentation.
<sysfs_interface>`, refer to :ref:`intervals_goal
<damon_usage_sysfs_monitoring_intervals_goal>` part of the documentation.
.. _damon_design_damos:
@ -564,9 +564,9 @@ aggressiveness (the quota) of the corresponding scheme. For example, if DAMOS
is under achieving the goal, DAMOS automatically increases the quota. If DAMOS
is over achieving the goal, it decreases the quota.
The goal can be specified with four parameters, namely ``target_metric``,
``target_value``, ``current_value`` and ``nid``. The auto-tuning mechanism
tries to make ``current_value`` of ``target_metric`` be same to
The goal can be specified with five parameters, namely ``target_metric``,
``target_value``, ``current_value``, ``nid`` and ``path``. The auto-tuning
mechanism tries to make ``current_value`` of ``target_metric`` be same to
``target_value``.
- ``user_input``: User-provided value. Users could use any metric that they
@ -581,9 +581,18 @@ tries to make ``current_value`` of ``target_metric`` be same to
set by users at the initial time. In other words, DAMOS does self-feedback.
- ``node_mem_used_bp``: Specific NUMA node's used memory ratio in bp (1/10,000).
- ``node_mem_free_bp``: Specific NUMA node's free memory ratio in bp (1/10,000).
- ``node_memcg_used_bp``: Specific cgroup's node used memory ratio for a
specific NUMA node, in bp (1/10,000).
- ``node_memcg_free_bp``: Specific cgroup's node unused memory ratio for a
specific NUMA node, in bp (1/10,000).
``nid`` is optionally required for only ``node_mem_used_bp`` and
``node_mem_free_bp`` to point the specific NUMA node.
``nid`` is optionally required for only ``node_mem_used_bp``,
``node_mem_free_bp``, ``node_memcg_used_bp`` and ``node_memcg_free_bp`` to
point the specific NUMA node.
``path`` is optionally required for only ``node_memcg_used_bp`` and
``node_memcg_free_bp`` to point the path to the cgroup. The value should be
the path of the memory cgroup from the cgroups mount point.
To know how user-space can set the tuning goal metric, the target value, and/or
the current value via :ref:`DAMON sysfs interface <sysfs_interface>`, refer to

10
Documentation/mm/damon/maintainer-profile.rst
View File

@ -27,8 +27,8 @@ maintainer.
Note again the patches for `mm-new tree
<https://git.kernel.org/akpm/mm/h/mm-new>`_ are queued by the memory management
subsystem maintainer. If the patches requires some patches in `damon/next tree
<https://git.kernel.org/sj/h/damon/next>`_ which not yet merged in mm-new,
subsystem maintainer. If the patches require some patches in `damon/next tree
<https://git.kernel.org/sj/h/damon/next>`_ which have not yet merged in mm-new,
please make sure the requirement is clearly specified.
Submit checklist addendum
@ -57,7 +57,7 @@ Key cycle dates
Patches can be sent anytime. Key cycle dates of the `mm-new
<https://git.kernel.org/akpm/mm/h/mm-new>`_, `mm-unstable
<https://git.kernel.org/akpm/mm/h/mm-unstable>`_and `mm-stable
<https://git.kernel.org/akpm/mm/h/mm-unstable>`_ and `mm-stable
<https://git.kernel.org/akpm/mm/h/mm-stable>`_ trees depend on the memory
management subsystem maintainer.
@ -99,5 +99,5 @@ Schedules and reservation status are available at the Google `doc
<https://docs.google.com/document/d/1v43Kcj3ly4CYqmAkMaZzLiM2GEnWfgdGbZAH3mi2vpM/edit?usp=sharing>`_.
There is also a public Google `calendar
<https://calendar.google.com/calendar/u/0?cid=ZDIwOTA4YTMxNjc2MDQ3NTIyMmUzYTM5ZmQyM2U4NDA0ZGIwZjBiYmJlZGQxNDM0MmY4ZTRjOTE0NjdhZDRiY0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t>`_
that has the events. Anyone can subscribe it. DAMON maintainer will also
provide periodic reminder to the mailing list (damon@lists.linux.dev).
that has the events. Anyone can subscribe to it. DAMON maintainer will also
provide periodic reminders to the mailing list (damon@lists.linux.dev).

2
Documentation/mm/memory-model.rst
View File

@ -165,7 +165,7 @@ The users of `ZONE_DEVICE` are:
* pmem: Map platform persistent memory to be used as a direct-I/O target
via DAX mappings.
* hmm: Extend `ZONE_DEVICE` with `->page_fault()` and `->page_free()`
* hmm: Extend `ZONE_DEVICE` with `->page_fault()` and `->folio_free()`
event callbacks to allow a device-driver to coordinate memory management
events related to device-memory, typically GPU memory. See
Documentation/mm/hmm.rst.

32
Documentation/mm/page_owner.rst
View File

@ -27,7 +27,10 @@ enabled. Other usages are more than welcome.
It can also be used to show all the stacks and their current number of
allocated base pages, which gives us a quick overview of where the memory
is going without the need to screen through all the pages and match the
allocation and free operation.
allocation and free operation. It's also possible to show only a numeric
identifier of all the stacks (without stack traces) and their number of
allocated base pages (faster to read and parse, eg, for monitoring) that
can be matched with stacks later (show_handles and show_stacks_handles).
page owner is disabled by default. So, if you'd like to use it, you need
to add "page_owner=on" to your boot cmdline. If the kernel is built
@ -116,6 +119,33 @@ Usage
nr_base_pages: 20824
...
cat /sys/kernel/debug/page_owner_stacks/show_handles > handles_7000.txt
cat handles_7000.txt
handle: 42
nr_base_pages: 20824
...
cat /sys/kernel/debug/page_owner_stacks/show_stacks_handles > stacks_handles.txt
cat stacks_handles.txt
post_alloc_hook+0x177/0x1a0
get_page_from_freelist+0xd01/0xd80
__alloc_pages+0x39e/0x7e0
alloc_pages_mpol+0x22e/0x490
folio_alloc+0xd5/0x110
filemap_alloc_folio+0x78/0x230
page_cache_ra_order+0x287/0x6f0
filemap_get_pages+0x517/0x1160
filemap_read+0x304/0x9f0
xfs_file_buffered_read+0xe6/0x1d0 [xfs]
xfs_file_read_iter+0x1f0/0x380 [xfs]
__kernel_read+0x3b9/0x730
kernel_read_file+0x309/0x4d0
__do_sys_finit_module+0x381/0x730
do_syscall_64+0x8d/0x150
entry_SYSCALL_64_after_hwframe+0x62/0x6a
handle: 42
...
cat /sys/kernel/debug/page_owner > page_owner_full.txt
./page_owner_sort page_owner_full.txt sorted_page_owner.txt

9
Documentation/mm/process_addrs.rst
View File

@ -48,7 +48,8 @@ Terminology
* **VMA locks** - The VMA lock is at VMA granularity (of course) which behaves
as a read/write semaphore in practice. A VMA read lock is obtained via
:c:func:`!lock_vma_under_rcu` (and unlocked via :c:func:`!vma_end_read`) and a
write lock via :c:func:`!vma_start_write` (all VMA write locks are unlocked
write lock via vma_start_write() or vma_start_write_killable()
(all VMA write locks are unlocked
automatically when the mmap write lock is released). To take a VMA write lock
you **must** have already acquired an :c:func:`!mmap_write_lock`.
* **rmap locks** - When trying to access VMAs through the reverse mapping via a
@ -907,3 +908,9 @@ Stack expansion
Stack expansion throws up additional complexities in that we cannot permit there
to be racing page faults, as a result we invoke :c:func:`!vma_start_write` to
prevent this in :c:func:`!expand_downwards` or :c:func:`!expand_upwards`.
------------------------
Functions and structures
------------------------
.. kernel-doc:: include/linux/mmap_lock.h

6
MAINTAINERS
View File

@ -11604,6 +11604,8 @@ F: mm/hugetlb.c
F: mm/hugetlb_cgroup.c
F: mm/hugetlb_cma.c
F: mm/hugetlb_cma.h
F: mm/hugetlb_sysctl.c
F: mm/hugetlb_sysfs.c
F: mm/hugetlb_vmemmap.c
F: mm/hugetlb_vmemmap.h
F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c
@ -11621,6 +11623,8 @@ M: Miaohe Lin <linmiaohe@huawei.com>
R: Naoya Horiguchi <nao.horiguchi@gmail.com>
L: linux-mm@kvack.org
S: Maintained
F: include/linux/memory-failure.h
F: include/trace/events/memory-failure.h
F: mm/hwpoison-inject.c
F: mm/memory-failure.c
@ -16346,6 +16350,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: include/linux/gfp.h
F: include/linux/gfp_types.h
F: include/linux/highmem.h
F: include/linux/leafops.h
F: include/linux/memory.h
F: include/linux/mm.h
F: include/linux/mm_*.h
@ -16353,6 +16358,7 @@ F: include/linux/mmzone.h
F: include/linux/mmdebug.h
F: include/linux/mmu_notifier.h
F: include/linux/pagewalk.h
F: include/linux/pgalloc.h
F: include/linux/pgtable.h
F: include/linux/ptdump.h
F: include/linux/vmpressure.h

2
arch/arm64/mm/mmu.c
View File

@ -49,8 +49,6 @@
#define NO_CONT_MAPPINGS BIT(1)
#define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */
#define INVALID_PHYS_ADDR (-1ULL)
DEFINE_STATIC_KEY_FALSE(arm64_ptdump_lock_key);
u64 kimage_voffset __ro_after_init;

3
arch/csky/include/asm/pgtable.h
View File

@ -263,7 +263,4 @@ void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma,
#define update_mmu_cache(vma, addr, ptep) \
update_mmu_cache_range(NULL, vma, addr, ptep, 1)
#define io_remap_pfn_range(vma, vaddr, pfn, size, prot) \
remap_pfn_range(vma, vaddr, pfn, size, prot)
#endif /* __ASM_CSKY_PGTABLE_H */

9
arch/mips/alchemy/common/setup.c
View File

@ -94,12 +94,13 @@ phys_addr_t fixup_bigphys_addr(phys_addr_t phys_addr, phys_addr_t size)
return phys_addr;
}
int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long vaddr,
unsigned long pfn, unsigned long size, pgprot_t prot)
static inline unsigned long io_remap_pfn_range_pfn(unsigned long pfn,
unsigned long size)
{
phys_addr_t phys_addr = fixup_bigphys_addr(pfn << PAGE_SHIFT, size);
return remap_pfn_range(vma, vaddr, phys_addr >> PAGE_SHIFT, size, prot);
return phys_addr >> PAGE_SHIFT;
}
EXPORT_SYMBOL(io_remap_pfn_range);
EXPORT_SYMBOL(io_remap_pfn_range_pfn);
#endif /* CONFIG_MIPS_FIXUP_BIGPHYS_ADDR */

5
arch/mips/include/asm/pgtable.h
View File

@ -604,9 +604,8 @@ static inline void update_mmu_cache_pmd(struct vm_area_struct *vma,
*/
#ifdef CONFIG_MIPS_FIXUP_BIGPHYS_ADDR
phys_addr_t fixup_bigphys_addr(phys_addr_t addr, phys_addr_t size);
int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long vaddr,
unsigned long pfn, unsigned long size, pgprot_t prot);
#define io_remap_pfn_range io_remap_pfn_range
unsigned long io_remap_pfn_range_pfn(unsigned long pfn, unsigned long size);
#define io_remap_pfn_range_pfn io_remap_pfn_range_pfn
#else
#define fixup_bigphys_addr(addr, size) (addr)
#endif /* CONFIG_MIPS_FIXUP_BIGPHYS_ADDR */

7
arch/powerpc/kvm/book3s_hv_uvmem.c
View File

@ -723,7 +723,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm)
dpage = pfn_to_page(uvmem_pfn);
dpage->zone_device_data = pvt;
zone_device_page_init(dpage);
zone_device_page_init(dpage, 0);
return dpage;
out_clear:
spin_lock(&kvmppc_uvmem_bitmap_lock);
@ -1014,8 +1014,9 @@ static vm_fault_t kvmppc_uvmem_migrate_to_ram(struct vm_fault *vmf)
* to a normal PFN during H_SVM_PAGE_OUT.
* Gets called with kvm->arch.uvmem_lock held.
*/
static void kvmppc_uvmem_page_free(struct page *page)
static void kvmppc_uvmem_folio_free(struct folio *folio)
{
struct page *page = &folio->page;
unsigned long pfn = page_to_pfn(page) -
(kvmppc_uvmem_pgmap.range.start >> PAGE_SHIFT);
struct kvmppc_uvmem_page_pvt *pvt;
@ -1034,7 +1035,7 @@ static void kvmppc_uvmem_page_free(struct page *page)
}
static const struct dev_pagemap_ops kvmppc_uvmem_ops = {
.page_free = kvmppc_uvmem_page_free,
.folio_free = kvmppc_uvmem_folio_free,
.migrate_to_ram = kvmppc_uvmem_migrate_to_ram,
};

2
arch/powerpc/platforms/pseries/pci_dlpar.c
View File

@ -29,7 +29,7 @@ struct pci_controller *init_phb_dynamic(struct device_node *dn)
nid = of_node_to_nid(dn);
if (likely((nid) >= 0)) {
if (!node_online(nid)) {
if (register_one_node(nid)) {
if (register_node(nid)) {
pr_err("PCI: Failed to register node %d\n", nid);
} else {
update_numa_distance(dn);

16
arch/riscv/Kconfig
View File

@ -142,11 +142,13 @@ config RISCV
select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET
select HAVE_ARCH_SECCOMP_FILTER
select HAVE_ARCH_SOFT_DIRTY if 64BIT && MMU && RISCV_ISA_SVRSW60T59B
select HAVE_ARCH_THREAD_STRUCT_WHITELIST
select HAVE_ARCH_TRACEHOOK
select HAVE_ARCH_TRANSPARENT_HUGEPAGE if 64BIT && MMU
select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if 64BIT && MMU
select HAVE_ARCH_USERFAULTFD_MINOR if 64BIT && USERFAULTFD
select HAVE_ARCH_USERFAULTFD_WP if 64BIT && MMU && USERFAULTFD && RISCV_ISA_SVRSW60T59B
select HAVE_ARCH_VMAP_STACK if MMU && 64BIT
select HAVE_ASM_MODVERSIONS
select HAVE_CONTEXT_TRACKING_USER
@ -849,6 +851,20 @@ config RISCV_ISA_ZICBOP
If you don't know what to do here, say Y.
config RISCV_ISA_SVRSW60T59B
bool "Svrsw60t59b extension support for using PTE bits 60 and 59"
depends on MMU && 64BIT
depends on RISCV_ALTERNATIVE
default y
help
Adds support to dynamically detect the presence of the Svrsw60t59b
extension and enable its usage.
The Svrsw60t59b extension allows to free the PTE reserved bits 60
and 59 for software to use.
If you don't know what to do here, say Y.
config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
def_bool y
# https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc

1
arch/riscv/include/asm/hwcap.h
View File

@ -106,6 +106,7 @@
#define RISCV_ISA_EXT_ZAAMO 97
#define RISCV_ISA_EXT_ZALRSC 98
#define RISCV_ISA_EXT_ZICBOP 99
#define RISCV_ISA_EXT_SVRSW60T59B 100
#define RISCV_ISA_EXT_XLINUXENVCFG 127

37
arch/riscv/include/asm/pgtable-bits.h
View File

@ -19,6 +19,43 @@
#define _PAGE_SOFT (3 << 8) /* Reserved for software */
#define _PAGE_SPECIAL (1 << 8) /* RSW: 0x1 */
#ifdef CONFIG_MEM_SOFT_DIRTY
/* ext_svrsw60t59b: bit 59 for soft-dirty tracking */
#define _PAGE_SOFT_DIRTY \
((riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)) ? \
(1UL << 59) : 0)
/*
* Bit 3 is always zero for swap entry computation, so we
* can borrow it for swap page soft-dirty tracking.
*/
#define _PAGE_SWP_SOFT_DIRTY \
((riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)) ? \
_PAGE_EXEC : 0)
#else
#define _PAGE_SOFT_DIRTY 0
#define _PAGE_SWP_SOFT_DIRTY 0
#endif /* CONFIG_MEM_SOFT_DIRTY */
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
/* ext_svrsw60t59b: Bit(60) for uffd-wp tracking */
#define _PAGE_UFFD_WP \
((riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)) ? \
(1UL << 60) : 0)
/*
* Bit 4 is not involved into swap entry computation, so we
* can borrow it for swap page uffd-wp tracking.
*/
#define _PAGE_SWP_UFFD_WP \
((riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)) ? \
_PAGE_USER : 0)
#else
#define _PAGE_UFFD_WP 0
#define _PAGE_SWP_UFFD_WP 0
#endif
#define _PAGE_TABLE _PAGE_PRESENT
/*

143
arch/riscv/include/asm/pgtable.h
View File

@ -417,6 +417,41 @@ static inline pte_t pte_wrprotect(pte_t pte)
return __pte(pte_val(pte) & ~(_PAGE_WRITE));
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
#define pgtable_supports_uffd_wp() \
riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)
static inline bool pte_uffd_wp(pte_t pte)
{
return !!(pte_val(pte) & _PAGE_UFFD_WP);
}
static inline pte_t pte_mkuffd_wp(pte_t pte)
{
return pte_wrprotect(__pte(pte_val(pte) | _PAGE_UFFD_WP));
}
static inline pte_t pte_clear_uffd_wp(pte_t pte)
{
return __pte(pte_val(pte) & ~(_PAGE_UFFD_WP));
}
static inline bool pte_swp_uffd_wp(pte_t pte)
{
return !!(pte_val(pte) & _PAGE_SWP_UFFD_WP);
}
static inline pte_t pte_swp_mkuffd_wp(pte_t pte)
{
return __pte(pte_val(pte) | _PAGE_SWP_UFFD_WP);
}
static inline pte_t pte_swp_clear_uffd_wp(pte_t pte)
{
return __pte(pte_val(pte) & ~(_PAGE_SWP_UFFD_WP));
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
/* static inline pte_t pte_mkread(pte_t pte) */
static inline pte_t pte_mkwrite_novma(pte_t pte)
@ -428,7 +463,7 @@ static inline pte_t pte_mkwrite_novma(pte_t pte)
static inline pte_t pte_mkdirty(pte_t pte)
{
return __pte(pte_val(pte) | _PAGE_DIRTY);
return __pte(pte_val(pte) | _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
}
static inline pte_t pte_mkclean(pte_t pte)
@ -456,6 +491,42 @@ static inline pte_t pte_mkhuge(pte_t pte)
return pte;
}
#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
#define pgtable_supports_soft_dirty() \
(IS_ENABLED(CONFIG_MEM_SOFT_DIRTY) && \
riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B))
static inline bool pte_soft_dirty(pte_t pte)
{
return !!(pte_val(pte) & _PAGE_SOFT_DIRTY);
}
static inline pte_t pte_mksoft_dirty(pte_t pte)
{
return __pte(pte_val(pte) | _PAGE_SOFT_DIRTY);
}
static inline pte_t pte_clear_soft_dirty(pte_t pte)
{
return __pte(pte_val(pte) & ~(_PAGE_SOFT_DIRTY));
}
static inline bool pte_swp_soft_dirty(pte_t pte)
{
return !!(pte_val(pte) & _PAGE_SWP_SOFT_DIRTY);
}
static inline pte_t pte_swp_mksoft_dirty(pte_t pte)
{
return __pte(pte_val(pte) | _PAGE_SWP_SOFT_DIRTY);
}
static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
{
return __pte(pte_val(pte) & ~(_PAGE_SWP_SOFT_DIRTY));
}
#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */
#ifdef CONFIG_RISCV_ISA_SVNAPOT
#define pte_leaf_size(pte) (pte_napot(pte) ? \
napot_cont_size(napot_cont_order(pte)) :\
@ -805,6 +876,72 @@ static inline pud_t pud_mkspecial(pud_t pud)
}
#endif
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
static inline bool pmd_uffd_wp(pmd_t pmd)
{
return pte_uffd_wp(pmd_pte(pmd));
}
static inline pmd_t pmd_mkuffd_wp(pmd_t pmd)
{
return pte_pmd(pte_mkuffd_wp(pmd_pte(pmd)));
}
static inline pmd_t pmd_clear_uffd_wp(pmd_t pmd)
{
return pte_pmd(pte_clear_uffd_wp(pmd_pte(pmd)));
}
static inline bool pmd_swp_uffd_wp(pmd_t pmd)
{
return pte_swp_uffd_wp(pmd_pte(pmd));
}
static inline pmd_t pmd_swp_mkuffd_wp(pmd_t pmd)
{
return pte_pmd(pte_swp_mkuffd_wp(pmd_pte(pmd)));
}
static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd)
{
return pte_pmd(pte_swp_clear_uffd_wp(pmd_pte(pmd)));
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
static inline bool pmd_soft_dirty(pmd_t pmd)
{
return pte_soft_dirty(pmd_pte(pmd));
}
static inline pmd_t pmd_mksoft_dirty(pmd_t pmd)
{
return pte_pmd(pte_mksoft_dirty(pmd_pte(pmd)));
}
static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd)
{
return pte_pmd(pte_clear_soft_dirty(pmd_pte(pmd)));
}
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
static inline bool pmd_swp_soft_dirty(pmd_t pmd)
{
return pte_swp_soft_dirty(pmd_pte(pmd));
}
static inline pmd_t pmd_swp_mksoft_dirty(pmd_t pmd)
{
return pte_pmd(pte_swp_mksoft_dirty(pmd_pte(pmd)));
}
static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
{
return pte_pmd(pte_swp_clear_soft_dirty(pmd_pte(pmd)));
}
#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */
#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */
static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,
pmd_t *pmdp, pmd_t pmd)
{
@ -1003,7 +1140,9 @@ static inline pud_t pud_modify(pud_t pud, pgprot_t newprot)
*
* Format of swap PTE:
* bit 0: _PAGE_PRESENT (zero)
* bit 1 to 3: _PAGE_LEAF (zero)
* bit 1 to 2: (zero)
* bit 3: _PAGE_SWP_SOFT_DIRTY
* bit 4: _PAGE_SWP_UFFD_WP
* bit 5: _PAGE_PROT_NONE (zero)
* bit 6: exclusive marker
* bits 7 to 11: swap type

1
arch/riscv/kernel/cpufeature.c
View File

@ -539,6 +539,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = {
__RISCV_ISA_EXT_DATA(svinval, RISCV_ISA_EXT_SVINVAL),
__RISCV_ISA_EXT_DATA(svnapot, RISCV_ISA_EXT_SVNAPOT),
__RISCV_ISA_EXT_DATA(svpbmt, RISCV_ISA_EXT_SVPBMT),
__RISCV_ISA_EXT_DATA(svrsw60t59b, RISCV_ISA_EXT_SVRSW60T59B),
__RISCV_ISA_EXT_DATA(svvptc, RISCV_ISA_EXT_SVVPTC),
};

1
arch/s390/boot/vmem.c
View File

@ -16,7 +16,6 @@
#include "decompressor.h"
#include "boot.h"
#define INVALID_PHYS_ADDR (~(phys_addr_t)0)
struct ctlreg __bootdata_preserved(s390_invalid_asce);
#ifdef CONFIG_PROC_FS

5
arch/s390/mm/gmap.c
View File

@ -596,8 +596,9 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)
| _SEGMENT_ENTRY_GMAP_UC
| _SEGMENT_ENTRY;
} else
*table = pmd_val(*pmd) &
_SEGMENT_ENTRY_HARDWARE_BITS;
*table = (pmd_val(*pmd) &
_SEGMENT_ENTRY_HARDWARE_BITS)
| _SEGMENT_ENTRY;
}
} else if (*table & _SEGMENT_ENTRY_PROTECT &&
!(pmd_val(*pmd) & _SEGMENT_ENTRY_PROTECT)) {

20
arch/s390/mm/gmap_helpers.c
View File

@ -11,27 +11,27 @@
#include <linux/mm.h>
#include <linux/hugetlb.h>
#include <linux/swap.h>
#include <linux/swapops.h>
#include <linux/leafops.h>
#include <linux/pagewalk.h>
#include <linux/ksm.h>
#include <asm/gmap_helpers.h>
#include <asm/pgtable.h>
/**
* ptep_zap_swap_entry() - discard a swap entry.
* ptep_zap_softleaf_entry() - discard a software leaf entry.
* @mm: the mm
* @entry: the swap entry that needs to be zapped
* @entry: the software leaf entry that needs to be zapped
*
* Discards the given swap entry. If the swap entry was an actual swap
* entry (and not a migration entry, for example), the actual swapped
* Discards the given software leaf entry. If the leaf entry was an actual
* swap entry (and not a migration entry, for example), the actual swapped
* page is also discarded from swap.
*/
static void ptep_zap_swap_entry(struct mm_struct *mm, swp_entry_t entry)
static void ptep_zap_softleaf_entry(struct mm_struct *mm, softleaf_t entry)
{
if (!non_swap_entry(entry))
if (softleaf_is_swap(entry))
dec_mm_counter(mm, MM_SWAPENTS);
else if (is_migration_entry(entry))
dec_mm_counter(mm, mm_counter(pfn_swap_entry_folio(entry)));
else if (softleaf_is_migration(entry))
dec_mm_counter(mm, mm_counter(softleaf_to_folio(entry)));
free_swap_and_cache(entry);
}
@ -66,7 +66,7 @@ void gmap_helper_zap_one_page(struct mm_struct *mm, unsigned long vmaddr)
preempt_disable();
pgste = pgste_get_lock(ptep);
ptep_zap_swap_entry(mm, pte_to_swp_entry(*ptep));
ptep_zap_softleaf_entry(mm, softleaf_from_pte(*ptep));
pte_clear(mm, vmaddr, ptep);
pgste_set_unlock(ptep, pgste);

12
arch/s390/mm/pgtable.c
View File

@ -16,7 +16,7 @@
#include <linux/spinlock.h>
#include <linux/rcupdate.h>
#include <linux/slab.h>
#include <linux/swapops.h>
#include <linux/leafops.h>
#include <linux/sysctl.h>
#include <linux/ksm.h>
#include <linux/mman.h>
@ -673,12 +673,12 @@ void ptep_unshadow_pte(struct mm_struct *mm, unsigned long saddr, pte_t *ptep)
pgste_set_unlock(ptep, pgste);
}
static void ptep_zap_swap_entry(struct mm_struct *mm, swp_entry_t entry)
static void ptep_zap_softleaf_entry(struct mm_struct *mm, softleaf_t entry)
{
if (!non_swap_entry(entry))
if (softleaf_is_swap(entry))
dec_mm_counter(mm, MM_SWAPENTS);
else if (is_migration_entry(entry)) {
struct folio *folio = pfn_swap_entry_folio(entry);
else if (softleaf_is_migration(entry)) {
struct folio *folio = softleaf_to_folio(entry);
dec_mm_counter(mm, mm_counter(folio));
}
@ -700,7 +700,7 @@ void ptep_zap_unused(struct mm_struct *mm, unsigned long addr,
if (!reset && pte_swap(pte) &&
((pgstev & _PGSTE_GPS_USAGE_MASK) == _PGSTE_GPS_USAGE_UNUSED ||
(pgstev & _PGSTE_GPS_ZERO))) {
ptep_zap_swap_entry(mm, pte_to_swp_entry(pte));
ptep_zap_softleaf_entry(mm, softleaf_from_pte(pte));
pte_clear(mm, addr, ptep);
}
if (reset)

12
arch/sparc/include/asm/pgtable_32.h
View File

@ -395,12 +395,8 @@ __get_iospace (unsigned long addr)
#define GET_IOSPACE(pfn) (pfn >> (BITS_PER_LONG - 4))
#define GET_PFN(pfn) (pfn & 0x0fffffffUL)
int remap_pfn_range(struct vm_area_struct *, unsigned long, unsigned long,
unsigned long, pgprot_t);
static inline int io_remap_pfn_range(struct vm_area_struct *vma,
unsigned long from, unsigned long pfn,
unsigned long size, pgprot_t prot)
static inline unsigned long io_remap_pfn_range_pfn(unsigned long pfn,
unsigned long size)
{
unsigned long long offset, space, phys_base;
@ -408,9 +404,9 @@ static inline int io_remap_pfn_range(struct vm_area_struct *vma,
space = GET_IOSPACE(pfn);
phys_base = offset | (space << 32ULL);
return remap_pfn_range(vma, from, phys_base >> PAGE_SHIFT, size, prot);
return phys_base >> PAGE_SHIFT;
}
#define io_remap_pfn_range io_remap_pfn_range
#define io_remap_pfn_range_pfn io_remap_pfn_range_pfn
#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS
#define ptep_set_access_flags(__vma, __address, __ptep, __entry, __dirty) \

12
arch/sparc/include/asm/pgtable_64.h
View File

@ -1048,9 +1048,6 @@ int page_in_phys_avail(unsigned long paddr);
#define GET_IOSPACE(pfn) (pfn >> (BITS_PER_LONG - 4))
#define GET_PFN(pfn) (pfn & 0x0fffffffffffffffUL)
int remap_pfn_range(struct vm_area_struct *, unsigned long, unsigned long,
unsigned long, pgprot_t);
void adi_restore_tags(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long addr, pte_t pte);
@ -1084,9 +1081,8 @@ static inline int arch_unmap_one(struct mm_struct *mm,
return 0;
}
static inline int io_remap_pfn_range(struct vm_area_struct *vma,
unsigned long from, unsigned long pfn,
unsigned long size, pgprot_t prot)
static inline unsigned long io_remap_pfn_range_pfn(unsigned long pfn,
unsigned long size)
{
unsigned long offset = GET_PFN(pfn) << PAGE_SHIFT;
int space = GET_IOSPACE(pfn);
@ -1094,9 +1090,9 @@ static inline int io_remap_pfn_range(struct vm_area_struct *vma,
phys_base = offset | (((unsigned long) space) << 32UL);
return remap_pfn_range(vma, from, phys_base >> PAGE_SHIFT, size, prot);
return phys_base >> PAGE_SHIFT;
}
#define io_remap_pfn_range io_remap_pfn_range
#define io_remap_pfn_range_pfn io_remap_pfn_range_pfn
static inline unsigned long __untagged_addr(unsigned long start)
{

6
arch/sparc/kernel/sys_sparc_64.c
View File

@ -241,7 +241,7 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u
if (flags & MAP_FIXED) {
/* Ok, don't mess with it. */
return mm_get_unmapped_area(current->mm, NULL, orig_addr, len, pgoff, flags);
return mm_get_unmapped_area(NULL, orig_addr, len, pgoff, flags);
}
flags &= ~MAP_SHARED;
@ -254,7 +254,7 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u
align_goal = (64UL * 1024);
do {
addr = mm_get_unmapped_area(current->mm, NULL, orig_addr,
addr = mm_get_unmapped_area(NULL, orig_addr,
len + (align_goal - PAGE_SIZE), pgoff, flags);
if (!(addr & ~PAGE_MASK)) {
addr = (addr + (align_goal - 1UL)) & ~(align_goal - 1UL);
@ -273,7 +273,7 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u
* be obtained.
*/
if (addr & ~PAGE_MASK)
addr = mm_get_unmapped_area(current->mm, NULL, orig_addr, len, pgoff, flags);
addr = mm_get_unmapped_area(NULL, orig_addr, len, pgoff, flags);
return addr;
}

1
arch/x86/Kconfig
View File

@ -281,6 +281,7 @@ config X86
select HAVE_PCI
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select ASYNC_KERNEL_PGTABLE_FREE if IOMMU_SVA
select MMU_GATHER_RCU_TABLE_FREE
select MMU_GATHER_MERGE_VMAS
select HAVE_POSIX_CPU_TIMERS_TASK_WORK

2
arch/x86/kernel/cpu/sgx/driver.c
View File

@ -130,7 +130,7 @@ static unsigned long sgx_get_unmapped_area(struct file *file,
if (flags & MAP_FIXED)
return addr;
return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags);
return mm_get_unmapped_area(file, addr, len, pgoff, flags);
}
#ifdef CONFIG_COMPAT

2
arch/x86/mm/init_64.c
View File

@ -1028,7 +1028,7 @@ static void __meminit free_pagetable(struct page *page, int order)
free_reserved_pages(page, nr_pages);
#endif
} else {
__free_pages(page, order);
pagetable_free(page_ptdesc(page));
}
}

4
arch/x86/mm/numa.c
View File

@ -262,7 +262,7 @@ void __init init_gi_nodes(void)
* bringup_nonboot_cpus
* cpu_up
* __try_online_node
* register_one_node
* register_node
* because node_subsys is not initialized yet.
* TODO remove dependency on node_online
*/
@ -303,7 +303,7 @@ void __init init_cpu_to_node(void)
* bringup_nonboot_cpus
* cpu_up
* __try_online_node
* register_one_node
* register_node
* because node_subsys is not initialized yet.
* TODO remove dependency on node_online
*/

2
arch/x86/mm/pat/set_memory.c
View File

@ -429,7 +429,7 @@ static void cpa_collapse_large_pages(struct cpa_data *cpa)
list_for_each_entry_safe(ptdesc, tmp, &pgtables, pt_list) {
list_del(&ptdesc->pt_list);
__free_page(ptdesc_page(ptdesc));
pagetable_free(ptdesc);
}
}

12
arch/x86/mm/pgtable.c
View File

@ -729,7 +729,7 @@ int pmd_clear_huge(pmd_t *pmd)
int pud_free_pmd_page(pud_t *pud, unsigned long addr)
{
pmd_t *pmd, *pmd_sv;
pte_t *pte;
struct ptdesc *pt;
int i;
pmd = pud_pgtable(*pud);
@ -750,8 +750,8 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr)
for (i = 0; i < PTRS_PER_PMD; i++) {
if (!pmd_none(pmd_sv[i])) {
pte = (pte_t *)pmd_page_vaddr(pmd_sv[i]);
pte_free_kernel(&init_mm, pte);
pt = page_ptdesc(pmd_page(pmd_sv[i]));
pagetable_dtor_free(pt);
}
}
@ -772,15 +772,15 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr)
*/
int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
{
pte_t *pte;
struct ptdesc *pt;
pte = (pte_t *)pmd_page_vaddr(*pmd);
pt = page_ptdesc(pmd_page(*pmd));
pmd_clear(pmd);
/* INVLPG to clear all paging-structure caches */
flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1);
pte_free_kernel(&init_mm, pte);
pagetable_dtor_free(pt);
return 1;
}

6
drivers/acpi/apei/ghes.c
View File

@ -506,12 +506,6 @@ static bool ghes_do_memory_failure(u64 physical_addr, int flags)
return false;
pfn = PHYS_PFN(physical_addr);
if (!pfn_valid(pfn) && !arch_is_platform_page(physical_addr)) {
pr_warn_ratelimited(FW_WARN GHES_PFX
"Invalid address in generic error data: %#llx\n",
physical_addr);
return false;
}
if (flags == MF_ACTION_REQUIRED && current->mm) {
twcb = (void *)gen_pool_alloc(ghes_estatus_pool, sizeof(*twcb));

6
drivers/base/memory.c
View File

@ -198,15 +198,15 @@ static ssize_t state_show(struct device *dev, struct device_attribute *attr,
break;
default:
WARN_ON(1);
return sysfs_emit(buf, "ERROR-UNKNOWN-%ld\n", mem->state);
return sysfs_emit(buf, "ERROR-UNKNOWN-%d\n", mem->state);
}
return sysfs_emit(buf, "%s\n", output);
}
int memory_notify(unsigned long val, void *v)
int memory_notify(enum memory_block_state state, void *v)
{
return blocking_notifier_call_chain(&memory_chain, val, v);
return blocking_notifier_call_chain(&memory_chain, state, v);
}
#if defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_MEMORY_HOTPLUG)

90
drivers/base/node.c
View File

@ -676,50 +676,6 @@ static void node_device_release(struct device *dev)
kfree(to_node(dev));
}
/*
* register_node - Setup a sysfs device for a node.
* @num - Node number to use when creating the device.
*
* Initialize and register the node device.
*/
static int register_node(struct node *node, int num)
{
int error;
node->dev.id = num;
node->dev.bus = &node_subsys;
node->dev.release = node_device_release;
node->dev.groups = node_dev_groups;
error = device_register(&node->dev);
if (error) {
put_device(&node->dev);
} else {
hugetlb_register_node(node);
compaction_register_node(node);
reclaim_register_node(node);
}
return error;
}
/**
* unregister_node - unregister a node device
* @node: node going away
*
* Unregisters a node device @node. All the devices on the node must be
* unregistered before calling this function.
*/
void unregister_node(struct node *node)
{
hugetlb_unregister_node(node);
compaction_unregister_node(node);
reclaim_unregister_node(node);
node_remove_accesses(node);
node_remove_caches(node);
device_unregister(&node->dev);
}
struct node *node_devices[MAX_NUMNODES];
/*
@ -907,7 +863,13 @@ void register_memory_blocks_under_node_hotplug(int nid, unsigned long start_pfn,
}
#endif /* CONFIG_MEMORY_HOTPLUG */
int register_one_node(int nid)
/**
* register_node - Initialize and register the node device.
* @nid: Node number to use when creating the device.
*
* Return: 0 on success, -errno otherwise
*/
int register_node(int nid)
{
int error;
int cpu;
@ -918,14 +880,23 @@ int register_one_node(int nid)
return -ENOMEM;
INIT_LIST_HEAD(&node->access_list);
node_devices[nid] = node;
error = register_node(node_devices[nid], nid);
node->dev.id = nid;
node->dev.bus = &node_subsys;
node->dev.release = node_device_release;
node->dev.groups = node_dev_groups;
error = device_register(&node->dev);
if (error) {
node_devices[nid] = NULL;
put_device(&node->dev);
return error;
}
node_devices[nid] = node;
hugetlb_register_node(node);
compaction_register_node(node);
reclaim_register_node(node);
/* link cpu under this node */
for_each_present_cpu(cpu) {
if (cpu_to_node(cpu) == nid)
@ -936,13 +907,26 @@ int register_one_node(int nid)
return error;
}
void unregister_one_node(int nid)
/**
* unregister_node - unregister a node device
* @nid: nid of the node going away
*
* Unregisters the node device at node id @nid. All the devices on the
* node must be unregistered before calling this function.
*/
void unregister_node(int nid)
{
if (!node_devices[nid])
struct node *node = node_devices[nid];
if (!node)
return;
unregister_node(node_devices[nid]);
hugetlb_unregister_node(node);
compaction_unregister_node(node);
reclaim_unregister_node(node);
node_remove_accesses(node);
node_remove_caches(node);
device_unregister(&node->dev);
node_devices[nid] = NULL;
}
@ -1018,7 +1002,7 @@ void __init node_dev_init(void)
* to already created cpu devices.
*/
for_each_online_node(i) {
ret = register_one_node(i);
ret = register_node(i);
if (ret)
panic("%s() failed to add node: %d\n", __func__, ret);
}

487
drivers/block/zram/zram_drv.c
View File

@ -500,8 +500,31 @@ static ssize_t idle_store(struct device *dev,
}
#ifdef CONFIG_ZRAM_WRITEBACK
#define INVALID_BDEV_BLOCK (~0UL)
struct zram_wb_ctl {
/* idle list is accessed only by the writeback task, no concurency */
struct list_head idle_reqs;
/* done list is accessed concurrently, protect by done_lock */
struct list_head done_reqs;
wait_queue_head_t done_wait;
spinlock_t done_lock;
atomic_t num_inflight;
};
struct zram_wb_req {
unsigned long blk_idx;
struct page *page;
struct zram_pp_slot *pps;
struct bio_vec bio_vec;
struct bio bio;
struct list_head entry;
};
static ssize_t writeback_limit_enable_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
struct device_attribute *attr,
const char *buf, size_t len)
{
struct zram *zram = dev_to_zram(dev);
u64 val;
@ -510,33 +533,31 @@ static ssize_t writeback_limit_enable_store(struct device *dev,
if (kstrtoull(buf, 10, &val))
return ret;
down_read(&zram->init_lock);
spin_lock(&zram->wb_limit_lock);
down_write(&zram->init_lock);
zram->wb_limit_enable = val;
spin_unlock(&zram->wb_limit_lock);
up_read(&zram->init_lock);
up_write(&zram->init_lock);
ret = len;
return ret;
}
static ssize_t writeback_limit_enable_show(struct device *dev,
struct device_attribute *attr, char *buf)
struct device_attribute *attr,
char *buf)
{
bool val;
struct zram *zram = dev_to_zram(dev);
down_read(&zram->init_lock);
spin_lock(&zram->wb_limit_lock);
val = zram->wb_limit_enable;
spin_unlock(&zram->wb_limit_lock);
up_read(&zram->init_lock);
return sysfs_emit(buf, "%d\n", val);
}
static ssize_t writeback_limit_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
struct device_attribute *attr,
const char *buf, size_t len)
{
struct zram *zram = dev_to_zram(dev);
u64 val;
@ -545,31 +566,71 @@ static ssize_t writeback_limit_store(struct device *dev,
if (kstrtoull(buf, 10, &val))
return ret;
down_read(&zram->init_lock);
spin_lock(&zram->wb_limit_lock);
/*
* When the page size is greater than 4KB, if bd_wb_limit is set to
* a value that is not page - size aligned, it will cause value
* wrapping. For example, when the page size is set to 16KB and
* bd_wb_limit is set to 3, a single write - back operation will
* cause bd_wb_limit to become -1. Even more terrifying is that
* bd_wb_limit is an unsigned number.
*/
val = rounddown(val, PAGE_SIZE / 4096);
down_write(&zram->init_lock);
zram->bd_wb_limit = val;
spin_unlock(&zram->wb_limit_lock);
up_read(&zram->init_lock);
up_write(&zram->init_lock);
ret = len;
return ret;
}
static ssize_t writeback_limit_show(struct device *dev,
struct device_attribute *attr, char *buf)
struct device_attribute *attr, char *buf)
{
u64 val;
struct zram *zram = dev_to_zram(dev);
down_read(&zram->init_lock);
spin_lock(&zram->wb_limit_lock);
val = zram->bd_wb_limit;
spin_unlock(&zram->wb_limit_lock);
up_read(&zram->init_lock);
return sysfs_emit(buf, "%llu\n", val);
}
static ssize_t writeback_batch_size_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
struct zram *zram = dev_to_zram(dev);
u32 val;
if (kstrtouint(buf, 10, &val))
return -EINVAL;
if (!val)
return -EINVAL;
down_write(&zram->init_lock);
zram->wb_batch_size = val;
up_write(&zram->init_lock);
return len;
}
static ssize_t writeback_batch_size_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
u32 val;
struct zram *zram = dev_to_zram(dev);
down_read(&zram->init_lock);
val = zram->wb_batch_size;
up_read(&zram->init_lock);
return sysfs_emit(buf, "%u\n", val);
}
static void reset_bdev(struct zram *zram)
{
if (!zram->backing_dev)
@ -697,23 +758,20 @@ static ssize_t backing_dev_store(struct device *dev,
return err;
}
static unsigned long alloc_block_bdev(struct zram *zram)
static unsigned long zram_reserve_bdev_block(struct zram *zram)
{
unsigned long blk_idx = 1;
retry:
/* skip 0 bit to confuse zram.handle = 0 */
blk_idx = find_next_zero_bit(zram->bitmap, zram->nr_pages, blk_idx);
unsigned long blk_idx;
blk_idx = find_next_zero_bit(zram->bitmap, zram->nr_pages, 0);
if (blk_idx == zram->nr_pages)
return 0;
if (test_and_set_bit(blk_idx, zram->bitmap))
goto retry;
return INVALID_BDEV_BLOCK;
set_bit(blk_idx, zram->bitmap);
atomic64_inc(&zram->stats.bd_count);
return blk_idx;
}
static void free_block_bdev(struct zram *zram, unsigned long blk_idx)
static void zram_release_bdev_block(struct zram *zram, unsigned long blk_idx)
{
int was_set;
@ -734,63 +792,234 @@ static void read_from_bdev_async(struct zram *zram, struct page *page,
submit_bio(bio);
}
static int zram_writeback_slots(struct zram *zram, struct zram_pp_ctl *ctl)
static void release_wb_req(struct zram_wb_req *req)
{
unsigned long blk_idx = 0;
struct page *page = NULL;
struct zram_pp_slot *pps;
struct bio_vec bio_vec;
struct bio bio;
int ret = 0, err;
u32 index;
__free_page(req->page);
kfree(req);
}
page = alloc_page(GFP_KERNEL);
if (!page)
return -ENOMEM;
static void release_wb_ctl(struct zram_wb_ctl *wb_ctl)
{
if (!wb_ctl)
return;
/* We should never have inflight requests at this point */
WARN_ON(atomic_read(&wb_ctl->num_inflight));
WARN_ON(!list_empty(&wb_ctl->done_reqs));
while (!list_empty(&wb_ctl->idle_reqs)) {
struct zram_wb_req *req;
req = list_first_entry(&wb_ctl->idle_reqs,
struct zram_wb_req, entry);
list_del(&req->entry);
release_wb_req(req);
}
kfree(wb_ctl);
}
static struct zram_wb_ctl *init_wb_ctl(struct zram *zram)
{
struct zram_wb_ctl *wb_ctl;
int i;
wb_ctl = kmalloc(sizeof(*wb_ctl), GFP_KERNEL);
if (!wb_ctl)
return NULL;
INIT_LIST_HEAD(&wb_ctl->idle_reqs);
INIT_LIST_HEAD(&wb_ctl->done_reqs);
atomic_set(&wb_ctl->num_inflight, 0);
init_waitqueue_head(&wb_ctl->done_wait);
spin_lock_init(&wb_ctl->done_lock);
for (i = 0; i < zram->wb_batch_size; i++) {
struct zram_wb_req *req;
/*
* This is fatal condition only if we couldn't allocate
* any requests at all. Otherwise we just work with the
* requests that we have successfully allocated, so that
* writeback can still proceed, even if there is only one
* request on the idle list.
*/
req = kzalloc(sizeof(*req), GFP_KERNEL | __GFP_NOWARN);
if (!req)
break;
req->page = alloc_page(GFP_KERNEL | __GFP_NOWARN);
if (!req->page) {
kfree(req);
break;
}
list_add(&req->entry, &wb_ctl->idle_reqs);
}
/* We couldn't allocate any requests, so writeabck is not possible */
if (list_empty(&wb_ctl->idle_reqs))
goto release_wb_ctl;
return wb_ctl;
release_wb_ctl:
release_wb_ctl(wb_ctl);
return NULL;
}
static void zram_account_writeback_rollback(struct zram *zram)
{
lockdep_assert_held_read(&zram->init_lock);
if (zram->wb_limit_enable)
zram->bd_wb_limit += 1UL << (PAGE_SHIFT - 12);
}
static void zram_account_writeback_submit(struct zram *zram)
{
lockdep_assert_held_read(&zram->init_lock);
if (zram->wb_limit_enable && zram->bd_wb_limit > 0)
zram->bd_wb_limit -= 1UL << (PAGE_SHIFT - 12);
}
static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req)
{
u32 index = req->pps->index;
int err;
err = blk_status_to_errno(req->bio.bi_status);
if (err) {
/*
* Failed wb requests should not be accounted in wb_limit
* (if enabled).
*/
zram_account_writeback_rollback(zram);
zram_release_bdev_block(zram, req->blk_idx);
return err;
}
atomic64_inc(&zram->stats.bd_writes);
zram_slot_lock(zram, index);
/*
* We release slot lock during writeback so slot can change under us:
* slot_free() or slot_free() and zram_write_page(). In both cases
* slot loses ZRAM_PP_SLOT flag. No concurrent post-processing can
* set ZRAM_PP_SLOT on such slots until current post-processing
* finishes.
*/
if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) {
zram_release_bdev_block(zram, req->blk_idx);
goto out;
}
zram_free_page(zram, index);
zram_set_flag(zram, index, ZRAM_WB);
zram_set_handle(zram, index, req->blk_idx);
atomic64_inc(&zram->stats.pages_stored);
out:
zram_slot_unlock(zram, index);
return 0;
}
static void zram_writeback_endio(struct bio *bio)
{
struct zram_wb_req *req = container_of(bio, struct zram_wb_req, bio);
struct zram_wb_ctl *wb_ctl = bio->bi_private;
unsigned long flags;
spin_lock_irqsave(&wb_ctl->done_lock, flags);
list_add(&req->entry, &wb_ctl->done_reqs);
spin_unlock_irqrestore(&wb_ctl->done_lock, flags);
wake_up(&wb_ctl->done_wait);
}
static void zram_submit_wb_request(struct zram *zram,
struct zram_wb_ctl *wb_ctl,
struct zram_wb_req *req)
{
/*
* wb_limit (if enabled) should be adjusted before submission,
* so that we don't over-submit.
*/
zram_account_writeback_submit(zram);
atomic_inc(&wb_ctl->num_inflight);
req->bio.bi_private = wb_ctl;
submit_bio(&req->bio);
}
static int zram_complete_done_reqs(struct zram *zram,
struct zram_wb_ctl *wb_ctl)
{
struct zram_wb_req *req;
unsigned long flags;
int ret = 0, err;
while (atomic_read(&wb_ctl->num_inflight) > 0) {
spin_lock_irqsave(&wb_ctl->done_lock, flags);
req = list_first_entry_or_null(&wb_ctl->done_reqs,
struct zram_wb_req, entry);
if (req)
list_del(&req->entry);
spin_unlock_irqrestore(&wb_ctl->done_lock, flags);
/* ->num_inflight > 0 doesn't mean we have done requests */
if (!req)
break;
err = zram_writeback_complete(zram, req);
if (err)
ret = err;
atomic_dec(&wb_ctl->num_inflight);
release_pp_slot(zram, req->pps);
req->pps = NULL;
list_add(&req->entry, &wb_ctl->idle_reqs);
}
return ret;
}
static struct zram_wb_req *zram_select_idle_req(struct zram_wb_ctl *wb_ctl)
{
struct zram_wb_req *req;
req = list_first_entry_or_null(&wb_ctl->idle_reqs,
struct zram_wb_req, entry);
if (req)
list_del(&req->entry);
return req;
}
static int zram_writeback_slots(struct zram *zram,
struct zram_pp_ctl *ctl,
struct zram_wb_ctl *wb_ctl)
{
unsigned long blk_idx = INVALID_BDEV_BLOCK;
struct zram_wb_req *req = NULL;
struct zram_pp_slot *pps;
int ret = 0, err = 0;
u32 index = 0;
while ((pps = select_pp_slot(ctl))) {
spin_lock(&zram->wb_limit_lock);
if (zram->wb_limit_enable && !zram->bd_wb_limit) {
spin_unlock(&zram->wb_limit_lock);
ret = -EIO;
break;
}
spin_unlock(&zram->wb_limit_lock);
if (!blk_idx) {
blk_idx = alloc_block_bdev(zram);
if (!blk_idx) {
ret = -ENOSPC;
while (!req) {
req = zram_select_idle_req(wb_ctl);
if (req)
break;
}
}
index = pps->index;
zram_slot_lock(zram, index);
/*
* scan_slots() sets ZRAM_PP_SLOT and relases slot lock, so
* slots can change in the meantime. If slots are accessed or
* freed they lose ZRAM_PP_SLOT flag and hence we don't
* post-process them.
*/
if (!zram_test_flag(zram, index, ZRAM_PP_SLOT))
goto next;
if (zram_read_from_zspool(zram, page, index))
goto next;
zram_slot_unlock(zram, index);
wait_event(wb_ctl->done_wait,
!list_empty(&wb_ctl->done_reqs));
bio_init(&bio, zram->bdev, &bio_vec, 1,
REQ_OP_WRITE | REQ_SYNC);
bio.bi_iter.bi_sector = blk_idx * (PAGE_SIZE >> 9);
__bio_add_page(&bio, page, PAGE_SIZE, 0);
/*
* XXX: A single page IO would be inefficient for write
* but it would be not bad as starter.
*/
err = submit_bio_wait(&bio);
if (err) {
release_pp_slot(zram, pps);
err = zram_complete_done_reqs(zram, wb_ctl);
/*
* BIO errors are not fatal, we continue and simply
* attempt to writeback the remaining objects (pages).
@ -799,43 +1028,69 @@ static int zram_writeback_slots(struct zram *zram, struct zram_pp_ctl *ctl)
* them) were not successful and we do so by returning
* the most recent BIO error.
*/
ret = err;
continue;
if (err)
ret = err;
}
atomic64_inc(&zram->stats.bd_writes);
if (blk_idx == INVALID_BDEV_BLOCK) {
blk_idx = zram_reserve_bdev_block(zram);
if (blk_idx == INVALID_BDEV_BLOCK) {
ret = -ENOSPC;
break;
}
}
index = pps->index;
zram_slot_lock(zram, index);
/*
* Same as above, we release slot lock during writeback so
* slot can change under us: slot_free() or slot_free() and
* reallocation (zram_write_page()). In both cases slot loses
* ZRAM_PP_SLOT flag. No concurrent post-processing can set
* ZRAM_PP_SLOT on such slots until current post-processing
* finishes.
* scan_slots() sets ZRAM_PP_SLOT and releases slot lock, so
* slots can change in the meantime. If slots are accessed or
* freed they lose ZRAM_PP_SLOT flag and hence we don't
* post-process them.
*/
if (!zram_test_flag(zram, index, ZRAM_PP_SLOT))
goto next;
if (zram_read_from_zspool(zram, req->page, index))
goto next;
zram_slot_unlock(zram, index);
/*
* From now on pp-slot is owned by the req, remove it from
* its pp bucket.
*/
list_del_init(&pps->entry);
req->blk_idx = blk_idx;
req->pps = pps;
bio_init(&req->bio, zram->bdev, &req->bio_vec, 1, REQ_OP_WRITE);
req->bio.bi_iter.bi_sector = req->blk_idx * (PAGE_SIZE >> 9);
req->bio.bi_end_io = zram_writeback_endio;
__bio_add_page(&req->bio, req->page, PAGE_SIZE, 0);
zram_submit_wb_request(zram, wb_ctl, req);
blk_idx = INVALID_BDEV_BLOCK;
req = NULL;
cond_resched();
continue;
zram_free_page(zram, index);
zram_set_flag(zram, index, ZRAM_WB);
zram_set_handle(zram, index, blk_idx);
blk_idx = 0;
atomic64_inc(&zram->stats.pages_stored);
spin_lock(&zram->wb_limit_lock);
if (zram->wb_limit_enable && zram->bd_wb_limit > 0)
zram->bd_wb_limit -= 1UL << (PAGE_SHIFT - 12);
spin_unlock(&zram->wb_limit_lock);
next:
zram_slot_unlock(zram, index);
release_pp_slot(zram, pps);
cond_resched();
}
if (blk_idx)
free_block_bdev(zram, blk_idx);
if (page)
__free_page(page);
/*
* Selected idle req, but never submitted it due to some error or
* wb limit.
*/
if (req)
release_wb_req(req);
while (atomic_read(&wb_ctl->num_inflight) > 0) {
wait_event(wb_ctl->done_wait, !list_empty(&wb_ctl->done_reqs));
err = zram_complete_done_reqs(zram, wb_ctl);
if (err)
ret = err;
}
return ret;
}
@ -948,7 +1203,8 @@ static ssize_t writeback_store(struct device *dev,
struct zram *zram = dev_to_zram(dev);
u64 nr_pages = zram->disksize >> PAGE_SHIFT;
unsigned long lo = 0, hi = nr_pages;
struct zram_pp_ctl *ctl = NULL;
struct zram_pp_ctl *pp_ctl = NULL;
struct zram_wb_ctl *wb_ctl = NULL;
char *args, *param, *val;
ssize_t ret = len;
int err, mode = 0;
@ -970,8 +1226,14 @@ static ssize_t writeback_store(struct device *dev,
goto release_init_lock;
}
ctl = init_pp_ctl();
if (!ctl) {
pp_ctl = init_pp_ctl();
if (!pp_ctl) {
ret = -ENOMEM;
goto release_init_lock;
}
wb_ctl = init_wb_ctl(zram);
if (!wb_ctl) {
ret = -ENOMEM;
goto release_init_lock;
}
@ -1000,7 +1262,7 @@ static ssize_t writeback_store(struct device *dev,
goto release_init_lock;
}
scan_slots_for_writeback(zram, mode, lo, hi, ctl);
scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);
break;
}
@ -1011,7 +1273,7 @@ static ssize_t writeback_store(struct device *dev,
goto release_init_lock;
}
scan_slots_for_writeback(zram, mode, lo, hi, ctl);
scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);
break;
}
@ -1022,7 +1284,7 @@ static ssize_t writeback_store(struct device *dev,
goto release_init_lock;
}
scan_slots_for_writeback(zram, mode, lo, hi, ctl);
scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);
continue;
}
@ -1033,17 +1295,18 @@ static ssize_t writeback_store(struct device *dev,
goto release_init_lock;
}
scan_slots_for_writeback(zram, mode, lo, hi, ctl);
scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);
continue;
}
}
err = zram_writeback_slots(zram, ctl);
err = zram_writeback_slots(zram, pp_ctl, wb_ctl);
if (err)
ret = err;
release_init_lock:
release_pp_ctl(zram, ctl);
release_pp_ctl(zram, pp_ctl);
release_wb_ctl(wb_ctl);
atomic_set(&zram->pp_in_progress, 0);
up_read(&zram->init_lock);
@ -1112,7 +1375,9 @@ static int read_from_bdev(struct zram *zram, struct page *page,
return -EIO;
}
static void free_block_bdev(struct zram *zram, unsigned long blk_idx) {};
static void zram_release_bdev_block(struct zram *zram, unsigned long blk_idx)
{
}
#endif
#ifdef CONFIG_ZRAM_MEMORY_TRACKING
@ -1634,7 +1899,7 @@ static void zram_free_page(struct zram *zram, size_t index)
if (zram_test_flag(zram, index, ZRAM_WB)) {
zram_clear_flag(zram, index, ZRAM_WB);
free_block_bdev(zram, zram_get_handle(zram, index));
zram_release_bdev_block(zram, zram_get_handle(zram, index));
goto out;
}
@ -1740,14 +2005,14 @@ static int zram_read_page(struct zram *zram, struct page *page, u32 index,
ret = zram_read_from_zspool(zram, page, index);
zram_slot_unlock(zram, index);
} else {
unsigned long blk_idx = zram_get_handle(zram, index);
/*
* The slot should be unlocked before reading from the backing
* device.
*/
zram_slot_unlock(zram, index);
ret = read_from_bdev(zram, page, zram_get_handle(zram, index),
parent);
ret = read_from_bdev(zram, page, blk_idx, parent);
}
/* Should NEVER happen. Return bio error if it does. */
@ -2610,6 +2875,7 @@ static DEVICE_ATTR_RW(backing_dev);
static DEVICE_ATTR_WO(writeback);
static DEVICE_ATTR_RW(writeback_limit);
static DEVICE_ATTR_RW(writeback_limit_enable);
static DEVICE_ATTR_RW(writeback_batch_size);
#endif
#ifdef CONFIG_ZRAM_MULTI_COMP
static DEVICE_ATTR_RW(recomp_algorithm);
@ -2631,6 +2897,7 @@ static struct attribute *zram_disk_attrs[] = {
&dev_attr_writeback.attr,
&dev_attr_writeback_limit.attr,
&dev_attr_writeback_limit_enable.attr,
&dev_attr_writeback_batch_size.attr,
#endif
&dev_attr_io_stat.attr,
&dev_attr_mm_stat.attr,
@ -2692,7 +2959,7 @@ static int zram_add(void)
init_rwsem(&zram->init_lock);
#ifdef CONFIG_ZRAM_WRITEBACK
spin_lock_init(&zram->wb_limit_lock);
zram->wb_batch_size = 32;
#endif
/* gendisk structure */

2
drivers/block/zram/zram_drv.h
View File

@ -127,8 +127,8 @@ struct zram {
bool claim; /* Protected by disk->open_mutex */
#ifdef CONFIG_ZRAM_WRITEBACK
struct file *backing_dev;
spinlock_t wb_limit_lock;
bool wb_limit_enable;
u32 wb_batch_size;
u64 bd_wb_limit;
struct block_device *bdev;
unsigned long *bitmap;

86
drivers/char/mem.c
View File

@ -304,13 +304,13 @@ static unsigned zero_mmap_capabilities(struct file *file)
}
/* can't do an in-place private mapping if there's no MMU */
static inline int private_mapping_ok(struct vm_area_struct *vma)
static inline int private_mapping_ok(struct vm_area_desc *desc)
{
return is_nommu_shared_mapping(vma->vm_flags);
return is_nommu_shared_mapping(desc->vm_flags);
}
#else
static inline int private_mapping_ok(struct vm_area_struct *vma)
static inline int private_mapping_ok(struct vm_area_desc *desc)
{
return 1;
}
@ -322,46 +322,49 @@ static const struct vm_operations_struct mmap_mem_ops = {
#endif
};
static int mmap_mem(struct file *file, struct vm_area_struct *vma)
static int mmap_filter_error(int err)
{
size_t size = vma->vm_end - vma->vm_start;
phys_addr_t offset = (phys_addr_t)vma->vm_pgoff << PAGE_SHIFT;
return -EAGAIN;
}
static int mmap_mem_prepare(struct vm_area_desc *desc)
{
struct file *file = desc->file;
const size_t size = vma_desc_size(desc);
const phys_addr_t offset = (phys_addr_t)desc->pgoff << PAGE_SHIFT;
/* Does it even fit in phys_addr_t? */
if (offset >> PAGE_SHIFT != vma->vm_pgoff)
if (offset >> PAGE_SHIFT != desc->pgoff)
return -EINVAL;
/* It's illegal to wrap around the end of the physical address space. */
if (offset + (phys_addr_t)size - 1 < offset)
return -EINVAL;
if (!valid_mmap_phys_addr_range(vma->vm_pgoff, size))
if (!valid_mmap_phys_addr_range(desc->pgoff, size))
return -EINVAL;
if (!private_mapping_ok(vma))
if (!private_mapping_ok(desc))
return -ENOSYS;
if (!range_is_allowed(vma->vm_pgoff, size))
if (!range_is_allowed(desc->pgoff, size))
return -EPERM;
if (!phys_mem_access_prot_allowed(file, vma->vm_pgoff, size,
&vma->vm_page_prot))
if (!phys_mem_access_prot_allowed(file, desc->pgoff, size,
&desc->page_prot))
return -EINVAL;
vma->vm_page_prot = phys_mem_access_prot(file, vma->vm_pgoff,
size,
vma->vm_page_prot);
desc->page_prot = phys_mem_access_prot(file, desc->pgoff,
size,
desc->page_prot);
vma->vm_ops = &mmap_mem_ops;
desc->vm_ops = &mmap_mem_ops;
/* Remap-pfn-range will mark the range VM_IO. */
mmap_action_remap_full(desc, desc->pgoff);
/* We filter remap errors to -EAGAIN. */
desc->action.error_hook = mmap_filter_error;
/* Remap-pfn-range will mark the range VM_IO */
if (remap_pfn_range(vma,
vma->vm_start,
vma->vm_pgoff,
size,
vma->vm_page_prot)) {
return -EAGAIN;
}
return 0;
}
@ -501,14 +504,26 @@ static ssize_t read_zero(struct file *file, char __user *buf,
return cleared;
}
static int mmap_zero(struct file *file, struct vm_area_struct *vma)
static int mmap_zero_private_success(const struct vm_area_struct *vma)
{
/*
* This is a highly unique situation where we mark a MAP_PRIVATE mapping
* of /dev/zero anonymous, despite it not being.
*/
vma_set_anonymous((struct vm_area_struct *)vma);
return 0;
}
static int mmap_zero_prepare(struct vm_area_desc *desc)
{
#ifndef CONFIG_MMU
return -ENOSYS;
#endif
if (vma->vm_flags & VM_SHARED)
return shmem_zero_setup(vma);
vma_set_anonymous(vma);
if (desc->vm_flags & VM_SHARED)
return shmem_zero_setup_desc(desc);
desc->action.success_hook = mmap_zero_private_success;
return 0;
}
@ -526,10 +541,11 @@ static unsigned long get_unmapped_area_zero(struct file *file,
{
if (flags & MAP_SHARED) {
/*
* mmap_zero() will call shmem_zero_setup() to create a file,
* so use shmem's get_unmapped_area in case it can be huge;
* and pass NULL for file as in mmap.c's get_unmapped_area(),
* so as not to confuse shmem with our handle on "/dev/zero".
* mmap_zero_prepare() will call shmem_zero_setup() to create a
* file, so use shmem's get_unmapped_area in case it can be
* huge; and pass NULL for file as in mmap.c's
* get_unmapped_area(), so as not to confuse shmem with our
* handle on "/dev/zero".
*/
return shmem_get_unmapped_area(NULL, addr, len, pgoff, flags);
}
@ -542,7 +558,7 @@ static unsigned long get_unmapped_area_zero(struct file *file,
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
return thp_get_unmapped_area(file, addr, len, pgoff, flags);
#else
return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags);
return mm_get_unmapped_area(file, addr, len, pgoff, flags);
#endif
}
#endif /* CONFIG_MMU */
@ -632,7 +648,7 @@ static const struct file_operations __maybe_unused mem_fops = {
.llseek = memory_lseek,
.read = read_mem,
.write = write_mem,
.mmap = mmap_mem,
.mmap_prepare = mmap_mem_prepare,
.open = open_mem,
#ifndef CONFIG_MMU
.get_unmapped_area = get_unmapped_area_mem,
@ -668,7 +684,7 @@ static const struct file_operations zero_fops = {
.write_iter = write_iter_zero,
.splice_read = copy_splice_read,
.splice_write = splice_write_zero,
.mmap = mmap_zero,
.mmap_prepare = mmap_zero_prepare,
.get_unmapped_area = get_unmapped_area_zero,
#ifndef CONFIG_MMU
.mmap_capabilities = zero_mmap_capabilities,

37
drivers/dax/device.c
View File

@ -13,8 +13,9 @@
#include "dax-private.h"
#include "bus.h"
static int check_vma(struct dev_dax *dev_dax, struct vm_area_struct *vma,
const char *func)
static int __check_vma(struct dev_dax *dev_dax, vm_flags_t vm_flags,
unsigned long start, unsigned long end, struct file *file,
const char *func)
{
struct device *dev = &dev_dax->dev;
unsigned long mask;
@ -23,7 +24,7 @@ static int check_vma(struct dev_dax *dev_dax, struct vm_area_struct *vma,
return -ENXIO;
/* prevent private mappings from being established */
if ((vma->vm_flags & VM_MAYSHARE) != VM_MAYSHARE) {
if ((vm_flags & VM_MAYSHARE) != VM_MAYSHARE) {
dev_info_ratelimited(dev,
"%s: %s: fail, attempted private mapping\n",
current->comm, func);
@ -31,15 +32,15 @@ static int check_vma(struct dev_dax *dev_dax, struct vm_area_struct *vma,
}
mask = dev_dax->align - 1;
if (vma->vm_start & mask || vma->vm_end & mask) {
if (start & mask || end & mask) {
dev_info_ratelimited(dev,
"%s: %s: fail, unaligned vma (%#lx - %#lx, %#lx)\n",
current->comm, func, vma->vm_start, vma->vm_end,
current->comm, func, start, end,
mask);
return -EINVAL;
}
if (!vma_is_dax(vma)) {
if (!file_is_dax(file)) {
dev_info_ratelimited(dev,
"%s: %s: fail, vma is not DAX capable\n",
current->comm, func);
@ -49,6 +50,13 @@ static int check_vma(struct dev_dax *dev_dax, struct vm_area_struct *vma,
return 0;
}
static int check_vma(struct dev_dax *dev_dax, struct vm_area_struct *vma,
const char *func)
{
return __check_vma(dev_dax, vma->vm_flags, vma->vm_start, vma->vm_end,
vma->vm_file, func);
}
/* see "strong" declaration in tools/testing/nvdimm/dax-dev.c */
__weak phys_addr_t dax_pgoff_to_phys(struct dev_dax *dev_dax, pgoff_t pgoff,
unsigned long size)
@ -285,8 +293,9 @@ static const struct vm_operations_struct dax_vm_ops = {
.pagesize = dev_dax_pagesize,
};
static int dax_mmap(struct file *filp, struct vm_area_struct *vma)
static int dax_mmap_prepare(struct vm_area_desc *desc)
{
struct file *filp = desc->file;
struct dev_dax *dev_dax = filp->private_data;
int rc, id;
@ -297,13 +306,14 @@ static int dax_mmap(struct file *filp, struct vm_area_struct *vma)
* fault time.
*/
id = dax_read_lock();
rc = check_vma(dev_dax, vma, __func__);
rc = __check_vma(dev_dax, desc->vm_flags, desc->start, desc->end, filp,
__func__);
dax_read_unlock(id);
if (rc)
return rc;
vma->vm_ops = &dax_vm_ops;
vm_flags_set(vma, VM_HUGEPAGE);
desc->vm_ops = &dax_vm_ops;
desc->vm_flags |= VM_HUGEPAGE;
return 0;
}
@ -330,14 +340,13 @@ static unsigned long dax_get_unmapped_area(struct file *filp,
if ((off + len_align) < off)
goto out;
addr_align = mm_get_unmapped_area(current->mm, filp, addr, len_align,
pgoff, flags);
addr_align = mm_get_unmapped_area(filp, addr, len_align, pgoff, flags);
if (!IS_ERR_VALUE(addr_align)) {
addr_align += (off - addr_align) & (align - 1);
return addr_align;
}
out:
return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags);
return mm_get_unmapped_area(filp, addr, len, pgoff, flags);
}
static const struct address_space_operations dev_dax_aops = {
@ -377,7 +386,7 @@ static const struct file_operations dax_fops = {
.open = dax_open,
.release = dax_release,
.get_unmapped_area = dax_get_unmapped_area,
.mmap = dax_mmap,
.mmap_prepare = dax_mmap_prepare,
.fop_flags = FOP_MMAP_SYNC,
};

4
drivers/firmware/efi/arm-runtime.c
View File

@ -12,18 +12,18 @@
#include <linux/io.h>
#include <linux/memblock.h>
#include <linux/mm_types.h>
#include <linux/pgalloc.h>
#include <linux/pgtable.h>
#include <linux/preempt.h>
#include <linux/rbtree.h>
#include <linux/rwsem.h>
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/pgtable.h>
#include <asm/cacheflush.h>
#include <asm/efi.h>
#include <asm/mmu.h>
#include <asm/pgalloc.h>
#if defined(CONFIG_PTDUMP_DEBUGFS) || defined(CONFIG_ARM_PTDUMP_DEBUGFS)
#include <asm/ptdump.h>

4
drivers/firmware/efi/riscv-runtime.c
View File

@ -14,18 +14,18 @@
#include <linux/io.h>
#include <linux/memblock.h>
#include <linux/mm_types.h>
#include <linux/pgalloc.h>
#include <linux/pgtable.h>
#include <linux/preempt.h>
#include <linux/rbtree.h>
#include <linux/rwsem.h>
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/pgtable.h>
#include <asm/cacheflush.h>
#include <asm/efi.h>
#include <asm/mmu.h>
#include <asm/pgalloc.h>
static bool __init efi_virtmap_init(void)
{

7
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
View File

@ -217,7 +217,7 @@ svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn)
page = pfn_to_page(pfn);
svm_range_bo_ref(prange->svm_bo);
page->zone_device_data = prange->svm_bo;
zone_device_page_init(page);
zone_device_page_init(page, 0);
}
static void
@ -567,8 +567,9 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
return r < 0 ? r : 0;
}
static void svm_migrate_page_free(struct page *page)
static void svm_migrate_folio_free(struct folio *folio)
{
struct page *page = &folio->page;
struct svm_range_bo *svm_bo = page->zone_device_data;
if (svm_bo) {
@ -1008,7 +1009,7 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf)
}
static const struct dev_pagemap_ops svm_migrate_pgmap_ops = {
.page_free = svm_migrate_page_free,
.folio_free = svm_migrate_folio_free,
.migrate_to_ram = svm_migrate_to_ram,
};

12
drivers/gpu/drm/drm_pagemap.c
View File

@ -196,7 +196,7 @@ static void drm_pagemap_get_devmem_page(struct page *page,
struct drm_pagemap_zdd *zdd)
{
page->zone_device_data = drm_pagemap_zdd_get(zdd);
zone_device_page_init(page);
zone_device_page_init(page, 0);
}
/**
@ -752,15 +752,15 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
}
/**
* drm_pagemap_page_free() - Put GPU SVM zone device data associated with a page
* @page: Pointer to the page
* drm_pagemap_folio_free() - Put GPU SVM zone device data associated with a folio
* @folio: Pointer to the folio
*
* This function is a callback used to put the GPU SVM zone device data
* associated with a page when it is being released.
*/
static void drm_pagemap_page_free(struct page *page)
static void drm_pagemap_folio_free(struct folio *folio)
{
drm_pagemap_zdd_put(page->zone_device_data);
drm_pagemap_zdd_put(folio->page.zone_device_data);
}
/**
@ -788,7 +788,7 @@ static vm_fault_t drm_pagemap_migrate_to_ram(struct vm_fault *vmf)
}
static const struct dev_pagemap_ops drm_pagemap_pagemap_ops = {
.page_free = drm_pagemap_page_free,
.folio_free = drm_pagemap_folio_free,
.migrate_to_ram = drm_pagemap_migrate_to_ram,
};

311
drivers/gpu/drm/nouveau/nouveau_dmem.c
View File

@ -50,6 +50,7 @@
*/
#define DMEM_CHUNK_SIZE (2UL << 20)
#define DMEM_CHUNK_NPAGES (DMEM_CHUNK_SIZE >> PAGE_SHIFT)
#define NR_CHUNKS (128)
enum nouveau_aper {
NOUVEAU_APER_VIRT,
@ -83,9 +84,15 @@ struct nouveau_dmem {
struct list_head chunks;
struct mutex mutex;
struct page *free_pages;
struct folio *free_folios;
spinlock_t lock;
};
struct nouveau_dmem_dma_info {
dma_addr_t dma_addr;
size_t size;
};
static struct nouveau_dmem_chunk *nouveau_page_to_chunk(struct page *page)
{
return container_of(page_pgmap(page), struct nouveau_dmem_chunk,
@ -108,14 +115,20 @@ unsigned long nouveau_dmem_page_addr(struct page *page)
return chunk->bo->offset + off;
}
static void nouveau_dmem_page_free(struct page *page)
static void nouveau_dmem_folio_free(struct folio *folio)
{
struct page *page = &folio->page;
struct nouveau_dmem_chunk *chunk = nouveau_page_to_chunk(page);
struct nouveau_dmem *dmem = chunk->drm->dmem;
spin_lock(&dmem->lock);
page->zone_device_data = dmem->free_pages;
dmem->free_pages = page;
if (folio_order(folio)) {
page->zone_device_data = dmem->free_folios;
dmem->free_folios = folio;
} else {
page->zone_device_data = dmem->free_pages;
dmem->free_pages = page;
}
WARN_ON(!chunk->callocated);
chunk->callocated--;
@ -139,20 +152,28 @@ static void nouveau_dmem_fence_done(struct nouveau_fence **fence)
}
}
static int nouveau_dmem_copy_one(struct nouveau_drm *drm, struct page *spage,
struct page *dpage, dma_addr_t *dma_addr)
static int nouveau_dmem_copy_folio(struct nouveau_drm *drm,
struct folio *sfolio, struct folio *dfolio,
struct nouveau_dmem_dma_info *dma_info)
{
struct device *dev = drm->dev->dev;
struct page *dpage = folio_page(dfolio, 0);
struct page *spage = folio_page(sfolio, 0);
lock_page(dpage);
folio_lock(dfolio);
*dma_addr = dma_map_page(dev, dpage, 0, PAGE_SIZE, DMA_BIDIRECTIONAL);
if (dma_mapping_error(dev, *dma_addr))
dma_info->dma_addr = dma_map_page(dev, dpage, 0, page_size(dpage),
DMA_BIDIRECTIONAL);
dma_info->size = page_size(dpage);
if (dma_mapping_error(dev, dma_info->dma_addr))
return -EIO;
if (drm->dmem->migrate.copy_func(drm, 1, NOUVEAU_APER_HOST, *dma_addr,
NOUVEAU_APER_VRAM, nouveau_dmem_page_addr(spage))) {
dma_unmap_page(dev, *dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL);
if (drm->dmem->migrate.copy_func(drm, folio_nr_pages(sfolio),
NOUVEAU_APER_HOST, dma_info->dma_addr,
NOUVEAU_APER_VRAM,
nouveau_dmem_page_addr(spage))) {
dma_unmap_page(dev, dma_info->dma_addr, page_size(dpage),
DMA_BIDIRECTIONAL);
return -EIO;
}
@ -165,21 +186,48 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
struct nouveau_dmem *dmem = drm->dmem;
struct nouveau_fence *fence;
struct nouveau_svmm *svmm;
struct page *spage, *dpage;
unsigned long src = 0, dst = 0;
dma_addr_t dma_addr = 0;
struct page *dpage;
vm_fault_t ret = 0;
int err;
struct migrate_vma args = {
.vma = vmf->vma,
.start = vmf->address,
.end = vmf->address + PAGE_SIZE,
.src = &src,
.dst = &dst,
.pgmap_owner = drm->dev,
.fault_page = vmf->page,
.flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
.flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE |
MIGRATE_VMA_SELECT_COMPOUND,
.src = NULL,
.dst = NULL,
};
unsigned int order, nr;
struct folio *sfolio, *dfolio;
struct nouveau_dmem_dma_info dma_info;
sfolio = page_folio(vmf->page);
order = folio_order(sfolio);
nr = 1 << order;
/*
* Handle partial unmap faults, where the folio is large, but
* the pmd is split.
*/
if (vmf->pte) {
order = 0;
nr = 1;
}
if (order)
args.flags |= MIGRATE_VMA_SELECT_COMPOUND;
args.start = ALIGN_DOWN(vmf->address, (PAGE_SIZE << order));
args.vma = vmf->vma;
args.end = args.start + (PAGE_SIZE << order);
args.src = kcalloc(nr, sizeof(*args.src), GFP_KERNEL);
args.dst = kcalloc(nr, sizeof(*args.dst), GFP_KERNEL);
if (!args.src || !args.dst) {
ret = VM_FAULT_OOM;
goto err;
}
/*
* FIXME what we really want is to find some heuristic to migrate more
* than just one page on CPU fault. When such fault happens it is very
@ -190,22 +238,28 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
if (!args.cpages)
return 0;
spage = migrate_pfn_to_page(src);
if (!spage || !(src & MIGRATE_PFN_MIGRATE))
if (order)
dpage = folio_page(vma_alloc_folio(GFP_HIGHUSER | __GFP_ZERO,
order, vmf->vma, vmf->address), 0);
else
dpage = alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO, vmf->vma,
vmf->address);
if (!dpage) {
ret = VM_FAULT_OOM;
goto done;
}
dpage = alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO, vmf->vma, vmf->address);
if (!dpage)
goto done;
args.dst[0] = migrate_pfn(page_to_pfn(dpage));
if (order)
args.dst[0] |= MIGRATE_PFN_COMPOUND;
dfolio = page_folio(dpage);
dst = migrate_pfn(page_to_pfn(dpage));
svmm = spage->zone_device_data;
svmm = folio_zone_device_data(sfolio);
mutex_lock(&svmm->mutex);
nouveau_svmm_invalidate(svmm, args.start, args.end);
ret = nouveau_dmem_copy_one(drm, spage, dpage, &dma_addr);
err = nouveau_dmem_copy_folio(drm, sfolio, dfolio, &dma_info);
mutex_unlock(&svmm->mutex);
if (ret) {
if (err) {
ret = VM_FAULT_SIGBUS;
goto done;
}
@ -213,25 +267,40 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
nouveau_fence_new(&fence, dmem->migrate.chan);
migrate_vma_pages(&args);
nouveau_dmem_fence_done(&fence);
dma_unmap_page(drm->dev->dev, dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL);
dma_unmap_page(drm->dev->dev, dma_info.dma_addr, PAGE_SIZE,
DMA_BIDIRECTIONAL);
done:
migrate_vma_finalize(&args);
err:
kfree(args.src);
kfree(args.dst);
return ret;
}
static void nouveau_dmem_folio_split(struct folio *head, struct folio *tail)
{
if (tail == NULL)
return;
tail->pgmap = head->pgmap;
tail->mapping = head->mapping;
folio_set_zone_device_data(tail, folio_zone_device_data(head));
}
static const struct dev_pagemap_ops nouveau_dmem_pagemap_ops = {
.page_free = nouveau_dmem_page_free,
.folio_free = nouveau_dmem_folio_free,
.migrate_to_ram = nouveau_dmem_migrate_to_ram,
.folio_split = nouveau_dmem_folio_split,
};
static int
nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage)
nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage,
bool is_large)
{
struct nouveau_dmem_chunk *chunk;
struct resource *res;
struct page *page;
void *ptr;
unsigned long i, pfn_first;
unsigned long i, pfn_first, pfn;
int ret;
chunk = kzalloc(sizeof(*chunk), GFP_KERNEL);
@ -241,7 +310,7 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage)
}
/* Allocate unused physical address space for device private pages. */
res = request_free_mem_region(&iomem_resource, DMEM_CHUNK_SIZE,
res = request_free_mem_region(&iomem_resource, DMEM_CHUNK_SIZE * NR_CHUNKS,
"nouveau_dmem");
if (IS_ERR(res)) {
ret = PTR_ERR(res);
@ -274,16 +343,40 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage)
pfn_first = chunk->pagemap.range.start >> PAGE_SHIFT;
page = pfn_to_page(pfn_first);
spin_lock(&drm->dmem->lock);
for (i = 0; i < DMEM_CHUNK_NPAGES - 1; ++i, ++page) {
page->zone_device_data = drm->dmem->free_pages;
drm->dmem->free_pages = page;
pfn = pfn_first;
for (i = 0; i < NR_CHUNKS; i++) {
int j;
if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) || !is_large) {
for (j = 0; j < DMEM_CHUNK_NPAGES - 1; j++, pfn++) {
page = pfn_to_page(pfn);
page->zone_device_data = drm->dmem->free_pages;
drm->dmem->free_pages = page;
}
} else {
page = pfn_to_page(pfn);
page->zone_device_data = drm->dmem->free_folios;
drm->dmem->free_folios = page_folio(page);
pfn += DMEM_CHUNK_NPAGES;
}
}
*ppage = page;
/* Move to next page */
if (is_large) {
*ppage = &drm->dmem->free_folios->page;
drm->dmem->free_folios = (*ppage)->zone_device_data;
} else {
*ppage = drm->dmem->free_pages;
drm->dmem->free_pages = (*ppage)->zone_device_data;
}
chunk->callocated++;
spin_unlock(&drm->dmem->lock);
NV_INFO(drm, "DMEM: registered %ldMB of device memory\n",
DMEM_CHUNK_SIZE >> 20);
NV_INFO(drm, "DMEM: registered %ldMB of %sdevice memory %lx %lx\n",
NR_CHUNKS * DMEM_CHUNK_SIZE >> 20, is_large ? "THP " : "", pfn_first,
nouveau_dmem_page_addr(page));
return 0;
@ -298,27 +391,41 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage)
}
static struct page *
nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm)
nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm, bool is_large)
{
struct nouveau_dmem_chunk *chunk;
struct page *page = NULL;
struct folio *folio = NULL;
int ret;
unsigned int order = 0;
spin_lock(&drm->dmem->lock);
if (drm->dmem->free_pages) {
if (is_large && drm->dmem->free_folios) {
folio = drm->dmem->free_folios;
page = &folio->page;
drm->dmem->free_folios = page->zone_device_data;
chunk = nouveau_page_to_chunk(&folio->page);
chunk->callocated++;
spin_unlock(&drm->dmem->lock);
order = ilog2(DMEM_CHUNK_NPAGES);
} else if (!is_large && drm->dmem->free_pages) {
page = drm->dmem->free_pages;
drm->dmem->free_pages = page->zone_device_data;
chunk = nouveau_page_to_chunk(page);
chunk->callocated++;
spin_unlock(&drm->dmem->lock);
folio = page_folio(page);
} else {
spin_unlock(&drm->dmem->lock);
ret = nouveau_dmem_chunk_alloc(drm, &page);
ret = nouveau_dmem_chunk_alloc(drm, &page, is_large);
if (ret)
return NULL;
folio = page_folio(page);
if (is_large)
order = ilog2(DMEM_CHUNK_NPAGES);
}
zone_device_page_init(page);
zone_device_folio_init(folio, order);
return page;
}
@ -369,12 +476,12 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk)
{
unsigned long i, npages = range_len(&chunk->pagemap.range) >> PAGE_SHIFT;
unsigned long *src_pfns, *dst_pfns;
dma_addr_t *dma_addrs;
struct nouveau_dmem_dma_info *dma_info;
struct nouveau_fence *fence;
src_pfns = kvcalloc(npages, sizeof(*src_pfns), GFP_KERNEL | __GFP_NOFAIL);
dst_pfns = kvcalloc(npages, sizeof(*dst_pfns), GFP_KERNEL | __GFP_NOFAIL);
dma_addrs = kvcalloc(npages, sizeof(*dma_addrs), GFP_KERNEL | __GFP_NOFAIL);
dma_info = kvcalloc(npages, sizeof(*dma_info), GFP_KERNEL | __GFP_NOFAIL);
migrate_device_range(src_pfns, chunk->pagemap.range.start >> PAGE_SHIFT,
npages);
@ -382,17 +489,28 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk)
for (i = 0; i < npages; i++) {
if (src_pfns[i] & MIGRATE_PFN_MIGRATE) {
struct page *dpage;
struct folio *folio = page_folio(
migrate_pfn_to_page(src_pfns[i]));
unsigned int order = folio_order(folio);
if (src_pfns[i] & MIGRATE_PFN_COMPOUND) {
dpage = folio_page(
folio_alloc(
GFP_HIGHUSER_MOVABLE, order), 0);
} else {
/*
* _GFP_NOFAIL because the GPU is going away and there
* is nothing sensible we can do if we can't copy the
* data back.
*/
dpage = alloc_page(GFP_HIGHUSER | __GFP_NOFAIL);
}
/*
* _GFP_NOFAIL because the GPU is going away and there
* is nothing sensible we can do if we can't copy the
* data back.
*/
dpage = alloc_page(GFP_HIGHUSER | __GFP_NOFAIL);
dst_pfns[i] = migrate_pfn(page_to_pfn(dpage));
nouveau_dmem_copy_one(chunk->drm,
migrate_pfn_to_page(src_pfns[i]), dpage,
&dma_addrs[i]);
nouveau_dmem_copy_folio(chunk->drm,
page_folio(migrate_pfn_to_page(src_pfns[i])),
page_folio(dpage),
&dma_info[i]);
}
}
@ -403,8 +521,9 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk)
kvfree(src_pfns);
kvfree(dst_pfns);
for (i = 0; i < npages; i++)
dma_unmap_page(chunk->drm->dev->dev, dma_addrs[i], PAGE_SIZE, DMA_BIDIRECTIONAL);
kvfree(dma_addrs);
dma_unmap_page(chunk->drm->dev->dev, dma_info[i].dma_addr,
dma_info[i].size, DMA_BIDIRECTIONAL);
kvfree(dma_info);
}
void
@ -607,31 +726,36 @@ nouveau_dmem_init(struct nouveau_drm *drm)
static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm,
struct nouveau_svmm *svmm, unsigned long src,
dma_addr_t *dma_addr, u64 *pfn)
struct nouveau_dmem_dma_info *dma_info, u64 *pfn)
{
struct device *dev = drm->dev->dev;
struct page *dpage, *spage;
unsigned long paddr;
bool is_large = false;
unsigned long mpfn;
spage = migrate_pfn_to_page(src);
if (!(src & MIGRATE_PFN_MIGRATE))
goto out;
dpage = nouveau_dmem_page_alloc_locked(drm);
is_large = src & MIGRATE_PFN_COMPOUND;
dpage = nouveau_dmem_page_alloc_locked(drm, is_large);
if (!dpage)
goto out;
paddr = nouveau_dmem_page_addr(dpage);
if (spage) {
*dma_addr = dma_map_page(dev, spage, 0, page_size(spage),
dma_info->dma_addr = dma_map_page(dev, spage, 0, page_size(spage),
DMA_BIDIRECTIONAL);
if (dma_mapping_error(dev, *dma_addr))
dma_info->size = page_size(spage);
if (dma_mapping_error(dev, dma_info->dma_addr))
goto out_free_page;
if (drm->dmem->migrate.copy_func(drm, 1,
NOUVEAU_APER_VRAM, paddr, NOUVEAU_APER_HOST, *dma_addr))
if (drm->dmem->migrate.copy_func(drm, folio_nr_pages(page_folio(spage)),
NOUVEAU_APER_VRAM, paddr, NOUVEAU_APER_HOST,
dma_info->dma_addr))
goto out_dma_unmap;
} else {
*dma_addr = DMA_MAPPING_ERROR;
dma_info->dma_addr = DMA_MAPPING_ERROR;
if (drm->dmem->migrate.clear_func(drm, page_size(dpage),
NOUVEAU_APER_VRAM, paddr))
goto out_free_page;
@ -642,10 +766,13 @@ static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm,
((paddr >> PAGE_SHIFT) << NVIF_VMM_PFNMAP_V0_ADDR_SHIFT);
if (src & MIGRATE_PFN_WRITE)
*pfn |= NVIF_VMM_PFNMAP_V0_W;
return migrate_pfn(page_to_pfn(dpage));
mpfn = migrate_pfn(page_to_pfn(dpage));
if (folio_order(page_folio(dpage)))
mpfn |= MIGRATE_PFN_COMPOUND;
return mpfn;
out_dma_unmap:
dma_unmap_page(dev, *dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL);
dma_unmap_page(dev, dma_info->dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL);
out_free_page:
nouveau_dmem_page_free_locked(drm, dpage);
out:
@ -655,27 +782,38 @@ static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm,
static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm,
struct nouveau_svmm *svmm, struct migrate_vma *args,
dma_addr_t *dma_addrs, u64 *pfns)
struct nouveau_dmem_dma_info *dma_info, u64 *pfns)
{
struct nouveau_fence *fence;
unsigned long addr = args->start, nr_dma = 0, i;
unsigned long order = 0;
for (i = 0; addr < args->end; ) {
struct folio *folio;
for (i = 0; addr < args->end; i++) {
args->dst[i] = nouveau_dmem_migrate_copy_one(drm, svmm,
args->src[i], dma_addrs + nr_dma, pfns + i);
if (!dma_mapping_error(drm->dev->dev, dma_addrs[nr_dma]))
args->src[i], dma_info + nr_dma, pfns + i);
if (!args->dst[i]) {
i++;
addr += PAGE_SIZE;
continue;
}
if (!dma_mapping_error(drm->dev->dev, dma_info[nr_dma].dma_addr))
nr_dma++;
addr += PAGE_SIZE;
folio = page_folio(migrate_pfn_to_page(args->dst[i]));
order = folio_order(folio);
i += 1 << order;
addr += (1 << order) * PAGE_SIZE;
}
nouveau_fence_new(&fence, drm->dmem->migrate.chan);
migrate_vma_pages(args);
nouveau_dmem_fence_done(&fence);
nouveau_pfns_map(svmm, args->vma->vm_mm, args->start, pfns, i);
nouveau_pfns_map(svmm, args->vma->vm_mm, args->start, pfns, i, order);
while (nr_dma--) {
dma_unmap_page(drm->dev->dev, dma_addrs[nr_dma], PAGE_SIZE,
DMA_BIDIRECTIONAL);
dma_unmap_page(drm->dev->dev, dma_info[nr_dma].dma_addr,
dma_info[nr_dma].size, DMA_BIDIRECTIONAL);
}
migrate_vma_finalize(args);
}
@ -688,20 +826,27 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm,
unsigned long end)
{
unsigned long npages = (end - start) >> PAGE_SHIFT;
unsigned long max = min(SG_MAX_SINGLE_ALLOC, npages);
dma_addr_t *dma_addrs;
unsigned long max = npages;
struct migrate_vma args = {
.vma = vma,
.start = start,
.pgmap_owner = drm->dev,
.flags = MIGRATE_VMA_SELECT_SYSTEM,
.flags = MIGRATE_VMA_SELECT_SYSTEM
| MIGRATE_VMA_SELECT_COMPOUND,
};
unsigned long i;
u64 *pfns;
int ret = -ENOMEM;
struct nouveau_dmem_dma_info *dma_info;
if (drm->dmem == NULL)
return -ENODEV;
if (drm->dmem == NULL) {
ret = -ENODEV;
goto out;
}
if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
if (max > (unsigned long)HPAGE_PMD_NR)
max = (unsigned long)HPAGE_PMD_NR;
args.src = kcalloc(max, sizeof(*args.src), GFP_KERNEL);
if (!args.src)
@ -710,8 +855,8 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm,
if (!args.dst)
goto out_free_src;
dma_addrs = kmalloc_array(max, sizeof(*dma_addrs), GFP_KERNEL);
if (!dma_addrs)
dma_info = kmalloc_array(max, sizeof(*dma_info), GFP_KERNEL);
if (!dma_info)
goto out_free_dst;
pfns = nouveau_pfns_alloc(max);
@ -729,7 +874,7 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm,
goto out_free_pfns;
if (args.cpages)
nouveau_dmem_migrate_chunk(drm, svmm, &args, dma_addrs,
nouveau_dmem_migrate_chunk(drm, svmm, &args, dma_info,
pfns);
args.start = args.end;
}
@ -738,7 +883,7 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm,
out_free_pfns:
nouveau_pfns_free(pfns);
out_free_dma:
kfree(dma_addrs);
kfree(dma_info);
out_free_dst:
kfree(args.dst);
out_free_src:

6
drivers/gpu/drm/nouveau/nouveau_svm.c
View File

@ -921,12 +921,14 @@ nouveau_pfns_free(u64 *pfns)
void
nouveau_pfns_map(struct nouveau_svmm *svmm, struct mm_struct *mm,
unsigned long addr, u64 *pfns, unsigned long npages)
unsigned long addr, u64 *pfns, unsigned long npages,
unsigned int page_shift)
{
struct nouveau_pfnmap_args *args = nouveau_pfns_to_args(pfns);
args->p.addr = addr;
args->p.size = npages << PAGE_SHIFT;
args->p.size = npages << page_shift;
args->p.page = page_shift;
mutex_lock(&svmm->mutex);

3
drivers/gpu/drm/nouveau/nouveau_svm.h
View File

@ -33,7 +33,8 @@ void nouveau_svmm_invalidate(struct nouveau_svmm *svmm, u64 start, u64 limit);
u64 *nouveau_pfns_alloc(unsigned long npages);
void nouveau_pfns_free(u64 *pfns);
void nouveau_pfns_map(struct nouveau_svmm *svmm, struct mm_struct *mm,
unsigned long addr, u64 *pfns, unsigned long npages);
unsigned long addr, u64 *pfns, unsigned long npages,
unsigned int page_shift);
#else /* IS_ENABLED(CONFIG_DRM_NOUVEAU_SVM) */
static inline void nouveau_svm_init(struct nouveau_drm *drm) {}
static inline void nouveau_svm_fini(struct nouveau_drm *drm) {}

29
drivers/iommu/iommu-sva.c
View File

@ -10,6 +10,8 @@
#include "iommu-priv.h"
static DEFINE_MUTEX(iommu_sva_lock);
static bool iommu_sva_present;
static LIST_HEAD(iommu_sva_mms);
static struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
struct mm_struct *mm);
@ -42,6 +44,7 @@ static struct iommu_mm_data *iommu_alloc_mm_data(struct mm_struct *mm, struct de
return ERR_PTR(-ENOSPC);
}
iommu_mm->pasid = pasid;
iommu_mm->mm = mm;
INIT_LIST_HEAD(&iommu_mm->sva_domains);
/*
* Make sure the write to mm->iommu_mm is not reordered in front of
@ -132,8 +135,13 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
if (ret)
goto out_free_domain;
domain->users = 1;
list_add(&domain->next, &mm->iommu_mm->sva_domains);
if (list_empty(&iommu_mm->sva_domains)) {
if (list_empty(&iommu_sva_mms))
iommu_sva_present = true;
list_add(&iommu_mm->mm_list_elm, &iommu_sva_mms);
}
list_add(&domain->next, &iommu_mm->sva_domains);
out:
refcount_set(&handle->users, 1);
mutex_unlock(&iommu_sva_lock);
@ -175,6 +183,13 @@ void iommu_sva_unbind_device(struct iommu_sva *handle)
list_del(&domain->next);
iommu_domain_free(domain);
}
if (list_empty(&iommu_mm->sva_domains)) {
list_del(&iommu_mm->mm_list_elm);
if (list_empty(&iommu_sva_mms))
iommu_sva_present = false;
}
mutex_unlock(&iommu_sva_lock);
kfree(handle);
}
@ -312,3 +327,15 @@ static struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
return domain;
}
void iommu_sva_invalidate_kva_range(unsigned long start, unsigned long end)
{
struct iommu_mm_data *iommu_mm;
guard(mutex)(&iommu_sva_lock);
if (!iommu_sva_present)
return;
list_for_each_entry(iommu_mm, &iommu_sva_mms, mm_list_elm)
mmu_notifier_arch_invalidate_secondary_tlbs(iommu_mm->mm, start, end);
}

5
drivers/pci/p2pdma.c
View File

@ -200,8 +200,9 @@ static const struct attribute_group p2pmem_group = {
.name = "p2pmem",
};
static void p2pdma_page_free(struct page *page)
static void p2pdma_folio_free(struct folio *folio)
{
struct page *page = &folio->page;
struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(page_pgmap(page));
/* safe to dereference while a reference is held to the percpu ref */
struct pci_p2pdma *p2pdma = rcu_dereference_protected(
@ -214,7 +215,7 @@ static void p2pdma_page_free(struct page *page)
}
static const struct dev_pagemap_ops p2pdma_pgmap_ops = {
.page_free = p2pdma_page_free,
.folio_free = p2pdma_folio_free,
};
static void pci_p2pdma_release(void *data)

3
drivers/s390/char/sclp_sd.c
View File

@ -16,8 +16,7 @@
#include <linux/vmalloc.h>
#include <linux/async.h>
#include <linux/mutex.h>
#include <asm/pgalloc.h>
#include <linux/pgalloc.h>
#include "sclp.h"

2
drivers/vdpa/vdpa_user/iova_domain.h
View File

@ -17,8 +17,6 @@
#define IOVA_START_PFN 1
#define INVALID_PHYS_ADDR (~(phys_addr_t)0)
#define BOUNCE_MAP_SHIFT 12
#define BOUNCE_MAP_SIZE (1 << BOUNCE_MAP_SHIFT)
#define BOUNCE_MAP_MASK (~(BOUNCE_MAP_SIZE - 1))

2
fs/dax.c
View File

@ -24,7 +24,7 @@
#include <linux/mmu_notifier.h>
#include <linux/iomap.h>
#include <linux/rmap.h>
#include <asm/pgalloc.h>
#include <linux/pgalloc.h>
#define CREATE_TRACE_POINTS
#include <trace/events/fs_dax.h>

49
fs/hugetlbfs/inode.c
View File

@ -96,8 +96,15 @@ static const struct fs_parameter_spec hugetlb_fs_parameters[] = {
#define PGOFF_LOFFT_MAX \
(((1UL << (PAGE_SHIFT + 1)) - 1) << (BITS_PER_LONG - (PAGE_SHIFT + 1)))
static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
static int hugetlb_file_mmap_prepare_success(const struct vm_area_struct *vma)
{
/* Unfortunate we have to reassign vma->vm_private_data. */
return hugetlb_vma_lock_alloc((struct vm_area_struct *)vma);
}
static int hugetlbfs_file_mmap_prepare(struct vm_area_desc *desc)
{
struct file *file = desc->file;
struct inode *inode = file_inode(file);
loff_t len, vma_len;
int ret;
@ -112,8 +119,8 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
* way when do_mmap unwinds (may be important on powerpc
* and ia64).
*/
vm_flags_set(vma, VM_HUGETLB | VM_DONTEXPAND);
vma->vm_ops = &hugetlb_vm_ops;
desc->vm_flags |= VM_HUGETLB | VM_DONTEXPAND;
desc->vm_ops = &hugetlb_vm_ops;
/*
* page based offset in vm_pgoff could be sufficiently large to
@ -122,16 +129,16 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
* sizeof(unsigned long). So, only check in those instances.
*/
if (sizeof(unsigned long) == sizeof(loff_t)) {
if (vma->vm_pgoff & PGOFF_LOFFT_MAX)
if (desc->pgoff & PGOFF_LOFFT_MAX)
return -EINVAL;
}
/* must be huge page aligned */
if (vma->vm_pgoff & (~huge_page_mask(h) >> PAGE_SHIFT))
if (desc->pgoff & (~huge_page_mask(h) >> PAGE_SHIFT))
return -EINVAL;
vma_len = (loff_t)(vma->vm_end - vma->vm_start);
len = vma_len + ((loff_t)vma->vm_pgoff << PAGE_SHIFT);
vma_len = (loff_t)vma_desc_size(desc);
len = vma_len + ((loff_t)desc->pgoff << PAGE_SHIFT);
/* check for overflow */
if (len < vma_len)
return -EINVAL;
@ -141,7 +148,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
ret = -ENOMEM;
vm_flags = vma->vm_flags;
vm_flags = desc->vm_flags;
/*
* for SHM_HUGETLB, the pages are reserved in the shmget() call so skip
* reserving here. Note: only for SHM hugetlbfs file, the inode
@ -151,17 +158,30 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
vm_flags |= VM_NORESERVE;
if (hugetlb_reserve_pages(inode,
vma->vm_pgoff >> huge_page_order(h),
len >> huge_page_shift(h), vma,
vm_flags) < 0)
desc->pgoff >> huge_page_order(h),
len >> huge_page_shift(h), desc,
vm_flags) < 0)
goto out;
ret = 0;
if (vma->vm_flags & VM_WRITE && inode->i_size < len)
if ((desc->vm_flags & VM_WRITE) && inode->i_size < len)
i_size_write(inode, len);
out:
inode_unlock(inode);
if (!ret) {
/* Allocate the VMA lock after we set it up. */
desc->action.success_hook = hugetlb_file_mmap_prepare_success;
/*
* We cannot permit the rmap finding this VMA in the time
* between the VMA being inserted into the VMA tree and the
* completion/success hook being invoked.
*
* This is because we establish a per-VMA hugetlb lock which can
* be raced by rmap.
*/
desc->action.hide_from_rmap_until_complete = true;
}
return ret;
}
@ -184,8 +204,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
if (addr)
addr0 = ALIGN(addr, huge_page_size(h));
return mm_get_unmapped_area_vmflags(current->mm, file, addr0, len, pgoff,
flags, 0);
return mm_get_unmapped_area_vmflags(file, addr0, len, pgoff, flags, 0);
}
/*
@ -1221,7 +1240,7 @@ static void init_once(void *foo)
static const struct file_operations hugetlbfs_file_operations = {
.read_iter = hugetlbfs_read_iter,
.mmap = hugetlbfs_file_mmap,
.mmap_prepare = hugetlbfs_file_mmap_prepare,
.fsync = noop_fsync,
.get_unmapped_area = hugetlb_get_unmapped_area,
.llseek = default_llseek,

2
fs/ntfs3/file.c
View File

@ -379,7 +379,7 @@ static int ntfs_file_mmap_prepare(struct vm_area_desc *desc)
if (rw) {
u64 to = min_t(loff_t, i_size_read(inode),
from + desc->end - desc->start);
from + vma_desc_size(desc));
if (is_sparsed(ni)) {
/* Allocate clusters for rw map. */

2
fs/proc/inode.c
View File

@ -443,7 +443,7 @@ pde_get_unmapped_area(struct proc_dir_entry *pde, struct file *file, unsigned lo
return pde->proc_ops->proc_get_unmapped_area(file, orig_addr, len, pgoff, flags);
#ifdef CONFIG_MMU
return mm_get_unmapped_area(current->mm, file, orig_addr, len, pgoff, flags);
return mm_get_unmapped_area(file, orig_addr, len, pgoff, flags);
#endif
return orig_addr;

326
fs/proc/task_mmu.c
View File

@ -14,7 +14,7 @@
#include <linux/rmap.h>
#include <linux/swap.h>
#include <linux/sched/mm.h>
#include <linux/swapops.h>
#include <linux/leafops.h>
#include <linux/mmu_notifier.h>
#include <linux/page_idle.h>
#include <linux/shmem_fs.h>
@ -1017,14 +1017,16 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr,
young = pte_young(ptent);
dirty = pte_dirty(ptent);
present = true;
} else if (is_swap_pte(ptent)) {
swp_entry_t swpent = pte_to_swp_entry(ptent);
} else if (pte_none(ptent)) {
smaps_pte_hole_lookup(addr, walk);
} else {
const softleaf_t entry = softleaf_from_pte(ptent);
if (!non_swap_entry(swpent)) {
if (softleaf_is_swap(entry)) {
int mapcount;
mss->swap += PAGE_SIZE;
mapcount = swp_swapcount(swpent);
mapcount = swp_swapcount(entry);
if (mapcount >= 2) {
u64 pss_delta = (u64)PAGE_SIZE << PSS_SHIFT;
@ -1033,14 +1035,11 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr,
} else {
mss->swap_pss += (u64)PAGE_SIZE << PSS_SHIFT;
}
} else if (is_pfn_swap_entry(swpent)) {
if (is_device_private_entry(swpent))
} else if (softleaf_has_pfn(entry)) {
if (softleaf_is_device_private(entry))
present = true;
page = pfn_swap_entry_to_page(swpent);
page = softleaf_to_page(entry);
}
} else {
smaps_pte_hole_lookup(addr, walk);
return;
}
if (!page)
@ -1060,14 +1059,16 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
bool present = false;
struct folio *folio;
if (pmd_none(*pmd))
return;
if (pmd_present(*pmd)) {
page = vm_normal_page_pmd(vma, addr, *pmd);
present = true;
} else if (unlikely(thp_migration_supported() && is_swap_pmd(*pmd))) {
swp_entry_t entry = pmd_to_swp_entry(*pmd);
} else if (unlikely(thp_migration_supported())) {
const softleaf_t entry = softleaf_from_pmd(*pmd);
if (is_pfn_swap_entry(entry))
page = pfn_swap_entry_to_page(entry);
if (softleaf_has_pfn(entry))
page = softleaf_to_page(entry);
}
if (IS_ERR_OR_NULL(page))
return;
@ -1146,6 +1147,7 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma)
[ilog2(VM_MAYSHARE)] = "ms",
[ilog2(VM_GROWSDOWN)] = "gd",
[ilog2(VM_PFNMAP)] = "pf",
[ilog2(VM_MAYBE_GUARD)] = "gu",
[ilog2(VM_LOCKED)] = "lo",
[ilog2(VM_IO)] = "io",
[ilog2(VM_SEQ_READ)] = "sr",
@ -1181,10 +1183,10 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma)
[ilog2(VM_PKEY_BIT0)] = "",
[ilog2(VM_PKEY_BIT1)] = "",
[ilog2(VM_PKEY_BIT2)] = "",
#if VM_PKEY_BIT3
#if CONFIG_ARCH_PKEY_BITS > 3
[ilog2(VM_PKEY_BIT3)] = "",
#endif
#if VM_PKEY_BIT4
#if CONFIG_ARCH_PKEY_BITS > 4
[ilog2(VM_PKEY_BIT4)] = "",
#endif
#endif /* CONFIG_ARCH_HAS_PKEYS */
@ -1230,11 +1232,11 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
if (pte_present(ptent)) {
folio = page_folio(pte_page(ptent));
present = true;
} else if (is_swap_pte(ptent)) {
swp_entry_t swpent = pte_to_swp_entry(ptent);
} else {
const softleaf_t entry = softleaf_from_pte(ptent);
if (is_pfn_swap_entry(swpent))
folio = pfn_swap_entry_folio(swpent);
if (softleaf_has_pfn(entry))
folio = softleaf_to_folio(entry);
}
if (folio) {
@ -1582,8 +1584,6 @@ struct clear_refs_private {
enum clear_refs_types type;
};
#ifdef CONFIG_MEM_SOFT_DIRTY
static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, pte_t pte)
{
struct folio *folio;
@ -1603,6 +1603,8 @@ static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr,
static inline void clear_soft_dirty(struct vm_area_struct *vma,
unsigned long addr, pte_t *pte)
{
if (!pgtable_supports_soft_dirty())
return;
/*
* The soft-dirty tracker uses #PF-s to catch writes
* to pages, so write-protect the pte as well. See the
@ -1611,6 +1613,9 @@ static inline void clear_soft_dirty(struct vm_area_struct *vma,
*/
pte_t ptent = ptep_get(pte);
if (pte_none(ptent))
return;
if (pte_present(ptent)) {
pte_t old_pte;
@ -1620,24 +1625,21 @@ static inline void clear_soft_dirty(struct vm_area_struct *vma,
ptent = pte_wrprotect(old_pte);
ptent = pte_clear_soft_dirty(ptent);
ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
} else if (is_swap_pte(ptent)) {
} else {
ptent = pte_swp_clear_soft_dirty(ptent);
set_pte_at(vma->vm_mm, addr, pte, ptent);
}
}
#else
static inline void clear_soft_dirty(struct vm_area_struct *vma,
unsigned long addr, pte_t *pte)
{
}
#endif
#if defined(CONFIG_MEM_SOFT_DIRTY) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
#if defined(CONFIG_TRANSPARENT_HUGEPAGE)
static inline void clear_soft_dirty_pmd(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmdp)
{
pmd_t old, pmd = *pmdp;
if (!pgtable_supports_soft_dirty())
return;
if (pmd_present(pmd)) {
/* See comment in change_huge_pmd() */
old = pmdp_invalidate(vma, addr, pmdp);
@ -1650,7 +1652,7 @@ static inline void clear_soft_dirty_pmd(struct vm_area_struct *vma,
pmd = pmd_clear_soft_dirty(pmd);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
} else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
} else if (pmd_is_migration_entry(pmd)) {
pmd = pmd_swp_clear_soft_dirty(pmd);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
}
@ -1923,6 +1925,9 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm,
struct page *page = NULL;
struct folio *folio;
if (pte_none(pte))
goto out;
if (pte_present(pte)) {
if (pm->show_pfn)
frame = pte_pfn(pte);
@ -1932,32 +1937,34 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm,
flags |= PM_SOFT_DIRTY;
if (pte_uffd_wp(pte))
flags |= PM_UFFD_WP;
} else if (is_swap_pte(pte)) {
swp_entry_t entry;
} else {
softleaf_t entry;
if (pte_swp_soft_dirty(pte))
flags |= PM_SOFT_DIRTY;
if (pte_swp_uffd_wp(pte))
flags |= PM_UFFD_WP;
entry = pte_to_swp_entry(pte);
entry = softleaf_from_pte(pte);
if (pm->show_pfn) {
pgoff_t offset;
/*
* For PFN swap offsets, keeping the offset field
* to be PFN only to be compatible with old smaps.
*/
if (is_pfn_swap_entry(entry))
offset = swp_offset_pfn(entry);
if (softleaf_has_pfn(entry))
offset = softleaf_to_pfn(entry);
else
offset = swp_offset(entry);
frame = swp_type(entry) |
(offset << MAX_SWAPFILES_SHIFT);
}
flags |= PM_SWAP;
if (is_pfn_swap_entry(entry))
page = pfn_swap_entry_to_page(entry);
if (pte_marker_entry_uffd_wp(entry))
if (softleaf_has_pfn(entry))
page = softleaf_to_page(entry);
if (softleaf_is_uffd_wp_marker(entry))
flags |= PM_UFFD_WP;
if (is_guard_swp_entry(entry))
if (softleaf_is_guard_marker(entry))
flags |= PM_GUARD_REGION;
}
@ -1969,12 +1976,93 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm,
__folio_page_mapped_exclusively(folio, page))
flags |= PM_MMAP_EXCLUSIVE;
}
out:
if (vma->vm_flags & VM_SOFTDIRTY)
flags |= PM_SOFT_DIRTY;
return make_pme(frame, flags);
}
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr,
unsigned long end, struct vm_area_struct *vma,
struct pagemapread *pm)
{
unsigned int idx = (addr & ~PMD_MASK) >> PAGE_SHIFT;
u64 flags = 0, frame = 0;
pmd_t pmd = *pmdp;
struct page *page = NULL;
struct folio *folio = NULL;
int err = 0;
if (vma->vm_flags & VM_SOFTDIRTY)
flags |= PM_SOFT_DIRTY;
if (pmd_none(pmd))
goto populate_pagemap;
if (pmd_present(pmd)) {
page = pmd_page(pmd);
flags |= PM_PRESENT;
if (pmd_soft_dirty(pmd))
flags |= PM_SOFT_DIRTY;
if (pmd_uffd_wp(pmd))
flags |= PM_UFFD_WP;
if (pm->show_pfn)
frame = pmd_pfn(pmd) + idx;
} else if (thp_migration_supported()) {
const softleaf_t entry = softleaf_from_pmd(pmd);
unsigned long offset;
if (pm->show_pfn) {
if (softleaf_has_pfn(entry))
offset = softleaf_to_pfn(entry) + idx;
else
offset = swp_offset(entry) + idx;
frame = swp_type(entry) |
(offset << MAX_SWAPFILES_SHIFT);
}
flags |= PM_SWAP;
if (pmd_swp_soft_dirty(pmd))
flags |= PM_SOFT_DIRTY;
if (pmd_swp_uffd_wp(pmd))
flags |= PM_UFFD_WP;
VM_WARN_ON_ONCE(!pmd_is_migration_entry(pmd));
page = softleaf_to_page(entry);
}
if (page) {
folio = page_folio(page);
if (!folio_test_anon(folio))
flags |= PM_FILE;
}
populate_pagemap:
for (; addr != end; addr += PAGE_SIZE, idx++) {
u64 cur_flags = flags;
pagemap_entry_t pme;
if (folio && (flags & PM_PRESENT) &&
__folio_page_mapped_exclusively(folio, page))
cur_flags |= PM_MMAP_EXCLUSIVE;
pme = make_pme(frame, cur_flags);
err = add_to_pagemap(&pme, pm);
if (err)
break;
if (pm->show_pfn) {
if (flags & PM_PRESENT)
frame++;
else if (flags & PM_SWAP)
frame += (1 << MAX_SWAPFILES_SHIFT);
}
}
return err;
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,
struct mm_walk *walk)
{
@ -1983,82 +2071,15 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,
spinlock_t *ptl;
pte_t *pte, *orig_pte;
int err = 0;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
ptl = pmd_trans_huge_lock(pmdp, vma);
if (ptl) {
unsigned int idx = (addr & ~PMD_MASK) >> PAGE_SHIFT;
u64 flags = 0, frame = 0;
pmd_t pmd = *pmdp;
struct page *page = NULL;
struct folio *folio = NULL;
if (vma->vm_flags & VM_SOFTDIRTY)
flags |= PM_SOFT_DIRTY;
if (pmd_present(pmd)) {
page = pmd_page(pmd);
flags |= PM_PRESENT;
if (pmd_soft_dirty(pmd))
flags |= PM_SOFT_DIRTY;
if (pmd_uffd_wp(pmd))
flags |= PM_UFFD_WP;
if (pm->show_pfn)
frame = pmd_pfn(pmd) + idx;
}
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
else if (is_swap_pmd(pmd)) {
swp_entry_t entry = pmd_to_swp_entry(pmd);
unsigned long offset;
if (pm->show_pfn) {
if (is_pfn_swap_entry(entry))
offset = swp_offset_pfn(entry) + idx;
else
offset = swp_offset(entry) + idx;
frame = swp_type(entry) |
(offset << MAX_SWAPFILES_SHIFT);
}
flags |= PM_SWAP;
if (pmd_swp_soft_dirty(pmd))
flags |= PM_SOFT_DIRTY;
if (pmd_swp_uffd_wp(pmd))
flags |= PM_UFFD_WP;
VM_BUG_ON(!is_pmd_migration_entry(pmd));
page = pfn_swap_entry_to_page(entry);
}
#endif
if (page) {
folio = page_folio(page);
if (!folio_test_anon(folio))
flags |= PM_FILE;
}
for (; addr != end; addr += PAGE_SIZE, idx++) {
u64 cur_flags = flags;
pagemap_entry_t pme;
if (folio && (flags & PM_PRESENT) &&
__folio_page_mapped_exclusively(folio, page))
cur_flags |= PM_MMAP_EXCLUSIVE;
pme = make_pme(frame, cur_flags);
err = add_to_pagemap(&pme, pm);
if (err)
break;
if (pm->show_pfn) {
if (flags & PM_PRESENT)
frame++;
else if (flags & PM_SWAP)
frame += (1 << MAX_SWAPFILES_SHIFT);
}
}
err = pagemap_pmd_range_thp(pmdp, addr, end, vma, pm);
spin_unlock(ptl);
return err;
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
#endif
/*
* We can assume that @vma always points to a valid one and @end never
@ -2310,12 +2331,16 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
struct vm_area_struct *vma,
unsigned long addr, pte_t pte)
{
unsigned long categories = 0;
unsigned long categories;
if (pte_none(pte))
return 0;
if (pte_present(pte)) {
struct page *page;
categories |= PAGE_IS_PRESENT;
categories = PAGE_IS_PRESENT;
if (!pte_uffd_wp(pte))
categories |= PAGE_IS_WRITTEN;
@ -2329,19 +2354,20 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
categories |= PAGE_IS_PFNZERO;
if (pte_soft_dirty(pte))
categories |= PAGE_IS_SOFT_DIRTY;
} else if (is_swap_pte(pte)) {
swp_entry_t swp;
} else {
softleaf_t entry;
categories = PAGE_IS_SWAPPED;
categories |= PAGE_IS_SWAPPED;
if (!pte_swp_uffd_wp_any(pte))
categories |= PAGE_IS_WRITTEN;
swp = pte_to_swp_entry(pte);
if (is_guard_swp_entry(swp))
entry = softleaf_from_pte(pte);
if (softleaf_is_guard_marker(entry))
categories |= PAGE_IS_GUARD;
else if ((p->masks_of_interest & PAGE_IS_FILE) &&
is_pfn_swap_entry(swp) &&
!folio_test_anon(pfn_swap_entry_folio(swp)))
softleaf_has_pfn(entry) &&
!folio_test_anon(softleaf_to_folio(entry)))
categories |= PAGE_IS_FILE;
if (pte_swp_soft_dirty(pte))
@ -2360,12 +2386,12 @@ static void make_uffd_wp_pte(struct vm_area_struct *vma,
old_pte = ptep_modify_prot_start(vma, addr, pte);
ptent = pte_mkuffd_wp(old_pte);
ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
} else if (is_swap_pte(ptent)) {
ptent = pte_swp_mkuffd_wp(ptent);
set_pte_at(vma->vm_mm, addr, pte, ptent);
} else {
} else if (pte_none(ptent)) {
set_pte_at(vma->vm_mm, addr, pte,
make_pte_marker(PTE_MARKER_UFFD_WP));
} else {
ptent = pte_swp_mkuffd_wp(ptent);
set_pte_at(vma->vm_mm, addr, pte, ptent);
}
}
@ -2376,6 +2402,9 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
{
unsigned long categories = PAGE_IS_HUGE;
if (pmd_none(pmd))
return categories;
if (pmd_present(pmd)) {
struct page *page;
@ -2393,9 +2422,7 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
categories |= PAGE_IS_PFNZERO;
if (pmd_soft_dirty(pmd))
categories |= PAGE_IS_SOFT_DIRTY;
} else if (is_swap_pmd(pmd)) {
swp_entry_t swp;
} else {
categories |= PAGE_IS_SWAPPED;
if (!pmd_swp_uffd_wp(pmd))
categories |= PAGE_IS_WRITTEN;
@ -2403,9 +2430,10 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
categories |= PAGE_IS_SOFT_DIRTY;
if (p->masks_of_interest & PAGE_IS_FILE) {
swp = pmd_to_swp_entry(pmd);
if (is_pfn_swap_entry(swp) &&
!folio_test_anon(pfn_swap_entry_folio(swp)))
const softleaf_t entry = softleaf_from_pmd(pmd);
if (softleaf_has_pfn(entry) &&
!folio_test_anon(softleaf_to_folio(entry)))
categories |= PAGE_IS_FILE;
}
}
@ -2422,7 +2450,7 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,
old = pmdp_invalidate_ad(vma, addr, pmdp);
pmd = pmd_mkuffd_wp(old);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
} else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
} else if (pmd_is_migration_entry(pmd)) {
pmd = pmd_swp_mkuffd_wp(pmd);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
}
@ -2434,6 +2462,9 @@ static unsigned long pagemap_hugetlb_category(pte_t pte)
{
unsigned long categories = PAGE_IS_HUGE;
if (pte_none(pte))
return categories;
/*
* According to pagemap_hugetlb_range(), file-backed HugeTLB
* page cannot be swapped. So PAGE_IS_FILE is not checked for
@ -2441,6 +2472,7 @@ static unsigned long pagemap_hugetlb_category(pte_t pte)
*/
if (pte_present(pte)) {
categories |= PAGE_IS_PRESENT;
if (!huge_pte_uffd_wp(pte))
categories |= PAGE_IS_WRITTEN;
if (!PageAnon(pte_page(pte)))
@ -2449,8 +2481,9 @@ static unsigned long pagemap_hugetlb_category(pte_t pte)
categories |= PAGE_IS_PFNZERO;
if (pte_soft_dirty(pte))
categories |= PAGE_IS_SOFT_DIRTY;
} else if (is_swap_pte(pte)) {
} else {
categories |= PAGE_IS_SWAPPED;
if (!pte_swp_uffd_wp_any(pte))
categories |= PAGE_IS_WRITTEN;
if (pte_swp_soft_dirty(pte))
@ -2464,22 +2497,25 @@ static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
pte_t ptent)
{
unsigned long psize;
const unsigned long psize = huge_page_size(hstate_vma(vma));
softleaf_t entry;
if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
return;
psize = huge_page_size(hstate_vma(vma));
if (is_hugetlb_entry_migration(ptent))
set_huge_pte_at(vma->vm_mm, addr, ptep,
pte_swp_mkuffd_wp(ptent), psize);
else if (!huge_pte_none(ptent))
huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
huge_pte_mkuffd_wp(ptent));
else
if (huge_pte_none(ptent)) {
set_huge_pte_at(vma->vm_mm, addr, ptep,
make_pte_marker(PTE_MARKER_UFFD_WP), psize);
return;
}
entry = softleaf_from_pte(ptent);
if (softleaf_is_hwpoison(entry) || softleaf_is_marker(entry))
return;
if (softleaf_is_migration(entry))
set_huge_pte_at(vma->vm_mm, addr, ptep,
pte_swp_mkuffd_wp(ptent), psize);
else
huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
huge_pte_mkuffd_wp(ptent));
}
#endif /* CONFIG_HUGETLB_PAGE */

2
fs/ramfs/file-mmu.c
View File

@ -35,7 +35,7 @@ static unsigned long ramfs_mmu_get_unmapped_area(struct file *file,
unsigned long addr, unsigned long len, unsigned long pgoff,
unsigned long flags)
{
return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags);
return mm_get_unmapped_area(file, addr, len, pgoff, flags);
}
const struct file_operations ramfs_file_operations = {

20
fs/resctrl/pseudo_lock.c
View File

@ -995,10 +995,11 @@ static const struct vm_operations_struct pseudo_mmap_ops = {
.mremap = pseudo_lock_dev_mremap,
};
static int pseudo_lock_dev_mmap(struct file *filp, struct vm_area_struct *vma)
static int pseudo_lock_dev_mmap_prepare(struct vm_area_desc *desc)
{
unsigned long vsize = vma->vm_end - vma->vm_start;
unsigned long off = vma->vm_pgoff << PAGE_SHIFT;
unsigned long off = desc->pgoff << PAGE_SHIFT;
unsigned long vsize = vma_desc_size(desc);
struct file *filp = desc->file;
struct pseudo_lock_region *plr;
struct rdtgroup *rdtgrp;
unsigned long physical;
@ -1043,7 +1044,7 @@ static int pseudo_lock_dev_mmap(struct file *filp, struct vm_area_struct *vma)
* Ensure changes are carried directly to the memory being mapped,
* do not allow copy-on-write mapping.
*/
if (!(vma->vm_flags & VM_SHARED)) {
if (!(desc->vm_flags & VM_SHARED)) {
mutex_unlock(&rdtgroup_mutex);
return -EINVAL;
}
@ -1055,12 +1056,9 @@ static int pseudo_lock_dev_mmap(struct file *filp, struct vm_area_struct *vma)
memset(plr->kmem + off, 0, vsize);
if (remap_pfn_range(vma, vma->vm_start, physical + vma->vm_pgoff,
vsize, vma->vm_page_prot)) {
mutex_unlock(&rdtgroup_mutex);
return -EAGAIN;
}
vma->vm_ops = &pseudo_mmap_ops;
desc->vm_ops = &pseudo_mmap_ops;
mmap_action_remap_full(desc, physical + desc->pgoff);
mutex_unlock(&rdtgroup_mutex);
return 0;
}
@ -1071,7 +1069,7 @@ static const struct file_operations pseudo_lock_dev_fops = {
.write = NULL,
.open = pseudo_lock_dev_open,
.release = pseudo_lock_dev_release,
.mmap = pseudo_lock_dev_mmap,
.mmap_prepare = pseudo_lock_dev_mmap_prepare,
};
int rdt_pseudo_lock_init(void)

135
fs/userfaultfd.c
View File

@ -29,7 +29,7 @@
#include <linux/ioctl.h>
#include <linux/security.h>
#include <linux/hugetlb.h>
#include <linux/swapops.h>
#include <linux/leafops.h>
#include <linux/miscdevice.h>
#include <linux/uio.h>
@ -233,40 +233,48 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
{
struct vm_area_struct *vma = vmf->vma;
pte_t *ptep, pte;
bool ret = true;
assert_fault_locked(vmf);
ptep = hugetlb_walk(vma, vmf->address, vma_mmu_pagesize(vma));
if (!ptep)
goto out;
return true;
ret = false;
pte = huge_ptep_get(vma->vm_mm, vmf->address, ptep);
/*
* Lockless access: we're in a wait_event so it's ok if it
* changes under us. PTE markers should be handled the same as none
* ptes here.
* changes under us.
*/
/* Entry is still missing, wait for userspace to resolve the fault. */
if (huge_pte_none(pte))
return true;
/* UFFD PTE markers require userspace to resolve the fault. */
if (pte_is_uffd_marker(pte))
return true;
/*
* If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
* resolve the fault.
*/
if (huge_pte_none_mostly(pte))
ret = true;
if (!huge_pte_write(pte) && (reason & VM_UFFD_WP))
ret = true;
out:
return ret;
return true;
return false;
}
#else
static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
struct vm_fault *vmf,
unsigned long reason)
{
return false; /* should never get here */
/* Should never get here. */
VM_WARN_ON_ONCE(1);
return false;
}
#endif /* CONFIG_HUGETLB_PAGE */
/*
* Verify the pagetables are still not ok after having reigstered into
* Verify the pagetables are still not ok after having registered into
* the fault_pending_wqh to avoid userland having to UFFDIO_WAKE any
* userfault that has already been resolved, if userfaultfd_read_iter and
* UFFDIO_COPY|ZEROPAGE are being run simultaneously on two different
@ -284,53 +292,63 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
pmd_t *pmd, _pmd;
pte_t *pte;
pte_t ptent;
bool ret = true;
bool ret;
assert_fault_locked(vmf);
pgd = pgd_offset(mm, address);
if (!pgd_present(*pgd))
goto out;
return true;
p4d = p4d_offset(pgd, address);
if (!p4d_present(*p4d))
goto out;
return true;
pud = pud_offset(p4d, address);
if (!pud_present(*pud))
goto out;
return true;
pmd = pmd_offset(pud, address);
again:
_pmd = pmdp_get_lockless(pmd);
if (pmd_none(_pmd))
return true;
/*
* A race could arise which would result in a softleaf entry such as
* migration entry unexpectedly being present in the PMD, so explicitly
* check for this and bail out if so.
*/
if (!pmd_present(_pmd))
return false;
if (pmd_trans_huge(_pmd))
return !pmd_write(_pmd) && (reason & VM_UFFD_WP);
pte = pte_offset_map(pmd, address);
if (!pte)
goto again;
/*
* Lockless access: we're in a wait_event so it's ok if it
* changes under us.
*/
ptent = ptep_get(pte);
ret = true;
/* Entry is still missing, wait for userspace to resolve the fault. */
if (pte_none(ptent))
goto out;
/* UFFD PTE markers require userspace to resolve the fault. */
if (pte_is_uffd_marker(ptent))
goto out;
/*
* If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
* resolve the fault.
*/
if (!pte_write(ptent) && (reason & VM_UFFD_WP))
goto out;
ret = false;
if (!pmd_present(_pmd))
goto out;
if (pmd_trans_huge(_pmd)) {
if (!pmd_write(_pmd) && (reason & VM_UFFD_WP))
ret = true;
goto out;
}
pte = pte_offset_map(pmd, address);
if (!pte) {
ret = true;
goto again;
}
/*
* Lockless access: we're in a wait_event so it's ok if it
* changes under us. PTE markers should be handled the same as none
* ptes here.
*/
ptent = ptep_get(pte);
if (pte_none_mostly(ptent))
ret = true;
if (!pte_write(ptent) && (reason & VM_UFFD_WP))
ret = true;
pte_unmap(pte);
out:
pte_unmap(pte);
return ret;
}
@ -490,12 +508,13 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)
set_current_state(blocking_state);
spin_unlock_irq(&ctx->fault_pending_wqh.lock);
if (!is_vm_hugetlb_page(vma))
must_wait = userfaultfd_must_wait(ctx, vmf, reason);
else
if (is_vm_hugetlb_page(vma)) {
must_wait = userfaultfd_huge_must_wait(ctx, vmf, reason);
if (is_vm_hugetlb_page(vma))
hugetlb_vma_unlock_read(vma);
} else {
must_wait = userfaultfd_must_wait(ctx, vmf, reason);
}
release_fault_lock(vmf);
if (likely(must_wait && !READ_ONCE(ctx->released))) {
@ -1270,9 +1289,9 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MISSING)
vm_flags |= VM_UFFD_MISSING;
if (uffdio_register.mode & UFFDIO_REGISTER_MODE_WP) {
#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP
goto out;
#endif
if (!pgtable_supports_uffd_wp())
goto out;
vm_flags |= VM_UFFD_WP;
}
if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR) {
@ -1980,14 +1999,14 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,
uffdio_api.features &=
~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM);
#endif
#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP
uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;
#endif
#ifndef CONFIG_PTE_MARKER_UFFD_WP
uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
#endif
if (!pgtable_supports_uffd_wp())
uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;
if (!uffd_supports_wp_marker()) {
uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
}
ret = -EINVAL;
if (features & ~uffdio_api.features)

8
include/asm-generic/hugetlb.h
View File

@ -97,14 +97,6 @@ static inline int huge_pte_none(pte_t pte)
}
#endif
/* Please refer to comments above pte_none_mostly() for the usage */
#ifndef __HAVE_ARCH_HUGE_PTE_NONE_MOSTLY
static inline int huge_pte_none_mostly(pte_t pte)
{
return huge_pte_none(pte) || is_pte_marker(pte);
}
#endif
#ifndef __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT
static inline void huge_ptep_set_wrprotect(struct mm_struct *mm,
unsigned long addr, pte_t *ptep)

24
include/asm-generic/pgalloc.h
View File

@ -18,8 +18,7 @@
*/
static inline pte_t *__pte_alloc_one_kernel_noprof(struct mm_struct *mm)
{
struct ptdesc *ptdesc = pagetable_alloc_noprof(GFP_PGTABLE_KERNEL &
~__GFP_HIGHMEM, 0);
struct ptdesc *ptdesc = pagetable_alloc_noprof(GFP_PGTABLE_KERNEL, 0);
if (!ptdesc)
return NULL;
@ -28,6 +27,8 @@ static inline pte_t *__pte_alloc_one_kernel_noprof(struct mm_struct *mm)
return NULL;
}
ptdesc_set_kernel(ptdesc);
return ptdesc_address(ptdesc);
}
#define __pte_alloc_one_kernel(...) alloc_hooks(__pte_alloc_one_kernel_noprof(__VA_ARGS__))
@ -146,6 +147,10 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long ad
pagetable_free(ptdesc);
return NULL;
}
if (mm == &init_mm)
ptdesc_set_kernel(ptdesc);
return ptdesc_address(ptdesc);
}
#define pmd_alloc_one(...) alloc_hooks(pmd_alloc_one_noprof(__VA_ARGS__))
@ -172,13 +177,16 @@ static inline pud_t *__pud_alloc_one_noprof(struct mm_struct *mm, unsigned long
if (mm == &init_mm)
gfp = GFP_PGTABLE_KERNEL;
gfp &= ~__GFP_HIGHMEM;
ptdesc = pagetable_alloc_noprof(gfp, 0);
if (!ptdesc)
return NULL;
pagetable_pud_ctor(ptdesc);
if (mm == &init_mm)
ptdesc_set_kernel(ptdesc);
return ptdesc_address(ptdesc);
}
#define __pud_alloc_one(...) alloc_hooks(__pud_alloc_one_noprof(__VA_ARGS__))
@ -226,13 +234,16 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long
if (mm == &init_mm)
gfp = GFP_PGTABLE_KERNEL;
gfp &= ~__GFP_HIGHMEM;
ptdesc = pagetable_alloc_noprof(gfp, 0);
if (!ptdesc)
return NULL;
pagetable_p4d_ctor(ptdesc);
if (mm == &init_mm)
ptdesc_set_kernel(ptdesc);
return ptdesc_address(ptdesc);
}
#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__))
@ -270,13 +281,16 @@ static inline pgd_t *__pgd_alloc_noprof(struct mm_struct *mm, unsigned int order
if (mm == &init_mm)
gfp = GFP_PGTABLE_KERNEL;
gfp &= ~__GFP_HIGHMEM;
ptdesc = pagetable_alloc_noprof(gfp, order);
if (!ptdesc)
return NULL;
pagetable_pgd_ctor(ptdesc);
if (mm == &init_mm)
ptdesc_set_kernel(ptdesc);
return ptdesc_address(ptdesc);
}
#define __pgd_alloc(...) alloc_hooks(__pgd_alloc_noprof(__VA_ARGS__))

17
include/asm-generic/pgtable_uffd.h
View File

@ -1,6 +1,23 @@
#ifndef _ASM_GENERIC_PGTABLE_UFFD_H
#define _ASM_GENERIC_PGTABLE_UFFD_H
/*
* Some platforms can customize the uffd-wp bit, making it unavailable
* even if the architecture provides the resource.
* Adding this API allows architectures to add their own checks for the
* devices on which the kernel is running.
* Note: When overriding it, please make sure the
* CONFIG_HAVE_ARCH_USERFAULTFD_WP is part of this macro.
*/
#ifndef pgtable_supports_uffd_wp
#define pgtable_supports_uffd_wp() IS_ENABLED(CONFIG_HAVE_ARCH_USERFAULTFD_WP)
#endif
static inline bool uffd_supports_wp_marker(void)
{
return pgtable_supports_uffd_wp() && IS_ENABLED(CONFIG_PTE_MARKER_UFFD_WP);
}
#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP
static __always_inline int pte_uffd_wp(pte_t pte)
{

39
include/linux/damon.h
View File

@ -91,17 +91,23 @@ struct damon_region {
* @nr_regions: Number of monitoring target regions of this target.
* @regions_list: Head of the monitoring target regions of this target.
* @list: List head for siblings.
* @obsolete: Whether the commit destination target is obsolete.
*
* Each monitoring context could have multiple targets. For example, a context
* for virtual memory address spaces could have multiple target processes. The
* @pid should be set for appropriate &struct damon_operations including the
* virtual address spaces monitoring operations.
*
* @obsolete is used only for damon_commit_targets() source targets, to specify
* the matching destination targets are obsolete. Read damon_commit_targets()
* to see how it is handled.
*/
struct damon_target {
struct pid *pid;
unsigned int nr_regions;
struct list_head regions_list;
struct list_head list;
bool obsolete;
};
/**
@ -147,6 +153,8 @@ enum damos_action {
* @DAMOS_QUOTA_SOME_MEM_PSI_US: System level some memory PSI in us.
* @DAMOS_QUOTA_NODE_MEM_USED_BP: MemUsed ratio of a node.
* @DAMOS_QUOTA_NODE_MEM_FREE_BP: MemFree ratio of a node.
* @DAMOS_QUOTA_NODE_MEMCG_USED_BP: MemUsed ratio of a node for a cgroup.
* @DAMOS_QUOTA_NODE_MEMCG_FREE_BP: MemFree ratio of a node for a cgroup.
* @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics.
*
* Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported.
@ -156,6 +164,8 @@ enum damos_quota_goal_metric {
DAMOS_QUOTA_SOME_MEM_PSI_US,
DAMOS_QUOTA_NODE_MEM_USED_BP,
DAMOS_QUOTA_NODE_MEM_FREE_BP,
DAMOS_QUOTA_NODE_MEMCG_USED_BP,
DAMOS_QUOTA_NODE_MEMCG_FREE_BP,
NR_DAMOS_QUOTA_GOAL_METRICS,
};
@ -166,6 +176,7 @@ enum damos_quota_goal_metric {
* @current_value: Current value of @metric.
* @last_psi_total: Last measured total PSI
* @nid: Node id.
* @memcg_id: Memcg id.
* @list: List head for siblings.
*
* Data structure for getting the current score of the quota tuning goal. The
@ -176,6 +187,12 @@ enum damos_quota_goal_metric {
* If @metric is DAMOS_QUOTA_USER_INPUT, @current_value should be manually
* entered by the user, probably inside the kdamond callbacks. Otherwise,
* DAMON sets @current_value with self-measured value of @metric.
*
* If @metric is DAMOS_QUOTA_NODE_MEM_{USED,FREE}_BP, @nid represents the node
* id of the target node to account the used/free memory.
*
* If @metric is DAMOS_QUOTA_NODE_MEMCG_{USED,FREE}_BP, @nid and @memcg_id
* represents the node id and the cgroup to account the used memory for.
*/
struct damos_quota_goal {
enum damos_quota_goal_metric metric;
@ -184,7 +201,10 @@ struct damos_quota_goal {
/* metric-dependent fields */
union {
u64 last_psi_total;
int nid;
struct {
int nid;
unsigned short memcg_id;
};
};
struct list_head list;
};
@ -472,7 +492,7 @@ struct damos_migrate_dests {
* @wmarks: Watermarks for automated (in)activation of this scheme.
* @migrate_dests: Destination nodes if @action is "migrate_{hot,cold}".
* @target_nid: Destination node if @action is "migrate_{hot,cold}".
* @filters: Additional set of &struct damos_filter for &action.
* @core_filters: Additional set of &struct damos_filter for &action.
* @ops_filters: ops layer handling &struct damos_filter objects list.
* @last_applied: Last @action applied ops-managing entity.
* @stat: Statistics of this scheme.
@ -498,7 +518,7 @@ struct damos_migrate_dests {
*
* Before applying the &action to a memory region, &struct damon_operations
* implementation could check pages of the region and skip &action to respect
* &filters
* &core_filters
*
* The minimum entity that @action can be applied depends on the underlying
* &struct damon_operations. Since it may not be aligned with the core layer
@ -542,7 +562,7 @@ struct damos {
struct damos_migrate_dests migrate_dests;
};
};
struct list_head filters;
struct list_head core_filters;
struct list_head ops_filters;
void *last_applied;
struct damos_stat stat;
@ -851,11 +871,11 @@ static inline unsigned long damon_sz_region(struct damon_region *r)
#define damos_for_each_quota_goal_safe(goal, next, quota) \
list_for_each_entry_safe(goal, next, &(quota)->goals, list)
#define damos_for_each_filter(f, scheme) \
list_for_each_entry(f, &(scheme)->filters, list)
#define damos_for_each_core_filter(f, scheme) \
list_for_each_entry(f, &(scheme)->core_filters, list)
#define damos_for_each_filter_safe(f, next, scheme) \
list_for_each_entry_safe(f, next, &(scheme)->filters, list)
#define damos_for_each_core_filter_safe(f, next, scheme) \
list_for_each_entry_safe(f, next, &(scheme)->core_filters, list)
#define damos_for_each_ops_filter(f, scheme) \
list_for_each_entry(f, &(scheme)->ops_filters, list)
@ -947,7 +967,8 @@ int damon_call(struct damon_ctx *ctx, struct damon_call_control *control);
int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control);
int damon_set_region_biggest_system_ram_default(struct damon_target *t,
unsigned long *start, unsigned long *end);
unsigned long *start, unsigned long *end,
unsigned long min_sz_region);
#endif /* CONFIG_DAMON */

6
include/linux/fs.h
View File

@ -2041,14 +2041,14 @@ static inline bool can_mmap_file(struct file *file)
return true;
}
int __compat_vma_mmap_prepare(const struct file_operations *f_op,
int __compat_vma_mmap(const struct file_operations *f_op,
struct file *file, struct vm_area_struct *vma);
int compat_vma_mmap_prepare(struct file *file, struct vm_area_struct *vma);
int compat_vma_mmap(struct file *file, struct vm_area_struct *vma);
static inline int vfs_mmap(struct file *file, struct vm_area_struct *vma)
{
if (file->f_op->mmap_prepare)
return compat_vma_mmap_prepare(file, vma);
return compat_vma_mmap(file, vma);
return file->f_op->mmap(file, vma);
}

2
include/linux/gfp.h
View File

@ -387,7 +387,7 @@ extern void free_pages(unsigned long addr, unsigned int order);
#define free_page(addr) free_pages((addr), 0)
void page_alloc_init_cpuhp(void);
int decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp);
bool decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp);
void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp);
void drain_all_pages(struct zone *zone);
void drain_local_pages(struct zone *zone);

112
include/linux/huge_mm.h
View File

@ -364,20 +364,35 @@ unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long add
unsigned long len, unsigned long pgoff, unsigned long flags,
vm_flags_t vm_flags);
enum split_type {
SPLIT_TYPE_UNIFORM,
SPLIT_TYPE_NON_UNIFORM,
};
bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins);
int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
unsigned int new_order);
int folio_split_unmapped(struct folio *folio, unsigned int new_order);
int min_order_for_split(struct folio *folio);
int split_folio_to_list(struct folio *folio, struct list_head *list);
bool uniform_split_supported(struct folio *folio, unsigned int new_order,
bool warns);
bool non_uniform_split_supported(struct folio *folio, unsigned int new_order,
bool warns);
bool folio_split_supported(struct folio *folio, unsigned int new_order,
enum split_type split_type, bool warns);
int folio_split(struct folio *folio, unsigned int new_order, struct page *page,
struct list_head *list);
/*
* try_folio_split_to_order - try to split a @folio at @page to @new_order using
* non uniform split.
static inline int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
unsigned int new_order)
{
return __split_huge_page_to_list_to_order(page, list, new_order);
}
static inline int split_huge_page_to_order(struct page *page, unsigned int new_order)
{
return split_huge_page_to_list_to_order(page, NULL, new_order);
}
/**
* try_folio_split_to_order() - try to split a @folio at @page to @new_order
* using non uniform split.
* @folio: folio to be split
* @page: split to @new_order at the given page
* @new_order: the target split order
@ -387,14 +402,13 @@ int folio_split(struct folio *folio, unsigned int new_order, struct page *page,
* folios are put back to LRU list. Use min_order_for_split() to get the lower
* bound of @new_order.
*
* Return: 0: split is successful, otherwise split failed.
* Return: 0 - split is successful, otherwise split failed.
*/
static inline int try_folio_split_to_order(struct folio *folio,
struct page *page, unsigned int new_order)
{
if (!non_uniform_split_supported(folio, new_order, /* warns= */ false))
return split_huge_page_to_list_to_order(&folio->page, NULL,
new_order);
if (!folio_split_supported(folio, new_order, SPLIT_TYPE_NON_UNIFORM, /* warns= */ false))
return split_huge_page_to_order(&folio->page, new_order);
return folio_split(folio, new_order, page, NULL);
}
static inline int split_huge_page(struct page *page)
@ -402,14 +416,43 @@ static inline int split_huge_page(struct page *page)
return split_huge_page_to_list_to_order(page, NULL, 0);
}
void deferred_split_folio(struct folio *folio, bool partially_mapped);
#ifdef CONFIG_MEMCG
void reparent_deferred_split_queue(struct mem_cgroup *memcg);
#endif
void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
unsigned long address, bool freeze);
/**
* pmd_is_huge() - Is this PMD either a huge PMD entry or a software leaf entry?
* @pmd: The PMD to check.
*
* A huge PMD entry is a non-empty entry which is present and marked huge or a
* software leaf entry. This check be performed without the appropriate locks
* held, in which case the condition should be rechecked after they are
* acquired.
*
* Returns: true if this PMD is huge, false otherwise.
*/
static inline bool pmd_is_huge(pmd_t pmd)
{
if (pmd_present(pmd)) {
return pmd_trans_huge(pmd);
} else if (!pmd_none(pmd)) {
/*
* Non-present PMDs must be valid huge non-present entries. We
* cannot assert that here due to header dependency issues.
*/
return true;
}
return false;
}
#define split_huge_pmd(__vma, __pmd, __address) \
do { \
pmd_t *____pmd = (__pmd); \
if (is_swap_pmd(*____pmd) || pmd_trans_huge(*____pmd)) \
if (pmd_is_huge(*____pmd)) \
__split_huge_pmd(__vma, __pmd, __address, \
false); \
} while (0)
@ -447,19 +490,14 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start,
spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma);
spinlock_t *__pud_trans_huge_lock(pud_t *pud, struct vm_area_struct *vma);
static inline int is_swap_pmd(pmd_t pmd)
{
return !pmd_none(pmd) && !pmd_present(pmd);
}
/* mmap_lock must be held on entry */
static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
struct vm_area_struct *vma)
{
if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd))
if (pmd_is_huge(*pmd))
return __pmd_trans_huge_lock(pmd, vma);
else
return NULL;
return NULL;
}
static inline spinlock_t *pud_trans_huge_lock(pud_t *pud,
struct vm_area_struct *vma)
@ -473,6 +511,8 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud,
/**
* folio_test_pmd_mappable - Can we map this folio with a PMD?
* @folio: The folio to test
*
* Return: true - @folio can be mapped, false - @folio cannot be mapped.
*/
static inline bool folio_test_pmd_mappable(struct folio *folio)
{
@ -481,6 +521,8 @@ static inline bool folio_test_pmd_mappable(struct folio *folio)
vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf);
vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf);
extern struct folio *huge_zero_folio;
extern unsigned long huge_zero_pfn;
@ -524,6 +566,8 @@ void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address,
pmd_t *pmd, bool freeze);
bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,
pmd_t *pmdp, struct folio *folio);
void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd,
struct vm_area_struct *vma, unsigned long haddr);
#else /* CONFIG_TRANSPARENT_HUGEPAGE */
@ -576,6 +620,11 @@ split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
VM_WARN_ON_ONCE_PAGE(1, page);
return -EINVAL;
}
static inline int split_huge_page_to_order(struct page *page, unsigned int new_order)
{
VM_WARN_ON_ONCE_PAGE(1, page);
return -EINVAL;
}
static inline int split_huge_page(struct page *page)
{
VM_WARN_ON_ONCE_PAGE(1, page);
@ -602,6 +651,7 @@ static inline int try_folio_split_to_order(struct folio *folio,
}
static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {}
static inline void reparent_deferred_split_queue(struct mem_cgroup *memcg) {}
#define split_huge_pmd(__vma, __pmd, __address) \
do { } while (0)
@ -642,10 +692,6 @@ static inline void vma_adjust_trans_huge(struct vm_area_struct *vma,
struct vm_area_struct *next)
{
}
static inline int is_swap_pmd(pmd_t pmd)
{
return 0;
}
static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
struct vm_area_struct *vma)
{
@ -662,6 +708,11 @@ static inline vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
return 0;
}
static inline vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf)
{
return 0;
}
static inline bool is_huge_zero_folio(const struct folio *folio)
{
return false;
@ -682,12 +733,6 @@ static inline void mm_put_huge_zero_folio(struct mm_struct *mm)
return;
}
static inline struct page *follow_devmap_pmd(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmd, int flags, struct dev_pagemap **pgmap)
{
return NULL;
}
static inline bool thp_migration_supported(void)
{
return false;
@ -720,6 +765,11 @@ static inline struct folio *get_persistent_huge_zero_folio(void)
{
return NULL;
}
static inline bool pmd_is_huge(pmd_t pmd)
{
return false;
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
static inline int split_folio_to_list_to_order(struct folio *folio,

13
include/linux/hugetlb.h
View File

@ -150,8 +150,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte,
struct folio **foliop);
#endif /* CONFIG_USERFAULTFD */
long hugetlb_reserve_pages(struct inode *inode, long from, long to,
struct vm_area_struct *vma,
vm_flags_t vm_flags);
struct vm_area_desc *desc, vm_flags_t vm_flags);
long hugetlb_unreserve_pages(struct inode *inode, long start, long end,
long freed);
bool folio_isolate_hugetlb(struct folio *folio, struct list_head *list);
@ -172,7 +171,7 @@ bool hugetlbfs_pagecache_present(struct hstate *h,
struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio);
extern int sysctl_hugetlb_shm_group;
extern int sysctl_hugetlb_shm_group __read_mostly;
extern struct list_head huge_boot_pages[MAX_NUMNODES];
void hugetlb_bootmem_alloc(void);
@ -275,11 +274,10 @@ void hugetlb_vma_lock_release(struct kref *kref);
long hugetlb_change_protection(struct vm_area_struct *vma,
unsigned long address, unsigned long end, pgprot_t newprot,
unsigned long cp_flags);
bool is_hugetlb_entry_migration(pte_t pte);
bool is_hugetlb_entry_hwpoisoned(pte_t pte);
void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
void fixup_hugetlb_reservations(struct vm_area_struct *vma);
void hugetlb_split(struct vm_area_struct *vma, unsigned long addr);
int hugetlb_vma_lock_alloc(struct vm_area_struct *vma);
#else /* !CONFIG_HUGETLB_PAGE */
@ -466,6 +464,11 @@ static inline void fixup_hugetlb_reservations(struct vm_area_struct *vma)
static inline void hugetlb_split(struct vm_area_struct *vma, unsigned long addr) {}
static inline int hugetlb_vma_lock_alloc(struct vm_area_struct *vma)
{
return 0;
}
#endif /* !CONFIG_HUGETLB_PAGE */
#ifndef pgd_write

15
include/linux/hugetlb_inline.h
View File

@ -2,22 +2,27 @@
#ifndef _LINUX_HUGETLB_INLINE_H
#define _LINUX_HUGETLB_INLINE_H
#ifdef CONFIG_HUGETLB_PAGE
#include <linux/mm.h>
static inline bool is_vm_hugetlb_page(struct vm_area_struct *vma)
#ifdef CONFIG_HUGETLB_PAGE
static inline bool is_vm_hugetlb_flags(vm_flags_t vm_flags)
{
return !!(vma->vm_flags & VM_HUGETLB);
return !!(vm_flags & VM_HUGETLB);
}
#else
static inline bool is_vm_hugetlb_page(struct vm_area_struct *vma)
static inline bool is_vm_hugetlb_flags(vm_flags_t vm_flags)
{
return false;
}
#endif
static inline bool is_vm_hugetlb_page(struct vm_area_struct *vma)
{
return is_vm_hugetlb_flags(vma->vm_flags);
}
#endif

4
include/linux/iommu.h
View File

@ -1135,7 +1135,9 @@ struct iommu_sva {
struct iommu_mm_data {
u32 pasid;
struct mm_struct *mm;
struct list_head sva_domains;
struct list_head mm_list_elm;
};
int iommu_fwspec_init(struct device *dev, struct fwnode_handle *iommu_fwnode);
@ -1616,6 +1618,7 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev,
struct mm_struct *mm);
void iommu_sva_unbind_device(struct iommu_sva *handle);
u32 iommu_sva_get_pasid(struct iommu_sva *handle);
void iommu_sva_invalidate_kva_range(unsigned long start, unsigned long end);
#else
static inline struct iommu_sva *
iommu_sva_bind_device(struct device *dev, struct mm_struct *mm)
@ -1640,6 +1643,7 @@ static inline u32 mm_get_enqcmd_pasid(struct mm_struct *mm)
}
static inline void mm_pasid_drop(struct mm_struct *mm) {}
static inline void iommu_sva_invalidate_kva_range(unsigned long start, unsigned long end) {}
#endif /* CONFIG_IOMMU_SVA */
#ifdef CONFIG_IOMMU_IOPF

20
include/linux/kasan.h
View File

@ -571,11 +571,27 @@ static inline void kasan_init_hw_tags(void) { }
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
void kasan_populate_early_vm_area_shadow(void *start, unsigned long size);
int kasan_populate_vmalloc(unsigned long addr, unsigned long size, gfp_t gfp_mask);
void kasan_release_vmalloc(unsigned long start, unsigned long end,
int __kasan_populate_vmalloc(unsigned long addr, unsigned long size, gfp_t gfp_mask);
static inline int kasan_populate_vmalloc(unsigned long addr,
unsigned long size, gfp_t gfp_mask)
{
if (kasan_enabled())
return __kasan_populate_vmalloc(addr, size, gfp_mask);
return 0;
}
void __kasan_release_vmalloc(unsigned long start, unsigned long end,
unsigned long free_region_start,
unsigned long free_region_end,
unsigned long flags);
static inline void kasan_release_vmalloc(unsigned long start, unsigned long end,
unsigned long free_region_start,
unsigned long free_region_end,
unsigned long flags)
{
if (kasan_enabled())
return __kasan_release_vmalloc(start, end, free_region_start,
free_region_end, flags);
}
#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */

6
include/linux/kmsan.h
View File

@ -133,6 +133,7 @@ void kmsan_kfree_large(const void *ptr);
* @prot: page protection flags used for vmap.
* @pages: array of pages.
* @page_shift: page_shift passed to vmap_range_noflush().
* @gfp_mask: gfp_mask to use internally.
*
* KMSAN maps shadow and origin pages of @pages into contiguous ranges in
* vmalloc metadata address range. Returns 0 on success, callers must check
@ -142,7 +143,8 @@ int __must_check kmsan_vmap_pages_range_noflush(unsigned long start,
unsigned long end,
pgprot_t prot,
struct page **pages,
unsigned int page_shift);
unsigned int page_shift,
gfp_t gfp_mask);
/**
* kmsan_vunmap_kernel_range_noflush() - Notify KMSAN about a vunmap.
@ -347,7 +349,7 @@ static inline void kmsan_kfree_large(const void *ptr)
static inline int __must_check kmsan_vmap_pages_range_noflush(
unsigned long start, unsigned long end, pgprot_t prot,
struct page **pages, unsigned int page_shift)
struct page **pages, unsigned int page_shift, gfp_t gfp_mask)
{
return 0;
}

4
include/linux/ksm.h
View File

@ -17,7 +17,7 @@
#ifdef CONFIG_KSM
int ksm_madvise(struct vm_area_struct *vma, unsigned long start,
unsigned long end, int advice, vm_flags_t *vm_flags);
vm_flags_t ksm_vma_flags(const struct mm_struct *mm, const struct file *file,
vm_flags_t ksm_vma_flags(struct mm_struct *mm, const struct file *file,
vm_flags_t vm_flags);
int ksm_enable_merge_any(struct mm_struct *mm);
int ksm_disable_merge_any(struct mm_struct *mm);
@ -103,7 +103,7 @@ bool ksm_process_mergeable(struct mm_struct *mm);
#else /* !CONFIG_KSM */
static inline vm_flags_t ksm_vma_flags(const struct mm_struct *mm,
static inline vm_flags_t ksm_vma_flags(struct mm_struct *mm,
const struct file *file, vm_flags_t vm_flags)
{
return vm_flags;

619
include/linux/leafops.h Normal file
View File

@ -0,0 +1,619 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Describes operations that can be performed on software-defined page table
* leaf entries. These are abstracted from the hardware page table entries
* themselves by the softleaf_t type, see mm_types.h.
*/
#ifndef _LINUX_LEAFOPS_H
#define _LINUX_LEAFOPS_H
#include <linux/mm_types.h>
#include <linux/swapops.h>
#include <linux/swap.h>
#ifdef CONFIG_MMU
/* Temporary until swp_entry_t eliminated. */
#define LEAF_TYPE_SHIFT SWP_TYPE_SHIFT
enum softleaf_type {
/* Fundamental types. */
SOFTLEAF_NONE,
SOFTLEAF_SWAP,
/* Migration types. */
SOFTLEAF_MIGRATION_READ,
SOFTLEAF_MIGRATION_READ_EXCLUSIVE,
SOFTLEAF_MIGRATION_WRITE,
/* Device types. */
SOFTLEAF_DEVICE_PRIVATE_READ,
SOFTLEAF_DEVICE_PRIVATE_WRITE,
SOFTLEAF_DEVICE_EXCLUSIVE,
/* H/W posion types. */
SOFTLEAF_HWPOISON,
/* Marker types. */
SOFTLEAF_MARKER,
};
/**
* softleaf_mk_none() - Create an empty ('none') leaf entry.
* Returns: empty leaf entry.
*/
static inline softleaf_t softleaf_mk_none(void)
{
return ((softleaf_t) { 0 });
}
/**
* softleaf_from_pte() - Obtain a leaf entry from a PTE entry.
* @pte: PTE entry.
*
* If @pte is present (therefore not a leaf entry) the function returns an empty
* leaf entry. Otherwise, it returns a leaf entry.
*
* Returns: Leaf entry.
*/
static inline softleaf_t softleaf_from_pte(pte_t pte)
{
softleaf_t arch_entry;
if (pte_present(pte) || pte_none(pte))
return softleaf_mk_none();
pte = pte_swp_clear_flags(pte);
arch_entry = __pte_to_swp_entry(pte);
/* Temporary until swp_entry_t eliminated. */
return swp_entry(__swp_type(arch_entry), __swp_offset(arch_entry));
}
/**
* softleaf_to_pte() - Obtain a PTE entry from a leaf entry.
* @entry: Leaf entry.
*
* This generates an architecture-specific PTE entry that can be utilised to
* encode the metadata the leaf entry encodes.
*
* Returns: Architecture-specific PTE entry encoding leaf entry.
*/
static inline pte_t softleaf_to_pte(softleaf_t entry)
{
/* Temporary until swp_entry_t eliminated. */
return swp_entry_to_pte(entry);
}
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
/**
* softleaf_from_pmd() - Obtain a leaf entry from a PMD entry.
* @pmd: PMD entry.
*
* If @pmd is present (therefore not a leaf entry) the function returns an empty
* leaf entry. Otherwise, it returns a leaf entry.
*
* Returns: Leaf entry.
*/
static inline softleaf_t softleaf_from_pmd(pmd_t pmd)
{
softleaf_t arch_entry;
if (pmd_present(pmd) || pmd_none(pmd))
return softleaf_mk_none();
if (pmd_swp_soft_dirty(pmd))
pmd = pmd_swp_clear_soft_dirty(pmd);
if (pmd_swp_uffd_wp(pmd))
pmd = pmd_swp_clear_uffd_wp(pmd);
arch_entry = __pmd_to_swp_entry(pmd);
/* Temporary until swp_entry_t eliminated. */
return swp_entry(__swp_type(arch_entry), __swp_offset(arch_entry));
}
#else
static inline softleaf_t softleaf_from_pmd(pmd_t pmd)
{
return softleaf_mk_none();
}
#endif
/**
* softleaf_is_none() - Is the leaf entry empty?
* @entry: Leaf entry.
*
* Empty entries are typically the result of a 'none' page table leaf entry
* being converted to a leaf entry.
*
* Returns: true if the entry is empty, false otherwise.
*/
static inline bool softleaf_is_none(softleaf_t entry)
{
return entry.val == 0;
}
/**
* softleaf_type() - Identify the type of leaf entry.
* @enntry: Leaf entry.
*
* Returns: the leaf entry type associated with @entry.
*/
static inline enum softleaf_type softleaf_type(softleaf_t entry)
{
unsigned int type_num;
if (softleaf_is_none(entry))
return SOFTLEAF_NONE;
type_num = entry.val >> LEAF_TYPE_SHIFT;
if (type_num < MAX_SWAPFILES)
return SOFTLEAF_SWAP;
switch (type_num) {
#ifdef CONFIG_MIGRATION
case SWP_MIGRATION_READ:
return SOFTLEAF_MIGRATION_READ;
case SWP_MIGRATION_READ_EXCLUSIVE:
return SOFTLEAF_MIGRATION_READ_EXCLUSIVE;
case SWP_MIGRATION_WRITE:
return SOFTLEAF_MIGRATION_WRITE;
#endif
#ifdef CONFIG_DEVICE_PRIVATE
case SWP_DEVICE_WRITE:
return SOFTLEAF_DEVICE_PRIVATE_WRITE;
case SWP_DEVICE_READ:
return SOFTLEAF_DEVICE_PRIVATE_READ;
case SWP_DEVICE_EXCLUSIVE:
return SOFTLEAF_DEVICE_EXCLUSIVE;
#endif
#ifdef CONFIG_MEMORY_FAILURE
case SWP_HWPOISON:
return SOFTLEAF_HWPOISON;
#endif
case SWP_PTE_MARKER:
return SOFTLEAF_MARKER;
}
/* Unknown entry type. */
VM_WARN_ON_ONCE(1);
return SOFTLEAF_NONE;
}
/**
* softleaf_is_swap() - Is this leaf entry a swap entry?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is a swap entry, otherwise false.
*/
static inline bool softleaf_is_swap(softleaf_t entry)
{
return softleaf_type(entry) == SOFTLEAF_SWAP;
}
/**
* softleaf_is_migration_write() - Is this leaf entry a writable migration entry?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is a writable migration entry, otherwise
* false.
*/
static inline bool softleaf_is_migration_write(softleaf_t entry)
{
return softleaf_type(entry) == SOFTLEAF_MIGRATION_WRITE;
}
/**
* softleaf_is_migration_read() - Is this leaf entry a readable migration entry?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is a readable migration entry, otherwise
* false.
*/
static inline bool softleaf_is_migration_read(softleaf_t entry)
{
return softleaf_type(entry) == SOFTLEAF_MIGRATION_READ;
}
/**
* softleaf_is_migration_read_exclusive() - Is this leaf entry an exclusive
* readable migration entry?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is an exclusive readable migration entry,
* otherwise false.
*/
static inline bool softleaf_is_migration_read_exclusive(softleaf_t entry)
{
return softleaf_type(entry) == SOFTLEAF_MIGRATION_READ_EXCLUSIVE;
}
/**
* softleaf_is_migration() - Is this leaf entry a migration entry?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is a migration entry, otherwise false.
*/
static inline bool softleaf_is_migration(softleaf_t entry)
{
switch (softleaf_type(entry)) {
case SOFTLEAF_MIGRATION_READ:
case SOFTLEAF_MIGRATION_READ_EXCLUSIVE:
case SOFTLEAF_MIGRATION_WRITE:
return true;
default:
return false;
}
}
/**
* softleaf_is_device_private_write() - Is this leaf entry a device private
* writable entry?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is a device private writable entry, otherwise
* false.
*/
static inline bool softleaf_is_device_private_write(softleaf_t entry)
{
return softleaf_type(entry) == SOFTLEAF_DEVICE_PRIVATE_WRITE;
}
/**
* softleaf_is_device_private() - Is this leaf entry a device private entry?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is a device private entry, otherwise false.
*/
static inline bool softleaf_is_device_private(softleaf_t entry)
{
switch (softleaf_type(entry)) {
case SOFTLEAF_DEVICE_PRIVATE_WRITE:
case SOFTLEAF_DEVICE_PRIVATE_READ:
return true;
default:
return false;
}
}
/**
* softleaf_is_device_exclusive() - Is this leaf entry a device-exclusive entry?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is a device-exclusive entry, otherwise false.
*/
static inline bool softleaf_is_device_exclusive(softleaf_t entry)
{
return softleaf_type(entry) == SOFTLEAF_DEVICE_EXCLUSIVE;
}
/**
* softleaf_is_hwpoison() - Is this leaf entry a hardware poison entry?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is a hardware poison entry, otherwise false.
*/
static inline bool softleaf_is_hwpoison(softleaf_t entry)
{
return softleaf_type(entry) == SOFTLEAF_HWPOISON;
}
/**
* softleaf_is_marker() - Is this leaf entry a marker?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is a marker entry, otherwise false.
*/
static inline bool softleaf_is_marker(softleaf_t entry)
{
return softleaf_type(entry) == SOFTLEAF_MARKER;
}
/**
* softleaf_to_marker() - Obtain marker associated with leaf entry.
* @entry: Leaf entry, softleaf_is_marker(@entry) must return true.
*
* Returns: Marker associated with the leaf entry.
*/
static inline pte_marker softleaf_to_marker(softleaf_t entry)
{
VM_WARN_ON_ONCE(!softleaf_is_marker(entry));
return swp_offset(entry) & PTE_MARKER_MASK;
}
/**
* softleaf_has_pfn() - Does this leaf entry encode a valid PFN number?
* @entry: Leaf entry.
*
* A pfn swap entry is a special type of swap entry that always has a pfn stored
* in the swap offset. They can either be used to represent unaddressable device
* memory, to restrict access to a page undergoing migration or to represent a
* pfn which has been hwpoisoned and unmapped.
*
* Returns: true if the leaf entry encodes a PFN, otherwise false.
*/
static inline bool softleaf_has_pfn(softleaf_t entry)
{
/* Make sure the swp offset can always store the needed fields. */
BUILD_BUG_ON(SWP_TYPE_SHIFT < SWP_PFN_BITS);
if (softleaf_is_migration(entry))
return true;
if (softleaf_is_device_private(entry))
return true;
if (softleaf_is_device_exclusive(entry))
return true;
if (softleaf_is_hwpoison(entry))
return true;
return false;
}
/**
* softleaf_to_pfn() - Obtain PFN encoded within leaf entry.
* @entry: Leaf entry, softleaf_has_pfn(@entry) must return true.
*
* Returns: The PFN associated with the leaf entry.
*/
static inline unsigned long softleaf_to_pfn(softleaf_t entry)
{
VM_WARN_ON_ONCE(!softleaf_has_pfn(entry));
/* Temporary until swp_entry_t eliminated. */
return swp_offset(entry) & SWP_PFN_MASK;
}
/**
* softleaf_to_page() - Obtains struct page for PFN encoded within leaf entry.
* @entry: Leaf entry, softleaf_has_pfn(@entry) must return true.
*
* Returns: Pointer to the struct page associated with the leaf entry's PFN.
*/
static inline struct page *softleaf_to_page(softleaf_t entry)
{
struct page *page = pfn_to_page(softleaf_to_pfn(entry));
VM_WARN_ON_ONCE(!softleaf_has_pfn(entry));
/*
* Any use of migration entries may only occur while the
* corresponding page is locked
*/
VM_WARN_ON_ONCE(softleaf_is_migration(entry) && !PageLocked(page));
return page;
}
/**
* softleaf_to_folio() - Obtains struct folio for PFN encoded within leaf entry.
* @entry: Leaf entry, softleaf_has_pfn(@entry) must return true.
*
* Returns: Pointer to the struct folio associated with the leaf entry's PFN.
*/
static inline struct folio *softleaf_to_folio(softleaf_t entry)
{
struct folio *folio = pfn_folio(softleaf_to_pfn(entry));
VM_WARN_ON_ONCE(!softleaf_has_pfn(entry));
/*
* Any use of migration entries may only occur while the
* corresponding folio is locked.
*/
VM_WARN_ON_ONCE(softleaf_is_migration(entry) &&
!folio_test_locked(folio));
return folio;
}
/**
* softleaf_is_poison_marker() - Is this leaf entry a poison marker?
* @entry: Leaf entry.
*
* The poison marker is set via UFFDIO_POISON. Userfaultfd-specific.
*
* Returns: true if the leaf entry is a poison marker, otherwise false.
*/
static inline bool softleaf_is_poison_marker(softleaf_t entry)
{
if (!softleaf_is_marker(entry))
return false;
return softleaf_to_marker(entry) & PTE_MARKER_POISONED;
}
/**
* softleaf_is_guard_marker() - Is this leaf entry a guard region marker?
* @entry: Leaf entry.
*
* Returns: true if the leaf entry is a guard marker, otherwise false.
*/
static inline bool softleaf_is_guard_marker(softleaf_t entry)
{
if (!softleaf_is_marker(entry))
return false;
return softleaf_to_marker(entry) & PTE_MARKER_GUARD;
}
/**
* softleaf_is_uffd_wp_marker() - Is this leaf entry a userfautlfd write protect
* marker?
* @entry: Leaf entry.
*
* Userfaultfd-specific.
*
* Returns: true if the leaf entry is a UFFD WP marker, otherwise false.
*/
static inline bool softleaf_is_uffd_wp_marker(softleaf_t entry)
{
if (!softleaf_is_marker(entry))
return false;
return softleaf_to_marker(entry) & PTE_MARKER_UFFD_WP;
}
#ifdef CONFIG_MIGRATION
/**
* softleaf_is_migration_young() - Does this migration entry contain an accessed
* bit?
* @entry: Leaf entry.
*
* If the architecture can support storing A/D bits in migration entries, this
* determines whether the accessed (or 'young') bit was set on the migrated page
* table entry.
*
* Returns: true if the entry contains an accessed bit, otherwise false.
*/
static inline bool softleaf_is_migration_young(softleaf_t entry)
{
VM_WARN_ON_ONCE(!softleaf_is_migration(entry));
if (migration_entry_supports_ad())
return swp_offset(entry) & SWP_MIG_YOUNG;
/* Keep the old behavior of aging page after migration */
return false;
}
/**
* softleaf_is_migration_dirty() - Does this migration entry contain a dirty bit?
* @entry: Leaf entry.
*
* If the architecture can support storing A/D bits in migration entries, this
* determines whether the dirty bit was set on the migrated page table entry.
*
* Returns: true if the entry contains a dirty bit, otherwise false.
*/
static inline bool softleaf_is_migration_dirty(softleaf_t entry)
{
VM_WARN_ON_ONCE(!softleaf_is_migration(entry));
if (migration_entry_supports_ad())
return swp_offset(entry) & SWP_MIG_DIRTY;
/* Keep the old behavior of clean page after migration */
return false;
}
#else /* CONFIG_MIGRATION */
static inline bool softleaf_is_migration_young(softleaf_t entry)
{
return false;
}
static inline bool softleaf_is_migration_dirty(softleaf_t entry)
{
return false;
}
#endif /* CONFIG_MIGRATION */
/**
* pte_is_marker() - Does the PTE entry encode a marker leaf entry?
* @pte: PTE entry.
*
* Returns: true if this PTE is a marker leaf entry, otherwise false.
*/
static inline bool pte_is_marker(pte_t pte)
{
return softleaf_is_marker(softleaf_from_pte(pte));
}
/**
* pte_is_uffd_wp_marker() - Does this PTE entry encode a userfaultfd write
* protect marker leaf entry?
* @pte: PTE entry.
*
* Returns: true if this PTE is a UFFD WP marker leaf entry, otherwise false.
*/
static inline bool pte_is_uffd_wp_marker(pte_t pte)
{
const softleaf_t entry = softleaf_from_pte(pte);
return softleaf_is_uffd_wp_marker(entry);
}
/**
* pte_is_uffd_marker() - Does this PTE entry encode a userfault-specific marker
* leaf entry?
* @entry: Leaf entry.
*
* It's useful to be able to determine which leaf entries encode UFFD-specific
* markers so we can handle these correctly.
*
* Returns: true if this PTE entry is a UFFD-specific marker, otherwise false.
*/
static inline bool pte_is_uffd_marker(pte_t pte)
{
const softleaf_t entry = softleaf_from_pte(pte);
if (!softleaf_is_marker(entry))
return false;
/* UFFD WP, poisoned swap entries are UFFD-handled. */
if (softleaf_is_uffd_wp_marker(entry))
return true;
if (softleaf_is_poison_marker(entry))
return true;
return false;
}
#if defined(CONFIG_ZONE_DEVICE) && defined(CONFIG_ARCH_ENABLE_THP_MIGRATION)
/**
* pmd_is_device_private_entry() - Check if PMD contains a device private swap
* entry.
* @pmd: The PMD to check.
*
* Returns true if the PMD contains a swap entry that represents a device private
* page mapping. This is used for zone device private pages that have been
* swapped out but still need special handling during various memory management
* operations.
*
* Return: true if PMD contains device private entry, false otherwise
*/
static inline bool pmd_is_device_private_entry(pmd_t pmd)
{
return softleaf_is_device_private(softleaf_from_pmd(pmd));
}
#else /* CONFIG_ZONE_DEVICE && CONFIG_ARCH_ENABLE_THP_MIGRATION */
static inline bool pmd_is_device_private_entry(pmd_t pmd)
{
return false;
}
#endif /* CONFIG_ZONE_DEVICE && CONFIG_ARCH_ENABLE_THP_MIGRATION */
/**
* pmd_is_migration_entry() - Does this PMD entry encode a migration entry?
* @pmd: PMD entry.
*
* Returns: true if the PMD encodes a migration entry, otherwise false.
*/
static inline bool pmd_is_migration_entry(pmd_t pmd)
{
return softleaf_is_migration(softleaf_from_pmd(pmd));
}
/**
* pmd_is_valid_softleaf() - Is this PMD entry a valid leaf entry?
* @pmd: PMD entry.
*
* PMD leaf entries are valid only if they are device private or migration
* entries. This function asserts that a PMD leaf entry is valid in this
* respect.
*
* Returns: true if the PMD entry is a valid leaf entry, otherwise false.
*/
static inline bool pmd_is_valid_softleaf(pmd_t pmd)
{
const softleaf_t entry = softleaf_from_pmd(pmd);
/* Only device private, migration entries valid for PMD. */
return softleaf_is_device_private(entry) ||
softleaf_is_migration(entry);
}
#endif /* CONFIG_MMU */
#endif /* _LINUX_LEAFOPS_H */

89
include/linux/memcontrol.h
View File

@ -52,6 +52,7 @@ enum memcg_memory_event {
MEMCG_SWAP_HIGH,
MEMCG_SWAP_MAX,
MEMCG_SWAP_FAIL,
MEMCG_SOCK_THROTTLED,
MEMCG_NR_MEMORY_EVENTS,
};
@ -956,17 +957,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,
void mem_cgroup_flush_stats(struct mem_cgroup *memcg);
void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg);
void __mod_lruvec_kmem_state(void *p, enum node_stat_item idx, int val);
static inline void mod_lruvec_kmem_state(void *p, enum node_stat_item idx,
int val)
{
unsigned long flags;
local_irq_save(flags);
__mod_lruvec_kmem_state(p, idx, val);
local_irq_restore(flags);
}
void mod_lruvec_kmem_state(void *p, enum node_stat_item idx, int val);
void count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx,
unsigned long count);
@ -1001,36 +992,8 @@ static inline void count_memcg_event_mm(struct mm_struct *mm,
count_memcg_events_mm(mm, idx, 1);
}
static inline void __memcg_memory_event(struct mem_cgroup *memcg,
enum memcg_memory_event event,
bool allow_spinning)
{
bool swap_event = event == MEMCG_SWAP_HIGH || event == MEMCG_SWAP_MAX ||
event == MEMCG_SWAP_FAIL;
/* For now only MEMCG_MAX can happen with !allow_spinning context. */
VM_WARN_ON_ONCE(!allow_spinning && event != MEMCG_MAX);
atomic_long_inc(&memcg->memory_events_local[event]);
if (!swap_event && allow_spinning)
cgroup_file_notify(&memcg->events_local_file);
do {
atomic_long_inc(&memcg->memory_events[event]);
if (allow_spinning) {
if (swap_event)
cgroup_file_notify(&memcg->swap_events_file);
else
cgroup_file_notify(&memcg->events_file);
}
if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
break;
if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_LOCAL_EVENTS)
break;
} while ((memcg = parent_mem_cgroup(memcg)) &&
!mem_cgroup_is_root(memcg));
}
void __memcg_memory_event(struct mem_cgroup *memcg,
enum memcg_memory_event event, bool allow_spinning);
static inline void memcg_memory_event(struct mem_cgroup *memcg,
enum memcg_memory_event event)
@ -1430,14 +1393,6 @@ static inline void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
{
}
static inline void __mod_lruvec_kmem_state(void *p, enum node_stat_item idx,
int val)
{
struct page *page = virt_to_head_page(p);
__mod_node_page_state(page_pgdat(page), idx, val);
}
static inline void mod_lruvec_kmem_state(void *p, enum node_stat_item idx,
int val)
{
@ -1497,16 +1452,6 @@ struct slabobj_ext {
#endif
} __aligned(8);
static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx)
{
__mod_lruvec_kmem_state(p, idx, 1);
}
static inline void __dec_lruvec_kmem_state(void *p, enum node_stat_item idx)
{
__mod_lruvec_kmem_state(p, idx, -1);
}
static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
{
struct mem_cgroup *memcg;
@ -1674,6 +1619,11 @@ int alloc_shrinker_info(struct mem_cgroup *memcg);
void free_shrinker_info(struct mem_cgroup *memcg);
void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id);
void reparent_shrinker_deferred(struct mem_cgroup *memcg);
static inline int shrinker_id(struct shrinker *shrinker)
{
return shrinker->id;
}
#else
#define mem_cgroup_sockets_enabled 0
@ -1705,6 +1655,11 @@ static inline void set_shrinker_bit(struct mem_cgroup *memcg,
int nid, int shrinker_id)
{
}
static inline int shrinker_id(struct shrinker *shrinker)
{
return -1;
}
#endif
#ifdef CONFIG_MEMCG
@ -1791,6 +1746,13 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,
bool mem_cgroup_node_allowed(struct mem_cgroup *memcg, int nid);
void mem_cgroup_show_protected_memory(struct mem_cgroup *memcg);
static inline bool memcg_is_dying(struct mem_cgroup *memcg)
{
return memcg ? css_is_dying(&memcg->css) : false;
}
#else
static inline bool mem_cgroup_kmem_disabled(void)
{
@ -1857,6 +1819,15 @@ static inline bool mem_cgroup_node_allowed(struct mem_cgroup *memcg, int nid)
{
return true;
}
static inline void mem_cgroup_show_protected_memory(struct mem_cgroup *memcg)
{
}
static inline bool memcg_is_dying(struct mem_cgroup *memcg)
{
return false;
}
#endif /* CONFIG_MEMCG */
#if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)

17
include/linux/memory-failure.h Normal file
View File

@ -0,0 +1,17 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_MEMORY_FAILURE_H
#define _LINUX_MEMORY_FAILURE_H
#include <linux/interval_tree.h>
struct pfn_address_space;
struct pfn_address_space {
struct interval_tree_node node;
struct address_space *mapping;
};
int register_pfn_address_space(struct pfn_address_space *pfn_space);
void unregister_pfn_address_space(struct pfn_address_space *pfn_space);
#endif /* _LINUX_MEMORY_FAILURE_H */

24
include/linux/memory.h
View File

@ -64,9 +64,19 @@ struct memory_group {
};
};
enum memory_block_state {
/* These states are exposed to userspace as text strings in sysfs */
MEM_ONLINE, /* exposed to userspace */
MEM_GOING_OFFLINE, /* exposed to userspace */
MEM_OFFLINE, /* exposed to userspace */
MEM_GOING_ONLINE,
MEM_CANCEL_ONLINE,
MEM_CANCEL_OFFLINE,
};
struct memory_block {
unsigned long start_section_nr;
unsigned long state; /* serialized by the dev->lock */
enum memory_block_state state; /* serialized by the dev->lock */
int online_type; /* for passing data to online routine */
int nid; /* NID for this memory block */
/*
@ -89,14 +99,6 @@ int arch_get_memory_phys_device(unsigned long start_pfn);
unsigned long memory_block_size_bytes(void);
int set_memory_block_size_order(unsigned int order);
/* These states are exposed to userspace as text strings in sysfs */
#define MEM_ONLINE (1<<0) /* exposed to userspace */
#define MEM_GOING_OFFLINE (1<<1) /* exposed to userspace */
#define MEM_OFFLINE (1<<2) /* exposed to userspace */
#define MEM_GOING_ONLINE (1<<3)
#define MEM_CANCEL_ONLINE (1<<4)
#define MEM_CANCEL_OFFLINE (1<<5)
struct memory_notify {
unsigned long start_pfn;
unsigned long nr_pages;
@ -130,7 +132,7 @@ static inline int register_memory_notifier(struct notifier_block *nb)
static inline void unregister_memory_notifier(struct notifier_block *nb)
{
}
static inline int memory_notify(unsigned long val, void *v)
static inline int memory_notify(enum memory_block_state state, void *v)
{
return 0;
}
@ -154,7 +156,7 @@ int create_memory_block_devices(unsigned long start, unsigned long size,
struct memory_group *group);
void remove_memory_block_devices(unsigned long start, unsigned long size);
extern void memory_dev_init(void);
extern int memory_notify(unsigned long val, void *v);
extern int memory_notify(enum memory_block_state state, void *v);
extern struct memory_block *find_memory_block(unsigned long section_nr);
typedef int (*walk_memory_blocks_func_t)(struct memory_block *, void *);
extern int walk_memory_blocks(unsigned long start, unsigned long size,

57
include/linux/memremap.h
View File

@ -76,11 +76,11 @@ enum memory_type {
struct dev_pagemap_ops {
/*
* Called once the page refcount reaches 0. The reference count will be
* Called once the folio refcount reaches 0. The reference count will be
* reset to one by the core code after the method is called to prepare
* for handing out the page again.
* for handing out the folio again.
*/
void (*page_free)(struct page *page);
void (*folio_free)(struct folio *folio);
/*
* Used for private (un-addressable) device memory only. Must migrate
@ -99,6 +99,13 @@ struct dev_pagemap_ops {
*/
int (*memory_failure)(struct dev_pagemap *pgmap, unsigned long pfn,
unsigned long nr_pages, int mf_flags);
/*
* Used for private (un-addressable) device memory only.
* This callback is used when a folio is split into
* a smaller folio
*/
void (*folio_split)(struct folio *head, struct folio *tail);
};
#define PGMAP_ALTMAP_VALID (1 << 0)
@ -176,6 +183,18 @@ static inline bool folio_is_pci_p2pdma(const struct folio *folio)
folio->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA;
}
static inline void *folio_zone_device_data(const struct folio *folio)
{
VM_WARN_ON_FOLIO(!folio_is_device_private(folio), folio);
return folio->page.zone_device_data;
}
static inline void folio_set_zone_device_data(struct folio *folio, void *data)
{
VM_WARN_ON_FOLIO(!folio_is_device_private(folio), folio);
folio->page.zone_device_data = data;
}
static inline bool is_pci_p2pdma_page(const struct page *page)
{
return IS_ENABLED(CONFIG_PCI_P2PDMA) &&
@ -205,7 +224,7 @@ static inline bool is_fsdax_page(const struct page *page)
}
#ifdef CONFIG_ZONE_DEVICE
void zone_device_page_init(struct page *page);
void zone_device_page_init(struct page *page, unsigned int order);
void *memremap_pages(struct dev_pagemap *pgmap, int nid);
void memunmap_pages(struct dev_pagemap *pgmap);
void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap);
@ -214,6 +233,31 @@ struct dev_pagemap *get_dev_pagemap(unsigned long pfn);
bool pgmap_pfn_valid(struct dev_pagemap *pgmap, unsigned long pfn);
unsigned long memremap_compat_align(void);
static inline void zone_device_folio_init(struct folio *folio, unsigned int order)
{
zone_device_page_init(&folio->page, order);
if (order)
folio_set_large_rmappable(folio);
}
static inline void zone_device_private_split_cb(struct folio *original_folio,
struct folio *new_folio)
{
if (folio_is_device_private(original_folio)) {
if (!original_folio->pgmap->ops->folio_split) {
if (new_folio) {
new_folio->pgmap = original_folio->pgmap;
new_folio->page.mapping =
original_folio->page.mapping;
}
} else {
original_folio->pgmap->ops->folio_split(original_folio,
new_folio);
}
}
}
#else
static inline void *devm_memremap_pages(struct device *dev,
struct dev_pagemap *pgmap)
@ -247,6 +291,11 @@ static inline unsigned long memremap_compat_align(void)
{
return PAGE_SIZE;
}
static inline void zone_device_private_split_cb(struct folio *original_folio,
struct folio *new_folio)
{
}
#endif /* CONFIG_ZONE_DEVICE */
static inline void put_dev_pagemap(struct dev_pagemap *pgmap)

4
include/linux/migrate.h
View File

@ -65,7 +65,7 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list);
int migrate_huge_page_move_mapping(struct address_space *mapping,
struct folio *dst, struct folio *src);
void migration_entry_wait_on_locked(swp_entry_t entry, spinlock_t *ptl)
void migration_entry_wait_on_locked(softleaf_t entry, spinlock_t *ptl)
__releases(ptl);
void folio_migrate_flags(struct folio *newfolio, struct folio *folio);
int folio_migrate_mapping(struct address_space *mapping,
@ -125,6 +125,7 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
#define MIGRATE_PFN_VALID (1UL << 0)
#define MIGRATE_PFN_MIGRATE (1UL << 1)
#define MIGRATE_PFN_WRITE (1UL << 3)
#define MIGRATE_PFN_COMPOUND (1UL << 4)
#define MIGRATE_PFN_SHIFT 6
static inline struct page *migrate_pfn_to_page(unsigned long mpfn)
@ -143,6 +144,7 @@ enum migrate_vma_direction {
MIGRATE_VMA_SELECT_SYSTEM = 1 << 0,
MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1,
MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2,
MIGRATE_VMA_SELECT_COMPOUND = 1 << 3,
};
struct migrate_vma {

699
include/linux/mm.h
View File

@ -105,6 +105,8 @@ extern int mmap_rnd_compat_bits __read_mostly;
# endif
#endif
#define INVALID_PHYS_ADDR (~(phys_addr_t)0)
#include <asm/page.h>
#include <asm/processor.h>
@ -273,180 +275,237 @@ extern unsigned int kobjsize(const void *objp);
* vm_flags in vm_area_struct, see mm_types.h.
* When changing, update also include/trace/events/mmflags.h
*/
#define VM_NONE 0x00000000
#define VM_READ 0x00000001 /* currently active flags */
#define VM_WRITE 0x00000002
#define VM_EXEC 0x00000004
#define VM_SHARED 0x00000008
/* mprotect() hardcodes VM_MAYREAD >> 4 == VM_READ, and so for r/w/x bits. */
#define VM_MAYREAD 0x00000010 /* limits for mprotect() etc */
#define VM_MAYWRITE 0x00000020
#define VM_MAYEXEC 0x00000040
#define VM_MAYSHARE 0x00000080
#define VM_GROWSDOWN 0x00000100 /* general info on the segment */
#ifdef CONFIG_MMU
#define VM_UFFD_MISSING 0x00000200 /* missing pages tracking */
#else /* CONFIG_MMU */
#define VM_MAYOVERLAY 0x00000200 /* nommu: R/O MAP_PRIVATE mapping that might overlay a file mapping */
#define VM_UFFD_MISSING 0
#endif /* CONFIG_MMU */
#define VM_PFNMAP 0x00000400 /* Page-ranges managed without "struct page", just pure PFN */
#define VM_UFFD_WP 0x00001000 /* wrprotect pages tracking */
#define VM_LOCKED 0x00002000
#define VM_IO 0x00004000 /* Memory mapped I/O or similar */
/* Used by sys_madvise() */
#define VM_SEQ_READ 0x00008000 /* App will access data sequentially */
#define VM_RAND_READ 0x00010000 /* App will not benefit from clustered reads */
#define VM_DONTCOPY 0x00020000 /* Do not copy this vma on fork */
#define VM_DONTEXPAND 0x00040000 /* Cannot expand with mremap() */
#define VM_LOCKONFAULT 0x00080000 /* Lock the pages covered when they are faulted in */
#define VM_ACCOUNT 0x00100000 /* Is a VM accounted object */
#define VM_NORESERVE 0x00200000 /* should the VM suppress accounting */
#define VM_HUGETLB 0x00400000 /* Huge TLB Page VM */
#define VM_SYNC 0x00800000 /* Synchronous page faults */
#define VM_ARCH_1 0x01000000 /* Architecture-specific flag */
#define VM_WIPEONFORK 0x02000000 /* Wipe VMA contents in child. */
#define VM_DONTDUMP 0x04000000 /* Do not include in the core dump */
#ifdef CONFIG_MEM_SOFT_DIRTY
# define VM_SOFTDIRTY 0x08000000 /* Not soft dirty clean area */
#else
# define VM_SOFTDIRTY 0
#endif
#define VM_MIXEDMAP 0x10000000 /* Can contain "struct page" and pure PFN pages */
#define VM_HUGEPAGE 0x20000000 /* MADV_HUGEPAGE marked this vma */
#define VM_NOHUGEPAGE 0x40000000 /* MADV_NOHUGEPAGE marked this vma */
#define VM_MERGEABLE BIT(31) /* KSM may merge identical pages */
#ifdef CONFIG_ARCH_USES_HIGH_VMA_FLAGS
#define VM_HIGH_ARCH_BIT_0 32 /* bit only usable on 64-bit architectures */
#define VM_HIGH_ARCH_BIT_1 33 /* bit only usable on 64-bit architectures */
#define VM_HIGH_ARCH_BIT_2 34 /* bit only usable on 64-bit architectures */
#define VM_HIGH_ARCH_BIT_3 35 /* bit only usable on 64-bit architectures */
#define VM_HIGH_ARCH_BIT_4 36 /* bit only usable on 64-bit architectures */
#define VM_HIGH_ARCH_BIT_5 37 /* bit only usable on 64-bit architectures */
#define VM_HIGH_ARCH_BIT_6 38 /* bit only usable on 64-bit architectures */
#define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0)
#define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1)
#define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2)
#define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3)
#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4)
#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5)
#define VM_HIGH_ARCH_6 BIT(VM_HIGH_ARCH_BIT_6)
#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */
#ifdef CONFIG_ARCH_HAS_PKEYS
# define VM_PKEY_SHIFT VM_HIGH_ARCH_BIT_0
# define VM_PKEY_BIT0 VM_HIGH_ARCH_0
# define VM_PKEY_BIT1 VM_HIGH_ARCH_1
# define VM_PKEY_BIT2 VM_HIGH_ARCH_2
#if CONFIG_ARCH_PKEY_BITS > 3
# define VM_PKEY_BIT3 VM_HIGH_ARCH_3
#else
# define VM_PKEY_BIT3 0
#endif
#if CONFIG_ARCH_PKEY_BITS > 4
# define VM_PKEY_BIT4 VM_HIGH_ARCH_4
#else
# define VM_PKEY_BIT4 0
#endif
#endif /* CONFIG_ARCH_HAS_PKEYS */
#ifdef CONFIG_X86_USER_SHADOW_STACK
/*
* VM_SHADOW_STACK should not be set with VM_SHARED because of lack of
* support core mm.
/**
* typedef vma_flag_t - specifies an individual VMA flag by bit number.
*
* These VMAs will get a single end guard page. This helps userspace protect
* itself from attacks. A single page is enough for current shadow stack archs
* (x86). See the comments near alloc_shstk() in arch/x86/kernel/shstk.c
* for more details on the guard size.
* This value is made type safe by sparse to avoid passing invalid flag values
* around.
*/
# define VM_SHADOW_STACK VM_HIGH_ARCH_5
#endif
typedef int __bitwise vma_flag_t;
#if defined(CONFIG_ARM64_GCS)
/*
* arm64's Guarded Control Stack implements similar functionality and
* has similar constraints to shadow stacks.
*/
# define VM_SHADOW_STACK VM_HIGH_ARCH_6
#define DECLARE_VMA_BIT(name, bitnum) \
VMA_ ## name ## _BIT = ((__force vma_flag_t)bitnum)
#define DECLARE_VMA_BIT_ALIAS(name, aliased) \
VMA_ ## name ## _BIT = (VMA_ ## aliased ## _BIT)
enum {
DECLARE_VMA_BIT(READ, 0),
DECLARE_VMA_BIT(WRITE, 1),
DECLARE_VMA_BIT(EXEC, 2),
DECLARE_VMA_BIT(SHARED, 3),
/* mprotect() hardcodes VM_MAYREAD >> 4 == VM_READ, and so for r/w/x bits. */
DECLARE_VMA_BIT(MAYREAD, 4), /* limits for mprotect() etc. */
DECLARE_VMA_BIT(MAYWRITE, 5),
DECLARE_VMA_BIT(MAYEXEC, 6),
DECLARE_VMA_BIT(MAYSHARE, 7),
DECLARE_VMA_BIT(GROWSDOWN, 8), /* general info on the segment */
#ifdef CONFIG_MMU
DECLARE_VMA_BIT(UFFD_MISSING, 9),/* missing pages tracking */
#else
/* nommu: R/O MAP_PRIVATE mapping that might overlay a file mapping */
DECLARE_VMA_BIT(MAYOVERLAY, 9),
#endif /* CONFIG_MMU */
/* Page-ranges managed without "struct page", just pure PFN */
DECLARE_VMA_BIT(PFNMAP, 10),
DECLARE_VMA_BIT(MAYBE_GUARD, 11),
DECLARE_VMA_BIT(UFFD_WP, 12), /* wrprotect pages tracking */
DECLARE_VMA_BIT(LOCKED, 13),
DECLARE_VMA_BIT(IO, 14), /* Memory mapped I/O or similar */
DECLARE_VMA_BIT(SEQ_READ, 15), /* App will access data sequentially */
DECLARE_VMA_BIT(RAND_READ, 16), /* App will not benefit from clustered reads */
DECLARE_VMA_BIT(DONTCOPY, 17), /* Do not copy this vma on fork */
DECLARE_VMA_BIT(DONTEXPAND, 18),/* Cannot expand with mremap() */
DECLARE_VMA_BIT(LOCKONFAULT, 19),/* Lock pages covered when faulted in */
DECLARE_VMA_BIT(ACCOUNT, 20), /* Is a VM accounted object */
DECLARE_VMA_BIT(NORESERVE, 21), /* should the VM suppress accounting */
DECLARE_VMA_BIT(HUGETLB, 22), /* Huge TLB Page VM */
DECLARE_VMA_BIT(SYNC, 23), /* Synchronous page faults */
DECLARE_VMA_BIT(ARCH_1, 24), /* Architecture-specific flag */
DECLARE_VMA_BIT(WIPEONFORK, 25),/* Wipe VMA contents in child. */
DECLARE_VMA_BIT(DONTDUMP, 26), /* Do not include in the core dump */
DECLARE_VMA_BIT(SOFTDIRTY, 27), /* NOT soft dirty clean area */
DECLARE_VMA_BIT(MIXEDMAP, 28), /* Can contain struct page and pure PFN pages */
DECLARE_VMA_BIT(HUGEPAGE, 29), /* MADV_HUGEPAGE marked this vma */
DECLARE_VMA_BIT(NOHUGEPAGE, 30),/* MADV_NOHUGEPAGE marked this vma */
DECLARE_VMA_BIT(MERGEABLE, 31), /* KSM may merge identical pages */
/* These bits are reused, we define specific uses below. */
DECLARE_VMA_BIT(HIGH_ARCH_0, 32),
DECLARE_VMA_BIT(HIGH_ARCH_1, 33),
DECLARE_VMA_BIT(HIGH_ARCH_2, 34),
DECLARE_VMA_BIT(HIGH_ARCH_3, 35),
DECLARE_VMA_BIT(HIGH_ARCH_4, 36),
DECLARE_VMA_BIT(HIGH_ARCH_5, 37),
DECLARE_VMA_BIT(HIGH_ARCH_6, 38),
/*
* This flag is used to connect VFIO to arch specific KVM code. It
* indicates that the memory under this VMA is safe for use with any
* non-cachable memory type inside KVM. Some VFIO devices, on some
* platforms, are thought to be unsafe and can cause machine crashes
* if KVM does not lock down the memory type.
*/
DECLARE_VMA_BIT(ALLOW_ANY_UNCACHED, 39),
#ifdef CONFIG_PPC32
DECLARE_VMA_BIT_ALIAS(DROPPABLE, ARCH_1),
#else
DECLARE_VMA_BIT(DROPPABLE, 40),
#endif
#ifndef VM_SHADOW_STACK
# define VM_SHADOW_STACK VM_NONE
DECLARE_VMA_BIT(UFFD_MINOR, 41),
DECLARE_VMA_BIT(SEALED, 42),
/* Flags that reuse flags above. */
DECLARE_VMA_BIT_ALIAS(PKEY_BIT0, HIGH_ARCH_0),
DECLARE_VMA_BIT_ALIAS(PKEY_BIT1, HIGH_ARCH_1),
DECLARE_VMA_BIT_ALIAS(PKEY_BIT2, HIGH_ARCH_2),
DECLARE_VMA_BIT_ALIAS(PKEY_BIT3, HIGH_ARCH_3),
DECLARE_VMA_BIT_ALIAS(PKEY_BIT4, HIGH_ARCH_4),
#if defined(CONFIG_X86_USER_SHADOW_STACK)
/*
* VM_SHADOW_STACK should not be set with VM_SHARED because of lack of
* support core mm.
*
* These VMAs will get a single end guard page. This helps userspace
* protect itself from attacks. A single page is enough for current
* shadow stack archs (x86). See the comments near alloc_shstk() in
* arch/x86/kernel/shstk.c for more details on the guard size.
*/
DECLARE_VMA_BIT_ALIAS(SHADOW_STACK, HIGH_ARCH_5),
#elif defined(CONFIG_ARM64_GCS)
/*
* arm64's Guarded Control Stack implements similar functionality and
* has similar constraints to shadow stacks.
*/
DECLARE_VMA_BIT_ALIAS(SHADOW_STACK, HIGH_ARCH_6),
#endif
DECLARE_VMA_BIT_ALIAS(SAO, ARCH_1), /* Strong Access Ordering (powerpc) */
DECLARE_VMA_BIT_ALIAS(GROWSUP, ARCH_1), /* parisc */
DECLARE_VMA_BIT_ALIAS(SPARC_ADI, ARCH_1), /* sparc64 */
DECLARE_VMA_BIT_ALIAS(ARM64_BTI, ARCH_1), /* arm64 */
DECLARE_VMA_BIT_ALIAS(ARCH_CLEAR, ARCH_1), /* sparc64, arm64 */
DECLARE_VMA_BIT_ALIAS(MAPPED_COPY, ARCH_1), /* !CONFIG_MMU */
DECLARE_VMA_BIT_ALIAS(MTE, HIGH_ARCH_4), /* arm64 */
DECLARE_VMA_BIT_ALIAS(MTE_ALLOWED, HIGH_ARCH_5),/* arm64 */
#ifdef CONFIG_STACK_GROWSUP
DECLARE_VMA_BIT_ALIAS(STACK, GROWSUP),
DECLARE_VMA_BIT_ALIAS(STACK_EARLY, GROWSDOWN),
#else
DECLARE_VMA_BIT_ALIAS(STACK, GROWSDOWN),
#endif
};
#undef DECLARE_VMA_BIT
#undef DECLARE_VMA_BIT_ALIAS
#define INIT_VM_FLAG(name) BIT((__force int) VMA_ ## name ## _BIT)
#define VM_READ INIT_VM_FLAG(READ)
#define VM_WRITE INIT_VM_FLAG(WRITE)
#define VM_EXEC INIT_VM_FLAG(EXEC)
#define VM_SHARED INIT_VM_FLAG(SHARED)
#define VM_MAYREAD INIT_VM_FLAG(MAYREAD)
#define VM_MAYWRITE INIT_VM_FLAG(MAYWRITE)
#define VM_MAYEXEC INIT_VM_FLAG(MAYEXEC)
#define VM_MAYSHARE INIT_VM_FLAG(MAYSHARE)
#define VM_GROWSDOWN INIT_VM_FLAG(GROWSDOWN)
#ifdef CONFIG_MMU
#define VM_UFFD_MISSING INIT_VM_FLAG(UFFD_MISSING)
#else
#define VM_UFFD_MISSING VM_NONE
#define VM_MAYOVERLAY INIT_VM_FLAG(MAYOVERLAY)
#endif
#define VM_PFNMAP INIT_VM_FLAG(PFNMAP)
#define VM_MAYBE_GUARD INIT_VM_FLAG(MAYBE_GUARD)
#define VM_UFFD_WP INIT_VM_FLAG(UFFD_WP)
#define VM_LOCKED INIT_VM_FLAG(LOCKED)
#define VM_IO INIT_VM_FLAG(IO)
#define VM_SEQ_READ INIT_VM_FLAG(SEQ_READ)
#define VM_RAND_READ INIT_VM_FLAG(RAND_READ)
#define VM_DONTCOPY INIT_VM_FLAG(DONTCOPY)
#define VM_DONTEXPAND INIT_VM_FLAG(DONTEXPAND)
#define VM_LOCKONFAULT INIT_VM_FLAG(LOCKONFAULT)
#define VM_ACCOUNT INIT_VM_FLAG(ACCOUNT)
#define VM_NORESERVE INIT_VM_FLAG(NORESERVE)
#define VM_HUGETLB INIT_VM_FLAG(HUGETLB)
#define VM_SYNC INIT_VM_FLAG(SYNC)
#define VM_ARCH_1 INIT_VM_FLAG(ARCH_1)
#define VM_WIPEONFORK INIT_VM_FLAG(WIPEONFORK)
#define VM_DONTDUMP INIT_VM_FLAG(DONTDUMP)
#ifdef CONFIG_MEM_SOFT_DIRTY
#define VM_SOFTDIRTY INIT_VM_FLAG(SOFTDIRTY)
#else
#define VM_SOFTDIRTY VM_NONE
#endif
#define VM_MIXEDMAP INIT_VM_FLAG(MIXEDMAP)
#define VM_HUGEPAGE INIT_VM_FLAG(HUGEPAGE)
#define VM_NOHUGEPAGE INIT_VM_FLAG(NOHUGEPAGE)
#define VM_MERGEABLE INIT_VM_FLAG(MERGEABLE)
#define VM_STACK INIT_VM_FLAG(STACK)
#ifdef CONFIG_STACK_GROWS_UP
#define VM_STACK_EARLY INIT_VM_FLAG(STACK_EARLY)
#else
#define VM_STACK_EARLY VM_NONE
#endif
#ifdef CONFIG_ARCH_HAS_PKEYS
#define VM_PKEY_SHIFT ((__force int)VMA_HIGH_ARCH_0_BIT)
/* Despite the naming, these are FLAGS not bits. */
#define VM_PKEY_BIT0 INIT_VM_FLAG(PKEY_BIT0)
#define VM_PKEY_BIT1 INIT_VM_FLAG(PKEY_BIT1)
#define VM_PKEY_BIT2 INIT_VM_FLAG(PKEY_BIT2)
#if CONFIG_ARCH_PKEY_BITS > 3
#define VM_PKEY_BIT3 INIT_VM_FLAG(PKEY_BIT3)
#else
#define VM_PKEY_BIT3 VM_NONE
#endif /* CONFIG_ARCH_PKEY_BITS > 3 */
#if CONFIG_ARCH_PKEY_BITS > 4
#define VM_PKEY_BIT4 INIT_VM_FLAG(PKEY_BIT4)
#else
#define VM_PKEY_BIT4 VM_NONE
#endif /* CONFIG_ARCH_PKEY_BITS > 4 */
#endif /* CONFIG_ARCH_HAS_PKEYS */
#if defined(CONFIG_X86_USER_SHADOW_STACK) || defined(CONFIG_ARM64_GCS)
#define VM_SHADOW_STACK INIT_VM_FLAG(SHADOW_STACK)
#else
#define VM_SHADOW_STACK VM_NONE
#endif
#if defined(CONFIG_PPC64)
# define VM_SAO VM_ARCH_1 /* Strong Access Ordering (powerpc) */
#define VM_SAO INIT_VM_FLAG(SAO)
#elif defined(CONFIG_PARISC)
# define VM_GROWSUP VM_ARCH_1
#define VM_GROWSUP INIT_VM_FLAG(GROWSUP)
#elif defined(CONFIG_SPARC64)
# define VM_SPARC_ADI VM_ARCH_1 /* Uses ADI tag for access control */
# define VM_ARCH_CLEAR VM_SPARC_ADI
#define VM_SPARC_ADI INIT_VM_FLAG(SPARC_ADI)
#define VM_ARCH_CLEAR INIT_VM_FLAG(ARCH_CLEAR)
#elif defined(CONFIG_ARM64)
# define VM_ARM64_BTI VM_ARCH_1 /* BTI guarded page, a.k.a. GP bit */
# define VM_ARCH_CLEAR VM_ARM64_BTI
#define VM_ARM64_BTI INIT_VM_FLAG(ARM64_BTI)
#define VM_ARCH_CLEAR INIT_VM_FLAG(ARCH_CLEAR)
#elif !defined(CONFIG_MMU)
# define VM_MAPPED_COPY VM_ARCH_1 /* T if mapped copy of data (nommu mmap) */
#define VM_MAPPED_COPY INIT_VM_FLAG(MAPPED_COPY)
#endif
#if defined(CONFIG_ARM64_MTE)
# define VM_MTE VM_HIGH_ARCH_4 /* Use Tagged memory for access control */
# define VM_MTE_ALLOWED VM_HIGH_ARCH_5 /* Tagged memory permitted */
#else
# define VM_MTE VM_NONE
# define VM_MTE_ALLOWED VM_NONE
#endif
#ifndef VM_GROWSUP
# define VM_GROWSUP VM_NONE
#define VM_GROWSUP VM_NONE
#endif
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
# define VM_UFFD_MINOR_BIT 41
# define VM_UFFD_MINOR BIT(VM_UFFD_MINOR_BIT) /* UFFD minor faults */
#else /* !CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */
# define VM_UFFD_MINOR VM_NONE
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */
/*
* This flag is used to connect VFIO to arch specific KVM code. It
* indicates that the memory under this VMA is safe for use with any
* non-cachable memory type inside KVM. Some VFIO devices, on some
* platforms, are thought to be unsafe and can cause machine crashes
* if KVM does not lock down the memory type.
*/
#ifdef CONFIG_64BIT
#define VM_ALLOW_ANY_UNCACHED_BIT 39
#define VM_ALLOW_ANY_UNCACHED BIT(VM_ALLOW_ANY_UNCACHED_BIT)
#ifdef CONFIG_ARM64_MTE
#define VM_MTE INIT_VM_FLAG(MTE)
#define VM_MTE_ALLOWED INIT_VM_FLAG(MTE_ALLOWED)
#else
#define VM_ALLOW_ANY_UNCACHED VM_NONE
#define VM_MTE VM_NONE
#define VM_MTE_ALLOWED VM_NONE
#endif
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
#define VM_UFFD_MINOR INIT_VM_FLAG(UFFD_MINOR)
#else
#define VM_UFFD_MINOR VM_NONE
#endif
#ifdef CONFIG_64BIT
#define VM_DROPPABLE_BIT 40
#define VM_DROPPABLE BIT(VM_DROPPABLE_BIT)
#elif defined(CONFIG_PPC32)
#define VM_DROPPABLE VM_ARCH_1
#define VM_ALLOW_ANY_UNCACHED INIT_VM_FLAG(ALLOW_ANY_UNCACHED)
#define VM_SEALED INIT_VM_FLAG(SEALED)
#else
#define VM_ALLOW_ANY_UNCACHED VM_NONE
#define VM_SEALED VM_NONE
#endif
#if defined(CONFIG_64BIT) || defined(CONFIG_PPC32)
#define VM_DROPPABLE INIT_VM_FLAG(DROPPABLE)
#else
#define VM_DROPPABLE VM_NONE
#endif
#ifdef CONFIG_64BIT
#define VM_SEALED_BIT 42
#define VM_SEALED BIT(VM_SEALED_BIT)
#else
#define VM_SEALED VM_NONE
#endif
/* Bits set in the VMA until the stack is in its final location */
#define VM_STACK_INCOMPLETE_SETUP (VM_RAND_READ | VM_SEQ_READ | VM_STACK_EARLY)
@ -470,12 +529,10 @@ extern unsigned int kobjsize(const void *objp);
#define VM_STARTGAP_FLAGS (VM_GROWSDOWN | VM_SHADOW_STACK)
#ifdef CONFIG_STACK_GROWSUP
#define VM_STACK VM_GROWSUP
#define VM_STACK_EARLY VM_GROWSDOWN
#ifdef CONFIG_MSEAL_SYSTEM_MAPPINGS
#define VM_SEALED_SYSMAP VM_SEALED
#else
#define VM_STACK VM_GROWSDOWN
#define VM_STACK_EARLY 0
#define VM_SEALED_SYSMAP VM_NONE
#endif
#define VM_STACK_FLAGS (VM_STACK | VM_STACK_DEFAULT_FLAGS | VM_ACCOUNT)
@ -483,12 +540,26 @@ extern unsigned int kobjsize(const void *objp);
/* VMA basic access permission flags */
#define VM_ACCESS_FLAGS (VM_READ | VM_WRITE | VM_EXEC)
/*
* Special vmas that are non-mergable, non-mlock()able.
*/
#define VM_SPECIAL (VM_IO | VM_DONTEXPAND | VM_PFNMAP | VM_MIXEDMAP)
/*
* Physically remapped pages are special. Tell the
* rest of the world about it:
* VM_IO tells people not to look at these pages
* (accesses can have side effects).
* VM_PFNMAP tells the core MM that the base pages are just
* raw PFN mappings, and do not have a "struct page" associated
* with them.
* VM_DONTEXPAND
* Disable vma merging and expanding with mremap().
* VM_DONTDUMP
* Omit vma from core dump, even when VM_IO turned off.
*/
#define VM_REMAP_FLAGS (VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP)
/* This mask prevents VMA from being scanned with khugepaged */
#define VM_NO_KHUGEPAGED (VM_SPECIAL | VM_HUGETLB)
@ -498,12 +569,68 @@ extern unsigned int kobjsize(const void *objp);
/* This mask represents all the VMA flag bits used by mlock */
#define VM_LOCKED_MASK (VM_LOCKED | VM_LOCKONFAULT)
/* These flags can be updated atomically via VMA/mmap read lock. */
#define VM_ATOMIC_SET_ALLOWED VM_MAYBE_GUARD
/* Arch-specific flags to clear when updating VM flags on protection change */
#ifndef VM_ARCH_CLEAR
# define VM_ARCH_CLEAR VM_NONE
#define VM_ARCH_CLEAR VM_NONE
#endif
#define VM_FLAGS_CLEAR (ARCH_VM_PKEY_FLAGS | VM_ARCH_CLEAR)
/*
* Flags which should be 'sticky' on merge - that is, flags which, when one VMA
* possesses it but the other does not, the merged VMA should nonetheless have
* applied to it:
*
* VM_SOFTDIRTY - if a VMA is marked soft-dirty, that is has not had its
* references cleared via /proc/$pid/clear_refs, any merged VMA
* should be considered soft-dirty also as it operates at a VMA
* granularity.
*
* VM_MAYBE_GUARD - If a VMA may have guard regions in place it implies that
* mapped page tables may contain metadata not described by the
* VMA and thus any merged VMA may also contain this metadata,
* and thus we must make this flag sticky.
*/
#define VM_STICKY (VM_SOFTDIRTY | VM_MAYBE_GUARD)
/*
* VMA flags we ignore for the purposes of merge, i.e. one VMA possessing one
* of these flags and the other not does not preclude a merge.
*
* VM_STICKY - When merging VMAs, VMA flags must match, unless they are
* 'sticky'. If any sticky flags exist in either VMA, we simply
* set all of them on the merged VMA.
*/
#define VM_IGNORE_MERGE VM_STICKY
/*
* Flags which should result in page tables being copied on fork. These are
* flags which indicate that the VMA maps page tables which cannot be
* reconsistuted upon page fault, so necessitate page table copying upon
*
* VM_PFNMAP / VM_MIXEDMAP - These contain kernel-mapped data which cannot be
* reasonably reconstructed on page fault.
*
* VM_UFFD_WP - Encodes metadata about an installed uffd
* write protect handler, which cannot be
* reconstructed on page fault.
*
* We always copy pgtables when dst_vma has uffd-wp
* enabled even if it's file-backed
* (e.g. shmem). Because when uffd-wp is enabled,
* pgtable contains uffd-wp protection information,
* that's something we can't retrieve from page cache,
* and skip copying will lose those info.
*
* VM_MAYBE_GUARD - Could contain page guard region markers which
* by design are a property of the page tables
* only and thus cannot be reconstructed on page
* fault.
*/
#define VM_COPY_ON_FORK (VM_PFNMAP | VM_MIXEDMAP | VM_UFFD_WP | VM_MAYBE_GUARD)
/*
* mapping from the currently active vm_flags protection bits (the
* low four bits) to a page protection mask..
@ -783,7 +910,9 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
static inline void vm_flags_init(struct vm_area_struct *vma,
vm_flags_t flags)
{
ACCESS_PRIVATE(vma, __vm_flags) = flags;
VM_WARN_ON_ONCE(!pgtable_supports_soft_dirty() && (flags & VM_SOFTDIRTY));
vma_flags_clear_all(&vma->flags);
vma_flags_overwrite_word(&vma->flags, flags);
}
/*
@ -794,6 +923,7 @@ static inline void vm_flags_init(struct vm_area_struct *vma,
static inline void vm_flags_reset(struct vm_area_struct *vma,
vm_flags_t flags)
{
VM_WARN_ON_ONCE(!pgtable_supports_soft_dirty() && (flags & VM_SOFTDIRTY));
vma_assert_write_locked(vma);
vm_flags_init(vma, flags);
}
@ -802,21 +932,33 @@ static inline void vm_flags_reset_once(struct vm_area_struct *vma,
vm_flags_t flags)
{
vma_assert_write_locked(vma);
WRITE_ONCE(ACCESS_PRIVATE(vma, __vm_flags), flags);
/*
* If VMA flags exist beyond the first system word, also clear these. It
* is assumed the write once behaviour is required only for the first
* system word.
*/
if (NUM_VMA_FLAG_BITS > BITS_PER_LONG) {
unsigned long *bitmap = ACCESS_PRIVATE(&vma->flags, __vma_flags);
bitmap_zero(&bitmap[1], NUM_VMA_FLAG_BITS - BITS_PER_LONG);
}
vma_flags_overwrite_word_once(&vma->flags, flags);
}
static inline void vm_flags_set(struct vm_area_struct *vma,
vm_flags_t flags)
{
vma_start_write(vma);
ACCESS_PRIVATE(vma, __vm_flags) |= flags;
vma_flags_set_word(&vma->flags, flags);
}
static inline void vm_flags_clear(struct vm_area_struct *vma,
vm_flags_t flags)
{
VM_WARN_ON_ONCE(!pgtable_supports_soft_dirty() && (flags & VM_SOFTDIRTY));
vma_start_write(vma);
ACCESS_PRIVATE(vma, __vm_flags) &= ~flags;
vma_flags_clear_word(&vma->flags, flags);
}
/*
@ -840,6 +982,51 @@ static inline void vm_flags_mod(struct vm_area_struct *vma,
__vm_flags_mod(vma, set, clear);
}
static inline bool __vma_flag_atomic_valid(struct vm_area_struct *vma,
vma_flag_t bit)
{
const vm_flags_t mask = BIT((__force int)bit);
/* Only specific flags are permitted */
if (WARN_ON_ONCE(!(mask & VM_ATOMIC_SET_ALLOWED)))
return false;
return true;
}
/*
* Set VMA flag atomically. Requires only VMA/mmap read lock. Only specific
* valid flags are allowed to do this.
*/
static inline void vma_flag_set_atomic(struct vm_area_struct *vma,
vma_flag_t bit)
{
unsigned long *bitmap = ACCESS_PRIVATE(&vma->flags, __vma_flags);
/* mmap read lock/VMA read lock must be held. */
if (!rwsem_is_locked(&vma->vm_mm->mmap_lock))
vma_assert_locked(vma);
if (__vma_flag_atomic_valid(vma, bit))
set_bit((__force int)bit, bitmap);
}
/*
* Test for VMA flag atomically. Requires no locks. Only specific valid flags
* are allowed to do this.
*
* This is necessarily racey, so callers must ensure that serialisation is
* achieved through some other means, or that races are permissible.
*/
static inline bool vma_flag_test_atomic(struct vm_area_struct *vma,
vma_flag_t bit)
{
if (__vma_flag_atomic_valid(vma, bit))
return test_bit((__force int)bit, &vma->vm_flags);
return false;
}
static inline void vma_set_anonymous(struct vm_area_struct *vma)
{
vma->vm_ops = NULL;
@ -2438,7 +2625,7 @@ static inline void zap_vma_pages(struct vm_area_struct *vma)
}
void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
struct vm_area_struct *start_vma, unsigned long start,
unsigned long end, unsigned long tree_end, bool mm_wr_locked);
unsigned long end, unsigned long tree_end);
struct mmu_notifier_range;
@ -2922,6 +3109,7 @@ static inline pmd_t *pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long a
#endif /* CONFIG_MMU */
enum pt_flags {
PT_kernel = PG_referenced,
PT_reserved = PG_reserved,
/* High bits are used for zone/node/section */
};
@ -2947,6 +3135,46 @@ static inline bool pagetable_is_reserved(struct ptdesc *pt)
return test_bit(PT_reserved, &pt->pt_flags.f);
}
/**
* ptdesc_set_kernel - Mark a ptdesc used to map the kernel
* @ptdesc: The ptdesc to be marked
*
* Kernel page tables often need special handling. Set a flag so that
* the handling code knows this ptdesc will not be used for userspace.
*/
static inline void ptdesc_set_kernel(struct ptdesc *ptdesc)
{
set_bit(PT_kernel, &ptdesc->pt_flags.f);
}
/**
* ptdesc_clear_kernel - Mark a ptdesc as no longer used to map the kernel
* @ptdesc: The ptdesc to be unmarked
*
* Use when the ptdesc is no longer used to map the kernel and no longer
* needs special handling.
*/
static inline void ptdesc_clear_kernel(struct ptdesc *ptdesc)
{
/*
* Note: the 'PG_referenced' bit does not strictly need to be
* cleared before freeing the page. But this is nice for
* symmetry.
*/
clear_bit(PT_kernel, &ptdesc->pt_flags.f);
}
/**
* ptdesc_test_kernel - Check if a ptdesc is used to map the kernel
* @ptdesc: The ptdesc being tested
*
* Call to tell if the ptdesc used to map the kernel.
*/
static inline bool ptdesc_test_kernel(const struct ptdesc *ptdesc)
{
return test_bit(PT_kernel, &ptdesc->pt_flags.f);
}
/**
* pagetable_alloc - Allocate pagetables
* @gfp: GFP flags
@ -2965,6 +3193,21 @@ static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int orde
}
#define pagetable_alloc(...) alloc_hooks(pagetable_alloc_noprof(__VA_ARGS__))
static inline void __pagetable_free(struct ptdesc *pt)
{
struct page *page = ptdesc_page(pt);
__free_pages(page, compound_order(page));
}
#ifdef CONFIG_ASYNC_KERNEL_PGTABLE_FREE
void pagetable_free_kernel(struct ptdesc *pt);
#else
static inline void pagetable_free_kernel(struct ptdesc *pt)
{
__pagetable_free(pt);
}
#endif
/**
* pagetable_free - Free pagetables
* @pt: The page table descriptor
@ -2974,9 +3217,12 @@ static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int orde
*/
static inline void pagetable_free(struct ptdesc *pt)
{
struct page *page = ptdesc_page(pt);
__free_pages(page, compound_order(page));
if (ptdesc_test_kernel(pt)) {
ptdesc_clear_kernel(pt);
pagetable_free_kernel(pt);
} else {
__pagetable_free(pt);
}
}
#if defined(CONFIG_SPLIT_PTE_PTLOCKS)
@ -3560,6 +3806,90 @@ static inline unsigned long vma_pages(const struct vm_area_struct *vma)
return (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
}
static inline unsigned long vma_desc_size(const struct vm_area_desc *desc)
{
return desc->end - desc->start;
}
static inline unsigned long vma_desc_pages(const struct vm_area_desc *desc)
{
return vma_desc_size(desc) >> PAGE_SHIFT;
}
/**
* mmap_action_remap - helper for mmap_prepare hook to specify that a pure PFN
* remap is required.
* @desc: The VMA descriptor for the VMA requiring remap.
* @start: The virtual address to start the remap from, must be within the VMA.
* @start_pfn: The first PFN in the range to remap.
* @size: The size of the range to remap, in bytes, at most spanning to the end
* of the VMA.
*/
static inline void mmap_action_remap(struct vm_area_desc *desc,
unsigned long start,
unsigned long start_pfn,
unsigned long size)
{
struct mmap_action *action = &desc->action;
/* [start, start + size) must be within the VMA. */
WARN_ON_ONCE(start < desc->start || start >= desc->end);
WARN_ON_ONCE(start + size > desc->end);
action->type = MMAP_REMAP_PFN;
action->remap.start = start;
action->remap.start_pfn = start_pfn;
action->remap.size = size;
action->remap.pgprot = desc->page_prot;
}
/**
* mmap_action_remap_full - helper for mmap_prepare hook to specify that the
* entirety of a VMA should be PFN remapped.
* @desc: The VMA descriptor for the VMA requiring remap.
* @start_pfn: The first PFN in the range to remap.
*/
static inline void mmap_action_remap_full(struct vm_area_desc *desc,
unsigned long start_pfn)
{
mmap_action_remap(desc, desc->start, start_pfn, vma_desc_size(desc));
}
/**
* mmap_action_ioremap - helper for mmap_prepare hook to specify that a pure PFN
* I/O remap is required.
* @desc: The VMA descriptor for the VMA requiring remap.
* @start: The virtual address to start the remap from, must be within the VMA.
* @start_pfn: The first PFN in the range to remap.
* @size: The size of the range to remap, in bytes, at most spanning to the end
* of the VMA.
*/
static inline void mmap_action_ioremap(struct vm_area_desc *desc,
unsigned long start,
unsigned long start_pfn,
unsigned long size)
{
mmap_action_remap(desc, start, start_pfn, size);
desc->action.type = MMAP_IO_REMAP_PFN;
}
/**
* mmap_action_ioremap_full - helper for mmap_prepare hook to specify that the
* entirety of a VMA should be PFN I/O remapped.
* @desc: The VMA descriptor for the VMA requiring remap.
* @start_pfn: The first PFN in the range to remap.
*/
static inline void mmap_action_ioremap_full(struct vm_area_desc *desc,
unsigned long start_pfn)
{
mmap_action_ioremap(desc, desc->start, start_pfn, vma_desc_size(desc));
}
void mmap_action_prepare(struct mmap_action *action,
struct vm_area_desc *desc);
int mmap_action_complete(struct mmap_action *action,
struct vm_area_struct *vma);
/* Look up the first VMA which exactly match the interval vm_start ... vm_end */
static inline struct vm_area_struct *find_exact_vma(struct mm_struct *mm,
unsigned long vm_start, unsigned long vm_end)
@ -3601,10 +3931,9 @@ unsigned long change_prot_numa(struct vm_area_struct *vma,
struct vm_area_struct *find_extend_vma_locked(struct mm_struct *,
unsigned long addr);
int remap_pfn_range(struct vm_area_struct *, unsigned long addr,
unsigned long pfn, unsigned long size, pgprot_t);
int remap_pfn_range_notrack(struct vm_area_struct *vma, unsigned long addr,
unsigned long pfn, unsigned long size, pgprot_t prot);
int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr,
unsigned long pfn, unsigned long size, pgprot_t pgprot);
int vm_insert_page(struct vm_area_struct *, unsigned long addr, struct page *);
int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr,
struct page **pages, unsigned long *num);
@ -3637,15 +3966,24 @@ static inline vm_fault_t vmf_insert_page(struct vm_area_struct *vma,
return VM_FAULT_NOPAGE;
}
#ifndef io_remap_pfn_range
static inline int io_remap_pfn_range(struct vm_area_struct *vma,
unsigned long addr, unsigned long pfn,
unsigned long size, pgprot_t prot)
#ifndef io_remap_pfn_range_pfn
static inline unsigned long io_remap_pfn_range_pfn(unsigned long pfn,
unsigned long size)
{
return remap_pfn_range(vma, addr, pfn, size, pgprot_decrypted(prot));
return pfn;
}
#endif
static inline int io_remap_pfn_range(struct vm_area_struct *vma,
unsigned long addr, unsigned long orig_pfn,
unsigned long size, pgprot_t orig_prot)
{
const unsigned long pfn = io_remap_pfn_range_pfn(orig_pfn, size);
const pgprot_t prot = pgprot_decrypted(orig_prot);
return remap_pfn_range(vma, addr, pfn, size, prot);
}
static inline vm_fault_t vmf_error(int err)
{
if (err == -ENOMEM)
@ -4094,6 +4432,7 @@ enum mf_action_page_type {
MF_MSG_DAX,
MF_MSG_UNSPLIT_THP,
MF_MSG_ALREADY_POISONED,
MF_MSG_PFN_MAP,
MF_MSG_UNKNOWN,
};
@ -4222,16 +4561,6 @@ int arch_get_shadow_stack_status(struct task_struct *t, unsigned long __user *st
int arch_set_shadow_stack_status(struct task_struct *t, unsigned long status);
int arch_lock_shadow_stack_status(struct task_struct *t, unsigned long status);
/*
* mseal of userspace process's system mappings.
*/
#ifdef CONFIG_MSEAL_SYSTEM_MAPPINGS
#define VM_SEALED_SYSMAP VM_SEALED
#else
#define VM_SEALED_SYSMAP VM_NONE
#endif
/*
* DMA mapping IDs for page_pool
*

16
include/linux/mm_inline.h
View File

@ -8,7 +8,7 @@
#include <linux/swap.h>
#include <linux/string.h>
#include <linux/userfaultfd_k.h>
#include <linux/swapops.h>
#include <linux/leafops.h>
/**
* folio_is_file_lru - Should the folio be on a file LRU or anon LRU?
@ -44,7 +44,7 @@ static __always_inline void __update_lru_size(struct lruvec *lruvec,
lockdep_assert_held(&lruvec->lru_lock);
WARN_ON_ONCE(nr_pages != (int)nr_pages);
__mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
__mod_zone_page_state(&pgdat->node_zones[zid],
NR_ZONE_LRU_BASE + lru, nr_pages);
}
@ -541,9 +541,9 @@ static inline bool mm_tlb_flush_nested(const struct mm_struct *mm)
* The caller should insert a new pte created with make_pte_marker().
*/
static inline pte_marker copy_pte_marker(
swp_entry_t entry, struct vm_area_struct *dst_vma)
softleaf_t entry, struct vm_area_struct *dst_vma)
{
pte_marker srcm = pte_marker_get(entry);
const pte_marker srcm = softleaf_to_marker(entry);
/* Always copy error entries. */
pte_marker dstm = srcm & (PTE_MARKER_POISONED | PTE_MARKER_GUARD);
@ -553,7 +553,6 @@ static inline pte_marker copy_pte_marker(
return dstm;
}
#endif
/*
* If this pte is wr-protected by uffd-wp in any form, arm the special pte to
@ -571,9 +570,11 @@ static inline bool
pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
pte_t *pte, pte_t pteval)
{
#ifdef CONFIG_PTE_MARKER_UFFD_WP
bool arm_uffd_pte = false;
if (!uffd_supports_wp_marker())
return false;
/* The current status of the pte should be "cleared" before calling */
WARN_ON_ONCE(!pte_none(ptep_get(pte)));
@ -602,7 +603,7 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
make_pte_marker(PTE_MARKER_UFFD_WP));
return true;
}
#endif
return false;
}
@ -616,6 +617,7 @@ static inline bool vma_has_recency(const struct vm_area_struct *vma)
return true;
}
#endif
/**
* num_pages_contiguous() - determine the number of contiguous pages

156
include/linux/mm_types.h
View File

@ -286,6 +286,31 @@ typedef struct {
unsigned long val;
} swp_entry_t;
/**
* typedef softleaf_t - Describes a page table software leaf entry, abstracted
* from its architecture-specific encoding.
*
* Page table leaf entries are those which do not reference any descendent page
* tables but rather either reference a data page, are an empty (or 'none'
* entry), or contain a non-present entry.
*
* If referencing another page table or a data page then the page table entry is
* pertinent to hardware - that is it tells the hardware how to decode the page
* table entry.
*
* Otherwise it is a software-defined leaf page table entry, which this type
* describes. See leafops.h and specifically @softleaf_type for a list of all
* possible kinds of software leaf entry.
*
* A softleaf_t entry is abstracted from the hardware page table entry, so is
* not architecture-specific.
*
* NOTE: While we transition from the confusing swp_entry_t type used for this
* purpose, we simply alias this type. This will be removed once the
* transition is complete.
*/
typedef swp_entry_t softleaf_t;
#if defined(CONFIG_MEMCG) || defined(CONFIG_SLAB_OBJ_EXT)
/* We have some extra room after the refcount in tail pages. */
#define NR_PAGES_IN_LARGE_FOLIO
@ -774,6 +799,65 @@ struct pfnmap_track_ctx {
};
#endif
/* What action should be taken after an .mmap_prepare call is complete? */
enum mmap_action_type {
MMAP_NOTHING, /* Mapping is complete, no further action. */
MMAP_REMAP_PFN, /* Remap PFN range. */
MMAP_IO_REMAP_PFN, /* I/O remap PFN range. */
};
/*
* Describes an action an mmap_prepare hook can instruct to be taken to complete
* the mapping of a VMA. Specified in vm_area_desc.
*/
struct mmap_action {
union {
/* Remap range. */
struct {
unsigned long start;
unsigned long start_pfn;
unsigned long size;
pgprot_t pgprot;
} remap;
};
enum mmap_action_type type;
/*
* If specified, this hook is invoked after the selected action has been
* successfully completed. Note that the VMA write lock still held.
*
* The absolute minimum ought to be done here.
*
* Returns 0 on success, or an error code.
*/
int (*success_hook)(const struct vm_area_struct *vma);
/*
* If specified, this hook is invoked when an error occurred when
* attempting the selection action.
*
* The hook can return an error code in order to filter the error, but
* it is not valid to clear the error here.
*/
int (*error_hook)(int err);
/*
* This should be set in rare instances where the operation required
* that the rmap should not be able to access the VMA until
* completely set up.
*/
bool hide_from_rmap_until_complete :1;
};
/*
* Opaque type representing current VMA (vm_area_struct) flag state. Must be
* accessed via vma_flags_xxx() helper functions.
*/
#define NUM_VMA_FLAG_BITS BITS_PER_LONG
typedef struct {
DECLARE_BITMAP(__vma_flags, NUM_VMA_FLAG_BITS);
} __private vma_flags_t;
/*
* Describes a VMA that is about to be mmap()'ed. Drivers may choose to
* manipulate mutable fields which will cause those fields to be updated in the
@ -791,12 +875,18 @@ struct vm_area_desc {
/* Mutable fields. Populated with initial state. */
pgoff_t pgoff;
struct file *vm_file;
vm_flags_t vm_flags;
union {
vm_flags_t vm_flags;
vma_flags_t vma_flags;
};
pgprot_t page_prot;
/* Write-only fields. */
const struct vm_operations_struct *vm_ops;
void *private_data;
/* Take further action? */
struct mmap_action action;
};
/*
@ -833,10 +923,12 @@ struct vm_area_struct {
/*
* Flags, see mm.h.
* To modify use vm_flags_{init|reset|set|clear|mod} functions.
* Preferably, use vma_flags_xxx() functions.
*/
union {
/* Temporary while VMA flags are being converted. */
const vm_flags_t vm_flags;
vm_flags_t __private __vm_flags;
vma_flags_t flags;
};
#ifdef CONFIG_PER_VMA_LOCK
@ -917,6 +1009,52 @@ struct vm_area_struct {
#endif
} __randomize_layout;
/* Clears all bits in the VMA flags bitmap, non-atomically. */
static inline void vma_flags_clear_all(vma_flags_t *flags)
{
bitmap_zero(ACCESS_PRIVATE(flags, __vma_flags), NUM_VMA_FLAG_BITS);
}
/*
* Copy value to the first system word of VMA flags, non-atomically.
*
* IMPORTANT: This does not overwrite bytes past the first system word. The
* caller must account for this.
*/
static inline void vma_flags_overwrite_word(vma_flags_t *flags, unsigned long value)
{
*ACCESS_PRIVATE(flags, __vma_flags) = value;
}
/*
* Copy value to the first system word of VMA flags ONCE, non-atomically.
*
* IMPORTANT: This does not overwrite bytes past the first system word. The
* caller must account for this.
*/
static inline void vma_flags_overwrite_word_once(vma_flags_t *flags, unsigned long value)
{
unsigned long *bitmap = ACCESS_PRIVATE(flags, __vma_flags);
WRITE_ONCE(*bitmap, value);
}
/* Update the first system word of VMA flags setting bits, non-atomically. */
static inline void vma_flags_set_word(vma_flags_t *flags, unsigned long value)
{
unsigned long *bitmap = ACCESS_PRIVATE(flags, __vma_flags);
*bitmap |= value;
}
/* Update the first system word of VMA flags clearing bits, non-atomically. */
static inline void vma_flags_clear_word(vma_flags_t *flags, unsigned long value)
{
unsigned long *bitmap = ACCESS_PRIVATE(flags, __vma_flags);
*bitmap &= ~value;
}
#ifdef CONFIG_NUMA
#define vma_policy(vma) ((vma)->vm_policy)
#else
@ -1194,15 +1332,13 @@ struct mm_struct {
unsigned long cpu_bitmap[];
};
/* Set the first system word of mm flags, non-atomically. */
static inline void __mm_flags_set_word(struct mm_struct *mm, unsigned long value)
/* Copy value to the first system word of mm flags, non-atomically. */
static inline void __mm_flags_overwrite_word(struct mm_struct *mm, unsigned long value)
{
unsigned long *bitmap = ACCESS_PRIVATE(&mm->flags, __mm_flags);
bitmap_copy(bitmap, &value, BITS_PER_LONG);
*ACCESS_PRIVATE(&mm->flags, __mm_flags) = value;
}
/* Obtain a read-only view of the bitmap. */
/* Obtain a read-only view of the mm flags bitmap. */
static inline const unsigned long *__mm_flags_get_bitmap(const struct mm_struct *mm)
{
return (const unsigned long *)ACCESS_PRIVATE(&mm->flags, __mm_flags);
@ -1211,9 +1347,7 @@ static inline const unsigned long *__mm_flags_get_bitmap(const struct mm_struct
/* Read the first system word of mm flags, non-atomically. */
static inline unsigned long __mm_flags_get_word(const struct mm_struct *mm)
{
const unsigned long *bitmap = __mm_flags_get_bitmap(mm);
return bitmap_read(bitmap, 0, BITS_PER_LONG);
return *__mm_flags_get_bitmap(mm);
}
/*

37
include/linux/mmap_lock.h
View File

@ -130,7 +130,7 @@ static inline bool is_vma_writer_only(int refcnt)
* a detached vma happens only in vma_mark_detached() and is a rare
* case, therefore most of the time there will be no unnecessary wakeup.
*/
return refcnt & VMA_LOCK_OFFSET && refcnt <= VMA_LOCK_OFFSET + 1;
return (refcnt & VMA_LOCK_OFFSET) && refcnt <= VMA_LOCK_OFFSET + 1;
}
static inline void vma_refcount_put(struct vm_area_struct *vma)
@ -183,7 +183,7 @@ static inline void vma_end_read(struct vm_area_struct *vma)
}
/* WARNING! Can only be used if mmap_lock is expected to be write-locked */
static bool __is_vma_write_locked(struct vm_area_struct *vma, unsigned int *mm_lock_seq)
static inline bool __is_vma_write_locked(struct vm_area_struct *vma, unsigned int *mm_lock_seq)
{
mmap_assert_write_locked(vma->vm_mm);
@ -195,7 +195,8 @@ static bool __is_vma_write_locked(struct vm_area_struct *vma, unsigned int *mm_l
return (vma->vm_lock_seq == *mm_lock_seq);
}
void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq);
int __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq,
int state);
/*
* Begin writing to a VMA.
@ -209,7 +210,30 @@ static inline void vma_start_write(struct vm_area_struct *vma)
if (__is_vma_write_locked(vma, &mm_lock_seq))
return;
__vma_start_write(vma, mm_lock_seq);
__vma_start_write(vma, mm_lock_seq, TASK_UNINTERRUPTIBLE);
}
/**
* vma_start_write_killable - Begin writing to a VMA.
* @vma: The VMA we are going to modify.
*
* Exclude concurrent readers under the per-VMA lock until the currently
* write-locked mmap_lock is dropped or downgraded.
*
* Context: May sleep while waiting for readers to drop the vma read lock.
* Caller must already hold the mmap_lock for write.
*
* Return: 0 for a successful acquisition. -EINTR if a fatal signal was
* received.
*/
static inline __must_check
int vma_start_write_killable(struct vm_area_struct *vma)
{
unsigned int mm_lock_seq;
if (__is_vma_write_locked(vma, &mm_lock_seq))
return 0;
return __vma_start_write(vma, mm_lock_seq, TASK_KILLABLE);
}
static inline void vma_assert_write_locked(struct vm_area_struct *vma)
@ -281,11 +305,10 @@ static inline bool mmap_lock_speculate_retry(struct mm_struct *mm, unsigned int
return true;
}
static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt) {}
static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
struct vm_area_struct *vma)
{ return NULL; }
static inline void vma_end_read(struct vm_area_struct *vma) {}
static inline void vma_start_write(struct vm_area_struct *vma) {}
static inline __must_check
int vma_start_write_killable(struct vm_area_struct *vma) { return 0; }
static inline void vma_assert_write_locked(struct vm_area_struct *vma)
{ mmap_assert_write_locked(vma->vm_mm); }
static inline void vma_assert_attached(struct vm_area_struct *vma) {}

4
include/linux/mmzone.h
View File

@ -1060,10 +1060,6 @@ struct zone {
} ____cacheline_internodealigned_in_smp;
enum pgdat_flags {
PGDAT_DIRTY, /* reclaim scanning has recently found
* many dirty file pages at the tail
* of the LRU.
*/
PGDAT_WRITEBACK, /* reclaim scanning has recently found
* many pages under writeback
*/

10
include/linux/node.h
View File

@ -132,8 +132,6 @@ static inline void register_memory_blocks_under_nodes(void)
}
#endif
extern void unregister_node(struct node *node);
struct node_notify {
int nid;
};
@ -176,8 +174,8 @@ static inline int hotplug_node_notifier(notifier_fn_t fn, int pri)
#ifdef CONFIG_NUMA
extern void node_dev_init(void);
/* Core of the node registration - only memory hotplug should use this */
extern int register_one_node(int nid);
extern void unregister_one_node(int nid);
int register_node(int nid);
void unregister_node(int nid);
extern int register_cpu_under_node(unsigned int cpu, unsigned int nid);
extern int unregister_cpu_under_node(unsigned int cpu, unsigned int nid);
extern void unregister_memory_block_under_nodes(struct memory_block *mem_blk);
@ -189,11 +187,11 @@ extern int register_memory_node_under_compute_node(unsigned int mem_nid,
static inline void node_dev_init(void)
{
}
static inline int register_one_node(int nid)
static inline int register_node(int nid)
{
return 0;
}
static inline int unregister_one_node(int nid)
static inline int unregister_node(int nid)
{
return 0;
}

12
include/linux/pgtable.h
View File

@ -1557,6 +1557,18 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)
#define arch_start_context_switch(prev) do {} while (0)
#endif
/*
* Some platforms can customize the PTE soft-dirty bit making it unavailable
* even if the architecture provides the resource.
* Adding this API allows architectures to add their own checks for the
* devices on which the kernel is running.
* Note: When overriding it, please make sure the CONFIG_MEM_SOFT_DIRTY
* is part of this macro.
*/
#ifndef pgtable_supports_soft_dirty
#define pgtable_supports_soft_dirty() IS_ENABLED(CONFIG_MEM_SOFT_DIRTY)
#endif
#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
#ifndef CONFIG_ARCH_ENABLE_THP_MIGRATION
static inline pmd_t pmd_swp_mksoft_dirty(pmd_t pmd)

12
include/linux/sched/mm.h
View File

@ -189,12 +189,11 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr,
unsigned long len, unsigned long pgoff,
unsigned long flags, vm_flags_t);
unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *filp,
unsigned long addr, unsigned long len,
unsigned long pgoff, unsigned long flags);
unsigned long mm_get_unmapped_area(struct file *filp, unsigned long addr,
unsigned long len, unsigned long pgoff,
unsigned long flags);
unsigned long mm_get_unmapped_area_vmflags(struct mm_struct *mm,
struct file *filp,
unsigned long mm_get_unmapped_area_vmflags(struct file *filp,
unsigned long addr,
unsigned long len,
unsigned long pgoff,
@ -318,6 +317,9 @@ static inline void might_alloc(gfp_t gfp_mask)
fs_reclaim_acquire(gfp_mask);
fs_reclaim_release(gfp_mask);
if (current->flags & PF_MEMALLOC)
return;
might_sleep_if(gfpflags_allow_blocking(gfp_mask));
}

9
include/linux/shmem_fs.h
View File

@ -94,7 +94,8 @@ extern struct file *shmem_kernel_file_setup(const char *name, loff_t size,
unsigned long flags);
extern struct file *shmem_file_setup_with_mnt(struct vfsmount *mnt,
const char *name, loff_t size, unsigned long flags);
extern int shmem_zero_setup(struct vm_area_struct *);
int shmem_zero_setup(struct vm_area_struct *vma);
int shmem_zero_setup_desc(struct vm_area_desc *desc);
extern unsigned long shmem_get_unmapped_area(struct file *, unsigned long addr,
unsigned long len, unsigned long pgoff, unsigned long flags);
extern int shmem_lock(struct file *file, int lock, struct ucounts *ucounts);
@ -135,11 +136,16 @@ static inline bool shmem_hpage_pmd_enabled(void)
#ifdef CONFIG_SHMEM
extern unsigned long shmem_swap_usage(struct vm_area_struct *vma);
extern void shmem_uncharge(struct inode *inode, long pages);
#else
static inline unsigned long shmem_swap_usage(struct vm_area_struct *vma)
{
return 0;
}
static inline void shmem_uncharge(struct inode *inode, long pages)
{
}
#endif
extern unsigned long shmem_partial_swap_usage(struct address_space *mapping,
pgoff_t start, pgoff_t end);
@ -193,7 +199,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof)
}
extern bool shmem_charge(struct inode *inode, long pages);
extern void shmem_uncharge(struct inode *inode, long pages);
#ifdef CONFIG_USERFAULTFD
#ifdef CONFIG_SHMEM

Some files were not shown because too many files have changed in this diff Show More