[Deepin-Kernel-SIG] [linux 6.6.y] [Upstream] mm/filemap: remove hugetlb special casing in filemap.c

[opsiff]

Link: https://lore.kernel.org/all/20230926192017.98183-1-sidhartha.kumar@oracle.com/T/#u

Remove special cased hugetlb handling code within the page cache by
changing the granularity of ->index to the base page size rather than the
huge page size. The motivation of this patch is to reduce complexity
within the filemap code while also increasing performance by removing
branches that are evaluated on every page cache lookup.

To support the change in index, new wrappers for hugetlb page cache
interactions are added. These wrappers perform the conversion to a linear
index which is now expected by the page cache for huge pages.

========================= PERFORMANCE ======================================

Perf was used to check the performance differences after the patch. Overall
the performance is similar to mainline with a very small larger overhead that
occurs in __filemap_add_folio() and hugetlb_add_to_page_cache(). This is because
of the larger overhead that occurs in xa_load() and xa_store() as the
xarray is now using more entries to store hugetlb folios in the page cache.

Timing

aarch64
2MB Page Size
6.5-rc3 + this patch:
[root@sidhakum-ol9-1 hugepages]# time fallocate -l 700GB test.txt
real 1m49.568s
user 0m0.000s
sys 1m49.461s

    6.5-rc3:
        [root]# time fallocate -l 700GB test.txt
        real    1m47.495s
        user    0m0.000s
        sys     1m47.370s
1GB Page Size
    6.5-rc3 + this patch:
        [root@sidhakum-ol9-1 hugepages1G]# time fallocate -l 700GB test.txt
        real    1m47.024s
        user    0m0.000s
        sys     1m46.921s

    6.5-rc3:
        [root@sidhakum-ol9-1 hugepages1G]# time fallocate -l 700GB test.txt
        real    1m44.551s
        user    0m0.000s
        sys     1m44.438s

x86
2MB Page Size
6.5-rc3 + this patch:
[root@sidhakum-ol9-2 hugepages]# time fallocate -l 100GB test.txt
real 0m22.383s
user 0m0.000s
sys 0m22.255s

    6.5-rc3:
        [opc@sidhakum-ol9-2 hugepages]$ time sudo fallocate -l 100GB /dev/hugepages/test.txt
        real    0m22.735s
        user    0m0.038s
        sys     0m22.567s

1GB Page Size
    6.5-rc3 + this patch:
        [root@sidhakum-ol9-2 hugepages1GB]# time fallocate -l 100GB test.txt
        real    0m25.786s
        user    0m0.001s
        sys     0m25.589s

    6.5-rc3:
        [root@sidhakum-ol9-2 hugepages1G]# time fallocate -l 100GB test.txt
        real    0m33.454s
        user    0m0.001s
        sys     0m33.193s

aarch64:
workload - fallocate a 700GB file backed by huge pages

6.5-rc3 + this patch:
    2MB Page Size:
        --100.00%--__arm64_sys_fallocate
                      ksys_fallocate
                      vfs_fallocate
                      hugetlbfs_fallocate
                      |
                      |--95.04%--__pi_clear_page
                      |
                      |--3.57%--clear_huge_page
                      |          |
                      |          |--2.63%--rcu_all_qs
                      |          |
                      |           --0.91%--__cond_resched
                      |
                       --0.67%--__cond_resched
        0.17%     0.00%             0  fallocate  [kernel.vmlinux]       [k] hugetlb_add_to_page_cache
        0.14%     0.10%            11  fallocate  [kernel.vmlinux]       [k] __filemap_add_folio

6.5-rc3
    2MB Page Size:
            --100.00%--__arm64_sys_fallocate
                      ksys_fallocate
                      vfs_fallocate
                      hugetlbfs_fallocate
                      |
                      |--94.91%--__pi_clear_page
                      |
                      |--4.11%--clear_huge_page
                      |          |
                      |          |--3.00%--rcu_all_qs
                      |          |
                      |           --1.10%--__cond_resched
                      |
                       --0.59%--__cond_resched
        0.08%     0.01%             1  fallocate  [kernel.kallsyms]  [k] hugetlb_add_to_page_cache
        0.05%     0.03%             3  fallocate  [kernel.kallsyms]  [k] __filemap_add_folio

x86
workload - fallocate a 100GB file backed by huge pages

6.5-rc3 + this patch:
    2MB Page Size:
        hugetlbfs_fallocate
        |
        --99.57%--clear_huge_page
            |
            --98.47%--clear_page_erms
                |
                --0.53%--asm_sysvec_apic_timer_interrupt

        0.04%     0.04%             1  fallocate  [kernel.kallsyms]     [k] xa_load
        0.04%     0.00%             0  fallocate  [kernel.kallsyms]     [k] hugetlb_add_to_page_cache
        0.04%     0.00%             0  fallocate  [kernel.kallsyms]     [k] __filemap_add_folio
        0.04%     0.00%             0  fallocate  [kernel.kallsyms]     [k] xas_store

6.5-rc3
    2MB Page Size:
            --99.93%--__x64_sys_fallocate
                      vfs_fallocate
                      hugetlbfs_fallocate
                      |
                       --99.38%--clear_huge_page
                                 |
                                 |--98.40%--clear_page_erms
                                 |
                                  --0.59%--__cond_resched
        0.03%     0.03%             1  fallocate  [kernel.kallsyms]  [k] __filemap_add_folio

========================= TESTING ======================================

This patch passes libhugetlbfs tests and LTP hugetlb tests

********** TEST SUMMARY

•                  2M
  

•                  32-bit 64-bit
  

• Total testcases:   110    113
  

•         Skipped:     0      0
  

•            PASS:   107    113
  

•            FAIL:     0      0
  

• Killed by signal: 3 0
• Bad configuration: 0 0

•   Expected FAIL:     0      0
  

• Unexpected PASS:     0      0
  

• Test not present: 0 0
• Strange test result: 0 0

---

Done executing testcases.
LTP Version:  20220527-178-g2761a81c4

page migration was also tested using Mike Kravetz's test program.8

Signed-off-by: Sidhartha Kumar sidhartha.kumar@oracle.com
Reported-and-tested-by: syzbot+c225dea486da4d5592bd@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=c225dea486da4d5592bd

rebased on mm-unstable 09/26/23

RFC v21 -> v12
-change direction of series to maintain both huge and base page size index
rather than try to get rid of all references to a huge page sized index.

v1 -> v23
- squash seperate filemap and hugetlb changes into one patch to allow
for bisection per Matthew
- get rid of page_to_index()
- fix errors in hugetlb_fallocate() and remove_inode_hugepages()

v2 -> v34
- gather performance data per Mike Kravetz
- remove start variable in remove_inode_hugepages() per Mike Kravetz
- remove hugetlb special case within folio_file_page()

v3 -> v45
- rebase to current mm-unstable
- include time data per Mike Kravetz

v4 -> v56
- fix build issue by removing hugetlb_basepage_index() definition
per intel test robot

v5 -> v67
- remove folio_more_pages() from result of incorrect rebase

v6 -> v79:
- clarify commit message to demonstrate the benefits of this patch.
- fix error in hugepage migration by using folio_expected_refs() rather
than 2 in migrate_huge_page_move_mapping() and increase the dst folio
ref count by folio_nr_pages().
- this error is due to __filemap_add_folio() increasing the ref
count of the huge folio by folio_nr_pages() which
migrate_huge_page_move_mapping() did not handle
- remove linear_hugepage_index() as it is no longer used per Matthew Wilcox.

v7 -> v8:
- fix syzbot reported error in hugetlbfs_read_iter() by using
filemap_lock_hugetlb_folio() for page cache lookup.

fs/hugetlbfs/inode.c | 37 +++++++++++++++++++------------------
include/linux/hugetlb.h | 12 ++++++++++++
include/linux/pagemap.h | 32 ++------------------------------
mm/filemap.c | 34 ++++++++++------------------------
mm/hugetlb.c | 32 ++++++--------------------------
mm/migrate.c | 6 +++---
6 files changed, 52 insertions(+), 101 deletions(-)

Summary by Sourcery

Simplify hugetlb page cache handling by indexing huge pages at base-page granularity and relying on common filemap paths, while updating hugetlb-specific users accordingly.

Bug Fixes:

• Fix hugetlbfs_read_iter page cache lookup to avoid the syzbot-reported error by using a dedicated hugetlb folio lock helper.
• Correct hugepage migration and refcount handling to account for multi-page folios, preventing refcount underflow or leaks.
• Adjust memory-failure extra refcount accounting for huge pages to subtract the correct number of folio references.
• Ensure userfaultfd hugetlb mfill logic uses the proper hugepage-based linear index for pagecache lookups.

Enhancements:

• Remove hugetlb-specific special casing from core filemap traversal and folio batching so all folios are handled uniformly in base-page units.
• Introduce hugetlb-aware helpers for linear page indexing and locking folios in the page cache, improving clarity of hugetlb interactions with filemap.
• Clean up obsolete hugetlb index conversion helpers now that huge page cache indices are expressed in base-page units.

[sourcery-ai]

Reviewer's Guide

This PR removes hugetlb-specific indexing and refcount handling from the core page cache (filemap) code by standardizing folio->index to PAGE_SIZE units, introducing hugetlb-specific helpers/wrappers to translate hugepage indices, and updating hugetlbfs, migration, memory-failure, and userfaultfd paths to use the new conventions while fixing related refcount and lookup bugs.

Sequence diagram for hugetlb page-cache lookup using filemap_lock_hugetlb_folio

sequenceDiagram
    participant HFS as hugetlbfs_read_iter
    participant HLPF as filemap_lock_hugetlb_folio
    participant FLF as filemap_lock_folio
    participant XA as xarray_pagecache

    HFS->>HLPF: filemap_lock_hugetlb_folio(h, mapping, idx)
    note over HLPF: idx << huge_page_order(h)
    HLPF->>FLF: filemap_lock_folio(mapping, idx_shifted)
    FLF->>XA: xa_load(mapping->i_pages, idx_shifted)
    XA-->>FLF: folio or error
    FLF-->>HLPF: folio (locked) or ERR_PTR
    HLPF-->>HFS: folio (locked) or ERR_PTR

Loading

File-Level Changes

Change	Details	Files
Standardize page cache xarray indexing and traversal on base-page units for all folios, removing hugetlb special-casing in filemap	Always set xarray order and slot count in page_cache_delete() and __filemap_add_folio() based on folio_order/folio_nr_pages without skipping hugetlb folios Remove hugetlb-specific handling when advancing iteration in find_get_entries(), find_lock_entries(), filemap_get_folios(), and filemap_get_folios_contig(), so *start is consistently bumped by folio_nr_pages() Simplify filemap_free_folio() to treat all large folios the same, dropping the hugetlb exemption when deciding how many refs to put	mm/filemap.c
Make folio/page index and pgoff helpers consistently operate in PAGE_SIZE units and drop legacy hugetlb index conversion helpers	Remove hugetlb branches from folio_file_page() and folio_contains() now that hugetlb folios use base-page indexing Collapse page_to_index()/page_to_pgoff() so page_to_pgoff() is the core helper; drop hugetlb_basepage_index() and linear_hugepage_index() usage in favor of base-page indices Simplify folio_pgoff() to directly return folio->index, removing hugetlb special handling	include/linux/pagemap.h mm/hugetlb.c
Introduce hugetlb-specific helpers to translate between hugepage indices and base-page indices for page-cache operations	Add hugetlb_linear_page_index() to compute hugepage-granularity offsets analogous to linear_page_index() Add filemap_lock_hugetlb_folio() wrapper which converts a hugepage index to the appropriate base-page index before calling filemap_lock_folio() Provide stub/no-op versions of new helpers for !CONFIG_HUGETLB_PAGE	include/linux/hugetlb.h
Update hugetlbfs and core hugetlb page-cache users to use base-page indexing and new helpers	Rewrite hugetlbfs_read_iter() to work on struct folio, using filemap_lock_hugetlb_folio(), folio_hwpoison helpers, and copy_folio_to_iter()/folio_put() Change remove_inode_hugepages() and hugetlbfs_zero_partial_page() to iterate and lock folios using base-page indices and convert folio->index back to hugepage units only where needed Adjust hugetlbfs_fallocate(), hugetlbfs_pagecache_present(), hugetlb_add_to_page_cache(), hugetlb_no_page(), hugetlb_fault(), and hugetlb_mfill_atomic_pte() to compute indices with linear_page_index()/hugetlb_linear_page_index() and call filemap_lock_hugetlb_folio()/shift indices as required	fs/hugetlbfs/inode.c mm/hugetlb.c mm/userfaultfd.c
Correct refcount expectations and moves for hugepage migration and memory failure paths under new indexing/refcount semantics	In migrate_huge_page_move_mapping(), replace hard-coded expected refcount with folio_expected_refs() and adjust dst/src ref operations to account for folio_nr_pages() In has_extra_refcount(), adjust the extra_pins handling for hugepages by subtracting folio_nr_pages(page_folio(p)) instead of 1	mm/migrate.c mm/memory-failure.c

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[deepin-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from opsiff. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• deepin/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sourcery-ai]

Hey - I've left some high level feedback:

• In hugetlbfs_read_iter(), any IS_ERR(folio) from filemap_lock_hugetlb_folio() is now treated as a hole and zero-filled, which can silently mask real lookup/allocation errors; consider distinguishing between "not present" (e.g., -ENOENT) and genuine failures (e.g., -ENOMEM) and propagating/handling those appropriately.
• In hugetlbfs_fallocate(), the folio lookup uses index << huge_page_order(h) while hugetlb_fault_mutex_hash() is still hashed on the unshifted hugepage index; please double-check that this intentional mismatch between mutex granularity (hugepage index) and pagecache index (base-page index) is safe and remains consistent with other hugetlb fault paths.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In hugetlbfs_read_iter(), any IS_ERR(folio) from filemap_lock_hugetlb_folio() is now treated as a hole and zero-filled, which can silently mask real lookup/allocation errors; consider distinguishing between "not present" (e.g., -ENOENT) and genuine failures (e.g., -ENOMEM) and propagating/handling those appropriately.
- In hugetlbfs_fallocate(), the folio lookup uses index << huge_page_order(h) while hugetlb_fault_mutex_hash() is still hashed on the unshifted hugepage index; please double-check that this intentional mismatch between mutex granularity (hugepage index) and pagecache index (base-page index) is safe and remains consistent with other hugetlb fault paths.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}