Skip to content

vm/map: Message buffers memory safety#810

Draft
etiaro wants to merge 2 commits into
masterfrom
etiaro/msg-safety
Draft

vm/map: Message buffers memory safety#810
etiaro wants to merge 2 commits into
masterfrom
etiaro/msg-safety

Conversation

@etiaro

@etiaro etiaro commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Description

Update message receiver map entries with corresponding amap/objects to ensure these pages won't get allocated by keeping reference counters non-zero even when message sender unmaps these sent pages.

Example that causes kernel panics on ia32 via the problem described above:

#include <stdio.h>
#include <sys/mman.h>
#include <sys/msg.h>
#include <pthread.h>
#include <fcntl.h>
#include <unistd.h>

void* buf = MAP_FAILED;

#define OP_SIZE 0x40000

void* thr2(void* a){
    for (;;) {
        while (buf == MAP_FAILED) {
            printf("thr2: waiting for buf\n");
            usleep(100000);
        }
        munmap((void*)buf, OP_SIZE);
        buf = MAP_FAILED;
        printf("freed\n");
    }
}



int main(void) {
    uint32_t port;
    pthread_t t;

    int fd = open("/test", O_RDWR | O_CREAT);

    buf = mmap(NULL, OP_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (buf == MAP_FAILED) {
        return 1;
    }
    for (int i = 0; i < OP_SIZE; i++) {
        ((char*)buf)[i] = 0xAB;
    }
    write(fd, buf, OP_SIZE);
    munmap(buf, OP_SIZE);

    // Memory saturation to speed up the exchange of available pages
    while ((buf = mmap(NULL, OP_SIZE*2, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0)) != MAP_FAILED) {
    }

    portCreate(&port);
    pthread_create(&t, NULL, thr2, NULL);

    for (;;) {
        buf = mmap(NULL, OP_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
        if (buf == MAP_FAILED) {
            continue;
        }

        printf("allocated buf: %p\n", buf);
        while (buf != MAP_FAILED) {
            lseek(fd, 0, SEEK_SET);
            // printf("written %d\n", write(fd, (void*)buf, OP_SIZE));
            printf("read %ld\n", read(fd, (void*)buf, OP_SIZE));
            printf("buf: %p\n", buf);
        }
    }

    return 0;
}

These changes exacerbate that issue: phoenix-rtos/phoenix-rtos-project#1714, , awaiting for fix

Motivation and Context

Current msg implementation maps pages without marking them anywhere, leading to potential reallocation and multimapping such pages in other places (even as kernel structures).

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Chore (refactoring, style fixes, git/CI config, submodule management, no code logic changes)

How Has This Been Tested?

  • Already covered by automatic testing.
  • New test added: (add PR link here).
  • Tested by hand on: (list targets here).

Checklist:

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • All new and existing linter checks and tests passed.
  • My changes generate no new compilation warnings for any of the targets.

Special treatment

  • This PR needs additional PRs to work (list the PRs, preferably in merge-order).
  • I will merge this PR by myself when appropriate.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces vm_mapBorrow to allow borrowing virtual memory map entries from a source map to a destination map, and updates proc/msg.c to use this function for bulk page mapping instead of mapping pages individually. Feedback on these changes highlights several critical issues in vm_mapBorrow, including an incorrect loop condition comparing a cumulative offset to a chunk size, a stale pointer risk due to an unreset variable e across iterations, and a security vulnerability where copy-on-write protections (MAP_NEEDSCOPY) are bypassed. Additionally, reviewers noted an uninitialized err variable in proc/msg.c and recommended refactoring error handling in vm_mapBorrow to adhere to MISRA 15.5 single-exit-point guidelines.

Comment thread vm/map.c Outdated
Comment thread vm/map.c
Comment thread vm/map.c
Comment thread proc/msg.c Outdated
Comment thread vm/map.c
Comment thread proc/msg.c Fixed
Comment thread vm/map.c Fixed
Comment thread vm/map.c Fixed
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown

Unit Test Results

10 813 tests   - 77   10 116 ✅  - 86   1h 5m 7s ⏱️ + 6m 18s
   680 suites ± 0      670 💤 ± 0 
     1 files   ± 0       27 ❌ + 9 

For more details on these failures, see this check.

Results for commit 15b8bd6. ± Comparison against base commit fb914a8.

This pull request removes 82 and adds 5 tests. Note that renamed tests count towards both.
phoenix-rtos-tests/libc/posixsrv ‑ aarch64a53-zynqmp-qemu:phoenix-rtos-tests/libc/posixsrv.tmpfile.multiple
phoenix-rtos-tests/libc/posixsrv ‑ armv7a7-imx6ull-evk:phoenix-rtos-tests/libc/posixsrv.tmpfile.multiple
phoenix-rtos-tests/libc/posixsrv ‑ armv7a9-zynq7000-qemu:phoenix-rtos-tests/libc/posixsrv.tmpfile.multiple
phoenix-rtos-tests/libc/posixsrv ‑ riscv64-generic-qemu:phoenix-rtos-tests/libc/posixsrv.tmpfile.multiple
phoenix-rtos-tests/libc/stdlib ‑ armv7a9-zynq7000-zedboard:phoenix-rtos-tests/libc/stdlib.stdlib_alloc.calloc_iterate
phoenix-rtos-tests/libc/stdlib ‑ armv7a9-zynq7000-zedboard:phoenix-rtos-tests/libc/stdlib.stdlib_alloc.calloc_large
phoenix-rtos-tests/libc/stdlib ‑ armv7a9-zynq7000-zedboard:phoenix-rtos-tests/libc/stdlib.stdlib_alloc.calloc_overflow
phoenix-rtos-tests/libc/stdlib ‑ armv7a9-zynq7000-zedboard:phoenix-rtos-tests/libc/stdlib.stdlib_alloc.free_null
phoenix-rtos-tests/libc/stdlib ‑ armv7a9-zynq7000-zedboard:phoenix-rtos-tests/libc/stdlib.stdlib_alloc.realloc_calloc_resize
phoenix-rtos-tests/libc/stdlib ‑ armv7a9-zynq7000-zedboard:phoenix-rtos-tests/libc/stdlib.stdlib_alloc.realloc_calloc_resize_larger
…
phoenix-rtos-tests/libc/posixsrv ‑ aarch64a53-zynqmp-qemu:phoenix-rtos-tests/libc/posixsrv
phoenix-rtos-tests/libc/posixsrv ‑ armv7a7-imx6ull-evk:phoenix-rtos-tests/libc/posixsrv
phoenix-rtos-tests/libc/posixsrv ‑ armv7a9-zynq7000-qemu:phoenix-rtos-tests/libc/posixsrv
phoenix-rtos-tests/libc/posixsrv ‑ riscv64-generic-qemu:phoenix-rtos-tests/libc/posixsrv
phoenix-rtos-tests/libc/stdlib ‑ armv7a9-zynq7000-zedboard:phoenix-rtos-tests/libc/stdlib

♻️ This comment has been updated with latest results.

Update message receiver map entries with corresponding amap/objects
to ensure these pages won't get allocated by keeping reference counters
non-zero even when message sender unmaps these sent pages.

TASK: RTOS-1393
@etiaro
etiaro force-pushed the etiaro/msg-safety branch from b6fccf3 to 15b8bd6 Compare July 16, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants