Skip to content

v3d.md

Latest commit

 

History

History
109 lines (77 loc) · 6.01 KB

v3d.md

File metadata and controls

109 lines (77 loc) · 6.01 KB
Raw

v3d driver

本文档描述了该项目的主要组成，V3D driver 的结构。

linux source code

路径linux-6.12.28/drivers/gpu/drm/v3d。以2025.5.19开始制作文档时，Raspberry Pi OS 最新预览版的6.12.28版本为准。

v3d_bo.c

负责内存分配。

v3d_debugfs.c

包含用于debug的各类函数。触发 replay 的外部接口在此文件中定义。

v3d_drv.c

包含 V3D 与设备及用户空间交互的函数。

v3d_fence.c

包含 fence 的创建等操作。fence 是一种类似信号量的结构，用于保证操作的有序性。

The idea behind fencing is to ensure ordering between operations—in particular to synchronize buffer sharing between the GPU and display drivers. It will make sure that the GPU does not write to a buffer that is still being displayed and that buffers won't be displayed while the GPU is still rendering to them. It allows the display driver and the GPU to operate asynchronously.

v3d_gem.c

包含 GEM - The Graphics Execution Manager 的初始化、终止等操作。GEM 是 DRM 用于管理内存分配与释放、指令执行、等功能的子系统。

v3d_gemgs.c

用于管理 GEM 的 file system？

v3d_irq.c

包含中断开关、中断处理等中断相关的函数。

When we take a bin, render, TFU done, or CSD done interrupt, we need to signal the fence for that job so that the scheduler can queue up the next one and unblock any waiters. When we take the binner out of memory interrupt, we need to allocate some new memory and pass it to the binner so that the current job can make progress.

v3d_mmu.c

包含内存管理。

The V3D 3.x hardware (compared to VC4) now includes an MMU. It has a single level of page tables for the V3D's 4GB address space to map to AXI bus addresses, thus it could need up to 4MB of physically contiguous memory to store the PTEs.

v3d_perfmon.c

用于记录 V3D 的性能？

v3d_sched.c

用于调度、运行 V3D job。

v3d_submit.c

用于释放、清理 V3D job，和用户空间做接口。

v3d_sysfs.c

用于创建 sysfs 属性，查看驱动内部状态？

v3d_trace_points.c

用于插桩，追踪各类事件。

GPU jobs

csd

一个调用流程：

  1. v3d_sched_init()：初始化调度器，让job最终能调用对应的工作函数。
  2. v3d_submit_csd_ioctl()：内部调用了v3d_push_job()，该函数再调用drm_sched_entity_push_job()，将job送入队列。有互斥锁保证job的FIFO性。

For simplicity, and in order to keep latency low for interactive jobs when bulk background jobs are queued up, we submit a new job to the HW only when it has completed the last one, instead of filling up the CT[01]Q FIFOs with jobs. Similarly, we use drm_sched_job_add_dependency() to manage the dependency between bin and render, instead of having the clients submit jobs using the HW’s semaphores to interlock between them.

  1. v3d_csd_job_run()：由drm_sched_run_job_work()调用，向 V3D 的 csd 相关寄存器写入数据，最终触发csd中断。fence返回到哪儿了？
  2. v3d_irq()：调用dma_fence_signal().

GPU Scheduling The shared DRM GPU scheduler is used to coordinate submitting jobs to the hardware. Each DRM fd (roughly a client process) gets its own scheduler entity, which will process jobs in order. The GPU scheduler will schedule the clients with a FIFO scheduling algorithm.

For simplicity, and in order to keep latency low for interactive jobs when bulk background jobs are queued up, we submit a new job to the HW only when it has completed the last one, instead of filling up the CT[01]Q FIFOs with jobs. Similarly, we use drm_sched_job_add_dependency() to manage the dependency between bin and render, instead of having the clients submit jobs using the HW’s semaphores to interlock between them.

这一段是不是暗示了csd job无法正常触发中断的原因？好像没太大关系。

构建

OP-TEE 应用形式的 replayer 本身并不是驱动，大部分逻辑也无需驱动支持，但现版本的中断部分仍需要 V3D 驱动支持。

由于从不同版本的内核编译出的模块无法挂载到另一版本的内核，所以应当在host machine (with buildroot) 的 linux-custom 构建模块。构建前只需把原 V3D driver 替换为本项目的 replayer。

cd /home/zhenlu/OPTEE-raspi5/buildroot/output/build/linux-custom
make M=/home/zhenlu/OPTEE-raspi5/buildroot/output/build/linux-custom/drivers/gpu/drm/v3d modules

然后再在 target machine 挂载模块。

加载

buildroot 的默认设置已经包含了V3D模块，但需要modprobe v3d手动加载，即使在config.txt加上dtoverlay似乎也得手动加载。大概是linux config的原因，但我没有找到自动加载模块的位置。

需要修改v3d_of_match才能正常匹配设备树节点。

sudo systemctl isolate multi-user.target # 关闭图形界面
sudo systemctl isolate graphical.target # 打开图形界面

关闭wayland后可以rmmod v3d。

安装

用modinfo -F depends v3d查看V3D依赖，并逐个insmod。依赖分别是drm、gpu-sched、drm_shmem_helper。

第一次insmod后要rmmod，需要解除V3D被引用，然后rmmod。好像又行不通。

echo -n "1002000000.v3d" | sudo tee /sys/bus/platform/drivers/v3d/unbind

参考资料

  1. https://bakhi.github.io/mobileGPU/v3d/ ：由本项目所依托的原论文的作者制作，全面描述了 V3D、Mali 等 GPU driver 的运行机制。
  2. https://docs.kernel.org/gpu/v3d.html ：Linux 官方文档，简要说明了 V3D driver 的部分机制。
  3. https://docs.kernel.org/gpu/drm-internals.html ：Linux 官方文档，描述了 drm driver 的共同机制。

个别机制

  1. https://stackoverflow.com/questions/10552085/difference-between-lock-memory-barrier-semaphore ：含有 fence 相关的资料。
  2. https://docs.kernel.org/gpu/drm-mm.html#the-graphics-execution-manager-gem ：Linux 官方文档，描述了 GEM 的机制。
  3. https://lwn.net/Articles/283798/ ：GEM 诞生之初的文档。
  4. https://docs.kernel.org/trace/tracepoints.html ：tracepoint 机制。