camerad: mainline kernel architecture proposal

[andiradulescu]

Goal

Port camerad to work on mainline Linux kernel (6.18+, SDM845/SM8150) while keeping all ISP and sensor register programming in userspace (openpilot). The kernel should be a thin hardware access layer. camerad owns the imaging pipeline.

Why

• Keep registers in openpilot: ISP tuning (gamma, linearization, CCM, vignetting, debayer), sensor configuration, and auto-exposure all live in the repo where they can be iterated on without kernel rebuilds.
• Enable advanced features: Two cameras on one IFE, per-frame ISP reconfiguration, custom AE strategies - all require userspace register control.
• Minimize latency: Direct control over when registers are written and when reg_update fires. No kernel-side pipeline management adding scheduling overhead.
• Maintainability: Thin kernel driver with stable interface. All imaging logic in one place (camerad).

Current architecture (downstream AGNOS kernel)

camerad talks to Qualcomm's proprietary camera subsystem via V4L2 subdevs that accept cam_packet command buffers. This is not standard V4L2 - it uses V4L2 as a transport for Qualcomm-specific ioctls.

Device paths

Device	Path	Purpose
Request manager	/dev/v4l/by-path/platform-soc:qcom_cam-req-mgr-video-index0	Session/link mgmt, buffer alloc, request scheduling, event polling
Cam sync	/dev/v4l/by-path/platform-cam_sync-video-index0	Fence creation/destruction for frame sync
ISP (IFE)	/dev/v4l-subdev{N} (cam-isp)	Accepts CDM packets with raw ISP register writes
ICP (BPS)	/dev/v4l-subdev{N} (cam-icp)	BPS pipeline for driver camera
Sensor x3	/dev/v4l-subdev{N} (cam-sensor-driver)	Proxies I2C/CCI register writes to sensors
CSIPHY x3	/dev/v4l-subdev{N} (cam-csiphy-driver)	MIPI CSI-2 PHY configuration

Per-frame request cycle

enqueue_frame(request_id):
  1. CAM_SYNC_CREATE           create fence for this frame slot
  2. CAM_REQ_MGR_SCHED_REQ     tell request manager "frame N coming"
  3. sensors_poke()             NOP I2C to keep sensor in sync
  4. config_ife(idx):           submit cam_packet containing:
       cmd_buf[0]               CDM register writes (ife.h values)
       cmd_buf[1]               clock/bandwidth config
       io_cfg[0]                mem_handle = buf_handle_yuv[idx]  (DMA target)
                                fence = sync_objs_ife[idx]        (signal when done)
       patches[]                fix up LUT DMA addresses (IOMMU translation)

Main loop

poll(video0_fd, POLLPRI)        block until SOF interrupt
VIDIOC_DQEVENT                  get cam_req_mgr_message (frame_id, request_id, timestamp)
CAM_SYNC_WAIT(fence)            block until ISP DMA complete -> frame is in buffer
sendFrameToVipc()               notify consumers (zero-copy, buffer is shared memory)
set_camera_exposure()           read Y pixels from buffer, compute new exposure
sensors_i2c(exposure_regs)      write new exposure/gain to sensor via I2C
enqueue_frame(req_id + 18)      re-queue this buffer slot for future frame

Key property: zero-copy frame output

VisionIPC buffers (DMA-BUF backed) are IOMMU-mapped into the ISP. The VFE hardware DMAs processed NV12 frames directly into shared memory. sendFrameToVipc() is just a notification to consumers (modeld, encoderd), not a copy.

Proposed mainline architecture

Design principle

The kernel provides: IOMMU mapping, register write access, interrupt delivery, power/clocks.
camerad provides: all register values, buffer management, frame scheduling, AE, sensor config.

No V4L2 video device for the frame path. The ISP writes directly into VisionIPC shared memory buffers. camerad manages the circular buffer queue itself (same as downstream). V4L2 subdevs remain for CSIPHY/CSID so the media controller pipeline links up and clocks/power are managed.

Kernel interface

A thin char device or V4L2 subdev ioctl extension on the VFE driver:

ioctl	Arguments	Kernel action
VFE_MAP_BUF	DMA-BUF fd	Map into VFE IOMMU, return IOVA
VFE_UNMAP_BUF	IOVA	Unmap from VFE IOMMU
VFE_WRITE_REGS	array of {offset, value} pairs	writel_relaxed(value, base + offset) for each
VFE_WRITE_DMI	{ram_select, data[], count}	Write DMI config register, upload LUT entries
VFE_SET_BUF	{wm_index, iova, stride, frame_inc}	Write VFE_BUS_WM_IMAGE_ADDR and related WM regs
VFE_REG_UPDATE	line_id (PIX or RDI)	Write VFE_REG_UPDATE_CMD, latch all pending writes
poll() / read()	-	Deliver SOF and buf_done interrupts with timestamps

Per-frame flow (mainline)

enqueue_frame(request_id):
  1. VFE_WRITE_REGS            same register values from ife.h (per-frame update)
  2. VFE_SET_BUF               point WM at VisionIPC buffer[idx] IOVA
  3. VFE_REG_UPDATE             latch all writes
  4. sensors_i2c()              poke sensor via /dev/i2c-*

main loop:
  poll(vfe_fd)                  SOF interrupt (timestamp, frame counter)
  poll(vfe_fd)                  buf_done interrupt (frame in buffer)
  sendFrameToVipc()             notify consumers (zero-copy, same as downstream)
  set_camera_exposure()         same AE logic, same Y pixel reads
  sensors_i2c(exposure_regs)    write exposure/gain via /dev/i2c-*
  enqueue_frame(req + depth)    re-queue slot

What stays where

camerad (userspace) - unchanged or minimal changes

Component	File(s)	Status
IFE register values	ife.h	Unchanged - same offsets, same values
CDM helpers	cdm.h, cdm.cc	Simplified - build {offset, value} lists instead of CDM HW packets
Sensor init registers	ox03c10_registers.h, os04c10_registers.h	Unchanged
Sensor exposure/gain	ox03c10.cc, os04c10.cc (getExposureRegisters)	Unchanged
Sensor base class	sensor.h (LUTs, CCM, linearization_pts, gamma)	Unchanged
Auto-exposure	camera_qcom2.cc (set_camera_exposure)	Unchanged
Frame sync logic	spectra.cc (syncFirstFrame)	Unchanged
Buffer management	camera_common.cc (CameraBuf, sendFrameToVipc)	Unchanged
Camera configs	hw.h (ALL_CAMERA_CONFIGS)	Unchanged

camerad - needs new backend

Component	Current (downstream)	New (mainline)
ISP register submission	CDM packets via cam_packet + CAM_CONFIG_DEV	VFE_WRITE_REGS ioctl
LUT uploads	CDM DMI commands with IOMMU patch descriptors	VFE_WRITE_DMI ioctl
Buffer IOMMU mapping	CAM_REQ_MGR_MAP_BUF on video0_fd	VFE_MAP_BUF on vfe_fd
Output buffer selection	io_cfg.mem_handle in cam_packet	VFE_SET_BUF ioctl
Frame scheduling	CAM_REQ_MGR_SCHED_REQ + CAM_SYNC_CREATE	VFE_REG_UPDATE + poll for buf_done
SOF events	VIDIOC_DQEVENT on video0_fd	poll() + read() on vfe_fd
Sensor I2C	cam-sensor-driver V4L2 subdev	/dev/i2c-* direct (same registers)
CSIPHY config	cam-csiphy-driver V4L2 subdev	Kernel-managed (standard mainline CSIPHY)
Session/link management	CAM_REQ_MGR_CREATE_SESSION, CAM_REQ_MGR_LINK	Not needed
Sync fences	CAM_SYNC_CREATE/WAIT/DESTROY	Not needed (poll for buf_done)

Kernel (mainline) - modifications to qcom-camss

Component	Current mainline state	Needed changes
CSIPHY	Working (standard mainline)	None
CSID	Working (camss-csid-gen2)	None
VFE power/clocks	Working (camss-vfe-17x)	None
VFE ISP pipeline	Hardcoded in vfe_pix_configure_isp()	Remove hardcoded config, add VFE_WRITE_REGS ioctl
VFE DMI upload	Hardcoded in vfe_dmi_upload_lut()	Add VFE_WRITE_DMI ioctl
VFE BUS/WM	Managed by V4L2 buffer layer	Add VFE_SET_BUF ioctl
VFE reg_update	Called internally on streamon	Add VFE_REG_UPDATE ioctl
IOMMU buf mapping	Handled by videobuf2	Add VFE_MAP_BUF / VFE_UNMAP_BUF ioctls
Interrupt delivery	ISR dispatches to V4L2 internals	Expose SOF + buf_done via poll/read
OX03C10 sensor driver	1780-line driver with init regs, V4L2 controls	Strip to power/probe stub (~200 lines)

Sensor I2C path

Current (downstream)

Sensor registers go through a dedicated kernel cam-sensor-driver V4L2 subdev. camerad builds cam_packet with CAM_SENSOR_PACKET_OPCODE_SENSOR_CONFIG containing i2c_random_wr_payload arrays, submits via CAM_CONFIG_DEV. The kernel driver translates these to CCI (I2C) transactions.

Proposed (mainline)

camerad writes sensor registers directly via /dev/i2c-* using standard Linux I2C userspace API. Same register addresses, same values, same 16-bit addr / 8-bit data format.

Sensor	I2C Address	Bus	Init regs	Per-frame regs
OX03C10 (wide)	0x36 (7-bit)	CCI0	~600 writes from ox03c10_registers.h	9 writes (exposure + gain)
OX03C10 (road)	0x10 (7-bit)	CCI0	~600 writes	9 writes
OS04C10 (driver)	0x36 (7-bit)	CCI1	~300 writes from os04c10_registers.h	6 writes

Power sequencing (regulators, clocks, reset GPIO) stays in the kernel sensor driver stub. It powers on during probe, camerad handles everything after that.

ISP register map (from ife.h)

These are the VFE Titan 170 registers that camerad programs. All offsets and values stay identical on mainline - it is the same hardware.

Initial config (once at startup)

Register block	Offset	Description	Source
CGC override	0x02c-0x03c	Clock gate force-on for all ISP modules	build_update
Module enables	0x040-0x04c	Enable debayer, WB, color correct, scaler, crop	build_update
Linearization	0x4dc-0x510	Kneepoint config (sensor->linearization_pts)	build_initial_config
Linearization LUT	DMI sel=9	36-entry piecewise linear table	build_initial_config
CAMIF config	0x478-0x49c	CAMIF clear, output enable, subsample	build_initial_config
Demux	0x560	Bayer pattern routing	build_update
Black level	0x6b0	Scale + offset (per-sensor black_level)	build_update
Vignetting	0x6bc-0x6d8	Correction grid config	build_initial_config
Vignetting LUTs	DMI sel=14,15	221-entry correction tables (GRR, GBB)	build_initial_config
White balance	0x6fc	Gain per channel (unity: 0x80)	build_update
Debayer	0x6f8, 0x71c	Demosaic config + coefficients	build_initial_config
Color correction	0x760	3x3 matrix (sensor->color_correct_matrix)	build_initial_config
Gamma	0x798	Bank select	build_initial_config
Gamma LUTs	DMI sel=26,28,30	64-entry curves, G/B/R (sensor->gamma_lut_rgb)	build_initial_config
Scaler Y	0xa3c	Width/height/coefficients	build_initial_config
Scaler UV	0xa68	Width/height/coefficients (2:1 subsample)	build_initial_config
Crop Y	0xe10-0xe18	Output crop window + rounding	build_initial_config
Crop UV	0xe30-0xe38	Output crop window + rounding	build_initial_config
YUV conversion	0xf30	BT.601 RGB-to-YUV matrix (12 registers)	build_common_ife_bps

Per-frame update

Register block	Offset	Description
CGC override	0x02c-0x03c	Re-written every frame
Module enables	0x040-0x04c	Re-written every frame
Demux	0x560	Re-written every frame
White balance	0x6fc	Re-written every frame
Black level	0x6b0	Re-written every frame
Crop widths	0xe0c, 0xe2c	Re-written every frame

Migration plan

Phase 1: Kernel interface

Modify camss-vfe-17x.c to expose the thin ioctl interface (VFE_WRITE_REGS, VFE_WRITE_DMI, VFE_MAP_BUF, VFE_SET_BUF, VFE_REG_UPDATE, interrupt delivery via poll). Remove vfe_pix_configure_isp() hardcoded register programming. Strip ox03c10.c kernel driver to power/probe stub.

Phase 2: camerad backend

Add a mainline backend in camerad (similar scope to current spectra.cc but simpler):

• Replace cam_packet submission with VFE_WRITE_REGS / VFE_WRITE_DMI calls
• Replace CAM_REQ_MGR_MAP_BUF with VFE_MAP_BUF
• Replace cam_sync fences with poll-based buf_done
• Replace cam-sensor-driver I2C proxy with direct /dev/i2c-*
• Keep ife.h, all sensor files, AE logic, buffer management, frame sync unchanged

Phase 3: Multi-camera

Bring up all three cameras (wide, road, driver). Implement frame synchronization across cameras using SOF timestamps (same logic as syncFirstFrame).

Open questions

1. IOMMU access: Does the mainline SMMU driver allow us to map arbitrary DMA-BUF fds into the VFE's IOMMU context from a custom ioctl, or do we need to go through DMA framework APIs?
2. CSIPHY from userspace: Should CSIPHY MIPI lane/timing config also be exposed to camerad, or is kernel-managed CSIPHY sufficient? Downstream camerad configures CSIPHY registers.
3. BPS/ICP for driver camera: The driver camera uses BPS (via ICP firmware) on downstream. On mainline, can we route all three cameras through IFE (VFE PIX), or do we need ICP support?
4. Interrupt granularity: Do we need separate SOF and buf_done events, or is composite done (SOF + both WMs done) sufficient?
5. Register write safety: Should the kernel validate register offsets (whitelist ISP-safe ranges), or trust camerad completely?