Prism

A hardware-agnostic perception accelerator

ComposableNode(
    package='prism_image_proc',          # was: 'image_proc'
    plugin='prism::ResizeNode',          # was: 'image_proc::ResizeNode'
    name='resize',
    parameters=[{'width': 640, 'height': 480}],
)

Get running in three steps

# Ubuntu 22.04 · ROS 2 Humble
sudo apt install \
    ros-humble-image-transport \
    ros-humble-image-transport-plugins \
    ros-humble-image-proc \
    libgstreamer1.0-dev \
    libgstreamer-plugins-base1.0-dev \
    gstreamer1.0-plugins-base \
    gstreamer1.0-plugins-good \
    gstreamer1.0-vaapi

mkdir -p ~/prism_ws/src && cd ~/prism_ws/src
git clone https://github.com/sohams25/prism-ros.git
cd ~/prism_ws
source /opt/ros/humble/setup.bash
colcon build --packages-select prism_image_proc
source install/setup.bash

ros2 launch prism_image_proc prism_image_proc_demo.launch.py

Detect, validate, dispatch

HardwareDetector probes the host; PipelineFactory queries the live GStreamer registry via gst_element_factory_find to validate each candidate — element presence is not the same as element working on this kernel, driver, and chroma-subsampling combination. Jetson and Intel probes are two-step (prefer nvvideoconvert / vapostproc; fall back to nvvidconv / vaapipostproc). The first backend that validates wins. Per-action routing within a backend is a hand-coded table from operator A/B measurement, not an autonomous runtime optimiser.

Fallback chain

Registered components

Helpers

Parameter reference

A/B captures on two hosts

A/B captures against stock ROS 2 Humble image_proc on two hosts: 4K BGR8 input at 10 Hz, 120 s per operation, two component_container processes. Latency is per-frame from publisher stamp to subscriber receive; full methodology, per-percentile / CPU / RSS / fps data, and the per-host findings live in intel_desktop_simple_summary.md and orin_simple_summary.md.

Intel desktop, GStreamer 1.20, direct-mode fallback

Intra-process composition + DDS round-trip elimination, not GPU offload — VAAPI is in fallback mode on this GStreamer 1.20 host.

colorconvert Δ% omitted — stock baseline is a Python NumPy node that cannot drain 4K BGR8 at 10 Hz (image_proc ships no C++ colorconvert). Full per-percentile data, methodology, and the throughput-ceiling explanation in intel_desktop_simple_summary.md.

Jetson Orin Nano Super, JetPack 6.2

Per-action backend on legacy nvvidconv (resize/chain GPU, crop CPU videocrop, colorconvert GPU). The BGR-CAPS gap that drives this routing, the Round-3 colorconvert contention finding (bench-harness CPU saturation, not BGR-adapter dominance), and the empirical intra-process verification are documented in the linked summary.

Stock-side image_proc::ResizeNode does not publish frames inside this container (an image_proc packaging issue, not a Prism finding) — so resize and chain are Prism-only; colorconvert Δ% is omitted for the same structural reason as Intel. Full per-percentile data, the Round-3 contention finding, and a single-process direct-mode 9.27 ms colorconvert reference in orin_simple_summary.md.

Reproducing

python3 bench/run.py --operation resize --video /path/to/4k.mp4 \
  --duration 120 --warmup 10 --output-dir bench/results/
python3 bench/analyze.py --results-dir bench/results/ \
  --output bench/results/summary.json
python3 bench/emit_simple_summary.py --summary bench/results/summary.json \
  --host-label "<host description>" --gst-version "$(gst-launch-1.0 --version | head -1)" \
  --out bench/results/<host>_simple_summary.md

Repeat with --operation {crop,colorconvert,chain}.

Forward-looking work

Short-term to exploratory, in roughly that order.

• Extend wrapper coverage to additional image_proc operations. Per-action prism::FlipNode first, then rectify and debayer.
• GStreamer 1.22+ Intel VA-API capture. A clean A/B run on Ubuntu 24.04 where vapostproc validates and the GPU resize kernel does the work, rather than direct-mode cv::resize.
• Jetson Orin capture against an image that ships nvvideoconvert. Closes the legacy nvvidconv BGR-CAPS gap by removing the CPU videoconvert ↔ BGRx adapter on each pipeline boundary.
• AMD Ryzen A/B capture. Same harness, different host class, no VA-API path.
• Rockchip RK3588 / Qualcomm QCS 6490 backend exploration. RubikPi 3 hardware on hand; the target is a Mali-G610 / RGA path validated against the local GStreamer registry the same way VA-API is.
• ROS Buildfarm submission for binary distribution.
• Extended test coverage — integration tests beyond the current gtest unit suite, alongside the Buildfarm submission noted above.