Perception

The perception module provides access to all on-board sensor data through a consistent interface. Whether you are running in simulation or on hardware, the same methods return data in identical formats, making sim-to-real transfer straightforward.

Sensor Overview

The 3we reference platform includes:

• USB Camera - 640x480 RGB at 30 fps
• LD06 LiDAR - 360-degree 2D scan, 12 m range, 10 Hz
• BNO055 IMU - 9-axis (accelerometer, gyroscope, magnetometer) at 100 Hz
• OAK-D Lite (optional) - Stereo depth camera for RGB-D

Methods

get_camera_image()

Capture a single frame from the RGB camera.

image = robot.get_camera_image()
# Returns: numpy.ndarray with shape (480, 640, 3), dtype uint8, BGR color order

Parameters:

Parameter	Type	Default	Description
camera_id	int	0	Camera index if multiple cameras are attached.
encoding	str	"bgr8"	Color encoding. Options: "bgr8", "rgb8", "gray".

get_lidar_scan()

Retrieve the latest 2D LiDAR scan.

scan = robot.get_lidar_scan()
# Returns: LaserScan(ranges, angles, timestamp)
# ranges: numpy.ndarray of float32, shape (N,), in meters
# angles: numpy.ndarray of float32, shape (N,), in radians

Parameters:

Parameter	Type	Default	Description
max_range	float	12.0	Discard points beyond this distance (meters).
min_range	float	0.05	Discard points closer than this distance (meters).

get_imu()

Read the current IMU state.

imu = robot.get_imu()
# Returns: IMUReading(linear_acceleration, angular_velocity, orientation, timestamp)

All vectors are returned as numpy.ndarray of shape (3,) in the body frame. Orientation is a quaternion (x, y, z, w).

get_rgbd_image()

Capture an aligned RGB-D frame (requires depth camera).

color, depth = robot.get_rgbd_image()
# color: numpy.ndarray, shape (H, W, 3), uint8
# depth: numpy.ndarray, shape (H, W), float32, meters

Parameters:

Parameter	Type	Default	Description
align	bool	True	Align depth to color frame.
max_depth	float	5.0	Clip depth values beyond this range.

Subscribing to Continuous Data

For real-time pipelines, use callbacks instead of polling:

def on_scan(scan):
    print(f"Got {len(scan.ranges)} points")

robot.subscribe("lidar", on_scan)

Available topics: "lidar", "camera", "imu", "rgbd".

Coordinate Frames

All sensor data is returned in the sensor’s own frame by default. Use robot.transform(data, target_frame="base_link") to convert to the robot base frame.