Perception & Sensors

The Perception & Sensors group lets the robot know where it is and what is around it. Members choose and mount sensors, calibrate them, and write software that turns raw camera images, LiDAR scans, GNSS signals and IMU readings into positions, maps and detected objects.

Good perception is what makes autonomy possible. An autonomous Robocon robot has to find blocks and building spots; a drone near tall buildings has to keep an accurate position where satellite signals are blocked; a guide robot dog has to see kerbs and stairs.

Mentors: Dr. Huang Feng, Xiao Naigui
Student representatives: To be announced after the 2026 placement
Tools: ROS 2, OpenCV, LiDAR, RTK GNSS, depth cameras, IMUs
Works on: Autonomous robot localisation for Robocon, drone positioning near buildings, vehicle mapping, robot dog obstacle detection

What the group works on

Sensor selection and mounting

Choosing cameras, LiDAR, GNSS and IMU for each robot, and mounting them rigidly.

Calibration

Camera intrinsics, camera-LiDAR and camera-IMU extrinsics, and time offsets between sensors.

Localisation

GNSS and RTK, inertial navigation and visual-inertial fusion, indoors and outdoors.

Mapping and SLAM

Building 2D and 3D maps and keeping the robot localised within them.

Detection and tracking

Finding game objects, obstacles, people, kerbs and stairs in camera and LiDAR data.

Data and evaluation

Time synchronisation, data logging, datasets and measuring accuracy against ground truth.

Engineering foundations

A few of the equations members use in this group. Training sessions build up to them step by step.

Pinhole camera projection

\[s\begin{bmatrix}u\\ v\\ 1\end{bmatrix} = K\,[\,R \mid t\,]\begin{bmatrix}X\\ Y\\ Z\\ 1\end{bmatrix}\]

How a 3D point appears in an image. Calibration estimates the intrinsic matrix K and the pose R, t.

Kalman filter: predict

\[\begin{aligned} \hat{x}_{k|k-1} &= F\,\hat{x}_{k-1} + B\,u_k \\ P_{k|k-1} &= F\,P_{k-1}F^{\top} + Q \end{aligned}\]

Propagates the robot state and its uncertainty with a motion model, for example from IMU or wheel odometry.

Kalman filter: update

\[\begin{aligned} S_k &= H P_{k|k-1} H^{\top} + R \\ K_k &= P_{k|k-1}H^{\top} S_k^{-1} \\ \hat{x}_k &= \hat{x}_{k|k-1} + K_k\left(z_k - H\,\hat{x}_{k|k-1}\right) \end{aligned}\]

Corrects the prediction with a measurement z such as a GNSS fix or a visual landmark.

GNSS pseudorange

\[\rho_i = \lVert \mathbf{p}_i - \mathbf{p} \rVert + c\,\delta t + \varepsilon_i\]

Distance to satellite i plus receiver clock error. Near tall buildings, reflections make ε large: a core research topic at PolyU.

Point cloud registration (ICP)

\[\min_{R,\,t}\; \sum_i \left\lVert R\,\mathbf{p}_i + t - \mathbf{q}_i \right\rVert^2\]

Aligns a new LiDAR scan with a map or the previous scan, the basis of LiDAR odometry and mapping.

Stereo depth

\[Z = \frac{f\,B}{d}\]

Depth from a stereo camera with focal length f, baseline B and pixel disparity d.

What we expect from members

Expectations grow with the member tier. Tiers are reviewed regularly, so members can move up as their skills grow.

Tier 3 · General

Interest in sensors and programming. Learns Linux and ROS 2 basics in introductory sessions.

Tier 2 · Training

Programs in Python or C++, uses Linux and ROS 2, can calibrate a camera and record and replay a sensor dataset.

Tier 1 · Advanced

Designs the perception and localisation pipeline of a robot, evaluates its accuracy, and mentors Tier 2 members.

Before you join: Linear algebra, probability, Python or C++ and some Linux experience are good starting points. No member is expected to know everything at the start.

Photos

Obstacle-avoidance test with a small ground robot.

Drone positioning beside a tall building.

A quadruped robot with sensors on campus.