AI ToolBox User Manual
How to use the calibration tools of the AI ToolBox field software — ToF + RGB sensor alignment and robot hand-eye calibration — and an overview and demo of K9/K10 automatic docking.
1. Introduction
AI ToolBox is a three-layer vision platform that learns in the cloud and decides at the edge. This manual covers the tools for calibration in the field software (ROVIX) — lining up the coordinates of the robot and the vision system.
| Tool | What it does | Chapter |
|---|---|---|
| ToF + RGB alignment (Helios Overlay Tool) | Aligns a ToF depth camera with an RGB camera to produce a colour point cloud in world coordinates | 4 |
| Robot hand-eye calibration | Works out where the camera is relative to the robot — robot-mounted (Eye-in-Hand) or fixed (Eye-to-Hand) | 5 |
An overview and demo videos of K9/K10 automatic docking, built with AI ToolBox, are in chapter 6.
2. Components and requirements
| Item | Details |
|---|---|
| 3D sensor | LUCID Helios2 (ToF) + RGB camera (Triton or similar), PoE+ switch or injector, GigE connection |
| Calibration board | ChArUco board (default 12 × 9, 30 mm squares) — the software can make a printable file. Mount it on a flat board |
| Robot | Now: robot poses entered by hand (any brand) · Next: UR (RTDE), Doosan · Rainbow, ROS2 |
| PC | Ubuntu 22.04 or later · Python 3.10 · 1920 × 1080 monitor |
| Sensor driver | LUCID Arena SDK (for real sensors) |
3. Installation
A PC installed by FLOYD is ready to use. On a new PC, unpack the FLOYD package; for real sensors, install the LUCID Arena SDK first. The first run sets up its own environment.
bash 31_helios_tool.sh --sim # ToF + RGB alignment — simulation (try it without hardware)
bash 31_helios_tool.sh # ToF + RGB alignment — real sensors
bash 30_calibration_tool.sh # robot hand-eye calibration (GUI)
--sim you can practise the whole process without hardware first.4. ToF + RGB alignment
The goal is a colour point cloud in world coordinates within 30 minutes of opening the box. Work through ① → ② → ③ on the left of the window.
① Connect
- Connect — finds the Helios2 and RGB camera automatically (give a serial number if there are several).
- Use Live ▶ to check that the board is visible in both cameras. If the ToF picture is dark, change the exposure preset and hold the board at 0.5–1.0 m.
② Calibrate
- Move the board a little each time (changing angle and distance) and press Capture pair 5–8 times. If registration fails (✗), the board is outside one camera's view or blurred.
- Calibrate (①+②) — the reprojection error is shown as a traffic light.
| Mark | Reprojection error | Meaning |
|---|---|---|
| ● OK | 1 px or less | Ready to use |
| △ Caution | 2 px or less | More pairs will improve it |
| ✗ Recalibrate | Over 2 px | Collect more pairs and calibrate again |
③ Register world · check
- Place the board where you want the world origin and press Register world (check the RMS).
- Show overlay (world) — check that colour bleeding at object edges is 2 px or less and that the board lies flat on the world z = 0 plane.
- If black or light-absorbing materials show false depth, turn on 저신뢰 depth 마스킹 (low-confidence depth masking).
- Export — saves
overlay_world.ply(point cloud) andchain.yaml(alignment result).
Maintenance
- Drift check: after recalibrating, press 이전 chain과 비교 (compare with previous chain). If it changed by more than 1 mm or 0.1°, check the camera mount.
- Even for sensors with factory calibration, calibrate with this tool from time to time and compare. If they differ a lot, use this tool's result.
5. Robot hand-eye calibration
Choose whether the camera is on the end of the robot (Eye-in-Hand) or fixed to the work area (Eye-to-Hand), then move the robot through several poses while photographing the board to work out where the camera is relative to the robot. A seven-step wizard guides you.
| Step | Task |
|---|---|
| ① Configuration | Choose Eye-in-Hand / Eye-to-Hand, name the session |
| ② Devices | Connect the sensor, choose the robot driver (skip connecting for manual entry) |
| ③ Board | Enter the ChArUco or checkerboard type and size, make a printable file, check detection. For Eye-to-Hand, mount the board on the robot flange |
| ④ Collect poses | Move the robot, then Collect current pose — enter pose values by hand or from CSV. Continue until the coverage gauge fills (12 or more poses recommended, rotating in several directions) |
| ⑤ Solve | Compare the errors of five solving methods, remove outlier poses and solve again. Under 1 mm is shown in colour |
| ⑥ Verify | Overlay the point cloud in robot coordinates and check the robot axes. If something looks wrong, go back to ④ or ⑤ |
| ⑦ Save | Save the session, export as YAML, JSON or ROS2 TF |
Export formats
| Format | Used by |
|---|---|
| YAML · JSON | Keeping results, connecting to other software |
| ROS2 TF | ROS2 robot systems |
AIToolBox convention (cam_K.txt, 4 × 4 transform matrix txt) | AI ToolBox pose estimation, docking and more |
From the command line
robot-calib demo # the whole collect → solve → save process in simulation (no hardware)
robot-calib demo --eye-to-hand # fixed-camera configuration
robot-calib solve sessions/demo # solve a saved session again
6. K9/K10 automatic docking
A Pre-POC of a vision solution for automatic docking (automatic connection) of the K9 and K10, built with AI ToolBox. Before going onto real equipment, it verifies docking performance, safety and computing feasibility in a digital twin (NVIDIA Isaac Sim).
Process flow
Test scenarios
Aligned start, position and angle offsets (the K9 stops at a random lateral position within ±25 cm), lighting changes and sensor noise, reflective and featureless surfaces, a partly hidden docking port, frame drops and communication faults, and safe stop → retreat → retry on collision risk were run repeatedly in batches. Beyond the plan, 6-degree-of-freedom estimation for parking on a slope (yaw ±10°, pitch and roll ±5°) was also verified.
Results (measured in simulation)
| Metric | Target | Result |
|---|---|---|
| Position accuracy | ±5 mm @ 3 m | Docking point error median 2.4 mm · mean 4.42 mm |
| Rotation accuracy | ±1° @ 3 m | 0.02° |
| Pose accuracy rate | ±5 mm and ±1° together | 94.8 % |
| Long-range detection | — | 100 % (192/192, with the docking port partly hidden) |
| Recognition rate | 15 FPS or more | Met (capture about 12 ms) |
| Automatic insertion | — | Inserted successfully in closed-loop docking |
Demo videos
One cycle, from operating the state controller web UI to a completed docking.
| Time | Action · step |
|---|---|
| 3 s | Tick jitter — places the K9 at random around the chosen position |
| 5 s | Click K9 position ↗ upper right — the K9 is repositioned |
| 10 s | Click Run → ① Approach — the K10 drives itself to the docking point behind the K9 |
| 15 s | ② Vision inference — detects the connector and estimates its pose |
| 18–39 s | ③ Align · extend — matches the yaw, up-down and slide axes while extending the conveyor. The Lidar Sensor and RGB Camera buttons switch the camera view on the way |
| 39 s | ④ Docked — the Dock lamp turns green |
State controller web UI
| Area | Contents |
|---|---|
| Top bar | Docking residual X · Y · Z, Dock lamp, COMM · CONTROL · SUB UNIT status lamps, mode |
| SIMULATION | Run (whole process) · Approach · Dock · Stop · Reset · Conveyor Home, endless repeat |
| K9 RESET POSITION | Nine K9 stop positions, jitter (random around the chosen position) |
| MANUAL CONTROL | Yaw · Up · Slide ± — the conveyor moves only when Manual is ON |
| CAMERA | Switch between RGB Camera and Lidar Sensor views |
| CONVEYOR | Lock · Unlock, current yaw, up-down and slide values |
7. Troubleshooting
| Symptom | What to check |
|---|---|
| The ToF picture is too dark to see the board | Change the exposure preset, hold the board at 0.5–1.0 m |
| Capture pair shows ✗ | Is the board inside both cameras' views, is it blurred by movement? |
| Reprojection error over 2 px | Collect more pairs at varied angles and distances, then calibrate again |
| Colours are misaligned in the overlay | Redo ② calibration; check that the camera mount does not shake |
| False points around black objects | Turn on 저신뢰 depth 마스킹 (low-confidence depth masking) |
| Large hand-eye error | 12 or more more-varied poses (including rotation), remove outliers and solve again |
| Recalibration differs a lot from before | Over 1 mm or 0.1°: check the mount |
8. Contact
| Company | FLOYD Inc. |
| floyd@floyd.kr | |
| Phone | +82-70-8676-7976 |
| Web | floyd.kr — Review request |
To review an application with your robot and sensor combination, send us a review request and we will get back to you.