AI TOOLBOX · USER MANUAL

AI ToolBox User Manual

How to use the calibration tools of the AI ToolBox field software — ToF + RGB sensor alignment and robot hand-eye calibration — and an overview and demo of K9/K10 automatic docking.

1. Introduction

AI ToolBox is a three-layer vision platform that learns in the cloud and decides at the edge. This manual covers the tools for calibration in the field software (ROVIX) — lining up the coordinates of the robot and the vision system.

ToolWhat it doesChapter
ToF + RGB alignment (Helios Overlay Tool)Aligns a ToF depth camera with an RGB camera to produce a colour point cloud in world coordinates4
Robot hand-eye calibrationWorks out where the camera is relative to the robot — robot-mounted (Eye-in-Hand) or fixed (Eye-to-Hand)5

An overview and demo videos of K9/K10 automatic docking, built with AI ToolBox, are in chapter 6.

Preview: the tool screens are in Korean for now. Both tools have been verified end to end in simulation, and verification with real sensors and robots is under way. Screens and steps may change in the release version.

2. Components and requirements

ItemDetails
3D sensorLUCID Helios2 (ToF) + RGB camera (Triton or similar), PoE+ switch or injector, GigE connection
Calibration boardChArUco board (default 12 × 9, 30 mm squares) — the software can make a printable file. Mount it on a flat board
RobotNow: robot poses entered by hand (any brand) · Next: UR (RTDE), Doosan · Rainbow, ROS2
PCUbuntu 22.04 or later · Python 3.10 · 1920 × 1080 monitor
Sensor driverLUCID Arena SDK (for real sensors)

3. Installation

A PC installed by FLOYD is ready to use. On a new PC, unpack the FLOYD package; for real sensors, install the LUCID Arena SDK first. The first run sets up its own environment.

bash 31_helios_tool.sh --sim        # ToF + RGB alignment — simulation (try it without hardware)
bash 31_helios_tool.sh              # ToF + RGB alignment — real sensors
bash 30_calibration_tool.sh         # robot hand-eye calibration (GUI)
With --sim you can practise the whole process without hardware first.

4. ToF + RGB alignment

The goal is a colour point cloud in world coordinates within 30 minutes of opening the box. Work through ① → ② → ③ on the left of the window.

ToF + RGB alignment in three steps — connect → collect pairs → calibrate → register world → check overlay (simulation)

① Connect

  1. Connect — finds the Helios2 and RGB camera automatically (give a serial number if there are several).
  2. Use Live ▶ to check that the board is visible in both cameras. If the ToF picture is dark, change the exposure preset and hold the board at 0.5–1.0 m.

② Calibrate

  1. Move the board a little each time (changing angle and distance) and press Capture pair 5–8 times. If registration fails (✗), the board is outside one camera's view or blurred.
  2. Calibrate (①+②) — the reprojection error is shown as a traffic light.
MarkReprojection errorMeaning
● OK1 px or lessReady to use
△ Caution2 px or lessMore pairs will improve it
✗ RecalibrateOver 2 pxCollect more pairs and calibrate again

③ Register world · check

  1. Place the board where you want the world origin and press Register world (check the RMS).
  2. Show overlay (world) — check that colour bleeding at object edges is 2 px or less and that the board lies flat on the world z = 0 plane.
  3. If black or light-absorbing materials show false depth, turn on 저신뢰 depth 마스킹 (low-confidence depth masking).
  4. Export — saves overlay_world.ply (point cloud) and chain.yaml (alignment result).

Maintenance

5. Robot hand-eye calibration

Choose whether the camera is on the end of the robot (Eye-in-Hand) or fixed to the work area (Eye-to-Hand), then move the robot through several poses while photographing the board to work out where the camera is relative to the robot. A seven-step wizard guides you.

StepTask
① ConfigurationChoose Eye-in-Hand / Eye-to-Hand, name the session
② DevicesConnect the sensor, choose the robot driver (skip connecting for manual entry)
③ BoardEnter the ChArUco or checkerboard type and size, make a printable file, check detection. For Eye-to-Hand, mount the board on the robot flange
④ Collect posesMove the robot, then Collect current pose — enter pose values by hand or from CSV. Continue until the coverage gauge fills (12 or more poses recommended, rotating in several directions)
⑤ SolveCompare the errors of five solving methods, remove outlier poses and solve again. Under 1 mm is shown in colour
⑥ VerifyOverlay the point cloud in robot coordinates and check the robot axes. If something looks wrong, go back to ④ or ⑤
⑦ SaveSave the session, export as YAML, JSON or ROS2 TF

Export formats

FormatUsed by
YAML · JSONKeeping results, connecting to other software
ROS2 TFROS2 robot systems
AIToolBox convention (cam_K.txt, 4 × 4 transform matrix txt)AI ToolBox pose estimation, docking and more

From the command line

robot-calib demo                     # the whole collect → solve → save process in simulation (no hardware)
robot-calib demo --eye-to-hand       # fixed-camera configuration
robot-calib solve sessions/demo      # solve a saved session again
In simulation with 12 poses it recovered the ground truth to about 0.9 mm and 0.04°. Accuracy on real hardware depends on the sensor, robot and board.

6. K9/K10 automatic docking

A Pre-POC of a vision solution for automatic docking (automatic connection) of the K9 and K10, built with AI ToolBox. Before going onto real equipment, it verifies docking performance, safety and computing feasibility in a digital twin (NVIDIA Isaac Sim).

Process flow

Closed loop — conveyor motion feeds back into the twin commands · status Digital twin K9 · K10 3D models (USD) Isaac Sim physics simulation automated scenarios Simulation PC 1 Sensors ToF (depth) · LiDAR + RGB camera chosen from ToF · stereo · LiDAR Simulation PC (virtual sensors) 2 Perception Connector detection · segmentation ROVI Detection Inference controller · Windows GPU 3 6D pose ROVI-3D pose estimation ICP registration · confidence filter Inference controller · Windows GPU 4 State machine SEARCH → APPROACH → ALIGN → DOCK → VERIFY tilt · residual · comms check → FAILSAFE Simulation PC 5 Docking K10 conveyor 3-axis align · extend yaw · up-down · slide → connect Simulation PC State controllerOperator screen · Run / Stop · status · video
Process flow — the orange dot follows one cycle: digital twin → sensors → perception → 6D pose → state machine → docking and back into the twin. The grey line in each box shows which machine runs that step.

Test scenarios

Aligned start, position and angle offsets (the K9 stops at a random lateral position within ±25 cm), lighting changes and sensor noise, reflective and featureless surfaces, a partly hidden docking port, frame drops and communication faults, and safe stop → retreat → retry on collision risk were run repeatedly in batches. Beyond the plan, 6-degree-of-freedom estimation for parking on a slope (yaw ±10°, pitch and roll ±5°) was also verified.

Results (measured in simulation)

MetricTargetResult
Position accuracy±5 mm @ 3 mDocking point error median 2.4 mm · mean 4.42 mm
Rotation accuracy±1° @ 3 m0.02°
Pose accuracy rate±5 mm and ±1° together94.8 %
Long-range detection—100 % (192/192, with the docking port partly hidden)
Recognition rate15 FPS or moreMet (capture about 12 ms)
Automatic insertion—Inserted successfully in closed-loop docking
All figures are measured in Isaac Sim simulation. A 1,000-run insertion test and a speed comparison with manual operation will be done in the real-equipment POC.

Demo videos

One cycle, from operating the state controller web UI to a completed docking.

Dashboard — 3D view top left, state controller web UI at original size top right, operator screen (HMI) bottom right, LiDAR view and step captions bottom left
TimeAction · step
3 sTick jitter — places the K9 at random around the chosen position
5 sClick K9 position ↗ upper right — the K9 is repositioned
10 sClick Run → ① Approach — the K10 drives itself to the docking point behind the K9
15 s② Vision inference — detects the connector and estimates its pose
18–⁠39 s③ Align · extend — matches the yaw, up-down and slide axes while extending the conveyor. The Lidar Sensor and RGB Camera buttons switch the camera view on the way
39 s④ Docked — the Dock lamp turns green
3D view with the state controller web UI at bottom right (1080p)
The state controller web UI alone — button operation and changing status

State controller web UI

AreaContents
Top barDocking residual X · Y · Z, Dock lamp, COMM · CONTROL · SUB UNIT status lamps, mode
SIMULATIONRun (whole process) · Approach · Dock · Stop · Reset · Conveyor Home, endless repeat
K9 RESET POSITIONNine K9 stop positions, jitter (random around the chosen position)
MANUAL CONTROLYaw · Up · Slide ± — the conveyor moves only when Manual is ON
CAMERASwitch between RGB Camera and Lidar Sensor views
CONVEYORLock · Unlock, current yaw, up-down and slide values
The vehicles and background in the videos are simulation models. On-screen captions are in Korean.

7. Troubleshooting

SymptomWhat to check
The ToF picture is too dark to see the boardChange the exposure preset, hold the board at 0.5–1.0 m
Capture pair shows ✗Is the board inside both cameras' views, is it blurred by movement?
Reprojection error over 2 pxCollect more pairs at varied angles and distances, then calibrate again
Colours are misaligned in the overlayRedo ② calibration; check that the camera mount does not shake
False points around black objectsTurn on 저신뢰 depth 마스킹 (low-confidence depth masking)
Large hand-eye error12 or more more-varied poses (including rotation), remove outliers and solve again
Recalibration differs a lot from beforeOver 1 mm or 0.1°: check the mount

8. Contact

CompanyFLOYD Inc.
Emailfloyd@floyd.kr
Phone+82-70-8676-7976
Webfloyd.kr — Review request

To review an application with your robot and sensor combination, send us a review request and we will get back to you.