ROVI-3D · WHITE PAPER

ROVI-3D White Paper

The architecture of ROVI-3D, which estimates the 6D pose of piled parts and picks them, and the physically based rendering (PBR) that creates its training data without on-site capture — explained with diagrams and video.

1. Summary

ROVI-3D is a vision system that looks at 3D parts piled at random in a bin with a depth camera, estimates each part's 6D pose (3 for position + 3 for orientation), and lets a robot pick them. It rests on two things.

KeyDetails
ArchitectureThe on-site device (ROVI-3D Vision) runs detection → 6D pose → grasp decision in real time; the training server (ROVI.net) builds the models and sends them down
PBR trainingPhysically based rendering (PBR) images are generated automatically from the part's CAD for training — from thousands up to hundreds of thousands of images, overlapping (occluded) scenes included, with labels

ROVI-3D's 6D pose estimation model entered the TOP 10 among commercial technologies in the BOP Challenge, the global 6D pose estimation benchmark (2023).

2. Why it is hard

Bin picking — picking parts one by one from a bin they were dumped into — is one of the hardest tasks in robot vision.

DifficultyDetails
Overlap and crossingParts hide and lean on each other, so only part of each is visible
Random poseNo fixed orientation, so all six degrees of freedom must be estimated
MaterialsMetal and glossy surfaces make the image and depth unstable through reflections
Picking orderPicking a part from underneath collapses those on top — order must be judged too
Training dataAI training needs many images with pixel-level labels (masks, 6D poses); people practically cannot make these labels, and on-site capture costs time and money

ROVI-3D solves the first four with its architecture and the last with PBR synthetic data.

3. ROVI-3D architecture

ROVI-3D has two parts: an on-site device and a training server.

ROVI-3D Vision (on-site device) ROVI.net server (training) ToF / RGB-D camera colour + depth Object detection find target regions 6D pose estimation FFB6D-PBR Grasp decision grasp point · order RL model Robot · 3D bin picking ROVIX 2.0 · hand-eye Inference (on site, real time) PBR training images auto-generated from CAD thousands → 100,000s Pose model training detection · 6D pose Grasp model training deep reinforcement learning Training frameworks · PyTorch etc. Training (server) upload site data download trained models
ROVI-3D architecture — orange: real-time inference on site (camera → object detection → 6D pose → grasp decision → robot); blue: the ROVI.net server trains models on PBR images and sends them to the on-site device
PartWhat it does
ROVI-3D Vision (on-site device)Detects parts and estimates their 6D pose from the colour + depth of a ToF / RGB-D camera, chooses the grasp point and order, and passes them to the robot. Real time on a compact high-performance PC
ROVI.net server (training)Generates PBR training images from the part CAD automatically and trains the pose and grasp models, using common frameworks such as PyTorch
Data flowDevice → server: upload site data / server → device: download trained models
ROVIX 2.0Operator software — camera and hand-eye calibration, NVIDIA Isaac, ROS and Unity integration

Here are the inference and training flows with real images.

ROVI-3D processing — top (ROVI-3D, inference): colour image → object detection → 6D pose estimation → 3D pose and segmentation. Bottom (training): simulation data generation (Unity + NVIDIA Isaac) → object training → into the pose model

4. 6D pose estimation — FFB6D-PBR

ROVI-3D's pose estimation is the FFB6D architecture (Full Flow Bidirectional 6D, He et al., CVPR 2021) trained on PBR synthetic data (FFB6D-PBR).

colour image H × W × 3 depth → point cloud N × 3 colour branch (CNN) geometry branch (point network) bidirectional fusion at every layer — appearance ↔ geometry per-point object labels 3D keypoint voting least-squares fit 6D pose (R, t)
FFB6D-PBR structure — the colour image goes through a CNN branch and the point cloud made from depth through a point-network branch, with the two branches exchanging information at every layer (bidirectional fusion). From the fused features each point is labelled by object and votes for 3D keypoints, and a least-squares fit gives the rotation R and translation t.
  1. Two inputs: the colour image carries the surface's texture and colour (appearance); the point cloud from depth carries its shape (geometry).
  2. Bidirectional fusion at every layer: most methods merge the two once at the end; FFB6D exchanges appearance ↔ geometry at every layer. When reflections disturb the image, geometry helps; when depth is lacking, appearance helps.
  3. 3D keypoint voting: every point votes for the location of predefined keypoints on the part surface. Even when part of it is hidden, the visible points' votes find the keypoints, so it copes with overlapping parts.
  4. Least-squares fit: matching the found 3D keypoints with the CAD model's keypoints gives the 6D pose (R, t).
The BOP Challenge is an international benchmark comparing 6D pose estimation methods on several public datasets under the same rules. ROVI-3D entered the TOP 10 among commercial technologies (2023, submission).

5. Grasp decision

Knowing the pose still leaves where, and in what order to pick.

6. What physically based rendering (PBR) is

PBR (Physically Based Rendering) makes images by computing how light is reflected and absorbed at surfaces in line with physics. It is the same technology behind realistic game and film graphics; for AI training, what matters is that it produces images close to what a real camera would capture.

smooth surface (low roughness) aligned microfacets reflect in one direction — a sharp highlight rough surface (high roughness) scattered microfacets spread the reflection — a soft highlight ● metal: reflection takes the metal colour, no diffuse light ● plastic: white reflection (about 4 %), the rest is diffuse in the surface colour energy conservation — reflected + diffuse ≤ incoming light
Microfacet model — a surface is a collection of tiny facets. On a smooth surface the facets are aligned and reflection gathers in one direction (left); on a rough surface they are scattered and reflection spreads (right).
PBR elementMeaning
Energy conservationReflected plus diffuse light can never exceed the incoming light — brightness stays physically right
Microfacet reflection (GGX)Roughness sets the size and sharpness of highlights
Fresnel reflectionReflection grows at grazing angles — metals reflect in their own colour, plastics give about 4 % white reflection
MetallicMetals only reflect, with no diffuse light; non-metals give diffuse light in the surface colour + white reflection
Shadows and interreflectionSoft shadows depending on light size, light bouncing between objects
PBR material chart — back: metal (metallic 1), front: plastic (metallic 0), roughness 0.05 → 0.90 from left to right. Under the same light, highlights and colour change with the material values (a simple PBR renderer made for this white paper).
PBR material chart — back: metal (metallic 1), front: plastic (metallic 0), roughness 0.05 → 0.90 from left to right. Under the same light, highlights and colour change with the material values (a simple PBR renderer made for this white paper).

Give a part's CAD model just its colour, metallic and roughness, and the glint of a metal bracket or the soft sheen of a plastic housing is reproduced close to reality. That is why a model trained on synthetic images also works on real camera images.

7. Making training data with PBR

CAD model 3D model of the part material (colour · metallic · roughness) 1 Physics simulation drop and pile into a bin overlaps generated automatically 2 Randomization light · material · camera background · part count 3 PBR rendering reflections · shadows · interreflection close to a real camera 4 Automatic labels depth · mask · 6D pose exact occlusion ratio 5 Training thousands → 100,000s no on-site capture 6 when site results show weak scenes, generate again under those conditions
PBR training data generation — CAD model → physics simulation (drop and pile into a bin) → randomization → PBR rendering → automatic labels → training. Scenes where site results are weak are generated again under those conditions.
  1. Physics simulation: CAD parts are dropped into a virtual bin so they overlap and lean as in reality. Occluded scenes are created automatically.
  2. Domain randomization: light position, strength and colour, floor and bin colours, part materials, camera position and part count change from scene to scene, so the model does not get used to one environment and learns many spaces and positions.
  3. PBR rendering: colour and depth images are rendered with reflections and shadows.
  4. Automatic labels: the renderer knows where every part is, so it saves depth, object masks, 6D poses and occlusion ratios exactly, pixel by pixel. No human labelling is needed.
  5. Scale: thousands up to hundreds of thousands of images are generated automatically to suit the environment.
PBR randomization for illustration — light, colour, material, camera and part layout change with every scene, and RGB, depth, object masks and 6D pose labels (boxes and axes) come out together with the render. 16 scenes rendered with a simple PBR renderer made for this white paper (actual training data is generated with the NVIDIA Isaac and Unity renderers).
On-site captured dataPBR synthetic data
PreparationParts, equipment and capture time neededReady as soon as CAD exists
LabelsHuman labelling — 6D poses practically impossibleAutomatic from the renderer, pixel-exact
VolumeHundreds to thousands of imagesThousands to hundreds of thousands
VarietyOnly the captured environmentLight, material and layout as varied as you like
OverlapOnly what happens by chanceDeliberately, by physics simulation

8. From simulation to the site

Models trained on synthetic data and the robot's motions are checked first in a digital twin. ROVI-3D and AI ToolBox work with NVIDIA Isaac Sim, Unity and ROS, so robot motions in simulation can be carried over to the real robot.

Digital twin — a robot in NVIDIA Isaac Sim (left) and a real collaborative robot making the same motion (bottom right). Motion is planned in simulation and the real robot is controlled in real time (FLOYD recording).

9. Strengths and limits

Strengths

StrengthWhy
Copes with overlapping partsKeypoint voting estimates pose from partial views; occluded scenes are learned in advance from training data
Minimal on-site capture and labellingPBR data generated automatically from CAD — new parts can be added quickly
Appearance + geometryColour and depth fused at every layer — handles both reflective and textureless parts
Down to picking orderGrasp model and work-order decision reduce collapses and collisions
Simulation linkIsaac, Unity, ROS — verified first in a digital twin

Limits

LimitDetailsWhat to do
CAD neededPBR data requires a 3D model of the partObtain a model by 3D scanning
Clear or strongly reflective partsThe depth camera itself loses depthPolarization-camera-based ROVI-Polar (in development)
Sim-to-real gapPerformance drops if materials or lighting differ greatly from realityDomain randomization, adding site data
ComputingReal-time inference needs a GPU-class PCCompact high-performance PC, Jetson
MaturityPOC stageSite POCs under way (chapter 10)

10. Applications and maturity

ItemDetails
ApplicationsVision guided robot (VGR), AI 6D pose estimation, AI grasping and 3D bin picking
MaturityPOC — development completed (2023)
POCs under wayMyungshin (Gunsan), LG Innotek (Startup TechBlaze)
BenchmarkBOP Challenge 6D pose estimation — TOP 10 among commercial technologies worldwide (2023)

Where the 2D-camera-based ROVI-2D/2.5D handles parts of known shape quickly and at low cost, ROVI-3D takes 3D parts piled at random. Clear parts are for ROVI-Polar, and AI ToolBox brings the vision systems together on one platform.

11. Configuration

ItemDetails
CameraToF or RGB-D camera
DeviceHigh-performance barebone PC
UXROVIX 2.0 — hand-eye and camera calibration, NVIDIA Isaac, ROS and Unity integration
OSUbuntu 18.04 or later / Windows
TrainingROVI.net server — PBR data generation, pose and grasp model training

Send application review requests through floyd.kr contact.

12. Glossary

TermMeaning
6D poseAn object's position (x, y, z) + orientation (rotation about 3 axes)
Bin pickingPicking parts one by one from a bin they were put into at random
RGB-D, ToFA camera giving colour and depth together; a camera measuring depth from the round-trip time of light
Point cloudA depth image turned into a set of 3D points
KeypointA predefined reference point on an object's surface — the pose is computed from where these points are
PBRPhysically based rendering. Making images by computing light reflection in line with physics
Microfacet, roughnessThe tiny facets making up a surface; how scattered those facets are
MetallicWhether a material is metal (1) or not (0) — sets the balance of reflection and diffuse light
Domain randomizationRandomly varying light, material, camera and so on in synthetic scenes so the model learns many environments
Digital twinA virtual model reproducing real equipment or robots
BOP ChallengeThe international benchmark for 6D pose estimation