Issue link: https://htpgraphics.uberflip.com/i/1385717
RAIN PROGRESS // We have a full ROS based implementation of our perception system. A stereo vision sensor is fitted to the end effector of the robot to provide a continuous stream of images, as it scans its working environment. The perception system by default can detect and recognise objects (objects are referred as known when they are a part of the dataset used to pre-train the object detection model), based on a real-time object detection pipeline along with their object poses. If any object is unknown to the system, it will give an indication to the operator to manually identify the new object and feed it to the world modelling data stream so that it can be automatically recognised subsequently. Once detected, the system is able to estimate the distances between the gripper and the detected objects, generate a dense point cloud, segment the point cloud, convert the segmented point cloud into triangulated 3D meshes and apply object textures on those meshes. UNIQUENESS // Our perception system is inspired by fully-convolutional model Yolact for real-time object class prediction, instance segmentation and 6D object pose estimation on a single RGB image. Most of the state-of-the-art approaches employ the PnP algorithm for estimating 6D pose, our method can output 6D pose directly without the PnP process, hence faster. The system takes an RGB image as input and can output object class, 2D bounding box, object mask and object rotation simultaneously. Following that, the translation of the object can be calculated by a reverse projection algorithm. Thirdly, we implemented an efficient pose refinement algorithm by aligning the object 2D projection and object mask contour, called Contour-Align, which can increase predicted 6D pose accuracy significantly. REMOTE HANDLING 59 FUTURE ASPIRATIONS // Opportunities are being explored to share the developments with industry and gain feedback. The perception system has been trialled in a mock-up work cell which offers a platform for industrial assessment. Such a configuration serves as a useful reference for future nuclear handling tasks where the containment structure is designed primarily for task effectiveness without the necessity for human interactions within the hazardous glovebox area.