We propose a real-time RGB-based pipeline for object detection and 6D pose estimation. Our novel 3D orientation estimation is based on a variant of the Denoising Autoencoder that is trained on simulated views of a 3D model using Domain Randomization. The so-called Augmented Autoencoder has several advantages over existing methods: it does not require real, pose-annotated training data, generalizes to various test sensors and inherently handles object and view symmetries. Instead of learning an explicit mapping from input images to object poses, our method provides an implicit representation of object orientations defined by samples in a latent space. The pipeline achieves state-of-the-art performance on the T-LESS dataset both in the RGB and RGB-D domain.

Estimating the 6D pose (3D translation and 3D rotation) of objects is a fundamental task in robotic manipulation and augmented reality. Recent approaches predominantly rely on deep learning methods trained on large amounts of annotated real data, which is costly and time-consuming to obtain. Moreover, many methods struggle with symmetric objects, as multiple orientations can produce identical visual appearances.

In this work, we propose the Augmented Autoencoder (AAE), a novel approach that learns 3D object orientations implicitly from synthetic data. The key insight is that by training a denoising autoencoder with domain randomization on the input, the encoder learns to map different appearances of the same object orientation to a similar latent code, while mapping different orientations to distinct codes. This implicit representation elegantly handles symmetries without requiring explicit symmetry annotations.

Traditional 6D pose estimation methods rely on feature matching between the observed scene and known 3D models. Approaches such as point pair features (PPF) and template matching have shown success but typically require depth information and are computationally expensive. Recent deep learning approaches, including PoseCNN and SSD-6D, directly regress poses from RGB images but require large annotated training sets and do not inherently handle object symmetries.

Domain Randomization has emerged as a powerful technique for bridging the sim-to-real gap. By randomizing textures, lighting, and backgrounds during training, models learn to be robust to domain shifts. Autoencoders and their variants have been used for representation learning, but their application to 6D pose estimation with implicit symmetry handling is novel.

Our pipeline consists of three stages: (1) 2D object detection using a standard detector (e.g., SSD or RetinaNet) to localize objects; (2) 3D orientation estimation via the Augmented Autoencoder; and (3) translation estimation from the 2D bounding box. The AAE is trained by rendering the target object at various orientations and augmenting the input with domain randomization — random backgrounds, lighting changes, and color jittering — while the reconstruction target remains the clean, canonical rendering.

During inference, the encoder maps a detected object crop to a latent code. Orientation is then retrieved by finding the nearest neighbor in a codebook of pre-computed latent codes from uniformly sampled orientations. For symmetric objects, multiple orientations that produce the same appearance will be mapped to the same or very similar latent codes — hence symmetries are implicitly captured without any special treatment. The cosine similarity metric is used for efficient codebook lookup.

The loss function trains the autoencoder to reconstruct the clean rendering from the augmented input: L = ||D(E(x_aug)) - x_clean||^2, where E is the encoder, D is the decoder, x_aug is the augmented input and x_clean is the canonical rendering. This forces the encoder to learn domain-invariant features that capture only the object's intrinsic appearance and orientation, while being robust to environmental variations.

We evaluate our approach on the challenging T-LESS dataset, which features 30 texture-less industrial objects with many symmetries and mutual similarities. Our AAE pipeline achieves state-of-the-art results with an average recall of 55.6% using VSD metric in the RGB-only setting, significantly outperforming methods like SSD-6D. In the RGB-D setting, our approach reaches recall of 67.2%, competitive with methods that use much more complex post-processing.

Ablation studies demonstrate the importance of each component: removing domain randomization drops performance by 12.3%, while using explicit orientation regression instead of the codebook approach fails entirely on symmetric objects. The inference speed of 42 FPS on a single GPU confirms the real-time capability, with the autoencoder encoding step taking only 2ms per object.

We have presented the Augmented Autoencoder, a novel approach for implicit 3D orientation learning that naturally handles object symmetries without explicit symmetry annotations. Our method achieves state-of-the-art results on the T-LESS benchmark while requiring only synthetic training data. The implicit representation paradigm offers a fundamentally different perspective on 6D pose estimation, and we believe it opens promising directions for category-level pose estimation and robotic grasping applications.