AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation

Xueyi Liu1, Xiaomeng Xu1, Anyi Rao2, Chuang Gan3, Li Yi1,4,
1Tsinghua University, 2The Chinese University of Hong Kong, 3MIT-IBM Watson AI Lab 4Shanghai Qi Zhi Institute
Paper arXiv Video Code

AutoGPart builds a supervision space and search intermediate supervisions from it for generalizable 3D part segmentation networks.

Abstract

Training a generalizable 3D part segmentation network is quite challenging but of great importance in real-world applications. To tackle this problem, some works design task-specific solutions by translating human understanding of the task to machine's learning process, which faces the risk of missing the optimal strategy since machines do not necessarily understand in the exact human way. Others try to use conventional task-agnostic approaches designed for domain generalization problems with no task prior knowledge considered. To solve the above issues, we propose AutoGPart, a generic method enabling training generalizable 3D part segmentation networks with the task prior considered. AutoGPart builds a supervision space with geometric prior knowledge encoded, and lets the machine to search for the optimal supervisions from the space for a specific segmentation task automatically. Extensive experiments on three generalizable 3D part segmentation tasks are conducted to demonstrate the effectiveness and versatility of AutoGPart. We demonstrate that the performance of segmentation networks using simple backbones can be significantly improved when trained with supervisions searched by our method.

Video

Problem Overview

AutoGPart searches for intermediate supervisions for a generalizable 3D part segmentation network. By training the network with searched features encoding correct part cues, the network could perform better when parsing an instance from a novel distribution.

AutoGPart Architecture

AutoGPart builds an intermediate supervision space based on prior knowledge of 3D segmentation tasks. The space contains all operations to generate supervision features from input geometry features and ground-truth labels. Then, we optimize the supervision space to fit it to a given part segmentation network via a ``propose, evaluate, and update'' approach. In each update cycle, an operation is first sampled to generate supervision features for each point in the shape. Then, it is evaluated by training the network together with the task-related supervision and the intermediate supervision. Finally, a reward value indicating the network's performance evaluated under a cross-validation process is used to update the intermediate supervision space. After that, a greedy search-like approach is performed to extract supervision features from the optimized space for further use.

Searched Intermediate Supervisions

We plot an intermediate supervision feature searched by AutoGPart for the Primitive Fitting task on shapes from training domains and test domains. Compared to ground-truth segmentations (as well as their possible variations), features searched by our strategy present some patterns more friendly for a network to learn (e.g. discriminative across different parts, while changes continuously in a single part).

Primitive Fitting

Click here for more examples on Primitive Fitting.

Mobility-based Part Segmentation

Click here for more examples on Mobility-based Part Segmentation.

Semantic-based Part Segmentation

Results

We evaluate AutoGPart on three part segmentation tasks, namely Mobility-based Part Segmentation, Primitive Fitting, and Semantic-based Part Segmentation. The generalizability of part segmentation networks could be improved by when trained together with intermediate supervisions searched by our method.

BibTeX

@InProceedings{liu2022autogpart,
      author    = {Liu, Xueyi and Xu, Xiaomeng and Rao, Anyi and Gan, Chuang and Yi, Li},
      title     = {AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation},
      booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      month     = {June},
      year      = {2022},
      pages     = {11624-11634}
  }