CLIPXPlore: Coupled CLIP and Shape Spaces for 3D Shape Exploration

Jingyu Hu*¹

Ka-Hei Hui*¹

Zhengzhe Liu¹

Hao Zhang²

Chi-Wing Fu¹

(* joint first authors)

SIGGRAPH Asia 2023 (Conference Track)

CLIPXPlore demonstrates new capabilities of leveraging a vision-language model to explore the 3D shape space: binary-attribute-guided, text-guided, and sketch-guided.

Overview of CLIPXPlore. (a) summarizes our employed data. (b) connects the CLIP and shape spaces. (c) co-optimizes the CLIP code 𝑐 and its coupled shape code 𝑧. (d) locates the latent direction associated with the given condition, allowing us to explore the shape space.

Abstract

This paper presents CLIPXPlore, a new framework that leverages a vision-language model to guide the exploration of the 3D shape space. Many recentmethods have been developed to encode 3D shapes into a learned latentshape space to enable generative design and modeling. Yet, existing methods lack effective exploration mechanisms, despite the rich information. To this end, we propose to leverage CLIP, a powerful pre-trained vision-language model, to aid the shape-space exploration. Our idea is threefold. First, we couple the CLIP and shape spaces by generating paired CLIP and shape codes through sketch images and training a mapper network to con-nect the two spaces. Second, to explore the space around a given shape, we formulate a co-optimization strategy to search for the CLIP code that better matches the geometry of the shape. Third, we design three exploration modes, binary-attribute-guided, text-guided, and sketch-guided, to locate suitable exploration trajectories in shape space and induce meaningful changes to the shape. We perform a series of experiments to quantitatively and visually compare CLIPXPlore with different baselines in each of the three exploration modes, showing that CLIPXPlore can produce many meaningful exploration results that cannot be achieved by the existing solutions.

Results

Results on binary-attribute-guided exploration. Our explored shapes are produced to match the binary attribute conditions; see the armrest growing from the chair and the drawer/shelf growing on the tables.

Results on text-guided exploration. By providing a diverse set of target shape descriptions, we can explore various input shapes to produce results that satisfy the corresponding description, e.g., the overall appearance of shapes (truck and jeep, short v.s. tall tables), and various topological structures (spindles/stretchers).

Results on sketch-guided exploration. Our framework can support changing the shape’s local properties, such as adding engines to the airplane and changing the top of the car. We can also add or remove some local structures, such as stretchers on tables

Paper and Supplementary Material

CLIPXPlore: Coupled CLIP and Shape Spaces for 3D Shape Exploration
In SIGGRAPH Asia 2023.
[Paper]

Acknowledgments

The authors thank the anonymous reviewers for their valuable comments. We also acknowledge the help from TANG Wai Lun for the data preparation. This work is supported by Shenzhen Portion of Shenzhen-Hong Kong Science and Technology Innovation Co- operation Zone (Project No. HZQB-KCZYB-20200089), Research Grants Council of the Hong Kong Special Administrative Region (Project no. CUHK 14206320 & 14201921), and Natural Sciences and Engineering Research Council of Canada (Project No. 611370).