We introduce an approach for selecting objects in neural volumetric 3D representations, such as multi-plane images (MPI) and neural radiance fields (NeRF). Our approach takes a set of foreground and background user scribbles in one or more views and automatically estimates a 3D segmentation of the desired object, which can be rendered into novel views. To achieve this result, we propose a novel voxel feature embedding that incorporates multi-view features from the neural volumetric 3D representation and image features from all input views. To evaluate our approach, we introduce a new dataset of human-provided segmentation masks for depicted objects in real-world multi-view scene captures. We show that our approach out-performs strong baselines, including 2D segmentation and 3D segmentation approaches adapted to our task.

Approach Overview. For each voxel in the 3D volume, we first compute a voxel feature embedding (left). We then train a 3D segmentation network to classify each voxel into foreground or background using as supervision the partial user scribbles (center). We apply the learned classifier and further refine the result using a 3D graph-cut which uses the 3D distance field of the scribbles (right).

For each scene, we show the reference view image and the input scribbles on the left. We then show the 2D mask and corresponding foreground image computed by different methods. On the right-most column, we show the ground-truth. Note that the foreground segmentation of the 2D baselines are not rendered; we apply the inferred mask to the ground truth test image.

Results on the NeRF-real360 using PlenOctrees as neural volumetric representation.

Scribbles: we annotate a fixed set of foreground-background scribbles per image. For each scene, we have one pair of foreground and background scribbles for the reference view. Note that a scribble may contain several strokes. Mask: we obtain the high-quality ground-truth annotations using a professional service.