We published an extended CVIU journal based on SHREC'19 and SHREC'18 Sketch/Image Tracks! Please see the CVIU paper on the bottom.

The objective of this track is to evaluate the performance of different 2D scene sketch-based 3D scene retrieval algorithms using a 2D sketch query dataset and a 3D Warehouse model dataset.

2D scene sketch-based 3D scene retrieval is to retrieve relevant 3D scenes (in either .OBJ or .SKP format) using a 2D scene sketch as input. This scheme is intuitive and convenient for users to learn and search for 3D scenes. It is also very promising and has great potentials in many applications such as autonomous driving car, 3D scene reconstruction, 3D geometry video retrieval, virtual reality (VR) and augmented reality (AR) in 3D Entertainment like Disney World's Avatar Flight of Passage Ride [1] [2] [3].

However, although there are many existing 2D sketch-based 3D shape retrieval systems, there is little existing research work on 2D scene sketch-based 3D scene retrieval due to two major reasons: 1) It is challenging to collect a large-scale 3D scene dataset and there exists very limited number of available 3D scene shape benchmarks. 2) Like 2D sketch-based 3D shape retrieval, there is a big semantic gap between the iconic representation of 2D scene sketches and the accurate 3D coordinate representations of 3D scenes. All of above reasons make the task of retrieving 3D scene models using 2D scene sketch queries a challenging, although interesting and promising, research direction.

Ye et al. [4] collected the Scene250 benchmark comprising 250 2D scene sketches of 10 classes, each with 25 sketches. It avoids the bias issue since they collected the same number of sketches for every class, while the sketch variation within one class is also adequate enough. However, there are no 3D scene dataset corresponding to this 2D scene sketch dataset.

Motivated by above obstacles, 100 3D scene models have been selected for each of the ten classes in Scene250 from 3D Warehouse [5], an open source library which allows SketchUp users to upload 3D models to share and download needed 3D models. The SketchUp (.SKP) type of the online 3D scene models can be transformed into many other formats, such as OBJ, PLY, and OFF.

We organize this track to foster this challenging research direction by soliciting retrieval results from current state-of-the-art 3D scene retrieval methods for comparison, especially in terms of scalability to 2D scene sketch queries. We will also provide corresponding evaluation code for computing a set of performance metrics similar to those used in the Query-by-Model retrieval technique.

Our 2D scene sketch-based 3D scene retrieval benchmark SceneSBR utilizes the 250 2D Scene sketches in Scene250 [4] as its 2D scene sketch dataset and 1000 SketchUp 3D scene models (.OBJ and .SKP format) as its 3D scene dataset. Each of the ten classes has the same number of 2D scene sketches (25) and 3D scene models (100).

To facilitate learning-based retrieval, we randomly select 18 sketches and 70 models from each class for training and use the remaining 7 sketches and 30 models for testing, as indicated in Table 1. Participants need to submit results on the testing dataset only if they use learning in their approach(es). Otherwise, the retrieval results based on the test (70 sketches, 300 models) and complete (250 sketches, 1000 models) datasets are needed. To provide a complete reference for future users of our benchmark, we will evaluate the participating algorithms on both the testing dataset (7 sketches and 30 models per query) and the complete SceneSBR benchmark (25 sketches and 100 models per class).
  Table1. Training and testing datasets (per class) of our SceneSBR benchmark.

The 2D scene sketch query set comprises 250 2D scene sketches (10 classes, each with 25 sketches), while all the classes have relevant models in the target 3D scene dataset which are downloaded from the 3D Warehouse. One example per class is demonstrated in Fig. 1.

Fig. 1 Example 2D scene sketches (one example per class, shown in one view) [4] in our SceneSBR benchmark.

The 3D scene dataset is built on the selected 1000 3D scene models downloaded from 3D Warehouse. Each class has 100 3D scene models. One example per class is shown in Fig. 2.

Fig. 2 Example 3D scene models (one example per class) in our SceneSBR benchmark.

To have a comprehensive evaluation of the retrieval algorithm, we employ seven commonly adopted performance metrics in 3D model retrieval technique. They are Precision-Recall (PR) diagram, Nearest Neighbor (NN), First Tier (FT), Second Tier (ST), E-Measures (E), Discounted Cumulated Gain (DCG) and Average Precision (AP). We also have developed the code to compute them.

The complete dataset will be made available on the 1^st of February and the results will be due in three weeks after that. Every participant will perform the queries and send us their retrieval results. We will then do the performance assessment. Participants and organizers will write a joint SHREC track competition report to detail the results and evaluations. Results of the track will be presented during the Eurographics 3DOR Workshop 2018 in Delft, Netherlands.

The following list is a step-by-step description of the activities:

• The participants must register by sending a message to juefei.yuan@gmail.com and Juefei Yuan. Early registration is encouraged, so that we get an impression of the number of participants at an early stage.
• The database will be made available via this website. Dataset.
• Participants will submit the rank lists on the test (for learning-based methods), or on both the test and the complete (for non-learning based approaches) datasets. Up to 5 matrices, either for the training or testing datasets, per group may be submitted, resulting from different runs. Each run may be a different algorithm, or a different parameter setting. More information on the dissimilarity matrix file format. More information on the rank list file format.
• Participants write a one-page description of their method with at most two figures and submit it at the same time when they submit their running results.
• The evaluations will be done automatically.
• The organization will release the evaluation scores of all the runs.
• The track results are combined into a joint paper, and then published in the proceedings of the Eurographics Workshop on 3D Object Retrieval after reviewed by the 3DOR and SHREC organizers.
• The description of the track and its results are presented at the 2018 Eurographics Workshop on 3D Object Retrieval (April 15-16, 2018).

 

January 22	- Call for participation.
January 25	- A few sample 2D scene sketches and 3D scene models will be available online.
January 31 (Extended!)	- Please register before this date.
February 1, 8:00 PM (UTC-6)	- Distribution of the database. Participants can start the retrieval or train their algorithms.
February 22, 11:59 PM (UTC-6)	- Submission of the results on the test (for learning-based methods) or test & complete (for non-learning based approaches) datasets and one-page description of their method(s).
February 25, 11:59 PM (UTC-6)	- Release of evaluation scores.
February 26	- Track is finished and results are ready for inclusion in a track report.
February 28	- Submit the track report for review.
March 3	- Reviews done, feedback and notifications given.
March 10	- Camera-ready track paper submitted for inclusion in the proceedings.
April 15-16	- Eurographics Workshop on 3D Object Retrieval 2018, featuring SHREC'2018.

Juefei Yuan - University of Southern Mississippi, USA
Bo Li - University of Southern Mississippi, USA
Yijuan Lu - Texas State University, USA

[1] Wikipedia. Avatar Flight of Passage. https://en.wikipedia.org/wiki/Avatar_Flight_of_Passage
[2] Youtube. Disney World Animal Kingdom: Avatar Flight of Passage Ride Video. https://www.youtube.com/watch?v=f-cw7iCUY3c
[3] Youtube. Pre-show in Pandora - The World of Avatar at Walt Disney World. https://www.youtube.com/watch?v=eM8f47Igtu8
[4] Yuxiang Ye, Yijuan Lu and Hao Jiang. Human's Scene Sketch Understanding. Annual ACM International Conference on Multimedia Retrieval (ICMR), 355-358, 2016
[5] 3D Warehouse. https://3dwarehouse.sketchup.com/?hl=en.

[1] Juefei Yuan, Hameed Abdul-Rashid, Bo Li, Yijuan Lu, Tobias Schreck, Song Bai, Xiang Bai, Ngoc-Minh Bui, Minh N. Do, Trong-Le Do, Anh-Duc Duong, Kai He, Xinwei He, Mike Holenderski, Dmitri Jarnikov, Tu-Khiem Le, Wenhui Li, Anan Liu, Xiaolong Liu, Vlado Menkovski, Khac-Tuan Nguyen, Thanh-An Nguyen, Vinh-Tiep Nguyen, Weizhi Nie, Van-Tu Ninh, Perez Rey, Yuting Su, Vinh Ton-That, Minh-Triet Tran, Tianyang Wang, Shu Xiang, Shandian Zhe, Heyu Zhou, Yang Zhou, Zhichao Zhou. A Comparison of Methods for 3D Scene Shape Retrieval. Computer Vision and Image Understanding, Vol. 201, December, 2020.

[2] Juefei Yuan, Bo Li, Yijuan Lu, Song Bai, Xiang Bai, Ngoc-Minh Bui, Minh N. Do, Trong-Le Do, Anh-Duc Duong, Xinwei He, Tu-Khiem Le, Wenhui Li, Anan Liu, Xiaolong Liu, Khac-Tuan Nguyen, Vinh-Tiep Nguyen, Weizhi Nie, Van-Tu Ninh, Yuting Su, Vinh Ton-That, Minh-Triet Tran, Shu Xiang, Heyu Zhou, Yang Zhou, Zhichao Zhou, In: Alex Telea, Theoharis Theoharis and Remco Veltkamp (eds.), SHREC'18 Track: 2D Scene Sketch-Based 3D Scene Retrieval, Eurographics Workshop on 3D Object Retrieval 2018 (3DOR 2018), Delft, The Netherlands, April 16, 2018 (PDF, Slides, BibTex)

If you have any question, please email to juefei.yuan@gmail.com and Juefei Yuan