An Efficient Approach to Creating Hand-Drawn Dataset for UI Manuscript Recognition

NAVIGATE

Table of Contents

STATISTICS

Viewed2157

Downloads1812

An Efficient Approach to Creating Hand-Drawn Dataset for UI Manuscript Recognition

[HTML] PDF下载 (1812)

[1]YANG Qi,LIU Mugeng,MA Yun.An Efficient Approach to Creating Hand-Drawn Dataset for UI Manuscript Recognition[J].Journal of Zhengzhou University (Engineering Science),2022,43(06):1-7.[doi:10.13705/j.issn.1671-6833.2022.06.009]

Copy

Journal of Zhengzhou University (Engineering Science)[ISSN 1671-6833/CN 41-1339/T] Volume: 43 Number of periods: 2022 06 Page number: 1-7 Column: Public date: 2022-09-02

Title:: An Efficient Approach to Creating Hand-Drawn Dataset for UI Manuscript Recognition

Author(s):: YANG Qi¹; LIU Mugeng²; MA Yun³; 1.School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School, Shenzhen 518055, China;
2.Department of Computer Science, Peking University, Beijing 100871, China;
3. Institute for Artificial Intelligence, Peking University, Beijing 100871, China

Keywords:: intelligent software developing service; UI manuscript; recognition; object detection; dataset; data enhancement

CLC:: TP311；O244

DOI:: 10.13705/j.issn.1671-6833.2022.06.009

Abstract:: UI manuscript recognition is one of the important applications of image object detection in the area of software engineering. Due to the significant difference between UI manuscript images and natural images where UI manuscript images usually need to be drawn manually, it is difficult to build UI manuscript dataset for deep learning because of the dependency on tremendous manual efforts. To address the issue, in this study an approach called UIsketcher was proposed to efficiently generate UI manuscript dataset based on optimizing the current workflow. In UIsketcher, users should just draw some basic elements without labeling, and then the dataset could be automatically generated for training deep learning model. According to the experiment with UIsketcher, only 25% drawing workload of the traditional methods could get the similar training results. If the workload was 75%, the final accuracy was even better than that of traditional methods.

References:: [1] 魏宏彬, 张端金, 杜广明, 等. 基于改进型 YOLO v3 的蔬菜识别算法[ J] . 郑州大学学报( 工学版) , 2020, 41(2) : 7-12, 31.
WEI H B, ZHANG D J, DU G M, et al. Vegetable recognition algorithm based on improved YOLOv3[ J] . Journal of Zhengzhou university ( engineering science) , 2020, 41(2) : 7-12, 31.
[2] 楼豪杰, 郑元林, 廖开阳, 等. 基于 SiameseYOLOv4 的印刷品缺陷目标检测[ J] . 计算机应用, 2021, 41(11) : 3206-3212.
LOU H J, ZHENG Y L, LIAO K Y, et al. Defect target detection for printed matter based on SiameseYOLOv4[ J] . Journal of computer applications, 2021, 41(11) : 3206-3212.
[3] 丁明宇, 牛玉磊, 卢志武, 等. 基于深度学习的图片中商品参数识别方法[ J] . 软件学报, 2018, 29 (4) : 1039-1048.

DING M Y, NIU Y L, LU Z W, et al. Deep learning for parameter recognition in commodity images [ J ] . Journal of software, 2018, 29(4) : 1039-1048.
[4] ALES Z, LUKÁS P, ANTONÍN R. Sketch2Code: automatic hand-drawn UI elements detection with faster R-CNN[EB / OL] . ( 2020 - 09 - 22) [ 2021 - 04 - 12] . http:∥ceur-ws. org / Vol-2696 / paper_82. pdf.
[5] WIMMER C, UNTERTRIFALLER A, GRECHENIG T. SketchingInterfaces: a tool for automatically generating high-fidelity user interface mockups from handdrawn sketches [ C ] ∥32nd Australian Conference on Human-Computer Interaction. New York: ACM, 2020: 538-545.
[6] JOÃO S F, ANDRÉ R, HUGO S F. Automatically generating websites from hand-drawn mockups [ C] ∥ International Conference on Computer Vision Theory and Applications. Vienna: VISAPP,2021: 48-58.
[7] FICHOU D, BERARI R, BRIE P, et al. Overview of the 2020 imageCLEFdrawnUI task: detection and recognition of hand drawn website UIs[EB / OL] . ( 2020- 09-22) [ 2021 - 04 - 12] . http:∥ceur-ws. org / Vol - 2696 / paper_245. pdf.
[8] CHEN J S, XIE M L, XING Z C, et al. Object detection for graphical user interface: old fashioned or deep learning or a combination[C]∥Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. New York: ACM, 2020: 1202-1214.
[9] GHIASI G, CUI Y, SRINIVAS A, et al. Simple copypaste is a strong data augmentation method for instance segmentation [ C ] ∥2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR) . Piscataway: IEEE, 2021: 2917-2927.
[10] REN S Q, HE K M, GIRSHICK R, et al. Faster RCNN: towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 39(6): 1137-1149.
[11] REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[ C] ∥2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 779-788.
[12] ULTRALYTICS. YOLOv5 [ EB / OL ] . [ 2021 - 04 - 12] . https:∥github. com / ultralytics/ yolov5.

Similar References:

Memo

Last Update: 2022-10-03