Journal of East China Normal University(Natural Science) ›› 2023, Vol. 2023 ›› Issue (5): 100-109.doi: 10.3969/j.issn.1000-5641.2023.05.009

• System for Learning from Data • Previous Articles    

Automatic generation of Web front-end code based on UI images

Jin GE, Xuesong LU*()   

  1. School of Data Science and Engineering, East China Normal University, Shanghai 200062, China
  • Received:2023-06-30 Accepted:2023-07-22 Online:2023-09-25 Published:2023-09-15
  • Contact: Xuesong LU E-mail:xslu@dase.ecnu.edu.cn

Abstract:

User interfaces (UIs) play a vital role in the interactions between an application and its users. The current popularity of mobile Internet has led to the large-scale migration of web-based applications from desktop to mobile. Web front-end development has become more extensive and in-depth in application development. Traditional web front-end development relies on designers to give initial design drafts and then programmers to write the corresponding UI code. This method has high industry barriers and slow development, which are not conducive to rapid product iteration. The development of deep learning makes it possible to automatically generate web front-end code based on UI images. Existing methods poorly capture the features of UI images, and the accuracy of the generated code is low. To mitigate these problems, we propose an encoder–decoder model, called image2code, based on the Swin Transformer, which is used to generate web front-end code from UI images. Image2code regards the process of generating web front-end code from UI images as an image captioning task and uses Swin Transformer with a sliding window design as the backbone network of the encoder and decoder. The sliding window operation limits the attention calculation to one window, which reduces the amount of calculation by the attention mechanism while simultaneously ensuring that feature connections remain across windows. In addition, image2code generates Emmet code, which is much simpler and can be directly converted to HTML code, improving the efficiency of model training. Experimental results show that image2code performs better than existing representative models, such as pix2code and image2emmet, in the task of web front-end code generation on existing and newly constructed datasets.

Key words: UI images, Web front-end code generation, attention mechanism, intelligent Web development

CLC Number: