http://orcid.org/0000-0001-6778-3650
Figures
Abstract
Aesthetic perception is a human instinct that is responsive to multimedia stimuli. Giving computers the ability to assess human sensory and perceptual experience of aesthetics is a well-recognized need for the intelligent design industry and multimedia intelligence study. In this work, we constructed a novel database for the aesthetic evaluation of design, using 2,918 images collected from the archives of two major design awards, and we also present a method of aesthetic evaluation that uses machine learning algorithms. Reviewers’ ratings of the design works are set as the ground-truth annotations for the dataset. Furthermore, multiple image features are extracted and fused. The experimental results demonstrate the validity of the proposed approach. Primary screening using aesthetic computing can be an intelligent assistant for various design evaluations and can reduce misjudgment in art and design review due to visual aesthetic fatigue after a long period of viewing. The study of computational aesthetic evaluation can provide positive effect on the efficiency of design review, and it is of great significance to aesthetic recognition exploration and applications development.
Citation: Xing B, Zhang K, Zhang L, Wu X, Si H, Zhang H, et al. (2020) And the nominees are: Using design-awards datasets to build computational aesthetic evaluation model. PLoS ONE 15(1): e0227754. https://doi.org/10.1371/journal.pone.0227754
Editor: Zhaoqing Pan, Nanjing University of Information Science and Technology, CHINA
Received: June 19, 2019; Accepted: December 27, 2019; Published: January 21, 2020
Copyright: © 2020 Xing et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data underlying the study is deposited on Figshare at https://doi.org/10.6084/m9.figshare.9741275.
Funding: This study is partly supported by the Natural Science Foundation of Zhejiang Province of China (LY19F020047, LZ19F020002), National Natural Science Foundation of China (61402141), the Philosophy and Social Science Planning Fund Project of Zhejiang Province (20NDQN260YB) and the Fundamental Research Funds for the Provincial Universities of Zhejiang (GB201901006). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Computer-aided design evaluation is becoming a well-recognized request in the intelligent design industry. Such a evaluation tool can be used as an intelligent assistant to human assessors, helping reduce misjudgments due to visual aesthetic fatigue and efficiently performing the manual work of primary screening. Vast amounts of design concepts have been created and nurtured for submission to various design awards competitions. The archives of original design submissions are rich potential data resource for aesthetic-aware modeling.
Design is an interdisciplinary major, combining engineering and art. Using visual and affective perception, human beings establish the direct image of design works, while on the other hand, layouts following design principles reveal certain forms created by rational thinking. Various factors should be considered in the concept creation process, including human factors, ergonomics, environment, psychology, and safety etc [1, 2]. Thus, excellent design layout is the creation of both art and engineering, and this combination of patterns is quite challenging for computer-based assessment study. Among all the factors, visual aesthetics is proved to be critical in product design evaluation [3, 4] by physiological analysis approaches [5] and user experience study [6]. The existing studies indicated that visual aesthetics is significantly influencing user preference [7] and stimulating users’ purchase behavior [8–10], which has a crucial effect on promoting product acceptance [11]. Consequently, quantifying the visual aesthetics of design works by computational means is promising for various industries.
In this study, we explore an aesthetic-aware model of design assessment using image-feature analysis. The main contributions of this work include the following content. A total of 2,918 images of original design works were collected from archived submissions to two industrial design awards to form two databases for design evaluation, and multiple machine-learning methods were compared to find the optimal method for automatic design grading. Specifically, the ranking information and reviewers’ ratings are the natural classification annotations of these design images. In this experiment, the following image features were extracted as hand-crafted features for aesthetic modeling by LibSVM, LibLinear, RBFNetwork and RandomSubSpace-Randomforest: local binary pattern (LBP), color histogram (HIST), and hue saturation value (HSV). VGG-19 and ResNet-50 were also used in the design aesthetic classification learning. The experimental results were assessed as optimal, attaining a classification accuracy of 80.19% on average, through the applying use of ResNet-50 in the dataset of submissions to the Electronic Home Applicants Design Awards. The methodology was then verified with the use of a second dataset, taken from submissions to the Electronic Tools Design Awards. The modeling performance was found to be stable, with an accuracy of 84.19% in selecting the nominees.
This paper is organized as follows: Section 2 summarizes the previous work on aesthetic evaluation using image processing methods; Section 3 introduces the methods of feature extraction and algorithms, including LibSVM, LibLinear, RBFNetwork, RandomSubSpace, VGG-19 and ResNet-50; Section 4 gives the experimental procedures, including reviewers’ evaluations based on design criteria and the design works evaluation using multi-modal modeling of image features; Section 5 presents the results and discusses the experiments; and Section 6 provides the conclusion of the study and directions for future study.
Related works
The joint study of aesthetic factors in art and design is of great importance for studies of multimedia computing, intelligent design, and aesthetics culture. In addition, aesthetic values in art and design are unique features of in cultural and social development, which develops various artistic forms over time. This can be an extremely important evidence for the study of features of visual perception. Thus, the aesthetic principles and patterns of multimedia works should be explored using computer models [12–15]. Here we review the related works of multimedia aesthetic computing, and the existing multimedia aesthetic databases are concluded.
2.1 Multimedia aesthetic modeling works
A review of aesthetic-aware modeling research is given in Table 1. Works that attempt to bridge computing and the perception of aesthetics have three main tasks. First, aesthetic images are to be evaluated using qualitative measures [16–18], including user interface design [19–21], photos, paintings, and filmed scenes. Second, multimedia retrieval is to be developed, based on aesthetic recognition. Third, aesthetic multimedia is to be generated using aesthetic-aware modeling [22, 23]. Work in this area can produce various applications, such as in the intelligent assistance of design evaluation and of design of computer games [24].
There are some specific guiding principles and aesthetic standards found in design theories, involving the treatment of colors and hues, saturation values, and layout formats. It is easy to select features of images that represent such characters. During the early period of aesthetic modeling study, evolutionary methods were commonly used. Ross et al. created an automatic synthesis of aesthetically pleasing images via genetic programming for the generation of textures [25]. Wong et al. presented a saliency-enhanced method for distinguishing professional photographs from amateur ones. A set of salient features and global features were utilized in this study [26]. Su et al. proposed a preference-aware image aesthetic model, which covered both implicit and explicit aesthetic features to meet users’ preferences, and the model achieved an accuracy rate of 92.06%. They also found that contrast features were most effective among the tested information [27]. Lovato et al. developed a personal aesthetic model as a novel behavioral biometrical trait, assessing low- and high-level features of Flickr images, using a LASSO (Least absolute shrinkage and selection operator) regressor [28]. Zhang et al. evaluated aesthetic quality in photographs, encoding local and global structural features [29]. Tarvainen et al. built a film dataset to develop assessments of style, aesthetics, and affect in films. Neural-network based Extreme Learning Machine was experimentally found to be slightly better than linear regression [30]. Temel et al. performed a comparative study of computational aesthetics and found that the feature of generic or hand-crafted was insufficient for aesthetics modeling, and the relationship between features and aesthetics was then explored through deep learning in the further study [31].
Various methods have proven to be useful in aesthetic learning, including SVM [32], GMM, Bayes, DCNNs [33–39], etc. Different images can be distinguished by style, and they often follow different aesthetic rules. Therefore, the best learning approach should differ according to the given dataset [40, 41]. The classic aesthetic database AVA, which contains over 250,000 images and a large amount of meta-data with aesthetic scores, has been used in many studies to produce modeling comparison and optimization [42]. Lu et al. investigated the effectiveness of deep neural networks on a 1.5-million image dataset for aesthetic assessment, finding an accuracy of 75.41% [33]. Using the AVA database, Jin et al. adopted DCNNs for prediction of image aesthetics, which achieved a high performance of means square error 0.3373 [34]. Meng et al. constructed a multi-layer aggregation network with various baseline networks from MobileNet, VGG-16, and Inception-v3. The experimental results indicated that the developed model exhibited superior performance to those found in existing studies [43]. Sidhu et al. explored aesthetic prediction on both beauty ratings and liking ratings for 240 abstract and 240 representational paintings in the study based on regression models. They used 4 subjective and 11objective predictors in the measurement and found that the results varied widely in modeling between abstract and representational paintings [44].
2.2 Multimedia aesthetic databases
Multimedia databases for aesthetic evaluation were constructed in various studies. The largest aesthetic database is AVA database (Aesthetic Visual Analysis) [42], which is a widely used aesthetic database containing 250,000 images in 60 classes. The other aesthetic databases include FLICKR-AES [17], HB [18], CUHK [29], PNE [29], MSR-ICD [37] and CUHKPQ [39].
In conclusion, machine learning algorithms have been widely used in aesthetic modeling in recent years for large image datasets, although an unexplored problem remains, namely, that of specific aesthetic learning regarding the styles of artworks and understanding user preferences. The selection of features for different images and differences in approaches to them should receive further exploration in a way that takes into account the goal of assessment.
Methodologies
In the existing aesthetic computing research, there are three main types of studies, the aesthetic ranking analysis [13], classification of aesthetic level (low/high or positive/negative) [15, 18, 24, 26–29] and the aesthetic score prediction [17, 31, 43]. In the majority of the related works, researchers conducted classification method on image aesthetic computing study. Thus, here we transformed the aesthetic computing problem into a three-class-classification problem of image aesthetic level. The contribution of this work is building a relatively objective aesthetic database with design awards submissions, which is a suitable carrier for aesthetic computing study. And an aesthetic evaluation model for product design is explored based on these datasets.
The work of aesthetic-aware modeling taken from design competition datasets can be considered a classification task. In this study, we built classification models using 10-fold cross-validation. Five algorithms (LibSVM, Liblinear, RBFNetwork, RandomSubSpace, VGG-19 and ResNet-50) were implemented for the three classification divisions, namely, “eliminated”, “middle class”, and “nominees”. The image features extraction approach and the applied methods are introduced below.
3.1 Image pre-processing
In the design work submission session, designers were requested to submit the proposal layout in an image format of 300 dpi. In the experiment, the input design layout images were resized to 640 × 480 without any cropping.
There are two original design layouts collections applied in this experiment, one is of 2216 pieces of design works collected from “Electronic Home Applicants Design Award” in 2012, while another collection has 639 submissions from “Electronic Tools Design Award” in 2015. We obtained permission to use the design layout images of the design awards from the awards organizers for this design aesthetic computing research. The images have been collected and classified by the design awards organizer during the design competition.
3.2 Feature extraction
Based on these image collections, two kinds of design aesthetics database were built in this study, including the databases of hand-crafted image features and the databases of deep-learning features extracted by deep learning approaches.
(1)Hand-crafted features database
In the hand-crafted databases, the image features of LBP [42] and color features (HSV and HIST) [19, 23, 27, 28, 44]were utilized for the modeling, in view of the previous studies on image aesthetics computing (see Session 2). On the other hand, the selection of features also took into account the industrial design theory of “Comprehensive Formation”, which is the general name of plane formation, color formation and three dimensional composition, the theoretical basis of industrial design. Inspired by the basic design theories, in this aesthetic evaluation study, we extracted the related image feature sets of LBP, HSV and HIST, that can represent contour features and color features of design work. Specifically, 64 Dimensions of LBP, 256 dimensions of HSV and 256 dimensions of HIST were extracted by OpenCV to form the databases.
(2)ResNet-50 features database
The deep-learning features databases were formed by image features extracted by ResNet-50. Firstly, we use ResNet-50 to extract a total of 25,088 dimensions of image features. A total of 2,048 features were obtained as the output vector. Then, it was reduced to be a 512-dimentional features vector by a fully connected network as the neural network input for the next step.
(3)VGG-19 features database
Firstly, a total of 25088 dimensions of image features were extracted by VGG-19. Then, 1000 features were obtained as the output vector. Then, it was reduced to be a 512-dimentional features vector by a fully connected network as the neural network input for the next step.
3.3 Algorithms
LibSVM.
LibSVM is an integrated tool used in multi-class classification and regression. A support vector machine (SVM) is a generalized linear classifier that uses binary classification, with a supervised learning method. Its decision boundary is the maximum-margin hyperplane for learning data samples. SVM uses hinge loss to compute empirical risk. It is regularized in the solution process to optimize its structure. A series of improved and extended algorithms have been developed for this, including multi-class classification, least-square SVM, support vector regression, support vector clustering, and semi-supervised SVM. This approach can be combined with other algorithms to optimize attributes to form various ensemble learning methods. SVM is widely used in pattern recognition and multimedia classification. It has been shown in many studies that this approach is highly efficient for the classification of small datasets.
Liblinear.
Liblinear is a linear classifier that is suitable for use in the classification of large datasets and multidimensional attributes [45]. It contains multiple classifiers for linear regression and SVM, including L2-regularized classifiers, L2-loss linear SVM, L1-loss linear SVM, and logistic regression (LR), L1-regularized classifiers, L2-loss linear SVM and logistic regression (LR), L2-regularized support vector regression, and L2-loss linear SVR and L1-loss linear SVR. For a sample of the form (xi, yi), i = 1,…,k, xi∈Dn, yi∈{−1,1}, this algorithm can solve problems of unconstrained optimization, as follows:
(1)
in which δ>0 is set as the penalty parameter, while β(α;xi,yi) represents the loss function. Previous work has shown the advantages of this approach in the classification of large datasets [44].
RBFNetwork.
A radial basis function network (RBFNetwork) is a neural network that uses a radial basis function for activation. It is usually constructed with three layers: an input layer, a hidden layer, and an output layer. The hidden layer can be represented by θi:Vn→V. The output of RBFNetwork is a scalar function of input vectors. This method is widely used to solve problems of function approximation, predication, classification, and regression. With this approach, complex, dimensional input data can be reduced and mapped into a new space. The kernel parameter is optimized with an estimation method. The resulting network output is formed of a combination of the radial basis functions of input data and neuron parameters [46].
RandomSubSpace.
RandomSubSpace is an ensemble learning method that can combine algorithms for classifiers. It constructs a classifier based on a decision tree and adapts to the highest performance on training dataset, improving its generalization accuracy as it grows in complexity. This algorithm incorporates multiple trees, which are constructed systematically by randomly selecting subsets from feature vectors. The trees are constructed in randomly chosen subspaces. For this situation, it is feasible that the number of features would be much larger than the number of samples, such as datasets of gene sequences and fMRI [47].
VGG-19.
VGG-19 can achieve great accuracy in the large-scale image recognition [48] and it is also applicable in a number of image aesthetic computing studies [18][39][43]. It is using architecture with small convolution filters. However, a significant result improvement can be achieved by constructing 16–19 weight layers. The VGG-19 network training process is presented as follows:
- Firstly, a fix-size 224 x 224 image is set to be the input during training.
- Secondly, a total of 1,000 dimensional features were extracted by VGG-19.
- Thirdly, the dimensions of image features vector were reduced to be a 512-dimentional features vector by the fully connected network for the classification.
Specifically, the soft-max layer is set as the final layer. All hidden layers are using ReLU as the activation function. The VGG-19 network architecture is presented in Fig 1, and the detailed feature extraction process is presented in Table 2.
Each plane is a feature map.
ResNet-50.
ResNet-50 is applied to extract multimodal features of design work images in view of its feasibility in existing studies [18, 39]. The ResNet-50 training process is shown as follows:
- Firstly, the input image is reset to 224 x 224 before the training process.
- Secondly, we extracted multimodal features by ResNet-50 to form an output vector of 2,048 features. The output vector is further fed into a fully connected network with four fully connected layers.
- Thirdly, the dimensions of image features vector were reduced by the fully connected network for aesthetics classification.
Specifically, various optimized methods were applied in features processing, including Standard Feature Normalization shift, rotation, zoom, nearest-fill and horizontal flip. Then we proceeded to use 2048-dimensional features and the normalized correlation value to train the fully connected network for 300 epochs. In the four-layer fully connected network (FCNN), the activation functions of the other layers are ReLU to prevent the gradient from disappearing or exploding.
Softmax (mapping the result to 0–1) is applied as the activation function in the output layer. The detailed feature extraction process is presented in Table 3.
As shown in Fig 2, a four-layer fully connected network was introduced to predict similarity after images features training. All activation functions in the first three layers of FCNN are ReLU to prevent gradients from disappearing or gradients exploding, and the activation functions of the output layer is softmax. During the training process, binary cross entropy was set as a loss function and Adam was set as the optimizer.
Each plane is a feature map.
Experiments
The datasets of applicants for the Electronic Home Applicants Design Awards and the Electronic Tools Design Awards were both divided into three categories: eliminated, middle class, and nominees. The experiments were intended to recognize the design works with low aesthetic scores (eliminated) and with high aesthetic scores (nominees). The competition results appear to indicate that, to some extent, the evaluation of aesthetic features is fundamental to judging design quality. The works of design that receive awards are judged to be outstanding in every aspect. The aesthetic level of a design can thus be a clue in assessing the design quality, transforming this latter task into an aesthetic-level classification question. An general aesthetic-aware modeling experimental procedure is presented in Fig 3.
4.1 Reviewers for design awards
Nine experts were reviewers for the 2012 Electronic Home Applicants Design Awards, and five experts were reviewers for the 2015 Electronic Tools Design Awards. These experts are renowned educators and practitioners in industrial design from top universities and design companies in China, and they all have rich experience in evaluating design.
4.2 Datasets
Two original datasets were used in this experiment. The first dataset contains images of 2216 design works collected from the 2012 Electronic Home Applicants Design Awards, and the other one contains images of 639 pieces from the 2015 Electronic Tools Design Awards. In the experiment, the Electronic Home Applicants Design Awards database was randomly divided for model exploration, in which 1777 pairs were set for training and 439 pairs were set for testing. While in the Electronic Tools Design Awards database, 519 pairs were randomly selected for training and 129 pairs were selected for testing.
For the aesthetic–aware classification modeling of Liblinear, LibSVM, RBFNetwork, and RSS-Randomforest, image features LBP, HSV, and HIST were used to form the datasets. LBP, HSV, and HIST were extracted in 64, 256, and 256 dimensions, respectively, were extracted using OpenCV. Then, VGG-19 and ResNet-50 were used as a comparison for modeling. We first built the aesthetic model using the Electronic Home Applicants Design Awards dataset, and then we tested it on the dataset from the Electronic Tools Design Award. The detailed feature extraction method is introduced in section 3.2.
4.3 Experts’ review procedure for design awards
The steps that the experts followed in their reviews procedure were introduced as follows.
Collection of design works.
A total of 2247 design works were collected from the Electronic Home Applicants Design Awards in 2012, of which 2216 were usable for image processing. A total of 671 design works were submitted to the Electronic Tool Design Awards in 2015, of which 639 were usable for image processing. The design works were created by college students pursuing an industrial design major and designers working in related industries around the world.
Design review.
After the submission deadline, a number of experts were invited to rate the design works according to several criteria. The design criteria of the two design awards are listed below: innovation in appearance and function (30%), market value and feasibility (20%), environmental aspect (20%), harmonious color design (10%), layout presentation quality (10%), and comprehensive evaluation (10%) (Table 4). It should be noted that design aesthetic is captured in relation to three items in the review, and it is a crucial element and part of the basic standard for design competitions. Submission scores were obtained for the first round of the experts’ review, with a numerical value on a scale from 0 to 100.
Primary screening of submissions.
Works that scored 0–60 were categorized as having poor design quality and were eliminated in this primary round of selection, such that 821 pieces of Electronic Home Applicants Design Awards and 182 pieces of Electronic Tools Design Awards were eliminated in this first round of selection.
Nomination.
The expert reviewers selected the nominees after the primary selection. These nominees were selected from the submissions that remained after primary screening. A total of 125 pieces were nominated from the pool for the Electronic Home Applicants Design Awards, and 77 pieces were selected as nominees from the pool for the Electronic Tools Design Awards.
Final awarding.
The awards list was generated by ranking the results for the list of nominees. In this session, the reviewers were invited to thoroughly discuss them and debate the final points. Fig 4 presents layouts of the top 20 design works from the Electronic Tools Design Awards and 20 layouts of the eliminated design works in this award as a comparison.
Fig (a) presents layouts of the top 20 design works from the Electronic Tools Design Awards and Fig (b) presents 20 layouts of the eliminated design works in this award as a comparison. Only part of the layout is presented here due to copyright protections.
4.4 Aesthetic-aware modeling
Visual aesthetics will influence user preference [7] and acceptance [11] of products. It is relatively difficult to evaluate the creativity level of each work, since it needs a large database of various design concepts, which integrate design factors of text, shape and ergonomics etc. Therefore, we took visual aesthetics character as a research point to carry out evaluation modeling.
In the current aesthetic related research, aesthetic score ranking computing and classification of aesthetic level (positive/negative) are two major modes of aesthetic computing methods. This study proposed an aesthetic classification problem in two design awards datasets. The significance of this paper lies in the utilization of an objective design competition database with ground-truth annotations, which is a suitable carrier of aesthetic computing research.
In this experiment, several algorithms were applied to the datasets to build the model, including LibSVM, Liblinear, RBFNetwork, RandomSubspace-RandomForest, VGG-19 and ResNet-50. Among all these approaches, the learning method of ResNet-50 attained the best accuracy, 74.32% for the dataset of the Electronic Home Applicants Design Award and 73.25% for the dataset of Electronic Tools Design Award. That is, design works were classified more efficiently and with superior accuracy by this approach.
Two experimental sessions were conducted in this study. Firstly, design works that had been scored by experts using design evaluation criteria were evaluated. Secondly, design proposal layouts were evaluated via machine learning. The dataset for the Electronic Home Applicants Design Awards was utilized for modeling exploration, and the method was proved to be effective by the verification using the dataset from the Electronic Tools Design Awards.
Results and discussion
In this study, we combined multiple features of images for aesthetic evaluation for two design award datasets. A comparison was conducted using LibSVM, Liblinear, RBFNetwork, RandomSubspace-RandomForest, VGG-19 and ResNet-50 to obtain the best model. Using aesthetic-aware model results, ResNet-50 were found to achieve the best classification accuracy.
Models comparison and optimization
The specific results of the comparison of the algorithms applied to the Electronic Home Applicants Design Awards dataset are presented in Table 5.
The performance of modeling could be limited by the scale of dataset, which would ultimately constrain the scalability of the method. Consequently, it was tested in the dataset of the Electronic Tools Design Awards to indicate its effectiveness in aesthetic evaluation. ResNet-50 also attained the best accuracy of 73.25% for the Electronic Tools Design Awards dataset. The verified modeling results of the Electronic Tools Design Awards are shown in Table 6.
The results of model comparisons in Table 5 and Table 6 show that ResNet-50 outperformed other algorithms in the average accuracy of classification.
Best features exploration for hand-crafted features.
Aesthetic evaluation by hand-crafted features is also an important method in this area. An investigation of best features can provide guidance to the further study. Consequently, the CfsSubset Evaluation via BestFirst method was applied in the feature selection to find the most relevant hand-crafted features with lookupCacheSize of 1 and searchTermination of 5. As a result, 16 relevant image features were selected for “Electronic Home Applicants Design Award” dataset and 26 relevant image features were selected for “Electronic Tools Design Award” dataset. The most relevant features for each dataset are listed in detail in Table 7. In the analysis result, 10 HSV features are most relative to aesthetic character of image for “Electronic Home Applicants Design Award” while 17 HIST features are most relevant for Electronic Tools Design Award dataset, see Table 7. The results indicate that the best features for aesthetic recognition can be differ in different datasets. It might be due to the difference of image content. It can be concluded that color related features should be concerned in design evaluation. Accordingly, HSV and HIST can be endowed with more concern in the further study.
Classification performance analysis.
The classification results indicate that a relatively higher classification effect was found for the class of nominees than the classes of the eliminated, see Table 8. The recognition of awarded design works was found to be relatively effective in the model exploration. The experimental result proved that the aesthetic level can be a cue for general design quality assessment. This may be because good presentation design is considered as a basic requirement of a design concept submission. Consequently, those design works that have a poor appearance are eliminated in the first round. Likewise, design works of higher quality share common design characteristics, such as detailed product images and descriptive text, along with visual and attractive color schemes. Nevertheless, when it comes to the final round of the selection for the design award, it is difficult to isolate the best works using layout and appearance alone, so the final ranking of the nominees might be hard to predict by aesthetic modeling. In the further study of intelligent design evaluation, semantic analysis should be incorporated to comprehend the highlights of design thinking.
We proceeded to explore the two-class image classification to distinguish the eliminated and un-eliminated ones, and classified the nominees from all the submissions by deep learning methods. In the ResNet-50 training process, the optimal classification accuracy achieves stable after 300 epochs of training, see Fig 5 and Fig 6. For the Electronic Home Applicants Design Award database, the Nominees classification accuracy achieves 93.70% after 300 epochs of training using ResNet-50, and the Eliminated classification accuracy achieves 66.67%. For the Electronic Tools Design Award database, the Nominees classification accuracy achieves 84.19% after 300 epochs of training using ResNet-50, and the Eliminated classification accuracy achieves 75.95%.As a result, the average classification accuracy of ResNet-50 outperforms the performance of VGG-19, see Table 8.
(Fig (a):Nominees classification accuracy; Fig(b): Eliminated classification accuracy).
(Fig (a): Nominees classification accuracy; Fig (b): Eliminated classification accuracy).
It is interesting to consider that if a design achieves low score in presentation alone, it could be considered not to qualify for consideration for an award. The principle that appearances matter holds true as well for artificial intelligence aesthetics perception. All the design works that received awards also conformed to high standards in the design of their presentation posters. Our experimental results confirmed our hypothesis that aesthetics can be assessed using a machine learning approach, and features that fuse the modeling of design layout images may be a feasible avenue for development of intelligent aesthetic perception.
Conclusion and directions for future work
Although the style of human cognition used in art and design is abstract and subjective, the scientific exploration of feature dimensions and data fusion can allow computers to obtain a sense of appreciation for design. In design work, layout follows certain formatting and color-combination rules. Comparing to paintings and abstract works of art, containing much personal understanding and preference, the aesthetic patterns of design layout can be studied using machine learning methods more readily.
In this study, we created an original database for the aesthetic evaluation of art and design, which may become a useful data resource for multimedia aesthetic computing. We also created an effective method for the aesthetic evaluation of design layouts based on multi-modal image features. In this work, 2,981 original design works taken from entrants of two design competitions were collected to build two sets of data for the design of aesthetic images. One dataset was used for the construction of models, and the other was used for aesthetic-aware model testing. In our experiment, the image features LBP, HIST, and HSV were extracted to form the dataset for traditional machine learning approaches. Subsequently, VGG-19 and ResNet-50 were used for comparison with the results of traditional method. The best aesthetic evaluation result reached a classification accuracy of around 80% for both datasets based on ResNet-50, and the classification was more accurate for the nominated designs than the eliminated ones. The experimental findings suggest that aesthetic-aware modeling based on image feature analysis is a feasible approach for automatic design evaluation. Software can acquire the ability of aesthetic appreciation by following this promising methodology.
Many possible avenues exist for future work. A larger dataset of design works should be built to improve modeling accuracy. Thus, it would be possible to use fusion deep learning methods for feature extraction and classification for model optimization. To verify this method, a system of design evaluation can be developed, based on the model explored for design competition review or self-assessment in designing. Moreover, this method can be used in possible applications in related areas of packaging and advertisement design involving image aesthetics assessment. Subsequent this study can involve the use of the method to address questions of aesthetic perception in various scenarios.
Supporting information
S1 File. Design aesthetic database description.
https://doi.org/10.1371/journal.pone.0227754.s001
(DOCX)
Acknowledgments
Much appreciated for the support of Linhai Xie and Bohan Zhu in Hangzhou Dianzi University for their support in the experiment of this work.
References
- 1.
Brunner R, Emery S, Hall R. Do You Matter? How Great Design Will Make People Love Your Company. FT Press, U.S., 2008.
- 2.
Norman D A. Emotional Design: Why We Love (Or Hate) Everyday Things. Basic Books, U.S., 2004.
- 3. Law D, Cheung M, Yip J, Yick K, Wong C. Scoliosis brace design: influence of visual aesthetics on user acceptance and compliance. Ergonomics. 2017; 876–886. pmid:27547883
- 4. Hou G, Lu G. The influence of design proposal viewing strategy: design aesthetics and professional background. Int J Technol Des Educ. 2019; 29:543–564.
- 5. Guo F, Li M, Hu M, Li F, Lin B. Distinguishing and quantifying the visual aesthetics of a product: An integrated approach of eye-tracking and EEG. International Journal of Industrial Ergonomics. 2019; 47–56.
- 6. Chien C, Kerh R, Lin K, Yu A P. Data-driven innovation to capture user-experience product design: An empirical study for notebook visual aesthetics design. Computers & Industrial Engineering. 2016; 162–173.
- 7. Bloch P H, Brunel F F, Arnold T J. Individual differences in the centrality of visual product aesthetics: concept and measurement. J. Consum. Res. 2003; 551–565.
- 8. Hsiao K L, Chen C C. What drives smartwatch purchase intention? Perspectives from hardware, software, design, and value. Telematics Inf. 2018; 103–113.
- 9. Toufani S, Stanton J P, Chikweche T. The importance of aesthetics on customers' intentions to purchase smartphones. Market. Intell. Plann. 2017; 316–338.
- 10. Simmonds G, Spence C. Thinking inside the box: how seeing products on, or through, the packaging influences consumer perceptions and purchase behaviour. Food Qual. Prefer. 2017; 340–351.
- 11. Nanda P, Bos J, Kramer K, Hay C, Ignacz J. Effect of smartphone aesthetic design on users' emotional reaction: an empirical study. The TQM Journal. 2008; 348–355.
- 12. Schindler I, Hosoya G, Menninghaus W, Ursula B, Valentin W, Michael E, et al. Measuring aesthetic emotions: A review of the literature and a new assessment tool. Plos One. 2017; 12(6): e0178899. pmid:28582467
- 13.
Tian X, Long Y, Lv H. Relative Aesthetic Quality Ranking. Proceedings of IEEE International Conference on Systems, Man, and Cybernetics. 2018; 2509–2516.
- 14.
Liao W, Chen P. Analysis of Visual Elements in Logo Design. Proceedings of International Symposium on Smart Graphics. 2014; 73–85.
- 15. Sheng K, Dong W, Ma C, Mei X, Huang F, Hu B. Attention-based Multi-Patch Aggregation for Image Aesthetic Assessment. Proceedings of ACM Mulimedia. 2018; 879–886.
- 16. Qian X, Li C, Lan K, Hou X, Li Z, Han J. POI Summarization by Aesthetics Evaluation from Crowd Source Social Media. IEEE TRANSACTIONS ON IMAGE PROCESSING. 2018; 27(3): 1178–1189. pmid:29220319
- 17. Ren J, Shen X, Lin Z, Mech R, Foran D J. Personalized Image Aesthetics. Proceedings of IEEE International Conference on Computer Vision. 2017; 638–647.
- 18. Kucer M, Loui A C, Messinger D W. Leveraging Expert Feature Knowledge for Predicting Image Aesthetics. IEEE TRANSACTIONS ON IMAGE PROCESSING. 2018; 27(10): 5100–5113.
- 19. Chen R, Hua L, Xie Y, Lin T, Tang N. A Fuzzy-Rule-Based Approach for Webpage Aesthetics Modeling. Proceedings of Nicograph International. 2016; 142–143.
- 20. Maity R, Bhattacharya S. Is My Interface Beautiful?—A Computational Model-Based Approach. IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS. 2019; 6(1): 149–162.
- 21. Persada A G, Pranata M W A, Ana A A. Aesthetics of Interaction Design on the Mobile-Based University Website. Proceedings of International Conference on Electrical Engineering and Computer Science. 2017; 137–143.
- 22.
Wu T, Zhang L, Yang J. Automatic Generation of Aesthetic Patterns with Cloud Model. Proceedings of 12th International Conference on Natural Computation. Fuzzy Systems and Knowledge Discovery. 2016; 1077–1084.
- 23. Zhang C, Lei K, Jia J. AI Painting: An Aesthetic Painting Generation System. Proceedings of ACM Multimedia. 2018; 1231–1234.
- 24. Erdem A N, Halici U. Applying Computational Aesthetics to a Video Game Application Using Machine Learning. IEEE Computer Graphics and Applications. 2016; 36(4): 23–33. pmid:27244720
- 25.
Ross B J, Ralph W, Zong H. Evolutionary Image Synthesis Using a Model of Aesthetics. Proceedings of IEEE Congress on Evolutionary Computation. 2006; 1087–1093.
- 26. Wong L, Low K. SALIENCY-ENHANCED IMAGE AESTHETICS CLASS PREDICTION. Proceedings of the International Conference on Image Processing. 2009; 997–1001.
- 27. Su H, Chen T, Kao C, Hsu W H, Chien S. Preference-Aware View Recommendation System for Scenic Photos Based on Bag-of-Aesthetics-Preserving Features. IEEE TRANSACTIONS ON MULTIMEDIA. 2012; 14(3): 833–844.
- 28. Lovato P, Bicego M, Segalin C, Perina A, Sebe N, Cristani M. Faved! Biometrics: Tell Me Which Image You Like and I’ll Tell You Who You Are. IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY. 2014; 9(3): 364–375.
- 29. Zhang L, Gao Y, Zimmermann R, Tian Q, Li X. Fusion of Multichannel Local and Global Structural Cues for Photo Aesthetics Evaluation. IEEE TRANSACTIONS ON IMAGE PROCESSING. 2014; 23(3): 1419–1430. pmid:24723537
- 30. Tarvainen J, Sjöberg M, Westman S, Laaksonen J, Oittinen P. Content-Based Prediction of Movie Style, Aesthetics, and Affect: Data Set and Baseline Experiments. IEEE TRANSACTIONS ON MULTIMEDIA. 2014; 16(8): 2085–2098.
- 31. Temel D, AlRegib G. A COMPARATIVE STUDY OF COMPUTATIONAL AESTHETICS. Proceedings of the International Conference on Image Processing. 2014; 590–595.
- 32. Wu O, Zuo H, Hu W, Li B. Multimodal Web Aesthetics Assessment Based on Structural SVM and Multitask Fusion Learning. IEEE TRANSACTIONS ON MULTIMEDIA. 2016; 18(6): 1062–1077.
- 33. Lu X, Lin Z, Jin H, Yang J, Wang J Z. Rating Image Aesthetics Using Deep Learning. IEEE TRANSACTIONS ON MULTIMEDIA. 2015; 17(11): 2021–2035.
- 34.
Jin B, Segovia M V O, Susstrunk S. IMAGE AESTHETIC PREDICTORS BASED ON WEIGHTED CNNS. Proceedings of the International Conference on Image Processing. 2016; 2291–2296.
- 35. Lee H, Hong K, Kang H, Lee S. Photo Aesthetics Analysis via DCNN Feature Encoding. IEEE TRANSACTIONS ON MULTIMEDIA. 2017; 19(8): 1921–1933.
- 36. Liu Z, Wang Z, Yao Y, Zhang L, Shao L. Deep Active Learning with Contaminated Tags for Image Aesthetics Assessment. IEEE TRANSACTIONS ON IMAGE PROCESSING. 2019; pmid:29993633
- 37.
Wang W, Shen J. Deep Cropping via Attention Box Prediction and Aesthetics Assessment. Proceedings of IEEE International Conference on Computer Vision. 2017; 2205–2214.
- 38.
Tong S, Liang X, Iwaki S, Tosa N. Learning the Cultural Consistent Facial Aesthetics by Convolutional Neural Network. Proceedings of International Conference on Culture and Computing. 2017; 97–104.
- 39.
Fu X, Yan J, Fan C. IMAGE AESTHETICS ASSESSMENT USING COMPOSITE FEATURES FROM OFF-THE-SHELF DEEP MODELS. Proceedings of the International Conference on Image Processing. 2018; 3528–3533.
- 40. Iqbal A, Heijden H V D, Guid M, Makhmali A. Evaluating the Aesthetics of Endgame Studies: A Computational Model of Human Aesthetic Perception. IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES. 2012; 4(3): 178–192.
- 41. Browne C. Elegance in Game Design. IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES. 2012; 4(3): 229–241.
- 42.
Murray N, Marchesotti L, Perronnin F. AVA: A Large-Scale Database for Aesthetic Visual Analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012; 2408–2416.
- 43.
Meng X, Gao F, Shi S, Zhu S, Zhu J. MLANs: Image Aesthetic Assessment via Multi-Layer Aggregation Networks. Proceedings of International Conference on Image Processing Theory, Tools and Applications. 2018; https://doi.org/10.1109/IPTA.2018.8608132
- 44. Sidhu D M, Mcdougall K H, Jalava S T, Glen E B. Prediction of beauty and liking ratings for abstract and representational paintings using subjective and objective measures. PLOS ONE. 2018; 13(7): e0200431. pmid:29979779
- 45. Fan R E, Chang K W, Hsieh C J, Wang X R, Lin C J. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research. 2008; 9: 1871–1874.
- 46. Chen W, Yan X, Zhao Z, Hong H, Bui D T, Pradhan B. Spatial prediction of landslide susceptibility using data mining-based kernel logistic regression, naive Bayes and RBFNetwork models for the Long County area (China). Bulletin of Engineering Geology and the Environment. 2018; 1–20.
- 47. Ho T K. The Random Subspace Method for Constructing Decision Forests. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1998; 20(8): 832–844.
- 48.
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. Proceedings of International Conference on Learning Representations. San Diego, USA. 2015.