De-Stijl: Facilitating Graphics Design with Interactive 2D Color Palette Recommendation (original) (raw)

Figures (16)

Figure 1: The workflow of how a marketer uses De-Stijl to make some graphics to promote the Valentine’s Day sale. (A) The user starts by creating or importing design elements into De-Stijl via the toolbox (shown on the left), adjusts their layout, then specifies a set of theme colors according to the theme (e.g., a romantic feeling), and ends by clicking the “RECOMMEND” button. (B) De-Stijl returns an automatically recolored graphic as well as a list of recommended color palettes for each element allowing for further refinement. In addition, De-Stijl offers a 2D palette-based overview of the entire design. (C) De-Stijl allows the user to not only easily create harmonic designs but also quickly obtain multiple design alternatives with different theme colors.

Figure 3: 2D palette reflecting the design overview with dif- ferent color quantity options. Users can control the color quantity via the slider to accommodate the designs and im- ages with different color complexity levels. The slider in- fluences both the design overview and also the 2D palette recommendation for images. Figure 2: User Interface of De-Stijl, consisting of (A) a design toolbox, (B) a canvas, and (C) a control panel. The control panel further includes (1) a theme color setting panel, (2) a theme effect level slider, (3) a color quantity control slider for the 2D palette, (4) a design overview palette, (5) a recommendation button, (6) a list of recommendations, and (7) an object layer panel.

Figure 4: Example of the four semantic design layers in an image-centric graphic.

Figure 6: Traditional 1D color palette (a) and design alterna- tives of the 2D palette: (b) showing fine-grained details about the image, (c) representing the image in a grid with superpix- els, and (d) grouping similar grid cells based on colors (our final design).

Figure 5: System architecture and workflow of De-Stijl. From an undesirable input graphic with no or incompatible colors, the system first decomposes the input into four semantic design layers (L & D) (i.e., image, decoration, text, and background) and extracts 2D color palettes for image objects through the 2D palette extractor (A). Next, the color recommender (B) considers the background and user specifications (e.g., theme colors) as design conditions (D) and suggests colors (palettes) for each of the elements (E). The system also automatically recolorizes all the elements based on the top-ranked recommendation, including images that are processed by a special image recolorizer (C). Together, it generates an aesthetically pleasing design with harmonic colors. A user can also interactively explore several design recommendations (E). They can also lock any elements’ colors (palettes) to further generate new design recommendations with additional design constraints.

Figure 7: Demonstration of input, design conditions, and output of the color recommenders for different design layers.

Figure 9: Example results of different theme color effect levels. E:neme = 0 indicates less influence of user-specified theme colors exerting on the recommended color palettes and E;peme = 1 indicates more influence. Figure 8: Model architecture of the color recommender in De-Stijl, consisting of a generator G connected by skip connections along with a Condition Feature Modulation (CFM) module and a discriminator D. Taking the image layer if as an example (input and output for other layers are shown in Figure 7), given the input x, the generator will output the recommended color palette 7 according to the design conditions c.

Table 1: User study tasks summary.

Figure 10: Participant-generated design examples during the user study. In each group of results, users start from the template (shown on the left of each group) with incompatible colors and use either the De-Stijl (results shown in the upper row) or the Baseline system (results shown in the lower row) to recolor the design. Task 1 requires the design to be compatible with the doughnuts picture. Task 2 requires the design to be harmonic with the yellow and red colors. Task 3 requires to reflect the season of fall. More examples are in Appendix C.

Figure 11: Expert’s ratings on the quality of participant-generated graphics (the higher the better).

Figure 12: Results of Creative Support Index (CSI) for De-Stijl and Baseline (the higher the better) for the factors of enjoy- ment, exploration, expressiveness, immersion, and results worth effort. The error bar indicates the 95% confidence in- terval.

Table 2: Quantitative comparison of different versions of our color recommender model on two metrics: Learned Percep- tual Image Patch Similarity (LPIPS) and Frechet Inception Distance (FID). Both metrics are the lower the better.

Figure 13: Model architecture of the color recommender in De-Stijl, consisting of a generator G connected by skip connections along with a Condition Feature Modulation (CFM) module and a discriminator D. Taking the image layer ii as an example, given the input x, the generator will output the recommended color palette j according to the design conditions c.

EEE EO Figure 14: Participant-generated design examples during the user study. In each group of results, users start from the template (shown in the left of each group) with incompatible colors and use either the De-Stijl (results shown in the upper row) or the Baseline system (results shown in the lower row) to recolor the design. Task 1 requires the design to be compatible with the salad picture. Task 2 requires the design to be harmonic with blue colors. Task 3 requires to reflect the feeling of Valentine’s Day.

Key takeaways

AI

1. De-Stijl offers an AI-driven 2D color palette tool that simplifies graphic design for novice users.
2. The system supports image-centric graphic designs, enhancing color selection through spatial and thematic recommendations.
3. De-Stijl incorporates a dataset of 706 annotated graphic designs for training its color recommendation models.
4. User studies show De-Stijl significantly improves task completion rates and design quality compared to existing tools.
5. The tool facilitates rapid design iterations and customization, promoting user engagement and creativity in graphic design.

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

References (60)

1. Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. on pattern analysis and machine intelligence 34, 11 (2012), 2274-2282.
2. George A Agoston. 2013. Color theory and its application in art and design. Vol. 19. Springer, New York, USA.
3. Hyojin Bahng, Seungjoo Yoo, Wonwoong Cho, David Keetae Park, Ziming Wu, Xiaojuan Ma, and Jaegul Choo. 2018. Coloring with words: Guiding image colorization through text-based palette generation. In ECCV. Springer, Munich, Germany, 431-447.
4. Malcolm Barnard. 2013. Graphic design as communication. Routledge, Abingdon, Oxfordshire.
5. Michael Mose Biskjaer, Peter Dalsgaard, and Kim Halskov. 2014. A Constraint- Based Understanding of Design Spaces. In Proc. of the 2014 Conf. on Designing Interactive Systems (Vancouver, BC, Canada) (DIS'14). ACM, New York, USA, 453-462. https://doi.org/10.1145/2598510.2598533
6. Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015. Palette-based photo recoloring. ACM Trans. Graph. 34, 4 (2015), 139-1.
7. Erin Cherry and Celine Latulipe. 2014. Quantifying the creativity support of digital tools through the creativity support index. ACM Trans. on Computer- Human Interaction (TOCHI) 21, 4 (2014), 1-25.
8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A Large-scale Hierarchical Image Database. In CVPR (Miami, FL, USA). IEEE, New York, USA, 248-255.
9. Aditya Deshpande, Jiajun Lu, Mao-Chuang Yeh, Min Jin Chong, and David Forsyth. 2017. Learning Diverse Image Colorization. In CVPR (Honolulu, HI, USA). IEEE, New York, USA, 6837-6845.
10. Leatrice Eiseman. 2000. Pantone's guide to communicating with color. Adams Media, Avon, MA, USA.
11. Yuki Endo, Satoshi Iizuka, Yoshihiro Kanamori, and Jun Mitani. 2016. Deepprop: Extracting deep features from a single image for edit propagation. Computer Graphics Forum 35, 2 (2016), 189-201.
12. Evelyn Fix and Joseph Lawson Hodges. 1989. Discriminatory analysis. Non- parametric discrimination: Consistency properties. International Statistical Re- view/Revue Internationale de Statistique 57, 3 (1989), 238-247.
13. Jorge Frascara. 2004. Communication design: principles, methods, and practice. Allworth Communications, Inc., New York, USA.
14. Tong Gao, Jessica R. Hullman, Eytan Adar, Brent Hecht, and Nicholas Diakopou- los. 2014. NewsViews: An Automated Pipeline for Creating Custom Geovisu- alizations for News. In Proc. of the CHI Conf. on Human Factors in Computing Systems (Toronto, ON, Canada) (CHI '14). ACM, New York, USA, 3005-3014. https://doi.org/10.1145/2556288.2557228
15. Arjan Gijsenij, Marjan Vazirian, Peter Spiers, Stephen Westland, and Pim Koeck- hoven. 2022. Determining key colors from a design perspective using dE-means color clustering. Color Research & Application 48 (2022), 69-87.
16. Connor C Gramazio, David H Laidlaw, and Karen B Schloss. 2016. Colorgorical: Creating Discriminable and Preferable Color Palettes for Information Visualiza- tion. IEEE TVCG 23, 1 (2016), 521-530.
17. Shunan Guo, Zhuochen Jin, Fuling Sun, Jingwen Li, Zhaorui Li, Yang Shi, and Nan Cao. 2021. Vinci: An Intelligent Graphic Design System for Generating Advertising Posters. In Proc. of the 2021 CHI Conf. on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). ACM, New York, USA, Article 577, 17 pages. https://doi.org/10.1145/3411764.3445117
18. Mitsuhiko Hanada. 2018. Correspondence analysis of color-emotion associations. Color Research & Application 43, 2 (2018), 224-237.
19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR (Las Vegas, NV, USA). IEEE, New York, USA, 770-778.
20. Mingming He, Dongdong Chen, Jing Liao, Pedro V Sander, and Lu Yuan. 2018. Deep exemplar-based colorization. ACM Trans. on Graphics 37, 4 (2018), 1-16.
21. Jefrey Heer. 2019. Agency plus automation: Designing artifcial intelligence into interactive systems. Proc. of the National Academy of Sciences 116, 6 (2019), 1844-1850.
22. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Proc. of the 31st International Conf. on Neural Information Processing Systems (Long Beach, California, USA) (NIPS'17). Curran Associates Inc., Red Hook, NY, USA, 6629-6640.
23. Laura Hummell. 2006. Synectics for creative thinking in technology education. Technology and Engineering Teacher 66, 3 (2006), 22.
24. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to- image translation with conditional adversarial networks. In CVPR (Honolulu, HI, USA). IEEE, New York, USA, 1125-1134.
25. Ali Jahanian, SVN Vishwanathan, and Jan P Allebach. 2015. Autonomous Color Theme Extraction from Images using Saliency. In Imaging and Multimedia Ana- lytics in a Web and Mobile World 2015, Vol. 9408. International Society for Optics and Photonics, San Francisco, CA, USA, 940807.
26. Ghita Jalal, Nolwenn Maudet, and Wendy E Mackay. 2015. Color portraits: From color picking to interacting with color. In Proc. of the 2015 CHI Conf. on Human Factors in Computing Systems. ACM, New York, USA, 4207-4216.
27. EunJin Kim and Hyeon-Jeong Suk. 2018. Image color adjustment for harmony with a target color. Color Research & Application 43, 1 (2018), 75-88.
28. Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization using optimiza- tion. In ACM SIGGRAPH 2004 Papers. ACM, New York, USA, 689-694.
29. Yuanzhen Li, Edward Adelson, and Aseem Agarwala. 2008. ScribbleBoost: Adding Classifcation to Edge-Aware Interpolation of Local Image and Video Adjustments. Computer Graphics Forum 27, 4 (2008), 1255-1264.
30. Simon Lok and Steven Feiner. 2001. A survey of automated layout techniques for information presentations. Proc. of SmartGraphics 2001 (2001), 61-68.
31. Kecheng Lu, Mi Feng, Xin Chen, Michael Sedlmair, Oliver Deussen, Dani Lischin- ski, Zhanglin Cheng, and Yunhai Wang. 2021. Palettailor: Discriminable Col- orization for Categorical Data. IEEE TVCG 27, 2 (2021), 475-484.
32. Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying-Qing Xu, and Heung- Yeung Shum. 2007. Natural Image Colorization. In Rendering Techniques, Jan Kautz and Sumanta Pattanaik (Eds.). The Eurographics Association, Eindhoven, Netherlands, 309-320. https://doi.org/10.2312/EGWR/EGSR07/309-320
33. Kazuo Misue. 2020. Tools for Developing Color Ramps for Representing Quanti- tative Data. In Proc. of the 13th International Symposium on Visual Information Communication and Interaction (Eindhoven, Netherlands) (VINCI '20). ACM, New York, USA, Article 17, 5 pages. https://doi.org/10.1145/3430036.3430056
34. Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2011. Color Com- patibility from Large Datasets. In ACM SIGGRAPH 2011 Papers (Vancouver, BC, Canada). ACM, New York, USA, Article 63, 12 pages. https://doi.org/10.1145/ 1964921.1964958
35. Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. DesignScape: Design with Interactive Layout Suggestions. In Proc. of the 2015 CHI Conf. on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). ACM, New York, USA, 1221-1224. https://doi.org/10.1145/2702123.2702149
36. Yingxue Pang, Jianxin Lin, Tao Qin, and Zhibo Chen. 2022. Image-to-Image Translation: Methods and Applications. IEEE Trans. on Multimedia 24 (2022), 3859-3881.
37. Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. Semantic Image Synthesis with Spatially-Adaptive Normalization. In CVPR. IEEE, New York, USA, 2337-2346.
38. Yuting Qiang, Yanwei Fu, Yanwen Guo, Zhi-Hua Zhou, and Leonid Sigal. 2016. Learning to generate posters of scientifc papers. In Proc. of the AAAI Conf. on Artifcial Intelligence, Vol. 30. AAAI Press, Palo Alto, CA, USA, 1.
39. Qianru Qiu, Mayu Otani, and Yuki Iwazaki. 2022. An Intelligent Color Recom- mendation Tool for Landing Page Design. In 27th International Conf. on Intelligent User Interfaces (Helsinki, Finland) (IUI '22 Companion). ACM, New York, USA, 26-29. https://doi.org/10.1145/3490100.3516450
40. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In International Conf. on Medical image computing and computer-assisted intervention, Vol. 9351. Springer, New York, USA, 234-241. https://doi.org/10.1007/978-3-319-24574-4\_28
41. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115, 3 (2015), 211-252. https: //doi.org/10.1007/s11263-015-0816-y
42. Maria Shugrina, Amlan Kar, Sanja Fidler, and Karan Singh. 2020. Nonlinear color triads for approximation, learning and direct manipulation of color distributions. ACM Trans. on Graphics 39, 4 (2020), 97-1.
43. KyoungHee Son, Seo Young Oh, Yongkwan Kim, Hayan Choi, Seok-Hyung Bae, and Ganguk Hwang. 2015. Color Sommelier: Interactive Color Recommendation System Based on Community-Generated Color Palettes. In Adjunct Proc. of the 28th Annual ACM Symposium on User Interface Software & Technology (Daegu, Kyungpook, Republic of Korea) (UIST '15 Adjunct). ACM, New York, USA, 95-96. https://doi.org/10.1145/2815585.2815736
44. Terry Lee Stone, Sean Adams, and Noreen Morioka. 2008. Color Design Workbook: A Real World Guide to Using Color in Graphic Design. Rockport Pub, Gloucester, MA, USA.
45. Christian Szegedy, Vincent Vanhoucke, Sergey Iofe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In CVPR (Las Vegas, NV, USA). IEEE, New York, USA, 2818-2826. https://doi.org/10.1109/ CVPR.2016.308
46. Jianchao Tan, Jose Echevarria, and Yotam Gingold. 2018. Efcient palette-based decomposition and recoloring of images via RGBXY-space geometry. ACM Trans. on Graphics 37, 6 (2018), 1-10.
47. Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and explore with a layout optimiser. In Proc. of the 2016 ACM Conf. on Designing Interactive Systems. ACM, New York, USA, 543-555.
48. Anjul Tyagi, Jian Zhao, Pushkar Patel, Swasti Khurana, and Klaus Mueller. 2022. Infographics Wizard: Flexible Infographics Authoring and Design Exploration. Computer Graphics Forum 41, 3 (2022), 121-132. https://doi.org/10.1111/cgf.14527
49. Baoyuan Wang, Yizhou Yu, Tien-Tsin Wong, Chun Chen, and Ying-Qing Xu. 2010. Data-driven image color theme enhancement. ACM Trans. on Graphics 29, 6 (2010), 1-10.
50. Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. 2021. Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data. In CVPRW. IEEE, New York, USA, 1905-1914.
51. Xintao Wang, Ke Yu, Chao Dong, Xiaoou Tang, and Chen Change Loy. 2019. Deep network interpolation for continuous imagery efect transition. In CVPR. IEEE, New York, USA, 1692-1701.
52. TW Whitfeld and TJ Whiltshire. 1990. Color psychology: a critical review. Genetic, social, and general psychology monographs 116, 4 (1990), 385-411.
53. Liron Yatziv and Guillermo Sapiro. 2006. Fast image and video colorization using chrominance blending. IEEE Trans. on image processing 15, 5 (2006), 1120-1129.
54. Wenyuan Yin, Tao Mei, and Chang Wen Chen. 2013. Automatic generation of social media snippets for mobile browsing. In Proc. of the 21st ACM international Conf. on Multimedia. ACM, New York, USA, 927-936.
55. Linping Yuan, Ziqi Zhou, Jian Zhao, Yiqiu Guo, Fan Du, and Huamin Qu. 2021. Infocolorizer: Interactive Recommendation of Color Palettes for Infographics. IEEE TVCG 28, 12 (2021), 4252-4266.
56. Qing Zhang, Chunxia Xiao, Hanqiu Sun, and Feng Tang. 2017. Palette-based image recoloring using color decomposition optimization. IEEE Trans. on Image Processing 26, 4 (2017), 1952-1964.
57. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Efectiveness of Deep Features as a Perceptual Metric. In CVPR (Salt Lake City, UT, USA). IEEE, New York, USA, 586-595.
58. Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros. 2017. Real-Time User-Guided Image Colorization with Learned Deep Priors. ACM Trans. on Graphics 36, 4, Article 119 (2017), 11 pages. https://doi.org/10.1145/3072959.3073703
59. Nanxuan Zhao, Quanlong Zheng, Jing Liao, Ying Cao, Hanspeter Pfster, and Rynson WH Lau. 2021. Selective Region-based Photo Color Adjustment for Graphic Designs. ACM Trans. on Graphics 40, 2 (2021), 1-16.
60. Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. 2019. Content- aware generative modeling of graphic design layouts. ACM Trans. on Graphics 38, 4 (2019), 1-15.