research-article
Authors: Enhao Zhang and Nikola Banovic
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
May 2021
Article No.: 76, Pages 1 - 15
Published: 07 May 2021 Publication History
- 18citation
- 1,286
- Downloads
Metrics
Total Citations18Total Downloads1,286Last 12 Months170
Last 6 weeks7
New Citation Alert added!
This alert has been successfully added and will be sent to:
You will be notified whenever a record that you have chosen has been cited.
To manage your alert preferences, click on the button below.
Manage my Alerts
New Citation Alert!
Please log in to your account
Get Access
- Get Access
- References
- Media
- Tables
- Share
Abstract
Generative Adversarial Networks (GANs) can automatically generate quality images from learned model parameters. However, it remains challenging to explore and objectively assess the quality of all possible images generated using a GAN. Currently, model creators evaluate their GANs via tedious visual examination of generated images sampled from narrow prior probability distributions on model parameters. Here, we introduce an interactive method to explore and sample quality images from GANs. Our first two user studies showed that participants can use the tool to explore a GAN and select quality images. Our third user study showed that images sampled from a posterior probability distribution using a Markov Chain Monte Carlo (MCMC) method on parameters of images collected in our first study resulted in on average higher quality and more diverse images than existing baselines. Our work enables principled qualitative GAN exploration and evaluation.
Supplementary Material
Supplemental video
- Download
- 59.16 MB
Preview video
- Download
- 12.36 MB
References
[1]
Amazon. 2020. Amazon Mechanical Turk. https://www.mturk.com/
[2]
Christophe Andrieu, Nando DeFreitas, Arnaud Doucet, and MichaelI Jordan. 2003. An introduction to MCMC for machine learning. Machine learning 50, 1-2 (2003), 5–43.
[3]
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, JoshuaB. Tenenbaum, WilliamT. Freeman, and Antonio Torralba. 2019. GAN Dissection: Visualizing and Understanding Generative Adversarial Networks. In Proceedings of the International Conference on Learning Representations (ICLR).
[4]
Ali Borji. 2019. Pros and cons of GAN evaluation measures. Computer Vision and Image Understanding 179 (2019), 41 – 65. https://doi.org/10.1016/j.cviu.2018.10.009
Digital Library
[5]
Eric Brochu, VladM Cora, and Nando DeFreitas. 2010. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599(2010).
[6]
Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large Scale GAN Training for High Fidelity Natural Image Synthesis. arxiv:1809.11096[cs.LG]
[7]
Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In International Conference on Learning Representations. https://openreview.net/forum?id=B1xsqj09Fm
[8]
Andrew Brock, Theodore Lim, JamesM Ritchie, and Nick Weston. 2016. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv:1609.07093(2016).
[9]
Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng(Eds.). 2011. Handbook of Markov Chain Monte Carlo. Chapman and Hall.
[10]
Wengling Chen and James Hays. 2018. SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[11]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.
[12]
Brochu Eric, NandoD Freitas, and Abhijeet Ghosh. 2008. Active preference learning with discrete choice data. In Advances in neural information processing systems. 409–416.
[13]
Edoardo Giacomello, PierLuca Lanzi, and Daniele Loiacono. 2019. Searching the Latent Space of a Generative Adversarial Network to Generate DOOM Levels. In 2019 IEEE Conference on Games (CoG). 1–8. https://doi.org/10.1109/CIG.2019.8848011
Digital Library
[14]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27, Z.Ghahramani, M.Welling, C.Cortes, N.D. Lawrence, and K.Q. Weinberger(Eds.). Curran Associates, Inc., 2672–2680. http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf
[15]
Aaron Hertzmann. 2020. Visual Indeterminacy in GAN Art. Leonardo 53, 4 (2020), 424–428. https://doi.org/10.1162/leon_a_01930arXiv:https://doi.org/10.1162/leon_a_01930
[16]
MatthewD Hoffman and Andrew Gelman. 2014. The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo.J. Mach. Learn. Res. 15, 1 (2014), 1593–1623.
[17]
Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal Unsupervised Image-to-image Translation. In Proceedings of the European Conference on Computer Vision (ECCV).
Digital Library
[18]
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. 2020. GANSpace: Discovering Interpretable GAN Controls. In Advances in Neural Information Processing Systems, Vol.33. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2020/file/6fe43269967adbb64ec6149852b5cc3e-Paper.pdf
[19]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and AlexeiA. Efros. 2017. Image-To-Image Translation With Conditional Adversarial Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[20]
Mario Klingemann. 2020. Trapping the Accident. In PhotographyDigitalPainting: Expanding Medium Interconnectivity in Contemporary Visual Art Practices(first ed.), Carl Robinson (Ed.). Cambridge Scholars Publishing, 77–98.
[21]
Yuki Koyama, Issei Sato, and Masataka Goto. 2020. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. 39, 4, Article 88 (July 2020), 12pages. https://doi.org/10.1145/3386569.3392444
Digital Library
[22]
Yuki Koyama, Issei Sato, Daisuke Sakamoto, and Takeo Igarashi. 2017. Sequential Line Search for Efficient Visual Design Optimization by Crowds. ACM Trans. Graph. 36, 4, Article 48 (July 2017), 11pages. https://doi.org/10.1145/3072959.3073598
Digital Library
[23]
J. Marks, B. Andalman, P.A. Beardsley, W. Freeman, S. Gibson, J. Hodgins, T. Kang, B. Mirtich, H. Pfister, W. Ruml, K. Ryall, J. Seims, and S. Shieber. 1997. Design Galleries: A General Approach to Setting Parameters for Computer Graphics and Animation. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’97). ACM Press/Addison-Wesley Publishing Co., USA, 389–400. https://doi.org/10.1145/258734.258887
Digital Library
[24]
Jakob Nielsen and Rolf Molich. 1990. Heuristic Evaluation of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI ’90). Association for Computing Machinery, New York, NY, USA, 249–256. https://doi.org/10.1145/97243.97281
Digital Library
[25]
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. Semantic Image Synthesis With Spatially-Adaptive Normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[26]
Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arxiv:1511.06434[cs.LG]
[27]
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. 2016. Improved Techniques for Training GANs. In Advances in Neural Information Processing Systems 29, D.D. Lee, M.Sugiyama, U.V. Luxburg, I.Guyon, and R.Garnett (Eds.). Curran Associates, Inc., 2234–2242. http://papers.nips.cc/paper/6125-improved-techniques-for-training-gans.pdf
[28]
John Salvatier, ThomasV Wiecki, and Christopher Fonnesbeck. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science 2(2016), e55.
[29]
Helena Sarin. 2019. The Books of GANesis: Divine Comedy in Tangled Representations. Helena Sarin.
[30]
Jacob Schrum, Jake Gutierrez, Vanessa Volz, Jialin Liu, Simon Lucas, and Sebastian Risi. 2020. Interactive Evolution and Exploration within Latent Level-Design Space of Generative Adversarial Networks. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference (Cancún, Mexico) (GECCO ’20). Association for Computing Machinery, New York, NY, USA, 148–156. https://doi.org/10.1145/3377930.3389821
Digital Library
[31]
Ben Shneiderman. 2007. Creativity Support Tools: Accelerating Discovery and Innovation. Commun. ACM 50, 12 (Dec. 2007), 20–32. https://doi.org/10.1145/1323688.1323689
Digital Library
[32]
JacobO. Wobbrock, Leah Findlater, Darren Gergle, and JamesJ. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 143–146. https://doi.org/10.1145/1978942.1978963
Digital Library
[33]
Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. 2019. GP-GAN: Towards Realistic High-Resolution Image Blending. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). Association for Computing Machinery, New York, NY, USA, 2487–2495. https://doi.org/10.1145/3343031.3350944
Digital Library
[34]
Sharon Zhou, Mitchell Gordon, Ranjay Krishna, Austin Narcomey, LiF Fei-Fei, and Michael Bernstein. 2019. HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models. In Advances in Neural Information Processing Systems 32, H.Wallach, H.Larochelle, A.Beygelzimer, F.d'Alché-Buc, E.Fox, and R.Garnett (Eds.). Curran Associates, Inc., 3449–3461. http://papers.nips.cc/paper/8605-hype-a-benchmark-for-human-eye-perceptual-evaluation-of-generative-models.pdf
[35]
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and AlexeiA. Efros. 2016. Generative Visual Manipulation on the Natural Image Manifold. In Computer Vision – ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling(Eds.). Springer International Publishing, Cham, 597–613.
Cited By
View all
- Vincenzi BStumpf STaylor ANakao Y(2024)Lay User Involvement in Developing Human-centric Responsible AI Systems: When and How?ACM Journal on Responsible Computing10.1145/36525921:2(1-25)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3652592
- Choi DHong SPark JChung JKim J(2024)CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AIProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642794(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642794
- Xiao SWang LMa XZeng W(2024)TypeDance: Creating Semantic Typographic Logos from Image through Personalized GenerationProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642185(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642185
- Show More Cited By
Index Terms
Method for Exploring Generative Adversarial Networks (GANs) via Automatically Generated Image Galleries
Computing methodologies
Artificial intelligence
Computer vision
Computer graphics
Machine learning
Index terms have been assigned to the content through auto-classification.
Recommendations
- Image Denoising via Generative Adversarial Networks with Detail Loss
ICISS '19: Proceedings of the 2nd International Conference on Information Science and Systems
Image denoising is a challenging task which aims to remove additional noise and preserve all useful information. Many existing image denoising algorithms focus on improving the typical object measure, peak signal-to-noise ratio (PSNR), and take the mean ...
Read More
- Image Dehazing Via Cycle Generative Adversarial Network
AISS '21: Proceedings of the 3rd International Conference on Advanced Information Science and System
Recovering a clear image from single hazy image has been widely investigated in recent researches. Due to the lack of the real hazed image dataset, most studies use artificially synthesized dataset to train the models. Nonetheless, the real word foggy ...
Read More
- A Method forFace Image Inpainting Based onAutoencoder andGenerative Adversarial Network
Image and Video Technology
Abstract
Face image inpainting has great value in the fields of computer vision and digital image processing. In this paper, we propose a face image inpainting method based on autoencoder and Generative Adversarial Network (GAN). The neural network for ...
Read More
Comments
Information & Contributors
Information
Published In
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
May 2021
10862 pages
ISBN:9781450380966
DOI:10.1145/3411764
- General Chairs:
- Yoshifumi Kitamura
Tohoku University, Japan
, - Aaron Quigley
University of New South Wales, Australia
, - Program Chairs:
- Katherine Isbister
University of California Santa Cruz, USA
, - Takeo Igarashi
The University of Tokyo, Japan
, - Publications Chairs:
- Pernille Bjørn
University of Copenhagen, Denmark
, - Steven Drucker
Microsoft Research, USA
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [emailprotected].
Sponsors
- SIGCHI: ACM Special Interest Group on Computer-Human Interaction
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 May 2021
Permissions
Request permissions for this article.
Check for updates
Author Tags
- Interactive model exploration
- qualitative model validation.
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
CHI '21
Sponsor:
- SIGCHI
Acceptance Rates
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%
Contributors
Other Metrics
View Article Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations
18
Total Citations
1,286
Total Downloads
- Downloads (Last 12 months)170
- Downloads (Last 6 weeks)7
Other Metrics
View Author Metrics
Citations
Cited By
View all
- Vincenzi BStumpf STaylor ANakao Y(2024)Lay User Involvement in Developing Human-centric Responsible AI Systems: When and How?ACM Journal on Responsible Computing10.1145/36525921:2(1-25)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3652592
- Choi DHong SPark JChung JKim J(2024)CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AIProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642794(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642794
- Xiao SWang LMa XZeng W(2024)TypeDance: Creating Semantic Typographic Logos from Image through Personalized GenerationProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642185(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642185
- Pang RSanty SJust RReinecke K(2024)BLIP: Facilitating the Exploration of Undesirable Consequences of Digital TechnologiesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642054(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642054
- Kim TLee YChang MKim J(2023)Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language ModelsProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606833(1-18)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606833
- Chung JAdar E(2023)PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like InteractionsProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606777(1-17)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606777
- Brachman MPan QDo HDugan CChaudhary AJohnson JRai PChakraborti TGschwind TLaredo JMiksovic CScotton PTalamadupula KThomas G(2023)Follow the Successful Herd: Towards Explanations for Improved Use and Mental Models of Natural Language SystemsProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584088(220-239)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584088
- Ko HPark GJeon HJo JKim JSeo J(2023)Large-scale Text-to-Image Generation Models for Visual Artists’ Creative WorksProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584078(919-933)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584078
- Antar AKratz ABanovic N(2023)Behavior Modeling Approach for Forecasting Physical Functioning of People with Multiple SclerosisProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808877:1(1-29)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3580887
- Banovic NYang ZRamesh ALiu A(2023)Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their TrustProceedings of the ACM on Human-Computer Interaction10.1145/35794607:CSCW1(1-17)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579460
- Show More Cited By
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in
Full Access
Get this Publication
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatMedia
Figures
Other
Tables