Dr. Marc A. Kastner

About me

Tell as You Imagine: Sentence Imageability-aware Image Captioning

Back to publications

Authors: Kazuki Umemura, Marc A. Kastner, Ichiro Ide, Yasutomo Kawanishi, Takatsugu Hirayama, Keisuke Doman, Daisuke Deguchi, Hiroshi Murase


Image captioning as a multimedia task is advancing in terms of performance in generating captions for general purposes. However, tailoring generated captions to different applications remains difficult. In this paper, we propose a sentence imageability-aware image captioning method to generate captions tailoring to various applications. Sentence imageability describes how easily the caption can be mentally imagined. This concept is applied to the captioning model to get a better understanding of the perception of a generated caption. First, we extend an existing image caption dataset by augmenting its captions' diversity. Then, a sentence imageability score for each augmented caption is calculated. A modified image captioning model is trained using this extended dataset to generate captions tailored to a specified imageability score. Experiments show promising results in generating imageability-aware captions. Especially, results from a subjective experiment shows that the perception of the generated captions correlates with the specified score.

Type: MultiMedia Modelling (MMM) 2021. Lecture Notes in Computer Science, vol 12573

Publication date: January 2021

DOI: 10.1007/978-3-030-67835-7_6


Attached Files

If you have questions or ideas about this research, feel free to leave a comment below or send me an email. I will reply quickly.
© 2013-2023 Marc A. Kastner. Powered by KirbyCMS. Some rights reserved. Privacy policy.