カストナー・マーク

博士(情報学)

Tell as You Imagine: Sentence Imageability-aware Image Captioning

研究業績へ戻る

著者: Kazuki Umemura, Marc A. Kastner, Ichiro Ide, Yasutomo Kawanishi, Takatsugu Hirayama, Keisuke Doman, Daisuke Deguchi, Hiroshi Murase

あらすじ:

Image captioning as a multimedia task is advancing in terms of performance in generating captions for general purposes. However, tailoring generated captions to different applications remains difficult. In this paper, we propose a sentence imageability-aware image captioning method to generate captions tailoring to various applications. Sentence imageability describes how easily the caption can be mentally imagined. This concept is applied to the captioning model to get a better understanding of the perception of a generated caption. First, we extend an existing image caption dataset by augmenting its captions' diversity. Then, a sentence imageability score for each augmented caption is calculated. A modified image captioning model is trained using this extended dataset to generate captions tailored to a specified imageability score. Experiments show promising results in generating imageability-aware captions. Especially, results from a subjective experiment shows that the perception of the generated captions correlates with the specified score.

種類: MultiMedia Modelling (MMM) 2021. Lecture Notes in Computer Science, vol 12573

日付: January 2021

DOI: 10.1007/978-3-030-67835-7_6


発表資料


添付ファイル


この研究についてコメントやご意見がある場合、ぜひ以下にコメントを投稿してくだい。メールにてご連絡も大歓迎です。
© 2013-2023 Marc A. Kastner. Powered by KirbyCMS. Some rights reserved. Privacy policy.