Dr. Marc A. Kastner

About me

Imageability- and length-controllable image captioning

Back to publications list »

Marc A. Kastner, Kazuki Umemura, Ichiro Ide, Yasutomo Kawanishi, Takatsugu Hirayama, Keisuke Doman, Daisuke Deguchi, Hiroshi Murase, Shin'ichi Satoh

Abstract:
Image captioning can show great performance for generating captions for general purposes, but it remains difficult to adjust the generated captions for different applications. In this paper, we propose an image captioning method which can generate both imageability- and length-controllable captions. The imageability parameter adjusts the level of visual descriptiveness of the caption, making it either more abstract or more concrete. In contrast, the length parameter only adjusts the length of the caption while keeping the visual descriptiveness on a similar degree. Based on a transformer architecture, our model is trained using an augmented dataset with diversified captions across different degrees of descriptiveness. The resulting model can control both imageability and length, making it possible to tailor output towards various applications. Experiments show that we can maintain a captioning performance similar to comparison methods, while being able to control the visual descriptiveness and the length of the generated captions. A subjective evaluation with human participants also shows a significant correlation of the target imageability in terms of human expectations. Thus, we confirmed that the proposed method provides a promising step towards tailoring image captions closer to certain applications.
Type: Journal paper at IEEE Access, vol. 9, pp. 162951-162961
Date: November 2021
Related files: [ pre-print ]

If you have questions or ideas about this research, feel free to leave a comment below or send me an email. I will reply quickly.
Powered by Hugo. © 2013-2022 Marc A. Kastner. Some rights reserved. Privacy policy.