Dr. Marc A. Kastner

Über mich

Image-Collection Summarization using Scene-Graph Generation with External Knowledge

Zurück zu Veröffentlichungen

Authoren: Itthisak Phueaksri, Marc A. Kastner, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide


Summarization tasks aim to summarize multiple pieces of information into a short description or representative information. A text summarization task is a task that summarizes textual information into a short description, whereas in an image collection summarization task, also known as the photo album summarization task, the goal is to find the representative visual information of all images in the collection. In recent years, scene-graph generation has shown the advantage of describing the visual contexts of a single- image, and incorporating external knowledge into the scene-graph generation model has also given effective directions for unseen single-image scene-graph generation. Following this trend, in this paper, we propose a novel scene-graph-based image-collection summarization model. The key idea of the proposed method is to enhance the relation predictor toward relationships between images in an image collection incorporating knowledge graphs as external knowledge for training a model. To evaluate the proposed method, we build an extended annotated MS-COCO dataset for this task and introduce an evaluation process that focuses on estimating the similarity between a summarized scene graph and ground-truth scene graphs. Traditional evaluation focuses on calculating precision and recall scores, which involve true positive predictions without balancing precision and recall. Meanwhile, the proposed evaluation process focuses on calculating the F- score of the similarity between a summarized scene graph and ground-truth scene graphs which aims to balance both false positives and false negatives. Experimental results show that the use of external knowledge in enhancing the relation predictor achieves better results compared with existing methods.

Typ: Journal paper at IEEE Access, vol. 12, pp. 17499-17512

Veröffentlichungsdatum: January 2024

DOI: 10.1109/ACCESS.2024.3360113


Wenn Sie Fragen oder Kommentare zu dieser Forschung haben, zögern Sie nicht einen Kommentar zu hinterlassen oder mir eine email zu schreiben. Ich werde mich zeitnahe zurückmelden.
© 2013-2023 Marc A. Kastner. Powered by KirbyCMS. Some rights reserved. Privacy policy.