Dr. Marc A. Kastner

Über mich

Nonword-to-Image Generation Considering Perceptual Association of Phonetically Similar Words

Zurück zu Veröffentlichungen

Authoren: Chihaya Matsuhira, Marc A. Kastner, Takahiro Komamizu, Takatsugu Hirayama, Keisuke Doman, Ichiro Ide


Text-to-Image (T2I) generation has long been a popular field of multimedia processing. Recent advances in large-scale vision and language pretraining have brought a number of models capable of very high-quality T2I generation. However, they are reported to generate unexpected images when users input words that have no definition within a language (nonwords), including coined words and pseudo-words. To make the behavior of T2I generation models against nonwords more intuitive, we propose a method that considers phonetic information of text inputs. The phonetic similarity is adopted so that the generated images from a nonword contain the concept of its phonetically similar words. This is based on the psycholinguistic finding that humans would also associate nonwords with their phonetically similar words when they perceive the sound. Our evaluations confirm a better agreement of the generated images of the proposed method with both phonetic relationships and human expectations than a conventional T2I generation model. The cross-lingual comparison of generated images for a nonword highlights the differences in language-specific nonword-imagery correspondences. These results provide insight into the usefulness of the proposed method in brand naming and language learning.

Typ: 1st International Workshop on Multimedia Content Generation and Evaluation (co-located at ACM Multimedia 2023)

Veröffentlichungsdatum: To be published in Oct 2023

Wenn Sie Fragen oder Kommentare zu dieser Forschung haben, zögern Sie nicht einen Kommentar zu hinterlassen oder mir eine email zu schreiben. Ich werde mich zeitnahe zurückmelden.
© 2013-2023 Marc A. Kastner. Powered by KirbyCMS. Some rights reserved. Privacy policy.