InfiMM: Advancing Multimodal Understanding with an Open-Sourced Visual Language Model

Haogeng Liu; Quanzeng You; Yiqi Wang; Xiaotian Han; Bohan Zhai; Yongfei Liu; Wentao Chen; Yiren Jian; Yunzhe Tao; Jianbo Yuan; Ran He; Hongxia Yang

doi:10.18653/v1/2024.findings-acl.27

InfiMM: Advancing Multimodal Understanding with an Open-Sourced Visual Language Model

Haogeng Liu
, Quanzeng You
, Yiqi Wang
, Xiaotian Han
, Bohan Zhai
, Yongfei Liu
, Wentao Chen
, Yiren Jian
, Yunzhe Tao
, Jianbo Yuan
, Ran He
, Hongxia Yang

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

5 Citations (Scopus)

Abstract

In this work, we present InfiMM, an advanced Multimodal Large Language Model that adapts to intricate vision-language tasks. InfiMM, inspired by the Flamingo architecture, distinguishes itself through the utilization of large-scale training data, three-stage training strategies, and diverse large language models. This approach ensures the preservation of Flamingo's foundational strengths while simultaneously introducing augmented capabilities. Empirical evaluations across a variety of benchmarks underscore InfiMM's remarkable capability in multimodal understanding. The code and model can be found at: https://huggingface.co/Infi-MM.

Original language	English
Title of host publication	The 62nd Annual Meeting of the Association for Computational Linguistics
Subtitle of host publication	Findings of the Association for Computational Linguistics, ACL 2024
Editors	Lun-Wei Ku, Andre Martins, Vivek Srikumar
Publisher	Association for Computational Linguistics (ACL)
Pages	485-492
Number of pages	8
ISBN (Electronic)	9798891760998
DOIs	https://doi.org/10.18653/v1/2024.findings-acl.27
Publication status	Published - Aug 2024
Externally published	Yes
Event	Findings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Hybrid, Bangkok, Thailand Duration: 11 Aug 2024 → 16 Aug 2024

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	Findings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024
Country/Territory	Thailand
City	Hybrid, Bangkok
Period	11/08/24 → 16/08/24

ASJC Scopus subject areas

Computer Science Applications
Linguistics and Language
Language and Linguistics

More information

10.18653/v1/2024.findings-acl.27

Cite this

Liu, H., You, Q., Wang, Y., Han, X., Zhai, B., Liu, Y., Chen, W., Jian, Y., Tao, Y., Yuan, J., He, R., & Yang, H. (2024). InfiMM: Advancing Multimodal Understanding with an Open-Sourced Visual Language Model. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), The 62nd Annual Meeting of the Association for Computational Linguistics: Findings of the Association for Computational Linguistics, ACL 2024 (pp. 485-492). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2024.findings-acl.27

Liu, Haogeng ; You, Quanzeng ; Wang, Yiqi et al. / InfiMM : Advancing Multimodal Understanding with an Open-Sourced Visual Language Model. The 62nd Annual Meeting of the Association for Computational Linguistics: Findings of the Association for Computational Linguistics, ACL 2024. editor / Lun-Wei Ku ; Andre Martins ; Vivek Srikumar. Association for Computational Linguistics (ACL), 2024. pp. 485-492 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{a94dc6a5d6e54e1eb1f499cbc08a8833,

title = "InfiMM: Advancing Multimodal Understanding with an Open-Sourced Visual Language Model",

abstract = "In this work, we present InfiMM, an advanced Multimodal Large Language Model that adapts to intricate vision-language tasks. InfiMM, inspired by the Flamingo architecture, distinguishes itself through the utilization of large-scale training data, three-stage training strategies, and diverse large language models. This approach ensures the preservation of Flamingo's foundational strengths while simultaneously introducing augmented capabilities. Empirical evaluations across a variety of benchmarks underscore InfiMM's remarkable capability in multimodal understanding. The code and model can be found at: https://huggingface.co/Infi-MM.",

author = "Haogeng Liu and Quanzeng You and Yiqi Wang and Xiaotian Han and Bohan Zhai and Yongfei Liu and Wentao Chen and Yiren Jian and Yunzhe Tao and Jianbo Yuan and Ran He and Hongxia Yang",

note = "Publisher Copyright: {\textcopyright} 2024 Association for Computational Linguistics.; Findings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 ; Conference date: 11-08-2024 Through 16-08-2024",

year = "2024",

month = aug,

doi = "10.18653/v1/2024.findings-acl.27",

language = "English",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "485--492",

editor = "Lun-Wei Ku and Andre Martins and Vivek Srikumar",

booktitle = "The 62nd Annual Meeting of the Association for Computational Linguistics",

address = "United States",

}

Liu, H, You, Q, Wang, Y, Han, X, Zhai, B, Liu, Y, Chen, W, Jian, Y, Tao, Y, Yuan, J, He, R & Yang, H 2024, InfiMM: Advancing Multimodal Understanding with an Open-Sourced Visual Language Model. in L-W Ku, A Martins & V Srikumar (eds), The 62nd Annual Meeting of the Association for Computational Linguistics: Findings of the Association for Computational Linguistics, ACL 2024. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 485-492, Findings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, Hybrid, Bangkok, Thailand, 11/08/24. https://doi.org/10.18653/v1/2024.findings-acl.27

InfiMM: Advancing Multimodal Understanding with an Open-Sourced Visual Language Model. / Liu, Haogeng; You, Quanzeng; Wang, Yiqi et al.
The 62nd Annual Meeting of the Association for Computational Linguistics: Findings of the Association for Computational Linguistics, ACL 2024. ed. / Lun-Wei Ku; Andre Martins; Vivek Srikumar. Association for Computational Linguistics (ACL), 2024. p. 485-492 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

TY - GEN

T1 - InfiMM

T2 - Findings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024

AU - Liu, Haogeng

AU - You, Quanzeng

AU - Wang, Yiqi

AU - Han, Xiaotian

AU - Zhai, Bohan

AU - Liu, Yongfei

AU - Chen, Wentao

AU - Jian, Yiren

AU - Tao, Yunzhe

AU - Yuan, Jianbo

AU - He, Ran

AU - Yang, Hongxia

PY - 2024/8

Y1 - 2024/8

N2 - In this work, we present InfiMM, an advanced Multimodal Large Language Model that adapts to intricate vision-language tasks. InfiMM, inspired by the Flamingo architecture, distinguishes itself through the utilization of large-scale training data, three-stage training strategies, and diverse large language models. This approach ensures the preservation of Flamingo's foundational strengths while simultaneously introducing augmented capabilities. Empirical evaluations across a variety of benchmarks underscore InfiMM's remarkable capability in multimodal understanding. The code and model can be found at: https://huggingface.co/Infi-MM.

AB - In this work, we present InfiMM, an advanced Multimodal Large Language Model that adapts to intricate vision-language tasks. InfiMM, inspired by the Flamingo architecture, distinguishes itself through the utilization of large-scale training data, three-stage training strategies, and diverse large language models. This approach ensures the preservation of Flamingo's foundational strengths while simultaneously introducing augmented capabilities. Empirical evaluations across a variety of benchmarks underscore InfiMM's remarkable capability in multimodal understanding. The code and model can be found at: https://huggingface.co/Infi-MM.

UR - https://www.scopus.com/pages/publications/85205305284

U2 - 10.18653/v1/2024.findings-acl.27

DO - 10.18653/v1/2024.findings-acl.27

M3 - Conference article published in proceeding or book

AN - SCOPUS:85205305284

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 485

EP - 492

BT - The 62nd Annual Meeting of the Association for Computational Linguistics

A2 - Ku, Lun-Wei

A2 - Martins, Andre

A2 - Srikumar, Vivek

PB - Association for Computational Linguistics (ACL)

Y2 - 11 August 2024 through 16 August 2024

ER -

Liu H, You Q, Wang Y, Han X, Zhai B, Liu Y et al. InfiMM: Advancing Multimodal Understanding with an Open-Sourced Visual Language Model. In Ku LW, Martins A, Srikumar V, editors, The 62nd Annual Meeting of the Association for Computational Linguistics: Findings of the Association for Computational Linguistics, ACL 2024. Association for Computational Linguistics (ACL). 2024. p. 485-492. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2024.findings-acl.27

InfiMM: Advancing Multimodal Understanding with an Open-Sourced Visual Language Model

Abstract

Publication series

Conference

ASJC Scopus subject areas

More information

Other files and links

Fingerprint

Cite this