MelodicBrush: A novel system for cross-modal digital art creation linking calligraphy and music

Michael Xuelin Huang; Will W.W. Tang; Kenneth W.K. Lo; C. K. Lau; Grace Ngai; Stephen Chan

doi:10.1145/2317956.2318018

MelodicBrush: A novel system for cross-modal digital art creation linking calligraphy and music

Michael Xuelin Huang
, Will W.W. Tang
, Kenneth W.K. Lo
, C. K. Lau
, Grace Ngai
, Stephen Chan

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

18 Citations (Scopus)

Abstract

MelodicBrush is a novel system that connects two ancient art forms: Chinese ink-brush calligraphy and Chinese music. Our system uses vision-based techniques to create a digitized ink-brush calligraphic writing surface with enhanced interaction functionalities. The music generation combines cross-modal stroke-note mapping and statistical language modeling techniques into a hybrid model that generates music as a real-time, auditory response and feedback to the user's calligraphic strokes. Our system is in fact a new cross-modal musical system that endows the ancient art of calligraphy writing with a novel auditory representation to provide the users with a natural and novel artistic experience. Experiment evaluations with real users suggest that MelodicBrush is intuitive and realistic, and can also be easily used to exercise creativity and support art generation.

Original language	English
Title of host publication	Proceedings of the Designing Interactive Systems Conference, DIS '12
Pages	418-427
Number of pages	10
DOIs	https://doi.org/10.1145/2317956.2318018
Publication status	Published - 8 Aug 2012
Event	Designing Interactive Systems Conference, DIS '12 - Newcastle Upon Tyne, United Kingdom Duration: 11 Jun 2012 → 15 Jun 2012

Conference

Conference	Designing Interactive Systems Conference, DIS '12
Country/Territory	United Kingdom
City	Newcastle Upon Tyne
Period	11/06/12 → 15/06/12

Keywords

Chinese calligraphy
Chinese guqin music
cross-modal interaction

ASJC Scopus subject areas

Computer Networks and Communications
Human-Computer Interaction

More information

10.1145/2317956.2318018

Cite this

@inproceedings{75f54183e0e74f93acf643caa7d1119e,

title = "MelodicBrush: A novel system for cross-modal digital art creation linking calligraphy and music",

abstract = "MelodicBrush is a novel system that connects two ancient art forms: Chinese ink-brush calligraphy and Chinese music. Our system uses vision-based techniques to create a digitized ink-brush calligraphic writing surface with enhanced interaction functionalities. The music generation combines cross-modal stroke-note mapping and statistical language modeling techniques into a hybrid model that generates music as a real-time, auditory response and feedback to the user's calligraphic strokes. Our system is in fact a new cross-modal musical system that endows the ancient art of calligraphy writing with a novel auditory representation to provide the users with a natural and novel artistic experience. Experiment evaluations with real users suggest that MelodicBrush is intuitive and realistic, and can also be easily used to exercise creativity and support art generation.",

keywords = "Chinese calligraphy, Chinese guqin music, cross-modal interaction",

author = "Huang, \{Michael Xuelin\} and Tang, \{Will W.W.\} and Lo, \{Kenneth W.K.\} and Lau, \{C. K.\} and Grace Ngai and Stephen Chan",

year = "2012",

month = aug,

day = "8",

doi = "10.1145/2317956.2318018",

language = "English",

isbn = "9781450312103",

pages = "418--427",

booktitle = "Proceedings of the Designing Interactive Systems Conference, DIS '12",

note = "Designing Interactive Systems Conference, DIS '12 ; Conference date: 11-06-2012 Through 15-06-2012",

}

Huang, MX, Tang, WWW, Lo, KWK, Lau, CK, Ngai, G & Chan, S 2012, MelodicBrush: A novel system for cross-modal digital art creation linking calligraphy and music. in Proceedings of the Designing Interactive Systems Conference, DIS '12. pp. 418-427, Designing Interactive Systems Conference, DIS '12, Newcastle Upon Tyne, United Kingdom, 11/06/12. https://doi.org/10.1145/2317956.2318018

MelodicBrush: A novel system for cross-modal digital art creation linking calligraphy and music. / Huang, Michael Xuelin; Tang, Will W.W.; Lo, Kenneth W.K. et al.
Proceedings of the Designing Interactive Systems Conference, DIS '12. 2012. p. 418-427.

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

TY - GEN

T1 - MelodicBrush: A novel system for cross-modal digital art creation linking calligraphy and music

AU - Huang, Michael Xuelin

AU - Tang, Will W.W.

AU - Lo, Kenneth W.K.

AU - Lau, C. K.

AU - Ngai, Grace

AU - Chan, Stephen

PY - 2012/8/8

Y1 - 2012/8/8

N2 - MelodicBrush is a novel system that connects two ancient art forms: Chinese ink-brush calligraphy and Chinese music. Our system uses vision-based techniques to create a digitized ink-brush calligraphic writing surface with enhanced interaction functionalities. The music generation combines cross-modal stroke-note mapping and statistical language modeling techniques into a hybrid model that generates music as a real-time, auditory response and feedback to the user's calligraphic strokes. Our system is in fact a new cross-modal musical system that endows the ancient art of calligraphy writing with a novel auditory representation to provide the users with a natural and novel artistic experience. Experiment evaluations with real users suggest that MelodicBrush is intuitive and realistic, and can also be easily used to exercise creativity and support art generation.

AB - MelodicBrush is a novel system that connects two ancient art forms: Chinese ink-brush calligraphy and Chinese music. Our system uses vision-based techniques to create a digitized ink-brush calligraphic writing surface with enhanced interaction functionalities. The music generation combines cross-modal stroke-note mapping and statistical language modeling techniques into a hybrid model that generates music as a real-time, auditory response and feedback to the user's calligraphic strokes. Our system is in fact a new cross-modal musical system that endows the ancient art of calligraphy writing with a novel auditory representation to provide the users with a natural and novel artistic experience. Experiment evaluations with real users suggest that MelodicBrush is intuitive and realistic, and can also be easily used to exercise creativity and support art generation.

KW - Chinese calligraphy

KW - Chinese guqin music

KW - cross-modal interaction

UR - https://www.scopus.com/pages/publications/84864744886

U2 - 10.1145/2317956.2318018

DO - 10.1145/2317956.2318018

M3 - Conference article published in proceeding or book

SN - 9781450312103

SP - 418

EP - 427

BT - Proceedings of the Designing Interactive Systems Conference, DIS '12

T2 - Designing Interactive Systems Conference, DIS '12

Y2 - 11 June 2012 through 15 June 2012

ER -

MelodicBrush: A novel system for cross-modal digital art creation linking calligraphy and music

Abstract

Conference

Keywords

ASJC Scopus subject areas

More information

Other files and links

Fingerprint

Cite this