MelodicBrush: A novel system for cross-modal digital art creation linking calligraphy and music

Michael Xuelin Huang, Will W.W. Tang, Kenneth W.K. Lo, C. K. Lau, Grace Ngai, Stephen Chan

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

8 Citations (Scopus)

Abstract

MelodicBrush is a novel system that connects two ancient art forms: Chinese ink-brush calligraphy and Chinese music. Our system uses vision-based techniques to create a digitized ink-brush calligraphic writing surface with enhanced interaction functionalities. The music generation combines cross-modal stroke-note mapping and statistical language modeling techniques into a hybrid model that generates music as a real-time, auditory response and feedback to the user's calligraphic strokes. Our system is in fact a new cross-modal musical system that endows the ancient art of calligraphy writing with a novel auditory representation to provide the users with a natural and novel artistic experience. Experiment evaluations with real users suggest that MelodicBrush is intuitive and realistic, and can also be easily used to exercise creativity and support art generation.
Original languageEnglish
Title of host publicationProceedings of the Designing Interactive Systems Conference, DIS '12
Pages418-427
Number of pages10
DOIs
Publication statusPublished - 8 Aug 2012
EventDesigning Interactive Systems Conference, DIS '12 - Newcastle Upon Tyne, United Kingdom
Duration: 11 Jun 201215 Jun 2012

Conference

ConferenceDesigning Interactive Systems Conference, DIS '12
CountryUnited Kingdom
CityNewcastle Upon Tyne
Period11/06/1215/06/12

Keywords

  • Chinese calligraphy
  • Chinese guqin music
  • cross-modal interaction

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Human-Computer Interaction

Cite this