MelodicBrush: A novel system for cross-modal digital art creation linking calligraphy and music

Michael Xuelin Huang, Will W.W. Tang, Kenneth W.K. Lo, C. K. Lau, Grace Ngai, Stephen Chan

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

12 Citations (Scopus)


MelodicBrush is a novel system that connects two ancient art forms: Chinese ink-brush calligraphy and Chinese music. Our system uses vision-based techniques to create a digitized ink-brush calligraphic writing surface with enhanced interaction functionalities. The music generation combines cross-modal stroke-note mapping and statistical language modeling techniques into a hybrid model that generates music as a real-time, auditory response and feedback to the user's calligraphic strokes. Our system is in fact a new cross-modal musical system that endows the ancient art of calligraphy writing with a novel auditory representation to provide the users with a natural and novel artistic experience. Experiment evaluations with real users suggest that MelodicBrush is intuitive and realistic, and can also be easily used to exercise creativity and support art generation.
Original languageEnglish
Title of host publicationProceedings of the Designing Interactive Systems Conference, DIS '12
Number of pages10
Publication statusPublished - 8 Aug 2012
EventDesigning Interactive Systems Conference, DIS '12 - Newcastle Upon Tyne, United Kingdom
Duration: 11 Jun 201215 Jun 2012


ConferenceDesigning Interactive Systems Conference, DIS '12
Country/TerritoryUnited Kingdom
CityNewcastle Upon Tyne


  • Chinese calligraphy
  • Chinese guqin music
  • cross-modal interaction

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Human-Computer Interaction


Dive into the research topics of 'MelodicBrush: A novel system for cross-modal digital art creation linking calligraphy and music'. Together they form a unique fingerprint.

Cite this