The technology is focused on the multilayer coding of music information by the XML (eXtensible Markup Language) and the multimodal access to that information; it consists of a set of functional units devoted to make music contents interactive; it is a new advanced technology for interacting with music contents and related media. It is mainly based on the adoption of the new IEEE1599 music standard, a multilayer XML coding format for music information which allows the publisher or even the user to produce new interactive music media where audio/video tracks, music scores, lyrics, and other music information are all automatically synchronized and active for interaction in the frame of multimedia playing.
The IEEE1599-2008 layers include:
- standard linear (Compact-Disc Digital Audio, .wav, DVD Audio, and others) and compressed :MP3, AAC (Advance Audio Coding), WMA (Windows Media Audio, and others), audio codes,
- MIDI ( Musical Instrument Digital Interface) codes,
- score sheets codes (Coda Finale files), Music XML, NIFF (Native Image File Format), and others),
- metadata such as those of the recording company archives,
- structural information such as graphs, schema, forms, etc.,
- ordering, spatial and timing data.
The core of the IEEE1599 approach is an XML code, containing logical information about the piece and synchronization information about the linked multimedia objects, according to its own multilayer coding format. An overall synchronization is provided, among graphic objects representing scores and related texts - such as libretto and lyrics - and audio and video clips containing human performances.
In this context, a rich but simple user interfaces can be designed, conceived for not cultured people, in order to listen/watch to an audio/video track one track freely chosen within a set of different audio/video interpretations and look at a score a score freely chosen within a set of music sheets related to the same piece of music. This kind of human-device interface can be adopted for any kind of device desktop and/or mobile devices equipped with audio and video output channels and cursors position control (joystick, mouse, arrows, etc.), even on the web.
During the playing of the music piece, users can switch the instrument to be followed, as well as the audio being played and the score to be observed. All these results can be achieved by clicking any point of the current graphical score (synchronization driven by spatial coordinates), by dragging the slider of the audio/video player (synchronization driven by time coordinates), and even by selecting syllables of the libretto/lyrics (navigation by text contents). Overall synchronization is immediately available.
What makes this new technology interesting is that there exists a set of prototypal automatic software tools i.e. the core of the technology - for converting original music media such as audio, video, score, text, and lyrics files into the IEEE1599 format, giving the automatic synchronization of all them at low cost of production, by means of a semi-automatic way of doing it. Those automatic tools can be even improved and a platform for developing IEEE1599 music software packages and authoring new interactive media producing is in progress.
This new technology can be considered both at the maximum level of interacting features, and at simpler application level such as automatically giving synchronization while playing to lyrics and/or chords to both audio and video components of music tracks.
Innovative Aspects:
The key advantage is association of different music data information from different data sources in different formats (music metadata, digital rights management, music structural schema i.e. forms, music sheets, MIDI sequences, stereo & multitrack audio recordings, single and multi angle video recordings, images, lyrics, other documents), fully integrated and synchronized in one only information unit. Currently, no competitor is known which can be compared to this integrated computer-based approach to music cultural heritage.
