Augmented Reality for Guitar Learning

Introduction

In the past years, online or CD-ROM books tend to replace the traditional paper based music books. E-learning has been on the focus of an enormous amount of research, both from academic and research communities. However, just a few experimental approaches have been developed trying to enable a complex computer interaction through the physical use of a musical instrument and digital information for the sake of learning. Here, an experimental AR system capable of simulating and superimposing 3D audio and 3D visual information is presented. The aim of this work is to teach the basics of electric guitar in a more efficient way than the traditional methods through the use of a prototype AR interface toolkit. Learners are not required to have previous musical experience although they must be familiar with a computing environment.

Method of Operation

The effectiveness of any technological self-learning system depends on both high quality video rendering and accurate 3D restitution of the sound space. The architecture of the application is based on the prototype AR interface toolkit which is capable of rendering in real time performance audio-visual information in indoor environments. Based on the robust integration of the above, an experimental application focused on guitar learning is designed.

A USB web-camera captures the real environment and sends the video stream into a computer. When one of the predefined marker cards is detected then the position and orientation of the camera is recovered. Graphics and sound are processed into the processing unit and meaningful audio-visual information is added to the real environment. The audio output is send to the speakers while the audio output to a standard display monitor. To operate the system, the learner is equipped with a set of marker cards and a guitar (which can be classic, acoustic, electric, etc). To learn a specific chord, the learner just positions the correspondent marker card on the environment. If the camera detects the marker, the application overlays a static image indicating how the fingers must be placed in the guitar. In addition, a pre-recorded sound of the chord is automatically played. The learner can naturally interact with the augmented information by manipulating the marker cards in 3D space. To practice in a different chord, the learner has simply to remove this card and replace it with the corresponding card. Other interactions within the system are performed similarly to the methods presented in the AR Interface Toolkit. Users can manipulate the virtual information either using standard I/O devices, using the interface menu, or by the natural manipulation of the generated marker cards.

Theoretical and Practical Scenarios

The purpose of this learning scenario is to provide multimedia information using a tangible AR interface. Learners can use this framework to combine theory and practice at the same time. The theoretical parts are presented through textual and pictorial augmentation while the practical is based on auditory augmentation. The practice session starts with a brief overview of chord theory using representative marker cards as the main interaction tool. The user selects a chord that wants to practice and places the correspondent marker card on the table so that the camera has the marker on the viewing frustum. An example of how the learner can understand where must place his/her fingers to play the ‘A' chord is shown below.

To obtain more information, as well as practice the theory, the learner can get an interactive audio-visual augmentation. Tablature information can be augmented in conjunction with descriptive textual information which presents three different ways of playing the chord. In addition, a pre-recorded sound file that corresponds to a chord can be inserted into the position of the marker. The sound file is played on a continuous loop with a delay of three seconds between intervals. When the learner is satisfied with the practice then another marker card can be placed on the table and continue the practice session.

Learning Complete Songs

Another learning visualization scenario is to help the learners to play easy but complete songs in any type of guitar. The minimum requirement of playing a song in a guitar includes some basic chords and the corresponding lyrics. Similarly to the previous tutorials, the appropriate markers have been designed and they are used to simultaneously provide textual information for the lyrics and the musical representation of a song. A multi-marker which consists of six smaller markers is employed for the chords while a bigger marker for the textual and auditory augmentation as illustrated below.

The user can read the lyrics (of a ‘Bod Dylan's' song called ‘Knocking on heavens door') and the corresponding chords from the monitor and try to improvise the song in the guitar. To aid the learning process more, the song is also inserted in the marker and automatically played in a loop in the same way as previously explained. The same process can be then repeated for other pre-recorded songs. The major difference between the previous tutorials is that seven markers have been used simultaneously. Although this reduces the efficiency of the operation of the system it is still a very nice tool which can be used to create unique learning combinations in a number of ways depending on the application demands.

• Liarokapis, F., Augmented Reality Scenarios for Guitar Learning, Proc. of Theory and Practice of Computer Graphics 2005, Eurographics UK Chapter, University of Kent, Canterbury, UK, 15th-17th June, 163-170, (2005). ISBN: 3-905673-56-8. Pdf.

Videos

The following videos illustrate part of the capabilities of the work performed in this application domain.