US007982122B2

(12) United States Patent

Gatzsche et al.
(54)

US 7,982,122 B2

(10) Patent No.:

(45) Date 0f Patent :
(56)

DEVICE AND METHOD FOR ANALYZING AN

Jul. 19, 2011

References Cited

AUDIO DATUM
Us. PATENT DGCUMFNTS

UMEhmOD2Daee@Enr0e)
nmYE
DeaiaWhe0.r5
( 372555oo6$20o0723561

2da1(nlgC

Oum.r3a(.0\elS e

e.m,uPT7e
.0S.l
2,
.DUANIGUFP1SQS(m
1.110N0lO\D..u.)1O@D
3T)
m1
2eC_.P0Ss
.fPc
r09b1/eO
C24C
0N
.P
Fte

WA.di.
\heh
n.1WM)02aeCu

Cenmn
ddba.wCJ07era0

m9,nmm06m.1
m
wrm
.n
.mm
ym)
2,
GMM
.ueu2_ N.pvyPuahS690unadaM
n0Pc
l.e,mh
m0.786cd1w<
u7kwam

nmiSgbaerC2.

sNomm)n,QPEFGswDUO@B2QM/ 3m%.wC._O~no>mME:mAJPES1UOE

s
..w
F
.C
3
m
6@a
d...

2o1Tn,F@rM0(.mupgeswacEz
Pp2705.tr1a0e4/.nn52r1ulng

Od).C44.

FMmnsmof/wMOfum/.Fs/h

m(\rZnDt0ayuhS

M
a\)0rdy.fal.Al6.11F1t(:\C9J

m.wmW4,50hcAgn OdZ2H590_P1a.67Ub MhwmBnWFmNdvweoUe8G3)9617?4,md mmsmoGih5nmNhca.56uE3mw2'h0Jl49?MARadmmsahP.(\cn,A0( mmTaBN.MDTmUH0JDH.wT@KmMCuwDUCedMom@emww%/Tt0>nsebLm)wodmnami>.dgmMeIAmeu.nbmhcw

6.1ua.1u000t.1ea

e3.

mm)mnw)u CeVJ1y4M0t3O.56na Ban.wRhmAc8dmmwWwmomcifAomr m0nsP./1ermBPT78\mfU9SOd1m9bCu@

Mmm,Ldn)k0S

auSwmf,6\1a

S0l.H4fEl7O.

0pmErde,C102r9l35,1b0

%P@oDA\dwUm \vm

t.1

,R03S5l

Mm7%m13f.a. l|

N4

chord variant

.tmmmSU01Eh8462.n15r6W7 n

M
hn?
LneoTiornnvdnf mmO.HdGrdnmVOSA/G.OZSg.flw

MdNmytOhummin ns M2uS1T:eeuO.Hd
oOcmahtwQmS.lauCLOma c&aSEm fda1vreuNm.wt,eBmo m.nm7SOhCumthLa amn81aA

,m.iSuawnvdmmAu.seftdmWpGLvuPeHompLim1mtVP1c.wa _rse
ihmaw n .

US 7,982,122 B2
Page 2
U.S. PATENT DOCUMENTS
1/1998
5,709,552 A
3/1998
5,723,802 A
7/1998
5,777,248 A
7,356,465 B2* 4/2008
2001/0052283 A1
12/2001
2002/0178896 A1
12/2002
2003/0209130 A1
11/2003
2004/0255759 A1
12/2004
2005/0114121 A1
5/2005
2006/0027080 A1
2/2006

LeGrange

Dorrell: Vector Analysis of Musical Intervals, What is Music:

Johnson et al.

Chapter 5: XP-002430508: 2004: http://vveb.archive.org/web/

Campbell

A1
U1
A1
A1
U1
U1
A1
A1
A1
U1
A1
U1
A1
A
A
A
A
A1
A1
A1
A1
A1
A1

20051124063808/http://vvhatismusic.info/download.html>:pp.

Tsingos et al. .............. .. 704/220

87-106.

Boyer
George

Gatzsche: Visualization of Musical Parameters in Music Theory,

Dissertation: Liszt School of Music Weimar: 2004: pp. 1-19.
Riemann: Ideas for a Study on the Imagination of Tone,Journal of

Anderson et al.

Gayama
Tsingos et al.
Schultz

FOREIGN PATENT DOCUMENTS

37 44 255
89 02 959.3
40 02 361
42 16 349
295 12 911
298 01 154
198 59 303
198 31 409
100 42 300
203 01 012
103 51 817
20 2005 009 551
1 533 786
2 312 085
2 324 642
11-126063
2004-286917
86/06193
90/07771
90/09650
96/36034
2005/122133
2006/005567

OTHER PUBLICATIONS

7/1989
8/1989
8/1991
11/1993
11/1995
6/1998
7/1999
1/2000
3/2002
4/2003
5/2005
3/2006
5/2005
10/1997
10/1998
5/1999
10/2004
10/1986
7/1990
8/1990
11/1996
12/2005
1/2006

Music Theory: vol. 36, No. 1: Spring 1992: pp. 81-117.

Shepard: Geometrical ApproXimations to the Structure of Musical
Pitch, Psychological Review: vol. 89, No. 4: Jul. 1982: pp. 305-333.
Gatzsche et al.: Device and Method for Generating a Note Signal
and Device and Method for Outputting an Output Signal Indicating a
Pitch Class,: U.S. Appl. No. 12/096,934, Iled Jun. 11, 2008.
Oflcial Communication issued in International Patent Application
No. PCT/EP2007/000560, mailed on May 11, 2007.
Izmirli: An Algorithm for Audio Key Finding, MireX 2005 Contest

Results: XP-002426658: http://vvvvvv.music-ir.org/evaluation/mireX

results/articles/key?audio/izmirli.pdfg 2005, 3 pages.
Gomez et al.: Tonality Visualization of Polyphonic Audio, ICMC

2005 Proceedings: XP-00246659: http://vvvvvv.iua.upf.es/~gomez/

TonalDescription/GomezBonada-ICMC2005.pdfg Sep. 5, 2005, 4
pages.

Chuan et al.: Polyphonic Audio Key Finding Using the Spiral Array
CEG Algorithm, Multimedia and EXpo, 2005: XP010843224: Jul. 6,
2005, pp. 21-24.
Bello et al.: A Robust Mid-Level Representation for Harmonic

Content in Music Signals, ISMIR 2005: XP-002426660: Sep. 2005,

pp. 304-31 1.

Purvvins et al.: Correspondence Analysis forVisualizing Interplay of

Pitch Class, Key, and Composer, Perspectives in Mathematical
Music Theory: XP-002426661: 2003, pp. 1-23.
Oflcial Communication issued in corresponding Korean Patent
Application No. Oct. 2008-7020716, mailed on Dec. 22, 2010.

* cited by examiner

U.S. Patent

Ju1.19,2o11

sheet 1 0f 16

Us 7,982,122 B2

F|GURE1

_volume 15m 15(3_2

intermediate vectors 1552

informatlon

1 55-1

semitones

analysis signal
(r. a)

FIGURE 2

U.S. Patent

Ju1.19,2o11

sheet 2 0116

100
1Y
.
1
.

FIGURE 3A

200

FIGURE 3B

Us 7,982,122 B2

U.S. Patent

Jul. 19, 2011

sheer 3 0f16

/210

US 7,982,122 B2

195

F205

FIGURE 3C

FIGURE 3D

U.S. Patent

Ju1.19,2o11

sheet 4 0116

Us 7,982,122 B2

FIGURE 3E

m#A:

Gn"
FIGURE 4A
30Gb

Bx

Fn"
FIGURE 4B

FIGURE 4C

Ax

C:

U.S. Patent

Jul. 19, 2011

sheer 5 0f16

US 7,982,122 B2

FIGURE 4D

FIGURE 4E

FIGURE 5A
inten

sity

310-C S10-E

FIGURE 5B

spatlal
position

U.S. Patent

Jul. 19, 2011

sheer 6 0f16

US 7,982,122 B2

32o-D

,B

@/5

.f

FIGURE 6A

32o-70'IEe/32o-A
Q i D/E
"

I(

FIGURE 6B
32U~G

Q/ ,B

32C-A

5 AJ

FIGURE 6C

CD

cu

FIGURE 6D

U.S. Patent

Jul. 19, 2011

390

sheer 7 0f16

370

US 7,982,122 B2

tonal center
minor third 380
S50-G

major third 385

semitone step 376
symmetry axis

symmetry pitch
35o-D

FIGURE 7

FIGURE 8

FIGUREQ

O-B

U.S. Patent

Jul. 19,2011

sheer 8 0f16

US 7,982,122 B2

geometrical representation
t a key at the position
of its symmetry axis

B
Q
E

number ot signatures
ot the respective key

_4

type oi signature

+ corresponds to sharp and

- corresponds to flat

FIGURE ii
405

flatkeys

Shafnkeys
adjacent key like, eg.,
G major and F-sharp major
Same key like,
eg., A minor and A major

opposite key like,

eg. C major and

F sharp maior

FIGURE I2

U.S. Patent

Jul. 19, 2011

sheer 9 0f16

FIGURE I3

US 7,982,122 B2

U.S. Patent

Jul. 19, 2011

sheer 10 0f16

FIGURE 15

US 7,982,122 B2

U.S. Patent

Jul. 19, 2011

_ 1,5

>

sheet 11 0f16

US 7,982,122 B2

~ /

1
A

E# 1

l1

05
H11/'00.0
0
f1
-05 Vf ww
I

4,5
-1 M

_2

` dit' I0

0 1y

time

FIGURE 16

4,05\ I.

4,00

3,00

moasvfynetcdgmuheolry

2,00
1,00

//

0,00 B/H dimi) D_minor' F-majorWC-major Eminor G-major'B/H dimi

_100 nlshed
/
nlshed

~2,00

-3,00
4,00

/
chord

FIGURE I7

U.S. Patent

Jul. 19,2011

sheet 12 0f16

US 7,982,122 B2

470
/

vaotthebsolute msvyoedcmutolry

p @N_mcn
C)

\`\

\\

prime' maior'minor' dimi-fourthI titth minorl maior

third third nished
titth

seoondseoond

interval

FIGURE 18

otthevabsolute msvyoecdmuteolry

p@
CD

480

prime major minor dirni- tourth iourth maior minor

third third nished
fifth
interval

FIGURE I9

secondsecond

U.S. Patent

Jul. 19,2011

sheer 13 0f16

US 7,982,122 B2

495

21n/1|._

tm

\ / \ n

Roo,
m,
n
m,

@@5E5Q2E3%m _5E25mE;

|OQ/..\
\

/ 490

@Qo @woo@o-

.o

chord variant

FIGURE 20

M,E_E2@5mNams@
W,

major

major
fifth

.r mihorseoond mirtor third fourth

E5?@EE@2g2:52wm 2as@5E53m2s@
4oo,\ 4510

5'oo

frequency ot the

bottom pitch
constant

FIGURE 21

5:50

600

frequency of the

top pitch

U.S. Patent

Jul. 19, 2011

sheet 14 0f16

US 7,982,122 B2

FIGURE 22

starting angle

FIGURE 23

U.S. Patent

Ju1.19,2o11

sheet 15 0116

b) numpad.

Us 7,982,122 B2

c) arrow keys

d) rotary-

730-1

hQ 730-3 a
atG

d h()
730-4

|10

730-2

FIGURE 24

b) fader
750

c) pitch number keys

-

FIGURE 25

/760 1 /760 3
3

U.S. Patent

Jul. 19, 2011

sheer 16 0f16

US 7,982,122 B2

,a2
f

825

mf 830

FIGURE 26

1 0
10106

1010

1020

auto

Signal semitone
.

analys's

1030 10303

cal ulation _* <

me

pich classes

Syggeglry 1 display i
|

circie ofthirds i

FIGURE 27

1040

..... ___/_

devce

US 7,982,122 B2
1

DEVICE AND METHOD FOR ANALYZING AN

AUDIO DATUM

devices coupled to the number of note selectors and to the

memory means for providing a note turn-off signal triggered

BACKGROUND OF THE INVENTION

by the note deselection signal which includes the note-desig

nating information stored when providing the note selection

signal.
The present invention relates to a device and a method for

The patent DE 4216349 C2 describes an electronic musical

analyzing an audio datum, inparticular to a device which may

be used, for example, in connection with a display device, an
accompaniment device or another evaluation device, for
example to enable a faster and simpler determination of a key
of the key change, a chord or a chord change.

instrument having a melody and an accompaniment key

board. The musical instrument described has a melody key
board whose melody keys include switches including two
switching stages, wherein those pitches corresponding to the
white keys are associated with the first switching stages and
those pitches corresponding to the black keys of a keyboard

When making music, but also when otherwise dealing with

a piece of music or an existing sequence of chords, an analysis

are associated with the second switching stages, and an

ofthe existing or sounding piece of music is required in many

situations, for example to enable improvising on the existing

accompaniment keyboard comprising accompaniment keys

which, when operated, may call an automatic chord accom

piece of music, i.e. creatively generating harmonically and

paniment, wherein the accompaniment keys are respectively

consonantly sounding melodies, or accompanying the exist

implemented as switches having at least two switching stages

which have different associated accompaniment chords. An
operation ofthe described electronic musical instrument does

ing piece of music, i.e. creating a sequence of chords and/or a

sequence of single tones which go with the melody and tend
to underline the same.

20

not request the knowledge of musical notation, but requires,

This frequently requires of a person a minimum measure of

due to the described modeling according to a fingerboard, an

experience in dealing with music, which may frequently only

operator who is educated in music theory, as in particular

certain combinations of individual pitches and chords, which

be learned by several years of working with music and/or a

musical instrument. ln addition to that, a corresponding
analysis frequently requires of a person a certain musical
talent, which may request partially even absolute hearing in
the case of very complex pieces of music. This, however,

25

may operate manually to generate an accompaniment chord.

excludes many people who lack the required background

knowledge of music theory, sufficient experience in dealing
with music and/ or a musical instrument, or the corresponding

are needed in particular for pedagogical purposes, are obvi

ous. ln particular, the document describes a musical instru
ment with a one-finger accompaniment system, which a user

30

talent.

The patent DE 2857808 C3 describes an electronic musical

instrument combined with an electronic clock. The invention
relates to an electronic musical instrument, wherein via input

and storage means any pitch sequences and pieces of music

ln literature, many teaching aids and means for learning

may be input and retrieved again. The described electronic

and/ or finding chords, harmonies and keys are known. These

are often templates, discs or other objects, in particular
mechanically connected, shiftable or rotatable templates on
which connections regarding music theory are illustrated.
Such learning aids and means are, for example, described in
the following documents DE 8005260 U1, DE 8902959 U1,
DE 3744255 A1, U.S. Pat. No. 5,709,552, DE 3690188 T1,
US 2002/0178896 A1, DE 4002361 A1, DE 19831409 A1,
DE 19859303 A1, DE 29801154 U1 and DE 20301012 U1. ln

musical instrument thus enables only an input with a subse

quent storage of a pitch sequence and a reproduction of the
stored pitch sequence via a pitch generator circuit to repro
duce the stored sequence of pitches in the form of a sequential

35

40

requires a certain minimum of theoretical musical knowl

edge, as otherwise a programming of the musical instrument

general, on one of the discs or the corresponding objects a

sequence of pitches is applied which in general either corre

sponds to the chromatic scale consisting of a sequence of
twelve semitones and thus all available pitches of an equal

will hardly be realizable.

45

The European patent EP 0834167 B1 refers to a virtual

musical instrument with a new input device. ln particular, the
above-mentioned patent application refers to a virtual musi
cal instrument having a portable accessory of a type which is
to be brought in contact with a musical instrument in order to

50

play this instrument, wherein the mentioned portable acces

temperament, or to the circle of fifths, wherein a pitch interval

of two adjacent pitches is a fifth (for example C-G or F-C). DE

8005260 shows a device for finding chords, harmonies and
keys with an arrangement in an interval of a third.

The utility model DE 29512911 U1 describes a teaching

and learning aid for a synthesis and analysis of connections
regarding music theory with several different templates and at

sory comprises a switch which generates an activation signal

as a reaction to a person holding the mentioned portable

least twelve gaming pieces provided with designations of

accessory causing the mentioned portable accessory to hit

another obj ect. The mentioned activation signal is received by

pitches.
The European patent EP 0452347 B1 refers to a universal

acoustic presentation. lt is in particular disadvantageous with

regard to the musical instrument described, that the input
and/or the programming of the pitch sequence takes place
via a 10-key pad, extended by several additional keys. ln
particular, the electronic musical instrument described also

55

a digital processor, which in turn generates a control signal

operating unit for an electronic musical instrument compris

ing a number of note selectors, each of which provides a note

which causes a synthesizer to generate a note which is repre

selection signal when a note is selected and a note deselection

signal with a deminishment of a note, note turn-on devices

patent application describes a virtual musical instrument,

wherein the mentioned portable accessory is a guitar plec

coupled to the number of note selectors for providing note

designating information associated with each note selector
and for providing a note turn-on signal triggered by the note

sented by a selected note data structure. ln particular, the

60

a predetermined amount of pitches sound via the synthesizer.

selection signal which includes the corresponding note-des

ignating information, a memory means for storing the note

designating information provided as triggered by the note

selection signal, means coupled to the note turn-on device for

changing the note-designating information and note tum-off

trum and wherein a user may only make pitches from within

65

The European patent EP 0632427 B1 relates to a method

and a device for inputting musical data. More specifically, the
mentioned patent relates to a musical data input device
including an input recording means for recording a hand
written input on it, a position detection means for detecting a
position on the input recording means where the hand-written

US 7,982,122 B2
3

input is performed to obtain pitch data representative of a

the audio datum with regard to a volume information distri

pitch of a musical note, an input detection means for detecting

bution over an amount of semitones; and a vector calculator,

which is implemented to calculate a sum vector over two
dimensional intermediate vectors for each semitone or each

the hand-written input performed on the input recording

means, wherein the input detection means comprises a means

element of the definition amount and to output an analysis

signal based on the sum vector, based on the volume infor

for detecting the number of pushing events performed on the

input recording means or for detecting a time period in which
the input recording means is pushed, or for detecting the
intensity of pressure which is exerted on the input recording
means during the hand-written input, or comprises a number

mation distribution or a distribution derived from the volume

detection means to detect a number written onto the input

recording means, or a line detection means to detect the

niment device, which is coupled to the device and imple

mented to receive the analysis signal and provide a note signal
based on the analysis signal.

information distribution, which comprises a definition

amount based on the amount of semitones; and an accompa

length of a line which is drawn onto the input recording

According to another embodiment, a measurement system

may have a device for analyzing an audio datum which may

means, a time designation means for designating time data

representative ofthe length of a musical pitch, on the basis of
the detected number of pushing events or the detected time
period or the detected intensity of pushing events or the
detected number or the detected length of a line detected by

have a semitone analyzer, which is implemented to analyze

the audio datum with regard to a volume information distri
bution over an amount of semitones; and a vector calculator,
which is implemented to calculate a sum vector over two
dimensional intermediate vectors for each semitone or each

the input detection device, and a musical pitch generation

means for detecting musical pitch data on the basis of pitch
level data obtained from the position detection means and the
time data obtained from the time designation means. ln par
ticular, the mentioned patent application describes a musical

mation distribution or a distribution derived from the volume

data input device having an LCD unit (LCDliquid crystal

information distribution, which comprises a definition

display) and a touch pad arranged on the same, via which,

with the help of a pen, pitches may be inserted into a pitch
system. The described musical data input device thus relates
to people having a sufficiently high knowledge of connec

element of the definition amount and to output an analysis

signal based on the sum vector, based on the volume infor

amount based on the amount of semitones; and a display

25

signal and implemented to provide an output signal indicating

an angle of the sum vector based on the output signal.

tions regarding music theory.

F[he patent application U.S. Pat. No. 5,415,071 relates to a
method and a device for generating relationships between
musical pitches. Here, an arrangement of offset lines or rows

device which is coupled to the device to receive the analysis

According to another embodiment, a detection system may

have an integrator, which is implemented to integrate a time
30

dependent audio input signal regarding time and provide the

same as an audio datum; a device for analyzing an audio

of symbols is described, wherein each symbol represents a

datum, which may have a semitone analyzer, which is imple

musical note. Each line includes a repeating series of twelve

symbols which forms a musical series of semitones which is
also known as the chromatic scale. Here, each line is offset

mented to analyze the audio datum with regard to a volume

information distribution over an amount of semitones; and a
35

vector calculator, which is implemented to calculate a sum

vector over two-dimensional intermediate vectors for each
semitone or each element of the definition amount and to
output an analysis signal based on the sum vector, based on
the volume information distribution or a distribution derived

40

from the volume information distribution, which comprises a

with regard to the adjacent lines so that groups of symbols

which represent the same musical relationship, i.e., for
example, intervals, scales, chords, etc., form the same visu

ally recognizable configurations, like, for example, diagonal

configurations or vertical configurations at certain locations

definition amount based on the amount of semitones, which is

in the arrangement. ln one embodiment, such a device which

includes such an arrangement may be used as a learning aid,

coupled to the integrator and provides the analysis signal; and

wherein the learning aid comprises two overlapping compo

an evaluation device, which is coupled to the device and is

implemented to analyze a time course of a length of the sum

nents which may be shifted against one another. Apart from

that, the patent application describes an arrangement of the

45

of the length of the sum vector comprises a maximum or a

contact area of a keyboard and/or a claviature of a musical

instrument with a claviature or a fingerboard of a musical

string instrument which are arranged in accordance with the

arrangement. The patent application thus describes a clavia
ture having keys arranged in the form of concentric circles.

vector based on the analysis signal and, when the time course

minimum, output a detection signal.

According to another embodiment, a key determination
system may have a device as mentioned above; and a key
50

determinator, which is coupled to the device and is imple

mented to generate a key signal indicating a key based on the
analysis signal of the device and provide the same at an

SUMMARY

output.
55

According to another embodiment, a method for analyzing

an audio datum may have the steps of analyzing the audio

vector calculator, which is implemented to calculate a sum

vector over two-dimensional intermediate vectors for each
semitone or each element of the definition amount and to 60
output an analysis signal based on the sum vector, based on
the volume information distribution or a distribution derived

datum with regard to a volume information distribution over

an amount of semitones; calculating a two-dimensional inter
mediate vector based on the volume information distribution
or a distribution derived from the volume information distri
bution, which comprises a definition amount based on the
amount of semitones, for each semitone or each element of
the definition amount; calculating a sum vector based on the

According to an embodiment, a device for analyzing an

audio datum may have a semitone analyzer, which is imple

mented to analyze the audio datum with regard to a volume
information distribution over an amount of semitones; and a

from the volume information distribution, which comprises a

definition amount based on the amount of semitones.

According to another embodiment, an accompaniment

system, may have a device for analyzing an audio datum,

having a semitone analyzer, which is implemented to analyze

65

two-dimensional intermediate vectors; and outputting an

analysis signal which is based on the sum vector.
According to another embodiment, a computer program
may have a program code for performing the method for
analyzing an audio datum, which may have the steps of ana