Episodic Memory and

Cognitive Capabilities

Andrew Nuxoll
24 May 2007
Outline

Review and Introduction

 Definitions
 Research Goals
 TankSoar Domain

Demonstrating Cognitive Capabilities
 Action Modeling
 Virtual Sensors
 Learning from Past Success and Failure

2
Long Term Memory
Memory

Long Term Short Term

Memory Memory

Declarative Procedural
Memory Memory

Semantic
Memory
Episodic
Memory

Episodic Memory
 Memories of
specific events in
your past
3
 Current Implementation
Long-term Procedural Memory
Encoding Production Rules
Initiation?

Storage

Retrieval
Output Working Memory Cue

Input Retrieved

When the agent takes an action.

4
 Current Implementation
Long-term Procedural Memory
Encoding Production Rules
Initiation
Content?
Storage

Retrieval
Output Working Memory Cue

Input Retrieved

A portion of working memory is stored in the episode

5
 Current Implementation
Long-term Procedural Memory Episodic
Encoding Production Rules Memory
Initiation
Content
Storage
Episode Structure?
Retrieval
Output Working Memory Cue

Episodic
Learning
Input Retrieved

Episodes are stored in a separate memory

6
 Current Implementation
Long-term Procedural Memory Episodic
Encoding Production Rules Memory
Initiation
Content
Storage
Episode Structure
Retrieval
Initiation/Cue? Output Working Memory Cue

Episodic
Learning
Input Retrieved

Cue is placed in an architecture specific buffer.

7
 Current Implementation
Long-term Procedural Memory Episodic
Encoding Production Rules Memory
Initiation
Content
Storage
Episode Structure
Retrieval
Initiation/Cue Output Working Memory Cue

Retrieval
Episodic
Learning
Input Retrieved

The closest partial match is retrieved.

8
Research Goals

Explore the cognitive capabilities granted to

an agent with an episodic memory

Explore what’s necessary to build an effective
episodic memory for a general cognitive
architecture
 Domain independence
 Performance

Take inspiration from cognitive psychology

9
Cognitive Capabilities:
How do we use Episodic Memory?

Sensing
Learning
 Detecting Repetition  Retroactive Learning
 Virtual Sensing  Reanalyzing with new
 Noticing Unusual Input Knowledge
 Sense of Identity  Explaining Behavior

Reasoning  “Boosting” other Learning
Mechanisms
 Action Modeling
 Recording Previous
Successes/Failures
 Modeling the Environment
 Managing Long Term
Goals

10
TankSoar Domain

Tanks in a maze

Sub-goals
 Shoot other tanks
 Don’t get shot
 Don’t run out of

Energy

Missiles

Multiple sensors
and actions

11
Action Modeling
Cognitive Capability: Action Modeling

Action Modeling
 Definition: Learning the
immediate effect of an action
 Analog: Case-Based
Reasoning

Task: Conserve Energy
 Selecting proper radar setting
to minimize energy
consumption

13
Agent Implementation

Radar Setting: 1
Radar Setting: 2 Evaluation Result:
X=11 … Radar Setting: 5
Y=3
Dir=south
Radar Setting: max
Agent
Confirms
Agent’s te y Location
a
State re r and
C mo e
e Direction
M Cu
Match

Episodic
Retrieval
X=11 X=11
Y=3 Y=3
Dir=south Dir=south

14
Initial Performance
Radar Tank Performance

1
0.9
0.8
Fraction Correct

0.7
0.6
0.5
0.4
0.3
0.2
0.1 Random
0
1 101 201 301 401 501 601 701 801 901
Radar Settings (time)

15
Nuggets Coal

Episodic memory is an
Requires agent have

effective medium for knowledge of whether a
action modeling retrieved episode is
useful

16
Virtual Sensors
Cognitive Capability: Virtual Sensors

Virtual Sensors


 Definition: Retrieving past
?
sensing that is relevant to the
current task

Task: Locate the Battery
 Using episodic memories to
construct a path

18
Agent Implementation

Move North
Evaluation Result:
Move West
Move North
Move East

a te y
Agent’s e r
Cr mo e
State e
M Cu

Episodic
Retrieval

19
Performance
Energy Search Time

1000
Number of Moves

100
Random
Ep. Mem.
10

1
1 2 3 4 5 6 7 8 9 10 11 12
Searches

20
Paths to Battery

21
Nuggets Coal

Episodic memory can
Limited investigation of

be used as a virtual integration of episodic
sensor and semantic memory

22
Learning from Past Success
and Failure
Cognitive Capability: Learning from Past
Success and Failure

Learning from Past
Success and Failure
 Definition: Using past
performance to guide future
behavior
 More emphasis on long term

Task: Combat
 Using episodic memory to
determine best tactics in the
“attack” subgoal

24
Agent Implementation
Move Forward
Move Forward + Fire Missile Evaluation Result:
… Move Forward + Fire = 2 points
Turn Right
Compare
Memories
Agent’s te In the Sequence
State r ea Cue
C ry
emo
M

Retrieve the Retrieve the

Episodic Next Memory Next Memory
in Sequence
… in Sequence
Retrieval

25
Performance

20

10

0
1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161
Margin of Victory

-10

-20

-30

-40

-50

-60
Successive Games

26
Tactics Learned

Fight or flight

Back away and shoot

Dodging

27
Without Heuristic Cue
Performance without Heuristic Cue Selection

20

10
Average Margin of Victory

0
1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161
-10

-20

-30

-40

-50

-60
Successive Games

28
Without Discount Factor
No Discount Factor

20

10
Average Margin of Victory

0
1 12 23 34 45 56 67 78 89 100 111 122 133 144 155 166 177
-10

-20

-30

-40

-50

-60
Successive Games

29
Nuggets Coal

Episodic memory can
Requires the use of a

be effective at learning discount factor
long term tactics
Requires heuristic cue
selection

30
Summary

Episodic memory enables multiple cognitive

capabilities including:
 Action modeling
 Virtual sensing
 Learning from past success and failure

31