Integrated Cognition
AAAI Technical Report FS-13-03
What’s between KISS and KIDS:
A Keep It Knowledgeable (KIKS) Principle for Cognitive Agent Design
Benjamin D. Nye
Institute for Intelligent Systems
University of Memphis, Memphis, TN 38111
benjamin.nye@gmail.com
Abstract
Axelrod and Hamilton (1981) proposed KISS, which stated
that agents should use the minimal number of variables and
mechanisms then add new ones if needed. On the converse, the KIDS principle states that modelers should include all variables and mechanisms that appear relevant and
then remove ones that do not add to the quality of the
model (Edmonds and Moss 2005). Neither of these approaches are completely satisfactory for cognitive agents,
which must combine general cognitive mechanisms with
other task-specific factors relevant to the situation being
modeled. A cognitive parameter that might be irrelevant for
one task might be pivotal for another.
Instead, cognitive agents are commonly developed using
cognitive architectures. Cognitive architectures are usually
designed to implement specific theoretical models drawn
from cognitive science and psychology experiments. The
architecture can then be applied to model specific cognitive tasks by specifying the task environment, altering initial
states (e.g., initial information, social network ties), or specifying context-specific mechanisms (e.g., production rules
for actions). Compared to a minimal model (KISS) or a
maximal model (KIDS), this approach has the advantage
that the architecture evolves in tandem with the theoretical
model it was based on, such as ACT-R, Soar, or CLARION
(Anderson 1996; Laird 2008; Sun 2007). Even normative
(e.g., not biologically-inspired) models, typically adhere to
standard theoretical or pragmatic constraints for their design.
However, this approach suffers significant drawbacks.
First, keeping a cognitive architecture up to date with new
research is difficult and time-consuming process. Modelers work hard to stay current with literature, but each year
the number of published studies increases and journals have
steadily diverged into silos with more specialized focuses
(Silverman 2010). As a result, cognitive models run the
risk of quietly falling out of sync with new findings. Second, practices for comparing different cognitive models see
limited use. Cognitive mechanisms seldom have quantifiable units and relationships between parameters may only
be known at a correlational level (Silverman et al. 2001).
As a result, there are usually infinitely many ways to implement the same combination of mechanisms. Model docking,
where the equivalency of two models is tested for some task,
has been proposed to compare models but docking tech-
The two common design principles for agent-based models,
KISS (Keep It Simple, Stupid) and KIDS (Keep It Descriptive, Stupid) offer limited traction for developing cognitive
agents, who typically have strong ties to research findings and
established theories of cognition. A KIKS principle (Keep It
Knowledgeable, Stupid) is proposed to capture the fact that
cognitive agents are grounded in published research findings
and theory, rather than simply selecting parameters in an adhoc way. In short, KIKS suggests that modelers should not
focus on how many parameters, but should instead focus on
choosing the right research papers and implement each of
their key parameters and mechanisms. Based on this principle, a design process for creating cognitive agents based
on cognitive models is proposed. This process is centered
around steps that cognitive agent designers are already consider (e.g., literature search, validation, implementing a computational model). However, the KIKS process suggests two
differences. First, KIKS calls for reporting explicit metadata
on the empirical and theoretical relationships that an agent’s
cognitive model is intended to capture. Each such relationship should be associated with a published paper that supports
it. This metadata would serve a valuable purpose for comprehending, validating, and comparing the cognitive models used by different agents. Second, KIKS calls for validation tests to be specified before creating an agent’s cognitive
model computationally. This process, known as test-driven
design, can be used to monitor the adherence of a cognitive
agent to its underlying knowledge base as it evolves through
different versions. Implications, advantages, and limitations
of the proposed process for KIKS are discussed.
1
Introduction
Cognitive agents are powerful tools for running simulated
experiments, forecasting political outcomes, powering intelligent tutoring systems, and driving agents in virtual worlds
(Sun 2006). However, significant debate exists over the appropriate design practices for such models. In particular,
selecting the parameters and mechanisms of an agent’s cognitive model remain more of an art than a science. The
agent-based modeling community has proposed two design
principles at opposite ends of a continuum: KISS (Keep
It Simple, Stupid) and KIDS (Keep It Descriptive, Stupid).
Copyright c 2013, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
63
niques are not well-standardized and are seldom performed
(Burton 2003). Third, it is difficult to rigorously validate
which relationships that a particular cognitive model incorporates as it evolves over multiple iterations. While the underlying theories and empirical relationships are reported in
publications, these mechanisms are spread across many papers and technical reports.
The purpose of this paper is to propose a process for developing cognitive agents meant to overcome these limitations. Cognitive agents should “Keep it Knowledgeable,
Stupid” (KIKS). Unlike other domains often modeled using
agents (e.g., political science, macro-economics), cognitive
agents benefit from the knowledge provided by a wealth of
relevant, well-controlled studies. These represent an embarrassment of riches: so many experiments are conducted that
cognitive agents cannot even use all of this knowledge. The
KIKS principle changes the focus from selecting the set of
parameters (e.g., KISS vs. KIDS) to the selecting a set published papers that contain the theories and empirical findings
used to design an agent’s cognitive model. These relationships capture the real knowledge that underpins the cognitive agent. By explicitly coupling the design of a cognitive
model to this knowledge, cognitive agent design, comparison, and validation might all be significantly improved.
2
or scenario being studied. During each phase, the following
tasks are performed:
1. Select papers and other information sources
2. Identify key parameters and relationships for each paper
3. Record structured metadata on each paper’s relationships
4. Create validation tests that check for these relationships
5. Implement parameters and mechanisms into a computational model
In the first phase, the general mechanisms of a cognitive
model are derived from core cognitive theories and empirical findings drawn from a specific set of published papers.
These provide the requirements for the cognitive architecture for the model. While the definition of a “key parameter” or relationship is fuzzy at the ontological level, most
well-written papers clearly state the main effects, interactions, or relationships that were studied. These requirements
should be formally represented as metadata and associated
with their published source. Based on these requirements,
tests should be specified first and then the cognitive architecture should be created so that those tests should be satisfied by cognitive agents built using this architecture. In the
second phase, context and task-specific knowledge is added,
such as training data or studies findings relevant only to a
specific task. The order of this process mirrors that of the
core knowledge, with relevant papers broken down into a
set of task-specific requirements that can be used to specify tests. This information can then be used to set up the
starting states for the cognitive agent. These include parameter values, model structure inferred from a training data set,
mechanisms present in the agents’ environment, or cognitive
mechanisms that are too specific to the task being studied to
be included in a more general model. After both phases are
complete, the completed cognitive agent can be subjected to
the general cognitive and task-specific validation tests specified during the earlier steps.
Different cognitive models place a different level of emphasis on these phases. Cognitive models built using architectures such as ACT-R place significant emphasis on
accommodating empirically-derived human cognitive principles (Anderson 1996). By comparison, a reinforcement
learning agent for machine vision might require very limited cognitive restrictions (e.g., only those for reinforcement
learning), but might require an extensive set of training examples or a carefully specified reward function for the task.
So then, the level of effort required for each step will often
depend on the purpose of the system.
In large part, model designers already perform many of
these steps. Two primary differences of KIKS should be
noted, however. First, this process emphasizes producing
explicit metadata about the theoretical and empirical relationships that the model is intended to implement. This
metadata should be publicly available to other researchers
and used to develop validity tests. Very few cognitive architectures state a process for collecting and recording data on
the literature underlying the model. To this author’s knowledge, only the PMFServ cognitive architecture has collected
metadata specifically on its source publications, through its
KIKS: Keep it Knowledgeable
The KIKS principle is that the design of cognitive agents
should be explicitly and formally tied to their underlying
literature, rather than implicitly tied to it. The foundation
of all cognitive agents rests on published scientific papers.
Cognitive modelers recognize the importance of theoretical
grounding and coherence, taking pains to report the basis of
models in their publications. However, the process of moving from literature to a validated cognitive agent is often informal. Based on KIKS, a structured process is proposed
here to break the development cycle of cognitive agents into
distinct phases and steps.
Figure 1: Cognitive Agent Design Process Based on KIKS
This KIKS process is outlined in Figure 2. The numbers
in this diagram indicate a suggested order a two-phase design and validation process. These two phases differ by the
knowledge considered: the knowledge behind the core architecture and the knowledge required for some specific task
64
eters and mechanisms can be added at a later time based on
additional literature.
Compared to KISS or KIDS, KIKS also offers the opportunity to improve the research ecosystem for developing cognitive agents. The following sections consider three
areas where KIKS could improve upon existing modeling
practices. First, the implications for KIKS on the goals of
cognitive agent modeling are briefly explored. Second, the
value of metadata for published papers is discussed. Last,
the benefits for validation and model comparison are noted.
anthology of human behavioral models (Silverman et al.
2001). However, this anthology was a collection of short reports, rather than formal, machine-readable metadata. With
that said, the KIKS approach is significantly inspired by
PMFServ’s process for breaking seminal papers down into
key effects and constructing cognitive mechanisms tied to
that literature (Silverman et al. 2006). However, KIKS is
intended to be architecture-neutral, which will be discussed
more in later sections.
The second major deviation from standard cognitive agent
design is that the specification of the validation tests occurs
before the computational implementation of the cognitive
model or agent. This approach is inspired by test-driven
design, a software design pattern where requirements and
validation tests are built first, then the system is built to conform to those standards. While test-driven design is uncommon in software design, it is very well-suited for cognitive
modeling. Fundamentally, a computational cognitive model
is built based on two things: 1. relationships derived from
well-established, peer-reviewed research and 2. the modeler’s best-guess at how to resolve ambiguities, gaps, and
underspecified mechanisms between this research. Clearly,
the former must be trusted far more than the latter. A testdriven design process prioritizes the established cognitive
mechanisms as constraints for model development, representing how the model should work after it is designed. It
also puts an up-front focus on what it means for the cognitive agent properly implement its underlying theory and
relationships.
Moreover, this process can be used to consider the implications of additional papers, which can be broken down into
requirements and used to develop tests to determine if (or
when) the agent’s cognitive model conforms to these findings. This is an explicit form of the scholarship often performed with cognitive theories, where their implications are
compared against new empirical studies. Cognitive models,
much like cognitive theories, do not need to agree with every study. However, it is very important to understand which
findings agree or disagree with a given cognitive model, as
these establish boundary conditions and drive scientific discussion.
Finally, compared to KISS or KIDS, this process greatly
simplifies parameter and mechanism selection. Under
KIKS, a cognitive agent should employ the parameters presented in the publications used to develop the model. For
each paper used to support the model, the designer should
identify the key parameters studied, empirically-derived relationships between them, and functional mechanisms proposed by theory. These parameters and their relationships
should be explicitly represented as metadata before starting
development of the computational model. So then, instead
of choosing parameters or mechanisms, the model designer
is actually choosing published references and explicitly deriving the model from them. In this way, literature search directly drives model design. Other parameters or mechanisms
might also be added to the model (e.g., for efficiency or taskspecific needs), so long as the model still behaves properly
and fulfills its specified requirements. For example, as part
of the two-phase development process, task-specific param-
3
The Objective of a Cognitive Agent
The KIKS process also has some implications over the objective of cognitive agent design. Cognitive agents serve
three main roles: an explicit representation of existing
knowledge, a predictive tool to discover new knowledge
(e.g., through simulation), and an artificial intelligence to
power human-like applications (e.g., intelligent tutors, virtual agents). The first role, as a representation, acts as a constraint on design: the cognitive model should embody prior
findings and well-established theorized relationships. The
second role, as a predictive tool, acts as an objective of the
design: to be able to forecast real-life outcomes. The third
role, as an AI, acts as an alternative objective: to be able to
emulate or replace a real human agent. Both of the later two
roles are objectives tied to a particular task, such as generating the distribution of human reaction time on a digit recall
experiment (a predictive objective) or passing a Turing test
(an AI objective). In both cases, the objective is connected
to the task for the cognitive agent.
So then, it is possible to consider a cognitive agent in
terms of an optimization problem where the cognitive model
for an agent is selected to optimize or satisfice some taskspecific objective function (FT ), but is subject to the con−→
straints (CC ) provided by the research findings and theory
that it was based on. The objective function is some measure of performance on one or more tasks, which establish
the domain the cognitive agent is intended to handle. For
example, the objective of a cognitive agent for digit recall
tasks might be to minimize the difference between the distribution of recall times for the model versus an empirical
distribution collected from human subjects. While an some
individual cognitive models only focus on a single task, cognitive architectures are often intended for broader scope of
tasks or even a unified theory of cognition (Newell 1994).
−→
The constraints (CC ) are the cognitive requirements that
fundamentally make an agent’s cognitive model represent
the cognition of a human or other agent, as opposed to a
general artificial intelligence for performing certain tasks.
While the objective function asks, “Is the model getting it
right,” the constraints check, “Does it get it right for the
right reasons?” In theory, a purely normative model does
not need any cognitive constraints: if only needs to perform well on its tasks. However, in practice, even machine
learning models designed often use variants of biologicallyinspired mechanisms such as reinforcement learning (Kaelbling, Littman, and Moore 1996). Also, normative assumptions are also typically rooted in published theories (e.g.,
65
rational actors). Overall, to design an effective cognitive
model, some constraints on the model are almost always required to keep the system theoretically coherent and computationally tractable. A second set of constraints also exists
−→
for the specific task (CT ). These constraints might be environmental rules or boundaries of the task, for example.
The design of a cognitive agent selects the parameters of
−→
the cognitive model (XC ), state transition functions (e.g.,
update rules, a transition matrix) that relate these parameters
(PC ), and the distribution of initial states for the cognitive
−→
model (SC ). Likewise, the task environment must be de−→
signed, with its own parameters (XT ), state transition func−
→
−
→
tions (PT ), and distribution of initial states (ST ). For the
task state and cognitive state to interact, a subset of state
transition functions must use parameters from both. Transitions that change cognitive state based on task state are often
→
−
called observations ( O ), while those that consider cognitive
state to change the task environment are often called actions
→
−
( A ). For this discussion, we will assume that a modeler primarily has control over the design of the cognitive model
−→ −→
(i.e., XC , SC , PC ), which corresponds with the first phase
of the KIKS process. With that said, the dynamics due to
observations and actions are also sometimes considered part
of the cognitive model (which would make them “free variables” for modeling also). Assume that the intention of the
modeler is to minimize the difference between the cognitive
agent task performance and some objective, as well as to sat−→
−→
isfy a set of boolean constraints (CC and CT ). This produces
an optimization problem in the form shown in Equation 3.
minimize
−
−
→ −→
XC ,SC ,PC
subject to
ter DataShop for educational data mining (Koedinger et al.
2011).
Secondly, as an optimization problem, it is possible to
consider the potential for constraint relaxations, such as variants of Lagrangian multipliers (Everett III 1963). By relaxing constraints (e.g., replacing them with weighted penalties
on the objective function), the formulation supports quantifying the tradeoffs of adhering to specific literature findings
versus performing optimally on a specific task. Moreover,
not all theories or empirical findings are equally trusted or
required in a system. As such, tying penalties to confidence
in research findings and theories offers one way to capture
these intuitions. Alternate representations, such as Bayesian
formulations, could also be used to unify constraints and objectives into a weighted objective function.
As a grounding example, assume a cognitive agent that
wanders a maze looking for rewards with two cognitive constraints: novelty-seeking (orient attention toward less familiar stimuli) and motivated attention (orient toward rewards).
Since these constraints would be based on empirical studies,
they might be recorded such as “The correlation between attention and novelty was 0.7, with a 95% confidence interval
of ±0.2” (within a particular study). One way to interpret
this constraint would be to require the cognitive agent to
maintain a correlation of 0.7±0.2 for at least 95% of runs
through the maze, rejecting non-conforming agents as invalid. A similar constraint could be constructed for noveltyseeking. By adding more constraints, the space of valid cognitive agents shrinks and created agents should better approximate the subjects from the underlying literature.
For certain agents, this would be an ideal approach (e.g.,
cognitive agents that represent rats, with constraints based
on studies of rats in mazes). However, the all-or-nothing
approach gives constraints effectively infinite importance:
no valid agent can violate them. For some agents, such
constraints may impose unsatisfactory impediments to task
performance (e.g., gathering rewards). Alternatively, certain constraints may be less relevant to the cognitive agents’
contexts (e.g., findings from laboratory studies used to build
a cognitive agent for more naturalistic settings). In these
cases, relaxed constraints that penalize the objective function for violations might be more appropriate. The relative weight given to the original objective function versus
the total weight for relaxed constraints would determine the
importance of task performance versus fidelity to literature.
Similarly, differential weights for constraints would capture
a relative importance for satisfying each constraint. While
more work is required to explore best-practices for building
agents that directly incorporate findings from studies, formalizing the purpose of a cognitive agent implies that existing optimization and machine learning algorithms can be
applied to the problem.
Finally, since this formulation focuses entirely on representing and testing against published literature, it is applicable to any cognitive agent design. Two major dimensions of
cognitive models for agents are their scope and their theoretical coherence. The scope of a cognitive model determines
its generality. Some models are intended to be a universal model of human cognition (general), while others only
−→ −→ −→ −
→ →
− →
−
FT (XC , XT , SC , ST , O , A , PC , PT )
−→ −→
ai (XC , SC , PC ) = 0, ai ∈ CC
−→ −→ −→ −
→ →
−
bj (XC , XT , SC , ST , O ,
→
−
A , PC , PT ) = 0, bj ∈ CT
KIKS posits a cognitive agent design, regardless of its
implementation details, is implicitly searching for a satisfactory solution to a problem in this form. In this view,
failed validation tests correspond to constraint violations and
model performance corresponds with the objective function.
This implies two corollaries. First, it is theoretically possible to design agents that explicitly optimize over this problem. This means that an architecture could be developed that
generates cognitive models based only on metadata about
published papers and a training data set for modeling performance of certain tasks. While such a system would almost certainly overfit a single task, one that was trained on a
variety of related domain tasks might produce a useful cognitive agent while reducing authoring effort. Such a system is one way to combine knowledge from well-controlled
experiments with big data, which is becoming increasingly
important due to social network data, crowd-sourced experiments using systems such as Mechanical Turk (Rand 2012),
and initiatives such as Pittsburgh Science of Learning Cen-
66
model a single interaction from an experiment (specific).
Framing KIKS as trying to optimize over an arbitrary group
of tasks (T ) captures this concept of scope. Some systems,
such as machine vision, treat this set of tasks very explicitly and have canonical databases for training and testing. In
other systems, the scope of design may only be in the modeler’s head and recorded in their published works.
Theoretical coherence refers to the neat versus scruffy
continuum (Minsky 1991). Neat models, such as dynamical systems, rely on a small number of mechanisms whose
implementation can often be tested using proofs. Scruffy
models use a larger breadth of literature and mechanisms,
whose implementation must usually be tested using validation tests on the completed system. Obviously, both dimensions lie on a continuum and hybrids are common, such as a
neat, generic core cognitive model supplemented by scruffy,
task-specific mechanisms. While neat and scruffy models
may use different techniques to validate their mechanisms
(i.e., implement the functions in CC and CT ) and measure
their performance (FT ), any two models based on the same
requirements (e.g., theory, empirical findings) that have the
same scope of tasks (T ) are effectively trying to solve the
same problem. As such, it should be possible to base either
type of design on the same metadata that describes these
requirements and objectives. This implies that the metadata that describes the specifications for a cognitive model
may have significant value for the cognitive modeling community, as it could be shared by different architectures approaching similar problems. The importance of such metadata is described in the next section.
4
onomies for cognitive science are currently an active area of
research. In neuroimaging (e.g., fMRI studies), the BrainMap taxonomy describes two decades of key components in
published results and the more recent Cognitive Paradigm
Ontology was developed to structure this into a domain ontology for describing neuroimaging experiments and results
(Turner and Laird 2011). The Cognitive Atlas is a similar project that contains an ontology for cognitive science
specifically, with over 400 tasks and 600 concepts currently
represented (Poldrack et al. 2011). Finally, the Cognitive
Modeling Repository is a recent project dedicated to storing implementations of cognitive models (Myung and Pitt
2010). While this project does not have an explicit ontology
for parameters, it does require structured metadata reporting
the input and output parameters of simple cognitive models.
There is undoubtably overlap of the concepts represented in
these projects with those used in the cognitive models used
by cognitive agents. As such, they might provide an excellent starting ontology for metadata about key relationships
presented in journals.
Journals have also increasingly recognized the value
of metadata. Structured abstracts are the most pervasive
method for structured data about publications, with significant use in medicine, education, biochemistry, and other
domains. A structured abstract breaks an abstract down into
required fields (e.g., Background, Objectives, Methods, Results, and Conclusions). Structured abstracts not only improve the ease of selecting relevant articles (Nakayama et
al. 2005), but there have been proposals for journals to hook
these into digital summaries and ontologies to provide semantic search capabilities (Gerstein, Seringhaus, and Fields
2007). Some biology researchers have even directly textmined publications to find patterns, though this approach is
likely limited to domains where the objects of study (e.g.,
protein sequences) have clear, unique names (Altman et al.
2008). With that said, the majority of biology metadata has
been created by human curators and offers an example that
could be imitated in cognitive modeling. KIKS would benefit from structured metadata for new journal publications, so
moves by relevant journals in this direction would strengthen
this approach. Reciprocally, KIKS would generate metadata about existing publications when they are used to design cognitive agents.
Obviously, this short discussion only scratches the surface
of the challenge of general metadata for cognitive research.
A database of metadata on findings and theorized relationships would have long-term value for cognitive science and
psychology as a whole. This section is primarily meant to
show how KIKS could contribute to and benefit from such
a repository. However, even in absence of a shared metadata format and repository, ad-hoc metadata that describes
how a cognitive model connects with literature would be a
valuable resource for understanding and validating a single
project. These issues are addressed in the following section.
Knowledge Base on Cognition
The first three steps of a KIKS phase (Select Papers, Identify
Relationships, and Record Metadata) focus entirely on distilling the knowledge present in publications. Since the findings from publications are constant, formal representations
of these relationships could be shared by different cognitive
architectures. If this metadata were stored in a shared repository, a structured knowledge base of cognitive science findings and theoretical principles would slowly emerge. The
value of such a knowledge base cannot be understated. The
value of even being able to search for papers based on their
findings would be enormous and might eventually facilitate real-time meta-analysis capabilities. However, for such
metadata to be useful, a workable domain ontology of cognitive concepts, experiments, and results would need to be
available.
Moreover, building this knowledge base does not necessarily need to be the modeler’s responsibility. Academic
journals and authors of scientific papers should have an
interest in making their findings easy queried and betterintegrated with other findings. Even putting aside cognitive
agents, the argument can be made that this type of work is
vital to managing the flood of scientific papers and findings
published each year. For this to be sustainable over the long
term, journals need to encourage structured data about published findings and theories.
Over the last twenty years, steps have already been made
in this direction by related fields. Ontologies and tax-
5
Validation and Comparison
By specifying requirements before designing a cognitive
agent, KIKS makes the validation process more principled.
67
KIKS suggests that authors should represent their underlying theoretical and empirically-derived relationships before implementing a cognitive model, then test the cognitive
agent against these requirements. This allows for test-driven
design, where tests are actually written before the implementation of the computational model. This approach can
be done iteratively, where additional literature findings provide new constraints that the cognitive model should satisfy.
Unlike the specific cognitive model implementation, these
constraints do not need to be specified at the level required
to create a computational model. Instead, they only need to
be specified at a level that allows a test to validate that the
constraint holds. For example, studies on motivated attention have found that being hungry correlates positively with
additional attention to videos or pictures of food (RoskosEwoldsen and Fazio 1992). Implementing this in a computational model requires filling in underspecified aspects of this
relationship (e.g., shape and parameters of the distribution
function of the effect of hunger on attention). By comparison, a validation test can be based closely on the empirical
tests conducted (e.g., a correlation test between those variables can be evaluated using a simple simulation).
Moreover, since the tests have already been created, they
can always be run on later iterations of the cognitive agent
to ensure that new relationships do not violate existing ones.
If such violations occur, either some problem exists with
the cognitive model implementation or, more interestingly,
it might imply that two findings or theories from literature
are incompatible for some reason (e.g., only valid for different sets of as-yet-unknown boundary conditions). Similar violations might occur when applying a model to a new
task. For example, the task dynamics might drive the cognitive model to behave in a way that violates its specifications
derived from literature. In either case, tests based on theory
and empirical findings provide a “canary” in the system to
detect such events quickly. This allows a modeler to correct their implementation or to look further into why certain
research implies conflicting outcomes.
The existence of metadata requirements for different cognitive agents also enables a limited form of model comparison. The gold standard for model comparison is probably model docking, where two models are directly applied
to the same task and their behavior is compared (Burton
2003). However, except for well-formed (e.g., neat) models that allow direct inferences of equivalence, this process
can be time-consuming and often requires a full experimental methodology. By comparison, comparing the metadata
about requirements is much simpler. Assuming that two different sets of metadata can be represented in the same format, a number of comparisons can be made between their
resulting models. First, the intersections and differences between the parameter sets of each model can be compared
(e.g., which concepts do both models study?). Provided
these can be expressed using a common ontology, the relative parameter sets would tell a great deal about the focus of
each model. Likewise, requirement sets could be compared
(e.g., how much do these models employ the same underlying assumptions about cognition?). If metadata about the
scope of tasks for each model was also available, this ap-
proach would allow comparing if two cognitive agents are
actually two different ways to approach the same problem.
Finally, as requirements are intended to be tied to papers,
this approach could be used to compare the distance between
literature behind different models (e.g., are they using different papers on the same theory or do they come from completely non-intersecting theoretical backgrounds?).
While these comparisons cannot replace the empirical
comparison between model behavior offered by model
docking, they could be performed easily across any two cognitive agents that have metadata about their requirements.
As such, they could detect pairs of models that could be
meaningfully docked for further study, improve searching
for cognitive models (e.g., “find similar models” functionality), and even cluster models by the similarity between
their parameters or literature. While these capabilities would
not replace the power of model docking, they would provide
simple and efficient comparison functionality that could be
used for semantic search, to compare different versions of
the same model, or to perform high-level comparisons between the properties of different cognitive models.
6
Limitations and Future Directions
For designing cognitive agents and their cognitive models, traditional paradigms such as KISS (Keep it Simple,
Stupid) and KIDS (Keep it Descriptive, Stupid) fail to capture the importance and traction offered by explicitly rooting design into the underlying literature. The KIKS (Keep It
Knowledgeable, Stupid) approach is proposed as a process
for designing cognitive agents that outlines a structure for
the modeling behaviors already used for modeling cognitive
agents: selecting relevant literature, identify key parameters
and relationships, implementing a computational cognitive
model, and validating that model. KIKS is not proposed as
a revolution to existing design practices for cognitive agents
and cognitive models. To the contrary, much of the KIKS
process merely outlines steps that many modelers already
consider. The primary differences between this approach
and existing modeling techniques currently in practice is that
it explicitly calls for producing metadata about the key parameters and their relationships from literature, then using
those to specify validation tests before actually making the
model.
These differences have benefits, as well as costs. The
primary cost of KIKS is the time invested to record metadata. With that said, the value of such metadata for literature search and meta-analysis purposes alone should offset
that investment. Additionally, there are already movements
in related research communities (e.g., neuroimaging) toward
similar metadata that would dovetail with this work. While
journals have only moved in this direction slowly, fields like
biology offer examples for how research can benefit from the
“big data” implicit in published literature and its reported
findings (Altman et al. 2008). Overall, the potential benefits for such metadata are considerable and worthwhile even
without KIKS.
The second cost is that KIKS asks modelers to specify
their validation tests prior to coding the cognitive agent. This
68
discovered through careful scientific exploration.
Issues also exist for streamlining validation tests for cognitive agents, though these problems are not specific to
KIKS. In general, there has been significant debate over best
practices for validating computational and agent-based models (Cooley and Solano 2011). Cognitive agents, for example, often incorporate relationships derived from statistical
tests based on findings from many subjects. So then, should
the a distribution of cognitive agent “subjects” be tested
for similar distribution properties? Alternatively, what if
the cognitive model always displays this relationship across
all conditions (e.g., hunger always leads to increased attention to food)? Is this a good model because it captures the
main relationship from literature, or is it a bad model because it lacks the noise empirically observed across subjects? These are fundamental philosophical questions about
validating the mechanisms of cognitive models that do not,
as yet, have cut-and-dry solutions. However, these should
not dissuade modelers from constructing validation tests. If
anything, they call for additional attention to validity-testing
and the implications of what is actually being tested.
Finally, the KIKS principle is entirely silent on best
practices for selecting appropriate literature for designing a
model and on the process of choosing how to implement
the model computationally. These gaps are intentional, as
the literature review and computational implementation are
both deeply connected to the domain expertise of research
groups. Despite these limitations and boundaries, KIKS
offers basic principles and a structured process for designing cognitive agents. While there is room for improvement,
KIKS focuses design specifically on the cognitive model and
the knowledge behind it, rather than viewing the design process in terms of a parameter search. This represents a significant improvement over other principles commonly invoked
for agent-based modeling. Additionally, KIKS accommodates the issue that cognitive agents require internal cognitive modeling as well as task-specific modeling related to
their intended application (e.g., simulated experiments, virtual agents, etc.). As such, the KIKS principle can hopefully
contribute to the larger discussion of how and why cognitive
agents are designed.
does not necessarily mean that the tests must be fully implemented, but their expected functional or statistical relationships should at least be outlined. Due to project scheduling
and delivery dates, it may not always be possible to follow a
test-driven design process. However, it is very much in line
with the scientific method: one would not design an experiment prior to establishing criteria for the success or failure of
that experiment. Why then would the design of a cognitive
agent, which might be used in many simulation experiments,
precede formulating tests to ensure the model works as intended? Worse, after a cognitive agent is finished, it is very
easy for rigorous validation to lose out to new applications or
extensions to the agent. Particularly for a predictive model,
leaving validation to the end of a design cycle has dangerous
implications for the quality of the model and the research it
produces. Given that cognitive agent modelers should be
developing validation measures at some point, KIKS suggests specifying these tests first rather than after modelers
are deeply invested in a particular implementation.
This paper is not intended to be the last word on KIKS.
Instead, KIKS is tentatively proposed as an alternative to the
KISS and KIDS principles, which do not offer guidelines
to harness the knowledge behind cognitive agents. Cognitive agent design, which relies heavily on cognitive model
design, has unique challenges and opportunities that KIKS
attempts to accommodate. However, given the breadth of
research on cognitive agents, KIKS should be significantly
improved by further discussion and refinements of the suggested process. The discussion of how to tighten the links
between cognitive models and published knowledge may ultimately be more important than the specific proposed design process.
While a high-level process has been presented, the pragmatics of KIKS also require further specification. For example, a simple and general metadata format for representing
theoretical and empirical relationships is needed to promote
sharing of such metadata. Likewise, while the results of
standard statistical tests (e.g., location tests) are relatively
easy to represent, advanced hypothesis tests (e.g., cluster
analysis) are not as standardized in their reports. Translating
these into metadata might offer challenges. Theorized relationships also offer challenges for formal markup, as these
might require representing functions or algorithms. While
methods exist for representing these relationships (e.g., symbolic algebra languages, logical languages such as Prolog),
cognitive modelers would only consider representing these
relationships if they could do so intuitively and with minimal effort. With that said, many cognitive models primarily
consider knowledge derived from classical statistical tests,
so these issues may be less critical in practice. Secondly, the
question must be asked: why focus on metadata at all? Why
not focus on raw experimental data instead of research findings or theories? Particularly with moves toward big data
in social science, this is an important question. However,
this author would argue that the value added by cognitive researchers in conducting, framing, and interpreting their analysis is pivotal. While there may also be merit in using raw
experimental observations to help design a cognitive agent,
it is doubtful that raw data could replace the relationships
Acknowledgements
Thank you to the Office of Naval Research, whose support for intelligent systems helps make this work possible.
Also, many thanks to my past and present colleagues at both
the University of Pennsylvania and the University of Memphis, whose vigorous scientific discussions helped set these
thoughts in motion.
References
Altman, R. B.; Bergman, C. M.; Blake, J.; Blaschke, C.;
Cohen, A.; Gannon, F.; Grivell, L.; Hahn, U.; Hersh, W.;
Hirschman, L.; Jensen, L. J.; Krallinger, M.; Mons, B.;
O’Donoghue, S. I.; Peitsch, M. C.; Rebholz-Schuhmann,
D.; Shatkay, H.; and Valencia, A. 2008. Text mining for
biology–the way forward: opinions from leading scientists.
Genome Biology 9 Suppl 2:S7.
69
Anderson, J. R. 1996. Act: A simple theory of complex
cognition. American Psychologist 51(4):355–365.
Axelrod, R., and Hamilton, W. D. 1981. The evolution of
cooperation. Science 211(4489):1390–1396.
Burton, R. M. 2003. Computational laboratories for organization science: Questions, validity and docking. Computational and Mathematical Organization Theory 9(2):91–108.
Cooley, P., and Solano, E. 2011. Agent-based model (ABM)
validation considerations. In The Third International Conference on Advances in System Simulation (SIMUL 2011),
126–131.
Edmonds, B., and Moss, S. 2005. From KISS to KIDS – an
antisimplistic modelling approach. In Davidsson, P.; Logan,
B.; and Takadama, K., eds., Multi-Agent and Multi-AgentBased Simulation, 130–144. Springer.
Everett III, H. 1963. Generalized Lagrange multiplier
method for solving problems of optimum allocation of resources. Operations Research 11(3):399–417.
Gerstein, M.; Seringhaus, M.; and Fields, S. 2007. Structured digital abstract makes text mining easy. Nature
447(7141):142.
Kaelbling, L. P.; Littman, M. L.; and Moore, A. W. 1996.
Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4:237–285.
Koedinger, K. R.; Baker, R. S. J.; Cunningham, K.;
Skogsholm, A.; Leber, B.; and Stamper, J. 2011. A data
repository for the EDM community: The PSLC datashop.
In Romero, C.; Ventura, S.; Pechenizkiy, M.; and Baker,
R. S., eds., Handbook of Educational Data Mining. Taylor
and Francis. 43–55.
Laird, J. E. 2008. Extending the Soar cognitive architecture.
In Proceedings of the 2008 conference on Artificial General
Intelligence, 224–235. Amsterdam, Netherlands: IOS Press.
Minsky, M. L. 1991. Logical versus analogical or symbolic
versus connectionist or neat versus scruffy. AI Magazine
12(2):34–51.
Myung, J. I., and Pitt, M. A. 2010. Cognitive modeling
repository. cmr.osu.edu (Retrieved: May 20, 2013).
Nakayama, T.; Hirai, N.; Yamazaki, S.; and Naito, M. 2005.
Adoption of structured abstracts by general medical journals
and format for a structured abstract. Journal of the Medical
Library Association 93(2):237–42.
Newell, A. 1994. Unified theories of cognition. Cambridge,
MA: Harvard University Press.
Poldrack, R. A.; Kittur, A.; Kalar, D.; Miller, E.; Seppa,
C.; Gil, Y.; Parker, D. S.; Sabb, F. W.; and Bilder, R. M.
2011. The cognitive atlas: toward a knowledge foundation
for cognitive neuroscience. Frontiers in Neuroinformatics
5(17).
Rand, D. G. 2012. The promise of Mechanical Turk: how
online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology 299:172–179.
Roskos-Ewoldsen, D. R., and Fazio, R. H. 1992. On the
orienting value of attitudes: Attitude accessibility as a deter-
minant of an object’s attraction of visual attention. Journal
of Personality and Social Psychology 63(2):198–211.
Silverman, B. G.; Might, R.; Dubois, R.; Shin, H.; Johns,
M.; and Weaver, R. 2001. Toward a human behavior models
anthology for synthetic agent development. In 10th Conference on Computer Generated Forces and Behavioral Representation, 277–285.
Silverman, B. G.; Johns, M.; Cornwell, J. B.; and O’Brien,
K. 2006. Human behavior models for agents in simulators
and games: Part I: Enabling science with PMFserv. Presence: Teleoperators and Virtual Environments 15(2):139–
162.
Silverman, B. G. 2010. Systems social science: A design inquiry approach for stabilization and reconstruction of social
systems. Intelligent Decision Technologies 4(1):51–74.
Sun, R. 2006. Cognition and Multi-Agent Interaction: From
Cognitive Modeling to Social Simulation. Cambridge University Press.
Sun, R. 2007. Cognitive social simulation incorporating
cognitive architectures. IEEE Intelligent Systems 22(5):33–
39.
Turner, J. A., and Laird, A. R. 2011. The cognitive
paradigm ontology: Design and application. Neuroinformatics 10(1):57–66.
70