I recently had to prepare a statement of research for tenure at Indiana University. This is the statement, and the purpose of the statement explains the self-congratulatory tone. Please ignore the tone. I am posting the description here so that the world will know about the types of questions being explored in our lab.

Research Statement

My research goal is to expand our understanding of perception, cognition, and their interactions. Traditionally, work in human perception has been disconnected from research on more sophisticated cognitive functioning. One of the first distinctions that an undergraduate psychology student learns is between "Low-level," simple, "merely" perceptual processes, and "high-level," "true" cognition. This distinction is echoed by philosophers who differentiate sense data from cognitive inferences about sense data. My research is based on the premise that this distinction is misleading and counterproductive. Perception is far more sophisticated than usually thought, raising it up to the level of "high-level" cognition. Reciprocally, "high-level" cognition is fundamentally grounded in our perceptual abilities. Even when we strive for disembodied, symbolic abstraction, our cognitive processes retain their connection to perception. My research is an attempt to build bridges between our perceptual and conceptual systems.

There have been a number of researchers who have argued that our environment is structured by our cognitive processes. Sapir and Whorf argued for systematic influences of language and culture on perceptual processes. The "New Look" school of psychology in the '50s and '60s argued that the motivation, knowledge, and experiences of an observer shape perception. Like these researchers, I am interested in how people learn new visual concepts, and what influences these newly learned concepts have on people's behavior. However, these researchers provided few formal theories for the mechanisms of conceptual influences on perception.

In studying this general question, I have developed a two-branched research program. The first branch is empirical research on concept learning and judgment in human subjects. The second branch is computational modeling and simulation of the human behavior observed in the laboratory. My goal is to develop theories of human concept learning that are precise and rigorous enough to be instantiated by computer models. The simulations that I develop are self-organizing. They involve many interacting simple units. Individually, none of the units is particularly important for the system's performance. However, when they are interacting together, they develop specializations, hierarchies, and permanent memories. Interestingly, global structure in the simulations can be obtained even without a standard recipe-like computer program. Global harmony can be created simply by specifying rules for how individual units locally interact. The simulations begin with very little specific knowledge, but spontaneously learn to organize their world into categories. After the simulation has been exposed to items from different visual categories, it can successfully categorize new items. Much of this concept learning involves perceptual change. In order to successfully categorize inputs, the simulation develops specialized perceptual "detectors." For example, if the simulation is presented several examples of "E" and "F," the computer will develop a "lower horizontal line" detector in order to distinguish the two letters.

The success of my simulations is evaluated by measuring how closely they correspond to human behavior. As such, my research programs are typically cyclic. First, I consider an issue in human cognition (e.g. "How are new concepts learned?"). Second, I design laboratory experiments that address the issue. Third, the data from the experiments are used to build a formal theory. Often, these theories are expressed as formalized, computer simulations. Fourth, the specific theory is then tested by further experimentation on human subjects, which in turn often leads to further refinement of the simulation.

Perceptual Similarity

In exploring links between "high-level" cognition (such as categorization, decision making, problem solving) and perception, I became interested in the nature of perceptual similarity. The question of "what makes two things similar" is of fundamental importance to cognitive psychology; William James calls similarity "the very keel and backbone of our cognition." Similarity seems to play a role in categorization (if A is similarto members of Category B, then A will be categorized as a B), memory (if A is similar to B, then A will often remind people of B), problem solving (if problem A is similar to B, then solving A will typically help solving B), and decision making (if A is similar to B, then people will often incorrectly infer that A causes B).

On the other hand, there is a recent, widespread (supported by Murphy, Rips, Keil, Carey, Gelman, Markman, and Atrans among others) theory that argues that high-level cognition is often not related to "simple" perceptual similarity. Instead, our high-level cognition depends on sophisticated "theories" that we have about the world.

My research suggests that perceptual similarity is more related to general cognition that would be suggested by this recent body of research. Perceptual similarity is quite sophisticated, and vestiges of perceptual similarity can be found in higher-level cognition.

Sophisticated Similarity

One of the reasons why similarity is able to provide the underpinnings for many cognitive processes is that perception itself is sophisticated and flexible. My early research explored the classes of properties that influence people's impressions of perceptual similarity. People's impressions of similarity are not simply based on superficial physical attributes, but are also influenced by abstract relations. For example, in many scenarios, XX is judged to be more similar to YY than it is to XY. People are sensitive to the abstract "both letters are identical" relation that is shared by XX and YY, even when they are making strictly physical judgments (Goldstone, Gentner, & Medin, 1989; Medin, Goldstone, & Gentner, 1990). Subsequent work described many factors that influence whether people base similarities on superficial physical properties or more abstract, relational aspects (Goldstone, Medin, & Gentner, 1991; Medin, Goldstone, & Gentner, 1993). This work is potentially useful for instruction and education. Many times, in order to properly deal with objects (e.g. algebra problems, paintings, diseases), it is important to ignore superficial, misleading properties, and concentrate on deeper aspects. In many cases, these deeper aspects involve relations between several aspects of the object rather than a simple feature of the object. We have found several factors (related to the overall similarity of objects, the type of judgment required, and the manner of presenting the objects) that can help people to notice relational similarities between objects.

Later work, which led to my dissertation in 1991, argued for a second way in which perceptual similarity is sophisticated. My empirical research and computational modeling efforts have indicated that when people evaluate similarity they carry out a process not unlike analogical reasoning (e.g., in research by Gentner, Hofstadter, and Holyoak). Namely, the parts from one object are placed in alignment or correspondence with the parts from the other object, and these correspondences mutually affect each other. If two scenes contain some matching property, then their similarity will be increased by some amount. To determine how much similarity will be increased, it is necessary to know how well aligned the scene parts are that have the matching property. For example, in comparing baseball to cricket, the property white is shared by the uniforms of cricket players and baseballs. However, because most people would place cricket balls into correspondence with baseballs rather than baseball players, the shared white property does not increase similarity much. The discovery that alignment, rather than sheer overlap between properties, determines similarity is important because virtually all traditional models of similarity have no way to account for this influence, and because it allies perceptual similarity judgments with conceptual comparisons such as metaphors, similes, and analogies.

People's similarity assessments are well captured by neural network model SIAM (Similarity as Interactive Activation and Mapping) that I have developed. The model shows how large-scale, well-structured visual interpretations can emerge even though very simple processing units only communicate with each other locally and without supervision. The model takes as input a description of two scenes, and produces as output an overall similarity estimate, and a set of alignments that shows how the various scene parts are related to each other. These similarity estimates are highly correlated with those produced by people in a variety of circumstances. The original theory of alignment-based similarity was described by Goldstone (1994c). Predictions of the computational model were empirically tested by Goldstone (1992), Goldstone and Medin (1994a) and Goldstone (in press-c). An overview of the model and its empirical support was presented in Goldstone and Medin (1994b). General rationales for alignment-based approaches to similarity were presented in Goldstone (1994b) and Medin, Goldstone, and Gentner (1993).

A third and final line of evidence that similarity is "smart" enough to be useful in explaining cognition concerns evidence for context-sensitive similarity. It is well known that our categories are context-sensitive -- an axe is generally a tool, but may belong to the category weapon in certain contexts. My colleagues and I have similarly found evidence that similarity is context- sensitive in surprising ways. This context sensitivity leads to violations of the assumptions of many traditional models of similarity, such as monotoniticy (if the same feature is added to two objects, their similarity should never decrease) and transitivity (if A is more similar to T than is B, and B is more similar to T than is C, then A should be more similar to T than is C). We find violations of these assumptions because the similarity of objects depends not simply on their individual attributes, but on the other objects simultaneously being compared, and even on contexts that are spontaneously created when the objects are displayed. Experiments showing the context-sensitivity of similarity are described in Goldstone, Medin, and Gentner (1991), Goldstone, Medin, and Halberstadt (in preparation), Medin, Goldstone, and Gentner (1993), and Medin and Goldstone (1995). The context-sensitivity of similarity is compared to other judgment context effects in Medin, Goldstone, and Markman (1995). The context effects in similarity are important because they provide persuasive evidence that similarity is constructed for particular purposes and circumstances rather than memorized. Objects do not have fixed similarities that people simply uncover; it would be more accurate (though less grammatical) to say that "people similaritize objects."

From Similarity to Cognition

Perceptual similarity may be a useful notion in understanding high-level cognition because it is sophisticated even though it is "merely" perceptual, but the converse also appears to be true. Similarity is useful because high-level cognition is not always very high-level. In many cases, categorization and decision making are not based on abstract rules. Instead, perceptual similarity intrudes on these judgments, even when it is inappropriate or irrelevant. Goldstone (1994b) presents a large review of literature suggesting the mandatory assessment of perceptual similarity during categorization. Empirically, the connection between categorization and similarity is explored by Kroska and Goldstone (in press), using emotion categories such as "anger," "fear," and "joy." Goldstone (in press-b) reviews evidence from my laboratory and others that indicates strong links between similarity and categorization. Medin, Goldstone, and Markman (1995) delineate relations between similarity and decision making. Medin, Goldstone, and Gentner (1993) consider relations between similarity and other comparison processes. Taken in total, this research argues that similarity plays a role even in highly symbolic processes because it is flexible enough to provide the groundwork for many cognitive processes, yet constrained enough to provide non-circular explanations for these cognitive processes.

This theme of perceptually-based cognition has been explored in a number of studies showing parallels between cognitive judgments and more perceptual judgments. The "confirmation bias" in judgment refers to people's tendency to selectively look for evidence that confirms, rather than disconfirms, their hypotheses. Goldstone (1993a) found a perceptual equivalent to the confirmation bias in a task where subjects simply estimated the number of black dots in a display. One speculation from the study is that several traditional judgment biases may actually have a perceptual or attentional basis (cf. Medin, Goldstone, & Markman, 1995). One mundane but pragmatic application of perceptual processes underlying judgment biases concerns the systematic under-reporting of copies by patrons of a communal copy machine (Goldstone & Chin, 1993). Likewise, Levine, Halberstadt, and Goldstone (in press) find evidence supporting a perceptual/attentional account of another judgment bias. Psychologists have recently found evidence suggesting that people often make better decisions when they do not reason about or justify their judgments. Our experiments indicate that one reason why people can be more optimal decision-makers when they are not reasoning is that they use more "holistic" or global perceptual processing strategies than when they are forced to reason.

Finally, Goldstone (1994d) describes a new method for collecting similarity judgments that takes advantage of psychologically natural perceptual and spatial constraints. While most researchers collect similarity assessments by simply asking subjects to provide similarity ratings, my new technique involves spatially rearranging items on a computer screen to reflect their similarity. Pragmatically, the technique collects data more efficiently than ratings, and appears to be more intuitive. Theoretically, the technique is interesting because its high concordance with existing measures suggests that purely symbolic similarity judgments have many of the same mathematical (topological and metric) constraints that perceptual distances do. It is literally true that to be similar is to be close in (psychological) space.

Concept Learning and its Perceptual Consequences

A full model of concept learning remains one of the most sought-after prizes in cognitive psychology. It is not hard to adopt a perspective whereby almost all cognition boils down to learning how to partition objects into useful groups. Concepts allow us to treat different objects equivalently, communicate, draw inferences, reason, and explain our world. To recognize a handwritten character as a B is to categorize into the class of B letters. To remember an article is to categorize the article as belonging to the class of familiar things. To make a clinical diagnosis is to place the patient in a disease category. The excitement and promise of studying concept learning stems from its astoundingly wide sphere of application. As such, I consider my main line of research to be one of exploring human concept learning, and building computational models of this process.

Conceptual Influences on Perception

In the previous section I described how our concepts depend on our perceptual abilities. Much of my research argues for a simultaneous, reciprocal influence of our concepts on our perception. This research stands in contrast to one of the great foundational ideas of cognitive science: that cognition involves operations on a fixed set of hard-wired perceptual features. The fixed set of features serve as the building blocks for representing objects. It's a great idea because of its parsimony - a huge variety of objects can be represented by combining a small number of existing features in different arrangements. This fixed-features approach has been popular in linguistics (the phoneme /f/ is represented by the features bilabial, fricative, consonantal in the work of Jacobsen), vision (suitcase is represented by a curved cylinder attached to a rectangular prism in the work of Biederman), and event representation (ordering food in a restaurant is represented using concepts such as ingest, propel, and physical transferÊin the work of Schank). In these of these fixed-feature approaches, a finite number of features (9 phonemic features, 36 geometric solids, and 23 primitive concepts respectively) represents all of the objects within a domain.

In my alternative to these fixed-features theories, I explore the possibility that the vocabulary of features that people use changes with their tasks and experiences. The categories that people need to learn seems to alter their lower-level featural descriptions. Certainly in many domains, experts (radiologists, wine tasters, chicken sorters, chess masters, and fishers) seem to develop specialized perceptual tools for analyzing the stimuli in their domain of expertise. I am exploring the nature of this perceptual learning due to categorization under more controlled laboratory conditions.

In early work on this topic David Aha and I found experimental evidence that people develop sensitivity to specific regions of a dimension that are useful for learning a categorization (Aha & Goldstone, 1990). Previous researchers had assumed that entire dimensions (such as size, brightness, location, or orientation) were sensitized. Our results indicated a surprising degree of flexibility in attention. Later, we developed a computational model of category learning that incorporated this flexible allocation of attention. This model was able to learn categories that could not be learned by several other computer models of categorization but could be learned by human subjects (Aha & Goldstone, 1992).

This work eventually led to a line of research on how category learning alters relatively low-level perceptual processes. In the experiments reported by Goldstone (1994a), subjects first conducted one of several trained categorizations. After this, they participated in a same/different judgment ("are these two squares physically identical?") involving dimensions that were either relevant or irrelevant during categorization training. I found that the categorizations that the subjects learned in the first phase of the experiment affected their ability to make the strictly physical judgments in the second phase.

Goldstone and Pevtzow (submitted; Pevtzow & Goldstone, 1994) have extended this logic to situations where categories are defined by a whole configuration of elements rather than simple dimensions such as brightness or size. In these two papers, we found evidence that the way in which an object is broken down into parts depends on how relevant those parts have been for previous categorizations. Natural ways of perceiving an object can be abandoned if less natural parsings involve parts that have been useful for categorization. A related approach was taken by Goldstone (in press-a), to show that people automatically categorize objects into groups (based on shapes), and these categories cause systematic distortions to peoples' perception of color. Even though the subjects' task is completely based on color ("Match the color of these two objects"), subjects are still influenced by "irrelevant" category-level information that is contained in the stimulus set.

I have also begun, in collaboration with overseas researchers, to work on theoretical aspects of the need to develop novel perceptual features. In Schyns, Goldstone, and Thibaut (submitted) we argue that the basic features used in representing objects are as flexibility tuned to environmental contingencies as concepts themselves are. Goldstone and Schyns (1994) also develop these arguments. These ideas led Schyns and I to co-organize a symposium entitled "Learning new features of representation" at the sixteenth annual conference of the Cognitive Science Society in 1994. Over 60 people attended the symposium.

Representation of Concepts

The previously cited research suggests some constraints on what our concepts must involve. Concepts must be at least partially based on perceptual inputs, and also must be capable of changing perception. What exactly are concepts? Are they photograph-like images, rules, nodes connected to other concepts, lists of instances, or dictionary entries? In my early work, my colleagues and I grappled with this general question of how are concepts represented (Goldstone & Kruschke, 1994; Matheus, Rendall, Medin, & Goldstone, 1989; Medin et al, 1990; Medin & Goldstone, 1991). I have published reviews of other researchers' work on this question (Goldstone, 1993c, 1994e, 1995), including one review that appeared in Science. In later work, I have begun to make progress on specific proposals for computationally representing concepts.

One proposal, still in its infancy, contends that there are two very different ideas about what concepts are like in cognitive science. Linguists, conceptual network researchers, and artificial intelligence researchers often stress the interrelations between concepts - concepts are defined in terms of their connections to other concepts in something like a semantic network. On the other hand, most work in concept learning has stressed relatively isolated representations of concepts. I have developed a neural network model that incorporates both isolated and interrelated aspects of concepts, and am currently empirically testing this model. Empirically, I have developed a number of indicators of the isolatedness/interrelatedness of a concept, and have also developed a number of manipulations that can alter the degree of relatedness of laboratory-trained concepts (Goldstone, 1991, 1993b). The computational model shows how a concept can be represented both in terms of isolated perceptual inputs and on other concepts (Goldstone, submitted).

With members of Douglas Hofstadter's laboratory, I have also been exploring structurally rich models of concept representation (McGraw, Rehling, & Goldstone, 1994; McGraw, Rehling, Goldstone, & Hofstadter, submitted). Surprisingly, virtually no contemporary optical character recognition systems treats letters as structured objects containing parts in specific relations to each other. We have been experimentally and computationally exploring the need for models that create letters from "roles" and relations between roles. For example, the letter "b" may be composed out of a "bowl" role combined with a "left-post" role. Our approach seems to recognize letters that are far more stylized than those handled by typical optical character recognition programs.

Summary

There are a few unifying themes in my research work. The first is that there are important interactions between concepts and perception. The "heaven" of ideas and the "ground" of perception are intricately related, and it is not simply the case that perception gives rise to concepts. Perceptual organization depends on experience and learned categories. The second theme is that cognition is much more fluid and dynamically evolving than has been supposed by most theories. Perceived similarity depends on several contexts. There may not even be fixed "building blocks" of representation; the building blocks themselves may be flexibly tuned to one's needs. The third theme is that one of the best ways to express theories in psychology is by developing computational models. These models establish a level of rigor in theory development and testing that is difficult to achieve with purely verbal theories. The models also tend to be elegant and parsimonious; complicated cognitive behaviors emerge from the simple interactions of a large number of units. Although we cannot get "something for nothing," it is surprising to see the degree of self-organized structure that can emerge in a system without building knowledge into it or specifying a step-by-step set of rules, but instead giving the system the ability to learn to organize its own concepts.

In the future, I hope to continue to develop computational models of concept learning and perception, and test them using human data. I am scheduled to have a contract with MIT Press to write a book, See- through Concepts, on interactions between perception and cognition. I have been asked to contribute an article to a special issue of Cognition on the role of perceptual similarity in cognition (the special issue will be released as a book through MIT Press). I have also been asked to prepare an article for Annual Review of Psychology on perceptual learning, and to a co-edit (with Douglas Medin) a special volume in the series The Psychology of Learning and Motivation on perceptual learning. I have ongoing collaborations with developmental psychologists on the acquisition of concepts by children, with neuroscientists on models of object recognition, and computer scientists on artificial intelligence approaches to categorization. At this stage, a sufficient number of studies have now been completed detailing the fact that there are bidirectional influences between our perceptual and conceptual systems. The challenge now will be to develop formal models to show the mechanisms underlying these influences.

References

Last updated October 18, 1995.