1. What do models do? They try to simulate a very complex system.

2. The system responds in a certain way to a certain input: when we see a light we push a button (assuming sufficient motivation).

3. The model tries to predict when we press light

4. Why study models?

5. So a model takes some input and based on that input responds in a certain way.

1. Previous models provided more general description of what happened during process:

1. Transformed along the way.

2. Take input in, transform it according to the rules of the network

3. Output pattern is said to represent some abstraction

4. For example, you might show a network a picture of a dog, it might convert that image into something it understands (a computer pictures) and then transform that picture somehow until it outputs some pattern that corresponds to the concept of "dog".

1. e.g. I see an "eye" or "it looks like part of a d"

2. They give this information to one person in the second row.

3. Person in the second row takes this information and makes their own guess as to what the puzzle says: "is it Last Tango in Paris?"

4. I know the answer, and I tell them. After they find out how they did, they consider this answer and then figure out who was most responsible for giving you good information. What you do is compare what was said by each person, and if it was right or not. Then you adjust how much you are going to listen to them in the future.

5. E.g. if person at the end guessed "Tango" then you realize that they are pretty smart or are looking at a valuable section of the board. Thus you pay more attention to them in the future.

6. We continue this, with the same input pattern (game board) after erasing everyone's memory for the previous time. However, we don't erase how much you are going to listen to each person.

7. Eventually you realize that persons 1, 4 and 7 are smart and are looking at valuable sections of the board and you listen only to them and not to the others.

8. This describes how an input pattern can be distributed across multiple nodes, and how learning can take place.

9. It doesn't explain how we represent the input and output patterns.

1. Input units: the initial representation of the dog.

2. Output units: the part of the network that is going to produce the pattern that corresponds to the "dog" concept.

3. Connections: Things that connect input units to output units. Each has a weight associated with it that describes how much the output nodes use information from the associated input unit.

4. Activation values

1. Input value: value between 0 and 1. When we have lots of input units these together will represent an input pattern

2. Connection weight. Value that we weight the activation from the input node by. These can change, and when they do it represents learning.

3. Output value: number that will represent the response of the system (arbitrarily defined by the experimenter).

4. If input is 0.8 and weight is 0.3, then output is 0.8 x 0.3 = 0.24. An input of 0.8 means something, and an output of 0.24 may or may not mean something. Don't worry right now about how we assign meanings to these numbers (it's arbitrary anyway).

1. The network learns by getting feedback from an all-knowing teacher (tom)

2. This makes it not really biologically plausible

3. Each time the network processes an input pattern, it gets an output pattern. It compares this output pattern to the correct pattern. The difference between the two is the error:

erroro = target output - actual output

4. The weights are changed by an amout that is the product of three numbers

5. This gives:

CWio = ai * lr * erroro

6. The new values for the weights are computed by:

New Wio = Old Wio + CWio

7. This continues though all patterns until the erroro's get close to zero (which implies no change in the weights as a result of equation in 5 above).

8. This is called backprop

8. Go through example on excel spreadsheet

1. Problems with delta-rule learning: we don't have a teacher in our heads

2. Instead use a self-directed learning rule

3. Based on a physiological property: neurons that fire together, wire together.

4. So if an input unit and an output unit are both highly active, the network increases the connection strength (the weight) between the two units.

5. The increase is in proportion to the product of the joint activation

New Wio = Old Wio + CWio

6. problems with hebbian learning: has trouble learning patterns that are similar to each other (correlated input patterns). It gets confused when trying to decide which is which

7. Not very powerful- it does what it can do, and can't do what it can't do.

1. either show letter alone or letter as part of a word

2. letter alone takes longer to recognize. Why?

3. Can't just be that it is flanked by any other letter (e.g. more letters helps you to attend to a given location). No speedup if letter in non-word

4. Other letters in a word place constraints on what letter can be

e.g. wor_ implies that last letter is either d or k or m, not a or b or etc.

5. However, if you do the experiment like this, person may be able to guess the letter just by seeing the other letters. Could get the w.s.e. even if didn't see last letter!

6. Equate this by 2-alternative forced choice with two valid letters (e.g. k or d)

person had to choose the correct one.

7. We've eliminated the effects of using the letters to help guess the last letter, but we still get the word superiority effect

1. Three levels

2. Inhibition within a level: what this means is that when a node starts becomeing active, it tends to surpress the other nodes. The rich get richer.

3. E.g. if one letter becomes more dominate like a "k", it tends to surpress the alternatives like "d"

4. The letter level also receives feedback from the word level, both excitatory and inhbitory.

5. Let's say the word presented is "trap" and the person then sees tr_p and has to choose beween "i" and "a". The "a" letter node startes becoming active, as do the "t" "r" and "p" letter nodes. However, the "i" letter node doesn't become very active.

6. Because the "t" "r" "a" and "p" letter nodes are becoming active, the "trap" node becomes active as well. As it becomes active it does two things:

7. How does this work when only a single letter is presented?

8. Word sup. effect is predicted by a model that assumes bottom up activation from the feature level, and top-down activation from the word level.

1. A company's stock goes up when it gets taken over

2. Lots of financial information about a company

3. Some of these companies get taken over. It would be nice to own this stock before this happens

4. How can predict which companies get taken over?

5. Develop a huge nework: need an input node for each piece of financial information we can get: cash on hand, stock price, etc.

6. Have a bunch of hidden units- just so the network has the ability to learn complex relationships between different indicators

7. Have one output unit that is active if the company was/will be taken over

8. Train network on previous data- include all companies, and set the desired output to zero if the company wasn't taken over, and 1 if it was (and stock went up).

9. Could train on one half of sample, test on second half

10. Doesn't have to be 100% accurate to give you an edge in the market.

1. Scientists often go on "fishing expeditions" looking for cancer-causing agents

2. Collect lots of data about what eat, drink, how much excerzise, etc.

3. Have lots of different cancer types that are recorded.

4. Use diet and exercize data as input, cancer rates as output

5. Train network on lots of people

6. See if network can predict cancer in patients in the future

7. Examine weights to determine the relationship between variables

8. E.g. red wine may make up for eating a fat-filled diet (not true, actually)

1. way too much information from satalites to view by people

2. Train computers to look for tanks, ships etc.

3. Apochraphal story:

"A major attraction of connectionist models is that they perform computations by means of processing utnits that behave on broadly similar principles to those used in the brain. Their appeal would be strngthened if it could be shown that analogous operations ont he elements of the two systems lead to similar consequences of rthe behavior of the overal systems. An obvious candedate operation is the effect of lesions. We take a realtively simple connectionist model and show that the lesioned model has properties that are similar to those already described in certain neurological patients."

1. Train a network to extract the features of letters, form letters and words, and then pronouce them

2. Look at two forms of dyslexia in people

3. Dyslexia is an umbrella term for many different disabilities, but the two that are described above are thought to result from a lesion in the information-processing pathway

4. To study this, first program an train a neural network to read words (like the word-superiority effect model, only one that can talk).

5. Once it is working, go in and "lesion" parts of the network by randomizing some of the connection weights.

6. Now look at the errors produced by various lesions.

7. Do get vision confusions (surface dyslexia) for lesions in one location

8. Get semantic confusions (deep dyslexia) for lesions in another location

9. Lesions in a third location produce both surface and deep errors

10. All three patterns are similar to those seen in humans

11. Interestingly, the lesion responsible for vision confusion is actually quite late in the process

1. Describe exactly how information is represented and transformed

2. Distributed, which is biologically plausible

3. Seems like an intuitive model of actual neurons.

1. recognizing patterns and forming abstract representations of those patterns

2. recognizing patterns in the presence of noise

3. recognizing partial patterns

4. distributes the representation and so can deal with "lesions"

1. may not be biologically plausible (much of human learning is self-directed)

2. Take a long time to learn big sets

3. researchers are designing specific models for each task; the brain cannot be as specialized.