AI versus Cognitive Models:
Should we be doing AI or Cognitive Models?
(written on 27 Feb. 2014; updated 01.05.2014 thanks to Bipin Indurkhya's feedback. updated 15.09.2014 thanks to Paulo Abelha's tip. Other comments welcome:)

The short answer: AI wins

What I mean by AI: I mean building systems to automate the types of tasks that humans do with ease, like robots cleaning or preparing food in a kitchen, or computer vision systems, or natural language processing systems.

What I mean by Cognitive Models: I mean a general model of some aspect of cognition, which does not attempt to compete with existing AI techniques in performance on any particular AI task. It is validated with data from a toy domain, or from psychology experiments measuring some isolated parameters of human performance on a set of lab tasks.

I had a change of mind on this issue since I started getting seriously interested in AI in 2002.
During 2002-2012 I didn't want to get bogged down in hard AI problems, and thought there was too much risk in getting very application specific programs that shed no light on how humans do it.
So I wanted to build toy models of some cognitive process (infant development mechanisms in my case).
But I now believe deep learning shows the way.
If you start from the right place, with some idea of a part of the solution to strong AI/general purpose AI, and try to build something general, then pursuing AI problems can even help you to take your general system in the right direction to get closer to what humans are doing.


I now think AI is the way to go, and I don't like cognitive modelling any more.
It's not the idea of a "model" that I object to, but the data used to "validate" it.

"When the authors invent the definition of the data, and they invent the problem, and they invent the error metric, and they make their own data, can we be surprised if they have high accuracy?"
 - Eamonn Keogh

If the data used to "validate" it is toy contrived data, then... the model is only as good as the data; i.e. it is a very artificial contrived model that probably has nothing to do with human cognition.
For me it's like Aristotle versus Galileo - cognitive models are typically not tested on the *real everyday tasks that people really can do* - and if they can't do that I think they might be a bit like Aristotelian theories of mechanics.
(I must clarify here that testing on some isolated parameter of human performance on a lab based task is insufficient. e.g. modelling a child's acquisition of knowledge of how to balance a scale by looking at competence in that isolated task misses the big picture of all the everyday situations where the knowledge can or can't be applied, and the experiences that led to the acquisition of the knowledge. Think of the intuition a human develops about the force needed to move something held in an outstretched arm, or to push a heavy door near the fulcrum. These experiences and the intuitions generalised from them are surely at the core of human cognition. You are not really modelling the child's mind unless you can model a wide range of the things the child can do.)
I actually think AI is the route into really understanding how the mind works.
Physics progressed by doing experiments to test theories and get results.
I think the analogue for progressing psychology is trying to apply your theory to a hard AI problem.
(If you don't: We have plenty of examples from the history of AI of AI programs (=theories) demonstrated on toy domains, and they did not scale to real domains,
and moreover the techniques which eventually (40 or so years later) performed better on those real domains were most often not extensions of the original theory, but completely different approaches.)
If we can really crack the problems of natural language understanding and vision and cognitive robotics performing at a human level,
then I think we will have learned a great deal about the human mind (partly because I also believe we will be successful by doing it in a pretty much human-like way, by copying what we know of humans).

I believe trying to tackle an AI problem is a discipline that a scientist must follow if he wants to test his theories. It's hard, and I've sometimes wanted to avoid it, but now I believe it's absolutely necessary.
The only way I can be sure I’m not deluding myself about some theory is if I can demonstrate it works in some “easy for humans hard for computers” domain, or preferably more than one such domain.
If the theory really has a claim to be part of the core of a human-like cognitive system, then why should it not be able to be evaluated on a real application where humans currently exceed AI ability?

Note that this is not only about testing your theory in a real domain to see if it works.
Working in a real domain lets you know the problems which need to be solved, and those which don't matter.
If working in an invented domain you might make a big deal over some "problem" that doesn't arise in a real domain, and you might completely gloss over a major issue that requires an elaborate solution in a real domain.

I am really impressed with the deep learning pioneers - they certainly started from a biological inspiration place, but they tried to apply their stuff to real AI problems, and defeated existing approaches in language, speech, vision.
They claim that they have captured something like part of how part of the brain works,
and I think, given the demonstrated results, they have a credible claim.

[2016 addition: note it is interesting to see how Deep Learning is helping to demystify operations of the brain,
Yamins and DiCarlo, "Using goal-driven deep learning models to understand sensory cortex", nature neuroscience]

Hinton says he thinks we will get to understanding what the brain is up to, by continuing his approach of working on real AI problems, and improving the learning techniques he has.
He says he thinks he will get there about a thousand years quicker than Henry Markram.
https://www.youtube.com/watch?v=vShMxxqtDDs
He says the kind of neural inspiration he likes is when making it more like the brain makes it work better.
Hinton says of Markram: "he just knows how to make it more and more like the brain without actually working
and that seems to me not the right approach...
there's not much point in making it more like the brain if it just makes it work worse".
(i.e. he is always primarily focussing on working better on real AI problems - that is the goal he is driving towards - and he believes that by focusing on that goal we will actually get an understanding of the brain faster than the guys who focus on modelling the brain itself and ignoring the actual information processing problems it solves.)

This mirrors something Marr said in 1975 in a negative review of models of the nervous system.
Marr came to believe that it was necessary to look at the real information processing problems.
He moved away from general theories (which he had in 1969-71) that didn't explain a specific information processing problem.

Above I looked at the example of Deep Learning vs. Neuro models. We can also look at other AI successes vs. psychology inspired models.
I think the cognitive models like SOAR, ACT-R and others seem to me to only capture a very narrow slice of human cognition - experts reasoning with knowledge for example.
IBM Watson or Google Car or Michael Beetz's Robots doing Everyday Tasks may be better models of human cognition,
although they didn't set out to model cognition, they set out to solve a practical real-world problem.
With the Google car or Beetz's work I mean they way they use simulation to model the world, and predict what will happen, and also make extensive use of probabilities.

"He who understands the baboon would do more towards metaphysics than Locke."
—Charles Darwin, 1838: Notebook M
Perhaps this is akin to saying something like
"Whoever builds an autonomous car (or a robot that cooks my breakfast, etc.) will do more for the understanding of how the mind works than so many "cognitive modelers"."

One way to summarise this is do you want to be a philosopher or a scientist? If you want to be a scientist discovering how cognition works then you need to do real experiments with real tasks that we know human cognition can do, but we don't know how human cognition does those tasks. Any model/theory you have about how cognition works needs to be tested on those real tasks - if you can't get performance comparable to humans on real tasks then you don't yet know how it works. Some peope stay at the philosophy level with theories of how it all works, but the journey from such a theory towards a system that solves a practical task is a very long one, and one that usually reveals that the original theory was wrong or at least very incomplete. That journey is the one we must take to find out about cognition.

Another postscript to this is that I used to (2000-2009) be really disappointed that the majority of people worked on "weak AI"/"applied AI" and so few worked on "strong AI". But now I am getting more impressed with weak/applied AI, and I think it might get there first.
I think the principal thing that changed over the years is that applied AI has been getting more and more ambitious, so that they are starting to hit the really hard problems, and they are tackling them.

A post-postscript - looking at the attendance of major conferences, e.g. in robotics ICRA, IROS get about 2000 people, also true in other areas CVPR,ICCV,ACL,ICML,NIPS. Conferences that deal with solving practical problems that lots of people care about get a lot of interest. If you want impact you need to go there. Cognitive Modelling events don't get that level of interest. A lot of their work, even if good, gets ignored.
If you have a good idea for AI you need to show its benefit on a problem a lot of people care about and take it to those people and promote it there.
I also see examples where those in the big practical problem AI communities can rediscover independently something that psychologists said earlier, because the AI people find it practically useful to solve their problem. The psychology or cognitive modelling can get bypassed.
Some examples of good impact include LDA and IBM Watson. These arose from work in concrete domains, like LDA builds on a long history of trying to understand the topics/gist of text, and IBM Wtson had an agenda to contribute to practical applications in question answering and information retrieval from the beginnning.


Acknowledgement: I was influenced a lot (against "models") by my PhD student John Alexander.
In fact John's key point is not expressed above. It is the machine learning perspective: if you showcase your learning program in one example task, as is common in developmental robotics, it is unlikely to generalise to other tasks. It relies on certain parameters being set the right way to work on that task. If it doesn't work in general then it's useless. It's very unlikely to work in general if you didn't have a goal to make it work in general, but rather had a goal to showcase it on one example. That's why machine learners try to prove things work in general, and find cases where they don't, and change their algorithms to work more generally.

David Marr. “Approaches to Biological Information Processing.” Science 190 (1975): 875–876.

Keogh: "When the authors invent the definition of the data, and they invent the problem, and they invent the error metric, and they make their own data, can we be surprised if they have high accuracy?"

another interesting paper:
Using goal-driven deep learning models to understand sensory cortex
Daniel L K Yamins1,2 & James J DiCarlo1,2
VOLUME 19 | NUMBER 3 | MARCH 2016 nature neuroscience
"Though the top hidden layers of these goal-driven models end up being predictive of IT cortex data, they were not explicitly tuned to do so; indeed, they were not exposed to neural data at all during the training procedure. Models thus succeeded in generalizing in two ways. First, the models were trained for category recognition using real-world photographs of objects in one set of semantic catego­ries, but were tested against neurons on a completely distinct set of synthetically created images containing objects whose semantic cat­egories were entirely non-overlapping with that used in training. Second, the objective function being used to train the network was not to fit neural data, but instead the downstream behavioral goal (for example, categorization). Model parameters were independently selected to optimize categorization performance, and were compared with neural data only after all intermediate parameters—for example, nonlinear model layers—had already been fixed."
connects to right problem txt. some probs drive us towards a technique, like deep learning for vision. Go better than chess because pattern recognition, but solved when deepRL became good, it did not drive developments in deeprl. object recogntiion was a better problem - drove a development of something very general to cognition. we need more problems like this. but in each case there is also an inspiration like animal behaviours for RL or neuro for DNN. autonomous driving is driving us towards models good for robot planning in everyday tasks.