I'll come back to this...

But I wanted to put it in here while I still remembered. This is a quote from an Ian Watson article on AI:

The subjective nature of awareness

A major assumption about AIs in the popular mind and in entertainment is that they will indeed be conscious and will have subjective experiences. The common image of an AI is one of self-awareness, not merely superintelligence. But how much self-awareness do human beings possess - and what is this "self" that we are aware of?

In 1985, the neurosurgeon Benjamin Libet performed some experiments with surprising results. He put electrodes on subjects to detect their brain waves and the flexing of their wrists. The subjects watched a revolving spot on a clock face. They could flex their wrists whenever they chose, but had to note the exact position of the spot when they made this decision. Libet was timing the beginning of the action, the precise moment of the decision to act, and the beginning of a particular brain wave pattern known as the readiness potential. When the brain preplans a series of movements, this pattern occurs just before the complex action.

Libet found that the readiness potential starts about one half of a second before the action, but the decision to act occurs about one-fifth of a second before the action. The conscious decision to act is not in fact the starting point. The event is already beginning before the person consciously chooses to start.

Conscious awareness lags behind what happens. You jerk your hand away from a hot surface before you consciously feel the pain. However, we do not realize this because of what Libet called subjective antedating. The brain puts events in order after the event. "I feel that I consciously did such and such," but tests prove otherwise.