Eric Holloway points out that the assumption that the human mind can be reduced to a computer program has never really been tested:
This test for intelligence, the Turing Test, was invented by and named after the mid-twentieth century computer pioneer Alan Turing. It is a subjective test in that it depends on whether an artificial intelligence is capable of convincing human testers that it is a human. But fooling humans, while impressive, is not really the same thing as actually possessing human-level intelligence. In any event, some judges may be biased in favor of the AI passing the Turing test and may thus be easier to persuade than skeptical ones.
In 2014, an AI chatbot named Eugene Goostman passed one Turing test competition, organized by the UK’s Reading University.The chatbot was developed to give the impression of a 13-year-old Ukrainian boy, so that errors and manipulation of conversation could be overlooked. Quantum physicist Scott Aaronson showed just how unconvincing this chatbot was in a short conversation.
Eric Holloway, “Current artificial intelligence research is unscientific” at Mind Matters News
See also: Math shows why the mind can’t be reduced to a formula. The Liar’s Paradox shows that even mathematics cannot be reduced to a fixed set of axioms. Gödel’s discovery brought back a sense of wonder to mathematics and to the rest of human knowledge. His incompleteness theorem underlies the fact that human investigation can never exhaust all that can be known. Every discovery builds a path to a new discovery.