

The only thing that worries me is that we don’t actually know how the human brain works. There’s an entire school of psychological theory and practice - really its oldest truly scientific branch, that holds that human intelligence actually does work a lot like an LLM. The hard core behaviorists believed that literally all human behavior was just a really complex version of Pavlov’s dogs. It sounds absurd, but they had good arguments. In principle even very complex behaviors can be the result of reinforcement and conditioning.
Exactly. We are beings of atoms and matter. If you want to believe in some immaterial soul, fine. But if you’re a materialist, then everything we are is atoms. We know atoms in one configuration can produce true intelligence. And there are likely many possible arrangements of atoms that can reproduce this effect. And since artificial minds are not subject to most of the constraints of biological minds, an artificial superhuman intelligence should be possible. Hell, even if biology was the only way to make it possible, you could always build an artificial biological brain and just make it a lot bigger than a human one. Even if human neurology really is the limit of what this universe allows for in terms of intelligence, we could best it by just making a bigger one.