www.greaterwrong.com/posts/XC7Kry5q6CD9TyG4K/no-evolutions-for-corporations-or-nanodevices

One of the misunderstandings of evolution: there's a meme going around that all kinds of things can evolve by natural selection, not just DNA/RNA but also self-copying nanodevices and even things like human corporations. After all, the market is a battleground where survival of the fittest applies, right? Presto, evolution! With time, the corps that survive will be stronger and stronger.

Except no. Many things have to go right for a force like evolution to meaningfully apply. A big thing missing from corps and nanodevices is copying fidelity.

Look. A corp may be outcompeted by another corp who learns the lesson their enemy failed to learn – but that one's successor in turn will easily have never learned this specific lesson, and so be vulnerable to the same mistake. The same way you carry many of your parents' life-lessons but not so many of your great-grandparents' life-lessons, especially if you've never met them, and so you repeat your great-grandparents' mistakes. The same way that post-WW2 was a great time for human rights, but a few generations later fascism is on the rise. Current generations do not have the life-lessons to see through the farce: the life-lessons failed to copy themselves for more than 2-3 generations.

All the old lessons of history, you are doomed to repeat them again and again.

—Leto II, God Emperor of Dune

Contrast with DNA evolution. Consider a genetic mutation that confers a 3% increase in fitness on average. DNA copies perfectly, so after about 768 generations(!), every member of the species has inherited it – and all copies are exactly identical (if just one molecule sits differently, we regard it a different gene).

DNA copies with something like 10^-8 errors per copy, several orders better copying fidelity than we can do with any machine for now – I think we're at 10^-2 or something? Our nanomachines would have something like a million times the mutation rate (i.e. rate of copying errors), which isn't viable for life.

www.greaterwrong.com/posts/q7Me34xvSG3Wm97As/but-there-s-still-a-chance-right

One of the chief morals of the mathematical analogy between thermodynamics and cognition is that the constraints of probability are inescapable; probability may be a subjective state of mind, but the laws of probability are harder than steel.

[…] Here's a little experiment: Smash an egg on the floor. The rule that says that the egg won't spontaneously reform and leap back into your hand is merely probabilistic. A suggestion, if you will.

[…] It may help to think of it this way—if you still have some lingering intuition that uncertain beliefs are not authoritative:

In reality, there may be a very small chance that the egg spontaneously reforms. But you cannot expect it to reform. You must expect it to smash. Your mandatory belief is that the egg's probability of spontaneous reformation is ~0. Probabilities are not certainties, but the laws of probability are theorems.

If you doubt this, try dropping the egg on the floor a few decillion times.

–www.greaterwrong.com/posts/zFuCxbY9E2E8HTbfZ/perpetual-motion-beliefs

A popular idea of "rationality" is Spock from Star Trek. Yet Spock is much more like a sub-par human statistician than a rational being, if you follow Cognitive science's definition of rationality. He's merely the autist of the TV-show, the straw Vulcan, and if anyone is the rationalist, it's Captain Kirk. Why? He makes the decision that saves everyone's butts, over and over again.

The Hollywood-Rationalist Spock says something like: "our probability of survival if we go into that nebula is 2.234%". Then Kirk says to go into it anyway, and nine episodes out of ten, the Enterprise is fine! Clearly he knew something Spock didn't.

Kirk doesn't survive despite irrationality; on the contrary, because he survives so reliably, you can see that his choices are pretty rational.

Rationality is about winning. It's making the decision that saves the Enterprise; the decision with the greatest chance of success, whatever you mean by "success", and no matter if you don't have an exact idea of the numbers involved.

It's not about ignoring your emotions and ignoring what your "gut" tells you in favour of a deliberate probability or utility calculation (though it can be!). If your gut has been reliable in the past in similar situations, why would you think it's not rational to listen to it?

Spock is not how a rationalist sounds… but he's often what an aspiring rationalist sounds like. The final form is Kirk.

Which leads us to another trope, emotionlessness. Yudkowsky:

For my part, I label an emotion as "not rational" if it rests on mistaken beliefs, or rather, on mistake-producing epistemic conduct: "If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm."

(www.greaterwrong.com/posts/iiWiHgtQekWNnmE6Q/if-you-demand-magic-magic-won-t-help)

If you need to live in a world with magic to take your life somewhere great and fulfil your dreams – 'if only magic were real' – you may be engaging in a fantasy that's comfortably out of reach so you need not even try. Realize that being in a world with magic would make it no longer special, the same way you treat lightbulbs as nothing special in this world. Thus, if you weren't studying difficult topics here, you wouldn't be studying magic there, as it'd be just another difficult topic that nobody regards as interesting. "What do you study?" "Magic." "Cool. I study art." "Bachelor's degree or…?"

Thus: if you're the sort of person to demand a magical world, actually living in that magical world wouldn't change anything, so what's the point?