SneerClub

is this post an extended retelling of the “I’m doing 1000 calculations per second and they’re all wrong” meme?

I don’t see how that works here. Humans don’t become impregnably narcissistic through bad management, rather insofar as management is the problem and as the scenario portrays it humans become incredibly good at managing information into increasingly tight self-serving loops. What the machine in this scenario would have to be able to do would not be “get super duper organised”. Rather it would have to be able to thoughtfully balance its own evolving systems against the input of other, perhaps significantly less powerful or efficient, systems in order to maintain a steady, manageable input of new information.

In other words, the machine would have to be able to slow down and become well-rounded. Or at least well-rounded in the somewhat perverse way that, for example, an eminent and uncorrupted historian is “well-rounded”.

In still other words it would have to be human, in the sense that human are already “open” information-processing creatures (rather than closed biological machines) who create processes for building systems out of that information. But the very problem faced by the machine’s designer is that humans like that don’t actually exist - no historian is actually that historian - and the human system-building processes that the machine’s designer will have to ape are fundamentally flawed, and flawed in the sense that there is, physically, no such unflawed process. You can only approach that historian by a constant careful balancing act, at best, and that as a matter just of sheer physical reality.

So the fanatics have to settle for a machine with a hard limit on what it can do and all they can do is speculate on how permissive that limit is. Quite likely, the machine has to do what the rest of us do: pick around in the available material to try to figure out what does and doesn’t work in context. Perhaps it can do so very fast, but so long as it isn’t to fold in on itself entirely it will have to slow down to a point at which it can co-operate effectively (this is how smart humans operate). At least, it will have to do all of this if it is to not be an impregnable narcissist.

That leaves a lot of wiggle room, but it dispenses with the most abject “to the moon” nonsense spouted by the anti-social man-children who come up with this shit.