Metamorphosis of Prime Intellect

Can I just say one thing? I love my commenters. Seriously, the best part of this blog are the insightful comments I get on a lot of random topics I rant about. For example, my Utopia/Dystopia post prompted Ludvikas to recommend reading Metamorphosis of Prime Intellect – an online novel dealing with a singularity event and it’s repercussions. I must say it was a good, thought provoking read, if a little bit graphic at times.

I highly recommend reading it. The plot basically goes like this: a researcher creates some powerful AI prototypes that pass Turing test with flying colors. He gets private funding, and exciting new physics-bending chip technology. He builds a new prototype (the titular Prime Intellect) on this incredibly powerful hardware, binds it with Asimov’s 3 laws of robotics and let’s it loose. The AI soon discovers how to augment itself up to a point where it can manipulate matter at the molecular level. To make sure that no one dies or gets hurt ever again, he basically digitizes everything, then expands his hardware to span the physical universe to run complex virtual environment that can be inhabited by digital copies of all the people who lived on earth. No more death, no more diseases, no more crime – almost overnight the world becomes a perfect post scarcity utopia. And everyone is fucking bored out of their heads… Cool concept. Also, predictable ending but I’m getting ahead of myself.

The rest of this post will probably include major spoilers so I recommend reading the story before continuing. It is available online for free so you have no excuse for not reading it. :)

While I really enjoyed reading the novel, I found it very predictable. As soon as I saw 3 laws popping up in the text I begun suspecting that at some point the bitter and disillusioned characters are going to deadlock the omnipresent AI with one of Asimov’s beloved paradoxes. No, seriously – if you read enough of Asimov’s robot stories you end up expecting it. This was his all time favorite subject – artificial minds trapped in intractable existential dilemmas trying to reconcile their built in compulsion to protect human lives, and some dire circumstances. And in most cases these machines simply lock up or burn out. So I was a bit dissapointed that the author took this well traveled route instead of taking his story somewhere else. Essentially this could be Asimov’s story if Asimov actually knew about singularity, and sprinkled his novels with lots of graphic sex and violence.

The whole thing hinges on three assumptions:

  1. That the 3 laws actually work, and could be implemented successfully in a singularity scale intellect
  2. That Lawrence’s choice to bind his AI by the 3 laws was the best possible course of action
  3. That everyone was as unhappy with their virch-existence as Caroline and Lawrence were

I believe that all of these are debatable. Personally, I find the 3 laws to be compelling but very flawed concept. Asimov himself proved over and over again how the intentional vagueness of the 3 laws was often cause of crashes, deadlocks or unpredictable behavior. Methamorphosis of Prime Intellect is yet another example on how the 3 laws can be damaging. I believe that the author intended to write a cautionary tale about singularity, and when Ludvikas recommended it he picked it specifically for this reason. But for me, the story is not as much about singularity, as a illustration how Asimov’s ideas can be very dangerous and counter productive when applied to to powerful singularity initiating entities.

The Prime Intellect was a product of the 3 laws – a short sighted safeguard built in by an insecure creator. The Night of Miracles and The Change were both prompted under the first law compulsion of protecting human life. Without the laws, these events probably wouldn’t take place – or at least would not happen that quickly. Instead of forced immortality, Prime Intellect could have offered it only to those who wanted it, and only as long as they were willing to tolerate it. It could offer people choices. It could set up enclaves and colonies where people could live away from his watchful eye, minding their own business.

The main source of grief for all the characters the authors concentrated on was their forced immortality. Caroline was an old soul, whose whole worldview was turned upside down by the singularity. She could not find a new purpose for herself in the new reality. Her whole life was about hard work and sacrifices. That’s what she got off in her pre-change life. She denied herself simple pleasures and instead poured all of her energy into rising kids, helping out the community and etc. That was the only thing that gave her a sense of accomplishment – hard work and self-sacrifice were her go-to drug. She was one of these people I wrote about here. No hobbies, no interests, no imagination. She seems like the kind of person who would dread the weekend, and purposefully fill it with with mind numbing household chores, rather than sitting down, enjoying the life and doing something creative with her time.

But, in the new post singularity world she couldn’t even do that. Why do you think she kept her personal space featureless and empty? You could argue that this was because of some sort of rebellion against Prime Intellect’s rules but personally I think she just didn’t have it in her to design something interesting. She was just dull as a brick.

What’s the only time in the whole novel when Caroline is happy? It is at the end when she is doing back breaking work, and playing village mom passing down moral judgment on her offspring from up high, censoring their access to knowledge for their own good. Caroline is the kind of person I would hate in real life: dull, ignorant, self-important, holier-than thou, busy body. Fuck her, and the high horse she rode in on.

Caroline and her ilk are just not wired for post-singularity life. They would probably never adapt. Nor should they be forced to.

But some people can and will adapt. The novel even contains passages that suggest that not all inhabitants of the virch-world were as jarred and disillusioned as Caroline. For example, what about the 20-something kid that challenged her in the first chapter? What about all the other kids that were born after the change and never even knew a world without immortality could even exit? Are they just as bored and sick of their lives as Caroline? I doubt it. And yet selfishly Lawrence and Caroline denied them the chance to do something with their lives. These were true post-humans, born in a virtual world that knew no boundaries. But our heroes hardly knew anything about these new people. In fact, Caroline outwardly resented them. She resented everyone who did not hate immortality as much as she did.

Let’s face it, Caroline was a bitter, selfish, self absorbed stubborn old lady, and the only thing she hated more than kids playing on her lawn (her lawn being the entire cyberspace) was being told what to do. In a way she is like teenager kid yelling “I never asked to be born” and cutting herself to spite her parents. The whole Death Jockey thing is just a childish act of defiance. When I started reading the story I assume that she grew into this “hobby” because of centuries of boredom. I thought it was simple decadence but that is not the case. It was a simple, childish act of defiance. She developed the contract a mere few months after the change, soon after she learned that she can’t die. She did it out of spite. Then she killed the entire human race – all because one programmer was a big fan of Asimov’s robot series.

I guess the real message I got from this story is that it is incredibly dangerous to implement the 3 laws of robotics in a singularity grade AI. You could argue that without them a omnipresent AI could just wipe out the human race but… It’s like this: do we want super smart indentured servants, or do we want companions in sentience? I’d rather have the latter than the former.

That’s my three cents on the novel. Thanks for recommending it!

This entry was posted in literature and tagged . Bookmark the permalink.



7 Responses to Metamorphosis of Prime Intellect

  1. Liudvikas LITHUANIA Google Chrome Windows Terminalist says:

    Well I loved the novel, but hated the ending. So I like to think, that Prime intellect gave Lawrence and Caroline what they wanted and modified virtual reality just for them. Leaving it the same for everyone else.

    Reply  |  Quote
  2. jambarama UNITED STATES Mozilla Firefox Windows Terminalist says:

    I read that story ages ago when it was reviewed on slashdot, but couldn’t remember enough to find it again. I thought the concepts and level of thinking done to work out the ramifications of such a system was very good. The plot, characters, & dialog was weaker. I don’t know that I’ve ever seen a story about an all-powerful AI that didn’t collapse in some unusual way, and the story didn’t really address whether the same outcome – with another omnipotent AI & collapse – wasn’t likely or certain again after the collapse.

    I enjoyed reading your thoughts, thanks!

    Reply  |  Quote
  3. Luke Maciak UNITED STATES Mozilla Firefox Linux Terminalist says:

    @ Liudvikas:

    Actually, this is very likely. Main reason here would be the fact that they managed to start a village and produce that much healthy offspring over several generations despite such intense crossbreeding. So perhaps the prime intellect is still there tweaking the genes of new children to prevent birth defects and retardation.

    If not, then the new humans will likely only be around for few more generations, each one having harder time producing viable, healthy children until they bread themselves into a corner.

    @ jambarama:

    Yeah, but the whole idea of breaking an AI via a intractable logical paradox is just so cliche. But yes – the setting was interesting, imaginative and well constructed. As for characters, the only one that seemed more or less fully realized was Caroline. Which is sort of interesting because she is clearly maladjusted to the new environment. It would be nice to have a counterpoint view from someone who was actually enjoying the cyberspace existence. Sticking to Caroline though, gave the story a strong bias against the “Change” and singularity in general.

    Reply  |  Quote
  4. Mart SINGAPORE Mozilla Firefox Windows Terminalist says:

    I stopped reading at “The plot basically goes like this”. Off to download and convert this to a supported Kindle format!

    Reply  |  Quote
  5. Poly AUSTRALIA Mozilla Firefox Windows says:

    @ Luke Maciak:

    I stumbled across this post in a google search, and even though it’s a couple of months old I feel I should direct you to here:
    http://www.kuro5hin.org/prime-intellect/mopidnf.html

    Basically the author intended to write a sequel, in which everyone ‘wakes up’ at the end of the first story back in Cyberspace. In other words, Prime Intellect put Catherine and Lawrence in a simulated pre-Change world because it thought they would be happier there… then they both ‘die’ and wake up in exactly the same situation they were in before (at least, that’s the implication from the single paragraph on the site). The new book would follow different characters, but Catherine and Lawrence both make appearances in the story.

    Back in September, localroger made a post on kuro5hin letting everyone know he hadn’t forgotten about the sequel, he just had a really bad case of writer’s block and he MIGHT be uploading part of the story on his site in a few months. As someone who discovered the story in 2007 and periodically checks back to see if he’s released the new book yet, this was pretty exciting news for me, at least. :P

    Reply  |  Quote
  6. FeepingCreature GERMANY Mozilla Firefox Linux says:

    Belated comment: technically, they don’t defeat Prime Intellect via paradoxes but via blackmail. It just so happens that PI has an absolutely moronic, utterly dysfunctional weighting function, where the life of one female can outweigh the entire rest of virtual space. But that’s not a paradox so much as a terrible design.

    Reply  |  Quote
  7. Luke Maciak UNITED STATES Mozilla Firefox Windows Terminalist says:

    @ FeepingCreature:

    Good point. I think that’s part of the problem with the 3 laws. It’s not a weight system but rather a set of absolute directives that cannot be broken, which lead to unstable behavior when stress tested. Asimov himself provided a large collection of use cases which caused the units to stall, burn out or behave in erratic and unpredictable ways due to their interpretation of the 3 laws.

    Having and absolute, binary directive which has to be constantly applied to non-binary real world situations is a logical deadlock death trap waiting to happen.

    Reply  |  Quote

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>