Last week I talked a bit about the future of UI design. I criticized Microsoft for the lack of vision but I never really tried to counter it with my own. Alas, I am not an UI designer. I do not ride the bleeding edge of technology, and I don’t get paid for imagining the interfaces of the future. I’m just a devilishly handsome, and humble back end programmer cum sysadmin. But if you asked me what my vision of future user interface was, I would probably say: none. By that I mean there would be no discernible interface to talk about. The whole point of UI design is to make technology intuitive and easy to use. So the ultimate UI would simply be almost imperceptible to the user. Let me give you and example.
Most mortals have a very specific vision of “the future” as a place where we basically just have conversations with our computers. A common future tech scenario pushed by many UI imagineers looks a little bit like this.
The Man of the Future, let’s call him Bob, comes back from work. His house intelligently detects his presence, turns on the lights and his house greets him with a pleasant female voice. Our hero makes a bee line for the fridge to fix up an evening snack for himself, while he conversationally chats with the house AI.
“Anything interesting in the news?” he asks while pulling milk out of the fridge. The friendly female voice gives him a brief summary. The fridge door temporarily becomes a screen displaying an array of articles and video clips. Bob lazily slides his hand over the display, scrolling through the data, and then waves it away. Fridge door becomes a door again – complete with digital sticky notes, and a holographic garbage pickup schedule.
“How are my appointments for tomorrow?” he queries as he starts to assemble a sandwich. The house dutifully reminds him about a big meeting he has the city at 8am sharp. Bob nods.
“Directions.”
Instantly a holographic display pops out in his field of vision, showing him a map with the best routes. He grabs his sandwich, and inspects the map for a few seconds than waves it away.
“Load it into my GPS.” he orders while carrying his sandwich into the living room. As he sits down on the couch, he politely asks the house to put on the TV. The far wall with dissolves into a huge panoramic 3D display. Bob finishes his sandwich watching some random move or TV show.
“Call my wife.”
Few seconds later the face of his significant other appears as an insert in the corner of the huge TV display.
Sounds vaguely familiar, right? You have probably seen some version of this vision somewhere before – perhaps in a SF movie or novel. But it is a short-sighted crock of shit. It is a pedestrian vision of the future which assumes that voice commands and gestures are the “most natural” and intuitive interface methods. It’s almost like having no UI at all. But alas, it is still there. Bob still has to order his house around, and explicitly state his orders. In fact using voice commands as input is painfully slow and it has no “speed dial” feature. If I had a talking house, as a power user I would train it to allow me to trigger specific tasks just by subtle hand gestures – like snapping my fingers, pointing or clicking my tongue because this whole talking to a semi-turingrade machine would totally annoy me. So in a way, I would be actually effectively circumventing the UI that was supposed to be imperceptible.
Here is my alternate vision of the future in which the UI layer simply goes away. There are no screens, no overlays, no HUD’s, no swiping, no gestures and no voice commands. The UI is made redundant. Granted, this comes from this transhumanist, progressive worldview of mine that has been fueled by a steady diet of science fiction. But hey – this is my vision.
In my version of the far future, Bob does not need to ask the house for his news. He simply becomes aware of them. His augmented mind reaches out to the internet and grabs news feeds from his favorite sources. They are not textual or video streams though. Bob is no longer forced to consume his media via the five analog senses humans were born with. He gets his news modeled as complex data packages that are side-loaded directly into his exo-memory. He does not need to read or view them like we, low-fi humans do – once they are downloaded and cached he recalls them just like any other memory – only more vivid and detailed. He can consume gigabytes of data in mere seconds. And since most of Bob’s private memory exists in the cloud, he does not really have to worry about storage.
The new Bob does not need to check his appointments. He knows about them. As the time nears, the appointment becomes a focus point within his awareness. The neural wetware he was born with can sometimes be fuzzy, and forget important things – especially when it gets preoccupied with personal hobbies, obsessions and etc. However the digital part of his mind never forgets. So the appointment is there – impossible to overlook or forget, unless he wills it to be forgotten. It is not an overlay, not a blinking icon. It is just the awareness of what, when and where.
Similarly, Bob does not need a watch. His exo-cortex has a built in time sense. Bob simply knows what time is it – always. Just like birds can always tell where North is, Bob can tell the time. When he travels, his time sense automatically adjusts to the correct time zone. His circadian rhythms may lag behind, but the digital part of his consciousness constantly syncs up with time servers. After all, time keeping is such an integral part of our lives. We surround ourselves with clocks now, so it would be foolish not to build it into ourselves.
Directions would be pointless for the neo-Bob as he can track the route in real time. Imagine Google maps, but modeled just like the news casts I mentioned earlier. Data not made to be consumed via the sense of sight, but destined for direct upload into the transhuman mind. Bob not only knows the road, he also knows the shortcuts, and he becomes aware of traffic jams as he is en-route. All without distractions such as overlays that could affect his driving.
Finally, he does not need to call his wife. He simply reaches out with his mind, and the connection is established between them. She becomes aware of his desire to communicate, and can accept or deny by simply willing it to happen. When they do establish a link, it is not a voice or video chat, a direct point-to-point conversation facilitated by their exo-cortexts. They can subvocalize if they want to, but they can also share images, smells, emotions or entire libraries of ideas. Their minds may touch this way for only a brief second or two, bur the amount of data exchanged in the process may amount to a long, heartfelt face-to-face conversation.
Artificial intelligence plays a large part in this, but it stays hidden. It works behind the scenes to facilitate, catalog and route information. But the goal is seamless integration of the human mind with the data-rich environment he creates. It is about data absorption, and rapid exchange without being slowed down by clunky interfaces operated via analog appendages better suited to biological tasks such as preparing and consuming food, or holding loved one’s hand for comfort.
Granted, all of this is strictly science fiction now. Whether or not all of this is possible is besides the point. The point is about having a vision of the future that goes beyond what we have now. The Microsoft showcase video I linked to last week is an example of limited, short sighted vision that aims to solve all our problems with “pictures under glass” interfaces only slightly more advanced than the ones we have right now. My vision goes the opposite way – far above and beyond and to the edges of imagination. It is my attempt at creating a polar extreme opposite to that video and the common notion that talking AI is the end-all, be-all of UI. The ultimate UI is no UI at all – it is a seamless meld of man and the machine.
this is by far the more interesting vision…
but how do you make a marketing-video of it? :D
@ Dr. Azrael Tod:
I know, I don’t think this would film well. It would just be a guy sitting in a room, eating a sandwich. Not quite impressive. :P
Also, this is a perfect example of why the written medium is so much more expressive than the visuals. Picture may be worth a thousand words, but carefully crafted narratives allow you to imagine and/or experience things that simply cannot be visualized.
Maaaaaan, to hell with pictures under glass. I want me some exo-brain.
Yes, absolutely, completely agree! I had the same thoughts not long ago, and coincidentally happened to start thinking about it when I needed to know the time but had no easy way to do so.
In a way, it’s similar to Matrix’s “I know Kung Fu” moment, although it would obviously had to be automated, more integrated and much faster. Your short time memory is read and changed in real time, giving it raw access to the digital information world.
It’s also why I find all those “digital 3D reality” futures so ridiculous; the writers are caught up in their own preconceived ideas on how we absorb and process information, so they assume we’d have to simulate them even in a pure digital medium.
One thing I didn’t consider in my thought experiment was communication. Being able to transmit much more than voice makes perfect sense, but I wonder if there wouldn’t be an “impedance mismatch” between the two parties; I mean, if the speech and body language don’t serve as an uniform interface between different patterns of thinking, and how could you augment that.
Good post!
I love this version of the future, but viruses and malware will be uploaded in our brains directly too :)
Also, I think that this vision requires much more than technological achievements – the whole perspective must shift to accept that. I have just read Windup Girl (thank you for that) and the whole thing about New People made me think hard on the subject.
I’m not going to lie Luke, I think you have something against Microsoft.
Their video was in short amazing in the perspective that it will actually happen- rather than being simply imaginary.
Anybody can make a video about an amazing world with no screens, etc. But I find this wasn’t Microsoft’s point anyway.
Whether you agree with their usefulness or not, the point of the video was to show what will be coming in the future, not what might; as in fact they already have invented half of the things displayed in the video.
I’m not arguing for whether it’s more functional or not, just that the video wasn’t about science fiction- and the title isn’t an indicator, just a selling point.
Sorry, addendum: the fact that I’ve thought about this recently is in no small part due to having been reading your blog; in many ways, it’s more significant than most of the sci-fi I’ve read, which has been mostly older stuff. I need to read more contemporary works, but they’re harder to come by around here. Even our bigger book stories only have a shelf for Sci-fi (and 20 or more for fantasy, which from what I can tell boils down to Meyer’s and endless clones).
@ icebrain:
Wow, very good point. This is a very interesting question that ought to be pondered at length – perhaps in it’s own post. That said, I imagine that there would be mediation protocols for this. When sending images or feelings they would be wrapped up in a digital package, indexed and annotated according to some algorithm. Upon receiving it would be unwrapped, and merged into your exo-memory with the intangibles being pointers to best-matching patterns stored in your memory. I guess fully abstract communication would be difficult for people who just met, but a married couple may build enough rapport, and shared experiences that could be cross linked to communicate wordlessly most of the time.
@ Victoria:
Yes, it is very scary! I’m sure there would be fail safes, but there would be also exploits around them. Have you ever seen Ghost in the Shell (movies, the series or the manga). They explore that very problem – in that universe people with fully prosthetic bodies, often get hacked and hijacked. Sometimes by kids out for a joyride, sometimes by terrorist groups. There is also one character in there who hacks peoples’s eyes in real time to become invisible.
It also reminds me of an old short story I once read, in which the hero jacks into the net via some neural implant, checks his messages and immediately pukes all over himself. As he cleans up, he remarks how he is glad that he wasn’t hit by the latest “shit your pants” exploit instead.
@ Andrew Zimmerman:
Yes, I know it’s a bad habit. When I was in HS and College Micrsosoft was the big and scary boogeyman who would FUD open source out of existence, and turn the technological clock back a decade any minute. Now they are sort of a big silly goof – like the dude from the Mac vs PC commercials. Good at corporate stuff, but completely incapable of getting ahead in the cool, hip markets like search, mobile technology, cloud computing, etc… Also Bill Gates is suddenly a good guy because he donates shitload of monies to charities. I still remember him as the great digital antichrist. Sometimes it is hard to turn those old notions off. :P
So yes, I do have a problem with Microsoft. Please forgive. I guess the video was cool on it’s own merit.
@ icebrain:
I know what you mean. My local Barnes & Nobile has a single section that lumps SF, Fantasy, Manga and few RPG related items together. Half the stuff there are Star Wars novels, Dragon Lance novels, Star Trek novels, and copious amounts of shitty fantasy. You actually have to look pretty hard to find something worthwhile.
That’s why I love Amazon. It has huge assortment of books in stock, and it also acts as a middleman for used book sellers for the stuff it does not currently sell directly. Sadly Amazon is probably not that useful if you live outside US.
Luke Maciak wrote:
GITS FTW!
Luke Maciak wrote:
Mmmm! Will we still be sitting and eat sandwiches by then?…
Luke Maciak wrote:
Of course, I’ve seen GITS and read manga :) It was a long time ago, right after the Matrix. It also reminds me how the agents in Matrix ‘borrowed’ people bodies – that too might become possible in the real world. Total control in the new age :)
I’m a little conflicted. On one hand I would love to clear up my ~200 book to-read list, on the other hand I love reading. I have to wonder how would it affect my entertainment experience if I instantly became aware of the contents of the book.
Liudvikas wrote:
Well the aviability of reading an excerpt didn’t strip you of the ability to read a book completely didn’t it? No reason to asume you would be forced to “know” the contents of every book when you can go the way to explore it on your own.
Even better: if you are allready manipulating your brain, why not read a good book, decide it was great to read it the first time, clear that area of your memory and read it the first time again? ;-)
@ Dr. Azrael Tod:
let me some kind of rephrase some of that to make a slightly different point…
What if you could know every one of your ~200 books to read, then decide what book you found really great of them and then read this exact book like you never would have known the story at all.
aditionally: why read a book, if you can live through the story like you would be the main character?
@ Dr. Azrael Tod:
Well books are specifically created as linear experiences.
@ Liudvikas:
…and you couldn’t do that with DNI? why?
As i see it, 75% of current computer-games are linear experiences too, although the medium does allow to do it otherwise (and a large bunch of games do). Movies are linear experiences too, but still differ completely from the way i see a good book. In my opinion a linear game is much nearer to a book then a movie is because, and not in spite of, the more space too let my mind wander, to see different angles and to experience the story on my own way.
The story is linear because it’s easiest to do this way, but would a non-linear book be less interesting then a ordinary one? (In fact you can really buy 1-2 non-linear books, where you make choices what should happen next, but this genre nearly died out with interactive media.)
Ok, I think you are not thinking deeply enough what Victoria is saying. I mean, a virus? Come on, no one is going to accept being “upgraded” if that implies being susceptible to viruses or hacking attacks. If a part of the system is going to need an OS, it is not going to be one that let these things happen, at least not the infections part. Data is also going to be safe because it will be encrypted using an algorithm based on a combination of your DNA and your brain waves’ patterns, so no one is going to have access to them unless you want it … well, maybe facebook, google and the neo-respawned digital-zombie Steve Jobs.
@ STop:
Sure, why not. Granted, you probably wouldn’t have a fridge in your house, because food would just be assembled in a nano-fab facility. But sandwiches may still be around.
@ Liudvikas:
Interesting point. I’m not sure how the experience would differ. I suspect that consuming data with your digital mind parts would feel different that actually absorbing it with your meat brain. Reading a book the old fashioned way would probably be more emotionally charged, and it would let you experience the buildup and katharsis whereas just side-loading the book into exo-memory would be more like reading a detailed synopsis, only it would be the actual book. So you would kinda know the contents all at once, and possibly miss out on the journey.
That said, if you could probably hack around it by linearly streaming data into over the interface between your exo-brain and meat-brain as fast as your mind can physically absorb it. So you could still get “traditional” experience in fraction of the time… Maybe.
@ Dr. Azrael Tod:
Excellent point. Direct neural interfaces like the ones we discussed would probably give rise to a whole new types of media. Think about this – a well crafted entertainment package could provide a complete life like experience – you could see, smell and feel everything in that world. You could feel the rage, sorrow and happiness of the characters. Hell, no reason why you couldn’t simultaneously experience the same story from the perspective of two or three different characters. It would be a whole new type of art we have not invented yet.
@ Agnosis:
Yes, but iPhones can still be rooted, and your encryption keys can still be social engineered out of you. Hell, if you can send data to it, there is a chance for a buffer overflow exploit somewhere on the line. So there is always a possibility of breach, even in supposedly secure closed systems – the more complex, the more space for trivial bugs to surface. And if there are bugs, there are exploits.
What Victoria said simply reminded me of Ghost in the Shell which dealt with these issues. Their cyber bodies did have security, encryption and security – and yet, there were still ways around them. Secret corporate back-doors for example. Or attacks against the weakest link – namely the meat-brain. Hell, I think at some point I have seen them use a Basslisk like idea – spam the visual cortex of the victim with seizure inducing, impossibly complex fractal imagery to cause a buffer overflow in an underlying optical processing software.
But you are right – it wouldn’t be as trivial as the shit we deal with today. So your relatives wouldn’t be carrying dozens of infections in their head on a daily basis.
@ Dr. Azrael Tod:
The camera zooms into his bionic eye and the matrix happens.
The main problem here is that the brain is a delicate structure. It can adapt to a lot of things, but that needs time, lot of training. Any such man-machine hybrid would have to be created artificially, with the machines interfacing with the wetware early on – embryo early.
If you have such powers, why remain ‘human’ at all? Why make sandwiches and go to work? It just seems redundant at that point.
@ peterix:
i did never say that i would want to stay human. If you can become something better, then why not? Eating Sandwitches is probably a bad example, it’s both, a source of energy (will need that in some way, so why not sandwitches?) and it’s some kind of thing you want to experience. Going to work on the other hand is something completely overrated. I don’t see why anyone would want something like this, be he human or something completely different.
Yeah, too much room for serious problems with neural interfaces, and you can’t just throw out an implant if you need to go off the grid. Though I must admit that I think uploading would become more and more attractive as the body begins to break down.
Luke Maciak wrote:
That reminds me of Transmet’s Buybombs: https://imgur.com/a/JTU1z
I wonder when will Google start offering free telepathic services in exchange for having to dream about ads ;)
@ peterix:
Well, that’s like asking why we still walk even though we have machines to ferry us around. We still walk for fun, for sport and for the hell of it. Eating is a pleasurable experience. I could see how people could possibly want to augment themselves to eliminate traditional waste evacuation, but why give up on eating regular food for the taste.
Granted, you could simulate the experience. Still, I suspect there would be enthusiasts more than willing to try the real thing.
@ Morghan:
True, but I suspect that you could flush/re-install the software that runs your wetware with anonymized credentials and drop off the grid that way. Maybe. Or just set of an EMP to fry it. Or get a quick surgical removal at the auto-doc booth on the corner. Or back-up and decant yourself a “clean” bio-body in the nearest full-body nano-fab tube.
@ icebrain:
TIL that I need to read Transmetropolitan. :P
Earlier today I wrote up a response to both this and your previous posting regarding this video. Check it out.
Ultimately I don’t think your (alternate) vision went far enough, but it’s in the right direction at least. Far better than the unimaginative Microsoft ad. While the brain will be the next interface, it just seems logical that the human body will eventually become obsolete as we embrace improving it. When we reach the stage of enhancing or replacing the human brain, there really is no limit to what mankind can achieve.
You had me at “sitting in a room, eating a sandwich”.
I like this vision, but itd make life a lot harder in some senses. Like you receive a call from your wife and you’re distracted and you think about some other woman, or something. It might be harder to mask feelings, like that you’re mad your boss is pestering you on a friday evening. There’s probably some other stuff that I could point out, thats just off the top of my head.
Would you really need to request communication with your wife?
You could have a pervasive, granular interest in being interrupted.
Happy to chat with anyone
Happy to chat with the wife only
Happy to make plans with the wife, but no mindless chatting etc
Happy to accept communications about Project Y from anyone
And how much discrete communication is needed? You would just already be transparently sharing your entire existence and thoughts to various friend groups. Total lifecasting of every moment, thought and sensation. Seamlessly tagged with the right privacy and notification setting. (Send it out /to/ someone, have it available if they ask for it etc)
I’m for modular external tech with a manual hardware interface. Of course I encrypt my grocery list. The thought of allow in a product to connect with my mind on more than a nerve connection for a replaced limb gives me countless nightmare scenarios and privacy/virus/rootkit concerns. I don’t trust any closed source project, or open source for that matter, but I rate them much higher.
The last thing I want is a Sony rootkit or some hack-job Microsoft/Apple OS implanted in me. I’m all for tech advances, but I want to be able to take it off at will and not have to worry about some program turning on network access at will or data mining my mind.
Call me paranoid, but if computers were DNI only I’d be living in a cave somewhere preaching about the dangers of technology.