{"id":38407,"date":"2017-05-09T06:00:39","date_gmt":"2017-05-09T10:00:39","guid":{"rendered":"http:\/\/www.shamusyoung.com\/twentysidedtale\/?p=38407"},"modified":"2017-05-09T06:12:10","modified_gmt":"2017-05-09T10:12:10","slug":"zenimax-vs-facebook-part-2-the-5-problems-of-vr","status":"publish","type":"post","link":"https:\/\/www.shamusyoung.com\/twentysidedtale\/?p=38407","title":{"rendered":"Zenimax vs. Facebook Part 2: The 5 Problems Of VR"},"content":{"rendered":"

Disclaimer: Like I said at the start of this series, I am not a lawyer. This is a complicated case and I am not an expert on the law, VR, or corporate contracts. I’m working with incomplete records of complex events where there was often more than two sides to every story. I’ve done what I could to be accurate, but series is intended as opinion and commentary, not authoritative historical record.<\/em><\/p>\n

VR is a strange thing. For people who haven’t tried it, it’s natural to assume this is just another technological advance like plasma screens or surround sound. They think this is just the next step up in fidelity. <\/p>\n

This is not the case. VR is as different from looking at a screen as a screen is different from a radio. VR engages parts of the brain that aren’t really involved or excited by traditional screen experiences. <\/p>\n

Presence<\/h3>\n

A screen grab of the VR demo at Valve in 2014. This is back when they were still using the Oculus Rift, before they developed the Vive, their own competing headset.<\/div>
A screen grab of the VR demo at Valve in 2014. This is back when they were still using the Oculus Rift, before they developed the Vive, their own competing headset.<\/div><\/p>\n

A notable example is one that Valve was offering in its VR labs in 2014. In the demo, the user would find themselves standing on a narrow stone platform floating in a vast open space. The space wasn’t even designed to look real. The skybox was comprised of old webpages. The platform texture looked like something out of Half-Life 2. If you looked at this on a traditional screen it would be incredibly boring. It looks like “Baby’s First Game Level”. It’s cheap and dull and you wouldn’t give it a second look.<\/p>\n

But in VR this stupid box room can be a visceral experience. If you’re at all nervous around heights then you’ll probably catch your breath, feel your knees lock up, and have an intense desire to grab onto something solid. You know you’re in a VR lab and you know it’s just a simulation, but the input reaches deep down and tickles the atavistic parts of your brain. You can see a similar idea at work in the Fear of Heights<\/a> VR demo. While FoH makes for a better demo to watch, I think the Valve demo makes the more dramatic case for VR, since it accomplishes the same effect using only rudimentary visuals. It manages to convince you using unconvincing graphics, thus driving home just how different it is from traditional screen experiences.<\/p>\n

This feeling of “being there” is called presence, and it’s only possible in VR. This effect isn’t a novelty. It persists, even in people who use VR regularly.<\/p>\n

This is good, because it makes VR an amazing product with new possibilities. But it’s also bad, because it’s very difficult to make people understand how different it is. You can’t just show it on television or have them download a demo. If you want someone to understand how amazing VR is, then you need to stick a VR headset on their noggin and stand back.<\/p>\n

In April of 2012 John Carmack reached out to Oculus and asked if he could try their prototype, called the Rift. Palmer Luckey – being a fan – sent him one. It was one of only two prototypes in existence. It’s entirely possible this was the best VR headset in the world.<\/p>\n

The Rift was good, but like so many times during the evolution of VR, this breakthrough only revealed that the VR was a little more complicated and challenging than everyone anticipated. There were several problems that would need to be solved before VR would be ready for the world.<\/p>\n

Problem #1: Field of View<\/h3>\n

This was what passed for VR in the 1970s. The past sucked.<\/div>
This was what passed for VR in the 1970s. The past sucked.<\/div><\/p>\n

The whole point of VR is to envelop you in a virtual world, which means the scenery needs to fill as much of your field of view as possible. You might notice that bringing up a photograph of a relaxing mountain scene on your mobile phone and mashing your face against the screen does not produce the desired effect. You can’t comfortably focus on things that close to your eyes, and even if you could it would ruin the sense of depth and being enveloped in the virtual world.<\/p>\n

This problem is easy to fix with lenses, assuming you don’t mind cutting off the user’s field of view. You can create something that’s basically a View Master<\/a>. You’ll have a comfortable view of things that appear to be in the distance, but you’ll have the same field of view of someone looking through binoculars. The drawback is that this pretty much ruins any sense of presence.<\/p>\n

Programmer Michael Abrash gave a presentation at Steam Dev Days in 2014, talking about this exact problem<\/a>. He pointed out that if you wanted to properly<\/em> bend a rendered image to fill the user’s field of view, it would require a complex chain of nine precise lenses, the largest of which would be over a foot in diameter. That’s obviously not the sort of thing you can comfortably wear on your face.<\/p>\n

The lenses required for un-distorted VR. Taken from the Abrash talk linked above.<\/div>
The lenses required for un-distorted VR. Taken from the Abrash talk linked above.<\/div><\/p>\n

Of course, that’s what you get if you want a perfect, undistorted image delivered to the eye. Palmer Luckey experimented with different lens arrangements and came up with a system that allowed the image fill the user’s view using only two<\/strong> sets of lenses. The lenses were reasonably small and lightweight and could fit within the expected volume of a VR headset.<\/p>\n

The problem with this solution is that bending an image that aggressively will inevitably cause distortions. When wearing the Rift prototype, the users would see the world as if through a fisheye lens<\/a>Actually, I think Luckey’s lens creates the OPPOSITE of a fisheye distortion, and fisheye is used to correct it. This is hard to confirm because I don’t think we have an accepted term for whatever the opposite of a fisheye lens is.<\/span>. <\/p>\n

Once he was able to try the headset for himself, Carmack was able to solve this problem on the software side. If the lenses create a fisheye effect, you can negate this by simplyActually it’s not simple at all.<\/span> creating an image with the opposite distortion so that it is “corrected” after passing through the lens.<\/p>\n

Problem #2: Chromatic Aberration<\/h3>\n

You can see the effect is strongest at the edges of the view, and basically vanishes in the center.<\/div>
You can see the effect is strongest at the edges of the view, and basically vanishes in the center.<\/div><\/p>\n

Unfortunately, the strong lenses have other side effects. Different wavelengths of light bend at different angles. Which means that while wearing the headset, the user will see the various color ranges appear displaced from one another. By coincidence, this looks sort of like stereoscopic images created using the red\/blue anaglyph system in old 3D movies. But the two things are actually unrelated. In fact, in an anaglyph 3D image the color separation is required for the effect to work, while in VR the color separation kind of ruins itOr just makes it crappy.<\/span>.<\/p>\n

Again, the solution seems to be to correct for the lens behavior in software. You can have the software render the image with the color ranges displaced in the opposite direction, so that after passing through the lens they will be properly re-combined into the final whole.<\/p>\n

This explains why VR screenshots always have those strange blurry color auras around them, even though people wearing VR headsets don’t see that effect.<\/p>\n

Problem #3: Head Tracking<\/h3>\n

The problem here is that your head doesn't pivot perfectly around an axis located between your eyes. When you tilt your head forward, you also MOVE your head forward, and the distance moved varies from person to person.<\/div>
The problem here is that your head doesn't pivot perfectly around an axis located between your eyes. When you tilt your head forward, you also MOVE your head forward, and the distance moved varies from person to person.<\/div><\/p>\n

I owned a Devkit 2<\/a>, the second-generation headset from Oculus. The original Rift didn’t have positional head tracking. If you turned your head, the virtual view would turn as expected. But if you moved<\/em> your head to the side then the world would feel like it was moving with you, as if the whole world was strapped to your face. (Because, you know, it was<\/b>.) <\/p>\n

With my Devkit 2, occasionally I’d move outside of the active area for head tracking and it would stop working. This gave me a few seconds to experience what the original Rift must have been like. The moment this happened, it would instantly bring waves of VR sickness.<\/p>\n

What does VR sickness feel like? It’s a bit like a headache. It’s also a bit like being dizzy, with maybe a bit of nausea. It feels kind of<\/b> like all of those things, but it really is a distinct sensation. Regardless of how you describe the sensation, it feels terrible. <\/p>\n

Worse, VR sickness can linger. You might start to feel a little uncomfortable in the simulation, so you take the headset off. But the problem doesn’t vanish immediately. If you continue to engage even after symptoms begin, the VR sickness might hang around for hours. It depends on the person.<\/p>\n

In any case, head tracking is a key component of avoiding VR sickness. While everyone has different tolerances for VR and experiences differing levels of VR discomfort, I think Devkit 2 – the version of the Rift with head tracking – represented the Minimum viable product<\/a> for VR. It had just enough technology to deliver on the basic premise of “being there” and giving that sense of presence. While a small minority of people can enjoy VR without head tracking, the vast majority of us need it to avoid sickness.<\/p>\n

Problem #4: Latency<\/h3>\n

The user's head moves. Then the camera takes a snapshot of the user. Then the computer processes the camera view and figures out where the user's head is now. Then a new frame is rendered at the updated position. Then it shows up on the screen. There are a lot of steps involved, and the longer it takes the worse it all feels.<\/div>
The user's head moves. Then the camera takes a snapshot of the user. Then the computer processes the camera view and figures out where the user's head is now. Then a new frame is rendered at the updated position. Then it shows up on the screen. There are a lot of steps involved, and the longer it takes the worse it all feels.<\/div><\/p>\n

Imagine you’re wearing a VR headset that, for some reason, is only rendering things at one frame a secondProtip: Don’t actually do this!<\/span>. You’re looking directly ahead at a virtual lamppost or some other landmark. Then you turn your head to the left. Because you’ve got such a horrible framerate, the display doesn’t update as you move your head. Instead, the lamppost remains in the center of the screen, which means that it will appear to “come with you”, always hovering directly in front of your eyes. (This will probably cause the VR sickness I mentioned above.) Then finally the rendering catches up. The headset updates, and the lamppost is suddenly on the right side of your field of view, where it should be. To someone wearing the headset, it feels like the lamppost floated to the left and then abruptly jumped over to the right.<\/p>\n

As you improve framerate this effect will diminish, but it’s very hard to get it to go away entirely. Even at 60fps, it means the lamppost will float to the left for 16 milliseconds and then appear to jump a tiny bit to the right as the display catches up. To someone wearing the headset, it feels like the lamppost is sort of “vibrating” as they turn their head side-to-side. This effect is called judder, and it’s hard to get rid of.<\/p>\n

You might think that this problem can be solved by buying a faster graphics card or rendering simpler scenes, but that’s not the case. Better graphics cards get us more throughput, but we need better (lower) latency.<\/p>\n

I need to deliver all these pixels on time!<\/div>
I need to deliver all these pixels on time!<\/div><\/p>\n

Let’s say Bob Nvidia is running a shipping company. His trucks can deliver me 1,000 packages by next week. Pretty soon he upgrades his fleet and now he can deliver 2,000 packages a week. A year after that, he can deliver 4,000 packages in a week.<\/p>\n

Now imagine I don’t necessarily need a lot of packages, but I need them tomorrow<\/b>. That’s the latency problem. The system is optimized for volume and now we’re trying to do something faster than anyone would have thought was reasonable when the system was designed. In 2012, most graphics cards, graphics drivers, display screens, and games had been designed under the assumption that the user wouldn’t have any use for more than sixty frames per second. <\/p>\n

I’m 45 years old. When I was younger I could really tell the difference between 30fps and 60fps. But at this age the difference is pretty slight and I often don’t notice. As long as it’s consistent, I’m fine. But even my worn-out eyeballs are sensitive to framerate in VR. 30fps suffers from horrible judder that makes me want to slam my eyes closed when I turn my head. I would say that 60fps is basically tolerable, but only if I’m playing something slow-paced. For an action game, I’d probably want something closer to 90 or even 120fps.<\/p>\n

To get the latency down to something comfortable, Carmack had to dig down in the rendering layer, look for bottlenecks, and then figure out ways to get around them. The raw power (in throughput) was there, but since there wasn’t previously a demand for extremely low latency rendering almost nobody had paid attention to this stuff until now.<\/p>\n

Problem #5: Center of Projection<\/h3>\n

A lot of processing power is spent rendering two views that are almost - but not quite - entirely similar.<\/div>
A lot of processing power is spent rendering two views that are almost - but not quite - entirely similar.<\/div><\/p>\n

According to the Zenimax complaint, there was a fifth problem that Carmack solved, which was the “center of projection” problem. This could mean a lot of things and I’m not sure what they’re talking about specifically. I’m sure this has something to do with how to position and orient the user’s virtual eyes within the simulation. <\/p>\n

The software needs to render two different views – one for each eye. You can think of their virtual eyes as a pair of cameras floating around the world. If you aim both cameras in exactly the same direction then the 3D effect won’t quite work as expected. You won’t get the feeling of close objects floating right in front of your face. To get that, you need to angle the cameras inward – to make them slightly cross-eyed as it were – to make objects “pop”. But hang on – isn’t that something that, in the real world, they do with their own eyes? If we render a cross-eyed view and then they cross their eyes to look at it, won’t that be… wrong? But we can’t cross their eyes for them and we can’t track where their eyeballs are looking, so how do we know where to put the cameras? Hm.<\/p>\n

This is complicated stuff<\/a> and there aren’t necessarily obvious answers. I wasn’t aware that Carmack made any particular breakthrough in this area. I’m not really disputing this claim. I’m just saying that I can’t nail down what specific<\/em> advance they might be talking about in this instance.<\/p>\n

And then the executives took credit for everything anyway.<\/div>
And then the executives took credit for everything anyway.<\/div><\/p>\n

I said above that Carmack solved these problems, but in truth it’s not at all clear who<\/em> did the solving, which is important because that’s basically what this case is all about. Zenimax is trying to make the case that once Carmack had his hands on the prototype, all further advancements were his work alone. <\/p>\n

But this was a collaboration between a hardware engineer and a programmer regarding a product that requires perfect integration of hardware and software. It would be a pretty big stretch to give either party credit for the whole thing. <\/p>\n

For example, Luckey was certainly aware of problems #1, #2, and #3. Possibly he envisioned the solution himself, but lacked the coding expertise to realize it. In which case, Luckey was the “inventor” and Carmack was simply the engineer who followed the blueprint. Carmack’s contibutions were no doubt significant, but it’s tough to prove<\/strong> he did any particular thing aside from write code. And since he now works for Oculus, Zenimax probably isn’t interested in asking either Palmer Luckey or John Carmack who did the heavy lifting when it comes to new ideas.<\/p>\n

On the other hand, innovation #4 is most certainly Carmack’s work. There probably aren’t many people in the world more uniquely qualified to optimize a rendering pipeline for low latency. Carmack even did some original research a few years earlier during the development of Quake Live<\/a>, trying to figure out just where all the processor cycles went between the moment the user pushes a button and the moment the result shows up on screen. <\/p>\n

This brings us to the details of the Zenimax complaint. We’ll get into that next time.<\/p>\n","protected":false},"excerpt":{"rendered":"

Disclaimer: Like I said at the start of this series, I am not a lawyer. This is a complicated case and I am not an expert on the law, VR, or corporate contracts. I’m working with incomplete records of complex events where there was often more than two sides to every story. I’ve done what […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[102],"tags":[],"class_list":["post-38407","post","type-post","status-publish","format-standard","hentry","category-weekly-column"],"_links":{"self":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/38407","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=38407"}],"version-history":[{"count":0,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/38407\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=38407"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=38407"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=38407"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}