One of the main issues with processing images for Virtual Reality is the intense workload it places on the computer’s GPU. With current steroscopic VR setups, there are two slightly different copies of each video image rendering each frame. One image is sent to one eye while the other image is sent to the other eye to get the 3D effect. Additionally, these images have to be refreshed a very high rate to keep the experience smooth and without judder. For VR applications, the refresh rate has to be even higher than normal because the image takes up our whole field of view and immerses us in the environment. This causes us to notice much more subtle differences that can lead to VR sickness. The goal for most VR companies has been to render both images at 90 Hz, which would mean 90 frames a second for each image. This is what the consumer version of the Oculus Rift and the HTC Vive will be running at. Needless to say, running two images, one for each eye at 90 frames per second is very processing intensive for a GPU. This is also the reason, the spec requirements are so high for the computers needed to run good VR headsets. The high specs and extra costs keeps VR out of the hands of many consumers.
Foveated Rendering looks to solve this issue by greatly reducing the processing power needed by the GPU to render the two images at 90Hz. The idea behind foveated rendering is that only the area the eye is focused on is rendered in full resolution. A bigger area surrounding the focused area is rendered in just a medium resolution, and the remaining areas of the screen are rendered in a low resolution. This enables the GPU to very rapidly render the low and medium resolution areas while only spending time to render the high resolution areas on a small portion of each image. The image above provides a example of the different layers of resolution that would be rendered for the image. This technique simulates the way we use our eyes where the area we are focused on is clear while our peripheral vision is blurred.
In addition to the rendering technique, foveated rendering also requires eye-tracking technology. As the user’s focus moves around a screen, it is neccesary to see where the user’s eye is focused so the system knows where to render the medium and high resolution areas. Obviously, you can imagine the eye-tracking and the rendering need to both happen very quickly to pull this off. Earlier this month, SensoMotoric Instruments (SMI), a German company specializing in eye-tracking technology has introduced their consumer-ready eye-tracking platform with the ability to do this as 250Hz, or 250 frames per second. Oculus has already reported they are working on eye-tracking technology for their 2nd generation headset, and I would expect to see foveated rendering incorporated. If so, this will go a long way in allowing more detailed graphics to be displayed but also in reducing the required specs a computer needs to display VR. This would make VR accesible to even more people. The video below gives a more detailed look at SMI’s foveated rendering in action.