What Is Augmented Reality?

AR enriches perception by adding virtual elements to the physical world

Augmented Reality
Coneyl Jay / Getty Images

If "augmented" is to mean anything increased or made better, then augmented reality (AR) can be understood as a form of virtual reality where the real world is expanded or enhanced in some way through the use of virtual elements.

AR can work in several different ways and is used for many different reasons, but in most cases, AR involves a scenario where virtual objects are overlaid and tracked atop real, physical objects to create the illusion that they’re in the same space.

AR devices have a display, input device, sensor, and processor. This can be accomplished through smartphones, monitors, head-mounted displays, eyeglasses, contact lenses, gaming consoles, and more. Sound and touch feedback can be included in an AR system as well.

Although AR is a form of VR, it is distinctly different in that unlike virtual reality where the entire experience is simulated, AR only uses some virtual aspects that are mixed with reality to form something different.

How Augmented Reality Works

Augmented reality is live, meaning that for it to work, it must allow the user to see the world as it is right now, and use that information to manipulate the space, pull information out of the environment, or alter the user’s perception of reality. This can be achieved in two ways…

One form of AR is when the user watches a live recording of the real world with virtual elements imposed on top of it.

Lots of sporting events utilize this type of AR where the user can watch the game live from their own TV but also see the scores overlaid on the game field.

The other type of AR is when the user can look around their environment normally apart from a screen but then a separate screen overlays information to create the augmented experience.

A prime example of this can be seen with Google Glass, which is like a regular pair of glasses but includes a small screen where the user can see GPS directions, check the weather, send photos, etc.

Once something virtual has been placed between the user and the real world, object recognition and computer vision can be used to allow the object to be manipulated by actual physical objects as well as let the user interact with the virtual elements using physical objects.

One example of the former includes mobile apps from retailers where the user can pick a virtual object of something they’re interested in buying, and then stick it into the real world through their phone. They can see their actual living room, for example, but the virtual couch they’ve chosen is now visible to them through their screen, letting them decide if it will fit in that room, which color best matches the room, etc.

An example of the latter where a physical element invokes something virtual, can be seen with mobile apps that can scan objects or special codes that the user can then interact with on their own screen. Retail apps might use this form of AR to let their customers read more information about a physical product before they buy it, see reviews from other buyers, or check what’s inside their unopened package.

Types of Augmented Reality Systems

There are a few kinds of AR implementations that all follow the same rules mentioned above, and some augmented reality devices might use some or all of them:

Marker and Markerless AR

When object recognition is used with augmented reality, the system recognizes what's being seen and then uses that information to react with the AR device. It's only when a specific marker is visible to the device that the user can interact with it to complete the AR experience.

These markers might be QR codes, serial numbers, or any other object that can be isolated from its environment for the camera to see.

Once registered, the augmented reality device might overlay information from that marker directly on the screen or open a link, play a sound, etc.

Markerless augmented reality is when the system uses a location or position-based anchor point, like the compass, GPS, or accelerometer. These types of augmented reality systems are implemented when location is key, like for navigation AR.

Layered AR

This type of AR is when the augmented reality device uses object recognition to identify the physical space, and then overlay virtual information on top of it.

Lots of popular AR devices use this form. It's how you can try on virtual clothes, display navigation steps in front of you, check whether a new piece of furniture can fit in your house, put on fun tattoos or masks, etc.

Projection AR

This might seem at first identical to layered, or superimposed augmented reality, but it's different in one specific way: actual light is projected onto a surface to simulate a physical object. Another way to think of projection AR is as a hologram.

One specific use for this kind of augmented reality might be to project a keypad or keyboard directly onto a surface so that you can press buttons or interact with the virtual items using real physical objects.

Augmented Reality Applications

There are several advantages to using augmented reality in areas like medicine, tourism, the workplace, maintenance, advertising, the military, and the following:

Education

In some senses, it can be easier and even more fun to learn with augmented reality, and there are tons of AR apps that can facilitate that.

A pair of glasses or a smartphone is normally all you need to learn more about physical objects around you, like paintings or books.

One example of a free AR app is SkyView, which lets you point your phone to the sky or ground and see where stars, satellites, planets, and constellations are located at that exact moment, both during the day and at night.

SkyView is considered a layered augmented reality app that uses GPS because it shows you the real world around you, like trees and other people, but also uses your location and the current time to teach you where these objects are located and give you more information about each of them.

Google Translate is another example of an AR app useful for learning. With it, you can scan text that you don’t understand and it will translate it for you in real time.

Navigation

Displaying navigation routes against a windshield or through a headset delivers augmented instructions for drivers, bicyclists, and other travellers so that they don’t have to look down at their GPS device or smartphone just to see which road to take ahead.

Pilots might use an AR system to display transparent speed and altitude markers directly within their line of sight for much the same reason.

Another use for an AR navigation app might be to overlay a restaurant’s ratings, customer comments, or menu items right on top of the building before you go inside, so that you can avoid having to search for those things online. Or maybe the augmented reality system will show the quickest route to the nearest Italian restaurant as you walk through an unfamiliar city.

Other GPS AR apps like Car Finder AR can be used to find your parked car, or a holographic GPS system like WayRay might overlay directions right on the road in front of you.

Games

There are lots of AR games and AR toys that can merge the physical and virtual world, and they come in many different forms for lots of devices.

One well-known example is Snapchat, which lets you use your smartphone to overlay fun masks and designs on your face before sending a message. The app uses a live version of your face to put a virtual image on top of it.

Other examples of augmented reality games include Pokemon GO!, INKHUNTER, Sharks in the Park (Android and iOS), SketchAR, Temple Treasure Hunt Game, and Quiver. See these AR iPhone games for more.

What Is Mixed Reality?

As the name very clearly suggests, mixed reality (MR) is when real and virtual environments are mixed together to form a hybrid reality. MR uses elements of both virtual reality and augmented reality to create something new.

It's tough to categorize MR as anything but augmented reality since the way it works is by overlaying virtual elements directly onto the real world, letting you see both at the same time, very much like AR.

However, one primary focus with mixed reality is that the objects are anchored to real, physical objects that are fully interactable in real time. This means that MR could achieve things like allowing virtual characters to sit in the actual chairs in the room, or for virtual rain to fall and hit the actual ground with life-like physics.

The basic idea behind mixed reality is to allow the user to seamlessly exist between both a real state with the real objects around them, and the virtual world with software-rendered objects interacting with them to create a fully immersive experience.

This Microsoft HoloLens demo video is a perfect example of what is meant by mixed reality.