What Is Augmented Reality?

AR enriches perception with virtual elements in the physical world

Augmented Reality
Coneyl Jay / Getty Images

The word "augment" means to increase, extend, or make better. Augmented reality (AR) can be understood as a form of virtual reality (VR) where the real world is expanded or enhanced through the use of virtual elements, usually overlaying those elements on the view of the real world through the use of a visual device.

AR can work in several different ways and is used for many different reasons, but in most cases AR virtual objects are overlaid on and tracked in a view of the real world, creating the illusion that they occupy the same space. AR devices have a display, input device, sensor, and processor. These devices can be monitors, head-mounted displays, eyeglasses, contact lenses, gaming consoles, and even smartphones, among others. Sound and touch feedback can be included in an AR system as well as through other non-visual methods and devices.

Although AR is a form of VR, it is distinctly different. Virtual reality is an entire experience that is entirely simulated — both the view of "reality" and the objects in it—while AR only uses some virtual aspects, which are mixed into reality to form something different.

How Augmented Reality Works

Augmented reality is live. For it to work, the user must be able to see the real world as it is right now. AR manipulates the real world space the user sees, altering the user’s perception of reality.

One form of AR, the user watches a live recording of the real world with virtual elements imposed on top of it. Lots of sporting events utilize this type of AR; the viewer can watch the game live from their own TV but also see the scores overlaid on the game field.

Another form of AR allows the user to look around their environment normally and in real time, but through a display that overlays information to create the augmented experience. An example of this is Google Glass, which is a device that appears much like a regular pair of glasses but includes a small screen on which the user can see GPS directions, check the weather, send photos, and many other functions.

When a virtual object is placed between the user and the real world, object recognition and computer vision can be used to allow the object to be manipulated by actual physical objects, and allow the user to interact with the virtual elements.

For example, some retailer mobile apps allow shoppers to select a virtual version of something they're considering purchasing, such as a piece of furniture and view it in the real space of their home through their phone. They can see their actual living room, for example, but the virtual couch they’ve chosen is now visible to them through their screen, letting them decide if it will fit in that room and if they like the look of it within that room.

Another example allows customers to scan products or special codes (such as UPC symbols) that use AR to show the customer more information about a physical product before they buy it, see reviews from other buyers, or check what’s inside their unopened package.

Marker and markerless AR

When object recognition is used with augmented reality, the system recognizes what's being seen and then uses that information to engage the AR device. It's only when a specific marker is visible to the device that the user can interact with it to complete the AR experience.

These markers might be QR codes, serial numbers, or any other object that can be isolated from its environment for the camera to see. Once registered, the augmented reality device might overlay information from that marker directly on the screen or open a link, play a sound, etc.

Markerless augmented reality allows a system to use a location or position-based anchor point, like the compass, GPS, or accelerometer. These types of augmented reality systems are implemented when location is key, such as with navigation AR.

Layered AR

This type of AR uses a device uses to recognize a physical space and then overlay virtual information on top of it. It's how you can try on virtual clothes, display navigation steps in front of you, check whether a new piece of furniture will fit in your house, put on fun tattoos, and more.

Projection AR

This might seem at first to be the same as layered, or superimposed augmented reality, but it's different in one specific way: actual light is projected onto a surface to simulate a physical object. Another way to think of projection AR is as a hologram.

One specific use for this kind of augmented reality might be to project a keypad or keyboard directly onto a surface allowing a user to type using the projected virtual keyboard.

There are many advantages to using augmented reality in areas like medicine, tourism, the workplace, maintenance, advertising, the military, and others.

AR in education

In some senses, it can be easier and more fun to learn with augmented reality, and there are tons of AR apps that can facilitate that. A pair of glasses or a smartphone can be all you need to learn more about physical objects around you, like paintings or books.

One example of a free AR app is SkyView, which allows you to point your phone at the sky or the ground and see where stars, satellites, planets, and constellations are located at that exact moment, both during the day, at night, and from the opposite side of the planet.

SkyView is considered a layered augmented reality app that uses GPS. It shows you the real world around you, like trees and other people, but also uses your location and the current time to teach you where these objects are located and give you more information about each of them.

Google Translate is another example of an AR app useful for learning. With it, you can scan text in a language you don’t understand, and it will translate it for you in real time.

AR in navigation

Displaying navigation routes against a windshield or through a headset delivers augmented instructions for drivers, bicyclists, and other travelers so that they don’t have to look down at their GPS device or smartphone just to see which road to take ahead. Pilots might use an AR system to display transparent speed and altitude markers directly within their line of sight for much the same reason.

Another use for an AR navigation app might be to overlay a restaurant’s ratings, customer comments, or menu items right on top of the building before you go inside. It might also show you the quickest route to the nearest Italian restaurant as you walk through an unfamiliar city.

GPS AR apps like Car Finder AR can be used to find your parked car, or a holographic GPS system like WayRay might overlay directions right on the road in front of you.

AR in games

There are lots of AR games and AR toys that can merge the physical and virtual world, and they come in many different forms for lots of devices. One well-known example is Snapchat, which lets users overlay fun masks and designs on their faces before sending a message through their smartphones. The app uses a live version of your face to put a virtual image on top of it.

Other examples of augmented reality games include Pokemon GO!, INKHUNTER, Sharks in the Park (Android and iOS), SketchAR, Temple Treasure Hunt Game, and Quiver.

What is mixed reality?

Mixed reality (MR), as the name suggests, mixes real and virtual environments to form a hybrid reality. MR uses elements of both virtual reality and augmented reality to create something new. It's tough to categorize MR as anything but augmented reality since the way it works is by overlaying virtual elements directly onto the real world, letting you see both at the same time, very much like AR.

However, one primary focus with mixed reality is that the objects are anchored to real, physical objects that may be interacted with in real time. This means that MR could allow virtual characters to sit in the actual chairs in a room, or for the virtual rain to fall and hit the actual ground with life-like physics.

Mixed reality lets the user seamlessly exist in both a real state with the real objects around them and the virtual world with software-rendered objects interacting with real-world objects to create a fully immersive experience. The Microsoft HoloLens demonstration is a good example of what is meant by mixed reality.