Augmented Reality is something we’ve seen cropping up more and more, but what exactly is it and how is it done? Everyone is familiar with Virtual Reality thanks to video games and movies like Ready Player One. In Virtual Reality you don a VR headset and experience a completely synthetic reality, removing you from the physical world and allowing you to immerse yourself in a totally different world. Meanwhile, Augmented Reality blurs the line between virtual and real. It allows the user to experience artificial content as part of their physical surroundings and is set to take ecommerce by storm, as Shopify is embracing the tech to enable AR for its store owners.
Recently Shopify integrated the use of Apple’s ARKit to provide AR content for its users and we at Eastside Co have been experimenting with it since. Most recently we worked with Lazy Oaf to deliver augmented reality versions of their new line of shoes developed in partnership with the legendary Doc Martens. Bringing the unique aesthetic of Lazy Oaf’s designs to augmented reality through 3D modelling was a fun challenge and, I believe, has added a level of interactivity to shopping online that is close to what was previously only available in retail stores.
So what goes into making a virtual product for AR and what are the problems that crop up along the way?
Getting Set up
Lazy Oaf were kind enough to send us the actual products pre-launch so I was able to get a good feel as to what the products were like. This can be hugely beneficial as sometimes photos just can’t capture the types or feel of materials used in the product and that can be important to making the product look right further on down the line. Our photographer and videographer, Dave, kindly photographed each of the Doc Marten shoes from the front, back, both sides, top and bottom as well as from various angles to help capture some details of the surfaces before I started the 3D modelling process.
Once I’d cropped and scaled the photos to be accurate to each other I laid them out as blueprints in 3DS Max 2018.
Basic Modelling
From here the modelling begins with simple shapes. The shoe began life as a single plane that was extruded and manipulated to fit the shape of the sole.
Once the basic shape was complete I cut in the edges that would make up the tread of the sole and extruded them followed by the rest of the sole, applying some modifiers to get the rounded bottom of the sole as a whole. After some further basic modelling the sole was ready to take into ZBrush to do the high resolution work.
High Resolution 3D Modelling
Taking a low resolution model and increasing the number of polygons smooths the model out. This process was done to a few levels giving a much smoother surface which enabled me to add stitches along the brim of the sole and manually hand sculpt the stitch holes into the surface to make it appear as though they were proper stitches.
Once the sole was smoothed out and polished up I started on the upper part of the shoe, creating a basic shape that followed the outline of the shoe itself.
Using this rough version as a base I was able to extract the shapes of all the pieces that made up the detail of the shoe. All the overlapping pieces of material that made the form were pulled from this shape and arranged accordingly. The laced area on the front of the toe was sculpted in.
From this stage it was a case of adding the accessories like the buckles and laces on the toe and of course more small details like the stitches for the panels.
The Hard Part
With some colour that model is surely finished, right? Unfortunately not. The Doc Marten shoe 3D model as a whole is made up of just under 10 million polygons. Even though the smartphones we carry in our pockets now are incredibly powerful computing devices, they would have a very hard time rendering that many polygons all at once. Much like a movie editor, you have to pick and choose the important parts of what it is you’re trying to present. You will notice that the inside of the shoe isn’t finished. This was the first piece that hit the cutting room floor, so to speak. The interior of the shoe held no significant importance to the overall aesthetic and so not modelling it reduced the complexity of the model (as well as saving me some time). But we’re still left with almost 10 million polygons.
Because augmented reality renders the model in real time we have to reduce the complexity. This involves reducing the polygon count down to manageable levels and it’s an incredibly tedious process. Fortunately most of the parts began life as low polygon versions and I was able to return to those parts and repurpose them for this task. In the screenshot below you can see that while the shape is representative of the shoe itself it lacks the finer details such as the stitches, slightly bevelled edges and the general smoothness that all those polygons afforded. However, you can also see that the polygon count has been reduced from almost 10 million to just over 6000 and could actually be reduced a little bit further if necessary, but 6000 polygons is well within budget and having geometry to support the shape can be handy for the next stage.
Bringing the Details Back
So now we have two versions of the shoe: an extremely high resolution version, and a low resolution version that will render fast and smooth on mobile hardware. But now we’re missing the details that I would argue make the shoe unique.
In order to transfer the high resolution details to the low resolution version we need to “bake” them into various texture maps that provide information to the virtual light in our augmented reality. To do so we first need to unwrap our model and flatten it out, creating a UV map. What is a UV Map? It’s a 2D representation of our model if it was unfolded like an origami model; it allows textures to be projected onto its surface later down the line.
To keep the details looking crisp and clean I decided to split the model into two parts. The sole, buckles and straps in one part, and then the upper part into another piece. This enabled me to use 2 small texture maps instead of a single large one which leads to a smaller file size so it loads quicker.
Below you can see the UV Maps - the flattened versions of both parts.
Once this is done we can bake our maps and work on the textures, adding the material information that will make the product look like the product.
The purple in the screenshot above may look strange but this is the main way we bring the detail back into our model. The surface of the high resolution model is projected onto the lower resolution model and recorded in values of red, green, or blue depending on which way the surface is pointing. When applied to the model and a light source hits it, the colour of the pixel that the light hits determines the direction the light bounces off, creating an artificially 3D surface without the polygon cost. This is most obvious with the stitches.
Once these maps have been created it is now just a case of applying materials and texture to the surface.
Getting Physical
Modern day realtime renderers use a system called PBR or Physical Based Rendering. This means that the lighting and rendering engine tries to accurately map the way light behaves when hitting the surface of a model based on what material that surface is. Modern texturing software has made creating realistic materials very easy. Substance Painter was instrumental in enabling the surfaces of these real life items to be recreated digitally. We primarily use 5 different texture maps to apply various properties to the model:
- Colour Map This is as it sounds, it is the base colour of the material.
- Metalness This is a black and white map that determines which part of the model is metal or non-metal.
- Roughness Just as it sounds, it defines how rough or smooth surfaces of the model are.
- Ambient occlusion A greyscale map that imitates those tiny little contact shadows where two surfaces meet in the corner, it adds extra depth to the lighting.
- Normal Map This is the purple map shown earlier and it is responsible for showing the detail of the surface - as shown below there is barely any detail when turning the normal map off on the sole and then the upper part of the shoe.
Once the texturing process is complete we have something that looks rather convincing. The final steps are to export the model and textures and convert them to an AR friendly format. In this case it is USDZ as that is the format that Apple ARKit uses.
If you’re still with me, well done! The result of AR is far more fascinating than the process of making it, but I hope this little glimpse behind the scenes shows how much goes into the process of manually recreating a virtual representation of a real product. New technologies such as 3D scanning and photogrammetry will make the process easier in the future but right now the quality isn’t quite there unless you invest a large sum of money into the technology.
We’re still in the early days of augmented reality technology - as it continues to advance and as mobile phones become more powerful and capable we’ll begin to see this tech more and more in everyday places. While Ready Player One and other sci-fi movies popularised Virtual Reality, I believe VR will remain in the realm of entertainment and education whereas AR will become as pervasive as the internet - especially when we all get our cybernetic eye upgrades. Maybe that’s a bit of a way off but until then at least I can put some shoes on the table and not get told off.
Visit Lazy Oaf and check out the Dr Marten shoes on your own iPhone or iPad running iOS and Safari 12.
