Outline shader techniques in Unity

Outlines are drawn on objects to visualize an object selection status, and constitute a main part of every toon shader. In this blog post, we describe some techniques to implement outline shaders in Unity.

Basic techniques

A popular technique is to draw the outline from the object vertices by translating these vertices along their normals. This technique requires 2 passes.
In the first pass, the object vertices are translated outwards along their normals and are rendered using the outline color. In this pass, front culling is used, so only the inner faces of the outline are shown. In the second pass, the object itself is rendered using back culling.
This technique is easy to implement and provides satisfying results when applied to complex objects.

However, when applied to simple objects, artifacts will appear if the camera captures the object at certain angles. They would be more noticeable if the width of the outline is large and the camera is close to the object.

These artifacts are often overlooked when writing outline shaders because most games contain more complicated objects that are placed relatively farther away from the camera. But if you are designing a Minecraft-style game where simple blocks can be displayed close to the camera, you will want to use other techniques.

One simple way to tackle this problem is to draw the outline by scaling the object up somewhat and coloring it with a uniform color of choice instead of shifting vertices along their normals in the outline pass.

But this technique requires the model to have the same order of magnitude for width and height. Otherwise, the outline won't be drawn evenly across the object border. Also, additional work must be done to make the outlines appear on models that have gaps in them.

In short, basic outline techniques are often tuned for use with specific kinds of objects. In order to construct an outline shader that can be used with a broader range of models, more advanced techniques must be used.

Advanced techniques

A more advanced technique to draw the outline is to draw the object wireframe in the first pass, then draw the object itself in the second pass. The wireframe would be drawn in thick lines and follow the normals of the object.

This method presents drawing artifacts at the corners of the model. To smooth the outline at the corners, we can draw low-resolution circles at each vertex before drawing the wireframe.

With this approach, one has to be sure the wireframe does not hide pixels belonging to the object. For this purpose, the Z-buffer of the outline pass can be altered so that the wireframe is drawn completely behind the object, but at the same time in front of other objects that are further away from the camera. One way to do this is to project the wireframe edges on the faces of the bounding box that are facing away from the camera.

A more practical approach would be to work with stencil buffers instead of altering the Z-Buffer. The model itself will be drawn in the first pass while setting the stencil buffer to a custom value at every pixel that belongs to the drawn object. Then, the outline is drawn by drawing the wireframe and circles at each vertex in the second pass.
In the second pass, the stencil buffer is configured to draw the outline only at pixels that do not belong to the drawn object. In order to identify these pixels, we compare the content of the stencil buffer with the custom value that is set in the first pass. If the values do not match, the pixel shader is called upon to draw the outline pixels.

Conclusion

The choice of a technique to draw outlines depends mainly on the required implementation complexity, performance and drawing infallibility. Advanced outline shaders address a broader range of models than basic shaders, and may be required if the game contains different kinds of objects that may be placed close enough to the camera that artifacts associated with simple outline shader techniques could become visible.