As another part of the “Advanced 3D Graphics Programming” module assessment, each student had to give a 20 minute presentation to the entire class about a given graphics-related subject, which we could not choose ourselves. Also, we only had one week’s notice to prepare our talk. My subject was “Real-Time Anti-Aliasing”, and I discussed the causes and effects of aliasing in computer graphics, along with the techniques used to address the problem.
Real-Time Anti-Aliasing – Presentation (.PPTX, 800 KB).
Read on to find out more.
The following is an approximate transcript of my presentation.
- What Causes Aliasing?
- What Are the Effects of Aliasing?
- Temporal Aliasing
- Anti-Aliasing Techniques
- Texture Filtering
- Temporal Anti-Aliasing
- Morphological Anti-Aliasing
Aliasing and anti-aliasing are not unfamiliar terms to gamers, even though we might not necessarily know what they mean. A casual gamer might assume that so long as they have a powerful enough computer, they can crank up the graphics settings as high as they’ll go and not really worry about it.
Aliasing can in fact happen to any kind of signal we might send to a device, it’s not just limited to computer graphics – this means that it’s quite an important issue, and there are dedicated fields of study to it. Since my presentation is based on graphics programming, however, my focus is on aliasing that occurs in image processing.
What Causes Aliasing?
If we want to understand what aliasing actually is, what causes it, and how to solve the issue it presents, we must consider how the graphics hardware actually works. When we view a computer image, the quality of that image is restricted by the capabilities of whatever device we are using to view it (whether it is a monitor, a printer, etc).
The device has to “reconstruct” the image that was sent to it from the graphics card. If the device is unable to reconstruct the image with sufficient resolution, some of the detail and quality gets lost, and so we end up with an “alias” of the original image.
What Are the Effects of Aliasing?
With a computer monitor, the pixels in the screen limit how much detail it can display. Pixels are tiny squares arranged in a grid pattern, so if our image has diagonal lines in it, the monitor will have difficulty displaying it because pixels can’t be half one colour, half another colour. Instead what we get is a “jagged” edge, like looking at a staircase from the side. Monitors also struggle to display small images, like distant objects in a 3D scene, because the pixels aren’t small enough to display them with sufficient detail.
Besides being unable to display small or angled images very well, monitors are also limited in how quickly they can update their display. Another gaming term you may have heard is “frame rate” or “frames per second”. Again, a higher capability is widely regarded as being better – but it is actually possible to have a frame rate that is too high. If your graphics card can render frames faster than your monitor can display them, a phenomenon known as “frame tearing” occurs, whereby half of the image appears to be different or out of sync with the rest of it, because your monitor hasn’t finished displaying the previous frame before being sent a new one.
To fix frame tearing, most graphics cards have a feature called “vertical synchronisation” (“v-sync” for short), which forces them to wait until the monitor has finished displaying the previous frame before a new one can be sent. It’s called “vertical” sync because monitors refresh their display from top to bottom. V-sync actually poses a new problem though – if objects in our rendered scene are moving faster than we can display them, they’ll appear to jump or stutter instead of moving smoothly. This is called Temporal Aliasing.
Temporal Aliasing is probably something you’ve seen thousands of times and never realised it. Have you ever watched a TV programme where a car has been driving along quickly next to the camera, and you noticed that it looks like the wheels are turning impossibly slowly, or perhaps even like they were turning backwards? You might see a similar effect with a propeller or helicopter rotors too.
Imagine a wheel with 5 spokes, 360 degrees divided by 5 is 72 degrees, meaning that every time the wheel rotates through a multiple of 72 degrees in either direction, it looks like it was in exactly the same position as when it started, because the spokes are symmetrical. Now imagine that this wheel rotates 70 degrees clockwise in the same amount of time as it takes us to render a frame from our graphics device – on the next frame, it will look exactly the same as if it had actually rotated 2 degrees anti-clockwise! This is called the “Wagon-wheel effect”, and because our eyes and brain are in fact part of the reconstruction process of the image, we simply accept what we see.
“Anti-aliasing” is a collective term referring to the techniques that alleviate the distortion and visual artefacts in our image, without making any changes to the hardware device on which we are viewing it.
There are many different techniques that can be used, each with their own pros and cons, but the common principle is to subtly blur the colour of the pixels along the edges of objects with the colour of the surrounding pixels, to make those edges look smoother instead of jagged or distorted. Here are some of those techniques:
One such approach is known as “super-sampling“, “over-sampling” or “full-screen anti-aliasing” (FSAA). This is a brute-force method that effectively draws the image to be a power-of-2 larger than the final resolution, before combining the pixel colours back down in to the final image. This is why you see anti-aliasing settings like “×2, ×4, ×8, ×16”, etc, because that’s how much bigger the image is drawn before it gets averaged down.
Consider “×4” super-sampling, for every pixel of the final image, 4 sub-pixel colour samples were taken – it was the average of those sub-pixel colour samples that decided the final colour for that pixel. Obviously, the bigger the image, the harder the graphics card has to work, so this technique is extremely processor intensive; but every single pixel gets anti-aliased by the colour averaging, so the overall image quality is improved dramatically.
Unfortunately, super-sampling isn’t much good for games. All of that extra rendering and averaging is just too slow; it would need very powerful hardware to run that at a good frame rate. Instead, many games use an optimised technique called “multi-sampling” (MSAA).
Multi-sampling creates just as many sub-pixels, but the colour information is not sampled from each one (which is a bigger difference than it sounds). First, multi-sampling looks to see if the edge of an object covers the centre point of a pixel, and if it does, then the pixel is set to the colour of that object as normal. Then, multi-sampling looks at how many of the sub-pixels were also covered by the object, and multiplies that colour by the percentage of them that were covered to get the final colour. This means that if we have a black object on a white background, any pixels along the edge of that object which have half of their sub-pixels covered will be set to black, then the black colour will be multiplied by 50%, resulting in a final colour of grey.
Despite being more efficient, multi-sampling still has downsides – only the edges of objects are anti-aliased, instead of the whole image, and it is possible for artefacts to occur when the edge of an object does not cover the centre point of a pixel, but it does cover some of the sub-pixels. In those circumstances, colour information will get sampled from outside of the object!
To make objects in our game actually look like what they are supposed to be, we have to apply textures, and textures suffer from aliasing too. Textures are grid-like arrangement of “texels”, just like the pixels of the screen, meaning that when we put a texture on a 3D object which has been rotated or scaled, the texels no longer map 1:1 with the screen pixels and we get distortion.
Scaling textures up isn’t such a problem – it just looks stretched; but when it’s scaled down on an object that is very small or far away, remember that the screen pixels won’t be able to display it with enough detail. Not only is it inefficient to draw a big texture on an object that’s only tiny on the screen, but it also makes the texture appear to “shimmer” or “crawl” with distortion (which is called “noise”). Because of the grid-like arrangement of the pixels, this aliasing often results in a phenomenon known as the “Moiré Pattern“. One way we can deal with this is to pre-create many versions of our texture, each one half the size of the previous one. We can then swap to a more appropriate sized texture based on how far away it is. This is called “Mip-Mapping“, another familiar gaming term.
When we have one really big object that spans multiple mip-map thresholds (like the level terrain), we need to be able to draw several different mip-maps on that same object. The ability to do this is called “bilinear filtering”; and yet this does not resolve the issue completely, because now we can see very noticeable seams where the texture changes between mip-levels. The solution to that is called “trilinear filtering”, which allows us to blend neighbouring mip-levels together in addition to blending pixels within each mip-map.
In what other ways can our image quality be improved? One thing you might notice is that objects with surfaces at oblique angles to the camera have blurry textures. The reason for this is again caused by the screen pixels having insufficient resolution to display objects with enough detail when they’re at such an angle.
Imagine a really big and high-definition wall texture, which is facing you directly. Now imagine tilting it backwards through almost 90 degrees (but not quite), so that it’s nearly flat and at eye-level. All of the detail and information in the texels is still there, but now the texture only spans a few rows of screen pixels! This is where Anisotropic Filtering (AF) comes in. “Anisotropy” literally means “directionally dependent”, as opposed to “isotropy” which means “directionally independent“. AF basically changes the number of colour samples we take from a texture based on what angle it’s at relative to the camera. The more angled it is, the more samples we take to compensate for the loss of detail in what we see on the screen.
One way to fix the problem of objects in our scene moving faster than we can display them can simply be fixed by being able to display them faster. Obviously, that isn’t possible otherwise we wouldn’t have this problem in the first place, but it is worthy of note that if we had a framerate of at least twice as fast as the quickest moving object, we’d have the minimum frequency at which no temporal aliasing would be noticeable. This frequency is called the “Nyquist Frequency” – a slightly lower or higher frequency would not work as well as this magic “sweet spot”. To achieve this, we’d need very high frame rates (which is impossible because of our display device limitations, we’d just end up with frame tearing), and in turn we would need low detail objects, which defeats the point of having nice graphics.
The real solution is simply to cheat. By storing old frames that we’ve already drawn, we can blend them with the current frame to create a pseudo motion-blur effect. If the objects in our scene are moving very fast indeed, they may leave a “ghostly trail” behind them of their position in previous frames, so this approach is not ideal, but true motion blur is rather expensive to compute.
Morphological Anti-Aliasing (MLAA) is a new, complicated and experimental technique being developed by Intel. They hope to achieve superior anti-aliasing capability more efficiently than ever before through the use of shaders. Shaders are special instructions running on the graphics card that modify the image it produces, so by having all of the anti-aliasing calculations done directly on the graphics card itself, the performance boost should be quite significant.
MLAA uses a shape-recognition algorithm to identify the jagged edges of objects in the scene, and then it uses a formula to recalculate the colours for the pixels along that edge. This approach is proving to be successful, and console games are already starting to appear with anti-aliasing of equivalent quality to what a powerful PC could achieve with a much more demanding technique.
- Aliasing is caused when we can’t reconstruct an image with enough resolution, or when we can’t sample fast enough to update it smoothly.
- Super-sampling deals with this by drawing a huge version of the image we want, then downsampling it by taking average pixel colours to get the final image.
- Multi-sampling is faster, but not as thorough. It only samples the colour once per pixel, then multiplies that colour by the percentage of covered sub-pixels to smooth the edges of objects.
- Texture filtering is the use of mip-mapping to use an appropriately sized texture for an object based on how far away it is. Combining different mip-levels gives us bilinear filtering, and blending those different mip-levels together as well gives us trilinear filtering.
- To preserve detail in textures at extreme angles relative to the camera, we use a technique called anisotropic filtering to take more colour information from the texture in those situations.
- When objects are moving in our scene too quickly, we get temporal aliasing. We mask this with motion blur by blending the current rendered image together with previous frames.
- In the future, we can expect to see morphological anti-aliasing becoming more wide-spread, it is a technique that uses post-processing to calculate anti-aliasing directly on the graphics card, allowing us to achieve higher-quality graphics more efficiently.