There are few topics amongst CG professionals and games developers that can so quickly raise a heated debate than that of facial animation for computer generated characters.  In a series of articles I discuss the aesthetic, technical, and practical considerations of building a first rate facial animation pipeline.

Getting Started

So you are starting a new project that requires facial animation content.  The debate on the team quickly kicks in:  “what’s the best approach? Should we use facial motion capture?  What about existing software solutions – is there a great piece of software out there already that is going to solve all of our problems for us, or should we develop something locally?”

It is a hot topic not just for technical directors and animators - talk to any programmer, modeller, director, designer or producer and they will quickly offer a series of very passionate opinions on the subject.  Each will often approach the problem from the perspective of their own focus areas.  Programmers may get excited by the procedural components of the system that allow them to control a character’s dialogue, emotional states and eye movements within the game code. Animators may gravitate more towards tools that will offer them a high degree of animation control and their passion probably lies more in producing great visual fidelity and expressiveness.  Producers and Art Directors often face the reality of creating hundreds or thousands of expressions and lines of dialogue with the need to balance the requirement for quality with the sheer volume of animation they need to produce. 

Combined with the range of differing concerns, there is the added fact that we are all by nature something of an expert on the topic of faces – each of us wakes up to one in the mirror every morning, and the way humans communicate through language and facial expression forms much of the core of human behaviour.  Everyone is going to be entering the debate with very strong ideas about what the best approach should be.

One thing many of the team members will likely agree on is that when it comes to the area of facial animation, particularly in games, there is room for improvement.  We complain that characters look wooden, zombie-like or just plain wrong.  Many other areas of CG such as lighting, environments and even character modelling have advanced to a level of detail and realism that, though not perfect, is high enough that the artist and animators can focus their efforts on creating beautiful content somewhat free of technical limitations.  Facial animation seems to lumber behind. 

You have probably decided you want better for your project.  You want it have great facial animation.  So how best to begin?

One approach is to grab for a quick ‘off-the-shelf’ solution.  Is there a piece of software out there that will comprehensively handle all of your facial animation requirements?  Or perhaps outsourcing the facial animation to a 3^rd party service provider is the answer?  Diving into a assembling a pipeline without doing your homework can be a bit like buying the first car on the lot – it might look flashy but not ultimately be the correct vehicle for the task at hand.  You don’t want to buy a Ferrari to deliver the milk.

One of the common pitfalls in choosing a facial pipeline is to base the choice on demo-videos. Software vendors and service providers are likely to have a few amazing animations to show you – but remember an animation is not a pipeline.   If one sticks a group of extremely talented animators in a room for a length of time, they are going to produce a great looking example of facial animation.  Often, when your project requires hundreds of minutes of animation, the quality may drop significantly. 

In terms of software, many existing solutions tend to do certain things quite well, and others less well if at all.  For example some may have excellent motion capture integration, possibly even deriving the facial animation from straight video source.  Others may have no mocap per-say but an excellent set of animation controls.  Some might be able to deliver an excellent degree of fidelity, but have output which is unusable for your application.  For games applications, if the result doesn’t run in engine, it may as well not work at all.

As every project is different, it’s likely your best solution may include a collection of different tools, services and software.  You may find that there are tools available which meet many of your requirements, or that you want to create some of your own.  Developing your own solutions can be costly in terms of R&D but offers the advantage that you can be sure its going to deliver exactly what you require. It can be equally expensive to adapt an existing solution to do something that it wasn’t originally intended to do.

Before deciding on the pipeline, it’s worth considering the following questions.  Having a comprehensive answer for each can help ensure you meet your technical, aesthetic and practical requirements. This will help focus your team’s energy so their efforts can be spent on creating great results rather than problem solving or fire-fighting.  Having looked at each in order can also help to ensure you are not placing too much emphasis on one area of concern, such as technical requirements, over another, such as animation control.  Later in this article I will go further into each, but here is a summary: 

1.                   What does good facial animation actually look like?  Before concentrating on stylistic and technical concerns, it’s worth considering first what convinces us that a facial animation is authentic. What are the common features that make a character’s face, regardless of whether it’s hyper-realistic or cartoony, spring to life? 

2.                   What are the specific requirements of the facial animation system?  What do your characters need to do for your animation or game play to work?  For example, do your characters need to speak in different languages with accurate lip-sync? This would impose restrictions on the solution you choose.

3.                   What does our animation need to look like stylistically? Are your characters highly cartoony or super-realistic?  Are they exaggerated in their expressions or more subtle. The visual style of the project is going to have a huge impact on how your characters look and behave, and this should be reflected in the facial animation.

4.                   What is the machinery that is going to drive it all? Once you have answered the above questions its time to start looking how you are technically going to achieve your targets.  If you are working in a game environment, remember your solution needs to work at runtime – many available solutions can provide great pre-rendered results, but are not practical for game applications.  Equally for pre-rendered applications remember that you can have the best interface available, but if the mechanics driving the actual facial deformation are not good, the final result is not going to be as good as it could be.  

5.                   How are the animator’s going to actually use the tools? Often this is the first area of concern, but I would argue it should probably one of the last; not because it isn’t extremely important, but because once you have a better idea of the why’s and how’s of your pipeline, creating a user interface to wrap around it is going to be a clearer and easier task.

6.                   How are you realistically going to make it all happen from a production and scheduling perspective? It’s a good time to think about the golden rule of production:  “Good, fast, cheap… pick any two”.  After you have decided what you want to do, and how you want to do it, its time to get realistic about what you can achieve within your time frames.  If you want to achieve the highest quality, be prepared to spend some time developing your solutions, or pay a premium in terms of cost.   If it has to be turned around very quickly and your budget is tight, then it’s likely the quality is going to drop.

As a final consideration in building your pipeline, consider that each of the final results will only be as good as the weakest components of your solution.  You can have a good mocap facility and a great set of tools, but if the actors you have chosen can’t deliver the right kind of performances, it will be reflected in the final quality.  Perhaps you have a brilliant animation team – if they don’t have the right tools to work with it’s going to be a slow process, at best, for them to create the level of quality you are looking for.  Each stage of the process needs to be solid and practical.

If you have gone through each of the areas above and feel confident you have answered all the questions its time to start building, buying and implementing your pipeline. You should be able to evaluate solutions and find out what’s out there to meet your specific needs – more than likely your pipeline will be a grab-bag of solutions, including internal or external software, or a combination of both. You might find you want to outsource most of the mocap, but have a more active hand in the selection and directing of talent. Or you may decide you have the time and resource to forgo mocap altogether and go for a hand-animated approach, using your best animators to deliver the results.   It will depend on what you are actually looking to achieve – there isn’t one ‘silver bullet’ solution – there are several. It’s all about choosing the right combination of approaches and assembling them together.

Spending the time at the beginning of the project plan mapping out your overall facial animation strategy can pay dividends later – countless projects have suffered towards the end of production when they have found themselves with hours of un-usable mocap, animation pipelines that are so cumbersome that the animators can’t iterate quickly enough to produce any kind quality, or the realization - too late – that the style of facial animation they have employed is a poor match for the style of the rest of the game.  Making good strategy early on will save on time, money and creative energy.  Deciding early what your benchmark for quality across all areas is going to be, and what elements you are going to ‘push the boat out’ with, will help your team focus and keep your project on track to deliver great facial animation.

In the next section (coming soon):  What does good look like?  I will go into detail about the most important question of all, what actually makes good facial animation.