Image-Metrics

 By Shamus Dec 7, 2007 27 comments

This is amazing. I almost didn’t link it because you have to visit the site to see it. They really need to think about making that into a YouTube video.

The demo on that site shows a woman making various expressions and faces for the camera, while a computer-generated face does an exceptional job of imitating her. The actual geometry of the face isn’t all that impressive. (It looks very plastic and lifeless. Haven’t these guys heard of texture maps? Then again, maybe the model is made this way on purpose, so that you focus on the motion and not the shape of the face.) The animation goes a long way to bringing the face to life, although I can’t help but wonder how it would look with a more convincing 3d model.

The obvious application here (and what they seem to be selling) is a system for making complex facial animations with minimal work. Doing this stuff by hand is a chore, and tends to look very mechanical. (I’ve dabbled with it. It sucks, and it takes a huge investment of time and skill to get rid of the mechanical feel.) Doing it “right” involves an actor putting on special makeup (usually dots all over their face for the computer to track) or other tricks to give the computer well-defined points of reference. But the system shown here seems to let anyone sit down in front of the camera and start mugging without any fancy setup.

I’m willing to bet it isn’t realtime, even though the demo movie seems designed to allow you to think so. I’m guessing you capture the video and then the software needs a jolly good think before you can see the results in action. I’m also confident that it would have trouble if you spoke. I doubt it attempts to track anything inside the mouth. Near the end of the demo the actress licks her lips, and the avatar doesn’t. It’s possible that the avatar doesn’t have a tongue, but I think it’s more likely that the software can’t quite comprehend tongues yet. This is a shame, since if that worked this would be the ultimate way to make lip synch look good. Just get a video of your voice actors as they do their thing. Most voice actors already emote visually when they do their thing, so the extra data would be “free”.

At any rate: It’s fun to watch the animation do its thing.

20727 comments. Hurry up and add yours before it becomes passé.


  1. Corvus says:

    Ken Perlin has something online that will be of interest to you as well, then.

    http://tinyurl.com/6cweq

  2. Henebry says:

    I wonder if this sort of thing would have made the difference in the recent Beowulf. I liked the movie on a number of levels (mostly for stuff attributable to the screenwriter, Neil Gaiman), but it was widely panned by critics for inexpressive faces. The cgi was evidently based on the same tech that gave us the icy mask worn by Tom Hanks in The Polar Express, though clearly Beowulf marked notable advances on that film.

    Anyway, it’s my guess that this tech will be seen in movies before its seen in videogames, since the quality of the renderings seen in movies is so much higher than in games. And since the expression of complex emotions remains much more central to the film industry than to the game industry.

  3. Nick Pitino says:

    That’s cool and all that and I can see the obvious implications, but I still can’t get past the feeling of this being about where we skip right off the edge of the uncanny valley and go plunging in.

  4. Joe says:

    my thought is that sometime in the nebulous future, this will become a part of online games. A camera on your PC watches your face, so your avatar may *look* like an Orc, but he’s mimicking your facial expressions in realtime.

    But that’s just my thought…

  5. Ryan says:

    But then most everyone would be walking around with a dead, eyes-glazed-over expression until that 14th level Wizard fireballs them into oblivion.

    Then they just look angry.

  6. asterismW says:

    I don’t think the avatar has any kind of inside mouth at all. Did you notice how the actor never opened her mouth? She didn’t yawn or show her teeth or anything. Still, it was really cool to watch.

  7. Alexis Li says:

    and at the risk of lowering the tone of this blog, some of those AFK paladins might have some very unusual expressions.

  8. Telas says:

    In Snow Crash, Neal Stephenson pointed out that in a virtual reality situation, it’s silly to make a computer read your facial expression when the human mind does it so well already. All the computer has to do is communicate your face’s movements to the virtual face, and the human mind on the other end will do the work for you. I believe he used the phrase, “condensing facts from the vapor of nuance”.

    This is a pretty big step in that direction. I’m impressed, and I’d like to see how this translates to non-human faces.

    IMHO, the virtual face needs more wrinkles. The uber-smooth skin looks too fake. (Notice especially the forehead during the frown; there’s a hint there, but it looks like her skin is really thick.)

  9. Oleyo says:

    I think the face is pretty good, for CG. Of course, I do some character animation here at work on low poly stuff, and know first hand how hard it is. (you know too Shamus!)

    I think she uses all procedural materials and no specific texture maps…those might make it more real but would be a ton more work. Not only that but there is more room to go wrong with texturing, if it isnt subtle and just right it will actually come out looking worse. So I think they just used the best off the shelf 3D head they could find.

    BTW Did anyone else get a Fallout flashback from watching that? :)

    A) “My, thats a black room you are in”

    B) “Tell me more about Shamus Young”

    C) “Give me all your money or I will kill you”

    D) “What do you have to trade?”

  10. Kris says:

    Hey Shamus, this covers into facial software AND your older post on the awesome FPS with the portal gun.

    Still Alive

    I don’t know, it just seemed right up your alley. :)

  11. Thad says:

    Mainly backing up the Beowulf point here: I’ve only seen the trailers so far, and while the characters look great, whenever anyone speaks the character’s face seem off. They haven’t quite got lip movements working right yet.

  12. Davesnot says:

    That’s pretty cool.. but the link givin by Corvus in the first post is a blast to play with!

  13. Mrs T says:

    Did they change the video since you posted this? Because I’m seeing talking and mouth opening, and I didn’t see the lip licking thing.

  14. asterismW says:

    Mrs T: Yes. It’s changed. Huh. The speed of technology is simply AMAZING.

  15. Randomscrub says:

    The video now involves talking, screaming, and gasping, as well as a host of other emotive expressions.

  16. Thanks everyone for the comments and thank you Shamus for posting the link. We are actively involved in many games and film projects (some of the films coming out are simply amazing) and the technology has come along way. The Llana character on our home page has just been updated to look more photo realistic. We can certainly go farther (and will in our next release).

    Someone above mentioned a youtube video. Does anyone have any thoughts on this topic?

  17. TrueBlue says:

    The textures, teeth, and tongue movement look excellent to me. Maybe they have updated the animation??

  18. M says:

    We definitely get to see the actress sticking out her tongue in the new animation…caution, though. She shouts at one point, and it’s way louder than the rest of the video.

  19. Davey says:

    It’s nice, I too was confused cause they updated the video (looks better than Beowulf IMO) and it woke up my baby… maybe some compression is in order on the video?? : )

  20. tocky says:

    It kind of sucks, too, because while the newer video is more complete technically, the older one was nevertheless much more impressive, by a combination of the other actress showing a more varied range of expression and the new actress’ smugness about her ability to growl and frown at you.

    It’s odd that even in a tech demo, the script could really put me off the subject matter.

  21. Kevin says:

    Haha, I went to high school with the girl in the new video.

    Anyway, it’s definitely impressive.

  22. ngthagg says:

    I disagree with tocky: I think the new video is a big improvement. The only thing I would like to see is more contrast between the real person and the animated face. Modeling an attractive woman’s face with . . . an attractive woman’s face, although no less difficult technically, seems to be kind of a waste.

    If Image Metrics is going to put something up on Youtube, I would suggest a three pronged approach: two videos showing contrasting faces, with the models being photorealistic. ie male vs. female, young vs. old, etc. (Using a baby’s face would be extremely cool.) And then, as a third video, use a non-photorealistic model. Possibly a cell-shaded model?

  23. ArchU says:

    Although nowhere near as advanced in expression as this, Logitech has featured similar real-time animation transforms for their webcam software (QuickCam) for quite some time. A simplistic facial map needs to be calibrated first.

    This is far superior but the technology has been around for a while already.

  24. Riksa says:

    This is really impressive. I look forward to seeing the technology hard at work. IMO one of the most lacking aspects of game character animation are the facial expressions. Looking forward to something of this calibre.

  25. acnellin says:

    Wish they use this in Fallout 3.

Leave a Reply

Comments are moderated and may not be posted immediately. Required fields are marked *

*
*

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun.

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!