Bridging Game enignes and compositing applications

[sgr]

Hey peps,

Today I was talking with the modelers here at VFS and they introduced me to some of the aspect of game engines such as Unreal Engine. As we got more into the speed and "fanciness" of it - I started to wonder if that could be Incorporated into compositing packages like after effects or Nuke.

With more and more material being produced for web and hand held devices it could perhaps be a interesting aspect to bridge game engines with flash and after effects or nuke.
To create interacive experineces which included the speed and quality of Game engines and live footage.

This was just a quick thought but someting I plan to look more into - but do you guys have any thought on it - pro / cons?!!

Best,
Sofus

[Aruna]

Ya know, I've been thinking into this a lot, and it can be done. A lot of the consoles and GPUs out there are capable of realtime transparent overlay, along with realtime fluid dynamics. I think that a smart programmer and dedicated art director and compositor could make a really realistic looking game that plays well at speed. I've just been playing Gears of War 2 extensively, and they take a lot of what we do in VFX and apply it to the game, with blooms, tracers, DOF, and so forth. The Gears of War 2 trailers were actually done at Digital Domain, using the Unreal Engine and our inhouse stuff, to make the trailer. Really quite cool. Depending on the type of game, realistic environments could be 'baked' into the textures of objects, along with their lighting. Games are slowly getting there, and the crossover between film and games is getting blurred. I wrote something a little while ago on my blog about it!

[Matt P]

The Crysis engine is worth having a look at as it's easily the most graphically convincing of current titles. The game shipped with a level editor which allows you to literally drag and drop 3d objects and design the level in realtime, with one button spawning you in on the camera to play test. No level compile or anything.

Film techniques have existed in gaming for a long while now. It's only recently that a.i. driven character reaction and graphical features which are nearing convincing have become a possibility. Though films and games will always be seperate as mediums due to how they seek to immerse (interaction/observation) I wouldn't be surprised to see gaming engine style tech appear in 3D packages.

I think the most interesting example of gaming immersion driven by filmic techniques is Left 4 Dead. In which an artificial intelligence controls both the gameplay and soundtrack catered around what is happening (a somewhat procedural and random process).

-Matt

[sgr]

Cool to hear Aruna - any chance on you being able to elaborate on how much of the "standard" engine was used ? I presume you tech staff was busy breaking it up for the needs of creating the trailers.

And indeed the Crysis engine looks to be really bad ass as well - I was reading the release PDF from Crysis where they cover the features and it's stunning to see what they are capable of doing in real time.

My biggest question is - will the compositor benefit from having a engine like this to generate and edit I.E. environment and props interactivly or is the current 3D work spaces (I am thinking about Nuke's) enough for a production.

Mainly I can see a plus to it if the production will move across media - TV, internet and console
but for large scale productions I presume that dedicated modeling/texture packages and users are more desirable.

But yeah I agree with you matt a more interactive workflow in the 3D packages is properly what will happen first - I am still looking forward to try out Mental Ray IPR in maya but it's still not as cool as seeing you models lit and textured like from the engine's with in the packages.

ANf thank for the reply gents.

Best,
Sofus

[Aruna]

IIRC, they used same game engine as the game, but just uprez'd the textures and 'rendered' it out in layers, as opposed to realtime. Then they just added the typical comp love, blooms, advanced DOF, tracer glows, better fx, etc.

I am using Nuke's 3D side quite a bit these days, especially with imported .objs, to either line up discrete cameras, match camera moves, or make wholly fake environments using projected photographs. While it does still take a while to render these out (most significant is the motion blur hurdle), a game engine compositor might be able to make do with less than a 6k environment or a 4k painting.

[sgr]

hey Aruna, thanks for the reply - on a side note can you link to the post you talked about - I am not sure which one you are referring to on your blog.

Thanks in advance
Sofus

[Aruna]

There are two entries on the blog where I've written about where the industry might be headed, where the joining of both film and gaming will come to a head..

http://www.digitalgypsy.com/vfxlog/archives/2008/12/five-years-into-the-future.php
http://www.digitalgypsy.com/vfxlog/archives/2005/03/3d-and-imax.php

[Gentle Fury]

Ya know, I've been thinking into this a lot, and it can be done. A lot of the consoles and GPUs out there are capable of realtime transparent overlay, along with realtime fluid dynamics. I think that a smart programmer and dedicated art director and compositor could make a really realistic looking game that plays well at speed. I've just been playing Gears of War 2 extensively, and they take a lot of what we do in VFX and apply it to the game, with blooms, tracers, DOF, and so forth. The Gears of War 2 trailers were actually done at Digital Domain, using the Unreal Engine and our inhouse stuff, to make the trailer. Really quite cool. Depending on the type of game, realistic environments could be 'baked' into the textures of objects, along with their lighting. Games are slowly getting there, and the crossover between film and games is getting blurred. I wrote something a little while ago on my blog about it!


Didn't DD do the first GOW trailer too with Mad World playing in the bed?

As far as real time gfx in compositing, Fusion 6 (if it ever comes out) supports Realtime CG Shaders...I saw it at Siggraph and it was quite impressive. The days of rendering 3D will soon come to an end, im sure of it......the days of rendering fully photo-real 3D in real time may be a few years off, but it is getting SO CLOSE! F.E.A.R. 2 was as close as I've seen lately....GOW 2 looks awesome too...but I don't have an xbox...only ps3 :( Maybe if it ever comes out for PC.

[Aruna]

Yep, you're right, DD did the Mad World GOW trailer as well.

[jokerxel]

As far as real time gfx in compositing, Fusion 6 (if it ever comes out) supports Realtime CG Shaders...


Looks like someone posted up pics that show examples of Fusion 6's shaders (http://www.vfxtalk.com/forum/fusion6-sneak-peek-t19332.html?t=19332).

[jbidwell]

Doesn't ILM have a real-time compositor called Zeno? I saw a demo of it and they made a previz scene that could eventually become a slap-comp.

I am impressed with how the retail version of NUKE has been developing too.

[Gentle Fury]

Looks like someone posted up pics that show examples of Fusion 6's shaders (http://www.vfxtalk.com/forum/fusion6-sneak-peek-t19332.html?t=19332).

I just saw that and it looks SWEET! I'm stoked.

[mboden]

Doesn't ILM have a real-time compositor called Zeno? I saw a demo of it and they made a previz scene that could eventually become a slap-comp.

I am impressed with how the retail version of NUKE has been developing too.

J Zeno is ILM's cracked out version of Maya. I think the comp software is called Comptime, but I could be wrong on this. They also have the SABRE but those Flames. Ask Vince or Catherine.

[jbidwell]

You're right, but they do have something that is based on a game engine which allows them to do previz and it goes right into "Comp-time."

It's been a while since I saw the demo at the Metreon...

[Gentle Fury]

So I just finished playing thru Crysis (gotta love downtime) and it has some AMAZING real time effects....some of the best I've seen since GOW......why is it that it seems like in 2007 the coolest games came out...and quality has kinda dropped since then? I spent half of Crysis just staring at the character models, they were SO GOOD....and the final battle on the air craft carrier flight deck was just unbelievable. What happened? Did we have a video game renaissance in 2007...then all of a boom....back snaps the time machine and we are back to games not looking as good. And we are still taking forever to render for production.

[Matt P]

So I just finished playing thru Crysis (gotta love downtime) and it has some AMAZING real time effects....some of the best I've seen since GOW......why is it that it seems like in 2007 the coolest games came out...and quality has kinda dropped since then? I spent half of Crysis just staring at the character models, they were SO GOOD....and the final battle on the air craft carrier flight deck was just unbelievable. What happened? Did we have a video game renaissance in 2007...then all of a boom....back snaps the time machine and we are back to games not looking as good. And we are still taking forever to render for production.

Not exactly a renaissance - the reason all games are not like Crysis is amusingly simple - they don't sell well enough. To run Crysis well you need a pretty damn good machine, and none of the 'next gen' consoles can run it. As a result, the audience isn't as wide as most games aim for.

Also, Crytek have a close relationship with NVidia so a lot of the tech that went into the graphics is still some way ahead of most modern releases. The very same things happened back when far Cry was released - a graphical leapfrog which wasn't caught up with for almost 2 years by the rest of the industry.

Take a look at the level editor for Crysis, it's also realtime. Fly around and place objects then one button and you're playing the game inside the editor. No compiling. It's awesome.

-Matt

[Gentle Fury]

Not exactly a renaissance - the reason all games are not like Crysis is amusingly simple - they don't sell well enough. To run Crysis well you need a pretty damn good machine, and none of the 'next gen' consoles can run it. As a result, the audience isn't as wide as most games aim for.

Also, Crytek have a close relationship with NVidia so a lot of the tech that went into the graphics is still some way ahead of most modern releases. The very same things happened back when far Cry was released - a graphical leapfrog which wasn't caught up with for almost 2 years by the rest of the industry.

Take a look at the level editor for Crysis, it's also realtime. Fly around and place objects then one button and you're playing the game inside the editor. No compiling. It's awesome.

-Matt

I just think it's sad that such amazing tech has to wait for the masses....the masses should be racing to catch up with it! Same reason there is still 100 dial up users...and there are still a couple vhs tapes floating around..... :(

[Matt P]

I don't know how I feel about it all. Whilst my hardware is up to date - it's no longer the same cutting edge it was a year ago. I'm not completely convinced I want applications to have a habit of demanding extreme hardware or other specialised bits of kit to run properly - It's for this reason I've never touched Flame.

With games and other entertainment items, the primary goal has got to be operating as a sustainable company and very few companies seem to be able to output genuinely progressive gaming experiences alongside this : Valve, Blizzard and Crytek being the handful I can think of. It's all about consumer affordability and target audience. Though every now and then a game like Crysis comes out which has ridiculous hardware requirements and you know what - The experience was worth it for me. Have you played it online yet? There is a weapon which, I kid you not - fires nuclear warheads. The scale of the explosion is completely ridiculous, my jaw dropped the first time I saw it.

With regard to gaming tech coming to film - An issue is that Film VFX requires photo-realism and realtime gaming engines just can't offer this yet. This isn't to say I wouldn't like to see some of the advances in real time render technology being used in an artistic environment though I feel it needs to advance to a stage of brilliant photo-realism before we can expect to see it prove useful in a VFX environment.

In the meantime, Nuke's 3D integration And Fusion's shader technology are still leaps from Shake - which is now viewed as archaic but itself was a vast improvement over it's predecessors. There's still progress! I think software-wise we're amidst a very exciting time!

-Matt

[Dotcommer]

Have you played it online yet? There is a weapon which, I kid you not - fires nuclear warheads. The scale of the explosion is completely ridiculous, my jaw dropped the first time I saw it.

Don't forget the fatman from Fallout 3 which shot mini nuclear warheads, and the experimental modified version which shot out 8 at a time. hahaha

[Aruna]

I have yet to play Crysis, and the demo I had barely ran on my machine, which was decked out almost for it! Unfortunately, trying to push anything at 1920x1200 in realtime at 60fps was hard. I think I got maybe 30fps at that res, I had to turn some stuff off. I can't imagine it with volumetrics.

I'm going to have to slightly disagree that game engines can't create photoreal environments and/or effects. They won't be in realtime, but they can be photoreal. The environments in GOW2, Fallout3, COD4, are all VERY VERY close to being photoreal. I haven't played Killzone 2 yet, but I hear that's pretty close too. It's a matter of bandwidth, texture resolution and detail, and the game community doesn't have the computing power to render them in realtime, yet. But the film community does.