multichannel exr and speed

[renzorosso]

hi all,

i have a question about how nuke handles the multichannel exr-s exactly.

i have noticed that when i have a multichannel exr sequence versus the same sequence with only rgba the last one is considerably faster on playback inside nuke.

this seems logical and all since the multichannel exr carries so much extra info with it, but is it somehow possible to make nuke read only one channel from the multichannel exr. if i add the remove node and remove everything but the rgba it is still as slow as before.


any tips or suggestions very helpful.

thanks.

[ringas]

Check if your EXR is tiled or scanline.

Nuke gets slow with tiled ones as it needs to read the whole image to display it. If it's scanline, it only needs to read the lines needed for display.

I used to have this problem with EXRs from Vray, they would hang the whole tree. Now, I import them and write them out as scanline ones and the speed has rocketed.

[renzorosso]

they are scanline. i think the main problem is that its slow because it reads all the channels with the rgba.

i like rendering my passes into one exr since it keeps stuff nicely organized in my flows and folders, but every now and then i would like to just quickly scroll through my sequences to check that all the frames are nicely rendered and on those occasions i wish i could somehow tell nuke not to cycle through all the channels but only rgba for quicker display.

thanks for your reply ringas.

[alkali]

This is an issue with Nuke that has been discussed, and currently I don't believe Nuke can request individual layers or channel sets from an EXR file.

[renzorosso]

sorry to hear that. thanks for the info.

[instinct-vfx]

This is not a nuke issue it is an exr issue and "by design". EXR channels are scanline interleaved. This is good if you need many channels and bad if you need only one.

Regards,
Thorsten

[alkali]

Hey Thorsten, good to see you hanging around here. And thanks for correcting me... I guess I always assumed the authors of OpenEXR had considered situations where it could be more efficient to be able to make single-layer IO requests instead of just image-wide, per-scanline.

[instinct-vfx]

It is a design issues where a file layout would either work very well for the one, but not the other. This has recently been discussed on the openexr mailing list. Here's a qute from Florian Kainz:

Hi Kora,

the IlmImf library does not allow writing the file channel by channel; all channels must be written at the same time. The OpenEXR file layout would channel-by-channel writing very inefficient.
However, you can write a few scanlines at a time or a few tiles at a time; your frame buffer only has to be large enough to hold the portion of the image you are writing. (In the extreme, a single scan line or a single tile is enough.)

Florian

[taa28]

Hmmmm... Perhaps I'm missing something here, but my assumption has been that Nuke's internal processing uses only the data from the particular channels called for in a script (via Shuffle/ShuffleCopy). IOW, if you've read-in an .exr, and plan to limit the data channels used w/in your working script, couldn't you merely use a Shuffle to isolate (and pass along) only the specific channel(s) from w/in the .exr that you need as Input for conected node(s) w/in the script? -much like using an RGBA image, but if needing to affect only the alpha data, one would switch off all input data but the alpha channel; saving on extraneous processing of the unneeded R, G, and B.

[renzorosso]

as i understand it through Thorsten's explanaition is that its not possible for nuke to throw away the channels in the multichannel exr in the read node. so while nuke processes (all the blurs and transforms and what nots) only on the channels you let through the shuffle (or the specific channel selections inside the nodes), the read node still reads in all the channels on every frame.

right?

[alkali]

Hmmmm... Perhaps I'm missing something here, but my assumption has been that Nuke's internal processing uses only the data from the particular channels called for in a script (via Shuffle/ShuffleCopy). IOW, if you've read-in an .exr, and plan to limit the data channels used w/in your working script, couldn't you merely use a Shuffle to isolate (and pass along) only the specific channel(s) from w/in the .exr that you need as Input for conected node(s) w/in the script? -much like using an RGBA image, but if needing to affect only the alpha data, one would switch off all input data but the alpha channel; saving on extraneous processing of the unneeded R, G, and B.

Nuke's internal processing is one thing, but we're talking about the actual file IO here. Even if you read in an EXR sequence and then immediately use Remove node(s) to dump all the layers besides RGBA, Nuke will still have to load all channels contained in a given scanline to access the RGBA channels due to the EXR architecture (as Thorsten explained).

[Zafar Iqbal]

I render out R, G, B and A at most since my 3D app won't do more. Would there be any increase in performance by converting these to scanlines?

I used Photoshop to do the convertion - Theres no options window popping up, so I cant say if it save them in a "better" way. Redrawing felt quicker in Nuke but I'm not sure. Is there any tool that could batch convert EXRs to EXR and with the desired options?

[instinct-vfx]

Hey there,

Nuke does its best at only loading data that is needed, but as mentioned above it still has to skip over the ones that are there in order to access the data that is needed. Converting to zip1 (zip compression per scanline) does indeed yield the best performance as nuke can grab a scanline at a time (as opposed to having to grab 16 for example with zip16 even if there are scanlines it doesnt need. Tiled EXRs are even worse. They offer different advantages, but as the data can be randomly scattered physically on the drive (per tile) file access is naturally rather slow.

We do have inhouse tools that convert EXRs automatically. If you got anyone that can handle a bit of c++ this aint to hard to do given the EXR examples.

The reason VRay's EXRs are tiled btw is to be able save a LOT of memory during rendering. As tiled EXRs can have their tiles written in random order VRay doesnt have to keep the whole framebuffer in memory during rendering, but only the tiles (buckets) it is currently processing. With big resolutions and/or a lot of channels (=renderelements) this makes a BIG difference.

Regards,
Thorsten

[Zafar Iqbal]

Thanks for the clarification, Thorsten. I understand bucket rendering, but tiles in EXR is new to me, and I'm thinking that it might be worth looking into to gain performance, as I typically do 5k renders.

[alkali]

If you don't feel like writing a custom app, you can also use Nuke (either manually, or by writing a quick Python script) to handle the conversion from tile-based to scanline-based EXRs.