rendering questions

[btha8086]

hi.

i'm editing this question 3/14:


is it possible to send packets out via different nics on a the render server so that each nic is assosiated with 1 segment of nodes?
i'm guessing this could double, triple or even quadruple network bandwith...depending upon the number of nics. of course the hdd bottleneck would be solved using RAID/high RPM scsi.

so for example - with 3 nics on the server, 3 nodes from 3 different segments could be receiving packets simultaneously? and with full duplex, the server could ALSO be receiving data at the same time through each 1 of those nics?

way to acheive: (a) assosiate nics with particular ip scope on server's routing table or routing software (b) group nodes into segments and assign ip's (c) connect nodes to router, and uplink each router to corresponding nic on server.

would this brain fart of mine actually work??? or am i just re-inventing the wheel? i know this is done for web servers/mail servers but i'm not sure about render farms.

anybody care to guess???

[btha8086]

why won't anyone answer my question?

::FEELING LONELY::


grouphug anyone???

[ShadowMaker SdR]

Maybe because noone KNOWS the answer? I don't even understand what you're asking :-)

[VFXTalkDotCom]

would this brain fart of mine actually work??? or am i just re-inventing the wheel? i know this is done for web servers/mail servers but i'm not sure about render farms.

anybody care to guess???

my take on this would be it wouldn't make any real difference - on a renderfarm the real bottlekneck is the size of the images that need to be moved around after they are rendered and caching of those images as they are rendered...

the data thats sent over to the servers to be rendered is mostly geometry and texture data so its minimal compared to the output frames. ie it may take 1/100th of a second to send the data for a frame to be rendered and 3 minutes to render it out...

with webservers and email servers the data is itself tinly little packets that can benefit from being moved around rapidly as they dont change on either side

the best ways to speed up a renderfarm are :

1. faster processors / more ram on your render nodes so they can crank frames out much faster
2. a good master server with a kick-ass raid and high speed or multi port ethernet access to the rest of your facility

hope this helps?

[SalaTar]

smart (managed) hub and bind/bridge a few nics on server

[modernprimitive]

A totally realistic concept - and I'm sure it's used in the big studios. You'de only use this in the case that the network really is your bottleneck. I'd imagine that we would be talking something like 24 quad or 8-way servers hanging off a managed switch to actually saturate Gigabit ethernet.

It also depends on what you're rendering - if it's CPU intensive the time between receiving data and returning data back to the Master server will be greater thus less bandwidth will be required.

The thing to do is to run a network analyser on the LAN and a performance monitor on the servers - in particular the Master. Based on the gathered data, you'll be able to see where the bottleneck is and where the upgrades should be. There's no point fixing the LAN if the hard drives can't write the received data fast enough or each server is taking 10 seconds to process a single frame, leaving the network lying unutilised during this period. (although bursts could be affected if all servers are sending data at once, but I'm sure that at some point they would loose synchronicity - also depends on the type of renderer.)

I've checked my bandwidth (Gigabit ethernet) with 3 render slaves (3GB CPUs) using Fusion's preview network renderer which is a lot more network throughput than a typical 3D render since it's 2D scanline with no raytracing and utilization didn't even hit 1%. Add 10Gb ethernet to the equation and I think your SCSI or your Fibre Channel SAN is going to suffer before your LAN.

...but definately a viable concept.

[btha8086]

o.k. - first of all, EVERYONE: THANK YOU.

jah - you have a remarkable way of putting things on track. thanks again.

mp - /excellent/ idea - didn't even think of a network analyzer. 1% huh? INTERSTING. "Add 10Gb ethernet to the equation and I think your SCSI or your Fibre Channel SAN is going to suffer before your LAN." interesting point again...(see far below)

salatar - good point...running to a single hub. anyone know if 3com is my only viable choice for gigabit? cisco=too much freakin' $$$!!!

shadowmaker - sorry 'bout the confusion buddy, sometimes /i/ don't even understand what the hell i'm talking about.


well...my goal/fantasy(?) is to get dual itanium2 servers, with 3ghz of itanium in each 3u case and wallops of RAM
so network bandwith, even with gigabit-rates /may/ have a problem. the itaniums are monsters y'know - i benchmarked with a buddy of mine and they almost tripled the dual opteron i was running - ghz for ghz. i knew they were good, but i was really blown away.

there's a real possibility of doing scientific stuff in addition to the rendering. i think it's called a LPAD clustering or something, beowulf's cousin where I/O is increased alot and there's inter-process communications. i just don't know yet.

the supervisor node doubles as the storage node - right, so with raid5, my accumulated storage throughput...hits 120MB/S - if i'm looking at sustained transfer rates/media transfer rates for the drives.
so the Actual storage throughput should sit comfortably around 120MB/s on the 125MB/s network. hmm.

so like you folks mentioned - will storage i/o be fast enough? or i could setup pvfs and strip across seperate i/o nodes??? i guess i'll find out when the time comes, but i'm thinking i'll strip across multiple i/o nodes, each node storing data at 120MB/s. so that'd be 240MB/s throughput with 2 nodes...still not peaking the actual network bandwith WHILE increasing the rate exponentially per i/o node added.

RAID is some bad-ass stuff

so i'd still set up ipscopes, but on the i/o nodes - IF the network became congested. hell, at that point i could even setup an actual SAN, scsi style. pvfs rocks

so it's going to be interesting to see how this all plays out. i'll optimize then analyze my current setup. you folks may have helped saved me some bones in the long run.

right now all i have is quad xeon's, a scattering of sun's, dual opterons (and a dual alpha on the horizon) - but hopefully this collection will grow

once this little experiment actually takes off, i'll check back with all you nice folk at vfxtalk.com!

[modernprimitive]

I think the Titaniums are an "true" 64-bit architecture unsupported by most applications other than large database apps like Oracle, SQL and Server OSes. Athlon/OPteron and P4-based 64-bit CPU's are actually dual 32-bits and therefore supported by all 32-bit apps. Check support for Titaniums from the software manufacturer before investing in this expensive hardware.

[Splash]

I think the Titaniums are an "true" 64-bit architecture unsupported by most applications other than large database apps like Oracle, SQL and Server OSes. Athlon/OPteron and P4-based 64-bit CPU's are actually dual 32-bits and therefore supported by all 32-bit apps. Check support for Titaniums from the software manufacturer before investing in this expensive hardware.

I belive the only render engine that support the Itaniuns is the Mental Ray ( in the fututre is possible that applications like Flame and Inferno run in Itaniun, http://usa.autodesk.com/adsk/servlet...linkID=5572949)

[SalaTar]

Itaniums were DOA and passed over, correct?

[btha8086]

well...my main goal is maya and 3ds, so i should be o.k.. lightwave supports IA_64 too from what i hear...don't qoute me on that though
- any open source software i figure i can compile...

there's ia32-EL - i have no expeience with it, but a friend says that gig-for-gig it really does allow an i2 to match a xeon for raw computing power...i'm kind of skeptical, but assuming he's right - there's not too much lost if i had to {sometimes} use [REAL-TIME]emulation (of course that's a matter of opinion too )

plus the thought of 12gb+ of ram per node makes me lightheaded...

tell me what you think of the following post from slashdot...
"Whichever processors you go with, make sure the entire farm uses the same type. Otherwise peculiar rendering differences might occur, in things like particles, hair/fur and fluids."

is (s)he talking about larger renderfarms only do you think? irght now i use an assortment of p4-d, xeons and opterons...and i haven' noticed anything.

---

for storage, is 240MB/s i/o good enough do you think? i'll raid 6 drives on 2 different nodes and use pvfs. i can add in more, i actually have the servers sitting around, so i could go as high as 480MB/S.
but i'm thinking that might be overkill, WAY-overkill. if anyone could please tell me what they think of that type of i/o rate...in what scenarios it would be actually used. right now i stripe across a 4-channel ide controller!


THANKS VFX GENIUS'S

[btha8086]

ITANIUM RUNNING 32bit code:

http://www.realworldtech.com/page.cf...2803224105&p=8.

[btha8086]

Itaniums were DOA and passed over, correct?

huh? don't quite follow, (i'm a bit slow)
not sure what you're saying - but from what i understand, mondern primitive's right - the itaniums usually run large databases, business type applications. for instance, m$ just put out an article saying vista would only support true IA_64bit software for business enviornments, whatever the hell that means. but i know there's no such "restrictions" on REAL operating systems
some folks think itanium's future is sketchy, but the itanium >2's< seem to really rock. i've read repeatedly that they benchmark better than even POWER 4'S, which are more expensive.
plus the worlds going 64-bit anyways, the hold up has been the software, which is changing-fast. so personally i believe their future is bright - and the next generation will probably be released sooner-than-later, so they're definately still on the radar

p.s. you may be thinking of 1st generation itaniums, which were short lived. the 2nd generation ("mackinely")came out way too soon afterwards and virtually doubled the performance rate - which made their predeccessors pretty much obsolete.

and get this, the next version of I2's - the "montecitos" - will include DUAL-CORES, and (don't pee yourself) - 24mb L3 cache. anybody care to guess how many gigaflops that'll acheive?
but wait! hold onto ya trousers! the tangelwoods are projected to have...(drum roll please)... ...16 cores. moore's law says what - 6 more years before the *average* renderfarm will be measured in TERAflops?