Budget Linux cluster suggestions?

I would suggest the dual core atom board for around $100, and sharing power supplies. But 10G T would be expensive. Could you go with multiple 1G T.

Thanks. Network latency is the killer in clusterized EM simulations, more than limited bandwidth. Lots of big clusters use Infiniband. I suppose I could get along without the 10 gig network if I got a really good gbE switch.

Cheers

Phil Hobbs

Don't use switch at all. For example, you can have 16 L1 w/ 1GbE and

Something like this, updated a bit?

Best regards, Spehro Pefhany

You might have a look at . There are two different price tables, one for Reserved Instances and another for Spot Instances. A Cluster Compute Spot Instance "Extra Large" runs about 17 cents per hour at the moment, vs "Quadruple Extra Large" at 81 cents per hour. I haven't looked at the specs on what that particular equipment actually is, but in cases I have looked at, Amazon cloud costs less than owning if you use less than half a cpu year per cpu.

Wow. A house fire waiting to happen!

But a pretty cool approach, especially finding out that IKEA cabinets are ATX-compatible.

Cheers

Phil Hobbs

I thought about that, or rather the Rackspace equivalent. My basement is already pretty jammed, so not having another box down there would be great, and the ability to resize the cluster in a minute or two is very attractive. There were two things that worried me a bit, though:

(1) the learning curve for their control scheme, and (2) possible ITAR restrictions if the project goes to a second phase. I don't think it's possible to guarantee ITAR compliance with a cloud model, more's the pity.

I'm also looking at an Asus dual-processor Opteron 61xx board. With two

Cheers

Phil Hobbs

Well, it all depends on how your software is written. If your software is optimized for clusters, you might not get much improvements in multi-cores. Before you jump into 12-core, I would suggest you try out some 2-cores or 4-cores systems.

Maybe a GPU might be useful, e.g. the NVIDIA Tesla, because then you'll need only one computer:

This one delivers 515 Gflops:

Of course, you'll need a big host and some swapping, because of the 6 GB on board RAM limit of the Tesla cards, but should be much faster than Ethernet transfer speed and latency. With CUDA it is easy to write high performance applications:

I wrote the code myself, so I'm pretty well up on how it works. ;) As long as the processor affinity is set up right, it works better on SMPs because of the low latency. It even works well on a cluster of SMPs--I tuned it on a system like that.

I guess I can try costing it out both ways. The Atoms are cheap enough that I can get a few to try out, I suppose. Having a 24-core desktop machine would give me a certain amount of bragging rights, of course.

Thanks,

Phil Hobbs

CUDA and the AMD equivalent are very interesting, but I know zilch about them. I thought about switching, but there's this learning curve again, and since I'm now coin-operated, it'll be a lot cheaper for the client if I just use general purpose machines. (They will also want to start right away.)

As I understand it, GPUs are only fast on fully vectorized code (i.e. they're SIMD machines), so I'd have to revise the code completely if I don't want to die from the effects of Amdahl's Law.

Or can I make it look like a cluster or a big SMP in some reasonable way?

Thanks

Phil Hobbs

two

ell

But running a 12-core chip scare me. If they are fully utilized, they are 200W to 300W soldering irons. There are claims that the 12-core is not a whole lot better than some 6-core

The AMD ones are 80 and 115 watts rated. And people always claim stuff like that. Seymour Cray described computer design as "bottlneckology", so it all depends on the application. From working at IBM Research, I understand the theoretical issues pretty well. However I know bupkis about what tradeoffs were made in building actual current commercial hardware.

What this application needs, in order of importance, are (a) really really good main memory bandwidth, (2) good inter-core communications, and (c) reasonable Gflops. That's because FDTD has to crunch through all of main memory once per time step. I'm planning to buy the fastest memory the mobo can use, and as much as I can afford.

Cheers

Phil Hobbs

It is not that difficult to write CUDA applications. Take a look at my example code:

All the calculations are in flux-cuda.cu (the source code comments are in English). The main problem is partioning the problem in threads and blocks. This was very easy for my example, see cCalculateFlux: the index for calculating a block is calculated based on the blockIdx and threadIdx, and then I just calculate subsets of the problem in cCalculatePoint in parallel (cCalculateFlux is called in parallel with different values for block and thread). There are some more advanced technics, like synchronization points etc., but you know the concepts already, if you have some experience with parallel programming.

I don't know, if this is as easy for FDTD, but a Google search for FDTD and CUDA shows some interesting results, so maybe you even don't have to implement your own solution, but can buy a finished one or hire someone who can help you.

I don't think so, because as you already noted, it is good at massive parallel vectorized code, but not so good at a small number of high performance serial tasks, but I guess a FDTD grid would fit very well for a massive parallel architecture.

If I were re-coding from scratch, sure. The existing code (over 90k lines, including the front end script) vectorizes reasonably well with an appropriate compiler, e.g. Intel C++, but since it works with chunks of variable length depending on the geometry, I get most of the parallelism by dividing the space up into shoe boxes and giving a few shoe boxes to each processor. With dielectrics, all the equations are the same, which would work beautifully in SIMD, but that isn't true for metals and perfectly-matched layers (PMLs).

The difference in the cost of the machine won't pay for the amount of recoding I'd need to do, not to mention the schedule impact.

If the project goes to phase 2, I'll have an actual budget for doing this, so CUDA and its brethren would be worth looking at, especially if by then I can get more RAM per board. (The code is all single-precision to save memory size and bandwidth.)

Cheers

Phil Hobbs

I guess only a small part of the code is the calculation algorithm and would be easy to port to CUDA, but you are right, that there is a risk that you'll need some weeks for it, if there are some problems, so just buying some hardware and using already working code might be less expensive for the customer and more predictable. But if you have already a NVIDIA or AMD graphics card (CUDA works on cheap consumer graphics cards, too), you should give it a try with a small problem. It's fun to use hundreds of parallel running threads :-)

That's a problem with CUDA: Looks like currently there is a limit of 6 GB for high end CUDA cards (and maybe for AMD cards, too), so you have to transfer it between main system memory and graphics card memory, which makes the implementation of algorithms, which requires much memory, more difficult. But internal bus transfer speed and latency is much lower than Ethernet speed.

Good suggestion. I'll probably do that as a background task when and if the project gets started--it's sort of a seedling, so I'll have probably the equivalent of two months full time.

The program splits the problem domain into 1-D sticks, with each stick being of uniform composition (and therefore the same set of updating equations and coefficients), and then the main loop iterates over the list of sticks. That's about twice as fast as the usual approach, which is to have three nested loops with a gigantic switch statement inside, so that you have to decide what material you're using for every single voxel on every single pass. It also makes it easy to extend, because the updating equations are member functions. There's only one function call per stick per pass, because the member function looks after the iteration.

It was a fun program to write--it took probably six or eight months' worth of my time spread over three years or so.

The cluster configuration is all manual, using a flat file. The plus side of that is that I don't need MPI or anything like that, so my code won't become unusable when the HPC flavour of the month changes. The minus side is that it's fully manual. Fortunately in practice it takes about 10 minutes a week to maintain.

Cheers

Phil Hobbs

Not really a bright idea, but ... the old dual G5 Xserve machines from Apple were coming in at 30 Gflop per node, you could look into getting a dozen of those. On the used market, it wouldn't be too pricey. They rackmount easily, are 1U size.

They're SIMD, of course, and the native OS is unix (but Linux could be put on, too). There's workgroup cluster software already worked out for 'em.

Thanks, I'll look.

Cheers

Phil Hobbs