Budget Linux cluster suggestions?

For a real computer builder, your way might well be a win.

I'm not one, though, so the amount of time it would take me to build it and especially to code around its peculiarities wouldn't be economic, I don't think. All that copying would have to be fully manual, whereas it's automatic (and very fast) in a big SMP.

I'm also very dubious about using PCIe for all the memory copying that has to happen in this design. Compared with multiple front-side busses in one SMP box, it would be quite slow, wouldn't it? Certainly in big servers, PCIe is the poor relative.

Cheers

Phil Hobbs

ssis

DR3

=A0That

ach,

=A0 $6220 =A0plus 7% tax =3D $6655

he

S)

=3D=3D=3D

=3D||

||||

le

||||

I can build them for you. That's the $2000 NRE for.

Why manual? You just need a memory mode linux network driver. The software would be unchanged from your linux cluster.

Subject to memory/bus contentions, as shown in various benchmarks.

There is no reason why we can't build 2G, 4G, 8G or 16G memory on the local board. All that data would need to go to the hard disk anyway, at a fraction of the PCIe bandwidth.

Because the big server designers are looking backward, not forward.

I hardly care about integer performance--the workload is generally >70% single precision floating point. Opterons rock at FP.

Depends. Having a whole bunch of multiply-accumulates happening in parallel is a big help.

Thanks.

But the memory per node in your design is too small, so it would have to be copied from elsewhere, a lot.

Sure, but yours is the same, isn't it? You're using a narrower, slower bus.

Only after some hours of crunching, generally, and much of it gets thrown away beforehand anyway--the simulation domain is usually quite a bit bigger than the part we actually care about (the extra is needed for accuracy though). The HDD isn't the main bottleneck, by a lot.

That's the speed of a single DDR3 1333 channel (10.6 GB/s). So yes, in this context, it's slow. It also doesn't have cache coherence, unlike the SMP, and there's all that additional copying.

No, because the processor-to-memory bandwidth is over 1 Tb/s per processor. PCIe is a peripheral bus.

Cheers

Phil Hobbs

(Who used to do on-chip optical interconnect research for large servers for a living.)

hassis

DDR3

=A0That

each,

$ 700

=A0 =A0 $6220 =A0plus 7% tax =3D $6655

=A0The

....

But it's relative performance that count. You will have the same bottleneck wiith FP.

re

The benchmarks are comparing pretty much the same class of CPU: PPC and X86. A properly built Xeon (2x6) can outperform an Opteron (2x12) in the benchmark.

LES)

=3D=3D=3D=3D

=3D=3D||

=A0||||

able

=A0||||

We can have 4Gb (512MB) with one SDRAM chip, 8Gb (1GB) with two, 16Gb (2GB) with four and 32Gb (4GB) with eight chips. Routing would be slightly more complicated. Four to Six layers PCB should be enough.

bus.

No, the local memories are dedicated 64 bits.

in

This should really be compared to your 10GbE, not your main memory. We can push as much local memories to the individual processors as possible.

This is same in my design. Only interprocessor comm. goes over PCIe.

I didn't mean to have the memory on PCI or PCIe. It should be on Northbeidge

--------------------------------+ || || || || PCIe cable || || ||

--------------------------------+ || || 1U PCIe/USB host |==== ||

-------------------------+ |============== +------+

Actually, it should be 64 bits 533MHz FSB. with 32Gb/s P2M BW. $5K budge only allow 8Gb(1GB) local memory.

---------------------------------------------------------------------- This is a 32 cores solution. Mount 16 dual core atoms on a thin rack. Run it up to 5m away, not that you have to. PCIe signals run between 2.5Gb/s and 80 Gb/s. Latency is less than a microsecond.

--------------------------------+ || || || || PCIe cable || || ||

--------------------------------+ || || 1U PCIe/USB host |=3D=3D=3D=3D ||

-------------------------+ |=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +------+

FSB: 64 bits 533 MHz Front Side Bus CPU: Intel Atom N270 (x2) 1.6GHz $44 or ARM A9 (x2) 1.5GHz Bridge: Pericom PI7C9X130DNDE $28 MEM: Micron MT46H64M32L2JG $27 MT46H128M32L2KQ $64 uSD: Sandisk 1G MicroSD boot drive $5

PCIe PCB (x16) CPU 88 PCIe to PCI 28 8Gb DRAM (x2) 128 Misc 50 -------------------- 294

------------------------ 4704

Atom is dead slow and dirt cheap, until you have to design with it. It is too damn fast and complicated.

You're going to build a custom motherboard? If the money comes in, I need something working by the first week of September. ASUS for me.

The memory busses are faster, and there are more of them. Each channel is as fast as the whole PCIe, and their latency is much less.

But with the SMP box, I don't need the Ethernet at all, which is more or less the point of the exercise. Plus it's already built--all I have to do is plug in the parts and install the OS.

Sure, but inter-core communication is faster than anything off-chip, including main memory, let alone PCIe.

Cheers

Phil Hobbs

chassis

GB DDR3

. =A0That

M.

=A0 $ 700

=A0 =A0 $6220 =A0plus 7% tax =3D $6655

=A0The

he

e-.....

core

SLES)

=3D=3D=3D=3D

=3D=3D||

=A0||||

cable

=A0||||

it

, I

he

to

No, just processor, memory and PCIe modules.

wer bus.

at

y,

or

es, in

ke

Yes, if you can get an SMP box with enough performance at the right price. I bet you will be disappointed.

On a sunny day (Wed, 18 Aug 2010 19:54:16 -0700 (PDT)) it happened linnix wrote in :

Sorry, I prefer numbers. Let us see, he has 24 cores at maybe 3.3 GHz, and a very powerful architecture. Youe Atom is at most 1.6 GHz and has a lousy architecture, so 1/2 the clock speed and 1/3 the speed because of the architecture, makes 6x slower But you have 2 cores per Atom, so you need 3 x 24 = 72 Atom boards, not counting the fact that data path is shorter from one core to the other in the

cture.

ck

That is unproven claim.

architecture.

The fastest Atom does 6.4 GFLOPS, the fastest i7 55 GFLOPS, the 12 core AMD is supposed to be around double that again.

nix

itecture.

clock

Top Xeon runs at 3.3GHz.

Not sure if the ASUS is using 2.2GHz or 2.4GHz opteron, but not

PCIe is 2.5Gb lane, your bridge is 4 lanes so after data stream encoding and packet over head will give about 800MB usable bandwidth per Atom. This rate will only be with large transfer sizes and higher latency. Smaller transfer sizes will give better latency but lower bandwidth.

Your memory is listed as samples at Micron and is not ECC. For large memory and critical applications ECC it a must along with error rate feedback. Does Atom even support this? One of the major parts of HPC bring up is memory burn in and getting rid of bad memory. You need not only detection but bad sim callout for replacement.

Running PCIe over a cable requires expensive connectors and cable - I didn't see this in your pricing.

There will be a bunch of special bios/firmware/device drivers needed for the PCIe connection. Is this covered in your NRE along with board design?

That is going to be a lot of cores, say 2000? Can you get away with

Hmmm. I don't think 10G Ethernet will be cost effective. I suggest eSATA instead, downright commodity priced and 3 Gb/s.

That is a very challenging price point for that much GFLOPS. In 2001 that would have been a competitive parallel super.

I suggest that you actually consider a lot more memory, say 512 to

Umm, that is GFLOPS, not GOPS.

That will be a wierd but interesting lawsuit, Sun was giving Java executable away free for any porpose.

Not all that similar, MS was trying to take over control of the language.

Andreas

pointers in C,

Yes and no. There is a lot of common dialog boxes that are applicable to almost all applications. These are already pretty well designed, coded up, and paramaterized in readily composable forms available. That constitutes a lot of code reuse in a lot of GUI environments.

OO is useful for low level design but does not always scale well.

What is still lacking is a general method for for decomposing a moderate/medium sized program cleanly. Nor is there any coherent underlying theory with which to do so. Ontology in the applicable area is mostly just arguments, as there are no testable metrics to begin to quantify things with.

Kind of like zoology 150 years ago. We got all kinds of critters but we really don't have much of a clue about how they relate to each other.