True Performance Comparison?

Only if you're planning on writing your programs in assembly language.

The only thing that matters in the end is the performance of the hardware/toolchain _combination_.

The performance of hand-tuned assembly only matters if you plan on writing your application in hand-tuned assembly.

--
Grant Edwards                   grante             Yow!  If elected, Zippy
                                  at               pledges to each and every
                               visi.com            American a 55-year-old
                                                   houseboy...

With larger projects, the view of software changes from a bits-and-bytes endeavor to an engineering endeavor. Creating code in 'C' is cheaper, pound for pound, then doing it in assembly language.

The goal is to finish the project. The goal is not to display prowess under the processor architecture.

Thus, I would argue that comparing 'C' is fair. It fairly reflects what will be done in practice. It is true that bad compilers will penalize the processor designers for inefficiences they did not create, but who said life is fair?

I think the lesson to be learned is clear. If you're designing a new processor, pay very close attention to the tool chain that will or will not be available to support it ...

Dave

No. Instead, using assembly would be unfair. How are you going to compare the following ARM instruction

ldreqb r0, [r1, -r2, lsr #27]

with equivalent for, say, AVR? For '51? For PIC?

For the record, this instruction loads byte (with zero extension) into the register r0 from the memory at base address in r1 minus offset in r2 shifted logically right 27 bits. And it only takes effect if Z flag is set. With proper sequencing of commands (eliminating register dependencies) this command executes in exactly one cycle.

"Use the proper compiler" is usually the answer. And if the code is "really" incorrect, what is the point of comparing it?

*

Vadim

IMHO, that kind of performance comparison is primarily of interesting to the marketing department (and perhaps CS professors). Engineers trying to get a product ready for market are more interested in the tool chain ease of use, power efficiency, memory size and MCU and support chip costs. The cycle-by- cycle efficiency of one instruction set over another isn't all that important when compared with the multitude of choices of tools, memory and package sizes, power efficiency, and chip cost.

Then you have to cope with the fact that instruction set efficiency measures depend a lot on the KIND of code you are writing. Some processors have horrible times with high-level languages, but do very nicely at bit-twiddling. (Else why would anyone buy a PIC chip???).

Mark Borgerson

To me, a high performance compiler is the one that lets me write the most functionality per time unit while being able to decipher it 5 years later with the usual documentation of the code. It lets me change some functionality 10 years later in the shortest time, by reading the available code documentation.

The amount of ASM statements is not reaaly interesting. The amount of flash is, hmm, tiny compared to 10 years later. The clockspeed of the cpu, is hmm, well it could take less current.

Rene

--
Ing.Buero R.Tschaggelar - http://www.ibrtses.com
& commercial newsgroups - http://www.talkto.net

Hi Michael,

I know we have discussed this before - just for the record below is a copy of my previous communication on the results published on the FreeRTOS WEB site (see

"I don't think an absolute and undisputable measure of a microcontrollers performance is practical when writing programs in C. What the results show is a 'comparison' of several 'systems' being used in a 'normal' manner.

'Comparison' because I don't attempt to give an absolute measure - only provide results so different systems can be compared to each other.

'System' in that both the compiler and hardware are included.

'Normal' in that no attempt is made to optimise to the particular hardware. Only C code is used. If for example the hardware included a hardware multiplier then the test does not specifically write assembler code to ensure the multiplication is done in the fastest possible way."

This is a topic that people often have a strong opinion on. As with everything it is always open to interpretation and dependent on what you want to do. Optimising for a particular microcontroller is contrary to wanting to write portable code for instance.

Regards, Richard.

michael_r snipped-for-privacy@yahoo.com (Michael Monteith) wrote in news: snipped-for-privacy@posting.google.com:

It depends on the type of performance comparison you are interested in. If you want to compare processors for a specific application, then it would make sense to use the development tools that would be used for said application to develop the benchmarks, and compare the results.

If you are strictly interested in processor performance, then, hand coding the most optimal assembly language using a very good assembly programmer could make sense.

The latter seems like a useless comparison for real world applications, unless they are also to be coded in assembly by a highly skilled assembly language programmer in the respective assembly languages.

--
Richard

That is why you never compare Apples to Oranges. Are you comparing Languages? CPUs, Languages? How good in the assembly? It is very possible to write bloated slow Assembly code. That is why bench marks should be taken with a grain of salt.

It depends upon which of the two questions you want to ask:

1) Is processor x faster than y? 2) Is the response time on x faster than y with application z?

If you're looking for raw processor performance testing (#1) to figure out if x is faster than y, then yes, assembly language is the only way to do it.

If you're looking to answer whether or not a computing experience is faster, then it's all going to be down to a combination of how well the program is written, what language it's in, and what compiler it's running on.

Now plenty of people have taken an example of #2 above and conclude that it means #1, including and especially Apple (remember the "G5 is the fastest computer on the planet" ads? That's blatant false advertising). Just because one app runs faster than another on a given platform doesn't mean that the platform is faster.

As someone already pointed out, the only thing that really matters is speed perception to the end user. To directly answer your question above, yes, any other programming language would absolutely be unfair if #1 is the goal.

-->Neil

Well, that would indicate that the ARM is capable of doing something in 1 instruction that the AVR, 8051, and PIC can't do in any less than a much larger number of instructions, therefore I'd conclude the ARM is faster than all 3. ;-)

-->Neil

For that kind of application you may be better off with a much simpler and less powerful processor and a CPLD or FPGA to do the shift register work.

--
Ben Jackson

http://www.ben.com/

Three words for you: $,$,$ :) Yes, it's been suggested. Blackboxing the problem like this always turns out to be more money though, even in an ASIC.

I'll put it bluntly. Performance as such is a non-entity. It simply doesn't exist. Might as well try to compare beauty.

Depends. On what performance it is you want to compare. In what circumstances.

That pair of statements is plain nonsense. Optimizing more or less well is not a question of generating correct code. If the code isn't correct, it would be terminally stupid to even let it participate in any performance comparison.

--
Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de)
Even if all the snow were burnt, ashes would remain.

Damn. There go my plans for the world's most optimizing C compiler:

---------------------------------8

LOL, no kidding. They wouldn't adevertise and fight over who optimizes the best. Guess I've seen WAY too many instances of compilers generating extra code only to make things easier on themselves. Ever looked at what compilers really generate and ask yourself "why in the world did they do that?" I guess in these days people don't care as long as the code works. I guess I came from a time when you had limited processing power and you squeezed every bit out of it and never assumed anything about a compiler. It certainly plays in when you're wanting to do performance comparisons.

Michael