Lies, damn Lies and Benchmarking!

Image

vs

Image

Making decent and objective Benchmark test is quite hard almost impossible task.

Why? as the different systems should be put in equal testing conditions.

Good example is the ARM commercial compilers, if you go to the web site of Keil, IAR, etc you will see on their web sites benchmarks which show how their compiler is best as code, speed etc.

How this is done? There is always example which you can make for your specific compiler which to execute faster and to have shorter code than the competitor, so this is cheap trick which less and less novices buy.

All these commercial compilers usually show GCC as slowest and less optimized compiler, needless to say they have run these tests with questionable options :-) real tests show that almost all ARM compilers produce about same code, the variations are few percents, also most of commercial compilers would fail to do the job where GCC shines, i.e. to compile and link HUGE projects like GB sources of Linux Kernel, so the focus which compiler to use should be on what additional value these compilers have around them, i.e. what flash loaders, processor support, demo examples are available which to kickstart your development.

So if we benchmark same architectures, on same code and same compiler with same settings it’s possible to have something relevant, but how do we benchmark different architectures?

Let’s face it – this is not possible to be done objectively and the recent debates for AnTuTu fake benchmarking for Intel new processors prove it.

In June, The Reg reported analyst firm ABI Research’s claim that it had pitted a Lenovo K900 smartphone based on Intel’s Atom Z2580 processor against a brace of devices build around ARM system-on-chip (SoC) components and found that not only did the Intel part perform better, but it also drew less power.
Jim McGregor of analyst firm Tirias Research smelled something fishy, and after investigating, he now says the surprise showing by Intel had less to do with the chip itself as it did with inconsistencies in the AnTuTu benchmark used to conduct the tests.
McGregor’s first clue was that different versions of the benchmark produced wildly different results.
“Going from the 2.9.3 version of the AnTuTu benchmark to the 3.3 version, the overall AnTuTu scores increased 122 percent, and the RAM score increased 292 percent for the Intel processor, while the scores for the Samsung processor increased only 59 percent and 53 percent, respectively,” McGregor wrote in a blog post at EE Times. “This was just from a change in the benchmark test, not the processors.”
Versions of AnTuTu for ARM chips are built using the open source GCC compiler. But beginning with version 2.9.4, AnTuTu for Intel is built using ICC, a proprietary optimizing compiler designed by Intel itself.
Working with AnTuTu and technology consulting firm BDTI, McGregor determined that the version of the benchmark built with ICC was allowing Intel processors to skip some of the instructions that make up the RAM performance test, leading to artificially inflated results.
AnTuTu released version 3.3.2 of the benchmark on Wednesday to address the problem, and according to McGregor, it negates Intel’s artificial advantage. Intel’s CPU and Overall scores are now about 20 per cent lower than they were with the previous build, and the RAM score is around 50 per cent lower.

It’s still questionable if these new results are valid as AnTuTu didn’t explain what changed in the benchmark – the result AnTuTu is IMO totally compromised as reliable source for benchmarking, switching to Intel compiler smells of corruption ;)

more on this subject:

http://www.theregister.co.uk/2013/07/12/intel_atom_didnt_beat_arm/

http://www.theregister.co.uk/2013/06/14/intel_clover_trail_plus_benchmark_comparison_with_arm/

http://www.eetimes.com/author.asp?section_id=36&doc_id=1318857

http://www.eetimes.com/author.asp?section_id=36&doc_id=1318894

http://news.cnet.com/8301-1001_3-57593426-92/debate-sparked-about-benchmark-for-intel-arm-chips/

DUINOMITE BASIC INSTRUCTIONS PER SECOND SPEED (BIPS)

Image

We got question on Duinomite support forum if small delays of 10 uS could be implemented with Duinomite Basic (DM-BASIC).

Duinomite Basic is interpreted language and it’s relatively slow, I write relatively as for some applications it’s still fast but for other slow. So I was interested how much time DM-BASIC needs to execute one BASIC instruction.

It was easy to implement as DM-BASIC have TIMER which increments every 1 mS, so I wrote this code:

10 TIMER = 0     ‘ clear the TIMER

20 I = I + 1         ‘ do one basic command like adding 1 to I variable

30 IF TIMER < 1000 THEN 20   ‘and execute this command until 1 second pass

40 PRINT “Speed approx.: “;I*3;” BIPS” ‘ print the result

RUN

Speed approx.: 31044 BIPS

BIPS = BASIC instructions per second

In the result we multiply I * 3 as the IF statement is composed by two instructions – the IF and the GOTO which forward the code to line 20

so the one simple BASIC instruction in DM-BASIC takes approximately 32 uS.

32 uS is slow for some processes, but if you want to do something fast you can always include it in the firmware as it’s open source and available for users to modify. this way was added support for the iButtons they require uS timing for communication and was impossible to do this in BASIC, so Gerard Sexton wrote additional BASIC commands which to communicate with MAXIM-DALLAS iBUTTONs.

Follow

Get every new post delivered to your Inbox.

Join 572 other followers