Thread metrics

I have completed some of the thread metric tests for a few RTOS.  I will be posting the results in this forum.  The test may be limited by the demo versions available and the capabilities of each RTOS.  If a particular vendor does not wish me to publish the performance figures then they should send me a private message saying so.  (I will send this post via email to vendors).   Open source projects should, by principle, should have no objection, and neither should ThreadX.   To be fair, I WILL post that company xyz has informed me not to publish the results.  Users of the forum can form their own opinions.
 
The RTOS that I have selected are
AVA - demo version, does not include semaphore and message Qs (I have an older version but that would not be fair)
AVIX - I have verified results on avix-rt web site
FreeRTOS
ThreadX  demo version
uCosII demo version
 
perhaps Picos - it is difficult with compile time only thread creation
perhaps TNKernel
Others?   
 
I am doing this work on my own free time.  The threadx thread metrics license specifically states that the test suite should not be used in the development of a RTOS.  I am not developing an RTOS.  I may also publish the source code on my own web site, if I ever get it back up.  Companies should refrain from asking for the source code since that may violate the license agreement.  To be fair, companies may have the right to inspect the specific build environment, compiler options, configuration, and the specific RTOS porting source code file so that they can verify that I have, to the best of my ability, correctly represented the RTOS.
 
This test suite, as with any benchmark or test suite, may not be comprehensive and any conclusions drawn are ones own responsibility.  I make no recommendations based upon the results.
 
I have not bothered with implementing or testing memory pools because of limited resources.  I am open to expanding the test suite and will accept source code contributions.  (mutex, pipes, heavy interrupt, timers, ...)
 
The selected processor is the p24H256GP610 although I would like to include a 33F given time.

Results for FreeRTOS
 
**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 30
Time Period Total:  89752
**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 60
Time Period Total:  89752
**** Thread-Metric Cooperative Scheduling Test **** Relative Time: 30
ERROR: Invalid counter value(s). Cooperative counters should not be more that 1 different than the average!
Time Period Total:  0
CORE-E0001: Trap due to stack error, occurred from instruction at 0x001d9e
CORE-E0003: Trap due to unimplemented RAM memory access, occurred from instruction at 0x001d9e

**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 30
ERROR: Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!
Time Period Total:  3717913
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 60
ERROR: Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!
Time Period Total:  3717884
 
**** Thread-Metric Interrupt Processing Test **** Relative Time: 30
Time Period Total:  1881892
**** Thread-Metric Interrupt Processing Test **** Relative Time: 60
Time Period Total:  1881849
 
**** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 30
Time Period Total:  2400967
**** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 60
Time Period Total:  2400906
 
**** Thread-Metric Message Processing Test **** Relative Time: 30
Time Period Total:  484691
**** Thread-Metric Message Processing Test **** Relative Time: 60
Time Period Total:  484685
 
**** Thread-Metric Synchronization Processing Test **** Relative Time: 30
Time Period Total:  1989999
**** Thread-Metric Synchronization Processing Test **** Relative Time: 60
Time Period Total:  1989952
 

results for ThreadX
 
Demo version is crippled.  Throws an address trap when built with any optimization level other than 0.  Must set up a timer otherwise thread sleep is infinite.  What can one do with a 2 hour demo time limit? 
 
**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 30
Time Period Total:  37667
**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 60
Time Period Total:  37667

**** Thread-Metric Cooperative Scheduling Test **** Relative Time: 30
Time Period Total:  10777579
**** Thread-Metric Cooperative Scheduling Test **** Relative Time: 60
Time Period Total:  10777414
 
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 30
Time Period Total:  2825565
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 60
Time Period Total:  2825508
 
this may have to be revisited.  The limitation of no local variables in an ISR is weird.
this may be slower if a C function is called within the ISR.
**** Thread-Metric Interrupt Processing Test **** Relative Time: 30
Time Period Total:  7717960
**** Thread-Metric Interrupt Processing Test **** Relative Time: 60
Time Period Total:  7717854
 
this may have to be revisited.  The limitation of no local variables in an ISR is weird.
this may be slower if a C function is called within the ISR.
**** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 30
Time Period Total:  2291784
**** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 60
Time Period Total:  2291753
 
**** Thread-Metric Message Processing Test **** Relative Time: 30
Time Period Total:  3363496
**** Thread-Metric Message Processing Test **** Relative Time: 60
Time Period Total:  3363454
 
**** Thread-Metric Synchronization Processing Test **** Relative Time: 30
Time Period Total:  10224691
**** Thread-Metric Synchronization Processing Test **** Relative Time: 60
Time Period Total:  10224467

I took the liberty to make a graphical representation of the results with the AVIX results included. You can see it below.
The Three RTOS's are placed in alphabetical order.
 
 


[image]local://30578/2732BAD7CEE242FFB77CD7257DFA09FD.gif[/image]

Attached Image(s)

I have re-run the ThreadX version of the ThreadMetrics test.  Another pair of eyes found a problem in the code.  I was not calling the _tx_thread_context_save and  _tx_thread_context_restore functions in the isr tests.  Now I find it strange that the RTOS should function at all given this omission.  I also added the memory test, but I havd to correct the test suite since it would fail.
 
 
Demo version is crippled.  Throws and address trap when built with any optimization level other than 0.
**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 30
Time Period Total:  37667
**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 60
Time Period Total:  37667

**** Thread-Metric Cooperative Scheduling Test **** Relative Time: 30
Time Period Total:  10777579
**** Thread-Metric Cooperative Scheduling Test **** Relative Time: 60
Time Period Total:  10777414
 
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 30
Time Period Total:  2825565
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 60
Time Period Total:  2825508
 
**** Thread-Metric Interrupt Processing Test **** Relative Time: 30
Time Period Total:  5669870
**** Thread-Metric Interrupt Processing Test **** Relative Time: 60
Time Period Total:  5669870
 
**** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 30
Time Period Total:  2125286
**** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 60
Time Period Total:  2125258
 
**** Thread-Metric Message Processing Test **** Relative Time: 30
Time Period Total:  3363496
**** Thread-Metric Message Processing Test **** Relative Time: 60
Time Period Total:  3363454
 
**** Thread-Metric Synchronization Processing Test **** Relative Time: 30
Time Period Total:  10224691
**** Thread-Metric Synchronization Processing Test **** Relative Time: 60
Time Period Total:  10224467

**** Thread-Metric Memory Allocation Test **** Relative Time: 30
Time Period Total:  9570268
**** Thread-Metric Memory Allocation Test **** Relative Time: 60
Time Period Total:  9570147

Had to correct the memory pool test since it would crash with an address trap.
This is similar to the source found in "demo_threadx.c" supplied with the PIC demo.
 
int  tm_memory_pool_create(int pool_id)
{
  UINT    status;
  /*  Create the memory pool.  */
  status =  tx_block_pool_create(&tm_block_pool_array[pool_id], "Thread-Metric test",
  sizeof(long), &tm_pool_memory_area[pool_id*TM_THREADX_MEMORY_POOL_SIZE], TM_THREADX_MEMORY_POOL_SIZE);
 
 
 

AVIX added at Microchip 3rd Party RTOS page:
http://www.microchip.com/stellent/idcplg?IdcService=SS_GET_PAGE&nodeId=1406&dDocName=en531543

Thanks to bosco for performing the ThreadMetric test.




[image]local://30578/08B972D2D52E4EE58A5C6719E3A8632A.gif[/image]

Attached Image(s)

the winner by 34% ahead of its closest competitor is........

 
For a newcomer I think this really is good or even outstanding. Congratulations zardos
 
When I am ever in need of a dsPIC RTOS I know where to look now.

I notified ucosii about the results a week ago.  No reply.  So I am posting the results.
I had to modify the 33F port provided by ucosii.  They do not provide a port for the 24H.  Simple task really, just remove all the 33F specific registers from the context switch code.
 
 
**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 30
Time Period Total:  37502
**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 60
Time Period Total:  37502
 
ucosii does not have a cooperative scheduler.  This should not be perceived as a hindrance since
it offers more than enough unique thread priorities.

**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 30
ERROR: Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!
Time Period Total:  2748027
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 60
ERROR: Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!
Time Period Total:  2747981
 
**** Thread-Metric Interrupt Processing Test **** Relative Time: 30
Time Period Total:  3733679
**** Thread-Metric Interrupt Processing Test **** Relative Time: 60
Time Period Total:  3733591
 
**** Thread-Metric Interrupt Preemption Processing Test ****
ucosii does not allow a call to OSTaskResume() from within an ISR.
 
**** Thread-Metric Message Processing Test **** Relative Time: 30
Time Period Total:  734759
**** Thread-Metric Message Processing Test **** Relative Time: 60
Time Period Total:  734748
 
**** Thread-Metric Synchronization Processing Test **** Relative Time: 30
Time Period Total:  7006146
**** Thread-Metric Synchronization Processing Test **** Relative Time: 60
Time Period Total:  7005973

**** Thread-Metric Memory Allocation Test **** Relative Time: 30
Time Period Total:  6814817
**** Thread-Metric Memory Allocation Test **** Relative Time: 60
Time Period Total:  6805827

the winner by 34% ahead of its closest competitor is........

That is an unreasonable claim considering those figures only include 2 other RTOSs.  As a business man you should be more careful on what you say because that does not reflect well on yourself.

Remark accepted. You are right.

ORIGINAL: maxcarson
That is an unreasonable claim considering those figures only include 2 other RTOSs. 

 
You could always help.  Pick an RTOS and apply the thread metric suite.  I am now working on AVA.  There is PICOS, which will be challanging because of the compile time only allocation of resources.  There is TNKernel.  Most of the other commercial RTOS are probably not interested, or they have their own internal benchmarks.

I am interested to see the tnkernel benchmark. I am between microc/os and tnkernel for my next project.
 

As for TNKernel (the last release of PIC24/dsPIC port, I'll post it soon):

 optimization:
   tnkernel lib = 02
   code         = 0s
 
 mem model:
   tnkernel lib = all code and data large
   code         = all code and data large
 
 
 1.1. Basic Processing Test
 =============================================================================
 
 **** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 30
 Time Period Total:  83568
 
 **** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 60
 Time Period Total:  83559
 
 
 1.2. Cooperative Scheduling Test
 =============================================================================
 
 TNKernel don't have cooperative scheduling, bacause if have round-robin for
 tasks with the same priority
 
 
 1.3. Preemptive Scheduling Test
 =============================================================================
 
 **** Thread-Metric Preemptive Scheduling Test **** Relative Time: 30
 Time Period Total:  3359814
 
 **** Thread-Metric Preemptive Scheduling Test **** Relative Time: 60
 Time Period Total:  3359296
 
 
 1.4. Interrupt Processing Test
 =============================================================================
 
 **** Thread-Metric Interrupt Processing Test **** Relative Time: 30
 Time Period Total:  5497238
 
 **** Thread-Metric Interrupt Processing Test **** Relative Time: 60
 Time Period Total:  5497907
 
 
 1.5. Interrupt Preemption Processing Test
 =============================================================================
 
 **** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 30
 Time Period Total:  2693630
 
 **** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 60
 Time Period Total:  2693429
 
 
 1.6. Message Processing Test
 =============================================================================
 
 **** Thread-Metric Message Processing Test **** Relative Time: 30
 Time Period Total:  4146914
 
 **** Thread-Metric Message Processing Test **** Relative Time: 60
 Time Period Total:  4147213
 
 
 1.7. Synchronization Processing Test
 =============================================================================
 
 **** Thread-Metric Synchronization Processing Test **** Relative Time: 30
 Time Period Total:  7353579
 
 **** Thread-Metric Synchronization Processing Test **** Relative Time: 60
 Time Period Total:  7353022
 
 
 1.8. RTOS Memory allocation
 =============================================================================
 
 **** Thread-Metric Memory Allocation Test **** Relative Time: 30
 Time Period Total:  5933761
 
 **** Thread-Metric Memory Allocation Test **** Relative Time: 60
 Time Period Total:  5933391
 

Hi Alex,
 
These are good results, congratulations.
 
I think it is fair that you publish your thread-metric port code on your website so the results can be verified by bosco, just like I did with AVIX.
 
I put the results until now in a graph you see below.
 
Note the AVIX figures are based on version 2.01 which will be published beginning next week and these results too are not yet verified by bosco.
 
 


[image]local://30578/691A39304F1A473788EE5AD776A7EDF7.gif[/image]

Attached Image(s)

Thank you Leon
Thread-metric port for TNKenrne + PIC24 link is
http://www.pic24.ru/upload/tnk_thread_metric.zip

Just create some MPLAB workspace and add all projects into it.
I create porting layer file for every test, because TNKernel use different services for the task and for the interrupt context.

By the way - thanks a lot to kt0157 for explain a errors and bugs of my port of TNKernel. And remember, I only port TNKernel for PIC24/dsPIC, author page is tnkernel.com
I'll post new TNKernel for PIC24/dsPIC sources soon

Thread Metrics for AVA
Kernel & modules & apps built with -O2, Optimization makes significant difference
Memory model Large Data, all others default
Modules Debug off
The source code is not available, commercial product.

**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 30
Time Period Total:  52669
**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 60
Time Period Total:  52676
 
Thread-Metric Cooperative Scheduling Test
N/A
 

**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 30
Time Period Total:  1724948
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 60
Time Period Total:  1725185
 
**** Thread-Metric Interrupt Processing Test **** Relative Time: 30
Time Period Total:  5207662
**** Thread-Metric Interrupt Processing Test **** Relative Time: 60
Time Period Total:  5208348
 
**** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 30
Time Period Total:  1260190
**** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 60
Time Period Total:  1260445
 
**** Thread-Metric Message Processing Test **** Relative Time: 30
Time Period Total:  2761154
**** Thread-Metric Message Processing Test **** Relative Time: 60
Time Period Total:  2761511
 
**** Thread-Metric Synchronization Processing Test **** Relative Time: 30
Time Period Total:  7514799
**** Thread-Metric Synchronization Processing Test **** Relative Time: 60
Time Period Total:  7515821
 
**** Thread-Metric Memory Allocation Test **** Relative Time: 30
Time Period Total:  10235182
**** Thread-Metric Memory Allocation Test **** Relative Time: 60
Time Period Total:  10236999

Thread Metrics ucosii  Update
Large data, large scalar, others default
Does not build with optimization other than -O0.  Most files will build with
-O2 and those files that fail are selectively set at -O0.
 lib_mem.c
 os_flag.c

**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 30
Time Period Total:  105693
**** Thread-Metric Basic Single Thread Processing Test **** Relative Time: 60
Time Period Total:  105692

Thread-Metric Cooperative Scheduling Test
N/A
 
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 30
ERROR: Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!
Time Period Total:  3909085
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 60
ERROR: Invalid counter value(s). Preemptive counters should not be more that 1 different than the average!
Time Period Total:  3909052

**** Thread-Metric Interrupt Processing Test **** Relative Time: 30
Time Period Total:  5259998
**** Thread-Metric Interrupt Processing Test **** Relative Time: 60
Time Period Total:  5259878

**** Thread-Metric Interrupt Preemption Processing Test ****
ucosii does not allow a call to OSTaskResume() from within an ISR.
 
**** Thread-Metric Message Processing Test **** Relative Time: 30
Time Period Total:  7387612
**** Thread-Metric Message Processing Test **** Relative Time: 60
Time Period Total:  7387447
 
**** Thread-Metric Synchronization Processing Test **** Relative Time: 30
Time Period Total:  10293318
**** Thread-Metric Synchronization Processing Test **** Relative Time: 60
Time Period Total:  10293006
 
memory test unchanged since the RTOS would
CORE-E0002: Trap due to misaligned data word access, occurred from instruction at 0x000d18
for any oprimization onther than -O0
 
 

Here are the results for the new AVIX 2.01 release which is available since today.

Downloadable from www.avix-rt.com is the testsuite with the AVIX port of the Thread-Metric interface layer in order to make the testresults veriviable.

Results of all tests, together in a graph can be seen below:



**** Thread-Metric Cooperative Scheduling Test **** Relative Time: 30
Time Period Total:  8880289
**** Thread-Metric Cooperative Scheduling Test **** Relative Time: 60
Time Period Total:  8880086

**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 30
Time Period Total:  4775165
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 60
Time Period Total:  4774991

**** Thread-Metric Interrupt Processing Test **** Relative Time: 30
Time Period Total:  7051598
**** Thread-Metric Interrupt Processing Test **** Relative Time: 60
Time Period Total:  7051440

**** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 30
Time Period Total:  3050405
**** Thread-Metric Interrupt Preemption Processing Test **** Relative Time: 60
Time Period Total:  3050326

**** Thread-Metric Message Processing Test **** Relative Time: 30
Time Period Total:  4705841
**** Thread-Metric Message Processing Test **** Relative Time: 60
Time Period Total:  4705739

**** Thread-Metric Synchronization Processing Test **** Relative Time: 30
Time Period Total:  18729971
**** Thread-Metric Synchronization Processing Test **** Relative Time: 60
Time Period Total:  18729411

**** Thread-Metric Memory Allocation Test **** Relative Time: 30
Time Period Total:  10702802
**** Thread-Metric Memory Allocation Test **** Relative Time: 60
Time Period Total:  10702506


[image]local://30578/E7BDEB2340B545ABBD28FFC59FF3B8F0.gif[/image]

Attached Image(s)

And here are the latest results of all tests in numeric format where green cells represent the fastest result for a test.
 
 


[image]local://30578/2C4090E5F3DB4431B16A0500603DA213.gif[/image]

Attached Image(s)