• AVR Freaks

Helpful ReplyXC16 - ELF file output - Symbols have zero "st_size"

Page: 12 > Showing page 1 of 2
Author
drodgers_ls
Starting Member
  • Total Posts : 9
  • Reward points : 0
  • Joined: 2013/02/08 15:02:58
  • Location: 0
  • Status: offline
2013/03/21 22:22:57 (permalink)
5 (1)

XC16 - ELF file output - Symbols have zero "st_size"

So, I've hit a brick wall, and I'd like to know if this is in fact a bug/oversight in the XC16 compiler, or if there's a reason behind what I'm seeing.

I've written an ELF file parser in C# to extract symbol information from a firmware image produced by the XC16 compiler, specifically for a dsPIC33 project.  We need to know the location and size of all the global variables in the ELF file in order to inspect memory contents.  I've completed my parser, and am able to extract the main header, section headers, program headers, section header name string table, symbol name string table, and symbol table entries, and display them all visually.  I've attached two dumps, one from the Microchip CE437 UART code example (with a few random globals tossed in), and one generated from an ELF file built by the Atmel AVR32 toolchain.
 
Unfortunately, I'm finding that in ELF files built by the XC16 compiler (I've tried multiple projects), the "st_size" field for every symbol is set to 0.  This field is supposed to declare the size of the item in question... for data items, it's the size of the variable; for functions, it's the size of the compiled code.  Without the "st_size" field, we have no way of knowing whether a given symbol is a char, int, long, or larger.  We don't need type information necessarily, but at a minimum, we need to know the range in memory the data symbol occupies.

ELF file documentation here (this is the section on symbol table entries): http://www.sco.com/develo...latest/ch4.symtab.html

Our customer requires the ability to parse ELF files, so we have to have the capability to read symbol names, addresses, AND sizes from ELF files.  If we had the size information, we'd be done.  Instead, I may now have to implement a DWARF debug info parser, for the sole purpose of extracting information that should be readily available in the ELF file.  For giggles, I shoved a different ELF file through my parser (from an older AVR32 project), and all of the function and data symbols in that file have non-zero "st_size" fields, as expected.  So this apparent defect is particular to the Microchip XC16 compiler.

Suggestions?
#1
shibanifa
Junior Member
  • Total Posts : 116
  • Reward points : 0
  • Joined: 2010/08/13 13:21:56
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/03/26 14:28:35 (permalink)
0
Seems like a bug.
Will be fixed for future releases.
 
Sorry for the inconvenience.
#2
drodgers_ls
Starting Member
  • Total Posts : 9
  • Reward points : 0
  • Joined: 2013/02/08 15:02:58
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/03/26 14:59:00 (permalink)
0
shibanifa Seems like a bug.
Will be fixed for future releases.

Sorry for the inconvenience.

Hi shibanifa, thanks for the reply.  Are you an employee of Microchip and/or a developer of the XC16 compiler?  I'm hoping that someone from Microchip can address this issue.  If you are an XC16 developer, can you provide a rough idea of how long it generally takes to release a new version of the compiler?  I would like to give my customer some idea of how long they may have to wait to see the issue resolved.  Thanks.
#3
Taliesin
Super Member
  • Total Posts : 130
  • Reward points : 0
  • Joined: 2008/10/08 11:39:35
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/03/27 07:15:05 (permalink)
0
I'd log an official support ticket if I were you -- that's definitely a bug, and an easy one to reproduce.
http://www.microchip.com/techsupport/
 
Taliesin
#4
drodgers_ls
Starting Member
  • Total Posts : 9
  • Reward points : 0
  • Joined: 2013/02/08 15:02:58
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/03/27 07:25:26 (permalink)
0
Taliesin I'd log an official support ticket if I were you -- that's definitely a bug, and an easy one to reproduce.
http://www.microchip.com/techsupport/

Taliesin

Thanks, I think I'll do that.
#5
Legato13
New Member
  • Total Posts : 2
  • Reward points : 0
  • Joined: 2013/06/11 02:59:08
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/06/11 03:16:02 (permalink)
0
Dear drodgers_ls,
 
I'm working on a same project as you worked on before.
I've successfully read the headers, the symbols and the string tables of the elf file with C#. But, as you wrote, the size field of the symbols contains only zeroes.
Have you found any solution since you had this problem?
Thank you in advance,
 
Legato13
#6
drodgers_ls
Starting Member
  • Total Posts : 9
  • Reward points : 0
  • Joined: 2013/02/08 15:02:58
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/06/11 07:33:11 (permalink)
+1 (1)
Legato13 I'm working on a same project as you worked on before.
I've successfully read the headers, the symbols and the string tables of the elf file with C#. But, as you wrote, the size field of the symbols contains only zeroes.
Have you found any solution since you had this problem?

Unfortunately, no, I have not, not directly anyway.  I did submit this as a support ticket to Freescale, and they acknowledged that it is a defect and will be fixed at some future time.  Because I also needed to access struct and array members, I then had to write a DWARF parser, to parse the .debug_info/abbrev/str sections.  I then discovered a second bug, in that all of the DWARF sections are "striped" with zeroes; that is, for every byte in each DWARF section, there is an additional 0x00 byte inserted.  Microchip also acknowledges this as a bug in the XC16 compiler.  This is why, if you take an XC16 ELF object and run it through dwarfdump, dwarfdump fails.  I wrote a C# utility to strip out the extraneous zeroes and repackage the ELF file, and the repackaged ELF file mostly passes dwarfdump (there is an error parsing .debug_frame, but I don't need that section).
 
You may be able to get the symbol size information you need by parsing the DWARF info and then correlating that information back to the ELF symbols, but it's considerably more complicated.  Writing a DWARF parser is not trivial.
 
Suffice to say, I am rather dismayed at the quality of the XC16 compiler, as it's clear that Microchip never tested the output file formats with any well-known third-party tools.
#7
Legato13
New Member
  • Total Posts : 2
  • Reward points : 0
  • Joined: 2013/06/11 02:59:08
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/06/12 10:10:56 (permalink)
0
Thank you for the information and the quick reply.
So if I write a DWARF parser, will I be able to get the inner elements of the structures and the parameters of arrays?
I would not like to use third-party tool, if I can write my own parser.
#8
drodgers_ls
Starting Member
  • Total Posts : 9
  • Reward points : 0
  • Joined: 2013/02/08 15:02:58
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/06/12 11:56:07 (permalink)
+3 (3)
Legato13 So if I write a DWARF parser, will I be able to get the inner elements of the structures and the parameters of arrays?
I would not like to use third-party tool, if I can write my own parser.

Yes, you can write your own parser.  It's trickier than an ELF parser, the data structures are completely different.  I'd provide the parser I wrote, but it was paid for by our customer.  I can provide some basic pointers, though:
  1. You'll need to extract the .debug_info, .debug_abbrev, and .debug_str sections from the ELF parser; these contain your symbol information.
  2. You'll need to fix the XC16 brain damage by removing the "stripes" of zeroes in the DWARF data.  If you read in each DWARF section, write it to a file, and view it in a hex editor, you'll see what I mean, every other byte is a zero.  Just detect and strip out the zeroes before using the data (I say "detect" because hopefully they'll fix the zeroes issue in a future version).
  3. Go to www.dwarfstd.org and download the latest DWARF standard (version 4 is backward-compatible to a version 2 ELF/DWARF file).  Pay attention to the examples provided.
  4. .debug_info contains the actual data for a given DIE or CU header.  .debug_abbrev contains the formatting structures for the debug info.  You parse the debug_abbrev section first, then parse the debug_info section, which references abbreviation entries.
I strongly recommend that you get some ELF files from other sources as well, such as a simple Linux application (build a "Hello World" style application on a Linux box, and turn on debug "-g" when you build it).  Then run those ELF files through "dwarfdump" (you can install a dwarfdump package on any Linux distro, or compile your own using libdwarf).  You'll get a complete breakdown of the DWARF contents, it'll help you see how the data is structured.  I did my basic DWARF parser in about 60-80 hours, then took another 40-80 hours to take the DIE information and convert it into simple nested objects that my application could use.  Good luck.
#9
iwo_andonow
Starting Member
  • Total Posts : 35
  • Reward points : 0
  • Joined: 2008/12/04 04:13:43
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/08/22 07:14:38 (permalink) ☄ Helpful
+1 (1)
Hello drodgers_ls,
I lost 3 days with the same 2 bugs. I just wanted to post the information about the additional byte 0x00 and I found your post.
Other confusing thing is that ELF specification have header for the Note Section - page 2-4 http://refspecs.linuxbase.org/elf/elf.pdf.
The section .debug_info type is SHT_NOTE. DWARF says how the entry begins, but it doesn't say whether it does from the begining of the section or where the "name" part ot Note section header.
I don't know why GCC override the previously defined ELF Note section header (http://www.kpitgnutools.com/ELF_Specifications.pdf) but if somebody is wondering - the CU starts from the begining.
Best Regards
Ivo
 
#10
drodgers_ls
Starting Member
  • Total Posts : 9
  • Reward points : 0
  • Joined: 2013/02/08 15:02:58
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/08/22 07:44:22 (permalink)
+1 (1)
iwo_andonow I lost 3 days with the same 2 bugs. I just wanted to post the information about the additional byte 0x00 and I found your post.

Sorry that you lost time on the same bug that I found.  It just so happens that Microchip Support replied to me half an hour ago, regarding the support ticket I filed months ago regarding the zero-striping issue.  This was their response:
Regarding this issue, the compiler group have indicated that this is intended behaviour and we acknowledge we do not produce compliant DWARF.
So, there you have it.  That's a giant "WONT FIX" from their compiler group.  Apparently they have little interest in conforming to standards.  Do with that as you will.
#11
iwo_andonow
Starting Member
  • Total Posts : 35
  • Reward points : 0
  • Joined: 2008/12/04 04:13:43
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/08/23 01:41:20 (permalink)
+2 (2)
drodgers_ls
Regarding this issue, the compiler group have indicated that this is intended behaviour and we acknowledge we do not produce compliant DWARF.
So, there you have it.  That's a giant "WONT FIX" from their compiler group.  Apparently they have little interest in conforming to standards.  Do with that as you will.

I'm not surprised. I had similar experience with their compiler group before - very big delay in ticket answering and answers like this.
Nobody cares that many people lose days in bugs where they have saved hours for not testing.
I have arround 10 tickets for different bugs and I found the forum much more usefull - usually takes up to 2-3 days rather than few months.
The funny thing with the ticketing system is that the status os the ticket (resolved or not) is marked by the support, not by the ticket issuer. So that they marked it immediatelly as resolved and you don't know whther to wait more or this was all.
Ivo
#12
iwo_andonow
Starting Member
  • Total Posts : 35
  • Reward points : 0
  • Joined: 2008/12/04 04:13:43
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/08/27 08:36:19 (permalink)
0
Hello again,
One more question:
When you parsed debug_abbrev and debug_info there are attributes for reference for example DW_AT_type. The reference is object at the given offset from the begining of the current compilation unit.
Did you noticed that the referenced objects are wrong? For example I have DW_TAG_variable of type struct but the referenced tag is DW_TAG_array_type
Sometimes the second reference is wrong. For example array of structs. This way DW_TAG_variable has reference to DW_TAG_array_type and this tag has reference to DW_TAG_structure_type but the name of the structure is not right.
Did you find such problems or the problem is "in my TV" ?
 
Thanks!
Ivo
#13
drodgers_ls
Starting Member
  • Total Posts : 9
  • Reward points : 0
  • Joined: 2013/02/08 15:02:58
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/08/27 08:44:56 (permalink)
+1 (1)
iwo_andonow When you parsed debug_abbrev and debug_info there are attributes for reference for example DW_AT_type. The reference is object at the given offset from the begining of the current compilation unit.
Did you noticed that the referenced objects are wrong? For example I have DW_TAG_variable of type struct but the referenced tag is DW_TAG_array_type
Sometimes the second reference is wrong. For example array of structs. This way DW_TAG_variable has reference to DW_TAG_array_type and this tag has reference to DW_TAG_structure_type but the name of the structure is not right.
Did you find such problems or the problem is "in my TV" ?

When I wrote my C# ELF/DWARF parser, I only parsed the abbrev, info, and string sections of the dsPIC33 ELF file.  I had a test project with some functioning code as well as some zany nested arrays of structs of arrays and such, and it seemed to parse all that stuff correctly.  However, when we sent our parser to our customer, and they tried it with their dsPIC33 project, they were getting a DWARF parsing error.  After examining their ELF file (by stripping the zeroes, repacking it, then shoving it through dwarfdump), I confirmed that the debug info in that particular ELF file was malformed; there was some issue with the declared length of the CU or something.  So XC16 is capable of generating bad DWARF data for a given project.  I would suggest submitting a ticket if you're certain that you're parsing the DWARF data correctly, but I think we both know how much traction that would gain.  Good luck.
post edited by drodgers_ls - 2013/08/27 08:49:33
#14
iwo_andonow
Starting Member
  • Total Posts : 35
  • Reward points : 0
  • Joined: 2008/12/04 04:13:43
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/08/27 13:17:07 (permalink)
+1 (1)
drodgers_ls 
I confirmed that the debug info in that particular ELF file was malformed; there was some issue with the declared length of the CU or something.  So XC16 is capable of generating bad DWARF data for a given project.  I would suggest submitting a ticket if you're certain that you're parsing the DWARF data correctly, but I think we both know how much traction that would gain.  Good luck.

I found this issue earlier today: If you parse all the entries in the CU (even the null ones) the pointer is not at the same position like if you calculate it with the length of the CU and the starting value of the pointer. The behaviour is different in different compilers (I have tried newer version GCC compiler for ARM). I have fixed this when always recalculate the pointer with the length. This way the compilation unit header is always got from the right place (pointer size = 4, version = 2, offsets whithin the current file, etc.).
Probably there are more bugs in the references between the debug_info entries. The bad thing is that this is my second week working on this and my boss got angry. And what is even worse is that it seems all the work will be useless.
The funny thing is that they have written on page 61 of XC16 User's manual (DS52071B) that "The debugging format used for ELF object files is DWARF 2.0." which is not true! And what is even worse is that the support already confirmed that the don't want to fix ELF/DWARF bugs.
So it will be waste of time to open a ticket.
 
Thank you for your help!
Ivo
#15
jfd
Administrator
  • Total Posts : 24
  • Reward points : 23
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2013/11/12 15:05:16 (permalink)
+1 (1)

Microchip does not produce "standard" DWARF or ELF; for architectural reasons we have differences in our containers. If you have a valid business reason to inspect the ELF/DWARF file please contact your FAE or support.microchip.com and, if warranted, we can follow up with the necessary detail – after the completion of an NDA.

All the best,
jfd

#16
Stampede
Super Member
  • Total Posts : 394
  • Reward points : 0
  • Joined: 2006/10/04 05:59:28
  • Location: Germany
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2017/03/01 01:53:10 (permalink)
0
Hi guys,
 
struggling with the same problem. Does anyone have a free parser that generates "standard" COF or ELF files?
Any help is kindly appreciated.

Regards
Stampede
#17
cawilkie
Administrator
  • Total Posts : 1977
  • Reward points : 0
  • Joined: 2003/11/07 12:49:11
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2017/03/01 07:51:17 (permalink)
+1 (1)
There can be nothing 'standard' about dsPIC(tm) COFF / ELF files, mainly we do this to squeeze 24-bits of information into a 16-bit hole.
 
However, st_size is filled in for function objects.  The DWARF information is standard DWARF-2 (with padding).
 
There are many tools that read Microchip object files/ debug information including objdump and readelf provided with the compiler, and 3rd party tools.
 
Regards
Calum
#18
Stampede
Super Member
  • Total Posts : 394
  • Reward points : 0
  • Joined: 2006/10/04 05:59:28
  • Location: Germany
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2017/03/01 11:26:28 (permalink)
+1 (1)
Hi Calum,
 
the only 3rd party tool that is somehow able to process the XC16 elf is the Visu-It parser. All other parser (gnu binutils report errors) are not able to parse without errors. An the elf-objdump.exe that comes with the XC16 somehow doesn't allow me to see debug info in an easy (like the members of strut).
 
Any ideas on that?

Regards
Stampede
#19
dsPIC33
New Member
  • Total Posts : 2
  • Reward points : 0
  • Joined: 2017/02/22 05:55:12
  • Location: 0
  • Status: offline
Re:XC16 - ELF file output - Symbols have zero "st_size" 2017/04/03 00:13:54 (permalink)
0
Hello. I have the same task on dsPIC33 project. I have to extract names, sizes, addresses of all global variables. Can someone write some guidelines for beginners how to start dealing with that ? Language, what i need to start and how ?? Thank you
#20
Page: 12 > Showing page 1 of 2
Jump to:
© 2019 APG vNext Commercial Version 4.5