• AVR Freaks

[Compiler bug?] XC8 v2.05 - Incorrect optimization size of pointer to global variable

Author
adaml
New Member
  • Total Posts : 4
  • Reward points : 0
  • Joined: 2019/11/18 00:43:07
  • Location: 0
  • Status: offline
2019/11/19 07:32:48 (permalink)
0

[Compiler bug?] XC8 v2.05 - Incorrect optimization size of pointer to global variable

Hi all, it's my first post here so hello everybody :) 
 
Some time ago I started some project on PIC18F55K42 using XC8 compiler (and custom makefiles). I've stuck at the moment when I wanted to port my implementation of communication protocol, based of general modbus mechanism (request/response, get register / set register) with some "compressing" method before handling received data. This protocol is working correctly on other MCUs (IAR/GCC compilers).
 
I saw on debug that I have received correctly all frame's bytes (raw input buffer) but after "decompressing" data i saw only zeros in my decompressed structure :( After deeper digging I saw on disassembly view that pointer to my input buffer (placed for example at address 0x240) is correctly placed in common RAM (LSB and MSB bytes), but after jumping into function...I saw that pointer is truncated to 1-byte size and function tries to get data from address..0x40.
 
In map file I saw the same, that pointer is incorrectly "optimized" to 1-byte. Somehow compiler lost tracking to what address pointer is assigned and started treating it as pointer to local variable. What it makes more weird is fact that some decompressing functions are not optimized and behave correctly! Such incorrect behaviour happens only on optimization O2 and higher. Below everything is fine but then I can place only half of files in the project due to exceed flash memory, so I have to use optimization.
 
I've snipped code as much as possible to leave only necessary stuff and here goes the most important parts of the code (I've attached complete project for MplabX)
 
main.c
#include "Protocol.h"

int main(void)
{
   Protocol_Init();

   while(1)
   {
      Protocol_Task();
   }
}

 
Nothing special here, just init protocol module and execute it's task. Of course original code has a lot more tasks.
 
Protocol.c
#include "Protocol.h"

#include <string.h>
#include <stdbool.h>

/************** COMMANDS STRUCTURES ******************/
typedef struct
{
   uint8_t Page;
   uint8_t Index;
} Protocol_CmdReqGetStructure_t;

typedef struct
{
   uint8_t ReqLength;
   uint8_t Page;
   uint8_t Index;
   uint8_t Payload[5];
} Protocol_CmdReqSetStructure_t;

typedef struct
{
   uint8_t ReqLength;
   uint8_t Page;
   uint8_t Index;
   uint8_t Payload[20];
} Protocol_CmdRspGetStructure_t;

typedef struct
{
   uint8_t ReqLength;
   uint8_t Page;
   uint8_t Index;
   uint8_t Payload[20];
} Protocol_CmdRspSetStructure_t;

typedef struct
{
   uint8_t CommandNumber;
   union
   {
      Protocol_CmdReqGetStructure_t ReqGetStructure;
      Protocol_CmdReqSetStructure_t ReqSetStructure;
   } ProtocolCmdReq;
} Protocol_CmdReq_t;

typedef union
{
   Protocol_CmdRspGetStructure_t RspGetStructure;
   Protocol_CmdRspSetStructure_t RspSetStructure;
} Protocol_CmdRsp_t;
/******************************************************************/

 
Whole protocol bases on request - response frames. Request/Commands which don't work correctly are GetStructure (to obtain some data from PIC) and SetStructure (to...set certain data). Page and Index are numbers to address structure. Actually 'Page' number is some kind of group/data type number. Structure addressing is done through it's index. Payload sizes are calculated in more complex way but I wanted to simplify code as much as possible.
 
/************** DATA LINK LAYER STRUCTURES ******************/
typedef uint8_t (*Protocol_CmdHandler_t)(const Protocol_CmdReq_t * pReqCmd, Protocol_CmdRsp_t * pRspCmd);

typedef struct
{
   struct
   {
      uint8_t OutputBuffer[ 200 ];
      uint8_t InputBuffer[ 200 ];
   } Buffers;
   uint8_t Counter;
   uint8_t Length;
} Protocol_Buffers_t;

typedef struct
{
   Protocol_Buffers_t IOBuffers;
   Protocol_CmdHandler_t const * CmdHandlers;
   Protocol_CmdHandler_t CurrentCmdHandler;
} Protocol_t;
/******************************************************************/

/**************** STRUCTURES MANAGEMENT *********************/
typedef struct
{
   uint8_t StructureGroup;
   uint8_t StructureIndex;
   uint8_t ReqLength;
   uint8_t RspLength;
   uint8_t * ReqStructurePointer;
   uint8_t * RspStructurePointer;
   bool (*StructurePacker)(uint8_t *ByteStream, uint8_t *UnpackedStructure);
   bool (*StructureUnpacker)(uint8_t *ByteStream, uint8_t *UnpackedStructure);
   void (*UnpackedStructureHandler)(void);
} Protocol_SetGetStructureCmd_t;
/******************************************************************/

/**************** UNPACKED STRUCTURES ***********************/
typedef struct { uint8_t GetStructure_Data_0[10]; } GetStructure_0;
typedef struct { uint8_t SetStructure_Data_1[4]; } SetStructure_1_Typedef;
typedef struct { uint8_t SetStructure_Data_2[3]; } SetStructure_2_Typedef;
typedef struct { uint8_t SetStructure_Data_3[2]; } SetStructure_3_Typedef;

typedef union
{
   GetStructure_0 GetStructure_0_field;
} Protocol_GetStructuresTypedef;

typedef union
{
   SetStructure_1_Typedef SetStructure_1;
   SetStructure_2_Typedef SetStructure_2;
   SetStructure_3_Typedef SetStructure_3;
} Protocol_SetStructuresTypedef;

typedef union
{
   Protocol_GetStructuresTypedef GetStructures;
   Protocol_SetStructuresTypedef SetStructures;
} Protocol_Structures_t;
/******************************************************************/

 
Data is sent as packed byte stream due to low baudrate (long cables) and packing/unpacking at both nodes is much faster. Due to fact that protocol handles single request when frame is received, any unpacked structure sits in the same place in RAM (that's why I use union instead of structure). Typedef Protocol_SetGetStructureCmd_t  contains all information about certain structure. It's pack and unpack function (via function pointer), address in RAM etc.
 

/**************** COMMANDS HANDLERS DECLARATIONS *************/
static uint8_t Protocol_HandleGetStructure( const Protocol_CmdReq_t * ReqCmd, Protocol_CmdRsp_t * RspCmd);
static uint8_t Protocol_HandleSetStructure( const Protocol_CmdReq_t * ReqCmd, Protocol_CmdRsp_t * RspCmd);
static bool Protocol_ExecuteGetStructureCmd(const Protocol_CmdReq_t * ReqCmd, Protocol_CmdRsp_t * RspCmd);
static bool Protocol_ExecuteSetStructureCmd(const Protocol_CmdReq_t * ReqCmd, Protocol_CmdRsp_t * RspCmd);
/******************************************************************/

/**************** STRUCTURES HANDLERS DECLARATIONS ***********/
static bool SetStructure_NotWorking_1_Unpack(uint8_t * ByteStream, uint8_t * UnpackedStructure);
static void SetStructure_NotWorking_1_Actions(void);
static bool SetStructure_NotWorking_1_Pack( uint8_t * ByteStream, uint8_t * UnpackedStructure);

static bool SetStructure_NotWorking_2_Unpack(uint8_t * ByteStream, uint8_t * UnpackedStructure);
static void SetStructure_NotWorking_2_Actions(void);
static bool SetStructure_NotWorking_2_Pack( uint8_t * ByteStream, uint8_t * UnpackedStructure);

static bool SetStructure_Working_3_Unpack( uint8_t * ByteStream, uint8_t * UnpackedStructure);
static void SetStructure_Working_3_Actions(void);
static bool SetStructure_Working_3_Pack( uint8_t * ByteStream, uint8_t * UnpackedStructure);
/******************************************************************/

 
Here goes declaration of functions assigned later to function pointers.

/**************** GLOBAL VARIABLES **************************/
static Protocol_Structures_t SetGetStructures;
static Protocol_t Protocol;
/******************************************************************/

/**************** CONTANT LOOKUP TABLES *********************/
static const Protocol_CmdHandler_t ProtocolCmdHandlers[2] =
{
   [0] = Protocol_HandleGetStructure,
   [1] = Protocol_HandleSetStructure
};

static const Protocol_SetGetStructureCmd_t CmdDataSetStructure[4] =
{
   [0] = {
      .StructureGroup = 0,
      .StructureIndex = 0,
      .ReqLength = 10,
      .RspLength = 10,
      .ReqStructurePointer = (uint8_t *) & SetGetStructures.SetStructures.SetStructure_1,
      .RspStructurePointer = (uint8_t *) & SetGetStructures.SetStructures.SetStructure_1,
      .StructurePacker = SetStructure_NotWorking_1_Pack,
      .StructureUnpacker = SetStructure_NotWorking_1_Unpack,
      .UnpackedStructureHandler = SetStructure_NotWorking_1_Actions,
   },

   [1] = {
      .StructureGroup = 0,
      .StructureIndex = 1,
      .ReqLength = 10,
      .RspLength = 10,
      .ReqStructurePointer = (uint8_t *) & SetGetStructures.SetStructures.SetStructure_2,
      .RspStructurePointer = (uint8_t *) & SetGetStructures.SetStructures.SetStructure_2,
      .StructurePacker = SetStructure_NotWorking_2_Pack,
      .StructureUnpacker = SetStructure_NotWorking_2_Unpack,
      .UnpackedStructureHandler = SetStructure_NotWorking_2_Actions,
   },

   [2] = {
      .StructureGroup = 0,
      .StructureIndex = 2,
      .ReqLength = 10,
      .RspLength = 10,
      .ReqStructurePointer = (uint8_t *) & SetGetStructures.SetStructures.SetStructure_3,
      .RspStructurePointer = (uint8_t *) & SetGetStructures.SetStructures.SetStructure_3,
      .StructurePacker = SetStructure_Working_3_Pack,
      .StructureUnpacker = SetStructure_Working_3_Unpack,
      .UnpackedStructureHandler = SetStructure_Working_3_Actions,
   },

   /** Termination entry */
   [3] = {
      .StructureGroup = 0xFF,
      .StructureIndex = 0xFF,
      .ReqLength = 0x00U,
      .RspLength = 0x00U,
      .ReqStructurePointer = NULL,
      .RspStructurePointer = NULL,
      .StructurePacker = NULL,
      .StructureUnpacker = NULL,
      .UnpackedStructureHandler = NULL,
   }
};
/******************************************************************/

 
Global variable SetGetStructures should contain unpacked data. Place where I see only zeros in my code.

/**************** INTERFACE FUNCTIONS DEFINITIONS ************/
void Protocol_Init(void)
{
   memset(&Protocol, 0x00, sizeof(Protocol));
   Protocol.CmdHandlers = ProtocolCmdHandlers;
}

void Protocol_Task(void)
{
      /** Simulate received frame for command 'Protocol_HandleSetStructure' */
      Protocol.CurrentCmdHandler = Protocol.CmdHandlers[1];

      Protocol.IOBuffers.Buffers.InputBuffer[0] = 0x0D;
      Protocol.IOBuffers.Buffers.InputBuffer[1] = 3;
      Protocol.IOBuffers.Buffers.InputBuffer[2] = 0;
      Protocol.IOBuffers.Buffers.InputBuffer[3] = 0;
      Protocol.IOBuffers.Buffers.InputBuffer[4] = 0xAA;
      Protocol.IOBuffers.Buffers.InputBuffer[5] = 0xBB;
      Protocol.IOBuffers.Buffers.InputBuffer[6] = 0xCC;
      Protocol.IOBuffers.Buffers.InputBuffer[7] = 0xDD;

      /** STEP 1: Assign pointers to input and output buffers. Const qualifier is added to
       * indicate that input/request data cannot be modified. Both of them sit in global
       * variable 'Protocol'. In the 'full' project is may sit anywhere in RAM, for
       * example at address 0x240 */
      const Protocol_CmdReq_t * ReqCmd = (const Protocol_CmdReq_t *)Protocol.IOBuffers.Buffers.InputBuffer;
      Protocol_CmdRsp_t * RspCmd = (Protocol_CmdRsp_t *)Protocol.IOBuffers.Buffers.OutputBuffer;

      /* If there is handler for certain command, execute it */
      if (Protocol.CurrentCmdHandler != NULL)
      {
         /** STEP 2: pointer to function is called when full frame is
          * received and checksum is correct. For this snippet code is
          * simulated as received command 'SetStructure'.
          *
          * JUMP TO FUCTION 'Protocol_HandleSetStructure' (line ~252)
          */
         Protocol.CurrentCmdHandler(ReqCmd, RspCmd);
      }
}
/******************************************************************/

/**************** COMMANDS HANDLERS DEFINITIONS **************/
static uint8_t Protocol_HandleGetStructure(const Protocol_CmdReq_t * ReqCmd, Protocol_CmdRsp_t * RspCmd)
{
   /** Commented out some data validation and handling return results */
   return Protocol_ExecuteGetStructureCmd(ReqCmd, RspCmd);
}

static uint8_t Protocol_HandleSetStructure(const Protocol_CmdReq_t * ReqCmd, Protocol_CmdRsp_t * RspCmd)
{
   /** Commented out some data validation and handling return results */
   return Protocol_ExecuteSetStructureCmd(ReqCmd, RspCmd);
}

static bool Protocol_ExecuteGetStructureCmd(const Protocol_CmdReq_t * ReqCmd, Protocol_CmdRsp_t * RspCmd)
{
   /** Similar like Set Structure */
   return true;
}

static bool Protocol_ExecuteSetStructureCmd(const Protocol_CmdReq_t * ReqCmd, Protocol_CmdRsp_t * RspCmd)
{
   Protocol_SetGetStructureCmd_t const * ReqStructure;
   bool Status = false;

   /** Commented out function for finding requested structure
    * index in lookup table 'CmdDataSetStructure' - assume structure index 0 */

   /** STEP 3: pointer to const data from flash 'ReqStructure'
    * is assigned to first structure in table: 'SetStructure_1'.
    * You can test other structures by changing 0 to 1 or 2.
    * For structure 2 everything works correctly.
    */
   ReqStructure = &CmdDataSetStructure[ 0 ];

   /** STEP 4:
    *
    * !!!WARNING!!!
    *
    * Here pointer 'ReqCmd->ProtocolCmdReq.ReqSetStructure.Payload' is 2 byte long to indicate
    * any point in RAM memory, on assembler view both bytes are correctly placed in ACCESS RAM.
    */

   /** STEP 5: 'StructureUnpacker' pointer is assigned to 'SetStructure_NotWorking_1_Unpack'
    * function. Const qualifier is abandoned because unpacker function is used also in other
    * places in the project where input buffer can be modified.
    *
    * JUMP TO FUNCTION 'SetStructure_NotWorking_1_Unpack' (line ~SetStructure_NotWorking_1_Unpack).
    */
   ReqStructure->StructureUnpacker((uint8_t *)ReqCmd->ProtocolCmdReq.ReqSetStructure.Payload,
                                             ReqStructure->ReqStructurePointer);
   /** Execute additional actions */
   ReqStructure->UnpackedStructureHandler();

   /** Pack structure data into byte stream */
   ReqStructure->StructurePacker(RspCmd->RspSetStructure.Payload, ReqStructure->RspStructurePointer);

   Status = true;

   return Status;
}
/******************************************************************/


Program flow:
Step 1 - just simulate receiving data. All low level program flow is simplified and input buffer is just filled with data.
Step 2 - call function assigned to pointer to function (command handler).
Step 3 - Finding structure by received index and group simplified to assigning to first structure.
Step 4 and 5 - Correct placing in RAM pointer to input buffer.

/**************** STRUCTURES HANDLERS DECLARATIONS ***********/
static bool SetStructure_NotWorking_1_Unpack(uint8_t * ByteStream, uint8_t * UnpackedStructure)
{
   bool status = false;
   volatile uint8_t PointerSize = sizeof(ByteStream);

   /** STEP 6a:
    *
    * !!!WARNING!!!
    *
    * Compiler probably lost tracked pointer 'ReqCmd' to input buffer
    * (which sits in global variable 'Protocol') and optimized it to 1-byte size
    * which causes that data is taken from wrong place in RAM!
    */

   if ((NULL != ByteStream) && (NULL != UnpackedStructure))
   {
      /** pack action */
      status = true;
   }

   return status;
}

static void SetStructure_NotWorking_1_Actions(void)
{
   /* Do something after unpack handler */

   /** STEP 7a:
    *
    * Unpacked data should be placed into variable 'SetGetStructures.SetStructures.SetStructure_1'
    * in previous step but due to incorrectly optimized pointer data has been unpacked somewhere
    * in 1st bank of RAM!
    */
}

static bool SetStructure_NotWorking_1_Pack(uint8_t * ByteStream, uint8_t * UnpackedStructure)
{
   bool status = false;
   volatile uint8_t PointerSize = sizeof(ByteStream);

   /** STEP 8a:
    *
    * !!!WARNING!!!
    *
    * Compiler here also incorrectly optimized pointer to output buffer so response data
    * will be placed somewhere in 1st bank of RAM instead of output buffer.
    * In worst case scenario when LSB address of 'ByteStream' pointer occupies the same place as
    * other local variables, we can corrupt some data!
    */
   if ((NULL != ByteStream) && (NULL != UnpackedStructure))
   {
      /** pack action */
      status = true;
   }

   return status;
}

static bool SetStructure_NotWorking_2_Unpack(uint8_t * ByteStream, uint8_t * UnpackedStructure)
{
   bool status = false;
   volatile uint8_t PointerSize = sizeof(ByteStream);

   /** STEP 6b:
    *
    * !!!WARNING!!!
    *
    * Compiler probably lost tracked pointer 'ReqCmd' to input buffer
    * (which sits in global variable 'Protocol') and optimized it to 1-byte size
    * which causes that data is taken from wrong place in RAM!
    */

   if ((NULL != ByteStream) && (NULL != UnpackedStructure))
   {
      /** Unpack action */
      status = true;
   }

   return status;
}

static void SetStructure_NotWorking_2_Actions(void)
{
   /* Do something after unpack handler */

   /** STEP 7b:
    *
    * Unpacked data should be placed into variable 'SetGetStructures.SetStructures.SetStructure_1'
    * in previous step but due to incorrectly optimized pointer data has been unpacked somewhere
    * in 1st bank of RAM!
    */
}

static bool SetStructure_NotWorking_2_Pack(uint8_t * ByteStream, uint8_t * UnpackedStructure)
{
   bool status = false;
   volatile uint8_t PointerSize = sizeof(ByteStream);

   /** STEP 8b:
    *
    * !!!WARNING!!!
    *
    * Compiler here also incorrectly optimized pointer to output buffer so response data
    * will be placed somewhere in 1st bank of RAM instead of output buffer.
    * In worst case scenario when LSB address of 'ByteStream' pointer occupies the same place as
    * other local variables, we can corrupt some data!
    */
   if ((NULL != ByteStream) && (NULL != UnpackedStructure))
   {
      /** pack action */
      status = true;
   }

   return status;
}

static bool SetStructure_Working_3_Unpack(uint8_t * ByteStream, uint8_t * UnpackedStructure)
{
   bool status = false;
   volatile uint8_t PointerSize = sizeof(ByteStream);

   /** STEP 6c:
    *
    * !!!WARNING!!!
    *
    * Somehow without any reason for this function 'ByteStream' pointer is not optimized
    * and is 2-bytes long! When modifying completely not related things, like structures
    * sizes or number of structures in lookup table 'CmdDataSetStructure', then compiler
    * may optimize is as well and makes pointer 1-byte long.
    */
   if ((NULL != ByteStream) && (NULL != UnpackedStructure))
   {
      /** Unpack action */
      status = true;
   }

   return status;
}

static void SetStructure_Working_3_Actions(void)
{
   /* Do some action after unpacking data */

   /** STEP 7c:
    *
    * For this structure data is correctly copied from input buffer to variable
    * 'SetGetStructures.SetStructures.SetStructure_3'
    */
}

static bool SetStructure_Working_3_Pack(uint8_t * ByteStream, uint8_t * UnpackedStructure)
{
   bool status = false;
   volatile uint8_t PointerSize = sizeof(ByteStream);

   /** STEP 8c:
    *
    * !!!WARNING!!!
    *
    * Here pointer is correctly tracked (you can see also map file) and data is correctly
    * read from variable 'SetGetStructures.SetStructures.SetStructure_3' and packed into
    * output buffer.
    */

   if ((NULL != ByteStream) && (NULL != UnpackedStructure))
   {
      /** Unpack action */
      status = true;
   }

   return status;
}
/******************************************************************/

 
Step 6x-8x  - cases 'a' and 'b' shows incorrect optimizing pointer, but what's weird, in case 'c' everything works as intended!
 
 
As I said, I've attached zipped project for MplabX so it's easy for everybody to reproduce it. Is it possible to set for compiler default pointer size? I saw that in previous compiler revision it was possible through "--CP=x" options but now compiler handles it...
 
For any help, suggestion I would appreciate, for any questions I will try to answer as fast as possible.
 
BR,
Adam
#1

2 Replies Related Threads

    du00000001
    Just Some Member
    • Total Posts : 3230
    • Reward points : 0
    • Joined: 2016/05/03 13:52:42
    • Location: Germany
    • Status: offline
    Re: [Compiler bug?] XC8 v2.05 - Incorrect optimization size of pointer to global variable 2019/11/19 09:02:49 (permalink)
    +1 (1)
    This seems to be a duplicate of https://www.microchip.com/forums/m1119330.aspx#1119330 ,
    but without the .zip attachment available only there.
     
    Suggest to cut this thread short.

    PEBKAC / EBKAC / POBCAK / PICNIC (eventually see en.wikipedia.org)
    #2
    adaml
    New Member
    • Total Posts : 4
    • Reward points : 0
    • Joined: 2019/11/18 00:43:07
    • Location: 0
    • Status: offline
    Re: [Compiler bug?] XC8 v2.05 - Incorrect optimization size of pointer to global variable 2019/11/19 23:45:58 (permalink)
    0
    Yes, it was duplicated and I cannot delete this thread, nor close it...
    #3
    Jump to:
    © 2019 APG vNext Commercial Version 4.5