• AVR Freaks

Helpful ReplyHot![SOLVED]Bug in the USART Harmony/FreeRTOS module

Author
BillP
Super Member
  • Total Posts : 436
  • Reward points : 0
  • Joined: 2014/09/28 07:53:35
  • Location: CA
  • Status: offline
2019/02/26 09:04:20 (permalink)
0

[SOLVED]Bug in the USART Harmony/FreeRTOS module

Platform: Curiosity PIC32MZ EF (100 pin) with 2 Clik board slots
WiFi4 Clik Board
MPLABX IDE v4.20
Harmony 2.06
XC32 v2.10
FreeRTOS v10.0.1 (the default version selected in MHC)

The WiFi4 uses a USART (Tx/Rx) interface to the PIC32MZ host. MHC configures the USART and a RESET signal to control the WiFi4 board.  The USART driver uses an interrupt with byte-mode at a 115200 baud rate.

I used MHC to configure 2 COM ports with the USB CDC library -- COM1 is a console port to enter AT commands for the WiFi4 Clik board and COM2 is a log port to display logging messages created during run-time.

I developed a superloop app that talks to the WiFi4 board.  All of the interfaces and features work as expected.  The logic running under the superloop is a little complicated because it relies on multiple state machines and a lot of flags to implement a command/response element.  [This is a standard interface characteristic in many serial interfaces, such as Modbus.]  

An RTOS simplifies the implementation of the command/response element by including delays and waits for the back-and-forth communications, so I converted the superloop logic to use the FreeRTOS module included in the Harmony distro. It crashed.  After many hours of checking configurations, debugging (sometimes) showed the PC was in the port_asm.S file (a FreeRTOS source file). No documentation on what that file does but it appears to deal with some interrupt vector locations.  

I was able to isolate the "bug" to be in the Rx and Error callbacks defined in the USART driver (byte-mode).  If I commented out the callbacks, the code did not crash (but it did not execute properly).  These same callbacks worked correctly in the superloop version.

Conclusion:  there appears to be a bug in the USART Harmony/FreeRTOS module.
post edited by BillP - 2019/03/04 08:50:27
#1
Twi
New Member
  • Total Posts : 2
  • Reward points : 0
  • Joined: 2019/02/22 01:39:12
  • Location: 0
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/02/26 11:08:57 (permalink)
0
Hey,
Maybe not? What's in those callbacks ?
#2
Paul PortSol
Super Member
  • Total Posts : 636
  • Reward points : 0
  • Joined: 2015/07/03 11:52:03
  • Location: Newfoundland, Canada
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/02/27 07:13:34 (permalink)
0
Anything in the HarmonyV206 Errata or the latest Errata for your PIC32 chip?
#3
BillP
Super Member
  • Total Posts : 436
  • Reward points : 0
  • Joined: 2014/09/28 07:53:35
  • Location: CA
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/02/27 08:24:03 (permalink)
0
I cannot find anything in the Harmony release notes.  I really doubt this is a Harmony problem since everything works in the superloop version.  It must be FreeRTOS doing something that interferes with the Harmony framework.  If no one in this forum has seen something like this before, then I will contact FreeRTOS and I will post any results.
#4
BillP
Super Member
  • Total Posts : 436
  • Reward points : 0
  • Joined: 2014/09/28 07:53:35
  • Location: CA
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/02/27 08:30:45 (permalink)
5 (1)
Here is the code in the Event Handlers.  This is the same in both the superloop and FreeRTOS. The RX event handler reads one byte and stores it in a circular buffer and returns. The check for an "OK" response is also done here to avoid more complex logic elsewhere.
 
void RXEventHandler(const SYS_MODULE_INDEX index)
{
    int pbm1;
    // a byte has been received. Store it in the rbuf
    rbuf[putbyte] = DRV_USART_ReadByte(serialData.usartHandle);
    // check for an "OK" response
    if(putbyte == 0) pbm1 = SERIAL_CIRCBUFSIZE;
    else pbm1 = putbyte - 1;
    if(rbuf[putbyte] == 'K' && rbuf[pbm1] == 'O') isOK = true;
    // update the counter and account for wrap-around
    putbyte++;
    if(putbyte >= SERIAL_CIRCBUFSIZE) putbyte = 0;
}

void ErrorEventHandler(const SYS_MODULE_INDEX index)
{
    serialData.isError = true;
}

#5
Twi
New Member
  • Total Posts : 2
  • Reward points : 0
  • Joined: 2019/02/22 01:39:12
  • Location: 0
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/02/28 01:36:18 (permalink)
0
Please provide FreeRTOS forum post link.
#6
BillP
Super Member
  • Total Posts : 436
  • Reward points : 0
  • Joined: 2014/09/28 07:53:35
  • Location: CA
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/02/28 12:11:45 (permalink)
#7
friesen
Super Member
  • Total Posts : 2155
  • Reward points : 0
  • Joined: 2008/05/08 05:23:35
  • Location: Indiana, USA
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/02/28 16:52:12 (permalink)
0
I kind of doubt it is a FreeRTOS problem. Have you isolated at all where the crash happens?
 
FreeRTOS is only a RTOS, it doesn't do as much as one may think.  You do have to be sure that the ISR's are set up to be compatible with their way of doing things, but beyond that I haven't run into much, other than that bootloaders need to be set up on 8192 instead of 4096 page boundaries for some unknown reason.

Erik Friesen
#8
BillP
Super Member
  • Total Posts : 436
  • Reward points : 0
  • Joined: 2014/09/28 07:53:35
  • Location: CA
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/03/04 08:52:23 (permalink)
5 (1)
Found it!  It is a Harmony bug, not FreeRTOS.  Richard Barry (FreeRTOS) doubted it was a FreeRTOS problem because the interrupt processing in FreeRTOS is a simple wrapper around the Harmony-generated code.  I reviewed his suggestions and the FreeRTOS code and I agree with him, so then I had to look at the USART driver.  I was confused because the application code ran fine in the superloop version, but locked up (not crashed) in the FreeRTOS version.  

The problem occurs in the USART interrupt/byte-mode driver.  The offending code is in the file framework/driver/usart/src/dynamic/drv_uasrt_byte_model.c.  The program hangs whenever a read or write is executed because there is some undocumented (surprise!) feature that tries to lock a mutex before executing the read or write.  I removed those mutex calls and the application works correctly.  

I am not suggesting this is the "universal" fix for the Harmony driver because I have no idea what MCHP had in mind when they put this logic in the driver.  Hopefully, they will review this and fix it in some future Harmony release.  For now, I will have to use my hack version (see below).

void DRV_USART_WriteByte( const DRV_HANDLE handle, const uint8_t byte)
{
    DRV_USART_CLIENT_OBJ * client;
    DRV_USART_OBJ * hDriver;

    /* Validate the client handle */
    client = _DRV_USART_DriverHandleValidate(handle);

    if(client == NULL)
    {
        SYS_DEBUG(0, "Invalid Driver Handle");
        return;
    }

    hDriver = client->hDriver;
    // Send one byte
    PLIB_USART_TransmitterByteSend(hDriver->moduleId, byte);
    _DRV_USART_InterruptSourceEnable(hDriver->txInterruptSource);

/***** -- v2.06 original code *****
    /* This function needs to be thread safe *

    if(OSAL_MUTEX_Lock(&(hDriver->mutexDriverInstance), OSAL_WAIT_FOREVER) == OSAL_RESULT_TRUE)
    {
        /* Send one byte
        PLIB_USART_TransmitterByteSend(hDriver->moduleId, byte);
        _DRV_USART_InterruptSourceEnable(hDriver->txInterruptSource);
        OSAL_MUTEX_Unlock(&(hDriver->mutexDriverInstance));
    }
    else
    {
        SYS_DEBUG(0, "Hardware Instance Mutex Time out in DRV_USART_WriteByte() function");
    }
****/
}

uint8_t DRV_USART_ReadByte( const DRV_HANDLE handle )
{
    DRV_USART_CLIENT_OBJ * client;
    DRV_USART_OBJ * hDriver;
    uint8_t readValue;

    /* Validate the client handle */
    client = _DRV_USART_DriverHandleValidate(handle);

    if(client == NULL)
    {
        SYS_DEBUG(0, "Invalid Driver Handle");
        return 0;
    }

    hDriver = client->hDriver;
    return PLIB_USART_ReceiverByteReceive(hDriver->moduleId);

/***** -- v2.06 original code *****

    /* This function needs to be thread safe
    if(OSAL_MUTEX_Lock(&(hDriver->mutexDriverInstance), OSAL_WAIT_FOREVER) == OSAL_RESULT_TRUE)
    {
        /* Read one byte
        readValue = PLIB_USART_ReceiverByteReceive(hDriver->moduleId);
        OSAL_MUTEX_Unlock(&(hDriver->mutexDriverInstance));
    }
    else
    {
        SYS_DEBUG(0, "Hardware Instance Mutex Time out in DRV_USART_ReadByte() function");
        return 0;
    }
 return readValue;
***/
}

#9
friesen
Super Member
  • Total Posts : 2155
  • Reward points : 0
  • Joined: 2008/05/08 05:23:35
  • Location: Indiana, USA
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/03/04 09:48:16 (permalink)
0
You really should figure out what is holding the mutex, or you'll probably end up running into some subtle bug thats rather hard to track down.

Erik Friesen
#10
friesen
Super Member
  • Total Posts : 2155
  • Reward points : 0
  • Joined: 2008/05/08 05:23:35
  • Location: Indiana, USA
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/03/04 09:51:15 (permalink)
0
However, if they are blocking on PLIB_USART_ReceiverByteReceive then..  Uh..  Do the troublesome ticket thing or move on.
 

Erik Friesen
#11
BillP
Super Member
  • Total Posts : 436
  • Reward points : 0
  • Joined: 2014/09/28 07:53:35
  • Location: CA
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/03/04 09:54:32 (permalink)
5 (1)
I agree, but since I identified the bug, I think it is up to MCHP to fix it so other subtle bugs are not created.
#12
NKurzman
A Guy on the Net
  • Total Posts : 19039
  • Reward points : 0
  • Joined: 2008/01/16 19:33:48
  • Location: 0
  • Status: online
Re: Bug in the USART Harmony/FreeRTOS module(?) 2019/03/04 10:17:20 (permalink) ☄ Helpfulby lamdaelectronics 2019/03/05 04:32:49
5 (3)
BillP
I agree, but since I identified the bug, I think it is up to MCHP to fix it so other subtle bugs are not created.

That would be interesting to see.
The Harmony Team historically has not been interested in bug fixes. They are now on an ARM adventure. So you should insure their code is not your malfunction. The problems are yours.
#13
T Yorky
Super (Thick) Member
  • Total Posts : 574
  • Reward points : 0
  • Joined: 2012/08/28 02:07:35
  • Location: UK
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2020/07/01 07:21:06 (permalink)
0
(old thread, but relevant)
@BillP, Thanks for posting the above. I have just experienced the same issue. Used Harmony V2 quite a bit. Needed to add a serial protocol to an application. Done it before. One of those 'ten minute jobs'. Three and a half days later came to the same conclusion as above. This is my first RTOS use of the serial port. Using an adapted MChip demo. Found the Rx interrupt causes the problem. It causes an exception trap (Code 5. Store Address Error). The app locks up in _general_exception_handler() with the code and an address.
I found an interrupt from a single Rx character caused the harmony handler to be repeatedly called 67 times before crashing (Stack overflow).
The Uart is used in Byte mode, Dynamic, interrupt. Even opening the port in Exclusive mode does not avoid the problem. Your mod along with Exclusive mode results in only single port access by one task and problem gone.
Agree with NKurzman, do not hold your breath waiting for a bug fix. :)
T Yorky.
post edited by T Yorky - 2020/07/01 07:23:54
#14
turtlepants
New Member
  • Total Posts : 9
  • Reward points : 0
  • Joined: 2019/06/13 13:19:13
  • Location: 0
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2020/08/19 07:17:38 (permalink)
0
This is a little over my head but I think I just experienced this bug too with Harmany v3. I am using UART to output Debug info using the Console and I also have RTOS set up.
 
It seemed to work fine for debug messages when doing simple button press tasks but I started seeing issues when trying to implement USB. The debug messages coming from the USBFS driver would get cut off and hang everything else. So my USB device would never get configured and I thought I set USB up wrong. 
 
With @BillP's guidance I did the following to fix my issue with UART mutex...
 
sys_console_uart.c

static bool Console_UART_ResourceLock(CONSOLE_UART_DATA* pConsoleUartData)
{
    // if(OSAL_MUTEX_Lock(&(pConsoleUartData->mutexTransferObjects), OSAL_WAIT_FOREVER) == OSAL_RESULT_FALSE)
    // {
    // return false;
    // }
    // else
    // {
    // return true;
    // }
    return true;
}

static void Console_UART_ResourceUnlock(CONSOLE_UART_DATA* pConsoleUartData)
{
// /* Release mutex */
// OSAL_MUTEX_Unlock(&(pConsoleUartData->mutexTransferObjects));
}

void Console_UART_Initialize(uint32_t index, const void* initData)
{
    CONSOLE_UART_DATA* pConsoleUartData = CONSOLE_UART_GET_INSTANCE(index);
    const SYS_CONSOLE_UART_INIT_DATA* consoleUsartInitData = (const SYS_CONSOLE_UART_INIT_DATA*)initData;

    if (pConsoleUartData == NULL)
    {
        return;
    }

    // if(OSAL_MUTEX_Create(&(pConsoleUartData->mutexTransferObjects)) != OSAL_RESULT_TRUE)
    // {
    // return;
    // }

    /* Assign the USART PLIB instance APIs to use */
    pConsoleUartData->uartPLIB = consoleUsartInitData->uartPLIB;

    pConsoleUartData->status = SYS_CONSOLE_STATUS_CONFIGURED;
}

 
Not sure if this is the best way but it does work for me. This has been an issue for me for a while now and I could not figure it out until now. Thanks!
#15
stbnrivas
New Member
  • Total Posts : 18
  • Reward points : 0
  • Joined: 2020/01/03 11:43:47
  • Location: 0
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2020/08/24 04:07:53 (permalink)
0
same here I have trouble with UART and SYS_DEBUG UART using Harmony 3
my hardware is:
- pic32 mz2048EFM144 into Curiosity PIC32MZ EF 2.0 Development Board
- mikrobus quectel bg96 https://www.mikroe.com/lte-iot-2-click
 
I am using AT commands to control the chip... in some point of execution UART driver would get cut off and hang everything else... Don't matter which AT command is ...
 
I put a logic analizer in the middle, se attach image:
 
The chronogram show how non pic32 chip return a right response "QIACT: 1,1,1 "IP" ..." but PIC32 UART read bad response...
the point is that the same code is working with all previous at commands (even more 1000bytes)... note that I need read single character one time until ends because the other chip use terminator multiple time in every response...
I can not ignore this response because if there a error connection to Internet won't be not available...
 
 
            case T_GSM_ONLY_QIACT_ASK:
                // AT+QIDEACT=<contextID>
                DRV_USART_WriteBuffer(uart_handler,"AT+QIACT?\r",10);
                at_response_double_init(&response_double);
                _data.state = T_GSM_ONLY_QIACT_ASK_READ;
               break;
            case T_GSM_ONLY_QIACT_ASK_READ:
                DRV_USART_ReadBuffer(uart_handler,&char_readed,1);
                at_response_double_read_and_enqueue(char_readed,&response_double);
                if(at_response_double_end(&response_double)){
                    _data.state = T_GSM_ONLY_QIACT_ASK_RESP;
                }
                break;
            case T_GSM_ONLY_QIACT_ASK_RESP:
                SYS_DEBUG_PRINT(SYS_ERROR_DEBUG,"resp %s (QIACT?)\r\n",response_double.data);
                if(strcmp("ERROR",response_double.data)==0){
                    _data.state = T_GSM_ONLY_QIDEACT;
                } else {
                    _data.state = T_GSM_ONLY_NOTIFY_TO_TASK_X;
                }
                break;
 
 
 
Also I tryed:
- replace AT+QIACT? for another command without success
- call at command, put 3second delay (to ignore) and call again same comand to check if the blame is RTOS task context switching without success
- DRV_USART_Close and DRV_USART_Open without success
- there are multiples RTOS task but all have lesser priority and also are Suspended to prevent switching context
- also try commenting the @turtlepants without success
 
 
any help is welcome, thanks in advanced
 
 
#16
stbnrivas
New Member
  • Total Posts : 18
  • Reward points : 0
  • Joined: 2020/01/03 11:43:47
  • Location: 0
  • Status: offline
Re: Bug in the USART Harmony/FreeRTOS module(?) 2020/08/25 06:55:00 (permalink)
4 (1)
for me replace the SYS_DEBUG_PRINT with snprintf + DRV_USART_WRITE do the trick
 
            SYS_DEBUG_PRINT(SYS_ERROR_DEBUG,"resp %s (QIACT?)\r\n",response_double.data);
with
           snprintf(debug_buffer,DBG_BUF_SIZE,"resp %s (QIACT?)\r\n",response_double.data);
           DRV_USART_WriteBuffer(uart_debug,debug_buffer,strlen(debug_buffer));
 
thanks
#17
Jump to:
© 2020 APG vNext Commercial Version 4.5