• AVR Freaks

Hot!PIC32MZ EF Harmony3 SQI issue with odd size buffer

Author
Luca Pascarella
Starting Member
  • Total Posts : 73
  • Reward points : 0
  • Joined: 2007/05/28 00:53:17
  • Location: The Netherlands
  • Status: offline
2019/09/10 08:42:18 (permalink)
0

PIC32MZ EF Harmony3 SQI issue with odd size buffer

I am porting my working project based on a PIC32MZ EF from Harmony2 to Harmony3 and I am struggling with the SST26 driver.
 
The project implements a web server that includes the MPFS driver to access an SST26VF064B through a SQI peripheral.
I fixed the cache handling in mpfs.c (since SQI DMA writes in not cachable space) but a strange problem is present in the submodule sys_fs_media_manager.c.
 
After a while a discovered that SYS_FS_MEDIA_MANAGER_Read(diskNum, buffer, source, nBytes) works properly only when nBytes is an even number. For odd numbers, instead, the buffer is filled with a progressive shifted data.
 
// Three consecutive calls to SYS_FS_MEDIA_MANAGER_Read create these pattern
 
// nBytes is 15, and for every call, the 0x4D byte is rotated of a position
0x4D 50 46 53 02 01 91 00 0A 90 AC CE F0 94 2C 00
0x2C 4D 50 46 53 02 01 91 00 0A 90 AC CE F0 94 00
0x94 2C 4D 50 46 53 02 01 91 00 0A 90 AC CE F0 00

// If nBytes is 14 it returns the correct results for every call
0x4D 50 46 53 02 01 91 00 0A 90 AC CE F0 94 00 00
0x4D 50 46 53 02 01 91 00 0A 90 AC CE F0 94 00 00
0x4D 50 46 53 02 01 91 00 0A 90 AC CE F0 94 00 00

 
At the moment I don't have a clue on what to check. Perhaps it may be an issue with the drv_sst26.c.
Does someone have the same issue?
 
Luca
post edited by Luca Pascarella - 2019/09/10 08:45:12
#1

7 Replies Related Threads

    optimus_jack
    Starting Member
    • Total Posts : 54
    • Reward points : 0
    • Joined: 2017/02/16 03:02:47
    • Location: 0
    • Status: offline
    Re: PIC32MZ EF Harmony3 SQI issue with odd size buffer 2019/09/10 19:55:43 (permalink)
    0
    Hi Luca,
    In Harmony 3 if you want to perform File operation or interact with memory media's like NVM, SST26(SQI) using the file system layer you need to use Memory Driver.
    Refer to https://github.com/Microchip-MPLAB-Harmony/core/tree/master/apps/fs/sqi_flash_fat/firmware/pic32mz_ef_sk.X for understanding the configurations
     
    Can you share the project? to better understand the problem
    #2
    Luca Pascarella
    Starting Member
    • Total Posts : 73
    • Reward points : 0
    • Joined: 2007/05/28 00:53:17
    • Location: The Netherlands
    • Status: offline
    Re: PIC32MZ EF Harmony3 SQI issue with odd size buffer 2019/09/10 22:55:36 (permalink)
    0
    In my project, the SYS_FS_MEDIA_MANAGER_Read is used by MPFS module in a web server application. I do not need to access MPFS File Systems from userspace, at least the SST26, userspace accesses FAT through mass storage but it is not implemented now.
    I arrived at that odd size buffer by debugging why web pages are not always loaded properly into the browser.

    I discovered that sometimes it works and others do not and this condition depends on the length of the filename.
     
    When the request is an odd length SYS_FS_MEDIA_MANAGER_Read returns incorrect data. Then, I just added a simple check to recall SYS_FS_MEDIA_MANAGER_Read several times discovering this strange behaviour for odd size.
     
    I build this project by combining Harmony 3 examples on apps/fs/sqi_flash_fat, web_net_server_qspi_mpfs, and web_net_server_nvm_mpfs. I am positive about the hardware because loading my old project developed with Harmony 2 it works properly.
     
     
    To be more specific, I found the issue in mpfs.c when temp is odd.

    // SQI_CACHE_WORKAROUND defines a temporary workaround to force non cachable space. hashBuffer and fileName must be moved back on the stack to allows reentrant calls.
     
    #if defined (SQI_CACHE_WORKAROUND)
    static uint16_t __ALIGNED(16) __COHERENT hashBuffer[16] = {0};
    static uint8_t __ALIGNED(16) __COHERENT fileName[FAT_FS_MAX_LFN];
    #endif
    static int MPFSFindFile
    (
        uint8_t diskNum,
        uint8_t *file,
        MPFS_FILE_RECORD *fileRecord
    )
    {
        uint32_t address = 0;
        uint32_t temp = 0;
        uint8_t *ptr = NULL;
        int32_t index = 0;
        uint16_t hash = 0;
    #if !defined (SQI_CACHE_WORKAROUND)
        uint16_t hashBuffer[16] = {0};
        uint8_t fileName[FAT_FS_MAX_LFN];
    #endif
        
        /* Calculate the hash value for the file name. */
        ptr = file;
        while (*ptr != '\0')
        {
            hash += *ptr++;
            hash <<= 1;
        }

        /* Read in hashes, and check remainder on a match. Store 8 in cache for
         * performance. */
        for (index = 0; index < gSysMpfsObj.numFiles; index++)
        {
            if ((index & 0x07) == 0u)
            {
                /* Address of the hash location. */
                address = 8 + (index << 1);
                temp = gSysMpfsObj.numFiles - index;
                if (temp > 8)
                {
                    temp = 8;
                }
                if (MPFSGetArray (diskNum, address, temp << 1, (uint8_t *)hashBuffer) == false)
                {
                    return -1;
                }
            }

            /* If the hash matches, compare the full filename. */
            if (hashBuffer[index & 0x07] == hash)
            {
                address = 8 + (gSysMpfsObj.numFiles * 2) + (index * 22);
                MPFSGetArray (diskNum, address, 22, (uint8_t *)fileRecord);
                temp = strlen ((const char *)file);

                if (MPFSGetArray (diskNum, fileRecord->fileNameOffset, temp, (uint8_t *)fileName) == false)
                {
                    return -1;
                }

                if (strncmp ((const char *)fileName, (const char *)file, temp) == 0)
                {
                    /* Found the matching file. */
                    return index;
                }
            }
        }

        return -1;
    }

    post edited by Luca Pascarella - 2019/09/10 23:12:10

    Attached Image(s)

    #3
    optimus_jack
    Starting Member
    • Total Posts : 54
    • Reward points : 0
    • Joined: 2017/02/16 03:02:47
    • Location: 0
    • Status: offline
    Re: PIC32MZ EF Harmony3 SQI issue with odd size buffer 2019/09/10 23:58:04 (permalink)
    0
    Hi Luca,
     
    Will look into the project and see if i can help you out.
    #4
    Luca Pascarella
    Starting Member
    • Total Posts : 73
    • Reward points : 0
    • Joined: 2007/05/28 00:53:17
    • Location: The Netherlands
    • Status: offline
    Re: PIC32MZ EF Harmony3 SQI issue with odd size buffer 2019/09/29 07:27:02 (permalink)
    0
    Some news on this topic? Should I share more details?
     
    I checked with a signal analyzer and the communication seems ok.
    In other words, the bytes sent by the SST match the requests, therefore, I suppose that the request is built correctly but some issues happen with an internal buffer.
     
    I have the same problem even usgin high level file reading. The first byte of the second read conicides with last byte of the first call.
    uint8_t __ALIGNED(16) __COHERENT buffer[128];
    char *fileName = "/mnt/sst/index.htm";
    [...]
    main_appData.fsHandle = SYS_FS_FileOpen(fileName, SYS_FS_FILE_OPEN_READ);
    if (main_appData.fsHandle != SYS_FS_HANDLE_INVALID) {
        SYS_CONSOLE_PRINT("Reading %s\r\n", fileName);
        if (SYS_FS_FileRead(main_appData.fsHandle, buffer, 7)) {
            SYS_CONSOLE_PRINT("0x%02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X\r\n", buffer[i + 0], buffer[i + 1], buffer[i + 2], buffer[i + 3], buffer[i + 4], buffer[i + 5], buffer[i + 6], buffer[i + 7], buffer[i + 8], buffer[i + 9], buffer[i + 10], buffer[i + 11], buffer[i + 12], buffer[i + 13], buffer[i + 14], buffer[i + 15]);
            if (SYS_FS_FileRead(main_appData.fsHandle, buffer, 7)) {
                SYS_CONSOLE_PRINT("0x%02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X %02X\r\n", buffer[i + 0], buffer[i + 1], buffer[i + 2], buffer[i + 3], buffer[i + 4], buffer[i + 5], buffer[i + 6], buffer[i + 7], buffer[i + 8], buffer[i + 9], buffer[i + 10], buffer[i + 11], buffer[i + 12], buffer[i + 13], buffer[i + 14], buffer[i + 15]);
            }
        }
        SYS_FS_FileClose(main_appData.fsHandle);
    } else {
        SYS_CONSOLE_PRINT("Cannot open %s \r\n", fileName);
    }

     
    Luca
    post edited by Luca Pascarella - 2019/09/29 08:43:55
    #5
    Luca Pascarella
    Starting Member
    • Total Posts : 73
    • Reward points : 0
    • Joined: 2007/05/28 00:53:17
    • Location: The Netherlands
    • Status: offline
    Re: PIC32MZ EF Harmony3 SQI issue with odd size buffer 2019/09/30 11:32:57 (permalink)
    0
    I create a simple project that partially addresses the cache issue with MPFS and SQI drivers but highlights the problem of odd size buffer (see mpfs.c for a temporary not perfect workaround).
    It contains two examples. The first uses the MPFS driver to access files on SQI flash:
     
            case MAIN_APP_FILE_OPEN:
     
                SYS_CONSOLE_PRINT("Open %s\r\n", fileName);
                main_appData.fsHandle = SYS_FS_FileOpen(fileName, SYS_FS_FILE_OPEN_READ);
                if (main_appData.fsHandle != SYS_FS_HANDLE_INVALID) {
                    main_appData.state = MAIN_APP_FILE_READ_1;
                } else {
                    SYS_CONSOLE_PRINT("Cannot open %s\r\n", fileName);
                    main_appData.state = MAIN_APP_MEDIA_BLOCK_READ;
                }
                break;

            case MAIN_APP_FILE_READ_1:
                read = 11;
                if (SYS_FS_FileRead(main_appData.fsHandle, buffer, read)) {
                    printBytes(buffer, read);
                    buffer[read] = '\0';
                    SYS_CONSOLE_PRINT("Read: %s\r\n", buffer);
                    main_appData.state = MAIN_APP_FILE_READ_2;
                } else {
                    SYS_CONSOLE_PRINT("Cannot open %s\r\n", fileName);
                    main_appData.state = MAIN_APP_STATE_SERVICE_TASKS;
                }
                break;

            case MAIN_APP_FILE_READ_2:
                read = 11;
                if (SYS_FS_FileRead(main_appData.fsHandle, buffer, read)) {
                    printBytes(buffer, read);
                    buffer[read] = '\0';
                    SYS_CONSOLE_PRINT("Read: %s\r\n", buffer);
                    main_appData.state = MAIN_APP_FILE_CLOSE;
                } else {
                    SYS_CONSOLE_PRINT("Cannot open %s\r\n", fileName);
                    main_appData.state = MAIN_APP_STATE_SERVICE_TASKS;
                }
                break;

    The above code returns a message with duplicate h during the second read:
    TCP/IP Stack: Initialization Started
    My first debug message in Harmony3
    TCP/IP Stack: Initialization Ended - success
        Interface PIC32INT on host MCHPBOARD_E - NBNS disabled
    /mnt/sst mounted
    PIC32INT IP Address: 0.0.0.0
    Open /mnt/sst/index.htm
    0x3C 21 44 4F 43 54 59 50 45 20 68
    Read: <!DOCTYPE h
    0x68 74 6D 6C 3E 0A 3C 68 74 6D 6C
    Read: html>
               <html
    Close /mnt/sst/index.htm
    0x6C 20 50
    0x50 46 53

    This issue happens only for odd size requests.
     
    The second example bypasses the MPFS driver and directly uses the SYS_FS_MEDIA_MANAGER_Read call to request 3 bytes from the SQI flash.
            case MAIN_APP_MEDIA_BLOCK_READ:
            {
                uint16_t j;
                for (j = 0; j < 2; j++) {
                    // Test MPFS
                    uint16_t diskNum = 0;
                    uint32_t address = 0x00000001;
                    uint32_t nBytes = 3;
                    uint8_t *source;

                    SYS_FS_MEDIA_BLOCK_COMMAND_HANDLE commandHandle = SYS_FS_MEDIA_BLOCK_COMMAND_HANDLE_INVALID;
                    SYS_FS_MEDIA_COMMAND_STATUS commandStatus = SYS_FS_MEDIA_COMMAND_UNKNOWN;

                    source = ((uint8_t *) address);
                    commandHandle = SYS_FS_MEDIA_MANAGER_Read(diskNum, buffer, source, nBytes);

                    if (commandHandle != SYS_FS_MEDIA_BLOCK_COMMAND_HANDLE_INVALID) {
                        commandStatus = SYS_FS_MEDIA_MANAGER_CommandStatusGet(diskNum, commandHandle);
                        while ((commandStatus == SYS_FS_MEDIA_COMMAND_IN_PROGRESS) || (commandStatus == SYS_FS_MEDIA_COMMAND_QUEUED)) {
                            SYS_FS_MEDIA_MANAGER_TransferTask(diskNum);
                            commandStatus = SYS_FS_MEDIA_MANAGER_CommandStatusGet(diskNum, commandHandle);
                        }
                        printBytes(buffer, nBytes);
                    }
                }
                main_appData.state = MAIN_APP_STATE_SERVICE_TASKS;
                break;
            }

    The output is mixed in the above result where last byte of the first call coincides with the first byte of the second call.
     
    I'll open an issue for the cache problem on the Harmony core repository, but I don't know how to catch and fix the odd size problem. Some hints I should try?
    post edited by Luca Pascarella - 2019/09/30 11:37:51
    #6
    Luca Pascarella
    Starting Member
    • Total Posts : 73
    • Reward points : 0
    • Joined: 2007/05/28 00:53:17
    • Location: The Netherlands
    • Status: offline
    Re: PIC32MZ EF Harmony3 SQI issue with odd size buffer 2019/09/30 11:35:51 (permalink)
    #7
    boatbodger
    Starting Member
    • Total Posts : 46
    • Reward points : 0
    • Joined: 2011/03/27 15:39:07
    • Location: 0
    • Status: offline
    Re: PIC32MZ EF Harmony3 SQI issue with odd size buffer 2019/11/05 11:27:21 (permalink)
    5 (1)
    I too am struggling with this.  I have also found that read requests have to be multiples of two bytes, AND that if you don't read to a static buffer (as opposed to memory allocated on the fly in the heap) it does not work.  I am therefore going through mpfs.c changing all the buffers to which sys_fs_manager_read has to write from dynamic to static memory allocation e.g. in mpfs.c, function FindFile, I have changed:
        uint16_t hashBuffer[16] = {0};
    to 
        static CACHE_ALIGN uint16_t hashBuffer[16] = {0};
    Having done that, the cache chunks are correctly read to memory.  When using the original version, hashBuffer contained all zeroes even after the read.
     
    EDIT: I now understand this to be because it does not work if you pass the DMA controller (through sys_fs_media_manager, drv_memory, drv_SST26) an address which is a cached address (in the range 0x8000_xxxx), as the DMA can't write to the L1 cache, it goes belly up.  By using the CACHE_ALIGN macro, this forces the variables to be accessed directly rather than through cache.  A tad slower but so what?
     
    That, of course is nothing to do with the odd size buffer - which I am about to address.  That latter point seems to be me to represent an underlying problem with the sys_fs_media_manager functions, as surely they should handle this issue?  The file management stuff is riddled with buffers all over the place, but they don't seem to be being used intelligently.
     
    I note the comment at the top of SYS_FS_MEDIA_MANAGER_Read which says:
    "This function is intended to work with NVM media only, which can have byte level addressing".
    So I guess the fact that Harmony lets you couple an SST16 + SQI driver underneath this module is the root of the problem.
     
    I think to make this work is going to need a lot of effort to either rip out the (doubtless very efficient) DMA stuff, and go to "bit banging", or to do some major re-engineering of sys_fs_media_manager (or maybe even several layers below that).
     
    What you and I are trying to do should be "out of the box" - but it seems that definitely isn't the case.
    post edited by boatbodger - 2019/11/06 04:57:18
    #8
    Jump to:
    © 2019 APG vNext Commercial Version 4.5