• AVR Freaks

Hot!Enable CRC for SD card communication with Harmony 2

Author
moser
Super Member
  • Total Posts : 637
  • Reward points : 0
  • Joined: 2015/06/16 02:53:47
  • Location: Germany
  • Status: offline
2021/01/19 09:45:57 (permalink)
5 (1)

Enable CRC for SD card communication with Harmony 2

I would like to know, if anyone has some experience with implementing CRC support to the SD card driver (drv_sdcard) of Harmony 2, because I would like to do that.
 
Has someone ever done this with Harmony? Or maybe just with an own implementation?
Could you share some experience or give some hints about potential issues and problems?
 
So far we estimate the effort to:
- get CRC7 and CRC16 calculation running correctly (probably only as software implementation).
- calculate CRC7s correctly for any "command send" dynamically (at least for any command with variable argument)
- turn CRC "ON" (CMD59) before SEND_OP_COND (ACMD41), instead of turning it "OFF". 
- calculate CRC16 correctly for WRITE_SINGLE_BLOCK (/ WRITE_MULTI_BLOCK)
- check received CRC16 in data responses (for READ_SINGLE_BLOCK, READ_MULTI_BLOCK and receiving CSD)
- check received CRC7 in received CSD
We will try to follow the Physical Layer Simplified Specification (8.00) from the SD Association for this.
 
 
Environment:
- Harmony 2.06 (recently upgraded from 2.03b)
- FatFs R0.11a from ChaN as part of Harmony 2.06 (recently upgraded from FatFs R0.09b)
- xc32 2.30 (recently upgraded from xc 1.42)
- PIC32MZ2048EFM144
- microSD SDHC, 4 GB, Intenso (will very likely be changed in near future)
- Custom PCB
 
 
Background in detail:
 
We have an own board with a PIC32MZ2048EFM144 and a microSD card slot. We were using Harmony 2.03b and recently upgraded to 2.06, which drives the microSD in SPI mode. I have an application, which uses the microSD to write two log files (one text, one binary), always by appending at the end. Both files are kept open all the time, but I use a SYS_FS_FileSync() call immediately after every SYS_FS_FileWrite(). Furthermore, a http web server is able to deliver both log files by using the same file handles as the writing process. And the web server also delivers some static files from a folder on the microSD. Devices are usually running 24/7, sometimes with a lot of power cycles, sometimes without a single one for years. We always used the same microSD card of the same brand (Intenso) and this worked quite reliable for a long time (several years). 
 
But lately it seems the cards have changed under the hood. Although the cards look the same, and have the same packaging, we believe that the production was changed. A lot of the later cards seem to have various kinds of errors which include:
- silently having a whole sector within the log file filled with zeros (0x00), without reporting any error at the SYS_FS layer.
- after some errors, no response at all anymore until next power cycle or detach+attach.
- after some errors, card can not be mounted anymore at all (even after a power cycle or detach+attach).
- root directory completely empty.
- root directory sector contains log file data instead (looks quite funny in the explorer).
- filling a large part of consecutive sectors in the SD with "random" garbage. However, often a repeating pattern of 1024 bytes (2 sectors) can be seen.
- in one case, partition table messed up. It displays a 5,7 GB volume, not formatted, on 4 GB microSD.
- in one case, but probably related to a power loss of the board, "random" bit flips, across 4 sectors of file data and 28 empty sectors. Within the empty sectors (0x00) it was roughly about 20 to 30 bits set to 1, within 64 bytes.
 
I created a small stress test with my application. We found out, there are cards which run (at least) for days without any issue. And there are other cards which cause some errors usually within a few seconds to about ten minutes. And results are reproducible.
 
We believe that some issues, like the overwritten root directory, are follow-up errors of some disc errors, due to bad error handling in the FAT library. This is why we switched from Harmony 2.03b (using FatFs R0.09b from ChaN) to Harmony 2.06 (using FatFs R0.11a from ChaN). But this does not remove the root issue of the errors.
 
So, we decided on the one hand to switch SD cards to a brand which probably has better SPI support (e.g. Sandisk or Panasonic). We have tested some Panasonic so far, and we were not able to reproduce any error with those.
 
And on the other hand we want to activate the CRCs. Without CRC basically anything could happen, and we would not even have any chance to get aware of this. And on the long run in our environment, such things might happen. And that is the reason for my question.
#1

4 Replies Related Threads

    Jim Nickerson
    User 452
    • Total Posts : 6933
    • Reward points : 0
    • Joined: 2003/11/07 12:35:10
    • Location: San Diego, CA
    • Status: offline
    Re: Enable CRC for SD card communication with Harmony 2 2021/01/19 10:19:32 (permalink)
    4 (1)
    I am using SD cards in one of my boards with a Pic32Mx795 and MLA.
    I have had quite a few problems, some cards are more trouble than others.
    I am running them in single channel SPI mode.
    I have debugged the drivers extensively and my conclusion was there are subtle timing differences in the cards when running single channel SPI, the only recovery with a troublesome card was a power cycle.
    I will not use SD cards in any future Pic32Mx boards.
    I find it interesting that these same troublesome cards work fine in my Windows PC.
    #2
    moser
    Super Member
    • Total Posts : 637
    • Reward points : 0
    • Joined: 2015/06/16 02:53:47
    • Location: Germany
    • Status: offline
    Re: Enable CRC for SD card communication with Harmony 2 2021/01/20 02:38:09 (permalink)
    0
    I made the same experience. I tested several of my troublesome cards at a Windows PC with h2testw ( https://www.heise.de/download/product/h2testw-50539 ) and they didn't show any errors. No empty sectors. No bit flips. No strange behavior. 
     
    However, at a PC they very likely use the SD Bus Operation Mode and not the SPI Bus Operation Mode. I guess this might explain some different behavior.
     
    My PCB has the capability to force a power cycle for the microSD, so this usually helps. However, if the last operation in progress already did something bad to the file system, then the FatFS implementation doesn't stop you from running into a catastrophic failure.
    #3
    Jim Nickerson
    User 452
    • Total Posts : 6933
    • Reward points : 0
    • Joined: 2003/11/07 12:35:10
    • Location: San Diego, CA
    • Status: offline
    Re: Enable CRC for SD card communication with Harmony 2 2021/01/20 08:34:19 (permalink)
    4 (2)
    When I started with the SD I was surprised by the power draw spike on start up.
    #4
    moser
    Super Member
    • Total Posts : 637
    • Reward points : 0
    • Joined: 2015/06/16 02:53:47
    • Location: Germany
    • Status: offline
    Re: Enable CRC for SD card communication with Harmony 2 2021/03/03 05:25:52 (permalink)
    4 (1)
    We postponed implementing the CRCs for the microSD cards, as we have currently other more important issues.
    We fixed the problems by selecting the right microSD cards. But I hope I can come back to it at a later point in time.
     
    I experimented a bit with my stress test which uses the SPI mode.
     
    Reducing SPI speed from 20MHz to 2MHz did not have an impact on the microSD cards which show errors. The errors only appear slower.
     
    Also we really verified that at some point a random sector gets erased, and the SD card driver does not see any error, not even at the lowest levels. As far as we can tell, we verified that the implementation follows the SD specification and does all necessary checks. Only the optional CRCs are avoided. Not one of checks fails, but we get empty sectors. 
     
    If this happens within the data sectors, then only your data is damaged, but if it happens elsewhere, for example in the directory entries of the FAT, then this causes really bad things. The FatFS driver has only one sector in the cache. However, after opening files, it remembers where file entries and other things are located. And when FatFS needs to change this stuff, it just reads in the right sector without checking what was read, then overwrites only the bytes which need to get changed and writes back the whole sector. We did not verify, but we believe, this combination of silently deleted sectors and blindly overwriting stuff, could theoretically explain almost all the different phenomena, which we have seen.
     
    Also I tested several microSD card brands. Results were varying. We found a few other brands which also had errors within minutes. We even had even one brand which almost never made it over the SD initialization phase, and even if it did then it immediately failed after. But we also found some brands which worked perfectly and were from manufactures which we believe are probably also more trustworthy. Since microSD cards are not replaced in our products, this solves our problem completely.
     
     
    JANickerson
    When I started with the SD I was surprised by the power draw spike on start up.
    I know I have a power draw spike, but it is not coming from the microSD. Power is really not an issue on my board. My external power supply has 10 A output current at 48 V. And I don't see any spike or dent in my 3.3 V level with the oscilloscope on startup or any different behavior which is depending on whether a microSD card is present or not.
    #5
    Jump to:
    © 2021 APG vNext Commercial Version 4.5