• AVR Freaks

FSfopen speed

Page: 123 > Showing page 1 of 3
Author
microcoder
Starting Member
  • Total Posts : 75
  • Reward points : 0
  • Status: offline
2012/04/18 00:03:41 (permalink)
0

FSfopen speed

Hello,

I want to share my experience on using FSfopen() :

with a 80Mhz pic32, on a freshly FAT16 formatted 2GB sdcard it takes 120ms to execute FSfopen("myfile","w")

what about you?
#1

44 Replies Related Threads

    ceegeek
    New Member
    • Total Posts : 6
    • Reward points : 0
    • Joined: 2011/11/22 17:55:23
    • Location: 0
    • Status: offline
    Re:FSfopen speed 2012/04/18 09:56:17 (permalink)
    0
    Hi,
    I'd be interested in hearing anything you've learned about storing data on SD cards. It sounds like you're using the MDD file system that Microchip provided. We tried that and it turned out to be too slow for our purposes. We use FatFs with FAT32 and 32 Gbyte SDHC cards. FatFs works amazingly well for an open-source file system. The problems I'm having are related to short pauses caused by the internal workings of the SDHC cards themselves. Different manufacturers' cards have longer pauses than others when in SPI mode.
     
    If you ever learn anything about why these pauses occur, I'd be interested in hearing about it.
     
    Rob
    #2
    Muenchow
    Super Member
    • Total Posts : 282
    • Reward points : 0
    • Joined: 2012/02/20 02:25:22
    • Location: Bochum Germany
    • Status: offline
    Re:FSfopen speed 2012/04/18 15:42:26 (permalink)
    0
    The "pauses" are an unavoidable effect of the large write and erase block granularity involved with NAND flash storage.
     
    http://www.microchip.com/forums/treeinterface.aspx?m=647058 
    #3
    microcoder
    Starting Member
    • Total Posts : 75
    • Reward points : 0
    • Status: offline
    Re:FSfopen speed 2012/04/19 07:27:43 (permalink)
    0
    concerning the sd card pauses, take a look at this post.
    http://blog.frankvh.com/2...ass-10-sd-card-or-not/

    You said that you use FatFS with PIC32 with good results. I'm interested!
    What implementation did you take? Did you modify it?
    Have you found any bug ? How much faster than MDD is it?

    regards
    #4
    Muenchow
    Super Member
    • Total Posts : 282
    • Reward points : 0
    • Joined: 2012/02/20 02:25:22
    • Location: Bochum Germany
    • Status: offline
    Re:FSfopen speed 2012/04/19 08:21:31 (permalink)
    0
    I'm not using FatFS with PIC32, because my present application needs advanced features like LFN support, while write speed is less critical.

    Up to now, I just noticed the "not mind blowing" speed of MDD FS, particularly when creating files. It didn't yet try to narrow down where MDD misses the performance.

    Regards,
    Frank 
    #5
    bnell
    Starting Member
    • Total Posts : 74
    • Reward points : 0
    • Status: offline
    Re:FSfopen speed 2012/04/19 08:54:16 (permalink)
    0
    The MAL SD MDD implementation has been frustrating to say the least.  I've heard alot of good things bout FatFs but haven't found, nor had the time to create, a turn-key PIC32 FatFs + MAL MSD example.

    I vaguely thought I remember reading MCHP was going to switch over to FatFs in the MAL?  Maybe not...
    #6
    Muenchow
    Super Member
    • Total Posts : 282
    • Reward points : 0
    • Joined: 2012/02/20 02:25:22
    • Location: Bochum Germany
    • Status: offline
    Re:FSfopen speed 2012/04/19 23:41:40 (permalink)
    0
    As a "not so good" point, full featured FSIO with LFN enabled and SD_SPI are consumbing > 32k flash (without optimization). But it's basically working, after some corrections even LFN. 
    #7
    bnell
    Starting Member
    • Total Posts : 74
    • Reward points : 0
    • Status: offline
    Re:FSfopen speed 2012/04/20 08:46:47 (permalink)
    0
    Biggest problem I have is with FSInit() but I haven't tried the MAL that was released a week or so ago yet.
    #8
    Muenchow
    Super Member
    • Total Posts : 282
    • Reward points : 0
    • Joined: 2012/02/20 02:25:22
    • Location: Bochum Germany
    • Status: offline
    Re:FSfopen speed 2012/04/20 10:35:54 (permalink)
    0
    After some MPLAB stopwatch measurements and reviewing FSIO FSfopen() operation, I can confirm the observations of a remarkable slow operation.
     
    There are two points causing this behavior:
    - The function FILECreateHeadCluster() called from CreateFileEntry() is overwriting the newly created cluster with all zero (through function EraseCluster())
    - SD card writes are exclusively performed as ineffective WRITE_SINGLE_BLOCK. In case of a FAT16 formatted 2 GB card with 32k clusters, EraseCluster is performing not less than 64(!) single block writes.
     
    If I understand right, EraseCluster() is needed when generating new empty directory clusters. Erasing the data area of a newly created file is not really needed. In so far there's a simple workaround to increase FSfopen speed. But preferably, the sector zero-fill should be performed as WRITE_MULTI_BLOCK.
     
    Regards,
    Frank
    #9
    Muenchow
    Super Member
    • Total Posts : 282
    • Reward points : 0
    • Joined: 2012/02/20 02:25:22
    • Location: Bochum Germany
    • Status: offline
    Re:FSfopen speed 2012/04/21 14:18:30 (permalink)
    0
    After disabling the reported unmotivated EraseCluster() action in CreateFileEntry(), I made some SD-card operation speed measurements to identify the performance bottleneck. Operation condition have been: 80 MHz CPU clock, instruction cache and prefetch enabled, 20 MHz SPI clock, 2 GB Sandisk microSD card.  
     
    An append action, writing one sector takes 38 ms. About 26 ms of this time, the processor is waiting for the SD card to finish write operations, in addition several ms for the deployment of read data. The read and write operations are prescribed by the FAT format, not arbitrary MDD actions. All in all, there's little potential for timing improvements. About 3.5 ms can be saved by using DMA instead of programmed I/O for sector data transfer.
     
    With a 4 GB micro SDHC card, the same small append action takes only 23 ms, apparently due to a considerably faster internal operation.
     
    A final comment on WRITE_SINGLE_BLOCK versus WRITE_MULTIPLE_BLOCK problem. It's rather easy to write the all zero data for EraseCluster() in a multi block write. This could save about 25 % of total write time for FSfopen file creation with 2 GB card, less than assumed.
     
    Regards,
    Frank 
     
     
     
     
     
     
    #10
    asmallri
    Super Member
    • Total Posts : 1864
    • Reward points : 0
    • Joined: 2004/05/26 09:00:05
    • Location: Perth, Australia
    • Status: offline
    Re:FSfopen speed 2012/04/21 14:43:44 (permalink)
    0
    I conducted speed test on a large range of cards a few years back. At that time I found that the rated speed (or class) of the card gave no indication of performance when the card was operated in SPI mode. In a lot of cases the expensive high performance cards performed slower than standard cards. The data structures on the card also made a significant impact. Windows does NOT perform a low level format. Performing speed tests on different cards that have been formatted by windows prior to the test does not give a true indication as to the performance of the card. A better approach is to format the card in a device, such as a camera, which does perform a low level format. Once the cards have been formatted with a camera, and therefore have a consistent layout, windows can be used to reformat the cards between tests.

    Regards, Andrew

    http://www.brushelectronics.com/index.php?page=software
    Home of Ethernet, SD Card, and Encrypted Serial and USB Bootloaders for PICs!!
    #11
    Muenchow
    Super Member
    • Total Posts : 282
    • Reward points : 0
    • Joined: 2012/02/20 02:25:22
    • Location: Bochum Germany
    • Status: offline
    Re:FSfopen speed 2012/04/22 05:29:33 (permalink)
    0
    As far as I understand, the impact of low level format is mainly related to the alignment of card clusters to media physical blocks. It shows when writing larger amounts of data in a single write operation. A typical logging application, as the one one described in post #9 shouldn't suffer from cluster misalignment.

    At present, I don't need to optimize throughput for larger data blocks. I'm mainly concerned about possibly exceeding reasonable time limits for atomar (open/write/close) operations because it would require a rather complex asynchronous file handling. The original MDD CreateFileEntry() implementation is a case where reasonable time limits are exceeded. With a 2 GB FAT16 formatted card and 7 ms sector write time, about 400 ms are needed for EraseCluster().

    Performance reduction in internal operation speed in SPI mode is an interesting point. Thanks for mentioning it.
    #12
    rk1genius
    New Member
    • Total Posts : 9
    • Reward points : 0
    • Joined: 2012/01/24 03:27:52
    • Location: 0
    • Status: offline
    Re:FSfopen speed 2012/04/23 00:02:46 (permalink)
    0
    how did you calculate the execution time ............... i assume you might have used the i/o toggle method for the purpose .
    #13
    microcoder
    Starting Member
    • Total Posts : 75
    • Reward points : 0
    • Status: offline
    Re:FSfopen speed 2012/04/23 00:05:25 (permalink)
    0
    Thanks Muenchow for your investigation. Is the difference between WRITE_SINGLE_BLOCK and WRITE_MULTIPLE_BLOCK the same as writing an eeprom byte by byte or with a 32-byte page?

    asmallri, I don't understand what is a low level format.
    I agree with Muenchow that Windows doesn't always use the optimized cluster size when formatting, that's why one should use this tool :
    https://www.sdcard.org/downloads/formatter_3/

    Right now I have received two new 2gb sdcard from SanDisk and Transcend where the FSfinit() fails so I have to check this.
    #14
    Muenchow
    Super Member
    • Total Posts : 282
    • Reward points : 0
    • Joined: 2012/02/20 02:25:22
    • Location: Bochum Germany
    • Status: offline
    Re:FSfopen speed 2012/04/23 02:43:48 (permalink)
    0
    how did you calculate the execution time ............... i assume you might have used the i/o toggle method for the purpose

    I made the first measurements with stopwatch in MPLAB hardware debugger, to narrow down the most time consuming actions in FSfopen().
     
    Later I monitored SPI signals by a DSO with logic probe and deep memory. By zooming into the waveform, you can easily identify, when a sector is read/written, and when the processor is waiting for the card to get ready.
     
    Is the difference between WRITE_SINGLE_BLOCK and WRITE_MULTIPLE_BLOCK the same as writing an eeprom byte by byte or with a 32-byte page?

     
    It's not that dramatic. It depends on the operation details of the SD card controller's flash translation layer. The basic problem is caused by the size of physical NAND flash pages (smallest write size) and blocks (erase size). Regularly, the card has to start a new page for every WRITE_SINGLE_BLOCK. Recent 2 to 32 GB flash memories have e.g. 4 kB page size  and 256 to 1024 kB block size.
     
    The advantage of writing multiple sectors at once mainly shows with larger files. Unfortunately, it usually implies large buffers.
     
    Right now I have received two new 2gb sdcard from SanDisk and Transcend where the FSfinit() fails so I have to check this.

    I would assume a signal integrity issue at first sight. Newer cards tend to use faster chip technology that is able to "see" ringing clock edges that are ignored by older ones. Also bugs with SPI mode support have been reported from time to time.
     
    I take your post as a pretty strong hint to check the newest cards with our product...
    #15
    microcoder
    Starting Member
    • Total Posts : 75
    • Reward points : 0
    • Status: offline
    Re:FSfopen speed 2012/04/24 08:02:48 (permalink)
    0
    Ok I have upgraded to the last MDD library version (from 1.2.4 to 1.3.4) and now the sdcards are working.

    It is just sad that the library needs more RAM than before.
    post edited by microcoder - 2012/04/24 08:07:20
    #16
    Muenchow
    Super Member
    • Total Posts : 282
    • Reward points : 0
    • Joined: 2012/02/20 02:25:22
    • Location: Bochum Germany
    • Status: offline
    Re:FSfopen speed 2012/04/25 02:14:12 (permalink)
    0
    Most recent MDD FS version is V1.3.6, as shipped with MAL v2012-02-15. There aren't much changes to V1.3.4 however.  If you don't need working LFN support, V1.3.4/V1.3.6 are basically O.K. The reported time consuming EraseCluster() with CreateFileEntry() applies to the MDD FS recent versions.

    I recently posted an attempt to fix the LFN problem in MDD FS. See http://www.microchip.com/forums/m603948.aspx

    #17
    microcoder
    Starting Member
    • Total Posts : 75
    • Reward points : 0
    • Status: offline
    Re:FSfopen speed 2012/05/16 04:08:46 (permalink)
    0
    I have made a benchmark in order to test fopen speed.
     
    On a freshly-formatted sandisk 2go sdcard, I do a loop with:
     
    f = fopen("filexxxx.txt","w");
    fwrite(f, "******************************",...);
    fclose(f);
     
    Here are my results.
    It's clear that at a certain point, the fopen takes too much time and the watchdog timer will trigger!

     

    post edited by microcoder - 2012/05/16 04:18:18
    #18
    Muenchow
    Super Member
    • Total Posts : 282
    • Reward points : 0
    • Joined: 2012/02/20 02:25:22
    • Location: Bochum Germany
    • Status: offline
    Re:FSfopen speed 2012/05/16 06:54:07 (permalink)
    0
    The results are plausible and indicating a basic problem of FAT file system handling. It's simply the effort to search the FAT and directory for a free entry. I fear, there isn't much you can do about it without caching larger parts of the administrative data structures.
     
    There are specific options depending on the way, the file system is utilized in your application. You can e.g. operate a background task, that is scanning for the next free FAT entry sequentially. This will be particularly helpful, if you are working with a limited numer of large files. If many files are also involved as in your benchmark, it may be interesting to speed up directory operations.
     
    Regards,
    Frank
    post edited by Muenchow - 2012/05/16 06:55:30
    #19
    microcoder
    Starting Member
    • Total Posts : 75
    • Reward points : 0
    • Status: offline
    Re:FSfopen speed 2012/05/21 02:03:41 (permalink)
    0

    After some MPLAB stopwatch measurements and reviewing FSIO FSfopen() operation, I can confirm the observations of a remarkable slow operation.
    There are two points causing this behavior:
    - The function FILECreateHeadCluster() called from CreateFileEntry() is overwriting the newly created cluster with all zero (through function EraseCluster())


    If I understand right, EraseCluster() is needed when generating new empty directory clusters. Erasing the data area of a newly created file is not really needed. In so far there's a simple workaround to increase FSfopen speed. But preferably, the sector zero-fill should be performed as WRITE_MULTI_BLOCK.

    If you don't erase the cluster before writing to it, how is the end of file defined? Only by the size in the directory entry?
     

    - SD card writes are exclusively performed as ineffective WRITE_SINGLE_BLOCK. In case of a FAT16 formatted 2 GB card with 32k clusters, EraseCluster is performing not less than 64(!) single block writes.

     

    A final comment on WRITE_SINGLE_BLOCK versus WRITE_MULTIPLE_BLOCK problem. It's rather easy to write the all zero data for EraseCluster() in a multi block write. This could save about 25 % of total write time for FSfopen file creation with 2 GB card, less than assumed.

    Is it possible to implement WRITE_MULTI_BLOCK without increasing the buffer size in RAM (512-bytes)? Do you have some code available?
     
    regards
     
    #20
    Page: 123 > Showing page 1 of 3
    Jump to:
    © 2019 APG vNext Commercial Version 4.5