2011/03/04 10:23:13
Aiden.Morrison
Hi All,

I'm working on a project where I need to stream data to SD at a high rate, and found that most of the openly available implementations I could find out there were just not up to the job - for example the microchip MDD library had the following performance -

Generic 512 M - FAT32/4096
8:52:41 to >9:03:00 - test cancelled - less than 13 kB/sec


Micro SD 4GB SDHC
9:21:18 to 9:27:00  - 342 seconds - 23kB/sec
9:27:00 to 9:32:40  - 340 seconds - 24kB/sec
9:32:40 to 9:38:21  - 341 seconds - 24kB/sec


Lexar 2GB fat32 formatted to 4kb allocation units
10:03:21 to 10:03:57 - 36 seconds to copy 8MB - 227kB/sec
10:03:57 to 10:04:33 - 36 seconds to copy 8MB - 227kB/sec
10:04:33 to 10:05:10 - 37 seconds to copy 8MB - 221kB/sec


Highly dependent on card type, and downright slow on many types.  I heard good things about FatFS
http://elm-chan.org/fsw/ff/00index_e.html
but couldn't find anyone who had implemented it for PIC32 that was willing to post code (fernando says he has too much proprietary stuff bolted on to share it, and dogbertius seems to be non communicative).

In the current implementation the benchmarks are now as follows:
Generic 512 M - FAT32/4096
seconds to copy 8MB
2:43:18 to 2:43:59 = 41 seconds to copy 8MB = 199.8 kB/sec
2:43:59 to 2:44:38 = 39 seconds to copy 8MB = 210.0 kB/sec
2:44:38 to 2:45:18 = 40 seconds to copy 8MB = 204.8 kB/sec

Micro SD 4GB SDHC
seconds to copy 8MB
2:36:30 to 2:36:54 = 24 seconds to copy 8MB = 341.33 kB/sec
2:36:54 to 2:37:16 = 22 seconds to copy 8MB = 372.36 kB/sec
2:37:16 to 2:37:37 = 21 seconds to copy 8MB = 390.09 kB/sec


Lexar 2GB fat32 formatted to 4kb allocation units
20 seconds =409.6 kB/sec
20 seconds =409.6 kB/sec
20 seconds =409.6 kB/sec



Performance can be improved further as this is simply a minimalistic implementation derived from the basic FatFS code, and using the pic24 example posted on the FatFS website.  I believe dogbertius was working on a version that would use DMA to do the bulk writes.

I likely won't be updating this code since I now have to roll it into a project that I shouldn't share, but my next tasks include trying to optimize the write and erase block sizes for each disk inserted by reading the card specific data and adapting to it (this should increase write speeds), as well as by utilizing DMA and SPI interrupts to arbitrate data streaming - I don't need to interleave reads with writes so this will destroy the generality of the file system.  


Some notes are included on the board used and what needs to be done to duplicate this simple setup.
Code should be attached here PIC32FatFS.zip
2011/03/06 06:27:22
Davis
UP!wink
2011/03/06 14:08:19
Aiden.Morrison
Two more notes:

1)  I'm still learning to use the file access controls like CREATE_ALWAYS etc. - the main.c in the rar archive doesn't properly do what I intended (create a new log file at each startup) - the updated main.c is attached, so it actually creates a new file at each startup for a logging application.

2) I forgot to mention that I disabled the timeout timer aswell, so while I haven't seen a read or a write timeout yet, there's no protection against this in the current code - originally ChaN indicates that 2 timer counters should be used that are decremented at 1kHz in a timer interrupt, but I just left them.

Hope these haven't tripped anyone up.
--An updated main.c is that addresses no.1 is attached.

Attachment(s)

main.c (4.96 KB)
2011/03/24 14:44:24
Aiden.Morrison
Bump.

Just wanted to mention that in this project there are two things you can change to get higher speed:

1) look up the definition of the fast clock setting for SPI - it currently sets SPI2BRG=2;  this can be set to SPIBRG=0, to raise the SPI clock from 6.7 MHz to 20 MHz.

2) changing the optimization level from O0 to O1 provides another small bump.

With those two changes the lexar 2G card now works at around 730 kB/sec write.
2011/04/04 11:03:11
mhark
Is this different from

http://www.microchip.com/...4&appnote=en539236


-mhark
2011/04/04 11:36:54
Aiden.Morrison
Yes, insofar as I couldn't get the microchip project to even compile.

It's from 2009 so my guess is it was built with tools that are now out of date...


This one is built with a 2011 version of FAT-FS as well.
2011/10/05 06:41:53
erupter
I just wanted to thank Aiden.Morrison for sharing his code.
I've just tested it with my Sandisk 4GB SD.

 Microchip : don't event think of it
FatFS std : 62s - 394 KB/s
FatFS BRG0: 35s - 686 KB/s
FatFS OP1 : 27s - 872 KB/s


Now onto integrating this with MSD and possibly FreeRTOS.
Any suggestions here?
2011/11/22 15:59:20
rileonar
  Thanks, Aiden.Morrison! I've ported your code to to PIC32MX795F512L.
This chip is used into chipKIT Max32 board (an Arduino clone).
These are the changes made to your code:
  • device changed to PIC32MX795F512L (used into chipKIT Max32)
  • Header files included via "Include Search Path" option in project
  • System config performance optimization
  • LEDs removed (I don't have them in my "shield")
  • FatFs version updated to R0.09 September 6, 2011 (multi-partition)
  • get_fattime routine moved to main
  • disk_timerproc call added (handled by ISR)
  • parametrization of SPI port number
  • core_timer added for rtc and benchmarking purposes

Here is some benchmarking on old 64 MB MMC and 1 GB Kingston SD:

Type FileSystem BRG Optimization Seconds     KB/s
SD     FAT16         2         0                 37,43     641,16
SD     FAT16         1         0                 32,2       745,32
SD     FAT16         1         1                 24,14     994,04
SD     FAT16         1         s                 24,06     997,55
SD     FAT16         1         3                 22,52     1065,91
SD     FAT16         0         0                 24,48     980,43
SD     FAT16         0         1                 17,53     1369,08 (unstable)
SD     FAT32         2         0                 61,83     388,14
SD     FAT32         1         0                 56,69     423,36
SD     FAT32         0         0                 49,07     489,06
SD     FAT32         0         1                 41,71     575,37

MMC FAT16         2         0                  51,06     470,01
MMC FAT16         1         0                  45,63     525,96
MMC FAT16         1         1                  36,90     650,41
MMC FAT16         1         s                  37,70     636,55
MMC FAT16         1         3                  35,85     669,47
MMC FAT16         0         0                     n/a         n/a

Spreadsheet with above data is included into the project.
Now I begin thinking about your DMA solution...
 
Thanks again,
Riccardo
2012/03/21 13:30:17
rcdeang
Thanks guys for all your work and sharing your code with everyone. I'm still reading through and studying everything.

I have a couple questions about USB and FatFs. Right now I have code that writes to a USB drive using the Microchip MDD file system library. I also need to be able to read/write to an SD card and I saw the Microchip's library can only access one drive at a time, which is why I'm looking into FatFs.

Can FatFs read/write to a USB drive in addition to an SD card? If so, would it be best to use Microchip's library for USB and FatFs for SD or just use FatFs for both?
2012/04/12 17:04:40
dogbertius
  Sorry about that. I've tried to at least keep up-to-speed on private messages, but have been swamped with work this last year. I hardly get any time for hobby electronics. It's sad really :( Glad to see the board members are still contributing actively. Personally I think MC should be hiring some of the board members so they can have demo apps that are both working and current.
© 2019 APG vNext Commercial Version 4.5

Use My Existing Forum Account