Approaching the end of my rope with DMA+SPI

Hi there,

I've tried to use the microchip provided SPI DMA example in the following situation:

I am streaming data to disk using fatFS.  Performance is great when it is all the chip is doing, but since it's just throwing one byte transfers any time there is an interrupt this degrades write throughput.  To minimize this I'm switching the block write operations to use DMA.

Here's the pertinent code:

volatile int DmaTxIntFlag; // flag used in interrupts, signal that DMA transfer ended
volatile int DmaRxIntFlag; // flag used in interrupts, signal that DMA transfer ended

/*-----------------------------------------------------------------------*/
/* Start a DMA Sector Transfer between host to card     */
/*-----------------------------------------------------------------------*/
BYTE sectorsendDMA(const BYTE *buff)
{
unsigned char resp;
int evFlags;
int done=0; 
//clear SPI TX empty int
INTClearFlag(INT_SOURCE_SPI(_SPI3A_TX_IRQ)); 
DmaChnOpen(DMA_CHANNEL1, DMA_CHN_PRI3, DMA_OPEN_DEFAULT);    // Tx 
DmaChnSetEventControl(DMA_CHANNEL1, DMA_EV_START_IRQ_EN|DMA_EV_START_IRQ(_SPI3A_TX_IRQ));
DmaChnSetTxfer(DMA_CHANNEL1,(unsigned char*)buff, (void*)&SPI3ABUF, 512, 1, 1); 
DmaChnSetEvEnableFlags(DMA_CHANNEL1, DMA_EV_BLOCK_DONE);

INTSetVectorPriority(INT_VECTOR_DMA(DMA_CHANNEL1), INT_PRIORITY_LEVEL_5); // set INT controller priority
INTSetVectorSubPriority(INT_VECTOR_DMA(DMA_CHANNEL1), INT_SUB_PRIORITY_LEVEL_3); // set INT controller sub-priority

INTEnable(INT_SOURCE_DMA(DMA_CHANNEL1), INT_ENABLED); // enable the chn interrupt in the INT controller
DmaTxIntFlag=0; // clear the interrupt flag we're  waiting on
//clear SPI TX empty int
DmaChnEnable(DMA_CHANNEL1);
DmaChnStartTxfer(DMA_CHANNEL1, DMA_WAIT_NOT, 0); 

while(!DmaTxIntFlag) {}
//DmaChnDisable(DMA_CHANNEL1);
//disable spi interrupt
INTEnable(INT_SOURCE_DMA(DMA_CHANNEL1), INT_DISABLED);
volatile unsigned char discard=SPI3ABUF;



}




// handler for the DMA channel 1 interrupt
void __ISR(_DMA1_VECTOR, ipl5) DmaHandler1(void)
{
int evFlags; // event flags when getting the interrupt
SPI3ASTATbits.SPIROV=0;
INTClearFlag(INT_SOURCE_DMA(DMA_CHANNEL1)); // acknowledge the INT controller, we're servicing int
evFlags=DmaChnGetEvFlags(DMA_CHANNEL1); // get the event flags
    if(evFlags&DMA_EV_BLOCK_DONE)
    { // just a sanity check. we enabled just the DMA_EV_BLOCK_DONE transfer done interrupt
     DmaTxIntFlag=1;
  DmaChnClrEvFlags(DMA_CHANNEL1, DMA_EV_BLOCK_DONE);
    }
}



If I don't clear the SPIROV bit in the DMA ISR the SPI module simply locks up!  This is interesting since the microchip example code doesn't seem to concern itself with handling this.

Before and after the code there are other SPI transactions happening, but nothing interrupt driven.  Just simple put/take byte operations that either send a byte and discard the received, or send 0xFF to receive a byte.

it seems to me that the DMA module isn't actually sending 512 bytes.

Hey Frost,

I tried that for the heck of it, but sure enough it made things far worse ;)

cell size can be thought of as 'how many of these bytes do I send before I wait for the trigger event to poke me again....?'.

Since the trigger is 'the spi3a transmitter is empty' and the buffer size of the spi module is 1 byte (not in enhanced buffer mode) I have to stop every 1 byte to wait for the spi transmitter to catch up.

Edit - I've figured out how to get this to work:

volatile int DmaTxIntFlag; // flag used in interrupts, signal that DMA transfer ended
volatile int DmaRxIntFlag; // flag used in interrupts, signal that DMA transfer ended
volatile int DMAINTCOUNT=0;
/*-----------------------------------------------------------------------*/
/* Start a DMA Sector Transfer between host to card     */
/*-----------------------------------------------------------------------*/
BYTE sectorsendDMA(const BYTE *buff)
{
unsigned char resp;
int evFlags;
int done=0; 
//clear SPI TX empty int
INTClearFlag(INT_SOURCE_SPI(_SPI3A_TX_IRQ)); 
DmaChnOpen(DMA_CHANNEL1, DMA_CHN_PRI3, DMA_OPEN_DEFAULT);    // Tx 
DmaChnSetEventControl(DMA_CHANNEL1, DMA_EV_START_IRQ_EN|DMA_EV_START_IRQ(_SPI3A_TX_IRQ));
DmaChnSetTxfer(DMA_CHANNEL1,(unsigned char*)buff, (void*)&SPI3ABUF, 512, 1, 1); 
DmaChnSetEvEnableFlags(DMA_CHANNEL1, DMA_EV_BLOCK_DONE);

INTSetVectorPriority(INT_VECTOR_DMA(DMA_CHANNEL1), INT_PRIORITY_LEVEL_5); // set INT controller priority
INTSetVectorSubPriority(INT_VECTOR_DMA(DMA_CHANNEL1), INT_SUB_PRIORITY_LEVEL_3); // set INT controller sub-priority

INTEnable(INT_SOURCE_DMA(DMA_CHANNEL1), INT_ENABLED); // enable the chn interrupt in the INT controller
DmaTxIntFlag=0; // clear the interrupt flag we're  waiting on
//clear SPI TX empty int
//DmaChnEnable(DMA_CHANNEL1);
DmaChnStartTxfer(DMA_CHANNEL1, DMA_WAIT_NOT, 0); 

while(!DmaTxIntFlag) {}
//DmaChnDisable(DMA_CHANNEL1);
//disable spi interrupt
INTEnable(INT_SOURCE_DMA(DMA_CHANNEL1), INT_DISABLED);
while(SPI3ASTATbits.SPIBUSY==1){}
volatile unsigned char discard=SPI3ABUF;



}




// handler for the DMA channel 1 interrupt
void __ISR(_DMA1_VECTOR, ipl5) DmaHandler1(void)
{
int evFlags; // event flags when getting the interrupt
SPI3ASTATbits.SPIROV=0;
volatile unsigned char discard=SPI3ABUF;
INTClearFlag(INT_SOURCE_DMA(DMA_CHANNEL1)); // acknowledge the INT controller, we're servicing int
evFlags=DmaChnGetEvFlags(DMA_CHANNEL1); // get the event flags
    if(evFlags&DMA_EV_BLOCK_DONE)
    { // just a sanity check. we enabled just the DMA_EV_BLOCK_DONE transfer done interrupt
     DmaTxIntFlag=1;
  DmaChnClrEvFlags(DMA_CHANNEL1, DMA_EV_BLOCK_DONE);
    }
DMAINTCOUNT++;
}


The DMAINTCOUNT variable was just me making sure that the interrupt was only called once per 512 byte block transfer.  The reason it wasn't working was that I had forgotten to account for the fact that the interrupt is called when the DMA finishes sending its last byte to the SPI module - we have to wait for it to finish sending before reading the last byte out of the module.  Interestingly, the overflow of the spi receive buffer doesn't appear to bother it when it's working in DMA mode like this - reception is unecessary, though I do clear the overflow bit once at the end of the 511th transaction/start of the 512th.

I noticed the 256 byte limit on the 3/4 series early on - similar to the UART and SPI peripherals having less RX/TX buffer on the 3/4 series than on the 6/7.
 
When I noticed this I found it amusing how much effort has gone into making helper functions that are compatible over the whole family, but still having these small but critical differences in peripherals that could cause some issues - though I suppose the problems would only been seen if one was trying to downsize the chip in use.
 
I've attached an image of the file system benchmarks I've obtained so far accross the 8 SD/SDHC cards I have available. Each of these cards has been benchmarked using a FAT32 format with default sector sizes (4kB-32kB) and with a block transfer size in my code of 4kB.  Note that higher performance can be obtained by writing larger blocks, but this gets expensive in terms of RAM usage pretty fast.
 
1) Microchip Disk Drive library running on a pic32
2) Fat-FS running on the same pic32 with an SPI clock of 6.67 MHz
3) Fat-FS running on the same pic32 with an SPI clock of 20.0 MHz
4) Fat-FS running on a '695 with an SPI of 22.5 MHz and DMA transfers of the 512 byte blocks 
 
Vertical axis is in kB/sec of write throughput.

Yes

It should be hiding somewhere under your C32 folder, i think in the plib examples area.

Hi jnadelman,
Can you post your test project for the FATFS modification with DMA.


Regards,
Gerrit

Hi Yuantuh, I should clarify what I mean by block size.

In the code I posted it's using a 4096 byte write size, but the individual write commands (multi-block write) are issued on 512 byte blocks that are convenient to use with DMA.

I think the biggest limitation on speed now is the SPI limit of ~20 MHz.  Since SD cards support up to 50 MHz SPI now, I hope the next iteration of the pic32 will support a ~40 MHz SPI bus within spec.

To get much faster we would need the 4 bit SD interface peripheral which seems too specialized for anything but multimedia processors.

Sorry, I made mistakes,  8000 DMA 4KB block transaction should be 8000 DMA 512 byte block transaction.

 You may double check your stopwatch implementation and confirm time statistics for writing 4MB data, same 4KB data loop 1000 times. 

Oh I use a very simple stopwatch implementation - I write down the time when I start the program running, and then write down the time it turns on the 'done' LED :)


Not the most refined or precise method, but pretty close.