Hot!TCPIP issue 2.04

Author
friesen
Super Member
  • Total Posts : 1689
  • Reward points : 0
  • Joined: 2008/05/08 05:23:35
  • Location: Indiana, USA
  • Status: offline
2017/10/02 06:27:17 (permalink)
0

TCPIP issue 2.04

I have been puzzling through a http server issue where I get files that won't load, or load with delays from chrome.  I have tracked this down through the tcp source and a number of wireshark captures to where I feel like something is borking within the tcpip stack.  Device is PIC32MZ2064DAH176
 
I am attaching two wireshark captures, one from the PC, the other from the PIC. 192.168.0.18 is the PIC. Source for wireshark capture is included as well, with the exception of umm_malloc for DDR.
 
It may be possible that my wireshark capture locations are incorrect, please advise if so.  At any rate, I definitely am getting garbage in the TX side, stuff that wireshark shows at strange frames, as well as short or long frames.
 
The symptom is slow loading, or delayed, and the captures show definite issues, what I would perceive as a PIC TX problem.
 
In drv_ethmac_lib.c
/****************************************************************************
 * Function: DRV_ETHMAC_LibTxSendPacket
 *****************************************************************************/
DRV_ETHMAC_RESULT DRV_ETHMAC_LibTxSendPacket(DRV_ETHMAC_INSTANCE_DCPT* pMacD, const DRV_ETHMAC_PKT_DCPT* pPkt)
{
    DRV_ETHMAC_RESULT res;
    DRV_ETHMAC_DCPT_LIST* pNewList;
    uint8_t newList [(DRV_ETHMAC_DCPT_LIST_ALIGN -1) + sizeof(DRV_ETHMAC_DCPT_LIST)];

    pNewList = DRV_ETHMAC_LIB_ListInit(_EthAlignAdjust(newList));

    res = DRV_ETHMAC_RES_OK;
    while (pPkt != 0 && pPkt->pBuff != 0 && pPkt->nBytes != 0 && res == DRV_ETHMAC_RES_OK) {
 res = _EthTxSchedBuffer(pMacD, pPkt->pBuff, pPkt->nBytes, pNewList);
#ifdef STACK_USE_WIRESHARK
 if (res == DRV_ETHMAC_RES_OK) {
     WireSharkCallBack((void*) pPkt->pBuff, pPkt->nBytes);
 }
#endif
 pPkt = pPkt->next; // next buffer in packet
    }


    if(res==DRV_ETHMAC_RES_OK)
    { // all's well
 _EthTxSchedList(pMacD, pNewList);
    }
    else if(!DRV_ETHMAC_LIB_ListIsEmpty(pNewList))
    { // we failed, put back the removed nodes
        DRV_ETHMAC_LIB_ListAppendTail(pMacD->mData._EnetTxFreePtr, pNewList);
    }

    return res;

} //DRV_ETHMAC_LibTxSendPacket

 
In DRV_ETHMAC_PIC32MACPacketRx
 
TCPIP_MAC_PACKET* DRV_ETHMAC_PIC32MACPacketRx (DRV_HANDLE hMac, TCPIP_MAC_RES* pRes, const TCPIP_MAC_PACKET_RX_STAT** ppPktStat)
{
 DRV_ETHMAC_RESULT ethRes;
    DRV_ETHMAC_PKT_DCPT *pRootDcpt, *pLastDcpt;
    TCPIP_MAC_RES mRes;
    uint16_t buffOffset;
#if (TCPIP_EMAC_RX_FRAGMENTS > 1)
    TCPIP_MAC_PACKET *pCurrPkt, *pPrevPkt;
    TCPIP_MAC_DATA_SEGMENT *pCurrDSeg;
    DRV_ETHMAC_PKT_DCPT *pCurrDcpt;
#endif // (TCPIP_EMAC_RX_FRAGMENTS > 1)
    TCPIP_MAC_PACKET* pRxPkt = 0;
 const DRV_ETHMAC_PKT_STAT_RX* pRxPktStat = 0;
    int buffsPerRxPkt = 0;
    DRV_ETHMAC_INSTANCE_DCPT *pMacD = (DRV_ETHMAC_INSTANCE_DCPT*)hMac;


    // prepare the receive packet descriptor
    // for receiving a new packet
    pLastDcpt = pRootDcpt = pMacD->mData.rxPktDcpt;
#if (TCPIP_EMAC_RX_FRAGMENTS > 1)
    int ix;
    for(ix = 0; ix < sizeof(pMacD->mData.rxPktDcpt)/sizeof(*pMacD->mData.rxPktDcpt); ix++)
    {
        pLastDcpt->next = pLastDcpt + 1;
        pLastDcpt++;
    }
    // adjust last descriptor
    pLastDcpt--;
    pLastDcpt->next = 0;
#endif // (TCPIP_EMAC_RX_FRAGMENTS > 1)

    _DRV_ETHMAC_RxLock(pMacD);
    ethRes = DRV_ETHMAC_LibRxGetPacket (pMacD, pRootDcpt, &buffsPerRxPkt, &pRxPktStat );
    _DRV_ETHMAC_RxUnlock(pMacD);

    if(ethRes == DRV_ETHMAC_RES_PACKET_QUEUED || ethRes == DRV_ETHMAC_RES_NO_PACKET)
    { // done, no more packets
        mRes = TCPIP_MAC_RES_PENDING;
    }
    else if(ethRes == DRV_ETHMAC_RES_OK)
    { // available packet; minimum check
#ifdef STACK_USE_WIRESHARK
 WireSharkCallBack(pRootDcpt->pBuff, pRootDcpt->nBytes);
#endif
        if(pRxPktStat->rxOk == 0 || pRxPktStat->runtPkt != 0 || pRxPktStat->crcError != 0)


Erik Friesen
#1

12 Replies Related Threads

    friesen
    Super Member
    • Total Posts : 1689
    • Reward points : 0
    • Joined: 2008/05/08 05:23:35
    • Location: Indiana, USA
    • Status: offline
    Re: TCPIP issue 2.04 2017/10/02 13:26:22 (permalink)
    0
    Update, tx wireshark capture was wrong, I had to iiterate through the segments like
     
     
    void WireSharkCallBackPkt(TCPIP_MAC_DATA_SEGMENT * segptr){
        unsigned char buff[4096];
        int buflen = 0;
        if (ActiveCapture) {
     while (segptr != 0) {
         if (buflen + segptr->segLen>sizeof (buff)) {
      break;
         }
         memcpy(&buff[buflen], segptr->segLoad, segptr->segLen);
         buflen += segptr->segLen;
         segptr = segptr->next;
     }
     WireSharkCallBack(buff, buflen);
        }
    }

     
    The end result is now that whenever multiple large packets (4 * 1514 bytes) are sent close together, tx shows on the outbound capture, but PC isn't showing all the packets.  
     
    It is a custom board using a lan8741 in mii.

    Erik Friesen
    #2
    friesen
    Super Member
    • Total Posts : 1689
    • Reward points : 0
    • Joined: 2008/05/08 05:23:35
    • Location: Indiana, USA
    • Status: offline
    Re: TCPIP issue 2.04 2017/10/02 15:05:29 (permalink)
    0
    One question I do have though, why is the mac tx fed from multiple descriptors for the same packet?  See DRV_ETHMAC_LibTxSendPacket where it loops through the pPkt buffers.  I know from testing that these splits are within the same on the wire packet, which seems funny.  Each descriptor is 1536, so what is the point of splitting?
    post edited by friesen - 2017/10/02 15:27:14

    Erik Friesen
    #3
    friesen
    Super Member
    • Total Posts : 1689
    • Reward points : 0
    • Joined: 2008/05/08 05:23:35
    • Location: Indiana, USA
    • Status: offline
    Re: TCPIP issue 2.04 2017/10/03 07:10:07 (permalink)
    0
    (Edit)Erased, unrelevant.
    post edited by friesen - 2017/10/03 07:36:20

    Erik Friesen
    #4
    friesen
    Super Member
    • Total Posts : 1689
    • Reward points : 0
    • Joined: 2008/05/08 05:23:35
    • Location: Indiana, USA
    • Status: offline
    Re: TCPIP issue 2.04 2017/10/03 11:16:39 (permalink)
    0
    I confirmed this behavior with the DA internal DDR starter kit as well.  
     
    As a side note, the chip starter kit's MAC is FF's, so I had to put something there, 2+ hours wasted on that.

    Erik Friesen
    #5
    friesen
    Super Member
    • Total Posts : 1689
    • Reward points : 0
    • Joined: 2008/05/08 05:23:35
    • Location: Indiana, USA
    • Status: offline
    Re: TCPIP issue 2.04 2017/10/05 18:18:04 (permalink)
    0
    I have determined that the eth tx engine is getting underruns. Further discussion will be here http://www.microchip.com/...spx?m=1018402&fp=1

    Erik Friesen
    #6
    rainad
    Moderator
    • Total Posts : 812
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: online
    Re: TCPIP issue 2.04 2017/10/06 08:58:09 (permalink)
    3 (1)
    That's usually caused by the system being late in transferring data to the Ethernet controller.
    The data on the bus has to meet a timing requirement and the transfer will be aborted if the data is not there.
    Please check what other bus/DMA transfers you have in your application that may cause this behavior.
     
     
    #7
    friesen
    Super Member
    • Total Posts : 1689
    • Reward points : 0
    • Joined: 2008/05/08 05:23:35
    • Location: Indiana, USA
    • Status: offline
    Re: TCPIP issue 2.04 2017/10/06 09:57:21 (permalink)
    0
    I basically don't have any significant dma transfers going on.  I have a couple audio spi ins and out going to coherent cache locations.  The stack is set up to use ddr, so the ethernet dma would be happening from a different target, not? What type of accesses would actually block the ethernet dma for significant periods?
     
    What are the timing requirements?  Does DDR meet those requirements?
     
    Is there a possibility this can happen from an EOWN descriptor race condition or other mis set?  If I may say so, the harmony stack is too complex in the descriptor handling, I see a margin for error there.
     
    These don't seem to show up until there are significant transfers going on.  I get underruns showing up in ackpackets, but the interrupt doesn't actually happen unless the packet doesn't go out, it seems.

    Erik Friesen
    #8
    rainad
    Moderator
    • Total Posts : 812
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: online
    Re: TCPIP issue 2.04 2017/10/06 11:14:45 (permalink)
    0
    The timing requirement is obvious: 100 Mbps = 12.5 MBps.
    And you'd need some reserve on that. I'd say you need to support at least 20 MBps to be on the safe side.
    Please check the data sheet for the DDR timing.
    I'll try to get some info on this and will tell you what I find.
    Yo can do some simple tests though:
    - use a 10 Mbps link and check the behavior. If everything works fine, most likely the DDR access is late.
    - do a large memory to memory copy involving DDR, repeat it for a number of times and calculate the transfer speed that you can get. That would give a fairly good idea of where you are. Make sure you use optimized/assembly code to get some real numbers.
     
    Not likely, in my opinion, that there is a race with the descriptor ownership in the MAC driver.
    Not impossible, of course, but this should have been noticed before, especially with fast memories.
     
    #9
    friesen
    Super Member
    • Total Posts : 1689
    • Reward points : 0
    • Joined: 2008/05/08 05:23:35
    • Location: Indiana, USA
    • Status: offline
    Re: TCPIP issue 2.04 2017/10/06 11:35:06 (permalink)
    0
    So far I have confirmed that operating this code outside of ddr does not trip the interrupt.  However I still get loads of software underruns, on the scale of 50 or so per 20kb file.  Are these expected?
     
    I am checking this way
     
    /****************************************************************************
     * Function: EthTxAcknowledgeBuffer
     *****************************************************************************/
    DRV_ETHMAC_RESULT DRV_ETHMAC_LibTxAcknowledgeBuffer(DRV_ETHMAC_INSTANCE_DCPT* pMacD, const void* pBuff, DRV_ETHMAC_BUFF_AckF ackFnc, void* fParam)
    {
        DRV_ETHMAC_RESULT res;
        DRV_ETHMAC_DCPT_NODE* pEDcpt;
        DRV_ETHMAC_DCPT_LIST* pAckList;
        uint8_t ackList [(DRV_ETHMAC_DCPT_LIST_ALIGN -1) + sizeof(DRV_ETHMAC_DCPT_LIST)];
        
        pAckList = DRV_ETHMAC_LIB_ListInit(_EthAlignAdjust(ackList));
        
        res = _EthAckPacket(pBuff, pMacD->mData._EnetTxBusyPtr, pAckList, ackFnc, fParam, 1);

        while((pEDcpt=DRV_ETHMAC_LIB_ListRemoveHead(pAckList)))
        {
            pEDcpt->hwDcpt.pEDBuff = 0; // corresponding buffer is not owned
            DRV_ETHMAC_LIB_ListAddTail(pMacD->mData._EnetTxFreePtr, pEDcpt);
        }

        return res;
    }

     
     
    static DRV_ETHMAC_RESULT _EthAckPacket(const void* pPkt, DRV_ETHMAC_DCPT_LIST* pRemList, DRV_ETHMAC_DCPT_LIST* pAddList, DRV_ETHMAC_BUFF_AckF ackFnc, void* fParam, int Tx)
    {
        static unsigned int pktid = 0;
        DRV_ETHMAC_DCPT_NODE *pEDcpt;
        DRV_ETHMAC_DCPT_NODE *prev, *next;
        int nAcks;
        int pktFound;
        int buffIx;

        prev=next=0;
        nAcks=pktFound=0;
        
        for(pEDcpt=pRemList->head; pEDcpt!=0; pEDcpt=next)
        {
            if(pEDcpt->hwDcpt.hdr.SOP && (pPkt==0 || pEDcpt->hwDcpt.pEDBuff==(unsigned char*)KVA_TO_PA(pPkt)))
            { // found the beg of a packet
                pktFound=1;

                if(pEDcpt->hwDcpt.hdr.EOWN)
         {
      break; // hw not done with it
         }
         pktid++;
         next = pEDcpt;
         buffIx = 0;
         do {
      pEDcpt = next;
                    next=pEDcpt->next;
                    while(pEDcpt->hwDcpt.hdr.EOWN); // shouldn't happen
                    DRV_ETHMAC_LIB_ListAddTail(pAddList, pEDcpt); // ack this node
                    if(ackFnc)
                    {
                        void* pBuff;
                        pBuff=(pEDcpt->hwDcpt.hdr.kv0?PA_TO_KVA0((int)pEDcpt->hwDcpt.pEDBuff):PA_TO_KVA1((int)pEDcpt->hwDcpt.pEDBuff));
                        (*ackFnc)(pBuff, buffIx++, fParam); // call user's acknowledge
                    }
      if (Tx) {
          if (pEDcpt->hwDcpt.stat & (1 << 31)) {
       void* pBuff2;
       pBuff2 = (pEDcpt->hwDcpt.hdr.kv0 ? PA_TO_KVA0((int) pEDcpt->hwDcpt.pEDBuff) : PA_TO_KVA1((int) pEDcpt->hwDcpt.pEDBuff));
       Debug("TUR:%u s%u e%u C=%u 0x%X\r\n", pktid,
        pEDcpt->hwDcpt.hdr.SOP, pEDcpt->hwDcpt.hdr.EOP,
        pEDcpt->hwDcpt.hdr.bCount, (unsigned int) pBuff2);
          }
      }
                }while(!pEDcpt->hwDcpt.hdr.EOP);

                nAcks++;
                // reconstruct the removed list
                if(prev)
                {
                    prev->next=next;
                    // prev->next_ed shouldn't matter here!
                }
                else
                {
                    pRemList->head=next;
                }

                if(pPkt)
                { // done, just one packet ack-ed
                    break; // done
                }
            }
            else
            {
                prev=pEDcpt;
                next=pEDcpt->next;
            }
        }

        return nAcks?DRV_ETHMAC_RES_OK:(pktFound?DRV_ETHMAC_RES_PACKET_QUEUED:DRV_ETHMAC_RES_NO_PACKET);

    } //_EthAckPacket

     
    Loading the 20kb file results in
     
    TUR:8 s1 e1 C=87 0xA002026A
    TUR:9 s1 e1 C=42 0xA002DE46
    TUR:11 s1 e1 C=72 0xA002CF0A
    TUR:16 s1 e1 C=90 0xA001FFAA
    TUR:45 s1 e1 C=42 0xA002DE46
    TUR:47 s1 e1 C=62 0xA0024B32
    TUR:50 s1 e1 C=54 0xA0024B32
    TUR:51 s1 e0 C=54 0xA001FF02
    TUR:52 s1 e0 C=54 0xA0024B32
    TUR:53 s1 e0 C=54 0xA001FF02
    TUR:53 s0 e1 C=1460 0xA0026178
    TUR:55 s1 e0 C=54 0xA0024B32
    TUR:55 s0 e1 C=1140 0xA002672C
    TUR:57 s1 e0 C=54 0xA0024B32
    TUR:57 s0 e0 C=1 0xA0026BA0
    TUR:57 s0 e1 C=1023 0xA0025BA0
    TUR:58 s1 e0 C=54 0xA001FF02
    TUR:58 s0 e1 C=1460 0xA0025F9F
    TUR:59 s1 e0 C=54 0xA0024B32
    TUR:59 s0 e1 C=1460 0xA0026553
    TUR:61 s1 e0 C=54 0xA0024B32
    TUR:63 s1 e0 C=54 0xA0024B32
    TUR:63 s0 e1 C=1022 0xA0025BA0
    TUR:64 s1 e0 C=54 0xA001FF02
    TUR:64 s0 e1 C=1460 0xA0025F9E
    TUR:65 s1 e0 C=54 0xA0024B32
    TUR:65 s0 e1 C=1460 0xA0026552
    TUR:67 s1 e0 C=54 0xA0024B32
    TUR:67 s0 e1 C=152 0xA0026B06
    TUR:69 s1 e0 C=54 0xA0024B32
    TUR:69 s0 e0 C=3 0xA0026B9E
    TUR:69 s0 e1 C=1021 0xA0025BA0
    TUR:70 s1 e0 C=54 0xA001FF02
    TUR:70 s0 e1 C=1460 0xA0025F9D
    TUR:71 s1 e0 C=54 0xA0024B32
    TUR:71 s0 e1 C=1460 0xA0026551
    TUR:73 s1 e0 C=54 0xA0024B32
    TUR:73 s0 e1 C=152 0xA0026B05
    TUR:75 s1 e0 C=54 0xA0024B32
    TUR:75 s0 e0 C=4 0xA0026B9D
    TUR:75 s0 e1 C=1020 0xA0025BA0
    TUR:76 s1 e0 C=54 0xA001FF02
    TUR:76 s0 e1 C=1460 0xA0025F9C
    TUR:77 s1 e0 C=54 0xA0024B32
    TUR:77 s0 e1 C=1460 0xA0026550
    TUR:79 s1 e0 C=54 0xA0024B32
    TUR:79 s0 e1 C=152 0xA0026B04
    TUR:81 s1 e0 C=54 0xA0024B32
    TUR:81 s0 e0 C=5 0xA0026B9C
    TUR:81 s0 e1 C=1019 0xA0025BA0
    TUR:82 s1 e0 C=54 0xA001FF02
    TUR:82 s0 e1 C=1460 0xA0025F9B
    TUR:83 s1 e0 C=54 0xA0024B32
    TUR:83 s0 e1 C=1460 0xA002654F
    TUR:85 s1 e0 C=54 0xA0024B32
    TUR:85 s0 e1 C=152 0xA0026B03
    TUR:87 s1 e0 C=54 0xA0024B32
    TUR:87 s0 e0 C=6 0xA0026B9B
    TUR:87 s0 e1 C=1018 0xA0025BA0
    TUR:88 s1 e0 C=54 0xA001FF02
    TUR:88 s0 e1 C=1460 0xA0025F9A
    TUR:89 s1 e0 C=54 0xA0024B32
    TUR:89 s0 e1 C=1460 0xA002654E
    TUR:91 s1 e0 C=54 0xA0024B32
    TUR:91 s0 e1 C=152 0xA0026B02
    TUR:93 s1 e0 C=54 0xA0024B32
    TUR:93 s0 e0 C=7 0xA0026B9A
    TUR:93 s0 e1 C=1017 0xA0025BA0
    TUR:94 s1 e0 C=54 0xA001FF02
    TUR:94 s0 e1 C=1460 0xA0025F99
    TUR:95 s1 e0 C=54 0xA0024B32
    TUR:95 s0 e1 C=1460 0xA002654D
    TUR:97 s1 e0 C=54 0xA0024B32
    TUR:97 s0 e1 C=152 0xA0026B01
    TUR:99 s1 e0 C=54 0xA0024B32
    TUR:99 s0 e0 C=8 0xA0026B99
    TUR:99 s0 e1 C=620 0xA0025BA0
    TUR:100 s1 e1 C=54 0xA0024B32
    TUR:103 s1 e1 C=54 0xA0024B32

    post edited by friesen - 2017/10/06 11:38:25

    Erik Friesen
    #10
    friesen
    Super Member
    • Total Posts : 1689
    • Reward points : 0
    • Joined: 2008/05/08 05:23:35
    • Location: Indiana, USA
    • Status: offline
    Re: TCPIP issue 2.04 2017/10/06 17:28:32 (permalink)
    0
    I think the above is rather caused by incorrect masking of the long long stat reg. I suppose it will be the ddr access speed or some as yet unknown errata, so Ill have to either re buffer the data in kseg1 or retransmit the packet, not sure which will be easier.

    Erik Friesen
    #11
    friesen
    Super Member
    • Total Posts : 1689
    • Reward points : 0
    • Joined: 2008/05/08 05:23:35
    • Location: Indiana, USA
    • Status: offline
    Re: TCPIP issue 2.04 2017/10/07 08:29:06 (permalink)
    3 (1)
    Anyway, Rebuffering the tx descriptors in kesg1 space does fix this problem.
     
    I seems like a hardware problem to me, as the ETH -> DDR RX doesn't seem to have the issue.

    Erik Friesen
    #12
    rainad
    Moderator
    • Total Posts : 812
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: online
    Re: TCPIP issue 2.04 2017/10/09 09:20:02 (permalink)
    3 (1)
    The DDR access is optimized for GFX accesses on this chip.
    ETH doesn't have high priority, it can be starved by CPU or other higher priority masters. On top of that, the ETH DMA transfers don't occur in bursts but cycle by cycle. Overall the memory access is slow. This is what you are noticing.
    I think that you have the right solution, having the ETH buffers in internal/faster RAM.
    #13
    Jump to:
    © 2017 APG vNext Commercial Version 4.5