• AVR Freaks

AnsweredHot!Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full

Author
jdeguire
Super Member
  • Total Posts : 467
  • Reward points : 0
  • Joined: 2012/01/13 07:48:44
  • Location: United States
  • Status: offline
2019/07/19 09:09:34 (permalink)
0

Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full

I'm evaluating Harmony 3 using a few different devices--the SAME70 on the SAME70 Xplained board, the SAME54 on the SAME54 Xplained Pro board, and the PIC32MZ EF on one of our own boards.
 
I've run into an issue with the TCP/IP Stack that I believe I was able to correct.  Unfortunately, I had to put the details (my original post) in a Word document because these forums are awful ("ACCESS DENIED!!!").
#1
rainad
Moderator
  • Total Posts : 1202
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full 2019/07/19 13:54:48 (permalink)
0
Hi jdeguire,
 
This seems to be a problem in the GMAC (SAME70 and probably other SAM devices) driver which will be corrected immediately.
As explained in the tcpip_mac.h, the function TCPIP_MAC_PacketTx() is required to return either TCPIP_MAC_RES_OK or an error code. 
But inadvertently, as you discovered, the DRV_GMAC_PacketTx() returns the result of _MacTxPendingPackets() (which is wrong as that's just an internal result).
The driver is required to queue the packets to be transmitted in an internal queue ( a  very cheap operation).
 
So the fix is, at the end of DRV_GMAC_PacketTx():
macRes = _MacTxPendingPackets(pMACDrv,queueIdx);
_MACTxAcknowledgeEth(pMACDrv,queueIdx);
_DRV_GMAC_TxUnlock(pMACDrv);
// return macRes;
return TCPIP_MAC_RES_OK.
 
Regarding the 2nd point:
jdeguire
 On another note, I also noticed that the stack does not seem to have a mechanism to retry sending pending Tx packets.

 
There is no queue for storing the pending TX packets in the stack. The driver, as explained above, takes care of that. So there's no modification needed in the stack code.
 
Thank you for finding the inconsistency in the GMAC driver.
The fix will be part of the next release.
 
 
 
#2
jdeguire
Super Member
  • Total Posts : 467
  • Reward points : 0
  • Joined: 2012/01/13 07:48:44
  • Location: United States
  • Status: offline
Re: Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full 2019/07/22 07:13:00 (permalink)
0
Hi rainad,
 
Thanks for the response!  I just took a closer look at the PIC32 Ethmac driver and, sure enough, it indeed ensures to never return the PENDING status when sending packets.  I'll put your fix into our working copy as that seems much simpler than what I had.
 
For the second point, I suspect that we're talking about the same thing but my terminology is incorrect.  In the GMAC driver, I'm referring to packets stored in the sGmacData.gmac_queue[queueIdx]._TxStartQueue, but there's a similar queue in the Ethmac driver.  From what I can tell, any packets that were not immediately sent because there weren't enough descriptors are stored in the queue.  The drivers' DRV_WHATEVER_Process() functions appear to be what sends the still-pending packets.  This function is called in TCPIP_STACK_Tasks() via a function pointer only if the stack's internal event counter (totTpcipEventsCnt) is non-zero.  That counter is incremented via a callback in the driver's ISR (_TcpNotifyFnc).  From what I can tell, by default the interrupt does NOT fire when a packet was successfully transmitted because the PIC32C_GMAC_ISR_TX and ETH_PIC32_INT_MAC_ISR_TX macros are never defined.  Thus, the interrupt fires only on a receive interrupt (see GMAC_INT_BITS in drv_gmac_lib_whatever.c).  Thus, the DRV_WHATEVER_Process() function is called, so and the Tx queue is processed, only when a new packet is received (or you have a receive error).
 
Is that correct?  If so, then I can come up with situations in which pending Tx packets are sitting in the driver queue for a while.  Either the firmware needs to send more packets or the remote device needs to send a response.  If neither of those happen for a while, then won't any pending Tx packets just sit there?
#3
rainad
Moderator
  • Total Posts : 1202
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full 2019/07/23 09:01:25 (permalink)
0
If the TX interrupts are not enabled and there's no event when done with transmitting a packet, then what you describe can happen. If there's no RX event either, yes, packets will just be waiting to be transmitted until a new packet needs to be transmitted.
For the PIC32M MAC driver the TX interrupts are enabled.
I'll have to check for the GMAC and I'll get back to you.
 
 
#4
jdeguire
Super Member
  • Total Posts : 467
  • Reward points : 0
  • Joined: 2012/01/13 07:48:44
  • Location: United States
  • Status: offline
Re: Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full 2019/07/23 10:57:24 (permalink)
0
In both my copy and the latest on GitHub, it looks like that line 103 of drv_ethmac_local.h defines ETH_PIC32_INT_MAC_ISR_TX, but that line is commented out.  This is also true for line 173 of drv_gmac_local.h and PIC32C_GMAC_ISR_TX.
 
Should those lines be uncommented or do I need to enable the Tx interrupt in some other way (MHC option?)?  I couldn't find anywhere else where the macros are defined; just some places they are checked.
#5
rainad
Moderator
  • Total Posts : 1202
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full 2019/07/23 11:24:21 (permalink)
5 (1)
The symbol ETH_PIC32_INT_MAC_ISR_TX serves a different purpose, it's not for enabling the interrupts.
It is not an option to be played with, as it is an internal setting and it should be always disabled.
The same is true for the PIC32C_GMAC_ISR_TX, the symbol should not be enabled.
 
I'll look into the GMAC interrupts to check if indeed the TX interrupts are not enabled.
For the PIC32M they are enabled, so there should not be an issue.
 
 
#6
rainad
Moderator
  • Total Posts : 1202
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full 2019/07/24 08:30:47 (permalink) ☼ Best Answerby jdeguire 2019/07/25 06:35:41
5 (1)
OK, we have verified the GMAC driver behavior and the TX interrupts were not enabled.
So, the issue that you described could potentially happen.
Thank you for finding and reporting this.
It will be fixed in the next release.
 
 
#7
jdeguire
Super Member
  • Total Posts : 467
  • Reward points : 0
  • Joined: 2012/01/13 07:48:44
  • Location: United States
  • Status: offline
Re: Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full 2019/07/25 06:34:53 (permalink)
0
Gotcha, thanks for checking that for me!
 
Is the fix to just remove this bit from DRV_GMAC_EventMaskSet?  I'll update my own copy if so.
 
#if !defined(PIC32C_GMAC_ISR_TX)
        //mask TX DONE event/interrupt when TX Interrupt processing is disabled
        macEvMask &= ~TCPIP_MAC_EV_TX_DONE;
#endif // defined(PIC32C_GMAC_ISR_TX)

 
#8
rainad
Moderator
  • Total Posts : 1202
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full 2019/07/25 14:02:27 (permalink)
#9
jdeguire
Super Member
  • Total Posts : 467
  • Reward points : 0
  • Joined: 2012/01/13 07:48:44
  • Location: United States
  • Status: offline
Re: Harmony 3 TCPIP Stack Potential Lockup when MAC Tx Descriptors Full 2019/07/26 12:00:09 (permalink)
0
Oh, excellent.  Thanks for the update!
#10
Jump to:
© 2019 APG vNext Commercial Version 4.5