• AVR Freaks

Hot!TCPIP Stack (UDP only) stops working after a while?

Author
Frankaas
Starting Member
  • Total Posts : 46
  • Reward points : 0
  • Joined: 2018/12/12 02:02:21
  • Location: 0
  • Status: offline
2019/01/11 06:39:58 (permalink)
0

TCPIP Stack (UDP only) stops working after a while?

Hello all,
 
I am working with a PIC32MX795F512L with SMSC 8720 PHY. I am using MPLAB X IDE v5.10, Harmony v2.06 and XC32 v2.10.
 
With a lot of blood, sweat and tears I've managed to reduce the harmony example: "tcpip_udp_client" to a fairly simple version with just UDP and CMPv4 enabled which is what I need for my project. For starters I am trying to send some packets, which works (I use wireshark to check), for the most part... The problem is that after some time (sometimes 15 minutes, sometimes an hour or more), no more packets are being sent and it is no longer possible to ping the board.
 
In my app task, I create an UDP client socket and when this was successful, I periodically (when I want data to be sent) check whether the socket is connected and PutIsReady before sending the data (ArrayPut and Flush). I also check whether GetIsReady and if yes, read data from the socket, I have no idea whether this is actually necessary.
 
Ultimately the idea is that I can send and receive data from the same port. However, when opening the client socket, I can only set the remote port, the manual states that "The local port for client sockets will be automatically picked by the UDP module", is there any way around that? Or should I open a server socket as well, with the same port and use that for receiving? But then should I also keep checking the RX of the client socket to avoid overflow?
 
Any advice is veeerrrrryyyy welcome! Thank you!!
 
Franka
#1

12 Replies Related Threads

    rainad
    Super Member
    • Total Posts : 1192
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/01/11 09:13:46 (permalink)
    0
    Yes, reading the pending data from the socket is absolutely necessary, or at least discarding the packets. Otherwise the socket RX will lock up and the packet buffers won't be released.
     
    If you need to set the local port for the UDP socket you can use the call: 
    TCPIP_UDP_Bind(skt, IP_ADDRESS_TYPE_IPV4, localPort,  0);
    Or, as you said, open a server socket that specifies the local port to listen on in the socket creation call.
     
    Not being able to ping the board anymore after a while indicates a serious problem.
    It may be caused by the fact that you don't consume the UDP RX packets that are sent to you and those keep accumulating until you run out of memory.
    Do you have a serial console attached to that board?
    That would allow us to check the status dynamically when running into this situation.
     
    Probably increasing the size of the TCP/IP heap, if possible, would also help.
     
    #2
    Frankaas
    Starting Member
    • Total Posts : 46
    • Reward points : 0
    • Joined: 2018/12/12 02:02:21
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/01/11 09:41:51 (permalink)
    0
    Hey Rainad,
     
    First of all, thank you very much for your reply, I have been at this for quite some time now and a little desperate ;). 
     
    Just found the TCPIP_UDP_Bind(), it works :), .
     
    As far as I know, I am consuming the UDP RX packets calling while(TCPIP_UDP_GetIsReady( clientSocket ) and if  yes, use TCPIP_UDP_ArrayGet followed by TCPIP_UDP_Discard. What could be wrong here?
     
    I've increased the size of the TCPIP heap to 30000 (I ran into trouble while initializing the stack) which, from what I have read in the "TCP/IP Stack Libraries Help" should be more than enough, I have a maximum of 4 UDP sockets set in MHC of which I now use 1, aside from this, only ARP and CMPv4 are enabled. I've kept track of how long it takes for things to go wrong and it is around 25mins, maybe I can try to increase heap size and see whether the problem shows up later? I am sending a packet every second, but checking the RX buffer happens a lot more frequent in the main (apptask).
     
    I unfortunately am forced to use a board that has made no uart available, it does have a USB connector but I have spent a lot of time earlier trying to get that to work for debugging purposes but to no avail. I could not get my Windhoes 7 PC to recognize it.
     
    Thanks again, I hope there is a fix for this, I just need verrry simple, basic but hopefully stable UDP :(
     
    Franka
    post edited by Frankaas - 2019/01/11 09:44:23
    #3
    rainad
    Super Member
    • Total Posts : 1192
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/01/11 11:55:15 (permalink)
    0
    I'd try first to increase the project heap size and the TCP/IP heap size. 
    Also, you can use the model from the tcpip_commands.c and call the functions for the "heapinfo" command when you run into this situation to see what's happening. Also, there is a "udpinfo" function that might be interesting, it will show if there's pending data for the socket - basically just call TCPIP_UDP_SocketInfo.
     
    Or, better, if possible, move your tests on a standard PIC32 ESK board where you have a serial line and that will help with the debugging, at least until we understand what's going on.
    The lack of memory seems to be the most probable cause for what you see. I assume the board is otherwise working, it's not stuck in an exception, it's still blinking and LED or shows somehow that it's up and running.
     
     
    #4
    Frankaas
    Starting Member
    • Total Posts : 46
    • Reward points : 0
    • Joined: 2018/12/12 02:02:21
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/01/11 14:23:12 (permalink)
    0
    Hello Rainad,
     
    Thanks so much for helping me!
     
    I will try to increase the heap size further, fingers crossed that it helps. 
     
    The weird thing is that it has been running without problems for a record time now (without the increased heap size), the only difference is that I do not have the PICkit3 connected now, all the previous times I had it connected. Could that somehow make a difference?
     
    Also, when I started with this project I set the tick rate for the stack to 35ms instead of the 5ms in the example project, could this somehow be making it more difficult to process the data in time causing the stack to require more heap than it would otherwise?
     
    I will now wait for the stack to fail (hopefully it won't though and the behaviour can somehow be blamed on the PICkit ;)) and then try increasing the heap and checking the heapinfo and udpinfo. 
     
    Franka
    post edited by Frankaas - 2019/01/11 14:28:49
    #5
    rainad
    Super Member
    • Total Posts : 1192
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/01/12 08:57:22 (permalink)
    0
    Hi Franka,
     
    I think that it's possible that decreasing the tick rate to 35 ms creates issues with the heap. The ETH controller works in interrupts and reports the events to the stack, so it shouldn't matter. However some services may become less responsive because of the decreased tick frequency and will process their pending packets less frequently . Do you have any particular reason to change from the default 5 ms tick rate? If not, maybe it's worth considering  changing it back. Of course, it depends on the frequency that you have set up for your system timer.
     
    From my experience, I've noticed pretty often that I cannot run lengthy tests when the application runs under the debugger. Not sure what the cause is but the IDE tends to lose the connection to the board after a while and the application halts. I'd recommend that you stick to the approach you took, run your tests without the debugger.
     
    Let me know what you get.
     
     
    #6
    Frankaas
    Starting Member
    • Total Posts : 46
    • Reward points : 0
    • Joined: 2018/12/12 02:02:21
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/01/12 12:20:37 (permalink)
    0
    Hi Rainad,
     
    Changing the tick rate was something I did very early on before I understood what it did exactly.
    I have experience with microcontrollers however not with a PIC, MPLABX or Harmony so it was and is quite the learning curve ;). I really appreciate you taking the time to help me along! I will change the tick rate back because there is no reason for it to be this slow. 
     
    The stack is still running and UDP packets are still being sent, for more than 24 hours now. Before it never lasted longer than approx. 25 minutes so all things seem fine now. The weird thing is that the problem also occurred when I simply had the PICkit3 connected to the board and to the PC but was not debugging, I just used it for programming mostly. Even though I would very much like to know the explanation, if it keeps working now I accept that it might behave a little differently when I have the PICkit3 connected but that it works fine without it. Also, when the problem with the stack occurred, the rest of the program was still running fine, blinking LEDs, AD-converting etc.
     
    I will do some more tests and report back here if I find anything.
     
    Thank you!!!
     
    Franka
    #7
    rainad
    Super Member
    • Total Posts : 1192
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/01/14 08:07:59 (permalink)
    0
    Well, if you mean that when the lock up occurred, with the PICkit3 connected, the rest of the system was working fine, then it seems to be a TCP/IP stack issue. If it's completely dead then the console would help, at least to give an idea what's going on. Without a console, we need to run it under debugger, wait for the lock up to occur and then start investigating- usually a tedious process. If you have a way to make it lock up I can take a look into it.
    I haven't seen this kind of behavior, the stack has been stable for a number of years now and we ran extensive, lengthy tests. However, it's not impossible that something like this might occur. The 1st thing that would have an impact would be not enough heap memory for the stack.
    Let me know what your experiments show and w'll take it from there.
     
     
    #8
    Frankaas
    Starting Member
    • Total Posts : 46
    • Reward points : 0
    • Joined: 2018/12/12 02:02:21
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/02/07 03:06:29 (permalink)
    0
    So, I am back. Some other parts of my project needed attention but now I am back testing the udp comms. It seemed to work well but the stack unfortunately still fails. 
     
    Heap is now 45000,
    max sockets is 2 (1 is used),
    default TX buffer size is 1024,
    TX and RX queue limits are both 10
    Stack tick rate is 5ms
     
    I am very certain that all incoming data is read from the RX buffer for the client socket that I created and also that there is not too much data in the TX queue since I am receiving and sending packets (of approximately 12 bytes each) at around 20 times per second. I cannot imagine that the stack can't handle that?
     
    Can you give me some pointers (pun intended) on how to use "udpinfo" and "heapinfo" to attempt to debug this? It seems that the stack gets killed, do these debug commands still work/provide useful info then? Unfortunately still no console, so I will have to use the debugger.
     
    Thanks!
     
    Franka
    #9
    Frankaas
    Starting Member
    • Total Posts : 46
    • Reward points : 0
    • Joined: 2018/12/12 02:02:21
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/02/07 04:03:30 (permalink)
    0
    By the way, there is another weird thing. I have no idea if it is related to my problem but I've set the PICs MAC to 66.55.44.33.22.11, however, in the packets it sends, I see that it is actually 66.00.00.00.00.00. What could that mean?
    #10
    rainad
    Super Member
    • Total Posts : 1192
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/02/07 07:23:31 (permalink)
    0
    Where do you see the 66:55:44:... MAC address, in the ETH registers?
    If the packets go out with a different MAC address, probably some other part of the code steps over the packet buffers. Does this happen for all packets or just once in a while? Something seems to be really wrong in your set up. What is the MAC address of the interface at initialization and at run time (use TCPIP_STACK_NetAddressMac call).
     
    Without some sort of communicating with the board at run time, to know what's going on, it's going to be difficult to get to the bottom of the issue. 
    Obviously you can add code to ask for socket and memory status (udpinfo, heapinfo calls in the tcpip_command.c show an example), but you need a way to call those at will and to get their results. If you can do that somehow, then we may learn more.
     
    I'd try this first:
    Simplify your application, i.e. create a very simple test app that doesn't do anything else other than sending periodic  UDP messages and receive what the socket data is available. So basically like the real app does but use static buffers for both TX and RX traffic and do proper length checking to avoid any possibility of overflowing the buffer boundaries. For the TX side you can use even a constant buffer.
    Let's see how this one behaves.
     
     
     
    #11
    Frankaas
    Starting Member
    • Total Posts : 46
    • Reward points : 0
    • Joined: 2018/12/12 02:02:21
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/02/10 14:21:11 (permalink)
    0
    Oke, the wrong MAC was my fault entirely, something regarding using points instead of semicolons to separate the bytes...
     
    Furthermore, I've cleaned up my code some more, avoiding the use of dynamic buffers as much as possible. My application has run all day without the stack failing. Unfortunately I still haven't found something that can conclusively explain its failing.
     
    Will run some more tests in the coming days and see if it continues to behave. I'll also try to get an ethernet starter kit although I'd think I should be able to get the USB console working on my custom board as the ethernet and USB part is identical to Microchip's starter kit.
    #12
    Frankaas
    Starting Member
    • Total Posts : 46
    • Reward points : 0
    • Joined: 2018/12/12 02:02:21
    • Location: 0
    • Status: offline
    Re: TCPIP Stack (UDP only) stops working after a while? 2019/02/28 14:15:52 (permalink)
    0
    Just to report back:
     
    The stack has been behaving very well. What I have done is:
    - Change "max (udp) sockets" to 1 since I only use one.
    - Use only static buffers.
    - Did a clean & build.
     
    So I have unfortunately not been able to pinpoint a specific cause but it has been performing very well.
    Thank you for all your help Rainad!
     
    I did however run into another thing which I think is related to the PHY but will create a new topic for this.
     
    Franka
     
     
    #13
    Jump to:
    © 2019 APG vNext Commercial Version 4.5