• AVR Freaks

Hot!Loss of Network on Harmony V1.11

Author
NKurzman
A Guy on the Net
  • Total Posts : 18975
  • Reward points : 0
  • Joined: 2008/01/16 19:33:48
  • Location: 0
  • Status: online
2020/09/30 15:20:46 (permalink)
0

Loss of Network on Harmony V1.11

Rainad:
I am loosing network connection on some units. At some locations. This is related to my previous net work issues.
I have a V1.11 Stack with V2.06 DHCP.
The Failure appears as a DNS Failure first.
Then DHCP will fail on the Renewal.   if gets stuck at state 10, TCPIP_DHCP_GET_RENEW_ACK
I do not have the console.
My Goal is to get the same Information and either send it to a file, or display it on the screen.
Does DHCP and DNS keep their sockets and memory, or return it when not used?
Could this be insufficient Memory, or Sockets?
 
In a Previous post to some one else you suggested looking at:
>netinfo
>heapinfo
>macinfo
Does the same advice apply here?
Can you suggest a project that has the console that I can copy?
Or do you know the Functions needs to get the information?
#1

19 Replies Related Threads

    friesen
    Super Member
    • Total Posts : 2154
    • Reward points : 0
    • Joined: 2008/05/08 05:23:35
    • Location: Indiana, USA
    • Status: offline
    Re: Loss of Network on Harmony V1.11 2020/10/01 06:27:15 (permalink)
    5 (1)
    Do you have a way to log socket id's?  This bug bit me, I think in 2.04.
     
    #1. Process 1 uses socket id 10, sleeps for some period.
    #2. socket 10 dies.  The stack closes this socket and flags it ready to use
    #3. Process 2 asks for and receives socket 10
    #4. Process 1 uses socket 10 because the stack says its fine.  Everything blows up now.
     
    The issue is that the stack can close and reissue the socket id without specific approval from the opener.

    Erik Friesen
    #2
    rainad
    Moderator
    • Total Posts : 1412
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: Loss of Network on Harmony V1.11 2020/10/01 07:54:03 (permalink)
    0
    The issue mentioned by Friesen refers to TCP sockets.
    The DHCP and DNS use UDP, so this should not have any impact.
    I think that the 'netinfo, macinfo, heapinfo' would still be useful, because it may be something more serious than just a socket getting somehow lost.
    The fact that DHCP gets stuck in TCPIP_DHCP_GET_RENEW_ACK is suspicious, as there is a timeout and the state should change.
     
    Is it possible to ping the board when this happens or it's completely dead? Is there any other network connection to the board, and if so, is it running? 
    If there's no console, you'll need to instrument the code to output some data. You could try with the same data that those commands output.
    Do you have a communication channel?
     
    Is this issue relatively easy to reproduce? Maybe you can give it a try with a newer version of the stack? Lots of issues have been addressed, etc.
     
     
    #3
    NKurzman
    A Guy on the Net
    • Total Posts : 18975
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Loss of Network on Harmony V1.11 2020/10/01 08:30:20 (permalink)
    0
    I am guessing that some one either used more sockets, or is not returning them. Or maybe the TCP/IP Heap is too small. If DHCP and DNS held their sockets, then that would not be a possible failure mode.
    How would you even resolve your issue?
    #4
    rainad
    Moderator
    • Total Posts : 1412
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: Loss of Network on Harmony V1.11 2020/10/01 09:43:16 (permalink)
    0
    DNS holds its socket.
    DHCP releases and tries to open again.
    But if it's in TCPIP_DHCP_GET_RENEW_ACK, it means it has a socket.
     
    Anyway, is the board otherwise working when this issue happens, can you ping it?
    #5
    NKurzman
    A Guy on the Net
    • Total Posts : 18975
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Loss of Network on Harmony V1.11 2020/10/01 10:57:22 (permalink)
    0
    I am doing Heap Info Now.
    My Problem is that It works perfectly in most locations, but not others.
    I Think have one at my office that does this.  I hope it stays broken.
    I added a lease renew button, but that did not restore operation.  I can not tell if it is stuck, or just goes through the states fast. I will try a Ping.  I guess the IP will be valid until the Lease expires.
    Note the DNS Fails First.
    The DHCP did not fail until I renewed the Lease.  I did not want to wait 10 days.
    SO you do not feel it is the number of sockets in the pool?
     
    The New Stack, what are the chance it will work with a V1.11 Application?  V2.06
    I can probably move the application to V2.06 but I would need to deal with the Graphics library.  That is why I never moved up.  I am assume a Move to V3.XX would require major rewriting of My Application.
    #6
    rainad
    Moderator
    • Total Posts : 1412
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: Loss of Network on Harmony V1.11 2020/10/01 13:28:31 (permalink)
    0
    I guess you can modify the DHCP function to actually skip the closing of the socket: TCPIP_UDP_Close() if you fear that this may be the reason.
    Are there many UDP sockets in use (and how many UDP sockets are created)?
    It can be because of the lack of memory.
    That's why a ping would be useful to see if it reacts properly.
    I was not suggesting moving the whole application to a newer Harmony version, just the TCP/IP. Or DHCP and DNS only. I can help if you run into issues with the port.
    But I think the main thing is to be able to reproduce the issue.
     
     
     
    #7
    NKurzman
    A Guy on the Net
    • Total Posts : 18975
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Loss of Network on Harmony V1.11 2020/10/01 14:18:55 (permalink)
    4 (1)
    My concern about moving pieces of the Stack is does any of it depend on MHC to properly configure it.
    I maybe better off moving the whole thing to V2.06.
    You would know better since you worked on it.
    Note I am bringing this Post back from the Dead.
    https://www.microchip.com/forums/m992214.aspx
    Short term my boss wants to restart the Stack if we detect it failed.  Apparently TCPIP_STACK_Deinitialize() has a bug that will prevent this.
    Moving to V2.06 Would require major retesting.
    How do you think the V2.06 stack is? (is there any major bugs that were fixed in V3.XX)  I assume you are not talking about bringing the 3.0 stack in to my V1.11 Application.
    #8
    rainad
    Moderator
    • Total Posts : 1412
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: Loss of Network on Harmony V1.11 2020/10/01 15:03:11 (permalink)
    0
    The configuration is taken from a header file: system_config.h.
    The MHC is just a helper to make it easier to put the stuff in that file.
    That file can be modified by hand. Sure, not that easy like using MHC.
     
    I understand the point about moving the whole stack which requires major retesting.
    That's why I was suggesting replacing only DHCP and DNS if you're sure that these ones fail - you'll have to do something about those anyway.
     
    Regarding the TCPIP_STACK_Deinitialize(), yes, may be a valid approach if you know how to detect this situation you're running into.
    There was some memory not de-allocated, if I remember correctly.
    This can be fixed, should not be difficult. I can look into the logs to see what was the issue and you can test the fix.
    Let me know if you want to take this path.
     
     
     
     
     
     
     
    #9
    NKurzman
    A Guy on the Net
    • Total Posts : 18975
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Loss of Network on Harmony V1.11 2020/10/01 23:04:46 (permalink)
    0
    Rainad
    I need to take both paths.
    TCPIP_STACK_Deinitialize() as a temporary patch.
    I know I can manually configure things, the Issue would be fighting MHC if I need to run that.
    MHC also changes the project files too.
    I will need to make a V2.06 Project and see how system_config.h compares
    It is not a static project. we intend to add features during its life.  At least for several years.
     
    I do not think it is the DNS or DHCP DNS failed first, DHCP only when I forced a Renew.
    I will try The Ping and see if the Stack is alive.
    netinfo, heapinfo and  I almost have macinfo done
     
    Moving to V2.06 gets me all the Bugs fixed in all the Drivers.  I just need to deal with the V1 GFX library.
    since my code is based on the Old V1.04 version I should be able to make it work.  I think I can just tell V2.06 I have no GFX library.  
    Do you think the V3.XX is possible to use?
    #10
    rainad
    Moderator
    • Total Posts : 1412
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: Loss of Network on Harmony V1.11 2020/10/02 10:20:11 (permalink)
    0
    Hi NKurzman,
    I think that the easiest way to fix the issue with the TCPIP_STACK_Deinitialize() is to adjust the tcpip_heap_internal.c::_TCPIP_HEAP_Delete() function.
    Change the code sequence:
        if(hDcpt->_heapHead)
        {
        #ifdef TCPIP_STACK_DRAM_DEBUG_ENABLE
            if(hDcpt->_heapAllocatedUnits != 0 || hDcpt->_heapHead != hDcpt->_heapStart || hDcpt->_heapHead->units != hDcpt->_heapUnits)
        #else
            if(hDcpt->_heapAllocatedUnits != 0)
        #endif
            {
                //  deallocating a heap not completely de-allocated or corrupted
               // return (hDcpt->_lastHeapErr = TCPIP_STACK_HEAP_RES_IN_USE);
            }
     
    So that you comment out the error return, and continue with the freeing of the memory anyway.
    That's just supposed to be a notification that some module didn't clean up, but you can ignore that for now.
    Give it a try and let me know if it works. It's easy to test and check if that works.
     
     
    post edited by rainad - 2020/10/02 11:26:34
    #11
    NKurzman
    A Guy on the Net
    • Total Posts : 18975
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Loss of Network on Harmony V1.11 2020/10/08 14:53:46 (permalink)
    0
    I have not looked at the code, but my issue may be related to DIsabling ICMP Ping
    TCPIP_ICMP_User_Disable()  / TCPIP_ICMP_User_Enable()
    When I tried to ping the Units I relized Ping was disabled.  Turning on Ping instantly restored the operation of the Stack. This only affects certain units, in certain places. Does this issue sound familiar? 
    #12
    rainad
    Moderator
    • Total Posts : 1412
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: Loss of Network on Harmony V1.11 2020/10/09 06:16:14 (permalink)
    0
    Unfortunately it doesn't seem familiar.
    So I take it that the board is not dead when the issue occurs, it can still be pinged OK.
    What about the 'heapinfo, netinfo, macinfo', what do they show?
     
     
    #13
    NKurzman
    A Guy on the Net
    • Total Posts : 18975
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Loss of Network on Harmony V1.11 2020/10/09 08:24:34 (permalink)
    0
    I will need to turn off the ping, and install my software that displays that information.
    I can’t do that till Tuesday.
    It is certainly an obscure bug.
    Unit on the second floor of my building appear to work, but not the ones on the first floor.
    They appear to be on different Servers.
    I guess I will also have to do wire shark on one of them to see what happens when it fails.
    I assume this is probably rare since most people don’t bother to turn off ping if they’ve added it to the build.
    #14
    rainad
    Moderator
    • Total Posts : 1412
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: Loss of Network on Harmony V1.11 2020/10/09 14:31:18 (permalink)
    0
    Please let me know when you have the new details and I'll try to help the best I can.
     
    #15
    NKurzman
    A Guy on the Net
    • Total Posts : 18975
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Loss of Network on Harmony V1.11 2020/10/09 14:39:04 (permalink)
    0
    Thank You.  I will try to see what additional information I can find.
    #16
    NKurzman
    A Guy on the Net
    • Total Posts : 18975
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Loss of Network on Harmony V1.11 2020/10/18 22:28:01 (permalink)
    0
    This Issue does not appear to be an issue in the Stack.
    The Issue appears to be the on some networks ICMP Ping is used to determine if the Unit is still present on the network.  If not it appears to be locked out.  enabling Ping on the Unit will immediately restore network access.
    I need to review the Wire Shark logs to confirm this.
    #17
    rainad
    Moderator
    • Total Posts : 1412
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: Loss of Network on Harmony V1.11 2020/10/20 06:16:18 (permalink)
    0
    Interesting find. Good to hear you've figured it out. Let me know once you have the logs.
     
    #18
    NKurzman
    A Guy on the Net
    • Total Posts : 18975
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Loss of Network on Harmony V1.11 2020/10/20 08:29:02 (permalink)
    0
    I have logs, Can I send them to you?
    #19
    rainad
    Moderator
    • Total Posts : 1412
    • Reward points : 0
    • Joined: 2009/05/01 13:39:25
    • Location: 0
    • Status: offline
    Re: Loss of Network on Harmony V1.11 2020/10/20 09:08:29 (permalink)
    0
    Sure. You can use PM or post them here.
     
    #20
    Jump to:
    © 2020 APG vNext Commercial Version 4.5