• AVR Freaks

Hot!How to implement Harmony connectionless UDP server

Author
jtron
Starting Member
  • Total Posts : 48
  • Reward points : 0
  • Status: offline
2020/10/23 15:17:37 (permalink)
4 (1)

How to implement Harmony connectionless UDP server

I want to use Harmony's UDP module to implement a connectionless UDP server running under ThreadX. In my app, there's no such thing as a "connection": different threads are used to send and receive UDP packets in a completely decoupled manner. As in all connectionless servers, there must be a way to determine the source endpoint for every received packet so that the app knows where to reply. Note that due to decoupled send/receive operations, the reply may not be sent until after additional packets have been received from other clients.
 
Problem: I can't find an example of a connectionless UDP server, and it seems that Harmony's UDP server socket only supports connection-based operation. I assume that to send a packet to an unconnected client, I must first set the socket's destination (via TCPIP_UDP_DestinationIPAddressSet and TCPIP_UDP_DestinationPortSet) and then send the packet. However, if the socket receives a packet from a different client before the packet is sent, won't that cause the socket to automatically establish a new connection (which overwrites the socket's destination) and thus corrupt my sendto?
 
Am I missing something here? How can I receive a packet without changing the socket's destination? Also, how can I get the source endpoint of each received packet? Finally, how can I get my receive thread to block until a receive packet is available (essential for multi-threaded apps)? Any help or advice would be greatly appreciated (especially relevant code examples).
#1
rainad
Moderator
  • Total Posts : 1430
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/25 15:45:00 (permalink)
5 (2)
UDP is connectionless, no matter what.
If we think of the terminology client/server, usually the server just waits for another party (the 'client') to send something and then the server socket can reply to the socket that connects to it. Obviously the server can start sending data out - but, as you mentioned, it needs to know where to send that data, therefore you have to set a destination.
Since there is one side to initiate the data transfer, i.e. the 'connection', the normal usage of the terminology is that the client connects first to the server, which listens.
 
The Harmony UDP socket works in packets. So, when a packet is available, the source of that packet is stored by the socket and when the socket user sends data with that socket, data will be automatically sent to the that source,
When you retrieve another packet from the socket, then this will have the updated source for that packet and so on.
Meaning that you can receive 10 different packets from 10 different sources and you can reply to all of them in turn.
If you want to know where you got that packet from, simply call UDP_SocketInfo and you'll see those details.
 
The difference in Harmony between creating a server and client UDP socket is that the server API call takes local parameters (the address and port to listen on) while the client one takes the remote parameters - the address and port to connect to. Other than that there are just some minor differences in the default settings, but those can be changed at run time. So basically there is no working difference between the server and the client sockets, except that you can start sending data immediately with a client socket - since you had to specify the remote address and port when you created the socket.
A server socket usually waits for someone else to talk to it first. But nobody stops a different usage model: set the destination address  and port (a remote bind operation) and then send data. It's just a matter of convenience.
 
Hope this clarifies the usage mode.
 
#2
jtron
Starting Member
  • Total Posts : 48
  • Reward points : 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/26 07:34:47 (permalink)
0
rainad
But nobody stops a different usage model: set the destination address  and port (a remote bind operation) and then send data

 
Should they be set via RemoteBind, or with DestinationIPAddressSet and DestinationPortSet? It's not clear to me what the difference is.
 
rainad
when a packet is available, the source of that packet is stored by the socket and when the socket user sends data with that socket, data will be automatically sent to the that source

 
That's exactly the problem: It treats every rx packet as if related to a connection, and it automatically fills in the socket's destination address/port accordingly. This will step on the sendto address/port set up by my transmit thread, won't it?
 
Also, I can't afford to poll for received packets because that would starve other threads. How can I get my receive thread to block while it waits for an rx packet?



#3
rainad
Moderator
  • Total Posts : 1430
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/26 07:59:34 (permalink)
3 (1)
I understand that you want your socket to receive from whatever host, let's say 192.168.1.100, but to always reply to 192.168.1.1, for example, without specifically calling the DestinationSet before sending data?
You need to call the DestinationSet for now, no other way to do it.
We may add an option in the future, to ignore where it received the packet from and always reply to something else.
For now, I'd just open a socket to 192.168.1.1 and all the data received by the other socket pass to this one, to send it always to 192.168.1.1. That allows you to not always call DestinationSet before sending something - although that's not too heavy either.
 
 
#4
jtron
Starting Member
  • Total Posts : 48
  • Reward points : 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/26 08:55:18 (permalink)
0
rainad
For now, I'd just open a socket to 192.168.1.1 and all the data received by the other socket pass to this one, to send it always to 192.168.1.1

 
That's an interesting idea but I don't think it would work. Suppose I create a server socket at local address 192.168.1.10 to receive packets from all clients -- my "receive only" socket. Upon receiving a packet from 192.168.1.11, I would create a "transmit only" socket and set its destination to 192.168.1.11. Now I have two sockets, both having the same local address. When the next packet arrives, will it be received by the "receive only" or by the "transmit only" socket?
 
What I really need, and what the UDP module should provide, are truly connectionless sockets. The choice of where to send packets should never be automatic and should always be explicitly specified by the app. I just want to receive and send independent packets like Berkeley recv and sendto, but the UDP module doesn't allow this, and I can't use Harmony's Berkeley module because I assume it's based on the UDP module.
#5
rainad
Moderator
  • Total Posts : 1430
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/26 10:00:37 (permalink)
4 (1)
You can have as many sockets as you want on one IP address as long as the ports are different.
So, the server socket should listen to a port known to all its clients, so that all the clients can send data to it.
The 'transmit only' socket will be a client socket which will have a different local port (that's done automatically by the stack) and it will have only the remote host specified, i.e. the address and port of that remote machine server which will wait for this 'data concentrator'.
 
Alternatively you can use BSD. This is the purpose of 'sendto' and it has the destination address and port part of the call, so it should work.
 
P.S. I think that setting the destination for the socket each time before sending data is probably quicker than using 'sendto' because BSD has some extra code in there, does extra checks, etc.
 
post edited by rainad - 2020/10/26 10:50:04
#6
jtron
Starting Member
  • Total Posts : 48
  • Reward points : 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/26 10:53:10 (permalink)
0
rainad
The 'transmit only' socket will be a client socket which will have a different local port (that's done automatically by the stack)

 
That's not feasible because in a UDP server, the local port specifies the associated service. If the server replies using a different local port then the client cannot determine the associated service.
 
rainad

Alternatively you can use BSD. This is the purpose of 'sendto' and it has the destination address and port part of the call, so it should work.

 
I agree that this should work but I'm concerned because Harmony BSD seems to be based on the UDP module, which contains this warning in the source code:
 

//  - A socket cannot be used concurrently from multiple threads!
//    It's ok to pass a socket from one thread to another as long as
//    there's is no access from more than one thread at a time

 
Can you please confirm that Harmony BSD allows access a socket to be used concurrently by different threads?
 
Also, in Harmony BSD, how do I get my receive thread to block while it waits for the next rx packet?
#7
rainad
Moderator
  • Total Posts : 1430
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/26 11:31:02 (permalink)
5 (1)
jtron
 That's not feasible because in a UDP server, the local port specifies the associated service. If the server replies using a different local port then the client cannot determine the associated service.
 

Sorry, but then I definitely do not understand what you're trying to achieve - not sure what the 'client to determine the associated service' is. What I know is that a server listens on a well known port and that's how the clients connect to it, using that listening port. If the client doesn't know where to send its data, then I do not know what can be done.


Regarding the multi-threading, the same is true for the regular UDP sockets: a socket needs to be used from one single thread.
Otherwise you end up adding a lot of synchronization code just for this, and for no good reason.
Just create another socket for a different thread.
 
#8
BillP
Super Member
  • Total Posts : 436
  • Reward points : 0
  • Joined: 2014/09/28 07:53:35
  • Location: CA
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/26 14:00:34 (permalink)
0
@jtron.  What about multicasting?  The server writes everything to the multicast address and it gets sent to each joined address.  The client can write to the multicast address and include some data that tells which service it wants to use.  Would this work for your app?

New book: Learn Harmony v3.  Go to amazon.com/books, then search for Bill Petrick
#9
jtron
Starting Member
  • Total Posts : 48
  • Reward points : 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/26 14:41:14 (permalink)
0
BillP
@jtron. What about multicasting? The server writes everything to the multicast address and it gets sent to each joined address. The client can write to the multicast address and include some data that tells which service it wants to use.

 
Thanks for the great idea, Bill! Unfortunately I can't do that because this project involves porting a new processor onto a legacy device and requires that I preserve the existing protocol and client-side binaries.
#10
jtron
Starting Member
  • Total Posts : 48
  • Reward points : 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/26 15:34:00 (permalink)
0
rainad
I definitely do not understand what you're trying to achieve - not sure what the 'client to determine the associated service' is. What I know is that a server listens on a well known port and that's how the clients connect to it, using that listening port

 
In connectionless UDP, a client doesn't "connect" to a server; it simply sends packets to the server's well-known port and, upon receiving a packet, uses the sender's port to identify the server. This is essential because every UDP server provides unique services. For example, suppose an app has two UDP servers: one at port 10000 for analog I/O and another at port 10001 for a UART bridge. To read analog inputs, the client sends a request to and receives a reply from port 10000. To fetch received UART chars, it sends a request to and receives a reply from port 10001. If the server replied using a local port which is not its own well-known port, the client couldn't determine which server the packet came or what type of data the packet contains (analog data vs. UART chars). This is why a UDP server must always reply with its own well-known port -- and why a UDP server can only have one socket.
 
rainad
Regarding the multi-threading, the same is true for the regular UDP sockets: a socket needs to be used from one single thread. Otherwise you end up adding a lot of synchronization code just for this, and for no good reason.

 
It seems that I do have to use synchronization code, because (1) a UDP server can only have one socket, and (2) Harmony's stack don't doesn't allow a socket to be used by two threads at the same time. To resolve this, each server must have a single UDP server socket that's shared by the server's rx and tx threads and protected by a resource lock to ensure only one thread can use the socket at a time.
 
I still don't know how to block the receiver thread while it waits for an rx packet. Any ideas would be appreciated.
post edited by jtron - 2020/10/26 15:36:10
#11
rainad
Moderator
  • Total Posts : 1430
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/27 07:14:36 (permalink)
5 (1)
jtron
To read analog inputs, the client sends a request to and receives a reply from port 10000. To fetch received UART chars, it sends a request to and receives a reply from port 10001. If the server replied using a local port which is not its own well-known port, the client couldn't determine which server the packet came or what type of data the packet contains (analog data vs. UART chars). This is why a UDP server must always reply with its own well-known port -- and why a UDP server can only have one socket.



That's exactly what happens, nobody said that the UDP server changes its local port on which it's listening on. That remains constant, of course, and it's used in the response to all clients.
So, you can have a UDP server listening on port 10000 and another on port 10001, no interference.
However, if the UDP socket listening on port 10000 receives requests from multiple clients it will try to respond to the client that requested the data, not to another destination address. The remote address and port will be different, not the local well-known server port or address.
If you want that when the server received a request from client 1 to respond to client 2, then you have to set the destination yourself. But the server local address and port will be the same, that won't change.
 
Anyway, we do not seem to make much progress. Better than discussing solutions and semantics, maybe you can let us know what the problem is, what needs to be achieved, and then maybe we can come up with a helpful idea.
 
 
 
#12
jtron
Starting Member
  • Total Posts : 48
  • Reward points : 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/27 14:29:20 (permalink)
0
rainad
if the UDP socket listening on port 10000 receives requests from multiple clients it will try to respond to the client that requested the data

 
I think I know what you mean, but it's not said quite right. It's the job of the server app -- not the socket -- to respond to client requests. A socket's job is to forward received packets to the server app, and to accept outgoing data from the server app and send it wherever the app tells it to. It's inappropriate for a socket to make any assumptions about where to send packets, and doing so can make it impossible for the server app to do its job, yet this is exactly what Harmony's UDP server sockets seem do. At least, that's what's indicated by the documentation and all of the examples I've seen.
 
rainad
maybe you can let us know ... what needs to be achieved

 
In my ThreadX app, a "receiver" thread receives packets from a UDP server socket, and the receiver and other threads transmit on the same socket. This is necessary to facilitate deferred command processing, retry management, and other required functions. The original firmware (which I'm porting to ATSAME53J18A) does this with a standard BSD socket which, unlike Harmony, supports simultaneous transmit/receive and allows concurrent transmit on multiple threads (by internally serializing "concurrent" transmits). I don't want get bogged down in details, so here's a simple example that shows what I must be able to do with a single UDP server socket -- in this case, deferred command processing:
begin ReceiverThread
  create udp server socket (public)
  create command queue (public)
  create CommandThread
  loop forever:
    block until rx packet available in server socket
    read packet from server socket and get command.message (data payload) and command.endpoint (sender's address/port)
    if command queue full
      send "queue full" message to command.endpoint via server socket
    else
      add command to command queue
  end loop
end ReceiverThread
 
begin CommandThread
  loop forever:
    block until command available in command queue
    read command from command queue
    perform command.message action
    send reply to command.endpoint via server socket
  end loop   
end CommandThread

#13
rainad
Moderator
  • Total Posts : 1430
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/28 15:40:20 (permalink)
5 (1)
jtron
 
 

 
 
 
 read packet from server socket and get command.message (data payload) and command.endpoint (sender's address/port)




If I understand correctly, there are multiple clients (command endpoints) sending data to the server socket.
Since the reply to the client is usually delayed and done from another queue/thread, there is no way that the server socket can know by itself where to reply. So, the application must set the client/endpoint destination before sending the response. I assume it was the same in the previous app, and sendto() must have been used, specifying the destination address + port, as indicated by the command in the queue. So no difference here.
 
To block until rx data is available, use the TCPIP_UDP_SignalHandlerRegister - with TCPIP_UDP_SIGNAL_RX_DATA signal. The RX thread could be blocked until it receives this signal, which it wakes the thread up.
 
The real problem is with the Receiver and Command threads synchronization.
Since the 1st thread deals with RX only and the 2nd with TX, there shouldn't be any issue.
However, the RX thread may need to transmit something too - and that's a problem.
If the client has a timeout and knows that if it hasn't received an ack it needs to re-transmit, then the "queue full" message could be eliminated. I guess the client will have to retransmit anyway if it received it.
 
Another solution - and this is how I'd do it, the RX thread should have a queue too to communicate with the command thread. When it receives something, it passes the message to the command queue. The command thread executes it and then passes the reply back to the RX thread. This now transmits the result to the client.
This way, there's only a single thread messing with the socket - end of story.
 
 
 
 
post edited by rainad - 2020/10/28 15:59:29
#14
jtron
Starting Member
  • Total Posts : 48
  • Reward points : 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/29 12:29:00 (permalink)
0
@rainad: Thanks for your ideas (and patience). Since Harmony sockets aren't thread-safe, it makes sense to have only one thread interact with the socket. I envision something like this:
 
The UDP server socket is handled exclusively by a SocketManager thread (except for rx signal callbacks), which processes service requests from SocketRequestQueue. There are two request types: RxRequest (indicates rx packet available in socket) and TxRequest (transmit request). Every request includes the associated client endpoint (address and port). Each TxRequest includes tx data; a TxRequest is processed by sending tx data to the associated client. An RxRequest is processed by moving it to CommandQueue; if this fails because CommandQueue is full, SocketManager sends a "queue full" message to associated client.
 
The following code shows the general idea. For clarity, I've omitted error checks, buffer over/underflow checks, and other irrelevant details. Do you see any problems with how I'm using Harmony UDP?
 
static void UdpSend(UDP_SOCKET s, SOCKET_REQUEST *item)
{
    TCPIP_UDP_DestinationIPAddressSet(s, IP_ADDRESS_TYPE_IPV4, item->addr);
    TCPIP_UDP_DestinationPortSet(s, item->port);
    TCPIP_UDP_ArrayPut(s, item->data, item->len);
    TCPIP_UDP_Flush(s);
}
 
void SocketManagerThread()
{
    UDP_SOCKET s = TCPIP_UDP_ServerOpen(IP_ADDRESS_TYPE_IPV4, SVR_PORT, NULL);          // create and bind server socket
    TCPIP_UDP_SignalHandlerRegister(s, TCPIP_UDP_SIGNAL_RX_DATA, RxSignalHandler, 0);   // register rx callback
    while (1) {  // Repeat forever ...
   
        SOCKET_REQUEST *item = ReadSocketRequestQueue(WAIT_FOREVER);   // block until socket i/o requested
       
        if (item->type == TX_REQUEST) {             // process TX_REQUEST:
            UdpSend(s, item);                           // send tx data to client
        }
        else {                                      // process RX_REQUEST:
       
            UDP_SOCKET_INFO info;
           
            item->len = TCPIP_UDP_GetIsReady(s);        // advance to next packet in socket's rx queue
            TCPIP_UDP_SocketInfoGet(s, &info);          // get client endpoint
            item->addr = info.remoteIPaddress;
            item->port = info.remotePort;
           
            if (isCommandQueueFull()) {                 // if no room in command queue
                TCPIP_UDP_Discard(s);                   //   drop rx data
                item->data = QueueFullMsg;              //   send "queue full" to client
                item->len = sizeof(QueueFullMsg);
                UdpSend(s, item);
                FreeItemToPool(item);
            }
            else {                                      // else
                TCPIP_UDP_ArrayGet(s, item->data, item->len);  //   copy rx data to item
                WriteCommandQueue(item);                //   send item to command processor thread
            }
        }
    }
}
 
void RxSignalHandler() // Notes: (1) Not executed in SocketManagerThread context; (2) Harmony stack requires this to be fast.
{
    SOCKET_REQUEST *item = AllocItemFromPool();
    item->type = RX_REQUEST;
    WriteSocketRequestQueue(item);
}

#15
rainad
Moderator
  • Total Posts : 1430
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/30 09:35:17 (permalink)
0
I think everything looks in good order.
 
I was thinking that the RxSignalHandler just releases a semaphore/sets an event flag to wake up the SocketManagerThread, which could check that there's messages pending directly anyway, no need for a RX_REQUEST item in the queue. But these are just minor details, of course and probably the 'AllocItemFromPool' is very fast anyway.
 
"Also, since the execution of the commands is probably completely asynchronous to the receive process, I think that the SocketManagerThread should wake up when there's something to transmit, not wait to receive something. So that it replies, if there's a reply pending, even if no client requested something else."
 
P.S. Ignore the last paragraph,  I think that the code already takes care of that. that queue carries both TX and RX events.
post edited by rainad - 2020/10/30 09:44:16
#16
jtron
Starting Member
  • Total Posts : 48
  • Reward points : 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/30 11:01:08 (permalink)
0
@rainad: Thanks for the code review and helpful info.
 
In case anyone is wondering, the reason for having a common SocketRequestQueue for tx and rx requests is (1) to serialize socket I/O (required because Harmony sockets are not thread-safe), and (2) to enable SocketManagerThread to block while it waits for I/O requests (required for event-driven operation). The latter eliminates the need for event objects and "WaitForMultipleEvents" on tx and rx requests. In my case, rx must have priority over tx, so RxSignalHandler will add rx requests at the head of the queue and other threads will add tx requests at the tail.
#17
rainad
Moderator
  • Total Posts : 1430
  • Reward points : 0
  • Joined: 2009/05/01 13:39:25
  • Location: 0
  • Status: offline
Re: How to implement Harmony connectionless UDP server 2020/10/30 16:02:06 (permalink)
0
Sure, glad to be of help, if possible.
Let me know if there's any issue.
But the approach seems sound.
 
#18
Jump to:
© 2020 APG vNext Commercial Version 4.5