• AVR Freaks

Hot!Best advice for making a USB to serial converter

Page: 123 > Showing page 1 of 3
Author
swmcl
Super Member
  • Total Posts : 271
  • Reward points : 0
  • Joined: 2014/05/10 13:54:42
  • Location: Queensland
  • Status: offline
2019/10/16 02:11:15 (permalink)
0

Best advice for making a USB to serial converter

Hi,
The behavior of the MCP2200 in that it chops up an incoming byte stream into chunks that are a maximum of 32 bytes of data in each packet means I will need to take on the task of creating my own USB to serial converter.
I think the current production device called an ATMEGA8U2 or 16U2 is about the least pin-count 5V device I can see at the moment.

Can I ask for advice on this journey please from whatever direction.

I'm wanting a 64-bytes of data in a single USB packet. 
I need full speed USB 2.0 performance. 
I also need hand-solderable chips.  The TQFP-32 devices are possible.

If you have a previous project you could share (even for money) then PM me !

Rgds,
Steve
#1

55 Replies Related Threads

    NorthGuy
    Super Member
    • Total Posts : 5754
    • Reward points : 0
    • Joined: 2014/02/23 14:23:23
    • Location: Northern Canada
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 07:06:52 (permalink)
    +1 (1)
    You're restricted to the set of supported USB classes which your OS has drivers for. If you want full bandwidth and 64-bit packet transfer, you can use custom USB class and write your own driver. Instead of the driver, you can also use libusb, but it is not kernel-side as drivers would be, so don't expect to get top performance.
     
    If you use CDC (USB serial emulator class), you will get more characters per packet if you use large writes (as opposed to writing byte-by-byte). Think about it. You write a single character. What is the driver supposed to do - send it out quickly (then you complain about bad throughput) or wait until you give it more characters (then you complain about bad latency)? The only solution is to give it lots of characters at once.
    #2
    Antipodean
    Super Member
    • Total Posts : 1767
    • Reward points : 0
    • Joined: 2008/12/09 10:19:08
    • Location: Didcot, United Kingdom
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 07:41:55 (permalink)
    +1 (1)
    swmcl
    Hi,
    The behavior of the MCP2200 in that it chops up an incoming byte stream into chunks that are a maximum of 32 bytes of data in each packet means I will need to take on the task of creating my own USB to serial converter.

     
    What baud rate is the serial side running at? 
     
    swmcl
    I'm wanting a 64-bytes of data in a single USB packet. 
    I need full speed USB 2.0 performance. 

     
    I cannot understand your chasing 64 bytes per USB transaction. The converter gets polled by the host every millisecond and will transfer whatever it has in its buffer at that time. I did a project with a PIC24 and the MLA where I was transferring the maximum possible amount per transfer every ms, and the limitation there was the windows host, if the screen saver kicked in it slowed down the USB transfer something shocking, and that was with a dual core processor at the windows end.
     
    Personally I would say you should use a PIC18F14K50 with the MLA that Microchip provide. If you cannot get the debug header for the 14k50 then do the development on a chip where you can debug and then port the code down.
     

    Do not use my alias in your message body when replying, your message will disappear ...

    Alan
    #3
    swmcl
    Super Member
    • Total Posts : 271
    • Reward points : 0
    • Joined: 2014/05/10 13:54:42
    • Location: Queensland
    • Status: offline
    Re: Best advice for making a USB to serial converter 2019/10/16 12:45:16 (permalink)
    0
    Thanks guys,
     
    Baud rate on the serial would be 38k4 at least. This figure is not set in stone yet because I've never been up to that speed !
     
    Frame size varies between 16 bytes and 48 bytes.  Would much prefer it if the frame wasn't broken up by the USB packets for processing on the other side.  My frames are indeterminate with respect to time.  They can arrive at any time.

    I do wonder if USB is the best way to transfer to the computer in this application.  A serial card seems so much better but computers don't have serial cards as standard anymore ...
    #4
    ric
    Super Member
    • Total Posts : 24278
    • Reward points : 0
    • Joined: 2003/11/07 12:41:26
    • Location: Australia, Melbourne
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 12:51:28 (permalink)
    +1 (1)
    38k4 is 3840 bytes per second (allowing for one start, one stop, and no parity).
    That is 3.84 bytes per millisecond, so CDC could easily handle this doing alternate packets of 3 or 4 bytes.
    I assume it is waiting for 32 bytes, then sending a 32 byte packet, which will happen about 120 times a second, or every eight and a bit milliseconds.
    So, CDC has plenty of throughput, it's just the irregular timing due to packetising that seems to be causing you concern.
     

    I also post at: PicForum
    Links to useful PIC information: http://picforum.ric323.co...opic.php?f=59&t=15
    NEW USERS: Posting images, links and code - workaround for restrictions.
    To get a useful answer, always state which PIC you are using!
    #5
    NorthGuy
    Super Member
    • Total Posts : 5754
    • Reward points : 0
    • Joined: 2014/02/23 14:23:23
    • Location: Northern Canada
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 12:52:36 (permalink)
    +1 (1)
    swmcl
    Baud rate on the serial would be 38k4 at least. This figure is not set in stone yet because I've never been up to that speed !



    This is 3.8kBytes/sec, roughly 0.3% of full speed USB bandwidth. Why do you want full USB bandwidth?
    #6
    NKurzman
    A Guy on the Net
    • Total Posts : 17939
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 12:59:15 (permalink)
    +1 (1)
    You are focusing on the wrong area.  You need to figure out why your PC program expects all the bytes to arrive at once.  And what can you do to deal with the fact it does not.
    #7
    mpgmike
    Super Member
    • Total Posts : 324
    • Reward points : 0
    • Joined: 2014/01/23 17:27:06
    • Location: NJ
    • Status: offline
    Re: Best advice for making a USB to serial converter 2019/10/16 14:20:47 (permalink)
    0
    On another note (take-2 as I mentioned a user name that cancelled out my post), Post #3 suggested using a PIC18F14K50.  It only has a 32-byte buffer.  The PIC18F2x_4xK50 varieties have the 64-byte buffer you specified.
     

    I don't need the world to know my name, but I want to live a life so all my great-grandchildren proudly remember me.
    #8
    swmcl
    Super Member
    • Total Posts : 271
    • Reward points : 0
    • Joined: 2014/05/10 13:54:42
    • Location: Queensland
    • Status: offline
    Re: Best advice for making a USB to serial converter 2019/10/16 15:09:40 (permalink)
    0
    In a discussion with technical help at FTDI concerning the way the FT232BL operates I found out that it has a timeout that is programmable after which the buffered serial data is packetised onto the USB.  My 'frame' has within it a frame-length byte so it is possible to tell whether a frame is broken.

    I am not doing the pc side of things.  I'm actually paying for others to do the programming.  My task is to get something running with the hardware.

    Here's a later email from FTDI:
     

    Hi Steve,

    The baud rate is set by the PC application using either the COM port API of your chosen language (if using VCP drivers) or via our FT_SetBaud call in our DLL
    (if using our D2xx). The baud rate is set on each time you open the port and has no setting in the EEPROM. FT_Prog is only for changing EEPROM settings
    but any others like baud rate, handshaking mode etc. are done by the application after opening the port. If you use a terminal program, then this
    usually has a drop-down menu which will set the rate in the chip. There is no default rate as such as we always recommend to set it as part of your
    initialising routine.

    The description string would be via FT_Prog as you mentioned. I believe there may be a few third-party tools for Linux similar to FT_Prog but these
    were developed completely independently of us and we have not tested or reviewed them. It's a one-time operation for each device and so you could
    possibly run it on a Windows machine for production test even if they will be used on Linux in the main application. The FT232R or FT231X may be better
    choices as they have internal EEPROM.

    Our devices don’t have specific detection for the gaps between messages. In most cases it is recommended to receive the data and allow it to buffer up for
    most efficiency and to parse the data once it arrives on the PC to separate the messages from each other. If a protocol is used on the serial side which uses
    only gaps in the messages, the variable delays over USB protocol scheduling and the OS itself may cause the data to be mixed and so in these cases we
    recommend a small MCU to do the serial bus interaction and then passing the messages back to the PC via the FTDI chip with a small message header/structure.
    This allows you to ensure the accurate timing on the serial bus if inter-byte spacing is the only way to separate messages. Our FT930 MCUs have USB peripheral
    interfaces and so you could combine both functions into the same chip but would need to write firmware.

    It is possible to set the timer to 2ms to minimise delay of any residual bytes or to have your external device change state of the handshake line such as
    DSR after each packet/message and this will trigger a send. However, we still recommend to parse the data on the PC application rather than partition it
    before sending back to the PC over USB.

    Thanks,

     
    In my experience so far, I've had to get a serial port monitor written to be able to accurately see the data being sent
    to the port. Other monitors showed a breakup of the incoming data.

     
    My weak understanding at this stage is to make a USB-serial converter that basically packages the buffered frame for USB about 1.5 bytes after the last byte of the frame has been received.  This would be regardless of the length of the frame.  If the converter were capable of packaging up to a 64-byte frame then only one USB packet is sent sometime soon after the frame has been received by the converter.  This is very different to the way the converters operate at the moment.

    I am not understanding how USB functions from the Linux computer side of things.  I have purchased Jan Axelson's 'USB Complete' but I need an educational source that show block diagrams and clearly explains concepts rather than spewing forth acronyms and jargon.  Getting good educational material is hard.

    Rgds,
    #9
    NorthGuy
    Super Member
    • Total Posts : 5754
    • Reward points : 0
    • Joined: 2014/02/23 14:23:23
    • Location: Northern Canada
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 15:43:39 (permalink)
    +3 (3)
    If you want frames - use HID instead of CDC. It is based on frames (called reports). It has a limit of one frame per 1 ms (1000 frames/s), but this is still 25 times more traffic than you need.
     
    #10
    ric
    Super Member
    • Total Posts : 24278
    • Reward points : 0
    • Joined: 2003/11/07 12:41:26
    • Location: Australia, Melbourne
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 15:44:26 (permalink)
    +1 (1)
    swmcl
    I am not doing the pc side of things.  I'm actually paying for others to do the programming.  My task is to get something running with the hardware.
    ...

    Why not just pay them to do their job properly?
    It sounds like they have a too crude method of detecting frame boundaries. If your data is packetised in a way that can be simply detected, why not just get them to reassemble the packets correctly at their end?
    As multiple people have pointed out, throughput is no problem, it's just that the data is going to be arriving in packets that are not synchronised with YOUR packets. That's just how CDC works normally.
     

    I also post at: PicForum
    Links to useful PIC information: http://picforum.ric323.co...opic.php?f=59&t=15
    NEW USERS: Posting images, links and code - workaround for restrictions.
    To get a useful answer, always state which PIC you are using!
    #11
    NKurzman
    A Guy on the Net
    • Total Posts : 17939
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 15:54:51 (permalink)
    +1 (1)
    mpgmike
    On another note (take-2 as I mentioned a user name that cancelled out my post), Post #3 suggested using a PIC18F14K50.  It only has a 32-byte buffer.  The PIC18F2x_4xK50 varieties have the 64-byte buffer you specified.

     
    How Big a Buffer does the Standard CDC Driver use?
    How Big a Buffer does the MCP2200 Driver use?
    How Big a Buffer does the FTDI's  Driver use? 
    #12
    NorthGuy
    Super Member
    • Total Posts : 5754
    • Reward points : 0
    • Joined: 2014/02/23 14:23:23
    • Location: Northern Canada
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 15:56:44 (permalink)
    +3 (3)
    ric
    That's just how CDC works normally.



    I completely agree. That's the important thing. CDC is designed to use USB packets to carry a byte stream. Byte stream - it's all you've got. If you want packets on top of CDC, you need some sort of protocol which creates packets on top of the byte stream.
    #13
    swmcl
    Super Member
    • Total Posts : 271
    • Reward points : 0
    • Joined: 2014/05/10 13:54:42
    • Location: Queensland
    • Status: offline
    Re: Best advice for making a USB to serial converter 2019/10/16 16:16:53 (permalink)
    0
    Guys,

    There are 2 aspects.  The broken frames at the pc and the timeout issue at the converter.

    My understanding of the MCP2200 comes from my case with the Microchip Technical people who created a 70-byte frame and sent it to a pc.  They had a USB analyser on the line to see what was happening.  They told me it was sending 32 then 32 then 6 bytes to the pc.  What this means is the 64 byte buffer in the MCP2200 is actually 2 x 32 byte buffers.  When one gets filled, the USB packet is created with those 32 bytes.  This is how the MCP2200 works in that it waits for the 32 bytes to fill before a timeout triggers a packaging event.  So the last 6 bytes in this case would have waited in the buffer of the device for at least 26-bytes worth of time (I assume).

    This timeout is configurable with a FT232BL BUT ... it is still triggered from the start of a frame being sent to the device.  If I have a datastream that has a variable frame length from 16 bytes to 48 bytes with the vast majority of those frames being 16 bytes or thereabouts, then setting a timeout for 48 or 64 bytes will be introducing a significant delay in proceedings.   I would like to attain as close to RTOS as I can as the information coming in is being used to make significant decisions.

    Packaging for the USB transfer a short time after no more bytes have been received by the converter is a best-case scenario in that it will be sent on the USB bus fairly soon after being received on the serial bus.  There is no significant waiting.

    Converters that have even a configurable timeout are best suited to larger bulk data transfers, I don't see USB as being setup with my kind of situation in mind. 

    The 64-byte package is some sort of a USB rule or 'accepted norm' it seems and is conveniently bigger than my largest frame.  To have a converter with a large enough buffer to not break up 64 bytes into smaller chunks like the MCP2200 is important (hence the interest in the FT232BL).  It means if I could get it to package things as described then there would be no special considerations at the pc end to recover divided frames.  I don't want to know what would happen if there were more than one of my devices connected into a USB hub ...

    As you could see also from the discussions with FTDI, the baud rate downstream of the FT232BL needs to be set every time the damn thing is plugged in to the pc.  This I didn't know before buying a number of them ...
     
    #14
    ric
    Super Member
    • Total Posts : 24278
    • Reward points : 0
    • Joined: 2003/11/07 12:41:26
    • Location: Australia, Melbourne
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 16:28:41 (permalink)
    +2 (2)
    swmcl
    ...
    This timeout is configurable with a FT232BL BUT ... it is still triggered from the start of a frame being sent to the device.  If I have a datastream that has a variable frame length from 16 bytes to 48 bytes with the vast majority of those frames being 16 bytes or thereabouts, then setting a timeout for 48 or 64 bytes will be introducing a significant delay in proceedings.

    From the FTDI post, the delay in in miliseconds, not bytes.
    What do you call a "significant delay" ?
     

    Packaging for the USB transfer a short time after no more bytes have been received by the converter is a best-case scenario in that it will be sent on the USB bus fairly soon after being received on the serial bus.  There is no significant waiting.

    Isn't that what a 2ms timeout will do?
     

    Converters that have even a configurable timeout are best suited to larger bulk data transfers, I don't see USB as being setup with my kind of situation in mind. 

    You still have not accurately described what your situation is.
     
    Are your PC programmers expecting to receive the whole packet with a single Windows call?
    You couldn't even expect that with a real serial port.
    If your programmers cannot handle breaking out your packets from a continuous stream, they're not worth what you're paying them.
     

    As you could see also from the discussions with FTDI, the baud rate downstream of the FT232BL needs to be set every time the damn thing is plugged in to the pc.  This I didn't know before buying a number of them ...

    So?
    That's how serial ports have always worked. I think you are misunderstanding something here.
     
    This whole topic seems to be a whole lot of misunderstandings piled onto misconceptions, and the attempted solutions are attacking the wrong problem.
     

    I also post at: PicForum
    Links to useful PIC information: http://picforum.ric323.co...opic.php?f=59&t=15
    NEW USERS: Posting images, links and code - workaround for restrictions.
    To get a useful answer, always state which PIC you are using!
    #15
    jtemples
    عُضْوٌ جَدِيد
    • Total Posts : 11407
    • Reward points : 0
    • Joined: 2004/02/13 12:31:19
    • Location: Southern California
    • Status: offline
    Re: Best advice for making a USB to serial converter 2019/10/16 16:34:36 (permalink)
    +3 (3)
    My 'frame' has within it a frame-length byte

     
    So tell your PC guys to use it.  With a length field, there is no reason to care about packets being split up, or about any aspect of what's happening on the USB.  You get the length byte, then you wait for that many more bytes to come in, then you process the packet.
    #16
    NKurzman
    A Guy on the Net
    • Total Posts : 17939
    • Reward points : 0
    • Joined: 2008/01/16 19:33:48
    • Location: 0
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 17:10:13 (permalink)
    +1 (1)
     Are the Packets are time delimited, with no internal structure?  No return Ack?  how long is the time delimiter?
    You may have painted your self into a Corner.  Even with a Real H/W serial port, windows can create time gaps.  Bytes get copies from the H/W to the S/W buffers in chunks.  Like the USB does.
    #17
    NorthGuy
    Super Member
    • Total Posts : 5754
    • Reward points : 0
    • Joined: 2014/02/23 14:23:23
    • Location: Northern Canada
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/16 17:42:27 (permalink)
    +1 (1)
    Try to put things in perspective. You spend 4 ms to send your 16-byte frame. During this time, USB can send almost 100 64-byte packets. So, why worry about bytes being split, scarce distribution of your bytes in the frames and such.
     
    Why not to eliminate UART instead - instead of external chip, get an MCU with built-in USB - you'll save yourself the whole 4 ms.
     
    Also make sure you don't slow it down with JavaPythons on the PC side.
    #18
    crosland
    Super Member
    • Total Posts : 1673
    • Reward points : 0
    • Joined: 2005/05/10 10:55:05
    • Location: Warks, UK
    • Status: offline
    Re: Best advice for making a USB to serial converter 2019/10/17 05:13:11 (permalink)
    +1 (1)
    swmcl
    This timeout is configurable with a FT232BL BUT ... it is still triggered from the start of a frame being sent to the device.  If I have a datastream that has a variable frame length from 16 bytes to 48 bytes with the vast majority of those frames being 16 bytes or thereabouts, then setting a timeout for 48 or 64 bytes will be introducing a significant delay in proceedings.   I would like to attain as close to RTOS as I can as the information coming in is being used to make significant decisions.



    You need to remember that USB is polled and the best you will get is one frame every millisecond. Write your firmware to reply with whatever is in the buffer regardless. If it was more than 64 bytes it will have to wait for the next 1ms time slot to complete in any case. The FTDI app notes explain all this quite well.
     
    You need to carefully consider
     
    How many bytes per frame are arriving/sending on the serial side?
     
    How many frames per second?
     
    What latency can you cope with?
     
    That should allow you to answer: What data rate do you actually need?
     
    Is suspect the answer to: Do you really need USB full speed performance? is no.
    #19
    Antipodean
    Super Member
    • Total Posts : 1767
    • Reward points : 0
    • Joined: 2008/12/09 10:19:08
    • Location: Didcot, United Kingdom
    • Status: online
    Re: Best advice for making a USB to serial converter 2019/10/17 05:24:20 (permalink)
    +1 (1)
    crosland
    swmcl
    This timeout is configurable with a FT232BL BUT ... it is still triggered from the start of a frame being sent to the device.  If I have a datastream that has a variable frame length from 16 bytes to 48 bytes with the vast majority of those frames being 16 bytes or thereabouts, then setting a timeout for 48 or 64 bytes will be introducing a significant delay in proceedings.   I would like to attain as close to RTOS as I can as the information coming in is being used to make significant decisions.



    You need to remember that USB is polled and the best you will get is one frame every millisecond. Write your firmware to reply with whatever is in the buffer regardless. If it was more than 64 bytes it will have to wait for the next 1ms time slot to complete in any case. 



    Umm, when I was playing with the MLA code some years ago you could send up to 1.9MB/ms as it packages up multiple 64 byte frames into one transfer, then the hardware unpacks it at the other end back into 64 byte packets.
     
    See "USB Complete"
    https://www.amazon.co.uk/USB-Complete-Guides-Jan-Axelson/dp/1931448280/ref=sr_1_5?keywords=usb+complete&qid=1571315034&sr=8-5
     
     

    Do not use my alias in your message body when replying, your message will disappear ...

    Alan
    #20
    Page: 123 > Showing page 1 of 3
    Jump to:
    © 2019 APG vNext Commercial Version 4.5