Recently, I’ve made a new feature (but now removed) to ScreenSherpa where it loads the screen instantly after pressing the connect button. When I mean instant, I mean in about 1 second you’ll see the host screen immediately. This was meant to fix the long handshake times via webrtc. I still have no clue why the ice gathering process on the Mac sometimes take about 15 seconds. Sometimes for some reason the ice gathering fails too.
Anyway, the instant screen share, was done by sending the h264 packets to a nodejs + socket io server and routed to the client. The result was I would say poor and I had to remove it. The fps is horrible, sometimes the fps would go up really high then it will freeze and fps will go down. and what the supposed to be live screen sharing experience is more like a recorded one as the ‘live’ screen the client viewing at the moment is about only 30 seconds ago.
The problem is that socket io because it is based on websockets and TCP, the packets of the host will be in ordered sequence and every packet must be completed before the next one is accepted. With UDP this is not the case, with each packet sent to the client will be received no matter the order it arrives. The h264 packets are very small but they are very high in volume so you will have to decode them all as they arrive. What will look like if you have unordered packets with h264? you get some portions of the frames pixelated or garbled but its negligible and the picture you are seeing will recover eventually but the benefit is that the fps is consistent and the latency is low (it feels as if the remote desktop is real)
So I don’t know the solution yet. UDP is not implemented on browsers and it won’t probably for the reasons listed https://gafferongames.com/post/why_cant_i_send_udp_packets_from_a_browser
Somehow, I probably need to drop packets if the video stream gets late for the viewer
if you have suggestions please comment below.