FMS, Live Streaming and Sync Between Publisher and Subscribers

Here I would like to discuss the FMS, Live Streaming and Sync issues between publisher and subscribers. Also I would discuss about possible solutions 🙂

The problem: Let’s say we are publishing a live stream using FMLE (Flash Media Live Encoder) or our custom publisher to FMS (Flash Media Server). There are 5 subscribers who are watching this live stream (given that those subscribers connected to stream after the stream has been already published). Now NetStream.time will not give absolute stream time for any of the subscriber connected to live stream because it gives time since the playback of live stream is started by subscriber. Now let’s say I (as a publisher) want to take certain action (such as displaying an image, or some message, or slide) on subscribers on specific point of time (in terms of viewing the live stream). Let’s say from publisher side, when I see (or speak) something (a tortoise in scene!) on screen, I want to show another picture of that same thing to all subscribers (dancing tortoises!) exactly at the same time when each one of them see tortoise on their live stream.

SharedObject can not be used because that will not provide any kind of sync (in terms of live streaming) because they are asynchronous and live streams may get delayed due to slow bandwidths on subscribers.

Even we can not directly call a function on FMS which would in turn call functions to all subscribers because that process would be instantaneous (compared to delay in live stream) and if there are number of subscribers this process may hang FMS (and publisher) application for a fraction of second.

The Solutions:

1) Somehow give all subscribers the absolute stream time. This would be an ideal solution but not easy. From FMLE, I managed to find a solutions and that is timecode. Let’s say if the publishing video frame rate is 15 fps then FMLE would encode a keyframe at each group of 15 frames (given that timecode interval is same as frame rate of video). Now the timecode is embedded as system time and date which we need to take care of. The condition here is the video must be recorded on server so that we can know the start time of publishing. On subscribers we get onFI() event at each 1 second which would give you current date and time. From this current time subtract the initial time (which is found by playing recorded stream) and that would give you absolute live stream time 🙂

2) Another way is kind of events instead of focusing on time but yet it keeps solid track of sync. From publisher side, call NetStream.send() function along with necessary parameters. This will actually embed metadata on current frame being encoded. When subscribers get this frame, they get the embedded event and they can take the necessary action by analyzing the data given in that event. Usually I would prefer to use onMetaData event for this stuff 🙂

Phue! It’s been late night now (1:43 AM already) and I am going for sleep now. If any one have any questions, post me a comment and I will be back there right away 🙂

Thanks, Naresh

Live Streaming, FMS and Absolute Stream Time

Think of a live stream is being published to FMS and all subscribers are watching that live stream. Now one of the subscriber is late in joining this live stream but still he does that successfully and watches the live stream from the point when he joined it. Everything looks perfect!

But wait, one thing is not perfect definitely – the time of the stream been published. Yes, the subscribers does not see the absolute stream time from start of its publish. So what does that mean? They are not in sync of each other. Let’s say I want to notify all subscribers at certain point of time in video. Now I would be watching something at time t and I will send message to all subscribers. What I want is : all subscribers should see this message at exact point of time when they see what I am watching in video. But as there is no time sync, subscribers won’t know that at what point of time the message should be displayed.

As work around, what I did is I passed timecode from FMLE. That means that at each frame of video, FMLE would encode system time in the frame. All subscribers would get onFI() event with that time code at each second (given that the timecode rate and frame rate of video is same at FMLE). A subscriber would have to save the first timecode as the start point of time. On next onFI() event they can just subtract the start timecode from the current timecode and that would give it a absolute time of stream!!

Once absolute time of stream is found, you can synchronize the message around 🙂 I did not written down any code here but if any one wants the code, let me know – I can provide it.