FMS, Live Streaming and Sync Between Publisher and Subscribers

Here I would like to discuss the FMS, Live Streaming and Sync issues between publisher and subscribers. Also I would discuss about possible solutions 🙂

The problem: Let’s say we are publishing a live stream using FMLE (Flash Media Live Encoder) or our custom publisher to FMS (Flash Media Server). There are 5 subscribers who are watching this live stream (given that those subscribers connected to stream after the stream has been already published). Now NetStream.time will not give absolute stream time for any of the subscriber connected to live stream because it gives time since the playback of live stream is started by subscriber. Now let’s say I (as a publisher) want to take certain action (such as displaying an image, or some message, or slide) on subscribers on specific point of time (in terms of viewing the live stream). Let’s say from publisher side, when I see (or speak) something (a tortoise in scene!) on screen, I want to show another picture of that same thing to all subscribers (dancing tortoises!) exactly at the same time when each one of them see tortoise on their live stream.

SharedObject can not be used because that will not provide any kind of sync (in terms of live streaming) because they are asynchronous and live streams may get delayed due to slow bandwidths on subscribers.

Even we can not directly call a function on FMS which would in turn call functions to all subscribers because that process would be instantaneous (compared to delay in live stream) and if there are number of subscribers this process may hang FMS (and publisher) application for a fraction of second.

The Solutions:

1) Somehow give all subscribers the absolute stream time. This would be an ideal solution but not easy. From FMLE, I managed to find a solutions and that is timecode. Let’s say if the publishing video frame rate is 15 fps then FMLE would encode a keyframe at each group of 15 frames (given that timecode interval is same as frame rate of video). Now the timecode is embedded as system time and date which we need to take care of. The condition here is the video must be recorded on server so that we can know the start time of publishing. On subscribers we get onFI() event at each 1 second which would give you current date and time. From this current time subtract the initial time (which is found by playing recorded stream) and that would give you absolute live stream time 🙂

2) Another way is kind of events instead of focusing on time but yet it keeps solid track of sync. From publisher side, call NetStream.send() function along with necessary parameters. This will actually embed metadata on current frame being encoded. When subscribers get this frame, they get the embedded event and they can take the necessary action by analyzing the data given in that event. Usually I would prefer to use onMetaData event for this stuff 🙂

Phue! It’s been late night now (1:43 AM already) and I am going for sleep now. If any one have any questions, post me a comment and I will be back there right away 🙂

Thanks, Naresh

FMS, Remote SharedObject and Arrays

I have bad experience with above 3 at the same time. When ever you use an array in FMS side coding and pass along that in Remote SharedObject you would be in trouble in in terms of bandwidth and memory usage of application. When I change anything an array i.e. add an item, update an item or delete an item, you would need to pass the whole array to shared object and that will send the full array to all connected clients to that shared object. So even if you have change single item in the full array it eats the bandwidth for full array.

So, what I learnt for it? NEVER use an array in FMS side coding when you have to pass it to Shared Object. Always use property names of shared object to identify the objects inside the shared object.


Live Streaming, FMS and Absolute Stream Time

Think of a live stream is being published to FMS and all subscribers are watching that live stream. Now one of the subscriber is late in joining this live stream but still he does that successfully and watches the live stream from the point when he joined it. Everything looks perfect!

But wait, one thing is not perfect definitely – the time of the stream been published. Yes, the subscribers does not see the absolute stream time from start of its publish. So what does that mean? They are not in sync of each other. Let’s say I want to notify all subscribers at certain point of time in video. Now I would be watching something at time t and I will send message to all subscribers. What I want is : all subscribers should see this message at exact point of time when they see what I am watching in video. But as there is no time sync, subscribers won’t know that at what point of time the message should be displayed.

As work around, what I did is I passed timecode from FMLE. That means that at each frame of video, FMLE would encode system time in the frame. All subscribers would get onFI() event with that time code at each second (given that the timecode rate and frame rate of video is same at FMLE). A subscriber would have to save the first timecode as the start point of time. On next onFI() event they can just subtract the start timecode from the current timecode and that would give it a absolute time of stream!!

Once absolute time of stream is found, you can synchronize the message around 🙂 I did not written down any code here but if any one wants the code, let me know – I can provide it.


Flash Media Server and NetStream Publish

When we use NetStream.publish method in append mode for any flash video file, the metadata is not updated in that flash video file. Actually it remains same as whatever was previously while the stream was recorded at the first time in recording mode.

Here you can find more details on the same … FMS Release Notes

Naresh Khokhaneshiya