Saturday, June 27, 2009

How does Google video chat work in gmail?

This post is just a speculation (aka guess work).

Google has a video chat function from within the Gmail web pages. This function is not available in the GTalk client yet. Google requires you to download a plugin which enables the video chat function form gmail. The video is rendered using Flash Player. In this article I present my understanding of how it works.

Flash Player exposes certain audio/video functions to the (SWF) application. But the Flash Player does not give access to the raw real-time audio/video data to the application. There are some ActionScript API classes and methods: the Camera class allows you to capture video from your camera, the Microphone class allows you to capture audio from your microphone, the NetConnection/NetStream classes allow you to stream the video from Flash Player to remote server and vice-versa, the Video class allows you to render video either captured by Camera or received on NetStream. Given these, to display the video in Flash Player the video must be either captured by Camera object or received from remote server on NetStream. Luckily, ActionScript allows you to choose which Camera to use for capture.

When the Google plugin is installed, it exposes itself as two Camera devices; actually virtual device drivers. These devices are called 'Google Camera Adaptor 0' and 'Google Camera Adaptor 1' which you can see in the Flash Player settings, when you right click on the video. One of the device is used to display local video and the other to display the remote participant video. The Google plugin also implements the full networking protocol and stack, which I think are based on the GTalk protocol. In particular, it implements XMPP with (P2P) Jingle extension, and UDP-based media transport for transporting real-time audio/video. The audio path is completely independent of the Flash Player. In the video path: the plugin captures video from the actual camera device installed on your PC, and sends it to the Flash Player via one of the virtual camera device driver. It also encodes and sends the video to the remote user. In the reverse direction, it receives video (over UDP) from the remote user, and gives it to the Flash Player via the second of the virtual camera device drivers. The SWF application running in the browser creates two Video objects, and attaches them to two Camera object, one each for the two virtual video device, instead of attaching it to your real camera device. This way, the SWF application can display both the local and remote video in the Flash application.

What this means is that for multi-party video calls, either (1) the plugin will have to expose as more video devices (is there any limit on devices?), or (2) somehow multiplex multiple videos in same video stream (which is CPU expensive), or (3) show only one active remote participant in the call (which gives bad user experience).

An open question to ask: will it be possible to use the Google's plugin to build our own Flash application and somehow use our own network application/protocol to implement video call? Hopefully Google will make the plugin API available to public some day.

6 comments:

elias said...

Oh ! how great would it be if we could substitute RTMP for a real ubiquous quality Video/Audio conferencing system embedded in the Browser !!! I bet someone must be doing the reverse engineering... let's find out !

:D

curious guy said...

Wow!! that makes me sick ;) ...
I'm on the way to do a project which consists of 1 way live video and audio . Is there any open source projects or papers on this subject ??
I'm tired of searching ..
Links to any papers or open source projects will be highly appreciated.

alengcm@gmail.com

Rinosh said...

hai
i want to make a voice chat appication in c#, I am able to communicate it through LAN or Internet by giving IP of machines , but if the two systems are located itwo lans then how can i communicate with each other , because theses two machines are having its Local LAN IP and both are connected to internet through proxy
any help is appreciated, which protocol i have to use , any use ful links or sample codes
my mail address is rinosh.sasidharan@gmail.com , www.rinosh.co.cc

Rinosh said...

How to Voice chat over internet from Two LANS
hai
i want to make a voice chat appication in c#, I am able to communicate it through LAN or Internet by giving IP of machines , but if the two systems are located itwo lans then how can i communicate with each other , because theses two machines are having its Local LAN IP and both are connected to internet through proxy
any help is appreciated, which protocol i have to use , any use ful links or sample codes
my mail address is rinosh.sasidharan@gmail.com , www.rinosh.co.cc

techie said...

Cool!
This is what I have been searching for hours!!
Even I had noticed the two virtual adapters "google.. 0" & "google.. 1".

I was just thinking of a simple problems's solution. How could we record gmail's video chat?

IMO if we could use the VLC media player's capture feature to successfully capture feed coming into adapter 1, then in that case we would be able to simple get the feed off the adapter in no time! Simple!?

Well we need some answers.. :)

video chat said...

Meeting new people in the video chat world can sometimes be just as difficult as it is fun. Let's face it - we all love the opportunity to get to know new and interesting people but we also dread the possibility of running into weirdos and creeps.

video chat