Pre-processing Audio and Video capture to Flash in Mac OS X

Web streaming services like UStream.tv and Livestream (previously known as Mogulus) use a Flash applet to capture audio and video signals on source computer. Media stream is encoded by Flash and sent over to broadcasting server. Currently this approach has two shortcomings:

1. Flash applet audio capture is very limited: only one device a time, only one stereo channel pair and doesn’t support devices without volume level mixer. It doesn’t allow any advanced setup like multi-channel digital mixers.
2. Audio and video are encoded on the fly and no local copy is recorded. Streaming services offer server-side recording, but after downsampling and packet loss, quality is very low. Even worse: any connection fault will interrupt your recording.

To achieve multi-channel audio capture and client-side recording, live events media producers traditionally have few options:

1. Process audio and video signals with external mixer and recording tools and then acquire analog output signals with an AD capture card. This makes multi-channel recording more expensive. Furthermore, if signal processing passes through analog components, streaming quality is degraded and HD video must be downscaled.
2. Record your audio-video using your camera and then use camera Firewire or analog output to stream to computer. To do this, you need to have audio inputs on your camera. This has three downsides: your recorded session may have lower video quality (miniDV or miniHDV tape recording quality may be lower than firewire output quality), you cannot record multitrack audio and you need to connect your audio mixer to your camera.
3. Use a digital mixer coupled with Flash Media Encoder software, which is only available on Windows, so if you’re a Mac OS X or Linux user, you’re out of luck. Also, you cannot record high quality encoded files but only FLV as they’re sent over to the broadcast.
4. Use UStream Producer Pro, a $199 tool released by UStream.tv for Windows and Mac OS X, that allows multi-camera management, HDV input, PIP and broadcasting output recording, at the same quality of the output stream.

For my live streaming setup, I wanted to use my MacBook laptop with Mac OS X, a Sony HVR-V1 HDV camera connected through Firewire and a Edirol M-16DX 16-channel digital audio mixer connected via USB to capture audio from a few condenser microphones. Methods mentioned above aren’t applicable, because they would require additional hardware and recording quality would be very low, compared to original signal quality.

After some research, I found a way to process audio and video signals before submitting them to Flash applet in Mac OS X. First, the requirements:

1. CamTwist: it’s a freeware software that creates a loopback video interface and allows multiple client apps to access a single video device at the same time. A number of video effects can be applied to the video stream by CamTwist.
2. Jack audio router: an Open Source application that allows custom routing of audio channels between multiple devices and programs simultaneously with very low latency.
3. An account on UStream.tv with an available broadcasting channel setup.

Once you’ve installed the programs and created the account, you can use the following checklist to configure your system to route your advanced audio-video stream both to UStream.tv and to your custom recording tools:

1. Connect your external camera and audio sources and wait until they’re properly recognised by the system.
2. Run CamTwist and select your video input from the Step 1 column and adjust input settings in the “Settings” column on the right. If you want, you can add effects in column Step 2 (check CamTwist documentation for details). If you need a preview, you can use Tools -> Preview menu (close it afterwards, so that it doesn’t eat up cpu cycles).
3. Run JackPilot program and configure access to hardware device from JackPilot -> Preferences menu (it should open automatically on first use). You need to select your audio driver (typically coreaudio), your hardware interface (select your multi-channel card), your hardware-specific setup (frequency, number input and output of channels) and then set your virtual Jack card channels and untick “Auto connect with physical ports”.
4. Once you’re done with Jack setup, close Preferences dialog and start Jack audio server by clicking on Run button in main dialog.
5. Open your audio-video recording tool and select CamTwist from available video input interfaces and JackRouter from audio input interfaces and start a new recording. For instance, if you use QuickTime to record your session, go to QuickTime -> Preferences menu and select Recording tab. In Video Source drop-down menu select CamTwist and in Microphone menu select JackRouter, then close dialog and open a new session with File -> New video recording menu and click on Record button to start recording. Important: if you don’t start recording, your tool may not show up in your Jack router setup, so do as you’re told
6. Open your browser (Firefox is better) and connect to UStream.tv, login with your account and click on Broadcast now button. A popup will open and once Flash applet is loaded, you can select your audio and video inputs just below the preview frame. Select CamTwist from Video drop-down menu and JackRouter from Audio menu. Check that you see video from your source in preview frame.
7. Go back to JackPilot and click on Routing button to open Routing panel. The panel contains 3 frames: on the first frame, you have your physical captures and your application audio outputs, on the second frame you have your physical outputs and your application inputs and on the third frame there’s a list of connection. What you have to do is create a series of mappings between items in first frame to items in second frame. To create a mapping, you need to select one item in first frame and double-click on item on second frame. To unbind the mapping, just repeat the same action again. In my setup, I create a mapping between stereo mix-down channels 17 and 18 of my mixer and Flash (Firefox) stereo input channels and then a mapping between the same mix-down channels and QuickTime stereo input. If you use a multi-track recording program like Cubase, you can add more virtual channels to Jack Router (in Preferences) and map each mixer input to its own virtual channel. Remember: if you need to change Jack Router settings to add more channels or change interface settings, you have to shut down Jack with Stop button first.
8. Now you’re ready to click on Broadcast button on your UStream.tv channel and go live. Check audio level with Flash Vu-meter and adjust volume slider accordingly. For what regards audio and video quality sliders, my advice is to keep audio above 70 and video at least 50: the more bandwith you have, the more you can raise video.

Be careful: audio and video streams are not synchronized, because there’s no common clock or timecoding between the source peripherals. This means that long audio-video sessions might get slightly out of sync if processing time is different between audio and video. My advice is to check wether your camera and audio mixer support timecoding synchronization in a common format: if they do, then set up a link between the two. If they don’t (which is probably your situation, if you’re using consumer devices), avoid running other cpu-intensive or disk-intensive processes during the streaming/recording activity and use a post-production tool like FinalCut Pro or Adobe Premiere Pro to match audio and video on recorded files by shifting and stretching appropriately. I will write more on this later.

That’s all. Have you found other ways to accomplish the same result? Need some hints for your setup or troubleshooting? Please share your thoughts using the comments box below and remember to subscribe to my feed for additional tips on AV techniques.

P.S.: Special thanks goes out to Maurizio Tognoni, who shared the hacking and testing efforts.