Golan Levin and Collaborators

Flong Blog + News

Open-Source Panoramic Video: Bloggie + OpenFrameworks & Processing

23 June 2010 / code, project

I’m pleased to announce the release of free code and tools, created in openFrameworks and Processing, for unwarping the panoramic videos produced by the popular Sony Bloggie solid-state camera. The inexpensive Bloggie ($160-190), in many ways similar to the Flip, is a perfectly decent little pocket camera with one exceptionally notable feature: it allows for the creation of 360-degree panoramic digital video, at relatively high resolution, at a consumer price-point. Here’s an example:

Obviously, this “donut vision” (annular) video needs to be transformed into something more useful, such as a long panoramic strip — a process variously called unwarping, dewarping, inverse warping, remapping, reprojection, rectilinearization, or simply distortion correction. Although Sony released unwarping software for Microsoft Windows, they — inexplicably — didn’t release Bloggie unwarping software for Mac OSX. Well, that’s problematic. Perhaps because panoramic imaging is considered a “niche” market, commercial tools for panoramic unwarping can be quite pricey. And although Christophe Maillot has lately released ThreeSixZero, a free OSX tool for viewing Bloggie panoramas, his project is not open-source and doesn’t export converted panoramic video! That’s the motivation for the releases presented here. Fortunately, the mathematics for polar-to-cartesian geometry conversion are relatively straightforward. Behold, a 360-degree panoramic video, remapped to a rectilinear perspective with our BloggieUnwarper software:

Projects and Source Code

Below are links to BloggieUnwarper, my open-source code projects for unwarping the panoramic videos produced by the Sony Bloggie. These projects are created in openFrameworks (a C++ library for creative coding) and Processing (a Java-based programming environment for people who want to create images, animations, and interactions). Both OpenFrameworks and Processing are free, open-source, cross-platform toolkits, optimized for visual computing, which take most of the headache out of creating graphically-oriented software. Both programming environments work across multiple operating systems (Mac, Windows, and Linux), and multiple compilers (e.g., openFrameworks compiles in XCode, Visual Studio, Code::Blocks, and more). If you’re interested in interactive art or computational design, these environments are a great way to start making projects fast.

Here’s a screenshot of the openFrameworks BloggieUnwarper software, showing the original (annular) video at upper left; the reprojected panoramic strip at the bottom; and (in the upper right) a view of the panorama from within a texture-mapped 3D cylinder:

Complete source code can be found here:

  • BloggieStillUnwarp.zip (for Processing; 1.3Mb download) — This is a single-page Processing sketch which performs inverse warping on a single panoramic still image stored in the sketch’s data folder. The program supports both nearest-neighbor and bicubic interpolation, and can save out the unwarped panorama to disk. To get started, download the Processing environment here; unzip this project into your “Sketchbook” folder (in “Documents”); open the sketch in the Processing development environment, and press play.
  • BloggieVideoUnwarp.zip (for Processing; 20.2Mb download) –This is a single-page Processing sketch which unwarps a Quicktime video stored in the sketch’s data folder, and exports an unwarped version of the video — either as a new Quicktime movie, or as a folder full of still images. Parameters for the unwarping (such as the optic center) are loaded from a settings file. The audio from the original file is (regrettably) not maintained; see my notes below about audio. (Please note that the bulk of this download is a 20Mb demonstration video.)
  • BloggieUnwarpOF.zip (for OSX & openFrameworks, 52.7Mb download) — This is a complete openFrameworks project for viewing, projecting, dewarping, and exporting panoramic videos shot with a catadioptric reflector like the Sony Bloggie’s. This project is built for Mac OSX 10.6.3, using XCode, and openFrameworks v.0.061, which is available here; be sure to download the FAT version. The project makes use of the ofxOpenCv and ofxXmlSettings addons which are included in the download. To avoid compilation errors, install the project at the same hierarchical directory level as the various “example” apps which come with the FAT download. There is some additional information about getting OF set up on Snow Leopard here. This project is available two ways:

More About the BloggieUnwarper

Here’s some more information about panoramic video on the Sony Bloggie, our app, and its implementation.

Sourcing the Bloggie and its Panoramic Lens Attachment.
The Sony Bloggie MHS-PM5K (about $160-180) comes with an included 360-degree panoramic lens attachment. You may hear this accessory erroneously referred to as a “fisheye lens“, but it’s actually a hyperbolic or parabolic mirror (with an optical path folded by a flat mirror, as it turns out) known as a catadioptric reflector. This panoramic lens attachment, which easily clips on and off the body of the camera, is the black piece shown above. The attachment is Sony Part Number X25467051, a “360 VIDEO ACCESSORY”, labeled VCL-BPP1 on its body. The attachment really does come with the camera! But as an isolated component, this accessory is also available here through Sony’s services site (Tel. 1-800-488-7669) for $82.95 USD. Now given that this is, like, a $5 piece of plastic, this is still a minor bummer, but oh well: competing mirror-based optics for one-shot 360-degree imaging, like the Palnon parabolic C-mount panoramic lenses from JBC, the Go Pano from EyeSee360, the OmniAlert360 from RemoteReality, the PentaOne P360 series ($~1000), the EggSolution EGG Lens (~$2000), or the 0-360 Panoramic Optic ($495), can cost hundreds or even thousands of dollars; there’s simply nothing else quite like the Bloggie at the low end. In any case, it’s worth pointing out that you can duct-tape this Sony pano accessory onto just about any micro-format camera if your application demands a different hardware scenario. I made relatively decent panoramic images by simply holding it in front of the cameras on my laptop and my mobile phone, and I also had good luck affixing the mirror module onto standard 12mm “micro-video” lenses such as those used in many web-cams. If you’re interested in doing something low-budget like DIY-Streetview, the Bloggie and/or this lens attachment is the camera for you.

Limits on the Resolution of Resulting Panoramas.
The Sony Bloggie is capable of recording non-panoramic HD video at resolutions up to 1920×1080 (30fps). Unfortunately (and this isn’t well-advertised), the Bloggie only records panoramic digital video at 1280×720. In video mode, the camera auto-detects the panoramic lens attachment (magnetically, it turns out) and switches to the lower resolution. Moreover, by the time the unwarping process has removed the wasted black pixels (inside and around the warped video annulus), it becomes clear that the maximum effective vertical resolution of the Bloggie’s panoramic video is only about 256 pixels, which may or may not suit your purposes. Used in still-picture mode, the Bloggie is able to create panoramic source images with its full (5 megapixel, 2592×1944) resolution; the effective vertical resolution of its panoramic stills is then 512 pixels. Since the Bloggie lens accessory has a vertical FOV of approximately 54.5 degrees (according to our measurements; others have measured the vertical field of view at 52.5°), panoramas produced with these optics should have aspect ratios close to (360/54.5) ≈ 6.6:1. For panoramas produced with the BloggieUnwarper, I therefore recommend dimensions like 1280×196, 1680×256, or (for 5MP stills) 3360×512. (Note that the Bloggie has a 1/2.5” CMOS imaging sensor and F3.6 optics. This means: expect relatively poor low-light performance, and some very exotic rolling-shutter artifacts on fast-moving subjects after your source video is unwarped.) Update: in a new blog post, “Modding the Bloggie Panoramic Lens Accessory“, I detail a simple modification to the Bloggie accessory which enables 1920×1080 panoramic video recording.

An Opportunity for Real-Time Panoramic Video
The BloggieUnwarper software can compute unwarped panoramas in real-time or faster. And happily, the Bloggie camera can also produce real-time panoramic video, through its analog NTSC/PAL composite video output, using the proprietary Sony HandyCam Audio/Video Cable (VMC-15FS A/V) which comes with the camera. This interlaced analog video signal produced by the Bloggie is then suitable for 720×480 capture via standard analog capture cards such as the Dazzle. An opportunity thus exists for (e.g.) interactive computer-vision based applications that use 360-degree sensing. To be perfectly clear, the Bloggie does not support use as a real-time USB webcam; you’ll need to run its analog video output through a capture card (or fit the Bloggie lens attachment onto some other camera) if you intend to build apps that incorporate real-time panoramic video.

The code projects provided here are configured for unwarping Bloggie videos stored on disk — but modifying these apps to use real-time video is straightforward; you’ll simply need to swap out the source of input pixels from the pre-stored video to the live video input:

  • If you’re working in openFrameworks, integrate the ofVideoGrabber class into the project, as demonstrated by the movieGrabberExample example included in the FAT download, to understand how to access the live video stream and its pixels.
  • For Processing, on the other hand, integrate the Capture class within the Video library, taking a careful look at the included GettingStartedCapture and Mirror examples to understand how to access the live video and its pixels.

Special Considerations, Features, and Known Bugs
Unwarping panoramic video from the Bloggie has a few special considerations, mentioned here.

  • Handling Inexact Centering of the Panoramic Optics.
    Sad to say, the Bloggie’s removable panoramic attachment is a wee jiggly. As a result, it’s unwise to assume that the optical center of the video donut is located precisely at the location [width/2, height/2]; indeed, the optical center has a movement range of approximately ±2% of the image dimensions. This has significant implications for an unwarping tool; even small differences of a quarter pixel can cause an undesirable sinusoidal “wobble” that runs the length of the panorama. The apps provided here allow for the user to adjust, set and save the center point of the video annulus, with sub-pixel accuracy.
  • Handling Corrections for Radial/Vertical Nonlinearity
    If the Bloggie had a ‘perfect’ catadioptric mirror, there would be no vertical distortion in the unwarped panoramas. Unfortunately, our careful measurements of rectilinear grids photographed with the Bloggie’s panoramic lens attachment proved otherwise; we discovered a minor nonlinear distortion in the radial/vertical dimension of the Bloggie lens. We also discovered that the Bloggie panoramic lens has an asymmetrical vertical field of view, with approximately 80% of the image falling above the horizon, and 20% below. Taking both perspectival distortion and the Bloggie’s off-center vertical optics into account, we computed this distortion, modeled it with a 2nd-order polynomial, and have corrected for this in our unwarping procedure. You’ll need to recalculate these fitting parameters if you wish to use this software with a different catadioptric mirror.
  • Providing Azimuthal/Horizontal Shifting
    When unwarping an annular video to a horizontal panoramic strip, the placement of the “zero angle” becomes an important artistic choice. Modifying this angle has the effect of determining the horizontal placement of objects in the scene. BloggieUnwarper allows for user-determined shifting of this kind.
  • Bugs with Preserving Source Audio
    As of this writing, BloggieUnwarper doesn’t handle audio nearly as well as I would like. The Processing version does not preserve the source audio at all; if you want to keep it, you’ll need to use a tool like FinalCut or AfterEffects to recombine the original source audio with the corrected panorama video. The openFrameworks version is a little more strange. It currently produces “non-standard” videos, which contain the original audio alongside the unwarped video, and are playable in Quicktime7+ on the Mac, but which fail to play in Windows, and are rejected by Vimeo for unspecified incompatibilities. I have successfully circumvented this by re-compressing the results using Quicktime and/or MPEG Streamclip.

A brief note about speed and library dependencies.
To accomplish fast (better than real-time) inverse warping, we compared four methods: a well-optimized, hand-coded inverse lookup table; the cvRemap() function available within the free OpenCV library; the fwiRemap() function in AMD’s free FrameWave signal processing library; and the ippiRemap() function from the commercial Intel Performance Primitives (IPP) 6.0 libraries. That’s right — we had a shootout. All of our implementations were coded for 3-channel (RGB) images, and for each method we tested both Nearest Neighbor and Bicubic interpolation methods. We set compiler optimizations to level -O3 in XCode and compiled for 10.6|Release. Under these conditions, the quick summary of the libraries’ relative performance was as follows:

  • IPP ippiRemap: 1 time unit
  • OpenCV cvRemap: 2 time units
  • Framewave fwiRemap: 4 time units
  • Hand-coded version: 2.5-10 time units, depending on the extent of optimization

In other words: IPP was the fastest; OpenCV was twice as slow as IPP; Framewave was four times as slow; and (short of getting all medieval with assembly code), my own best hand-coded algorithm was about 25% slower than OpenCV. Based on the above shootout, we elected to use OpenCV with our openFrameworks build (see the ofxOpenCV addon example, which comes with the download), and a hand-coded unwarping algorithm for the Processing version. Researchers who are serious about speed may wish to consider the (non-free, non-open-source) IPP libraries. As an aside, OpenCV is documented as being able to “take advantage of IPP if it is available“, but this feature of OpenCV is being phased out.

Acknowledgments
Ivan Poupyrev first alerted me to the existence of the Sony Bloggie. Solomon Bisker helped port my original Processing sketch to openFrameworks, and also set up the four-way shootout between IPP, Framewave, OpenCV, and my hand-coded algorithms. Amisha Gadani tracked down purchasing details for the X25467051 lens accessory. Kyle McDonald patiently walked me through learning Git, and collaborated in many stimulating discussions and experimentation related to low-cost panoramic video. This (side-)project was supported by the STUDIO for Creative Inquiry at Carnegie Mellon University, with the help of grants from the Fine Foundation and the National Endowment for the Arts. This project has been featured at Hack-a-Day.

Keywords
Golan Levin, Flong, Processing, openFrameworks, #Processing, #openFrameworks, Sony Bloggie camera, 360° video, 360-degree pano, panoramic video, catadioptric reflector, hyperbolic reflector, parabolic mirror, unwarp, dewarp, inverse warp, remap, reprojection, rectilinearization, distortion correction, free open-source code, Java, C++, software, free video tools, optics, lens, attachment, one-shot panoramic imaging, download, Bloggie software for Mac OSX, free Bloggie for MaxOS X, Mac OS X.

References
A Theory of Single-Viewpoint Catadioptric Image Formation“, S. Baker and S.K. Nayar. International Journal on Computer Vision, Vol.35, No.2, pp.175-196, Nov, 1999. [PDF]

Image Warping / Distortion“, P. Bourke. http://local.wasp.uwa.edu.au/~pbourke/miscellaneous/imagewarp/, Dec. 2002.

Licence
This software is licensed under the CC-GNU LGPL version 2.1 or later.
CC-GNU LGPL


TykeType! a free, open-source application for toddler typing

13 June 2010 / code, life, project

TykeType! A free application for toddler typing

TykeType! is a free, open-source application I created for my toddler to type at the computer keyboard. The app is very minimal, though it does have a few features (like automatically separating numbers from letters) that make reading text aloud easier. You can see a quick demonstration video below.

TykeType! is built for Mac OSX and uses openFrameworks, an open-source toolkit for creative coding. You’re invited to download the application (and all source code) here. The software and code are free for non-commercial purposes, and are provided without any support.

Creative Commons License
TykeType! by Golan Levin is licensed under a Creative Commons Attribution-Noncommercial 3.0 Unported License. Keywords: children’s software, software for children, free toddler typing game, free typing program for kids, toddler typing software, pressing keys, pushing buttons, keyboarding, introductory keyboard skills, basic reading skills, keyboard interaction, typing software, computer keyboard play. Also see related software: TykeWriter, ToddlerType.


A Brief Note on Infrared Filters: 87 vs 87C

6 June 2010 / reference

Many computer vision projects require a camera which blocks visible light and exclusively sees infrared light. Unless you have money to spend on optical glass units, such filters generally take the form of a thin sliver of black gelatine or polyester sheet, inserted between your camera’s sensor chip and your lens. Representative products include the Kodak Wratten series (about $0.80 per square cm), and the Lee filter series (~$0.25 per square cm). These filters each have several varieties with different spectral properties: Type 87 passes IR longer than 750 nanometers, while type 87C passes IR light longer than 800nm. In the photo above, I’m holding a sample of each of these filters. (The camera can see through these, but they’re both optically black to humans.) Here you can see the filters’ spectral responses:

Meanwhile, most CCD sensors have spectral responses which taper off in the infrared regions above ~700nm:

Thus, the photo at the top of this post confirms what you would logically expect: as the sensitivity of the CCD diminishes for longer wavelengths of IR light, IR-pass filters with higher cutoff frequencies will allow less and less light through to the camera. How much less? To judge from my casual test, I’d say about 10%, 25 grayscale levels, or roughly ½ of an F-stop. No big surprises here, but it’s nice to see the theory confirmed. The quick upshot: Choose an 87 filter over an 87C if you need to get the most out of your infrared illumination.

For more information about IR imaging, I recommend the resources:

(Please note that this article refers to cameras and filters for “near IR” [close to red, with wavelengths around 700-900 nanometers], not the “deep IR” thermal-imaging cameras [sensitive to wavelengths in the 7-14 µm range], such as those made by FLIR.)


Computational Gastronomy

2 June 2010 / general, life

It can be a rewarding and entertaining exercise to think through the implications of prepending “computational” to various subjects of consideration — particularly those fields traditionally considered “outside” the purview of procedural science, like the arts and humanities. (As communications technologies inexorably revolutionize every aspect of contemporary life, such a gedankenexperiment is merely a reasonable strategy for planning for the very near future.) A recent conversation with my friend Joshua got me wondering about a new one, “computational gastronomy”, and reminded me that we would do well to consider computation in combination with the messiest possible subjects, like sexuality, spirituality and psychology. My initial research on computational gastronomy turned up relatively little (just a passing mention in a short story and a brief tweet from February) so I thought it could be helpful to assemble some preliminary proceedings for a virtual congress on this hot new topic. Here, then, I tender an index to some (very real) overviews and articles in emerging sub-concentrations of the field. Some of the titles are genuinely mind-bending, suggesting even stranger things to come:

Computation and Agriculture:

Recipes and Planning:

Computation and Food Preparation Technologies:

Analysis of Eating Behavior:

Food Interactions and Gustatory Displays – Proposals:

Star Trek Replicator - Computational Gastronomy

The Star Trek replicator, a computational gastronomy system.

XKCD #720: Recipes

From XKCD (#720): A speculation on genetic algorithms and recipe generation. (Thx @zzkt)


Announcing: An Exhibition of Student Projects!

25 April 2010 / announcement, exhibition, pedagogy, studio

What: Projects from “Special Topics in Interactive Art & Computational Design
Where:
Carnegie Mellon STUDIO for Creative Inquiry, CFA-111
When:
Wednesday, 28 April 2010, 4:30-6:00pm

Golan LEVIN (Associate Professor of Electronic Art) and Patrick Gage KELLEY (Teaching Assistant and PhD Student, CS), are proud to present capstone projects and other treats from our interdisciplinary Spring 2010 course, Special Topics in Interactive Art & Computational Design, in which 22 students from seven departments — Art, Architecture, Design, Computer Science, Human-Computer Interaction, Electrical Engineering, and Entertainment Technology — spanning the full range from freshmen to doctoral students — have explored computation as a medium for curiosity-driven experimentation!

COME SEE information visualizations … augmented realities … interactive robotics … generative form fabrication … contestational retrosurveillance …mobile software … speech-driven gaming … computational inflatables … crowd-sourced psychoanalytics … 2D and 3D game design … culture analysis machines … useful web services … and freestyle computing hacks … all developed in arts-oriented computing environments like PROCESSING, OPENFRAMEWORKS, FLASH and ARDUINO.

Poster for Class Exhibition. 4-28-2010

Featuring new works by: Caryn AUDENRIED, Solomon BISKER, Amanda BURRIDGE, Richard CAMERON, Justin EDMUND, Mehrdad GHODS, Max HAWKINS, Michael HILL, Xiaoyuan JIANG, Nara KASBERGEN, Jessica MENG, Jon MILLER, Alyssa REUTER, Matthew SANDLER, Paul SHEN, Jordan SINCLAIR, Gabriela URIBE, Karl D.D. WILLIS, Kuan Ju WU, Cheng XU, David YEN, & Raymond YUN. This exhibition was made possible with generous support from the Fine Foundation; the STUDIO for Creative Inquiry at Carnegie Mellon, and the CMU School of Art.