Finally one libvlc_video_set_format_callbacks() example

libVLC new (2.0) API for video size and chroma (libvlc_video_set_format_callbacks) is much more powerful than libvlc_video_set_format. It allows either to get video size and chroma dynamically so that your rendering pipeline can adapt, or to modify such values to get the values you need.

However, I find it a little more complicated to use and spent some time looking for an example. Apparently I am not alone, so I'll share my finding.

The main source example I found is from FBVLC, a web browser plugin based on VLC. You can read the full source here: http://code.google.com/p/fbvlc/source/browse/FBVLC.cpp

Basically you need to create two functions with this exact signature

static unsigned video_setup_cb(void **opaque, char *chroma, unsigned *width, unsigned *height, unsigned *pitches, unsigned *lines);

static void video_cleanup_cb(void *opaque);

and as the names go initialize all variables in the first function and cleanup in the second one. You can find sample implementation in the previous source code, under the name of

video_format_cb and video_cleanup_cb.

I spent some time in selecting the right chroma to adopt (and missed the fact that you had to memcpy there) for my standard h264 420 file "RV32" was working fine. Also the pitches variable is important, it represents the the video width in luma components, so for 420 it corresponds to width * 4.