Dangerous history editing with Mercurial

...let's be revisionist for once

I decided to stop worrying and love the bomb, it was time to edit the immutable repository history of Mercurial! Thanks to the power of hg I almost lost every changeset I was working on but in the end I succeeded in modifying a few past changesets.

I spent some time between documentation and tutorials so I'll sum up the most important commands with a brief description.

You start off by populating your queue with a few revisions with

hg qimport -r revision_start:revision_end

this will generate one .diff file in .hg/patches for each revision between revision_start and revision_end. Afterwards you remove every (with -a in the example) patch from your local repo, so that only the ones in .hg/patches exist. You do so with

hg qpop -a

and then you can edit whatever revision simply by modifying the equivalent rev.diff file. You can merge, delete and add changesets withqdelete qnew qfold
When you are done with modifications you just have to re-apply your revised patches with  

hg qpush -a

and complete the editing with

hg qfinish -a

Not bad.

Achievements in BitFighter

...8 BitFighters make 1 ByteFighter

Today I got mentioned in the credits of Bitfigher, a neat two space shooter multiplayer game.

* BFLogBot Commit: 4a4fe6ac0503 | Author: buckyballreaction | Message: koda now belongs in the credits 

I really appreciate the endless support from the developers while I break their sources!
Check out this awesome game: www.bitfighter.org.

Breaking the Great Firewall

...breaking the law, breaking the law!!!

I'm in the main country of Asia on a work trip and I don't really like the censoring applied here. I will briefly explain how to use the Internet like it should be.

1. Set up a computer with an accessible IP address with an SSH server (I recommend doing this while you are still in your country)
2. Connect using this line

   ssh -ND port usrname@host
   

   -N disables shell input
   -D is the port which will be forwarded to
   username@host is your ssh server
3. Go to Firefox under Options->Advanced->Network->Settings, select Manually enter proxy information" and in the SOCKS section insert 127.0.0.1 and
4. Enjoy your free internet

Now, traffic might be a little slow so I recommend using the Foxy Proxy Firefox extension and manually set the website that are needing it. This extension replaces Firefox proxy implementation so you'll have to configure it again by creating a 'New Proxy' and inserting the site rules like in the example. Don't forget to activate the proxy and Foxy Proxy itself in the main page.

By the way Blogger is normally blocked here, so you'll have to read this blog before leaving.

How to cross compile from Linux to Windows, x264, VLC and Libav

...using latest modern tools!

X264 and VLC are two of the most awesomest opensource software you can find on-line and of course the pose no problem when you compile them on a Unix environment. Too bad that sometimes you need to think of Windowze as well, so we need a way to crosscompile that software: in this blogpost, I'll describe how to achieve that, using modern tools on a Ubuntu 12.04 installation.

[0] Sources
It goes without saying that without the following guides, I'd have had a much harder time!
http://alex.jurkiewi.cz/blog/2010/cross-compiling-x264-for-win32-on-ubuntu-linux
https://bbs.archlinux.org/viewtopic.php?id=138128
http://wiki.videolan.org/Win32Compile
http://forum.videolan.org/viewtopic.php?f=32&t=101489
So a big thanks to all the original authors!

[1] Introduction
When you crosscompile you just use the same tools and toolchains that you are used to, gcc, ld and so on, but configured (and compiled) so that they produce executable code for a different platform. This platform can vary both in software and in hardware and it is usually identified by a triplet: the processor architecture, the ABI and the operating system.

What we are going to use here is i686-w64-mingw32, which identifies any x86 cpu since the Pentium III, the w64 ABI used on modern Windows NT systems (if I'm not wrong), and the mingw32 architecture, that is the Windows gcc variant.


[2] Prerequisites
Note that the name of the packages might be slightly different according to your distribution. We are going to need a quite recent mingw-runtime for VLC (>=3.00) which has not yet landed on Ubuntu, so we'll take it from our Debian cousins.

Execute this command

$ wget http://ftp.jp.debian.org/debian/pool/main/m/mingw-w64/mingw-w64-dev_3.0~svn4933-1_all.deb
$ sudo dpkg -i mingw-w64-dev_3.0~svn4933-1_all.deb

and then install stock dependencies




$ sudo dpkg -i gcc-mingw-w64 g++-mingw-w64
$ sudo dpkg -i pkg-config yasm subversion cvs git-core

[3] x264 and libav 
x264 has very few dependencies, just pthreads and zlib, but it reaches its full potential when all of them are satisfied (encapsulation, avisynth support and so on).

Loosely following Alex Jurkiewicz's work, we create a user-writable folder and then we prepare a script that sets some useful variables every time.

$ mkdir -p ~/win32-cross/{src,lib,include,share,bin}

#!/bin/sh

TRIPLET=i686-w64-mingw32

export CC=$TRIPLET-gcc
export CXX=$TRIPLET-g++
export CPP=$TRIPLET-cpp
export AR=$TRIPLET-ar
export RANLIB=$TRIPLET-ranlib
export ADD2LINE=$TRIPLET-addr2line
export AS=$TRIPLET-as
export LD=$TRIPLET-ld
export NM=$TRIPLET-nm
export STRIP=$TRIPLET-strip

export PATH="/usr/i586-mingw32msvc/bin:$PATH"
export PKG_CONFIG_PATH="$HOME/win32-cross/lib/pkgconfig/"

export CFLAGS="-static -static-libgcc -static-libstdc++ -I$HOME/win32-cross/include -L$HOME/win32-cross/lib -I/usr/$TRIPLET/include -L/usr/$TRIPLET/lib"
export CXXFLAGS="$CFLAGS"

exec "$@"

Please not the use of the CFLAGS variables: without all the static parameters, the executable will dynamically link gcc, so you'll need to bundle the equivalent dll. I prefer to have one single exe, so everything goes static, but I'm not really sure which flag is actually needed. If you have any idea, please drop me a line.

Anyway, let's compile latest revision of pthreads (2.9.1 as of this writing)

$ cd ~/win32-cross/src
$ wget -qO - ftp://sourceware.org/pub/pthreads-win32/pthreads-w32-2-9-1-release.tar.gz | tar xzvf -
$ cd pthreads-w32-2-9-1-release
$ make GC-static CROSS=i686-w64-mingw32-
$ cp libpthreadGC2.a ../../lib
$ cp *.h ../../include

and zlib (1.2.7) - we need to remove the references to the libc library (which is implied anyway) otherwise we will get a linkage failure

$ cd ~/win32-cross/src
$ wget -qO - http://zlib.net/zlib-1.2.7.tar.gz | tar xzvf -
$ cd zlib-1.2.7
$ ../../mingw ./configure
$ sed -i"" -e 's/-lc//' Makefile
$ make
$ DESTDIR=../.. make install prefix=

Now it's turn for libav, so that x264 can use different input chroma and other stuff. If you need libav exececutables, you might want to change the configure line so that it suits you

$ cd ~/win32-cross/src
$ git clone git://git.libav.org/libav.git
$ cd libav
$ ./configure \
      --target-os=mingw32 --cross-prefix=i686-w64-mingw32- --arch=x86 --prefix=../.. \
      --enable-memalign-hack --enable-gpl --enable-avisynth --enable-runtime-cpudetect \
      --disable-encoders --disable-muxers --disable-network --disable-devices
$ make
$ make install

and the nice tools that give more output options

$ cd ~/win32-cross/src
$ svn checkout http://ffmpegsource.googlecode.com/svn/trunk/ ffms
$ cd ffms
$ ../../mingw ./configure --host=mingw32 --with-zlib=../.. --prefix=$HOME/win32-cross
$ ../../mingw make
$ make install

$ cd $HOME/win32-x264/src
# Create a CVS auth file on your machine
$ cvs -d:pserver:anonymous@gpac.cvs.sourceforge.net:/cvsroot/gpac login
$ cvs -z3 -d:pserver:anonymous@gpac.cvs.sourceforge.net:/cvsroot/gpac co -P gpac
$ cd gpac
$ chmod +rwx configure src/Makefile
# Hardcode cross-prefix
$ sed -i'' -e 's/cross_prefix=""/cross_prefix="i686-w64-mingw32-"/' configure
$ ../../mingw ./configure --static --use-js=no --use-ft=no --use-jpeg=no \

      --use-png=no --use-faad=no --use-mad=no --use-xvid=no --use-ffmpeg=no \

      --use-ogg=no --use-vorbis=no --use-theora=no --use-openjpeg=no \

      --disable-ssl --disable-opengl --disable-wx --disable-oss-audio \

      --disable-x11-shm --disable-x11-xv --disable-fragments--use-a52=no \

      --disable-xmlrpc --disable-dvb --disable-alsa --static-mp4box \

      --extra-cflags="-I$HOME/win32-cross/include -I/usr/i686-w64-mingw32/include" \

      --extra-ldflags="-L$HOME/win32-cross/lib -L/usr/i686-w64-mingw32/lib"
# Fix pthread lib name
$ sed -i"" -e 's/pthread/pthreadGC2/' config.mak
# Add extra libs that are required but not included
$ sed -i"" -e 's/-lpthreadGC2/-lpthreadGC2 -lwinmm -lwsock32 -lopengl32 -lglu32/' config.mak
$ make
# Make will fail a few commands after building libgpac_static.a

# (i586-mingw32msvc-ar cr ../bin/gcc/libgpac_static.a ...).

# That's fine, we just need libgpac_static.a 

i686-w64-mingw32-ranlib bin/gcc/libgpac_static.a 

$ cp bin/gcc/libgpac_static.a ../../lib/
$ cp -r include/gpac ../../include/

Finally we can compile x264 at full power! The configure script will provide a list of what features have been activated, make sure everything you need is there!

$ cd ~/win32-cross/src
$ git clone git://git.videolan.org/x264.git
$ cd x264
$ ./configure --cross-prefix=i686-w64-mingw32- --host=i686-w64-mingw32 \

      --extra-cflags="-static -static-libgcc -static-libstdc++ -I$HOME/win32-cross/include" \

      --extra-ldflags="-static -static-libgcc -static-libstdc++ -L$HOME/win32-cross/lib" \

      --enable-win32thread
$ make

And you're done! Take that x264.exe file and use it wherever you want!
Most of the work here has been outlined by Alex Jurkiewicz in this guide so checkout his blog for more nice guides!


[4] VideoLAN
On the other hand, VLC has a LOT of dependencies, but thankfully it also has a nice way to get them working quickly. If you read the wiki guide, you'll notice that it will use i586-mingw32msvc everywhere, but you should definitely avoid that! In fact that one offers a very old toolchain, under which VLC will fail to compile! Also the latest versions provides much better code, x264 will weight 46MB against 38MB in one case!

So let's update every script to the more modern version i686-w64-mingw32! As usual, first of all get the sources

$ git clone git://git.videolan.org/vlc.git vlc

$ cd vlc 

And let's get the dependencies through the contrib scripts, qt4 needs to be compiled by hand as the version in Ubuntu repositories doesn't cope well with the rest of the process. I also had to remove some of the files because they were of the wrong architecture (mileage might vary here) .

$ mkdir -p contrib/win32
$ cd contrib/win32
$ ../bootstrap --host=i686-w64-mingw32
$ make prebuilt

$ make .qt4

$ rm ../i686-w64-mingw32/bin/{moc,uic,rcc}

$ cd -

We now return to the main sources folder and launch the boostrap and configure process; you need some standard automake/libtool dependencies for this.

$ ./bootstrap

$ mkdir win32 && cd win32

$ ../extras/package/win32/configure.sh --host=i686-w64-mingw32

$ ./compile

$ make package-win-common

Let's grab something to drink and celebrate when the compilation ends! You'll find all the necessary files in the vlc-x.x.x folder. A big thanks goes to the wiki authors and j-b who gave me pointers on #videolan irc.

[5] Conclusions
Whelp, that was a long run! As additional benefit you are able to customize every single piece of software to your need, eg. you can modify the libav version that you are going to use for Vlc as you wish! Also crosscompiling is often treated as black magic, but in reality is a simple process that just needs more careful configuration. Errors often are related to wrong paths or missing dependencies and sometimes a combination of both; don't lose hope and keep going until you get what you want!