What filesystem did you end up using? Ever looked at GlusterFS (http://www.gluster.org/)? From an (admittedly unscientific) comparison of the various free/open-source distributed filesystems, it seems the best to me.

Ceph is still in beta, and the fact that Lustre only supports a single Metadata Server is really scary to me.