For unknown reasons the Mac OSX afsctool only compresses files smaller than 2 GB (This may have to do with limitation of some older file-systems). Afsctool can be patched in order to compress larger files and it works.
Here is the the relevant code afsctool.c, the 2147483647 (=2^31-1) can be changed to a bigger number which will be the new upper limit.
177 numBlocks = (filesize + compblksize - 1) / compblksize;
178 if ((filesize + 0x13A + (numBlocks * 9)) > 2147483647)
179 return;
180
Here is the link to the original afsctool
It can be compiled (Command Line Tools For XCode (or complete XCode) required) with this command:
gcc -arch x86_64 -arch i386 -lz -o afsctool afsctool.c
If you need more explanation or a compiled version, please send me an email: oliver at brausch dot org