FFmpeg lossy compression problems

I'm trying to compress very large video to something uploadable to Vimeo, preserving weekly limit.. Any command I use that tries to reduce image quality results in videos that crash VLC or simply do not play in other players. Codes I've used include:

Just compressing:

ffmpeg -i test.mp4 -vcodec libx264 -crf 30 test.mp4

Converting .mov to .mp4 and compressing:

ffmpeg -i test.mov -qscale 30 video2.mp4

Any idea why this is happening? I'm working in OSX bash (Mountain Lion).

The latter case resulted in this report:

unknown-4c:8d:79:e6:e3:fc:birthday_int3 robinedwards$ ffmpeg -i test.mov -q:v 30 video2.mp4
ffmpeg version 1.1.2 Copyright (c) 2000-2013 the FFmpeg developers
  built on May 14 2013 16:56:21 with Apple LLVM version 4.2 (clang-425.0.24) (based on LLVM 3.2svn)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/1.1.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-nonfree --enable-hardcoded-tables --enable-avresample --cc=cc --host-cflags= --host-ldflags= --enable-libx264 --enable-libfaac --enable-libmp3lame --enable-libxvid
  libavutil      52. 13.100 / 52. 13.100
  libavcodec     54. 86.100 / 54. 86.100
  libavformat    54. 59.106 / 54. 59.106
  libavdevice    54.  3.102 / 54.  3.102
  libavfilter     3. 32.100 /  3. 32.100
  libswscale      2.  1.103 /  2.  1.103
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    creation_time   : 2013-07-14 15:44:54
  Duration: 00:00:21.77, start: 0.000000, bitrate: 124460 kb/s
    Stream #0:0(eng): Video: png (png  / 0x20676E70), rgb24, 1280x720, 124459 kb/s, 30 fps, 30 tbr, 3k tbn, 3k tbc
    Metadata:
      creation_time   : 2013-07-14 15:44:54
      handler_name    : Apple Alias Data Handler
File 'video2.mp4' already exists. Overwrite ? [y/N] y
using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2 AVX
[libx264 @ 0x7fb71b82ec00] profile High 4:4:4 Predictive, level 3.1, 4:4:4 8-bit
[libx264 @ 0x7fb71b82ec00] 264 - core 125 - H.264/MPEG-4 AVC codec - Copyleft 2003-2012 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'video2.mp4':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    encoder         : Lavf54.59.106
    Stream #0:0(eng): Video: h264 ([33][0][0][0] / 0x0021), yuv444p, 1280x720, q=-1--1, 15360 tbn, 30 tbc
    Metadata:
      creation_time   : 2013-07-14 15:44:54
      handler_name    : Apple Alias Data Handler
Stream mapping:
  Stream #0:0 -> #0:0 (png -> libx264)
Press [q] to stop, [?] for help
frame=  653 fps= 41 q=-1.0 Lsize=    3211kB time=00:00:21.70 bitrate=1212.2kbits/s    
video:3203kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.260798%
[libx264 @ 0x7fb71b82ec00] frame I:3     Avg QP:16.09  size: 90551
[libx264 @ 0x7fb71b82ec00] frame P:203   Avg QP:22.52  size: 12759
[libx264 @ 0x7fb71b82ec00] frame B:447   Avg QP:28.39  size:   933
[libx264 @ 0x7fb71b82ec00] consecutive B-frames:  8.6%  0.3%  0.5% 90.7%
[libx264 @ 0x7fb71b82ec00] mb I  I16..4: 23.8% 30.3% 45.9%
[libx264 @ 0x7fb71b82ec00] mb P  I16..4:  0.2%  0.7%  0.5%  P16..4: 17.2%  6.8%  5.7%  0.0%  0.0%    skip:68.8%
[libx264 @ 0x7fb71b82ec00] mb B  I16..4:  0.0%  0.1%  0.0%  B16..8: 14.5%  0.5%  0.2%  direct: 0.6%  skip:84.2%  L0:36.1% L1:61.8% BI: 2.1%
[libx264 @ 0x7fb71b82ec00] 8x8 transform intra:43.6% inter:62.5%
[libx264 @ 0x7fb71b82ec00] coded y,u,v intra: 51.4% 2.6% 27.7% inter: 6.4% 0.0% 2.6%
[libx264 @ 0x7fb71b82ec00] i16 v,h,dc,p: 52% 42%  5%  0%
[libx264 @ 0x7fb71b82ec00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  9% 18% 35%  6%  7%  7%  6%  6%  6%
[libx264 @ 0x7fb71b82ec00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 13% 24% 23%  6%  9%  6%  9%  3%  7%
[libx264 @ 0x7fb71b82ec00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fb71b82ec00] ref P L0: 72.4%  9.8% 14.2%  3.6%
[libx264 @ 0x7fb71b82ec00] ref B L0: 90.1%  8.6%  1.4%
[libx264 @ 0x7fb71b82ec00] ref B L1: 92.4%  7.6%
[libx264 @ 0x7fb71b82ec00] kb/s:1205.09

Your input video uses an RGB colorspace – rather unusual. What happens is that ffmpeg converts that color space to planar YUV, without subsampling (yuv444p). Unfortunately, this color space is not widely supported. Many consumer programs or hardware simply won't play it. 4:4:4 subsampling is more common in broadcasting.

Per Vimeo's guidelines you can upload High Profile and Main Profile H.264 video, and these only support 4:2:0 subsampling. The proper ffmpeg command would therefore be:

ffmpeg -i input.mov -c:v libx264 -crf 30 -pix_fmt yuv420p test.mp4

Some tips:

• Add -profile:v high to force the High Profile.
• Having RGB24-encoded input could mean you're encoding an animation or something pre-rendered, maybe from an individual set of images. Have a look at the -tune animation option, which optimizes encoding for animated video.
• A CRF of 30 might result in bad quality – I'd start off with 23–25 and only increase the value if the video is still too big for you.
• q:v or qscale shouldn't be used with the libx264 encoder, which is what ffmpeg tries to auto-select for MP4 video. Use the CRF here, like you did in your first example.