Psyk's stuff

Fixing choppy sound in Fedora and VMware

2010-10-03T14:24:00.000+11:00

Sony and Ableton etc. don't support Linux with their major high-end audio applications which is such a damn shame. Applications such as Ableton Live, Sony Acid Pro and Sony Vegas Pro demand a lot of CPU resources and my system is dual-boot to get the best experience with these apps, but sometimes one needs to do a quick re-edit, re-render and sometimes most of the material is stored on the Linux partitions in ext3/lvm format.... It would be easier if Apple, Sony and Ableton supported Linux... oh well. The current answer is to run them up in a Windows virtual machine and I've had a lot of success with VMware Workstation.

One of the biggest issues is choppy sound under load within the VM's. Ableton is extra sensitive and allows you to tweak additional parameters to get the most optimum configuration, since you're virtualized you have multiple layers of tweaking...

Since Fedora (currently on 13) runs Pulseaudio, it's much easier now days to configure and setup the audio parameters. Below are a settings that helped in getting better audio (i.e. without stuttering) in a Vista VM.

1) Run up in a terminal

gedit /etc/pulse/daemon.conf

2)Change the following (most of these will be commented out with a ';' just remove the semi-colon):

high-priority = yes 
nice-level = -11
realtime-scheduling = yes 
realtime-priority = 5 
default-fragments = 25 
default-fragment-size-msec = 25 
resample-method = speex-float-3

3) Bring up a terminal window as user (i.e. non-root):

[psyk@psyk ~]$ killall pulseaudio
[psyk@psyk ~]$ pulseaudio

The last entry should restart the pulse daemon.

Fire up your VM and you should get trouble-free sound :)

Mounting Truecrypt/Realcrypt with read/write access for users

2010-09-05T00:49:00.000+10:00

Mounting realcrypt or truecrypt volumes as a regular user disallows writing to the volumes unless you're root.

A quick fix is as follows. When mounting your realcrypt volume, click mount -> Options. You should get the screen below. In the mount options field enter the relevant uid and gid entries as shown.

Options screen in Realcrypt when mounting a volume

Setting up Password-less logins via SSH

2010-06-12T17:07:00.001+10:00

This allows you to SSH into many machines (great for managing a number of VM's) without having to continually enter login and password.

On your main server (e.g. your management server), configure the SSH keys:

# mkdir -p $HOME/.ssh
# chmod 0700 $HOME/.ssh
# ssh-keygen -t dsa -f $HOME/.ssh/id_dsa -P ''

This command creates two files: $HOME/.ssh/id_dsa (private key) and $HOME/.ssh/
id_dsa.pub (public key).

# ssh-copy-id -i $HOME/.ssh/id_dsa.pub (your.other.linux.servers)

From your 'management server' test the results by ssh'ing into each of the servers:

# ssh (your.other.linux.servers)

Update to PS3 encoding using mencoder in MP4 format

2010-01-24T00:23:00.003+11:00

Further to my old (and probably obsolete) entry here I thought it worthwhile to add an update in regards to encoding to MP4 format playable on Windows Media Centres and PS3's, although it should work on other platforms such as the Xbox but can't be sure as I don't have one for testing.

One of the biggest issues I had with the previous method was the reliability in muxing it into an MP4 via MP4Box and especially when trying to get the audio, video sync right which I think had to do with it not being able to detect the FPS correctly. In any case, the procedure involves converting everthing to an AVI (mencoder's preferred format) and then ripping/converting them raw and muxing it back into an MP4 using MP4box, mencoder can do MP4's but it's broken.

First things first, and I can't take all the credit for this. Credit goes to the h264enc project which uses mencoder heavily and has some great built-in profiles. Excellent display of show-casing mencoder's power and versatility.

Lets, start...

Add a new profile entry in ~/.mplayer/mencoder.conf

 [H264ENC]  
 profile-desc="taken from h264enc, PS3 compatible encode with high quality."  
 vf=pp=al:c,softskip,scale=720:-10,harddup   
 aspect=16:9  
 af=volnorm=2,lavcresample=48000:16:1   
 srate=48000  
 oac=faac=yes   
 faacopts=mpeg=4:br=160:tns=yes:object=2  
 ovc=x264=yes  
 x264encopts=crf=19:me=umh:me_range=24:nodct_decimate:nointerlaced:8x8dct:nofast_pskip:trellis=1:partitions=p8x8,b8x8,i8x8,i4x4:mixed_refs:keyint=240:keyint_min=24:psy_rd=0.8,0.2:frameref=3:bframes=3:b_adapt=2:b_pyramid:weight_b:direct_pred=auto:subq=7:chroma_me:cabac:aud:aq_mode=1:deblock:vbv_maxrate=20000:vbv_bufsize=20000:level_idc=41:threads=auto:ssim:psnr

Obviously you can add your own parameters here especially in the "vf" section where you can adjust the scale, add some additional features (e.g. pullup,softskip, yadif etc.)

I won't go into all the details here, but basically this scales the image down to a 720p type format, normalizes the volume and resamples it to 48kHz, encodes it to AAC at 160kb and uses a constant rate factor for h264 of '19' for higher-end quality without much regard for final file size (i.e. it focuses on maximizing the video quality).

Once you have your profile you can run:

 mencoder -profile H264ENC "source_movie_file.ext" -o "source_movie_file.avi"

This is the encoding process and the lengthiest bit...

If you want to hardsub an SRT file you can specify the -sub "name_of_srt_file.srt" and the -subfont-autoscale 1 parameters to the mencoder line above.

For example:

 mencoder -profile H264ENC "source_movie_file.ext" -o "source_movie_file.ext.avi" -sub "name_of_srt_file.srt" -subfont-autoscale 1

Once this is finished, you'll have a h264 encoded video and aac encoded audio muxed into an AVI format, which is not very friendly format and container combo, only mencoder/mplayer/vlc player friendly :)

Next process requires us to split the video and audio into 'raw' input files that MP4Box can handle. We'll do the video first:

 mencoder "source_movie_file.avi" -nosound -ovc copy -of rawvideo -o h264_video.h264

This will output a file called "h264_video.h264" in a raw stream that MP4Box can understand. As we've already done the encoding in the first step this will be quick as we're just ripping it from the AVI file.

Next, is the audio:

 MP4Box -aviraw audio "source_movie_file.avi" -out aac.raw

Here we're using MP4Box to rip the audio from the AVI to a file called "aac_audio.raw" as MP4Box will rename the output.

Rename it so that MP4Box can manage it as an AAC input:

 mv aac_audio.raw aac_audio.aac

The next step is to pick up the framerate (FPS) of the original movie source for correct interpretation into MP4.

 mplayer "source_movie_file.avi" -noconfig all -loop 1 -identify -nosound -vo null -nocache -frames 1 2>/dev/null | grep '^ID_VIDEO_FPS' | tail -n 1 | awk -F= '{print $2}' > mp4fps.fps

This outputs the FPS of the original converted movie (AVI file) to a file called mp4fps.fps.

The final step muxes the video and audio into an MP4 container compatible with the many MP4 hardware devices and media centres. This may not work with an iPod as it uses some advanced parameters in the encoding. I'll try and post up the iPod parameters another time...

 MP4Box -fps $(cat "mp4fps.fps") -add "h264_video.h264" -add "aac_audio.aac"#audio:name="LC-AAC Stereo" -itags name="source_movie_file.avi":comment="Tagged by Psyk on $(date)" -mpeg4 -new "source_movie_file.avi".mp4

The final output will be something like "Source_movie_file.avi.mp4". You can obviously change these output files and edit the tags.

Lastly, the cleanup:

 rm h264_video.h264  
 rm aac_audio.aac  
 rm mp4fps.fps

To make things easier I put all of these into a very simple shell-script.

 mencoder -profile H264ENC "$1" -o "$1".avi  
 sleep 2  
 mencoder "$1".avi -nosound -ovc copy -of rawvideo -o h264_video.h264  
 sleep 2  
 MP4Box -aviraw audio "$1".avi -out aac.raw  
 sleep 2  
 mv aac_audio.raw aac_audio.aac  
 mplayer "$1".avi -noconfig all -loop 1 -identify -nosound -vo null -nocache -frames 1 2>/dev/null | grep '^ID_VIDEO_FPS' | tail -n 1 | awk -F= '{print $2}' > mp4fps  
 .fps  
 MP4Box -fps $(cat "mp4fps.fps") -add "h264_video.h264" -add "aac_audio.aac"#audio:name="LC-AAC Stereo" -itags name="$1":comment="Tagged by lazza on $(date)" -mpeg4  
  -new "$1".mp4  
 sleep 2  
 rm h264_video.h264  
 rm aac_audio.aac  
 rm mp4fps.fps

Using Handbrake for conversion

2010-01-20T00:31:00.013+11:00

I've been playing around with HandBrake for a little while now and it's a very good and versatile encoder. One of the reasons I like it so much is that it combines two of my favourite encoders being FFmpeg and Mencoder.

One of the more challenging items is adjusting the aspect ratio of your final video. It requires some understanding of Pixel Aspect Ratio which is different to your standard aspect ratio. I won't go into detail here, but basically to change your video format from 4:3 to 16:9 you'll need to feed the PAR parameters.

Here are the rough PAR specs on standard DVD's.
NTSC 4:3 - storage resolution 720 x 480,PAR 8/9 - display resolution 640 x 480 (= 4:3 aspect ratio)
NTSC 16:9 - storage resolution 720 x 480,PAR 32/27 - display resolution 853 x 480 (= 16:9 aspect ratio)
PAL 4:3 - storage resolution 720 x 576,PAR 16/15 - display resolution 768 x 576 (= 4:3 aspect ratio)
PAL 16:9 - storage resolution 720 x 576,PAR 64/45 - display resolution 1024 x 576 (= 16:9 aspect ratio)

Under HandBrake I've been using the preset settings for doing the encode and they're working well. Here's a list of the preset settings as per 0.94 of Handbrake:

 Apple:  
 Universal: -e x264 -q 20.0 -a 1,1 -E faac,ac3 -B 160,160 -6 dpl2,auto -R 48,Auto -D 0.0,0.0 -f mp4 -X 720 --loose-anamorphic -m -x cabac=0:ref=2:me=umh:bframes=0:8x8dct=0:trellis=0:subme=6  
 iPod: -e x264 -b 700 -a 1 -E faac -B 160 -6 dpl2 -R 48 -D 0.0 -f mp4 -I -X 320 -m -x level=30:bframes=0:cabac=0:ref=1:vbv-maxrate=768:vbv-bufsize=2000:analyse=all:me=umh:no-fast-pskip=1:subme=6:8x8dct=0:trellis=0  
 iPhone & iPod Touch: -e x264 -q 20.0 -a 1 -E faac -B 128 -6 dpl2 -R 48 -D 0.0 -f mp4 -X 480 -m -x cabac=0:ref=2:me=umh:bframes=0:subme=6:8x8dct=0:trellis=0  
  AppleTV: -e x264 -q 20.0 -a 1,1 -E faac,ac3 -B 160,160 -6 dpl2,auto -R 48,Auto -D 0.0,0.0 -f mp4 -4 -X 960 --loose-anamorphic -m -x cabac=0:ref=2:me=umh:b-adapt=2:weightb=0:trellis=0:weightp=0  
 ----  
 Regular  
 Normal: -e x264 -q 20.0 -a 1 -E faac -B 160 -6 dpl2 -R 48 -D 0.0 -f mp4 --strict-anamorphic -m -x ref=2:bframes=2:subme=6:mixed-refs=0:weightb=0:8x8dct=0:trellis=0  
 High Profile: -e x264 -q 20.0 -a 1,1 -E faac,ac3 -B 160,160 -6 dpl2,auto -R 48,Auto -D 0.0,0.0 -f mp4 --detelecine --decomb --loose-anamorphic -m -x b-adapt=2:rc-lookahead=50  
 ----  
 Legacy  
 Classic: -b 1000 -a 1 -E faac -B 160 -6 dpl2 -R 48 -D 0.0 -f mp4  
 AppleTV Legacy: -e x264 -b 2500 -a 1,1 -E faac,ac3 -B 160,160 -6 dpl2,auto -R 48,Auto -D 0.0,0.0 -f mp4 -4 --strict-anamorphic -m -x ref=1:subme=5:me=umh:no-fast-pskip=1:cabac=0:weightb=0:8x8dct=0:trellis=0  
 iPhone Legacy: -e x264 -b 960 -a 1 -E faac -B 128 -6 dpl2 -R 48 -D 0.0 -f mp4 -I -X 480 -m -x level=30:cabac=0:ref=1:analyse=all:me=umh:no-fast-pskip=1:psy-rd=0,0:bframes=0:subme=6:8x8dct=0:trellis=0  
 iPod Legacy: -e x264 -b 1500 -a 1 -E faac -B 160 -6 dpl2 -R 48 -D 0.0 -f mp4 -I -X 640 -m -x level=30:bframes=0:cabac=0:ref=1:vbv-maxrate=1500:vbv-bufsize=2000:analyse=all:me=umh:no-fast-pskip=1:psy-rd=0,0:subme=6:8x8dct=0:trellis=0

The Normal and High-Profile settings should work just find on a PS3.

As an example, here are the details on a source DVD I'm encoding for the kids:

+ title 1:
+ vts 0, ttn 0, cells 0->0 (0 blocks)
+ angle(s) 0
+ duration: 00:39:46
+ size: 720x576, pixel aspect: 16/15, display aspect: 1.33, 25.000 fps
+ autocrop: 0/0/12/6
+ chapters:
+ 1: cells 0->0, 0 blocks, duration 00:39:46
+ audio tracks:
+ 1, Unknown (AC3) (2.0 ch) (iso639-2: und), 48000Hz, 224000bps
+ subtitle tracks:
+ combing detected, may be interlaced or telecined

The aspect is 4:3 with a pixel aspect of 16/15.

Another way to pick up the aspect and Pixel Aspect Ratio (PAR) parameters is to use FFmpeg.
For example, you could do something like ffmpeg -i movie.vob :



$ffmpeg -i movie.vob  
 Seems stream 0 codec frame rate differs from container frame rate: 59.94 (60000/1001) -&gt; 29.97 (30000/1001) 
 Input #0, mpeg, from 'mvoie.vob': 
  Duration: 00:03:08.18, start: 0.060000, bitrate: 6939 kb/s 
   Stream #0.0[0x1e0]: Video: mpeg2video, yuv420p, 720x480 [PAR 8:9 DAR 4:3], 8000 kb/s, 29.97 tbr, 90k tbn, 59.94 tbc 
   Stream #0.1[0xa0]: Audio: pcm_s16be, 48000 Hz, stereo, s16, 1536 kb/s 
 At least one output file must be specified

You'll note the PAR is 8:9 and DAR is 4:3 which matches an NTSC DVD as per above. Applying a
--pixel-aspect 32:27 in Handbrake would change this to a Widescreen format.

By using HandBrake I was able to re-encode to a compliant MP4 with the correct wide-screen aspect display. The size of the file was slightly larger and the resolution increased slightly from 720x576 (or 704 with cropping) to 1001x576 (with cropping) in Widescreen format. Very handy tool this HandBrake :)

Here's the command I used:

 HandBrakeCLI -v -i source.avi -o target.mp4 --deinterlace --size 700 -e x264 -a 1 -E faac -B 160 -6 dpl2 -R 48 -D 0.0 -f mp4 --custom-anamorphic --pixel-aspect 64:45 -m -x ref=2:bframes=2:subme=6:mixed-refs=0:weightb=0:8x8dct=0:trellis=0

Ripping audio from video files with mplayer

2010-01-01T10:27:00.001+11:00

Here's a quick fire way to get that sound clip, soundtrack or whatever it is into an audio format from a video file. For example, on youtube, there might be a good video clip with a great soundtrack, and the video is nothing more than just a fixed image and/or you want to stick on your iPod/iPhone etc. Or you might want to get an audio sample for use in a ring tone etc.

Below is a script that I use to rip audio files out of pretty much any video.

 #!/bin/bash  
 echo "Ripping audio......."   
 mplayer -nocorrect-pts -vo null -vc null -ao pcm:fast:file=audiodump.wav "$1"   
 normalize -v audiodump.wav   
 faac -b 256 -c 48000 --mpeg-vers 4 -o audiodump.aac audiodump.wav

The first line uses mplayer to dump the audio to a standard wav file called "audiodump.wav". The "-nocorrect-pts" corrects errors in some certain video files.

I then normalize the audio file via the "normalize" command (make sure you have the normalize package installed).

It's then encoded into AAC format using faac. You could easily do an mp3 here by using lame. For example, change the line starting with faac to something like:


lame --preset extreme audiodump.wav audiodump.mp3

Creating animated GIFS with mplayer

2009-11-08T22:31:00.006+11:00

Looking for a quick fire way to create an animated GIF from sections of video's I didn't need to look any further than the multimedia swiss-army knife of the linux world, mplayer/mencoder and ffmpeg.

mplayer -ao null -loop 0 -ss 0:11:22 -endpos 5 file.avi

First is to playback the video and find the section you want. Tweak the "-ss" parameter to find the starting point and the "-endpos" for the finishing point of your video clip.

 mplayer file.avi -ao null -ss 0:11:22 -endpos 5 -vo gif89a:fps=25:output=animated.gif -aspect 16/9 -vf scale=-2:100

Use mplayer again to write out the animated GIF file. Adjust the "scale" parameter to adjust the size of the animated GIF which is useful if you're going to use it as a signature for web forum etc.

Encoding for the PS3, and potentially other MP4 compatible devices

2009-08-12T21:04:00.011+10:00

Playing around with encoding settings for playback on the PS3 led to alot of trial and error and tweaking. It seems to be easier to find a way on how "not" to do it than how to do it as there's so many utilities and tools out there.

I stuck with mencoder/mplayer as I've used it over the years and extremely comfortable with how it behaves and works. The only real challenge is that it can't really mux properly into an MP4 container that other devices can properly read (i.e. it can read it's own 'corrupted' MP4 but nothing else can!). If that's mencoder's only blemish on an otherwise powerful video encoding tool, then that's really nothing at all.

So the process is to use mencoder, faac for some additional flexibility in bitrates, normalize for levelling out the audio and then MP4Box to mux it all together into a universally compliant MP4 container.

If you're running Fedora, you'll need to install these applications as follows:

yum install mplayer mencoder gpac normalize faac

Firstly, create edit your ~/.mplayer/mencoder.conf and add the following:

 [PS3]  
 vf=pullup,softskip,pp=fd,scale=:-10,hqdn3d,harddup  
 ovc=x264=yes   
 x264encopts=level_idc=41:crf=18:bframes=16:b_pyramid=yes:partitions=all:threads=auto:frameref=4:mixed_refs=yes:weight_b=yes  
 oac=pcm=yes  
 of=rawvideo=yes  
 sws=9  
 aspect=16/9

Then create a file called "enc-to-ps3.sh" in your favourite editer (e.g. gedit). This will be the executable filename and you can choose any name you like.

Add the following lines:

mplayer -fps 29.97 -nocorrect-pts -vo null -vc null -ao pcm:fast:waveheader:file="$1"_temp.wav "$1" && \
normalize -a 0.5 -v "$1"_temp.wav && \
faac -b 256 -c 48000 --mpeg-vers 4 -o "$1"_temp.aac "$1"_temp.wav && \
mencoder -profile PS3 "$1" -ofps 29.97 -o "$1"_temp.264 && \
MP4Box -fps 29.97 -add "$1"_temp.264 -add "$1"_temp.aac "$1".mp4 && \
rm "$1"_temp.wav && \
rm "$1"_temp.264 && \
rm "$1"_temp.aac

Make "enc-to-ps3.sh" executable by running:

chmod +x enc-to-ps3.sh

So what does it do?

mplayer -vc null -vo null -ao pcm:fast:waveheader:file=temp_"$1"_temp.wav "$1" && \

This line rips the audio from your video track and writes it out to a wave file. The name of the 'temp' file is your original file prefixed and suffixed with "temp" for easy identification.

normalize -v temp_"$1"_temp.wav && \

Quickly normalize the audio to prevent clipping and clean up the sound. I have a lot of clips where the audio is too load/soft and this helps adjust that.

faac -b 256 -c 48000 --mpeg-vers 4 --tns -o temp_"$1"_temp.aac temp_"$1"_temp.wav && \

This line encodes the resulting normalized WAV audio file into AAC at 256kb bitrate.

mencoder -profile PS3 "$1" -o temp_"$1"_temp.264 && \

Mencoder is used here to encode the original video using the profile settings we created earlier. This is the lengthiest piece of the encoding process depending on your source, so grab a coffee and come back in a little while. The output is a raw h.264 (x264) stream.

MP4Box -add temp_"$1"_temp.264 -add temp_"$1"_temp.aac "$1".mp4 && \

Using MP4Box from the gpac utilities, the resulting h.264 stream and AAC audio file are then muxed together into an MP4 file using the detected frame rate of the original video to keep things in sync.

And then your done.

Usage is as easy as:

enc-to-ps3.sh <name_of_video>

The resulting file should be playable on the PS3...

The script is quite rudimentary, but the basis of keeping it like this allows understanding of the process and then empower you to tweak settings if you so desire.

Windows Media Centre, Linux and .mkv files

2009-02-10T23:12:00.010+11:00

I've been running Vista MCE for sometime now and while it's doing the job, I would've really liked to be running Mythdora... That'll be a project for another time.

In any case, I'm running ffdshow and the haali splitter. Haali is used to handle the MKV's (Matroska) and MP4's whereas ffdshow will pretty much handle any other format including Divx/XviD and anything else. As a result it's really all you need to prevent serious 'codec-hell' in Windows.

First and foremost, a lot of the credit needs to go to this UbuntuForums thread where the original poster walks through a process of ripping DVD's to MKV format.

Also, the following method is an alternative to my previous post in regards to encoding dvr-ms files into MP4 containers for PS3 compatible playback.

If you want to automate things and not really worry about tweaking or understanding the process (which will lend itself to allow you to tweak), then you can use something like avidemux.

However, I like tweaking and tinkering with the settings and understanding the process, so the manual method described in the Ubuntu link works for me.

One of the things I do, is use a different set of parameters for the H264/x264 encode in addition to using AAC encoded audio as opposed to ogg. Here is the 'to-mkv' profile entry in my ~/.mplayer/mencoder.conf

[to-mkv]
profile-desc="        Makes a x264 raw video stream for muxing into MKV"
lavdopts=threads=2
ovc=x264=yes 
x264encopts=keyint_min=24:keyint=240:crf=18:qp_min=5:qp_max=51:qcomp=0.75:me=umh:subq=6:frameref=6:bframes=3:ip_factor=1.25:
pb_factor=1.33:deblock=-1,-1:nopsnr:b_pyramid=yes:brdo=yes:weight_b=yes:mixed_refs=yes:bime=yes:chroma_me=yes:partitions=all
:trellis=1:8x8dct=yes:threads=auto
oac=copy=yes
of=rawvideo=yes

The resulting mencoder command is:

$mencoder -profile to-mkv -vf scale=720:-10,harddup <video_source> -o <output_video>.264

The above command outputs only a raw h.264 video format with no audio. I'm asking it to convert "video_source" into a DVD scaled (720x576/480 etc.) h.264 "output_video".264. The ".264" extension helps identify it as a raw h.264 video and is of very good quality depending on your source. In my case they're usually TV recorded hi-def channels of 1920x1080 MPG format and AC3 audio.

Mux the .264 file into an MP4 container (the mkv utilities handles the mp4 format a little better):

$MP4Box -add output_video.264 output_video.mp4

If you haven't already separated the audio from the video, then you'll need to do that. Once done, normalize the audio file by doing:

$normalize audio.wav

Once you have separated and normalized the audio, encode to AAC format with the following:

$faac audio.wav -o audio.aac

Once you have the MP4 and AAC files you can use the mkv gui tools to mux them all into the Matroska (MKV) container. You can start the gui utility by running:

$mmg

...and that's all there is too it in creating mkv files in linux. I deliberately left out a lot of detail and only high-lighted some of the key points in the process in doing it as a lot of it is documented elsewhere.

Refer to the Ubuntuforums page for more detail: http://ubuntuforums.org/showthread.php?t=273635

Joining AVI files together

2009-02-01T14:03:00.003+11:00

Here's a simple way to join two AVI files together. The AVI's should be the same in regards to type (i.e. XvID/DivX, audio bitrate etc.):


$cat movie1.avi movie2.avi movie3.avi > movie_temp.avi
$mencoder -forceidx -ovc copy -oac copy movie_temp.avi -o movie_final.avi

Protecting ones data on removable drives

2009-01-25T16:28:00.026+11:00

With large capacity drives and USB sticks getting cheaper for more capacity, the tendency to carry more and more around with you increases. With so much information being digitzed, losing a USB-key or a small passport sized backup USB drive would be akin to losing your wallet, keys and credit cards...

Truecrypt

I've been looking at Truecrypt which is an open-source encryption application which can encrypt entire partitions, drives and/or create a virtual drive as a file on an existing file-system. It has a rich set of encryption algorithms and can also chain them together. For example, it can chain AES256 (Rijndael), Twofish and Serpent the latter two being finalists for AES encryption standard. There's plenty of information out there on the strengths of these ciphers, so I won't go into detail here...

However, if you are interested, you can read NIST's entire 116 page report on the AES encryption round and their decisions to choose Rijndael. In summary none have any known security attacks let alone cracked (it would've never made it this far if it had been cracked before :) ). The strength testing they used was on purposely weakened versions of the algorithms - and usually provided by the team themselves. Twofish and Serpent came out slightly stronger in the 'weakend' versions submitted for analysis, but didn't fare so well in things like implementation (i.e. smartcards, memory footprint for asic implementation etc.).

Unfortunately, Truecrypt only comes with packages ready for SuSE and Ubuntu, however the source is available for download.

A guide is available here if you want to build and compile it yourself.
http://arbitness.blogspot.com/2008/07/installing-truecrypt-on-fedora-9-howto.html

There's good information here as well:
http://penguinenclave.blogspot.com/2008/12/truecrypt-61-install-guide-for-fedora.html

However, what is nice is that there are binaries for Fedora available from here:
http://www.lfarkas.org/linux/packages/fedora/9/

Since I run Fedora 9 x86_64 version, there was no binary, so I downloaded the srpm file from here:
http://www.lfarkas.org/linux/packages/fedora/9/SRPMS/

The basic process for building it from SRPM is as follows:


cd ~/rpmbuild/SRPMS

wget -c http://www.lfarkas.org/linux/packages/fedora/9/SRPMS/truecrypt-6.1-1.fc9.src.rpm

rpmbuild --rebuild truecrypt-6.1-1.fc9.src.rpm

cd ~/rpmbuild/RPMS/x86_64/

su -c"rpm -Uvh truecrypt-6.1-1.fc9.src.rpm"

Truecrypt is easy to use. Once it's up and running it has a gui for mounting encrypted volumes and for creating them. Refer to the documentation at Truecrypts website.

PGP and GnuPG

Finally the other method I use is PGP or the linux implementation of it called GPG.

I've primarily been looking GPG to do symmetric cipher encryption on a single file, which basically asks for a passphrase prior to encryption. You just then need the passphrase to decrypt the file. Not as a secure as using public/private key encryption of course...

gpg -v --cipher-algo TWOFISH --symmetric --output <encrypted file> <name of file you want encrypted>

GPG defaults to 'CAST5' (or CAST-128) so I've forced it to use TWOFISH in this implementation. GPG supports a few ciphers, and you can check what's available on your system by running 'gpg --version'.

$ gpg --version
gpg (GnuPG) 1.4.9
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, RSA-E, RSA-S, ELG-E, DSA
Cipher: 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

The best method to modify the cipher algorithm to use is to modify the ~/.gnupg/gpg.conf file and add the following items at the end. Modifying the 'personal-cipher-preferences' option as shown defaults TWOFISH as the cipher to use first.

personal-cipher-preferences TWOFISH AES256 AES192 AES BLOWFISH CAST5 3DES
personal-digest-preferences SHA256 SHA1 SHA512 SHA384 SHA224 RIPEMD160 MD5
personal-compress-preferences ZIP ZLIB BZIP2 Z0

You probably don't need to adjust the digest-preferences unless you're wanting to use it with private/public keys etc.

To decrypt you would just type:

gpg <name of encrypted file>

If you want to encrypt multiple files, it's a lot easier to zip or tar them and then encrypt the final compressed file.

Windows users can use the win32 implementation of GnuPG available from here:
http://www.pgpi.org/download/gnupg/

A GUI front-end to GPG called WinPT is available from here:
http://winpt.gnupt.de/int/

Otherwise PGP is compatible.
http://www.pgp.com/downloads/desktoptrial/desktoptrial2.html#trial_or_freeware

Quick update, Truecrypt is now available in Fedora 11 called Realcrypt from the rpmfusion repositories. Make sure you have rpmfusion enabled and then install via yum.


yum install realcrypt

Mediatomb and the PS3 with mencoder

2009-01-07T00:02:00.010+11:00

After some mucking around, I'm successfully streaming from my Fedora/Linux PC to the PS3 over wireless.

A great guide to get this going is from the mediatomb website and more specifically the transcoding section:

http://mediatomb.cc/dokuwiki/transcoding:transcoding

Take note to adjust your config.xml.

The site describes using ffmpeg and/or VLC as the transcoding engine, but I've always liked mencoder as I'm more familiar with it and have a lot of profiles defined already in ~/.mplayer/mencoder.conf.

In regards to the mediatomb config.xml file (located at ~/.mediatomb/config.xml), like with most things, it needs to be heavily customised. I have a stack of AVI's that aren't in DivX/XviD format (which is what mediatomb tells the PS3 by default) and I have some AVI files which ffmpeg borks on... however, mencoder works very well on all these files. As a result, I transcode everything rather than pass the DivX/XviD to the PS3 - and surprisingly it streams well over wireless.

So here's the "~/bin/mencoder-tr" script I use that mediatomb calls to transcode the video's from the PC to a format the PS3 can understand. In this case I'm transcoding to a DVD compliant MPEG2 stream and AC3 audio in 720x576 format:

exec mencoder \
-oac lavc -ovc lavc \
-of mpeg \
-mpegopts format=dvd:tsaf \
-vf softskip,scale=720:576,hqdn3d,harddup \
-srate 48000 \
-af lavcresample=48000:volnorm=2 \
-ofps 25 \
-lavcopts vcodec=mpeg2video:vrc_buf_size=1835:vrc_maxrate=5120:vstrict=0:keyint=15:vbitrate=5120:acodec=ac3:abitrate=192:autoaspect \
"$1" -o "$2"

The "$1" and "$2" designations are mediatomb specific...

Lastly, I have the following "transcoding" section in my "~/.mediatomb/config.xml" file:

<transcoding enabled="yes">
   <mimetype-profile-mappings> 
     <transcode mimetype="audio/mpeg" using="vlcwav"/>   
     <transcode mimetype="video/x-flv" using="transvideo"/>
     <transcode mimetype="video/mp4" using="transvideo"/>
     <transcode mimetype="video/x-quicktime" using="transvideo"/>
     <transcode mimetype="application/ogg" using="vlcwav"/>
     <transcode mimetype="audio/x-ms-wma" using="vlcwav"/>
     <transcode mimetype="audio/x-ms-asf" using="vlcwav"/>
     <transcode mimetype="audio/x-flac" using="ffmpegwav"/>
     <transcode mimetype="audio/x-aac" using="vlcwav"/>
     <transcode mimetype="audio/mp4" using="vlcwav"/>
     <transcode mimetype="video/x-msvideo" using="transvideo"/>
     <transcode mimetype="video/x-ms-wmv" using="transvideo"/>
     <transcode mimetype="video/mpeg" using="mpeg2trans"/>
     <transcode mimetype="video/x-matroska" using="transvideo"/>
     <transcode mimetype="image/jpeg" using="rescalejpeg"/>
   </mimetype-profile-mappings>
   <profiles>
     <profile name="transvideo" enabled="yes" type="external">
       <mimetype>video/mpeg</mimetype>
       <accept-url>no</accept-url>
       <first-resource>yes</first-resource>
       <agent command="mencoder-tr" arguments="%in %out"/>
       <buffer size="10485760" chunk-size="262144" fill-size="524288"/>
     </profile>
     <profile name="mpeg2trans" enabled="yes" type="external">
       <mimetype>video/mpeg</mimetype>
       <accept-url>no</accept-url>
       <first-resource>yes</first-resource>
       <hide-original-resource>yes</hide-original-resource>
       <agent command="mencoder-tr" arguments="%in %out"/>
       <buffer size="28800000" chunk-size="512000" fill-size="120000"/>
     </profile>
     <profile name="ffmpegwav" enabled="yes" type="external">
       <use-chunked-encoding>no</use-chunked-encoding>
       <mimetype>audio/wav</mimetype>
       <accept-url>no</accept-url>
       <first-resource>yes</first-resource>
       <agent command="ffmpegaudio" arguments="%in %out"/>
       <buffer size="1048576" chunk-size="131072" fill-size="262144"/>
     </profile>
     <profile name="vlcwav" enabled="yes" type="external">
   <use-chunked-encoding>no</use-chunked-encoding>
       <mimetype>audio/wav</mimetype>
       <accept-url>yes</accept-url>
       <first-resource>yes</first-resource>
       <agent command="vlc" arguments="-I dummy %in --sout #transcode{acodec=s16l,ab=192,channels=2}:standard{access=file,mux=wav,dst=%out} vlc:quit"/>
       <buffer size="512000" chunk-size="32000" fill-size="64000"/>
     </profile>
     <profile name="rescalejpeg" enabled="yes" type="external">
       <mimetype>image/jpeg</mimetype>
       <accept-url>no</accept-url>
       <first-resource>yes</first-resource>
       <accept-ogg-theora>no</accept-ogg-theora>
       <agent command="convert" arguments="-size 1080x720 %in -auto-orient -resize 1080x720 +profile '*' %out"/>
       <buffer size="50000" chunk-size="100" fill-size="100"/>
     </profile>
    </profiles>
</transcoding>

Note the "mencoder-tr" entries that allows mediatomb to execute the required script to perform the transcoding.

The advantage of this method means that using mencoder's engine it can read pretty much any format and convert to any format, in this case a standard DVD mpeg stream.

As requested, here is my entire ~/.mediatomb/config.xml

<?xml version="1.0" encoding="UTF-8"?>
<config version="1" xmlns="http://mediatomb.cc/config/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://mediatomb.cc/config/1 http://mediatomb.cc/config/1.xsd">
  <server>
    <ui enabled="yes">
      <accounts enabled="no" session-timeout="30">
        <account user="mediatomb" password="mediatomb"/>
      </accounts>
    </ui>
    <name>MediaTomb</name>
    <udn>uuid:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</udn>
    <home>/home/xxxxxxxxxxxxx</home>
    <webroot>/usr/share/mediatomb/web</webroot>
    <storage>
      <sqlite3 enabled="yes">
        <database-file>mediatomb.db</database-file>
      </sqlite3>
      <mysql enabled="no">
        <host>localhost</host>
        <username>mediatomb</username>
        <database>mediatomb</database>
      </mysql>
    </storage>
    <protocolInfo extend="yes"/>
  </server>
  <import hidden-files="no">
    <scripting script-charset="UTF-8">
      <common-script>/usr/share/mediatomb/js/common.js</common-script>
      <playlist-script>/usr/share/mediatomb/js/playlists.js</playlist-script>
      <virtual-layout type="builtin">
        <import-script>/usr/share/mediatomb/js/import.js</import-script>
      </virtual-layout>
    </scripting>
    <mappings>
      <extension-mimetype ignore-unknown="no">
        <map from="mp3" to="audio/mpeg"/>
        <map from="ogg" to="application/ogg"/>
        <map from="asf" to="video/x-ms-asf"/>
        <map from="asx" to="video/x-ms-asf"/>
        <map from="wma" to="audio/x-ms-wma"/>
        <map from="wax" to="audio/x-ms-wax"/>
        <map from="wmv" to="video/x-ms-wmv"/>
        <map from="wvx" to="video/x-ms-wvx"/>
        <map from="wm" to="video/x-ms-wm"/>
        <map from="wmx" to="video/x-ms-wmx"/>
        <map from="m3u" to="audio/x-mpegurl"/>
        <map from="pls" to="audio/x-scpls"/>
        <map from="ts" to="video/mpeg"/>
    <map from="vob" to="video/mpeg"/>
    <map from="wav" to="audio/wav"/>
    <map from="mpg" to="video/mpeg"/>
    <map from="aac" to="audio/x-aac"/>
    <map from="m4a" to="audio/mp4"/>
    <map from="mkv" to="video/x-matroska"/>
    <map from="mov" to="video/x-quicktime"/> 
    <map from="flv" to="video/x-flv"/>
    <map from="divx" to="video/x-divx"/>
      </extension-mimetype>
      <mimetype-upnpclass>
        <map from="audio/*" to="object.item.audioItem.musicTrack"/>
        <map from="video/*" to="object.item.videoItem"/>
        <map from="image/*" to="object.item.imageItem"/>
      </mimetype-upnpclass>
      <mimetype-contenttype>
        <treat mimetype="audio/mpeg" as="mp3"/>
        <treat mimetype="application/ogg" as="ogg"/>
        <treat mimetype="audio/x-flac" as="flac"/>
        <treat mimetype="image/jpeg" as="jpg"/>
        <treat mimetype="audio/x-mpegurl" as="playlist"/>
        <treat mimetype="audio/x-scpls" as="playlist"/>
        <treat mimetype="audio/x-wav" as="pcm"/>
        <treat mimetype="audio/L16" as="pcm"/>
      </mimetype-contenttype>
    </mappings>
  </import>
<transcoding enabled="yes">
    <mimetype-profile-mappings>  
      <transcode mimetype="audio/mpeg" using="vlcwav"/>    
      <transcode mimetype="video/x-flv" using="transvideo"/>
      <transcode mimetype="video/mp4" using="transvideo"/>
      <transcode mimetype="video/x-quicktime" using="transvideo"/>
      <transcode mimetype="application/ogg" using="vlcwav"/>
      <transcode mimetype="audio/x-ms-wma" using="vlcwav"/>
      <transcode mimetype="audio/x-ms-asf" using="vlcwav"/>
      <transcode mimetype="audio/x-flac" using="ffmpegwav"/>
      <transcode mimetype="audio/x-aac" using="vlcwav"/>
      <transcode mimetype="audio/mp4" using="vlcwav"/>
      <transcode mimetype="video/x-msvideo" using="transvideo"/>
      <transcode mimetype="video/x-ms-wmv" using="transvideo"/>
      <transcode mimetype="video/mpeg" using="mpeg2trans"/>
      <transcode mimetype="video/x-matroska" using="transvideo"/>
      <transcode mimetype="image/jpeg" using="rescalejpeg"/>
    </mimetype-profile-mappings>
    <profiles>
      <profile name="transvideo" enabled="yes" type="external">
        <mimetype>video/mpeg</mimetype>
        <accept-url>no</accept-url>
        <first-resource>yes</first-resource>
        <agent command="mencoder-tr" arguments="%in %out"/>
    <buffer size="10485760" chunk-size="262144" fill-size="524288"/>
      </profile>
      <profile name="mpeg2trans" enabled="yes" type="external">
        <mimetype>video/mpeg</mimetype>
        <accept-url>no</accept-url>
        <first-resource>yes</first-resource>
        <hide-original-resource>yes</hide-original-resource>
        <agent command="mencoder-tr" arguments="%in %out"/>
        <buffer size="28800000" chunk-size="512000" fill-size="120000"/>
      </profile>
      <profile name="ffmpegwav" enabled="yes" type="external">
        <use-chunked-encoding>no</use-chunked-encoding>
        <mimetype>audio/wav</mimetype>
        <accept-url>no</accept-url>
        <first-resource>yes</first-resource>
        <agent command="ffmpegaudio" arguments="%in %out"/>
        <buffer size="1048576" chunk-size="131072" fill-size="262144"/>
      </profile>
      <profile name="vlcwav" enabled="yes" type="external">
    <use-chunked-encoding>no</use-chunked-encoding> 
        <mimetype>audio/wav</mimetype>
        <accept-url>yes</accept-url>
        <first-resource>yes</first-resource>
        <agent command="vlc" arguments="-I dummy %in --sout #transcode{acodec=s16l,ab=192,channels=2}:standard{access=file,mux=wav,dst=%out} vlc:quit"/>
        <buffer size="512000" chunk-size="32000" fill-size="64000"/>
      </profile>
      <profile name="rescalejpeg" enabled="yes" type="external">
        <mimetype>image/jpeg</mimetype>
        <accept-url>no</accept-url>
        <first-resource>yes</first-resource>
        <accept-ogg-theora>no</accept-ogg-theora>
        <agent command="convert" arguments="-size 1080x720 %in -auto-orient -resize 1080x720 +profile '*' %out"/>
        <buffer size="50000" chunk-size="100" fill-size="100"/>
      </profile>
      <profile name="vlcyoutube" enabled="no" type="external">
        <mimetype>video/mpeg</mimetype>
        <accept-url>yes</accept-url>
        <first-resource>yes</first-resource>
        <accept-ogg-theora>yes</accept-ogg-theora>
        <agent command="vlc" arguments="-I dummy %in --sout #transcode{vcodec=mp2v,vb=4096,canvas-width=448,canvas-height=252,acodec=mpga,ab=64,samplerate=44100,channels=1}:standard{access=file,mux=ts,dst=%out} vlc:quit"/> 
        <buffer size="14400000" chunk-size="256000" fill-size="80000"/>
      </profile> 
    </profiles>
  </transcoding>
</config>

Note the bold and italicized regions. The home designation should be your "/home/(username)/.mediatomb" directory, and the uuid is a unique 32 alpha-numeric string.

Lastly, my config.xml probably needs a lot of cleaning up also. I basically heavily modified the one avialable from the mediatomb website and didn't go and clean it up. Fo example, I could probably delete the "vlcyoutube" transcoding trigger as I'm using the transvideo one instead.

Building ffmpeg from SVN

2008-12-31T10:43:00.003+11:00

A couple of quick notes to build ffmpeg from svn under Fedora.

$svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg
$cd ffmpeg
$./configure --prefix=/usr --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --enable-libdc1394 --enable-libfaac --enable-libfaad --enable-libgsm --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-x11grab --enable-avfilter --enable-avfilter-lavf --enable-postproc --enable-swscale --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-optimizations --disable-stripping
$make
$su -
Password:
#make install

I basically used the same configure parameters (or as close too) as the original rpm packaged fedora build from rpmfusion. Satisfy the missing library requests if any errors come up after running the 'configure' script. Typically it will just be installing the 'devel' versions of the package.

Encoding to h.264/x264 with mencoder and ffmpeg

2008-11-02T23:25:00.005+11:00

Looking at ways to compress and encode movies/videos to a more compact size with minimal noticeable quality loss, led a search to some solid and well developed open source utilities. I run both Windows (home theatre) and Linux on a separate workstation so an open source solution made it much more flexible to use on both platforms.

Windows dvr-ms format is an absolute pain. Don't know why they don't just stick with an open standard like Matroska :) anyways...

First things first. Mencoder etc. has a hard time with DVR-MS files, so strip it using ffmpeg:

ffmpeg -y -i movie.dvr-ms -vcodec copy -acodec copy -f dvd movie.mpg

This extracts the mpg from the dvr-ms file to a more user-friendly format. Yes you'll lose the metadata information...

The following mencoder command creates a high quality h264 movie scaled to roughly 720x480 with aac audio (it all needs to be on one line).

mencoder -vf pullup,softskip,pp=fd,scale=720:-10,hqdn3d,harddup -lavdopts threads=2 -ovc x264 -x264encopts bitrate=1200:subq=6:frameref=6:qcomp=0.8:8x8dct:b_pyramid:weight_b:me=umh:partitions=p8x8,i4x4:bime:brdo:nodct_decimate:trellis=1:direct_pred=auto:level_idc=30:nocabac:threads=auto -oac faac -faacopts br=128:raw:mpeg=4:tns:object=2 -of lavf -lavfopts format=mp4 -sws 9 -ofps 24000/1001 -srate 48000 -alang en movie.mpg -o movie.mp4

Rather than typing all that out, you can use mencoder's "-profile" parameter by creating a mencoder.conf file in ~/.mplayer/mencoder.conf with the following contents:

[h264mp4]
profile-desc="H.264 MP4"
vf=pullup,softskip,pp=fd,scale=720:-10,hqdn3d,harddup
lavdopts=threads=2
ovc=x264=yes
x264encopts=bitrate=1200:subq=6:frameref=6:qcomp=0.8:8x8dct=yes:b_pyramid=yes:weight_b=yes:me=umh:partitions=p8x8,i4x4:bime=yes:brdo=yes:nodct_dec
imate=yes:trellis=1:direct_pred=auto:level_idc=30:nocabac=yes:threads=auto
oac=faac=yes
faacopts=br=128:raw=yes:mpeg=4:tns=yes:object=2
of=lavf=yes
lavfopts=format=mp4
sws=9
ofps=24000/1001
srate=48000

In which case you can then run :

mencoder -profile h264mp4 movie.mpg -o movie.mp4

This creates a h.264 video stream and an aac encoded audio stream perfect for muxing into an MP4 container. Unfortunately mencoder's MP4 muxing is a little broken and will more than likely not play correctly in anything other than mplayer.

To mux it back into a more compliant MP4 format, run ffmpeg again:

ffmpeg -y -i movie.mp4 -vcodec copy -acodec copy movie-remuxed.mp4

Make sure you use a different name for the mp4 output from ffmpeg above so you don't overwrite the original input file.