11.04.08

Simple Transcoding Script for Linux - dvd2mkv

Posted in programming at 13:24 by dalore

I wrote a simple bash script that would let me convert my anime DVDs to a digital file for ease of use. Now in the past I’ve used tools like nandub and gordian knot to create high quality xvids from DVDs on windows. This time I wanted some better and on Linux.

I recently heard about a new container format called Matroska, that has a few new features over the avi container. Some of the features are:

• Complete open format
• Ability to store multiple audio tracks (anime typically comes in both english and japanese, a purist would watch it in japanese, but I wanted the option)
• Soft subtitles (with avi you could burn in the subtitles or provide it as a separate file, but with matroska you have the option of turning it off and on and having it all in 1 file)
• Chapter support (say you want to skip the beginning credits without having to fastforward to the exact spot, with chapters you can just say next chapter and it starts playing for the marker as specificed and since most DVDs comes with chapters I didn’t want to lose this information

Now I could have gone with the OGG but Matroska is superior is features and so decided to go with that.

For the video codec I chose to go with the latest high quality standard of H.264 (otherwise known as MPEG-4 AVC, not to be confused with MPEG-4 SP/ASP which is what is used in xvid/dvix etc). There is an open source library for encoding H.264 called x264, and as of August 2008, x264 implements more H.264 features than any other encoder. It also has won many a codec shoot-out competition.

As for the audio codec, I chose not to do any transcoding of the audio (as one knows everytime lossy transcoding is done, artifacts can be introduced and quality is lost). I will just take the audio tracks (which are usually AC3), and just put them in the container as is.

Here is my simple bash script (note it doesn’t do any error checking), I have called it dvd2mkv:

#!/bin/bash

TRACK=$1
MOVIE_TITLE=$2
BITRATE=$3
DVDDEV=$4
RIPPATH=.
SID=0
HEXSID=0x2${SID}
AID1=128
AID2=129
OUTFILE=${RIPPATH}/${MOVIE_TITLE}.ac3.mkv
DEF_LANG=eng
AID2_LANG=jpn

if [ -z "$MOVIE_TITLE" ] ; then
  echo "USAGE: $0 trackno title [bitrate] [dvddev]"
  echo "EXAMPLE: $0 2 elfien.lied.s01e01 2000 /dev/dvd"
  echo "bitrate defaults to 2000"
  echo "dvddev defaults to /mnt/dvd"
  exit 1
fi

if [ -z "$BITRATE" ] ; then
  BITRATE=2000
fi

if [ -z "$DVDDEV" ] ; then
  DVDDEV=/mnt/dvd
fi

VIDEO_FPS=`lsdvd -x -t ${TRACK} ${DVDDEV} | grep FPS | cut -f6 -d' ' | cut -f1 -d,`

mplayer dvd://${TRACK} -dumpstream -dumpfile ${RIPPATH}/movie.vob -dvd-device ${DVDDEV}

dvdxchap -t ${TRACK} ${DVDDEV} > ${RIPPATH}/chapters.txt

cp ${DVDDEV}/VIDEO_TS/VTS_01_0.IFO ${RIPPATH}/

tccat -i ${RIPPATH}/movie.vob -L | tcextract -x ps1 -t vob -a ${HEXSID} > ${RIPPATH}/subs-${SID}
subtitle2vobsub -o ${RIPPATH}/vobsubs -i ${RIPPATH}/VTS_01_0.IFO -a ${SID} < ${RIPPATH}/subs-${SID}

mplayer ${RIPPATH}/movie.vob -aid ${AID1} -dumpaudio -dumpfile ${RIPPATH}/audio${AID1}.ac3
mplayer ${RIPPATH}/movie.vob -aid ${AID2} -dumpaudio -dumpfile ${RIPPATH}/audio${AID2}.ac3

mencoder ${RIPPATH}/movie.vob -vf pullup,softskip,harddup -nosound -ovc x264 -x264encopts bitrate=${BITRATE}:subq=1:bframes=3:b_pyramid:weight_b:turbo=1:threads=auto:pass=1 -of rawvideo -o /dev/null

mencoder ${RIPPATH}/movie.vob -vf pullup,softskip,harddup -nosound -ovc x264 -x264encopts bitrate=${BITRATE}:subq=5:8x8dct:frameref=2:bframes=3:b_pyramid:weight_b:threads=auto:pass=2 -of rawvideo -o ${RIPPATH}/movie.264

AID_CMD="${RIPPATH}/audio${AID1}.ac3"
if [ -s "${RIPPATH}/audio${AID2}.ac3" ] ; then
  AID_CMD="${AID_CMD} --language 0:${AID2_LANG} ${RIPPATH}/audio${AID2}.ac3"
fi

mkvmerge --title "${MOVIE_TITLE}" --default-language ${DEF_LANG} -o ${OUTFILE} --default-duration 0:${VIDEO_FPS}fps --chapters ${RIPPATH}/chapters.txt ${RIPPATH}/movie.264 ${AID_CMD} vobsubs.idx

# cleanup
rm -rf movie.vob chapters.txt VTS_01_0.IFO vobsubs.sub vobsubs.idx subs-${SID} audio${AID1}.ac3 audio${AID2}.ac3 divx2pass.log movie.264

Basically you run it by giving it the required arguments of the track number and the title you would like for it (note it will automatically prepend .ac3.mkv to your title when saving the file). It will automatically find out the FPS of the track (important as it can be PAL or NTSC which are different framerates). It will then dump the vob from the dvd as well as copying the chapters to a text file for use later and copying some other necessary files (like the IFO). It will then rip out 2 audio tracks, the first is usually english and the second japanese (the script assumes and labels them accordingly), but it could be any language. It then encodes the video using x264 with a two-pass encoder. Which means encoding it quickly and temporarily a first time to get information to help it encode better the second time round. Then it merges all these components into a single mkv file and cleans up after itself.

Now what track number do I give it you ask? Well to find that out use the lsdvd command. It will give you the length of each track and using that you can determine which track you want. If you have a dvd of anime which contains a few episodes there should be a few roughly around the same size, they are the ones you want. If your dvd is a movie, there should be one big one. There are always a few other smaller tracks that contain stuff like previews, extra content etc. Feel free to rip/ignore those at your leisure.

Now the other extra optional arguments are bitrate and device. The bitrate if not given is set to 2000 which is fairly high for a anime dvd. Setting it higher acheives a higher quality rip but at the expense of file size. Of course one cannot make a higher quality rip then the source material. To encode a dvd movie to a size of 1 cd, the average bitrate would be 860. The bitrate of 2000 is reasonably high quality for a DVD rip, but for HD content one would most likely go higher (but since this script doesn’t support ripping from blueray/hddvd, its a moot point).

The device argument is used for when your dvd is not at the default place for linux. One common method is to have an iso file which you mount on the loopback and then you can specify that mount point as the device. Or maybe someone has done a straight file copy of the dvd to a folder, then that folder can be your device argument.

The list of free software packages used (which could be installed by your package management system, such as apt-get, or installed from source):

• lsdvd
• mplayer
• dvdxchap (most likely from ogmtools
• transcode
• mkvtoolnix

If I have missed anything, or anyone can offer improvements, please feel free to comment.

Permalink Comments

10.01.08

“the eclipse executable launcher was unable to locate its companion shared library” AKA Fixing eclipse for mac os x

Posted in programming at 15:11 by dalore

Recently I did an upgrade of eclipse within eclipse using the “software updates” feature as well as adding some packages like svn and coldfusion for eclipse. When I restarted I ran into a small issue, it wouldn’t start.

The error message I kept getting was

the eclipse executable launcher was unable to locate its companion shared library

Now googling error messages normally shows someone whose experienced it and a solution, not in this case.

Digging around I found the the eclipse binary in:

/Applications/eclipse/Eclipse.app/Contents/MacOS

There is a file called eclipse and an eclipse.ini. Running this binary gave me this error:

 ./eclipse
dlopen(../../../plugins/org.eclipse.equinox.launcher.carbon.macosx_1.0.100.v20080509-1800, 2): image not found

Indeed I verified that such a file no longer existed, I did notice something similarly named. I edited the eclipse.ini file and replaced it:

../../../plugins/org.eclipse.equinox.launcher.carbon.macosx_1.0.101.R34x_v20080731

It then started up as normal.

Permalink 42 Comments

06.13.08

Quick backup solution for mysql with daily and monthly rotated backups

Posted in programming at 16:10 by dalore

Using a combination of logrotate, cron and mysqldump one can easily create a backup solution that does daily and monthly backups.

First I’d recommend setting up a mysql user that has mysqldump privileges only, in my case I created one called backup and it was only accessiable by localhost. I didn’t put a password on it. Also create a directory for your backups to go in, in my case I used /var/backups/mysql/

Create a shell script in /etc/cron.daily to be run every day. I called mine mysqldump.daily and don’t forget to chmod +x it. Here is the contents:

#!/bin/bash
DEST=/var/backups/mysql/daily.dump
mysqldump --all-database -u backup > $DEST

For the monthly, create a similar file in /etc/cron.monthly/mysqldump.monthly

#!/bin/bash
DEST=/var/backups/mysql/monthly.dump
mysqldump --all-database -u backup > $DEST

Test it out by executing them at the command line. It should create the daily.dump and monthly.dump file, verify they exist and filled with your data. You can fine tune the dump command to be just the databases your interested in, but I wanted to do all of them.

To make it rotate so you get to keep the last 7 backups for daily and 6 backups for monthly, create a logrotate configuration file in /etc/logrotate.d/mysqldump

/var/backups/mysql/daily.dump {
    daily
    rotate 8
    compress
    missingok
}
/var/backups/mysql/monthly.dump {
    monthly
    rotate 7
    compress
    missingok
}

This will rotate the daily.dump file every day keeping 8 days worth, and the monthly.dump every month keeping 7 months worth.

Test it out by running logrotate in verbose and force mode:
logrotate -fv /etc/logrotate.d/mysqldump

You should see the backups being gzipped and rotated as per your schedule. Now if that works there is nothing else to do as putting the files in the specified directories will cause them to be executed at the correct times. Good luck!

Permalink Comments

05.19.08

Loop through dates in perl

Posted in programming at 23:51 by dalore

A little code tidbit, how to loop through dates a day at a time, can easily be modifed for other time periods but this is all I needed:

#!/usr/bin/perl -w

use Date::Calc qw(Add_Delta_Days Delta_Days);

my @date = (2008, 1, 1);
my @enddate = (2009, 1, 1);

while (Delta_Days(@date, @enddate) >= 0)
{
  print "$date[0]-$date[1]-$date[2]\n";
  #... do stuff

  # move to next day
  @date = Add_Delta_Days(@date, 1);
}

Permalink Comments

05.12.08

python amazon stock checker

Posted in Shopping, programming at 23:01 by dalore

I just wrote my first python program. It checks a given ASIN to see if amazon has it in stock.

#!/usr/bin/python

import sys
import urllib

AWSAccessKeyId = "removed"
AmazonMerchantId = "A3P5ROKL5A1OLE"

def checkStock(asin):
  try:
    fh = urllib.urlopen("http://ecs.amazonaws.co.uk/onca/xml?Service=AWSECommerceService&AWSAccessKeyId="+AWSAccessKeyId+"&Operation=ItemLookup&ItemId="+asin+"&ResponseGroup=OfferFull&Version=2008-04-07")
  except:
    print "fail"
    return -1
  page = fh.read()
  fh.close()
  found = page.find(AmazonMerchantId)
  return found != -1

try:
  asin = sys.argv[1]
except:
  print "Usage: " + sys.argv[0] + " asin"
else:
  if checkStock(asin):
    print "in stock"
  else:
    print "not in stock"

You of course need to sign up to amazon webservices and insert your own AWS access key.

Sample output:

# Mario Kart (Wii)
./stockcheck.py B000XJNTNS
not in stock

# Nintendo Wii Console
./stockcheck.py B0007UATDG
in stock


Permalink Comments