linux

pysshbackup

I’ve always been very serious about data safety. I’ve lost data more than once, and so when I moved into the linux world I searched high and low for a backup system that would suit me. At the time, not many options existed, but one that stood out was the rsync + cp with hard links.
To sum up the idea, you make a copy of all the files on your hard drive to a backup system (another computer with a large attached hard drive), then when you want to update the backup, you copy the backup using hard links and then backup only the files that have changed since the previous backup (think snapshots). Let me explain hardlinks.
In linux there are symbolic links and hardlinks. Symbolic links are like most other OSes links… If you have a file and make a symbolic link, if you delete the link the file still exists, if you delete the original file, the link no longer works (its target is missing).
Hardlinks is a way for two (or more) files to ‘point’ to the same data. If I have a file and make a hard link, if I delete the link the file remains, if I delete the file the link still works… the actual file will exist as long as one hardlink points to it, when the last hardlink is removed, the space is regained and the data is released.
Using nothing more than a couple of command line utilities, one can have the most sophisticated incremental backups (like Apple’s time machine, just around for 15-20 years before time machine ever existed): rsync (makes copies of files that have changed), ssh (allows this to work over a network or the internet) and ‘cp -l’ (which makes hardlink copies of files).
I wanted a way to make this a little more automated, idiot proof if you wish, so I wrote a python program called pysshbackup. I’ve been using this program on my systems for some years and it works really well. It is command line only, it has a light menu system (similar to fdisk), and settings are stored in a xml file.
I recently uploaded the code to my github account in case anyone thinks this may be up their alley as well.
As always, because I am part of the open source software community who has given me all the tools I used to create the script, this is open source as well, GPL version2.
The program can be found here: https://github.com/eugenecormier/pysshbackup
I’m not sure if this is useful to anyone, but in the spirit of openness: have at it!
~Eugene

The original information I was using can be found here (last updated 2004!, still relevent): http://www.mikerubel.org/computers/rsync_snapshots/

music

My initiation to programming: Rudi

Exciting news: I’ve created a new program which randomly generates music theory/rudiments worksheets and corresponding answer keys. I’ve been working on this a few months and I’ve got most of the bugs worked out.
To use the program itself you need to download the source and you have to have Python3 and lilypond installed on linux.
The code can be found here:
https://github.com/eugenecormier/rudi

If you don’t want to do it the ‘hard’ way, a former guitar student of mine is creating a web interface, you just go to the website, click the options you want and poof! you get two pdfs to print off and use.
Justin’s website can be found here:
http://rudiapp.com/

Of course all the code is free to use, change… basically do whatever you want with… I wrote this to help my life out at university, now I give it away freely to whoever else wants it (my thanks to the open source community, Python, and of course Lilypond)

linux

Linux DVD conversion to h264 aac mp4

So I get a week off, and what really needs to be done?!? I’ve been noticing the deterioration of my massive, 200+, DVD collection which contains ALL my former concerts, ensembles, studio recitals, grad recitals etc…

This of course is very precious to me and maintaining my data/media is always a balance of: newest and greatest way, good compression, high quality audio and video, longevity, and last but not least convenience.

For a long time now, the best option for me has been a transfer to the DVD format, and I have well over 200+ DVD vids of our concerts. I’ve been watching video codecs with hope, and finally with h264 and aac I have to say it really is incredible quality at a highly compressed bitrate.

So with the h264 (x264 in linux) codec (the video/size ratio is very good) and my dying DVDs, I decided to get this done, and I’ve learnt a few key commands that I’d like to share with whoever will listen. I’ll break it into steps:

1) Getting the info off the disc
Since these are all personal discs, a conversation about libdvdcss as it’s not needed. There are two good ways of doing this (the former is quicker, while the latter is far better at regaining lost data

ddrescue works really well if there are no/few problems with the disc (free from scratches):
dd_rescue -n -b 2048 /dev/cdrom dvd.iso

dd can be brutally slow, but gets everything:
dd if=/dev/cdrom of=dvd.iso conv=noerror,sync

2) joining VOBs
There are two ways to do this too. I’ll give the most logical and easiest first although I notice no difference at all between the two.

use cat:
cat VTS_01_1.VOB VTS_01_2.VOB VTS_01_3.VOB > combined.vob

use ffmpeg (you need 1.1 or newer for concat):
ffmpeg -i concat:VTS_01_1.VOB\|VTS_01_2.VOB\|VTS_01_3.VOB -c copy combined.vob

One neat thing about the ffmpeg command is you can include everything else and come up with a finished product like:
ffmpeg -i concat:VTS_01_1.VOB\|VTS_01_2.VOB\|VTS_01_3.VOB -c:v libx264 -crf 23 -c:a libfaac -ac 2 -ab 192k finished-video.mp4

3) Encoding video
I use ffmpeg for this:
ffmpeg -i combined.vob -c:v libx264 -crf 23 -c:a libfaac -ac 2 -ab 192k test.mp4

to break this last example down:
-i combined.vob -> is the concatted vob file
-c:v libx264 -> this is the codec (x264)
-crf 23 -> this is how you set the constant quality, a lower number means higher quality, bigger file
-c:a libfaac -> audio codec (faac)
-ac 2 -> stereo sound (channels = 2)
-ab 192k -> the audio bitrate, higher is better/bigger

and that’s pretty much it. For a 1 hour concert at 720p I’m getting about a 500-600mb file which is really good, and I’m using all open source software. I’m plowing through these videos, getting them backed up and safe. As soon as I complete this, the next stage is to run out to pick up a new 2TB HD, copy the files to that and move them offsite (just in case the house burns down…. you can NEVER be too safe with your data!

I like the quote:

“There are two types of people in this world, those who have lost data, and those who are going to lose data”

gentoo

Gentoo on the MacBook Air

It’s been a very long time since I posted anything, so it’s my new year’s resolution to do more this year. This first post of the year will be a computer post. Over the holidays I picked up a MacBook Air, and I have to say I love the design. Some of the highlights include: back-lit keyboard, multi-touch trackpad, fans & speakers hidden in the joint between the display and the keyboard & aluminium case. Apple really does a great job with design. But of course, I hate the simplicity of MacOSX. Yes I know it’s built on top of Darwin, but I like to be able to tinker with “everything” and run all my favorite programs. So the first thing to do when I got it was to delete everything and put on Linux. I won’t go through every detail, but I will mention the things that did not work out of the box. Hope this helps someone else!

##################################
# MacOSX fixes – Do this before
# removing MacOSX
##################################

If you use only Linux, boot hangs for 30 seconds waiting for the Mac partition. To override this issue the following command (from MacOSX):
bless –device /dev/disk0s1 –setBoot –legacy –verbose

To stop the boot mac sound, issue the following command:
/usr/sbin/nvram SystemAudioVolume=%01

##################################
# Wireless Card
# note the two <M>, these MUST be
# built as modules for the wifi
# card to work
##################################

edit: in kernel 3.2+ you must turn off BCMA to get BRCM softmac enabled, which is now under drivers/network (not staging)

 

Install: sys-kernel/linux-firmware

[*] Networking support —>
-*- Wireless —>
<*> cfg80211 – wireless configuration API
[*] Wireless extensions sysfs files
<*> Common routines for IEEE802.11 drivers
<*> Generic IEEE 802.11 Networking Stack (mac80211)

Device Drivers —>
Broadcom specific AMBA —>
<M> BCMA support
[*] Support for BCMA on PCI-host bus
[*] Staging drivers —>
<M> Broadcom IEEE802.11n PCIe SoftMAC WLAN driver

##################################
# Intel Video Card
##################################

edit: the hack below is not needed in kernels newer than 3.2

I found this solution here: https://bugs.freedesktop.org/show_bug.cgi?id=39533#c25

first edit /usr/src/linux/drivers/gpu/drm/i915/intel_bios.c
and after the:
DRM_DEBUG_KMS(“Found panel mode in BIOS VBT tables:\n”);
drm_mode_debug_printmodeline(panel_fixed_mode);
lines add the following:
panel_fixed_mode->hdisplay = 1366;
panel_fixed_mode->hsync_start = 1398;
panel_fixed_mode->hsync_end = 1566;
panel_fixed_mode->htotal = 1734;
panel_fixed_mode->vdisplay = 768;
panel_fixed_mode->vsync_start = 772;
panel_fixed_mode->vsync_end = 776;
panel_fixed_mode->vtotal = 792;
panel_fixed_mode->clock = 72500;
panel_fixed_mode->type = 0x48;
panel_fixed_mode->flags = 0xa;
drm_mode_set_name(panel_fixed_mode);

Graphics support —>
<*> Direct Rendering Manager (XFree86 4.1.0 and higher DRI support) —>
<*> Intel 8xx/9xx/G3x/G4x/HD Graphics
[*] Enable modesetting on intel by default

##################################
# Intel Audio Card
##################################

This worked automatically with pulseaudio, BUT I had to install gnome-alsamixer and turn up the “Surround” volume to get the speakers working

##################################
# Webcam
##################################

install the media-video/isight-firmware-tools
download: http://www.mediafire.com/?81xtkqyttjt

then ift-extract –apple-driver AppleUSBVideoSupport

##################################
# SSD enhancements
##################################

fstab should include (in this case sda1 is the ssd, the tmpfs is for gentoo to not compile on the ssd):
/dev/sda1 / ext4 discard,noatime,data=ordered 0 1
tmpfs /var/tmp/portage tmpfs nodev,nosuid,mode=1777 0 0

add the following to the /boot/grub/grub.conf kernel line:
elevator=noop

linux

Using ART Tubefire 8 with Linux

About a month ago I decided that it was time for me to pick up some decent recording gear and get moving on a recording. Going to a studio was out of the question for me because 1) I love computers and recording and I want to be involved in every part of the creation and 2) due to my job (I’ve taught some recording courses) I have a substantial working knowledge of recording techniques. I went to my local music shop and after a few days of reading and browsing and going home and researching on the net I decided to grab a nice mic (AKG C-214, this is for classical guitar) and a tube mic preamp (ART Tubefire 8).

I was really impressed with the versatility of the Tubefire. You can read more about it here, but to sum up it’s points: it is an eight channel, tube preamp which has been getting very good reviews for it’s price point…..and the real kicker, it has a built in ADC/DAC which runs into the computer via firewire. Even without the computer sound input though I was really interested in this unit because it has: eight channels – eight ins, eight outs (great for a live setting with a band), real tubes (push the input gain on the unit and you can really get some nice tube warmth), and phantom power.

So I had made up my mind on this product….it really was everything I could possibly hope for…one last step (as a Linux guy)…..check for Linux support…I mean I’m pretty good with Linux these days, so even if I have to compile and hack a little….no problem (and before you start saying….oh why doesn’t he just reinstall windows/macosx and stop complaining….here’s the thing…..it’s not about using the other operating system…..it’s about not wanting to have to always keep the other operating system + recording software around so that I can get back to my original takes…..I use Ardour these days and it is definitely everything I will ever need in a recording program and I don’t feel like moving from a pro-tools powerhouse sort of program into say cubase or garageband)
Anyways, I hit the net with my fingers crossed and learned that the project/program for firewire audio in Linux is ffado. After hitting their website and looking up Tubefire in the device support list (yes it was there at least) it was listed as “unknown”…..Damn….it was so close….it was everything I needed but no Linux support. So I thought about it and I came to the conclusion I could still use it as a preamp and run it into my computer via my Edirol UA-25 USB soundcard….which gives me all the nice tube warmth for recording, but I lose the ability to record 8 tracks at once (into individual tracks in Ardour) and play back eight tracks at once (again individually)….of course Ardour can record/playback as many tracks as your hard drive can keep up with….but it would be really nice to be able to mic a drum kit with eight mics, all running through a tube preamp into individual tracks on the computer for mixing a tweaking later….but oh well….now ….on to the fun and exciting bit

Last night after owning the unit for a month….I decided to screw around with the computer side of things a bit and see what I could learn (I have been using it successfully as a stand alone preamp)…..so I plugged it in to my computer via the firewire and sure enough, the kernel detected the firewire interface and setup a file (/dev/raw1394) to handle input/output from the device….good start 🙂
I’m using the now current Ubuntu (ver.9.04 Jaunty) so I installed the following packages: linux-image-rt, linux-restricted-modules-rt, jackd, qjackctl, ardour, ffado-dbus-server, ffado-mixer-qt4, ffado-tools and libffado0
Next I ran the command (in a terminal): sudo chown myusername /dev/raw1394
to give myself permission to access the firewire device and I fired up the ffado mixer. it discovered the Tubefire, there were some settings (which seemed to do nothing) and there was nothing in the mixer….ok what’s next….let’s try jack
So I started up Jack control, went into settings and changed: driver = firewire, interface = hw:0
hit the start button and voila….Jack was talking to something….I take a look at the input/output channels….OMG!!!! Jack is showing 8 inputs and 8 outputs…..now I’m starting to get really excited.
Last thing to check…..I started up Ardour and again everything works….It looks like multiple inputs should work no problem (I only recorded one channel at a time, but I can’t see there being any problem) and I tested multiple playback channels at the same time…ran them into my Mackie and everything was fine….so I’m assuming (I’ll test more in the coming weeks) that recording 8 channels at once should work and playing back 8 separate channels should work too

In the Jack setup if you select the sample rate to be any of the device’s supported rates and then start Jack, the unit correctly displays the correct rate by lighting the proper led light

wow….I should have done this long ago….lets put this in the “I can’t believe it just worked out of the box category!!!!!”

linux

Mencoder – my hero

I’m just starting to fully realize how truly amazing mencoder actually is. It’s helping me do things that I previously thought were impossible.

For example, I have a few avi files which were split to fit on multiple cds. Now every time in the past that I’ve tried to combine these I’ve seen disaster (problems with audio/video syncing) until I tried mencoder. (I’ve also posted some other commands I find useful)

To combine files:

mencoder -forceidx -ovc copy -oac copy -o output.avi input1.avi input2.avi

To encode subtitles into the video (the xvidencopts bitrate in this example is set to 400mb, you should change this to your desired file size and remember to leave in the minus sign before the number):

mencoder -ovc xvid -oac mp3lame -xvidencopts bitrate=-400000 -subfont-text-scale 3 -sub subtitles-file.srt -o output.avi input.avi

To encode video for a PlayStation3 dlna server:

mencoder -ovc xvid -oac mp3lame -xvidencopts bitrate=-400000 -o output.avi input.avi

Convert any video file to a DVD compliant MPG:

mencoder -oac lavc -ovc lavc -of mpeg -mpegopts format=dvd:tsaf -vf scale=352:240,harddup -srate 48000 -af lavcresample=48000 -lavcopts vcodec=mpeg2video:vrc_buf_size=1835:vrc_maxrate=9800: vbitrate=5000:keyint=18:vstrict=0:acodec=ac3:abitrate=192: aspect=16/9 -ofps 30000/1001 -o outputfile.mpg inputfile.avi

linux

How to recover data from a damaged cd/dvd

EDIT: see my new post about this here:
Linux DVD conversion to h264 aac mp4
———————————————————————

Simply use ddrescue…to install in Ubuntu:
sudo apt-get install gddrescue

To use:
ddrescue -d /dev/cdrom output.iso

It is by far the best/easiest program I’ve seen for rescuing data and it will recover everything possible. (It works great on CDs and DVDs)