The ABC’s of creating MP3s

---

Being content with GUI ripping software was something that didn’t happen to me using Linux. I had expected my music player software to handle `the task but I can’t remember any that did (not remembering to me is the same as working poorly I’m discovering). As for stand-alone rippers I haven’t heard any that were notable. Because I’m a big fan of software being efficient and to the point (do one thing and do it well) I was a bit nonplussed when I began wondering how I was going to import my CDs to MP3s. A good number of tasks that I had regularly done through the GUI, I discovered are better done through the command line and though I haven’t tested every MP3-related application this looks like it may be true for them as well. Here’s a complete-ish guide to ripping, organizing, repairing, and volume normalizing an audio collection well, done mostly through the CLI.

Rip

RipIT is program that can do just about anything that a GUI version to do. It’s default options will be good enough for most cases (running ripit is all that is needed). Having a greater amount of control however can save time in the end. A wrapper script can be created to help with this:

The ripcd script below defines:

• The ripping preset (extreme here because storage space isn’t an issue).
• The directory creation template. RipIT goes online and gets the album tag information which can be used organize directories by tag (here the common "$artist/$album" is used”).
• Looping (prompts when for new CD when ripping is done)
• Ripping priority so RipIT plays nice with other programs.
• Query the MusicBrainz music database instead as it is usually more accurate (editor approval required).
• The Audio sub-directory to rip it (my Audio directory is divided as such: # ls ~/Audio/ Audiobooks Music Others Podcasts)

Normalize

Normalizing audio means to adjust the volume of audio files to a standard level. This is often a good idea as average volumes levels per album usually differ to some degree. A great program called mp3gain can do this easily. I created a script for this that first normalizes by type (either Music collection, or Audiobook collection… since there are usually differing recording standards for each), then normalize relative to other albums in that catagory. Here’s the script:

Repair

Lame is used by RipIT for encoding of the audio files and does a very good job of it, occasionally though I’ve found it to make a mistake. For these MP3s, previous rips, and for MP3s that have been previously downloaded it is good idea to check them and see if they are in good shape. An excellent tool called MP3 Diags can test MP3s and fix common problems. Repairing MP3s I’ve discovered makes inter-operability between different players play nice. MP3 Diags also includes a very nice (though basic) tag editor.

Bash Script Templates

The basic and expanded templates I use to begin most of my bash scripts:

template-basic

template

Syntax Highlighting in Blog Posts with Vim

Update: Reader Elder Marco has pointed out that WordPress.com does have support for syntax highlighting of source code built-in (which I had never heard of before) that might be a preferred alternative for some. An example of both is below.

Vim is a great all-around editor, it also does very good at syntax highlighting. With the plugin “TOhtml” included with Vim it’s easy to put that highlighting into a blog post. I created a blogscrpt bash script that when run on another script will produce a file defining the syntax highlighting in HTML code. From there it can be pasted into the blog post.

blogscrpt syntax highlighting:

#!/bin/bash
# Create HTML code from Vim syntax highlighting (for use in coloring scripts)

filename=$@
background=light
colorscheme=beauty256
scrpt=${0##*/}  # filename of script

# Display usage if no parameters given
if [[ -z "$@" ]]; then
  echo " $scrpt <filename> - create HTML code from Vim syntax highlighting"
  exit
fi

# Syntax highlighting to HTML export
vim -f  +"syntax on"                  \
        +"set background=$background" \
        +"colorscheme $colorscheme"   \
        +"let html_use_css = 0"       \
        +"let html_no_pre = 1"        \
        +"let html_number_lines = 0"  \
        +"TOhtml"                     \
        +"x"                          \
        +"q" $filename

# Clean up HTML code
tidy -utf8 -f /dev/null --wrap -m $filename.html

# Delete the HTML meta page information.
sed -i '1,/body bgcolor=/d' $filename.html

# Remove line breaks (needed for some things like blog posts)
sed -i 's|<br>||g' $filename.html

# Remove the closing HTML tags
sed -i 's~</body[^>]*>~~g' $filename.html
sed -i 's~</html[^>]*>~~g' $filename.html

# Add preformatting tabs <pre> and </pre>
#sed -i '1 i <pre>' $filename.html
#sed -i '$ a </pre>' $filename.html

# Remove trailing blank lines
while [ "$(tail -n 1 $filename.html)" == "\n" ]; do
  sed -i '$d' $filename.html
done

# Delete newline of last <font> line for better formatting
sed -i ':a;N;$!ba;s/\(.*\)\n/\1/' $filename.html
sed -i ':a;N;$!ba;s/\(.*\)\n/\1/' $filename.html

# Delete final newline
perl -i -e 'local $/; $_ = <>; s/\n$//; print' $filename.html

WordPress built-in syntax highlight support example:

#!/bin/bash
# Create HTML code from Vim syntax highlighting (for use in coloring scripts)

filename=$@
background=light
colorscheme=beauty256
scrpt=${0##*/}  # filename of script

# Display usage if no parameters given
if [[ -z "$@" ]]; then
  echo " $scrpt <filename> - create HTML code from Vim syntax highlighting"
  exit
fi

# Syntax highlighting to HTML export
vim -f  +"syntax on"                  \
        +"set background=$background" \
        +"colorscheme $colorscheme"   \
        +"let html_use_css = 0"       \
        +"let html_no_pre = 1"        \
        +"let html_number_lines = 0"  \
        +"TOhtml"                     \
        +"x"                          \
        +"q" $filename

# Clean up HTML code
tidy -utf8 -f /dev/null --wrap -m $filename.html

# Delete the HTML meta page information.
sed -i '1,/body bgcolor=/d' $filename.html

# Remove line breaks (needed for some things like blog posts)
sed -i 's|<br>||g' $filename.html

# Remove the closing HTML tags
sed -i 's~</body[^>]*>~~g' $filename.html
sed -i 's~</html[^>]*>~~g' $filename.html

# Add preformatting tabs <pre> and </pre>
#sed -i '1 i <pre>' $filename.html
#sed -i '$ a </pre>' $filename.html

# Remove trailing blank lines
while [ "$(tail -n 1 $filename.html)" == "\n" ]; do
  sed -i '$d' $filename.html
done

# Delete newline of last <font> line for better formatting
sed -i ':a;N;$!ba;s/\(.*\)\n/\1/' $filename.html
sed -i ':a;N;$!ba;s/\(.*\)\n/\1/' $filename.html

# Delete final newline
perl -i -e 'local $/; $_ = <>; s/\n$//; print' $filename.html

Ubuntu Oneiric: Final Touches

Note: A month ago I meant to write this article but experienced hardware issues. I wrote that in places that Oneiric was slow… I was wrong. Apologize for any inconvenience.

Here are some edits, additions, and subtractions that help complete the feel of the of Ubuntu’s 11.10 Oneiric desktop. Note that a couple modifications are made only for performance reasons for use with an older computer.

Installing

When installing Ubuntu, it is still recommended to do a clean (fresh) install of Ubuntu. Ubuntu/Debian engineers primarily focus resources on the install route therefore making it the recommended method.

Home folder on a dedicated partition

“How we work can be almost as important as what we do.”

Putting application preferences back together can be a lengthy process. A good work flow can dramatically improve productivity. Putting settings and documents on a dedicated partition will allow them to be easily built on from install to install. In Linux, configurations rarely ever cause problems. The Parted Magic Maintenance CD is a good tool to start with that can help with the process. More on how to do this can be found here.

When doing a clean install with a dedicated home partition, the partition needs to be defined during installation being sure to have it remain unformatted:

Tools

For future reference here is a package management helper script. It makes common package management related tasks easier to execute (and remember).

If planning to stick around with Linux, learn Vim. Vim is an excellent command line editor. Learning Vim can save time and be pleasurable to use (here to edit configuration files). More about Vim can be found here.

Hardware Setup

The first detail to focus on after installing Ubuntu is to get all hardware up and running. Ubuntu does good at discovering/setting up hardware but it isn’t able to do everything. First, the Additional Drivers control panel in System Settings may have hardware needed to be installed (some hardware setup requires user confirmation and is done here). After this, testing all devices and peripherals is recommended. It may in the end be necessary to visit the manufacturers website and download drivers. In most cases though to get the hardware working, information is usually available on the wiki.

Desktop Preferences

A number of options can be made to make a more efficient desktop; these programs will be needed to make the edits:

sudo apt-get install dconf-tools gconf-editor

Remove Unnecessary Startup Applications

To restore the ability to edit the Startup Applications do:

mkdir -p ~/.config/autostart
cd ~/.config/autostart
cp /etc/xdg/autostart/*.desktop .
sed -i "s/NoDisplay=true/NoDisplay=false/g" *.desktop

To save resources, select what is needed in Startup Applications. If not needed, Ubuntu One, Desktop Sharing, and Check Hardware Drivers can be removed. Removing Update Notifier too can save a good bit or resources if willing to update manually. To complete Update Notifier disabling:

dconf write /com/ubuntu/update-notifier/auto-launch false
sudo apt-get remove apt-xapian-index  # actually an old Synaptic plugin remnant

Launcher

The Launcher with a couple edits can become more able to streamline the workspace.

Disable Auto-hide:

To have the Launcher always visible (usually recommended) do:

dconf write /com/canonical/unity-2d/launcher/use-strut true

Remove Multiple Desktops/Workspaces:

Save space on the Launcher if not using the multiple desktops feature:

gconftool-2 -s /apps/metacity/general/num_workspaces --type int 1
sudo cp /usr/share/unity-2d/launcher/Launcher.qml{,.bck}
sudo sed -i '/items.appendModel(workspaces)/d' /usr/share/unity-2d/launcher/Launcher.qml

This edit is temporary and will need to be run again when the unity-2d-launcher package is updated.

Add Show Desktop:

The ability to show the desktop can be done with the Super + D keypress (thats usually the Windows key) but to have the icon available on the Launcher an Xorg server interface tool will be needed:

sudo apt-get install xdotool

Create the .desktop so it can be pinned to the launcher:

echo "[Desktop Entry]
Name=Show Desktop
Exec=xdotool key --delay 300 super+d
Icon=desktop
Terminal=false
Type=Application
StartupNotify=false" >>   ~/.local/share/applications/show-desktop.desktop

Open the file manager and drag the .desktop to the Launcher:

nautilus ~/.local/share/applications/

Numlock Enabled on Login

Because the numberpad exists on most keyboards and since it’s primary use is for doing calculations having the Numlock on by default is usually is the preferred option:

sudo apt-get install numlockx
sudo sed -i 's|^exit 0.*$|# Numlock enable\n[ -x /usr/bin/numlockx ] \&\& numlockx on\n\nexit 0|' /etc/rc.local

Turn Off Resume from Sleep Lock

More obstruction than protection for some the resume from sleep lock can be disabled:

gsettings set org.gnome.desktop.lockdown disable-lock-screen 'true'

File Manager Possibilities

Once the behavior is adapted to this feature can save time; however this behavior can be persistent: to streamline workflow consider using a single-click for files in the file manager/desktop. Set this in the File Manager under > Edit > Preferences > Behavior > Single click. For a slight speedup in the file manager, lower the preview values (Nautilus > Edit > Pref > Preview > No text icons, Thumbs for smaller file sizes, and Count number).

Application Indicators

Application indicators are the feedback icons in the menu bar on the top right. Here are some edits/considerations (changes to application indicators area don’t take effect until Logout/Login).

Switch Users Unneeded:

For single-user computer or if the feature is never used, save space by disabling the Switch Users indicator:

dconf write /apps/indicator-session/user-show-menu false

Google Web Mail:

Because of its’ efficient use of space and it’s connectivity possibilities the web interface of Google mail is preferred over email programs by a good number of people. There is an application indicator to notify of new Gmail email called gm-notify:

sudo apt-get install gm-notify

gm-notify can be configured additionally to play a sound when new mail arrives, check /usr/lib/libreoffice/basis3.4/share/gallery/sounds/curve.wav ia a possibility.

Other Indicators:

Additional application indicators can be found at Ask Ubuntu.

Laptop Touches

For regular laptop users the thought of limiting the touchpad from accidental scrolling and mouse click tapping is kept in the front of the mind. Consider using two-finger scrolling and disabling touchpad tapping instead.

Firefox Security

If on the Internet a lot, it’s a good idea to protect the application that primarily accesses it. There is a nice script written by Ignorant Guru that puts Firefox in a sandbox. To learn more read here. First install the PPA then install the script through the package manager:

sudo apt-get update
sudo apt-get install sandfox

The script is most productive in protecting from Adobe Flash security holes. A perk of the script is that it allows Flash preferences to be saved; a disadvantage is this allows a security hole. To plug the hole change the preference directories to read-only only by root:

cd ~
rm -rf .adobe .macromedia
sudo mkdir .adobe .macromedia
sudo chmod ugo-wx .adobe .macromedia

Then bind the folders read-only in the script:

sed -i 's_^hide=/home/\\$user/.adobe.*$_bindro=/home/\\$user/.adobe      # bind folder read-only_g' /usr/bin/sandfox
sed -i 's_^hide=/home/\\$user/.macromedia.*$_bindro=/home/\\$user/.macromedia # bind folder read-only_g' /usr/bin/sandfox

After this, the Sandfox package could be put on hold to prevent it from updating (thereby preserving changes made to the script):

echo sandfox hold | sudo dpkg --set-selections

Under the Hood

A few options on the system-level can help improve performance and help unexpected delays.

No Timestamping on File Access

Since Linuxs’ early days the kernel behavior has been to re-date a files’ timestamp every time a file is accessed. This reasoning goes back to its’ server days when users were more interested in knowing when a file was accessed rather then when it was edited (written to). For desktop users however the expected behavior is for the timestamp of a file to be when it was last edited. Tagging the option noatime to the filesystem will give the expected behavior, also this option additionally improves system performance by saving a number of writes to the disk. See more on this here.

Swap Value

For computers with plenty of memory available (1 Gigabyte will be enough for most uses), lowering swap priority can help improve performance. To change immediately do:

sudo sysctl -w vm.swappiness=20
sudo sysctl -w vm.vfs_cache_pressure=50

And to have it as this value used regularly add the values to /etc/sysctl.conf:

vm.swappiness=20
vm.vfs_cache_pressure=50

Match Filesystem Check Times

If more than one partition is used, having filesystem check times run at the same time will cause less number of unexpected boot delays. This can be done with tune2fs (Ubuntus’ default value is 33 mounts and six months):

sudo tune2fs -c 33 -C 0 -i 6m -T now /dev/partition1
sudo tune2fs -c 33 -C 0 -i 6m -T now /dev/partition2

Other Programs

Other useful programs are these (most are additional command line utilities that come in useful down the road):

sudo apt-get install cd-discid curl dos2unix dnsmasq epiphany-browser gdebi gimp gparted imagemagick inkscape iotop irssi lame librsvg2-bin links mp3gain msmtp openjdk-6-jre p7zip pdftk ppa-purge pwgen realpath ripit ruby tree unrar vim xclip

Vims is set up well as is, but to make it even better use a more-optioned configuration:

sudo mv /etc/vim/vimrc{,.bak}
sudo cp /usr/share/vim/vim73/vimrc_example.vim /etc/vim/vimrc

Being on the Internet a good deal a Domain Name Server address cache/query daemon can help a lot with improving web browser load times, particularly during busy hours (the NetworkManager connection will need to be re-established afterward for changes to take effect):

sudo sed -i 's:^#listen-address=:listen-address=127.0.0.1:g' /etc/dnsmasq.conf
sudo sed -i 's:^#prepend domain:prepend domain:' /etc/dhcp/dhclient.conf
sudo service dnsmasq restart

Extrenui

• Missed Touchpad Button Clicks – fix for a touchpad button that missed clicks regularly.
• Hosts File Help – Only really a good idea for aging computers that can’t process complex ad-laden webpages.
• Root Required – If around Linux for a bit eventually the root account will have to be used. To work in a familiar environment when it root link common home settings: sudo ln -s ~/.{bashrc,profile,vimrc,vim} /root

Editors’ Opinion

I’m happy with my setup. Originally I had thought I’d go straight to Gnome 3 Fallback but I’ve stuck with Unity and I like the simplicity of it; plus it runs well. With a desktop setup like this, I’m beginning to feel productive. Thanks to Linux and Ubuntu engineers that made this possible.

Links

• Gnome 3 (Shell) Perspective
• Gnome 3 Installation

Western Digital My Book Essential External Hard Drive on Linux

I decided to sell my desktop computer and use my laptop exclusively, I had no need to keep another computer and since I was only using it for doing backups I decided it would be better to save some space.

I choose to get a Western Digital because they have been so reliable to me in the past. Of the external hard drives available at Wal-Mart it initially appeared not the be the best value. A Seagate right next to it was also a terabtye in storage capacity but also had USB 3.0 capability for only $15 dollars more. The WD Essentials has only USB 2.0 and I know that 3.0 is supposed to be considerable faster than 2.0. However, for me, my laptop is only USB 1.0 so this didn’t factor into it; also, because I am only using this for backups, time isn’t much of a factor and I prefer to have the reliability of the Western Digital name.

The My Book Essential HD has a capacity meter on the front to display how full the disk is. I learned though, unfortunately, that this only works through the Windows driver and using the NTFS file system. Because I’m going to be using this for backups on Linux with ext4 this feature isn’t available.

Since I have a Windows system installed I retrospectively learned that it is a good idea to install the driver/software for the drive there to setup the drive for only the reason so that I could disable the VCD. The Virtual CD Drive is a built-in memory chip that registers to the operating system as a regular CD drive. On it it contains the driver/software installer and manual. As far as the driver/software goes its nicer than I’ve seen of other hardware’s software, it was lightweight, easy to use, and with no frills. For Linux though the driver/software is unecessary as it is automatically recognized and working out of the box. I disabled the VCD drive with the Windows software though to keep the VCD from popping up when I loaded my Linux desktop.

I ran a S.M.A.R.T. conveyance test and extended test on it then did a thorough badblocks write test that took about 24 hours… all tests passed.

Formatting to ext4, the drive works perfectly in Linux without any additional configuration (besides noatime. I’ve been using the hard drive the last couple of months and I’m real happy with it: it’s small, quiet, and has done it’s job without a hitch.

DisplaySize in xorg.conf… uhgg!

Update: This turns out to be done by xrandr which the X.org server hands off to now for dynamic use of monitors. man xrandr even reports that it is trying to keepaconstant DPI. Not sure just why it is doing it, but found a good way to get it done.

I just got a new monitor to be able to use as an external monitor for my laptop. While I was setting it up I noticed that the monitors display size wasn’t correctly detected. The Xorg server does a good job auto-configuring however this caught my eye:

xdpyinfo | grep -B2 resolution
dimensions:    1920x1080 pixels (508x286 millimeters)
resolution:    96x96 dots per inch

The monitor I got is a 21.5″ monitor so I figured the DPI was off. I decided to calculate it myself (this is a square pixel monitor):

res_horz=1920
res_vert=1080
res_diag=$(echo "scale=5;sqrt($res_horz^2+$res_vert^2)" | bc)
siz_diag=21.5
siz_horz=$(echo "scale=5;($siz_diag/$res_diag)*$res_horz*25.4" | bc)
siz_vert=$(echo "scale=5;($siz_diag/$res_diag)*$res_vert*25.4" | bc)
echo "$siz_horz"x"$siz_vert"
475.48800x267.46200

Also there are online DPI Calculators conferred by doubt (1, 2,) and xrandr:

em_ds_h=$(xrandr | grep VGA-0 | rev | cut -d " " -f 3 | rev | sed 's/mm//')
em_ds_v=$(xrandr | grep VGA-0 | rev | cut -d " " -f 1 | rev | sed 's/mm//')
em_ds="$em_ds_h"x"$em_ds_v"
echo $em_ds
477x268

My discovered value and theirs are a couple millimeters off overall so I just used theirs. I created a configuration to define the display size to the the Xorg server. A basic configuration to define display size can be done like this:

cat /usr/share/X11/xorg.conf.d/90-monitor-disp-size.conf
Section "Monitor"
  Identifier "<default monitor>"
  DisplaySize 477 268
EndSection

Ubuntu uses /usr/share/X11/xorg.conf.d/ Arch Linux and some other use /etc/X11/xorg.conf.d/ (better choice I think). However this won’t work on the external monitor. So I expanded on it (more than it probably needed to be) by defining both monitors and related sections:

Section "Monitor"
  Identifier    "Internal - Pavilion Laptop"
  DisplaySize    304.5 228.6
EndSection

Section "Monitor"
  Identifier    "External - Samsung Syncmaster SA350"
  VendorName    "Samsung"
  ModelName     "SA300/SA350"
  DisplaySize    476 267.7
EndSection

Section "Device"
  Identifier    "ATI Radeon Mobility IGP 330M"
  Option        "Monitor-VGA-0"  "External - Samsung Syncmaster SA350"
  Option        "Monitor-LVDS"   "Internal - Pavilion Laptop"
EndSection

Section "Screen"
  Identifier    "Default Screen"
  Monitor       "Internal - Pavilion Laptop"
EndSection

Section "ServerLayout"
  Identifier    "Default Layout"
  Screen        "Default Screen"
EndSection

I added VendorName and ModelName but I’m not sure they uniquely define the monitor so that the Xorg server acknowledges them. The VendorName I believe is just for reference, ModelName can usually be discovered by doing:

grep "Monitor name" /var/log/Xorg.0.log

Monitor-VGA-0 and Monitor-LVDS define the ports and hence by reference should uniquely define the monitor (xrandr -q shows them and both are found in the Xorg log).

After a bit of research I discovered that there is a good amount of history concerning the Xorg server having a bit of trouble in not being able to correctly discover the display size. I believe this may be related to some drivers. I’ve been told the open-source ATI driver have had problems and read in some other places of other people who have had similar issues. Defining the display size in the configuration and telling the Xorg server not to use the auto-detected value can be done by adding this to the Devices section (for Nvidia drivers use: Option "UseEDID" "FALSE"):

 Option        "NoDDC"

Unfortunately, this didn’t work either and left me completely at a loss. Unsure how to go further to define display size in the the Xorg server configuration I decided to define it through xrandr.

xrandr has an option to define the display size with the --fbmm option:

xrandr --output VGA-0 --auto -fbmm 476x267.7

--auto uses the default/preferred mode of the monitor.

A Journey of Fidelity (As Luck Has It)

A few weeks ago, my monitor kept blanking out on me. I’d been having a couple troubles with X crashes so I attributed them to that. I had been wanting to change my partitioning scheme so I decided to just re-install and use the older version for awhile (thinking the problem came after an update). So I installed Windows, then started Parted Magic’s Gparted to resize the NTFS partition only to get 22 Unaccounted Clusters errors. ‘ntfsresize‘ (which GParted uses) does a filesystem integrity check before resizing I found out. Thinking that Windows must not have unmounted the disk cleanly on shutdown, I rebooted and forced a filesystem check… no errors. When I went back and tried again, I got the same problem. I learned that there are different versions of the NTFS filesystem so I reinstalled Windows again and let the Windows installer format the partition instead of Gparted which I had done previously. When I went to resize again, I got the same problem. Here I eventually came to the conclusion that very possibly my hard drive was failing on me. This threw me off because my drive was only a year and a half old and because it was a Western Digital. Nonetheless, I had to check. So I ran a S.M.A.R.T conveyance test and then an extended test. Both tests passed. I knew (…Ughh!) that I’d have to run a ‘badblocks‘ test. I ran a non-destructive test (… long wait here) and discovered I had 44 bad sectors on my hard drive. I checked the Western Digital website (who had a very nice warranty check/RMA program) and thankfully my drive was still under warranty. I got a replacement drive (in only two days!!), did tests this time, and installed Windows again. When I went to resize… uggggh, I got the same problem again. At this point, I got out an older version of Parted Magic (6.3) and everything worked… perfectly.

Through all this the fun part was my monitor which kept blanking out on me (its just getting old) and was only able to read the screen for about a minute at a time.

I got a new monitor now too and am doing good again. It turns out that ‘ntfsresize‘ had a bug in it. I’m not sure what version of ‘ntfsresize‘ had the bug but it’s also on the Ubuntu 11.10 install disk. I upgraded to Parted Magic 11-11-11 and was able to resize my NTFS partition.

On this journey I learned is to never buy a a new drive and not test it, que sera sera. Because of this, I wrote badblocks page on the Arch Wiki for reference.