Category Archives: Command Line

decompress—a wrapper script to decompress various archive types

00-post

The Arch Linux BBS has a thread where people put up their scripts so that others can peruse them. A long time ago someone came up with the idea to create a script that would detect various archive formats and decompress them. That post is unfortunately gone now, but I kept the idea and have expanded on it a bit: I’ve added a couple archive types, file detection, program detection, and archive list support. I gave it a good, overall test so I feel comfortable with it.

Options can be in any order:

$ decompress archive-r.zip --help
decompress [*-l] ... — wrapper script to decompress various archive types
  -l, --list  - list archive contents

If an archive’s existence isn’t detected it will be displayed:

$ decompress archive-r.zip
archive non-existent: archive-r.zip

If a program’s existence isn’t detected it will be displayed:

$ decompress archive-q.zip
program required: unzip

Listing support is available:

$ decompress -l archive-q.zip
 archive-q.zip
       32  2016-04-11 10:39   file-q1
       32  2016-04-11 10:39   file-q2

Listing and decompressing can be done for multiple documents:

$ ls
archive-a.tar.bz2  archive-f.tgz       archive-k.txz  archive-p.xz
archive-b.tb2      archive-g.tar.lz    archive-l.7z   archive-q.zip
archive-c.tbz      archive-h.tar.lzma  archive-m.bz2
archive-d.tbz2     archive-i.tlz       archive-n.gz
archive-e.tar.gz   archive-j.tar.xz    archive-o.lz
$ decompress archive-*
archive-a.tar.bz2...
archive-b.tb2...
archive-c.tbz...
archive-d.tbz2...
archive-e.tar.gz...
archive-f.tgz...
archive-g.tar.lz...
archive-h.tar.lzma...
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
archive-i.tlz...
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
archive-j.tar.xz...
archive-k.txz...
archive-l.7z...
archive-m.bz2...
archive-n.gz...
archive-o.lz...
archive-p.xz...
archive-q.zip...

.exe and .rar files are untested because I was lazy. If there is an error its error message will be displayed.

decompress can be found in my general-scripts repository.

Advertisements

compress—a tar wrapper script to simplify archiving files

00-post

I have become accustomed to using long options over the years as they are easier to remember. I do however use tar in numerous ways. I needed to have a quick way to remember how to archive files; I wrote this script to make it real basic:

$ cd ~
$ compress .local/bin/ Development/general-scripts/
archive name [archive.tar.gz]: /dev/sda4/sc
scripture.css  scripts.tar.xz
archive name [archive.tar.gz]: /dev/sda4/scripts.tar.xz
archive exists, overwrite? (y/n): y
archive created: scripts.tar.xz

The compression type to be used will depend on which extension is typed; tar has a nice option called --auto-compress. So, in the above example, typing ...tar.xz will use the LZMA compression algorithm. Just typing Enter on the archive name and the default archive.tar.gz will be used. The script also supports tab-completion for typing the archive name to help navigate folders and files.

compress can be found in my general-scripts repository.

Perl module installation

If doing Perl programming or if another package requires a Perl module, learning how to install one may become necessary. The recommended way to install a Perl module is through the distribution’s repositories, however, they can be installed manually with Perl.

Configure

Perl has its own repository where programmers make available their modules called the comprehensive Perl archive network, which is better known as the CPAN. Perl includes a built-in module that can download, build, and install from the network. For some distributions this module may already be built, however, it is probably a good idea for all to build it… to be sure it is set up correctly. Begin by starting the CPAN module shell so that it may be configured:

perl -MCPAN -e shell

A configuration message will appear… most users will be good with the automatic configuration it recommends. If additional configuring needs to be done later typing o conf init will re-run the configuration dialog. To leave the shell type exit.

Install

The first requirement most people will need to do is build and/or update the CPAN module. Modules can be installed with the built-in module in three ways: from the module shell, from the perl command, or from the CPAN module binary.

From the shell (which was entered in the configuration section), the following command will install a new module, or in this case, update the CPAN module:

install Bundle::CPAN

From the perl command:

perl -MCPAN -e 'install HTML::Template'

From the cpan module binary:

cpan Module::Name

Note: CPAN itself recommends using the cpanm module for installation. Modules will need to be reloaded after being updated: reload cpan.

Execute

Modules are sometimes executable binaries and if they are known to the shell can be executed like any other command. Some modules are support modules and can only be used for programming or by use of another module. Information of installed modules can be discovered with the command perldoc perllocal.

Uninstall

Module maintenance is typically unexpected after installation and the built-in CPAN module has no ability to be able to do so. If the cpanm module is installed it does have the ability with the --uninstall/-U option. It will display the files to be removed and prompt for approval before uninstalling.

System backup to DVD

The purpose of this article is to explain how to span large archives into multiple files. One would do this, for example, to store on numerous DVDs.

Rational

I had an occurrence where the only form of storage media I had were DVDs. It became necessary to create a full operating system backup and I was able to do so with the DVDs. (A system backup with DVDs is atypical because DVDs have dyes which have limited time usage.) The following explanation is for an Operating System multi-volume archive creation that can also be used on other large archives.

Operating System archive creation

From an Install CD I did my archive creation. Such an archive will nearly always need to be done from such an external operating system (a running operating system is always adding or editing files that necessitates using an external operating system).

My disk partitions I label according to the Operating System. The archive name I used begins with my hostname which is my computer model:

mkdir               /mnt/LABEL
mount /dev/disk/by-label/LABEL
cd                  /mnt/LABEL
archive_name=$HOSTNAME_$(date +%F)_DISTRO.tar.xz
tar cavf $archive_name .

Archive Multi-voluming

I can now divide the archive to multiple volumes. I do this so that they fit on a DVD. A few explanations first:

  • DVD storage capacity varies. For me I used a 4.7 Gigabyte DVD.
  • DVD storage capacity will likely need to be converted so a multi-volume file will fit on a DVD. DVD storage capacity is almost always calculated with metric prefixes (i.e. a base of 1000); however, typical computer numerology uses the binary prefixes a base of 1024. The metric 4.7 Gigabyte converted to binary is 4.37721 Gibibyte (GB to GiB conversion fraction: 1,000,000,000/[1024 x 1024 x 1024] = 0.9313226).
  • The UDF file system, typically used for DVD data storage, at the time of this writing was still experimental on Linux and I choose to use the ISO-9660 file system for reliability. However, this file system has a file size limit of four GiB. Since my writable DVDs had a capacity of 4.7 GB I had to split up the files to two per DVD. File system overhead also has to be factored in and I reduced 2.188608100 GiB to 2.188000 GiB.

There are two methods that typically do this:

split

I default to the split command because the open source philosophy “do one job and do it good”. It is easy to operate: -b/--bytes= is converted to Mebibytes and -d appends a numerical suffix. (Notice the period following to the second value.)

split -b 2240M -d HOSTNAME_DATE_DISTRO.tar.xz HOSTNAME_DATE_DISTRO.tar.xz.

tar

tar can also do it. Gibibytes is converted to Kibibytes (1024 bits). To _c_reate a _L_ength-limited _M_ulti-volume archive:

tar -cML 2284284 -f HOSTNAME_DATE_DISTRO.tar.xz.00 HOSTNAME_DATE_DISTRO.tar.xz

The first multi-volume file will be created and then the name of the next file will be requested:

Prepare volume #2 for HOSTNAME_DATE_DISTRO.tar.xz.00
# n HOSTNAME_DATE_DISTRO.tar.xz.01

If the save location is in another location the path will need to be entered on every new entry (e.g. n /mnt/Backup/HOSTNAME_DATE_DISTRO.tar.xz.02).

DVD writing

I believe growisofs is a good way to do this:

growisofs -Z /dev/dvd -rJ HOSTNAME_DATE_DISTRO.tar.xz.{00,01}

Operating System archive restore

Boot from the Install CD, then:

mkdir /mnt/LABEL
mount /dev/dvd /mnt/dvd
mount /dev/disk/by-label/LABEL /mnt/LABEL

File system creation I am assuming is already done. To join the multi-volume archives (if done by split and tar respectively):

cat      HOSTNAME_DATE_DISTRO.tar.xz.*  > HOSTNAME_DATE_DISTRO.tar.xz
tar -xMf HOSTNAME_DATE_DISTRO.tar.xz.00   HOSTNAME_DATE_DISTRO.tar.xz

Then the archive will just need decompressing:

tar xvf HOSTNAME_DATE_DISTRO.tar.xz -C /mnt/LABEL

External links

Command line dictionary

command-line-dictionary

As a person who likes to write it has always been helpful for me to have a dictionary nearby. As a regular command line user to have a dictionary I could access from there was something I really wanted. I hadn’t predicted this would be much of a task, however, I found it an uphill battle.

What I felt a command line dictionary should offer:

1) a basic description that is accurate
2) the capability to be accessed offline
3) a formatting that is easy to read

Availability

In my original attempt I didn’t find any. I looked at a number of programs but most were inadequate in one way or another. I was baffled and I almost gave up looking. I did eventually find one but before that the two most promising programs were dictd and sdcv.

Dictd

Dictd is a protocol/software-framework for a networking dictionary, it contains both a server and a client. The idea is to have a server where numerous clients can connect to it. This would be useful for local network use or for something like an online dictionary group. However, it seems that the development has been quiet, and I had trouble installing several of its dictionaries… I could never get it to work.

The basic setup steps that are required to make it function are:

  1. install package and a dictionary for it
  2. start the dictd daemon (requires very little overhead) and check if the dictionaries are available (dict -I).
  3. look up a word definition using a particular dictionary (e.g. dict --database gcide)

sdcv

I used to use this program (Stardict console version) for years. It provided a basic, easy-to-use, unambiguous, definition. These days, however, the parent program StarDict is no longer in development. Additionaly, there were formatting problems that broke reading flow, and made it difficult to read.

Forest through the trees

I may not always get what I want, at other times, if I’m paying attention, I’ll find what I need. I discovered a program that while not a full-blown dictionary does pretty good. It technically might not even be a dictionary. From the man page:

wn - command line interface to the WordNet lexical database... it outputs synsets and relations to be displayed as formatted text.

In more human-speak: it details relationships between words. Its use as a thesaurus would be of a more direct comparison; however it can work for a dictionary as it does provide definitions and contextual examples. The definitions may be basic, but they are to the point. The only feature it does not provide that I use sometimes is word pronunciations.

wn lexical -over
...
The adj lexical has 2 senses (first 1 from tagged texts)
...
1. (2) lexical -- (of or relating to words; "lexical decision task")
2. lexical -- (of or relating to dictionaries) 

Creating good enough alone

The output of wn can be difficult to read: it jumbles a lot of information together, and only roughly organizes it. (FYI, in the above example I’ve filtered out a couple lines.) To help the reading of it in a smooth natural way, I’ve created a couple scripts to format the output. One script is called dict and the other is called thes.

dict

I’ve put them in a repository for any who are interested.

Two fine DAE scripts

fine-dae-scripts

Anybody that knows my command line habits, or me in general, knows that my memory could be better. It can be good when I need it to be, however, if I don’t have to remember a thing I’m writing it down. This is why I built my DAE scripts. DAE, which I pronounce as day, is an acronym for Digital Audio Extraction, also known as ripping audio CDs. The scripts are a wrapper for a command that has only a few options yet I have no way to remember the command options that I may not use again for awhile. Hence I created these basic scripts. They are straightforward scripts that just cover the essentials.

(These scripts are only basic wrappers, most of the work is done by the RipIt developer(s)… I thank them very much for their effort.)

How they look

There are two scripts. They are demonstrated here in use, as it is the best way to describe them.

daeme (pronounced like lame) is for MP3s:

daefe

daefe is for MP4s:

daefe

After these steps RipIt does a CDDB query from the Internet (if available) and allows tag editing if desired.

Settings

I bypassed adding a few settings in the script and rather allowed them to be specified in the RipIt configuration file as their values will likely remain the same:

faacopt=-s
dirtemplate="${artist} — ${album}"
playlist=0
eject=1

Audiobooks

The daefe script can also be used for audiobooks. The procedure encodes an entire CD to a file and writes a track/chapter index to another file. The chapter index file can be merged into the audiobook for an integrated audiobook, read ArchWiki:Audiobook for more details.

Download

Both scripts have error checking and I consider them reliable. I have put them in a repository for anyone interested.

gurl—a general downloader

I like to keep things basic. Because a command-line, download program is already a part of the base package installation, it is all I need. Once I learned curl I liked it quite a bit. As always I need help remembering the options so I wrote a general wrapper script and it seems to be all I need. It features redirect following, progress bar, and resume support. It looks like this:

# gurl http://.../archlinux-2014.09.03-dual.iso
###############################                                           43.4%

gurl can be found in my general scripts repository.