Category Archives: Command Line

decompress—a wrapper script to decompress various archive types

00-post

The Arch Linux BBS has a thread where people put up their scripts so that others can peruse them. A long time ago someone came up with the idea to create a script that would detect various archive formats and decompress them. That post is unfortunately gone now, but I kept the idea and have expanded on it a bit: I’ve added a couple archive types, file detection, program detection, and archive list support. I gave it a good, overall test so I feel comfortable with it.

Options can be in any order:

$ decompress archive-r.zip --help
decompress [*-l] ... — wrapper script to decompress various archive types
  -l, --list  - list archive contents

If an archive’s existence isn’t detected it will be displayed:

$ decompress archive-r.zip
archive non-existent: archive-r.zip

If a program’s existence isn’t detected it will be displayed:

$ decompress archive-q.zip
program required: unzip

Listing support is available:

$ decompress -l archive-q.zip
 archive-q.zip
       32  2016-04-11 10:39   file-q1
       32  2016-04-11 10:39   file-q2

Listing and decompressing can be done for multiple documents:

$ ls
archive-a.tar.bz2  archive-f.tgz       archive-k.txz  archive-p.xz
archive-b.tb2      archive-g.tar.lz    archive-l.7z   archive-q.zip
archive-c.tbz      archive-h.tar.lzma  archive-m.bz2
archive-d.tbz2     archive-i.tlz       archive-n.gz
archive-e.tar.gz   archive-j.tar.xz    archive-o.lz
$ decompress archive-*
archive-a.tar.bz2...
archive-b.tb2...
archive-c.tbz...
archive-d.tbz2...
archive-e.tar.gz...
archive-f.tgz...
archive-g.tar.lz...
archive-h.tar.lzma...
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
archive-i.tlz...
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
archive-j.tar.xz...
archive-k.txz...
archive-l.7z...
archive-m.bz2...
archive-n.gz...
archive-o.lz...
archive-p.xz...
archive-q.zip...

.exe and .rar files are untested because I was lazy. If there is an error its error message will be displayed.

decompress can be found in my general-scripts repository.

compress—a tar wrapper script to simplify archiving files

Perl module installation

Configure

Perl has its own repository where programmers make available their modules called the comprehensive Perl archive network, which is better known as the CPAN. Perl includes a built-in module that can download, build, and install from the network. For some distributions this module may already be built, however, it is probably a good idea for all to build it… to be sure it is set up correctly. Begin by starting the CPAN module shell so that it may be configured:

perl -MCPAN -e shell

A configuration message will appear… most users will be good with the automatic configuration it recommends. If additional configuring needs to be done later typing o conf init will re-run the configuration dialog. To leave the shell type exit.

Install

The first requirement most people will need to do is build and/or update the CPAN module. Modules can be installed with the built-in module in three ways: from the module shell, from the perl command, or from the CPAN module binary.

From the shell (which was entered in the configuration section), the following command will install a new module, or in this case, update the CPAN module:

install Bundle::CPAN

From the perl command:

perl -MCPAN -e 'install HTML::Template'

From the cpan module binary:

cpan Module::Name

Note: CPAN itself recommends using the cpanm module for installation. Modules will need to be reloaded after being updated: reload cpan.

Execute

Modules are sometimes executable binaries and if they are known to the shell can be executed like any other command. Some modules are support modules and can only be used for programming or by use of another module. Information of installed modules can be discovered with the command perldoc perllocal.

Uninstall

Module maintenance is typically unexpected after installation and the built-in CPAN module has no ability to be able to do so. If the cpanm module is installed it does have the ability with the --uninstall/-U option. It will display the files to be removed and prompt for approval before uninstalling.

System backup to DVD

1 Reply

The purpose of this article is to explain how to span large archives into multiple files. One would do this, for example, to store on numerous DVDs.

Rational

I had an occurrence where the only form of storage media I had were DVDs. It became necessary to create a full operating system backup and I was able to do so with the DVDs. (A system backup with DVDs is atypical because DVDs have dyes which have limited time usage.) The following explanation is for an Operating System multi-volume archive creation that can also be used on other large archives.

Operating System archive creation

From an Install CD I did my archive creation. Such an archive will nearly always need to be done from such an external operating system (a running operating system is always adding or editing files that necessitates using an external operating system).

My disk partitions I label according to the Operating System. The archive name I used begins with my hostname which is my computer model:

mkdir               /mnt/LABEL
mount /dev/disk/by-label/LABEL
cd                  /mnt/LABEL
archive_name=$HOSTNAME_$(date +%F)_DISTRO.tar.xz
tar cavf $archive_name .

Archive Multi-voluming

I can now divide the archive to multiple volumes. I do this so that they fit on a DVD. A few explanations first:

DVD storage capacity varies. For me I used a 4.7 Gigabyte DVD.
DVD storage capacity will likely need to be converted so a multi-volume file will fit on a DVD. DVD storage capacity is almost always calculated with metric prefixes (i.e. a base of 1000); however, typical computer numerology uses the binary prefixes a base of 1024. The metric 4.7 Gigabyte converted to binary is 4.37721 Gibibyte (GB to GiB conversion fraction: 1,000,000,000/[1024 x 1024 x 1024] = 0.9313226).
The UDF file system, typically used for DVD data storage, at the time of this writing was still experimental on Linux and I choose to use the ISO-9660 file system for reliability. However, this file system has a file size limit of four GiB. Since my writable DVDs had a capacity of 4.7 GB I had to split up the files to two per DVD. File system overhead also has to be factored in and I reduced 2.188608100 GiB to 2.188000 GiB.

There are two methods that typically do this:

split

I default to the split command because the open source philosophy “do one job and do it good”. It is easy to operate: -b/--bytes= is converted to Mebibytes and -d appends a numerical suffix. (Notice the period following to the second value.)

split -b 2240M -d HOSTNAME_DATE_DISTRO.tar.xz HOSTNAME_DATE_DISTRO.tar.xz.

tar

tar can also do it. Gibibytes is converted to Kibibytes (1024 bits). To _c_reate a _L_ength-limited _M_ulti-volume archive:

tar -cML 2284284 -f HOSTNAME_DATE_DISTRO.tar.xz.00 HOSTNAME_DATE_DISTRO.tar.xz

The first multi-volume file will be created and then the name of the next file will be requested:

Prepare volume #2 for HOSTNAME_DATE_DISTRO.tar.xz.00
# n HOSTNAME_DATE_DISTRO.tar.xz.01

If the save location is in another location the path will need to be entered on every new entry (e.g. n /mnt/Backup/HOSTNAME_DATE_DISTRO.tar.xz.02).

DVD writing

I believe growisofs is a good way to do this:

growisofs -Z /dev/dvd -rJ HOSTNAME_DATE_DISTRO.tar.xz.{00,01}

Operating System archive restore

Boot from the Install CD, then:

mkdir /mnt/LABEL
mount /dev/dvd /mnt/dvd
mount /dev/disk/by-label/LABEL /mnt/LABEL

File system creation I am assuming is already done. To join the multi-volume archives (if done by split and tar respectively):

cat      HOSTNAME_DATE_DISTRO.tar.xz.*  &gt; HOSTNAME_DATE_DISTRO.tar.xz
tar -xMf HOSTNAME_DATE_DISTRO.tar.xz.00   HOSTNAME_DATE_DISTRO.tar.xz

Then the archive will just need decompressing:

tar xvf HOSTNAME_DATE_DISTRO.tar.xz -C /mnt/LABEL

External links

Command line dictionary

command-line-dictionary

As a person who likes to write it has always been helpful for me to have a dictionary nearby. As a regular command line user to have a dictionary I could access from there was something I really wanted. I hadn’t predicted this would be much of a task, however, I found it an uphill battle.

What I felt a command line dictionary should offer:

1) a basic description that is accurate
2) the capability to be accessed offline
3) a formatting that is easy to read

Availability

In my original attempt I didn’t find any. I looked at a number of programs but most were inadequate in one way or another. I was baffled and I almost gave up looking. I did eventually find one but before that the two most promising programs were dictd and sdcv.

Dictd

Dictd is a protocol/software-framework for a networking dictionary, it contains both a server and a client. The idea is to have a server where numerous clients can connect to it. This would be useful for local network use or for something like an online dictionary group. However, it seems that the development has been quiet, and I had trouble installing several of its dictionaries… I could never get it to work.

The basic setup steps that are required to make it function are:

install package and a dictionary for it
start the dictd daemon (requires very little overhead) and check if the dictionaries are available (dict -I).
look up a word definition using a particular dictionary (e.g. dict --database gcide)

sdcv

I used to use this program (Stardict console version) for years. It provided a basic, easy-to-use, unambiguous, definition. These days, however, the parent program StarDict is no longer in development. Additionaly, there were formatting problems that broke reading flow, and made it difficult to read.

Forest through the trees

I may not always get what I want, at other times, if I’m paying attention, I’ll find what I need. I discovered a program that while not a full-blown dictionary does pretty good. It technically might not even be a dictionary. From the man page:

wn - command line interface to the WordNet lexical database... it outputs synsets and relations to be displayed as formatted text.

In more human-speak: it details relationships between words. Its use as a thesaurus would be of a more direct comparison; however it can work for a dictionary as it does provide definitions and contextual examples. The definitions may be basic, but they are to the point. The only feature it does not provide that I use sometimes is word pronunciations.

wn lexical -over
...
The adj lexical has 2 senses (first 1 from tagged texts)
...
1. (2) lexical -- (of or relating to words; "lexical decision task")
2. lexical -- (of or relating to dictionaries)

Creating good enough alone

The output of wn can be difficult to read: it jumbles a lot of information together, and only roughly organizes it. (FYI, in the above example I’ve filtered out a couple lines.) To help the reading of it in a smooth natural way, I’ve created a couple scripts to format the output. One script is called dict and the other is called thes.

I’ve put them in a repository for any who are interested.

Two fine DAE scripts

Anybody that knows my command line habits, or me in general, knows that my memory could be better. It can be good when I need it to be, however, if I don’t have to remember a thing I’m writing it down. This is why I built my DAE scripts. DAE, which I pronounce as day, is an acronym for Digital Audio Extraction, also known as ripping audio CDs. The scripts are a wrapper for a command that has only a few options yet I have no way to remember the command options that I may not use again for awhile. Hence I created these basic scripts. They are straightforward scripts that just cover the essentials.

(These scripts are only basic wrappers, most of the work is done by the RipIt developer(s)… I thank them very much for their effort.)

How they look

There are two scripts. They are demonstrated here in use, as it is the best way to describe them.

daeme (pronounced like lame) is for MP3s:

daefe

daefe is for MP4s:

daefe

After these steps RipIt does a CDDB query from the Internet (if available) and allows tag editing if desired.

Settings

I bypassed adding a few settings in the script and rather allowed them to be specified in the RipIt configuration file as their values will likely remain the same:

faacopt=-s
dirtemplate="${artist} — ${album}"
playlist=0
eject=1

Audiobooks

The daefe script can also be used for audiobooks. The procedure encodes an entire CD to a file and writes a track/chapter index to another file. The chapter index file can be merged into the audiobook for an integrated audiobook, read ArchWiki:Audiobook for more details.

Download

Both scripts have error checking and I consider them reliable. I have put them in a repository for anyone interested.

gurl—a general downloader

I like to keep things basic. Because a command-line, download program is already a part of the base package installation, it is all I need. Once I learned curl I liked it quite a bit. As always I need help remembering the options so I wrote a general wrapper script and it seems to be all I need. It features redirect following, progress bar, and resume support. It looks like this:

# gurl http://.../archlinux-2014.09.03-dual.iso
###############################                                           43.4%

gurl can be found in my general scripts repository.

bckfile—backup a file with sequential numbering

I have discovered over the years that protecting a file, its content, and developing in a controlled, deliberate method is usually something good to keep in mind. I have learned that if I feel an document, project… is important, then to backup the data and then do the edit is the methodology I need to learn.

When I decide to backup a file, first thing I do is to see if there is a _vault directory. In any location where I had to backup previously, I created this directory. After the first time I did this I realized I was going to have to number these file backups. I reasoned out that filename_[0-9][0-9] would be a format that would be sufficient, if there was an extension it would be filename_[0-9][0-9].ext.

As I could see that file backups were something that I would regularly do, I decided to create a script that would automate this task. It tests the destination directory if the file exists. For the first backup, the script prepends 00 to the file, after that its prepend the sequential number.

The usage is basic: I define the source file and optionally the destination directory. The current directory is assumed if only the source file is specified.

An example:

$ bckfile file.txt _vault
‘file.txt’ -> ‘_vault/file_01.txt’

The script does have one limitation: the filename can only contain one period and it must be for the extension. This is necessary as determination for an otherwise intention would take a lot of work 😊.

bckfile can be found in my general scripts repository.

gt—create a gedit scratchpad

geditmp—a-script-to-create-a-gedit-scratchpad

I am really lazy with my editors. I have aliases in my shell configuration for gedit and vim to open them as quickly as possible:

alias v="vim -p"
alias sv="sudo vim -p"
alias g="bgcmd gedit"
alias sg="bgcmd gksudo geany"

This is very nice for me because I use my editors quite a bit. One thing I needed though was a command that would create and open a temporary file in gedit. The main reason for this is that, there are times, I don’t know how to name or place the file properly yet. At other times, the reason is, I like to have a scratchpad that would save information that I might forget if it was just a new file and I forgot about it or a crash happened.

The bash script

Running gt will create and open a file named with the current time (MMDDhhmm) and will be saved in the trash folder. If gt is followed by a name (e.g. gt geditmp.md) the name will be appended to the time.

The name is helpful if wanting to dig the file out of the trash folder at a later time.

#!/usr/bin/bash
# (g)edit (t)emp. file — create gedit scratchpad

tmp_dir=~/.local/share/Trash/files
tme_day=$(date +%m%d%H%M)

[ ! -d "$tmp_dir" ] && mkdir -pv "$tmp_dir"

# gedit existence test
hash gedit 2>&- || { echo "Requires program \"gedit\"."; exit 1; }

# Usage
if [ "$1" = -h -o "$1" = --help ]; then
  echo "${0##*/} [name1] [name2*] — create a gedit scratchpad"
  exit 2
fi

if [ "$1" ]; then
  for name in "$@"; do
    nohup gedit "$tmp_dir"/"$tme_day"-"$name" &> /dev/null &
  done
else
  nohup gedit "$tmp_dir"/"$tme_day" &> /dev/null &
fi

Write The Fine Manual with pod2man

Disclaimer: This is a guide for basic manual writing. It uses a simple markup language and converter to create a manual. It is good for beginners. For a good general overview of man pages read this, for more common approaches to writing manuals read this post.

An example

To get a good idea of how to write a manual seeing the markup of a POD (Perl’s Plain Old Document) works well. This is a POD for a fictitious program called dr-smile:


=encoding UTF8

=head1 NAME

dr-smile — output platitudes to help to improve one's mood

=head1 SYNOPSIS

dr-smile [B<--happy>] [B<--joke>=F<type>]

=head1 DESCRIPTION

B<dr-smile> outputs jokes and happy thoughts designed to make one feel F<--better>. It has the ability to output pleasant thoughts, jokes, good memories, optimistic fortunes, or just give you a big hub. B<dr-smile> is for I<entertainment> and for light mood change requirements. Please use responsibly and with greater change requirements please see an appropriate professional. Additional options are available in the configuration file F</etc/dr-smile.conf>.

=over 4

=item B<-h>, B<--happy>

A general happy thought will be output (e.g. C<dr-smile --happy>).

=item B<-j> F<type>, B<--joke>=F<type>

A good-sense joke will be told.  The types are: F<knock> and F<oddball>.  Read more in B<jokes(1)>.

=back

Text formatting

Man pages are mainly created with these types of formatting:

Format	Markup
bold	B<text>
italic	I<text>
link	L<http://address.domain >
file name	F<file-name>
email	E<name@address.domain>

Note: E<> is technically for escapes, however, I have seen it used before in another pod2man document—whether intentional or not I am not sure. Read the manual perlpod for markup information.

From a general observation over the years, bold and italic formatting get used in these situations:

Bold:

program names
options
other manuals

Italic:

keywords
emphasis
arguments
occasionally manuals

Category formatting

Manual page categories are generally formatted with a certain order. The order of categories vary a bit per man page but the first four are usually present. Here is the general structure and their purpose

NAME
  The name of the command followed by a one-line description of what it does.
SYNOPSIS
  A command's invocation with its listed options, or a functions parameter listings with its header file.
DESCRIPTION
  An in-depth explanation of the command or function.
OPTIONS
  A explanation(s) of what an option does these are sometimes put in the DESCRIPTION.
EXAMPLES
  Some examples of common usage.
FILES
  Files related to the command or function.
BUGS
  List of known bugs.
SEE ALSO
  A list of referenced or related commands.
AUTHOR, HISTORY, COPYRIGHT, LICENSE
  Information about the author...

For how to create the categorical entries, I only know the basic formatting. Look at the above example to see how it is done.

Category begin is defined with the head1 tag.
Indenting is done with the over 4 tag.\
Category item is defined with the =item tag.
Category end to define a new category is done with the =back tag.

Document to manual conversion

pod2man usage is basic. First look at manual section definitions here to know what section it should belong to. For my miscellaneous, fictitious program:

pod2man dr-smile.pod > dr-smile.7

Additional information may be wanted to add to the manual along with compression:

pod2man --section=7 --center="General Commands Manual" --release="1.0" \
dr-smile.pod | gzip > dr-smile.7.gz

To view the man page use man -l dr-smile.7.gz.

screenshot-manpage

Resources

pod2man a great way to write UNIX man pages

xuserrun—run a command as the currently active X.org server user

xuserrun is a bash script to detect the first X.org server environmental values, these values are then used to transmit a command to it. The script is primarily useful from another environment: a different user, from the tty console, from cron, a boot script…. xuserrun requires systemd.

Running it is basic:

xuserrun xclock -digital

Another example:

xuserrun notify-send "Remember to get bread."

I have put it in its own repository.