Category Archives: Command Line

Getting Help from Console

If you’re in console (aka virtual console) doing an install or repairs on a system, it’s good to know how to get help if problems occur.

“Ground Control…”

To get help in console you can use a chat client. Read this page on how to set up irssi – a terminal/console IRC program. The guide will walk you through setting up irssi and connecting to freenode where many Linux distribution chat channels are located.

“Waiting for details, Houston…”

When you tell the people in the chat-room what your problem is, sometimes they will need to know additional information. This could be the output of a command or the contents of a configuration file. To do a command without leaving irssi do Ctrl+Alt+F2 (F3, F4… can also be used) to enter another console, then enter the command.

Be better not to have to write everything down on a notepad and then type it into irssi, this is where it becomes useful to use a collaborative debugging tool like pastebin. Pastebin is a website that temporarily holds configurations, bug outputs… that you can refer other people to get help. There are several tools that can be used from the command line that can send files to a pastebin service, for example pastebinit. Add pastebinit from your distro, then upload a file. For example, your xorg.conf file:

pastebinit /etc/X11/xorg.conf

For uploading the output of a command, first you have to put it into a file:

sudo fdisk -l &> partitions.txt

&> will redirect all output to a text file (both standard output and error output) and now it can be uploaded.

“I have visual…”

Occasionally you might need to actually show a picture of what your question is about (e.g. if you have a question about a console-based installer). For this you can use fbshot. fbshot is a framebuffer screenshot program. To take a screenshot of the first console (Ctrl+Alt+F1):

fbshot -c 1 console1.png

Then you can use a console web-browser and a image-hosting website to upload the image.

Customize man page colors with ‘less’ definitions

Man pages by default use less for displaying. I’ve used vim before to for colored text in man pages but something got bjorked in an update. To have color with man pages termcap will need to be invoked. Thanks to nico for the tip.

All that needs to be done is to export bold and underline values of termcap. Adding the values to the ~/.bashrc will make sure that they are always used:

# Less Colors for Man Pages
export LESS_TERMCAP_mb=$'\E[01;31m'       # begin blinking
export LESS_TERMCAP_md=$'\E[01;38;5;74m'  # begin bold
export LESS_TERMCAP_me=$'\E[0m'           # end mode
export LESS_TERMCAP_se=$'\E[0m'           # end standout-mode
export LESS_TERMCAP_so=$'\E[38;5;246m'    # begin standout-mode - info box
export LESS_TERMCAP_ue=$'\E[0m'           # end underline
export LESS_TERMCAP_us=$'\E[04;38;5;146m' # begin underline

And source the ~/.bashrc to have it work:

source ~/.bashrc

Notice I used Arch and Gentoo colors, my two favorite distros :) :

Updating BIOS with Linux

If you don’t have Windows installed and you need to upgrade your BIOS, Linux does have the tools to be able to create a BIOS flash CD. Not many companies make Linux flash utilities and alot of these utilites are DOS utilities so a bootable DOS disk is needed. This is a simple, easy way to create a BIOS flash CD.

First, get a BIOS image. You’ll need to download a BIOS image for your board. For information on what Flash utility to use, a good place to look is your computer manufacturers homepage. Award BIOS and American Megatrends BIOS are the most popular BIOS’s used on motherboards.

Editing FreeDOS Minimal Boot Image

Note: This didn’t work for me but plenty of people have had success with it, fdboot.img is a bit old and may not work on newer hardware. Look at flashrom below for another alternative.

FreeDOS provides a bootable DOS image. Download the DOS image to the Desktop:

wget http://www.ibiblio.org/pub/micro/pc-stuff/FreeDOS/files/distributions/1.0/fdboot.img

and mount it:

sudo mount -t vfat -o loop /home/user/Desktop/fdboot.img /media/ISO

The BIOS flash utility and BIOS image will need to be added to the FreeDOS image. I prefer to use /media/ISO but any empty directory will do. The bootable image has a fixed size (1,440 Kb, the size of a floppy disk) and hence /media/ISO will also have that limited memory. The size needs to remain fixed in order to create a bootable floppy of it. You can see the space used in the image by:

du -b /media/ISO

Add the flash utility DOS executable and the BIOS image (there should be just enough room for it). I prefer to put these in a new directory but it’s up to you.

cd /media/ISO
mkdir bios
cp /home/user/Desktop/flashprog.exe /home/user/Desktop/bios-image /media/ISO/bios

The data added to the FreeDOS image will be saved when the ISO is unmounted:

sudo umount /media/ISO

Now return to the Desktop and convert the appended FreeDOS image to a bootable ISO:

mkisofs -r -b fdboot.img -c boot.cat -o fdboot-bios.iso fdboot.img

The -b option defines the floppy image used for booting; the -c option will create a boot.cat file that directs to fdboot.img and is necessary for booting; the -o option defines the output file, in this case a bootable iso; and finally the image file needs to be added.

Now burn the iso to the CD/DVD however you like. For example, from the command line:

cdrecord fdboot-bios.iso

Flash BIOS in Linux with Flashrom

Flashrom is a utility to directly flash the bios directly in Linux. It’s design to be a comprehensive utility and supports a good number of hardware devices. Above that, flashrom is easy to use. Check their page for compatibility, or install flashrom and see if it recognizes your chipset. I’d tell more but the flashrom website does a good job of telling about the utility. I also updated the Gentoo ebuild for flashrom.

Questions

Because BIOS sizes are getting larger, we may need to learn how to create larger bootable images. mkisofs mentions that is can create an El Torito (bootable) iso with either 1200 Kb, 1440 Kb, or 2880 Kb images. I know how to create an empty vfat image can be created with:

mkfs.msdos -C newimage.img 2880

And, of course, it can be mounted and the FreeDOS files can be copied there, but how could we make it bootable?

Resources

Color Output on Bash Scripts (Advanced)

I talked in a previous post about basic bash script colored output using the tput command. The tput command works for basic coloring (providing seven colors to choose from) but ANSI also provides a 256 color palette.

Note: Not all terminals support ANSI, but most do.

ANSI color coding is in this form:

\033[01;38;5;160m

The ANSI sequence: {ESC}[{attr};{bg};{256colors};{fg}m

{ESC} or \033 represents the ANSI escape-sequence. {attr} represents the outputs attributes (properties such as blinking and bold text), {fg} is the foreground color, {bg} is the background color, m means the sequence ends.

Note: The escape-sequence \033 works fine but at times you might have to use \e.

An example:

echo -e "My favorite color is \033[38;5;148mYellow-Green\033[39m"

The variable -e is required because echo doesn’t normally interpret backslashes and 033[39m tells bash to end the seqeunce. The 38 Value will use no background color. Notice too that I omitted the attribution value which if isn’t used will use the default (regular text) value.

To get color values a good program is colortest.

colortest -w

Colortest will show the ANSI color value to the corresponding hex value. Then just insert the ANSI value into the either the foreground or background value.
There’s also a program called conv-rgb2xterm that a hex value can be put in and it will give and ANSI sequence for the foreground color.

References

Color Output on Bash Scripts

Users who have been using Linux for awhile often learn that creating a basic script is a good way to run multiple, often-repeated commands. Adding a little color to scripts can additionally provide nice feedback. This can be done in a fairly straight-forward way by using the tput command.

A common way of doing this is to define the colors that tput can produce by putting them at the beginning of the bash script:

#!/bin/bash
# scriptname - description of script

# Text color variables
txtund=$(tput sgr 0 1)          # Underline
txtbld=$(tput bold)             # Bold
bldred=${txtbld}$(tput setaf 1) #  red
bldblu=${txtbld}$(tput setaf 4) #  blue
bldwht=${txtbld}$(tput setaf 7) #  white
txtrst=$(tput sgr0)             # Reset
info=${bldwht}*${txtrst}        # Feedback
pass=${bldblu}*${txtrst}
warn=${bldred}*${txtrst}
ques=${bldblu}?${txtrst}

When writing new scripts using templates with these variables already defined can quicken the creation process and help keep scripts organized (my Bash Templates).

If just needing to use tput colors for specific instances this script can display the tput definitions and their corresponding possibilities:

#!/bin/bash
# tputcolors

echo
echo -e "$(tput bold) reg  bld  und   tput-command-colors$(tput sgr0)"

for i in $(seq 1 7); do
  echo " $(tput setaf $i)Text$(tput sgr0) $(tput bold)$(tput setaf $i)Text$(tput sgr0) $(tput sgr 0 1)$(tput setaf $i)Text$(tput sgr0)  \$(tput setaf $i)"
done

echo ' Bold            $(tput bold)'
echo ' Underline       $(tput sgr 0 1)'
echo ' Reset           $(tput sgr0)'
echo

To use additional colors see: Color Output on Bash Scripts (Advanced).

Linux Tidbits

Linux Tidbits

Regular commands I use every day in Linux plus a few eclectic ones. Basically geared to the new user. Tips or thoughts, please let me know.

Man pages

In Linux there is a manual for just about anything. Learn about almost everything by “man command” in the terminal. (e.g. man ls). Or type “command –help” for a basic description. Also, many man pages also cover configuration files (man resolv.conf).

Basic Commands

Navigating:

  • Up key- Command History
  • Tab – Auto-completion, nice and handy for completing file-names, directory names, and commands.
  • Commands are in this form: command -arguments

Files ( + directories )

  • ls ( list ), -l ( long ), -a ( shows hidden )
  • cp ( copy )
  • mv ( move or rename ), mv filename1 filename2
  • rm ( remove ) Very dangerous to use as root. Use with caution. -r ( recursive ) -f ( force – needed to remove a link)

Wildcards to expand definitions:

  • * (matches any character), cp *.txt ~/Desktop
  • ? (matches any single character), cp file?.txt ~/Desktop
  • [characters] (Matches a range/set of characters), cp [a-n]*.txt ~/Desktop</li>
  • [!characters] (Matches any character that is not a member of the set characters)

Directories

cd    #( change directory )
pwd   #( print working directory )
mkdir #( make directory )

A name followed by a / means it’s a directory, bash will figure it out if you don’t use it but some apps don’t. A safe syntax would be cd myfiles/

Command Output to Text (Standard Output)

ls /usr/bin > /home/user/Desktop/programs.txt

Add to an existing text file (append):

ls /sbin >> /home/user/Desktop/programs.txt

Pipes ( | )

Useful for using programs in conjunction with others:

ls -l | less

Filters

Popular filters used after piping:

  • sort – alpha-numeric ordering
  • uniq – removes duplicate lines of data
  • grep – returns the output of a specified pattern of characters
  • head, tail – outputs the first of last lines of output
  • tr – translates characters – can be used for upper/lower case conversion

Use grep to extract patterns from files

grep EE /var/log/Xorg.0.log
ls ~/Documents | grep recipe
glxinfo | grep -i direct

Files and File Permissions

View file permissions

List the files in “long” view:

ls -l

-rwxr--r-- 1 user user 225444 2007-05-01 21:58 abc.pdf
||  |  |     |    |
||  |  world |    group name
||  group    owner name
|owner/user
directory
r = read = 4
w = write = 2
x = executable = 1

Change File Permissions

owner = rwx = 4+2+1 = 7; group = r = 4 ...

The above file’s permissions numerically is 744. To change permissions of the above file:

chmod 750 abc.pdf

Change Ownership

chown user:group /home/user/document.txt

Lazy way of make a file executable ;)

chmod +x ~/.scripts/example

File Systems

Show all partitions and their types:

sudo fdisk -l

Show mounted partitions used/available space:

df -h | grep ^/dev

See all file systems mounted::

cat /proc/mounts

Sort Directories by How much space they consume:

du | sort -nr

Define Disks and Partitions and how they mount

sudo mkdir /mnt/USB-Drive
sudo mount -t vfat -o rw /mnt/USB-Drive

types include hfsplus, vfat…

The fstab file defines how to mount available disks/partitions that can automatically be mounted it at boot: Enter in /etc/fstab:

/dev/sda2    /mnt/OSX          hfsplus ro,exec,auto,users    0      0
/dev/sda4    /mnt/Shared_Disk  vfat users,auto,uid=1000,gid=100,umask=007  0 0

ro – read-only, rw – read-write, auto mounts filesystem on boot

If the disk will resemble another disk it is better to use a unique device ID (UUID) rather than /dev/disk:

sudo blkid /dev/disk

Unmount all possible file systems:

umount -a

File System Check

Only filesystems that are able to be unmounted can have the filesystem checked.

Reboot immediately and check for errors:

sudo shutdown -Fr now

For specific mounted filesystems they can be check on next boot by placing a forcefsck file at the root of the disk/partition:

sudo touch /forcefsck
sudo touch /home/forcefsck

This will not run a file system check though if the file system is marked clean. For this boot a rescue disk and run fsck -f to do this.

Change how often fsck runs at boot (-c = count boots, -i = time interval):

sudo tune2fs -c 30 -i 6m /dev/disk

Check and mark bad blocks on damaged drives:

fsck -vcck /dev/disk

Mounted file systems should be checked from the Installer CD/DVD or on boot.

Swap

Create swapfile

dd if=/dev/zero of=/swapfile bs=1024 count=2097152

Swap is recommended to be 1-1/2 to 2x the value of the RAM to use for hibernation.

1 GB = 1024 MB = 1024 x 1024 kB = 1048576 kB = 1048576 kB x 1024 bytes/kB = 1,073,741,800 bytes10
mkswap /swapfile
swapon /swapfile

Add to /etc/fstab:

/swapfile              swap             none     defaults

Controlling Swap

Turn off swap:

swapoff -a /swapfile

Swapiness is the input/output priority of swap. To measure the current value:

cat /proc/sys/vm/swappiness

To change the swap priority (higher value means more swapping):

sudo sysctl vm.swappiness=10

to use this value permanently add it to /etc/sysctl.conf (vm.swappiness=0). Values of 20 or lower are better for laptops.

File Compression

Pack:

tar gunzip:
tar cvpzf /AreaToSaveTo/yourcompressedfile.tgz --exclude=/this/folderorfile /CompressionStarts/Here
tar.bz2(tbz2) (block sorted, better compression):
tar -cvjf files.tar.bz2 fileorfolder
bzcat linux-2.6.XX.tar.bz2 | tar x

Unpack:

tar xvf file.tgz
unrar e file.part01.rar

Span Multiple Volumes

Create:

tar -c -M --tape-length=2294900 --file=part1.tar too-large-archive.tgz

Extract:

tar -x -M --file=part1.tar too-large-archive.tgz

At prompt specify new (n),
then specify volume name (e.g. n part2.tar)
tape-length is 1024 bytes measurement or (1 computer kilo)

Or use “split” to break a large volume (2m = 2 megabytes, LF is the prefix for new name):

split -b 2m largefile LF_
tar -cvj /full/path/to/mybigfile | split -b 650m

Put back together:

cat file* > newfile

Backup and Restore

Tar – From Install CD

cd /mnt/gentoo
tar -czpvf /mnt/gentoo/MacBook-Gentoo-Backup.tgz *

Rsync – Full Backup

rsync -avtp --delete --exclude=/home/user/somedir /source/dir /destination/dir
  • -a archive, -v verbose
  • -t preserve modification times, -p permissions
  • --delete removes destination file if has been removed from source
  • --links recreate symlinks
  • -z compress from source to destination – good for slow connections.
  • use “-a e ssh source name@hostname:dest” for ssh

Users

Add user

useradd -m -G adm,audio,cdrom,cdrw,cron,games,plugdev,portage,shutdown,usb,users,video,wheel -s /bin/bash user

Groups may vary some per distribution, this one is for Gentoo. Some groups will not be available until a certain program is installed.

Add/delete user to group:

gpasswd -a user plugdev
gpasswd -d user plugdev

See what groups user belongs to:

id

Remove user:

userdel username

CD/DVD

Writing to CD/DVD with Rock-Ridge support

Rock-ridge support add Unix file extensions and attributes for iso9660 standard disks.
DVD are marked as 4.7GB capacity but thats just the marketing measure. In terms the computer understand the space on a DVD is 4.368 GB’s (1 GB = 1048576 kB x 1024 bytes/kB). DVD +R at 4x or 8x for best performance

DVD

growisofs -Z -lrJ -joliet-long /path/to/files
  • -Z means to start at the beginning of the dvd
  • -l allows long filenames (breaks DOS compatability)
  • -r Rock-ridge support
  • -J Add Joiliet support
  • -joliet-long – allows Joliet filenames to be 103 characters long instead of 64 – breaks joliet compatibility but works in most cases.

CD

mkisofs -o my.iso -lrJ /path/to/files

Then burn iso to CD (not sure if I can write directly to CD, from what I’ve seen it would seem that I can’t).

Blanking a Disk
If you want to blank a disk or it already has a file-system on it you’ll see an error like “WARNING /dev/hda already carries isofs!” then reinitialize the filesystem:

DVD

dvd+rw-format -f /dev/sr0
growisofs -Z /dev/hda=/dev/zero

CD

cdrecord -v dev=/dev/hda blank=fast
cdrecord -v dev=/dev/hda speed=2 blank=fast
cdrecord -vv dev=1,0 blank=all

ISO

Write ISO to CD/Drive:

dd if=name.iso of=/dev/sdb1

Mount ISO:

mount -t iso9660 -o loop,ro name.iso

Create an ISO from a DVD or CD:

dd if=/dev/sr0 of=name.iso

Create and ISO from a file/directory:

mkisofs -o name.iso /path/to/file_or_directory

CDRWin (.bin/.cue) images to ISO:

bchunk name.bin name.cue name.iso
bin2iso name.cue

Converting CloneCD images to ISO:

ccd2iso name.img name.iso

Converting nrg (Nero) images to ISO:

nrg2iso name.nrg name.iso

Support for writting large file sizes

ISO has file size limit of 4GB (untested -udf support is still in alpha):

mkisofs -o my.iso -lrJ -allow-limited-size -udf file-or-pathtofiles
growisofo -Z /dev/sr0 -lrJ -allow-limited-size -udf file-or-pathtofiles

Mouse/Keyboard

Change keymaps:

setxkbmap dvorak

Map pointer buttons to keyboard:

xmodmap -e 'keycode 116 = Pointer_Button2'
xmodmap -e 'keycode 108 = Pointer_Button3' 
xkbset exp m

Hardware Info

Kernel messages about hardware

dmesg | less

Cpu info:

cat /proc/cpuinfo

List all PCI/USB devices

lspci
lsusb

Detect hardware as it’s plugged in

sudo tail -f /var/log/messages

lshal –monitor # more detail

Icons / Cursors / Fonts …

Reset Icon Cache

gtk-update-icon-cache -f /usr/share/icons/hicolor/

Convert Windows Icons to Linux

Link

Reset cache for fonts:

fc-cache -vf

Build font info per directory:

mkfontscale
mkfontdir

Take screenshot of selected area

import filename.png

Set gamma

If you have ability to calibrate your own icc profile (Macintosh’s do) copy the icc profile to Linux and use “xcalib icc.profile“, otherwise a basic gamma can be set:

xgamma -bgamma 0.925 -ggamma 0.925 -rgamma 0.925

System

Shutdown at a specific time

shutdown -h 22:33
shutdown -P now

date

Use “date” to check date and to set system clock:

date MonthDayHourMinuteYear

Find out kernel version:

uname -r

Start Program that isn’t in the Systems Path

Only programs that are in a system’s $PATH setting can be started by typing the command, otherwise:

./program

Disable Touchpad whilest Typing

syndaemon -d -t -i 2

Networking

Samba

Change or add password to smbconf:

sudo smbpasswd -L -a user

Mount SMB share to folder:

sudo smbmount //192.168.1.105/user/ mnt/directory -o username=username,password=pass,uid=1000,mask=000

Mount all Samba Shares in fstab:

mount -a -t smbfs

SSH/SCP

SSH Howto

Remote login with SSH with username (diferent than the one you’re using):

ssh -l username 192.168.1.101

Copy remote file to local file:

scp -p user@192.168.1.101:~/Desktop/file.name file.name

Download entire website:

wget -r http://www.example.com/

Advanced

Bash

The Bash configuration file (~/.bashrc) file:

Adding PATHs to the ~/.bashrc file will make the system aware of another folder that has executables and shortcuts can be created for common commands:

export PATH=$PATH:/home/user/.scripts:
alias capscreen="import ~/Desktop/screen.png"

To see the preset variables already defined for bash:

set

Search History

ctrl-r

Cron
Cron is the system program scheduler. It checks every minute for commands to run. To edit a crontab (cron jobs):

crontab -e

#   minute (0-59),
#   |   hour (0-23),
#   |   |   day of the month (1-31),
#   |   |   |   month of the year (1-12),
#   |   |   |   |   day of the week (0-6 with 0=Sunday).
#   |   |   |   |   |   user

43 08 * * * env DISPLAY=:0.0 audacious /home/user/My\ Music/Other/Alarms/301gq.mp3

chroot – (changing root)
Userful for logging into your current Linux from an installtion CD:

su
mkdir /mnt/osname
mount /dev/sda3 /mnt/osname
mount -t proc none /mnt/osname/proc
mount -o bind /dev /mnt/osname/dev
chroot /mnt/osname /bin/bash

Compile Kernel

make oldconfig
make menuconfig
make clean zImage modules modules_install install

For PPC “make pmac32_defconfig” will generate a basic config.

Find Kernel Modules:

find /lib/modules/*Kernel-Version* -type f -iname '*.o*' -or -iname '*.ko*'

Add the screen program to be able to background a terminal process
screen command (CTRL + A + D to background it, to return it: screen -r).

Use noup to continue a process even if you log out

noup command

Unsorted/Less Used

sudo echo >> no work

echo "my text" | sudo tee /etc/portage/package.use

See whats taking up ram:

ps auxf --sort size

Allow window executables to run directly (will need Wine and misc. binaries enabled in kernel)

In /etc/sysctl.conf add:

fs.binfmt_misc.register = :WINEXE:M::MZ::/usr/bin/wine:

and add to fstab:

none /proc/sys/fs/binfmt_misc  binfmt_misc  defaults 0 0

Re-size Images from the command line

Requires imagemagick to be installed (convert writes new image, mogrify overwrites):

convert image.jpg --resize 800x600 newresized.png
mogrify -geometry 1024x768 *.png

Copy ALL Files (+invisible, hard links, softlinks)

find . -depth -print0 | cpio -null -sparse -pvd /mnt/newhome/

Create random numbers, hex letters

dd if=/dev/random bs=1 count=5 2>/dev/null | xxd -ps

Run programs sequentially or concurrently

program1 && program2
program1 & program2

A simple web server

Share files in directory and all subfolders:

python -c "import SimpleHTTPServer;SimpleHTTPServer.test()"

View at http://localhost:8000 or http://your_ip:8000/.

Debian Specific:

drive space show taken by installed packages

dpkg-query -W --showformat='${Installed-Size;10}\t${Package}\n' | sort -k1,1n

Rebuild Font Directory

dpkg-reconfigure fontconfig

Gentoo Linux Tidbits

I’ve been using Gentoo for about two years now and I took notes on managing my box. These are those notes. If you’re interested in installing Gentoo take a look at Gentoo Quick Install.

Update: bash script

This section is an addition. I’ve since created a bash script that does many of the functions and make reading the rest less necessary. The script is self-explanatory: link.

About Portage

Gentoo Linux uses a package management system called Portage. Portage offers one of the most extensible and customizable package systems available in Linux.

System Update

Update all packages on the computer. This process involves: syncing Portage, creating a text file to review updates, updating the system, merging new configuration files, remove orphaned dependencies.

Sync the portage tree

emerge --sync # or
eix-sync      # preferrable for faster searches (eix search)

Examine update before install

emerge --pretend --verbose --update --newuse --deep @world | less

If unexpected dependencies are being pull in, use the --tree option to track it down.

Complete update

emerge --update --newuse --deep --with-bdeps=y @world
dispatch-conf
revdep-rebuild  # ...
emerge --depclean
  • In the first command, portage wiil update all packages on the system
  • dispatch-conf is configuration updating and merge tool
  • revdep-rebuild will check that all programs and libraries are linked correctly and rebuild them if necessary.
  • emerge --depclean will remove orphaned dependencies
  • Additionaly a kernel update may need compiling again, distfiles may need to be cleaned up…

Failed package emerge in compilation string

At times in a long list of package emerges (like a system update) a package will fail to emerge. The bug-tracker and the forums usually have information about known problems with the package. If not, it’s possible the package needs a newer version of a package that has not yet been installed in the compilation string. Skipping the problem package and emerging it again when the rest of the packages are compiled may fix the problem.

emerge --resume --skipfirst

Blocked packages

Packages that block other packages from being emerged can be fixed by removing the obstructing packages then reinstalling it after blocked package is emerged.

quickpkg $BLOCKER
emerge -C $BLOCKER
emerge $BLOCKED
emerge --usepkgonly $BLOCKER

Specify USE Flags per package

In /etc/portage/package.use add pkg-category/pkg useflag useflag2. To add a USE flag temporarily (not recommended):

USE="useflag" emerge package

Masked Packages (keyword)

The “missing keyword” mask states an ebuild doesn’t support or hasn’t been tested on the current architecture (x86, amd, ppc… ). Keywording can be added to /etc/portage/package.accept_keywords for example: media-libs/libquicktime ~ppc.

Masked Packages (hard)

Gentoo hards masks some packages for security concerns, collisions… Packages are hard masked in /usr/portage/profiles/package.mask. If liking to live on the edge packages can be unmasked in /etc/portage/package.unmask.

Emerge dependency of a package

Packages that are dependencies of other packages (i.e. have no use on their own) should be emerged as oneshot. This is because if the main package is removed so too will this package when emerge –depclean is run. Otherwise these dependent packages are added to the world file.

emerge --oneshot package

Freeze a Package

If a rebuild of a package is undesired a package can be frozen. This is useful for kernels and other such packages.

  • Mask the entire package (i.e. without version) in /etc/portage/package.mask with sys-kernel/gentoo-sources.
  • Add the specific version to /etc/portage/package.unmask with sys-kernel/gentoo-sources-version.

Create a Binary Package

If enough disk space is available its may be a good idea to create and archived-binary of a package so it is quicker to re-install:

emerge --buildpkg zim
emerge --usepkg zim    # to reinstall

Info About the Portage System

This information can be useful for reporting bugs:

emerge --info

Other Portage Tools

Info about USE flags (equery is part of gentoolkit):

equery uses package

Programs built with a specific USE flag:

equery hasuse useflag

View what files are installed by program:

equery files alsa-lib

View what packages install to a folder:

equery belongs /usr/share/fonts/misc

List all installed packages:

equery list

Select a new system profile (With each new revision (i.e. 2006.1 to 2007) new profiles are added. Profiles define basic system USE flags…):

eselect profile list
eselect profile set 4

Select new kernel:

eselect kernel list
eselect kernel set 2

Rebuild external modules (added from Portage). Drivers build against the kernel (video drivers, sound drivers…) and will need to be rebuilt with new kernels:

emerge --ask @module-rebuild

Compile with specific compiler version. Some packages require a specific version of the compiler. See installed GCC profiles and select one:

gcc-config -l
gcc-config 2
source /etc/profile

Clean portage world file:

Remove entries that are dependencies only in /var/lib/portage/world to help emerge times. Careful what to remove thought. For example, “epiphany-extensions” requires “epiphany” so if “epiphany” is removed and later its decided that “epiphany-extensions” are no longer needed than running `emerge –depclean’ will removed “epiphany”.

Layman / Overlays

Overlays are package repositories that can be added to the portage tree. A good number of third party overlays are available in necessary.

Local Overlay

Create a local overlay to create your own ebuilds or edit an existing one. If the later, it’s best to see if the program has an ebuild in bugzilla or is in one of the third party overlays.

To make a personal overlay create an overlay directory and let make.conf known to it (for example) mkdir /home/user/.portage-local and add to make.conf:

PORTDIR_OVERLAY="/home/username/.portage-local"

Ebuilds must be placed in a category that already exists in Portage

mv package.ebuild ~/.portage-local/media-plugins/

Keyword if necessary:

emerge gentoolkit-dev
ekeyword ~ppc package-1.2.1.2.ebuild

Create a manifest and build:

ebuild ~/.portage-local/category/program/program-version.ebuild digest
emerge package

Layman

Layman is a manager for third party overlays.

emerge layman subversion

Add to make.conf:

PORTDIR_OVERLAY="/home/username/.portage-local"
source source /var/lib/layman/make.conf

To add an overlay:

layman --list
layman --add overlay

Update layman overlays:

layman -S

Creating Quality Web Sized PNGs with ImageMagick

Note: This is a post with me messing around with ImageMagick and it’s good food for thought, however as a reader pointed out I accidentally started with a lossy image and turned it to a lossless image which was a mistake on my part.

I recently posted The Battle for Wesnoth – 1.4 picture review, a look at the new version of turn based strategy game. I had uploaded the post and the screenshots from a super fast pc at work. I gave it no thought when it was as easy as clicking a button. No thought until I loaded the blog at home yesterday and I patiently waited as the browser icon kept looping. Scrolling to the Wesnoth post and seeing images were only half loaded, I looked into my images folder and discovered my Wesnoth screenshots took up a whooping 4MB. Yikes! To make matters worse, I had lazily used html image resizing for thumbnails. This was a time warp I’d forgotten with fast networking but a basic html blunder: when writing for a general audience, write for the lowest common denominator. Scrolling my other posts and their images were still loading I thought, if I had visited this page for the first time and it was taking this long to load, I probably would have closed the tab by now.

And such it is on todays modern web. Not everyone has that T3 spliced into their home. On this slow connection, I have learned to be patient but I really like those pages that load quickly. Portable Network Graphics (PNG)’s the graphic format I use are designed for modern networks. A lossless format for good quality images with a reasonable file size for optic networks, cable modems and dsl. A good number of people though still use standard phone modems or experience congested networks and some images can take awhile to download. Well compressed PNG’s can be made though that load reasonably well while maintaining good image quality.

So I decided to find out what I could do to reduce the load of my web graphics. I did numerous tests over differing quality and quantity. I tested all conversions from a standard jpeg photo. The image is scaled down to 500 width (aspect preserved), a good size for viewing on this blog. First of all I have to say that GIMP does this very well – accurate color reproduction, smoothing, no color bleeds, all in a 270K file.

But often I don’t wanna fire up gimp just to convert one image in such instance I like to use ImageMagick’s command line programs: convert/mogrify.

Resizing is simple:

convert -resize 500 a-friend-in-need.jpg a-friend-in-need-500.png

The image produced is as good as quality as the one produced by GIMP but doesn’t particularly do well at compression this image is a whopping 871K which explains alot about my web page load yesterday. So next I reduced the colorspace to 256 to see if this could produce an acceptable image:

Not too bad. At 109K its a good size for the internet but after looking awhile the dithering becomes obvious. The number of colors is a big factor in determining the size of the image. I played with the colors a bit and came to that around 2000 colors as images began to look non-dithered.

convert -resize 500 -colors 2048 -depth 16 -quality 95 a-friend-in-need.jpg a-friend-in-need-500-c256.png

Quality is the compression amount and type. The first digit is the compression level (nine is the highest) and the next digit is compression type (five is adaptive). I also used optipng that can add up to 30% further compression.

optipng -o5 a-friend-in-need-500-c256.png

The images still look a bit too sharp so I used the enhance variable to smooth. Enhance is it’s own filter and will over-rule other variables so setting color and depth needs to be done in another step.

Doing these steps gives me nice quality, lightweight (comparably – 280k) images that I can use for my blog. The final product:

Creating Quality Web Sized PNGs with Imagemagick

To simplify the process, I wrote a script to make it easier:

#!/bin/bash
# resize-image-new - makes new image and resizes.
# http://www.imagemagick.org/www/command-line-options.html

SIZE=$1
NAME=$2
COLORS=2048
DEPTH=16
RESIZEDNAME="${NAME%.*}"-"$SIZE".png
#ENHANCE - options to enhance image -enhance smoothes rough images

if [[ -z $NAME ]]; then
    echo "resize-image <WIDTHxHEIGHT> <original-image>"
    exit;
fi

# Convert (SIZE is proportional least value is used and only x needs specifid.)
convert -resize "$SIZE" -enhance "$NAME" "$RESIZEDNAME"

mogrify -colors $COLORS -depth $DEPTH "$RESIZEDNAME"

# Compress PNG
optipng -o5 "$RESIZEDNAME" 

I’d like to be able to learn if I could adapt this script to batch conversions.

 

Command Line to Clipboard

Update: Script has been updated to add pipe support. Thanks to Nathan who allowed me to use his improvements.

It’s really something to be learning Linux. The more I learn about Linux the more I learn it’s about manipulating letters and numbers (well, this is more programming than anything but Linux is a lot about that). Bash I’m discovering is great; I’m just getting into it and now have made things a good deal easier by learning how to copy and paste text from the command line via the Xorg server clipboard. Here is a couple commands that can do it with examples, following them are a couple bash scripts that make this easy as can beasy.

The Programs

xsel and xclip are command line programs that can redirect the contents of the Xorg server clipboard. The Xorg server has two clipboards: the common right-click > Copy, and one for the middle-mouse click. For those that don’t know of it yet, the middle-click clipboard allows quick copy and pasting without having to enter a menu or using Ctrl + v. Anytime you select text on the Xorg server there is a separate register that records this text, then clicking the middle-mouse button (sometimes called the mouse button three [usually done by clicking down the scroll wheel] will paste the text. The Xorg server defines the the middle-click clipboard as primary and the right-click clipboard as secondary.

xclip

Here are the basics of using xclip. xclip, I prefer over xsel because I have found that xsel can have problems pasting to java apps.

xclip can be used in a variety of ways. First, for example, it can be piped to:

echo "hi" | xclip -selection clipboard

This will copy to the standard clipboard. For abbreviation, you can use c instead of clipboard. You can specify primary or p here too to copy to the middle-mouse button, but isn’t necessary as this is the default for xclip.

echo "hello" | xclip

To direct a file to xclip the -in or -out options are needed:

xclip -in -selection c <filename>
xclip -out -selection c <filename>

Which will respectively put a file into the clipboard, and write to a file from the clipboard contents.

To make the process quicker, I’ve created a couple scripts to automate the tasks called cb-in and cb-out and can be used like a standard command:

cb-in pack 
 File pack copied to the clipboard

cb-in

cb-out

xsel

To copy to the context-menu clipboard:

xsel --clipboard < /etc/fstab

To copy a text to the middle mouse button clipboard:

xsel < /etc/fstab

xsel can be piped too:

echo "a-bit-of-text" | xsel -b
cat /etc/make.conf | xsel -b

To output directly to the terminal:

xsel --clipboard

And to redirect and append to a file:

xsel --clipboard > Baada-Boom.txt
xsel --clipboard >> ~/.Baada-Boom

cp2clip (xsel)

#!/bin/bash
# cp2clip - copy to the clipboard the contents of a file

# Program name from it's filename
prog=${0##*/}

# Text color variables
bldblu='\e[1;34m'         # blue
bldred='\e[1;31m'         # red
bldwht='\e[1;37m'         # white
txtbld=$(tput bold)       # bold
txtund=$(tput sgr 0 1)    # underline
txtrst='\e[0m'            # text reset
info=${bldwht}*${txtrst}
pass=${bldblu}*${txtrst}
warn=${bldred}!${txtrst}

filename=$@

# Display usage if full argument isn't given
if [[ -z $filename ]]; then
    echo " $prog <filename> - copy a file to the clipboard"
    exit
fi

# Check that file exists
if [[ ! -f $filename ]]; then
  echo -e "$warn File ${txtund}$filename${txtrst} doesn't exist"
  exit
fi

# Check user is not root (root doesn't have access to user Xorg server)
if [[ $(whoami) == root ]]; then
  echo -e "$warn Must be regular user to copy a file to the clipboard"
  exit
fi

# Copy file to clipboard, give feedback
xsel --clipboard < "$filename"
echo -e "$pass ${txtund}"${filename##*/}"${txtrst} copied to clipboard"

clippaste (xsel)

#!/bin/bash
# clippaste - Paste contents of clipboard to file in terminal.
# use 'xclip -out -selection primary' for middle click clipboard

# Program name from it's filename
prog=${0##*/}

# Text color variables
bldblu='\e[1;34m'         # blue
bldred='\e[1;31m'         # red
bldwht='\e[1;37m'         # white
txtbld=$(tput bold)       # bold
txtund=$(tput sgr 0 1)    # underline
txtrst='\e[0m'            # text reset
info=${bldwht}*${txtrst}
pass=${bldblu}*${txtrst}
warn=${bldred}!${txtrst}

filename=$@
pasteinfo="clipboard contents"

# usage if argument isn't given
if [[ -z $filename ]]; then
  echo "clippaste <filename> - paste contents of context-menu clipboard to file"
  exit
fi

# check if file exists, prompt to append or override, else create new
if [[ -f $filename ]]; then
  echo -en "$warn File ${txtund}$filename${txtrst} already exists - (${txtbld}e${txtrst})xit, (${txtbld}a${txtrst})ppend, (${txtbld}o${txtrst})verwrite: "
  read edit
  case "$edit" in
    [aA] )  xsel --clipboard >> $filename
            echo -e "$pass File ${txtund}$filename${txtrst} appended with clipboard contents"
            ;;
    [oO] )  xsel --clipboard > $filename
            echo -e "$pass File ${txtund}$filename${txtrst} overwrote with clipboard contents"
            ;;
    * )     exit
    esac; else
    xsel --clipboard >> $filename
    echo -e "$pass File ${txtund}"$filename"${txtrst} created with clipboard contents"
fi

Purge memory trick

Purge Memory

Linux does a good job when it comes to memory allocation. If memory isn’t being used or hasn’t been used for awhile it gets put into a cache where it can readily pulled. At times though this cache can become pretty big (especially for a program that has a memory leak). If a good number of programs are being used or if memory is limited then Linux will begin using hard disk swap which can really bog down performance. In these instances, it may help to purge the memory.

In the terminal type free -m to see memory usage. Flushing the filesystem buffers and to drop extra caches can be done by doing:

sudo sync
sudo echo 3 | sudo tee /proc/sys/vm/drop_caches

Look once more at free -m and memory usage should be improved. Freeing memory is most effective by shutting down whatever programs can be. To be really effective shutdown X server first.

This can be put in a script if you need to regularly do this:

Resources

Background a Process/Program

At times it is useful to see the output of what a program produces by typing its command line name in the terminal (for instance for debugging), at other times typing a program in the terminal just takes up space that could ordinarily be utilized for something else. Launching programs from the terminal can be put in the background easily with a bash script.

nohup is used to prevent hangups and then you redirect the output of the command to /dev/null (the great Linux blackhole). Here’s the script:

Then in the terminal use the bgcmd command with whatever program needed to be put in the background:

Backgrounding Already Running Processes

Already running applications can be backgrounded as well. First type Ctrl + Z to release the application, then use bg to background it’s output.

Keep in mind though that if the terminal or tab is closed the program will close with it. Also too the bg command doesn’t suppress all output.

Follow

Get every new post delivered to your Inbox.

Join 52 other followers