Extreme Multitasking with GNU Screen + Tabbed Client

Mbm329 16:45, March 13, 2012 (UTC)

WARNING: THIS DOCUMENT IS HERE FOR LEGACY PURPOSES ONLY. IF YOU CURRENTLY USE GNU SCREEN, MORE POWER TO YOU. I DECIDED TO SWITCH TO TMUX AS IT COULD DO EVERYTHING THAT SCREEN COULD DO FOR ME AND MORE WITH LESS OVERHEAD AND NO NEED OF A SPECIAL CLIENT. PLEASE REFERENCE MY SIMILAR DOC FOR TMUX CONFIGURATION.

There are plenty of methods people use to administer UNIX hosts. From separate telnet windows directly into the individual hosts, to tabbed terminal clients. Everybody has their own preferences. That is what this doc is; my preference.

This method of accessing UNIX hosts pretty much involves a tab for each "project", and then "tabs" within those tabs indicating which host I'm logged into when the tab is idle, and what command is being run when it's not. This is purely my preference, and it works for me. Breaking away from a single window per system methodology is a lifestyle change. It does take some getting used to. But after using this, I would probably never go back.

The benefits you will gain with the following guide:


 * Instantly pick up where you left off after a crash/reboot/network disconnect/etc...
 * Access all your hosts seamlessly by entering a single password once (each time you boot your laptop/desktop).
 * Single window for accessing all hosts as many times as you want, virtually uncluttered.
 * Simplistic separation of projects or tasks from one another.
 * Always know which host you're logged into.
 * Always know what command is running in your sessions at a glance.



Shell Server
The shell server is key to centralized management of an array of systems. From the shell server, you can have a single point of contact to access all systems and host scripts and utilities that assist in the management of your systems. Another important key aspect of the shell server is that it is not a system that would need to be shutdown when you go home, and less likely to crash than your laptop w/ a billion browser tabs open or that you just recompiled your kernel on so you could get support for that new camera you just bought.

I happen to favor a Linux system for a shell server as it is extremely versatile in capability, and usually has all the luxury tools available at it's disposal. Software outside of package management is generally easy to compile and less error-prone during build times.

Simply put, the shell server can be any system that is in your place of business running 24/7 that will act as your gateway to all your systems. With this setup, you generally would not access your systems directly from your workstation. It's worth noting that the shell server does not have to be a new expense to your business. Any *nix system that can run SSHD and GNU Screen will do; a virtual machine running Linux does the trick for me.

SSH Daemon
To communicate with your systems, you will want to use SSH. Telnet is older and less secure. And forget about that $150 suite of software that acts as an X-server+remsh client that, in turn, runs xterm on the UNIX server you're trying to access. SSH can provide key authentication, port tunneling, X11 forwarding, and much other goodness. This is a necessity. Oh, btw, don't pay for it; it's free. Use OpenSSH, the others don't get you anything special no matter how much they say they will. By now, most UNIX/Linux/BSD systems run ssh as standard.

Used ssh before, but never setup the publickey access because it seemed too complicated? No worries, I'll take you through it further in the doc. It's simple. You will wonder how you ever got along without it.

GNU Screen
Remember the shell server? Remember when I mentioned that you don't have to shut it down when you go home? Run GNU Screen on your shell server, and you will be able to keep your terminal sessions open when you shutdown your laptop and take it home. GNU Screen will spawn it's own session once you're on the shell server. You can do whatever inside the screen session, then detach from the session and go home. This will be key as you will not have to remember where you left off on certain windows or recover from a dropped wifi connection when you're on-call at a Waffle House at 2AM trying to sober up. Yes, you know you've been there. If not, you will be now.

GNU Screen basically has two levels of abstraction. The session. This level occurs automatically when you spawn Screen. Each session can have multiple windows. While in screen, all commands you feed to it begin with "^a" (that's CTRL+A). To open a new window within a screen session, you would use ^a,c. You now would have two windows within the screen session. More details to follow!

Like with SSH, most variants and flavors of UNIX have a GNU Screen package available for install.

Terminal/SSH Client
So you got all this goodness, how the hell are you gonna interface with it? You need a terminal/ssh client. It's pretty simple here. If your workstation/laptop is running Linux, you most likely have an assortment of flavors to choose from, all free. If you're running Windows, I'd go with PuTTY. I will say this though: make sure it has tabs. Preferrably w/ the ability to remember the names of your tabs (we'll cover this later). Tabs will be helpful to your management of the connections to your systems. Similar to how tabs are helpful to your web browsing needs, they keep windows in one central place, semi-managable.

If you're running Linux, there's Gnome Terminal and Konsole among others. I'll use Gnome Terminal throughout this guide. If you're on Windows, PuTTY Connection Manager (PCM) will wrap PuTTY windows into tabs. I'll also be covering PCM within this document. If you've got money burning a hole in your pocket, SecureCRT is pretty decent also and has tab capability.

Getting from your laptop to your shell server
First things first. You need to connect to your shell server using your terminal client. Sure, you could just open your terminal client up and make an ssh connection to your shell server. But we're a little more hardcore than that. When we can spend a little time up front to setup things, it will save us time on the back end applying them. Trust me, when you can enter your password once, make two mouse clicks to have 15 tabs open, and re-attach to the 40+ connections you had open into your server farm yesterday before your workstation crashed or you recompiled your kernel for that new supercool USB gadget, you will appreciate it.

Generate Key Pair
Remember when I said ssh has key authentication? Key authentication will allow you to enter your password once from your laptop to access the shell server as many times as you need to. You'd be a fool not utilize such delicious functionality.
 * Windows: Download PuTTY's "PuTTYgen" utility and execute it.  During key generation, you will need to enter a passphrase.  If you don't, and your key is stolen, it can be used to access all the systems you can access.  This is extremely important.  Remember, I said you could just enter the password once on your system and connect to the shell server as many times as you want.  It's a small price to pay for peace of mind (and in many cases, your employment).  After your key is generated, save the public key and the private key to a directory only you have access to if other users share your laptop ("My Documents" or something).

$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/mbm/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/mbm/.ssh/id_rsa Your public key has been saved in /home/mbm/.ssh/id_rsa The key fingerprint is: ce:c4:a5:06:3a:c0:e2:8b:a1:85:1c:9d:64:87:c8:49 mbm@hostname
 * Linux: Execute /usr/bin/ssh-keygen and follow the prompts.  Your key should be stored in ~/.ssh.  Read the "Windows" section above about entering passwords.

Place Public Key on Shell Server
Place the contents of the public key that was generated into ~/.ssh/authorized_keys file on the shell server, ensuring no linebreaks are present. Also, re-perm this file so you are the only one able to read/write to it. $ chmod 600 ~/.ssh/authorized_keys

SSH Authentication Agent
Here is where the ability to enter one password for all the connections to your shell server happens.
 * Windows: Download PuTTY's "Pageant" utility and execute it.  An icon will appear in your taskbar.  Right-click on it, and select "Add Key".  Give it the path to where your private key (.ppk) is stored.  I've found that it's best to create a shortcut to pageant.exe in the "Startup" folder under the Applications menu so that it runs on bootup.  Right-click the shortcut, choose "Properties", and use the following as the "Target" line:  "C:\full\path\to\pageant.exe" "C:\full\path\to\rsa.ppk".  This will also be the window that will show when your system boots if you placed the shortcut in your Startup folder.



SSH_ENV="$HOME/.ssh/environment" function start_agent { echo "Initialising new SSH agent..." /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}" echo succeeded chmod 600 "${SSH_ENV}" . "${SSH_ENV}" > /dev/null usr/bin/ssh-add; } if [ -f "${SSH_ENV}" ]; then . "${SSH_ENV}" > /dev/null ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || { start_agent; } else start_agent; fi
 * Linux: Place the following script in your .profile (you may use .bashrc or .bash_profile):
 * 1) Credit: http://mah.everybody.org/docs/ssh#run-ssh-agent
 * 1) Source SSH settings, if applicable

Configure PuTTY (Windows Only)
Setup a profile in putty by executing the PuTTY binary. Your profile window will open, and you can enter a new name for your session. Perhaps the hostname, or maybe just "Shell Server". I went with naming it "publickey_authentication". You won't need to enter a hostname as that is what PCM will handle; you are simply building a profile. In this window, you can select other options you would like to use. Here are some I would configure: After you've configured everything, head back over to the "Session" screen and select, "Save".
 * 1) Window -> Lines of Scrollback = 1000 (For the "Oops, what did I do" moment)
 * 2) Window -> Translation -> Received data assumed to be in which character set = UTF-8 (Needed for some ncurses-based applications like "dialog")
 * 3) SSH -> Protocol="2 only"
 * 4) SSH -> Auth -> Attempt authentication using Pageant (Most Important One)
 * 5) SSH -> Auth -> Private key file for authentication = c:\full\path\to\rsa.ppk
 * 6) SSH -> X11 -> Enable X11 forwarding (Important if you use X11 applications that need to display to your laptop.)

Firing up your ssh authentication agent for the first time

 * Windows: Honestly, I'd just go ahead and reboot to make sure Pageant starts up and prompts for a password on bootup.  You're running Windows, you should be used to this. :)
 * Linux: Open another terminal window to read your .profile with added ssh-agent code. You should be prompted for your key's password.

SSH To Shell Server
Lets see how the password-less login works out.
 * Windows: Open PuTTY, double-click your saved profile, type in @ at the blank at the top, and click "Open".
 * Linux: Open Gnome Terminal, and type "ssh @ ".

If this is your first time accessing the shell server, you will be prompted to accept the SSH server's host public key.

Setup SSH Key access from shell server to all hosts
You will want to run commands in bulk in a lot of circumstances. You will not want to have to enter a password. Similar to how you don't want to enter one for logging into the shell server. The good news is, when you set it up, it'll run until either the ssh-agent process dies, or when the system reboots. If the ssh-agent process dies, just login again and it'll get fired back up. For instructions on how to do this, see the sections above detailing the generation of keys for and setup of the ssh-agent under Linux. Setup the ssh-agent in the shell server's ~/.profile. And put the public key in the ~/.ssh/authorized_keys file of all the hosts you access. Once this is done, scp a file to your homedir on all hosts as a test. A simpler way to do this would be to place the public key of the shell server into the authorized_keys file on the shell server, then scp the authorized_keys file to all hosts you would be accessing. This is assuming there are no other public keys listed in the remote hosts' authorized_keys file for this user. $ for host in host1 host2 host3 host4 host5 host6 ;do echo ${host} ;scp -p ~/.ssh/authorized_keys ${host}:~/.ssh/authorized_keys ;done

If you know you do not have .ssh directories in your homedir on each host, you should create it first like so: $ for host in host1 host2 host3 host4 host5 host6 ;do echo ${host} ;ssh ${host} "mkdir ~/.ssh ;chmod 700 ~/.ssh" ;done

Setting up GNU Screen on The Shell Server
The setup of screen is, for the most part, done for you to some degree. Usually screen will come with a default config file and all you really need to do is create your own additional features, or deviations from the global config. These user-specific configs are held in the ~/.screenrc file. Here are some configurations I use in my setup.

Caption/Hardstatus
GNU Screen comes w/ the ability to list all windows associated w/ the session you're in (^a,"). The list is selectable using the up/down arrows.  I've found this cumbersome and a pain if you are switching between windows a bit.  Why not have them on your screen while you're working so you can know which window you would like to switch to?  Use the "caption" line.  Example:

caption always '%{= kg}[%L=%{+u b} %=%{-u W}%?%-Lw%?%45L>%{Y}(%n*%f %t)%?(%u)%?%{W}%+Lw%{+u b} %=%-1=%{-u g}]'

...will produce the following. Window 1 is the current window. Therefore, it is highlighted yellow. If you switched to window 3, the highlighting would move to window 3. There are other goodies you can include in this caption line, but I've found I like to have as much screen realestate as possible on this line. Also of note, if you are using Gnome Terminal and you notice that your caption line is a lighter color than the rest of the screen, you should check your color schemes in your preferences.

As I said, I like to have maximum screen realestate for the caption line because of the amount of windows I may have open, but there are other things I like to see at the bottom of my screen window. The date/time is one of them. Also, I like to have a quick list of shortcuts for rarely used actions like splitting the window and logging the terminal output. Enter the "hardstatus" line. The hardstatus line will remain as a single line at the bottom of the session, even if you split your windows. However, your caption line will follow each window on a split.

ShellTitle
Knowing what host your window is logged into and what your window is doing is pretty helpful. If you have many hosts, you may want this to be the hostname of the host you're ssh'd into. If you're running a command, you may want to know what command is running in the window. Along with the "caption" config above, you can get this information easily by using the following line in your ~/.screenrc: shelltitle "$ |idle"

...And the following in your ~/.profile on all the hosts you connect to (including your shell server):

export PS1="\[\033]0;${USER}@${HOSTNAME}\007\]" ##display user@host in titlebar or "%h" screen string escape export PS1=${PS1}'\[\033k\033\\\]' ##display running command for window name in screen's caption line export PS1=${PS1}'\[\033k'${HOSTNAME}'\033\\\]' ##show hostname for window name on screen's caption line when idle export PS1='<\u@\h:\w>\n'${PS1}'\$ '
 * Linux:

Basically, in order to get the pathing set in the prompt, it takes a little bit more code, but the first three "export PS1" lines are the same as above except that because HPUX's posix shell cannot parse the octal characters, we have to place the literals by using the "CTRL+V, CTRL+[" and "CTRL+V, CTRL+g" keystrokes in their place. For the last "export PS1" line, we have to use a literal new line to have the prompt return to the next line. export USER=$(whoami) export HOSTNAME=$(hostname) mcd { my_path=${@:-} ##set path to homedir if $@ not set cd ${my_path} my_pwd=$(pwd) export PS1="^[]0;${USER}@${HOSTNAME}^G" ##display user@host in titlebar or "%h" screen string escape export PS1=${PS1}'^[k^[\\' ##display running command for window name in screen's caption line export PS1=${PS1}'^[k'${HOSTNAME}'^[\\' ##show hostname for window name on screen's caption line when idle export PS1="<${USER}@${HOSTNAME}:${my_pwd}> ${PS1}$ " } mcd alias cd='mcd'
 * HPUX:

After these two edits, your caption line will appear as such:

Notice that window 2 is running top and windows 0, 1, and 3 are logged into various hosts. The focused window is still window 1.

Brief Tutorial
A brief synopsis of how screen works. The man page has much more useful information:
 * 1) Log into machine.
 * 2) Run "screen"
 * ^a,c == creates new window
 * ^a,a == switches to previously accessed window
 * ^a,0 == switches to screen number 0
 * ^a,8 == switches to screen number 8
 * ^a,x == locks screen session w/ password of owning user
 * ^a,d == detaches from current session
 * ^a," == as indicated above, gives a listing of windows for the session
 * ^a, == copy mode (which also lets you utilize your GNU screen scrollback buffer); use vi-keys to navigate

Sample ~/.screenrc
For details on how to configure the caption and hardstatus line with proper padding, please refer to Truncation/Padding Escapes vbell off ##turn off visual-bell shell -${SHELL} ##gives a login shell defscrollback 30000 ##sets scrollback buffer autodetach on ##autodetach when terminal disconnects term vt100 ##set $TERM to xterm after screen generation shelltitle "$ |idle" ##tells what to title the window if its idle termcapinfo xterm ti@:te@ ##allows scrollbar to inherit current window data attrcolor i "+b" ##allow SecureCRT and possibly other terminal emulators show bright colors as bright colors (by using bold attributes) caption always '%{= kg}[%L=%{+u b} %=%{-u W}%?%-Lw%?%45L>%{Y}(%n*%f %t)%?(%u)%?%{W}%+Lw%{+u b} %=%-1=%{-u g}]' ##windowlist hardstatus alwayslastline '%L>%{= wk}[c=screen|d=detatch|K=kill]%=[ =copy|<]>=paste]%=[H=log|h=hardcopy]%=[S=split|Q=only|X=remove| =focus]%=[?=help]%=%-18=%{= kG}[ %m/%d %c:%s ]' ##help and date/time

Multiple Screen Sessions Inside Tabs Of Terminal Client
Using the method above makes for great organization and minimizes clutter on your screen. If you're a seasoned admin, and you have used multiple separate windows for some time, it does take some getting used to and you will have to re-train your brain for this new methodology. To take full advantage of the caption line, you will most likely want to use a full-screen window for your terminal window. Having two monitors helps. I use a laptop screen, and a monitor. The more monitors, the merrier, but if you use a terminal window for a living, at least one dedicated for terminal window works best.

There will be times when even the above organization becomes un-manageable. You could end up with 20 or more windows inside your screen session, and trying to remember what each is doing and in what order they are in can become unwieldy. To remedy this, we invoke the tabs of the terminal client. Remember earlier when I mentioned " preferrably w/ the ability to remember the names of your tabs"? Here's where that comes in handy. One thing you can do with both Gnome Terminal and PuTTY (with the help of the PCM), is to assign the tab to a purpose. This is easily accomplished via GNU Screen sessions on the shell server.

Essentially, each tab is attached to a screen session. Therefore, each screen session can have multiple windows, and each session can be for a particular project or purpose. This is MUCH MUCH easier to work with. Not to mention, they all keep running after you log off for the day and go home. With the ability of the client to remember names, you can easily open the client back up, and you're back to where you left off yesterday. Here's how to do it.

Gnome Terminal
Gnome Terminal is slightly easier to configure, yet, not as flexible for on-the fly connection creations as PCM is. Nonetheless, the objective is still there and works well. To configure Gnome Terminal, it's easiest to write a script and re-run the script when an edit has been made. I called this script "gtabs". Just replace "SHELL_SERVER" with the hostname of your shell server. Then as time goes, add or remove sessions from the "sessions_list" list.
 * 1) !/bin/sh

session_list=" base cluster new_session "

for session in ${session_list} ;do args="${args:-} --tab --title \"${session}\" -e \"ssh SHELL_SERVER screen -DR ${session}\"" done echo "${args}" | xargs /usr/bin/gnome-terminal

PuTTY Connection Manager
Within PCM you can setup configs for each connection. Some people would use this to manage connections to different hosts. However, with this method, we will use them to manage multiple connections to a single host. Each with it's own screen session.

When you first install PCM, your window will look like this (as reference):

Session Config

 * 1) On the "Connection Manager" tab (at the bottom right, next to the "PuTTY session" tab), right-click in the blank space above and select "Create Database".  Give it a name, (ex. "Connection Manager").
 * 2) This would be a good time to save the database.  This will create the database file on the system which will yield you less potential headaches when you close PCM for the first time.  File > "Save All Databases" > Make sure to use the full path to PCM and name it something like, "connection_manager".
 * 3) Answer "Yes" to the question: "Do you want to set "Connection Manager" as your default database opened at startup?"
 * 4) You can remove the directories it creates in there by default if you like and create a new directory under "Connection Manager" (I called mine "Screen Sessions").
 * 5) Right-click the directory you created, and select "New -> Connection".  This will place you on the "Connection" section of this window.
 * 6) In "Name" field, give your first session a name.  I called mine "base".  For me, this session is used for general purpose misc. activities that don't pertain to other more specific oriented sessions.
 * 7) Continue to the "Login Macro" section of the window.  In the "Login" field, type your username on the shell server.
 * 8) No password is necessary due to the Pageant setup we performed earlier.  It will utilize your Pageant ssh-agent.  However, because PCM seems to really really really want a password, just place a simple ":" in the Password field.  This will basically run an empty command in the shell after you are logged in via publickey auth thereby satisfying PCM and not typing your password on the commandline when you login.
 * 1) No password is necessary due to the Pageant setup we performed earlier.  It will utilize your Pageant ssh-agent.  However, because PCM seems to really really really want a password, just place a simple ":" in the Password field.  This will basically run an empty command in the shell after you are logged in via publickey auth thereby satisfying PCM and not typing your password on the commandline when you login.

Test The Configuration
Double-click the "base" listing. A new tab should be formed called "base" that will ssh to your shell server and spawn a new screen session on the shell server called "base".

Lets say you find yourself working on a project to cluster some servers. You want to create a new screen session to handle this so that windows you're working on within that project don't get mixed around with windows from other projects or daily work.


 * 1) You can easily create a new session from within PCM by right-clicking "base" and choosing "Copy".  Then, right-click the "Screen Sessions" folder and select "Paste".  A new connection will be created called "base".
 * 2) Right-click the second "base" listing, and select "Configuration".
 * 3) On the "Connection" tab, change the "Name" field to "cluster".
 * 4) On the "Login Macro" tab, change the parameter passed to the "screen -DR" command to "cluster".  Click "Apply", then "OK".
 * 5) Double-click the "cluster" listing, and watch it login and create a new session for you.  From there, you can open as many windows within your "cluster" screen session as you want.

When you're done with that tab, just log out of each screen window you have open in that tab. When you log out of the last one, the screen session will die off on the shell server. You can remove the "cluster" connection from PCM by right-clicking on the connection and selecting "Delete".

Further Configuration
PCM gives you the ability to remove some of the un-necessary toolbars also. With one of your connections open, you can right-click next to the open tab, and mouseover "View" and uncheck some of the toolbars you don't want to see. By selecting the pushpin in the top right corner of the "Connection Manager" frame, it will minimize to the right side of the window when not in use. This will give you maximum screen realestate with your terminal window.

There are a few extra things I do to slim down the PCM app and mitigate potential usability issues.
 * 1) I don't use the "Putty Sessions" tab next to the "Connection Manager" frame, so I choose to disable this from showing.  View > Deselect "PuTTY sessions"
 * 2) By default PCM will minimize to the taskbar's tray.  I like having an icon on the taskbar.  Tools > Options > General > Uncheck "Hide when minimized"
 * 3) PCM will ask to close the databases.  This is irritating enough to disable considering that it is also set up to auto-save the databases.  Tools > Options > Database > Uncheck "Ask confirm before closing database"
 * 4) Some of the hotkeys are also used within some vi commands, so I choose to disable them all (I don't do much with PCM in actual daily use anyways).  Tools > Options > Hotkeys > Uncheck "Enable hotkeys"

As mentioned above, I run PCM fullscreen with a mimimalist layout. When you're done adding connections, your window should look similar to the image at the top of this document. Also when you are finished for the day, you can just close PCM and your screen sessions should detach automatically. Simple as that.

X11 Forwarding Issues
You may notice if you use X11 Forwarding through SSH that with multiple screen sessions, your $DISPLAY might not be current. To remedy this, here's a script that can be placed in your ~/.profile. When you encounter this issue, just run "update_display" on the shell server, and give "xclock" a go. You should see the clock open up on your laptop. Close it, and login to any of your systems, and X11 should forward properly to your Xserver locally on your laptop. update_display { good_display=$(netstat -an | /bin/grep 0\ [0-9,:,.]*:60..\ | awk '{print $4}' | tail -n 1) good_display=${good_display: -2} export DISPLAY=${HOSTNAME}:${good_display}.0 }

Renumbering GNU Screen Windows
See the screnum script page for details.

Resurrecting GNU Screen Sessions After Reboot
See the resurrect script page for details.

Profiling Hosts
See the loadvars script page for details.