Resurrecting GNU Screen Sessions After Reboot

Mbm329 04:38, March 12, 2012 (UTC)

Should you use your shell server for any length of time, you will undoubtedly need to reboot it due to kernel patching, or possibly faulty hardware. When you do this, all your screen sessions will die. Unfortunately, this is a byproduct of how screen was designed. All your session data resides in volatile memory. Screen communicates to this memory through unix sockets. So when your system reboots, the sockets will exist, but the session information that once was stored in memory will be lost.

To make this less of a burden, I wrote a script that will attempt to do the following:
 * Re-create the screen windows
 * Populate them with the session history you once had before the reboot
 * Log you into your previously logged in host via ssh
 * Place you into the same working directory you were once in.

Most of this data is derived from the prompt. Running any previously ran commands would be extremely dangerous, so this is as far as I can get you.

Basically, the way this is accomplished is via a cron that runs as often as you would like snapshots of your sessions. I prefer to have it run only when I'm actively using my sessions (basically my standard workday (every hour, from 7AM to 7PM, Mon. - Fri.). You might ask, "Why not just every minute, 24 hours a day?" The answer is - Because if you did it that often, should your system reboot on its own, you would have just potentially wiped out the "good" state with crap data. By sticking to the hours you work, you would lessen the risk of overwriting good data with bad states (or no states at all). The script will keep X amount of copies so you an roll back to an hour or day that you would like. Should you feel you would work at any time of the day, you should keep more copies of snapshots. Here's the basics for configuration:

Settings
There are four main settings with the resurrect script. max_backups=12 ignore_sessions="hosts" backup_dir=~/.resurrect/screen_snapshot
 * 1) Config Variables
 * 1) Config Variables

get_state { prompt='^\<.*@.*:' prompt_line=$(grep $(eval echo ${prompt}) ${window_full} | tail -n 1) host=$(printf "${prompt_line}" | cut -d@ -f2 | cut -d: -f1) working_dir=$(printf "${prompt_line}" | cut -d: -f2 | cut -d\> -f1) }
 * 1) Use prompt to get state of window.
 * 1) Use prompt to get state of window.


 * 1) max_backups: I set this to 12 as I backup each hour for 12 hours. The earliest backup gets removed, and every backup up to the latest gets moved into the next larger number.
 * 2) ignore_sessions: This is simply a list of session names that you would like to ignore from being backed up. These may be ignored because you have a special .screenrc that creates them, or they do not follow your convention for how your prompt is setup. For example, I have a session called "hosts" that is a login into most of our hosts in case my account expires accidently, I can still get into the host. This session is created by a special .screenrc config I use.
 * 3) backup_dir: This is simply where you would like your latest snapshot stored. I store mine in ~/.resurrect/screen_snapshot.
 * 4) prompt: This is probably the most important setting as it is used to determine how to rebuild your sessions' states.

Backup
The script has two options -b for backup and -r for resurrect. The cron will run the -b option. This cron should also be run as your user account (not root). 0 7-19 * * 1-5 /full/path/to/resurrect -b
 * 1) Backup screen sessions

Session Repository
In the script, I store the screen sessions snapshots in ~/.resurrect. This session directory contains a snapshot of each of it's windows. $ ls ~/.resurrect screen_snapshot   screen_snapshot.10  screen_snapshot.12  screen_snapshot.3 screen_snapshot.5 screen_snapshot.7  screen_snapshot.9 $ ls ~/.resurrect/screen_snapshot 10621.user_cleanup 12523.rsync  12877.oracle_mail_cleanup  14735.unavail 15336.dr 15764.prog  23356.vgapps  26098.base  26844.monitor  7056.samba $ ls ~/.resurrect/screen_snapshot/26098.base 0 1  10  11  2  3  4  5  6  7  8  9

Script (resurrect)

 * 1) !/bin/sh

usage { echo "USAGE: ${0} -b | -r [backup_number]" exit 1 }
 * 1) Usage
 * 1) Usage

max_backups=12 ignore_sessions="hosts" backup_dir=~/.resurrect/screen_snapshot
 * 1) Config Variables
 * 1) Config Variables

get_state { prompt='^\<.*@.*:' prompt_line=$(grep $(eval echo ${prompt}) ${window_full} | tail -n 1) host=$(printf "${prompt_line}" | cut -d@ -f2 | cut -d: -f1) working_dir=$(printf "${prompt_line}" | cut -d: -f2 | cut -d\> -f1) }
 * 1) Use prompt to get state of window.
 * 1) Use prompt to get state of window.

if [ ! ${1} ] ;then usage fi
 * 1) Code
 * 1) Code

while [ ${1} ] ;do case ${1} in   -b)      task=backup      shift 1    ;;    -r) task=restore if [ ${2} ] ;then backup_number=${2} shift 2 else shift 1 fi   ;; *)     usage    ;;  esac done

if [ "${task}" = 'backup' ] ;then ##Rotate previous backups. i=${max_backups} while ${i} != 0  ;do if [ -d ${backup_dir}.${i} ] ;then if ${i} = ${max_backups}  ;then rm -r ${backup_dir}.${i} else mv ${backup_dir}.${i} ${backup_dir}.$((${i}+1)) fi   fi    i=$((${i}-1)) done if [ -d ${backup_dir} ] ;then mv ${backup_dir} ${backup_dir}.1 fi

##Dump hardcopy from all windows in all available screen sessions. if [ ! -d ${backup_dir} ] ;then mkdir -p ${backup_dir} fi for session_dir in $(screen -ls | grep tached\)$ | awk '{print $1}') ;do    session=$(echo ${session_dir} | cut -d. -f2)   for ignore_session in ${ignore_sessions} ;do      if [ ${session} = ${ignore_session} ] ;then        continue 2      fi    done    if [ ! -d ${backup_dir}/${session_dir} ] ;then      mkdir -p ${backup_dir}/${session_dir}    fi    for win in $(seq 0 30) ;do      window=${backup_dir}/${session_dir}/${win}      screen -S ${session_dir} -p ${win} -X hardcopy -h ${window}      if [ ! -s ${window} ] ;then        sleep 1        rm ${window}      fi    done  done

elif [ "${task}" = 'restore' ] ;then

##Check for specified number backup. If none, then use latest. if [ "${backup_number}" != '' ] ;then backup_dir=${backup_dir}.${backup_number} fi

##Restore sessions and windows from backups. for session_dir in $(ls ${backup_dir}) ;do session_full=${backup_dir}/${session_dir} session=$(echo ${session_dir} | cut -d. -f2) screen -d -m -S ${session} i=0 for window in $(ls ${session_full}) ;do window_full=${session_full}/${window} get_state echo "${session}:${window}:${host}:${working_dir}" if [ ${window} -ne 0 ] ;then screen -S ${session} -p \- -X screen screen -S ${session} -p ${i} -X number ${window} fi "    screen -S ${session} -p ${window} -X stuff "cat ${window_full} if [ "${host}" != "$(hostname)" ] ;then "      screen -S ${session} -p ${window} -X stuff "ssh ${host} sleep 2 fi "    screen -S ${session} -p ${window} -X stuff "cd ${working_dir} if [ ${window} -eq ${i} ] ;then i=$((${i}+1)) fi   done done

fi