Resurrecting tmux Sessions After Reboot

Mbm329 04:41, March 12, 2012 (UTC)

Should you use your shell server for any length of time, you will undoubtedly need to reboot it due to kernel patching, or possibly faulty hardware. When you do this, all your tmux sessions will die. Unfortunately, this is a byproduct of how tmux was designed. All your session data resides in volatile memory. tmux communicates to this memory through unix sockets. So when your system reboots, the sockets will exist, but the session information that once was stored in memory will be lost.

To make this less of a burden, I wrote a script that will attempt to do the following:
 * Re-create the tmux sessions and windows
 * Populate them with the session history you once had before the reboot
 * Log you into your previously logged in host via ssh
 * Place you into the same working directory you were once in.

Most of this data is derived from the prompt ($PS1). Running any previously ran commands would be extremely dangerous, so this is as far as I can get you.

Basically, the way this is accomplished is via a cron that runs as often as you would like snapshots of your sessions. I prefer to have it run only when I'm actively using my sessions (basically my standard workday (every hour, from 7AM to 7PM, Mon. - Fri.). You might ask, "Why not just every minute, 24 hours a day?" The answer is - Because if you did it that often, should your system reboot on its own, you would have just potentially wiped out the "good" state with crap data. By sticking to the hours you work, you would lessen the risk of overwriting good data with bad states (or no states at all). The script will keep X amount of copies so you can roll back to an hour or day that you would like. Should you feel you would work at any time of the day, you should keep more copies of snapshots. Here's the basics for configuration:

Settings
There are four main settings with the tmux_resurrect script. max_backups=12 ignore_session_list="hosts" backup_dir=~/.resurrect/tmux_snapshot export PATH="${PATH}:/usr/local/bin"
 * 1) Config Variables
 * 1) Config Variables

mimic_state { prompt='^\<.*@.*:' prompt_line=$(grep $(eval echo ${prompt}) ${backup_dir}/${pane_full} | tail -n 1) host=$(printf "${prompt_line}" | cut -d@ -f2 | cut -d: -f1) working_dir=$(printf "${prompt_line}" | cut -d: -f2 | cut -d\> -f1)
 * 1) Use prompt to get state of pane.
 * 1) Use prompt to get state of pane.

if [ "${host}" != "$(hostname)" ] && [ "${host}" != "" ] ;then tmux send-keys -t ${pane_full} "ssh ${host}^M" echo "SSH'd to ${host} in ${pane_full}. Sleeping 5 sec..." sleep 5 fi tmux send-keys -t ${pane_full} "cd ${working_dir} " #carriage return from previous line }


 * 1) max_backups: I set this to 12 as I backup each hour for 12 hours. The earliest backup gets removed, and every backup up to the latest gets moved into the next larger number.
 * 2) ignore_sessions: This is simply a list of session names that you would like to ignore from being backed up. These may be ignored because you have a special .tmux.conf that creates them, or they do not follow your convention for how your prompt is setup. For example, I have a session called "hosts" that is a login into most of our hosts in case my account gets locked out accidentally, or if something seriously wrong occured and no logins are available, I can still get into the host. This session is created by a special .tmux.conf config I use.
 * 3) backup_dir: This is simply where you would like your latest snapshot stored. I store mine in ~/.resurrect/tmux_snapshot.
 * 4) prompt: This is probably the most important setting as it is used to determine how to rebuild your sessions' states. It is nothing more than a regex to get your prompt out of the sea of other data in your session windows. To give reference to what the regex above is looking for, with bash shell, my prompt is set with PS1='<\u@\h:\w>\n$ '

Backup
The script has two options -b for backup and -r for resurrect. The cron will run the -b option. This cron should also be run as your user account (not root).

Unfortunately, I could never find a way to keep the output of "set option: mode-keys -> vi" from being output when the resurrect -b command runs, and appending a ">/dev/null" to the crontab entry would hang the "tmux set-window-option mode-keys vi ; copy-mode -t" commands, I strip out any output (not stderr) with a crontab entry like so: 0 7-19 * * 1-5 /home/user/bin/resurrect -b | sed '/.*/d'
 * 1) Backup tmux sessions

Session Repository
In the script, I store the tmux sessions snapshots in ~/.resurrect. This session directory contains a snapshot of each of its windows. $ ls ~/.tmux_resurrect tmux_snapshot   tmux_snapshot.10  tmux_snapshot.12  tmux_snapshot.3  tmux_snapshot.5  tmux_snapshot.7  tmux_snapshot.9 tmux_snapshot.1 tmux_snapshot.11  tmux_snapshot.2   tmux_snapshot.4  tmux_snapshot.6  tmux_snapshot.8 $ ls ~/.resurrect/screen_snapshot base:0.0  base:5.0      foglight:1.0   foglight:2.0  foglight:8.0  monitor:4.0    rep_perf:2.0  rep_perf:8.0  rsync:4.0    storage:2.0  vacs:0.0  vacs:8.0 base:1.0  base:6.0      foglight:10.0  foglight:3.0  foglight:9.0  rep_perf:0.0   rep_perf:3.0  rep_perf:9.0  rsync:5.0    storage:3.0  vacs:1.0  vacs:9.0 base:10.0 base:7.0      foglight:11.0  foglight:4.0  monitor:0.0   rep_perf:1.0   rep_perf:4.0  rsync:0.0     rsync:6.0    storage:4.0  vacs:4.0 base:2.0  base:8.0      foglight:12.0  foglight:5.0  monitor:1.0   rep_perf:10.0  rep_perf:5.0  rsync:1.0     rsync:7.0    storage:5.0  vacs:5.0 base:3.0  base:9.0      foglight:13.0  foglight:6.0  monitor:2.0   rep_perf:11.0  rep_perf:6.0  rsync:2.0     storage:0.0  upgrade:0.0  vacs:6.0 base:4.0  foglight:0.0  foglight:14.0  foglight:7.0  monitor:3.0   rep_perf:12.0  rep_perf:7.0  rsync:3.0     storage:1.0  upgrade:1.0  vacs:7.0

Script (tmux_resurrect)
usage { echo "USAGE: ${0} -b | -r [backup_number]" exit 1 }
 * 1) !/bin/bash
 * 2) Usage
 * 1) Usage

max_backups=12 ignore_session_list="hosts" backup_dir=~/.resurrect/tmux_snapshot export PATH="${PATH}:/usr/local/bin"
 * 1) Config Variables
 * 1) Config Variables

mimic_state { prompt='^\<.*@.*:' prompt_line=$(grep $(eval echo ${prompt}) ${backup_dir}/${pane_full} | tail -n 1) host=$(printf "${prompt_line}" | cut -d@ -f2 | cut -d: -f1) working_dir=$(printf "${prompt_line}" | cut -d: -f2 | cut -d\> -f1)
 * 1) Use prompt to get state of pane.
 * 1) Use prompt to get state of pane.

if [ "${host}" != "$(hostname)" ] && [ "${host}" != "" ] ;then tmux send-keys -t ${pane_full} "ssh ${host} " #carriage return from previous line echo "SSH'd to ${host} in ${pane_full}. Sleeping 5 sec..." sleep 5 fi tmux send-keys -t ${pane_full} "cd ${working_dir} " #carriage return from previous line }

if [ ! ${1} ] ;then usage fi
 * 1) Code
 * 1) Code

while [ ${1} ] ;do case ${1} in   -b)      task=backup      shift 1    ;;    -r) task=restore if [ ${2} ] ;then backup_number=${2} shift 2 else shift 1 fi   ;; *)     usage    ;;  esac done

if [ "${task}" = 'backup' ] ;then ##Rotate previous backups. i=${max_backups} while ${i} != 0  ;do if [ -d ${backup_dir}.${i} ] ;then if ${i} = ${max_backups}  ;then rm -r ${backup_dir}.${i} else mv ${backup_dir}.${i} ${backup_dir}.$((${i}+1)) fi   fi    i=$((${i}-1)) done if [ -d ${backup_dir} ] ;then mv ${backup_dir} ${backup_dir}.1 fi

##Dump copy from all windows in all available tmux sessions. if [ ! -d ${backup_dir} ] ;then mkdir -p ${backup_dir} fi for session in $(tmux list-sessions | awk -F: '{print $1}') ;do for ignore_session in ${ignore_session_list} ;do if [ "${session}" = "${ignore_session}" ] ;then continue 2 fi   done session_size=$(tmux list-sessions | awk -F'[' '/^'${session}': / {print $2}' | awk -F']' '{print $1}') echo "${session_size}" >${backup_dir}/${session}.size for window in $(tmux list-windows -t ${session} | awk -F: '{print $1}') ;do for pane in $(tmux list-panes -t ${session}:${window} | awk -F: '{print $1}') ;do ##backup pane layout if [ ${pane} -gt 0 ] ;then tmux list-windows -t ${session} | awk -F'\\[layout ' '/^'${window}':/ {print $2}' | awk '{print $1}' | sed 's/\]$//' >${backup_dir}/${session}:${window}.layout fi       pane_full=${session}:${window}.${pane} ##figure out the editing mode so we can select text in history mode_keys=$(tmux show-window-options -g | awk '/^mode-keys/ {print $2}') if [ "${mode_keys}" = 'emacs' ] ;then tmux copy-mode -t ${session}:${window}.${pane} \; send-keys -t ${session}:${window}.${pane} 'M-<' 'C-Space' 'M->' 'C-e' 'M-w' \; save-buffer -b 0 ${backup_dir}/${pane_full} \; delete-buffer -b 0 elif [ "${mode_keys}" = 'vi' ] ;then tmux copy-mode -t ${session}:${window}.${pane} \; send-keys -t ${session}:${window}.${pane} g Space G '$' Enter \; save-buffer -b 0 ${backup_dir}/${pane_full} \; delete-buffer -b 0 fi       if [ ! -s ${backup_dir}/${pane_full} ] ;then rm ${backup_dir}/${pane_full} fi     done done done

elif [ "${task}" = 'restore' ] ;then ##Allow for base-index configuration base_index=0 if [ -f ~/.tmux.conf ] ;then if grep -v ^# ~/.tmux.conf | egrep 'base-index +[0-9]' >/dev/null ;then base_index=$(grep -v ^# ~/.tmux.conf | egrep 'base-index +[0-9]' | awk -F'base-index ' '{print $2}') fi fi

##Check for specified number backup. If none, then use latest. if [ "${backup_number}" != '' ] ;then backup_dir=${backup_dir}.${backup_number} fi

##Restore proper number of sessions, windows, and panes for session in $(ls ${backup_dir} | grep -v '\.size$' | cut -d: -f1 | sort -du) ;do window_list=$(ls ${backup_dir} | awk -F: '/^'${session}':/ {print $2}' | cut -d\. -f1 | sort -nu) window_list_rev=$(echo "${window_list}" | sort -nr) num_windows=$(echo ${window_list} | wc -w)

##create session, and first window. echo "* Creating new session, \"${session}\"..." session_width=$(cut -d x -f 1 ${backup_dir}/${session}.size) session_height=$(($(cut -d x -f 2 ${backup_dir}/${session}.size)+1)) tmux new-session -d -s ${session} -x ${session_width} -y ${session_height}

##create additional windows if # of windows is > 1 if [ ${num_windows} -gt 1 ] ;then echo "* Creating $((${num_windows}-1)) additional windows for session, \"${session}\"..." i=1 while [ ${i} -lt ${num_windows} ] ;do tmux new-window -d -a -t ${session}:${base_index} i=$((${i}+1)) done fi

##re-number the windows to reflect backed-up window number echo "* Re-numbering windows to reflect original scheme:" #always get $i to 1 regardless of base-index i=$((${num_windows}+(${base_index}-1))) for window in ${window_list_rev} ;do if [ ${i} -ne ${window} ] ;then echo "${session}:${i} -> ${session}:${window}" tmux move-window -d -s ${session}:${i} -t ${session}:${window} fi     i=$((${i}-1)) done

##if windows had multiple panes, populate with proper number of panes for window in ${window_list} ;do pane_list=$(ls ${backup_dir} | grep -v 'layout$' | awk -F\. '/^'${session}':'${window}'\./ {print $2}') num_panes=$(echo ${pane_list} | wc -w) if [ ${num_panes} -gt 1 ] ;then echo "* Adding panes to window, \"${session}:${window}\"." i=1 while [ ${i} -lt ${num_panes} ] ;do tmux split-window -d -v -t ${session}:${window}.0 rval=${?} if [ ${rval} -gt 0 ] ;then echo "Error on split-window with \"${session}:${window}\", compensating by re-arranging panes and trying again." tmux select-layout -t ${session}:${window} tiled tmux split-window -d -v -t ${session}:${window}.0 fi         i=$((${i}+1)) done echo "Applying saved layout to panes in window, \"${session}:${window}\"." layout=$(cat ${backup_dir}/${session}:${window}.layout) tmux select-layout -t ${session}:${window} "${layout}" for pane in ${pane_list} ;do pane_full="${session}:${window}.${pane}" tmux send-keys -t ${pane_full} "cat ${backup_dir}/${pane_full} " #carriage return from previous line mimic_state done else pane_full="${session}:${window}.0" tmux send-keys -t ${pane_full} "grep -v -e 'strings you dont want to see' -e 'another string you dont want to see' ${backup_dir}/${pane_full} | cat -s " #carriage return from previous line mimic_state fi   done done fi