BrainScraps Wiki
Register
Advertisement

Mbm329 04:41, March 12, 2012 (UTC)

Should you use your shell server for any length of time, you will undoubtedly need to reboot it due to kernel patching, or possibly faulty hardware. When you do this, all your tmux sessions will die. Unfortunately, this is a byproduct of how tmux was designed. All your session data resides in volatile memory. tmux communicates to this memory through unix sockets. So when your system reboots, the sockets will exist, but the session information that once was stored in memory will be lost.

To make this less of a burden, I wrote a script that will attempt to do the following:

  • Re-create the tmux sessions and windows
  • Populate them with the session history you once had before the reboot
  • Log you into your previously logged in host via ssh
  • Place you into the same working directory you were once in.

Most of this data is derived from the prompt ($PS1). Running any previously ran commands would be extremely dangerous, so this is as far as I can get you.

Basically, the way this is accomplished is via a cron that runs as often as you would like snapshots of your sessions. I prefer to have it run only when I'm actively using my sessions (basically my standard workday (every hour, from 7AM to 7PM, Mon. - Fri.). You might ask, "Why not just every minute, 24 hours a day?" The answer is - Because if you did it that often, should your system reboot on its own, you would have just potentially wiped out the "good" state with crap data. By sticking to the hours you work, you would lessen the risk of overwriting good data with bad states (or no states at all). The script will keep X amount of copies so you can roll back to an hour or day that you would like. Should you feel you would work at any time of the day, you should keep more copies of snapshots. Here's the basics for configuration:

NOTE: THIS WORKS AGAINST TMUX v1.5 ONLY.

The script uses options only supplied in tmux v1.5 and up. It will break on v1.4 and earlier. I have not tested past v1.5.

Settings

There are four main settings with the tmux_resurrect script.

#######################
# Config Variables
max_backups=12
ignore_session_list="hosts"
backup_dir=~/.resurrect/tmux_snapshot
export PATH="${PATH}:/usr/local/bin"

#######################
# Use prompt to get state of pane.
mimic_state() {
   prompt='^\<.*@.*:'
   prompt_line=$(grep $(eval echo ${prompt}) ${backup_dir}/${pane_full} | tail -n 1)
   host=$(printf "${prompt_line}" | cut -d@ -f2 | cut -d: -f1)
   working_dir=$(printf "${prompt_line}" | cut -d: -f2 | cut -d\> -f1)

  if [ "${host}" != "$(hostname)" ] && [ "${host}" != "" ] ;then
    tmux send-keys -t ${pane_full} "ssh ${host}^M"
    echo "SSH'd to ${host} in ${pane_full}.  Sleeping 5 sec..."
    sleep 5
  fi
  tmux send-keys -t ${pane_full} "cd ${working_dir}
" #carriage return from previous line
}

  1. max_backups: I set this to 12 as I backup each hour for 12 hours. The earliest backup gets removed, and every backup up to the latest gets moved into the next larger number.
  2. ignore_sessions: This is simply a list of session names that you would like to ignore from being backed up. These may be ignored because you have a special .tmux.conf that creates them, or they do not follow your convention for how your prompt is setup. For example, I have a session called "hosts" that is a login into most of our hosts in case my account gets locked out accidentally, or if something seriously wrong occured and no logins are available, I can still get into the host. This session is created by a special .tmux.conf config I use.
  3. backup_dir: This is simply where you would like your latest snapshot stored. I store mine in ~/.resurrect/tmux_snapshot.
  4. prompt: This is probably the most important setting as it is used to determine how to rebuild your sessions' states. It is nothing more than a regex to get your prompt out of the sea of other data in your session windows. To give reference to what the regex above is looking for, with bash shell, my prompt is set with PS1='<\u@\h:\w>\n$ '

Backup

The script has two options -b for backup and -r for resurrect. The cron will run the -b option. This cron should also be run as your user account (not root).

Unfortunately, I could never find a way to keep the output of "set option: mode-keys -> vi" from being output when the resurrect -b command runs, and appending a ">/dev/null" to the crontab entry would hang the "tmux set-window-option mode-keys vi ; copy-mode -t" commands, I strip out any output (not stderr) with a crontab entry like so:

# Backup tmux sessions
0 7-19 * * 1-5 /home/user/bin/resurrect -b | sed '/.*/d'

Session Repository

In the script, I store the tmux sessions snapshots in ~/.resurrect. This session directory contains a snapshot of each of its windows.

$ ls ~/.tmux_resurrect
tmux_snapshot    tmux_snapshot.10  tmux_snapshot.12  tmux_snapshot.3  tmux_snapshot.5  tmux_snapshot.7  tmux_snapshot.9
tmux_snapshot.1  tmux_snapshot.11  tmux_snapshot.2   tmux_snapshot.4  tmux_snapshot.6  tmux_snapshot.8
$ ls ~/.resurrect/screen_snapshot
base:0.0   base:5.0      foglight:1.0   foglight:2.0  foglight:8.0  monitor:4.0    rep_perf:2.0  rep_perf:8.0  rsync:4.0    storage:2.0  vacs:0.0  vacs:8.0
base:1.0   base:6.0      foglight:10.0  foglight:3.0  foglight:9.0  rep_perf:0.0   rep_perf:3.0  rep_perf:9.0  rsync:5.0    storage:3.0  vacs:1.0  vacs:9.0
base:10.0  base:7.0      foglight:11.0  foglight:4.0  monitor:0.0   rep_perf:1.0   rep_perf:4.0  rsync:0.0     rsync:6.0    storage:4.0  vacs:4.0
base:2.0   base:8.0      foglight:12.0  foglight:5.0  monitor:1.0   rep_perf:10.0  rep_perf:5.0  rsync:1.0     rsync:7.0    storage:5.0  vacs:5.0
base:3.0   base:9.0      foglight:13.0  foglight:6.0  monitor:2.0   rep_perf:11.0  rep_perf:6.0  rsync:2.0     storage:0.0  upgrade:0.0  vacs:6.0
base:4.0   foglight:0.0  foglight:14.0  foglight:7.0  monitor:3.0   rep_perf:12.0  rep_perf:7.0  rsync:3.0     storage:1.0  upgrade:1.0  vacs:7.0

Script (tmux_resurrect)

#!/bin/bash
#######################
# Usage
usage() {
  echo "USAGE: ${0} -b | -r [backup_number]"
  exit 1
}

#######################
# Config Variables
max_backups=12
ignore_session_list="hosts"
backup_dir=~/.resurrect/tmux_snapshot
export PATH="${PATH}:/usr/local/bin"

#######################
# Use prompt to get state of pane.
mimic_state() {
   prompt='^\<.*@.*:'
   prompt_line=$(grep $(eval echo ${prompt}) ${backup_dir}/${pane_full} | tail -n 1)
   host=$(printf "${prompt_line}" | cut -d@ -f2 | cut -d: -f1)
   working_dir=$(printf "${prompt_line}" | cut -d: -f2 | cut -d\> -f1)

  if [ "${host}" != "$(hostname)" ] && [ "${host}" != "" ] ;then
    tmux send-keys -t ${pane_full} "ssh ${host}
" #carriage return from previous line
    echo "SSH'd to ${host} in ${pane_full}.  Sleeping 5 sec..."
    sleep 5
  fi
  tmux send-keys -t ${pane_full} "cd ${working_dir}
" #carriage return from previous line
}

#######################
# Code
if [ ! ${1} ] ;then
  usage
fi

while [ ${1} ] ;do
  case ${1} in
    -b)
      task=backup
      shift 1
    ;;
    -r)
      task=restore
      if [ ${2} ] ;then
        backup_number=${2}
        shift 2
      else
        shift 1
      fi
    ;;
    *)
      usage
    ;;
  esac
done

if [ "${task}" = 'backup' ] ;then
  ##Rotate previous backups.
  i=${max_backups}
  while [[ ${i} != 0 ]] ;do
    if [ -d ${backup_dir}.${i} ] ;then
      if [[ ${i} = ${max_backups} ]] ;then
        rm -r ${backup_dir}.${i}
      else
        mv ${backup_dir}.${i} ${backup_dir}.$((${i}+1))
      fi
    fi
    i=$((${i}-1))
  done
  if [ -d ${backup_dir} ] ;then
    mv ${backup_dir} ${backup_dir}.1
  fi

  ##Dump copy from all windows in all available tmux sessions.
  if [ ! -d ${backup_dir} ] ;then
    mkdir -p ${backup_dir}
  fi
  for session in $(tmux list-sessions | awk -F: '{print $1}') ;do
    for ignore_session in ${ignore_session_list} ;do
      if [ "${session}" = "${ignore_session}" ] ;then
        continue 2
      fi
    done
    session_size=$(tmux list-sessions | awk -F'[' '/^'${session}': / {print $2}' | awk -F']' '{print $1}')
    echo "${session_size}" >${backup_dir}/${session}.size
    for window in $(tmux list-windows -t ${session} | awk -F: '{print $1}') ;do
      for pane in $(tmux list-panes -t ${session}:${window} | awk -F: '{print $1}') ;do
        ##backup pane layout
        if [ ${pane} -gt 0 ] ;then
          tmux list-windows -t ${session} | awk -F'\\[layout ' '/^'${window}':/ {print $2}' | awk '{print $1}' | sed 's/\]$//' >${backup_dir}/${session}:${window}.layout
        fi
        pane_full=${session}:${window}.${pane}
        ##figure out the editing mode so we can select text in history
        mode_keys=$(tmux show-window-options -g | awk '/^mode-keys/ {print $2}')
        if [ "${mode_keys}" = 'emacs' ] ;then
          tmux copy-mode -t ${session}:${window}.${pane} \; send-keys -t ${session}:${window}.${pane} 'M-<' 'C-Space' 'M->' 'C-e' 'M-w' \; save-buffer -b 0 ${backup_dir}/${pane_full} \; delete-buffer -b 0
        elif [ "${mode_keys}" = 'vi' ] ;then
          tmux copy-mode -t ${session}:${window}.${pane} \; send-keys -t ${session}:${window}.${pane} g Space G '$' Enter \; save-buffer -b 0 ${backup_dir}/${pane_full} \; delete-buffer -b 0
        fi
        if [ ! -s ${backup_dir}/${pane_full} ] ;then
          rm ${backup_dir}/${pane_full}
        fi
      done
    done
  done

elif [ "${task}" = 'restore' ] ;then
  ##Allow for base-index configuration
  base_index=0
  if [ -f ~/.tmux.conf ] ;then
    if grep -v ^# ~/.tmux.conf | egrep 'base-index +[0-9]' >/dev/null ;then
      base_index=$(grep -v ^# ~/.tmux.conf | egrep 'base-index +[0-9]' | awk -F'base-index ' '{print $2}')
    fi
  fi

  ##Check for specified number backup.  If none, then use latest.
  if [ "${backup_number}" != '' ] ;then
    backup_dir=${backup_dir}.${backup_number}
  fi

  ##Restore proper number of sessions, windows, and panes
  for session in $(ls ${backup_dir} | grep -v '\.size$' | cut -d: -f1 | sort -du) ;do
    window_list=$(ls ${backup_dir} | awk -F: '/^'${session}':/ {print $2}' | cut -d\. -f1 | sort -nu)
    window_list_rev=$(echo "${window_list}" | sort -nr)
    num_windows=$(echo ${window_list} | wc -w)

    ##create session, and first window.
    echo "* Creating new session, \"${session}\"..."
    session_width=$(cut -d x -f 1 ${backup_dir}/${session}.size)
    session_height=$(($(cut -d x -f 2 ${backup_dir}/${session}.size)+1))
    tmux new-session -d -s ${session} -x ${session_width} -y ${session_height}

    ##create additional windows if # of windows is > 1
    if [ ${num_windows} -gt 1 ] ;then
      echo "* Creating $((${num_windows}-1)) additional windows for session, \"${session}\"..."
      i=1
      while [ ${i} -lt ${num_windows} ] ;do
        tmux new-window -d -a -t ${session}:${base_index}
        i=$((${i}+1))
      done
    fi

    ##re-number the windows to reflect backed-up window number
    echo "* Re-numbering windows to reflect original scheme:"
    #always get $i to 1 regardless of base-index
    i=$((${num_windows}+(${base_index}-1)))
    for window in ${window_list_rev} ;do
      if [ ${i} -ne ${window} ] ;then
        echo "${session}:${i} -> ${session}:${window}"
        tmux move-window -d -s ${session}:${i} -t ${session}:${window}
      fi
      i=$((${i}-1))
    done

    ##if windows had multiple panes, populate with proper number of panes
    for window in ${window_list} ;do
      pane_list=$(ls ${backup_dir} | grep -v 'layout$' | awk -F\. '/^'${session}':'${window}'\./ {print $2}')
      num_panes=$(echo ${pane_list} | wc -w)
      if [ ${num_panes} -gt 1 ] ;then
        echo "* Adding panes to window, \"${session}:${window}\"."
        i=1
        while [ ${i} -lt ${num_panes} ] ;do
          tmux split-window -d -v -t ${session}:${window}.0
          rval=${?}
          if [ ${rval} -gt 0 ] ;then
            echo "Error on split-window with \"${session}:${window}\", compensating by re-arranging panes and trying again."
            tmux select-layout -t ${session}:${window} tiled
            tmux split-window -d -v -t ${session}:${window}.0
          fi
          i=$((${i}+1))
        done
        echo "Applying saved layout to panes in window, \"${session}:${window}\"."
        layout=$(cat ${backup_dir}/${session}:${window}.layout)
        tmux select-layout -t ${session}:${window} "${layout}"
        for pane in ${pane_list} ;do
          pane_full="${session}:${window}.${pane}"
          tmux send-keys -t ${pane_full} "cat ${backup_dir}/${pane_full}
" #carriage return from previous line
          mimic_state
        done
      else
        pane_full="${session}:${window}.0"
        tmux send-keys -t ${pane_full} "grep -v -e 'strings you dont want to see' -e 'another string you dont want to see' ${backup_dir}/${pane_full} | cat -s
" #carriage return from previous line
        mimic_state
      fi
    done
  done
fi
Advertisement