srimux

This post is a journey of abstractions. These abstractions stand on the shoulders of a few command line utilities that combine well. If you are the kind of programmer who dwells in command line terminals for much of your waking hours, read on. Your productivity is about to get booster engines.

Let's start bottom up with the utilities. Most of these should be familiar to you. I just focus here on features that suit my abstractions.

tmux

If you work on terminals and are not using tmux, you should by all means. tmux is a terminal multiplexer. It allows you to create and switch between sessions. It allows you to attach to, detach from and reattach to any session. With different terminals with different form factors too. It allows you to create elaborate window-pane layouts that you can keep switching out and returning back to. This helps minimize context switching costs. This also allows you to use laptops and desktops interchangeably while attaching to sessions in remote servers. Not only that, tmux is 100 % scriptable. Everything you do in tmux can also be done by firing a tmux command from your terminal. This includes simulating typing by sending keys to different sessions, windows and panes programmatically. There are a ton of other tmux features that make it the natural fit for screen sharing, pair programming, training, code-reviews etc. It really is a work of art.

vim

vim is my preferred editor of choice. I consider it a programming language for modifying text with well designed commands. An expert vim user can achieve feats that can only be described as magic by lesser mortals. You can take a lifetime to acquire God level vim skills. There is no ceiling to how good you can get. It really is a work of art.

Clojure, lein and n-repl

Clojure is a Lisp that runs on the JVM. It's an incredibly well designed language. It's minimal syntax comprises of s-expressions. Lein is a project automation tool for hacking clojure/clojurescript projects. 1 of the things lein does is fire up a network repl (read-eval-print-loop) or n-repl. Think of n-repl as a repl but reachable over the internet. n-repl is a server process that listens on a socket for s-expressions coming in remotely from clients. As it receives them, it evals them in the actual repl it wraps over and sends the results of the evaluation back to the client. Classic client-server architecture. The clojure - nrepl combo is really a work of art.

vim-fireplace

If n-repl is the server running on a port. vim-fireplace is it's client. Once a connection is established, Clojure s-expressions written and stored in vim can be sent over to the server, just by navigating to the start of the expression and hitting the keys c-p-p. There is no need to even save the file in which you have entered that thing. The results are evald on the server and shown within vim itself. With vim-fireplace you never have to leave vim ever. The fireplace vim plugin is really a work of art.

Abstraction 1 - A project is a tmux session. Nesting among these is necessary at times

Developers almost always work concurrently on several projects. The above abstraction says that if you have 7 projects you are working on at the moment, you should have at least 7 tmux sessions running in your system (actually it's 8 as you will soon find out. srimux being the additional and necessary mother session). Each tmux session can have a unique name and represents 1 project. In the course of a work day, you switch between these projects all the time. If your workflow is not optimized, switching costs can be really draining. Abstraction 1 treats each project as it's own tmux session. A tmux session can have an elaborate layout of windows and panes. Like 1 window for code, 1 for tests, 1 for monitoring log files etc. It's also common practice for some windows in a project, to be attached to yet another tmux session. e.g in 1 window of your project named "smoke-tests", you connect to a AWS EC2 via ssh and then further attach to an already existing tmux session in that server. This last attachment causes tmux nesting and assumed to be a routine part of your workflow.

A project with elaborate windows and panes layout

In the screenshot below, there are 2 tmux sessions nested. The 2 sessions are named srimux and kodeship-website. srimux is the outer session. It has 2 windows. window 1 is invisible and has a Clojure repl running. window 2 (highlighted in red - active) is made up of 2 panes. The process that is running on window 2 pane 2 is another tmux session called kodeship-website. kodeship-website has 6 windows, with the 4 th window (active) having 3 panes. These kinds of elaborate layouts can be automated with a few keystrokes in srimux.

Abstraction 2 - Most of a programmer's output come from simply evaling an ordered sequence of s-expressions

You intuitively know this. Most of your output comes from typing on the keyboard. In srimux, typing a keyboard is a special case of evaling s-expressions. Good abstractions can be powerful. More generic the abstraction, further it travels. I could have worded the above by stating that a programmer's output is just sending the right keystrokes to the right places. But that would have limited it's applicability. There are many things in automation that are more than typing commands. Like saying don't type anything for 5 seconds before these 2 sets of keystrokes. Delays like these could be expressed as s-expressions - (Thread/sleep 5000). In srimux, all keystrokes are sent hardcoded to srimux window 2, pane 2. This is a special address. Let's call it 2.2 from here on. Because of this hardcoding for anything to work, you need to be able to 'summon' the right project to 2.2. Not only that, you also have to switch to the required window and pane as a prerequisite. Once you do this, the process that's running inside the switched pane becomes the recipient of the keystrokes. You can summon any project to 2.2, by calling the summon-child function. Switching to the right window of a project happens by calling a meta keystroke function. There are 2 types of keystrokes. Meta and others (or non meta). Meta keystrokes are sent by calling either g, b or gg functions. Non meta keystrokes are sent by calling the tsne and ts functions. What are these meta keystrokes? These are keystrokes that send a command to the appropriate 'depth' of the nested tmux session visible in 2.2. When you summon a project, the tmux representing the project is loaded in 2.2. This is 1 level of nesting. The function g addresses the first level of nested tmux. Example if you need to create another window in the just summoned project you need to eval (g "c"). If you want to switch to window 3 of the summoned project you eval (g "3"). If 3 also happens to attach to a remote tmux session then you have 2 levels of nesting. When in this mode, to reach the second nested tmux i.e the innermost and remote tmux you use the command gg. Example if you need to create a new window in the remote tmux, you eval (gg "c"). On rare occasions, you may not have access to configure your tmux.conf in the remote session. In this case the tmux prefix by default is ^b and the gg function will not work. In such situations you can use the b function instead of gg to reach such non configurable tmuxes.

• tmux primer
  • tmux new -s "sessionname"
  • tmux a -t "sessionname"
  • detach
    • Prefix d
  • list
    • Prefix w
  • copy mode
    • Prefix Esc
      • Once in copy mode ? to go to the search word. v lll will start selection. y will yank selection and put it in the buffer. yanking will also exit copy mode. Prefix p to paste wherever
      • Enter in copy mode to exit copy mode
  • name
    • Prefix ,
  • Pane
    • Prefix |
    • Prefix -
  • goto pane
    • Prefix hjkl
  • resize pane
    • Prefix HHHLLLJJJKKK
  • zoom toggle pane
    • Prefix z
  • command
    • Prefix :
    • checkout the command choose-buffer

A project with 2 level tmux nesting

In the screenshot below, there are 3 tmux sessions with 2 level nesting. The 3 sessions are named srimux, cubelive and prod (cant be seen clearly because of bad contrast). srimux is the outer session. It has 2 windows. window 1 is invisible and has a Clojure repl running. window 2 (highlighted in red - active) is made up of 2 panes. The program that is running on window 2 pane 2 is another tmux session called cubelive. cubelive is level 1 nested tmux. cubelive has 2 windows, with the 1 st window (called test) active. This window has attached to another server side tmux session called prod. prod is level 2 nested tmux. prod has 5 windows with it's 2nd window active. This is a sql client connected to a live database. You can automate sending sql commands to production using ts from local vim. The keystrokes originate from your keyboard and flys tunneling down till it reaches the desired process - in this case a remote sql client. srimux rocks !!

If summoning projects and switching windows is this easy, then I can now imagine s-expressions from within vim that summons any project, runs any command via ts, wait 500 ms for output via (Thread/sleep 500), summon yet another project and then send another set of commands by compounding several s-expression through the do form. Which is also an s-expression. This is what makes intra and inter project automation possible under an uniform API. Obviously each of the tmux sessions you create in srimux are independent entities and can be worked on outside of srimux. After all they are first class tmux sessions. Go ahead and work on them on another terminal tab as if srimux was never there. You summon these projects to 2.2, only when you want s-expression automation. The gohome function clears 2.2 from all summoned projects and is treated as the default ready state. srimux in this state is ready to work on any project. This is the 0 level nested state. You just have to summon your next project to 2.2 and become 1 level nested.

Before you can start experiencing all this s-expressions based automation in your machine, you need to get bootstrapped, which is a straightforward process...

Installation and demo

I have assumed an ubuntu/debian distro for illustrating installation here. All apt-get packages should also be available for your distro.

Install srimux

cd ~/
git clone https://sparasur@bitbucket.org/sparasur/srimux.git
cd srimux

#This makes tmux a easy transition for old screen and vim users
cp tmux.conf.sample ~/.tmux.conf

#srimux_secrets is an ignored file in git. this is so that you don't accidentally commit ur secrets and push it to public repos
cp src/srimux_secrets_sample.clj src/srimux_secrets.clj

#modify your .bash_profile. we will use nesting extensively. tmux thinks it's an error to nest. Need to silence that by blanking out the TMUX variable
echo 'export TMUX=' >>~/.bash_profile

Install tmux

sudo apt-get update
sudo apt-get install tmux

Start the mother project - a tmux session called srimux

tmux new -s srimux

Install other utilities

#Install vim
sudo apt-get install vim

#Install jdk
sudo apt-get install default-jdk


#Install lein
sudo apt-get install wget

#https://leiningen.org/#install
wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
sudo mv lein /usr/bin
sudo chmod a+x  /usr/bin/lein

#Install git
sudo apt-get install git 

#Install python
sudo apt-get install python 

Install fireplace

mkdir -p ~/.vim/pack/tpope/start
pushd ~/.vim/pack/tpope/start
git clone https://tpope.io/vim/fireplace.git
vim -u NONE -c "helptags fireplace/doc" -c q
popd

srimux window 1 hosts the nrepl server

cd ~/srimux

#will take time on first run
lein repl

#when the installation is complete you should see the user=> prompt in the repl
# look for the nrepl://127.0.0.1:32923 string - we will have vim connect to it in srimux window 2

srimux window 2 hosts a vim session as the nrepl client

#The tmux prefix key is ^a - and the command to create a new window is c
^a -c
cd ~/srimux
vim src/srimux.clj

#type and enter this vim command
:Connect

#enter the port 32923 you saw in window 1. ^a - 1 will take you to window 1

#take the cursor to the opening bracket of the first (ns line 
#eval the ns s-expression
c-p-p

All further interactions in srimux involve the same motions - navigating to the start of the s-expression and hitting c-p-p. The results will be available in the bottom of window 2. You may also want to type the vim command :set cmdheight=2, to silence the "hit enter to continue" message after every eval

Very rarely, fireplace loses it's connection to the server. When that happens evals wont happen. To fix, save srimux.clj and quit vim. Reopen the file and you are good. No need to :Connect again. Just re opening the file, reestablishes the connection.

The first command you run programatically is (resize). It splits window 2 in 2 panes in a 25:75 split. Pane 1 is the srimux.clj file opened in vim. Functions tsend and tsne sends keystrokes to pane2. Care should be taken that the right project is loaded (as a nested tmux) in 2.2

Abstraction 3: Treat srimux.clj as your s-expressions repository in a vim file. You "vim-navigate" to the beginning of any and eval it by hitting c-p-p. Most expressions send side effects to pane 2. These trigger secondary effects and all these can be visually inspected in real time as if a human typed them there. This flow is the REPL for your automation

Abstraction 3 is the last and most profound abstraction. It says that you can get anything done by navigating in vim and finding an s-expression and c-p-p ing it. If the s-expression you just c-p-p'd has side effects, it's immediately fedback to 2.2. This causes a repl visual effect. You switch projects, hit passwords, reconnect to remote servers, configure remote tmux layouts from curated s-expressions cpp-ing on pane 1 and get immediate feedback on pane 2. You never leave vim during this entire process. Even as you switch from project to project - hop from server to server, there is really no need to leave vim or reach for the mouse. Since you never leave vim, if you have a process to navigate via creative commenting or bookmarking, you develop incredible muscle memory over time. We live in times when attention is the most scarce resource. As mentally exhausting tasks like context switching gradually turn into muscle memory, your freed up mental energy can be redirected to other matters. Another non-trivial consequence of this is that if you can have some basic vim macros like wrapping strings between (ts) or (tsne), then you can type commands in vim, hit the macro instead of typing commands in bash or vim in various projects. This makes your workday self documenting with minimal overhead. One word of caution - make sure you don't have any s-expressions lying in the top level in the srimux-namespace. Always put them under some top level function. This is so that when you eval the namespace, such rogue s-expressions don't get executed. I personally create 1 function like (defn d_20200519 []) everyday, I try new s-expressions inside that function. Also please be aware of this architecture, before you start issuing destructive commands like "drop table". With great power comes great responsibilities.

Feel free to try your own expressions based on the demo. In time your repository may be completely different than the original srimux.clj. After all, your workflow should reflect your working style.

Example recipes

tsne send keystrokes

(tsne "Enter")

ts send keystrokes suffixed with an "Enter"

(ts "ls -l")

Ctrl keys may be prefixed with ^. Apart from the special string Enter, the following special strings can be used to represent various keys: Up, Down, Left, Right, BSpace, BTab, End, Escape, F1 to F12, Home, IC (Insert), NPage/PageDown/PgDn, PPage/PageUp/PgUp, Space, Tab

send a constant

(ts "echo \"blah\"")

send Ctrl C and Enter

(ts "^c")

send "Enter" 5 times

(dotimes [_ 5]
        (tsne "Enter")
    )

write a program using vim, execute it using python3, use clojures let and lexical scoping rules

    (let [fname "/tmp/deleteme1.py"]
      (ts (format "rm %s" fname))
      (Thread/sleep 500)
      (ts (format "vim %s" fname))
      ;give vim some time to open
      (Thread/sleep 500)
      (tsne "i")
      (ts "import sys")
      (tsne "print (sys.path)")
      (tsne "Escape")
      (ts ":wq!")
      (ts (format "python3 %s" fname))
    )

tnewsession, the first step in defining a new project

   (tnewsession "project1" "test")

Hardcode variables for the repl automation to work

   (reset! session-name "srimux")
   (reset! window-number 2)
   (reset! pane-number 2)

summon-child summons the tmux workspace representing project 1 to 2.2

   (summon-child "project1")

g meta keystroke command create a new window in the just summoned project1

   (g "c")

3 s-expressions that rename the window you just created

   (g ",")
   (dotimes [_ 4] (tsne "BSpace"))
   (ts "p1 code")

create project 2, but this time batch up all individual s-expressions into 1 compound s-expression using do

  (do
    (tnewsession "project2" "test")
    (reset! session-name "srimux")
    (reset! window-number 2)
    (reset! pane-number 2)
    ;gets the tmux workspace of project 1 in the work pane
    (summon-child "project2")

    ;create another window in project 2
    (Thread/sleep 500)
    (g "c")
    (g ",")
    (dotimes [_ 4]
      (tsne "BSpace")
    )
    (ts "p2 code")
  )

switch between the 2

(summon-child "project1")
(summon-child "project2")

wto waits till commands are over. Also observe below how backslashes are escaped

(ts "export PS1=\"\\u@[\\h] [\\t] [\\w] [\\$?]> \"")
(dotimes [_ 65]
  (tsne "Enter")
)

(do
    (ts "sleep 10")
    ;this waits till last command is over
    (wto :nest-level 1 :attempts 90 :delay-between-polls 1000)
    (ts "ls -l")
)

For the wto command to work, you need the prompt to be set up a certain way and the pane buffer to be non empty. Hence hit Enter key about 65 times to fill the buffer

in case the last char is ; in a string to be sent, escape only that. Useful in sending sql commands

(ts "# create ;database srimux\\;")

Sometimes projects like project2, may have windows with tmux to say remote server. Use gg meta keystroke function to reach the remote tmux and rename windows

(ts "tmux new -s p2tmux")
#use gg meta command to reach the inner most tmux
(gg ",")
(dotimes [_ 4]
    (tsne "BSpace")
)
(ts "remote tmux")

#this is now sent to the remote bash process that you just started above, logging you out
(ts "^d")

you can killsession by name. Useful in reconnection recipes

(tkillsession "project2")

command ready mode. now back in 0 level nesting

(gohome)

recipe to re-establish a ssh connection with a remote server ...

... possibly running it's own tmux. this assumes you already had a datalayer project that connected via ssh in it's window 1 to a remote server and further attaches to a remote tmux there. In this s-expression, you kill the level 1 nested tmux and re initiate the connection

(do
    (tkillsession "datalayerlive")
    (Thread/sleep 500)
    (tnewsession "datalayerlive" "test")
    (Thread/sleep 500)
    (reset! session-name "srimux")
    (reset! window-number 2)
    (reset! pane-number 2)

    (summon-child "datalayerlive")
    (Thread/sleep 500)
    ;open a side window for scp etc also
    (g "c")
    (Thread/sleep 500)
    (g "1")
    (Thread/sleep 500)
    ;ssh into machine
    (ts (format "ssh -i \"%s/Downloads/foo.pem\" username@aa.bb.cc.dd" "~"))
    (Thread/sleep 3000)

    ;since we are resuming, simply attach and thats that
    ;inner tmux. you would need a gg to reach this tmux
    (ts "tmux a -t prod")
)

play the piano

#adapted from https://github.com/stuarthalloway/programming-clojure/blob/master/src/examples/midi.clj
(import '(javax.sound.midi MidiSystem Synthesizer))

(defprotocol MidiNote
  (to-msec [this tempo])
  (key-number [this])
  (play [this tempo midi-channel]))

(defn perform [notes & {:keys [tempo] :or {tempo 88}}]
  (with-open [synth (doto (MidiSystem/getSynthesizer).open)]
    (let [channel (aget (.getChannels synth) 0)]
      (doseq [note notes]
        (play note tempo channel)))))

(defrecord Note [pitch octave duration]
  MidiNote
  (to-msec [this tempo]
    (let [duration-to-bpm {1 240, 1/2 120, 1/4 60, 1/8 30, 1/16 15}]
      (* 1000 (/ (duration-to-bpm (:duration this))
                 tempo))))
  (key-number [this]
    (let [scale {:C 0,  :C# 1, :Db 1,  :D 2,
                 :D# 3, :Eb 3, :E 4,   :F 5,
                 :F# 6, :Gb 6, :G 7,   :G# 8,
                 :Ab 8, :A 9,  :A# 10, :Bb 10,
                 :B 11 :c 12 :c# 13, :db 13, :d 14, :d# 15, :eb 15, :e 16,
                 :f 17, :f# 18, :gb 18, :g 19, :g# 20, :ab 20, :a 21, :a# 22, 
                 :bb 22, :b 23}]
      (+ (* 12 (inc (:octave this)))
         (scale (:pitch this)))))
  (play [this tempo midi-channel]
    (let [velocity (or (:velocity this) 127)]
      (.noteOn midi-channel (key-number this) velocity)
      (Thread/sleep (to-msec this tempo)))))

(defn pp [notes]
  (perform (for [x (clojure.string/split notes #" ")]
             (cond (= \+ (last x)) (->Note (keyword  (apply str (drop-last 1 x))) 5 1/4)
                   (= \- (last x)) (->Note (keyword  (apply str (drop-last 1 x))) 3 1/4)
                   :else (->Note (keyword  x) 4 1/4)
             )
             )))

;yeh raaate yeh mausam nadi ka kinara
(pp "F G# G F G# G F G# G F E C C G G F F A# C+ C+ A# C+ C#+")

;happy bday
(pp "C C D C F E C C D C G F C C C A F F E D")

Isn't this hurtingly flexible!! We've covered a lot of ground here, and also barely scratched the surface. Hope you enjoyed it. Would love to hear your feedback, recipes and enhancement ideas.