Greenspun, his 10th and I

This post is about my adversarial relationship with Greenspun's 10th rule and how it all changed when I chanced upon an article in May 2012.

2012 was a super busy year for us Engineers at Netcore Solutions. By then I had already maintained, designed or built several types of Rules Engines from scratch. Not only at Netcore, but also at previous gigs and on a few personal projects. I appreciated the universality of Rules Engines. I understood its business significance. At its core, a Rules Engine is about the separation of concerns at the Organization level. The Rules Engine our team had deployed was indispensable to our Mobility division sending 100s of millions of SMSs daily. The economic and regulatory landscape of Indian telecoms at the time was changing rapidly. Rules Engine was a necessity. Rules Engine empowered our Operations team to control key aspects critical to the business. e.g Cost based routing and segregation of transactional and promotional messages. They could control this without talking to Engineers or raising the dreaded Software Change Request.

By then I had acquired a taste for styles that scaled. I was intimate with the gifts interpreted languages like Python especially the luxury its eval function afforded the Rules Engine implementer. I had a ring side view of various trade offs played out between control and security. Between flexibility and scale. Between control and operator training. Between business agility and performance. The effectiveness of regular expressions pattern matching in the Rules Engine implementation was tangible. The over engineeredness of solutions like Drools and algorithms like Rete were apparent.

Thanks to Paul Graham's essays I also knew what these Rules Engines lacked. None of it had reached its peak expressive potential. My intuition of why I felt so is elucidated in this pithy aphorism by Greenspun. Jokingly called the 10th for effect. There are no preceding 9 rules even. "Any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp." I hated this rule. Not because it was not true, but because it was. It was nagging like hell. I could almost hear Greenspun laughing at me with ridicule. I knew about Lisp. I knew what it looked like, but I didn't know what it really was. I had taken multiple stabs at it over the years (like trying emacs for a while before giving up) but there was no way I could integrate Lisp into my reality. I couldn't drink the kool aid. Lisp and I were never meant to be together. Forever starcrossed. I had regular expressions, Python and eval as reasonable consolations.

On May 22, 2012, I read an article that changed everything. It was Peter Norvig's 90 line implementation of Scheme here and an enhanced version of lispy2 here. With this crystal clear demonstration of the idea of Lisp in code, Norvig had flung open the doors to a new beautiful world. Every design decision of Lisp comes out like an epiphany in these lines. The impact following the code was like being struck by lightning. The elegance of the code! The simplicity of its constructs! I am not a religious person, but on May 22, 2012 I had a religious experience!

How can a program be so beautiful? How can code and data be treated the same? By working through the program in my own Python REPL, I grokked the true nature of parsing, tokens, symbols, atoms, quoting, control flow, lexical scope, namespaces, syntax quoting, macros, syntax and source code. So complete was my understanding that in that Python REPL, I recognized that Python's REPL was not actually a true REPL in the manner its embedded Scheme's REPL was ;)

If you can follow Python code, I urge you to experience it yourself by downloading it and running it. To understand Lisp, you just need to understand eval in Norvig's code. You pass an s expression to eval, and in some cases parts of the s expression would be further evald (example of recursion) using simple rules. Basically there are only a few "forms". It's very much like the rules of Chess. Easy to explain to a newcomer but with infinite implications and sublime beauty.

I could now also easily embed Norvig's lispy2 into my own Python programs. In fact in the following weeks, I made my own Lisp dialect by extending the code to help with various data transforms riding on top of Python. I also wrote my first macro exasperated of thinking inside out all the time. Lisp does that to you. Later when I learnt Clojure (another Lisp dialect), I realized that I had actually implemented the thread last macro.

So on May 22, 2012, I finally found my response to Greenspun's 10th. Since that day Greenspun doesn't nag me anymore. In fact a few days back when I deployed yet another embedded rules engine in the form of Fun Dojo, I couldn't help flashing back to my summer awakening of 2012.