boss-level.com
13Jan/130

Inference as a Value

tl;dr

Clojure brings different weapons to scientific computing to the traditional matlab/R/... environments. These can help you understand your problem differently/better. I have released some toy code demonstrating some aspects of this idea.

Introduction

I'm reading Kevin Murphy's tremendous book on machine learning. There is a lot to like about this book, and the inclusion MATLAB code for all the figure and examples puts it right over the top. As a Clojure paradigm fiend, though, MATLAB isn't going to cut it for me so I've been recoding things in Clojure.

I've noticed right out of the gate how differently you can do things in Clojure to MATLAB and this post is an exploration of these ideas. I'm basing the discussion around Tenenbaum's Number Game which is used in an example early in the book.

The Sapir Whorf Hypothesis

In the realm of human languistics it is debated whether the language in which you speak/think affects what it is that you think. Whether this is true in human linguistics or not is moot to us - its certainly true, in the strong sense, for computer languages.

This is interesting for scientific computing in which the environments we use are well and widely established. It's a tantalising thought that you might reap a cognitive advantage by using a different language to the mainstream. Certainly Paul Graham has argued that he reaped a commercial advantage out of his lisp weenishnes.

The Code

The MATLAB code for this example is here. The equivalent Clojure code is on Github but you really want to start with the marginalia. I have an earlier, stunted version, which is basically a direct translation of the MATLAB to Clojure, useful for comparison.

The Argument

Obey the Domimatrix

The first thing about the MATLAB code is that everything is a matrix. Well, it does stand for MATrix LABoratory, so this is unsurprising. The trouble is that there is nothing intrinsically 'matrixy' about this problem, yet in MATLAB we are forced to use matrices as the data structure in which to express our problem.

This particular problem, in fact a lot of Bayesian computation, is most naturally expressed as a graph. As MATLAB just won't give you that datastructure, this forces you to hunt about looking for a way to express yourself in matrix-speak. Anybody who's written graphical model code in MATLAB knows too well of which I speak.

This matrix slavery expresses itself in this particular example is an order dependence in the data. When you define your likelihood functions and priors at different points in the code you'd better remember to get the order correct, or you're sunk. If you change something, you'd better remember to change the other thing. This implicit, but unstated, stricture in code is enough rope to hang yourself.

The Structure

But it's more than that.

The MATLAB expression of this solution is to place the objects of computation (priors, likelihoods, etc) into separate matrices, and then bind them together using functions. While this can easily be expressed in MATLAB or Clojure, and this is usually the obviously correct way to do things, in this example its not so.

The dependency between the objects (priors, etc) carries meaning which becomes inaccessible once expressed as a function. The dependencies induce a graph, the manipulation of which is useful in the problem domain. In Clojure this graph is easily expressed, in fact I've based the actual computation on @stuartsierra's excellent Flow library. This gives us a natural expression of the dependency structure in the problem.

The main benefit of this is that we can consider the entire inference a value!

Inference as a Value

Here's the code to play the numbers game this way. The key bit demosntrating inference-as-a-value is line 152. A previous inference currently lives as a map in the var called inference. We then modify this data structure to change a prior (we could do anything, like add hypotheses, remove them, change the likelihood function for an hypothesis etc etc) and recalculate. However, because the inference lives in data structure which we can manipulate, we do not have to rerun the entire calculation. We can naturally work out those bits of the calculation that need to be rerun, by examining the structure of the inference, and execute only them. For this toy problem that's not a big issue, but for serious work it frequently can be.

Having the calculation as a value, which is manipulable and communicable has many advantages. For instance in more complex examples you might like to change the structure of the problem dynamically, having it available as a graph enables such wizardry. In addition when you make changes to the value, you still have the old one for comparison, which is also often very useful. The point is that by turning a calculation into a data structure you have handles on it.

Conclusion

It's nothing much to claim that different languages have different tools that allow you to do different things. However, Clojure brings with it not only a weighty tool belt, but also a philosophy ("Data all the Things") which is radically different to mainline scientific computing. Applying this philosophy will lead to different ways of thinking about and solving problems.

Comments (0) Trackbacks (0)

No comments yet.


Leave a comment

No trackbacks yet.