Working notes by Matthew Rocklin - SymPy

SymPy and Theano -- Matrix Expressions

Fri, 05 Apr 2013 00:00:00 -0700

Introduction

This post uses some LaTeX. You may want to read it on the original site.

This is the last of a three part series connecting SymPy and Theano to transform mathematical expressions into efficient numeric code (see part 1 and part 2). We have seen that it is simple and computationally profitable to combine the best parts of both projects.

In this post we’ll switch from computing scalar expressionss to computing matrix expressions. We’ll define the Kalman filter in SymPy and send it to Theano for code generation. We’ll then use SymPy to define a more performant blocked version of the same algorithm.

Kalman Filter

The Kalman filter is an algorithm to compute the Bayesian update of a normal random variable given a linear observation with normal noise. It is commonly used when an uncertain quantity is updated with the results of noisy observations. For example it is used in weather forecasting after weather stations report in with new measurements, in aircraft/car control to automatically adjust for external conditions real-time, or even on your smartphone’s GPS navigation as you update your position based on fuzzy GPS signals. It’s everywhere, it’s important, and it needs to be computed quickly and continuously. It suits our needs today because it can be completely defined with a pair of matrix expressions.

from sympy import MatrixSymbol, latex
n       = 1000                          # Number of variables in our system/current state
k       = 500                           # Number of variables in the observation
mu      = MatrixSymbol('mu', n, 1)      # Mean of current state
Sigma   = MatrixSymbol('Sigma', n, n)   # Covariance of current state
H       = MatrixSymbol('H', k, n)       # A measurement operator on current state
R       = MatrixSymbol('R', k, k)       # Covariance of measurement noise
data    = MatrixSymbol('data', k, 1)    # Observed measurement data

newmu   = mu + Sigma*H.T * (R + H*Sigma*H.T).I * (H*mu - data)      # Updated mean
newSigma= Sigma - Sigma*H.T * (R + H*Sigma*H.T).I * H * Sigma       # Updated covariance

print latex(newmu)
print latex(newSigma)

$$ \Sigma H^T \left(H \Sigma H^T + R\right)^{-1} \left(-data + H \mu\right) + \mu $$ $$ - \Sigma H^T \left(H \Sigma H^T + R\right)^{-1} H \Sigma + \Sigma $$

Theano Execution

The objects above are for symbolic mathematics, not for numeric computation. If we want to compute this expression we pass our expressions to Theano.

inputs = [mu, Sigma, H, R, data]
outputs = [newmu, newSigma]
dtypes = {inp: 'float64' for inp in inputs}

from sympy.printing.theanocode import theano_function
f = theano_function(inputs, outputs, dtypes=dtypes)

Theano builds a Python function that calls down to a combination of low-level C code, scipy functions, and calls to the highly optimized DGEMM routine for matrix multiplication. As input this function takes five numpy arrays corresponding to our five symbolic inputs and produces two numpy arrays corresponding to our two symbolic outputs. Recent work allows any SymPy matrix expression to be translated to and run by Theano.

import numpy
ninputs = [numpy.random.rand(*i.shape).astype('float64') for i in inputs]
nmu, nSigma = f(*ninputs)

Blocked Execution

These arrays are too large to fit comfortably in the fastest parts of the memory hierarchy. As a result each sequential C, scipy, or DGEMM call needs to move big chunks of memory around while it computes. After one operation completes the next operation moves around the same memory while it performs its task. This repeated memory shuffling hurts performance.

A common approach to reduce memory shuffling is to cut the computation into smaller blocks. We then perform as many computations as possible on a single block before moving on. This is a standard technique in matrix multiplication.

from sympy import BlockMatrix, block_collapse
A, B, C, D, E, F, G, K = [MatrixSymbol(a, n, n) for a in 'ABCDEFGK']
X = BlockMatrix([[A, B],
                 [C, D]])
Y = BlockMatrix([[E, F],
                 [G, K]])
print latex(X*Y)

$$ \begin{bmatrix} A & B \\ C & D \end{bmatrix} \begin{bmatrix} E & F \\ G & K \end{bmatrix}$$

print latex(block_collapse(X*Y))

$$ \begin{bmatrix} A E + B G & A F + B K \\ C E + D G & C F + D K\end{bmatrix} $$

We are now able to focus on substantially smaller chunks of the array. For example we can choose to keep A in local memory and perform all computations that involve A. We will still need to shuffle some memory around (this is inevitable) but by organizing with blocks we’re able to shuffle less.

This idea extends beyond matrix multiplication. For example, SymPy knows how to block a matrix inverse

print latex(block_collapse(X.I))

$$ \begin{bmatrix} \left(- B D^{-1} C + A\right)^{-1} & - A^{-1} B \left(- C A^{-1} B + D\right)^{-1} \\ - \left(- C A^{-1} B + D\right)^{-1} C A^{-1} & \left(- C A^{-1} B + D\right)^{-1} \end{bmatrix} $$

High performance dense linear algebra libraries hard-code all of these tricks into each individual routine. The call to the general matrix multiply routine DGEMM performs blocked matrix multiply within the call. The call to the general matrix solve routine DGESV can perform blocked matrix solve. Unfortunately these routines are unable to coordinate blocked computation between calls.

Fortunately, SymPy and Theano can.

SymPy can define and reduce the blocked matrix expressions using relations like what are shown above.

from sympy import blockcut, block_collapse
blocksizes = {
        Sigma: [(n/2, n/2), (n/2, n/2)],
        H:     [(k/2, k/2), (n/2, n/2)],
        R:     [(k/2, k/2), (k/2, k/2)],
        mu:    [(n/2, n/2), (1,)],
        data:  [(k/2, k/2), (1,)]
        }
blockinputs = [blockcut(i, *blocksizes[i]) for i in inputs]
blockoutputs = [o.subs(dict(zip(inputs, blockinputs))) for o in outputs]
collapsed_outputs = map(block_collapse, blockoutputs)

fblocked = theano_function(inputs, collapsed_outputs, dtypes=dtypes)

Theano is then able to coordinate this computation and compile it to low-level code. At this stage the expresssions/computations are fairly complex and difficult to present. Here is an image of the computation (click for zoomable PDF) as a directed acyclic graph.

Results

Lets time each function on the same inputs and see which is faster

>>> timeit f(*ninputs)
1 loops, best of 3: 2.69 s per loop

>>> timeit fblocked(*ninputs)
1 loops, best of 3: 2.12 s per loop

That’s a 20% performance increase from just a few lines of high-level code.

Blocked matrix multiply and blocked solve routines have long been established as a good idea. High level mathematical and array programming libraries like SymPy and Theano allow us to extend this good idea to arbitrary array computations.

Analysis

Good Things

First, lets note that we’re not introducing a new library for dense linear algebra. Instead we’re noting that pre-existing general purpose high-level tools can be composed to that effect.

Second, lets acknoledge that we could take this further. For example Theano seemlessly handles GPU interactions. We could take this same code to a GPU accelerated machine and it would just run faster without any action on our part.

Bad Things

However, there are some drawbacks.

Frequent readers of my blog might recall a previous post about the Kalman filter. In it I showed how we could use SymPy’s inference engine to select appropriate BLAS/LAPACK calls. For example we could infer that because $ H \Sigma H^T + R $ was symmetric positive definite we could use the substantially more efficient POSV routine for matrix solve rather than GESV (POSV uses the Cholesky algorithm for decomposition rather than straight LU). Theano doesn’t support the specialized BLAS/LAPACK routines though, so we are unable to take advantage of this benefit. The lower-level interface (Theano) is not sufficiently rich to use all information captured in the higher-level (SymPy) representation.

Also, I’ve noticed that the blocked version of this computation experiences some significant roundoff errors (on the order of 1e-3). I’m in the process of tracking this down. The problem must occur somewhere in the following tool-chain

SymPy -> Blocking -> Theano -> SciPy -> C routines -> BLAS

Debugging in this context can be wonderful if all elements are well unit-tested. If they’re not (they’re not) then tracking down errors like this requires an unfortunate breadth of expertise.

References

Scripts

SymPy and Theano -- Scalar Simplification

Thu, 28 Mar 2013 00:00:00 -0700

Introduction

This post uses some LaTeX. You may want to read it on the original site.

In my last post I showed how SymPy can benefit from Theano. In particular Theano provided a mature platform for code generation that outperformed SymPy’s attempt at the same problem. I argued that projects should stick to one specialty and depend on others for secondary concerns. Interfaces are better than add-ons.

In this post I’ll show how Theano can benefit from SymPy. In particular I’ll demonstrate the practicality of SymPy’s impressive scalar simplification routines for generating efficient programs.

After re-reading over this post I realize that it’s somewhat long. I’ve decided to put the results first in hopes that it’ll motivate you to keep reading.

Project	operation count
SymPy	27
Theano	24
SymPy+Theano	17

Now, lets find out what those numbers mean.

Example problem

We use a larger version of our problem from last time; a radial wavefunction corresponding to n = 6 and l = 2 for Carbon (Z = 6)

from sympy.physics.hydrogen import R_nl
from sympy.abc import x
n, l, Z = 6, 2, 6
expr = R_nl(n, l, x, Z)
print latex(expr)

$$\frac{1}{210} \sqrt{70} x^{2} \left(- \frac{4}{3} x^{3} + 16 x^{2} - 56 x + 56\right) e^{- x}$$

We want to generate code to compute both this expression and its derivative. Both SymPy and Theano can compute and simplify derivatives. In this post we’ll measure the complexity of a computation that simultaneously computes both the above expression and its derivative. We’ll arrive at this computation through a couple of different routes that use overlapping parts of SymPy and Theano. This will supply a couple of direct comparisons.

Disclaimer: I’ve chosen a larger expression here to exaggerate results. Simpler expressions yield less impressive results.

Simplification

We show the expression, it’s derivative, and SymPy’s simplification of that derivative. In each case we quantify the complexity of the expression by the number of algebraic operations

The target expression:

print latex(expr)

$$\frac{1}{210} \sqrt{70} x^{2} \left(- \frac{4}{3} x^{3} + 16 x^{2} - 56 x + 56\right) e^{- x}$$

print "Operations: ", count_ops(expr)
Operations:  17

It’s derivative

print latex(expr.diff(x))

$$ \frac{1}{210} \sqrt{70} x^{2} \left(- 4 x^{2} + 32 x - 56\right) e^{- x} - \frac{1}{210} \sqrt{70} x^{2} \left(- \frac{4}{3} x^{3} + 16 x^{2} - 56 x + 56\right) e^{- x} + \frac{1}{105} \sqrt{70} x \left(- \frac{4}{3} x^{3} + 16 x^{2} - 56 x + 56\right) e^{- x} $$

print "Operations: ", count_ops(expr.diff(x))
Operations:  48

The result of simplify on the derivative. Note the significant cancellation of the above expression.

print latex(simplify(expr.diff(x)))

$$ \frac{2}{315} \sqrt{70} x \left(x^{4} - 17 x^{3} + 90 x^{2} - 168 x + 84\right) e^{- x} $$

print "Operations: ", count_ops(simplify(expr.diff(x)))
Operations:  18

An unevaluated derivative object. We’ll end up passing this to Theano so that it computes the derivative on its own.

print latex(Derivative(expr, x))

$$ \frac{\partial}{\partial x}\left(\frac{1}{210} \sqrt{70} x^{2} \left(- \frac{4}{3} x^{3} + 16 x^{2} - 56 x + 56\right) e^{- x}\right) $$

Bounds on the cost of Differentiation

Scalar differentiation is actually a very simple transformation.

You need to know how to transform all of the elementary functions (exp, log, sin, cos, polynomials, etc...), the chain rule, and that’s it. Theorems behind automatic differentiation state that the cost of a derivative will be at most five times the cost of the original. In this case we’re guaranteed to have at most 17*5 == 85 operations in the derivative computation; this holds in our case because 48 < 85

However derivatives are often far simpler than this upper bound. We see that after simplification the operation count of the derivative is 18, only one more than the original. This is common in practice.

Theano Simplification

Like SymPy, Theano transforms graphs to mathematically equivalent but computationally more efficient representations. It provides standard compiler optimizations like constant folding, and common sub-expressions as well as array specific optimizations like the element-wise operation fusion.

Because users regularly handle mathematical terms Theano also provides a set of optimizations to simplify some common scalar expressions. For example Theano will convert expressions like x*y/x to y. In this sense it overlaps with SymPy’s simplify functions. This post is largely a demonstration that SymPy’s scalar simplifications are far more powerful than Theano’s and that their use can result in significant improvements. This shouldn’t be surprising. Sympians are devoted to scalar simplification to a degree that far exceeds the Theano community’s devotion to this topic.

Experiment

We’ll compute the derivative of our radial wavefunction and then simplify the result. We’ll do this using both SymPy’s derivative and simplify routines and using Theano’s derivative and simplify routines. We’ll then compare the two results by counting the number of required operations.

Here is some setup code that you can safely ignore:

def fgraph_of(*exprs):
    """ Transform SymPy expressions into Theano Computation """
    outs = map(theano_code, exprs)
    ins = theano.gof.graph.inputs(outs)
    ins, outs = theano.gof.graph.clone(ins, outs)
    return theano.gof.FunctionGraph(ins, outs)

def theano_simplify(fgraph):
    """ Simplify a Theano Computation """
    mode = theano.compile.get_default_mode().excluding("fusion")
    fgraph = fgraph.clone()
    mode.optimizer.optimize(fgraph)
    return fgraph

def theano_count_ops(fgraph):
    """ Count the number of Scalar operations in a Theano Computation """
    return len(filter(lambda n: isinstance(n.op, theano.tensor.Elemwise),
                      fgraph.apply_nodes))

In SymPy we create both an unevaluated derivative and a fully evaluated and sympy-simplified version. We translate each to Theano, simplify within Theano, and then count the number of operations both before and after simplification. In this way we can see the value added by both SymPy’s and Theano’s optimizations.

exprs = [Derivative(expr, x),    # derivative computed in Theano
         simplify(expr.diff(x))] # derivative computed in SymPy, also sympy-simplified

for expr in exprs:
    fgraph = fgraph_of(expr)
    simp_fgraph = theano_simplify(fgraph)
    print latex(expr)
    print "Operations:                             ", theano_count_ops(fgraph)
    print "Operations after Theano Simplification: ", theano_count_ops(simp_fgraph)

Theano Only

$$ \frac{\partial}{\partial x}\left(\frac{1}{210} \sqrt{70} x^{2} \left(- \frac{4}{3} x^{3} + 16 x^{2} - 56 x + 56\right) e^{- x}\right) $$

Operations:                              40
Operations after Theano Simplification:  21

SymPy + Theano

$$ \frac{2}{315} \sqrt{70} x \left(x^{4} - 17 x^{3} + 90 x^{2} - 168 x + 84\right) e^{- x} $$

Operations:                              13
Operations after Theano Simplification:  10

Analysis

On its own Theano produces a derivative expression that is about as complex as the unsimplified SymPy version. Theano simplification then does a surprisingly good job, roughly halving the amount of work needed (40 -> 21) to compute the result. If you dig deeper however you find that this isn’t because it was able to algebraically simplify the computation (it wasn’t) but rather because the computation contained several common sub-expressions. The Theano version looks a lot like the unsimplified SymPy version. Note the common sub-expressions like 56*x below.

The pure-SymPy simplified result is again substantially more efficient (13 operations). Interestingly Theano is still able to improve on this, again not because of additional algebraic simplification but rather due to constant folding. The two projects simplify in orthogonal ways.

Simultaneous Computation

When we compute both the expression and its derivative simultaneously we find substantial benefits from using the two projects together.

orig_expr = R_nl(n, l, x, Z)
for expr in exprs:
    fgraph = fgraph_of(expr, orig_expr)
    simp_fgraph = theano_simplify(fgraph)
    print latex((expr, orig_expr))
    print "Operations:                             ", len(fgraph.apply_nodes)
    print "Operations after Theano Simplification: ", len(simp_fgraph.apply_nodes)

$$ \begin{pmatrix}\frac{\partial}{\partial x}\left(\frac{1}{210} \sqrt{70} x^{2} \left(- \frac{4}{3} x^{3} + 16 x^{2} - 56 x + 56\right) e^{- x}\right), & \frac{1}{210} \sqrt{70} x^{2} \left(- \frac{4}{3} x^{3} + 16 x^{2} - 56 x + 56\right) e^{- x}\end{pmatrix} $$

Operations:                              57
Operations after Theano Simplification:  24

$$ \begin{pmatrix}\frac{2}{315} \sqrt{70} x \left(x^{4} - 17 x^{3} + 90 x^{2} - 168 x + 84\right) e^{- x}, & \frac{1}{210} \sqrt{70} x^{2} \left(- \frac{4}{3} x^{3} + 16 x^{2} - 56 x + 56\right) e^{- x}\end{pmatrix} $$

Operations:                              27
Operations after Theano Simplification:  17

The combination of SymPy’s scalar simplification and Theano’s common sub-expression optimization yields a significantly simpler computation than either project could do independently.

To summarize

Project	operation count
SymPy	27
Theano	24
SymPy+Theano	17

References

A script of this session

SymPy and Theano -- Code Generation

Tue, 19 Mar 2013 00:00:00 -0700

No one is good at everything, that’s why we have society.

No project is good at everything, that’s why we have interfaces.

This is the first of three posts that join SymPy, a library for symbolic mathematics, and Theano, a library for mathematical compilation to numeric code. Each library does a few things really well. Each library also over-reaches bit and does a few things not-as-well. Fortunately the two libraries have clear and simple data structures and so can be used together effectively.

In this post I’ll focus on how SymPy can use Theano to generate efficient code.

Physics

SymPy knows Physics. For example, here is the radial wavefunction corresponding to n = 3 and l = 1 for Carbon (Z = 6)

from sympy.physics.hydrogen import R_nl
from sympy.abc import x
expr = R_nl(3, 1, x, 6)
print latex(expr)

$$\frac{8}{3} x \left(- 4 x + 4\right) e^{- 2 x}$$

SymPy is great at this. It can manipulate high level mathematical expressions very naturally. When it comes to numeric computation it is less effective.

Numerics

Fortunately there are methods to offload the work to numerical projects like numpy or to generate and compile straight Fortran code. Here we use two existing methods to create two identical vectorized functions to compute the above expression.

from sympy.utilities.autowrap import ufuncify
from sympy.utilities.lambdify import lambdify
fn_numpy   = lambdify(x, expr, 'numpy')
fn_fortran = ufuncify([x], expr)

fn_numpy replaces each of the SymPy operations with the equivalent function from the popular NumPy package. fn_fortran generates and compiles low-level Fortran code and uses f2py to bind it to a Python function. They each use numpy arrays as common data structures, supporting broad interoperability with the rest of the Scientific Python ecosystem. They both work well and produce identical results.

>>> from numpy import linspace
>>> xx = linspace(0, 1, 5)
>>> fn_numpy(xx)
[ 0.          1.21306132  0.98101184  0.44626032  0.        ]
>>> fn_fortran(xx)
[ 0.          1.21306132  0.98101184  0.44626032  0.        ]

We use these functions and matplotlib to plot the original equation

from pylab import plot, show, legend
xx = linspace(0, 5, 50000)
plot(xx, fn_numpy(xx))

Performance

When we profile these functions we find that the Fortran solution runs a bit faster. This is because it is able to fuse all of the scalar operations into one loop while the numpy solution walks over memory several times, performing each operation individually. Jensen wrote a more thorough blogpost about this when he worked on code generation. He shows substantial performance increases as the complexity of the mathematical expression increases.

>>> timeit fn_numpy(xx)
1000 loops, best of 3: 1.4 ms per loop
>>> timeit fn_fortran(xx)
1000 loops, best of 3: 884 us per loop

This weekend I built up a translation from SymPy expressions to Theano computations. This builds off of old work done with Frederic Bastien at SciPy2012.

>>> from sympy.printing.theanocode import theano_function
>>> fn_theano  = theano_function([x], [expr], dims={x: 1}, dtypes={x: 'float64'})
>>> timeit fn_theano(xx)
1000 loops, best of 3: 1.04 ms per loop

Theano generates C code that performs the same loop fusion done in Fortran but it incurs a bit more startup time. It performs somewhere between the numpy and Fortran solutions.

However, the SymPy to Theano translation interface only takes up about a page of code while the lambdify and autowrap modules are substantially more complex. Additionally, Theano is actively developed and is sure to improve and track changes in hardware well into the future. lambdify and autowrap have been relatively untouched over the past year. For example Theano is able to seemlessly compile these computations to the GPU.

Leveraging Theano

In the above example we used Theano to copy the behavior of SymPy’s existing numpy and Fortran numeric solutions. Theano is capable of substantially more than this. To show a simple example we’ll compute both our original output and the derivative simultaneously.

outputs = expr, simplify(expr.diff(x))
print latex(outputs)

$$ \begin{pmatrix}\frac{8}{3} x \left(- 4 x + 4\right) e^{- 2 x}, & \frac{32}{3} \left(2 x^{2} - 4 x + 1\right) e^{- 2 x}\end{pmatrix} $$

We redefine our functions to produce both outputs, instead of just expr alone

fn_numpy  = lambdify([x], outputs, 'numpy')
fn_theano = theano_function([x], outputs, dims={x: 1}, dtypes={x: 'float64'})

fns_fortran = [ufuncify([x], output) for output in outputs]
fn_fortran  = lambda xx: [fn_fortran(xx) for fn_fortran in fns_fortran]

The expression and its derivative look like this:

for y in fn_theano(xx):
    plot(xx, y)
legend(['$R_{31}$', "$R'_{31}$"])

Because Theano handles common subexpressions well it is able to perform the extra computation with only a very slight increase in runtime, easily eclipsing either of the other two options.

>>> timeit fn_numpy(xx)
100 loops, best of 3: 2.85 ms per loop
>>> timeit fn_fortran(xx)
1000 loops, best of 3: 1.8 ms per loop
>>> timeit fn_theano(xx)
1000 loops, best of 3: 1.16 ms per loop

When we extend this experiment and vary the number of simultaneous derivatives we observe the following runtimes

In the case of highly structured computation Theano is able to scale very favorably.

Conclusion

The Theano project is devoted to code generation at a level that exceeds the devotion of SymPy to this same topic. This is natural and prevalent. When we combine the good parts of both projects we often achieve a better result than with an in-house solution

In-house solutions to foreign problems lack persistence. As programmers within an ecosystem we should make projects that do one thing well and provide clean interfaces and simple data structures to encourage inter-project communication.

References

Maximum a Posteriori Estimation

Mon, 25 Feb 2013 00:00:00 -0800

Disclaimer: I know relatively little about this application. Corrections welcome.

In this post we see how SymPy can simplify common numeric calculations, particularly in Bayesian inference problems.

Imagine you are a scientist studying some counting process (like radioactive decay or the number of page requests on a web server). You describe this process with a Poisson random variable and try to learn the rate parameter of this distribution by observing some random samples.

If you have no preconceptions about the rate then this problem is easy. You just divide total counts by total time and you’re done.

A more complex problem arises when external theory provides prior information about your rate parameter (for example physics might impose rules on the rate of radioactive decay). Lets model this problem in SymPy. For the sake of concreteness lets arbitrarily assume that $\lambda$, the rate parameter, follows a Beta distribution with parameters a and b.

a, b = symbols('a,b', positive=True)
lam = Symbol('lambda', positive=True)
rate = Beta(lam, a, b)
count = Poisson('X', rate)

In the lab we observe many samples $x_i$ taken from count. From these we wish to find the most likely value of rate. The probability of any single value of rate given our data can be rewritten with Bayes’ rule.

$$ p(\lambda \vert x_i) \propto \prod_i p(x_i \vert \lambda) \cdot p(\lambda) $$

In this case the distributions are given by

pdf = density(count, rate);  print latex(pdf(x))  # density of count, given rate
pdf = density(rate);         print latex(pdf(lam))

$$ p(x_i \vert \lambda) = \frac{\lambda^{x}}{e^{\lambda} x!} \;\;\;\; p(\lambda) = \frac{\lambda^{a - 1} \left(- \lambda + 1\right)^{b - 1} \Gamma\left(a + b\right)}{\Gamma\left(a\right) \Gamma\left(b\right)}$$

To find the maximizer of $p(\lambda \vert x_i)$ we set the derivative equal to zero. We simplify the computation by taking the log. Because log is monotonic this does not change the solution.

$$ 0 = \frac{d}{d\lambda} \log\left( \prod_i p(x_i \vert \lambda) \cdot p(\lambda)\right) = \frac{d}{d\lambda} \sum_i \log(p(x_i \vert \lambda) \cdot p(\lambda)) $$

We can accomplish this in SymPy with the following code

# Model `n` observations with a function `data` indexed by integer `i`
i, n = symbols('i,n', integer=True)
data = Function('data')

# Compute log likelihood
loglikelihood = log(Product(density(count, rate)(data(i)) * density(rate)(lam), (i, 1, n)))
Eq(simplify(loglikeihood.diff(lam)), 0)

$$ \sum_{i=1}^{n} \frac{a \left(\lambda - 1\right) + b \lambda - \lambda \left(\lambda - 1\right) - 2 \lambda + \left(\lambda - 1\right) \operatorname{data}{\left[i \right]} + 1}{\lambda \left(\lambda - 1\right)} = 0 $$

Discussion

SymPy reduces this Bayesian inference problem to finding roots of the above equation. I suspect that many prevalent numeric problems could be similarly accelerated through a symbolic preprocessing step.

Looking at the equation above it’s clear that this problem can be simplified further. However I like the existing solution because it does not depend on the user possessing any mathematical expertise beyond the ability to describe their mathematical model (the derivatives, log, etc… are generally applicable to this problem). In what other automated ways can SymPy further process computations like this? What are other ways that aren’t in SymPy but could be developed in the future?

I suspect that the problem given here is analytically solvable. To the extent possible SymPy should try to solve these problems. However for the vast number of problems without analytic solutions I suspect there is still a great deal we can do, either by reducing the problem as above or through the mathematically informed selection of numeric algorithms.

Various root finding algorithms are appropriate in different cases. Wikipedia suggests Householder’s Method, a generalization on Newton’s method for scalar systems with known derivatives. Perhaps in cases where SymPy is unable to solve the problem analytically it could select the correct numeric algorithm. Is this a reasonable use case for SymPy?

References

Assuming assumptions

Tue, 05 Feb 2013 00:00:00 -0800

SymPy has two assumptions systems called (unimaginatively) “old assumptions” and “new assumptions.” They differ in how they manage mathematical attributes.

Old Assumptions

In old assumptions attributes are bound to variables

>>> x = Symbol('x', positive=True)
>>> y = Symbol('y', positive=True)

These are then composed into expressions.

>>> expr = 2*x + y

And we query these expressions directly

>>> expr.is_positive
True

The expression and the attributes are woven into the same object.

New Assumptions

In new assumptions variables and attributes are maintained separately.

>>> x = Symbol('x')
>>> y = Symbol('y')

>>> facts = Q.positive(x) & Q.positive(y)

The construction of mathematical expressions remains unchanged

>>> expr = 2*x + y

But querying now requires two inputs, both the expression and the facts.

>>> ask(Q.positive(expr), facts)
True

The separation of facts from expressions enables rich logical inference but it requires the management of two separate variables, expr and facts, rather than just one, expr. It is difficult to consistently pass the extra variable through all relevant function calls.

Global assumptions

One solution to the management problem is to keep all facts in a globally accessible collection. This removes the need to pass an extra argument between function calls.

This little known feature is already accessible in SymPy

>>> # Setup
>>> global_assumptions.add(Q.positive(x))
>>> global_assumptions.add(Q.positive(y))

>>> # Compute in this context
>>> ask(Q.positive(2*x + y))
True

Unfortunately global variables often cause confusion. We will invariably add an experimental fact to the global collection and then forget to clean up, polluting future computations. In this case we need to always remember to clean up after we’re done.

>>> # Cleanup
>>> global_assumptions.remove(Q.positive(x))
>>> global_assumptions.remove(Q.positive(y))

This cleanup step is both crucial and forgettable. We can not trust ourselves to remember it.

Introducing `assuming`

Context managers provide the convenience of global variables with side-effect free security. This is accomplished through consistent cleanup.

SymPy now includes, assuming, a context manager for mathematical assumptions. Here is an example

>>> facts = Q.positive(x), Q.positive(y)

>>> with assuming(*facts):
...     ask(Q.positive(2*x + y))
True

All ask calls within this block have global-like access to the knowledge Q.positive(x) and Q.positive(y). These calls may be at top level as in the example above or buried deeply within function calls. This arrangement is convenient because we do not need to pass down facts through all function calls. This knowledge is pervasive like a global variable but contained within the with assuming clause.

Commutative Unification

Fri, 25 Jan 2013 00:00:00 -0800

LogPy now supports commutative and associative pattern matching on expression trees. This is a standard requirement for computer algebra systems like SymPy but not a traditional feature of logic programming systems.

Pattern-matching in LogPy is expressed by the eq goal. This goal relies on unification to match trees of tuples. Unification is a computational cornerstone of LogPy. Traditionally eq performs exact structural pattern matching. For example

(1, x, (5, y, 7))  matches  (1, (2, 3, 4), (5, 6, 7))

with the following substitution

{x: (2, 3, 4), y: 6}

Expression Trees

We traditionally represent both mathematical expressions and computer programs with expression trees. For example $y * (1 + x)$ can be visualized as follows

We represent this expression in LogPy with tuples. The head/first element of each tuple is an operation like add or mul. All subsequent elements (the tail) are the arguments/children of that expression.

y * (x + 1) -> (mul, y, (add, x, 1))

Matching Expression Trees

This form is exactly what we use for unification. We could match this pattern against the following expression, treating x and y as wildcard logic variables

(mul, y, (add, x, 1))  matches  (mul, (pow, 2, 10), (add, 3, 1))

with the following substitution

{x: 3, y: (pow, 2, 10)}

But what about the following?

(add, x, 1)  matches?  (add, 1, 3)

This doesn’t unify. While the first (add, add) and second (x, 1) elements can match (if {x: 1}) the third elements (1, 3) will not.

As mathematicians however we know that because add is commutative these expressions should match if we are allowed to rearrange the terms in the tail and match 1 to 1 and x to 3. LogPy doesn’t know this by default. LogPy is not a math library.

Building Commutative Equality

Given the goal seteq for set unification and a goal conso for head-tail pattern matching we build eq_commutative for commutative matching.

Example of seteq

run(0, x, seteq((1, 2, x), (3, 1, 2)))  # seteq matches within sets
(3,)

Example of conso

run(0, head,  conso(head, tail, (1, 2, 3, 4)))
(1,)
run(0, tail,  conso(head, tail, (1, 2, 3, 4)))
((2, 3, 4),)

Given these two it is easy to build eq_commutative

def eq_commutative(u, v):
    operation, utail, vtail = var(), var(), var()
    return lall(conso(operation, utail, u),
                conso(operation, vtail, v),
                commutative(operation),
                seteq(utail, vtail))

That is we require all of the following (lall is logical all).

u must be of the form (operation, utail....)
v must be of the form (operation, vtail....). Note that the same variable operation must be the same in both expressions.
The operation must be commutative (operations register themselves beforehand, see example below)
utail and vtail must unify under set equality.

I am glossing over some details here, like “what about associative matching” and “how does seteq work?” but this should give a high-level view of how logic programs are made. Lets see an example of associative/commutative matching

Example

This is the standard example for commutative matching found in the repository

from logpy import run, var, fact
from logpy.assoccomm import eq_assoccomm as eq
from logpy.assoccomm import commutative, associative

# Define some dummy Operationss
add = 'add'
mul = 'mul'

# Register that these ops are commutative using the facts system
fact(commutative, mul)
fact(commutative, add)
fact(associative, mul)
fact(associative, add)

# Define some wild variables
x, y = var('x'), var('y')

# Two expressions to match
pattern = (mul, (add, 1, x), y)                # (1 + x) * y
expr    = (mul, 2, (add, 3, 1))                # 2 * (3 + 1)
print run(0, (x,y), eq(pattern, expr))         
# ((3, 2),) #  meaning one result with x matches to 3 and y matches to 2

Conclusion

With this LogPy contains all of the functionality of SymPy’s old unify module but in a cleaner and much more extensible form.

LogPy - Facts and Relations

Thu, 17 Jan 2013 00:00:00 -0800

In my last post I introduced LogPy, a library for logic and relational programming in Python. In this post I show how LogPy can be used as a quick and dirty in-memory database.

Data

As an example we’ll look at the 50 states in the US. We know two things about each state.

Is it coastal? For example California (CA) is coastal because it is next to the Pacific Ocean, Arizona (AZ) is not.
To which other states is it adjacent? For example California (CA) is adjacent to Oregon (OR), Arizona (AZ) and Nevada (NV).

We express data in LogPy using relations and facts

>>> from logpy import Relation, fact, facts
>>> coastal = Relation()
>>> fact(coastal, 'CA')

here we have asserted the fact that 'CA' is coastal. Lets quickly do this for all of the coastal states

>>> coastal_states = 'WA,OR,CA,TX,LA,MS,AL,GA,FL,SC,NC,VA,MD,DE,NJ,NY,CT,RI,MA,ME,NH,AK,HI'
>>> for state in coastal_states.split(','):
...     fact(coastal, state)

Adjacency is only slightly more complex to express. The following code asserts that California (CA) is adjacent to Arizona (AZ) and that California (CA) is adjacent to Oregon (OR).

>>> adjacent = Relation()
>>> fact(adjacent, 'CA', 'AZ')
>>> fact(adjacent, 'CA', 'OR')

Now we need a list of all adjacent pairs of states. Fortunately someone else has already compiled such a list. His data looks like this

AK
AL,MS,TN,GA,FL
AR,MO,TN,MS,LA,TX,OK
AZ,CA,NV,UT,CO,NM
CA,OR,NV,AZ
CO,WY,NE,KS,OK,NM,AZ,UT
...

Each line says that the first element is adjacent to the following ones. So for example Alaska (AK) is adjacent to no states and California (CA) is adjacent to Oregon (OR), Nevada (NV) and Arizona (AZ). We can parse this file and assert the relevant facts with fairly standard Python code

f = open('examples/data/adjacent-states.txt')  # lines like 'CA,OR,NV,AZ'
adjlist = [line.strip().split(',') for line in f]
f.close()

for L in adjlist:                   # ['CA', 'OR', 'NV', 'AZ']
    head, tail = L[0], L[1:]        # 'CA', ['OR', 'NV', 'AZ']
    for state in tail:
        fact(adjacent, head, state) # e.g. 'CA' is adjacent to 'OR',
                                    #      'CA' is adjacent to 'NV', etc...

Queries

Once have asserted the relevant facts we can run queries with the logical expressions of LogPy. Recall from the last post that we can use relations to express logical goals and use run to search for cases that satisfy those goals. Here are two simple queries

>>> from logpy import var, run
>>> x = var()
>>> print run(0, x, adjacent('CA', 'NY')) # is California adjacent to New York?
()

>>> print run(0, x, adjacent('CA', x))    # all states next to California
('OR', 'NV', 'AZ')

We can construct more complex queries with multiple goals. In SQL the following queries would require a JOIN

>>> y = var()  # create second variable for complex queries

>>> print run(0, x, adjacent('TX', x),    # all coastal states next to Texas
...                 coastal(x))
('LA',)

>>> print run(5, x, coastal(y),           # five states that border a coastal state
...                 adjacent(x, y))
('VT', 'AL', 'WV', 'DE', 'MA')

>>> print run(0, x, adjacent('TN', x),    # all states adjacent to Tennessee
...                 adjacent('FL', x))    #        and adjacent to Florida
('GA', 'AL')

Facts and relations are currently indexed by default, yielding relatively fast query times.

Conclusion

LogPy provides a declarative interface to query complex data. Data is stored as facts/tuples and queries are expressed as logical goals. This system is expressive and can match SQL in many respects. The use of Logic programming languages for database queries has roots in Datalog a subset of Prolog designed for databases.

Introducing LogPy

Mon, 14 Jan 2013 00:00:00 -0800

LogPy is a library for logic and relational programming in Python. This post contains some introductory examples.

Informative Examples

LogPy enables the expression of relations and the search for values which satisfy them. The following code is the “Hello, world!” of logic programming. It asks for 1 number, x, such that x == 5

>>> from logpy import run, eq, membero, var, conde
>>> x = var()
>>> run(1, x, eq(5, x))
(5,)

Multiple variables and multiple goals can be used simultaneously. The following code asks for a number x such that x == z and z == 3

>>> z = var()
>>> run(1, x, eq(x, z),
              eq(z, 3))
(3,)

LogPy uses unification, an advanced form of pattern matching, to match within expression trees. The following code asks for a number, x, such that (1, 2) == (1, x) holds.

>>> run(1, x, eq((1, 2), (1, x)))
(2,)

The above examples use eq, a goal to state that two expressions are equal. Other goals exist. membero(item, coll), a goal, states that item is a member of coll, a collection.

The following example uses membero twice to ask for 2 values of x, such that x is a member of (1, 2, 3) and that x is a member of (2, 3, 4).

>>> run(2, x, membero(x, (1, 2, 3)),  # x is a member of (1, 2, 3)
              membero(x, (2, 3, 4)))  # x is a member of (2, 3, 4)
(2, 3)

We can write other fancier goals too. Here is a list of all prime numbers within 1..10. primo depends on the traditional prime and isprime functions found in sympy.

>>> from logpy.math import primo
>>> run(0, x, (membero, x, (1,2,3,4,5,6,7,8,9,10)),
              (primo, x))
(3, 2, 7, 5)

Want just a few primes? Here are five numbers that satisfy the primo goal

>>> run(5, x, primo(x))
(2, 3, 5, 7, 11)

Relations

We often want to state and then query data. Logic programming represents data a set of facts and represents queries with logical goals. In the following examples we assert some facts about the Simpsons family, construct queries through logical goals and then run the queries to obtain results.

The following code defines a parent relation and uses it to state who fathered whom.

>>> from logpy import Relation, facts
>>> parent = Relation()
>>> facts(parent, ('Homer', 'Bart'),
...               ('Homer', 'Lisa'),
...               ('Abe',  'Homer'))

We ask some questions using the parent relation as a goal constructor. Who is Bart’s father?

>>> run(1, x, parent(x, 'Bart'))  # one x such that x is a parent of Bart
('Homer',)

>>> run(2, x, parent('Homer', x)) # two xs such that Homer is a parent of x
('Lisa', 'Bart')

We can use intermediate variables for more complex queries. Who is Bart’s grandfather?

>>> y = var()
>>> run(1, x, parent(x, y), 
              parent(y, 'Bart'))  
('Abe',)

We can express the grandfather relationship separately. In this example we use conde, a goal constructor for logical and and or.

>>> def grandparent(x, z):
...     y = var()
...     return conde((parent(x, y), parent(y, z)))

>>> run(1, x, grandparent(x, 'Bart'))
('Abe,')

grandparent demonstrates that we can construct complex relations programmatically. How would you define sibling? How about uncle or aunt? How about descendant?

If you’d like to play with LogPy you can install it with pip or easy_install using

pip install logic

or clone it directly from github

git clone git@github.com:logpy/logpy.git

Source is available at http://github.com/logpy/logpy/, design input and contributions are much appreciated.

Logic Programming in General

Logic and relational programming are making a comeback. They were popular in the 80s, died during the AI dark ages, and have recently begun a resurgence in the functional programming community. Logic programs write music, search databases, write numeric algorithms, and build testing frameworks. It is expressive for a wide class of problems.

The design of LogPy is based off of miniKanren, a simple and powerful implementation in Scheme popularized through the core.logic Clojure library.

References

Statistical Simplification

Tue, 11 Dec 2012 00:00:00 -0800

Lawrence Leemis, a statistician at Williams and Mary, recently published a wonderful interactive visualization on the reduction relationships of statistical distributions. (found via John Cook’s blog)

This excites me because it touches on one of my favorite topics

How do we reusably encode expert knowledge into computational systems?

The Big Challenge

Correct use of mathematical information can accelerate important computations by several orders of magnitude. Unfortunately the people who know the mathematics are not always the ones doing the computation. This results in substantial waste.

How do we integrate expert mathematical knowledge everywhere? One solution is to collaborate more. While collaboration is generally good it doesn’t scale well. As problems become more complex it is more difficult to find all of the necessary experts, especially for smaller relatively unimportant projects. Also, collaboration rarely results in reusable infrastructure. Statistical chemistry projects are rarely applicable to statistical biology problems despite their shared interest in statistics.

One Solution - Multi-Stage Compilation

We could write each expert’s knowledge into a single project and then connect many such projects into a multi-stage compiler. At each each stage we simplify the expression with the knowledge relevant at that stage. We must create a transformation between each pair of connecting stages. Ideally the conceptual distance between connected stages is small and so these transformations are easy.

This isn’t clearly the right solution. It is difficult to chain many small projects together and end up with efficient code. You need to find the right sequence of intermediate layers that are able to communicate mathematical information down to the lowest-level of computational code.

Relevance to SymPy.stats

SymPy.stats endeavors to be a transformation in such a toolchain. It converts stochastic expressions into integral expressions.

The surrounding infrastructure looks like this

When SymPy expressions are imbued with random variables they form stochastic expressions. Sympy.stats transforms these into integral expressions which are then again converted through a variety of methods, either numeric (like Monte Carlo) or again symbolic.

Each stage within this pipeline presents us with the opportunity to simplify the expression with knowledge relevant to that stage. For example at the input and output SymPy Expr layers we make algebraic and trigonometric simplifications like the following

X + X -> 2*X
sin(x)**2 + cos(x)**2 -> 1

At the integration stage we might separate multivariate integrals if possible or use integration by parts.

Notice that there is no such simplification self-loop at the Stochastic Expr node. This is where the information in Leemis’ chart would fit.

A Failing of sympy.stats

Currently sympy.stats does not simplify stochastic expressions with expert knowledge; it converts directly from stochastic expressions to integral expressions without first applying known simplifications like what is encoded in Leemis’ chart. This causes some fairly embarassing failures

In [1]: from sympy.stats import *

In [2]: X = Normal('X', 0, 1)  # A standard normal random variable

In [3]: density(X**2)  
Out[3]: 
<< failure: unevaluated integral >>

Any statistician could tell you that the expression X**2 has a Chi Squared distribution which has a simple and well understood density. This relation is commonly known and commonly occurs in practice.

Because sympy.stats doesn’t know this it blindly takes the expression density(X**2) and converts it directly into an integral. The resulting integral is difficult and stumps the integration routines.* In this case knowing basic statistics would have turned an impossible problem into a trivial one.

Future work

We should encode relations on distributions into SymPy. The knowledge in Leemis’s chart could be written down as a knowledgebase of known transformations. Transformations like the following could solve our immediate problem.

Normal(0, 1) -> StandardNormal()
StandardNormal()**2 -> ChiSquared(1)
StandardNormal()**2 + ChiSquared(n) -> ChiSquared(n+1)

Each stage in the compilation pipeline presents us with an opportunity to apply expert knowledge. The Stochastic Expr stage is such an opportunity of which we are not currently taking advantage.

Leemis’s chart is written declaratively, highlighting which logical transformations are possible under which conditions. The new modules on unification and strategies should provide all of the necessary infrastructure to translate Leemis’ chart to functioning code. Writing a minimal simpliication scheme for the above problem might be as simple as

#    rewriterule(from-pattern, to-pattern, wilds)
p1 = rewriterule(Normal(name, 0, 1), StandardNormal(name), wilds=(name,))
p2 = rewriterule(StandardNormal(name)**2, ChiSquared(name, 1), wilds=(name,))
p3 = rewriterule(StandardNormal(name)**2 + ChiSquared(name, n), 
                 ChiSquared(name, n+1), wilds=(name, n))

statsimp = exhaust(bottom_up(multiplex(p1, p2, p3)))

If anyone is interested in this I’d be happy to help out. This is the sort of project that really excites me but that I can’t currently justify time-wise.

Characteristic Functions

Mon, 03 Dec 2012 00:00:00 -0800

In a recent post, Characteristic Functions and scipy.stats, Josef Perktold created functions for the characteristic functions of the Normal (easy) and t (hard) distributions. The t-distribution is challenging because the solution involves special functions and has numerically challenging behavior around 0 for high degrees of freedom. Some quotes

The characteristic function for the normal distribution is easy, but looking at the characteristic function of the t-distribution, I wish someone had translated it into code already

Since I haven’t seen it yet, I sat down and tried it myself. I managed to code the characteristic function of the t-distribution, but it returns NaNs when it is evaluated close to zero for large df. I didn’t find a Bessel “k” function that works in this case

He then includes his code and discusses a particular application of the characteristic function which I won’t discuss here.

Symbolic Solution

Josef wished that this code had been written already. Characteristic functions aren’t built into SymPy but we may be able to derive them automatically.

Wikipedia says that the characteristic function $\phi(t)$ of a random variable X is defined as follows

$$ \phi_X(t) = E(e^{itX}) $$

It equal to the expectation of exp(i*t*X). Lets do this in SymPy

>>> from sympy.stats import *
>>> mu = Symbol('mu', bounded=True)
>>> sigma = Symbol('sigma', positive=True, bounded=True)
>>> t = Symbol('t', positive=True)

>>> X = Normal('X', mu, sigma)  # Normal random variable
>>> simplify(E(exp(I*t*X)))     # Expectation of exp(I*t*X)
              2  2
             σ ⋅t 
     ⅈ⋅μ⋅t - ─────
               2  
   ℯ

I was actually pretty surprised that this worked as smoothly as it did. SymPy stats wasn’t designed for this.

Here are some gists for the Cauchy and Student-T distributions. Cauchy simplifies down pretty well but the Student-T characteristic function has a few special functions included.

Behavior near zero

Josef says that the behavior of the characteristic function of the t distribution near zero for high degrees of freedom (the nu parameter) is challenging to compute numerically. Can SymPy handle this symbolically?

>>> nu = Symbol('nu', positive=True, integer=True)
>>> t = Symbol('t', positive=True)

>>> X = StudentT('X', nu)
>>> simplify(E(exp(I*t*X)))

The bold scripted I’s are besseli functions. We restrict this expression to a specific number of degrees of freedom, setting nu = 50

>>> simplify(E(exp(I*t*X))).subs(nu, 50)  # replace nu with 50
         ⎛         ___  ⎞            ⎛        ___  ⎞
∞⋅besseli⎝-25, 5⋅╲╱ 2 ⋅t⎠ - ∞⋅besseli⎝25, 5⋅╲╱ 2 ⋅t⎠

Whoops, simple substitution at that number of degrees of freedom fails, giving us infinities. What if we build the expression with 50 from the beginning? This gives the integration routines more information when they compute the expectation.

>>> X = StudentT('X', 50)
>>> simplify(E(exp(I*t*X)))
        ⎛              │     2  -ⅈ⋅π⎞           ⎛              │     2  ⅈ⋅π⎞
╭─╮3, 1 ⎜   1/2        │ 25⋅t ⋅ℯ    ⎟   ╭─╮3, 1 ⎜   1/2        │ 25⋅t ⋅ℯ   ⎟
│╶┐     ⎜              │ ───────────⎟ + │╶┐     ⎜              │ ──────────⎟
╰─╯1, 3 ⎝25, 0, 1/2    │      2     ⎠   ╰─╯1, 3 ⎝25, 0, 1/2    │     2     ⎠
────────────────────────────────────────────────────────────────────────────
                        1240896803466478878720000⋅π

The solution is in terms of Meijer-G functions. Can we evaluate this close to t = 0?

>>> simplify(E(exp(I*t*X))).subs(t, 1e-6).evalf()
0.999999999999479 + 1.56564942264937e-29⋅ⅈ

>>> simplify(E(exp(I*t*X))).subs(t, 1e-30).evalf()  
1.0 - 1.2950748484704e-53⋅ⅈ

This is where scipy’s special functions failed in Josef’s post, yielding infinity instead of 1.

And finally we plot the behavior around 0.

>>> plot(re(simplify(E(exp(I*t*X)))), (t, 1e-7, 1e-1))

Closing Notes

The sympy.stats module was not designed with characteristic functions in mind. I was pleasantly surprised when I entered code almost identical to the mathematical definition

X = Normal('X', mu, sigma)
E(exp(I*t*X))

and received the correct answer. I am always happy when projects work on problems for which they were not originally designed. The fact that this works is largely due to SymPy’s excellent handling of integrals of special functions, due largely to Tom Bachmann.

If you do complex work on statistical functions I recommend you take a look at sympy.stats. It’s able to perform some interesting high level transformations. Thanks to recent work by Raoul Bourquin many of the common distributions found in scipy.stats are now available (with even more in a pull request).

Working notes by Matthew Rocklin - SymPy

SymPy and Theano -- Matrix Expressions

Introduction

Kalman Filter

Theano Execution

Blocked Execution

Results

Analysis

Good Things

Bad Things

References

SymPy and Theano -- Scalar Simplification

Introduction

Example problem

Simplification

Bounds on the cost of Differentiation

Theano Simplification

Experiment

Analysis

Simultaneous Computation

References

SymPy and Theano -- Code Generation

Physics

Numerics

Performance

Leveraging Theano

Conclusion

References

Maximum a Posteriori Estimation

Discussion

References

Assuming assumptions

Old Assumptions

New Assumptions

Global assumptions

Introducing assuming

Commutative Unification

Expression Trees

Matching Expression Trees

Building Commutative Equality

Example

Conclusion

LogPy - Facts and Relations

Data

Queries

Conclusion

Introducing LogPy

Informative Examples

Relations

Logic Programming in General

References

Statistical Simplification

The Big Challenge

One Solution - Multi-Stage Compilation

Relevance to SymPy.stats

A Failing of sympy.stats

Future work

Characteristic Functions

Symbolic Solution

Behavior near zero

Closing Notes

Introducing `assuming`