Every tool you ever build will have some goal it supports. Start with writing the code/examples you want your tool to enable! They wont be quite right, but it gives you a start and a set of examples to evaluate your tool on. Then go write the dumbest implementation of the operations needed to make those examples run. Then write more examples, and discover better refinements for that API, and thence iterate back and forth quickly between top-down and bottom-up approaches to library engineering.
I've spent too much of the past year bogged down in bottom-up only, and just now now starting to see the power of an "alternating minimization" approach.
Speaking of which, I hope to start sharing some interesting writing about how to write and use Numerical solvers and optimization routines in the coming weeks, and Alternating Minimization will (I hope) be one class of optimization routines I'll discuss.
Its been a busy year, and its about to get even busier and much more awesome. Numerical Computing, Compilers, Debugging, Optimization Routines, oh my!