Dave Hudson


Compression and how I want to keep track of information and ideas

2021-01-04 22:38 - 3 min read

For some time I’ve been giving a lot of thought as to how I keep track of information and ideas. Information vis the most important thing I work with.

Over the last 30+ years, I’ve tried all sorts of approaches. I have dozens of hand-written journals detailing my technical thoughts. I’ve tried keeping online journals, including this one. None work well enough. They all end up disconnected and incomplete.

The problem is more complex because I have information that I gather in my personal life and also information that I gather in the course of my work. I clearly can’t just mix the two, and definitely can’t have commercially sensitive data uploaded to arbitrary cloud storage.

This has got me thinking that I’m missing a critical tool.

There are lots of applications that try to make note-taking and linking easier, but they always seem to have way too much user friction for me to work with them. I think that may be because none are designed for the problem I’m looking to solve.

My problem isn’t taking notes, nor is it hyperlinking back to things. Both of these are important, but neither does anything to help me compress the complexity of concepts to something useful. How can I compress the things I know about into the smallest most useful form?

Compressibility is really interesting to me.

I suspect my interest comes from starting software development in the late 70s where home computers just didn’t have lots of storage capacity. As I got to being a teenager and playing computer games, I would marvel at the ingenuity of software developers to cram content into such little spaces. I think I ended up getting as much enjoyment from disassembling games to work out how they did this as I did from playing them!

This has deeply affected the sorts of software I’ve developed. Not only do I love the compression of data but I also love the compression of software. Finding the most compact way to represent a complex problem is something I can’t resist. What I’ve come to realise in the last few months is that this is also the way I like to think about architectural and other knowledge problems.

What I’ve also realised is I hate loss of fidelity. Lossy compression has always upset me at some fairly deep level. It can be incredibly useful at times (e.g. JPEGs and MPEGs) but the results are only approximations of the originals and thus feel diminished.

Normalisation of data and refactoring of software feel related as both serve to reduce redundancy. A quick web search shows this isn’t a new thought. I found the same sort of idea on the c2 wiki but I think my interests are slightly different.

I think I’m interested in compressibility because compressing things implies having found a shortcut to generate the original content. That’s what I’m looking to do with information. Now I probably need to design some software to help me do this!

More journal entries