madbean

Matt's first law of software complexity

25 Jul 2003

The first law of thermodynamics states something like this: the total energy of the system plus the surroundings is constant. Or basically energy is conserved; it cannot be created or destroyed, just moved around. This first law is an important realization for chemists/physicists, and underpins the way they look at problems.

I think there is a similar realization that can be made regarding the complexity of software systems, and that it can change the way you look at and design software. It would go something like this:

Matt's first law of software complexity:

The underlying complexity of a given problem is constant. It can be hidden, but it does not go away.
Or:
Complexity is conserved by abstractions. In fact, apparent complexity can be increased by abstractions, but the underlying complexity can never be reduced.

Complex problems remain complex, now matter how much good design and architecture you heap on top. Abstractions hide the complexity. But this is a good thing! A good abstraction divides a problem in two, putting part of the complexity on either side. A good abstraction puts most of the complexity where it can be solved by an existing solution to that subproblem.

But complex problems remain complex, and abstractions leak. If you come up with a good design that lets you hide the complexity of a problem, and then you forget about that complexity, rest assured that it will eventually come back to bite you. Hide complexity, but don't ignore it.

You may have easily implemented a huge, performant web application because you used off-the-shelf components for presentation, persistence, messaging, clustering and fail-over. You applied well known abstractions like J2EE, RDBMSes, Web Services, MVC... the list goes on. But abstractions leak. Clustering and fail-over is inherently hard no matter what the vendors tell you, and the complexity often leaks through under load. HTTP and TCP do a great job of getting bytes from one place to another, until a bug feature in the OS or router shows itself (like MTU size or sockets closing on DHCP lease re-acquisition).

Use abstractions to move complexity to where it is more easily dealt with. But don't forget that it is still there.

Mathematicians know a great trick that can be used to sidestep my first law of complexity: redefine the problem. The underlying complexity of a given problem is constant, so change the problem. This can be the single most useful outcome of a requirements gathering exercise. Someone may come up and ask you to solve this problem: "we need an enterprise-bus messaging system". But if you dig a little bit, you may find out they would be happy with occasional automated emails, a .forward script and a CGI.

Use abstractions to move complexity to where it is more easily dealt with. But remember complexity is conserved by abstractions; and abstractions leak.

  • Home
  • Blog