Boolean algebra

The thing to notice is that a bit ( or ) can be represented as a vector: and

An identity (NoOp) gate is then and a NOT gate is .

To operate on multiple bits at the same time we tensor them together by applying the Krondecker product. So the boolean number is equal to , for example. We end up with a vector of length with an entry for each possible value of the binary number, in order¹.

Higher order operators are easier here than in the quantum case. We can just write out the result that we want as columns in a matrix, where each column represents , , , , respectively. So for example the gate is simply: .

When we multiply a pair of binary digits (tensored together) by this matrix then we get the answer we want. Example: $ AND * (T F) F$.

The beautiful thing is that we can then use wiring diagrams to build more elaborate calculations, like we do in quantum computing circuits. It works because this boolean algebra is an example of a symmetric monoidal category having a braiding that is essentially a SWAP operation between bits. This category allows us to compose operations by shifting bits into place with an appropriate sequence of swaps. This is very similar to the quantum computing equivalents but without the restriction to unitary (reversible) operations. Dropping this restriction means that we can give the category copy² and discard³ operations, turning it into a copy-discard category⁴.

Examples like this make applied category theory such a powerful thinking tool.

Kauffman’s example

A simple circuit for Kauffman’s 3 node example is fairly easy to construct. The three operators correspond to the three truth tables, one for each node. The node value is cloned/copied. This is not allowed in quantum computing circuits but no problem here.

This circuit does a trick to avoid an extra copy. The copied bit is dropped at the end to make a routine with matching inputs and outputs. This allows multiple similar operations to be chained together to evolve the system.

Some more tricks are required⁵ to implement the swapping and shifting of registers in code but luckily I had already done something similar for the quantum computing simulator I wrote some years ago. The and gates were relatively easy to think through and the tensor product just worked perfectly for all these unfamiliar operations (there are no-cloning and no-deleting restrictions in quantum circuits which make things harder in that context).

Of course with this simple example (and the benefit of hindsight) the matrix corresponding to this entire circuit could have been been constructed from scratch from the extended truth table. However this way the actual relations themselves are baked into the circuit. The final matrix, let’s call it , will be the same anyway. Here’s what it looks like in code (note that matrix multiplication is from the right, the opposite of the circuit diagram):

OPX = op(kauffman_truth_table[1,:])
OPY = op(kauffman_truth_table[2,:])
OPZ = op(kauffman_truth_table[3,:])

M = lift(3,3,DROP)*lift(3,3,OPZ)*lift(3,2,OPY)
        *row_swap(4,2)*row_swap(4,1)*lift(3,2,OPX)
        *row_swap(4,2)*lift(3,3,CLONE)*row_swap(3,2)

> 8×8 SparseMatrixCSC{Int64, Int64} with 8 stored entries:
    ⋅  ⋅  ⋅  ⋅  1  ⋅  ⋅  ⋅
    1  ⋅  1  1  ⋅  ⋅  1  ⋅
    ⋅  ⋅  ⋅  ⋅  ⋅  ⋅  ⋅  ⋅
    ⋅  ⋅  ⋅  ⋅  ⋅  ⋅  ⋅  1
    ⋅  ⋅  ⋅  ⋅  ⋅  ⋅  ⋅  ⋅
    ⋅  1  ⋅  ⋅  ⋅  ⋅  ⋅  ⋅
    ⋅  ⋅  ⋅  ⋅  ⋅  1  ⋅  ⋅
    ⋅  ⋅  ⋅  ⋅  ⋅  ⋅  ⋅  ⋅

Quite a lot of work went into getting that matrix mostly full of zeros!

Testing out the circuit we do get back the results from the original paper:

for state in 0:7
    s = Bool.(digits(state, base=2, pad=3))
    S = transform(s)
    println(s => measure(M*S))
end

[0, 0, 0] => [0, 0, 1]
[0, 0, 1] => [1, 0, 1]
[0, 1, 0] => [0, 0, 1]
[0, 1, 1] => [0, 0, 1]
[1, 0, 0] => [0, 0, 0]
[1, 0, 1] => [1, 1, 0]
[1, 1, 0] => [0, 0, 1]
[1, 1, 1] => [0, 1, 1]

So now the whole point of doing this wasn’t just to build the circuit in a new way. It was to use the power of linear algebra to tell us something about the network and its properties. Being a matrix we can calculate its eigenvalues and eigenvectors. If we can find eigenvalues with a value of exactly 1 then we will have found stationary points in the network, i.e. cycles.

The eigenvalues $= { _1, _2, …, _8 } $ of turn out to be:

λ = eigvals(M)

8-element Vector{ComplexF64}:
   -0.5 - 0.8660254037844388im
   -0.5 + 0.8660254037844388im
    0.0 + 0.0im
    0.0 + 0.0im
    0.0 + 0.0im
    0.0 + 0.0im
    0.0 + 0.0im
    1.0 + 0.0im

Now it just so happens that those non-zero values are the cube roots of unity. Using the fact that the cubes of the eigenvalues of a matrix are the eigenvalues of the matrix cubed then we should get all ones.

λ = eigvals(M^3)

8-element Vector{Float64}:
   0.0
   0.0
   0.0
   0.0
   0.0
   1.0
   1.0
   1.0

The last 3 eigenvalues are now exactly one! So let’s look at the last 3 eigenvectors and convert them back to the corresponding states:

[measure(ν) for ν in eachcol(eigvecs(M^3)[:,6:8])]

3-element Vector{Vector{Bool}}:
 [1, 1, 0]
 [0, 0, 1]
 [1, 0, 1]

These are exactly the states which form the only cycle in the example kimatograph.

It is not coincidence that the other eigenvalues are zero. The fact that is also no coincidence. There is much more to learn about the spectral properties of these matrices.

Wolfram’s cellular automata

Once that had worked, I was interested in matrix operations for bigger boolean networks. As before we can apply the same process to Wolfram’s cellular automata which are just specialisations of the general boolean network. This network can be as big as you like. How can we go about building it systematically? This was an interesting problem to mull over and I eventually worked it out. First we can come up with a trinary operator representing the rule.

This operator has 3 inputs, two of which just pass through to the outputs. The other output is the result of the Wolfram coded rule. The reason for passing through the two outputs is to be able to chain them together easily. Of course we could just copy the values but this way seems neater. To create the complete circuit it also has to wrap around the ends to form a periodic boundary. That’s done by just copying the bottom bit to the top and the top bit to the bottom. The extra two bits are dropped at the end to make a composable operation. So we’re left with this circuit:

Note that there are input states and output states so the operation can again be chained by matrix multiplication on the left.

What does this look like in code? Something like this although I expect there are many ways to do it:

function Rule(code::Integer)
    outputs = binary(code, 8)
    R = zeros(Bool, 8, 8)
    for k in 0:7
        X,Y,Z = reverse(binary(k,3))
        R[:,k+1] = transform([outputs[k+1], Y, Z])
    end
    sb(R)
end

function update(N, rule)
    # Clone the ends and shift to top or bottom respectively
    WRAP = row_shift(N+2,1,N+1)*lift(N+1,N,CLONE)*row_shift(N+1,N+1,2)*lift(N,1,CLONE)

    # Build the rule gate
    R = Rule(rule)

    # Chain the rule N times from the left
    Rᴺ = foldl(*, lift(N+2-2,i,R) for i in N:-1:1)

    # Drop the last two bits
    PRUNE = lift(N,N,DROP)*lift(N+1,N+1,DROP)

    # Build the full circuit
    PRUNE*Rᴺ*WRAP
end

Took a while to get the bit order of all this right but the idea was right. I did some nice testing and got it working as it should. As a big test I used the circuit to build and run different systems and compare to existing catalogues. This was the result and I’m pretty happy with it.

Looking at the spectra of the update matrices is pretty interesting. As a simple example we can discover eigensystem of the different rules.

In fact we can look at all the eigenvalues of the same group of rules as above. These are plots of the complex plane from (-1,1) in both axes.

A few interesting things immediately jump out. First, every rule has zero eigenvalues and the rest are on the unit circle. This is consistent with what we saw for the simple boolean network above.

The rules with eigenvalues in $ ,0,1 $ correspond to the simplest rules: 123, 127, 128, 132, 136. In fact, rules with widely and evenly spaced eigenvalues seems to lead to simple periodic behaviour. Rules 130 and 134 are examples.

Rules 122, 126 and 129 share the same eigenvalue distributions have triangular run-in evolutions although all different in detail. Rule 131 also has uneven eigenvalues and a triangular update rule.

Rules known to be complex like 124 and 137 have a large number of eigenvalues. There can be a maximum of eigenvalues so the natural question is if any reach this maximum? Rule 120 has many eigenvalues but the pattern is simple. I suspect that in this case the behaviour is highly dependent on the initial state.

There is a strong resemblance here with the permutation matrices and that’s not surprising. The update rules are, in a sense, permutations with degeneracies. Those degeneracies are the “spider’s legs” in the state diagram. The results suggest that those degeneracies correspond to the zero eigenvalues of the matrix. The degeneracies are zero rows in the transition matrix. Powers of the transition matrix essentially zero out rows associated with the spider’s legs of the permutations, eventually converging on the cycles of a true permutation.

More questions

Some things that have occurred to me as I was doing this:

• Curious now for a deeper understanding of the Krohn-Rhodes theorem, to understand it and to see it in the light of category theory. What, if any, is the relation to signal-flow diagrams?
• What would the phase space of a probability or complex state be like as different operators are applied to them. Do they tend to go be attracted to the fixed points or do they spins about in other ways? It would be quite easy to set up a 2D (or maybe 3D is minimum?) state and just apply random operators to it and display the path. It would be quite cool if there were strange attractors hidden there somewhere.
• Intuition is telling me that we could use cycles to represent classical probabilities and “compress” deterministic algorithms to probabilistic ones. In other words, move computation from the time domain into a wider state space. I’m not sure if -machine⁹ does something similar. Curious to look more at this.
• Can we construct a fourier transform operator by combining cycles of lengths and working backwards from the transition matrix to a boolean network? Is there there anything like a Shor’s algorithm, for example?
• Is there such a thing as an “indivisible” permutation which would link this to probability in the way that indivisible unitary matrices link probability to Hilbert spaces? See Barandes.

The setup

The paper begins like this:

Proto-organisms probably were randomly aggregated nets of chemical reactions. The hypothesis that contemporary organisms are also randomly constructed molecular automata is examined by modeling the gene as a binary (on-off) device and studying the behavior of large, randomly constructed nets of these binary “genes”.

First some terminology. Kauffman refers to his net as “a binary (on-off) device”. We will call this a boolean network because each node has a boolean value attached to it, on or off. When he uses the word “gene” (in quotes) he’s referring to the individual nodes of the network.

There are nodes and each node is attached to other nodes. By way of example Kauffman uses a simple network with 3 nodes and 2 connections.

The network is allowed to evolve in a particular way:

… the inputs to each binary “gene” may be chosen at random; the effect of those inputs on the recipient element’s output behavior may be randomly decided by assigning at random to each element one of the possible Boolean functions of its inputs.

For example, if a node has two inputs A and B (K=2) then the rule might be or AND or XOR , assigned at random, and the node’s value would be updated accordingly.

Alternatively, these rules can be represented in full generality as a truth table. Since for this study we don’t care what the actual rules are in terms of ANDs and ORs, we will construct the update rules simply by taking a random truth table.

To reproduce Kauffman’s simple example in the paper we assign the following truth table to the network:

As an aside, there are entries in the truth table. This suggests to me that a tensor representation might be more efficient for simulation, especially for larger values of .

Node	Inputs
		0	1	0	0
		0	0	0	1
		1	1	0	1

Enumerating the small number of possible states in this example we find the following update rule:

 [0, 0, 0] => [0, 0, 1]
 [0, 0, 1] => [1, 0, 1]
 [0, 1, 0] => [0, 0, 1]
 [0, 1, 1] => [0, 0, 1]
 [1, 0, 0] => [0, 0, 0]
 [1, 0, 1] => [1, 1, 0]
 [1, 1, 0] => [0, 0, 1]
 [1, 1, 1] => [0, 1, 1]

When the update rule is applied multiple times, the values of the nodes enter a dynamic wholly defined by the truth table. Since this is a finite system, at some point the values will enter a cycle. The maximum possible cycle length is (or in our small example) but often the cycles will be shorter. By tracing the dynamics for each of the possible state values we can draw a graph of their trajectories³.

By looking at the diagram and following the arrows, it’s clear that all the states eventually fall into the same cyclic behaviour. There is a transient (or run-in) period between the initial state and the first state encountered on a cycle. Kauffman defines a confluent as the set of states leading into, or on, a cycle. In this case there is only one.

The results

In the paper, Kauffman studied the dynamics of large networks, keeping . Just by way of example a graph representation of a network with might look like the following diagram.

This network has 14 nodes labelled from 1 to 14 (). Each node has two incoming connections ().

The situation clearly becomes more complicated very quickly and you’d expect the dynamics to be equally complicated. However, Kauffman goes on to say:

The results suggest that, if each “gene” is directly affected by two or three other “genes”, then such random nets: behave with great order and stability.

This is the crux of the study and mostly borne out but we will see that there are important exceptions. Kauffman realised this and the nuance became of central importance in the development of complex systems theory although it is understated in this early paper.

Take, for example the histogram of cycle length. Kauffman discovered that the average cycle length for () nets with random truth tables was smaller than might be expected. We can reproduce the result fairly well.

There are immediately some things to note. First the similarity is remarkable, especially considering that this is a run of 200 random samples out of a possible states. There are peaks at 1, 2, 4, 6, 8, 10, 12, 16, 20, 24, etc. and the trend is decreasing with larger cycle lengths.

There are two important caveats, though.

The preponderance of cycles of length two in Kauffman’s results is not present in mine. Initially, I thought this was a bug in my code but I checked and double checked the calculations by hand and couldn’t find a problem. Moreover, closer study reveals a second, related caveat worth mentioning.

I have artificially chopped the histogram at the value of 55 to match Kauffman’s. However, my results include cycles of lengths much greater then 55, some having exceeded the limit of 10,000 which I had to add to stop the simulations running forever. I have individual instances of cycle lengths of 60, 62, 72, 78, 80, 84, 96, 102, 111, 116, 168, 186, 248, 282, 292, 320, 381, 438, 458, 482, 508, 1017, 1260, 1281, 1552, 3066, 3500, 3628, 6527 and eight more reaching the upper limit.

Suspiciously, my results typically show a similar number of cycle lengths above 55 as the difference in Kauffman’s and my results for 2-cycles. For example, the 2-cycle count from my experiment shown in the figure, was 9. The number of cycles of length greater than 55 was 39. Kauffman’s 2-cycle count was 40.

I don’t have Kauffman’s original code and I can’t formally prove my code is correct. Nonetheless I have found an interesting way of testing it which is instructive in itself. The idea is to narrow in on Stephen Wolfram’s well-known 1D cellular automata as a special case of Kauffman’s boolean nets.

Wolfram’s classes

It turns out that Wolfram’s famous study of elementary 1D cellular automata (which he discusses in great depth in his book “A New Kind of Science (2002)”) can easily be modelled as a boolean net. The good thing is that the results are visually quite distinctive and can serve as a reference.

Make a truth table based on one of the possible rules. Wolfram codified them as numbers from 0 to 255. The truth table is identical for each of the nodes.

Allow the system to evolve, keeping records on the history of states as the rows of an image. This is what you get, for example for rule 26:

Wolfram and others have studies these nets in depth and famously found that they exhibit behaviour that fits into 4 classes:

The first class leads to a single terminal, stationary state. All 0s or all 1s. In terms of Kauffman’s nets, the cycle length in this class is 1 and we have seen that many rules lead to this outcome. Kauffman notes that removing rules which always resolve to 0 or 1⁴ significantly reduces the occurrence of this class.

The second class reduces to fixed patterns with simple periodic behaviour. Rule 26 shown above falls into this class. While it is a fairly complicated (using the term advisedly) pattern it is periodic and predictable.

The third class is entirely chaotic. The pattern does not converge and is only periodic to the extent of the finite state space (which may be extremely large). The appearance is of white noise and randomness and, in fact, is available as a random number generator in Mathematica.

The fourth class demonstrates complexity. The patterns are non-periodic but also not chaotic. These configurations are on the edge of chaos, neither periodic nor chaotic. One interesting property that at least two of the rules in this category have is universality⁵, or in other words the ability to perform any calculation.

What’s the relevance of all this? Well, by converting the well-documented Wolfram rules to Boolean nets I’m much more comfortable saying that my simulation is working as expected. I’ve tried many different rules and the patterns are identical to the published ones.

On top of that, since the simple cellular automata studied by Wolfram is an example of very simple Kauffman network then the Kauffman networks in general must necessarily admit such classes of behaviour, including class 3 and 4, the chaotic and universal ones. In this light, Kauffman’s discovery that the average cycle length remains smaller than might be expected is just a part of a much richer picture.

At this point I got side tracked onto a parallel way of modelling the networks which has some promise. The write up of that will have to wait until the next post.

A model is a model

In any real team the batch size won’t be constant but, all other things being equal, this model is good enough to tell us that regular delivery of smaller batches is preferable to larger ones.¹

Also in real teams the work items are generally not completely independent. This can be due to internal team dynamics, one piece of work being a prerequisite of another, etc. The practical effect of this is to increase the variance further. One nice thing about the negative binomial as a modelling tool is that it can easily be tuned to the teams actual data by increasing its variance slightly while keeping the mean constant. In the past I have found this to be a very good approximation of data gathered from real teams.

Another nice thing about using a known distribution is that we can go beyond normal Monte Carlo and use Bayesian inference for forecasting. Having a Bayesian model has the advantage of being deterministic and smoother, regulating and reducing noise or small sample effects that are sometimes evident in Monte Carlo simulations.

As always these results need to be used with caution and a full understanding of what they mean. Nonetheless having models to help us understand the mechanisms and principles behind our intuitions can be very handy at times.

Reproducing Dunbar’s exact results

So yesterday, with a combination of OCR and careful copying, I recovered Dunbar’s original data from his paper. Here’s a reproduction of a plot of the data - satisfyingly similar to the original.

Not sure why those axis limits were chosen, maybe just rounding to orders of ten. In any case group sizes less than 1 make no sense and neocortex ratios beyond that of humans are not useful. From now on the plots will focus on the meaningful ranges. They’ll also have the vertical axis on the right so that predicted values are easier to read off.

To this graph Dunbar’s original fit is overlaid. One again his famous number, 150, appears as the predicted value for human group size.

Using this data and using exactly the same RMA (aka Geometric) linear regression we can recover his results almost² exactly. The next plot shows the agreement along with the 95% confidence interval as calculated following Rayner³.

This agreement was exactly what I was hoping for by using the original data and reproduces Dunbar’s result satisfactorily.

Residuals

One again we can apply simple residual checks on the data to ensure we are satisfying the linear regression assumptions. The qqplot shows the residuals close to a normal distribution.

The residuals plot itself shows some variance but no obvious heteroscedasticity.

Confidence in the mean

While the fit itself is statistically sound, if we look again at Figure 3, we can see that the logarithmic scale underplays the numerical range. Looking at the same graph with the vertical axis scaled linearly then the problem is clear.

The confidence interval (which is only for the mean itself) is from under 100 to nearly 250. This is worse than the previous data because the slope is greater. I think you will agree that this is a wide error margin which is partially obscured by the log-log view. If we don’t believe the confidence interval calculation (and we shouldn’t, some doubt has been placed on it⁴, see below) then let’s take a different approach and compare.

Assuming the parameters of the fit are normal, as we must because it was an assumption of the regression itself and has been somewhat confirmed empirically, then we can generate random samples of regression lines drawn from those distributions. This is a so-called Monte-Carlo simulation of the regression distributions and gives us an alternative way to histogram the predicted values. The following plot shows 3000 such samples together with a histogram of the predicted values for human group size.

Again we find an average in the right range (see below) but once more with very wide error margins for the prediction. Actually it’s worse. The distribution is clearly heavy tailed towards higher group sizes. This is an expected effect of converting a Normal distribution on a log scale to a linear scale where it becomes Log Normal and therefore long tailed. Taking 1 million such predictions it’s evident that it conforms very closely.

Taking these results we can confirm that the 95% confidence interval has widened further, from 71 as a lower bound to 368 as the upper. Remember that this is still the range for the mean itself and does not take into account the variability in the samples.

There is another insight here too. The mean of the LogNormal distribution is not 150 but rather 176. The reason for this is exactly as stated in Feng et al. and can be confirmed by direct calculation. The mean of the distribution when transformed back into natural units is not but rather . The variance shifts the mean.

Confidence in the prediction

We’ve stated several times that until now we have been looking at inferences related only to the prediction of the mean of the regression line, not including the variance in the data itself. Let’s take that into account now. We know the variance in the residuals and under the assumption they are normally distributed in log space then, in principle, we can model the residuals in the predicted value.

The normal distribution of the mean regression and the normal distribution of the residuals is combined using the relation: . We can then carefully extend this to linear space by expressing as a Lognormal distribution with parameters and .

The factors are necessary because the original data was transformed base 10 and the relation between the normal and the Lognormal is via the natural logarithm.

Using this new distribution the confidence interval has now widened still further. The 95% interval ranging between 31 to over 740. The mean now is above 200 due to the upward effect of the increased variance.

Conclusion

With the original data it’s been possible to reproduce Dunbar’s famous results almost perfectly.

However it has become even clearer that the extrapolation of the regression line to human scales is highly questionable. The prediction is well outside the existing data, leading to wide error margins, and the log transformation exaggerates the error still further. If we also take into account the residual variance in the data, the confidence interval of any prediction widens beyond useful limits (according to my analysis, the 95% interval is from 71 up to 740).

This is not an issue related to the type of regression nor any internal structure in the samples but rather the overwhelming error margins that make sensible prediction based on the available data unreasonable.

This is all consistent with the Stockholm University group’s “deconstruction”⁵ of not only the number itself, but the notion that such a number is even sensible to talk about.

Fitting the data

On a double log-log plot, my grandmother fits on a straight line - Fritz Houtermans

Here’s an example of a similar data set to the one used in the original studies. The data is tabulated in the paper but it’s a lot to copy out so I took a similar data set published by the same author in a later paper. Figure 1 is a plot of the data. For multiple species of animal, it shows the ratio of the size of the neocortext to the size of the rest of the brain () on the horizontal axis and the average social group size () for that species on the vertical axis.

Neither Dunbar nor his critics used the data in this form however, they used its logarithm. This is a common technique used under the assumption that it will make skewed data (as this is) appear more symmetrical (normal). There is no explanation of the reasons for the transform in this particular case. Nonetheless it has an important effect on the results. We’ll come back to this point later. Here’s the transformed data.

I’ve presented it in exactly the same way as Dunbar’s paper. Although the data is slightly different the similarity is clear. Also, it does look a bit more like a straight line now and the variance is a little less pronounced. I’m not sure how valid this kind of subjective judgement is.

Figure 3 overlays Dunbar’s original result onto this data set and there is no surprise. The line (the equation for which is given explicitly in the paper) is projected out to show the predicted human mean group size which can be seen to be close to 150. This is the source of Dunbar’s number.

I’m tried to reproduce this result but for now we note that for the analysis to work we have to make the strong assumption that an errors in the measurements are normally distributed. The choice of using the log-log transformation another strong assumption that we’re taking as fact for the moment but we will return to it.

The parameters of the line are determined using linear regression which attempts to find the best straight line through the points. The question arises: what is the best straight line? It turns out that there are multiple ways of defining best fit and depending on which you chose you can get different results.

The most common type of linear regression is most commonly called Ordinary Least Squares³ (OLS) where the the best line is chosen to be the one that minimises the squares of the vertical difference between each point and itself. This is the most basic regression technique taught in statistics classes.

The problem with OLS, as Dunbar points out in his original paper and again in his recent article, is that by minimising only the vertical distances there is an implicit assumption that there are no errors in the measurements on the x-axis, which is not true in this case.

To take these (unknown) error margins into account in both axes we can choose to minimise the geometric distance between the points and the line. This is called Geometric Mean Regression⁴ in much of the modern literature. The geometric distance is the area between the point and the line, the triangle spanning both the distance along the x-axis and the distance along the y-axis.

One cited problem with geometric regression is the extra difficulty in the interpretation of confidence intervals for the parameters. I haven’t seen that as being any more of a problem for this type of regression than for any other but I might be wrong. Please read the paper if you are interested.

I did both geometric and OLS regression to compare the difference. Figure 4 shows the results of both types of linear regression transformed back into the original, untransformed scales.

This confirms Dunbar’s assertion that OLS considerably underestimates the slope of the regression line, as compared to geometric one.

There are two other important things to note here beyond the difference between the two lines. First notice the upward curve of the regression lines over the original data. This is a result of the log transformations, essentially producing an approximately exponential fit due to the differing scales of the two axes.

The second thing to notice is the wide spread of the confidence interval for this line. This is also exaggerated by the log transformation. In fact, the validity of the confidence intervals after transformation is highly questionable. See Feng et al. for more details.

Notwithstanding the above caveats, what happens if we extrapolate the curve in an attempt to predict a typical group size for humans?

A direct extrapolation of this data using the geometric fit gives a group size prediction for humans as very approximately ~100. This well below Dunbar’s result but some difference should be expected due to the different data. It does however highlight the disproportionate effect of the log-log transformation on the prediction. It also puts into question the conceptual validity of extrapolating so far outside the available data. Remember that this is the spread of the confidence interval of just the mean itself and does not extrapolate from the variance seen in the rest of the data. This is a key point and an interesting realisation for me personally.

Until now we have collected a series of assumptions about the data that we’ll look at in turn in light of the analysis.

Conclusion

So, after all this analysis, I learnt a great deal about regression considerations. Least squares never seemed so controversial until I started this. My results came in below Dunbar’s original results. This means nothing, of course, but it does lend some credence to the critics suspicions which would be compatible with my little investigation here.

I also learnt a lot about what can and can’t be said about Dunbar’s number as a concept.

First, none of this says anything about the nested structure of the group sizes and other predictions made subsequently, which have also been used widely⁵.

Secondly, it is most definitely not a maximum or upper limit, as often stated. We need to be careful not to confuse the confidence interval of the sample mean with a measure of confidence of the prediction.

It is clear to me now that stating Dunbar’s number as 150 is misleading and misses a great deal of context. Even assuming the validity of the model and the regression, it is a predicted mean value extrapolated far outside the underlying data with extremely wide error bars and needs to be treated with a great deal of caution. The prediction has even wider margins to the point of losing some of its meaning as a useful concept.

So how much did the map help?

To answer this question I had an idea for a toy model to compare Dora’s and Diego’s adventures and did a bit of programming to try it out. The images above are actual screenshots. Here’s the setup.

The island is a grid. Getting from one point on the grid to an adjacent point has a difficulty representing the lie of the land (the difficulty of movement). This is calculated numerically as a cost function where the cost depends on the change in height¹. This corresponds to the intuitive notion that steeper sections are more difficult to traverse than flat sections and any bumps on the ground make the going harder.

Then we consider two grids. One grid is set up like a map, an idealised representation of the contours of valleys and mountains and a few of the largest features. Another grid represents the actual terrain, with additional detail of smaller obstacles which are not marked on the map. The ruggedness² of the terrain is applied randomly but its magnitude is parametrisable.

For different levels of ruggedness, I ran a shortest path finding algorithm ³ across the map (Dora’s planned route) and across the terrain using step sizes of varying lengths. Think about the size of these steps as corresponding to different amounts of fog for Diego: visibility is the second parameter. In every case calculating the total difficulty over the whole journey.

Since this is a numerical model we can get a feeling for what’s happening by studying how the model behaves as the parameters (ruggedness and visibility) change. I ran the model thousands of times to get a distribution of results for different parameter settings.

Diego vs Dora

First let’s see how Diego’s and Dora’s experience compares in general as the ruggedness of the terrain changes.

As you might guess, Dora did well when the map was highly accurate (the far left of the graph), but, unfortunately for her, any additional ruggedness of the terrain which was not marked on the map made her experience much more difficult than expected when following the route she planned in advance.

Diego on the other hand didn’t stick to a predefined route so sometimes was able to adapt to obstacles not on the map. As the ruggedness of the terrain increases past a certain point he consistently does better than Dora.

This is the first observation: there is a trade-off between following a plan and adapting as you go and it depends (at least) on the ruggedness of your terrain. There is a book in that statement⁴.

A second observation here, often overlooked, is predictability. This can be interpreted on the graph as the width of the ribbon. A narrow ribbon means low variability which corresponds to greater predictability. Again Dora’s progress is very predictable when the map matches reality but that predictability decreases rapidly as reality bites. Interestingly, Diego’s progress is much more predictable than Dora’s as the terrain becomes more and more unpredictable; the variability in his routes is, in fact, fairly constant.

More predictability means being able to have greater confidence in the final outcome. In many projects involving multiple external stakeholders this predictability is actually more important than the details of any plan itself. Read that again.

Third observation: Past a certain point of ruggedness, not only does Diego consistently do better than Dora, he also tends to pick a path close to the best route in hindsight. Let’s see this on a graph.

Intuition holds in the sense that the best route also gets more and more difficult (and variable) as the ruggedness of the actual terrain increases. Diego hardly ever finds perfection but, interestingly, his experience tracks surprisingly closely the best possible one. Again we can see that Diego’s strategy is more predictable than Dora’s as he can “think on his feet” and avoid any trouble encountered on the way.

So how much does Dora’s experience deviate from her best made plans?

The relationship between the plan and reality has become a cliché. As Field Marshal Helmuth von Moltke is said to have said:

“No plan of operations extends with any certainty beyond the first encounter with the main enemy forces.”

Can we see this in our little model? The answer is yes and very strikingly.

When the map perfectly matches the terrain then her experience is as expected but as the map fails to show the detail, the difficulty and variability of her experience increases enormously. Compare this to Diego’s experience compared to the very best route. Field Marshal Helmuth von Moltke was definitely right.

There is another maxim which is usually attributed to Eisenhower:

“Plans are worthless, but planning is everything.”

The strategies used by Dora and Diego are two extremes but they can be combined. We can plan and we can adapt too. What happens if we lift the fog on Diego to allow him to “look ahead” with greater visibility to the future.

If we give Diego predictive superpowers he gets closer and closer to the best possible route. Perfect prediction is equivalent to 20/20 hindsight.

If we believe Winston Churchill who said:

“…the best generals are those who arrive at the results of planning without being tied to plans.”

Then Diego would have made a pretty good general!

A model is just a model

Erica Thomson, in her book Escape from Model Land: How Mathematical Models Can Lead Us Astray and What We Can Do About It (2022) wrote:

“Models are not simple tools that we can take up, use and put down again. The process of generating a model changes the way that we think about a situation, encourages rationalisation and storytelling, strengthens some concepts and weakens others.”

This is a model with many implicit assumptions, for example, there are no existing paths or roads on the map. Assuming these were clear of obstacles then these would potentially give Dora a significant advantage. What roads would do in effect is to reduce uncertainty in the map, making her strategy more effective. Likewise dead ends like impassable cliffs or rivers are unlikely to be present.

Also there is some flexibility in the destination, it’s not a specific location. If it were then Diego would have a more difficult time finding it because he wouldn’t be able to “go roughly east”. There is some scope here for further study. Are there strategies that Diego could follow to get to a specific spot in a reliable way?

The model is set up in such a way that ruggedness and difficulty are directly related, in fact linearly related. This is visible as the slope of the line on the graph. Similarly the intercept of the line is the minimal difficultly of the map. On the other hand, there are no units on the axes. That’s because it’s not clear to me how those units would map to the real world anyway. By removing the units I’m saying that I don’t know⁵. I could attempt to build them in but then it would be a different model.

In fact, where did the the parameters of ruggedness and visibility even come from? Well, they came from my intuition and I had to play around with the model to see if my intuition was valid or not. It turned out that it was but it doesn’t preclude other parametrisations and observations.

As it stand, it does do two things, however. Firstly, it reifies my intuition. It’s not just some fuzzy idea in my head, or me being persuaded by appeal to authority. It’s a numerical model that has built-in assumptions and a story to tell. You might have another.

Secondly, it uses data to demonstrate the results, not hand wavy suppositions. Assumptions in, observations of data out. Both ends require a certain leap of faith but at least it’s an honest, transparent and verifiable one. That’s one way to escape Model Land.

It took me a couple of full days to create this model and another couple to write up the results. My intuition is stronger now and I have a model which generates interpretable data. When someone says Alfred Korzybski’s phrase “the map is not the terrain” again, I will think back to this model. The next time someone asks me why we need a plan, I’ll have an honest answer.

Next draft:

Remove repetition of the “no axis” stuff.

Need a “So what?” section which explicitly interprets the model in terms of project management.

GPS?

The results

The difference in the distributions of the concurrent requests in with the different strategies is clear.

Notice how remnants of the input and output distributions are still visible in the round robin strategy. The least occupied and random 2 strategies tend to concentrate the distribution around its average. On the other hand the random strategy seems to spread the distribution still further.

How does the average vary with load factor?

All strategies perform progressively worse as load factor increases, as would be expected. However the least occupied and random 2 strategies are noticeably better that the others, even at higher load factors.

And how does it change with number of servers?

Clearly the best strategies require a minimum number of servers over which to spread the requests. That number is relatively small (in this example around 10) and very little improvement is observed with additional servers. The others are unaffected by number of servers.

Variance

The spread in concurrent requests will translate to a spread in latencies too. For a distributed systems we are usually interested in maintaining predictable latencies and minimising long-tails, so we want to minimise this spread. It’s visibly clear that least occupied is the best in this case as it has the narrowest distribution. Let’s have a look at the variance for each one, taking round robin as the base.

What is the relationship between number of servers and the variance?

We can see that when there is only a single server, all strategies are the same (obviously). Both the round robin and random strategies have little or no effect on the variance, no matter how many servers there are in the cluster. Although bot least occupied and random 2 fare better, the least occupied has a clear advantage here and seems to be able to capitalise on additional servers more effectively.

Shedding

We employ a shedding mechanism to help keep request distributions under control independently of the load balancing strategy. How does this affect the results? Here we apply a simple shedding of requests by disallowing more than 20 concurrent requests per server.

The mean of the worse strategies has actually gone down, however much more shedding is being done. That means that server throughput (serviced requests) should be lower. On the other hand the average latencies will also be higher due to the higher load on the server.

Conclusion

Of all the strategies round robin and random are disastrous and either do nothing to improve the distribution of requests or actually make it worse. However, the least occupied and random 2 strategies are able to take advantage of multiple servers to not only reduce the mean but also reduce the variance across the cluster.

While the least occupied is slightly better in terms of the spread of requests, the random 2 has some other advantages. Firstly, it’s slightly simpler and therefore faster in practice because only 2 servers are checked for each request rather than all of them. More importantly, it avoids servers which are (re)starting receiving all the load immediately. This is useful when the server needs some time to warmup caches, etc.

Dependency Hell

The first one I want to cover was discovered by mathematicians Paul Erdős and Alfréd Rényi in the late 1950s¹ and concerns the way networks tend to join together as new connections are added.

They considered what would happen if you start with a fixed set of independent things and then progressively add new connections between them at random.

One might think that the connectivity on the whole would increase in proportion with the number of connections, but this is not the case. The connectivity is not only non-linear² but undergoes a phase transition³ at which nearly everything joins together very quickly. Let’s see it happening.

We start with 20 completely independent objects (people or system components) then start adding connections randomly. What you find is that initially nothing much happens: you have some pairs of things and a few threesomes.

Adding a few more connections and we can see groups starting to coalesce.

However add just a couple more connections and the different connected groups⁴ of the network quickly connect together to form much a larger (so-called giant) group.

This happens every time. If we run the above scenario many many times with a thousand points and plot the size of the largest group, the tendency is remarkable.

For people networks this is the first part⁵ of the Kevin Bacon game, you are certainly connected to Kevin Bacon and indeed to everyone else on the planet.

Another lesson to be learnt from this scenario originated some years ago in a paper⁶ building on the originals mentioned above. It adds to the model a destructive process where connected groups are “burnt down” periodically. The key is that the building and burning down of the groups balances to a precarious and dynamic equilibrium⁷, so-called self-organised criticality.

The take-away for systems designers is that we sometimes see outages in our systems and then “repair” them by scaling or using tools like circuit-breakers and time-outs. At the same time other connections are being added by new features and services. We find ourselves more often than not on the critical line between stable and unstable.

Tipping Points

Let’s look at a second model. This time, rather than taking completely independent objects and gradually connecting them together, we take a set of object which are all connected together, but with varying weights.

This might represent the number of times that one person interrupts another person per day, or the number of requests from one service to another in a distributed system.

We then gradually increase all the weights using some common scaling factor.

This scenario was first proposed by Lord Robert May in a famous paper⁸ from 1972. He found⁹ that the interactions between the individual parts become unstable (that is move away from a balanced and steady equilibrium) at a certain sudden, non-linear transition.

We start with a network where all objects, this time just 10 of them, are connected to each other by differing strength interactions.

Using May’s model we can calculate the threshold where the loops and feedback in the network make the whole system unstable. As before, rather that try and study specific configurations, lets run the example many times with different random networks of this type to see the tendency.

May’s criteria for stability has drawn some controversy¹⁰ but the finding is still striking. There is a definite tipping point where network effects produce a phase shift in the dynamics of the system.

Imagine the network being a team of 10 people each communicating freely with everyone else. The team members which work most closely together would have the strongest interactions. At the tipping point, a slight change in the situation or team dynamic might be amplified by the network effects of interactions with multiple people and could cause a cascade that affects the team as a whole. This is of course just a model and no doubt unrealistic in detail but it wouldn’t be the first time that a seemingly trivial change would destabilise a team.

Alternatively, imagine the network being a distributed software system. If a peak in load is experienced and a component is pushed beyond its response threshold once again it can affect the entire system disproportionately.

As with the previous example, teams will naturally detect and counteract these tipping points - people with too many interactions will turn off Slack if they can’t keep up. Profilers will be brought to bear to optimise some code just enough to stop overloading the system. This has the result of leaving us constantly on the brink of catastrophe.

Wrap up

In this article there was no intention to make precise predictions about the dynamics of any particular team or system but rather demonstrate universal tendencies using basic ensemble techniques.

We’ve seen a couple of simple examples of how small increases in interactions, either in number or in strength, can have significant, non-linear effects for a system as a whole.

Furthermore, our naturally reactions tend to balance the network effects and leave us at a critical point¹¹ where dangerous tipping points are continually at the doorstep. To a certain extent this explains the universal saying “if it ain’t broke, don’t fix it”.

The combination of these dynamics is why a single new team member can bring a working team to a standstill or a small change in load can wreak havoc on a seemingly well oiled system. Note that this doesn’t mean that we should avoid change, just that we need to bear in mind how the system might respond.

Strict minimisation of direct and indirect dependencies is not just about clean architectures. Removal of dependencies and working towards additional quality measures beyond the minimal “it works” moves systems further from the critical point and hence makes them considerably more stable. On the other hand, increasing code entropy and poor design choices will push a system towards the critical point, making them less stable, even if they continue working.

I’m aware that the software industry is not used to talking about systems in these terms, some of these ideas could be considered technical and abstract. Nevertheless these examples are real and can help us increase our literacy with complex systems, our “lived, practical complexity”. They are just couple of the many results of this type that could be applied more generally.

Data

Imagine a random area of a memory chip full of transistors, each either on or off. One might suppose that those states mean something to someone but without any context it remains data and, actually, does not provide information at all.

That may seem counter intuitive because we are used to seeing data as information but, thinking about it, it can’t be. First, we don’t even know if it is a complete representation of anything. It’s just a state with no context. It might as well, in fact, be random¹.

Often, as programmers we confuse data with information and pay the price with bugs and errors, as we’ll see in a moment.