Silos Are A Good Thing ™

As in any human social network, we in the software world collect this bank of common wisdom that we can begin to believe and follow without question. One of the biggest, partially true, but mostly wrong bits of common wisdom in our space is the sense that “silos” are always a bad thing.

First – let’s start with a bit of a working definition, so we have a reference. And so that we have something to question if we don’t agree on the conclusions that we reach here. A silo is simply a group – a limited set of social connections that we leverage to complete work.

The “limited” nature of the set of social connections is the first part of this that people tend to find implicitly disagreeable. It feels exclusionary. Isn’t inclusiveness an absolute good? No – it’s not. For two reasons.

Firstly, opening a working group to the wrong individual can lower the output of the other individuals in the group. This can happen for a number of reasons – the wrong skill set, the wrong attitude, the wrong vision.

Secondly, no matter how aligned an individual is to the values of the group…no matter how skilled they are….no matter how much of a positive, team player they are….at a point social connections – that is, communication network nodes, slow progress more than they help. It’s the reason we have the beautiful idiom; and why we understand so painfully well what it means to design software “by committee”. Committees are good for vetting all the angles of a thing. But as a group grows in size, its ability to execute diminishes rapidly, because its communication paths multiply. And this is particularly problematic in software where the things we are communicating are so detail rich, context sensitive, and highly technically sophisticated.

So it’s important to build silos…..the right kind of silos. Silos that have just enough people to accomplish what we want to accomplish and no more. We should have the skills and authority to design, build, test and put to use the software that we intend to create embodied in as few people as we can get away with.

And we shouldn’t feel guilty about it. Limiting communication networks to the minimum possible to deliver well and rapidly is the utmost that we can do to add value to our organizations and its customers. This is the foundational piece of our work.

We can’t let “common wisdom” keep us from this.

Now there is a bad type of silo – and if it isn’t obvious what that is at this point, it’s the kind of silo that creates hard separation between the people and skills necessary to deliver our software from top to bottom. That is – if we create organizational units out of those that build a database for the system, from those that build the application code, and also from the people who put the software into operational use – if we do that, we are creating the bad kind of silos that should rightly be frowned upon, and more importantly – avoided.

So we should recognize the importance of structuring our organizations properly. And we should avoid fear of an abstract notion (the silo) that is only bad if it is poorly implemented.

Here’s to building amazing software!

Kyle

What The Big Data Revolution Teaches Us About The Innovating Organization

Tags

agile, influence, leadership

One of the fascinating technical advancements that’s been happening recently has been around distributed data, and distributed data processing. The whole “Big Data” revolution rode in on the back of Hadoop. Hadoop is a tool that used a simple paradigm, the idea of combining two logical set operations (viz. map and reduce) into one clean way to heavily parallelize processing of massive sets of data. Now given, the data and the problem have to be a fit for the solution – and not all data and problems are. But none-the-less, if one’s problem space can be fit to the solution shape, the payoff is huge. There are newer techniques that follow the same broad pattern, but the underlying principle is that problems are split into pieces that don’t rely on each other, and can thus be run in or out of order, on the same computer or a different one, while still eventually arriving at meaningful results.

Data storage has been similarly advancing, largely along the same lines. Several years ago – the line of “no-sql” tools started to crop up offering different technical trade-offs that weren’t available when we relied solely on relational databases for our production data storage. Most interestingly – they allowed for a thing called “eventual consistency”. The dogma throughout my early years as a software geek was that data should always be what it’s supposed to be – it should be – in the fastest way possible – completely and utterly consistent. This turns out to be unnecessary for large swaths of problems. And so our beliefs about this and thus the tools we build for ourselves evolve to support it. The general outcome is that if we say that our database can be running on a number of computers, geographically dispersed, and they don’t even have to be immediately consistent, suddenly storing that data becomes far less burdensome. We can scale by throwing hardware at the problem – so that we don’t have to spend so much expensive thinking time on making the data storage as fully optimal as possible. And now we are commonly in the range of storing more than a Petabyte of data.

I want to highlight that just like with Hadoop and distributed processing, the distributed storage problem has to match the solution shape. Eventual consistency (in this case – and please forgive the radical simplification I’m doing to keep this blog post under a Petabyte itself) has to be ok.

And this is the big idea – the thing that big data teaches us about team organization:

In order to scale out massively, we have to place particular limits on the shape of the problem we are solving.

Demanding 100% control over our data – so that we always know it is entirely consistent – is comforting in a way. But it demands a lot of things about the technical structure of the database. You can’t parallelize operations, you can’t do them out of order, you can’t distribute them. The benefit requires a trade off in restriction. Conversely, if you shape your application and your business logic such that it is useful regardless the immediate consistency of the data, you can scale broadly.

On the data processing side – if we serialize processing, depend on the results of previous operations, and share state, we limit ourself to nearly one thread of processing, with shared, controlled memory to hold state, and scaling is something we can only do by adding more power to the single computer.

In general we give up immediate control for immediate chaos – with an understanding that the answer will emerge from the chaos, because of how we’ve shaped it.

With a computer it’s easy to see why this is true, memory and processing on a single computer, or a tightly coordinated few computers has an upper limit. It always will, though that limit will increase slightly over long periods of time. Designing a solution that can arbitrarily scale to any number of computers makes us only limited by the number of computers we can get a hold of, which is a completely different, and much easier ceiling to push up.

With humans, it’s even worse, until we make a breakthrough and figure out how to implant the cybernetic interfaces directly into our brains, our capacity has been the same for thousands of years. And more discouragingly it will remain the same for many years to come. We can not buy a more powerful processor; we cannot add memory. Some models come installed with a trivial, incrementally larger amount of one or the other or both, but it’s not enough to really make a difference.

To solve large problems, you need more than one person. To solve problems requiring creativity and human ingenuity – humans need autonomy to explore with their intuition. This means that every human being involved in solving the problem will have to saturate their ability to think. No matter how smart the genius – it is a significant stunting of effort, if we pretend like they can serialize all the thinking through themselves. Even if it’s a couple or maybe a small team of geniuses that are tightly coordinated. To really achieve anything great in a modern software organization – or rather, in a modern information-work-based organization – and actually that’s EVERY organization – the thinking has to be fully distributed.

For the EXACT same reason as it has to be with distributed computing and distributed storage. The problem size has a limit if we want full control. It doesn’t if we can allow the solution to emerge from the chaos of coordinated but distributed activity.

And just like the distributed example above, this simply means we have to choose to forge our problems to match the solution shape. Organizationally this means at every level realizing that the leadership and initiative necessary to drive something to completion – the thing that drive us so quickly to top-down, command-and-control structure – has to be embedded in every doer. That positional leaders in the organization should be few and should be “Clock Builders” – to use Jim Collins’ brilliant phrase – shaping the system and providing just enough structure so things don’t fly apart.

And when we get down to the software – the teams have to be shaped so that there are no dependencies on one team to another. Not for architecture, database programming, testing or moving something into production. The small software team that works together day to day, to fully distribute the innovation and creation that we humans are designed so well for, should have everything it needs to handle the software from cradle to grave – including but not limited to – the necessary authority and skill sets – with every team member possessing the leadership and initiative to drive anything through, even if the rest of the organization burned to the ground.

Failure to distribute our creativity and innovation in this way will rightly be viewed as just as backwards as a software development team committing to only storing megabytes of data in a fully consistent relational database is today.

So let’s get organized and build some amazing software!!

Kyle

What Are Managers Good For Anyway (The Role of Leadership in Software Development)

Tags

agile, leadership, Software Craftsmanship

As an on-again-off-again software development manager, I have asked this question of myself on more than one occasion..what *are* managers good for anyway?

It’s a great question – and the answer is not entirely intuitive.

Software development managers, and the expectations about their duties have been modeled after earlier times, and different types of work. The measure of the success of a manager is often directly tied to the software within the purview of the team that is managed. Which, even, from the title of the role seems like a patently obvious way to do things. After all – software and manager are both in there – so it seems like the thing one manages should be the thing one is measured by.

But take a second look even at the title – software development manager – what is it that is being managed? Software? No – software development. And while software isn’t a bad proxy to measure to find the potency of our software development, it can often be a misleading one.

The reason for this is due to the nature of software. Recognized and memorialized by the Scrum framework, the duty of organizing and defining the nature of the software to be developed is vested in a member of the development team known as the Product Owner. This role is envisioned as a partner with the other members of the team and not one that controls the specific actions of the team. One that collaborates to find the proper priority of software features to be developed, but leaves specific action to be defined by the team itself.

The “what” question is separated from the “how” question. And both are meant to be collaborations on a team consisting of both types of roles.

Where is management? Where is leadership? The traditionalist that takes their cues from car factories and construction sites might say that someone, somewhere has abdicated their responsibility. We should fire whomever it was, and find someone that can do their job. And by the way – this Agile stuff only works in textbooks.

They’d, of course, be wrong.

If we’re looking closely – there’s a huge need for leadership in this situation. It’s just not with the first-order concerns of the specific software output. It’s with the second-order concerns of the system that the software is being built by. The software development manager – should be managing the development of software – setting up the systems and practices, hiring the right kind of people, and ensuring the cleanest code. Then, they’ll be managing software development, and then the software will practically write itself.

—

But let’s put this magic aside for a sec, and take a quick step back.

Let’s parse out management and leadership here for a second. Because the difference starts to make a difference.

When we talk about management, we talk about ticking HR checkboxes. For starters, in most organizations, we can’t very well have everyone reporting to the CEO. Training, timesheets, whatever regulatory things that need to be “managed” – that’s the job of the manager. With this comes a certain degree of leverage. This isn’t bad, but it is interesting as we will see in a little bit.

Leadership is about change. If an individual’s or an organization’s inertia will carry them through to whatever thing it is that they want to accomplish, leadership is unnecessary. It’s very similar to physics – acceleration is defined as changing velocity, which is a two-part vector that includes both direction and speed. If either of those change, you’ve accelerated – and more importantly, a force has acted upon you. Leadership is that force – whether applied by someone else, or by yourself – if your velocity – your speed or direction or both – if they change, leadership has been applied to your situation.

I won’t cover the various ways that this force can be applied, or the tools used to generate it. But it is sufficient to say that the organizational leverage that a manager might wield is one of the most obvious though least powerful methods.

So management for a leader is a tool to the end of changing momentum.

—

Software is an interesting beast, if I hadn’t been clear enough about it earlier. And it is so for several reasons.

Firstly, if left alone there is a natural drag that will decelerate its delivery until it stops entirely. That’s right – if nothing is done to push back against this natural force, an organization that starts to deliver software, given a certain amount of time, will be unable to continue.

Secondly, the problem solving capability of the software team is embodied in the knowledge of the individual team members and in the communication networks within the team. Unless the team is very weak or severely restricted, no single individual would be able to hold enough information to make a better, quicker decision than the team itself.

Following from this second situation, it’s important to recognize that direct manipulation of the team – ordering a specific action – will lead to a reorganization of the team’s collective understanding. It will also create a sense of dependence on the one giving the order — since blindly following a directive is easier, quicker to execute, and necessitates a lower level of responsibility for the results – all of which are pleasant in the short-term.

—

So hopefully the hole that the software leader can and should fill is becoming obvious. But to pull all of this together…

Our challenge is basically, as a software development manager, what do we do when output isn’t reaching potential.

The answer is that we have two options – the path most taken would be to exercise direct manipulation of the team. That is – we might give specific commands about how to deliver…break out the carrot and the stick, threaten, entice – go directly for the output and work back to specific actions that can get that for you.

This is problematic because it makes the system worse, as we said above, it necessitates a reorganization of the team to accommodate the adhoc input (which will be less optimal than it was prior to the command) and it creates dependency. Both of these make “software development” – the thing we are managing – the system that delivers the software, worse. Which means, next time you will be delivering less well.

The second option is to work on the system itself to make it better.

What is the system? It is all the things that go in to creating software, from hiring the team, to the workflow the team uses to create the software to the delivery methodologies the team has for putting software into production. It includes paring and mobbing, TDD and the taste your team members have to create modular, well-crafted software that’s a snap to understand and change. And probably more that I’ve forgotten to mention.

So – you focus on the system and the next time around your output is better. The only challenge is having the character to not stress out and react incorrectly and too quickly to the pain of disappointed expectations (whether or not they were in any way realistic).

Fantastic – so that’s a win. But it sounds like a one time thing. That’s not exactly a full-time job for a software development manager.

Right – except the whole thing was predicated on the challenge that “the output didn’t meet the potential”. Which, as much as this may hurt to hear, will in fact always be the case. Further, as we said above, software development is funny because it rots – the system itself that is. It becomes slower over time. So even to maintain the momentum we have currently, requires a “force”, requires leadership to overcome.

So – even something as boring as maintaining the status quo in software requires leadership. Certainly to make headway and to become a truly great example of software delivery won’t happen under anything less than heroic exertions by a gallant generation of brilliant software leaders and the brave souls that follow them.

Let’s be them!

Happy coding!

Kyle