What The Big Data Revolution Teaches Us About The Innovating Organization

Tags

agile, influence, leadership

One of the fascinating technical advancements that’s been happening recently has been around distributed data, and distributed data processing. The whole “Big Data” revolution rode in on the back of Hadoop. Hadoop is a tool that used a simple paradigm, the idea of combining two logical set operations (viz. map and reduce) into one clean way to heavily parallelize processing of massive sets of data. Now given, the data and the problem have to be a fit for the solution – and not all data and problems are. But none-the-less, if one’s problem space can be fit to the solution shape, the payoff is huge. There are newer techniques that follow the same broad pattern, but the underlying principle is that problems are split into pieces that don’t rely on each other, and can thus be run in or out of order, on the same computer or a different one, while still eventually arriving at meaningful results.

Data storage has been similarly advancing, largely along the same lines. Several years ago – the line of “no-sql” tools started to crop up offering different technical trade-offs that weren’t available when we relied solely on relational databases for our production data storage. Most interestingly – they allowed for a thing called “eventual consistency”. The dogma throughout my early years as a software geek was that data should always be what it’s supposed to be – it should be – in the fastest way possible – completely and utterly consistent. This turns out to be unnecessary for large swaths of problems. And so our beliefs about this and thus the tools we build for ourselves evolve to support it. The general outcome is that if we say that our database can be running on a number of computers, geographically dispersed, and they don’t even have to be immediately consistent, suddenly storing that data becomes far less burdensome. We can scale by throwing hardware at the problem – so that we don’t have to spend so much expensive thinking time on making the data storage as fully optimal as possible. And now we are commonly in the range of storing more than a Petabyte of data.

I want to highlight that just like with Hadoop and distributed processing, the distributed storage problem has to match the solution shape. Eventual consistency (in this case – and please forgive the radical simplification I’m doing to keep this blog post under a Petabyte itself) has to be ok.

And this is the big idea – the thing that big data teaches us about team organization:

In order to scale out massively, we have to place particular limits on the shape of the problem we are solving.

Demanding 100% control over our data – so that we always know it is entirely consistent – is comforting in a way. But it demands a lot of things about the technical structure of the database. You can’t parallelize operations, you can’t do them out of order, you can’t distribute them. The benefit requires a trade off in restriction. Conversely, if you shape your application and your business logic such that it is useful regardless the immediate consistency of the data, you can scale broadly.

On the data processing side – if we serialize processing, depend on the results of previous operations, and share state, we limit ourself to nearly one thread of processing, with shared, controlled memory to hold state, and scaling is something we can only do by adding more power to the single computer.

In general we give up immediate control for immediate chaos – with an understanding that the answer will emerge from the chaos, because of how we’ve shaped it.

With a computer it’s easy to see why this is true, memory and processing on a single computer, or a tightly coordinated few computers has an upper limit. It always will, though that limit will increase slightly over long periods of time. Designing a solution that can arbitrarily scale to any number of computers makes us only limited by the number of computers we can get a hold of, which is a completely different, and much easier ceiling to push up.

With humans, it’s even worse, until we make a breakthrough and figure out how to implant the cybernetic interfaces directly into our brains, our capacity has been the same for thousands of years. And more discouragingly it will remain the same for many years to come. We can not buy a more powerful processor; we cannot add memory. Some models come installed with a trivial, incrementally larger amount of one or the other or both, but it’s not enough to really make a difference.

To solve large problems, you need more than one person. To solve problems requiring creativity and human ingenuity – humans need autonomy to explore with their intuition. This means that every human being involved in solving the problem will have to saturate their ability to think. No matter how smart the genius – it is a significant stunting of effort, if we pretend like they can serialize all the thinking through themselves. Even if it’s a couple or maybe a small team of geniuses that are tightly coordinated. To really achieve anything great in a modern software organization – or rather, in a modern information-work-based organization – and actually that’s EVERY organization – the thinking has to be fully distributed.

For the EXACT same reason as it has to be with distributed computing and distributed storage. The problem size has a limit if we want full control. It doesn’t if we can allow the solution to emerge from the chaos of coordinated but distributed activity.

And just like the distributed example above, this simply means we have to choose to forge our problems to match the solution shape. Organizationally this means at every level realizing that the leadership and initiative necessary to drive something to completion – the thing that drive us so quickly to top-down, command-and-control structure – has to be embedded in every doer. That positional leaders in the organization should be few and should be “Clock Builders” – to use Jim Collins’ brilliant phrase – shaping the system and providing just enough structure so things don’t fly apart.

And when we get down to the software – the teams have to be shaped so that there are no dependencies on one team to another. Not for architecture, database programming, testing or moving something into production. The small software team that works together day to day, to fully distribute the innovation and creation that we humans are designed so well for, should have everything it needs to handle the software from cradle to grave – including but not limited to – the necessary authority and skill sets – with every team member possessing the leadership and initiative to drive anything through, even if the rest of the organization burned to the ground.

Failure to distribute our creativity and innovation in this way will rightly be viewed as just as backwards as a software development team committing to only storing megabytes of data in a fully consistent relational database is today.

So let’s get organized and build some amazing software!!

Kyle

What Are Managers Good For Anyway (The Role of Leadership in Software Development)

Tags

agile, leadership, Software Craftsmanship

As an on-again-off-again software development manager, I have asked this question of myself on more than one occasion..what *are* managers good for anyway?

It’s a great question – and the answer is not entirely intuitive.

Software development managers, and the expectations about their duties have been modeled after earlier times, and different types of work. The measure of the success of a manager is often directly tied to the software within the purview of the team that is managed. Which, even, from the title of the role seems like a patently obvious way to do things. After all – software and manager are both in there – so it seems like the thing one manages should be the thing one is measured by.

But take a second look even at the title – software development manager – what is it that is being managed? Software? No – software development. And while software isn’t a bad proxy to measure to find the potency of our software development, it can often be a misleading one.

The reason for this is due to the nature of software. Recognized and memorialized by the Scrum framework, the duty of organizing and defining the nature of the software to be developed is vested in a member of the development team known as the Product Owner. This role is envisioned as a partner with the other members of the team and not one that controls the specific actions of the team. One that collaborates to find the proper priority of software features to be developed, but leaves specific action to be defined by the team itself.

The “what” question is separated from the “how” question. And both are meant to be collaborations on a team consisting of both types of roles.

Where is management? Where is leadership? The traditionalist that takes their cues from car factories and construction sites might say that someone, somewhere has abdicated their responsibility. We should fire whomever it was, and find someone that can do their job. And by the way – this Agile stuff only works in textbooks.

They’d, of course, be wrong.

If we’re looking closely – there’s a huge need for leadership in this situation. It’s just not with the first-order concerns of the specific software output. It’s with the second-order concerns of the system that the software is being built by. The software development manager – should be managing the development of software – setting up the systems and practices, hiring the right kind of people, and ensuring the cleanest code. Then, they’ll be managing software development, and then the software will practically write itself.

—

But let’s put this magic aside for a sec, and take a quick step back.

Let’s parse out management and leadership here for a second. Because the difference starts to make a difference.

When we talk about management, we talk about ticking HR checkboxes. For starters, in most organizations, we can’t very well have everyone reporting to the CEO. Training, timesheets, whatever regulatory things that need to be “managed” – that’s the job of the manager. With this comes a certain degree of leverage. This isn’t bad, but it is interesting as we will see in a little bit.

Leadership is about change. If an individual’s or an organization’s inertia will carry them through to whatever thing it is that they want to accomplish, leadership is unnecessary. It’s very similar to physics – acceleration is defined as changing velocity, which is a two-part vector that includes both direction and speed. If either of those change, you’ve accelerated – and more importantly, a force has acted upon you. Leadership is that force – whether applied by someone else, or by yourself – if your velocity – your speed or direction or both – if they change, leadership has been applied to your situation.

I won’t cover the various ways that this force can be applied, or the tools used to generate it. But it is sufficient to say that the organizational leverage that a manager might wield is one of the most obvious though least powerful methods.

So management for a leader is a tool to the end of changing momentum.

—

Software is an interesting beast, if I hadn’t been clear enough about it earlier. And it is so for several reasons.

Firstly, if left alone there is a natural drag that will decelerate its delivery until it stops entirely. That’s right – if nothing is done to push back against this natural force, an organization that starts to deliver software, given a certain amount of time, will be unable to continue.

Secondly, the problem solving capability of the software team is embodied in the knowledge of the individual team members and in the communication networks within the team. Unless the team is very weak or severely restricted, no single individual would be able to hold enough information to make a better, quicker decision than the team itself.

Following from this second situation, it’s important to recognize that direct manipulation of the team – ordering a specific action – will lead to a reorganization of the team’s collective understanding. It will also create a sense of dependence on the one giving the order — since blindly following a directive is easier, quicker to execute, and necessitates a lower level of responsibility for the results – all of which are pleasant in the short-term.

—

So hopefully the hole that the software leader can and should fill is becoming obvious. But to pull all of this together…

Our challenge is basically, as a software development manager, what do we do when output isn’t reaching potential.

The answer is that we have two options – the path most taken would be to exercise direct manipulation of the team. That is – we might give specific commands about how to deliver…break out the carrot and the stick, threaten, entice – go directly for the output and work back to specific actions that can get that for you.

This is problematic because it makes the system worse, as we said above, it necessitates a reorganization of the team to accommodate the adhoc input (which will be less optimal than it was prior to the command) and it creates dependency. Both of these make “software development” – the thing we are managing – the system that delivers the software, worse. Which means, next time you will be delivering less well.

The second option is to work on the system itself to make it better.

What is the system? It is all the things that go in to creating software, from hiring the team, to the workflow the team uses to create the software to the delivery methodologies the team has for putting software into production. It includes paring and mobbing, TDD and the taste your team members have to create modular, well-crafted software that’s a snap to understand and change. And probably more that I’ve forgotten to mention.

So – you focus on the system and the next time around your output is better. The only challenge is having the character to not stress out and react incorrectly and too quickly to the pain of disappointed expectations (whether or not they were in any way realistic).

Fantastic – so that’s a win. But it sounds like a one time thing. That’s not exactly a full-time job for a software development manager.

Right – except the whole thing was predicated on the challenge that “the output didn’t meet the potential”. Which, as much as this may hurt to hear, will in fact always be the case. Further, as we said above, software development is funny because it rots – the system itself that is. It becomes slower over time. So even to maintain the momentum we have currently, requires a “force”, requires leadership to overcome.

So – even something as boring as maintaining the status quo in software requires leadership. Certainly to make headway and to become a truly great example of software delivery won’t happen under anything less than heroic exertions by a gallant generation of brilliant software leaders and the brave souls that follow them.

Let’s be them!

Happy coding!

Kyle

The one thing you need to do to build great software….

Killing Software Development Capacity, Part 1: The Performance Review

I’ve been on both sides of this morbid practice with several different organizations at this point.  The similarities are striking everywhere I’ve been.  The process usually goes something like this:

1. Set “goals” starting at the top of the organization
2. Make those goals more specific as you move down the org chart
3. For each individual
   1. Make sure that there’s awareness that financial incentives rest on meeting this
   2. Ostensibly allow for modifications to goals throughout the year
4. At the end of the year (and sometimes throughout the year)
   1. “Calibrate” performance across all disciplines and parts of the organization
   2. Use the goals starting at the leaf nodes and working up to the top of the org as the baseline
5. Individual managers rate their employees, but are held to the calibration done by the larger management team.  And the calibration is typically used to formulate a stack rank – a bell curve of performance that must be fit

As I noted above – there has been remarkable consistency in the different organizations I’ve been a part of with regards to the thinking about and implementation of this process.  I have to assume that there’s a common source of knowledge that leads to this – simply because the coincidence is a bit too stark.  Though, I am unaware of what the source is.

There are a lot of similarities, both in the form and the results of the practice between this system and the software development approach commonly known as Waterfall.  And as with Waterfall, I would concede that on the surface, it’s intuitively very appealing.  And I can imagine a lot of scenarios where it may be a very helpful technique.

Software Development is not one of these scenarios.

It cannot be, due to its nature.  I will do my best here to spell this out specifically for software development, though I imagine that the arguments apply equally to any information work.

Software development is a learning activity.  It tends to get conflated in the popular mind with the act of typing in a computer program (“coding”).  The act of actually entering the code is the least time consuming, least important part of developing software to meet a human need.

Understanding the problem space and the logical bounds that must be worked within, and how one might put the two together and model the specific solution is the bulk of the work.  Because of our cognitive limitations, that will always involve entering some code, seeing how well it serves the problem, and how clearly it reflects a model of the problem, fixing whatever we weren’t able to get right the first time and trying again.

That is, software development is always experimental, and the real value that is created as a part of that experimentation is learning;  learning that is reflected in the individual, and in the organization, and also in the software code itself.

The Appeal

The appeal of the system laid out above is two-fold.  And it is based on some undoubtedly real needs.

Appeal #1: Organizational Leaders want to be able to set direction.  That is – at every level the management team has a real desire to be able to feel comfortable that they’re driving the people working for them in a direction that is consistent with the direction that has been laid out for them.  The filtering down, with further specification of goals is a neat, clean way to do this, that almost seems to remove the messy human stuff from the system.

Appeal #2: Organizational Leaders want to have the sense that the organization that is accountable to them is moving with urgency and with an appropriate amount of energy.  The financial awards system is seen as the carrot that can be consistently applied, again in a neat and clean way that removes as much controversy as possible.

Again, both of these appeals come from genuinely important functions of authentic leadership – to provide direction, and to generate the collective energy to get there.

The Problem

Seeing the set of problems here requires some Systems Thinking.  I’d recommend (re-)reading Senge’s seminal work on the topic “The 5th Discipline“.

Problem #1: The Impossibility of Planning Software – Anyone who has worked in software for more than a day has come to realize with the rest of the industry, that predicting precisely what software we will be working on even as little as a couple of weeks out into the future is close to impossible.  Unless there is an extensive artificial structure in place to prevent it – the software you build today (again, because software development is primarily learning) informs the software you build tomorrow;  tomorrow’s software affects that of the next day, etc.  The uncertainty multiplies the further you get into the future.  This is why Agile has emerged from the software development world.  The  only way to build software that really meets needs is to try something and adjust.

If as a software developer I predicted what software I would be working on just 6 months into the future, even in the broadest terms, I would need to adjust many times over.  That adjustment would need to be fed back into the goals that had come down, potentially adjusting those with the learning that has occurred, disrupting a ton of work that was done by a ton of folks to figure out what those goals were in the first place.  And almost precisely like the Waterfall methodology – the force of this inertia makes this a practical impossibility.

Practically what happens is that the goals that an individual software developer writes has nothing to do with the actual software they’re writing, so that any changes won’t be disruptive to the larger system.  Which, in case it’s not obvious, completely blows away the usefulness of this tool as a mechanism for setting direction.

Problem #2: Individual Focus – the goals for a specific individual are always, almost as a logical extension of the system, formulated only between the individual and their immediate supervisor.  This means that teams aren’t being built around the things that their members are ostensibly working toward; because one team member is fully unaware of what any other team member is most interested in accomplishing.  And because of this it is almost impossible to not send the message that how someone performs as an individual is the key expectation from the leadership team.  This was heartbreaking as a software development manager, because no matter how hard I would focus on building a team, the system wouldn’t have it.

Problem #3: Motivation – One of the primary appeals of this system is that it gives a nice, clean way “to motivate people”.  Attach money to whatever it is you want people to do, and voila, problem solved – people will produce whatever you want.  This betrays a couple deep misunderstandings of human nature.  Firstly, money is not a real motivator – people will be anxious to make more money until they feel they are being paid fairly (to which, accomplishing more things is only one solution – leaving is another).  Secondly, it presumes the organization to take on the initiative to motivate someone.  People can be demotivated by others, but naturally we want to accomplish – we want to make a difference.  Ironically, attempting to bribe someone into doing more is really introducing a significant demotivating factor.

This is multiplied when the things that are being incentivized by financial pressure are unrelated to the work that they are primarily interested in doing (an unfortunate outcome of Problem #1).

Problem #4: Manager Bottleneck – Unlike in prior times or in other types of work, in information work, the doers possess the specific knowledge required to make something happen.  There is no way that a single individual would be able to possess a team’s worth of knowledge about a particular piece of software that is to be delivered.  Yet if the manager is expected to drive the team in a particular direction using the goals as his steering wheel, it requires that he know exactly what every single job requires, and how to execute it.   And further, if he or she is not actively directing the work, because their goals will be tied to the execution of the goals of those under them, they will have a powerful pressure on them to review the output of their reports.

This is absolutely antithetical to a strong software delivery team, where the full team shares the knowledge and responsibility necessary to deliver software.  And it makes the software development manager a bottleneck to delivery.  Which, in addition to being a major limitation on all but the most junior teams, means that there will be wide variance from team to team based solely on the manager’s skill level. (And none of this accounts for the burnout that will be the manager’s lot for being put in this kind of a situation)

Further, any technological innovation, process improvements, or other enhancements that the team is interested in doing to the way it works all require up-front planning with the manager, since it will have to fit in as a part of goals.  Which means, that instead of being group exercises, they are necessarily top-down driven initiatives that will rise or fall on the manager’s say rather than on the collective voice of those doing the work.

Problem #4 is probably the most dire.  But like Problem #1, the vent that usually lets the pressure off is that the actual goals are placed on things that are less relevant and less important than the actual software that is being delivered.  And since there is more and more pressure adding up to this end, less and less value gets placed on the usefulness of the tool.  And ultimately ….

The Ultimate Outcome

Because the upper levels of management create a system that actively necessitates avoidance, reduces its own value in the eyes of those using it, while creating an illusion of control and motivation, the resulting cynicism should be unsurprising.

The Solution

I’m a firm believer that you shouldn’t whine about a problem without pointing to the solution.

The solution here is to – and I steal from Jim Collins here – recognize that there are no silver bullets in leading an organization.  Directing people toward a common goal is a messy, human situation that requires a deep commitment to communicating unceasingly, to really understanding your business and the way the people fit into it.  And most importantly it requires a deep understanding of how software delivery (and really information work) differs from the old style of work that didn’t fall apart as readily under a heavy top-down hand.

1. Fully eliminate this idea (Performance Review) from any part of your organization that engages in software delivery.
2. To meet the direction needs, re-evaluate the role of the software development leader.  Make it their role to work on the system – to formulate a delivery practice based on agile principles.
3. Re-evaluate the manager to staff ratio.  The software development manager should have almost nothing to do with the specific software and everything to do with creating a healthy system.  Because of this the number of staff reporting to a single manager can be far higher.
4. With this decreased need for management, flatten your organization – eliminate as many intermediate layers of management as possible, so that top leaders can be more readily in contact with doers.  So that feedback can REALLY flow back up to the leaders making the strategic decisions.
5. Take all that money you’re saving – and really pay the doers what they are worth.

None of this is particularly exciting – especially because it involves a lot of communication.  Something, that if we asked people how they really feel, they might opine that it is not productive – know that it is.

It also eliminates a lot of positions of authority.  Authority is a thing that can attract some people.  Though, I would argue that it attracts the wrong kind of people.

Organizing like this will do nothing but lead to better software, happier people, and much more value being delivered to the world we serve.

Happy coding!

Kyle