submitted by Chris
Chris has submitted a story of how organisational dysfunction leads of the failure of a strategic software project. It is a classic example of how non-engineering led organizations often misunderstand what software development requires, putting faith in the wrong people and ignoring those with most experience – their software developers.
We are an events company and at the core of our IT architecture is a third party events management product. The product offers most of the features we need as a business but it is expensive so we only pay for the core features and build our own applications on top of that to get all the functionality we need.
One issue we have is that when someone wants to make a configuration change, such as a new type of event, we must update dozens of applications. In these applications, there is if this then that logic everywhere. So we need new ifs when we add a new event type. This is getting more and more costly and we regularly have many defects across the system because of it. Everyone recognises that we have a problem with our architecture.
Then both our software architects leave the company and there is a feeling in the company that we need a change, we need a new architecture. So our Events Management Architect has the idea to build a centralized suite of services that sit on top of the events management product and all applications communicate with these new services via REST APIs. These new services define a new layer of configuration and control regarding which applications can see what data and use what data etc. It also centralizes some core custom logic.
The vision of the Events Management Architect is that when an application uses these services it is given the list of event types, locations, languages, cultures etc that the application can use, including default values. Each application must be entirely dynamic, that is, when a new event type is added, the application sees the new event type and dynamically changes its behaviour. No more adding new if statements everywhere.
The trouble is that this Events Management Architect doesn’t actually know about software development, he knows about the events management product. Yet he has designed a technical solution to a technical problem and defined the architecture to boot. We don’t have an authoritative software architect anymore, no one to question the project and so it gets the go ahead.
We, the development team, pointed out that none of the applications are dynamic right now and that making them dynamic will take a couple of years of development time and might not be possible. Perhaps partially dynamic, but sometimes we need new behaviour for a new event type that just doesn’t exist right now. Given the huge number of different B2B services, customer services and integrations with third parties we have, it is unrealistic to expect that adding a new event type, or changing an existing an event will only be a configuration change. Not to mention the risks of causing an outage via a configuration change.
We were not listened to and were told, with some passion and frustration, that it will work.
The requirements on day 1 is a diagram with some circles on it. Each one representing a different functional area of the events management system. All told there are about 20 circles. Of those circles about 12 have been highlighted for the MVP. It is the minimum that can be delivered we are told. The Events Management Architect has decided that three months is the deadline.
At the same time, external consultants are brought in to help with the project. We get a software architect consultant and an Agile consultant.
First off the software architect decides that we need to go for Docker and a new tech stack. The Agile consultant decides we need to use Scrum. Us developers have experience of neither and more over we have never worked with each other. We were hand picked from various teams for being recognised as top talent in the development department. We are excited about using Docker for the first time and a new technology stack. It seems a bit risky given the tight timings and not really having an idea of the work that needs to be done, but we start with enthusiasm.
Things quickly go downhill though.
The first three months
Domineering Software Architect
First of all the software architect is a dominating character. In our Scrum meetings nobody gets to finish a sentence before he jumps in and takes over. His pattern is basically: be dismissive of the suggestion and then suggest the most minimal thing that would make the up coming demo work.
For example, rather than putting data in a database he said we should just store the data in a json file on disk so we could move quickly with agility. Why use a database when we don’t even know the details of what we need?
Technical debt starts piling up alarmingly. This json file idea is quickly shelved in favour of a database, but that kind of absolute minimum for the demo mindset is pushed on us relentlessly. This is backed up by the Scrum Master who is backed up by the Product Owner (our line manager).
Speaking Schedule Oriented Development
Our second issue with the architect consultant is that it is clear he is only willing to invest time in Docker. After a while of hammering out functionality it becomes obvious that the business domain is pretty complex and modelling it is going to be a challenge. At this point, a couple of weeks in, he checks out and leaves us to it and concentrates entirely on Docker. He still derails us in every meeting but he doesn’t get involved in the day to day design and coding work.
Our enthusiasm for Docker soon dissapates as it is clear that we are not going to be able to touch Docker. He actively blocks us from meetings about it and is always vague about it when we ask questions. It becomes clear that he is only interested in Docker. Our theory is that he was looking for a success story that he could use for his speaking schedule.
We spend about 25-30% of our time in Scrum meetings. We have the usual stand ups, sprint planning, backlog grooming, demo meetings and retrospectives.
Obviously this impacts our productivity. The planning and grooming sessions are pretty bad. The system we are building is complicated but the user stories we are given by the functional analyst consists of a meaningless As a, I want, So that, followed by about 4 lines of text. So inevitably we end up discussing each story for over half an hour. Most of that is arguing the meaning of concepts, arguing the viability of the story and simply just trying to understand it. With so little information we just often don’t know what to build.
We are told to better prepare for the meetings but what preparation is there to do? It takes two minutes to read the user story content, there is so little in them. In any case we estimate each story and make a commitment to complete the sprint. The word commitment is emphasised greatly.
Over the next three months we don’t meet our commitment once. We average about 50%. Basically it turns out that each user story is greatly more complicated than the 4 lines of requirements describe. We have all kinds of issues:
- we waste a lot of time chasing the functional analyst to understand the requirements
- the analyst works with a QA to test our software and rejects much of it which requires a lot of back and forth, often across multiple sprints
- we have a new tech stack and are learning it on the go, this slows us down a fair amount
Our retrospectives are not that useful. In every single one we call out the same problems:
- the requirements are too vague
- there are too many meetings and we never get into a state of flow
- we are constantly sprinting and our technical debt is rising fast
- we have no test automation because of the pressure to deliver something quick.
We get shutdown immediately about the requirements everytime. The Scrum Master tells use to lose the waterfall mentality, we need to prioritize conversations over documentation. First of all, none of us have ever done waterfall in our careers. In this company we’d always worked on small projects in iterations. We always started with a functional spec for the project, but project times were rarely past 2 months and the dev/QA cycle just repeated in the iterations and the functional spec evolved during that period.
The functional analyst says he was too busy to document the requirements. But the problem is that we can’t always find the analyst to talk to him, so we find ourselves blocked often.
The underlying tone is that we developers just are not moving fast enough. When we complain about things in the retrospectives, if it involves any changes outside of the immediate dev team then it gets shutdown. We cannot change the Scrum practices and we are locked into a one week sprint.
We lose most of each Friday to preparation for the demo and the demo itself. Half of Monday is lost to sprint planning. Then Wednesday is half lost to backlog grooming. So we have Tuesdays and Thursdays as our productive days.
One time we did a two week sprint because some stakeholders were away, so our manager figured we’d skip demo day and just do a two week sprint. Our productivity was way higher as we had a run of 7 work days without interruptions. But management and Scrum Master do not allow us to continue the two week sprints. One week sprints are considered the holy grail. We can deliver value each week! But the problem is that we deliver very little each week, and it is not going to production. We are going to go to production with the MVP as a whole.
Our product owner was not right choice. He is our line manager and not an expert in anything related to the project or even software development. So he does not prioritise our technical user stories related to fixing our technical debt and adding automated testing to previous work.
We are the software development experts of the company and yet we have to contantly justify any effort on technical practices. Finally we win the right to prioitise a couple of technical stories per sprint. So we start to break even on technical debt but we are still left with a lot. We have an architecture of quick fixes.
The retrospectives end up with way more complaints than the ones I have already described and they usually get abandoned immediately after the meeting. Little is done to address our complaints. The retrospectives are useless and just make us feel powerless. We begin to realise that the Agile consultant is totally inept and doesn’t understand the tenets of Agile. He has the rules down pretty good though.
Months 3 to 6
It has been three months and we have completed about 5% of the project. We are in serious trouble.
Docker is going nowhere, we don’t even have an environment set up yet. It turns out that our operations team doesn’t even know we are going to use a new tech stack and Docker. Our software architect is keeping us and the ops team in the dark about it all. But we manage to schedule a meeting between Ops and our team. Now they get involved and are surprised we never told them about our plans for Docker. They refuse to give us any dates, even rough dates, about when/if they will be ready to support Docker in production. At this point more alarm bells are ringing.
The software architect isn’t worried. His approach is to go ahead anyway, even if it means we won’t have an ops team that will be able to support Docker in production.
So we have a big argument in which one side of the dev team wants to put the brakes on the Docker work and concentrate purely on delivering the business functionality required. The other half wants to continue with Docker. I am on the Docker yes but not yet side. We argue that we could start on classic infrastructure at first and in phase two we could containerize it. If we put the right abstractions in place, we would make the software agnostic to hosting.
The software architect is furious and it becomes heated. Management steps in and fires the software architect and another dev team member. We are told to concentrate on delivering working software, it is not a technology demonstrator project. It is a victory of sorts for us though we don’t feel good about it. The project has been a disaster so far.
Requirements still bad
After three months of complaining about the quality of the requirements, nothing has changed. The requirements are still consisting of a few lines of text for complex functionality. We are not talking about complex technical problems but intricate business rules. Under-estimation of the work involved remains unchanged.
Neither the Scrum Master nor management are willing to change the process. We are told to be faster. This is the constant message. Why are you so slow? Work faster!
After 5 months we have done 10% of the project. It is obvious that the original deadline of three months was way off. But management have this feeling that we are on the cusp of a ramping up in speed. There seems to be an expectation that soon things will start speeding up.
So far we have been drip fed the requirements. We have the diagram with the circles but do not know what each one entails. Architecture, design and strategic planning do not come into it. Each sprint we must deliver a small piece of the system and focus on that. We have asked for more detail on the bigger picture but the request is continually ignored. We work out that the reason for this is that nothing has been defined for those circles that have yet to be started. The Events Management Architect has an overall vision but is defining the requirements on the fly.
Luckily, higher management want an estimation. Not a new date but an estimation of the remaining work. So we all sit down together and review each circle and at a very high level work out what is involved in each one. We look at our current velocity, comparing to previous tasks that are probably similar to these future requirements.
In the end we calculate that it will take us an additional year to year and a half to finish. The Scrum Master is obviously uncomfortable with this and immediately starts saying that this is only a rough estimation and not really meaningful. Our line manager doesn’t say anything. In the end, our estimation is canned and a new deadline is created. The new deadline would make it a 10 month project overall, giving us an additional 5 months to complete the project.
The MVP Grows
Our MVP, already pretty large, keeps growing in scope. Not only is it more complicated than the Events Management Architect thought, but now we need extra functionality on top.
In a retrospective we say that the team is not big enough. We’re five people and it would be pretty easy to develop these circles in parallel. We recommend three teams of five. We know that adding more people to a project doesn’t always work (some of us have read The Mythical Man Month), but much of the areas of the system are pretty independent of each other and have not been started yet. Management agree and start hiring. Management at this point are pretty nervous. The company executives want monthly updates on the project now.
Developers Speak Out
In the team we have a couple of guys with deep knowledge of the events product. It is a pretty powerful and complicated product. They are the go-to guys related to the system and combine business domain knowledge as well as technical knowledge.
We get together as a dev team (no Scrum Master present) to discuss the bad state of the project and these two guys tell us their doubts about the value of the project. They think that the issues the company has experienced regarding the difficulty of adding new events and other config changes is not solved by this project. In fact these services we are building only seem to add complexity. We all agree. We never really understood the project, it was the brainchild of the Events Management Architect and we were told to follow his vision and just make it work. But we didn’t believe in it, we never did. It was about time we stood up and spoke up.
In the next retrospective we present this opinion. Our line manager (our product owner) seemed unphased. He says we should sit down with the Events Management Architect and talk about it. The trouble with that idea is that all that will happen is that we will get shouted at. Anytime we have any criticism of the design, or proposed functionality he tends to get frustrated and angry. But our manager says he will schedule a meeting.
We never have that meeting and the conversation is shoved under the carpet just like our estimate was.
In the end the developer with the closest relationship to the Events Management Architect speaks to him. It does not go well. The developer comes out of the discussion deeply troubled and makes his feelings known. Soon it is common knowledge of this run-in with the Events Management Architect. The developer is quite vocal about his feelings about the project and news spreads inside the development department. He asks to be moved to a different project but that is rejected and instead they just fire him.
He is our best developer, in both domain knowledge and technical skill. May be he should have been more professional and not complained so publically, but firing our best developer is a real loss.
The new devs get hired and soon we have three teams. But more reorganisation is soon required as the tech lead of one of those teams gets moved out because he also has a run-in with the Events Management Architect.
The product owner (our line manager) is pretty much vacant through out. He never challenges anyone except us developers. Again, a developer had spoken up about their doubts and opinions and they had gotten moved off the project. The message is clear and no-one speaks up again.
The next developer they lost was myself. I simply burned out. The project was stressful and the team and wider organisation dysfunctional to the extreme. One day I woke up and my body and brain just rejected the project. Everytime I tried to start my tasks I suffered paralysis and panic attacks. I took a couple of weeks off sick and ultimately left the project.
There are now about twenty devs on the project and they have reached the 10 month deadline date. They have reached the 30% done mark. I keep well away from the project now, I seriosuly don’t want to know the details of what is going on. It’s a death march project and it ain’t over. I am told though that the MVP keeps getting larger.
So what went wrong?
Our software architect screwed things up for us. He should have been the guy to knock some sense into the product owner and Events Management Architect, but he was in it only for getting real experience of Docker so he could speak at conferences.
Our Scrum Master screwed things up for us. He had a Scrum Master certificate and two years as a junior developer under his belt. Us developers had between 8 and 15 years experience each and a proven track record in the company, hence being hand picked for the project.
Yet he lauded over us with his superior knowledge of how to develop software. We were told by management to follow his lead. We were going Agile and the Scrum Master was the expert.
Our Scrum Master rammed the Scrum religion down our throats and yet failed to see the lack of agility. He also failed to see that our Product Owner was anything but that role.
Our product owner had zero knowledge of the events system, zero knowledge of software development and was just the manager closest to the project. Mostly he was an empty space who attended meetings but contributed nothing. When we needed him he was not there and backed up our painfully inept Scrum Master every time. He decided on prioritization. It was comical to watch him in the meetings prioritize the various stories because he truly had no understanding of any of it.
The true overlord of the project (our Events Management Architect) was a fantasist. He had a vision but it just didn’t stack up. He wasn’t a software architect, or even a developer in the past and yet he designed the entire system. But he had no control over the project scope, letting it get larger and larger over time. He wasn’t willing to budge an inch on anything. If we pushed back he got angry and moved developers out of the team or got them fired if they disagreed.
We developers screwed up too. We should have been more courageous and stepped up when our software architect consultant got fired. But we were already downtrodden by the process and all the stakeholders involved. We were relegated to the bottom tier of decision making and then kicked for not going fast enough. I got a real psychological hammering which ended up in burnout and a kind of professional mental crisis. I had gotten to such a low point that I didn’t even recognise myself. Finally my wife said enough is enough and made me request out of the project. She was right.
My Lessons Learned
So ultimately I think the lessons I have learned are that when your company has a visionary project that is identified as a strategic project for the company, certain things are needed:
- Executive level needs to communicate clearly to middle management that the project exists and their support is needed. The various managers and teams need to work together, in a common direction, with visibility of the various aspects of the project that affect them.
- The vision needs to be understood by all relevant parties and the value it brings must be clear. In our case it was a technical solution for a technical problem, but the technical experts were actively ignored and pushed out.
- Use a team with a proven record. Throwing a bunch of devs together to make a new team doesn’t always work. Once you have a team for the project, listen to and respect them. Often they are domain experts as well. Ignore them at your peril.
- Beware of Agile consultants. The barrier to entry is a three day course. If you have a team that already functions well and has delivered previous projects successfully, don’t throw in a process oriented towards control. Give them the freedom necessary to be self-organising professionals and give them the necessary tools to get the work done.
If you want to contribute an anonymous developer story then please me an email at firstname.lastname@example.org