By Adrian Bridgett | 2021-02-21
Introduction
Scrum sprints are a technique of software development. All the pieces of work form a “backlog”. The team plans and estimates how much work they’ll produce for the next “sprint” (typically two weeks) and places the most important tickets from the backlog into the sprint until they it’s full.
Each day, everyone attends a “standup” meeting stating what they did yesterday, what they will do today and if anything is blocking them.
At the end of the two weeks a review takes place (maybe also a demo). A retrospective covers what went well, what didn’t go well, suggested improvements/changes.
This process is then repeated for the next sprint until the project is finished (or forever).
Welcome to the cult
For the last fifteen years almost every team I’ve worked in or worked with has claimed to be an “agile development team” as they do sprints, standups, retrospectives. However often the reality has often been a sorrier tale.
I won’t claim to be an agile practitioner or scrum master. Perhaps that’s part of my point - few people are. Our methods are often adopted by hearsay rather than because they are helpful. Some problems arise from doing scrum wrong, some because scrum is the wrong thing to do.
The following isn’t strict scrum, it’s what I’ve seen teams do.
The theory (as I understand it)
A large project is defined as (typically multiple) epics which in turn are broken down into stories. Each story is assigned a number of points to say how much effort it will take. These points are typically decided in “planning poker” sessions where simultaneously everyone declares how many points the story needs. Stories may need discussing in order to reach concensus.
As a team works through sprint after sprint, the number of points they are capable of completing within the sprint becomes more accurate. The first sprint they may attempt 200 points, completing 150. The next they may try 150 and find they only complete 140. The next they may try 140 and run out of work (and pull more tickets in).
Note: I’ll refer to stories/tickets/issues interchangeably in this article.
The good parts
As in a retrospective, let’s cover the good stuff first. It’s the shortest list here - often the case in a retrospective as well!
The amount of work “in flight” is limited to the sprint length. People are less disracted by what needs to happen in a month’s time - they are focused on the next two weeks.
During planning poker, the difficulty of a task can spotlight hidden knowledge. A task deemed simple by one team member may be rated “difficult” by another who has better knowledge that there’s a complicated migration required. Conversely a difficult task maybe much simpler due to an existing unreleased tool.
There is a sense of trying to “finish” a task by the end of a sprint. A sprint should have a “goal” and these can also focus energy on delivering a target.
Points
How do you plan your work? I’ll lay good money that you use time. It’ll take you 2 hours to paint the room, 40 minutes to cook dinner, 2 days to write that blog article…
However these times vary wildly between people and here is where points come into play. A team with senior developers will complete more points per sprint than one with junior developers. Or in other words we might say that writing the blog article is 5 points. Which for me take 2 days, for a professional writer just half a day.
No story is allowed to be too “large”, and in fact the points are often restricted to low Fibonacci numbers (1, 2, 3, 5, 8, 13).
I have many problem with this approach. Firstly how many points can a team accomplish per sprint? Tracking the whole team does not work as the team varies with holidays, people availability. So we do need to determine how many points each person can complete.
Secondly, people aren’t replacable cogs. We each have different skill sets. A new task will take much longer than a task we’ve performed dozens of times before. Scrum’s genesis is in software development where programming ability is key. For DevOps and SRE teams though, there is a such a range of technologies that this single-axis assumption fails. Prior experience with a technology (which may not correleate with overall experience) becomes a principle metric.
In most planning sessions, it’s known who will be working on a task. Anna may fix Cassandra as she’s the guru, Bob might setup DNS so that he can learn a new skill. The amount of time it will take them and the effective “points value” of the ticket needs to change to reflect this. Scrum’s use of a pool of tasks with defined points just doesn’t match reality.
How about points themselves? Revisiting my first point - everyone I know thinks in terms of time. I’ve seen too many planning poker sessions where someone will say “it’s 3 days work so that’s 8 points”. Let’s just cut out the middle man here and say that points should reflect time rather than pretending it doesn’t.
Estimations
We need to have an idea of when a project will finish. We need estimates. There’s no way we can plan exactly who delivers a story for the next few months and allocate hours per-person. I’m only suggesting we do this when tasks are assigned (i.e. near term).
For long term planning, it’s competely acceptable to say that points are a rough approximation for an average person in the team. However, I think we are fooling ourselves to claim that we can do this accurately. We’ll omit tasks, work will hit hurdles, people will be absent (holidays, illness, new jobs).
My aim here is to remove what I see as needless complexity, bring things back to the natural way of thinking. Fibonacci numbers is for me a classic example of the cargo cult - why can’t you say a ticket is about 10 points rather than 8 or 13? One good reason please…
Spreadsheet reality
This is the term I use when (generally project managers) tot up the story points and set delivery dates based upon it.
A good project manager tries to ensure the system reflects reality. As one such person said to a development team - “I want you to tell me how it is, not what you hope it will be or think I want to hear”. Poor project managers try and insist that reality reflects their spreadsheet. They will be dissapointed.
Being “data driven” is an admirable concept and works well in many areas, including project management. We shouldn’t ignore “gut feel” though - how long a project will take can often be more accurately measured at the big scale than trying to break it down and then aggregate.
We should plan, try to ensure we’ve thought of the tasks, critical paths, blockers. There’s a sensible level to drill down to. Expecting to break down a project and get all the steps right at the outset is unrealistic, causes unneccssary paperwork and clutters up planning. Be realistic.
Standup
Engineers want to work in long, uninterrupted periods. Paul Graham’s “Maker’s schedule, manager’s schedule” is a classic (linked in References at the end of this article). Cal Newport’s “Deep work” book is another reference point. Most employees have flexible starting times - or work in different timezones so the start of one person’s day is part way through anothers. Disturbing everyone on the team each day is a high price to pay - it should be worth it.
“What I did yesterday”. Now’s your time to justify and/or defend yourself. Explain how you’ve fixed this script, wrote that program, helped so-and-so, spent 14 hours debugging a tricky problem (let me tell you all the details and how clever I am). Is any of that useful to the team? It may be genuinely helpful or interesting, if not “I worked on building the cluster” is all you need to say.
“What I intend to do today”. This is the most useful part, ensuring everyone is in sync with priorities (or crisies). Again, it’s easy to get sidetracked here or go into too much detail. Defer those to the end of the standup for interested parties.
“I’m blocked”. Why wait until the standup to ask for help? There’s probably a slack channel, post it there (perhaps say if it’s blocking something critical or if it’s less important). Asynchronous communication is a good thing and this also improves knowledge sharing.
As I write this during the COVID-19 pandemic, standups provide another purpose - keeping us together as a team. It’s easy to feel isolated, working on a problem by yourself with little support. Ask for help, or offer it if a colleague is struggling - even if just to bounce ideas around.
In general, I belive that such a rigid daily ritual is bordering on micro-management for most people. I prefer every few days, just to keep people in sync. Perhaps people could post briefly on slack anything of interest - a low-touch standup, just keep the updates brief.
I have one counterpoint where daily meetings are useful - fast moving projects/crisies. With a subset of people, they keep focus on the critical path elements and novel workarounds for blockers.
Rollover day
The last day of the sprint is often rather painful. In theory tickets complete at a constant rate throughout a sprint. I’ve rarely seen this in practice. Instead at the end of the sprint there’s always a rush to mark tickets as done and this has several bad impacts:
- Tickets marked “done” when they are 95% complete
- Tickets marked “done” when they are 95% complete and another ticket raised for the remaining 5%
- Tickets marked “done” by rushing the release (e.g. on a Friday evening)
Friday releases are bad because people should not have their weekends disturbed. Few places have adequate automated rollbacks. Top tip: always promise something for the start of the week, not the end of the week. This way you avoid Friday releases (and in the worst case you have the weekend to work on it). For most people, there’s no difference between 6pm on Friday and 9am on Monday.
Rollover day is also “meeting day”. Firstly a review of the tickets (sometimes combined into the planning meeting), followed by the retrospective and then planning.
Prepare for the planning meeting. You should have a clear idea of the top priority projects and low-hanging fruit. Respect the retrospective - the team’s concerns and ideas are important and you may want to include several straight away. Be flexible - don’t jam-pack the sprint so there’s no wiggle room left.
Planning meetings should be interactive. Write tickets up beforehand. Speak to the stakeholders and try and get a rough plan so that the team is reviewing the plan and signing it off. Use the teams time wisely - are there differing opinions about how long a task will take? It’s worth highlighting one good attribute of planning poker - it helps everyone have a voice (though it can be tempting to “low ball” points to avoid looking as if you are “slow”).
Kanban - sprints for non-developers
Instead of scrum sprints, I recommend teams use kanban instead.
In kanban, items still move through a pipeline from “backlog” through “in progress” to “completed” (some teams add more steps such as “verify”). Instead of a “sprint” or two-week period, it’s a continuous process. The main concept of kanban is to limit “work in progress”. In other words, don’t juggle too many tasks at once.
In sprints people should work on a single task at a time. Whilst in software development this is often possible, in my field it’s common to be juggling several tasks (one maybe blocked on another team, one is your main task, another is an urgent problem). The concept of the sprint deadline can unfortunately push people to work on many tickets as possible simultaneously. This reduces efficiency due to context switching and also dramatically increases the risks of mistakes and ommissions.
With fewer tasks in progress, each task is completed faster, bringing a sense of accomplishment and avoiding long running tasks. The business benefits as accomodating change is easier - it’s quicker to “clear the plates” of existing work (as less is “in progress”). This is not an excuse to avoid planning and foresight, it just lowers the damage caused by changes in plans.
With no sprint “deadline” Kanban is often thought of as a “less rigid” sprint. This is good for mental health and stress. It removes the “end-of-sprint” ticket problems - which also removes paperwork (no 5% tickets) and improves the accuracy of the ticket system (no need to split the ticket up).
We do now need some way to limit the field of view - as otherwise instead of a small set of sprint tickets we now have the entire backlog. Worse, those tickets have not been vetted by the planning-poker process!
One approach is to have two backlogs - one “near” and one “far”. Instead of sprint planning, have weekly meetings covering the same approximate areas. Whether new tickets go on the “near” or “far” backlog depends upon the team. In either case it’s good to make a single person each day responsible for triage.
Planning tips
To plan well, you need to keep the backlog managable. It’s a lot easier to do the right work when you have 20 tickets rather than 200 or 2000.
Group common tickets - with a tag, under a “forever” epic or just by standardising the title: “[BUILD] …“.
Remove duplicates. Eliminate repeats.
Close tickets waiting for “verification” if they havn’t responded within a sensible timeframe. Or perhaps eliminate the step entirely and set the expectation that they should reopen if there’s a problem.
Be realistic. The lowest priority tickets may never get done. You may still want to create them to say that they had been considered. Then close them - they are a distraction (I mark them as “deferred” and check them twice a year). Or put them on a roadmap.
Few is better than many. Is breaking a ticket into ten smaller tickets really helping you to deliver? Or is it hard to see the forest for the trees? Often it’s pure accounting overhead - or worse, best effort guesses. Remember the main aim here is to determine the future - to plan and estimate completion.
Prune the backlog. Take a bit of time in a team meeting to use some of these techniques to clean and reduce the backlog.
Tag. If your management software doesn’t support multiple backlog queues, you could tag tickets “up next” as an alternative.
Remember - even if you have 2000 tickets, you shouldn’t have to check them all every planning session. There will be some approach that allows you to bucket them into “near”, “soon”, “later”, “much later” at a suitable granularity for your situation.
Kannot Kanban
Perhaps you cannot work with Kanban, maybe your tools or organisation forbids you. Now what?
Firstly, minimise overheads - ignore that rollover buerocracy penalty if you can. If a release is due, just complete the ticket on the Monday in the next sprint and don’t worry about the mess it makes of the numbers.
If an urgent problem comes up, put it in the sprint (technically a no-no) and work on it - it’s the right thing to do. Hopefully people will see that it’s the results not the numbers that matter. Remember Goodhart’s law - “When a measure becomes a target, it ceases to be a good measure”.
Remain agile - perhaps have a mid-sprint correction where you re-prioritise (due to new ticket or new information) and re-assess (if a key story is going badly, add resource, inform the customer). The system is supposed to be there to help you - if it gets in your way then do want you must to keep it out of the way.
Summary
My belief is that for DevOps and SRE teams, a Kanban approach is a much better fit than sprints. It copes better with the high rate of important interrupts and eliminates the rollover penalties of sprints. The focus is on minimising work-in-progress - concentrating on a few high priority tasks, avoiding the death-by-multitasking that deadline based sprints encourage.
Good project management is essential, this needs to come from psycologically safe, open and realistic assessments of the work needed and progress rate. It needs revisiting on a weekly/fortnightly basis and the number one way that manager can assist is not through threats of broken promises but through servant leadership.
Most importantly - think for yourself, change your processes to see what works (or not). Even if you try a change and it doesn’t work, you may develop a new appreciate for the parts of your process that does work well.
Update: I recommend reading Shape up as well, which covers Basecamp’s process (which uses six week cycles).