In December of 2011 I documented the days of an Agile Timebox at Examiner.com. The idea was to illustrate how three distinct phases overlap in a given cycle. You have: Priorities and Overall Company Initiatives and Strategies, Functional Requirements, User Stories, and Artifacts, Technical Architecture and Feature Development. I hope that these posts provide insight into the ever evolving hybrid Agile methodology used.
An Introduction to the Next 15 Work Days
I've been writing a lot about process lately. Examiner.com is one of the largest Drupal sites in the wild. It was migrated from Cold Fusion to Drupal - a process that began about two years ago.
The Examiner development experience has moved through a wide variety of styles, tools, and methodologies over the last two years. Through blog posts and presentations, a fair bit of information is being shared about how we operate. When I was in Austin presenting, a fairly significant number of folks asked specific questions about our "Generic Timebox". I wanted to answer all the questions, but ran out of time that afternoon.
For this timebox we are using a compressed time-line. Our normal interval is 20 days - in this case we are shaving off a week and making it 15 days because of the holidays. Still, all the elements that are in our normal period are reflected in this series of days making it pretty much perfect to (hopefully) answer the questions I was being asked.
I'm going to write a blog post each day covering what we did and how we went about it. I'll be keeping specifics and names out of the picture - but it should give the sense of what a development cycle looks like with the Examiner team. Today was the first day of a new timebox - #5 - so I'm going to start my posts today with this one. If you have questions, just post them!
Day 1 - Story Review/Technical Architecture
Day 1 is a little misleading. By the time we have made it to Day 1 of a timebox, user stories have been written, comps designed, wireframes completed. The requirements are largely done. In fact, at Day -2 (day 14 of this example), the time box finishes being locked down for the next cycle. However, this is Day 1 of the actual software development.
The two planning days are half days - focused in the mornings to allow for overlap between our European and North American teams - because we found in earlier time boxes that a full day was exhausting. In fact, one of our developers approached me after today's planning phase and said, "The difference between this planning day and the first one I was in on is amazing - so much easier." By shifting our initial review of user stories to the previous time box, we're able engage in the technical planning rather quickly during our planning days.
So what did we do today?
- Reviewed stories for half our features in this time box
- Discussed technical solutions for the tougher features
- Discussed the weight and performance implications of third party Java Script
- Contacted a third party regarding a potential contract
- Broke out into a few smaller groups to discuss individual features
After our planning, some folks started working on user stories that were well understood. Others continued to review stories for tomorrow's planning meeting. I spent part of the day reading a contract and giving feedback on that contract. We also continued planning for time boxes that are 40 and 60+ days out.
Finally, we continued to address bugs that emerged, from the previous release, in our production environment. We are near the end of the period where we tightly align production and staging in an "environment lockdown".
Tomorrow we'll continue our planning - and I'll write my second post in this series.
Day 2 - More Planning
Day 2 of this shortened timebox was the second of our Technical Planning days. As I mentioned in my previous post, we have two half days of Technical Architecture/Planning at the beginning of the time box. We continued to review details on the user stories for this box and also tightened our estimates a little bit for remaining stories. We look at whether we need to write code from scratch, we can make use of community code, or enhance community code from the Drupal project.
We colour code our stories - white for not scheduled, yellow for scheduled and committed to, greeen for QA ready, salmon when we need more information, and grey for stories that we defer for future timeboxes. We re-reviewed stories from yesterday for which we needed clarifications. These visual queues make it really easy to scan the Google spreadsheet(s) and see the status of a given story.
Our story spreadsheet header describes the information we're looking to capture for hand off to the development team.
We capture a story ID - and will do point increments when we have sub-stories related to a given feature. Each story must be associated with a ticket and includes the description of the story, questions from various team members, and points to illustrate the level of effort. One point is a half day in our world.
After The Technical Planning
The three project managers then finished breaking up the projects between them. We try to all take part in the planning days so we're in the loop on all the projects. Projects are typically broken up into development groups with an attempt to have as distinct a team as possible with little overlap. This is sometimes difficult as all code is reviewed by another developer and another themer before it even has a chance of making it out for QA.
We also ended up having our retrospective from the previous timebox today. Typically that would be scheduled for day 20 in the timebox, but constraints delayed this meeting. We tweaked our process in a few places. For example, much of our discussion is done in IRC - we decided that for our releases we would add voice to the mix for the end of this timebox. We use logged IRC to keep our decisions and conversations logged, but sometimes you need a more rapid way of getting another person's attention.
We're Starting the Next Timebox?
Finally, Day 3 of the timebox is the point when priorities for the next timebox are shared by the Executive Committee. Today was a day where a representative from the product team, from the project management team, and from the release team met to put final "t-shirt" LOEs on the prioritized feature backlog. The list of desired priorities will be delivered tomorrow and user stories, wireframes, and comps for the next timebox will begin to be generated. In a very real way Day 2 represents the first day of the next timebox.
Tomorrow will be the first day of our code sprint which will last for 7 work days. The team will also start on user stories based on the priorities set by the executive committee.
In the first two posts in this series I wrote about the planning days at Examiner.com and how those planning days set up the beginning of the code sprint. I also discussing the ancillary activities that set us up for the next timebox - locking down the priorities for the next time box.
Today, Day 3, the coding portion of the sprint begins.
The scrums started out by reviewing the salmon stories (salmon are stories that still need more information to be actionable) in the Google spreadsheet. Each story, as clarified was turned yellow - indicating the team in committing to completing the effort to complete that work. After these stories have been reviewed and clarified, the different groups are tasked with different tasks.
The morning began with three scrums. Each scrum represents a practicing unit that includes:
- A Project Manager
- A QA Person
- A Product Manager
- Code Reviewer
- Theme Reviewer
- Release Manager
There is some overlap between the scrum teams - but we try and keep the units as separated as possible.
Developers and Themers
coding coding coding
The QA team works on building test cases to support the next period of the timebox. They take part in meetings, as needed, in the development of use cases and review of artifacts for the next timebox.
The reviewers start to be fed features to engage in code review on. If code passes - it moves to the next development step. This could be theming or it could be Manual QA. If it is ready for QA it is pushed to staging for functional testing.
pushing pushing pushing
Product managers are responsible for delivering the prioritized list for the next timebox today (Day 3) and begin to work through writing user stories for the next timebox. They are also responsible for answering questions about user stories in the current timebox. Any discovery for two timeboxes out and beyond is happening as well.
Project Managers act as defense during this period of time. They are responsible for clearing blockers and preventing the development team from being distracted by tasks outside of directly programming new features, enhancements, and attacking the defect backlog. During this time the Project Managers are also working with the Product Team to develop user stories for the next timebox. The Project Managers are also socializing updates, considering requests, and trouble shooting problems during this period in the spirit of keeping the development team from being distracted.
The day also included reviewing processes that were identified as needing adjustment during the previous retrospective. I, personally, worked on setting up meetings with various parties to support future development tasks including a potential upgrade to Open Atrium. We use Open Atrium as our Intranet for Examiners (our writers). We reviewed our defect backlog to start scheduling the highest priority bugs into the current timebox. People in the company make use of this time to discuss potential approaches for new features. Discussions with the Project Management team direct new features towards contributed modules as much as possible. Plans may start for code reviews on contributed modules that might be used in the next time box.
The take away is this - the developers, themers, testers, and reviewers are all focused on what we are developing now. The focus of the product team is starting to move strongly to the next time box. Project management is straddling these two different sets of activities while herding the collective cats.
Photo Credit: cc511 on Flickr
Day 4 of a Drupal Timebox
Yesterday was the first day that the Examiner.com development team was really coding. That became conversations in IRC and the beginning of features being sent to code and theme reviewers. The spreadsheet that we keep our user stories in is starting to move yellow stories to green and the salmon stories are being turned yellow (white is uncommitted stories, grey is deferred, yellow are committed to being developed, salmon need more information, and green are ready for manual QA). This is the time when the developers really get their heads down and are working hard to complete the stories they have allotted in the time period available and work on the bug backlog.
The morning began with our regular scrums - three teams, three scrums, three sets of projects. This post doesn't need to rehash our morning scrums.
The day was peppered with meetings including a review of current Infrastructure needs and ideation of new Ad targeting techniques with a Product manager. It included working through planning a business trip to work with another company on an Examiner project. We're also still working through retrospective improvements from the previous timebox.
The Backlog Mambo
We embrace the idea that the current work we're developing should be predictable. The current timebox development should be pretty much set in stone. There are times with opportunities or threats are so great that we need to suffer the challenges of context switching. This should be rare. Anything after current development should be negotiable. That is what a large chunk of today was all about.
Yesterday, the Product team brought to the Project Managers a prioritized backlog for the next timebox. This list was a first pass based on Executive Committee directives. The Executive Committee convened today with the Product Team, and representatives from Business Intelligence and Development. The top items in the list were discussed and adjusted. Scope was shifted and priorities were moved. There were requests to fatten out some additional initiatives. The next two weeks will have the Product team focused on developing user stories in conjunction with development and Creative working with themers on wireframes and comps.
The dance continues.
The last few days have pretty much been nose down to the grindstone coding like crazy for the development team. Days 5-9 were pretty much the same with a few exceptions.
On this day the Epics and High Level User Stories were delivered and these helped define user stories that needed design artifacts. The project team started sifting through the stories to identify any pitfalls or problems. As the Technical Product Architect, I start to think really hard about approaches that we might be able take when we get down to the Drupal part of the implementation. Are there any contributed modules that might fit the bill? Sitting down with various Product Managers we began detailing how the product vision would technically be handled.
Largely the user stories were delivered. As a team we really started doing a great job of reviewing individual stories. Certain members of the development team had their features code complete which opened them up to help review the stories that were being delivered. We started looking at wireframes.
During these two days, the Executive Committee were helping continue form priorities for the next Timebox.
Today was a transition day. If you look at the first post in this series, you'll see that the days turned from blue to yellow today. This represents a handoff of the environment to the QA department to start reviewing code that has been completed and pushed to the staging environment. This transition is also the first day that the development team throws themselves largely into helping review stories and wireframes. It is early enough in testing that there are few defects being sent back for rework.
First draft comps for the next timebox are also due. In our case, wou can watch the box.net directory synch as comp after comp are dropped making them available for the team to review. I've noticed that the transition day tends to be heavy on meetings. A sampling for example, today, had:
- Three Scrums
- One Scrum of Scrums
- Part 1 of a Config Dry Run for a new set of features
- Ideation for a new project that will lead to User Stories
- A ticket system transition meeting
- A reports meeting
The next three timebox days will finalise our user stories for the next time box with both the development and product groups. We will continue testing and reworks.
We Were Busy
Today was an extraordinarily busy day in the timebox. We're in the midst of attending to revealing defects in our current release, some fairly significant configuration work to support America Inspired, reviewing tons of user stories, writing user stories, and continued work on setting up JIRA with Greenhopper to replace our ticketing system.
The day started out, as usual, with three scrums that cover the work our three distinct teams are working through. The scrums at this point are less about the developers telling us what they've worked on, what they are working on, and what blockers they have and more about the QA team reporting on results that have come through testing. We do look at our story board spreadsheet to confirm that stories are either green or nearly green. We also make hard decisions on features that might not be ready for release in the time box. Those are noted and we socialise that news with stakeholders as appropriate. We're also looking at any stories that might be at risk and identify alternate plans for those stories. That might comprise a) deferring the story for completion in the next timebox or b) completing the work during the QA week or c) releasing the new feature partially. Finally, the project management team got together for a scrum of scrums where we identify any problem areas that might have emerged.
Stories, Stories, and More Stories
This day quickly moved into several hours of team leads reviewing user stories with a great deal of detail. On or around the Backlog Mambo, the project team is giving a rough pointed estimate to the stories as they are completed. This is to have a general sense of the needed burn rate to compete the next timebox's work. It gives the Executive team a little more information to be able to prioritise the work. During the QA week, the development team is challenging these assumptions and either confirming or disputing the time frames. We're generally pretty good about our estimates. It is common for priorities to shuffle a bit during this period of time as well. Scope on one of our initiatives increased while we read through the stories and realised there were a fair number of cases that hadn't be identified and drawn up. There was, also, new functionality being discussed. All of this is fine as long as we have our stories nailed down along with the accompanying artifacts by the time we lock the timebox down. This sprint, it will happen on day 15 although during our normal sprint it occurs on day 19. We reviewed about half of the current user stories and pointed all of them. Looking at the needed burn rate, the list needs to go back to our Executive team for prioritization. User story review will continue on Tuesday and Wednesday.
Ideation, Performance, and Configuration! OH MY!
The day also had us looking at solutions to improve overall performance on the site. These conversations have ranged from reducing the amount of third party Java Script on the site to more agressive caching to creating a Boost-like flat HTML methodology. Stacey and I started the next timebox's planning for who will do what - the general breakdown of what our teams will look like this time round. One each, of our product managers, designers, QA member, developers, and I started new configurations in our staging environment to script out a process on the live site for support of the America Inspired initiative. There are many blocks to be configured, placed, and visibility settings to be tweaked. Articles will need to be grouped into the five areas and labeled correctly. In short, we're drawing up a plan for the next two phases of that program.
Finally, we are engaged in defining how our instance of JIRA will behave once we have finishing configuring it. This is requiring our reviewing each and every kind of task we engage in and defining workflows for each. It is a significant effort.
That rounds out Day 11 of our Agile timebox on this very large Drupal site.
The next few days are like a timer counting down. Sometimes the conversations feel a bit like a NASA control room. We review and re-review our plans because this deployment isn't going to be a small one. It is likely in the top 3 in scope since I began over two years ago.
Day 12 - Testing
We started with Scrum, Scrum, Scrum, and Scrum of Scrums.
Testing continued on Day 12 - and all the while we continued to work through Product Manager user stories. Supporting artifacts were shared as well. This segued into the Project Managers reviewing points and starting to assign teams for the next timebox. For this cycle, we found that our story points were roughly double to the number of the points available for the next timebox. This requires a feedback loop with the Executive committee to prioritize completed set of user stories.
The Executive Technical Leadership Committee met where we discussed a variety of issues, successes, and follow ups from the prior week. This committee fulfills the role of a CTO.
Finally, I also reviewed a patent application.
Day 13 - Testing, Stories, and Horse Trading
We continued to work through user stories and test. We had a 90 minute meeting with the Executive Committee to discuss the priorities where some items were shifted lower on the priority list and, for intents, got knocked out of the upcoming timebox and into the next.
Testing is coming to an end at this point. Day 14 will be release day and so only the most critical items are prioritised to be addressed before we release. This means several status meetings throughout the day. In this timebox, we were still engaged with status meetings until later in the evening. This was partially due to a timebox that was 25% smaller than our normal 20 day period. It was also in part because we launched our new mobile experience transitioning from a third party, Verve, to an entirely Drupal based mobile site. This shift greatly enhances the user's experience on a mobile device by leveraging block logic from the desktop site and providing access to all Examiner.com content. The Verve experience was limited to the previous 30 days.
User stories and the supporting artifacts are being heavily examined at this point. We have to lock down our next timebox on Day 14 - and that is now less that a day away. We also anticipated that the morning release would be long based on the new features that have been made ready during the current timebox and previous ones. The good news is that the work of creating stories is largely done - there are a few holes that need to be filled - but there is also bad news. Even with the prioritization of scope, the points that will be needed for Timebox 6 exceed the points we have to expend. This will mean and tertiary review prioritisation prior to the planning days 1 and 2 for timebox 6. Fortunately the holidays have afforded us a week in which the team will be largely working on defects giving enough breathing room before the next timebox starting to do that evaluation.
Work went late into the evening. There was a 6 am MT release start time.
Day 14 - Deployment Day
We start at 6 am MT - a time which is a compromise between low traffic and a time when our entire distributed team can be available. Most of us work on deployment from our homes rather than in the office. We have several communication lines available during our deployments.
- A logged "Live" IRC channel
- A logged "Post Deployment Validation" IRC channel
- This timebox we added a Voice line using Skype
We want everything logged during deployment. If we need to go back and see what someone was doing at any given time, the logged channels help in those forensics. Anything that is spoken in Skype is to be logged as well. The goal here is to allow for extremely rapid communication in an emergency that doesn't rely on the typing speed of any one person. One of the lessons learned in a previous timebox was just that - fingers, in a pinch, cannot trump the speed of someone saying out loud, "STOP".
Normally deployments, from start of code push to end of post deployment validation, take 2-3 hours. We were launching a new mobile site making this deployment a longer process. It was about 10 hours total if you include hot fixes that occurred after validation to mitigate defects that manifested after push. After the code push is complete and post deployment validation is done, team leaders huddle and discuss which issues that have arisen need to be addressed immediately. A rapid triage occurs were we identify what order the needed changes will be pushed. They are then fixed, pushed to staging, validated, pushed to production, and validated one at a time.
After deployment is complete, it is time to do the final lockdown of stories for the next timebox. Folks have one last opportunity to lobby for switching priorities. Stacey is busy preparing our demo plan for the next day and informing folks who will be presenting what order and for how long.
Day 15 - Demo and Retrospective
Demo Day - when we arrive we will start to prepare the Examiner lounge - it has a large screen that we hook laptops up to, a video camera so our virtual team can peek in and take part. We set up a Skype call and a video feed for folks to access. The demo lasts for about an hour with each Product Manager going over the work we've completed during the timebox. Generally several of the Development Team will present something as well. This time around quite a bit of time was spent on the new mobile experience and America Inspired which will, on the 9th of January, will boast a brand new flexible polling module with different states for roles and voting status.
The retrospective was held as a round table. It is designed to be a safe place where we can honestly discuss what worked and what didn't work in the timebox. It allows us to make adjustments to our process. We have changed things like timing for artifacts, length of our testing period, adding a environment synch and freeze between prod and staging for a few days after deployment, and adding voice to our deployments.
Rinse and Repeat
The process repeats itself. Currently we are moving into development for our 6th timebox, user stories and artifacts for timebox 7, and ideation for timeboxes 8 through infinity. We track our burn rates pretty closely and are able do a good job of estimating the time our stories will take. Every timebox we complete improves our team's efficiency. We learn new things about how to do our jobs better every 20 days. I am incredibly proud of our team and how our process has grown.
Image Credit from Flicker. "Stopwatch" by William Warby
sed under a Creative Commons License.