Solving cross-team priorities with a Product Council - an experience report

Today, I’ll share an interesting approach we used at New Relic to solve executive alignment on product direction, and cross-team execution.

What is a Product Council?

A Product Council is a way to create a leadership group that coordinates the most critical product work for the company. It provides a product strategy, and a way to prioritize and protect critical projects. It can help solve the challenges of cross-team projects. And it can also be a way to get executives aligned on product strategy.

The Product Council is led by the head of product. They also involve other key people in the council: perhaps the CEO, some Product leaders, and the head of engineering.

Key responsibilities for the Council are to write and communicate a product strategy, and to prioritize and communicate the key initiatives in the company.

That seems straightforward, so why is this useful?

How does a Product Council help with cross-team projects?

The year prior to us forming a Product Council at New Relic, I was responsible for the engineering effort behind a new product. It was an ambitious project, requiring work from many teams in the organization.

Everyone knew that the project was important, because we talked about its importance continually. I was able to secure the commitments of five or six teams to do the work we needed to deliver this new analytics product.

But as any experienced project manager will tell you, we were building on sand. There were other important projects too. And emergencies that came up. And one by one, the teams who have given me iron-clad assurances started to waver. New priorities came in, and piece by piece, the plan fell apart.

I came to the conclusion that a project of that size was simply infeasible given the way we were organized.

After suffering for a couple years of failed cross-team projects, we finally hired a consultant, Jim Shore, who helped us dig our way out of the situation. An expert in high performing agile teams at scale, Jim created a set of solutions to help address our cross-team project problem.

I’ll describe how it worked in a second, but what it did for cross-team projects was transformative. Simply put, it gave us the ability to execute on large projects and to have confidence they would move forward. It also gave us assurance that smaller critical projects could be guaranteed protection. It was a huge improvement.

How does a Product Council help get executives aligned on product strategy?

Another reason to consider using a Product Council is to align your leaders on a product strategy. Nadya Duke Boone, Chief Product Officer at Huntress, told me she is setting up a Product Council there: “We’re setting up a Product Council and have just eight teams [New Relic had fifty when we established the Product Council]. The first problem we want to solve with the Product Council is making sure the right people are involved in decisions about what teams are working on at the ‘big rock’ level. We have three executives who have unique insights that I want to make sure are heard.” Nadya also invited the functional heads / directors of engineering, product, and design to participate.

Executive misalignment is incredibly expensive. A Product Council can help.

How does a Product Council Work?

“The biggest benefits of a Product Council are the cross-functional membership of the council and that it creates a single official system for the approval and prioritization of work. Without a Product Council or General Manager-ish, the only place where there is cross-functional conversation about priorities is on the CEOs staff. That might work fine for a tiny company, but very soon the distance from an individual worker up to the nearest resolution point becomes vast. The Product Council addresses this explicitly by saying ‘everyone needed for a decision is here’” – Nic Benders

The Product Council was led by the head of product. They also chose the other members. At New Relic, it was composed initially of three VPs of Product, the head of engineering, and the chief architect. But I could easily imagine a CEO or cofounder participating.

The output of the Product Council was a written product strategy, and an ordered list of Product Priorities. At New Relic, the product strategy was revised at least annually, and the Product Priorities were updated about quarterly.

The Product Priorities was a merged list of these two types of projects:

1. Programs: Important initiatives that required coordination across a lot of teams — programs rather than projects. For example, an RBAC program that needed almost every team to do work for it.
2. Critical projects: Projects that required protection from the rest of the organization, even if they didn’t require a lot of coordination. For example, the team working on an absolutely business critical feature, that didn’t need to participate in the RBAC program or anything else until it was done.

Participating in the Product Council is almost a full time role – it’s not just attending a meeting. Being on the Council was basically a full time role. Writing a product strategy, communicating that strategy, and synthesizing and coordinating the most important projects of a large organization is a lot of work. Be sure to set appropriate expectations!

The Product Council also served a governance function. We focused on delivering milestones instead of projects, and made our milestones small so they shouldn’t need to be interrupted. We further ruled that to interrupt a milestone in progress required the consent of the Product Council. Nadya Duke Boone: “The need to go to the Product Council to interrupt work in progress was an A+ feature of the Product Council – it really put meat into protecting a milestone and letting teams deliver before pivoting.”

Why a Product Council?

Companies with software teams tend to have centralized priorities. Or they have team-by-team priorities. Both can work fine, but will fail to scale as you get large enough. Eventually, you will need both. Why?

• Centralized prioritization eventually fails because the amount of things to prioritize increases with the number of teams. The central team has to maintain an understanding of an ever-increasing amount of things. And they lack the knowledge of the local teams, so will prioritize many things poorly.
• Decentralized prioritization will eventually fail because dependencies between teams can increase in complexity at an exponential rate. Without some form of coordination, all requests that cross teams have equal weight, and important projects will just be unable to move forward because there won’t be a shared understanding of what is important.

How the Product Council worked with local teams

A Product Council merges the advantages of centralized and decentralized prioritization. You get local teams using their local expertise to develop plans that make sense for their part of the organization. And you have a centralized coordinating function that provides overall business context and direction, and coordinates the success of the most critical projects.

The Product Council existed to ensure the big boulders could be protected. But the emphasis was on building strong local teams, with expertise in their area. We had product managers on each local team (and for teams that didn’t, we added a transitional role until we could hire a product manager for each team). These local teams thus also had their own roadmaps.

Product managers would take high-level direction from the Product Council’s product strategy. They’d develop their own local plans and have conversations with adjacent groups and the product council leadership to rationalize their plans. They could also pitch ideas to the Product Council and shape the overall direction of the company.

So the product strategy was influential because it was essentially a distillation of the market needs and product thinking of the leadership. This set the overall direction. And the local teams would craft their own strategies in response. But what was interesting was the interplay between the two.

The Product Priorities were also influential and helpful to local teams.

For product teams, they made our plans realistic. Prior to the Product Council, I could proceed with the illusion that my new product might possibly happen. After the Product Council existed, that illusion was dispelled, because everything had to be ranked. Product teams were forced to make realistic plans without excessive dependencies, unless they were near the top of the priority list. This is where the independent executor model comes from: make plans, but any dependency on another team needed to either be a Product Council priority, or it needed to be optional.

For platform teams, they found this prioritization useful as well. A platform team gets competing requests from all over the organization. If you are helping expand your API for several different teams, it’s hard to know which of them to prioritize. Prior to the Product Council, you’d guess or make your own assessment. Or you would be influenced by the most influential leader. But after the Product Council, you could reply to the #2 priority that the other team had to come first, because they were first on the list. This allowed them to give a clear “not able to do that for at least 3-6 months” answer, which allowed product teams to reconsider their plans with better information on what was actually feasible.

Product Council tips

The Product Council should be very restrained in what it prioritizes. In general, leaders will want to tackle everything. But doing this takes up all the spare capacity of engineering, and leaves the local teams with little autonomy or ability to use their local expertise in service of the company. Because the opportunity cost isn’t very visible to the Product Council leaders, they need to remind themselves of this all the time.

We found that there was a limited number of programs that could operate successfully at the same time. Programs tend to each spread their tendrils throughout the organization, because they are large and tend to involve a lot of work across teams. The top program always went well. As we got further down the list, our confidence it would succeed dropped. At some point there was a clear line – that represented our program capacity. We capped the number of program managers to that natural number, and tried not to exceed that number of programs we prioritized.

One danger I’ve seen of centralized prioritization is that the project that gets the “top priority” billing can use that designation to request unlimited resources from the rest of the product development group. It’s important to restrain things to the most minimal size possible, because you’re locking up the engineering organization. There should be some sort of scrutiny of the top priorities so they stay lean.

You may have a need to set some ground rules for reliability and security work. At New Relic, we had a rule that “do not repeat incidents” work trumped project priorities, even if deadlines would not be met. Just because something is the top project priority, doesn’t mean it’s more important than reliability or security matters.

Each program has a web of teams it hits within the organization. The top three priorities can often hit the same platform or infra team. We found that these hot spots were often fully dedicated to Product Council priorities. This meant they had no ability to do any other work. This can result in unanticipated tradeoffs. Be on the lookout for where the bottlenecks are, and always be fixing your bottlenecks. At the very least, talk with those teams about the impact it is having on them.

Also, you do need your leaders to play by the rules. Brent Miller pointed out: “One of the failure modes the Product Council had was there was an attitude from [some] leadership that they wanted all the things, so the reality on the ground was that there wasn’t one top priority, there would be five. For bottleneck teams that was just untenable; they would get in trouble for blocking somebody’s pet project no matter what they did.” This is a cultural challenge – you have to constantly reinforce good behavior and discourage people from pressuring teams for lower priority work.

Another important factor in a Product Council is that the conversations have to go both ways. Encourage people to critique the priorities or pitch other things to be the top priority. I saw some pretty dramatic improvements in the product strategy at New Relic when people successfully pitched new directions to the Product Council. That was one of its best features, but it was also one of the hardest to achieve:

“Making this work was really hard and it never worked particularly well from my view. Many people complained that the Product Council were the ‘secret masters’ who decided everything in secret. So even though the entire strategy would be made up of work proposed by various people in the org, we were always accused of making the whole thing up ourselves. Part of that is life, some is the communication strategy, but some is around a need for real transparency and openness in how you run it.” – Nic Benders.

Finally, over time a natural complement to the Product Council developed, which was an Execution Meeting for key projects. This was a large, tightly run meeting where we reviewed the key projects each week. We went over all the projects, talked through the red/yellow/green status for each, and called out risks and concerns. A typical meeting resulted in a couple of realizations of ways the projects were at odds with each other, or needed attention or assistance of various kinds.

When to use a Product Council

The main value of the Product Council is in creating a cross-functional, unified product direction. And the second value is in setting priorities that help resolve cross-team deadlock.

So I would consider using a Product Council when you have a group of executives or founders that are poorly aligned on the direction the product should go. Or when the company direction is poorly distributed throughout the organization. Another indication that it might be a good time to consider a Product Council is if you’re seeing cross-team project problems.

A Product Council can work in engineering organizations as small as eight teams, or as large as fifty. Nic Benders said this felt a bit like an upper bound of the size that the Product Council could be effective.

Scaling Product Councils

“Already with fifty-ish teams the Product Council felt overwhelming. We couldn’t manage fifty workstreams and milestones per month. So some way to scale it, either in a fractal nature or by changing altitude, is needed. We ended up moving to a GM [General Manager] model as we outgrew the single Product Council” – Nic Benders.

For changing altitude, what Nic is referring to is the granularity of the work the Product Council looks at. The size of the organization influences the altitude of the work the Product Council will be capable of evaluating. Originally, the plan was for the Product Council to focus on milestones instead of projects. But this quickly became infeasible.

If your organization gets larger, you either need to subdivide into multiple Product Councils, reduce the amount of things the Product Council considers, or change the scope of what the Product Council considers. Or you might want to consider other approaches.

New Relic was about 50 software engineering teams when we added the Product Council. It would have been helpful with many fewer teams. But I wouldn’t probably do it before cross-team coordination was a large problem.

Spend more time implementing strategy, less on writing strategy

Alex Kroman was on the Product Council for many years. When I asked him his thoughts on it, he shared these thoughts:

The Product Council focused too much on writing strategy and not enough on implementing strategy.

It’s relatively easy and risk free to write a good strategy doc. But it is difficult, and involves a lot of risk, to implement. Implementing strategy involves making difficult tradeoffs, and can be really unpopular and use a lot of political capital.

Decisions to support strategy almost always cross organizational lines or functional boundaries.

What I would do if I were doing it over again is identify a small number of cross-functional initiatives that will have the most impact on the business. Then communicate about them.

Then I would set up the Product Council to work with the leaders of those cross-functional initiatives, and help the organization make the right tradeoff decisions. These things won’t happen on their own because they cross boundaries. So you need to align leaders across boundaries on these priority tradeoffs.

I would judge the effectiveness of the Product Council based on the number of hard tradeoff decisions the leaders make and implement themselves, in support of the highest priority initiatives.

Too often, I see organizations give “directly responsible individual” type accountability for projects. You tell this leader they have our permission to work across teams and do this work and do all these things. What I typically see is they don’t give that person that authority. What needs to happen is the leaders need to actually make hard decisions based on that person’s recommendations.

Product council versus other coordination models

One alternative to a Product Council is a General manager approach (which is a type of Single Threaded Owner) or a Division structure. This subdivides the problem in smaller pieces, which can scale.

Jim Shore said: “Since the work at New Relic, I’ve been looking at ways to vertically scale (fewer bigger teams) rather than horizontally scale (lots of little teams). If I were to do it again today, I’d use FAST to create a handful of large teams, each responsible for a well-defined area, similar to how the General Manager model had people responsible for particular areas. FAST solves a lot of the problems I’ve seen with horizontal scaling: it’s more amenable to specialties like UX, more flexible to changing business goals, and—particularly valuable—allows for incremental change rather than requiring a big-bang transformation.”

You can also opt to make teams so independent that what teams offer is essentially self-service.

You could attempt to resolve this through some sort of currency, real or virtual.

You could also do this work directly through the Product Management hierarchy.

Coordination models

The Product Council is one of many coordination models. Coordination models give you a menu of choices to choose from when solving your leadership coordination issues.

Feedback

See anything I missed? Disagree with this? Please let me know your thoughts!

Thank you

Jim Shore introduced me (and all of New Relic) to the Product Council. It was part of a larger set of changes he introduced – the most successful organizational transformation I’ve seen. Jim reviewed this post and shared his thoughts on FAST as an alternative approach.

Alex Kroman left voice memos about the Product Council and product strategy that were far more coherent than I would have been able to make while driving. I edited them into the section above on implementing the product strategy of the Product Council.

Nic Benders was one of the key executives behind the agile transformation at New Relic. He worked with Jim Shore and myself to implement these changes and was the executive responsible for the work. He was on the Product Council for many years, so he brings a unique perspective to bear. Nic reviewed a draft of this article and I included a quote of his on the cross-functional and prioritization value the Product Council provides. He emphasized Brent’s point about the time commitment of being on the Product Council. Nic also contributed a quote about the power dynamics and perception issues of the Product Council, and the need for transparency.

Nadya Duke Boone shared her experience planning a Product Council at another company. She also pointed out some things that I had forgotten about, such as the Product Council being a point of escalation for interrupting in-flight milestones. Nadya pointed out the importance of “do not repeat incidents” and security priorities, and how having those ground rules in place help. She also reminded me of the project execution meetings, which emerged after the Product Council as a way to track execution on key projects.

Brent Miller pointed out some of the challenges local teams would have from leaders who didn’t necessarily pay attention to the Product Council priorities. He also mentioned that the time commitment for participation in the Product Council was something the leaders tended to underestimate. He also reminded me of some of the challenges around the size of work to focus on.

Thank you to Bessi on Pixabay for the cover image.