Agile Product Development: 2023 Expert Guide | TCGen

What is Agile Product Development?

Agile development for new products allows companies to develop products to respond rapidly to change. This involves self-organizing teams that create fast prototypes in collaboration with each other and with customers.

These prototypes are then iterated through repeated cycles where customers interact with them and provide feedback that is then incorporated into the product. Often, organizations start with software development before applying the Agile approach to other product development activities (such as hardware). If you would like to learn more about TCGen’s Agile Consulting, please check out our approach.

Agile first came to prominence through the Agile Manifesto, a document published in 2001 and written collaboratively by seventeen software development leaders. The Agile Manifesto proposed twelve Agile principles that express the spirit of Agile development through the Agile methodology.

Subsequently, Agile has evolved into a set of frameworks and practices (Scrum, sprints & sprint planning, daily stand-up meetings, burndown charts, etc.) that are frequently used, especially by software development organizations using Agile. But these specific practices are not essential to Agile. Preeminently, Agile is a set of principles that favor empowered teams that are granted the freedom to innovate and create products that delight customers. It also implies a learning organization that adapts nimbly to changes in markets and technologies.  

We view Agile as a continuum, not just a black-and-white proposition. We also apply Agile methods to our approach to the Product Development Process, regardless of the product or service you’re offering.

Agile for New Product Development | Agile Methodology Essentials

If you must find out more about agile product development and how the basic agile processes are applied to product development teams – either in software, complex system projects involving both hardware and software, or tangible products, read our updated 2023 guide that describes the essentials. It includes five tips for adopting agile for your product development teams.

Download Your Agile Product Development Guide Here.

What is Agile?

Boiled down to its essential characteristics, the Agile methodology happens in teams, and it is about adapting to new information rather than executing a pre-ordained plan. Ultimately the benefits of Agile are better products, faster time to market, and greater customer satisfaction. The main characteristics of these fast-moving teams are:

• Customer involvement: A development team creates an evolving prototype that allows them to enter a dialog at early stages in the process where they can receive repeated customer feedback.
• Iteration: A development team works fast, dividing larger tasks into smaller chunks and then performing cycles of building and testing to learn quickly from incomplete prototypes. The goal of each cycle is to create a deliverable/prototype that receives customer feedback.
• Learning: Each team, and the organization at large, reflects on past projects and digests this information. Reflection enhances organizational learning, improves processes, and builds trust/creates better relationships with stakeholders.

What is Scrum for Product Design and Development?

Scrum is a framework often associated with Agile, although it is not synonymous with Agile. The main principles of Scrum are self-management, transparency, and iteration.

The Scrum approach divides projects into sprints. A sprint is nothing more than a small, time-bound portion of a longer product development process. Sprints are most often two weeks but can be as little as a week and as much as a month. A sprint focuses on achieving a pre-defined goal.

Whether the goal is to test a portion of a product, create a detailed interface design, or prove the concept of key functionality, each sprint must have a definite deliverable or prototype demonstration presented to some customer, external or internal. This deliverable must be specific, observable, and tangible. Kanban is an agile method, but it is a more stripped-down process for managing unplanned work (work in, doing, done).

Planned in this manner, sprints are a forcing function that moves the product development process ahead in a rapid fashion, holding teams accountable for result metrics and doing so on a regular basis. Sprints also provide early and rich feedback from those outside the team that dramatically increases the probability and predictability that the new product will satisfy, if not delight, customers.

Iteration in Scrum: Maximizing Product-Market Fit and UX

Iteration is a key to Scrum. Through an iterative approach, prototypes are put in front of customers early and often. The team iterates and reworks and then puts them in the hands of customers again to optimize their product definition. Repeated iterations help to secure a good fit between product and market while also improving the User Experience (UX).

Roles in the Scrum process

In the Scrum process, there are three main roles:

• Product owner: serves as the link between customers and the development team.
• Scrum Master: responsible for driving the project, owning the process, and ensuring that the team maintains Agile principles and practices.
• Team Member: an individual contributor on a cross-functional development team.

Scrum also directs teams to perform sprint retrospectives. At the end of each sprint, the team meets and reviews the project. This enables continuous improvement. It is a facet of the learning organization and correlates with success in the long term.

Why is Agile Good for Product Development?

The Agile methodology mindset has numerous benefits over traditional approaches for new product development:

• Enabling companies to respond rapidly to change
• Empowering teams to do their best work
• Cutting non-value-added time by limiting upper management meddling

Enabling companies to respond rapidly to change

There is a lot to be said for having flexibility in product definition, especially given dynamic markets and fast-changing customer needs. Furthermore, it takes a long time to get all the requirements documented. Waiting for this to happen before the team starts work delays time to value.

Rapid response times start with having a dedicated Product Owner – and a Process for documenting and prioritizing product requirements. If hardware teams have these two elements, they can gain some substantial benefits from an Agile Product Development Process (or internal IT process) without getting bogged down by the perception that you can’t create stories for systems or hardware. In fact, you can.

Empowering teams to do their best work with Agile product management

We’ve discovered that much of Agile’s success comes from the Scrum team itself. We have found that teams perform best when they hire good people, define the leadership system, and have leaders who can clear obstacles from their path.

• Upper management has empowered teams to plan and prioritize their work (backlog)
• Middle management changes their focus from micromanaging to removing roadblocks
• Team members have both domain expertise and collaboration expertise
• Everyone puts product allegiance over functional allegiance

Improving Product Managers’ influence by limiting upper management meddling

Executives want to know that teams are on track and that their investments in new product development will earn a profitable return. Agile development empowers teams to define projects that fit the company’s strategic intent, hone the product definition with care, and then collaborate with management to establish the key parameters of the project in terms of features, cost, timing, quality, etc.

In App development, if the team is performing to specification, then upper management should leave the team alone. The project management team should only interfere if it looks as though the project will not stay within its defined parameters. Then, it is Management’s job to help the team redefine the parameters in question and to remove roadblocks for the team, giving whatever resources or other help they need to get the project back on track.

In the digital world, this means that much more code that gets written gets shipped because teams are left alone to interact with potential customers and write the code that they will care about. The same principle holds in the non-software world. Empowering cross-functional, self-organizing teams to do what they do best reduces non-value-added time and organizational churn. This puts the emphasis where it should be: on creating products that satisfy customers.

How Agile Product Development Process Influences Product Teams

When you apply agile to your process, you may wonder how that impacts the functioning of a product team or product management. In general, agile practices define three roles (Product Owner, Scrum Master, and Team itself). All throughout the development phase, the product development teams function similarly to their function in a more typical waterfall methodology. 

Scrum Master is very similar to a project manager & the Product Owner is very similar to a product manager, and well the other team members function as they typically do in waterfall development. The big difference is not in the product team itself but rather in the practices these teams follow.

While the product manager’s role is the most different, agile product management basically focuses less on the product roadmap but rather on the current release, focussing on creating stories for the agile development team. 

The agile environment also leverages tooling, such as Jira, for many of the details. This is especially true when developing high-quality software or when really focussing the development cycle on the widest range of customer requirements – because Jira is really well suited for managing large numbers of requirements. Agile product management often goes hand in hand with agile product development.

Improving Product Strategy: Product Vision & Agile Product Management

Although product strategy is often viewed as a yearly process, it does not have to be. Agile practices can be applied to strategic planning if there is a willingness by management to allow iteration as the market, competitors, and customers change. Although the product vision should be a steady guide during the development cycle, if there are drastic changes in the market, the vision for the product should evolve, just like it does in a startup where pivots are a common occurrence.

Product managers and those leading agile product development or software development should take advantage of the increments of learning when the product concept is exposed to end users. This is one of the most important benefits of an Agile project, although there might be a better fit with a product management consulting engagement.

Is Agile Software Development Methodology Generally Applicable?

By now, the business agility movement and protocols like SAFe, have at last exploded the myth that Agile is for software product development only. Of the twelve principles contained in the Agile Manifesto that defines the agile methodology, 75 percent are not specific to software. For example, such precepts as “Business people and…developers work together daily”; “Projects require motivated individuals, support & trust,” and “Face-to-face conversation is most efficient” are not unique to digital development. There’s no reason that these principles cannot operate in any product development environment.

When you’re developing a tangible product, as opposed to a purely digital one, an essential component manufactured overseas might be delayed, which can derail the best-laid plans (aka your product roadmap), Scrum, or no Scrum. However, our experience with clients shows that successful firms are using a very small number of select practices from the Agile methodology, such as sprints and demos, to manage product development programs to marketplace success.

Proprietary research we conducted on twenty large companies showed that using 1) sprints, 2) having more iterations, and 3) developing a local build capacity were used effectively by product developers across industries. We have found that applying Agile frameworks beyond the digital world had numerous benefits, including:

• Bringing product teams closer to customers
• Improving the team’s ability to adapt to market or technology shifts
• Reducing team burnout

What developers who operate outside of the purely digital world want is agile with a small “a.” Small “a” agile means transparency. It means that the status of programs is easy to view and frequently tracked. It means crystal clear program definitions. And it means measurable indications of progress, including the ability to demonstrate working software at the end of each sprint.

Agile with a small “a” consists of three things:

• Sprints or iterations of a fixed-length
• A demo or other artifact at the end of each sprint indicating the status of the project
• A transparent tracking mechanism for projects

Agile with a small “a” is practical, implementable, and scaled to your organization. It frees the product team to work on what’s most important to customers while providing the transparency, frequent updates, and clear demonstrations of project progress that Senior project managers need.

Can You Reconcile Agile vs. Waterfall?

Public companies are managed and held to yearly milestones because Wall Street manages by measuring companies quarterly. Companies make commitments to investors and are rewarded for fulfilling them. The outward-facing part of your organization operates according to “waterfall” assumptions and related cycles of planning.

We have helped teams with a phased process, with milestones and gate reviews, create a hybrid approach that embeds sprints within a more structured “waterfall” product development process.

Some of these teams had lead times of as long as 18 months. They cut this long timeframe into sprints and then nested them within their legacy process, as shown in the diagram below. The company found that embedding sprints within a larger structure speeded decision-making. Dividing a longer process into small, achievable steps, with deliverables on a fixed schedule, preserved the team’s focus and kept the project on the boil, compressing timelines.

How did they do it? Through release planning. Release plans look at an individual project and its dependencies over an approximately nine-month period. Release plans span many sprints in order to set the point of release. Milestones drive the release plan, while sprints are the means to reach the next milestone. Release Plans treat each milestone as a release point. They form a natural and strategic link between agile and waterfall.

The release plan manages the following:

• Critical dependencies between projects
• The internal milestones, nesting sprints to achieve them
• Launching to customers

One of our clients found that nesting agile within a waterfall process using release planning eliminated status meetings. Since demos and deliverables are defined for each sprint, the status of projects remained clear throughout the process. As long as the team performed and delivered as planned in each sprint, managers left them alone. Only when the team appears to be veering off course will they inform management and trigger a new agreement around a baseline for the sprint.

Nesting sprints within a larger process restores the proper role of management – enabling the team to be successful.

One culture change challenge that remains is for managers to adapt to a world where they must empower teams and give up some control. Management has its structures of control, while agile teams need the freedom to be creative. A hybrid process that takes advantage of the planning advantages of the waterfall approach, with the speed, creativity, and energy of agile, reconciles these two needs.

Can You Make Inflexible Requirements, Like a Regulatory Requirement, Agile?

Most companies must follow some business processes that follow inflexible requirements with prescribed deliverables that are tied to specific dates. Some of these processes are required by law. For example, many companies must comply with financial reporting regulations or consumer privacy laws. These processes have dependencies, milestones, and processes that are necessary for doing business.

But even inflexible requirements stemming from parties outside of your organization, such as auditing and regulatory compliance processes, can benefit from applying Agile principles. Organizations that are “agile with a small ‘a’” can convene self-organizing, cross-functional teams with the freedom and skills to solve problems across many domains, be it Legal, regulatory, or HR.

Apply the Agile product development process concept of a minimum viable product (MVP). Think beyond the product development life cycle to the needs of other cycles within your company, be they regulatory or audit requirements. These requirements may also have an MVP, which may take the form of a submission of data to an auditing body or a final report. Since such requirements are likely to have dates attached, include the date in the MVP.

Also, look at the processes you would use to meet regulatory or audit requirements. Break these larger activities up into their components, and you will uncover the sub-process that drives each of these larger activities. Reduce these sub-processes into requirements and treat them as you would user stories in Agile development. Then use cross-functional teams to get the job done, operating in a manner similar to Agile teams: meeting frequently, having sprints, and perhaps even early “demos” of a work product. You will find that you can control an agile development project.

5 Tips for Successfully Implementing the Methodology for New Agile Product Development

1. Implement a few basics like sprints, release plans, or self-organizing teams.

Too many zealous practitioners think Agile is an all-or-nothing deal. In fact, companies tailor their own processes using Agile frameworks as they see fit. Given the vast differences in industries, cultures, and products, that’s a no-brainer. You’re not constrained to use what doesn’t work for your organization. Start with a few practices, such as sprints or daily stand-up meetings, and build from there. For programs just beginning, you should also create a product backlog.

2. Think about an agile implementation as a change management project with classic change management pitfalls.

Go back to change management basics. Know that you are embarking on a long process likely to have some breakdowns and breakthroughs along the way. Think about how to implement and then evangelize Agile so that your organization sees its benefits and wants to embrace its principles.

3. Understand where there are challenges to Agile in your culture (including product management).

Expect skepticism and a certain amount of resistance. Agile challenges are functional and managerial authority. Letting go of control can be challenging. Agile is a culture change so be mindful of the culture and what the barriers are likely to be within it. Also, the roles of the Product Owner differ slightly from the roles of a product manager – make sure you bring them on board.

4. Make sure your initial agile teams are well-staffed with proven product owners (product managers), scrum masters, and team leads.

If you’re using the classic Scrum framework, then having proven, experienced people in the key roles is essential. Agile needs strong teamwork, and that means hiring and maintaining the right talent. Train, recruit internally, and hire with these roles in mind.

5. Scrum Masters need to have Agile mastery; Invest in training them.

Scrum Masters own the process. They are the Agile gatekeepers. Invest in people by investing in Agile training. Continuous development of talent is a hidden principle of Agile. And don’t forget to train the Senior Management.