Agile Product Development: 2024 Expert Guide | TCGen

What is Agile Product Development?

Agile development for new products allows companies to develop products to respond rapidly to change. This involves self-organizing teams that create fast prototypes in collaboration with each other and customers.

These prototypes are then iterated through repeated cycles where customers interact with them and provide feedback incorporated into the product. Often, organizations start with software development before applying the Agile approach to other product development activities (such as hardware). If you want to learn more about TCGen’s Agile Consulting, please check out our approach.

Agile first came to prominence through the Agile Manifesto, a document published in 2001 and written collaboratively by seventeen software development leaders. The Agile Manifesto proposed twelve Agile principles that express the spirit of Agile development through the Agile methodology.

Subsequently, Agile has evolved into a set of frameworks and practices (Scrum, sprints & sprint planning, daily stand-up meetings, burndown charts, etc.) frequently used, especially by software development organizations. However, these specific practices are not essential to Agile. Preeminently, Agile is a principle that favors empowered teams that are free to innovate and create products that delight customers. It also implies a learning organization that adapts nimbly to changes in markets and technologies.  

We view Agile as a continuum, not just a black-and-white proposition. We also apply Agile methods to our approach to the Product Development Process, regardless of the product or service you’re offering.

Agile for New Product Development | Agile Methodology Essentials

To learn more about agile product development and how the basic agile processes are applied to product development teams – either in software, complex system projects involving hardware and software, or tangible products- read our updated 2023 guide that describes the essentials. It includes five tips for adopting agile for your product development teams.

Download Your Agile Product Development Guide Here.

What is Agile?

Boiled down to its essential characteristics, the Agile methodology happens in teams, and it is about adapting to new information rather than executing a pre-ordained plan. Ultimately, the benefits of Agile are better products, faster time to market, and greater customer satisfaction. The main characteristics of these fast-moving teams are:

• Customer involvement: A development team creates an evolving prototype that allows them to enter a dialog at early stages where they can receive repeated customer feedback.
• Iteration: A development team works fast, dividing larger tasks into smaller chunks and then performing cycles of building and testing to learn quickly from incomplete prototypes. Each process aims to create a deliverable/prototype that receives customer feedback.
• Learning: Each team and the organization collectively reflect on past projects and digest this information. Reflection enhances organizational learning, improves processes, and builds trust/creates better relationships with stakeholders both internally and externally like product management consultants.

What is Scrum for Product Design and Development?

Scrum is a framework often associated with Agile, although it is not synonymous with Agile. The main principles of Scrum are self-management, transparency, and iteration.

The Scrum approach divides projects into sprints. A sprint is a small, time-bound portion of a longer product development process. Sprints are often two weeks but can be as little as a week and as much as a month. A sprint focuses on achieving a pre-defined goal.

Whether the goal is to test a portion of a product, create a detailed interface design, or prove the concept of crucial functionality, each sprint must have a definite deliverable or prototype demonstration presented to some customer, external or internal. This deliverable must be specific, observable, and tangible. Kanban is an agile method but a more stripped-down process for managing unplanned work (work in progress, doing, done).

Planned in this manner, sprints are a forcing function that moves the product development process ahead rapidly, holding teams accountable for result metrics and doing so regularly. Sprint also provides early and rich feedback from those outside the team that dramatically increases the probability and predictability that the new product will satisfy, if not delight, customers.

Iteration in Scrum: Maximizing Product-Market Fit and UX

Iteration is a key to Scrum. Prototypes are put in front of customers early and often through an iterative approach. The team iterates and reworks and then puts them in customers’ hands again to optimize the product definition. Repeated iterations help secure a good fit between the product and market while improving the User Experience (UX).

Roles in the Scrum process

In the Scrum process, there are three leading roles:

• Product owner: serves as the link between customers and the development team.
• Scrum Master: responsible for driving the project, owning the process, and ensuring the team maintains Agile principles and practices.
• Team Member: an individual contributor on a cross-functional development team.

Scrum also directs teams to perform sprint retrospectives. At the end of each sprint, the team meets and reviews the project. This enables continuous improvement. It is a facet of the learning organization and correlates with success in the long term.

Why is Agile Good for Product Development?

The Agile methodology mindset has numerous benefits over traditional approaches for new product development:

• Enabling companies to respond rapidly to change
• Empowering teams to do their best work
• Cutting non-value-added time by limiting upper management meddling

Allowing the companies to react quickly to change

There is much to be said for having flexibility in product definition, especially given dynamic markets and fast-changing customer needs. Furthermore, it takes a long time to get all the requirements documented. Waiting for this to happen before the team starts work delays time to value.

Rapid response times start with having a dedicated Product Owner – and a Process for documenting and prioritizing product requirements. If hardware teams have these two elements, they can gain substantial benefits from an Agile Product Development Process (or internal IT process) without getting bogged down by the perception that you can’t create stories for systems or hardware. You can.

Empowering teams to do their best work with Agile product management

We’ve discovered that much of Agile’s success comes from the Scrum team. We have found that teams perform best when they hire good people, define the management leadership, and have leaders who can clear obstacles from their path.

• Upper management has empowered teams to plan and prioritize their work (backlog)
• Middle management changes their focus from micromanaging to removing roadblocks.
• Team members have both domain expertise and collaboration expertise
• Everyone puts product allegiance over functional allegiance

Improving Product Managers’ influence by limiting upper management meddling

Executives want to know in real time that teams are on track and that their investments in new product development will earn a profitable return. Agile development empowers teams to define projects that fit the company’s strategic intent, hone the product definition with care, and then collaborate with management to establish the critical parameters of the project in terms of features, cost, timing, quality, etc.

In-App development, upper management should leave the team alone if the team is performing to specifications. The project management team should only interfere if it looks like the project will not stay within its defined parameters. Then, it is the management’s job to help the team redefine the parameters in question and to remove roadblocks for the team, giving whatever resources or other help they need to get the project back on track.

Much more code gets shipped in the digital world because teams are left alone to interact with potential customers and write the code they care about. The same principle holds in the non-software world. Empowering cross-functional, self-organizing teams to do what they do best reduces non-value-added time and organizational churn. This puts the emphasis where it should be: on creating products that satisfy customers.

How Agile Product Development Process Influences Product Teams

When you apply agile to your process, you may wonder how that impacts the functioning of a product team or product management. Agile practices generally define three roles (Product Owner, Scrum Master, and the Team itself). Throughout the development phase, the product development teams function similarly to their function in a more typical waterfall methodology. 

Scrum Master is very similar to a project manager & the Product Owner is very similar to a product manager, and the other team members function as they typically do in waterfall development. The big difference is not in the product team, but in the practices these teams follow.

In agile product management, the product manager’s role is very different, focusing less on the product roadmap and more on the current release, creating stories for the agile development team. 

The agile environment also leverages tooling, such as Jira, for many details. This is especially true when developing high-quality software or focusing the development cycle on the broadest customer requirements. Jira is well suited for managing large numbers of conditions. Agile project management often goes hand in hand with Agile product development.

Improving Product Strategy: Product Vision & Agile Product Management

Although product strategy is often viewed as a yearly process, it does not have to be. Agile practices can be applied to strategic planning if management allows iteration as the market, competitors, and customers change. Although the product vision should be a steady guide during the development cycle, if there are drastic changes in the market, the idea for the product should evolve, just like it does in a startup where pivots are joined.

Product managers and those leading agile product development or software development should take advantage of the learning increments when the product concept is exposed to end users. This is one of the most important benefits of an Agile project.

Is Agile Software Development Methodology Generally Applicable?

By now, the business agility movement and protocols like SAFe have at last exploded the myth that Agile is for software product development only. Of the twelve principles in the Agile Manifesto defining the agile methodology, 75 percent are not specific to software. For example, precepts such as “Business people and…developers work together daily,” “Projects require motivated individuals, support & trust,” and “Face-to-face conversation is most efficient” are not unique to digital development. There’s no reason that these principles cannot operate in any product development environment.

When you’re developing a tangible product, as opposed to a purely digital one, an essential component manufactured overseas might be delayed, derailing the best-laid plans (aka your product roadmap), Scrum, or no Scrum, however, our experience with clients shows that successful firms use a few select practices from the Agile methodology, such as sprints and demos, to manage product development programs to market success.

Proprietary research we conducted on twenty large companies showed that using 1) sprints, 2) having more iterations, and 3) developing a local build capacity were used effectively by product developers across industries. We have found that applying Agile frameworks beyond the digital world had numerous benefits, including:

• Bringing product teams closer to customers
• Improving the team’s ability to adapt to market or technology shifts
• Reducing team burnout

What developers who operate outside the purely digital world want is agile with a small “a.” Small “a” agile means transparency. It means that the status of programs is easy to view and frequently tracked. It means crystal clear program definitions. It means measurable indications of progress, including the ability to demonstrate working software at the end of each sprint.

Agile with a small “a” consists of three things:

• Sprints or iterations of a fixed-length
• A demo or other artifact at the end of each sprint indicating the status of the project
• A transparent tracking mechanism for projects

Agile with a small “a” is practical, implementable, and scaled to your organization. It frees the product team to work on what’s most important to customers while providing the transparency, frequent updates, and clear demonstrations of project progress that Senior project managers need.

Can You Reconcile Agile and Waterfall?

Public companies are managed and held to yearly milestones because Wall Street works by measuring companies quarterly. Companies make commitments to investors and are rewarded for fulfilling them. The outward-facing part of your organization operates according to “waterfall” assumptions and related cycles of planning.

We have helped teams with a phased process, with milestones and gate reviews, create a hybrid approach that embeds sprints within a more structured “waterfall” product development process.

Some of these teams had lead times of as long as 18 months. The diagram below shows they cut this long timeframe into sprints and nested them within their legacy process. The company found that embedding sprints within a larger structure speeded decision-making. Dividing a longer process into small, achievable steps, with deliverables on a fixed schedule, preserved the team’s focus and kept the project on the boil, compressing timelines.

How did they do it? Through release planning. Release plans examine an individual project and its dependencies over approximately nine months. Release plans span many sprints to set the point of release. Milestones drive the release plan, while sprints are the means to reach the next milestone. Release Plans treat each milestone as a release point. They form a natural and strategic link between agile and waterfall.

The release plan manages the following:

• Critical dependencies between projects
• The internal milestones, nesting sprints to achieve them
• Launching to customers

One of our clients found that releasing planning to nest agile within a waterfall process eliminated status meetings. Since demos and deliverables are defined for each sprint, the status of projects remains evident throughout the process. Managers left them alone as long as the team performed and delivered as planned in each sprint. Only when the team appears to be veering off course will they inform management and trigger a new agreement around a baseline for the sprint.

Nesting sprints within a more extensive process restores the proper management role – enabling the team to succeed.

One culture change challenge remains for managers to adapt to a world where they must empower teams and give up some control. Management has its structures of control, while agile teams need the freedom to be creative. A hybrid process that takes advantage of the planning advantages of the waterfall approach, with agile’s speed, creativity, and energy, reconciles these two needs. It’s not a question of Waterfall vs. Agile. It can be both.

Can You Make Inflexible Requirements, Like a Regulatory Requirement, Agile?

Most companies must follow some business processes that follow inflexible requirements with prescribed deliverables tied to specific dates. Some of these processes are required by law. For example, many companies must comply with financial reporting regulations or consumer privacy laws. These processes have dependencies, milestones, and techniques necessary for business.

However, even inflexible requirements from parties outside your organization, such as auditing and regulatory compliance processes, can benefit from applying Agile principles. Organizations that are “agile with a small ‘a'” can convene self-organizing, cross-functional teams with the freedom and skills to solve problems across many domains, whether Legal, regulatory, or HR.

Apply the Agile product development process concept of a minimum viable product (MVP). Think beyond the product development life cycle to the needs of other cycles within your company, be they regulatory or audit requirements. These requirements may also have an MVP, which may take the form of data submission to an auditing body or a final report. Since such conditions are likely to have dates attached, include the date in the MVP.

Also, look at the processes you would use to meet regulatory or audit requirements. Break these more significant activities into their components, and you will uncover the sub-process that drives each more practical activity. Reduce these sub-processes into requirements and treat them like user stories in Agile development. Then, cross-functional teams can be used to get the job done, operating like Agile teams: meeting frequently, having sprints, and perhaps even early “demos” of a work product. You will find that you can control an Agile development project.

5 Tips for Successfully Implementing the Methodology for New Agile Product Development

1. Implement a few basics like sprints, release plans, or self-organizing teams.

Too many zealous practitioners think Agile is an all-or-nothing deal. Companies tailor their processes using Agile frameworks as they see fit. That’s a no-brainer, given the vast differences in industries, cultures, and products. You’re not constrained to use what doesn’t work for your organization. Start with a few practices, such as sprints or daily stand-up meetings, and build from there. For programs just beginning, you should also create a product backlog.

2. Consider an agile implementation as a change management project with classic pitfalls.

Go back to change management basics. Know that you are embarking on a long process that is likely to have some breakdowns and breakthroughs. Think about implementing and evangelizing Agile so your organization sees its benefits and wants to embrace its principles.

3. Understand where Agile challenges your culture (including product management).

Expect skepticism and a certain amount of resistance. Agile challenges are the functional and managerial authority. Letting go of control can be challenging. Agile is a culture change, so be mindful of the culture and the barriers to it. Also, the roles of the Product Owner differ slightly from those of a product manager – make sure you bring them on board.

4. Ensure your initial Agile teams are well-staffed with proven product owners (product managers), scrum masters, and team leads.

If you’re using the classic Scrum framework, having proven, experienced people in the key roles is essential. Agile needs strong teamwork, which means hiring and maintaining the right talent, Training, recruiting internally, and engaging with these roles in mind.

5. Scrum Masters need Agile mastery; Invest in training them.

Scrum Masters own the process. They are the Agile gatekeepers. Invest in people by investing in Agile training. Continuous development of talent is a hidden principle of Agile. And don’t forget to train the Senior Management.