A Step-by-step Guide to Feature Rollout | VWO (original) (raw)

What is feature rollout?

Feature rollout is a process of strategically and gradually introducing a new software feature in the market. To ensure a good user experience and less process fatigue, a feature is rolled out in a controlled environment before rolling out to the environment where real or all users exist and use the feature.

Not so long ago, product managers used to roll out a new software feature once every week or two with a number of changes bundled together and then monitored at scale. They believed that their developed features or updates would work effectively in the existing software environment and across all domains. However, most of the time, the lack of market due diligence led to major glitches and malfunctions in the software, thereby hampering the end user’s overall product experience.

Such irregularities also led to necessitating a full roll-back leaving the product managers and engineers to investigate the issue and conduct the entire rollout again.

Over the years, forward-thinking software companies, especially Facebook, Twitter, Samsung, Uber, etc. have learned from their past mistakes and have now established new ways to introduce new features or upgrades to their target audience. They’ve understood that agility is essential for innovation. The more beta tested (especially A/B tested) their developed feature is, the less will be the chances of hasty roll-backs and hotfixes.

Electrons, protons, and neutrons of the feature rollout atom

Over the years and across organizations, there have been massive process changes when it comes to feature delivery. Software companies including Facebook, have come up with several new and faster ways to deliver codes with better quality to enable both product managers and marketing folks to launch and promote their new additions swiftly.

Many of these efforts also revolve around relatively new rollout techniques such as continuous integration, continuous delivery, continuous agile development, and test-drive developments.

Of these, continuous integration (CI), continuous delivery (CD), and continuous agile development (CAD) hold maximum importance. Consider them as vectors that have the same direction and goal – to make new feature development and release processes faster, safer, and more robust. However, their magnitudes are quite different. They’re not mutually exclusive, but include each other.

Let’s understand the role of each of these vectors in detail:

Continuous integration – the process protons

As the name suggests, continuous integration (CI) refers to a development process wherein developers can integrate new codes into the system as frequently as possible. Here, developer changes get validated instantly through an automated build. This build also looks for errors and helps rectify them way before the release date of the new feature.

Continuous integration lays much emphasis on automated testing which helps ensure things do not break whenever new commits are integrated into the release branch. You can also call continuous integration “the process protons”. Without continuous integration, the entire feature rollout process would crumble down.

Benefits of continuous integration:

• Less errors go into production as regressions get identified at an early stage, thanks to automated build and rigorous testing.
• It’s easy to build a release since all integration issues get resolved early.
• The system helps alert product managers and developers as soon as there’s a break in the build. This helps them fix the issues instantly, before they move to another task.
• Ideal CI servers can run hundreds of tests in a matter of minutes. This means drastically low testing cost.
• The QA team spends less time testing new integrations and gets to focus on more significant improvements to maintain a culture of quality.

Continuous delivery – the process electrons

Consider this as an extension of continuous integration since it automatically deploys all code changes into testing and production cycle after the code gets a green card in the build stage.

This means that besides automated testing, you also have an automated feature release process which helps deploy new applications anytime you want. Continuous delivery ideally has the same features as an electron and hence, we call it “the process electrons.”

In theoretical terms, continuous delivery gives you the advantage of deciding the release date – daily, weekly, fortnightly, or a time that suits your business needs. However, many experts suggest that the best way to truly benefit from continuous delivery is to deploy your newly developed codes into production as soon as possible to ensure you release in small batches and troubleshoot problems, if any, quickly and easily.

Benefits of continuous delivery (CD):

• CD takes away the complexity of deploying a new feature into the market. Teams can deploy a change as per their own timelines.
• CD enables more frequent feature releases, thus accelerating the customer feedback loop.
• It reduces the pressure of making decisions for smaller changes, thus encouraging faster iterations.

You can read more about continuous delivery in Jez Humble and David Farley’s book, Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation.

Continuous agile deployment – the process neutrons

This is considered as the third and final step in the new feature rollout process. You can also gauge its features similar to an atom’s neutrons.

With continuous agile deployment, every change that passes through all the other stages of production pipelines gets released in the market for customer use. There’s practically no human involvement and only a failed test can prevent the new changes from getting deployed to production.

Continuous agile deployment is also an excellent way to accelerate the customer feedback loop and help product managers create a robust feature.

Benefits of continuous agile deployment (CAD):

• CAD ensures that teams are able to develop and deploy new features at a faster rate.
• It creates a deployment pipeline that triggers automatic release of a new change in the market at regular intervals.
• It’s easier and less risky to fix any problems that surface the deployment process as changes are released in small batches.
• Customers see a continuous flow of improvements, which increases the quality of the product every day and builds a loyal customer base for the company.

Feature rollout process

Any and every activity that you do involves a process, be it preparing a meal, giving an exam, writing an essay, or washing clothes. A process basically lays down a series of necessary steps to follow which ensure you achieve your end result. And, feature rollout is no exception.

A good feature rollout process enables you to create and deploy product features that serve its actual purpose. It also gives you the ability to control what you’re making. For instance, adding or developing only those software elements that are mentioned in your blueprint document. Sticking to a process helps create new features quickly and with less hassle.

So, what’s an ideal feature rollout process? Let’s find out!

1. Design

Draft and design a new functionality you plan to launch in the market. Examine all the related use cases and seek answers to important questions like what the feature will do, who will use it, and how they will use it. Lastly, define a timeline.

2. Develop a strategy

Create a good release strategy using the above-drafted functionality blueprint. It helps set essential feature release parameters and a plan to incorporate end-user feedback.

3. Build

Develop the feature and manage its progress through multiple development cycles.

4. Test

Run ample tests to gauge the effectiveness of the newly developed feature using feature flags. This helps control rollouts, roll-backs, user targeting, and gets performance feedback from users.

5. Release

Once your feature passes all the functionality tests, push it off for final production and market release.

6. Gather feedback

Create a constant user feedback loop as it helps product managers to improve their designed products and their strategies.

7. Release continuously

Constantly make incremental changes to your product based on customer feedback and continue optimizing.

Once you release the product in the market, constantly monitor the feature and its functionality. This will enable you to make incremental improvements and optimizations based on customer feedback.

How to launch a feature?

Launching a new product update is often just the beginning. It’s basically the phase where you make a claim that, “Yes, we’re adding a new feature to our existing product.” The next important step is to test and rollout your feature in a manner that reduces the chances of a misfire.

How you launch your new integration in the market ideally depends on many essential factors:

• The maturity of your build
• Your existing customer base
• The type of feature you’re planning to launch
• The goal or metric you want to achieve with the rollout

While there’s no specific recipe to launching your update in the market, it’s more about relying on your experience and product sense. Still, we’re laying down three common types of rollouts that we believe aid to the success of your hard work.

1. Limited rollout with A/B testing

One of the biggest advantages of choosing the limited rollout process is that it gives product managers the benefit to control the size and types of users they want to rollout their product to. Before running a limited rollout, set a quantitative goal that you want to achieve that becomes your benchmark for larger release.

Four common variations of limited rollout are as follows:

Alpha release — Also known as employee only release or dogfooding, the concept of alpha release was first formulated by Microsoft. Since the company develops features for a vast customer base, it needs to do its due diligence at an extensive scale.

Alpha release is a low risk way to test new features. You release the feature to internal audiences or employees only and gather their feedback. Such feedback is also considered the most rigorous one as the product’s functionality is scrutinized to all possible extent.

Beta rollout — Many product managers also call it restricted release. Here, product managers decide to rollout the feature to a restricted user base. This restricted group could be, for instance, paid users only, users from a specific geographic location, demography like people aging between 25 and 50, specific platforms such as iOS only, and so on.

The primary purpose behind rolling out an update in a restricted manner is that you get to test every aspect of your update.

For instance, a company decides to add a new personalized feature to its software. Instead of rolling out the update at large, it selects a section of users from each of its customer pools. It includes a section of:

• Loyal customers
• Enterprise grade customers
• Customers from one geography, say the USA
• Customers using one specific OS, say Windows
• Customers belonging to one particular industry, say pharmaceutical

When you rollout the update to this specific group of customers, you’re in a better position to gauge the performance of your product based on the feedback you get. You can easily rectify your errors without rolling back the feature and make amendments when you plan to launch the same update to another restricted group.

Invite-only release — Here, the product team invites a section of users to try out their new update. This section often involves only the most interested users. Just like beta or restricted rollout, this is another great way to understand the merits and pitfalls of your developed update.

Another best part about invite-only release is that your users won’t judge you as a company if your product fails to interest them. They’d rather help you improve and come out stronger in the market when the time is right. They serve as pillars to your success.

A/B test — This is usually used to examine product improvements with the objective to optimize a metric, say increase in-app engagement. You can test many versions of your feature with many specific changes against the control (the originally developed feature) and compare how the change affected the metric.

2. Gradual rollout

Gradual rollout is quite similar to limited rollout. The only difference is that in case of the latter, you don’t select a specific group of users, rather a percentage of them, say 10%, 20%, 50%, etc. For instance, selecting 10% of your loyal users or 20% of the users from a specific geographic location, and so on.

Usually gradual rollout is used by companies that

• Have a relatively small customer base
• Are releasing an update that doesn’t require rigorous user testing

Rolling out through this method reduces rollback risks, server crashes due to a large number of requests, etc.

3. Full rollout

Such a rollout is manyatimes required when a company decides to launch a new feature. This could be because of a major product launch or product campaign running against it, or when a feature requires constant customer feedback to make the product more robust in nature. For instance, addition of new social features or those which require network efforts.

Note, even if you plan to go for a full rollout, it’s always advised to leave 1 to 5 percent of your users on the control or original experience. This helps you to monitor the long-term impact of your release and helps to troubleshoot problems that surface from the very roots.

Feature rollout – metrics and variables

Every new feature launched in the market has its own list of metrics to map and KPIs to achieve. While some may be launched with an aim to increase engagement on the platform, many others may be released to get more conversions.

However, some metrics and KPIs are applicable to nearly all product launches. Some of these are as follows:

Launch campaign metrics

You can’t expect a new feature to be successful in the market without the help of a smart marketing strategy. Marketing metrics help you to effectively inform your customers and those eyeing your product about your new developments.

When launching a new software feature, include these common marketing metrics to your list:

• Lead generation: The first metric to your product launch should be to increase the interest in your product, further get more leads onboard. These leads could be free trials, demo requests, content downloads, or anything that matches the nature of your new release.
• Promotional metrics: Most often, channels like email marketing, social media marketing, and paid advertising become your primary lead generation sources. So, measure your email open rate, click through rates, social outreach, and the effectiveness of your main product’s image in the market. In case of paid advertising check the cost and click through rate to understand how successful your efforts will be.
• Website traffic/page views: Ensure to measure your launch content’s views as well. This typically includes overall website traffic, page views, new product pages, landing pages, and related content pages.

Product adoption metrics

Product adoption metrics basically map the success or failure of your launch and marketing efforts and whether you’ve achieved revenue and other related business goals. Here are some product adoption metrics to add to your list:

• Feature trials: If customers are starting feature trials, this can be a great metric to analyze their interest in your launch. Such customers can also become your target audience that ends up in your conversion funnel.
• Customer usage and retention: Track goals around customers who continue to use your new release beyond the trial periods. This is a good way to evaluate the real interest of people in what you’ve launched and its success in the long run.

Market impact metrics

Measuring the penetration of your new release in the market and impact on sales is a key indicator of the success of your launch. Key marketing metrics to include are as follows:

• Revenue gains: No matter if your new update is just to enhance the performance of your parent product, you should still map the revenue gains. Manyatimes, people tend to buy your parent product looking at the updates you add to it to make it more robust and user friendly. Revenue shows the true picture of whether or not you’ve delivered value with your new feature rollout.
• Market share: It’s quite possible that many of your competitors already offer the new feature you’ve just rolled out in the market. The difference lies in market share. The KPI tells you how well your update is performing in the market as compared to your top competitors.
• Competitive win rate: This metric will help define your success or failure over your competitors in the market. If your parent product competes head-to-head with existing solutions from competitors, your new launch should add to your success rate.

Qualitative feedback metrics

Besides the above-mentioned quantitative key metrics, it’s equally important to have quantitative metrics in your arsenal. Get feedback from your internal as well as external audiences.

• Internal feedback: Internal audience includes your sales reps, marketing folks, executives, product managers, and any other organization folks who can represent that target audience. Get them to review (or scrutinize) your product aggressively. Be sure to achieve their desired outcomes before you prepare for external launch.
• External feedback: Collect feedback from your existing customers as well as prospects to understand the feasibility of your product. Based on the feedback, discover areas of improvements and make amendments.

Admittedly, launching a software update and choosing the right metrics to analyze its success can be challenging, considering nearly 75% of such launches fail to lure customer interest at the very initial stage. However, setting clear goals and tracking metrics for every feature you rollout is essential. Because ultimately it is these metrics that tell whether your efforts have paid off or gone in vain.

Benefits of feature rollout

There are many benefits to feature rollout. Some of them are as follows:

1. Thoughtful planning and team collaboration

Feature rollout encourages thoughtful planning along with team collaboration. It gets you to think deeply and methodically before introducing a new feature in the market and also collaborate with necessary stakeholders to ensure everything happens smoothly.

For instance, when we at VWO decide to introduce a new feature to our testing platform, we take the following necessary steps:

• Create a feature requirement document around the feature we’re planning to launch. This ideally contains the problem statement, reasons behind why it’s important to launch this feature, any supporting documents/tickets/conversations, market research, and success metrics
• Seek the opinion and technical guidance of all necessary stakeholders including product managers, engineers, marketers, and developers
• Gauge their feedback
• Prepare a release strategy, otherwise known as a solution document, and share the same with the development team to start the feature development process.

2. Opt for velocity testing and phased rollouts

feature rollout enables you to test everything in the beta stage. Teams involved in the launch process can rigorously test multiple hypotheses and get results in less time.

Besides velocity testing, when you’re planning to launch a feature to a large number of users, opt for phased rollouts. Segment your entire audience base into smaller groups. Gradually or one by one, test and release the feature to each segment.

Why do we advocate this theory?

Phased feature rollout gives you an opportunity to fully test the user interface and user experience of the new feature, and launch it in a manner that minimizes the chances of it firing back. People appreciate your new launches and look forward to more of them.

3. Speed to market

Owing to the ever changing nature of consumer behavior today, market conditions are always in a fluid state. A product can suddenly catch the attention of its users or even get rejected completely. At the same time, technological advancements and market competition can also affect the need and velocity to release new and advanced features more quickly to retain customers.

The process of feature rollout helps respond to such unprecedented market requirements and speed up delivery.

4. Better and quicker learning cycles

One significant advantage of frequently releasing new features to the market is that you get more timely feedback from your customer base.

This means the sooner you release the product in the market, the sooner your customers will use it, and the sooner they’ll share an opinion about your release’s functionality, especially whether or not it’s a useful update. Frequently releasing updates helps understand the psyche of your customers and build products that are worth their use.

Feature rollout and A/B testing

Amidst all the steps that you’ve taken to ensure a smooth rollout of your new feature, A/B testing your hypothesis is another crucial step that you must not avoid.

Experimentation plays a vital role in feature management and rollout. It’s one of the best and most effective ways to validate your ideas and hypotheses prior to your product’s launch. Experimentation also helps prevent errors and failures from creeping into the actual live environment.

But, how to do these tests? How many checks and reviews should you run before deploying a feature in the market? And, in what sequence should you run these tests? Here’s an ultimate A/B testing strategy to roll out a feature.

Note, every feature is different and the strategy defined below may or may not be applicable to every project. But, it’s a comprehensive and reasonable approach for organizations providing mid-sized and large products to millions of customers.

1. Development stage testing

Development is the primary stage where maximum scrutiny happens when it comes to discovering the true potential of a product. It’s also the stage where maximum iterations happen on a gut feeling or trial and error basis. This causes an increase in the iteration rate and delay in product deployment.

Rather than constantly testing the capabilities of the product on a trial and error basis, use A/B testing. It helps identify and resolve issues at an early stage and paces the production cycle.

Design and build the product, test it locally, and review and push it live on the staging server. Once done, this is where testing comes into play. With server side testing, you can check the nuances of the product you’ve just developed.

Rollout multiple A/B tests, multivariate tests or split URL tests and let them give you insights into what and how you can make your product more robust. The moment you feel confident with your build: all the changes seem to be working properly and as expected, and the customer requirements have been met, it’s time to take the next steps.

2. User testing in a controlled environment

Once you’ve successfully created your first prototype and tested at the development stage using server side testing, the next step is to make real users test your product.

There are two ways to do so:

• Invite users to test your product and share their feedback
• Run an A/B test and gauge user reactions through session recordings

The first method is quite similar to what we’ve explained in the above section, “How to launch a feature.” You can opt for one of the following methods to test the functionality of your newly developed feature.

• Limited rollout: Rolling out a feature to a limited audience such as internal employees, a specific set of customers, invite-only releases, and more.
• Gradual rollout: Rolling out a feature to a percentage of people. For instance, launching a personalization feature on the mobiles of 10% or 20% of users, etc.

Contrary to the above method where testing is done by voluntary users, all testers in the case of A/B testing are unknown of the release. This makes A/B testing more unique and a better way to get unbiased feedback from users who’re actually interested in the new release.

Depending on the type and size of feature you’re launching, you can either choose to run a regular A/B test, a split URL test or a multivariate test.

To ensure that you don’t break the existing habit of your users, run tests in phases. For instance, you can A/B test a part of your new feature, or select a segment of users to measure its success and retention. After all, your aim is to ensure your new version works at least as well as your control (the original version).

Identify where users are most engaging themselves and measure not only how much time they’re spending toying with your new feature, but also the time they take to come back and use it again.

The more often a user returns, the higher is the retention rate. This is a great indicator to measure the success of your release.

A/B testing not only helps analyze the performance of your release quickly but provides many other insights around the use of your product in the market. You can use the learnings to improve your release and even create newer strategies to make it more robust.

Conclusion

Feature rollout is not a linear process. It’s rather a twisted one that entails a lot of back and forth. To move one step forward, you will have to take two steps backwards. However, the end result it gives is exactly what you rigorously work for. The feature rollout workflow that we’ve discussed above may seem intensive but it’s one that has worked wonders for us, at VWO.

I've worked as a writer and editor in the B2B SaaS space for over 6 years. I read and write on all things CRO and experience optimization. I'm a chai fanatic and a paranoid parent. When I'm not copyediting or working on marketing projects, I love to spend my time reading. Besides B2B content, my passion lies in modern, behavioral astrology. I'm a dreamer and wish to write stories for children someday :)