The Importance of Standards in Experimentation: A Key to Unlocking Successful Innovation

As any organization delves into experimentation and innovation, it's essential to establish standards for assessing experiment results. This is particularly crucial when working on complex projects that require precise data analysis and interpretation. Without a shared language of evaluating experiment results, teams may encounter difficulties in understanding what constitutes a success or an inconclusive outcome.

This can lead to issues down the line, as changing standards mid-experiment can cause confusion and relationship problems within the team. It's essential for organizations to set up these standards beforehand and ensure that everyone is on the same page. This includes developing a clear definition of what constitutes an experiment, how to assess results, and how to evaluate the success of a particular experiment.

Statistical Literacy: A Key to Unlocking Successful Experimentation

Statistical literacy is critical for organizations operating at full capacity when it comes to experimentation. While textbooks provide a solid foundation for understanding designs of experiments, there are nuances that arise in real-world applications. For instance, considering the cost of reading an experiment result inaccurately can significantly vary depending on the industry and application.

A financial services company or e-commerce company may face severe consequences if they misinterpret data, potentially leading to significant financial losses. In contrast, a pharmaceutical company's experiments can have far-reaching implications for human life, and incorrect interpretations can be catastrophic. Therefore, it's essential to develop standards that account for the unique challenges of each industry.

Standards for Different Business Units

The degree of confidence required in experiment results varies depending on the business question being answered, the part of the funnel being addressed, and the level of risk involved. As a result, different teams within an organization may require tailored standards to suit their specific needs. For instance, teams working on high-stakes projects or those venturing into new areas may need to rely more heavily on intuition and business expertise.

In such cases, relying solely on experimentation may not be the most effective approach. However, for other applications, experimentation can provide valuable insights that would be difficult to achieve through intuition alone. It's essential for organizations to strike a balance between these two approaches, ensuring that each team has the necessary tools and expertise to make informed decisions.

The Limits of Experimentation

While experimentation is an incredibly powerful tool for answering questions and driving innovation, it's not without its limitations. In certain situations, experimentation may not be feasible or effective due to factors such as lack of data, high levels of uncertainty, or significant regulatory requirements.

In these cases, business leaders must take a stand and make decisions based on their expertise and judgment. This is particularly true when venturing into new markets, products, or processes that require significant investment and planning. While experimentation can provide valuable insights in such situations, it's not always the best approach.

The Role of Business Intuition

Ultimately, business intuition plays a critical role in balancing experimentation with other approaches to decision-making. By combining data-driven insights from experimentation with business expertise and judgment, organizations can make more informed decisions that account for the nuances of their specific context.

In conclusion, establishing standards for assessing experiment results is crucial for unlocking successful innovation within an organization. By developing tailored standards for different business units and recognizing the limitations of experimentation, teams can strike a balance between data-driven insights and business expertise.

"WEBVTTKind: captionsLanguage: enhow important is it to have a common data language or data literacy or maybe statistical literacy within the organization to be a able to scale experimentation similar to what you've done American Express so I'd love if you can maybe comment on the role that data skills data literacy plays when it comes to being able to scale this culture of experimentation look I mean data literacy is absolutely basic to an experimentation culture right um and if different parts of the company has different definitions for what is in effect the same data it's very hard to then compare as results of experiments um if different parts of the funnel Define the same item differently and it's not an unknown problem that can be its own compounder right because sometimes the experiment that you're running runs across you know teams that actually operate in different parts of the funnel so practically you need to be able to bring different teams together and have them speak the same language I think it's also important to have very robust statistical standards as I said we usually leverage frequent test statistics for this purpose um the statisticians on our team have looked at the Corpus of experiments that we have run and they have a set of standards that we enforce across all the experiments across the company but it's it's also up to individual teams because sometimes you have sample Siz issues especially when you're running experiments in a completely new product or a completely new channel you just may not have as much sample size so would it would it necessarily make sense to have the same set of standards that you have for your primary product or your primary channels where um traffic is usually not a problem probably that's that's that's you know a question to be answered by individual companies and individual teams on how to uh react to problems like that but if you do not have that singular language of assessing experiment results that can cause its own issues later on so I would strongly recommend teams before they get into a you know widespread experimentation across uh you know a large company to have and set up those standards and make sure everybody aligns because again that um that really defines which experiment is considered a success which experiment is considered inconclusive and you do not want to change standards in the middle of a particular experiment uh that can cause a lot of relationship issues I can tell you yeah I can imagine and then you know you're talking about these these standards right and I'm sure this is something that your team is the one that's kind of building these standards advocating for them within the organization it's part of kind of that overall statistical literacy that we've discussed maybe walk us through what those standards look like in a bit more depth and you know if you add like a magic wand what does like statistical lit look like within an organization that is operating at full cylinders when it when it comes to like experimentation I mean I think you know the statistical standards I don't think they necessarily deviate much from the textbooks that um you can pick up on designs of experiments right but I do think a few things change I can give an example um and this is an example that's perhaps a little hypothetical but hopefully makes sense what is the cost of reading an experiment result inaccurately say for a financial services company for an e-commerce company versus a pharmaceutical company if you are a pharmaceutical company and by the way I have no experience working for a pharmaceutical company so this is a hypothetical example um people could die if the way you have looked at the data and you have used statistical standards that are not aligned with um standards Reg dat is required you could be as a company be sued out of existence now if you are trying to improve conversion rates in a funnel that's a somewhat different type of impact yes you could have a financial loss but you will not be losing you know you'll not be impacting people's lives in a way that's extremely detrimental so your ability to accept errors in experiment result readout depends on which part of the funnel you're operating in what is the business question you're trying to answer how critical it is for you to have the right answer uh the degree of confidence ultimately determines you know the standards that you put in so there's probably not one single answer that I can give it um you know we we understand the different needs different business units and different parts of a funnel have and based on that we have set up standards and you know you're hinting here at kind of different different standards across different teams and also mentioned previously you know when you were discussing is that depending on the area the channel the product the standards of experimentation may be different but as well the kind of how you approach experiments may be different you know uh I remember Brian chesky CEO of Airbnb mentions one mentioned once the limits of experimentation especially when venturing into new areas right um you know there's not a lot of data that you can work with if you're in a completely new area to run experiments on so maybe can you walk us through the limits of experimentation what point is there may be too much Reliance on experimentation or too much experimentation happening so I love to see kind of the Counterpoint here yeah no I um personally I love Airbnb and it's amazing you know the business that Airbnb has built over such a short time uh and the brand it has created um I have to say I don't necessarily agree with the statement that there can be too much experimentation but I do think what Brian wanted to say is that experimentation cannot answer every question and not not every situation can be converted into a specific experiment so there is always this need to rely on business intuition uh but in in effect business intuition also helps you choose the path you want to take and some of those are very difficult to decide ahead of time or experiment into um in certain cases I I certainly believe it is absolutely critical for Business Leaders to take a stand especially when they are thinking of completely net new markets net new products disruptive processes disruptive products it's hard to experiment into that level of specificity and gain enough knowledge without significant Investments and those you know sometimes yes it's possible to experiment into disruptive change but usually that is very difficult you have taken a decision that you must follow through and you know it's the it's the path that you take that determines success there so yes I would have to agree with him that there are certain areas where it is not possible to experiment but in almost all other areas where you know you know what you want you have some prior knowledge you have the infrastructure and the willingness to experiment it's usually a better way of answering questions than taking a shot in the dark and relying entirely on intuitionhow important is it to have a common data language or data literacy or maybe statistical literacy within the organization to be a able to scale experimentation similar to what you've done American Express so I'd love if you can maybe comment on the role that data skills data literacy plays when it comes to being able to scale this culture of experimentation look I mean data literacy is absolutely basic to an experimentation culture right um and if different parts of the company has different definitions for what is in effect the same data it's very hard to then compare as results of experiments um if different parts of the funnel Define the same item differently and it's not an unknown problem that can be its own compounder right because sometimes the experiment that you're running runs across you know teams that actually operate in different parts of the funnel so practically you need to be able to bring different teams together and have them speak the same language I think it's also important to have very robust statistical standards as I said we usually leverage frequent test statistics for this purpose um the statisticians on our team have looked at the Corpus of experiments that we have run and they have a set of standards that we enforce across all the experiments across the company but it's it's also up to individual teams because sometimes you have sample Siz issues especially when you're running experiments in a completely new product or a completely new channel you just may not have as much sample size so would it would it necessarily make sense to have the same set of standards that you have for your primary product or your primary channels where um traffic is usually not a problem probably that's that's that's you know a question to be answered by individual companies and individual teams on how to uh react to problems like that but if you do not have that singular language of assessing experiment results that can cause its own issues later on so I would strongly recommend teams before they get into a you know widespread experimentation across uh you know a large company to have and set up those standards and make sure everybody aligns because again that um that really defines which experiment is considered a success which experiment is considered inconclusive and you do not want to change standards in the middle of a particular experiment uh that can cause a lot of relationship issues I can tell you yeah I can imagine and then you know you're talking about these these standards right and I'm sure this is something that your team is the one that's kind of building these standards advocating for them within the organization it's part of kind of that overall statistical literacy that we've discussed maybe walk us through what those standards look like in a bit more depth and you know if you add like a magic wand what does like statistical lit look like within an organization that is operating at full cylinders when it when it comes to like experimentation I mean I think you know the statistical standards I don't think they necessarily deviate much from the textbooks that um you can pick up on designs of experiments right but I do think a few things change I can give an example um and this is an example that's perhaps a little hypothetical but hopefully makes sense what is the cost of reading an experiment result inaccurately say for a financial services company for an e-commerce company versus a pharmaceutical company if you are a pharmaceutical company and by the way I have no experience working for a pharmaceutical company so this is a hypothetical example um people could die if the way you have looked at the data and you have used statistical standards that are not aligned with um standards Reg dat is required you could be as a company be sued out of existence now if you are trying to improve conversion rates in a funnel that's a somewhat different type of impact yes you could have a financial loss but you will not be losing you know you'll not be impacting people's lives in a way that's extremely detrimental so your ability to accept errors in experiment result readout depends on which part of the funnel you're operating in what is the business question you're trying to answer how critical it is for you to have the right answer uh the degree of confidence ultimately determines you know the standards that you put in so there's probably not one single answer that I can give it um you know we we understand the different needs different business units and different parts of a funnel have and based on that we have set up standards and you know you're hinting here at kind of different different standards across different teams and also mentioned previously you know when you were discussing is that depending on the area the channel the product the standards of experimentation may be different but as well the kind of how you approach experiments may be different you know uh I remember Brian chesky CEO of Airbnb mentions one mentioned once the limits of experimentation especially when venturing into new areas right um you know there's not a lot of data that you can work with if you're in a completely new area to run experiments on so maybe can you walk us through the limits of experimentation what point is there may be too much Reliance on experimentation or too much experimentation happening so I love to see kind of the Counterpoint here yeah no I um personally I love Airbnb and it's amazing you know the business that Airbnb has built over such a short time uh and the brand it has created um I have to say I don't necessarily agree with the statement that there can be too much experimentation but I do think what Brian wanted to say is that experimentation cannot answer every question and not not every situation can be converted into a specific experiment so there is always this need to rely on business intuition uh but in in effect business intuition also helps you choose the path you want to take and some of those are very difficult to decide ahead of time or experiment into um in certain cases I I certainly believe it is absolutely critical for Business Leaders to take a stand especially when they are thinking of completely net new markets net new products disruptive processes disruptive products it's hard to experiment into that level of specificity and gain enough knowledge without significant Investments and those you know sometimes yes it's possible to experiment into disruptive change but usually that is very difficult you have taken a decision that you must follow through and you know it's the it's the path that you take that determines success there so yes I would have to agree with him that there are certain areas where it is not possible to experiment but in almost all other areas where you know you know what you want you have some prior knowledge you have the infrastructure and the willingness to experiment it's usually a better way of answering questions than taking a shot in the dark and relying entirely on intuition\n"