Wyatt Jenkins, a VP at Shuttestock, wrote a great HBR blog post about A/B testing and creating a culture of experimentation. In it, he makes an important point that is not made enough in all the books and blog posts that are promoting new and better ways to lead and manage our organizations: every answer has a down side.

Every time you figure something out, you’ve also created a problem. That is just the nature of the universe, I think. In complex systems, there’s not a single right answer. You’re never done.

Jenkins describes the power of a culture that values experimentation, particularly to deal with the HiPPO problem (Highest Paid Person’s Opinion). In a lot of organizations, the HIPPOs drive what happens, and not always for the best. But at Shutterstock, according to the post, when a senior executive suggests a solution, the response is “Great, let’s test that.” More ideas see the light of day in this culture, and employees across the board are more motivated because they get to see their ideas come to life. But he also points out the down side:

It’s not all roses, of course. Experimentation culture has some downsides, too. One of the big ones is that experimentation teams sometimes miss the next big innovation because they’re constantly making incremental improvements that show quickly in test results. Remember, some test results will show a negative outcome in the short term, but be better in the long term due to user change aversion. Also, testing strategy is hard, and there’s still a place for strategic thinking that moves your organization in new directions.

So it’s complex. It’s a both/and. You have to know when to experiment and when NOT to experiment. But such is life. The days when we could ignore that complexity (and simply rely on HIPPOs as the answer) are gone. Today’s “answers” look different.