5 tips for product managers working with machine learning

Machine learning based products are springing up like mushrooms after the rain. As a results more product managers are likely to face ML algorithms in their careers. In fact, if you’re reading this, you probably fall under one of two categories: either you work already with ML algorithms, or you will work with ML algorithms in the future.

But what effect does ML have, if at all, on our work as product managers? Is there anything different about being a PM for an ML product vs. any other product? How much should product managers know about these algorithms?

Here are 5 pieces of advice on how to better manage an ML product based on my experience working with ML algorithms:

1. Commit on the process not on the results - Imagine a head of a clinical research lab who says to her researchers “we’re going to find a cure to cancer by end of Q2, then in Q3 we’re going to pass clinical trials and in Q4 we’re going to get FDA approval”. not very likely, right? I’m not saying machine learning is akin to curing cancer but ML is a bit like research work — it is hard to predict when the breakthrough will come. In most areas of product management, you write your spec, pass grooming with R&D and get effort estimations (for example in dev days, or T-shirt size). However there is an inherent unpredictability nature to ML that makes it hard to asses when development will be finished. If you’re new to ML you might be tempted to commit too early to the business and management on results. Instead I suggest you commit on the process and set expectations with your management in the way you communicate about your product. For example, instead of saying “we will develop an ML algorithm in Q1 that does X, then in Q2 we will start beta testing with 5 clients” you can say: “in Q1 we will test algorithmic approach A. If the test succeeds we will move forward in Q2 to testing with 5 clients. If the test fails, we will test algorithmic approach B in Q2 instead”. This is called scenario planning. When you start talking this way you signal your business stakeholders that you are dealing with a complex process that is hard to predict. Also you set the expectations that if an experiment fails it is not a failure — it is a good thing! it takes many failures in order to refute hypotheses until you get to the right one.
2. Set a dumb benchmark - while committing on the process will help you setting expectations, it won’t help you with getting things done. The following 3 tips could help you expedite the ML product development and achieve better results. So let’s say you started developing an ML algorithm in your company. Congratulations! A month has passed since the algorithm engineers started to work on the project : do you know how much progress was achieved? Now two months have passed, new models are tested and new signals are taken into account in those models. Can you tell now how much progress was achieved? If you don’t want to stay in the dark about the progress of your ML product I suggest you constantly compare to a dumb benchmark. A dumb benchmark is the naive solution. It can be a simple rule, a random data set or a toss of a coin. There is a known fable story about a government that spent millions of dollars on building a weather prediction satellite. When they were finally done after few years of development they found out that the satellite can predict tomorrow’s weather with 80% accuracy. Not bad, everyone thought. But then they found out if they simply predicted tomorrow weather would be like today’s weather, the prediction would be also 80% accurate. The morale of this story you should always benchmark against the simple “dumb” solution. It sounds trivial but it’s very useful. If you do that you gain two things; First you might find the dumb solution is good enough. If all what that government needed was 80% prediction accuracy then they could have saved millions of dollars and years of development. However even if it’s not the case you gain a powerful tool to asses the progress of your algorithm development, by constantly AB testing the ML algo against the naive one.
3. Be super-duper specific with your KPI- suppose your KPI is indeed 80% accuracy of tomorrow’s weather prediction. Now you start testing your model and you find out in the northern part of the country you get 83% accuracy and in the south you get 74% accuracy. Is it success? suppose you were 89% accurate in April, August and November but 75% accurate the rest of the year. Is it success? and what if you are better predicting temperature rise than temperature decline — is this success? Maybe you would like to consult with more people in the company about how to interpret the results? maybe you need more analysis? maybe you need to get into a quiet room and think this through. In any case more time is wasted and you don’t have much time to spare. What I’m trying to say is you should think in advance about all the possible results and be as specific as you can with defining success. Being specific with your KPI is always a good thing, but since ML projects have a natural tendency to take time and not meet deadlines, you don’t want to waste additional time in trying to understand what’s your next move.
4. Roll up your sleeves, know your Algorithm — You might think that because you don’t have a PhD in mathematics or computer science, you don’t need to know how the algorithm works or that you don’t have anything to teach the algorithm developers. Well… you’re wrong. big time. If you don’t get yourself involved in understanding what the algorithm actually does, the algorithm developers might end up solving a mathematical problem but not a business problem. You are the product manager, you understand the client need, you understand the business environment. Your job is to help shape the algorithm by setting the product constraints and determine what to optimize for. But how do you deep dive into a fancy algorithm without having a PhD in mathematics? For that I suggest the blackbox approach (also known as input/output approach): if you’re trying to understand a complicated algorithm, try instead to focus on understanding what are the inputs and outputs and keep the algorithm as a blackbox. In most cases, if you have good understanding of what are all the signals going into your ML algorithm and what is coming out, this will be enough to start adding value and direct the development to a good direction.
5. Be empathic to your clients anxiety — ML algorithms usually imitate or replace human decisions. However most people find it difficult to trust the machine in making decisions for them. Especially when those decisions have something to do with money. Would you trust an algorithm to make investments decisions for you with your pension funds? Would you trust an algorithm to automatically choose a hotel for you for you next vacation? Would you feel a bit of anxiety letting the machine make these decision for you? I’m guessing most people would say yes… When you develop an ML product take into account it might make your clients a bit nervous, and the best way to relieve anxiety is to bring back sense of control. Do that with your product — think of a parameter in the algorithm you can externalize and turn into a lever for the client to pull. Alternatively, offer the client a way to revert the decision made by the algorithm. Or even just explain to the user why the algorithm has made a specific decision. Here are few examples I’m especially fond of:

• Netflix explaining to you why they are recommending you a particular movie because you watched something similar.
• Waze allowing you to find an alternative route if you’re not happy with the route they have chosen.
• Moovit letting you decide between least walking or least transfers.

To be honest most of the above pieces of advice might be applicable to any product not just ML one. However I found these ones to be particularly useful when your product involves ML.

Do you have more words of advice on how to better manage ML products? If so please share them in the comments .

This blog post is based on a talk I gave on ProductX 2019 at Tel-Aviv University. The video is accessible here (Hebrew only )