As a Sr. Product Manager at AWS Machine Learning, I led the development of AWS DeepRacer from the ground up. AWS DeepRacer is a 1/18th autonomous racing car that aims to teach developers reinforcement learning (RL). As you can guess, the product has fairly large scope covering hardware, device software, and RL simulation on the Cloud. However, the most critical piece that determines the success of the product is data science. I’m lucky to have the chance to work with a few brilliant RL experts and learned tons from them. Since this is a brand new area for PM especially for AI products, there are not many best practices we can refer to. So I think it will be good to share my experience and hopefully valuable to the fellow PMs.
The Challenge
Science work usually determines whether an AI product can meet the expected performance, such as prediction accuracy, and in turn result in the success or failure of an AI product. As a product manager, I generally enjoy interacting with our scientists who taught me a lot. However, I also found there are several unique challenges when working with the science team when I tried to drive the product forward.
First, scientific research is not as predictable as software development. When running agile scrum, engineering team can estimate the required efforts for each task with relatively high accuracy. For science work, especially research or exploration related, scientists usually have no idea how long it will take. All we know is a few ideas or papers we can try out. No result or timeline can be guaranteed. When building DeepRacer, there was a time we have to decide which simulation engine to use. We originally thought it only needed a couple of weeks but ended up with spending more than six weeks to make the final call.
Second, science team usually has a different report structure from the product development team. At Amazon, most of the teams work independently, which means we define the vision, roadmap, design and build the product. There are lots of autonomy and ownership of the team. When it comes to data scientists, most of them are assigned to a specific product and dedicated to that project. However, they usually don’t report to the same manager as the product team. I don’t think this is uncommon at other organizations. In fact, some companies have shared science teams who handle all the requests across different product teams. This structure, together with many other factors, lead to lots of challenges to coordinate development work.
Third, scientists usually have a different mindset as engineers. Coming from academia, most scientists are driven by curiosity instead of results. They are interested in trying the latest paper or attending research conferences. I don’t think this is a bad thing but it does add risk to manage the schedule when you are trying to deliver a product under a tight deadline. Additionally, most scientists enjoy running experiments but may not necessarily have rich experience of writing production code. All of these make it pretty challenging when they need to collaborate with the engineering team closely to build a product. I often feel the two teams were speaking two different languages. In these situations, misunderstanding, even mistrust, can be easily developed.
My Approach
Because scientists are so critical to the product and the challenges above, I believe PMs need to pay special attention and put more effort to drive science work moving forward. On the other hand, since working with data scientists are relatively new for most products, I found there is little direct experience to leverage. As a result, I have to explore and experiments with different methodologies over time. They are by no means the best practice but could be used as a starting point for you to find the approach that works best for your team.
Earn trust
It is one of the Amazon leadership principles, but I found it applies well when working with our science team. There are two aspects I did to build trust with them.
First, learn the basics of the machine learning technology needed for your product so you can speak the same languages as data scientists. When I joined DeepRacer team, I know nothing about RL except the name. So I asked one of our scientists to literally give me a lecture of RL introduction. Be humble and ask stupid questions, especially in the beginning, helps me a lot to gain domain knowledge. I found most data scientists are pretty easy going and are willing to help if you ask. Additionally, there are a lot of educational blogs, videos, courses to teach machine learning techniques, which I leveraged as much as I can. As I learned more, I was able to understand the jargons and concepts when talking to scientists, which helps a lot to facilitate communications.
Second, bring value to the science team by offering information about customers, business strategy, etc. As a PM, I was able to access the information from cross-functional teams such as UX, marketing, leadership, etc. Such information is valuable to help the team make decisions. For example, I did lots of user studies during the UX design of DeepRacer and found something that was particularly confusing for our targeting users. When I brought the data back to the science team to prioritize their work, they respect my input, which eventually builds my credibility. Once you’ve earned trust, it will much easier to collaborate with the team.
Define the vision and goals
Perhaps one of the most valuable things that a PM can bring to all the teams including the science team is to align everyone with the long vision and business goals. It is especially important when you try to motivate the team.
Since the science team is more likely to be distracted by different factors such as the latest research progress, it is your responsibility as PM to keep them focused on the most important things for the product. In terms of product vision, I think putting something that is bold and inspiring is very helpful when the team disagrees on certain topics. For DeepRacer, I clarified our vision is to help developers learn RL through a fun and hands-on approach. We particularly want to educate people who have little knowledge of machine learning. The vision enables us to settle on a lot of arguments. When it comes to long term planning, I think PM needs to spend significant amount of time on building a clear roadmap. The mid to long term goals should be challenging, exciting but still achievable. I actually listened to the ideas from our science team carefully before deciding what should be our biggest deliverable by the end of the year. Once the team gained buy-in of the target, I can feel our scientists are inspired and keep working hard towards that goal.
Set the priorities
As the ultimate owner of your product, I believe PM should always prioritize your team’s work ruthlessly. It also applies to science work. Actually, it is one of the most valuable contributions PM can make to the science team.
Each team usually has its unique way to prioritize their work. My experience is that it often requires some experiments and iterations to find the best approach for a new team. There is a book called Product Roadmaps Relaunched: How to Set Direction while Embracing Uncertainty that I found very helpful and served as a starting point for my way of prioritization. In my team, using pure score system to rank everything makes people confused. So I tweak the approach by defining the tasks by putting it in higher level categories, such as must have, highly desired, nice to have, etc., then using scores to stack rank each of the task inside each category. This approach works well so far and gives the science team a clear idea of what needs to be done first.
With priorities and deliverables aligned, the science team is able to focus on the work that is most important to the product and less likely to explore too many irrelevant areas.
Set the stages for complex tasks
When it comes to complex tasks like exploring an idea based on a recent paper, there are a lot of uncertainties. It would be challenging to have an even rough estimation when it will be done or what the result will be. To address this challenge, I would divide and conquer, which means split the big task into smaller stages and have checkpoints for each of the stage. Even though you don’t know what this research will lead you by the end, it is much easier to evaluate the efforts needed for the first one or two stages.
When building DeepRacer, we wanted to explore a number of reinforcement learning algorithms that can work for our use case. This is a big task that requires lots of experiments and development. To reduce the risk, we decided the first step is to set up the infrastructure needed and then experiment with two popular algorithms. This helped us checkpoint in two weeks. Comparing the result, I proposed to stop further exploration and just focus on the most promising algorithm. So instead of continuing trying different ideas, we just focused on one algorithm until launch, which saved the science team a tremendous amount of time.
Track the progress
Let’s talk about program management even your title is a product manager. It is common for PM to do some program management work, especially for the science team. A program manager who has many years of experience as a researcher in academia shared with me a lot of good suggestions. To minimize the risk of science work, you could work with the team to define 3 things to every science task:
- Deliverables
- Acceptance criteria
- Deadline
Then ask a single science owner to commit on the work, and make the owner accountable. Most team use tools like JIRA to track development work, I just created a template for science tasks with the three components above. So far it removes a lot of ambiguity and makes the team more productive. But from time to time, the science work may not be able to meet your expectations or there is disagreement that can’t be settled. Under these circumstances, it will be necessary to escalate to the management.
Future Experiment
Like many other things in life, we shouldn’t treat the procedure we have so far as a static system. Instead, we should periodically take a step back and retrospect what has worked well and what hasn’t, and make changes accordingly. For me, there are still many ideas I want to try. For example, we haven’t really followed the strict agile process for science work. Although it is probably not entirely necessary to do everything required, I do think we can try to increase the communication frequency with daily or twice a week standups in addition to weekly sync. Also, I’d like to have science team provide input for the features for future development which will offer opportunities for them to make direct impact on our product.
There are definitely more than one ways to work efficiently with the science team. So you should keep iterating to find the approach that works well for you. As it is still the early stage for AI product management, I hope to learn a lot more from you in the future. Cheers!
