Your data smells

You know what a product manager eats for breakfast, lunch and sometimes dinner?

Data.

It’s always about the data, and it’s not always palatable.

When I started working as a product manager, I had absolutely no experience with any kind of data analysis tool—hell, I didn’t even know how to do a VLOOKUP in Excel. I remember spending about a week drowning in videos explaining Google Analytics and its features, but I’ve learnt over the past year that using a tool to get data is the easy part. Understanding what to make of the data is the tricky part.

There are several times I’ve nearly torn my hair out because I didn’t know what to make of questionable data. I also found this post (thanks to the Mind the Product newsletter) that was unfortunately all-too relatable. The main problem is that data comes from multiple sources.

Imagine multiple water pipes going into a single water tank. Each pipe is of a different diameter, so the rate at which each pipe fills up the tank is different. The speed at which water comes into the tank is different, but all the pipes together achieve the final goal of filling up the tank. Data is what flows through different pipes, and each “pipe” is a function of the source of data and the unit of measurement. One pipe would have my traffic data (sessions) coming in from Google Analytics, while my other pipe would have the number of users who clicked through an email from Mandrill. Depending on the size of the product and the company involved, this could mean anything between 5 pipes and 50. It can be a nightmare to figure out why a certain metric went up and another went down because they come from different systems. Converting all the metrics to comparable units means even more data is lost; I think of it as a “conversion tax”. What this means though in practice is to always account for weaknesses in the measurement process.

You can have sampling errors, cases where the sample size was just too small, where wrong segments were defined—the scope for errors is omnipresent. Apart from being careful while analyzing data, I have always found it helpful to do a “smell test”. Check if your numbers stink! The easiest way to do this is to compare to historical data. Most numbers don’t change significantly on a daily or even a weekly basis, so if a datapoint went from 0.2 to 20, you know something is wrong. This would be an ideal example though, because usually the differences are comparable enough to be realistic, but also large enough to raise doubt about the correctness of data. In these cases, you have to just go with your gut.

I’ll be the first to admit that my gut has been wrong on several occasions. As the article suggests, when you do decide to go with your gut, the first step is to eliminate bad data and the systems that are contributing to bad data. After that, make an educated guess from the data that you have left at the table. It’s not likely to be 100% correct, but it will be a much closer representation of what the data is actually saying as opposed to what gets lost in translation due to its misinterpretation. Finally, this is something I’d like to record here for posterity

It is critical that managers appreciate that measurement — all measurement — is fraught. Good measurements enlighten, but bad ones mislead.

Thomas C. Redman

The unbearable heaviness of being late

If it’s one thing I am terrible at, it’s being on time.

late_clockNow don’t get me wrong, I’m not the worst. I can compare myself to friends and acquaintances who are much worse than I am at showing up when they’re supposed to, but the whole point is to compare myself to the best, not the worst.

In the workplace, showing up on time can make a huge difference to how you’re perceived. Every single time I’m late to a morning meeting, a part of me dies inside. I’ve often been on the receiving end, waiting for ages for a person to show up and it sucks. Whether or not it was intentional, being late indicates a lack of respect for the other person’s time. It’s plain rude and I know I’m guilty of it.

Tim Urban puts it really well. There are two types of lateness

  1. Okay lateness. This is when the late person being late does not negatively impact anyone else—like being late to a group hangout or a party. Things can start on time and proceed as normal with or without the late person being there yet.
  2. Not okay lateness. This is when the late person being late does negatively impact others—like being late to a two-person dinner or meeting or anything else that simply can’t start until the late party arrives.

I fall firmly in the first camp—I’m usually late to meetings that don’t require my presence to start. I highly doubt it negatively impacts anyone else, but it definitely does negatively impact me and how I’m perceived. I’m careful to avoid the second type of lateness, usually because the guilt itself would drive me nuts. The reasons why I’m late usually boil down to

  1. Denial. “I still have plenty of time left!”
  2. Instant Gratification. “Let me just catch up on all the Instagram posts I missed when I was sleeping”
  3. Resignation. “I’m going to be late anyway, what’s a few more minutes?”

The good thing is though, I recognize the error of my ways and I know that I have to do better. The underlying psychology of being late is pretty interesting too.

That’s a little-known concept called the planning fallacy, which is a strong tendency to chronically underestimate task completion. The planning fallacy is one of the most difficult behavioral patterns to change, experts say.

To put it simply, people who are late underestimate how long a task will take. Now as a product manager, a huge part of my job is to coordinate releases and plan product cycles. This means estimating how long tasks will take and sticking to a predetermined release date. A delay could be really costly not just in terms of money, but in terms of precious development time. Underestimating time in this context is fatal, and the weird thing is, I’m usually on-point with these type of estimates. It seems my problem with estimating how long things will take is limited to my personal life.

I’ve spent some time thinking and reading about it—I’ve come up with a few ways to combat perennial lateness

  1. Track how long usual activities like bathing, working out and driving to work take on average. Measure everyday until you’ve come up with a realistic number.
  2. Use these values while planning for your day. Overestimate and plan for trouble in each case.
  3. Set out everything you need for the morning (clothes, accessories) the night before.
  4. Don’t hit snooze on the alarm when you wake up in the morning.

I’m experimenting with a combination of these, I plan to update this blog once I do. Let me know if you have any suggestions that have worked for you.

Image credit: http://careergirlnetwork.com/

Both Sides of the Table

I’m going to be a bit of a fangirl here so bear with me.

Twitter is proving to be a great source of reading material, and I’m starting to see the appeal of it now more than ever. I’ve also come to the inevitable conclusion that there is no single formula to become a great entrepreneur, venture capitalist or product manager. In order to truly develop meaningful insights, you need to do a lot more than just read a single book or a single article on Medium. Like mastering any other craft, it requires concentrated effort and practice.

Until you’re able to construct your own system based on what works for you and what doesn’t (morally, creatively and otherwise), the next best thing is to follow a system that comes from a source you trust. For me at this point, it’s Mark Suster. I think my Tweet summarizes his appeal for me

While my dad can give me philosophical advice about life when I need it, more often than not I call him to ask random questions about filing my taxes. It’s practical and easier to understand the process from someone who has experience with it, and you don’t need to go into more details of the subject matter than you need to. Mark Suster’s advice is along the same lines—realistic and backed by a shit ton of experience.

I am subscribed to his blog of course, but I actually reinstalled Snapchat after being incredibly annoyed with it just to keep an eye on Mark’s Snapchat stories. Very helpfully, his snaps are transcribed for people who aren’t on Snapchat, but I find that watching him speak is a welcome break from reading, and his fist bumps at the end of each story never fail to put a smile on my face. In any case, some of his best hits from his various Snapstorms are below (huge thanks to @lazalberto)

  • 50 coffee meetings. If you committed to just 1 coffee meeting per week, you’d have 50 meetings per year – imagine how many connections you’d actually make.
  • Always send a thank-you email. You’d be surprised how very few people are thankful for the time they actually do get.
  • It’s really important, even at an early stage team, to have diversity of skills. If you’re more introverted you might want someone more extroverted. If you’re more process based, you might want someone more freeform.
  • Another big mistake, particularly in Silicon Valley, is people underestimating product management. The success of Facebook and Google is the exception, not the rule. Of course you need great engineers, but you need to mix it with great product management to get great product.
  • Inexperienced investors or board members will always try to get you to hire people with great resumes; I like to tell companies to hire people who are one punch above their weight class. Hire the person who was number 2 at sales or finance or marketing, who wanted the top job, so they’re hungry and will work harder.
  • A players beget A players, and B players beget C players. It’s because A players are confident enough to hire great people, while B players often lack the confidence so they hire people less good than themselves.
  • If you have a business, you need to think about distribution. The app store is only one distribution point. Think Facebook, think Imgur, think Reddit, think Slack… And for all you Apple fanboys, think Google.

The Design Sprint

When I was at the Google Women Techmakers event, I participated in the design sprint that was conducted by Melinda Klayman, UX Program Manager. We were given a problem statement that we had to solve using the 5 Day Sprint method developed by Jake Knapp over at Google Ventures. I was completely unfamiliar with it before I attended the workshop and although it was only a brief introduction to the method, I was sold.

The sprint gives teams a shortcut to learning without building and launching.
The sprint gives teams a shortcut to learning without building and launching.

So this is how things normally work. The CEO or the Head of Product decides that we need to develop a new feature that users are asking for. The product team comprising of product managers and designers are hauled into a brainstorming session, which sooner or later devolves into a discussion backed by purely by opinions and intuitions (as opposed to data, facts and real user behaviour).

The kicker here is that it has been well established that group brainstorming does not work. Sitting in a group, talking about ideas makes you feel like you’re being productive but in reality, the ideas that are produced from such interactions are subpar. This is because of several reasons

  1. The herd effect: Once the conversation heads in a particular direction, everyone follows. You tend to miss out on unique ideas that could have been generated without a preconceived sense of direction.
  2. Sheer laziness: There are always a few vocal people who take over the conversation, inspiring others to take a back seat and enjoy their mid-afternoon siesta.

This is exactly why the Sprint works. It combines the good parts of a brainstorming session while getting rid of most of its vices. In Thinking, Fast and Slow, Daniel Kahneman has pointed out how easy it is for humans to jump to conclusions without having all the necessary information—a huge problem in the brainstorming process. I’m quoting from the book here

Let’s decorrelate errors by obtaining separate judgments on the issue before any discussion. We will get more information from independent assessments.

The Sprint does exactly this. The creators of the Sprint know that individuals working alone generate better solutions, in fact they call it “working alone together”. The method also identifies the importance of involving representatives from all parts of the organization (sales, marketing, customer service) in the design of a product. It’s a unique mix of creativity and structure, something I’m starting to associate with Google instinctively.

I’m not going to describe every step involved in the Sprint (read the book for that). Group projects are also a huge chunk of an MBA program, and the groups consist of students with varied backgrounds and personalities—exactly what Sprints were designed for. I look forward to trying it out in that context as well.

 

Moonshot thinking

I was at Google’s Women Techmakers event this Saturday and it was a somewhat productive way to spend my day. I did expect it to be better than it was, but as always, that’s a case of my own mismanaged expectations. There were several speakers at the event, some of them good and some not so good, but I really enjoyed Karishma Shah’s talk about Moonshots. This isn’t necessarily a new topic of discussion, the concept of a moonshot has been around for a while. Google[x] is probably the loudest advocate of moonshot thinking, and I was personally very inspired by Ken Norton’s essay 10x not 10 percent.

venn-diagram

For the uninitiated though, the basic idea is think big. As a society, there are so many problems we still haven’t found solutions to. Global warming. Poverty. Hunger. Disease. The list goes on. These are the kind of problems that moonshot thinking will help you solve. I’d sort of mentioned this earlier in the context of the kind of companies I’d like to work for—if it’s not an important problem with a unique solution, I don’t want to solve it. This rules out Instagram for cats and Tinder for travellers from my job search.

Instead of looking at incremental improvements to existing solutions, moonshot thinking means starting with a blank slate—completely reimagining the problem statement. This type of thinking I’ve realized is much harder than it sounds. It requires us to exercise brain muscles that we haven’t really exercised before. It requires courage and conviction. But once you have all the prerequisites in place,

It’s often easier to make something 10x better than it is to make it 10% better. – Astro Teller, Google[x]

There was an interesting question that all of us at the event had to answer.

What is your moonshot?

I heard the question and I was pretty stumped. I know about the large problems – all of us do, but I always dismissed them as being too large for me to solve. I think that’s another aspect of moonshot thinking, you’ve got to believe that you can. As for my answer to the question – I still don’t have a perfect answer. The biggest step for me though is that I’ve started to think about it.

With that said, watch this video, it gave me goosebumps.

Combining form, function and design

I came across what I think is a stunning product – one that combines form, function, and design. It’s called a soft scale and what it does is deceptively simple.

  • It converts an ordinary bathmat into a weight and BMI scale
  • Fiber optic LEDs sewn into the mat display weight trends of the past two weeks against a goal
  • Wireless connection to the web permits realtime data analysis and graphing on a smartphone

Soft Scale

What I love about this is the fact that it is so unobtrusive. Most people don’t like to check their weight very often. It’s just a reminder of how badly you’ve been eating and how many sessions at the gym you’ve missed. But the soft scale is already a built-in part of your daily activities. The minute you step out of the shower, you towel off on your bath mat, and the best part—you’re on it long enough for an accurate reading of your weight. It has been designed for the human psyche of avoidance—there is no new learning curve here and no new habit to be cultivated. I can’t think of a better example for behavioural design.

Unfortunately as far as I could tell, the soft scale is still a concept and not an actual product. I first saw it in this presentation by David Rose, where he uses magic and enchantment to discover what delights human behaviour. He specifically calls out:

  • Omniscience
  • Communication
  • Protection
  • Health
  • Teleportation
  • Expression

I’m kind of stunned by how the manifestation of all human desires that are found in music, movies, art, and literature repeat themselves in technology. I mean in hindsight it seems obvious, but all the majorly successful tech products fall into one of these buckets. I highly recommend this presentation, it’s a refreshingly unique take on how to design for usability.

 

Behavioural Design

I happened to read a really interesting paper on Behavioural Design and the impact it can have on development policies. In technology, decoding user behaviour is one of the most important components of the design process, however these tenets of product management are yet to be adopted effectively outside the tech industry. The implications of assessing user behaviour could be huge—especially to improve the impact of several programs aimed at reducing poverty and disease.

Behavioural design process
Taken from Behavioural Design – A New Approach to Development Policy

I don’t want to regurgitate the entire paper, but there are many aspects of it that I found interesting and relevant, especially in the Indian context that I am familiar with. As privileged members of society who design these policies and models, we fail to recognize that the same problems that plague us also plague those who have less than us. I think our tendency to think “they don’t have enough food to eat so clearly that’s the only thing they think about” is wrong.

Lack of self-control, forgetfulness, procrastination, and laziness

are not just first-world problems!

Farmers are given access to fertilizers, but studies indicate that crop growth remains unaffected. We immediately think of reasons why this could be – perhaps the farmers are uneducated, perhaps the fertilizer needs to be subsidized further… but what if farmers had every intention of using the fertilizer, but it was just too much work? The example cited in the paper is spot-on. How many of us decide to go the gym  but find it difficult to go despite knowing the many health benefits of working out? How many of us know that junk food is terrible for our bodies, but we still stuff our faces with KFC every chance we get?

Behavioral economics leads us to a very different question. Our intentions do not always translate into actions. It is not merely about marketing or better tools of persuasion (sometimes it is that too). It is a deeper perspective on what makes people behave.

I’ve been reading Daniel Kahneman’s excellent book “Thinking Fast and Slow” and it really brings home the fact that exerting self-control depletes physical energy (leading to a faster pulse and decreased skin conductance) and that we only have a finite amount of self-control (also known as ego depletion).

Baumeister’s group has repeatedly found that an effort of will or self-control is tiring; if you have had to force yourself to do something, you are less willing or less able to exert self-control when the next challenge comes around. The phenomenon has been named ego depletion.

So really, an interesting solution for the ineffective use of fertilizers would be to remind the farmers each day by virtue of an SMS. Reminders often set off a “mental guilt” alarm and push us to do the things we need to do, especially when the push comes from a third party. We have apps to remind us when to go the gym, we make ourselves more accountable by making a public commitment on social media and we have “gym-buddies” who motivate us to train – there’s no reason why the same levers that control us can’t be used to push everyone else.

This is just one aspect of behavioural design that I found to be fascinating. The paper discusses several others, some of which include:

  • People are much more responsive to being informed of what they lose by not doing something than they are to being told how much it benefits them.
  • Comparing a person to his peers, neighbours, friends, etc. is an extremely effective way to change behaviour.

My key takeaway from this though is that we cannot underestimate the importance of human behaviour while designing anything that is to be used by humans, be it technology or development programs.

Words with Friends

I recently started playing Words with Friends on my phone and I quite like it apart from the pesky ads that show up after every single move. It was an interesting study for me though, so I dug a bit deeper and learnt about the gaming industry in the process. Every single number in there was plugged in using publicly available data; app usage data is notoriously hard to find even with AppAnnie and the likes.


 

Note: This is something I created and has no affiliation to the game or the company whatsoever.

The quagmire of A/B Testing

When I first started as a product manager, I didn’t know what an A/B test was. I read about it briefly and I thought “Okay, all you have to do is compare two versions of the same thing. No big deal”. It wasn’t until I actually implemented my first A/B test that I realized there were so many nuances I hadn’t even considered. As I’ve learnt and continue to learn over time, the concept of an A/B test is simple, if you know where and when to use it.

When you are immersed in a culture where you must test everything before coming to a conclusion, it becomes a habit to test even the most trivial things. If you don’t have a hypothesis to test or if you know for certain that the hypothesis is wrong—don’t A/B test it. This is something I’ve battled with constantly.

Whenever our CTRs (click through rates) are down, the first culprit pulled up is the user interface because this is the easiest, most visible thing to fix (ignoring other sources of the problem like page load times, seasonality, sources of traffic). So you jump in with your designer, do a revamp of the page and A/B test the old and new versions without an actual hypothesis. A wasted exercise.

The statistical significance of an A/B test is vital, and hence you can only do these exercises if you have a reasonable number of users. If you are using an A/B testing framework like Optimizely, the math is done for you. However, when we actually ran A/B tests on our own platform, knowing when to start (minimum number of users) and when to stop (how long to run the test) was much more complicated than we realized. The numbers at the end were shrouded in confusion and the result of the experiment was inconclusive. The lesson I learnt here was that unless you are confident about the statistical rigour of your model, rely on the experts in the industry.

There was an excellent post by Kissmetrics that I’m going to quote from here, which lays out the steps required to run an A/B test correctly.

  1. Decide the minimum improvement you care about. (Do you care if a variant results in an improvement of less than 10%?)
  2. Determine how many samples you need in order to know within a tolerable percentage of certainty that the variant is better than the original by at least the amount you decided in step 1.
  3. Start your test but DO NOT look at the results until you have the number of examples you determined you need in step 2.
  4. Set a certainty of improvement that you want to use to determine if the variant is better (usually 95%).
  5. After you have seen the observations decided in step 2, then put your results into a t-test (or other favorite significance test) and see if your confidence is greater than the threshold set in step 4.
  6. If the results of step 5 indicate that your variant is better, go with it. Otherwise, keep the original.

Image credit: https://vwo.com/ab-testing/

Mind games

You know what’s worse than mind games in relationships?

Mind games in interviews… or as they’re more commonly known—puzzles.

I had my first experience with them in college during on-campus recruiting. Instead of being quizzed on my engineering chops, I had to answer brainteasers about ants, triangles, and eggs. I was incredibly frustrated to say the least, because even as a lowly undergraduate with barely any work experience, I wasn’t sure how solving puzzles was an indication of my fit for the role. I mean I was nervous enough interviewing for jobs for the very first time, but to top that off, I had to do mental acrobatics while a panel of interviewers looked at me expectantly.

I hated it.

Ken Norton wrote what is now a legendary blog post about “How to Hire a Product Manager” nearly 10 years ago and his very first point talks about how to hire smart people.

I usually ask an interview candidate a series of analytical questions to gauge intelligence and problem-solving ability. Generally I’ll ask questions until I’m sure the candidate is smarter than me. For some reason, lots of people I know are reluctant to do that. They argue that it’s insulting to the candidate. I think the right candidate will relish the challenge. In fact, that’s the first test – how do they react when I say “I’d like to pose some theoretical problems, is that okay?” The best of the bunch are usually bouncing out of their chairs with excitement. The super smart sometimes counter with questions of their own.

From my admittedly limited experience, the only people bouncing out of their chairs with excitement are the ones who’ve pre-Googled solutions to all the popular puzzles online. Every single question I’ve ever been asked has been a variation of an existing question – so really, the only thing these questions test is your preparedness. But because Ken Norton is awesome, he admitted that his emphasis on logic puzzles was misguided. I agree with his sentiment that product managers need to be smart, however there are more effective ways to judge smartness. As Lazlo Block, Google’s SVP of People Operations says:

At worst, puzzles rely on some trivial bit of information or insight that is withheld from the candidate, and serve primarily to make the interviewer feel clever and self-satisfied. They have little if any ability to predict how candidates will perform in a job.

It is hard to solve a complex problem when put on the spot with a time constraint. In the real world, you can step out, take a break, drink some coffee or just spend more time to come up with an answer. But you can’t do any of that in an interview. Essentially, you’re weeding out some really smart people who may not necessarily think fast when under pressure. We all have different ways of solving the same problem and we all have different speeds at which we do it. Is one better than the other? I don’t think so.

With that said, it seems like most companies have finally seen the light. Google for instance has stopped asking brainteasers in their interviews and I’m hoping that this trickles down to companies in India as well. It is also amazing that companies like Stripe are so transparent about what to expect in their interviews. This comprehensive document demystifies what they are looking for and helps aligns expectations accordingly.

Now if only more companies would follow, maybe we can stop scouring the internet for hints of what to expect before every job interview.

Image credit: http://www.puzzlevilla.com/