I won’t preach to you about the importance of user interviews when validating a hypothesis but I can merely hope that by now you acknowledge their implication.

Nevertheless, the problem with user interviews is that most times they are done improperly. And when this is the case, the decisions based on their outcome is actually worse than those based on no user interviews whatsoever. Because a false positive (or negative) path is a very misleading one. 

Let’s assume that you have a new idea for a feature that you, a smart guy, consider that is going to break the charts. And since you know that the best practices say that you gotta’ talk to the users first, you select a few of them and give them a call.

“Hey! We are doing this feature, do you like it?”

“Yes, sounds good.”

Fantastic! Now, you need to know if they would pay for it.

“Would you pay for it?”

“Yes sure, let me know when it’s ready!”

Jackpot! Excited by the confirmation received, you start to allocate resources into something that, surprisingly, users don’t find valuable. And no, they are not going to pay for it.

So what happened?

The users have lied to you, that is what happened. And mostly for not hurting your feelings. 

I never actually thought I was about to write this but the problem with most people is that they are good people. And good people will tell you what you want to hear because they don’t want to be perceived as bad, insensitive humans.

“I like your idea, sure, just don’t think of me as a bad person!”

The rule of thumb is that, in user interviews, people mean good but do bad.

How to do it right?

Not actually an interview

First thing, don’t make the interview sound like an interview. In fact, it’s actually better if the user doesn’t know she is being interviewed. The more structure you add to your interview the more people are going to tell you what you want to hear.

You can successfully achieve this setup especially when testing a new product hypothesis by going where they are actually hanging around, such as conferences, meetings and other sorts of events. While networking, find out what their problems are.

Focus on them

The main principle behind user interviews done right is that the questions to ask should be about the lives of your customers with their problems, goals, dreams, pain points, barriers, etc. In product management, you are always shooting blind before you understand what the problems really are. Once you figure out, you are one step closer to validating the solution. 

For testing new feature hypotheses on an existing app in use then you do have to add some structure to the interview. The secret sauce here is to keep it as casual as possible. Be very subtle, make it about them, and for the first couple of interviews don’t even mention the solution. Keep it in the back of your mind, compare it with what you learn, and refine it accordingly.

Is the problem important?

And speaking about problems, how can you tell they are important? That they are a real hassle and that by solving them you are going to increase the value of the product and get paid more?

The answer is: 

Check out to see if the customer has already actively looked for ways to solve it.

If they desire to achieve something that your app is not currently delivering and they have some painful and time-consuming workarounds to accomplish it, then YES, the problem is important.

If they “somehow thought about it” but didn’t do anything to address it, or that “it happens every day but I don’t mind it”, or if the workaround is quick and painless, then the problem is not there and your feature would be pointless.

What to ask to increase your confidence that the problem is real:

• “When did the last time X happen?”
• “How did you deal with it?”
• “What else have you tried?”
• “How do you deal with it now?”
• “How’s that coming along?”

Start digging

At the point where you have identified a real problem, start riding it. Get specific, ask for the number of times it has happened before, its frequency, related aspects, and look for emotional signs. Are they excited, do they seem bothered, do they express any concern? Or are they simply emotionless?

The whole point is to learn and be brutally honest regarding your solution. Don’t dig for answers that would just confirm your hypothesis. Don’t forcefully link your solution with the problem and say that “yes it might actually work if we do…”. Don’t try to be pleased with what you hear. Don’t lie to yourself!

Again, be brutally honest, curious and skeptical. Look for clues about why it wouldn’t work and further polish the idea based on unemotional empirical evidence.

What to ask:

• “How many times it happened?”
• “How often does it happen?”
• “What makes it so awful?”
• “Why haven’t you been able to solve this already?”
• “You seem pretty excited, is this something important to you?”

Avoid compliments

In user interviews, compliments are dangerous and sometimes highly misleading. They feel good and grow up the excitement around a certain idea but send product managers on an erroneous track. Compliments are nothing but unintentionally lies told to protect your feelings.

• “Yes, I like it’.
• “I would definitely buy this.”
• “When will it be available?”

Feels good, right? It is almost as you can clearly picture victory. 

Avoid compliments by averting mentioning the solution for as long as possible. Users will not “like” something that they don’t know about.

When you do reach the right moment to mention it (you are more confident that the problem is real and that your solution is the right one for it) and compliments somehow keep coming then Mike Tyson them. Deflect them and subtly go back to asking the right questions meant for objective learning.

“Great, but let’s go back to focus a bit more on…”

Disregard (most of the) ideas

If you have ever done any user interviews then you know how “helpful” people can be. 

• “Why don’t you do this?”
• “I’m sure this would work.”
• “It would be great if your product can do X.”

Most times ideas are compliments in disguise. And compliments are lies in disguise. They are unintentionally aimed at maintaining the good spirits of the conversation and protecting the feelings of the interviewer. Your feelings.

Some other times though, ideas might express solutions to a not so clear problem that they are having.

Like when users ask for X to do Z, but it proves that either Z is not important, or, if it is, Y might be a better way to solve it.

The way to deal with ideas and feature requests is to start digging into the “whys” “whats” and “hows” behind them. 

• “Why do you want that?”
• “What would that let you do?”
• “How are you dealing without it?”
• “Why do you think we should do it now and not later?”
• “Why do you think we should do this before that?”
•  “How would that fit into your work process?”

Bottom line:

If there is one thing to remember about correctly doing user interviews is that they own the problem, you own the solution. The purpose of the interview is to dig into the problem and see what is the best way to crack it.

In terms of numbers, three to four interviews done right should be enough to get a clear picture of the problem, its importance, and add more confidence to the whole hypothesis validation purpose.