Price testing: monadic purchase intent test

[tweetmeme source="chenelaine" only_single="false" service="bit.ly"

This is the tenth post in my Customer Research series.

Today I read a fantastic post on the Vovici Blog by Jeffrey Henning on monadic price testing.  You can read it here.   In fact this is such a fantastic blog that if you are interested in customer research at all, you should subscribe to the feed – I haven’t found any post that’s not insightful and educational on this blog yet.

The post on monadic testing was particularly apropos for today since we had some new team members come on board, and we reviewed the methodology and outcome of a purchase intent at price study we did early last year that used this methodology. In a nutshell, here’s what you do:

• Come up with a screening questionnaire that will only let in the respondents you care about (e.g. you will want to screen for people that fit your primary buyer persona.)
• Come up with a “product concept” that describes your product or service’s key value proposition. This tends to be a two page text and image based presentation in an on-line survey.
• Pick the price points you want to test.
• Send out the survey to several groups of respondents, each of which will see the exact same product concept, at one of the prices you want to test.  Respondents are asked to rate their likelihood to purchase on a 5 point scale.
• Keep collecting completed surveys until you have at least 100 samples per price point (Best practice suggests collecting 100-400 samples per cell).
• Now tally the results and see if the top-two-box scores (percentage of people who respond with “very likely” or “completely likely” purchase intents) show any kind of a trend as you vary the price.

A lot of times people assume adoption will rise as price drops. In reality that’s true only if the product or service is not valued by the target respondents or if there is an obvious comparable product that has a low price.  If a monadic study like this shows no obvious trend when you vary the price for the exact same product concept, that is a clear cue to product management to hold the price and not cave in to channel pressure to drop the price.

The reality is that retail and other channels will always pressure a business to drop the price – but that’s not always going to result in an uplift in customer acquisition.  If the customer has substantial pricing elasticity, then dropping the price only serves to reduce your margin, and doesn’t necessarily buy you additional market share.  Something to keep in mind when thinking about how to price your products and services.

Beware the focus group of one

I read a great post by Jeff Bussgang (an entrepreneur turned VC) where he talked about “Mother in law market research”.  He shared a quote by his MBA classmate:

“I think [this consumer product] will be a hit because I can see my mother-in-law buying it.”

I don’t have to explain why such an assertion should never be made without supporting data (e.g. does the MIL match the primary or secondary personas?  Does she have the right demographic / psychographic profile?  Was there qualitative and quantitative research to prove the same?)

The “focus group of one” happens to the best of us, regardless of our roles.  Sometimes we use the MIL, most times we project our own views onto the target personas.

It generally goes like this.  Someone in the company becomes fixated to some product feature.   99% of the time, he or she is not a good match to the target personas.   (He can be a man commenting on a product to fix hot flashes for menopausal women.)  He or she would share his/her opinion:

“Yesterday I saw a demo of <product feature>, and it immediately made me think people can <achieve improbable application of product feature to unrealistic use case>.  I am now absolutely convinced we must line up all our resources to optimize the user experience for <unrealistic use case> because if I thought of this,  lots of other people would want to do that too.”

I’ve seen it happen to  founders, CEO’s, CTO’s, COO’s, or SVP’s of something-or-other.  For all their brains and success (present and past), they fall into the trap of believing they can project themselves into the minds of target end users, without taking the time to really understand the latter.

Unfortunately, these guys routinely underestimate the magnitude of thrashing they can cause.   Let’s face it, if the SVP Sales declares that product feature X must do Y, the product team isn’t going to spend 2 minutes convincing her otherwise.  Instead, it is going to spend 2 business days putting together a well structured argument, based on facts.  Then they will make an appointment with her to present their arguments.   They may even commission a new research study to put the argument to bed.

And let’s face it… if she remains unconvinced despite hard facts, and the company is set up in that way, Feature X shall do Y in the next release.  Forget about the research results and the needs of the end users.

So, to folks on the management team:   please don’t become the focus group of one.  At best, you will waste much more time of your product team than you realize.  At worst, your stray comments could cost your company its ability to develop great products.

Beta programs

This is the eighth post in my Customer Research series.

What is a beta program? The Wikipedia has the following definition:

“Beta” is a nickname for software which has passed the alpha testing stage of development and has been released to users for software testing before its official release. It is the prototype of the software that is released to the public. Beta testing allows the software to undergo usability testing with users who provide feedback, so that any malfunctions these users find in the software can be reported to the developers and fixed. Beta software can be unstable and could cause crashes or data loss.

In my mind, the beta program provides an early window into how the market will receive the product release.  It generally happens at the very end of the development cycle.  I generally run small-scale beta programs as follows:

• Recruit 10-20 beta testers to match the primary and secondary personas that the product is designed for
• Do a kickoff meeting (either one on one or in a group) to set expectations on what’s in the new release, and how we expect to collect feedback from beta testers
• Ask beta testers to use the product in the target environment of use
• Schedule a call with each tester on the phone 1 week into the program, to ensure everything is going smoothly
• Use phone calls and email to keep track of progress during the program
• At the end of the beta program, schedule a phone conference or an in person debrief to collect feedback.

Since beta programs occur at the very end of the development cycle, typically weeks before the target release date, it is really only useful for testing things that can be iterated right before the release: positioning and messaging, delivery and support mechanisms and the like.  There is a great recent post on beta programs by Dave Daniels of Pragmatic Marketing that outlines all this – do take a look, it brings into sharp relief many of the questionable practices a lot of software companies take for granted.

Findings from beta programs can also be used to pull a release if (gasp!) a customer discovers a fatal bug that the QA department failed to find.  Lastly, it can also be used as a vehicle to collect customer feedback for the next release.  It is NOT a vehicle for usability testing – it is way too late in the game for that!  Usability studies (whether in lab or extended use tests) should be done early in the development cycle, before the product is finalized and when there is still time to effect change.

Using beta programs to test positioning is a great idea.  One can save a lot of money in marketing programs by iterating the messaging with target buyers until winning messages are arrived at.

Usability research in the lab

This is the seventh post in my Customer Research series.

Usability research for consumer electronics products can be very costly.  There are companies that specialize in doing it the right way, with high end audio/video equipment and multi-stream video editing and compositing integrated into the program.  The deliverable is typically an incredibly insightful presentation with snippets of video that tell a compelling story all by themselves.

Since I work with startups and small businesses, I have never had the luxury of doing it “right”.   My theory is that some research trumps no research.  So I butcher best practices until they become unrecognizable but affordable (deepest apologies to Scott Weiss who taught me how to do it the right way!)

I usually start with a research protocol that clearly states the questions we want to answer, provides a guideline for recruitment and lists the props required for the session, which includes a mini-DVD camcorder and a tripod.   Then I design the session, which is videotaped throughout.  I try to stay inside of 1h if at all possible.

Let’s say I am comparing the usability of two smart phones for working moms aged 35-45, with at least one child under the age of 12 living in their house.  The session could look something like this:

• Introductions and orientation – explain purpose of research to subject and let them know what to expect (5 min)
• Execute any paperwork, such as an NDA, a photo and video release forms, and a profiling questionnaire (5 min)
• Ask the subject to familiarize themselves with Device 1. Product manuals are provided to the subject. (5 min)
• Repeat with Device 2. (5 min)
• Ask the subject to execute a scripted task list for Device 1.  Tasks tend to be fairly specific – for instance, I could ask them to make a call, send and receive a text message, check traffic, take a picture, upload a picture to a computer, take a video, etc.  Ask them to verbalize what they are doing as they try things out (but do not offer hints or commentary – we are there to watch and learn, not to talk.)  (10 min)
• Repeat with Device 2. (10 min)
• Debrief – loosely guided interview to ask subjects to rate the usability of each task on a scale of 1 to 5, as well as answer some open ended questions about their general impressions and perceptions (20 minutes)
• Present the incentive check (typically $50-100, depending on the nature of the study).

This format is great at providing a sanity check for the out-of-box experience for consumer electronics devices.  Can the end user figure out how to set up a new device and get it working without groaning and gnashing of teeth?  Lots of times they can’t.  I’ve learned so much about what’s wrong with the current packaging design and documentation from watching subjects struggle through product setup.   It is very hard to keep quiet and not offer suggestions along the way… but the learnings are priceless.

As with all other kinds of research, I am aggressive about inviting engineering team members to be observers in these sessions.  This is the best way to help them understand who they are designing for and why certain feature enhancements are necessary to ensure an awesome user experience.

Focus groups vs. roundtable discussions

This is the sixth post in my Customer Research series.

When I do qualitative research, invariably someone would say: “oh, so you are doing focus groups!” which usually makes me cringe.  The reality is that 99% of the time I am not doing focus groups.   I may be doing detailed interviews or observations which are 1-on-1 techniques.  Or I may be doing a photo essay or journal study.  Or I may be doing roundtable discussions.

For those, I adhere to the same best practices one uses to run focus groups:

• Include no more than 8 participants per session.
• Craft a screening questionnaire and recruit carefully to ensure you get the right crowd.
• Separate the genders. You get much more candid discussions that way (especially for younger demographics).
• Control the discussion. Ensure everyone at the table has a turn (including the reticent ones).
• Record the discussion on video for future reference.

Such a roundtable discussion becomes a focus group only if two more criteria are met:

• The moderator is an independent third party who is engaged by the company to do this research.
• The discussion is held in a research facility with one-way glass.

I’m old schooled.  I insist on using an independent moderator for focus groups.  In my experience, employees tend to be too close to the products and services provided by the employer.  They have assumptions and expectations that may impact their ability to be impartial.  A third party moderator has no such baggage. He or she is free to learn about the product by asking probing questions, then lead a discussion where more probing questions are asked.  The resulting quality of the discussion tends to be much higher and more unbounded for this reason.

As for the facility, the one-way-glass room has superb benefits. It allows a lot more people from the company to observe. It also  removes employees from the participants, helping to foster a more genuine discussion. It costs more than using the company lunch room, but I have never felt like I didn’t get my money’s worth in such a facility.

What’s your take? Would you do a focus group in a hotel meeting room? I would love to hear your thoughts.

Using detailed interviews to build personas

This is the fifth post in my Customer Research series.

There are many awesome posts on personas, including this primer by Steve Johnson at Pragmatic Marketing, and this post about how personas serve the CEO and the executive team.  So I am not going to belabor the point.  I’ll simply provide a case study of exactly how I go about building personas.

Before we begin, we must decide whether we are building buyer personas or user personas.   Since I have always worked on the PD/PM side (instead of the PMM side), I almost always end up making user personas first.   Here is the process I use.

1.  Pick a market segment.  Size it.  Make sure the total addressable market (TAM) is big enough to matter.

2. Generate prototype personas.  Draw on your company’s tribal knowledge / theories about customers and prospects, as well as from your own life experiences for this.  Make sure you state assumptions about needs, wants and expectations.

3. Create a screening questionnaire.  Work with a recruiter to find subjects.  Evolve screener as necessary (tweaking the prototype personas as needed).

4. Plan the interviews.

• Pull together the equipment: a mini DVD camcorder, a tripod, fully charged batteries, camera, laptop for notetaking. (Mini-DVD camcorders result in the least postprocessing overhead.  Finalize the disk and you can watch it instantly.)
• Staff each interview with 1 researcher who will lead the conversation, and at least 1 support person (in the role of audio/video monkey and notetaker).
• Every researcher should plan on going to several interviews (overlapping with each other for some of them).
• Cycle as many staff members through the support role as possible. This exposes everyone to the process and gets people out of the office and into customers’ environments, which is always good.
• Plan mid term and final debrief sessions with the research team.
• Plan other work around this work.  It takes at least 1 – 1.5 person days to process each interview. You also need time to reflect on the results.  Expect each researcher to be completely consumed for a good 75% of the time.
• Share intermediate work products with the team early and often.  Transparency is key to buy-in.

5. Do the interviews. Here are some tips I’ve collected over the past 14 years (your mileage may vary).

• For best results, detailed interviews should be done in the target environment for product use. That said, an interview at a coffee shop or some other neutral environment is better than no interview at all.
• Start with an introduction and a warm up section where the subject gets comfortable with the researchers, then ease into the topic of interest.
• Keep the interview guide short and sweet. Questions should be open ended, inviting the interviewee to talk in their own style and to show their personalities.
• Be prepared to take the conversation where it wants to go, not where you want it to go.
• Make sure the subject talks more than you.
• For certain types of products, you may have to match gender. (e.g. research around a feminine hygiene product will require a female interviewer).
• Don’t even think about showing a product or product concepts. Discuss your product at the end only if there’s time left over.

6. Build the personas.  With luck, you will have recruited carefully to get the right people, and after 10-20 interviews, they will have self-organized.  You should now be able to build composite personas in each bucket of subjects who share key needs, wants, expectations, and attributes.  Check these personas against the people you met and tweak until they work well.

Once the primary, secondary and non-personas are built, you can begin to use them as a proxy to help you envision what benefits they require, what products or services can deliver these benefits, and how those products and services may look like.

Common ethnographic techniques

This is the fourth post in my Customer Research series.

The word “ethnography” has such a grand sound to it.  I’ve seen seasoned executives get intimidated by it. They say things like “Whoa – that’s big agency stuff! We don’t have budget for that!”  To that, my standard response is: “It’s not rocket science!  We can do this!”

And it really isn’t rocket science. It’s just hard, detailed work.  Lots of it. You don’t need a degree in anthropology or psychology. Anyone with an open mind and great listening skills can learn to do it.  If you can’t listen more than you talk… well, perhaps you shouldn’t be a product person after all.

And the budget can vary wildly.  You can spend $250,000 if you outsource it to a big agency and you have a global market.  You can train your staff to do it themselves under some guidance for well under $25,000, using a recruiter to find subjects for you.  I’ve run projects under $2500 where I did the recruitment myself using Craig’s List.  So your mileage may vary.

Now what exactly is ethnographical research in the context of product development?  In the simplest terms, it’s a set of qualitative techniques that places the researcher in the environment and/or the mindset of the subjects.  One listens and observes subjects in the environment that the product is meant to be used in, without showing the product or product concepts. One then derives the needs, wants, expectations and workarounds for the subjects, and uses this information to drive product definition.

There are three techniques that I am a big fan of (mainly due to their high bang for the buck):

1. Detailed Interviews. Researchers meet with a subject for one to one and a half hours.  A researcher asks open ended questions to get the subject to tell a story about the domain of interest.  The interview guide tends to be very high level and the researcher is trained to mix things up and respond to new threads that come up in conversation.  I like to have two to three researchers at each interview so one person can drive the conversation while the other person mans the audio/video equipment and takes notes.
2. Observation or shadowing. Researchers set up their audio/video equipment in the environment where a product or set of products is being used, and simply hang out with the subject while the subject uses the product.  The researchers asks questions when necessary, but by and large they behave like flies on the wall.  They are there to watch and learn, not to talk.  The subject is disturbed as little as possible.  This could be a multi-hour proposition – I once shadowed someone for 8h with a coworker.
3. Immersion. Researchers use the product or a related product for an extended period of time to get a first hand understanding of long term product use. Researchers can record their extended findings via a journal or a photo essay. I also like to write a debrief document at the end of the immersion to sum up my key takeaways.

There are other techniques too, based more on self-reporting by end users. Examples include journal studies or photo essays involving customers.  These are all valid approaches.  When used with detailed interviews and/or observation, they can help round out the picture of the customer’s problems, needs, wants, expectations and so on.

The biggest challenge for this technique is the amount of time it takes to do a good job. Since research is done one customer at a time, and best practices suggest we work with 10-20 customers to allow the data to converge and allow time for tweaking the technique and/or the recruitment criteria, it takes corporate level commitment to pull off a project like this.  It’s a lot of long nights too since sometimes the research must be done after hours when the customers have time to work with the researchers.  But if we keep the focus, and we recruit correctly, generally by the 5th or 6th interview or observation session, a pattern will begin to emerge.

You know you’ve nailed it when you start hearing the same thing from everybody over and over again.  It’s an incredible feeling to get there and know you’ve built the knowledge that will guide the product team moving forward.

Asking the right questions of the right people

This is the third post in my Customer Research series.

One of the trickiest things about customer research is subject recruitment.  The quality and applicability of research results is directly proportional to one’s ability to ask the right questions of the right people.

There are generally two types of customer research that directly impact product development:

• Product discovery research.  This is what typically happens at the beginning of a new product development effort. The goal is to understand who we are developing products for, what their problems are and what solutions may serve their needs and meet their expectations.
• Product research. This is typically done after a product launch and the goal is to investigate product utility and usability and to gauge customer satisfaction.

Product discovery research should be done with prospective customers, not existing customers, for two reasons:

• Existing customers of a previous generation product may not fit the primary persona for the new product, so we may end up asking the right questions of the wrong people.
• Even if existing customers are smack in the sweet spot for the new product too, their expectations for the company’s value proposition have been set by the specific feature set offered by the previous product. Their needs may not be any different but their wants will be shaped by what they know about the previous product. We would be asking the wrong questions of the right people then – these folks are much better equipped to help with product research instead of product discovery research.

Product research, on the contrary, is best done with real customers and end users who would have much more to share about long term use of the product.  One can do some product research with prospective customers too, but generally unless there is a desire to start selling the product to a completely different buyer/user persona than before, it is better to stick with existing customers.

That said, nothing is absolute in customer research.  A certain amount of insight can always be generated by a bit of cross-recruiting.   For instance, during product discovery research, one can test product features with a customer advisory board who can provide a rapid sniff test without imposing the overhead of recruiting new subjects.  They can also comment on whether key features are missing in their mainstream workflos.  During product research, one might want to test product usage with a completely new target demographic and see whether the product as it stands today can be expanded to serve new market segments.

In general I believe in being highly creative and mixing things up when doing research. Every product development project is different and when one looks at the facts at hand, it will be quite obvious which types of research efforts will yield the most bang for the buck.

From Personas to Functional Specifications

This is the second post in my Customer Research series.

Many people have asked me how persona work leads to product definition.   Here is my personal take.

In any new product development effort, we need to ask the following questions:

1. What is the market segment I am shooting for, and is it big enough?
2. Who am I making this product for?
3. What are the problems that customers are trying to solve?
4. How may our core competencies be applied to solve these problems?
5. What are the product features that may deliver these solutions?
6. What is the end user’s mainstream workflow, and what minimum feature set is necessary to service that workflow?
7. How would the end user want these features to work for Version 1?
8. What would the roadmap look like for future releases?

Item 1 is a market sizing and analysis exercise.  It is based on quantitative data from analyst reports and from internal market analysis.  It builds the business case for a new product, and helps establish a basis for interest that justifies further effort.

Items 2 and 3 are addressed by persona development.  I like to interview prospective customers in the environment they will be using the product in about their thoughts on the problems that the future product may address.  I listen between the lines and then derive the needs, wants and expectations of the prospective customers.

The key words are “prospective” and “expectations”.   Expectations are set by experiences, which is why you will get a different result if you interview existing customers of a previous-generation product, as opposed to prospective customers that represent your target market.

Item 4 is a cross functional discussion involving the product manager, the engineering lead, and a proxy for an archetypical end user in the sweet spot, representing the primary persona we are trying to target.   For instance, if the target user is a 65 year old, there should be an older person in the discussion to represent their point of view.

If the core competencies do not support a viable solution to the customers’ problems, the team would have to go back to Item 1 and rethink the target market and the business case.   Developing new core competencies is an engineering research activity that precedes product development.

Item 5 is about envisioning features that will deliver the solutions needed by end users, and Item 6 places them in the context of user workflows.   One is meaningless without the other.

At this point is generally a lively discussion with all the possible ways to serve up benefits to customers.   Rather than going around in circles in the office, I reach out to customers to get their feedback.  I employ a few techniques for this:

• Get customer feedback with paper prototypes.  This can be done either as a round-table discussion (2-3 groups of 8-12 participants each) or remotely by phone on a one-on-one basis.  This should be done with both prospective customers and existing customers so we can get the full range of perspectives for people in the sweet spot and for early adopters. Results should be interpreted separately, of course.
• Beta testing with hacked prototypes where feasible. This should run for 2-4 weeks with prospects AND existing customers with separated analysis of results for the two constituencies.
• A product concept test for the new product, executed as an on-line survey to prospective customers only.
• A comparative product concept test, pitting the proposed product against a control product, executed as an on-line survey to existing customers and prospective customers.

Once we get through 1-6, we will have a clear idea of what features ought to be included.  The next step, Item 7, addresses how those features should work.  I like focus groups for this detailed phase, but if budget doesn’t permit, we can cover this with informal round tables or virtual focus groups.  I like qualitative techniques for this because it is difficult to get into enough detail with quantitative techniques.

When we wrap up Item 7, we have nailed all the requirements and are ready to develop a Functional Specification for Version 1 of the new product. This should represent a minimum set of features that service the mainstream workflows of end users in the primary persona.  The product team can now execute on the new product.

The learnings from 1-7 will also feed Item 8 – roadmapping.  We will have a ready list of features that deliver secondary benefits, and we can tentatively bracket those into future releases.

This process can sound like a drag. It is not. It is simply a practical way to frame the challenge of understanding customer problems.  The whole thing can take anywhere from a few days to a few months, depending on how much the product team already knows about the market and the product, and the scope for the new product development effort.   A little thinking goes a long way here – much better to invest time up front than start development without a clear understanding of what one’s doing and why.

Why do qualitative research?

This is the first post in my Customer Research series.

I will start with a discussion on why I do qualitative research.  Some companies place a disproportionate amount of trust upon quantitative, statistically significant results presented in analyst reports.  Anything involving a handful of research subjects is viewed with deep suspicion.  Insights that their own product teams bring back from the field are dismissed as anecdotal, and the knowledge is not used for decision making.

In my opinion, this suspicion is dangerously misplaced.   Numbers are sterile. They are necessary; any moderately skilled product person would stay on top of market sizing reports and industry trends by following analyst reports.  However, that provides only a sliver of the knowledge needed to design the right products.

Qualitative techniques allow us to probe deep into personas, needs and wants, use cases, habits and practices and so forth with a small sample size.  We step into the customers’ shoes and get a taste of their motivations, problems they encounter, and what they may be trying to achieve with a current or future product, in their environment with a workflow that works for them.  This, coupled with quantitative research, is what drives great product development.  It is incredibly time consuming,  but if you plan and execute it properly, every minute is worth the work.

Please join the conversation if you are a product person – I would love to hear what you think.