What Are You Weighting For?

Naira Musallam, PhDNov 14 2018
What Are You Weighting For?

In research, weighting is used to adjust the results of a study to bring them more in line with what is known about a population. And it’s a frequent topic of conversation.

It’s often the case that there is confusion around when to use it, how to use it, and its purpose.

So What is Weighting?

Weighting is an applied correction technique. It is done by multiplying each response that is collected from a consumer by a ratio to bring the total results more in line with what is known about the population.

For example, only 20% of the consumers that responded to your survey were males. However, we know that males make up roughly 50% of the general census population. So that 20% needs to be adjusted to bring the results more in line with the overall characteristics of the population.

In general, for responses that belong to an under-represented category, you assign a certain weight that is larger than the value of 1, and for responses that belong to an over-represented category, you assign a weight less than 1.

But Why Weight?

Ideally, you collect your survey data from a representative sample at the start, so that no weighting is needed.

If you’ve done the work upfront and you simply sample your target population in a representative way that meets the goals of your research, then the analysis is the very next step. No weighting, cleaning, or re-coding is needed.

Weighting techniques become important when there are discrepancies between the actual population you are trying to analyze, and the breakdown of the consumers you engaged. When this happens, no reliable conclusion can be drawn from your collected responses- so we weight.

To apply weighting properly, consumer insights teams and market researchers need to rely on what are called “auxiliary variables”. These are variables that have been measured in the survey itself, and their distribution in the population is already known. Like males in the example above.

Typically, these include demographic variables such as age, sex, gender, marital status, etc. obtained from credible national statistical institutions such as the Census Bureau. They can help consumer insights and research teams make estimates on the larger population.

Populations can be anything from all voting-age citizens of the United States to craft beer enthusiasts in Portland, Oregon, or cat owners in Pennsylvania. Each is unique, and a representative sample for each will look different. 

Now for the tough question: to weight or not to weight?

In published research, top-notch empirical scholars make conflicting choices about whether and how to weight, often providing little or no rationale for their choices.

Additionally, in private discussions among experts, it has been repeatedly found that accomplished researchers express confusion or provide faulty reasons for their weighting choices. This debate is captured below:

Pros:

  • Enable researchers to argue for more accuracy and representation of results for the larger population by adjusting for over-or under-represented segments of the population

Cons:

  • It increases the standard errors of the statistical analysis, making the overall findings less precise and more variable
  • In other words, if you are up-weighting respondents, and counting each as more than one person, the more their answers are exaggerated
  • All analysis is effected, to include reported descriptive statistics (means, percentiles, medians, modes), and inferential statistics such as regressions and coefficients

Again, this is a correction technique. So by that definition, you are fundamentally altering the data you’ve collected from your consumers.

There is a general consensus in statistics literature that weights can be useful for descriptive statistics information like mean, median, mode, and standard deviation.

Our view is that insights professionals and researchers should avoid relying on weighting as much as possible. Opt for focusing more time upfront, identifying and targeting a representative and balanced sample before you get yourself into the muddy area of weighting. 

It will not only save you time on the back-end, but generally make your life a little easier.

Naira Musallam, PhD

Naira Musallam, PhD

Ready to meet the next generation
of market research technology?

More from SightX

Just Like Content, Context is King

Just Like Content, Context is King

Suppose you are asked to assess the impact of a brand's messaging, an educational program sponsored by the government, or perhaps even a leadership development course for corporate America.

by Tim Lawton

Five Ways to Get Creative with Heatmaps

We’ve said it before- and we'll most likely say it again: consumers are changing.

It should come as no surprise that consumer behavior has evolved quite a bit in recent years, but that evolution was fast-tracked in 2020. From where they shop to how they want to connect with their favorite brands- consumers demand engagement on their terms.

Effective engagement can mean speed and efficiency, but more often than not, it also demands creativity.

For insights teams, in particular, this can be a challenge. However, a modern, effective, and creative way to get impactful feedback from consumers is through a heatmap experiment.

A heatmap is a visual storytelling exercise. It organizes data about an image using color-coded zones representing the frequency of activities, interactions, or sentiments.

Historically, heatmaps have been a popular visualization tool with data-driven researchers across industries. Given current consumer trends, it shouldn’t come as a surprise that heatmaps have been gaining popularity in recent years amongst leading researchers. While they remain a key tool in user interface and experience research, their usage in concept and product testing research continues to gain popularity.

To help spark some creativity and curiosity, we’ve put together a list of simple ways you can incorporate heatmap techniques in your own research:

Whitespace & Prototype Testing
Exploring white space and researching prototypes are important initial steps in the product innovation process. If you have some initial ideas or mock-ups for a product, heatmaps can be an important early indicator about which attributes your potential customers would be compelled by, or (just as importantly) be repelled by.

Efficient and effective prototype feedback allows you to refine your products earlier in the development process- before you even begin building your minimum viable product (MVP).

Design Testing
Getting feedback on visual design elements like fonts, colors, layouts, and imagery is an important step in the research process, and heatmap experiments are one of the most cost- and time-efficient ways to do it.

Using heatmaps for design testing allows you to identify what works and what doesn’t for any customer-facing visuals.


Package Testing
Most products go through many iterations of packaging designs before launch. Testing various concepts with heat mapping allows you to gain detailed insights into potential customers' preferences surrounding specific packaging attributes.

Respondents have the opportunity to select and react to design elements, logo placements, packaging types, and other details - allowing you to understand where consumers focus their attention and in what order.

Ad & Message Testing
Your go-to-market messaging and content strategy can make or break your product launch. However, message testing isn’t just about the words themselves - the taglines, logos, and other copy in the ad are just as important as the package and product designs.

Using heatmaps, you can test which ad or message garners the most positive or frequent interaction, and which drives more viewers to engage with the Call-to-Action. Consumers indicate to researchers where the messaging is catching their attention, if that attention is positive or negative, and why they feel that way.

Shelf Placement
Even though most of us are primarily shopping online, the in-store experience cannot be overlooked. Pandemics aside, consumers will continue walking into stores for the foreseeable future. By testing how a consumer responds to different shopping environments, you can understand how to maximize value both for the customer and your brand during in-store shopping experiences.

Of course, the shelf is a critical point in the in-store customer journey. Heatmaps are a great way to understand optimal shelf placement and product combinations that will entice consumers to reach for your products. They can also help with the design of the shelf itself!

These are just five primary examples of how heatmaps can enhance your consumer research to provide visual, data-driven insights. They are a quick, fun way for consumers to provide insights in a survey setting, and make a great addition to any research report.

Start exploring new use cases and research projects with heatmaps! And of course, reach out to the team at SightX to learn more.

by Naira Musallam, PhD
Image of a sample price research study featuring ice cream flavors and toppings, alongside a graphical output of pricing research data.

Your Guide to Pricing Research

Finding the perfect pricing for your product is no easy feat. 

by Savannah Trotter
SightX

Research Services

Business