top of page
  • Writer's pictureChelsea Killam

Yes, I'm a Gatekeeper

I’ve you’ve spent any time on the socials in the past year, you know how criminal it is to be a gatekeeper. So it is with some hesitation that I confess: I’m a gatekeeper.

But hear me out! In a professional context, it’s sometimes necessary. There are some things we should leave to the professionals, and as a research professional I’m gonna have to argue that surveys are one of them.

As ‘data-driven’ has achieved the business jargon status of “synergistic,” “agile,” and “paradigm,” more and more leaders are looking for ways to infuse evidence and insights into their decision-making. And the market has responded. Welcome to the jungle of survey crafting, where the vines of Google Forms and Survey Monkey have overgrown the once-sacred temples of professional research. It’s a wild, wild world where anyone with a keyboard and an opinion can whip up a survey. This has massively democratized research, but it’s also led to some pretty questionable data.

And while I’d obviously love for each of you to drop me a line the next time you need a survey, I realize that there are constraints that sometimes make it necessary for you to find a way to collect data on your own. So here are a few things you should know before you set out on your next survey safari.

There are three big phases in a survey project:

1. Write the questions

2. Collect the data

3. Analyze the results

Each phase introduces the opportunity for error and bias (technical terms in the field of survey methodology). This article is an excellent primer on the various forms of error and how they introduce bias into your project in case you’re a big curious nerd (#mypeople).

The form of error we’re concerned with when it comes to writing survey questions is known as Measurement Error (the Qualtrics article refers to it as Observation Error). I wrote about Measurement Error in my last blog post (Hanging Out With Toddlers Will Help You Write Better Surveys), too, but I want to get into some more instructional details here.

Concepts, Constructs, and Other Cool Stuff

Every research endeavor starts with a problem. In a professional setting, these usually come back to the business challenges you're facing that you want to inform through some sort of research.

Here is the framework I use to get myself ready to write a survey:

Here is the framework I use to get myself ready to write a survey:

Step 1: State your business challenge as one or more Hypotheses and/or Research Questions (remember these from college?).

Step 2: Identify the concepts that are embedded within your hypotheses/research questions. Another way to think about concepts would be as dependent variables.

Step 3: Operationalize you concepts into constructs. Constructs are the individual, discrete metrics that explain your concepts.

Step 4: Now, and only now, can you write survey questions that measure your constructs.

This is what it looks like to write just ONE survey question.

Business Challenge

Our customers don’t seem to come back to us the next time they need a

product like ours, causing a lot of attrition in our customer base. Competitive

research has revealed that other brands in our category have robust mobile apps

that they use to keep in touch with customers after the purchase event.

Expressed as a Hypothesis

Our customers will be more engaged with our brand

if they can interact with us via a mobile app.

Converted Into Concepts

In our hypothesis, we have concepts like customers,

engagement, brand, interaction, channels, and mobile.

Operationalized As Constructs

Focusing on the “engagement” concept, your operationalized constructs

might be things like, "feels loyal to our brand," "turns to our brand first,"

"stays up to date with our new product/feature releases," "recommends our

brand to others," "uses brand swag like stickers, shirts, etc."

Measured In A Survey Question

Now you can finally write some survey questions. Continuing with ‘engagement,' you

might want to go with a 5-pt likelihood likert scale question that asks, “How likely would

you be to use each of the following to stay up to date with our product and service

updates?” with a list of potential communication channels that includes “mobile app.”

I know this probably seems like a lot of work. And it is! But it is the best way to get yourself very smart, and very locked in on what your survey needs to accomplish. Without these steps, you’re likely to end up asking your respondents, “Do you think you’d be more engaged with our brand if we had a mobile app?” And yeah, they’ll answer that questions, but neither you nor the respondent really knows what that means or what is behind all of those concepts which leaves you making a lot of judgement calls and assumptions in analysis… but that’s a whole other form of error and a whole other blog post!

30 views0 comments

Recent Posts

See All


bottom of page