Never answer surveys!

The number of sites asking me to take a brief survey has exploded in the past six months. I’m also seeing email and paper mail requests to participate in surveys creep up too. If this increase is an indicator of an improving economy—as we interpret the worsening traffic on US101—that’s great, but it doesn’t affect my goal, which is to convince you that you should never answer surveys.

There are three reasons I’m trying to persuade you to avoid surveys. First, after almost a year at a startup, I’m very sensitive to time wasting activities. Second, I don’t believe survey originators are valuing that time fairly. Finally, since the survey terms rarely limit use of the submitted information, I want you to recognize that, in addition to being undercompensated for the information you shared, you have no control over how or for what end the information will be used.

We’ll visit each of these in turn, but before that, let me identify two exceptions where you might choose to submit a survey. Much of my arguments are going to depend on viewing one or both of the time and information that the survey involves as having some value. So, if the survey giver has no ability to compensate you for your time, you might choose to view your completion of the survey as a donation. That would allow you to return surveys to a selection of charities. Similarly, if you support the survey giving organization’s goals—like a Swedish car manufacturer with a fine product, but struggling in the market—you could answer their surveys in the hope that the value of the information you shared incrementally helps them to reach their goals. (Acting on a survey request being substantially less investment than purchasing a car.) A shareholder of a company that is surveyed by that same company might choose to answer in the hopes of seeing their investment succeed.

If the survey organization doesn’t fit into those categories, it’s very likely that you shouldn’t respond. Let’s see why.

“I recommend to you to take care of minutes: for hours will take care of themselves.” – Earl of Chesterfield

The first concern is that surveys require time, and return no information. Just the request, as a rejected dialogue on a web page or an email to be deleted, has already cost a minute. Typical surveys run from 3 – 5 minutes to 30 minutes or more, plus that initial minute of interruption. What do we receive for that minute? Presumably the survey giver now knows more about us, and can adjust their products or pricing to be more appealing to those people who answered the survey identically. We receive nothing—not even information shared by other respondents, so that we might get some comparative demographics about our fellow customers.

I think we would all get sensitized to a stream of requests for work (the surveys) for which we got nothing back. Eventually, even the most accommodating personality would just refuse. They might even slam down the telephone, or press the delete key really hard. That’s why surveys often offer to enter submitters into a draw for one or more prizes; this tactic is meant to suggest that you are being compensated for your effort.

Before we analyze a specific survey request, let’s establish a useful baseline for comparison. Minimum wage in California since 2008 is $8/hr, which corresponds to $16,000 annually. The survey requests all claim to need only minutes of our time, so a more natural unit is a minute of compensation. For the California, the minimum per-minute wage is $0.13.

I just received a survey offer today. It’s from a for-profit company, whose goals aren’t of particular interest to me. That means it’s a convenient example to evaluate. The survey claims to require 3 minutes for a chance at $2500. (We’ll omit the minute we lost reading the offer; it makes the numbers worse.) To determine whether or not the survey’s compensation is worthwhile, we will compare the minimum wage for 3 minutes:

$$ 3\thinspace\rm{minutes} \times $0.13\thinspace/\thinspace\rm{minutes} = $0.39 $$

against the expectation value of the lottery draw. The expectation value, written ⟨value⟩, is defined as the sum over all of the outcome values multiplied by the probability that each particular outcome occurs. Since, when we lose, the outcome value is 0, the only contributing term is when we win:

$$ \langle \rm{value} \rangle = \rm{value} \times P [\rm{we\ win\ lottery}] $$

The value of the prize is $2500. To determine the probability, we assume each respondent has an equal chance to win. Then we are left to estimate the likely number of respondents to the survey. If each respondent has an equal chance, then the probability is

$$ P[\rm{we\ win\ lottery}] = {1 \over \rm{number\ of\ respondents}} $$

The company in question, after a brief search, had 3 – 4 million customers in a recent report. If we assume they sampled 1% of their customers, that’s a pool of 30,000 – 40,000 potential respondents. If 1000 people respond, our expectation value is $2.50; if 10,000 people respond, our expectation value is $0.25, less than our minimal compensation. (The expectation value is equal when 6,410 people respond.)

So, for a reasonable assumption about number of respondents, we see that the expectation value of the prize is in the vicinity of the minimum wage for the time contributed. If our example company sampled a larger set, then it’s very likely the expectation value will be less than minimum wage; if you value your time substantially higher than minimum wage, the expectation value of the prize will not be compensating. Thus, in most cases, when you submitted your survey, you received neither useful information nor compensating value.

(Before moving on, we should flip our perspective: how did the surveying company value the information submitted and the impact on customer time? There’s the amount of the prize. We could divide that amount by the number of respondents, and see the valuation above; we might divide by the number of invitations, which is substantially lower. If data acquisition costs only a few cents per customer, is it likely that the data will be valued more highly in a strategic discussion?)

The last reason to avoid submitting surveys is that you are sharing information about yourself, or your household, with an organization into which you likely have limited insight. Beyond whether or not the data you submitted in your response has a meaningful impact on the company’s decisions, there are questions about the use of that data after its initial analysis. Some surveys say they won’t share your email or contact information, or explicitly ask for permissions around contact; in the bulk of these, though, no limitations are placed on the use of the submitted data. That’s worrying, but abstract.

The more concrete concern is that your information will be used by the company for goals other than the product improvement ones we’ve assumed above. Your information might be used to assess your ability to contribute to a political campaign, or to determine the compensation rates of your employer. Your disclosure of a preference for Scandinavian automobiles to your bank might be eventually shared with a domestic or foreign car manufacturer, who would rather you didn’t hold such a preference. Since there is no agreement on how the information can be used, you should expect that some of the survey data you submitted has been shared, or sold, and has circulated wildly. Why continue to refresh it?

Responding to surveys is a flawed activity in three ways: you receive no information in return for the information you submitted, you receive inadequate compensation for the time taken, and you have no assurance or control over the use of the information over any period of time. I suppose, as we look at those three concerns, that an idealized survey can be described as

• a survey that includes a simple summary of the responses to each question, returned within a reasonable time following the survey period,
• a survey with guaranteed payment in some useful form, in an amount that reflects the value of the responder’s time and the giver’s use of the information, and
• a survey that covers the use and retention of the submitted information with a actual, easily understood license.

If you receive a survey with those characteristics, I suppose you might consider submitting it. Otherwise, you should stop.