Recommendations for designing high-quality online surveys

The quality of data collected through a survey questionnaire depends directly on how well the questionnaire is designed. Poorly worded questions, an unsuitable structure, flawed logic or technical shortcomings can lead to biased, incomplete or even unusable results.

The recommendations below provide a systematic guide to preparing a survey questionnaire in 1KA. They cover basic formal considerations, fundamental and advanced methodological principles, technical guidance and additional aspects of questionnaire quality.

The recommendations are intended for all users, regardless of their level of experience. They are equally useful for those preparing a survey for the first time and for more experienced users seeking to further improve the quality of their questionnaires and, in turn, the reliability of the data they collect.

Formal aspects of questionnaire design

Formal aspects provide the foundation for a clear and comprehensible survey questionnaire. Although they may often appear to be purely technical, they can have a significant impact, as even minor errors may lead to questions being misinterpreted or responses becoming invalid.

1. Using the correct question type

Each question should use the appropriate response type. Checkboxes should be used when respondents may select more than one answer, while radio buttons should be used when only one answer may be selected. Using an inappropriate question type can result in unusable data.

2. Using conditions

If a question is not relevant to all respondents, use a condition (IF) to ensure that it is displayed only to respondents who meet the relevant criteria. Question skipping should not be managed through written instructions (for example, “If you selected female, skip question …”).

3. Checking spelling and grammar

Before publishing the questionnaire, check its spelling and grammar. Also ensure consistent use of semicolons, commas, full stops, spaces and capitalisation. Pay particular attention to consistency at the beginning and end of subquestions and response categories. Where appropriate, ensure that gendered forms are used consistently, or opt for neutral wording.

4. Page breaks

Use page breaks thoughtfully. As a general rule, a single section should not extend much beyond one screen on a desktop computer, helping to keep the questionnaire clear and manageable.

5. Questionnaire design

Unless you have professional design expertise, avoid extensive custom styling and use the standard survey themes instead. 1KA offers ten general-purpose themes, all of which are also optimised for mobile devices.

6. Introduction, invitation and closing message

Respondents first need to be encouraged to click the survey link and then, through a clear introductory message, motivated to complete the questionnaire. The introduction should briefly explain the purpose of the research, what participation involves and the conditions of participation.

The introduction or invitation should provide information about the research, including:

• who is conducting the research and whom respondents can contact with questions,
• what respondents will be asked about,
• how respondents were selected,
• the purpose of the research,
• why participation is important,
• the approximate time required to complete the survey,
• assurance that responses will be anonymous,
• the option to decline participation,
• how and where the results will be published, and
• clear instructions on how to access the survey and begin completing it.

In the closing message, state where and when the research results will be published. After completing the survey, respondents can also be redirected to a selected webpage, and a message can be displayed once the survey has been deactivated, for example with a link to the published results.

Both the introduction and the closing message should be friendly and tailored to the respondents.

Methodological recommendations

When preparing a questionnaire, it is important to consider methodological recommendations alongside the formal aspects of questionnaire design. These principles guide how questions are worded, how concepts are measured and how the questionnaire is structured in order to minimise bias, ambiguity and avoidable errors.

1. Using scales instead of dichotomous questions

Where possible, replace dichotomous YES/NO questions with scales that allow attitudes to be measured more precisely. Scales capture the intensity of an opinion rather than simply whether it is present or absent.

Example: Instead of asking “Do you support the measure? YES/NO”, ask “To what extent do you support the measure? (1 – do not support at all, 5 – fully support)”.

2. Testing the questionnaire and using comments

The questionnaire should be thoroughly tested before use, as practical testing often reveals ambiguities, logical errors or difficulties in understanding particular questions. The questionnaire should be tested by at least one person.

In test mode, it is also advisable to make use of user comments, as these make it easier to identify errors and opportunities for improvement as they arise. If several evaluators are involved, consider enabling the entry of initials so that each comment can be attributed to the person who submitted it.

Before collecting live data, it is advisable to delete all test entries.

3. Checking performance across different devices

The questionnaire should be tested on different devices and operating systems, as its appearance on a desktop computer may differ considerably from its appearance on a mobile phone or tablet. Pay particular attention to tables, scales and longer passages of text, which may be laid out differently on smaller screens. For example, a table that is easy to read on a desktop computer may be misaligned or difficult to read on a phone and should therefore be checked separately. Also check the length of the survey title to ensure that it displays correctly on mobile devices and does not take up excessive space.

4. Using mandatory questions

Use mandatory questions only where they are essential to the questionnaire logic or subsequent branching. Excessive use of mandatory questions can increase respondent burden and may lead respondents to abandon the survey before completing it.

5. Randomising response categories and subquestions

For nominal variables, such as regions, brands or political parties, and for table questions, it is advisable to randomise response categories or subquestions when their order is not substantively important. This helps to reduce the effect of item position on respondents’ choices.

6. Limiting the use of “Don’t know” responses

Include a “Don’t know” option only when it is methodologically justified, as providing this option may encourage superficial responding.

7. Defining variable types

For all categorical variables, clearly define whether they are nominal or ordinal. This will later affect how calculations and results are presented in the analysis.

8. Using a clear and engaging survey title

The survey title should be concise, clear and informative. A good title immediately signals the subject of the research and can influence whether a respondent is willing to participate.

9. GDPR compliance

When collecting personal data, we recommend using the default 1KA GDPR introduction. The introduction should explain the purpose of collecting the data, how the data will be used and the rights of respondents. If a respondent does not consent to the collection of personal data, they will not be able to continue with the questionnaire.

Advanced methodological and quality considerations

In addition to the basic methodological recommendations, it is advisable to consider more advanced methodological and quality-related aspects of questionnaire design in order to improve the quality of the resulting data.

10. Optimising questions and measurement scales

When designing a questionnaire, use closed-ended questions with predefined response options, including an “Other” option where appropriate. Simplify scales by removing numerical labels and using descriptive response categories tailored to the content of the question. For example, when measuring concern, use a scale such as “How concerned are you about …? Not at all concerned … Very concerned” rather than a general agreement scale such as “Please indicate how strongly you agree that you are concerned about … Strongly disagree … Strongly agree”.

It is also important to ensure that the strength of statements used in scales is appropriate so that responses are distributed normally rather than being concentrated exclusively at the left- or right-hand end of the scale.

11. Quality control and the respondent experience

Include at least one control question to assess respondent attention, such as a “trap” question, a self-assessment item or a similar check. At the end of the survey, also include standardised questions about the completion experience, such as perceived response burden and satisfaction with the survey.

12. Planning the sample

Define the sampling approach in advance, whether probability or non-probability sampling, and estimate the required sample size according to the type of question—for example, a minimum of 30 observations, or in exceptional cases 10 for questions measured on ordinal scales—and according to the target subgroups, for which 10 to 30 observations are required per subgroup. Ensure that there are enough observations for reliable group-level analyses and, where appropriate, consider randomising individual sections of the questionnaire if doing so can reduce its length or respondent burden.

13. Checking questionnaire length

Use testing to determine how long the survey takes to complete. On this basis, assess whether respondents are likely to remain willing to answer and whether the number of questions should be reduced or adjusted.

14. Including control variables for analysis

Decide in advance which control variables or categories, such as gender or age, will be used to analyse and break down the results, and include these variables in the questionnaire.

15. Wording and a clear questionnaire flow

Use consistent and systematic wording throughout the questionnaire. Each question should measure a single dimension, use clear and familiar language, and rely on precise and concrete descriptions. Sentences should be simple and non-leading, while less familiar terms should be explained. Particular care should be taken to avoid negative wording, especially double negatives, and to formulate sensitive questions or questions about past behaviour appropriately.

At the same time, ensure that the questionnaire has a clear and logical structure. The flow should be straightforward and consistent, with questions grouped into coherent thematic sections and arranged in an appropriate order. Avoid abrupt or illogical shifts between topics so that the questionnaire remains easy to follow throughout.

Technical recommendations

Finally, the technical implementation of the survey is also important. The main technical recommendations to consider before data collection begins are outlined below.

1. Developing the questionnaire and copying text

It is advisable to build the questionnaire directly in 1KA rather than drafting it first in Word or Google Docs, as this helps to avoid problems caused by copying and additional formatting. If the questionnaire has nevertheless been prepared outside 1KA, remove formatting when copying and pasting the text into the questionnaire and check that the font, size, colour and other formatting are consistent across all questions.

2. Survey activation and duration

Before data collection begins, check that the survey has been activated and is therefore accessible to respondents. Also review the survey’s default time settings, such as automatic deactivation after three months, and adjust them where necessary to match the planned data-collection period.

Check that the progress indicator is enabled, as it can improve the user experience and increase the likelihood that respondents will complete the survey.

3. Checking questionnaire logic

Ensure that all questions are displayed correctly according to the conditions (IF), blocks and loops that have been set up, as errors in the questionnaire logic can result in questions being omitted or displayed incorrectly.

4. Preventing multiple submissions

Take steps to prevent the same respondent from completing the survey more than once, for example by using 1KA invitations, cookies or other access restrictions.

5. Archiving the questionnaire

Before making major changes to the questionnaire, create an archived copy so that the earlier version is preserved and can be restored if necessary. Once the questionnaire has been finalised, archive the final version as well.

6. Running the automated diagnostic check

In 1KA, an automated diagnostic tool is available under the “Testing” tab. It highlights potential errors and provides recommendations based on established methodological guidelines for high-quality question design.

7. Progress indicator and the “Continue later” option

For longer surveys, it is useful to enable the “Continue later” feature, which allows respondents to pause the questionnaire and use a link to resume later from the page where they stopped. A progress indicator can also be helpful, as it shows respondents how much of the survey they have already completed.

Conclusion

We have brought together the key recommendations for preparing a survey questionnaire, ranging from basic formal requirements to methodological and technical decisions. The overall aim is to ensure that the questionnaire is clear, logically structured and suitable for analysis, while placing as little burden as possible on respondents.