Sociologický časopis / Czech Sociological Review 2010, 46(6): 1011-1034 | DOI: 10.13060/00380288.2010.46.6.06

Approaching Quality in Survey Research: Towards a Comprehensive Perspective

Jindřich Krejčí
Institute of Sociology AS CR

The article has two goals: (1) bring attention to the problem of inappropriate treatment of survey data quality issues in the social sciences, and (2) introduce the basic principles of contemporary approaches to survey quality. If quality evaluation focuses solely on sampling error, most aspects of data quality are ignored and surveys are assumed to have 'ideal' statistical characteristics that are rarely attainable in the pragmatic world of survey fieldwork. A complex overview of the entire process of data collection provides a more solid foundation for evaluating data quality. Under this approach, quality is ensured by controlling the whole survey process. Accuracy, which is commonly elaborated using the concept of survey error, ceases to be the only dimension of quality. Nevertheless, this data quality component is crucial for data analysis and statistical testing. A comprehensive approach to survey data quality requires us to take account of complex sample designs when evaluating sampling error and to identify and distinguish between different dimensions of nonsampling error. Analysts who are not directly involved in data collection have limited ability to obtain information necessary for data quality evaluation. There are two types of quality standards: administratively imposed standards (ISO20252:2006) and the technical and ethical criteria of professional associations (e.g. ICC/ESOMAR, AAPOR/WAPOR, SIMAR). These help break this information barrier between data producers and data users.

Keywords: survey research methods, survey accuracy, total survey error, quality standards

Published: December 1, 2010Show citation

ACS AIP APA ASA Harvard Chicago IEEE ISO690 MLA NLM Turabian Vancouver
Krejčí, J. (2010). Approaching Quality in Survey Research: Towards a Comprehensive Perspective. Sociologický časopis / Czech Sociological Review, 46(6), 1011-1034. doi: 10.13060/00380288.2010.46.6.06.
Download citation
  • BibTeX (.bib)
  • Bookends (.ris)
  • EasyBib (.ris)
  • EndNote (.enw)
  • EndNote 8 (.xml)
  • ISI WoS (.isi)
  • Medlars (.medlars)
  • Mendeley (.ris)
  • MODS (.xml)
  • Papers (.ris)
  • RefWorks (.txt)
  • RefManager (.ris)
  • RIS (.ris)
  • MS Word (.xml)
  • Zotero (.ris)
    • Open full article

    References

    1. AAPOR. 1997. Best Practices for Survey and Public Opinion Research. Deerfield: The American Association for Public Opinion Rsearch. Retrieved 5 August 2010 (http://www.aapor.org/Best_Practices/1480.htm).
    2. AAPOR. 2009. Standard Definitions. Final Dispositions of Case Codes and Outcome Rates for Surveys. Deerfield: The American Association for Public Opinion Research. Retrieved 5 August 2010 (http://www.aapor.org/AM/Template.cfm?Section=Standard_Definitions&Template=/CM/ContentDisplay.cfm&ContentID=1819).
    3. Andersen, R., J. Kasper and M. Frankel (eds.). 1979. Total Survey Error. San Francisco: Jossey-Bass.
    4. Babbie, Earl. 2004. The Practice of Social Research. 10th ed. Belmont: Thomson Learning.
    5. Bell, John F. 1991 'Big Is Not Necessarily Beautiful in Survey Design: Measurement Error and the APU Science Survey.' The Statistician 40 (3: Special Issue: Survey Design, Methodology and Analysis 2): 291-300.
    6. Biemer, Paul P. and Lars E. Lyberg. 2003. Introduction to Survey Quality. Hoboken: John Wiley & Sons. Go to original source...
    7. Blalock, Hubert M., Jr. 1989. 'The Real and Unrealised Contributions of Quantitative Sociology.' American Sociological Review 54: 447-460. Go to original source...
    8. Brehm, John. 1993. The Phantom Respondents: Opinions Surveys and Political Representation. Ann Arbor: University of Michigan Press. Go to original source...
    9. Colledge, Michael and Mary March. 1997. 'Quality Policies, Standards, Guidelines, and Recommended Practices at National Statistical Agencies.' Pp. 501-522 in Survey Measurement and Process Quality, edited by L. E. Lyberg, P. Biemer, M. Collins, E. D. de Leeuw, C. Dippo, N. Schwarz and D. Trewin. New York: John Wiley. Go to original source...
    10. Curtin, R., S. Presser and E. Singer. 2000. 'The Effects of Response Rate Changes on the Index of Consumer Sentiment.' Public Opinion Quarterly 64 (4): 413-428. Go to original source...
    11. Dalenius, Tore. 1977. 'Bibliography of Nonsampling Errors in Surveys.' International Statistical Review 45: 71-81, 181-197.
    12. Dalenius, Tore. 1981. 'The Survey Statistician's Responsibility for both Sampling and Measurement Errors.' Pp. 17-29 in Current Topics in Survey Sampling, edited by D. Krewski, R. Platek and J. N. Rao. New York: Academic Press. Go to original source...
    13. Denis, Daniel J. 2003. 'Alternatives to Null Hypothesis Significance Testing.' Theory & Science 14 (1). Retrieved 17 August 2010 (http://theoryandscience.icaap.org/content/vol4.1/02_denis.html).
    14. ESOMAR. 2008. ICC/ESOMAR International Code of Marketing and Social Research Practice. ESOMAR World Research Codes and Guidelines. Amsterdam: ESOMAR. Retrieved 5 August 2010 (http://www.esomar.org/uploads/pdf/professional-standards/ICCESOMAR_Code_English_.pdf).
    15. ESOMAR. 2009. Notes on How to Apply the ICC/ESOMAR International Code of Marketing and Social Research Practice. ESOMAR World Research Codes and Guidelines. Amsterdam: ESOMAR. Retrieved 5 August 2010 (http://www.esomar.org/uploads/professional_standards/guidelines/ESOMAR_Codes&Guidelines_NotesHowToApplyCode.pdf).
    16. ESOMAR/WAPOR. 2009. ESOMAR/WAPOR Guide to Opinion Polls and Published Surveys. ESOMAR World Research Codes and Guidelines. Amsterdam: ESOMAR. Retrieved 5 August 2010 (http://www.esomar.org/uploads/professional_standards/guidelines/WAPOR-ESOMAR_Guidelines.pdf).
    17. Eurostat. 2002. Quality in the European Statistical System - The Way Forward. Luxembourg: Office for Official Publications of the European Communities. Retrieved 5 August 2010 (http://epp.eurostat.ec.europa.eu/cache/ITY_PUBLIC/KS-45-02-814/EN/KS-45-02-814-EN.PDF).
    18. Eurostat. 2003. Methodological Documents - Definition of Quality in Statistics. Working Group on Assessment of Quality in Statistics, Luxembourg, 2-3 October 2003. Luxembourg: Eurostat. Retrieved 5 August 2010 (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/ess%20quality%20definition.pdf).
    19. Eurostat. 2009. ESS Handbook for Quality Reports. Luxembourg: Office for Official Publications of the European Communities. Retrieved 5 August 2010 (http://epp.eurostat.ec.europa.eu/cache/ITY_OFFPUB/KS-RA-08-016/EN/KS-RA-08-016-EN.PDF).
    20. Gigerenzer, G., S. Krauss and O. Vitouch. 2004. 'The Null Ritual: What You always Wanted to Know about Significance Testing but Were Afraid to Ask.' Pp. 391-408 in The Sage Handbook of Quantitative Methodology for the Social Sciences, edited by D. Kaplan. Thousand Oaks, CA: Sage.
    21. Gill, J. 1999. 'The Insignificance of Null Hypothesis Significance Testing.' Political Research Quarterly 52 (3): 647-674. Go to original source...
    22. Groves, Robert M. [1989] 2004. Survey Errors and Survey Costs. 2nd ed. Hoboken: John Wiley & Sons.
    23. Groves, Robert M. 2006. 'Nonresponse Rates and Nonresponse Bias in Household Surveys.' Public Opinion Quarterly 70 (5): 646-675. Go to original source...
    24. Groves, R. M. and M. P. Couper. 1998. Non-Response in Household Interview Surveys. New York: Wiley. Go to original source...
    25. Groves, Robert M., Floyd J. Fowler, Mick P. Couper, James M. Lepkowski, Eleanor Singer and Roger Tourangeau. 2004. Survey Methodology. Hoboken: John Wiley & Sons.
    26. Hansen, M. H., W. N. Hurwitz and L. Pritzker. 1967. 'Standardization of Procedures for the Evaluation of Data: Measurement Errors and Statistical Standards in the Bureau of the Census.' Bulletin of the International Statistical Institute (Proceedings of the 36th Session): 49-66.
    27. ISO. 1994. ISO 8402:1994. Quality - Vocabulary. Geneva: International Organization for Standardization.
    28. Juran, J. M. and Frank M. Gryna. 1993. Quality Planning and Analysis: From Product Development Through Use. 3rd ed. New York: McGraw-Hill.
    29. Kaase, Max (ed.). 1999. Qualitätskrieterien der Umfrageforschung - Quality Criteria for Survey Research. Berlin: Akademie Verlag GmbH.
    30. Keeter, S., C. Miller, A. Kohut, R. M. Groves and S. Presser. 2000. 'Consequences of Reducing Non-response in a National Telephone Survey.' Public Opinion Quarterly 64 (2): 125-148. Go to original source...
    31. Kish, Leslie. 1965. Survey Sampling. New York: John Wiley & Sons.
    32. Laliberté, Lucie, Werner Grünewald and Laurent Probst. 2004. Data Quality: A Comparison of IMF's Data Quality Assessment Framework (DQAF) and Eurostat's Quality Definition. IMF Reference Material. Retrieved 5 August 2010 (http://dsbb.imf.org/vgn/images/pdfs/dqaf_eurostat.pdf).
    33. Lessler, Judith T. and William D. Kalsbeek. 1992. Nonsampling Errors in Surveys. New York: John Wiley & Sons.
    34. Lynn, Peter. 2001. A Quality Framework for Longitudinal Studies. Reference Document of the ESRC National Strategy Committee for Longitudinal Studies. Colchester: ESRC. Retrieved 5 August 2010 (http://natstrat.essex.ac.uk/pages/documentation/reference/framework.doc).
    35. Martin, E. 2004. 'Presidential Address: Unfinished Business.' Public Opinion Quarterly 68 (3): 439-450. Go to original source...
    36. Merton, Robert K. and Paul F. Lazarsfeld (eds.). 1950. Continuities in Social Research: Studies in the Scope and Method of 'The American Soldier'. Glencoe: Free Press.
    37. Morganstein, David and David A. Marker. 1997. 'Continuous Quality Improvement in Statistical Agencies.' Pp. 475-500 in Survey Measurement and Process Quality, edited by L. E. Lyberg, P. Biemer, M. Collins, E. D. de Leeuw, C. Dippo, N. Schwarz and D. Trewin. New York: John Wiley.
    38. Noelle, Elisabeth. 1963. Umfragen in der Massengesellschaft. Einführung in die Methoden der Demoskopie. Reinbek bei Hamburg: Rowohlt Tagensbuch Verlag.
    39. OECD. 2002. Quality Framework for OECD Statistics. Paris: OECD. Retrieved 5 August 2010 (http://www.oecd.org/document/43/0,3343,en_2649_33715_21571947_1_1_1_1,00. html).
    40. OECD. 2007. OECD Principles and Guidelines for Access to Research Data from Public Funding. Paris: Organisation for Economic Co-Operation and Development. Retrieved 5 August 2010 (http://www.oecd.org/dataoecd/9/61/38500813.pdf).
    41. SIMAR. 1996-2008. Kvalitativní standardy (Quality standards). On-line version. Retrieved 5 August 2010 (http://www.simar.cz/standardy-kvality/kvalitativni-standardy/index.php).
    42. Smith, Tom W. 2002. 'Developing Nonresponse Standards.' Pp. 27-40 in Survey Nonresponse, edited by Robert M. Groves, Don A. Dillman, John L. Eltinge and Roderick J. A. Little. New York: John Wiley & Sons.
    43. Soukup, Petr and Ladislav Rabušic. 2007. 'Několik poznámek k jedné obsesi českých sociálních věd - statistické významnosti.' (Some Notes on the Obsession of the Czech Social Sciences with Statistical Significance) Sociologický časopis/Czech Sociological Review 43 (2): 379-395.
    44. Statistics Canada. 2002. Statistics Canada's Quality Assurance Framework. Ottawa: Statistics Canada. Retrieved 5 August 2010 (http://www.statcan.gc.ca/pub/12-586-x/12-586-x2002001-eng.pdf).
    45. Stoop I. A. L and F. Louwen. 2000. 'Do Respondents Differ?' Paper presented at the Eleventh International Workshop on Survey Nonresponse, 27-29 September, Budapest, Hungary.
    46. Tourangeau, Roger, Lance J. Rips and Kenneth Rasinski. 2000. The Psychology of Survey Response. New York: Cambridge University Press. Go to original source...
    47. Zahradníček, Stanislav. 2006. Komentář k ČSN 20252:2006 'Výzkum trhu a veřejného mínění a sociální výzkum.' (A Commentary on the ČSN 20252:2006 'Market, Public Opinion and Social Research) Prague: Český normalizační institut.

    This is an open access article distributed under the terms of the Attribution-NonCommercial 4.0 International (CC BY-NC 4.0), which permits use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.


    Return to the content