Knowledge, Perceptions and Reporting Practices of Theoretical Design in Causal Observational Epidemiological Studies on the Role of Antibiotic Use in the Occurrence of Asthma in Children

Background

Good research is driven by study design, encompassing theoretical design, design of data collection and design of data processing (‘analysis’). The theoretical design of a study is the translation of the research question into a research object. In epidemiological research, the object is designed in order to take into account the directionality of the research question. The key elements of the theoretical design depend on the nature (etiologic, diagnostic, prognostic or intervention related) of the functional relationship.^1,2 For example, the functional relationship for a diagnostic research question would be between current prevalence of the target illness and the current diagnostic profile (“current” referring to the moment of diagnostic probability setting). Key elements of theoretical design are: measure of occurrence, case (event or state) definition, conceptualization (and operationalization) of the exposure, temporal relation between outcome and exposure and confounders and effect modifiers taken into account. For the purpose of statistical management, the functional relationship is then expressed as an occurrence function in mathematical terms. The theoretical design is completed with a prespecified (designed) domain (ie, that part of the theoretical population for which the results are relevant and the context). The theoretical design is essential (the backbone, the motivation) for the design of data collection, the design of data processing and the interpretation of the study results. It is therefore to be expected that apart from a research question, also the theoretical design is explicitly reported in an article.

Scientific knowledge in medicine is expanded by confronting existing evidence with new empirical evidence. Relevant research is therefore replicated taking into account the eventual weaknesses of previous studies. It is to be expected that studies investigating the same exposure–outcome relationship yield slightly different results. Less expected is that study results are contradicting or that differences between study results are large. Contradicting results are usually explained by biases such as residual confounding, information bias, selection bias,… or by unobserved effect modification. Often, design issues are only presented as a weakness (typically without discussing the potential impact on the study results) when the research is not a randomized-controlled trial (RCT), the recommended design to provide evidence of efficacy of interventions^3,4 When conducting a RCT is not feasible (for causal research), the current recommendation is that the next best option is to conduct a cohort study. Eventually, a case-control study can be considered as an alternative, being that a case-control study can be interpreted as an efficient version of a cohort study.⁵ This point of view, inspired by interventional causal research, is considered as a paradigm for any causal research whether interventional or observational or whether it is aimed at studying causal aspects of an intervention or aimed at causally explaining the genesis of an illness. However, these two types of causality are different and should lead to a different theoretical design. So far, theoretical design is (to our knowledge) never considered in the discussion of the study results.

In our critical appraisal on the use and reporting of theoretical design in studies on the relationship between asthma occurrence and early life antibiotic use, none of the 63 articles reviewed reported a theoretical design and even key elements of theoretical design were not consistently reported.⁶ This underreporting could have several reasons, which could not be traced solely based on the critical appraisal: was the theoretical design conceptualized without reporting it, was it not conceptualized at all or was it conceptualized and reported but as a consequence of reviewers’ comments they decided to remove it from the manuscript before publication?

To gain insight into the role of theoretical design in the entire research process leading to the publication of an article, we decided to informally question the corresponding authors of a selection of the reviewed articles in our previous work. The objective was to appraise the perceived knowledgeability of theoretical design among the authors of these articles. From the 53 articles, for which we could formulate a theoretical design,⁶ we selected 30 articles (20 most recent and 10 less recent, leaving a gap of approximately 10 years in between). The oldest articles were not selected because of practical reasons (eg, traceability of the authors,…). The theoretical designs that were assigned in consensus in the previous work,⁶ can be consulted online: https://zenodo.org/record/3562255#.YVLhUH2xXIU. Other details concerning the questionnaire can be consulted in Appendix 1.^7–10,14

Perception of Knowledgeability of the Concepts of Theoretical Design, Causal Theory, Confounding and Effect Modification Among the Authors

Fifteen authors filled out the questionnaire. The questionnaire of one author was omitted, because he/she specifically requested not to quote the answers.

Knowledgeability was assessed with a 5-level Likert scale (5: extremely knowledgeable; 4: moderately knowledgeable; 3: somewhat knowledgeable; 2: slightly knowledgeable; 1: not at all knowledgeable). Most authors considered themselves to be moderately to extremely knowledgeable with the concepts of confounding and effect modification, whereas the perceived knowledgeability of causal theory and theoretical design was more diverse. Based on the sum scores for the perception of knowledgeability of theoretical design and causal theory, four groups can (arbitrarily) be distinguished.

Group 1: Three authors perceiving themselves to be extremely knowledgeable with the concepts of theoretical design and causal theory, with a sum score of 10.

Group 2: Six authors perceiving themselves to be slightly to moderately knowledgeable with the concepts of theoretical design and causal theory, with a sum score of 6–9.

Group 3: Four authors perceiving themselves to be not at all to slightly knowledgeable with the concept of theoretical design and slightly to moderately knowledgeable with the concept of causal theory, with a sum score of 3–5.

Group 4: One author perceiving him/herself to be not at all knowledgeable with all concepts except for confounding, with a sum score of 2.

The authors were asked to formulate a research question for their study and to translate this research question into a theoretical design (including the occurrence function and the domain). To have an idea about their opinion on the directionality of their (implicit) research question, the authors were also asked to select the occurrence function behind their study from one of the seven presented options: (1) Current prevalence of asthma as a function of past exposure to antibiotics; (2) Current incidence of asthma as a function of past exposure to antibiotics; (3) Current prevalence of asthma as a function of current exposure to antibiotics; (4) Current incidence of asthma as a function of current exposure to antibiotics; (5) Future prevalence of asthma as a function of current exposure to antibiotics; (6) Future incidence of asthma as a function of current exposure to antibiotics and (7) Other (specify). In group 1, only one author formulated a research question (with all three essential elements). The same author was able to translate this research question into a theoretical design. Among the three authors, one selected “current occurrence as a function of past exposure”. The remaining author selected “future occurrence as a function of current exposure”. In group 2, only one author formulated a research question and none were able to formulate a theoretical design for their study. All authors selected “current occurrence as a function of past exposure”. In group 3, only one author formulated a research question and none were able to formulate a theoretical design. All authors selected “current occurrence as a function of past exposure”. The author in group 4 did not formulate a research question nor a theoretical design. The author selected “future occurrence as a function of current exposure”.

Overall Appreciation of the Answers of the Authors

The anonymized answers of the authors to the questions (per group) can be consulted in Appendix 2. The vast majority of the authors (all except for one) did not formulate an occurrence function. This was not entirely surprising, as more than half of the authors did not formulate a research question. The vast majority (all except two) selected “current occurrence as a function of current/past exposure” from the presented options. However, when comparing this with the theoretical design (occurrence function) deduced from the article based on the reported information, half of these authors conducted their study based on “future occurrence as a function of current exposure”. An explanation for this could be the above mentioned preference to conduct a “cohort study” when conducting a RCT is not feasible. No major differences were observed in the answers to the questions between the groups.

Domain seems not to be a known concept. Apparently this is not a commonly used term. Moreover, several definitions can be found for domain. However, domain is an essential part of the theoretical design. Designing the domain, ie, defining that part of the theoretical population for which the results are relevant in an as well-defined setting (context), will guide an appropriate selection of the study population (eg: in intervention research, the domain refers to the population for which the drug is indicated).

Most authors agreed with the comments of the reviewers and for the few that did not agree, changes made to the manuscript before publication were not related to the reporting of a theoretical design. Also, in all studies but one, an epidemiologist was involved.

Comment

Even though some of the corresponding authors perceive themselves to be knowledgeable with the concept of theoretical design, this cannot be deduced from what is reported in their article. The existing guidelines, such as “the Strengthening the Reporting of Observational Studies in Epidemiology” (STROBE) and “Responsible Epidemiological Research Practice” (RERP), acknowledge the importance of transparent reporting.^11,12 However, though STROBE requests the reporting of key aspects of study design, the guideline does not address in detail what these key aspects are, probably under the assumption that these are well known among researchers.

In medical science, research is often conducted by health professionals lacking a formal training for conducting scientific research. These professionals might have been introduced to scientific research in some of their courses within their basic training, and obviously this introduction is more than appropriate for the qualifications needed to become a good practitioner. However, in order to become a good researcher conducting high-quality scientific research, an in-depth training is essential.¹³ Moreover, it appears that merely the presence of an epidemiologist in the team of researchers does not guarantee the appropriate transparency of the report.

We call upon the epidemiological community to reflect on what we observed. We attempted to get an impression of knowledgeability with a few questions only. Should the researchers’ knowledgeability of basic research concepts (such as formulating an appropriate research question, translating this question in an appropriate theoretical design, designing a procedure for data collection matching the theoretical design,…) be investigated in a more formal way? In the Netherlands, training programs for epidemiological researchers are not only accredited by the government but also by the Society for Epidemiology (VvE). Should this example be followed, generalized?

We are convinced that the concepts presented in this commentary should be well known and understood, implemented and explicitly reported by researchers conducting medical scientific research. Beyond that, guidelines for reporting could be more explicit in defining what the key elements of study design are. Appropriate reporting of theoretical design probably will increase the emphasis on this part of study design and without doubt consequently improve the quality of medical scientific research (data collection, data management and reporting).

Conclusion

There is diversity in how authors perceive themselves to be knowledgeable with the concept of theoretical design. This is not reflected in the answers of the authors, since the vast majority did not formulate an appropriate theoretical design. Almost all authors selected “current occurrence as a function of past exposure” for the directionality of their research question when presented with options. However, half of these authors conducted their study based on “future occurrence as a function of past exposure”. The reporting of theoretical design in the articles was not influenced by the peer review process. Changes to the manuscripts were not related to the reporting of theoretical design. Basic epidemiological concepts should be well known among researchers conducting medical scientific epidemiological research. We call on the epidemiological community to reflect on the current practice in causal observational epidemiologic research and on the importance of theoretical design both in research practice and training.