Back to Journals » Advances in Medical Education and Practice » Volume 6

Five-year review of an international clinical research-training program

Authors Suemoto CK, Esmail S, Correa P, Khawaja F, Jerves T, Pesantez L, Germani AC, Zaina F, Soares dos Santos Junior AC, Ferreira R, Singh P, Paulo J, Matsubayashi S, Vidor L, Andretta G, Tomas R, Illigens B, Fregni F

Received 22 April 2014

Accepted for publication 28 July 2014

Published 1 April 2015 Volume 2015:6 Pages 249—257

DOI https://doi.org/10.2147/AMEP.S66627

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 5

Editor who approved publication: Dr Md Anwarul Azim Majumder



Claudia Kimie Suemoto,1,2 Sherine Ismail,1,3 Paulo César Rodrigues Pinto Corrêa,1,4,5 Faiza Khawaja,1,6 Teodoro Jerves,1 Laura Pesantez,1 Ana Claudia Camargo Gonçalves Germani,1,7 Fabio Zaina,1,8 Augusto Cesar Soares dos Santos Junior,1,9,10 Ricardo Jorge de Oliveira Ferreira,1,11 Priyamvada Singh,1,12 Judy Vicente Paulo,1,13 Suely Reiko Matsubayashi,1,14 Liliane Pinto Vidor,1,15 Guilherme Andretta,1,16 Rita Tomás,1,17 Ben MW Illigens,1,18 Felipe Fregni1,18,19

1Collaborative Learning in Clinical Research Program, Principles and Practice of Clinical Research (PPCR), Department of Physical Medicine and Rehabilitation, Spaulding Rehabilitation Hospital and Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA; 2Discipline of Geriatrics, Department of Internal Medicine, University of São Paulo Medical School, São Paulo, Brazil; 3King Abdullah International Medical Research Center, King Saud bin Abdulaziz University for Health Sciences, Pharmaceutical Care Department, King Khalid Hospital, NGHA, Jeddah, Saudi Arabia; 4Discipline of Internal Medicine and Medical Semiology, Department of Internal Medicine, Federal University of Ouro Preto (UFOP) Medical School, Ouro Preto, Brazil; 5Discipline of Pneumology, Department of Internal Medicine, Centro Universitário de Belo Horizonte (Uni-BH), Belo Horizonte, Brazil; 6Canadian Centre for Advanced Eye Therapeutics, Mississauga, ON, Canada; 7Department of Preventive Medicine, University of São Paulo Medical School, São Paulo, Brazil; 8Italian Scientific Spine Institute (ISICO), Milan, Italy; 9Hospital Osvaldo Rezende Franco, Betim, Brazil; 10Nucleo de Avaliação de Tecnologia em Saude, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil; 11Department of Rheumatology, Centro Hospitalar e Universitário de Coimbra, Coimbra, Portugal; 12Department of Internal Medicine, Saint Vincent Hospital, Worcester, MA, USA; 13Portuguese Institute of Oncology, Coimbra, Portugal; 14Acupuncture Center, Orthopedics and Traumatology Institute, University of Sao Paulo Medical School, São Paulo, Brazil; 15Department of Medical Science, Faculty of Medicine, University Federal of Rio Grande do Sul, Porto Alegre, Brazil; 16Quintiles Transnational, São Paulo, Brazil; 17Department of Physical Medicine and Rehabilitation, Hospital de Curry Cabral, Centro Hospitalar de Lisboa Central, EPE, Lisbon, Portugal; 18Department of Neurology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA; 19Department of Physical Medicine and Rehabilitation, Spaulding Rehabilitation Hospital and Massachusetts General Hospital, Boston, MA, USA

Abstract: The exponential increase in clinical research has profoundly changed medical sciences. Evidence that has accumulated in the past three decades from clinical trials has led to the proposal that clinical care should not be based solely on clinical expertise and patient values, and should integrate robust data from systematic research. As a consequence, clinical research has become more complex and methods have become more rigorous, and evidence is usually not easily translated into clinical practice. Therefore, the instruction of clinical research methods for scientists and clinicians must adapt to this new reality. To address this challenge, a global distance-learning clinical research-training program was developed, based on collaborative learning, the pedagogical goal of which was to develop critical thinking skills in clinical research. We describe and analyze the challenges and possible solutions of this course after 5 years of experience (2008–2012) with this program. Through evaluation by students and faculty, we identified and reviewed the following challenges of our program: 1) student engagement and motivation, 2) impact of heterogeneous audience on learning, 3) learning in large groups, 4) enhancing group learning, 5) enhancing social presence, 6) dropouts, 7) quality control, and 8) course management. We discuss these issues and potential alternatives with regard to our research and background.

Keywords: education, distance learning, biomedical research, critical thinking, e-learning

Introduction

Improvements in clinical research methods have effected widespread support of evidence-based medicine.1,2 Although evidence-based medicine aims to improve clinical care, it challenges clinicians and researchers.3,4 Moderate-to-advanced knowledge of the methodology is necessary to apply the resulting findings into practice and assess the cost-effectiveness of new interventions.5

The success of a research-methodology program depends on the course format; sole transfer of content is not adequate. For instance, knowing the meaning of “stratified block randomization” is insufficient; it is critical to understand the impact of using this method and to analyze potential biases that could affect the results. Training should develop critical thinking skills,6 wherein subjects are self-correcting and assessing the rationality of their thinking in an iterative process.7

According to Stanovich et al, the process of decision-making can be divided into two systems.8 System 1 is associated with fast, intuitive, and usually unconscious thinking,9 and is used for complex cognitive tasks, such as making quick decisions, based on a problem.9,10 System 1 is evident when assessing the question: “Which statistical test do you use to compare two means?” The answer will simply depend on basic knowledge of the criteria that are used to select a statistical test (eg, type of variable, study outcome, and possible need to adjust for covariates). Although system 1 is the principal operating system, it is frequently associated with bias8 and informal logical fallacies, requiring further correction and override by system 2.

Conversely, system 2 represents slow, analytical, and conscious thinking. For example, if a study demonstrates an association between coffee consumption and lung cancer, system 1 would quickly advise coffee drinkers to stop drinking coffee, whereas system 2 could infer that coffee might be a confounder in the relationship between smoking and lung cancer. Therefore, it is crucial to encourage critical thinking and reasoning skills through an effective learning program to train system 2 processes.

International distance-learning clinical research program

Our goal was to design a course to develop critical thinking skills, and it was even more challenging to achieve this aim in an international distance-learning setting. The development of the worldwide web has helped tremendously in overcoming this hurdle, enabling collaboration and interaction between students.11,12 Our 9-month course was offered every year from 2008 to 2012 to health care professionals worldwide, for which English was the official language.13

The course comprised 24 topics that were grouped into four modules. Module 1 introduced the basics of clinical research and clinical trial design; in module 2, statistical concepts and the most standard statistical approaches were taught; module 3 addressed the practical aspects of clinical research; and module 4 covered more specific study design-related topics. Each topic was covered over 1 week, during which students prepared the material in advance until they met during a live session at the end of the week.

The course uses a blended format of synchronous and asynchronous methods of interaction. The asynchronous interactions take place in an online forum, in which participants are divided into groups of 20. The goal is to promote an interstudent debate to foster collaborative learning. We use phpBB® software, which allows posting, the use of attachments, polls, drafting, pictures, real-time statistics, forum pruning, moderation, and email notification.14 Forum participation is a graded activity. In 2012, approximately 36,536 posts were made, which represents an average of five posts per participant per lecture. Another synchronous type of interaction is the chat, which takes place in predetermined live “office hours” during the week, hosted by teaching assistants (TAs; primarily former students of the course). Finally, through weekly assignments, participants must answer questions that are related to important articles in the field of clinical research.

At the end of each week, a synchronous interaction takes place during 3-hour classes that can be attended live or remotely via video conference.15 The web-conference platform Adobe® Connect™ supports the synchronous interaction (with a choice of customized layouts that offer interaction via “pods” that allow, for example, polling, questions and answers [Q&As], chats, group discussion, and notes); four versions of the software were used, updated to match the most recent version that was available. The initial 45 minutes of each 3-hour class is allocated to a hypothetical case discussion, followed by a 45-minute lecture by a faculty member in Boston, MA, a 45-minute Q&A session, and another 45 minutes of interactive discussion. The classes are recorded and hosted to allow for repeated viewing.

In parallel to the regular class schedule, participants develop a hypothetical grant project with their group members using online collaborative tools (Wiki engine). Every year, the number of participants has increased progressively (Figure 1). Although the course has been successful, expanding worldwide, we decided to assess our main challenges to redesign certain aspects of our program.

Figure 1 Number of students per year and number of countries where they came from (2008–2012).

The aim of this paper is to describe and analyze the challenges and possible solutions of this course. We have identified key points from student and faculty feedback, conducting an informal group discussion and a narrative review about these issues. We illustrate some aspects with data from 2012 participants (Table 1).

Table 1 Student characteristics in 2012 and distribution of participants per site
Abbreviations: SD, standard deviation; IQR, interquartile range.

Challenges and possible solutions

Student engagement and motivation

The course is divided into two halves, separated by a 6-week break, when students may review previous modules and work on the group project (hypothetical grant application). However, after the break, students tend to be less motivated. Therefore, we want to discuss options to increase their engagement and motivation.16 According to Keller’s attention/relevance/confidence/satisfaction motivation model, these four elements are required to keep students motivated.17 The content must be relevant, because adults learn best when they recognize the need for the information. Moreover, the learner has to understand the content (Table 2). From our experience, relatively high dropout rates occur during the statistical module, because statistics is perceived as a difficult subject, and in fact students may not feel satisfied with their performance, thus dropping out the course.

Table 2 Major challenges faced in distance-learning courses

Instructor characteristics are also a key component in promoting student motivation. Five characteristics were identified: expertise, empathy, enthusiasm, clarity, and cultural responsiveness.18 Portraying enthusiasm and engagement during online lectures is challenging, because the video might filter important emotional cues. Nevertheless, timely feedback from instructors and weekly emails with suggested schedules and reminders were useful motivating tools in our course. The students’ confidence levels also rose once they became more familiar with the online course site through an introductory session, which provided them with an orientation of the online course site setup and design and recorded instructional videos, providing guidance for each required task. Confidence was also fostered through stepwise construction of knowledge, online quizzes, and self-assessment opportunities. Therefore, satisfaction was facilitated by giving attention, relevance, and confidence to students.

Several methods to enhance motivation and engagement in distance-learning courses are suggested (Table 3), some of which are already used in our course, whereas others are being developed.

Table 3 Methods to enhance motivation and engagement and their use in our course
Abbreviation: TAs, teaching assistants.

Impact of heterogeneous audience on learning

Our course has a diverse audience. In 2012, 180 (55.1%) participants had already graduated, and 59 (18.0%) had a PhD. On this occasion, we had participants from 27 countries, mixed among 12 groups. In 2012, in addition to having onsite teaching centers in 10 countries, we had 64 (19.5%) participants who attended the lectures remotely via Adobe Connect from all over the world (Figure 2). Therefore, these cultural differences must be addressed to make the course meaningful for all students.

Figure 2 Countries from where students attended the course in 2012.

The diversity of students influences the rate of participation, level of expectation and satisfaction, and the degree of learning.1922 A survey study by Hannon and D’Netto23 assessed student satisfaction in various domains of an online course, comparing on-site versus international off-site students. On-site local students were more comfortable regarding organizational issues.23 Therefore, universal access and an inclusive design must be provided in relation to organizational issues. Although most students gave us positive feedback in the course assessment, individual support should be provided on a more regular basis.

Further, the effect of language must be considered in diverse international groups. Language proficiency strongly impacts learning, and nonnative English speakers, who represented nearly 75% of 2012 students, require more time to process reading assignments and submit their written contributions through the course website.24 Another student characteristic is computer literacy. Higher skills are associated with better performance,23,25 and having taken an online course before increases the likelihood of successfully passing the course.26 In 2012, 31 (9.4%) students reported having only basic computer knowledge, 139 (42.3%) declared intermediate computer expertise, and 159 (48.3%) were advanced users. Regarding previous participation in distance-learning courses, 197 (59.88%) had no such experience.

Another heterogeneous aspect of our audience is the differences in graduation backgrounds. Interprofessional education can help enhance collaborative and nonhierarchical relationships in effective teams.23 Besides physicians, previous participants have been nurses, pharmacists, audiologists, biologists, psychotherapists, and biostatisticians. Promoting interactions from the outset of the course in an intensive manner is important.

In terms of ethnicity, whereas Asians and Hispanics rated educational barriers more highly than Caucasians and African Americans, the difference was low and might have been confounded by other factors, such as previous access to online courses.26 Notably, some participants experienced prejudices due to their ethnicity, which interfered with their learning in traditional classroom environments.26 Therefore, online courses might be beneficial with regard to the inclusion of minority groups.

Physical issues that are related to aging, such as impairments in working memory, vision, and hearing, can affect learning.18 However, because the majority of our audience was aged under 65 years, physical limitations did not seem to be an issue in our course. Conversely, older professionals might be more available to take an online course than a traditional course. Being in a learning environment again, even if online, can be highly satisfactory to them.27 In summary, cultural, ethnic, and age-related factors in isolation can have minor effects for online learning; other individual factors that are associated with motivation and social interaction might be more important.

Learning in large groups

The weekly classes are challenging, given the large number of students. Low attendance low and uneven emotional engagement, lack of student preparedness, immediate feedback on students’ understanding and students’ metacognition, and dissatisfying learning outcomes are potential problems.28

One important issue regarding interaction is the use of appropriate tools. Increasing access and interactions for students during lectures is important;29 however, this is technically challenging for a 300-student course. Adobe Connect played a fundamental role during class activities through text chats between small groups or the entire group, audio access to all participants, notepads that summarize the main concepts, PowerPoint slide presentations with screen sharing, a whiteboard to develop explanations, and polls with key questions.

Another challenge is managing questions during the Q&A session. We receive 50–100 questions per lecture, and have time to address only 15 of them. Google Moderator is an interesting tool, as it allows students to vote on the best questions using an intuitive platform. Moreover, by including students in the process of choosing questions, they will be more motivated and engaged during the lecture. To adapt this system to our course, we created a system that combines the available components with the concepts of Google Moderator (Figure 2).

Although students learn as well or better through active learning techniques, they may feel less confident in their skills, because there is little formal presentation of information in active learning compared with traditional environments. In addition, with active learning methods, students might place less value on attending class. The use of a blended method that comprises lectures and active learning activities might be optimal.

Enhancing group learning

Although collaborative learning is possible in the asynchronous component of our course, it is challenging for live lectures. The use of a web-conferencing environment is a possible strategy to effect collaboration, but it is sometimes technically difficult.

Another important aspect is the group project. Students may use two strategies to complete it: cooperative (tasks are divided between students in each group and completed individually) and collaborative (tasks completed together through dialog and sharing of complementary skills).30

It is important to present this concept to students early to stimulate collaboration. Considering the collaborative dialog, Paulus31 opines that larger groups (more than ten participants) are more effective. In our course, groups of approximately 20 participants collaborate weekly to discuss course readings through the forum, and the level of interaction/collaboration across participants reaches its peak during the group project.

Assessing students’ attitudes toward group work is essential. Some students might display resistance, preferring traditional forms of learning.32 In this case, it may be useful to explain the benefits of group work and clarify the assessment criteria.

Various degrees of relationships can develop within groups and influence teamwork.33 Working together, discussing, and finally creating associations outside the course are ways of achieving progressive states of a relationship. Increased levels of community lead to greater participation in the class and vice versa.33

The selection of groups is also critical. This is less problematic in online courses than in face-to-face lectures, in which students may want to work with their friends. In fact, it is possible to choose groups randomly or select heterogeneous or homogeneous groups, based on common interests.34

Enhancing social presence

Social presence is important for enhancing learning. Students who prefer to work alone have less satisfaction with collaborative learning.27 Therefore, it may be useful to identify these students during the registration process.

Participants can enroll in our course as part of a site center or individually if a site is not accessible. Taking the course at a site center promotes interactions between site-center students and creates a more personal environment.35 Moreover, during the live lecture, the course instructor interacts primarily with participants who are at the sites. Our system can identify which students are present, facilitating cold calls to increase student participation and improve attention during the lectures.

In one-on-one interactions, the two people who are engaged in a conversation are constantly giving feedback to each other through immediate responses or body language. This lack of feedback in online environments hinders such interactions; therefore, the use of additional tools, such as emoticons, can address some of these limitations. Moreover, participants’ posts in the forum must be answered by other students; otherwise, if faculty start answering posts there, this may decrease students’ motivation to post.

There are other tools that enhance immediacy in online courses, such as using the first names of students during the discussion and summarizing the most interesting posts to the group.35 Finally, grading participation in the forum is useful to enhance participation, and is performed in two levels. First, the number and timing of posts are considered using a predetermined algorithm. Second, the TA evaluations use rubric grading that assesses quality and determines whether a certain post has promoted subsequent discussion.

Dropouts

On average, 30% of students drop out of our course, although this number decreased in 2012 to 17%. There are two peaks of dropouts: at the beginning of the course and during the statistical module. Our dropout rate is similar to other online courses, ranging from 20% to 50%, and is estimated to be 10%–20% higher than traditional live courses.36

According to Tyler-Smith, the main reasons for dropouts include lack of personal motivation, weak instructional design, lack of time, and learning.37 Because the online-learning audience might comprise primarily professional adults, dropouts in this setting are more indicative of the decision of an informed adult, assessing competing factors, such as work and personal life.

Another significant cause of dropouts is cognitive overload.37 In fact, this may explain the early peak of dropouts in our course, because it may be too challenging for some learners. Finally, technical challenges can also contribute to cognitive overload. We use several methods to reduce cognitive overload: 1) an introductory session 3 weeks before the start of the course to present the methods and grading system, and another workshop about the statistical software before the statistical module; 2) additional content support, such as tutorials on the website; and 3) a design in which the demands of the course are initially lower, increasing progressively over time.

Students can be classified into four categories of learning styles: auditory, kinesthetic, tactile, and visual. Participants with the auditory learning style were more likely to drop out of online courses.38 Therefore, presenting verbal information and promoting interactions during the live lectures are useful. Finally, previous academic success can also be a significant predictor of online persistence.

Additionally, it is important to monitor student participation and follow up if a student is absent from the course. One option is to use tools that predict students who are at risk of dropping out, based on patterns of participation, and implement early interventions to avoid it.38 We publish indicators of participation on the website weekly so that students can monitor their participation and adjust it accordingly. We noted a significant rise in participation after being more explicit with this method. Finally, measures to enhance motivation, as discussed in Table 1, are critical to decrease dropout rates.

Quality control

Online education has been growing rapidly. Technology is seductive and might give the impression of easy revenue, leading to lower-quality courses.39 Although quality control in online courses should follow the same basic principles of traditional learning, certain differences should be considered: time measurement, credential definition, documentation of student-learning outcomes, student attendance, and performance patterns.39

Some questionnaires are used to assess quality. One example is a detailed 60-item questionnaire on various aspects of online courses: flexibility, responsiveness and student support, learning, interaction, technical support, and student satisfaction.39 Technical support is an essential aspect in quality control in distance-learning courses: students should have constant assistance, because technology should not be a limiting factor.

We assess the quality of our course through a feedback questionnaire after each module that addresses technical, professional, and personal support to identify the level of satisfaction of participants. After each module, the course director meets with TAs and technical staff to discuss student comments and ways to improve the course.

Course management

Distance-learning courses are challenging to manage, because many organizational issues, such as time demands, quality measurement, marketing, and budgeting, must be addressed. Cost might be a major issue for online courses in the absence of experience or structure with this educational format. Usually, hidden costs are not fully appreciated, and an erroneous impression of the course profits is common. Costs include server, technology maintenance, course development, and instructor time. Collectively, these costs might be higher than in traditional settings.40

Another sensitive topic is how to manage private information. Restricted access to participants’ baseline characteristics, confidentiality of performance, and exam results for group TAs are some of the strategies that adapted in our course.

Moreover, time management is an important issue for students, faculty, and TAs.41 It is important to identify students who are experiencing time-management problems and help them, including those with minimal or no participation, poor organization, communication anxiety, technical difficulties, and excessive posting that bothers other students. TAs should also be guided and informed about improving time management to increase effectiveness and reduce anxiety.

With regard to course structure, the software should be easy to use, have a good visual layout to stimulate participation, and offer technical support.41 A successful distance-learning program must have continuous expert website support and testing of novel web tools. It is critical that students be prepared to handle technical difficulties.

Finally, marketing might also be a challenge. In our experience, cold-type marketing has little impact. Students enroll in our course usually after having heard of the experience from a previous student, which might be the consequence of the novelty of this educational method or the characteristics of our learners, usually busy professionals.

Conclusion

Knowledge transfer and exchange seem to be growing worldwide in health care, and constitute significant motivators of the success of distance-learning courses. However, distance-learning techniques are still under development.42 Although our experience confirms that most of our challenges are common to any educational program, such as student motivation and engagement, several issues are exclusive to online programs. Specific factors must be carefully considered in the redesign of our program (Table 4).

Table 4 Factors to be considered in the redesign of a collaborative learning program
Abbreviation: TAs, teaching assistants.

One of the main challenges of our course is the impact of the heterogeneous audience on learning. To overcome this issue, we will perform a better baseline assessment of student characteristics. In our registration, students will be required to answer several questions to determine the characteristics that might influence their performance, such as learning style. In addition, based on our experience and also from other established online courses, we will implement other changes in our design: implementing the possibility of taking notes on our website portal when participants are studying, making the navigation through our website smoother and increasing technical support, and improving the system to moderate questions during class.

Although we have faced some challenges, the number of participants has increased and the dropout rates have decreased over the years, possibly as a result of our growing experience with online education and better support for students and staff. Constant evaluation and redesign might elevate online courses as the leading method for adult education in the near future. Joint efforts will transform this format of adult education into an important and accessible method for developing critical thinking skills in research-methodology programs.

Acknowledgment

Funding for this study was provided by Principles and Practice of Clinical Research (PPCR), Department of Physical Medicine and Rehabilitation, Spaulding Rehabilitation Hospital and Massachusetts General Hospital, Harvard Medical School.

Disclosure

The authors report no conflicts of interest in this work.


References

1.

Whitley R, Rousseau C, Carpenter-Song E, Kirmayer LJ. Evidence-based medicine: opportunities and challenges in a diverse society. Can J Psychiatry. 2011;56(9):514–522.

2.

Lu YC, Li YC. How doctors practice evidence-based medicine. J Eval Clin Pract. 2013;19(1):44–49.

3.

Lenfant C. Shattuck lecture – clinical research to clinical practice – lost in translation? N Engl J Med. 2003;349(9):868–874.

4.

Zarin DA, Young JL, West JC. Challenges to evidence-based medicine: a comparison of patients and treatments in randomized controlled trials with patients and treatments in a practice research network. Soc Psychiatry Psychiatr Epidemiol. 2005;40(1):27–35.

5.

Brunoni AR, Tadini L, Fregni F. Changes in clinical trials methodology over time: a systematic review of six decades of research in psychopharmacology. PloS One. 2010;5(3):e9479.

6.

Kowalczyk N. Review of teaching methods and critical thinking skills. Radiol Technol. 2011;83(2):120–132.

7.

Behar-Horenstein LS, Schneider-Mitchell G, Graff R. Promoting the teaching of critical thinking skills through faculty development. J Dent Educ. 2009;73(6):665–675.

8.

Stanovich KE, Toplak ME, West RF. The development of rational thought: a taxonomy of heuristics and biases. Adv Child Dev Behav. 2008;36:251–285.

9.

Kahneman D. A perspective on judgment and choice: mapping bounded rationality. Am Psychol. 2003;58(9):697–720.

10.

Koenig CS, Platt RD, Griggs RA. Using dual-process theory and analogical transfer to explain facilitation on a hypothetico-deductive reasoning task. Psychol Res. 2007;71(4):495–502.

11.

Sandars J, Haythornthwaite C. New horizons for e-learning in medical education: ecological and Web 2.0 perspectives. Med Teach. 2007;29(4):307–310.

12.

Sanderson PE. E-learning: strategies for delivering knowledge in the digital age. Internet High Educ. 2002;5(2):185–188.

13.

Imamura M, Hsing WT, Platt LD, Fregni F. How to develop research capacity using a collaborative training approach: the International Society of Physical and Rehabilitation Medicine (ISPRM) international training program experience. J Rehabil Med. 2009;41(4):295–296.

14.

Topçu A. Perceived learning in the online asynchronous discussion forum provided by PHPBB. In: McFerrin K, editor. Proceedings of Society for Information Technology and Teacher Education International Conference 2008. Chesapeake (VA): AACE; 2008:3180–3187.

15.

Singh H. Building effective blended learning programs. Educ Technol. 2003;43(6):51–54.

16.

Kusurkar RA, Croiset G, Mann KV, Custers E, Ten Cate O. Have motivation theories guided the development and reform of medical education curricula? A review of the literature. Acad Med. 2012;87(6):735–743.

17.

Keller JM. Using the ARCS motivational process in computer-based instruction and distance education. New Dir Teach Learn. 1999;78: 39–47.

18.

Wlodkowski RJ. Enhancing Adult Motivation to Learn: A Comprehensive Guide for Teaching All Adults. Hoboken (NJ): Wiley; 2011.

19.

Blanchard E, Razaki R, Frasson C. Cross-cultural adaptation of e-learning contents: a methodology. In: Richards G, editor. Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2005. Chesapeake (VA): AACE; 2005:1895–1902.

20.

Macfadyen LP, Chase MM, Reeder K, Roche J. Matches and mismatches in intercultural learning: designing and moderating an online intercultural course. Poster presented at: UNESCO Conference on Intercultural Education; June 15–18, 2003; Jyväskylä, Finland.

21.

McLoughlin C, Oliver R. Designing learning environments for cultural inclusivity: a case study of indigenous online learning at tertiary level. Aust J Educ Technol. 2000;16(1):58–72.

22.

Reeder K, Macfadyen LP, Roche J, Chase M. Negotiating cultures in cyberspace: participation patterns and problematics. Lang Learn Technol. 2004;8(2):88–105.

23.

Hannon J, D’Netto B. Cultural diversity online: student engagement with learning technologies. Int J Educ Manag. 2007;21(5):418–432.

24.

Zhang Z, Kenny R. Learning in an online distance education course: Experiences of three international students. Int Rev Res Open Distance Learn. 2010;11(1):17–36.

25.

Lim HC, Lee SG, Nam KC. Validating E-learning factors affecting training effectiveness. Int J Inf Manage. 2007;27(1):22–35.

26.

Muilenburg LY, Berge ZL. Student barriers to online learning: a factor analytic study. Distance Educ. 2005;26(1):29–48.

27.

So HJ, Brush TA. Student perceptions of collaborative learning, social presence and satisfaction in a blended learning environment: relationships and critical factors. Comput Educ. 2008;51(1):318–336.

28.

Walker JD, Cotner SH, Baepler PM, Decker MD. A delicate balance: integrating active learning into a large lecture course. CBE Life Sci Educ. 2008;7(4):361–367.

29.

Bower M. Redesigning a web-conferencing environment to scaffold computing students’ creative design processes. Educ Technol Soc. 2011;14(1):27–42.

30.

Schrage M. Shared Minds: The New Technologies of Collaboration. New York: Random House; 1990.

31.

Paulus TM. Collaborative and cooperative approaches to online group work: the impact of task type. Distance Educ. 2005;26(1):111–125.

32.

Alexander S. E-learning developments and experiences. Educ Train. 2001;43(4–5):240–248.

33.

Brown RE. The process of community-building in distance learning classes. J Asynchronous Learn Netw. 2001;5(2):18–35.

34.

Roberts TS, McInnerney JM. Seven Problems of online group learning (and their solutions). Educ Technol Soc. 2007;10(4):257–268.

35.

Collins J. Education techniques for lifelong learning: giving a PowerPoint presentation: the art of communicating effectively. Radiographics. 2004;24(4):1185–1192.

36.

Frankola K. Why online learners drop out. Workforce. 2001;80(10):52–61.

37.

Tyler-Smith K. Early attrition among first time elearners: a review of factors that contribute to drop-out, withdrawal and non-completion rates of adult learners undertaking elearning programmes. J Online Learn Teach. 2006;2(2):73–85.

38.

Harrell IL, Bower BL. Student characteristics that predict persistence in community college online courses. Am J Distance Educ. 2011;25(3):178–191.

39.

Eaton JS. Distance Learning: Academic and Political Challenges for Higher Education Accreditation. Washington: Council for Higher Education Accreditation; 2001.

40.

Billings DM, Skiba DJ, Connors HR. Best practices in web-based courses: generational differences across undergraduate and graduate nursing students. J Prof Nur. 2005;21(2):126–133.

41.

Palloff RM, Pratt K. Building Online Learning Communities: Effective Strategies for the Virtual Classroom. Hoboken (NJ): Wiley; 2007.

42.

Mitton C, Adair CE, McKenzie E, Patten SB, Waye Perry B. Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q. 2007;85(4):729–768.

Creative Commons License © 2015 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.