A tool to evaluate physiotherapy clinical education in South Africa

Background Physiotherapy clinical education is complex. The dynamic learning milieu is fluid and multidimensional, which contributes to the complexity of the clinical learning experience. Consequently, there are numerous factors which impact the clinical learning experience which cannot be measured objectively – a gap which led to the development of our study. Objectives To develop, validate, and test the reliability of an assessment tool that evaluates the effectiveness and quality of physiotherapy clinical education programmes. Method A mixed methods approach in three phases included physiotherapy academics, clinical educators, and clinicians throughout South Africa. Phase One was a qualitative study: focus group discussions determined items and domains of the tool. Phase Two established the content and construct validity of the tool, a scoring system and a name for the tool, using the Delphi method. In Phase Three, factor analysis reduced the number of items, and the feasibility and utility of the tool was determined cross-sectionally. Results The Vaneshveri Naidoo Clinical Programme Evaluation Tool (VN-CPET) of 58 items and six domains was developed and found to be valid, reliable (α = 0.75) and useful. The six domains of VN-CPET include governance; academic processes; learning exposure; clinical orientation; clinical supervision and quality assurance and monitoring and evaluation. Conclusion The Vaneshveri Naidoo Clinical Programme Evaluation Tool is a valid, reliable and standardised tool, that evaluates the quality and effectiveness of physiotherapy clinical education programmes. Clinical implications This tool can objectively evaluate the quality and effectiveness of physiotherapy clinical education programmes in South Africa, and other health science education programmes, both locally and globally, with minor modification.


Introduction
Currently, there are no tools to evaluate the quality and effectiveness of physiotherapy clinical education programmes. Several scholars have attempted to do this, but were unsuccessful because clinical education is complex, diverse, and multidimensional (Higgs 1993;Jette et al. 2014;McCallum et al. 2013;Stachura, Garven & Reed 2000;Strohschein, Hagler & May 2002). The curriculum review process by academic departments largely focusses on the theoretical component (learning objectives, activities and outcomes), while the structure and processes of clinical education are largely overlooked; yet clinical education is a core component of a physiotherapy undergraduate programme Chetty et al. 2018;Delany & Bragge 2009;Higgs 1993;McCallum et al. 2013;Moghadam, Kashfe & Abdi 2017).
A standardised, valid, and reliable monitoring and evaluation tool will facilitate the summative and formative evaluation of physiotherapy clinical education programmes (Frye & Hemmer 2012;Persky, Joyner & Cox 2012;Stachura et al. 2000). The programme's structure and processes will be analysed, not only the outcomes (Frye & Hemmer 2012;Owston 2008). This kind of inquiry will enable strategic quality assurance mechanisms to be incorporated, and high-quality clinical learning experiences for students are likely to be achieved. Therefore, it is imperative that the clinical education component of the curriculum is independently and objectively evaluated.
Background: Physiotherapy clinical education is complex. The dynamic learning milieu is fluid and multidimensional, which contributes to the complexity of the clinical learning experience. Consequently, there are numerous factors which impact the clinical learning experience which cannot be measured objectively -a gap which led to the development of our study.
Evaluation tools provide the data required to track the implementation of processes, to determine the programme's intended and unintended effects, and to establish the programme's effectiveness. Moreover, valid, and reliable tools are needed to facilitate the complex evaluation process of physiotherapy clinical education, with its unique, multifaceted constructs. Thus, the purpose of our study was to develop and validate the reliability of an assessment tool that evaluates the effectiveness and quality of a physiotherapy undergraduate clinical education programme: A Programme Evaluation Tool.

Methods and procedure
A three-phase exploratory, sequential design that included mixed methods was used to develop the tool (Ivankova, Creswell & Stick 2006). Data collection commenced countrywide following ethical clearance to conduct this study from the Human Research Ethics Committee of the University of the Witwatersrand (Wits) -(ethical clearance number: M210160), as well as ethical clearance and/or permission, and informed consent from seven out of eight academic departments and clinical departments: University of Cape Town (UCT); University of Stellenbosch (US); University of the Free state (UFS); University of Kwazulu-Natal (UKZN); University of the Western Cape (UWC); Sefako Makgatho Health Sciences University (SMU); Chris Hani Baragwaneth Academic Hospital (CHBAH); Steve Biko Academic Hospital (SBAH) and Helen Joseph Academic Hospital (HJ).
After the pilot study, and through purposive sampling, 81 key stakeholders involved in student training (academics, clinical managers and clinicians, including new graduates) participated in the focus group discussions (FGDs). Focus group discussions allowed the first author to collect national data, efficiently and cost-effectively (Jayasekara 2012). Fourteen FGDs, each with approximately eight participants, took place. Data saturation occurred after the eighth FGD. However, appointments for the FGDs had been made 1 year in advance, so 14 FGDs were conducted. The FGDs consisted of mixed groups of participants, or distinct groups, depending on participants' availability. A broadly structured script with prompts was used. The recorded FGDs were transcribed verbatim, coded, categorised and themed inductively by the first author, using Tesch's (1992) method of data analysis (Vuso & James 2017), and MaxQda, version 2018.2 (a qualitative data analysis tool). Thematic content analysis was also conducted by the co-authors and an independent qualitative expert. There was a high level of agreement on coding and themes, and disagreements were discussed. Prior to the thematic content analysis, the transcripts were checked by the first author for errors, and member-checks confirmed that true accounts of the FGD had been captured. Data and investigator triangulation ensured trustworthiness (Halcomb & Andrew 2005;Leech & Onwuegbuzie 2007), leading to Phase Two of our study. My bias as clinical co-ordinator (CC) was curbed by conducting FGDs, where a broad statement was used to elicit the data; as well as inductive coding, member checks and co-coders.
For the Delphi process, 79 FGD participants were invited to participate in Phase Two of our study to determine the face and content validity of the preliminary tool (two FGD participants from Phase One were excluded due to invalid email addresses). The preliminary tool of 131 questions was emailed to the participants, who were asked to decide on which items should remain in the tool. Two Delphi rounds were undertaken and an 80% agreement on items in each round was obtained to keep the item in the preliminary tool. There is no standard level of consensus, although 70% -80% is usually adopted (Diamond et al. 2014;Maleka, Stewart & Hale 2017;Trevelyan & Robinson 2015).The third Delphi round confirmed a scoring system and a name for our tool. Following each Delphi round, the first author and the co-authors reviewed the comments and edited questions, as recommended by the participants, and as appropriate. Descriptive statistics (frequencies and percentages) were used to analyse the data.
Following the Delphi process, a Research Electronic Data Capture (REDCap) link (a secure web platform for building and managing online databases and surveys) (projectredcap. org) of the preliminary tool was emailed to 13 participants (heads of departments [HOD] and/or CC and/or undergraduate co-ordinators [UG]) of the eight academic physiotherapy departments in South Africa, to enable principal component factor analysis to reduce the number of items in the tool (Abdi & Williams 2010). The participants were requested to answer all the questions. The internal consistency of the items was determined using Cronbach's alpha.
Phase Three of our study, a cross-sectional survey, was used to determine the construct validity of the tool. A REDcap link of the provisional tool and questions testing the feasibility and utility (Appendix 1) of the tool was purposively emailed to 35 participants nationally and internationally (HODs and/or CCs in universities in the countries listed in Appendix 2). The participants were requested to complete the 58 questions in the tool, and to answer the following open-end questions: 1. Does this tool evaluate what you consider to be important regarding clinical education? 2. What are the strengths of this tool?
3. Indicate the weaknesses of this tool. 4. Is this tool useful for your institution?
The data were tabulated, and descriptive statistics, frequencies, and percentages, were used to analyse the data. Principal component analysis was conducted using Stata (16.0).

Ethical considerations
Our study was coducted under a strict code of ethics; the anonymity of all participants were maintained where possible; there was no identifying data on any of the data collection sheets and the data was handled under utmost confidentiality. The raw data was stored in a locked cupboard M210160. 20210204. A second ethics cleareance certificate was applied for as the first one (M140706 -22/08/2014) expired.

Phase one
The preliminary tool of 131 items which emerged following the FGDs, contained three key areas: Governance, structure and experience; the macro-, meso-and micro-components, respectively, as seen in Figure 1. Table 1 provides an overview of the categories and subcategories under each theme. Figure 2 illustrates the outcomes the Delphi rounds, the exploratory factor analysis and the internal reliability of the tool. The 131 items were reduced to 85 items after the Delphi rounds. Principal component factor analysis reduced the items to 73, and five sections, which were edited and reorganised by the first author to produce the final tool of 58 questions and six sections.

Phase two
Vaneshveri Naidoo Clinical Programme Evaluation Tool (VN-CPET) was the name chosen for our tool, after numerous suggestions. The VN-CPET is a self-administered programme evaluation tool. Scoring options reflect self-assessment, as illustrated in Table 2.
Five completed questionnaires (38.5%) were used to run principal component factor analysis and determine the internal reliability of the questions. Exploratory factor analysis split the tool into five sections and a total of 73 questions, with an acceptable internal consistency of 0.75 (Bolarinwa 2015;Hulin, Netemeyer & Cudeck 2001).
The authors reviewed the provisional tool following factor analysis and further reduced items due to redundancy, which resulted in the final tool containing 58 questions. We also reorganised the sections of the tool based on my experience as a CC (praxis), which resulted in the final tool containing six domains: 1. Section 1 -Governance (5 questions) 2. Section 2 -Academic processes (5 questions) 3. Learning exposure (6 questions) 4. Clinical orientation (7 questions) 5. Clinical supervision (18 questions) 6. Monitoring and evaluation and quality assurance (19 questions) See Appendix 1 for the complete tool.

Phase three
In this phase, although 71% (25) responded (n = 35), only 68% (17) of the participants completed the entire questionnaire. Eight (32%) questionnaires were incomplete. Figure 3 illustrates that 88% of the respondents found that the VN-CPET evaluates useful constructs of clinical education,

Governance
Policy and procedure and resources

Structure
Academic and operaƟonal MICRO Experience clinical while 59% indicated that VN-CPET would be useful for their institution. Of the remainder, 29% thought it was likely to be useful for their institution, thus elevating the institutional usefulness of the tool to 88%.
In Figure 4, 53% of the respondents commented on the comprehensiveness of the tool as a remarkable strength, followed by the wide range of influences that were captured (18%); and 12% thought it was a bench-marker. The length of the tool was its major drawback, which was pointed out by 35% of the respondents ( Figure 5).

Discussion
A physiotherapy clinical education programme requires independent and objective evaluation to determine its merits and shortcomings. Such an evaluation enables educators/ academics to strengthen aspects, add quality assurance mechanisms where required, or remove unintended effects (Frye & Hemmer 2012;Stufflebeam 2003). Mixed methods (Strohschein et al. 2002) enabled us to explore the length, breadth and depth of physiotherapy clinical education by     Strohschein et al. 2002). Additionally, the impact on students' clinical learning, as a result of the structure and processes of a clinical education programme, was discussed.
The preliminary tool of 131 items was refined into the provisional tool (73 questions; five sections) by using two key processes: the Delphi process and exploratory factor analysis. The Delphi process, through a posteriori consensus (knowledge based on experience or personal observation), which was set at 80% (Maleka et al. 2017), was used to determine the items and domains of the tool. This was appropriate in our study, as we were able to cost-effectively include several participants in a field of study where there is a paucity of information (Diamond et al. 2014;Okoli & Pawlowski 2004;Powell 2003;Trevelyan & Robinson 2015). Exploratory factor analysis further delineated the items and domains of the tool by grouping similar variables into smaller groups, while eliminating variables that had a low factor loading and/or lowered the internal consistency of the items. (Portney & Watkins 2009 Weiner et al. (2017) confirmed that the acceptability, appropriateness, and feasibility of an instrument must be determined to ensure its implementation. In other words, is this innovation satisfactory, fit for purpose, and useable in its context? A decisive 'yes' by 88% (15 out of 17; n = 17) of the participants was the reply. The VN-CPET was deemed comprehensive and a bench-marker that captured a wide range of influences, although it is long. This tool, therefore, considers everything that is important in evaluating the effectiveness and quality of a physiotherapy clinical education programme, under these sections: governance; academic processes; learning exposure; clinical orientation; clinical supervision; monitoring and evaluation, and quality assurance.
Governance refers to a multitude of factors: people, roles, structures, and policies. It is a framework under which stakeholders perform activities within regulated boundaries (Bigdeli et al. 2020;Pyone et al. 2017). In this section of the tool, governance refers to policies and agreements that guide the educational programme at a macro, meso-and micro level. Programme governance establishes processes and provides a structure for communication, implementation, and monitoring. It also ensures that policies and best practices are followed. Additionally, it ensures that the programme's goals and objectives are aligned with the larger institutional and regulatory bodies.
Academic processes refers to the educational strategies (curriculum; teaching; learning; assessment; resourceshuman and others) that have been instituted to ensure adequate pre-clinical preparation, while learning exposure refers to the hands-on learning opportunities that students experience to ensure competency in all areas of physiotherapy, meeting the needs of their country inclusively, efficiently and cost-effectively (Hirsh et al. 2007). Clinical orientation programmes are aimed at enhancing students' transition (vertically or horizontally) and student success, as they adapt to a new environment (Nguyen et al. 2018;Perrine & Spain 2008  Monitoring is the continual assessment of a project or programme to determine its intended and unintended effects (formative evaluation), whereas evaluation is the periodic retrospective assessment of a project or programme to determine its worth: relevance, impact, effectiveness, efficiency and sustainability (summative evaluation) (Annecke 2008;Porter & Goldman 2013;Stem et al. 2005;Stone-Jovicich et al. 2019). Quality assurance, on the other hand, is the evaluation of activities against a gold standard or guideline (Stachura et al. 2000). Summative and formative evaluation of physiotherapy clinical education informs the evaluator of the length, breadth, and depth of physiotherapy clinical education. The VN-CPET enables the aforementioned and allows quality assurance measures to be inserted where necessary. Most importantly, the VN-CPET provides a standardised, valid and reliable way of evaluating a physiotherapy clinical education programme.

Conclusion
The VN-CPET reflects the complexity and diversity of clinical education, due to its ability to be 'comprehensive' and to capture a 'wide range of influences'. Although long, it was found to be acceptable, appropriate, and feasible. Furthermore, the VN-CPET is a valid and reliable tool and can be used to objectively evaluate the effectiveness and quality of a physiotherapy clinical education programme. Even though the scoring system is subjective, an evaluative response is obtained. A link to the online tool can be requested from the corresponding author.
The strength of the VN-CPET lies in its rigorous development using mixed methods (Strohschein et al. 2002), and the South African context is by no means a barrier to its global application in clinical physiotherapy education: the educational framework of physiotherapy clinical education is the same, despite different contexts. This tool will be shortened, and the scoring system refined in future studies.
The limitations include the subjectivity in the existing scoring system; the length of the tool, which is a potential barrier to its use; and the purposive sampling that was used to determine the feasibility and usefulness of this tool. Therefore, this tool should be used bearing these limitations in mind.
V.N. was responsible for conceptualisation, study design and execution, data collection, analysis and interpretation of data, drafting and critical revision of manuscript and final approval of the version to be published. A.V.S. and M.E.D.M. contributed to the conceptualisation and study design, review of data analysis, validation, as well as review and editing of manuscript.

Data availability
The data that support the findings of this study are available on request from the corresponding author.

Disclaimer
The views and opinions expressed in this article are that of the authors, and do not reflect official policy or position of any of the affilations.