A 360‐degree assessment model that fosters professional skills development in doctor of physical therapist students

Successful transition from academia to the twenty‐first century workplace requires that college graduates acquire technical skills in their field as well as professional skills for interacting effectively with people (McLester & McIntire, 2006). In 2006, The Conference Board collaboarated with several organizations to conduct an in‐depth study, that included surveys and interviews with 431 human resource representatives and senior executives to determine the corporate perspective on the readiness of new entrants into the US workforce (The Conference Board, 2006). The report, Are They Really Ready to Work?, includes a Workforce Readiness Report Card, which presents employers' views on the preparation level of new entrants to the US workforce. Results maintained that while employers expect young adults to arrive with a core set of basic knowledge and the ability to apply these skills in the workplace, the expectation is not being met. The missing essential professional skills identified included: teamwork, critical thinking, communication, personal accountability, and effective work habits. These skills are critical as factors such as incompetence and dishonesty have recently impacted the public's perception of the professions (Gardner & Shulman, 2005). Findings from the aforementioned studies are of great relevance in healthcare, where the professional role of health professionals requires regard for patients, compassion, effective communication, ethics, teamwork and accountability. These skills are quintessential for attaining positive health outcomes.

Our study focused on the development of professional skills in doctor of physical therapy (DPT) students. For the purpose of this research, the professional skills underpinning the DPT profession are: accountability, altruism, compassion and caring, excellence, integrity, professional duty, and social responsibility (American Physical Therapy Association, 2003). Because physical therapist students are entering adulthood, the attitudes, interests, values, and character development that underlie their behaviors may not be at a professional level (Hayward, Noonan & Shain, 1999; May, Morgan & Lemke, 1995). For example, novice physical therapists students often lack appreciation for a patient's perspective during a provider interaction, "yet this skill is an essential part of the professional skills that leads to successful patient outcome" (Hayward, Noonan & Shain, 1999, p. 37). Similar to graduates in other professional programs, in addition to acquiring skills necessary for professional practice, successful DPTs require strong professional skills.

Cooperative education (co‐op) or internships provide an opportunity to foster the academic knowledge transfer required for students entering the workplace (Hayward, Raelin & Blackmer, 2007). Academic institutions that include co‐op must find ways to research outcomes related to the integration of academic and experiential learning (Hayward, Raelin & Blackmer, 2007; Raelin, Glick, McLaughlin, Porter & Stellar, 2008; Schutte, 2007). Higher education faculty members are challenged to design pedagogy that explicitly teach and reward student development of professional skills crucial for co‐op and workplace success. 

The purpose of our exploratory study was to present and test the impact of a 360‐degree assessment model designed to explicitly teach and reward the acquisition and demonstration of professional skills in physical therapist students. Our study is aligned with two suggestions made by McLester and McIntire (2006) to improve professional skills development: 1) teach applied skills integrated with academic content; and 2) increase participation from the business community in defining the skills required by graduates.

THEORETICAL FRAMEWORK

360-Degree Assessment Model

The following is a description and theoretical background for a 360‐degree assessment model (Figure 1) that fosters the development of professional skills in DPT students. The model relies on standardized patients augmented by online communities of practice, and provides opportunities for reflection (Hayward, Blackmer & Markowski, 2006).

Standardized patients are an experience‐based pedagogy with potential for both teaching and assessing learning outcomes in students (Barrows, 1993). Standardized patients, while common in medical education, (Barrows, 1993) are a developing pedagogy in physical therapist education (Black & Marcoux, 2002; Hayward, Blackmer & Markowksi, 2006; Ladyshewsky, Jones, Baker & Nelson, 2000; Ladyshewsky & Gotjamanos, 1997). According to Barrows (1993), standardized patients are laypersons trained to mimic a patient condition and provide realistic teaching experiences for students and opportunities for faculty to assess professional skill acquisition. A community of practice is a group of individuals who are interconnected by a future‐oriented and shared learning goal (Doty, 2002; Howard & Kennedy‐ England, 2001; Lave & Wenger, 1991) such as preparing for treatment of standardized patients. A community of practice may exist among members at a designated location or may bridge geographic boundaries through Internet technology.

The model provides a 360‐degree feedback loop on a student physical therapist interaction with standardized patients (Figure 2). The 360‐degree feedback loop is a well‐described assessment strategy with origins in the business literature (Antonioni, 1996). In business, the feedback loop is defined as a process that provides an individual with insight on their performance in the workplace. The feedback process includes a self‐appraisal and insight from multiple perspectives, such as colleagues and managers. The feedback loop is a powerful tool because an individual receives performance appraisal that explicitly documents strengths and areas for development.

Benefits of the 360‐degree approach include an increased understanding of performance expectations and an appreciation for how actions affect others. The inclusion of self‐appraisal or reflection is useful for promoting self‐ directed learning and development (Dewey, 1933; Schön, 1983). In addition, coaching assists the individual to understand the data more easily, share feelings about results, and discuss a plan of action for areas targeted for improvement. As maintained by Antonioni, (1996) an individual faced with multiple sources of feedback is less likely to dismiss it as opinion and more likely to make changes to improve performance.

METHODOLOGY

We used an exploratory research design with three phases and two research questions:

1. What professional skills in DPT students do co‐op employers desire?
2. Will the model intervention impact student professional skill development?

Setting and Participants

The sample was obtained from a large urban institution in the US whose educational philosophy embraces practical, experience‐based learning throughout a student's college experience. Central to the educational philosophy of this institution is co‐op, in which students' alternate periods of classroom study with periods of full‐time employment related to career or personal interests.

The institution's physical therapy program is a 6.5‐year entry‐level clinical doctoral program in physical therapy. DPT students are required to complete two six‐month co‐op terms, Co‐op I in the third year and Co‐op II in the fourth year. Students on co‐op are typically employed full‐time as physical therapist aides. DPT students also participate in two short‐term opportunities for service learning, another form of experiential education. In year six, students participate in a three‐phase 28‐week (8, 8, & 12‐weeks) clinical education component in which they are supervised and mentored by a clinical preceptor while assuming the professional role of the physical therapist. Throughout the curriculum students experience many developmental transitions including high school to college, student to worker, worker to student, and worker to entry‐level professional.

Project participants were 104 (83 females) third‐year DPT students and eight co‐op employers. Co‐op employers were eight (4 females), with an average of 16 years experience. Two worked in hospitals and the remainder in private clinics. The employers' health care facilities staff supervises and assesses physical therapist student development during co‐op.

Intervention-Protocol

Phase 1: Pre-test

Focus groups were conducted with eight PT co‐op employers, a convenience sample, to determine their perceptions of student awareness of professional skills and communication in the workplace. Employers were asked to provide examples of professional skills observed in students that were both exemplary and non exemplary and to describe, how these behaviors impacted patient care or inter‐professional relationships. Focus groups were tape‐recorded and transcribed verbatim. Faculty used the information gathered from employers to develop the case studies. In addition, all 104 DPT students completed the Professionalism in Physical Therapy: Core Values survey to self‐assess their professional and communication skills. In 2003, the American Physical Therapy Association adopted the Professionalism in Physical Therapy: Core Values survey as a core document to foster awareness of the seven core values that define the DPT professional. Using a five point Likert scale, an individual self assesses in which of the core values they exhibit strength and/or areas for growth. For each core value, sample indicators are provided that delineate what a physical therapist would do in practice, education, and research (American Physical Therapy Association, 2003).

Phase 2: Intervention

All 104 students in "Physical Therapy Professional Seminar 1," participated in the model intervention (Figure 1). "Physical Therapy Professional Seminar I" is a 15‐week course designed for third‐year DPT students offered the semester immediately following Co‐op I. In this course, students are expected to reflect upon and integrate their co‐op experiences into course activities while continuing to develop communication and professional skills. 

To prepare standardized patients for the intervention, faculty members, in collaboration with co‐op employers, developed paper‐based case studies that depicted a patient with a complex medical condition. The cases were designed to challenge students to develop and apply communication skills and professional skills required for patient interviews. Each case was complicated by realistic dilemmas such as a cultural or ethical concern. The case and guiding questions were posted on a Blackboard^TM "Physical Therapy Professional Seminar 1" website. Communities of practices of four to five physical therapist students were created. In preparation for the experience with the standardized patients, student communities of practices and faculty communicated electronically about the case using the Blackboard^TM discussion board feature. Standardized patients were recruited and trained by course faculty to portray a patient case. Training each standardized patient took approximately one hour. Each standardized patient received a $75 gift certificate as compensation for serving as a standardized patient for two or more student interactions. Each standardized patient interaction lasted for 30 minutes and was videotaped. Upon completion of a standardized patient interaction faculty, standardized patients, and peers provided each student with "360‐degrees of feedback" on their professional, communication, and technical skill performance using grading rubrics (figure 1). To reflect on their learning, all students completed a self‐reflective paper (appendix), and at the conclusion of the standardized patients‐communities of practice interaction, students and faculty debriefed and reflected together about the entire learning experience.

Phase 3: Post-test

At course completion, all 104 students retook the Professionalism in Physical Therapy: Core Values survey.  

Data Collection and Analysis

Data were collected at three points during the research project using both qualitative and quantitative research methodologies. Qualitative data consisting of co‐op employer focus group transcripts were analyzed using content analysis (Miles & Huberman, 1994). Content analysis is a process of identifying, coding, and categorizing the principle patterns within a data set to arrive at themes that summarize results (Miles & Huberman, 1994). The qualitative analysis began with the authors organizing and reducing the data by assigning codes or descriptors to individual pieces of data. Coding categories began as descriptive, in that they allowed the researchers to describe what ideas emerged from reviewing the focus group transcripts. Then the researchers examined how the data clustered together and sorted it into preliminary descriptive coding categories. The preliminary coding categories emerged inductively during the data analysis process (Maxwell, 1996). Once the data were assigned preliminary descriptive codes, the authors examined and grouped the data together as they reflected larger patterns or meaning. The authors created pattern code names, which reflected the meaning, generated from the descriptive codes. Descriptive codes were collapsed and regrouped under the pattern codes. Pattern codes then became the study's three final themes (Miles & Huberman, 1994).

Quantitative data were analyzed to compare pre to post intervention scores on the Professionalism in Physical Therapy: Core Values survey. The survey is a 68 item, 7 scale, 5 point Likert scale survey instrument that was developed by a panel of 18 physical therapist experts convened by the education section of the American Physical Therapy Association in 2002. The number of questions allotted to each scale follows: Accountablity, 10 questions; Altruism 5 questions; Compassion and Caring 11 questions; Excellence 11 questions; Integrity 12 questions; Professional Duty 7 questions; and Social Responsibility 12 questions. The survey, which is grounded in the medical literature on professionalism, was developed in response to the challenge faced by the physical therapist profession to define professionalism. The American Physical Therapy Association board of directors reviewed the survey and adopted it in 2003 as a core document on professionalism in physical therapist practice, education and research (American Physical Therapy Association, 2003). Averages for aggregate pre‐ and post‐test Professionalism in Physical Therapy: Core Values survey scores were analyzed using a paired t‐test 0.05. Averages for each of the seven sections were compared pre‐ and post‐test using paired t‐tests, ∝ 0.05 (Portney & Watkins, 2009).

RESULTS

Co-op Employers

Focus group data was collected from the eight co‐op employers to learn about their perceptions of what professional skills are required for workplace success.These qualitative data were analyzed and summarized into three major themes: basic job skill, professional skills, and learning. Within each theme, the employers noted both desired and observed behaviors in DPT students. Quotes from the focus group illustrate each theme.

Desired Skills: Basic Job

Basic job skills identified by the employers as critical for the job included: time management, punctuality, organization, professional dress, knowing the scope of the job, understanding the needs of the workplace, and flexibility: "Organization and time management...they [co‐op student] need to be able to organize and manage their time so that they can accomplish all that they are expected to do rather than having [the supervisor] to lead them around" (Co‐op employer focus group, 12/7/07).

Observed-Basic Job

Basic job skills deficits included: limited attention to job rhythm, poor professional appearance, cyber multitasking and excessive cell phone use on the job:

One of the co‐op responsibilities is to help with some of the billing codes, note writing…so it's very easy for them to have an IM [instant messaging] chat open and a note open and a couple of other things going and checking their email. And they're very good at multitasking in that way. But sometimes when they're supposed to get things done, they get lost, they lose focus. (Co‐op employer focus group, 12/7/07)

Desired Skills‐Professional

Professional skills employers preferred were the ability to communicate needs and requests, be a team player, interact effectively with other people, demonstrate empathy, adapt one's communication to level of listener, and show respect for different ages and cultures. Employers also wanted students to become part of the fabric of the company:

Being respectful, that you're dealing with different nationalities, ages…a man talking to a woman is obviously different than a man talking to a man and then having a different ethnicity, race, color, creed, everything that we have to take into consideration when talking to our patients, their comfort level. I think they [co‐op student] need to have that adaptability to be able to do that. (Coop employer focus group, 12/13/07)

Observed Skills ‐ Professional

Inappropriate professional skills cited by employers were poor communication, not demonstrating professional pride, and not understanding the limits of the job.

I [the supervisor] had an interesting student once. There were a lot of things not exemplary; the biggest thing was his personal communication with patients…He tried to sell a snake to one of my patients. But just the personal things, such as talking about hangovers, that [conversation topics] shouldn't cross that personal professional line…he pushed the boundaries where he could. (Co‐op employer focus group, 12/13/07)

Desired Skills‐Learning

The traits related to learning desired in co‐op students included the ability to take initiative and be self‐directed and to look at a job as an educational opportunity: "So somebody who has that initiative and being proactive, wanting to learn and taking advantage of the opportunities of being in the clinic" (Co‐op employer focus group, 12/7/07).

Observed‐Learning

Observed behaviors included: lack of initiative and not turning mistakes into learning opportunities: "Everybody makes mistakes. Mistakes are just, you know, it's [co‐op] a learning process, we [supervisors] all expect that. We're co‐ ops [former]. We've all made mistakes. But that he [the co‐op student] didn't turn that mistake into a learning experience and translate it for the next thing" (Co‐op employer focus group, 12/7/07).

Student Survey Data

Quantitative data consisted of pre and post intervention Professionalism in Physical Therapy: Core Values survey results. Some 62 of the 104 students completed both the pre and the posttest surveys (ca. 60% were female). However, all 104 students completed the standardized patients‐communities of practice, intervention. Pre‐  versus post‐intervention paired t‐test comparisons were run for aggregate data average scores (pre‐ versus post‐), and for each of the seven scales of the survey. All comparisons, except Accountability, revealed a statistically significant higher post‐test score. These results reveal a significant change in the positive direction for students' development of professional skill awareness (Table 1).

DISCUSSION

Our research study had two questions. The first concerned identification of the professional skills required by co‐op employers in DPT students. To answer this question, we collected focus group data from co‐op employers that resulted in three major themes: basic job skills, professional skills, and preferred learning attitudes. Desired basic job skills included time management, punctuality, organization, professional dress, knowing the scope of the job, ability to adapt to change or understand needs of the workplace, and flexibility. Employers indicated that observed behaviors in DPT students included limited attention to job rhythm, poor professional appearance, cyber multitasking, and excessive cell phone use. Similar to other professions, DPT students must demonstrate effective work habits that are expected in the workplace (McLester & McIntire, 2006). Basic job skills are the foundation for most employment situations. Professional skills identified by employers were centered on student ability to communicate needs and requests, interact effectively with others, work as a team, demonstrate empathy, adapt one's communication to level of listener, and show respect for different ages and cultures. Employers in our study noted that some DPT students displayed poor communication skills, lack of professional pride, and a limited understanding of the scope of the job.

Teamwork, communication, and personal accountability are elements that have been identified as required for current workplace success by both the American Physical Therapy Association and employers in general (American Physical Therapy Association, 2003; Gardner & Shulman, 2005; McLester & McIntire, 2006; The Conference Board, 2006). These findings are of particular relevance in healthcare because compassion, effective communication, teamwork, and accountability are essential for positive patient health outcomes. 

Employers articulated specific desired learning traits, which included the ability to take initiative, be self‐directed, and view a job as an educational opportunity. These learning traits are hallmarks of a reflective practitioner (Dewey, 1933; Schön, 1983). Our data support the need for co‐op employers and faculty to incorporate opportunities for reflection on experience, which allows for enriched learning and the integration of academic and experiential education (Daudelin, 1996; Raelin et al, 2008; Schutte, 2007; Van Gyn, 1996). One benefit of feedback from co‐op employers, both anecdotal and statistical, is that their feedback can guide curricular adjustments (Harfman et al., 2008; Hutt, 1979). Including co‐op employer feedback can be powerful because employers are often concerned with broad educational goals in addition to specific workplace needs (Harfman et al., 2008). We used employer feedback to assist with the customization of the case studies used with the standardized patients. Our goal as faculty was to use employer feedback to focus student attention on both specific skills and broader goals required for workplace success. Inclusion of employer feedback strengthened our methodology and provided useful information for faculty on curriculum design and course objectives. We believe that academic faculty, students and employers can benefit if the curriculum is responsive to the realistic needs of the workplace. Increased participation from the business community in defining the skills required by graduates is supported in the literature (McLester & McIntire, 2006; The Conference Board, 2006; Van Gyn, 1996).

Our second research question sought to determine if our intervention would have an impact on student communication and professional skills awareness. We used the Professionalism in Physical Therapy: Core Values survey to measure the seven core values of the DPT professional: accountability, altruism, compassion and caring, excellence, integrity, professional duty, and social responsibility (American Physical Therapy Association, 2003). The survey data demonstrate that an experiential pedagogy that relied on standardized patient‐communities of practice, and 360‐degree assessment resulted in a statistically significant improvement in DPT students' self‐described awareness of professional skills. Increase in student awareness of six core values could be attributed to inclusion of opportunities for reflection within the model. Reflection has been used in co‐op as a means to link theory and practice (Hayward, Blackmer & Raelin, 2007; Raelin et al., 2008; Schutte, 2007; Van Gyn, 1996). In our model, there were several structured opportunities for student reflection. For example, we created communities of practices composed of 3‐5 students who used the asynchronous discussion board in Blackboard^TM to communicate about a case prior to interaction with standardized patients. This opportunity enabled our students to construct thoughtful written responses and reflect on and learn concurrently from the contributions of others (Hayward, Blackmer & Markowski, 2006; Hayward et al., 2001; Lave & Wenger, 1991; Raelin et al., 2008). We believe that the communities of practice activity facilitated student teamwork and communication skills. Communities of practices are useful for promoting teamwork when individuals have shared goals for learning (Lave & Wenger, 1991). In addition, students were graded on their participation, which was linked to preparation and performance during the standardized patient interaction. Assignments that are graded will most likely increase student attention and importance related to learning outcomes.

A second opportunity for reflection was provided during the student interaction with the standardized patients. The 360‐degree assessment process provided each student with multiple sources of feedback on their performance. Research supports that when an individual receives feedback from multiple sources he is less likely to dismiss it as opinion and more likely to make changes to improve performance (Antonioni, 1996). Our students received feedback from faculty, standardized patients and peers. In addition, all student interactions were videotaped. Students noted in their reflective papers that feedback from the standardized patients in particular was viewed as valuable and authentic and contributed to their awareness of personal performance and skill development.  

Finally, students were provided with two opportunities for guided reflection on the entire learning experience. At the conclusion of each standardized patients interaction, students reflected with their peers. A course faculty member facilitated the debriefing systematically to help them refine their thinking (Raelin et al., 2008; Van Gyn, 1996). Students were able to reflect immediately after the experience on what went well and what elements of the interaction could be improved. Reflection immediately after an experience and in the presence of others enables students to deeply examine a learning experience (Lave & Wenger, 1991; Raelin et al., 2008, Van Gyn, 1996). Students were also required to complete a final reflective paper using four guiding questions (Appendix). The inclusion of self‐appraisal in the assessment process is useful for promoting self‐directed learning and development in DPT students (Hayward, 2000; Jensen & Denton, 1991; Shepard & Jensen, 1990).

Our data indicate a statistically significant increase in student awareness of 6 of the 7 core values, with the exception of accountability. It is unclear why accountability, which is defined "as acceptability of the responsibilities for the diverse roles, obligations, and action of the physical therapist...that positively influence patient/client outcomes, the profession and health needs of society" (American Physical Therapy Association, 2003, p. 4) did not demonstrate a significant increase. One possible explanation is that the student on co‐op is working as a physical therapy aide, and cannot assume responsibility for decision making with respect to patient care. Thus, the development of accountability may require experience as decision maker that occurs during clinical education experience. 

Our study contains several limitations. First of all, the study was exploratory, did not contain a comparison group. The study was based on a small sample size of 62 students and 8 co‐op employers. In addition, it was conducted with physical therapist students and employers affiliated with a single institution.   Finally, the model was used with physical therapist students, and more research is indicated to determine if it has the potential for wider application to physical therapist students in other institutions or students enrolled in other professional programs. 

CONCLUSIONS

We created and pilot tested a 360‐degree assessment model that relied on standardized patients augmented by online communities of practices and opportunities for reflection to teach and evaluate student development of professional skills. Our project explored the effectiveness of the model for facilitating awareness of professional and non‐technical skills required for students aspiring to become a DPT professional. Feedback from employers was utilized to guide the development and assessment of a pedagogy that was experientially based. Our results indicated that the model was successful in promoting an increase in awareness of core professional values in DPT students. The model was structured to explicitly reward and offer constructive feedback to students on the professional skills that ultimately benefit patient treatment outcomes. Inclusion of employer perspectives on desired workplace skills was useful for curriculum design by academic faculty. While the model was used with physical therapist students, it has the potential for wider application to students enrolled in other professional programs whose graduates require strong non‐technical skills for workplace success. More research is indicated to examine the longitudinal effectiveness of our intervention on promoting student professional skills, and the learning traits desired by employers.