Developement of a Measure for Predicting Learning Advencement Through Cooperative Education: Reliability and Validity of the PLACE Scale¹

Cooperative education (co-op) originated in 1906 from the vision of Herman Scheider, Dean of Engineering at the University of Cincinnati (Barbeau, Godfrey, & Stull, 1998). Dr. Scheider developed a plan that featured alternating periods of industrial work with academic study. According to Hartley (1987), the cooperative education plan spread to 10 colleges and universities between 1906 and 1919. Each of these cooperative education programs offered placements exclusively to engineering students. Assisted in part by federal funding in the 1970s and 1980s, cooperative education has grown to include almost 1,000 post-secondary institutions, 200,000 students per year (representing a wide variety of majors), and 80,000 employers in the United States and abroad (Scippio, 1996).

A key feature of the cooperative education model is paid work experience that occurs over multiple semesters or quarters and which is related to the participating student's field of study (Cooperative Education Network, 1996). According to Cates and Jones (1999), early cooperative education programs usually involved alternating terms of academic study with terms of degree-related work experience. However, in an effort to accommodate non-technical majors, course scheduling limitations, and two-year institutions, many of the newer programs deviated from the alternating model in favor of parallel and/or one-term experiences (Gould, 1987). Thus, current debate in the field centers largely on structural issues, especially the amount of time which individual cooperative education programs require of students in the field (Perry, 1999).

According to Branton et al. (1990), cooperative education continues to function outside the mainstream of American higher education. Branton et al. (1990) attribute this lack of acceptance of cooperative education by American higher education practitioners to a dearth of research demonstrating that cooperative education students display superior academic progress as compared to non-cooperative education students. Yet, in order to survive and thrive, cooperative education programs must demonstrate their educational value (Demetriou, 1995). Thus, it is surprising that, as noted by Heinemann and Wilson, (1995, p. 47), "'little in the way of formal research has been undertaken to develop a better understanding of these programs."

Moreover, as noted by Ricks et al. (1993), there has been a paucity of systematic approaches to cooperative education research. Specifically, Ricks et al. identified the following nine problems endemic to published literature in this area: (1) lack of development and refinement of theory; (2) failure to integrate contrary research findings: (3) failure to conduct research within a theoretical framework; (4) lack of research rigor and standards; (5) failure to operationalize variables adequately; (6) failure to pose key research questions; (7) failure to utilize standardized measures of cooperative education outcomes; (8) a tendency to over-generalize findings; and (9) "a trend away from empirical studies and a movement toward general discussion and position papers" (p. 12).

Consistent with this position, Van Gyn, Cutt, Loken. and Ricks (1997) maintained that the three primary challenges facing researchers of cooperative education student outcomes are "operational definitions, standardized instrumentation, and choice of methodology" (p. 8 2). Notwithstanding, a way of simultaneously addressing several of these concerns is through development of a standardized instrument assessing student outcomes across various settings. It is critical that such an instrument be inexorably linked to a sound theoretical framework.

Although Heinemann and Wilson (1995) developed a taxonomy of institutional-sponsored work experience. no researcher appears to have developed an instrument that assesses students· perceptions of their cooperative learning experience. This was the major purpose of this investigation. A secondary goal was to use this instrument to compare outcomes across several demographic variables. Of particular interest was the examination of differences in student outcomes across varying numbers of cooperative education experiences.

As a first step in developing a standardized instrument, the researchers conducted an extensive review of the cooperative education literature, with particular emphasis on student learning outcomes. Indubitably, Fletcher (19 89 , 1991) has played a major role in identifying clusters of student outcomes believed to be related to participation in cooperative education programs. These clusters represented pre- and/or post-graduation outcomes. Specifically, Fletcher (1989) identified three clusters of student outcomes into which much of the previous research can be classified, that is, career development, career progress, and personal growth. Career development is defined as students' level of certainty about career choice, development of a realistic career identity, and a more realistic perception of one's strengths, weaknesses, and interests. Career progress includes post-graduation benefits such as increased likelihood of finding employment related to one's major, higher starting salaries, and faster advancement within the employing agency. Personal outcomes include increased confidence, development of social skills and maturity, and greater autonomy and independence.

Later, Fletcher (1991) identified the following three outcomes, that is, personal development. career development, and academic achievement. Both of Fletcher's clusters of outcomes are presented in Figure 1. Interestingly, not contained in either of Fletcher's conceptualization is the role of work-skills development. Work-skills development represents specific skills needed to succeed in the employment setting. Work-skills can be thought of as day-to-day micro-level functions. whereas career development skills are the macro-skills needed to manage one's career. Figure 1 also presents this newly-identified concept.

Our present focus was to develop a standardized instrument examining pre-graduation outcomes. The four-component model of cooperative education outcomes presented in Figure 2 fanned the basis of this measure. This model comprised the following variables: (l) career development, (2) academic functions/achievement, (3) work-skills development, and (4) personal growth/development. It was believed that the development of such an instrument would facilitate the creation of an extensive research base documenting the impact of participation in cooperative education programs upon student learning outcomes.

Method

Participants

Invitations to participate in the study were sent

Figure 1
Pre- and Post-Graduation Outcomes of Cooperative Education

to the entire Cooperative Education Network (CEN) membership.2 Directors from over 150 schools across the United States were invited to participate in the survey. Each school in the sampling frame was a subscriber to the CEN Attributes of Cooperative Education. Of these, 21 (14%) volunteered students from their colleges and universities to participate in the investigation. Participants in this convenience sample were selected by virtue of the fact that they were working as cooperative education students in one of these 21 colleges or universities during the Winter Quarter 1999, Spring Quarter 1999, or Spring Semester 1999. From these institutions, 3,687 cooperative education students became members of the sample. However, only those students who had worked at least one term were included in the final analysis, for a total of 2,309 participants. Students were strongly encouraged, although not required, to complete the instrument.

Participants ranged in age from 18 to 31 years. The majority of the sample was male (69.0%) and Caucasian-American (68.2%). Of the minority students who specified their ethnicity, 28.6% were African-American, 41.3% were Asian-Pacific, 15.8% were Hispanic, and 1.5% were Native American/ Indian. The number of terms worked by the sample ranged from l to 9 (M = 2.94, SD= 1.74).

Instruments

Participants were administered the Cooperative Education Evaluation (CEE). The CEE was a new instrument constructed by a working committee of the CEN.3 The items on the CEE were developed using the Four-Component Model of Cooperative Education Outcomes (see Figure 2). The CEE contained 34 seven-point, rating-scale items that relate to career development, academic functions/ achievement, work-skills development, and personal growth/development outcomes of the cooperative education experience.

Procedure

The data collection occurred at 21 colleges and universities in the United States. Directors of cooperative education at the individual schools were given the task of distributing and collecting the surveys. Standardized instructions regarding survey administration were provided to the participating schools. Surveys were given to students entering cooperative education programs starting January 1999. Students who worked between January 1999 and June 1999 were surveyed, thereby allowing students working three work periods (winter and spring quarters, and spring semester) to be surveyed.

Students were informed that their responses

Figure 2
Four-Component Model of Cooperative Education Outcomes

would be kept confidential. Completed survey instruments were collected by the individual schools and were sent to a central location for data entry. The data, once entered, were then forwarded to the current researchers for data analysis.

Results

Construct-Related Validity

Validity is the extent to which an instrument measures what it is supposed to measure (Gay & Airasian, 2000). Furthermore, construct-related validity is the extent to which an instrument can be interpreted as a meaningful measure of some characteristic or quality (Gay & Airasian, 2000). Establishing structural validity, also known as factorial validity (i.e., the degree to which the emergent factor structures can be appropriately interpreted in light of the theoretical assumptions) is an important step in providing evidence of construct validity.

Exploratory factor analysis was used to assess the structural validity of the CEE. Specifically, a maximum likelihood (ML) factor analysis was used to determine the number of factors underlying the CEE. This technique is more valid for identifying the number and nature of the latent factors that are responsible for covariation in a data set than is principal components factor analysis (Bickel & Doksum, 1977; Hatcher, 1994), and is, perhaps the most commonly used method of factor analysis (Lawley & Maxwell, 1971). Using a Varimax rotation. and a criterion of .5 or greater for deeming a factor loading practically significant (Hair, Anderson. Tatham, & Black, 1995), the ML factor analysis revealed three interpretable factors with eigenvalues greater than 1.0. These three factors combined. explained 57.8% of the total variance.

Loadings of items on each factor are presented in Table l . (Only items that met Hair et al. 's (1995) criterion are presented.) It can be seen from this table that the three factors were represented by 22 of the 34 items, with Factor I containing nine items, Factor 2 comprising seven items, and Factor 3 containing six items. Factor l , whose loadings ranged from .54 to .80, contained items dealing with work skill developments. Factor 2, whose loadings ranged from .59 to .81, comprised items pertaining to career development. Finally, Factor 3, whose loadings ranged from .52 to .75, contained items relating to academic functions/achievement.

As indicated above, 12 of the 34 items were removed from the instrument as a result of the factor analysis. Of these, 10 (e.g., personal finances, global and societal context, abilities and limitations. tolerance and understanding) had loadings of less than .5. Of the omitted items, self-confidence and follow-through, had loadings greater than .5; however, a two-item scale was not deemed adequate for representing the personal growth/development construct. In any case, the addition of the fourth factor to the selected three-factor model increased the proportion of variance explained by only 3%.

Cronbach's coefficient alpha, a measure of internal consistency, provides information about the degree to which the items in a scale measure similar characteristics (Campbell & Stanley. 1990; Gay & Airasian, 2000; Kerlinger, 1999). Coefficient alpha was determined for each of the three sub

Table 1
Factor Leadings for Three-Factor Common Factor Analysis (n= 2,309)

scales, yielding the following reliability estimates: .91 for the Work Skills Development subscale, .86 for the Career Development subscale, and .85 for the Academic Functions/Achievement subscale. In addition, the total scale yielded an alpha coefficient of .94. Using Nunnally and Bernstein's (1994) criterion for affective measures, these reliability estimates are more than adequate. Thus, the revised version of the CEE, which was termed Predicting Learner Advancement through Cooperative Education (i.e., PLACE), appeared to generate reliable and valid scores for the underlying sample.

Mormative Data

Descriptive statistics were computed for each scale. These are displayed in Table 2. Additionally. the percentile rank norms for the total sample were calculated. Although not presented, these norms can be obtained from the first author upon request.

Invariance of Scales

A series of independent t-tests (assuming unequal variances), using the Bonferroni adjustment (Huberty, 1994), revealed that females (M = 50.35, SD= 7.92) had statistically significantly (t = 4.65. p < .001; Cohen's d = .20) higher scores than did males (M = 48.58, SD = 9.13) with respect to the Work Skills Development subscale. Similarly. females (M = 35.48, SD = 5.02) had statistically significantly (f = 4.34, p < .00 I: Cohen's d = .20) higher scores than did males (M = 34.38, SD = 5.52) with respect to the Academic Functions/

Table 2
Means and Standart Deviations of PLACE Subscales

Achievement subscale. Also, females (M = 125.71, SD= 16.40) had statistically significantly (t= 4.65, p < .00 I; Cohen's d = .20) higher scores than did males (M = 122.07. SD= 18.43) with respect to the total scale. However, although females (M = 39.13, SD= 6.82) had higher scores than did their male counterparts (M = 38.46, SD= 6.66) with respect to the Career Development subscale, this difference was not statistically significant after the Bonferroni adjustment was applied (t = 2.11, p = .04). Using Cohen's (1988) criterion, the effect sizes pertaining to the significant differences were small to moderate.

With respect to ethnicity, a series of independent r-tests (assuming unequal variances), using the Bonferroni adjustment. revealed (1) no statistically significant (t = -2.04, p > .05) difference between White (M = 48.99. SD= 8.79) and minority (M = 50.13. SD= 8.58) students with respect to scores on the Work Skills Development subscale; (2) no statistically significant (t = -.16, p > .05) difference between White (M= 38.68, SD= 6.68) and minority (M = 38.74, SD = 6.42) students with regard to scores on the Career Development subscale; (3) no statistically significant (t = -1.42, p > .05) difference between White (M = 34.66, SD= 5.31) and minority (M = 35.16, SD= 5.38) students with respect to scores on the Academic Functions/ Achievement subscale; and (4) no statistically significant (t = -1.38, p > .05) difference between White (M = 122.74, SD= 17.81) and minority (M = 124.48, SD= 17.91) students with respect to the total scale scores.

Finally, a series of Pearson product-moment correlation coefficients indicated a statistically significant positive relationship between the number of cooperative education work terms and scores on the Career Development subscale (r = .10, p < .001). However, using Cohen's (1988) criterion, this relationship represented a very small effect size. Consistent with this finding, a trend analysis revealed a linear trend linking scores on the Career Development subscale to the number of cooperative education work terms; however, no higher-order trends (e.g., quadratic, cubic, quartic) were noted. Additionally, no statistically significant relationship was found between the number of work terms and the following subscales: Work Skills Development (r = .04, p > .05), Academic Functions/Achievement (r = .01, p > .05) , and the total scale (r = .05, p > .05). Similarly, no polynomial trends emerged involving these subscales.

Discussion

The purpose of the present investigation was to develop an instrument that assesses students' perceptions of their cooperative education learning experiences. Using statistical techniques, a scale was developed, entitled, Predicting Learner Advancement through Cooperative Education (i.e., PLACE), which appears to have good psychometric properties. As such, the current study makes an important contribution to the area of cooperative education by providing an instrument that can be used to assess students' attitudes toward their cooperative learning experience. Such an instrument can be used to make programmatic changes. However, an essential next step in the process of validating the PLACE is to administer it to other samples, and then to utilize confirmatory factor analytic techniques to test the three-dimensional structure that emerged in the current study (see for example, Onwuegbuzie, Bailey, & Daley, 2000). Moreover, future researchers should consider including in their sample a cohort of students enrolled but not yet placed in their respective cooperative education programs. These pre-co-op students would provide a benchmark against which a possible ceiling effect could be measured. In addition, longitudinal studies could be undertaken in which students' learning outcomes are tracked over time.

The PLACE instrument was found to contain the following three subscales: Work Skills Development, Career Development, and Academic Functions/ Achievement. This conceptualization is somewhat consistent with Fletcher (1989), who theorized the following clusters: career development, career progress, and personal growth; and to Fletcher (1991), who identified personal development, career development, and academic achievement. Moreover, the present findings provide partial support for the four-component model of cooperative education. Although a clear personal development factor did not emerge from the factor analysis, the fact that two personal-development items clustered together with loadings greater than .5 suggests that this fourth element of the model should not be dismissed. Indeed, replications of this study are needed to explore further the personal- development factor and the four-component model of cooperative education.

Interestingly, no ethnic differences were found with respect to the subscales of PLACE or to the total scale. This finding is encouraging because it suggests that the cooperative education experience is equally beneficial to students across ethnic lines. On the other hand, some gender differences were revealed. Specifically, females were more positive about their cooperative education experiences in the areas of work skills development and academic functions/achievement, as well as overall. This finding suggests that cooperative education is particularly beneficial for females. Unfortunately, it is beyond the scope of the present inquiry to determine why this occurred. Thus, future research should seek explanations for this interesting and encouraging finding.

Somewhat surprising was the finding of no statistically significant relationship between the number of cooperative education work terms and students' perceptions of whether their cooperative education provided relevant practical experience with respect to work skills development or academic functions/achievement. Additionally, although a statistically significant positive relationship emerged between the number of cooperative education work terms and the extent to which students perceived that their cooperative education provided relevant practical experience with respect to career development, this association was small.

Although it may appear that these findings do not appear to support educators (Cooperative Education Network, 1996) who advocate a multipleterm cooperative education model that comprises at least 20% of the students' curriculum, such a conclusion should not be made with any confidence. Indeed, an examination of the item responses revealed that the mean of all 34 items of the CEE ranged from 4.85 (oral presentation skills) to 6.00 (retention). Bearing in mind that these items represented a seven-point scale, these mean scores indicate strongly that students are overwhelmingly positive about their cooperative education experiences. Thus, it is very likely that in examining the relationship between the number of cooperative education work terms and students' perceptions of whether their cooperative education provided relevant practical experience with respect to work skills development, career development, and academic functions/achievement, statistical regression was a threat to internal validity (Gay & Airasian, 2000; Onwuegbuzie, in press).

Specifically, the item mean scores suggest that a ceiling effect was present, whereby students, regardless of the number of cooperative education work terms completed, were extremely positive about their cooperative education experiences, thereby attenuating the relationship between the amount of exposure to work terms and their attitudes towards the cooperative education program. Indeed, students who only had one cooperative education work term had mean item responses that ranged from 4.78 (oral presentation skills) to 6.02 (retention)-which suggests that even these students were extremely positive about their cooperative education experiences.

Because of the use of a convenience sample, the external validity of the findings was threatened. In other words, it is possible that the lack of random sampling utilized in this investigation led to an unrepresentative sample that undermines the generalizability of the results. However, the fact that the sample included three of the largest cooperative education programs in the United States likely increased its representativeness.

It is clear that the CEE. in its original form, was not sensitive enough to discriminate different populations of students-particularly with respect to the number of work terms. This poses a threat to internal validity. As such, the authors have decided to change the response options of future versions of the PLACE instrument to include explicit definitions for each answer choice (i.e., 1 = decreased significantly, 2= decreased moderately, 3 = decreased slightly, 4 = no change, 5 = increased slightly, 6 = increased moderately, and 7 = increased significantly). Inclusion of these specific definitions will transform the PLACE into a Likert-type scale. Indeed, Likert-type formats provide more explicit direction to respondents than do ordinary scales such as that utilized for the CEE.

In addition. the researchers intend to include several open-ended questions in the next PLACE instrument. This mixed-methods approach should yield richer and thicker data to provide additional information regarding cooperative education student learning outcomes. Interestingly, Van Gyn, Cutt, Loken, and Ricks (I 997) strongly argued for the use of qualitative methods in the examination of the educational outcomes of co-op programs.

Cooperative education administrators should not assume that learning occurs in students merely because they are placed in a cooperative education setting. Rather, regular program evaluations are needed. As Van Gyn et al. (1997) noted, finding important differences within cooperative education programs that facilitate student educational outcomes is just as important as identifying commonalities. Consequently, we hope that the PLACE instrument will provide educators with a means to conduct these assessments.