Standardization of methods and tools is recognized by most researchers as a powerful mechanism for increasing the generalizability of research outcomes (Anderson, Hughes and Permaul, 1984). In experience-based education research, whether oriented toward cooperative education or other experiential approaches, successful standardization efforts could generate broadly applicable research results. This is important, particularly in a young field such as ours, where most existing research derives from practitioners' self-developed evaluations of their own programs. While such examination is valid, and is in fact the source of most of our knowledge to date (Anderson, 1983), research documenting experience based learning outcomes in a generalizable way is far more persuasive. In this brief paper I will present examples illustrating the potential approaches for implementing standardization. It is my hope that cooperative educators, together with their colleagues in other areas of experiential education, will join in a concerted effort to develop and use standardized research tools and methods. Out of such cooperation, a field of proven educational excellence can emerge.
In our bibliography of experiential education research (1983), Jenny Anderson, Jane Permaul and I cited dozens of studies. Rather than forming a consistent, cohesive body of information, the results constitute scattered tidbits teasing us with findings that cannot be compared across studies. As already mentioned, most of this research results from local program evaluation efforts intended to meet local needs. This is unfortunate because experience-based learning, while increasingly recognized for its value as a career development tool, remains suspect as a legitimate learning approach in the traditional academic curriculum. Research allowing comparison of outcomes across studies would greatly enhance our understanding of both the educational impacts of experience-based learning and the most effective approaches to achieving desired outcomes. The key to achieving such comparable outcomes is development of standardized tools and methods.
One approach to standardization is in progress at the University of California at Davis where we have developed and pilot-tested a questionnaire intended for use by multiple researchers. Sixty percent of the items concern learning outcomes and demographic information of general interest to most experiential/co-op programs. The remaining forty percent of the instrument allows individual researchers to develop their own program-specific questions ( consistent with an established questionnaire format). Computerized analytical software capable of utilizing data from both sections of the questionnaire to compute statistical relationships is now under development. Ultimately, this questionnaire and the associated software will be made available to other experiential researchers and programs. While the variability in programs that might be thus examined limits the applicability of outcomes in the specific case, the capacity to develop a large cohort of responses (taken together with results from a companion instrument designed to identify characteristics of the program(s) involved) can, like the U.S. Census survey, provide a tremendous resource of general information about internship programs, users and outcomes. Such survey research, though often criticized for its focus on self-reported outcomes, its demographic emphasis, and its reliance on simplistic paper and pencil instruments (Hamilton, 1980), is a potentially powerful and very cost effective approach to research, dealing with large sample populations.
Another useful approach to developing standardized survey instruments might take the form of an "item bank." Such a resource would consist of a microcomputer data base storing a wide variety of refereed questions for use by researchers. As these items are tested through use in various instruments, validations and reliability indices can be defined. The microcomputer format of such an item bank would greatly facilitate its updating and dissemination.
A third approach to research that would be greatly enhanced by availability of standardized tools and methods is ethnography. Ethnographic studies examine behavioral and contextual variables using in depth observation. Typically they are based on small populations which provide a wealth of useful information for the specific case, but are difficult to use in developing statistically valid inferences. The use of standardized tools for examining learning settings and interactions would allow comparison and consolidation of findings across several studies.
As ethnographer David Moore of Cornell has suggested (1984), these studies must be more than the "fast and dirty ethnographies in which the observer spends one or two hours at the site." They require systematic, repeated visits, ""lasting long and occurring as frequently as possible." In this heavily time-intensive situation, researchers cannot study large, "statistically significant" populations. Such significance must be brought about by consolidating the findings of many studies.
Using standardized tools and methodologies, thirty researchers, each doing ten to fifteen in-depth studies, could generate nearly five hundred thorough observations. While some elements of these studies might not be consistent, those that are would represent a level of significance that could not be ascribed to chance alone. Resources facilitating such substantive collaboration include microcomputer support of data collection and analysis and video-based training and testing of observers.
To amplify briefly, video-based training of observers would increase the likelihood of consistent rating practices. Further, microcomputer programs could be developed to provide a "cross-check" of observers, thus highlighting possible observer and rating bias. Such tools would provide powerful support of ethnographic methods allowing their broader use, and consequent effect, in experiential education research. The wide use of such intense methods would have a high probability of elucidating the key features of effective field-based learning.
Studies of personal and cognitive development are another arena where standardization would be of tremendous benefit. For instance, armed with the findings of collaborative ethnographic studies, developmental research could be more effectively targeted to the examination of highly promising developmental features of experiential learning. Because of the control and treatment groups, and pre- and post-tests required, developmental studies are often prohibitively expensive to undertake. However, such approaches provide more tangible evidence of learning growth or change than simple self-report surveys are capable of demonstrating. Again, standardization in the form of consistent protocols and matching of key programmatic and participant characteristics can enable several small group studies to produce results on a par with a single large population study.
In all the examples cited, powerful computational support using affordable microcomputer resources is an assumed prerequisite. Computational tasks include data entry and storage in support of subsequent manipulation, item bank management, statistical analyses, report generation and instrument development ( even collection of field data for ethnographic studies could be supported). Without computer support, extensive collaborative development of research would be unlikely. Computers provide the necessary analytical base for comparing large amounts of data among different research projects. Simultaneously, they allow a level of communication, through electronic mail and exchange of data diskettes, that promotes interactions previously impossible.
In ethnographic and developmental studies, microcomputers make possible the organized and efficient storage and retrieval of extensive amounts of data. In survey research, microcomputer databases can readily maintain detailed item banks for use in formulating instruments. Furthermore, the modification of existing instruments, as well as the generation of new ones, is greatly facilitated.
Of course, the efficient utilization of microcomputer resources requires some expertise which, while not hard to learn, is beyond the focus of this paper. The following extract from Malyj, Smith and Horowitz (1984) is a useful summary of key points:
Specific brief recommendations on hardware and software are also in order:
In conclusion, while standardization is not appropriate for all research or researchers, in cooperative education and experiential education, the benefits of appropriate standardization are incontrovertible. With the new generation of tools available and the current national emphasis on quality education, it is particularly important and timely to implement standardized research tools and approaches in the study of experience-based learning.