Takako Hirokawa, Noah Finkelstein, and H. J. Lewandowski†
Department of Physics, University of Colorado Boulder, Boulder, CO 80309 (Dated: March 4, 2014) In response to national calls to better align physics laboratory courses with the way physicists engage in research, we have developed an epistemology and expectations survey to assess how students perceive the nature of physics experiments in the contexts of laboratory courses and the professional research laboratory. The Colorado Learning Attitudes about Science Survey for Experimental Physics (E-CLASS) evaluates students’ shifts in epistemology and affect at the beginning and end of a semester. Also, at the end of the semester, the E-CLASS assesses students’ reflections on their course’s expectations for earning a good grade. By basing survey statements on widely embraced learning goals and common critiques of teaching labs, the E-CLASS serves as an assessment tool for lab courses across the undergraduate curriculum and as a tool for PER research. We present the development, evidence of validation, and initial formative assessment results from a sample that includes 45 classes at 20 institutions. We also discuss feedback from instructors and reflect on the challenges of large-scale online administration and distribution of results.
I.
INTRODUCTION
Laboratory courses offer significant opportunities for engagement in the practices and core ideas of science. Laboratory course environments typically have apparatus, flexible classroom arrangements, low student/teacher ratios, and opportunities for collaborative work that promote students’ engagement in a range of scientific practices (e.g., asking questions, designing and carrying out experiments, analyzing data, developing and refining models, and presenting results to peers). Creating such opportunities requires significant investments in physical space, sophisticated equipment, and instructor support. Despite the abundant opportunities and resources in many laboratory courses, concerns are frequently raised about how effective such courses are at fulfilling their potential.1,2 Problems often manifest themselves as a gap between the kinds of practices going on in the laboratory classroom and the practices going on in professional scientific research and engineering labs. Sometimes gaps result from differing goals between lab courses and research experiences, while other times gaps result from good intentions, but poor implementation of the goals within the curriculum. There are many calls to transform lab courses coming from the physics education community,3 the life sciences,4 and national science policies promoting the retention of STEM majors and the development of the STEM workforce.5 One theme that spans these calls is students should develop habits of mind, experimental strategies, enthusiasm, and confidence in research through effective laboratory courses. A variety of responses have emerged for improving laboratory experiences within the physics curriculum. Some laboratories have introduced new educational technologies (e.g., microcomputer-based labs6 and VPython7,8 ), others have added an emphasis on particular scientific
practices (e.g., measurement and uncertainty9,10 , developing testable questions and designing experiments11,12 , and scientific argumentation13 ), while others have pushed the lab course closer to cutting edge research by introducing modern physics concepts and apparatus (e.g., single photon quantum optics experiments14,15 ), while others have demonstrated improved conceptual learning gains through research-based lab activities.16 The diversity of responses reflects both the diversity of goals for the laboratory and the flexibility and adaptability of the laboratory environment to meet many different goals. Given this wide range of modifications to the laboratory curriculum, there is a need for evaluation tools for lab courses that allow instructors to iteratively improve their course offerings, and for tools to give PER researchers insight into effects of different course modifications on student learning. We have developed, validated, and collected initial results on a national-scale for a new epistemology and expectations (E&E) survey17–22 called the Colorado Learning Attitudes about Science Survey for Experimental Physics (E-CLASS).23,24 An E&E survey is well-suited to assessing the present situation in laboratory instruction for three reasons. First, E&E surveys are typically not directly tied to specific content, which increases their applicability in the already-existing wide range of laboratory courses. Second, the habits of mind and ways of thinking probed in E&E surveys represent a significant course goal for many instructors. Third, E&E surveys are of most value when evaluating educational environments that have significant differences from professional practice. It is in these classroom environments where we expect to see the largest splits in epistemology and expectation between students and experts. As lab courses are transformed to align more closely to prepare students for research, we expect gaps between students’ and experts’ epistemological beliefs about experiments
2 will also narrow. The process for the development and validation of the E-CLASS as a tool for formative assessment of instruction broadly aligns with the procedures laid out in Adams and Wieman’s article on the Development and Validation of Instruments to Measure Learning of Expert-Like Thinking,25 which aligns with the Standards for Psychological and Educational Assessment.26 Our process begins with the identification of key areas of importance to instructors where students often differ from experts. We then present our overall design criteria for the survey development. Our development continues on to the validation and refinement of a ready-to-administer online assessment tool. Initial results from the Fall 2012 and Spring 2013 semesters are presented as they appear in a typical post-semester report sent to instructors as a formative assessment tool. We conclude by giving an overview of the level of participation across all classes, summarizing difficulties in achieving consistently high levels of participation, and looking at future research questions that can be answered using the E-CLASS. introductory lab course. Does this impact their general enthusiasm for doing experiments in a research setting? Students repeat historic experiments with known results rather than asking their own questions and designing experiments to investigate them. Does this impact how they think about the roles of asking questions, design, and confirmation in conducting research? Students are often confronted with a range of new measurement tools and apparatus. Do our students treat the apparatus as something to be understood and explored or as a “black box”? Uncertainty analysis and error propagation has played a significant role in our curriculum as well. Do our students see uncertainty as a tool for better understanding their data and refining their experiment, or is it just an algorithmic calculation that comes at the end of the lab report? As the final step of most of our lab activities, students complete a lengthy written report that often takes more time to complete than they spend working with the equipment and taking data. Do students see writing lab reports as an exercise in scientific communication or merely in meeting the instructor’s grading expectations? For fear of cheating in our large introductory course, students have often been required to work individually in the lab. When students work by themselves, does it affect the role they see for collaboration within scientific research or lessen the value they place on presenting ideas to peers? These kinds of concerns helped us target the E-CLASS statements on areas where we may see larger signal and provide relevant information for formative assessment. The final criteria, that the E-CLASS should accurately capture some aspects of students’ thinking and learning, is explored in the following sections as we articulate more clearly what is probed (Sec. III), and then present evidence of validity (Sec. IV).
II. IDENTIFYING DIFFERENCES BETWEEN EXPERTS AND NOVICES IN EXPERIMENTAL PHYSICS
Like any tool for assessment of instruction, the ECLASS must meet the triple criteria of (1) measuring something that experts and instructors care about (i.e., it should be aligned with widely accepted course goals), (2) targeting areas where students may not be meeting instructors’ goals, and (3) accurately capturing some aspects of student thinking and learning. In order to measure something that most instructors care about, we aligned the survey with a set of consensus learning goals developed for our lab curriculum for physics majors,27 though there is considerable overlap with similar goals established by AAPT for the introductory labs.3 Broadly, these goals were: modeling physical systems, modeling the measurement tools, statistical analysis, design of experiments and apparatus, troubleshooting, communication of scientific arguments, communicating in genres relevant to scientists, and technical lab skills using common lab equipment and software. Beyond these learning goals that emerged through a departmental consensus-building process, we followed other E&E surveys such as the Colorado Learning Attitudes about Science Survey (CLASS) by also considering students’ affect and confidence when doing physics experiments and their identity as researchers. In order to ensure the E-CLASS meets the second criteria of probing areas where students may not be meeting instructors’ goals, we aligned the survey with several common challenges that instructors have found in our lab courses at the University of Colorado Boulder and are common elsewhere. We knew many students found the labs very time-consuming and many students disliked our
III. A.
SURVEY DESIGN CONSIDERATIONS
Measuring epistemology and expectations in the lab
The E-CLASS was designed to survey students’ epistemological beliefs and their expectations of students. Epistemology refers to theories of the nature of knowledge, knowing, and learning in the discipline. Epistemology, in the context of the lab, means defining what is viewed as a good or valid experiment and what are the appropriate ways to understand the design and operation of an experiment and the communication of results. Expectations, on the other hand, deal with the expectations that an instructor has for what students should be doing in the class—the kinds of knowledge and learning that are expected and rewarded in the laboratory course. We included expectations statements in addition to epistemology statements because course expectations are something an instructor can influence through explicit framing, grading priorities, and classroom culture. In order
3 to assess the impact of the course, the E-CLASS provides pre and post measures of students’ epistemology, while also providing a post-only look at expectations. The expectations and epistemology statements cluster around a similar set of ideas to enable us to look at the relationship between epistemology and expectations in the course (see Appendix for the full list of statements). As a tool for formative assessment of a course, we wanted to assess many aspects of experimental physics. Probing a wide range of epistemological statements allows the survey to have relevance in courses that have a wide range of goals. We also take a resources perspective28 on the nature of these epistemological beliefs. This means that we don’t expect students to hold particularly coherent epistemological stances as though they had some well-developed world-view of doing physics experiments. Instead, we expect students to draw on a range of resources and experiences when responding to each statement, and responses might sometimes be in apparent contradiction with each other. Because of this resources perspective, we do not use the survey as a tool to evaluate individual students, but as a coarse measurement of the epistemological state of the class.17 be less robust.
IV. ITERATIVE VALIDATION AND REFINEMENT OF THE SURVEY A. Lessons from early student interviews
B.
Format and structure of the survey
For ease of administration, we followed the Colorado Learning Attitudes about Science Survey in the use of Likert-scale responses to statements.22 However, unlike the CLASS, we did not develop categories for clustering questions in the data analysis. The E-CLASS questions still form groups that align with the course goals described in Sec. II, but those groups were not used to create categories statistically robust categories via factor analysis. The reasons for the omission of categories are two-fold. The first deals with the nature of actual lab courses. It is possible for a course to prioritize one idea, while ignoring another related idea. In other words, the category may make sense to an expert, but the correlation may not be reflected in students’ responses. For instance, it seems reasonable that “communicating results using scientific arguments” and “communicating scientific results to peers” are ideas that could be grouped in the same “communication” category. Yet often courses are structured so that students’ results are only communicated to the instructor, while communication to peers is ignored as a course goal. So although audience and argumentation are each aspects of communication, they can be emphasized independently in a course. Rather than grouping those statements, we let them stand on their own when presenting survey results to instructors since they each deal with different actionable responses to the format of the lab course. The second, and less significant, reason to avoid categories is because we don’t expect students to have particularly coherent perspectives on the nature of experimental physics, implying related statements will have less correlation and categories will
The initial development of the E-CLASS survey was based closely on the well-studied CLASS survey22 that has found significant use in undergraduate physics courses both in the United States and internationally.29,30 We had reason to believe a straightforward adaptation might be possible. A similar process of adapting CLASS from physics to chemistry31 was accomplished through a straightforward modification of many statements by changing the word “physics” to “chemistry,” by focusing attention on chemical formulas in addition to mathematical formulas, and by adding 11 new statements involving chemistry-specific ideas. Validation interviews and faculty surveys for the CLASSChem showed the survey had a similar level of expert and student validity as the original CLASS-Phys. We developed our own minimal adaptation by replacing many uses of the word “physics” or “physics problem” with “experiment,” and developing several new questions as well. But in a series of 11 student validation interviews in Fall 2011, a substantial number of issues arose. One of the most significant issues was that the phrase “physics experiment” is used to refer to activities in a lab class and to the kinds of experiments that professional researchers engage in. Depending on the exact statement, students switched between a context of classroom laboratories, everyday life, and professional physics experiments, and their answers could depend very strongly on which context they chose. In addition, students often commented that they were unsure whether they should answer “What do I think?” or “What should I think?” when asked to rate their level of agreement about a statement like “When doing an experiment, I just follow the instructions without thinking about their purpose.” The final difficulty of this early version of the survey was that it did not probe many aspects of experimental physics that we viewed as important (i.e., it was too disconnected from our learning goals). Because of these early interviews and a desire to more strongly link to the consensus learning goals, the later iterations of E-CLASS began to differ more significantly from CLASS-Phys. The primary changes were (1) that we focused the context of students’ responses to either be about “experiments for class” or about “research,” (2) we also eliminated confusion about “should I...” vs “do I...” by asking students paired questions that distinguished between what they thought and what an expert might think, similar to the paired format using the FCI32,33 and CLASS-Phys,34 and (3) the statements in the survey more effectively spanned our assessment goals described in Sec. II.
4
1. When doing an experiment, I try to understand how the experimental setup works. Strongly Disagree 1 2 3 4 5 Strongly Agree What do YOU think when doing experiments for class? What would experimental physicists say about their research? 2. If I wanted to, I think I could be good at doing research. Strongly Disagree 1 2 3 4 5 Strongly Agree What do YOU think when doing experiments for class? What would experimental physicists say about their research?
C.
Evidence of validity through student interviews
FIG. 1. Two epistemological beliefs statements as they are presented to students in the pre- and post- E-CLASS online survey.
Very Unimportant 1 2 3 4 5 Important How important for earning a good grade in this class was understanding how the measurement tools and sensors work?
1.
Very Unimportant 1 2 3 4 5 Important How important for earning a good grade in this class was reading scientific journal articles? 2.
FIG. 2. Two expectations questions as they are presented to students in the post-semester E-CLASS online survey.
B.
Creation of a final version
Figures 1 and 2 show a few statements from the ECLASS in the format they were presented to students in the online surveys during Fall 2012 and Spring 2013. In order to come to this final format and question wording, thirty one additional interviews were conducted during Spring and Fall 2012. These interviews were focused on three aspects of the survey design. One aspect was refining the question wording to clarify the context of students’ epistemological reflections. Through these interviews, the paired survey questions evolved from “What do YOU think?” and “What would a physicist say?”, which were used in the paired CLASS-phys34 , toward the current wording “What do YOU think when doing experiments for class?” and “What would experimental physicists say about their research?” The second emphasis of the interviews was on the wording of individual statements to make sure they could be readily interpreted by all levels of undergraduate students. The third focus was on how students interpreted the phrase “experimental physicists” and whether that could be replaced with more general language of “scientists.” We discuss each of these aspect in turn.
In order to ensure reliable interpretation of the context for students’ responses to the epistemology statements, we found it necessary to clarify “What do YOU think?”, which was the prompt used in the paired CLASS-Phys. Most frequently, students answered “What do YOU think?” by reflecting on their prior experience in lab classes, but students with prior research experience, especially upper-division students, often would switch to a context of their own personal research experience if it seemed to fit. The final wording “What do YOU think when doing experiments for class?” ensures students maintain a consistent context for reflecting on “doing physics experiments.” This wording also aligns with the default interpretation of students who have never taken a physics lab prior to taking the E-CLASS. Such students typically referred to their experiences in a high school or introductory college-level science lab or to inclass demonstrations that involved observations of real phenomena. The question “What would experimental physicists say about their research?” also went through successive iterations. Because experiments exist in very different forms in both research and the teaching labs and professional physicists are part of both of those environments (as either teachers in teaching labs or researchers in research labs), we restricted the context to research. The use of “experimental physicists” rather than “physicists” arose in the interviews to clarify the question for upper-division students who are becoming more aware of the professional culture within physics. In the interviews, it was suggested theoretical physicists may hold differing views, particularly regarding statements about enjoyment while doing experiments or building things and working with their hands. In summary, the use of the paired statements “What do YOU think when doing experiments for class?” and “What would experimental physicists say about their research?” clarifies students’ interpretation of the questions and also clarifies the meaning of the E-CLASS as an assessment tool. In this final form, the E-CLASS measures students epistemological beliefs about physics experiments in the two contexts where such experiments occur: in the laboratory classroom and in the research lab. The E-CLASS becomes a tool to assess students’ perception of the gap between their own classroom experiences and what they perceive to be the nature of authentic science. The second focus of these validation interviews was to refine individual question wording. For instance, in a trial administration of the E-CLASS in Spring 2012 to introductory students at CU, we found that the majority of students agreed with the statement “I can read a journal article for understanding.” Given the difficulty of reading the professional literature for graduate students, we were surprised that so many introductory students would agree with this statement. Through targeted validation
5 interviews, we found that many students set a low bar for “reading for understanding” that was equated with “getting the gist of the article.” Also when discussing “journal articles,” some introductory students mentioned popular science articles (e.g., Scientific American) rather than professional research literature that was intended in our question. The final question wording was substantially modified to address these findings and now reads: “Scientific journal articles are helpful for answering my own questions and designing experiments.” For other statements, particular attention was paid to technical terms, and where appropriate, question wordings were simplified. For example, “Doing error analysis (such as calculating the propagated error) usually helps me understand my results better.” was simplified to “Calculating uncertainties usually helps me understand my results better.” Not all technical language was avoided, but it was simplified whenever possible. The remaining technical terminology was retained so that the survey would continue to address key aspects of experimentation for upper-division physics majors. The third aspect of the interviews dealt with the concern of some instructors that most introductory physics courses primarily serve non-physics majors, and the use of the phrase “experimental physicists” makes an unhelpful distinction between experimental physicists and other scientists and engineers. In particular, some faculty were worried the language may alienate students who are not physics majors by suggesting the material is only relevant to this small group of people called “experimental physicists.” A final series of interviews was conducted to better understand what comes to mind when students think about “physicists” and “experimental physicists.” The outcome was that most students were more aware of physicists famous for their theoretical ideas (e.g., Newton and Einstein), and had trouble naming any experimental physicists. In addition, many introductory-level students were unfamiliar with the distinctions of theorist and experimentalist, but they still interpreted “experimental physicists” straightforwardly as “physicists who do experiments.” So the clarification does not obscure students’ interpretation, but may help depending on whether a student is aware of the broader community of professional physicists. We also investigated replacing the term “experimental physicists” with “scientists.” In interviews, students found “scientists” too general to answer the questions because they realized that scientists’ typical activities could differ substantially between disciplines (e.g., an experimental physicists versus a field ecologist). Lastly, even though the context was specific to experimental physics, most students still felt that the statements emphasized broadly relevant experimental skills that could be applied to their own discipline. In order to gather evidence of validity across the broad population of students taking physics laboratory courses, altogether 42 interviews were conducted. There were 24 students interviewed who had never taken any college physics lab classes, 8 were currently enrolled in an introductory physics lab, and the remaining 10 were physics majors who had already taken upper-division physics lab classes. The pre-introductory and introductory-level students included a mix of physical science majors, life science majors, and engineering majors. The population of 42 interviewees included 22 male and 20 female.
D.
Content validity
Another key aspect of developing an assessment tool around epistemology is ensuring faculty have consistent responses to the various survey items. We establish the content validity of the E-CLASS when experts find the questions relevant (as described in Sec. II) and have a consistent response to the statements. To date, we have collected 23 expert responses (3 full time instructors and 20 with a blend of teaching and research responsibilities) from both primarily undergraduate serving institutions (N = 7) and PhD granting institutions (N = 16). Faculty were asked to respond to the thirty statements on the epistemological portion of the survey considering their own perspective as a faculty member and/or researcher. In these responses, 24 of the 30 statements had an expert consensus of 90% or higher, and all 30 statements had consensus of 70% or higher. The statements and distribution of responses with lower than 90% consensus are summarized in Table I. Despite the fact that a few of the questions had lower levels of expert consensus, we justify the inclusion of these statements for the following reasons. The first three statements in Table I all relate to key learning goals of many labs: developing scientific arguments based on data, evaluating uncertainty in data, and understanding the theoretical ideas underlying the lab. Although there was some small disagreement of the importance of these, they still remain important in many lab curricula and in the research programs of many faculty. The fourth statement, about asking help from an expert, has an awkward context in the faculty survey, but we left the statement in for completeness as it does have a clear meaning in a classroom context for students. Perhaps the most surprising and interesting results from the expert validation are two statements with the lowest consensus. Over 25% of respondents did not agree that working in a group is an important part of doing physics experiments, which might indicate that faculty have a variety of ways in which they go about their research depending on their particular research expertise and nature of their projects. We retain this question because group work is typically an attribute of authentic research and also because there are many pedagogical benefits to working in groups. Finally, responses to the statement with the lowest consensus showed that about 30% of instructors did not agree that nearly all students are capable of doing a physics experiment if they work at it. This finding seems to indicate that faculty and instructors think physics experiments are difficult. Most research faculty have many stories to
6 tell of highly qualified students struggling in the lab, so perhaps their own experience suggests not all students are capable of doing PhD-level experiments. We retain this statement because we want to know whether students view physics experiments as accessible to a broad population. A key motivation for improving laboratory instruction is improving retention in STEM, so it is critical that students see technically challenging aspects of STEM, such as doing physics experiments, as something accessible to many people.
When doing an experiment, I try to understand how the experimental setup works. What do YOU think when doing experiments for class? N=52 19 Strongly 2 7 19 Agree Agree 3 4 5
Post
Neutral Disagree Strongly Disagree
1
1
7 2
1
Evidence of convergent validity of an assessment tool shows that the assessment results are correlated with other established measures, such as course performance or GPA. On similar assessment tools, such as the CLASS, it is found that students with more expert-like perspectives on physics and learning physics tend to do better in their physics courses.35 To date, we have not had access to course grade data to correlate with E-CLASS scores, though we plan to do this analysis in upcoming semesters. On the other hand, our current data set does contain a student population that spans freshman through seniors. We may expect that students who are majoring in physics and are taking upper-division labs would tend to have more expert-like views. When comparing students in algebra-based physics labs to students in upper-division labs and averaging across all 30 statements, we find that upper-division students have larger fraction of expert-like responses in both the classroom context (mean expert-like fraction = 0.66 vs 0.61, p-value = 6 × 10−6 , effect size = 0.38) and in the context of professional research (mean expert-like fraction = 0.82 vs 0.78, p-value = 2 × 10−4 , effect size = 0.28).
FIG. 3. The pre and post response data for a single statement summarized as a 2D histogram. The number insider each box corresponds to the number of students with each (pre,post) response.
V.
E-CLASS RESULTS AS A FORMATIVE ASSESSMENT TOOL
The E-CLASS was designed with two purposes in mind. The first purpose was as an assessment tool for laboratory courses. The second was as a physics education research tool. The results that follow demonstrate how the E-CLASS has been used as a formative assessment tool for 45 classes during the Fall 2012 and Spring 2013 semesters. We postpone the discussion of E-CLASS as a PER tool for comparative evaluation of different laboratory curricula for a later publication. One significant feature of the E-CLASS is that at the end of the semester, instructors are provided with a detailed results report in PDF format with explanations, analysis, and figures. Figs. 3, 4, 5, 6, 7, and 8 and Table II are all full or abbreviated versions of those appearing in the instructor report. The report begins by using one of the 30 questions as an example for how the pre/post shifts are calculated. Fig. 3 shows the combined (pre,post)
data for a single statement. This information is then condensed to a pair of numbers—the fraction of student responses in agreement with experts on the pre-semester survey and on the post-semester survey. That pair of numbers is used to generate plots of pre/post shifts as shown in Fig. 4. Although Fig. 4 shows pre/post shifts for only three statements, instructors receive a full version with all 30 statements. Finally, the pre and post results from all questions can be further condensed into a single overall pre and post score for the class, as shown in Fig. 5. Whenever possible, we also provide a comparison with a group of students in similar level classes. The comparison group provides instructors with a baseline for evaluating whether or not their results are typical. Currently, we are using three comparison groups: non-calculus-based introductory physics, calculus-based introductory physics, or upper-division (anything after the introductory level). The results discussed so far in Figs. 3, 4 and 5 deal only with students’ responses to “What do YOU think when doing experiments for class?”, which is just one of the three statements that students respond to around a single idea. The second aspect of the E-CLASS survey is students’ views of what was expected of students for earning a good grade. The results of “How important for earning a good grade in this class was...” are shown in Fig. 6. Such a plot allows instructors to see whether students’ perceptions of the grading priorities for the class actually align with their own personal goals as instructors. The third area probed by the E-CLASS is students’ epistemology regarding physics experiments done for research. Data for this aspect of students’ epistemology are shown in green in Fig. 7. Although the data shown are just for 3 of the 30 questions, we typically find that stu-
DisStron ag gly r Dis ee ag ree Ne utr al Ag Str ree Ag ongl ree y
E.
Convergent validity
0
Pre
7
TABLE I. A list of E-CLASS statements with the faculty agreement less that 90%. Agree is the number of respondents who answered either “Agree” or “Strongly Agree”. Disagree is the number of respondents who answered either “Disagree” or “Strongly Disagree”. Consensus refers to the fraction of respondents in the consensus response. Statement If I am communicating results from an experiment, my main goal is to make conclusions based on my data. Calculating uncertainties usually helps me understand my results better. I am usually able to complete an experiment without understanding the equations and physics ideas that describe the system I am investigating. When I encounter difficulties in the lab, my first step is to ask an expert, like the instructor. Working in a group is an important part of doing physics experiments. Nearly all students are capable of doing a physics experiment if they work at it. Agree Neutral Disagree Consensus 20 19 0 2 2 4 1 2 19 0.87 0.83 0.83
0 17 16
5 4 4
18 2 3
0.78 0.74 0.70
What do YOU think when doing experiments for class?
Similar level classes Working in a group is an important part of doing physics experiments. I am usually able to complete an experiment without understanding the equations and physics ideas that describe the system I am investigating. When doing an experiment I usually think up my own questions to investigate. 0 0.2 0.4 0.6 0.8 1.0 Fraction of class with expert-like response
FIG. 4. Pre/Post changes in students’ personal views about “What do YOU think when doing experiments for class?” for your class (Red) and all students in similar level classes (i.e., introductory calculus-based physics classes) (Blue). The circles show the pre-semester survey values. The arrows indicate the pre/post changes. The shaded bars are 95% confidence intervals. The data shown are for a subset of 3 out of 30 statements.
Your class
TABLE II. Summary of class participation for a calculusbased physics lab class at a large public university. Number of valid pre-responses 69 Number of valid post-responses 65 Number of matched responses 52 Reported number of students in class 117 Fraction of class participating in pre and 0.44 post
Overall E-CLASS Score on "What do YOU think..." statements
Fraction of statements with expert-like responses
1.0 0.8 0.6 0.4 0.2 0.0
Your class N = 52
Similar level classes N = 174
0.65
0.69
0.61
0.68
Pre
Post
dents have much more expert-like views regarding what experimental physicists would say about their research than they do for their personal views about doing experiments for class. We also find that students’ views of researchers typically change less during the semester than their personal views. One immediate use of these data is to identify the statements with the largest epistemological splits between students’ views of classroom experiments and research experiments. For this particular class, the largest split occurred for “When doing an experiment, I usually think up my own questions to investigate.” (See Fig. 7) About 28% agreed when thinking
FIG. 5. Comparison between overall pre and post scores for students’ personal views about “What do YOU think when doing experiments for class?” Your class (Red) is compared with all students in similar level classes (i.e., introductory calculus-based physics classes) (Blue). The error bars show 95% confidence intervals. The overall mean shown here averages over all students and all statements on the survey.
about experiments done for class, while 90% thought experimental physicists doing research would agree. Other questions with large splits (a difference of 40% or larger)
8
Your class
How important for earning a good grade in this class was...
Unimportant
Similar level classes ...thinking up my own questions to investigate? ...understanding the equations and physics ideas that describe the system I am investigating? ...working in a group? Very Important
FIG. 6. Students’ views of the importance of different activities for earning a good grade in your class (Red) and in similar level classes (i.e., introductory calculus-based physics classes) (Blue). The data shown are for a subset of 3 out of 23 statements.
were: “I don’t enjoy doing physics experiments.” “Calculating uncertainties usually helps me understand my results better.” “Scientific journal articles are helpful for answering my own questions and designing experiments.” and “If I don’t have clear directions for analyzing data, I am not sure how to choose an appropriate analysis method.” We can also use the data presented in Fig. 7 to identify statements where students express the least expertlike view about professional research. For this class of calculus-based physics students, when asked “What would experimental physicists say...,” about 5% of students disagreed with the statement, “If I am communicating results from an experiment, my main goal is to create a report with the correct sections and formatting.” When faculty were given same statement, 96% disagreed, nearly the opposite result from students. This results persists across many classes. Among the 612 responses in the Spring 2013, 13% of all responses disagreed. Upperdivision classes had disagree fractions as high as 40% demonstrating upper-division students tended to have more expert-like views. However, the divide between students and experts is so striking that we plan to conduct follow-up interviews to see what students are attending to and how it might differ from experts. One hypothesis based on our own experience is that an overemphasis on well-formatted lab reports may be misrepresenting the priorities of scientific communication. The statement with the second least expert-like result is “The primary purpose of doing a physics experiment is to confirm previously known results.” Only about 40% of students disagreed when asked “What would experimental physicists say...,” while 100% of experts disagreed. In addition to summarizing the class’ responses to each individual statement and question, we also provide instructors with a summary of their students’ participation in the E-CLASS survey (Table II). The classroom participation data shown in Table II apply to the figures presented in Figs. 3–8. Finally, instructors are presented with the distribution of students’ majors in their own class and in similar-level classes, so they can see whether the composition of their class was similar to the comparison group or not. Also, instructors are provided with figures showing students’ interest in physics at the end of the course and their
change in interest in physics during the course (see Fig. 8). Currently, we know of six schools that are actively using the E-CLASS reports as an assessment tool for their curricula. Four schools are using it for evaluation of significant curricular changes to their introductory lab sequence, while two others are using it for evaluations of upper-division laboratory courses. We are actively soliciting feedback from instructors on how to make our survey and reports more useful for course evaluation. In response to feedback on the Fall 2012 E-CLASS reports, we now include: a summary table of class participation (Table II), an overall E-CLASS score for the entire class displayed as a bar graph (Fig. 5), and the “How important for earning a good grade...” information is presented graphically rather than as a table (Fig. 6). Additional input from instructors will allow us to further condense our reports and bring out the most salient features. Our efforts to provide efficient and helpful information to faculty about their courses and to have this information promote changes in classroom instruction is a goal we share with other current projects such as Data Explorer and Assessment Resources for Faculty (DEARFaculty), which is an assessment-focused extension of the PER Users’ Guide,36 and the 2013 American Association of Universities Undergraduate STEM Education Initiative,37 which is focusing on overcoming challenges to adopting best teaching practices, including assessing student learning and formative assessment of classroom practices.
VI. LARGE-SCALE SURVEY ADMINISTRATION AND PARTICIPATION A. Participation
During the Fall of 2012 and Spring of 2013, E-CLASS was administered to 45 different classes at 20 institutions in 3 countries. The institutions represent a wide crosssection of institution types (7 Private, 13 Public), sizes (5 Small (1-3K), 3 Medium (3-10K), 12 Large(10K+)), and degree-granting statuses (1 associates, 5 baccalaureate, 3 masters, 11 PhD). The 45 individual classes in-
9
Personal/Professional Split
What would experimental... Working in a group is an important part of doing physics experiments. I am usually able to complete an experiment without understanding the equations and physics ideas that describe the system I am investigating. When doing an experiment I usually think up my own questions to investigate. 0 0.2 0.4 0.6 0.8 1.0 Fraction of class with expert-like response
FIG. 7. Comparison of changes in students’ personal views versus their views about professional physicists. Red shows the change in students’ response to “What do YOU think when doing experiments for class?” Green shows the change in students’ responses to “What would experimental physicists say about their research?” The circles show the pre-semester survey values. The arrows indicate the pre/post shift. The shaded bars are 95% confidence intervals. The data shown are for a subset of 3 out of 30 statements.
What do YOU...
During the semester, my interest in physics? N (Your class) = 52, N (Similar level classes) = 174
De
FIG. 8. Change in students’ interest in physics. Your class (Red) refers to your own class. Similar level classes (Blue) refers to a set of students in all classes at the calculus-based introductory-level.
cluded: 11 algebra-based introductory-level classes, 18 calculus-based introductory-level classes, and 16 laboratory classes beyond the intro-level, which were typically for physics and engineering physics majors. The introductory classes tended to be larger, many in the range 50-200 students, while the upper-division classes were typically smaller, mostly in the range 8-25 students. The median completion time on the Spring 2013 pre E-CLASS was 8 minutes (N =745), while for the post E-CLASS was 11 minutes (N =521). Although we received responses from a large number of institution and classes, the response rate in about half of those classes was disappointingly low. Fig. 9 shows the distribution of E-CLASS response rates for all 45 classes. Only 20 of the 45 classes had a matching pre/post response rate higher than 40%. By comparison, when other surveys, such as CLASS-Phys, are routinely administered at CU for a small amount of credit and with multiple reminders from the instructor, the response rate is typically between 45% and 60%. The lowest E-CLASS response rates occurred when faculty chose not to give any credit
Sta y
Inc rea sed
cre ase d
0.5 0.4 0.3 0.2 0.1 0.0
Fraction of Students
Your class
Similar level classes
for completion of the survey, which is contrary to established recommendations for achieving high levels of participation.25
B.
Administration
Delivering the survey online made it easy for instructors to adopt the E-CLASS in their classes. However, the full administration of the survey was still highly labor intensive and required many steps for each individual class. Based on these experiences, future versions of the E-CLASS will likely be administered in a more unified online environment. In this unified environment, instructors would be able to create an account for their class, enter basic information about their class and institution, get a unique survey link to send to their students, have immediate access to lists of students completing the survey, and have immediate access to the aggregate ECLASS report after the close date on the survey. We hope that by providing an integrated environment for the survey and results, instructors will receive information in a timely manner, that the E-CLASS can more easily be integrated into courses, that students will respond at a higher rate, and that there will be fewer errors in selecting the appropriate course names and course sections.
ed
sam
e
VII.
CONCLUSIONS
The E-CLASS survey was motivated by the evident gap between common student practices in many laboratory courses and the epistemological beliefs, habits of mind, and scientific practices essential for successfully engaging in research. The E-CLASS was developed as an epistemology and expectations survey to directly assess students views of doing physics experiments in both the classroom context and in the context of professional research. Initial results show evidence of some significant gaps between students’ epistemology of classroom experiments and research experiments (e.g., the role of
10 assessment tool, partial results from the instructor report for a calculus-based physics lab at a large public research university were presented. On-going studies include comparative evaluation of different laboratory curricula and the evaluation of laboratory activities in a massive open online course (MOOC). As the administration and processing of results continues to be streamlined, we plan to provide access to any interested instructors nationally and internationally.
FIG. 9. Distribution of participation levels for all classes taking the E-CLASS survey. The percentage is calculated by diving the number of matched pre/post responses by the total number of students reported to be enrolled in the class.
asking questions when doing experiments). Because evidence of validation has been gathered from a wide student population, the E-CLASS can be administered in any undergraduate physics lab, and to-date has received responses from 45 different laboratory classes at 20 institutions. In order to demonstrate its use as a formative
The authors would like to thank the CU-Boulder Physics Department and Physics Education Research Group for contributing to the learning goals process and providing feedback on early versions of the survey. We would also like to particularly thank the Advanced Lab Physics Association (ALPhA) community for their support in disseminating and providing feedback on a number of aspects of the E-CLASS development. This work is supported by NSF CAREER PHY-0748742, NSF TUES DUE-1043028, JILA PFC PHY-0551010, the CU Science Education Initiative, and the Center for STEM Learning NSF DRL-0833364. The views expressed in this paper do not necessarily reflect those of the National Science Foundation.
Appendix: List of E-CLASS Statements
∗ †
1
2
3
4
5
6
7
benjamin.m.zwickl@rit.edu Also at JILA, University of Colorado Boulder, Boulder, CO 80309 Ricardo Trumper, “The Physics Laboratory A Historical Overview and Future Perspectives,” Science & Education 12, 645–670 (2003). Avi Hofstein and Vincent N. Lunetta, “The laboratory in science education: Foundations for the twenty-first century,” Science Education 88, 28–54 (2004). American Association of Physics Teachers Committee on Laboratories, “Goals of the Introductory Physics Laboratory,” American Journal of Physics 66, 483–485 (1998). Committee on Undergraduate Biology Education to Prepare Research Scientists for the 21st Century; National Research Council, BIO2010: Transforming Undergraduate Education for Future Research Biologists (National Academies Press, Washington, D.C., 2003) p. 208. President’s Council of Advisors on Science and Technology, Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and Mathematics, Tech. Rep. (2012). Ronald K. Thornton and David R. Sokoloff, “Learning motion concepts using real-time microcomputer-based laboratory tools,” American Journal of Physics 58, 858–867 (1990). Andy Buffler, Seshini Pillay, Fred Lubben, and Roger Fearick, “A model-based view of physics for computational
8
9
10
11
12
13
14
activities in the introductory physics course,” American Journal of Physics 76, 431–437 (2008). Marcos D. Caballero, Matthew A. Kohlmyer, and Michael F. Schatz, “Implementing and assessing computational modeling in introductory mechanics,” Physical Review Special Topics - Physics Education Research 8, 020106 (2012). Rebecca Lippmann Kung, “Teaching the concepts of measurement: An example of a concept-based laboratory course,” American Journal of Physics 73, 771–777 (2005). Saalih Allie, Andy Buffler, Bob Campbell, Fred Lubben, Dimitris Evangelinos, Dimitris Psillos, Odysseas Valassiades, Cape Town, and South Africa, “Teaching Measurement in the Introductory Physics Laboratory,” The Physics Teacher 41, 23–30 (2003). E Etkina and A Van Heuvelen, “Investigative Science Learning Environment A Science Process Approach to Learning Physics,” in PER-based reforms in calculus-based physics (AAPT, 2007) pp. 1–48. Eugenia Etkina, Sahana Murthy, and Xueli Zou, “Using introductory labs to engage students in experimental design,” American Journal of Physics 74, 979–986 (2006). Cary Moskovitz and David Kellogg, “Inquiry-based writing in the laboratory course.” Science 332, 919–20 (2011). E. J. Galvez, Charles H. Holbrow, M. J. Pysher, J. W. Martin, N. Courtemanche, L. Heilig, and J. Spencer, “Interference with correlated photons: Five quantum mechan-
11
TABLE III. Complete list of all statements for the E-CLASS Survey. Question 23 is omitted because it is a check question to make sure students are reading the statements. Personal/Professional Statement 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 How important for earning a good grade in this class was... When doing an experiment, I try to understand how the experimental ...understanding how the experimental setup setup works. works? If I wanted to, I think I could be good at doing research. NA When doing a physics experiment, I don’t think much about sources of ...thinking about sources of systematic error? systematic error. If I am communicating results from an experiment, my main goal is to ...communicating results with the correct create a report with the correct sections and formatting. sections and formatting? Calculating uncertainties usually helps me understand my results bet- ...calculating uncertainties to better underter. stand my results? Scientific journal articles are helpful for answering my own questions ...reading scientific journal articles? and designing experiments I don’t enjoy doing physics experiments. NA When doing an experiment, I try to understand the relevant equations. ...understanding the relevant equations? When I approach a new piece of lab equipment, I feel confident I can ...learning to use a new piece of laboratory learn how to use it well enough for my purposes. equipment? Whenever I use a new measurement tool, I try to understand its per- ...understanding the performance limitations formance limitations. of the measurement tools? Computers are helpful for plotting and analyzing data. ...using a computer for plotting and analyzing data? I don’t need to understand how the measurement tools and sensors work ...understanding how the measurement tools in order to carry out an experiment. and sensors work? If I try hard enough I can succeed at doing physics experiments. NA When doing an experiment I usually think up my own questions to ...thinking up my own questions to investiinvestigate. gate? Designing and building things is an important part of doing physics ...designing and building things? experiments. The primary purpose of doing a physics experiment is to confirm pre- ...confirming previously known results? viously known results. When I encounter difficulties in the lab, my first step is to ask an expert, ...overcoming difficulties without the instruclike the instructor. tor’s help? Communicating scientific results to peers is a valuable part of doing ...communicating scientific results to peers? physics experiments. Working in a group is an important part of doing physics experiments. ...working in a group? I enjoy building things and working with my hands. NA I am usually able to complete an experiment without understanding the ...understanding the equations and physics equations and physics ideas that describe the system I am investigating. ideas that describe the system I am investigating? If I am communicating results from an experiment, my main goal is to ...making conclusions based on data using make conclusions based on my data using scientific reasoning. scientific reasoning? When I am doing an experiment, I try to make predictions to see if my ...making predictions to see if my results are results are reasonable. reasonable? Nearly all students are capable of doing a physics experiment if they NA work at it. A common approach for fixing a problem with an experiment is to ...randomly changing things to fix a problem randomly change things until the problem goes away. with the experiment? It is helpful to understand the assumptions that go into making predic- ...understanding the approximations and tions. simplifications that are included in theoretical predictions? When doing an experiment, I just follow the instructions without think- ...thinking about the purpose of the instrucing about their purpose. tions in the lab guide? I do not expect doing an experiment to help my understanding of NA physics. If I don’t have clear directions for analyzing data, I am not sure how to ...choosing an appropriate method for anachoose an appropriate analysis method. lyzing data (without explicit direction)? Physics experiments contribute to the growth of scientific knowledge. NA Expert A A D D A A D A A A A D A A A D D A A A D
22 24 25 26 27
A A A D A
28 29 30 31
D D D A
12 ics experiments for undergraduates,” American Journal of Physics 73, 127–140 (2005). Brett J. Pearson and David P. Jackson, “A hands-on introduction to single photons and quantum mechanics for undergraduates,” American Journal of Physics 78, 471– 484 (2010). Edward F Redish, Jeffery M Saul, and Richard N Steinberg, “On the effectiveness of active-engagement microcomputer-based laboratories,” American Journal of Physics 65, 45–54 (1997). Andrew Elby, “Getting Started with Research on Epistemologies and Expectations,” Reviews in Physics Education Research 2, 1–33 (2011). Andrew Elby, “Helping physics students learn how to learn,” American Journal of Physics 69, S54 (2001). Ibrahim Halloun and David Hestenes, “Interpreting VASS Dimensions and Profiles for Physics Students,” Science & Education 7, 553–577 (1998). Edward F Redish, Jeffery M Saul, and Richard N Steinberg, “Student expectations in introductory physics,” American Journal of Physics 66, 212–224 (1998). Norm G. Lederman, Fouad Abd-El-Khalick, Randy L. Bell, and Rene S. Schwartz, “Views of nature of science questionnaire: Toward valid and meaningful assessment of learners’ conceptions of nature of science,” Journal of Research in Science Teaching 39, 497–521 (2002). W. K. Adams, K. K. Perkins, N. S. Podolefsky, M. Dubson, N. D. Finkelstein, and C. E. Wieman, “New instrument for measuring student beliefs about physics and learning physics: The Colorado Learning Attitudes about Science Survey,” Physical Review Special Topics - Physics Education Research 2, 010101 (2006). Benjamin M. Zwickl, Noah Finkelstein, and H. J. Lewandowski, “Development and validation of the Colorado learning attitudes about science survey for experimental physics,” in Proceedings of the Physics Education Research Conference (2013) pp. 442–445, arXiv:1207.2418. URL: http://tinyurl.com/E-CLASS-Sp13-Post. Wendy K Adams and Carl E. Wieman, “Development and Validation of Instruments to Measure Learning of ExpertLike Thinking,” International Journal of Science Education 33, 1289–1312 (2011). American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, Standards for Psychological and Educational Assessment (AERA, Washington, D.C., 1999). Benjamin M. Zwickl, Noah Finkelstein, and H. J. Lewandowski, “The process of transforming an advanced lab course: Goals, curriculum, and assessments,” American Journal of Physics 81, 63–70 (2013). David Hammer, Andrew Elby, Rachel E Scherr, and Edward F Redish, “Resources, framing, and transfer,” in Transfer of learning from a modern multidisciplinary perspective, edited by J Mestre (Information Age Publishing, Greenwich, CT, 2005) pp. 89–120. H. Alhadlaq, F. Alshaya, S. Alabdulkareem, K. K. Perkins, W. K. Adams, and C. E. Wieman, “Measuring Students Beliefs about Physics in Saudi Arabia,” Proceedings of the Physics Education Research Conference 69, 69–72 (2009). Ping Zhang and Lin Ding, “Large-scale survey of Chinese precollege students epistemological beliefs about physics: A progression or a regression?” Physical Review Special Topics - Physics Education Research 9, 010110 (2013). Wendy K. Adams, Carl E. Wieman, Katherine K. Perkins, and Jack Barbera, “Modifying and Validating the Colorado Learning Attitudes about Science Survey for Use in Chemistry,” Journal of Chemical Education 85, 1435 (2008). Timothy L. McCaskey, Melissa H Dancy, and Andrew Elby, “Effects on assessment caused by splits between belief and understanding,” 2003 Physics Education Research Conference 720, 37–40 (2004). Timothy L McCaskey and Andrew Elby, “Probing Students’ Epistemologies Using Split Tasks,” Proceedings of the Physics Education Research Conference 790, 57–60 (2004). Kara E. Gray, Wendy K. Adams, Carl E. Wieman, and Katherine K. Perkins, “Students know what physicists believe, but they dont agree: A study using the CLASS survey,” Physical Review Special Topics - Physics Education Research 4, 020106 (2008). K K Perkins, W K Adams, S J Pollock, N D Finkelstein, and C E Wieman, “Correlating Student Beliefs With Student Learning Using The Colorado Learning Attitudes about Science Survey,” in Proceedings of the Physics Education Research Conference (2005) pp. 6–9. URL: http://perusersguide.org/. Association of American Universities, “Undergraduate STEM Education Initiative Press Resease,” (2013).
