The relationship between instructional technology
and student achievement: A new model
Lorraine Sherry and Daniel Jesse
RMC Research Corporation, Denver
The WEB Project is a five-year Technology Innovation Challenge Grant that has just ended. Its intent was to infuse standards-based instruction in multimedia, digital art, music composition, and online discourse into the arts and humanities curricula of Vermont K-12 schools, and to have students share their work with a community of online experts so that they could improve their academic performance. The intervention has been stable over the past three years.
In our evaluation of this project, we posed the following question: “what is the impact of The WEB Project on student performance?” with the intent of generalizing our methodology to other instructional technology projects. Based on teacher responses to an online survey repeated over the past three years, we began to see a connection between student motivation, metacognition, and learning processes that was reminiscent of Sternberg’s (1998) Developing Expertise model. According to Sternberg, motivation drives metacognition, which, in turn, stimulates the development of thinking and learning skills. Thinking and learning skill development further stimulates metacognition, resulting in the development of expertise.
Methodology
In 1999, a 6-page pilot student survey instrument was designed using the Vermont Reasoning and Problem Solving Standards and other schools of thought (Sternberg, 1998; Perry, 1992; Gagne & Briggs, 1979; Bloom et al., 1956) to explore the Sternberg model using structural equation modeling. In May 1999, at the end of the spring term, the pilot survey was administered to 119 students in nine cooperating schools. Three scales were developed from the individual items in those questions that dealt with metacognition and learning processes using The WEB Project technology. Reliability (alpha) ranged from .72 to .84 for all scales. Partial correlations were high (metacognition -> inquiry learning, Beta = .75; metacognition -> application of skills, Beta = .53). This confirmed our belief that the Sternberg model might be used as an interpretive framework for our work.
Based on the strength of these correlations, a plan was devised for using Marzano et al.’s (1993) Dimensions of Learning Model to measure student performance along the path to expertise. A pair of ten-item motivation measures was derived from the literature and from previous work at RMC Research Corp., containing questions related to classroom motivation and school motivation. Concurrently, participating teachers developed and benchmarked rubrics designed to assess student work in art, music, multimedia production, and reflective online discourse.
In January 2000, a revised 7-page survey was administered to 165 students in nine schools that were participating in The WEB Project. 137 of these surveys were from students who had not yet been exposed to the intervention, and could therefore be used as a pre-test. Reliability (alpha) for all scales (motivation, metacognition, inquiry learning, and application of skills) ranged from .75 to .87.
In May 2000, at the end of the spring term, the survey was re-administered as a post-test to the same sample of students. 131 completed surveys were returned by seven of the nine schools as of July 26, 2000. About 75% of the students who responded were from high schools, 25% from middle schools. 50.4% were boys; 49.6% were girls. Preliminary analyses have been completed; additional matching will be completed as more surveys become available. A total of 58 pre-tests and post-tests were matched for repeated measures analyses. 131 post-test surveys were used for the path analyses.
1. Time Analysis of Technology
Skills
Students were asked to report their skill level along ten dimensions “before” the project (pre-test) and “now” (post-test), on a five point Likert-type scale (1 = “could not do it”; 5 = “expert, the best!”). These two measures were used as a proxy of how much students had developed their skills during the term. A preliminary set of repeated measures t-tests were conducted upon these data to determine significance levels. There were significant improvements (p<.05) from the “before” to the “now” skill levels for all skill items.
Next, 58 valid data sets were matched in order to conduct a true repeated measures methodology (pre-test vs. post-test). The results are displayed below for the key subscales of interest:
Subscale Class.Motiv. Metacog. Inq.Learn App.of.Skills
Pre-test 4.00 2.17 2.49 2.04
Post-test 3.97 2.36 2.52 1.87
t-value .347 -1.96 -.458 2.00
df 52 56 56 56
p (2-tailed) .730 .055 .649 .050
Only the metacognition variable changed significantly in the predicted direction (1-tailed test probability = .028). Students reported higher levels of metacognitive skills at the end of the course than they did at the beginning.
2. Preliminary Post-Test Path Analysis
Results
Teachers assigned a “product” score from 0 to 3 (0 = no evidence, 1 = approaches standards, 2 = meets standards, 3 = exceeds standards) for the students’ final products. They also assigned a “process” score from 1 (low) to 4 (high) for the quantity and depth of revisions of student products. These data constitute two independent measures of student achievement, which serve to complete the model, namely:
“This class” motivation -> metacognition -> learning processes (inquiry learning and application of skills) -> student achievement (product and process).
Exploratory analyses were conducted with the subscales described here and an achievement indicator for student products. The nonparametric correlation for application of skills and student product quality was significant (r (38) = .383, p = .031). Also, the nonparametric correlation between classroom motivation and student product quality was significant (r (32) = .383, p = .031). These results suggest the important role that the development of skill plays in student achievement. Additionally, it appears that all of the elements in this model are interrelated.
From the May 1999 pilot test of the student survey, structural equation modeling showed that there were strong partial correlations between metacognition and inquiry learning (Beta = .75) and between metacognition and application of skills (Beta = .53). In January 2000 (baseline for this term), there was a low to moderate partial correlation between “this class” motivation and metacognition (Beta = .24), a strong partial correlation between metacognition and inquiry learning (Beta = .68), and a low partial correlation between metacognition and application of skills (Beta = .04). At the end of the spring 2000 term, the partial correlations were approaching moderate for motivation and metacognition (Beta = .28), strong for metacognition and inquiry learning (Beta = .49), and moderate to strong for metacognition and application of skills (Beta = .35).
There appears to be some similarity between the extent of impact on student performance resulting from The WEB Project-related work as documented in the 1998-2000 online surveys of teachers, the student self-reported data, and Sternberg’s Developing Expertise Model. The use of a standard model must be balanced against the fact that this case study is based on a standards-driven instructional technology project in a small rural state, with a sample of students who report that they put out a lot of effort for the grades they receive.
The results, obtained from over 75% of the data set for spring 2000, indicate that over time, the relationship between metacognition and inquiry learning slipped after participation in the course. However, the relationship between metacognition and application of skills grew. This could be explained by the fact that as the course progressed, students were applying the skills they have learned on a regular basis, thus impacting metacognition. Moreover, this pattern of results is parallel to what was seen in May 1999, although they are not as dramatic.
Once all data have been received, we plan to examine correlations from teacher measurements of student achievement and the student survey data, thus confirming or rejecting the hypothesis that the developing expertise model can serve as one piece of the puzzle, and to explain increases in student performance as a result of WEB Project-related activities.
Bloom, B., Englehart, M., Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive Domain. NY: Longmans Green.
Gagne, R., & Briggs, L. (1979). Principles of Instructional Design (2nd Ed.). NY: Holt, Rinehart, & Winston.
Marzano., R.J., Pickering, D., & McTighe, J. (1993). Assessing student outcomes: Performance assessment using the Dimensions of Learning Model. Littleton CO: McREL Institute.
Perry, S.M. (1992). Building Better Thinking Skills. Available: Aurora Public Schools, 1085 Peoria Street, Aurora CO 80011.
Sternberg, R.J. (1998, April). Abilities are forms of developing expertise. Educational Researcher, 27 (3), 11-20.