Assessing the Achievement of General Education Objectives:
Three Years of Assessment
September 2001
Trudy Bers
Office of Research
Oakton Community College
1600 E. Golf Rd.
Des Plaines, IL 60016
(847) 635-1894
Assessing the Achievement of General Education Objectives:
Three Years of Assessment
EXECUTIVE SUMMARY
In spring 2001 the Oakton Student Academic Achievement Team (SAAT) conducted the third year of its project to assess general education learning outcomes. A sample of 1183 students enrolled in spring 2001 was given and answered questions about prompts covering selected general education objectives. Trained faculty used rubrics developed for the prompts to score students’ work. A total of 1763 scores was obtained, because one prompt covered two different general education areas. Rubrics were based on a 3-level scoring system, with the top two levels indicating a student met standards for general education objectives on that prompt. These general education objectives were covered:
·
Explain how information fits within an
historical context
·
Compare/ evaluate alternative solutions to a
problem
·
Identify problems
·
Effectively
communicate ideas in writing
The last two objectives were based on the same prompt and student responses. Two different rubrics were used to score the students’ work, one rubric dealing with the objective of indentifying problems and the other dealing with the objective of effectively communicating ideas in writing.
Researchers examined the association between a number of student demographic and academic characteristics and students’ achievement levels on the prompts. Results were:
·
Overall, 64 percent of the students’ responses were
scored at a “passing” level; i.e., 64 percent of the responses indicated that
students met general education achievement levels. Thirteen percent of responses were scored high pass and 51
percent were scored low pass. The
number of responses scored for each prompt and percent of students who passed
is presented below:
|
|
How information fits within an
historical context S. Carolina flag N = 467 |
Compare/ evaluate alternative solutions to a problem Illiniwek N=422 |
Define problems Clone-problems N= 437 |
Effectively communicate ideas
in writing Clone-write N = 437 |
||||
|
|
N |
Pct. |
N |
Pct. |
N |
Pct. |
N |
Pct. |
|
High pass |
48 |
10 |
106 |
25 |
44 |
10 |
34 |
8 |
|
Low pass |
241 |
52 |
142 |
34 |
241 |
55 |
280 |
64 |
|
No pass |
178 |
38 |
174 |
41 |
152 |
35 |
123 |
28 |
·
Seventy-two percent of students received passing scores
on effectively communicating ideas in writing, a substantial increase from past
years.
·
Fifty-five percent of students whose work was scored
for both identifying problems and effectively communcating ideas in writing—the
same prompt was used for both objectives--received passing scores on both
rubrics applied to the prompt.
· Statistically significant associations[1] between independent variables and achievements on general education objectives were:
· Gender was associated with the achievements on fitting information within an historical context (69% of males achieved passing scores and 59% of females achieved passing scores).
· Age was associated with fitting information within an historical context and effective writing. On both prompts, a larger proportion of younger (below age 25) students achieved standards than older ones (students aged 25 or above).
· Cumulative credits earned at Oakton was not associated with achievements on any prompt.
· Cumulative grade point average was associated with comparing and evaluating alternative solutions to a problem and effective writing. Students with higher GPAs were more likely to achieve passing scores.
· Mathematics competency was associated with achievements on fitting information within an historical context, but there was no logical explanation for why this might be the case. Students with college-level mathematics competency were most likely to achieve a passing score and students with remedial-level competency were least likely to do so.
·
English competency was associated with achievements on
all four prompts. In all cases,
students with college-level writing competency and those for whom no
information was available were substantially more likely to achieve a passing
score than students at the remedial level.
· The percentage of students who achieved passing scores on the general education assessment varied in each of the three years of this general education assessment project (1999, 2000 and 2001). In 1999, 59 percent passed, in 2000 77 percent passed and in 2001 64 percent passed. In 2000 one prompt was answered in a multiple choice format and 95 percent of students achieved passing scores, skewing results for that year. Differences in the prompts, scoring rubrics, general education objectives, instructions to students and faculty reader procedures across the years make it inappropriate to compare achievements from one year to the next.
INTRODUCTION
This report describes the three years of the Oakton Student Academic Achievement Team (SAAT) project to assess general education learning outcomes. Results from three years are presented, with special emphasis and description of the year three project. More details about the assessments in 1999 and 2000 are available in several articles (Bers, 2000; Bers, 2001; Bers, Davis and Mittler, 2001).
PROJECT DESCRIPTION
In the spring semesters of 1999, 2000 and 2001 a subcommittee of the SAAT implemented a locally developed institutional approach for assessing students’ achievements of general education objectives. [2] While refinements to the project were made in the second and third years based on experiences of prior years, the overall process was essentially the same over the three-year span. The approach used prompts selected to cover a subset of general education objectives. Prompts were narrative or graphic documents such as editorial cartoons, newspaper articles, or graphs that faculty determined addressed the selected general education objective.[3] Each prompt was accompanied by a series of questions students were asked to answer. For example, in 2001 one prompt described a controversy in South Carolina about flying the confederate flag over the state’s capital building. Students were asked to write a paragraph in which they explained to an international student not familiar with it, the historical context of this controversy.
Different prompts were used in each year, and different general education objectives were assessed. Prompts were administered to students enrolled in some 90 Oakton classes during the week of April 23-28, 2001. Classes were chosen purposefully, with the intent of identifying courses enrolling large numbers of students with 30 or more college credits.[4] However, all students enrolled in the selected classes completed the assessment. Prompts were distributed randomly among students, with each participant completing just one prompt in a class. Students themselves were asked to select a different prompt if they had already completed a prompt in a different class.
Trained faculty members using scoring rubrics developed by subcommittee members scored student responses. Faculty readers received training at the beginning of each scoring session. Initially two readers scored each prompt. Students who received two scores of 3 were awarded a “high pass,” those who received two scores of 2 or scores of 2 and 3 were awarded a “low pass,” and those with two scores of 1 were awarded a “no pass.” Where readers differed on whether or not the student passed, a third reader scored the prompt. If the third reader scored the response as a pass, the student was given a “pass.” If the third reader scored the response as a “no pass,” then the student was given a “no pass.” Scores were not added together to compute a numeric total. Rather, the number of “high pass,” “pass” and “no pass” evaluations were used to determine the student’s final score.
Prompts in 2001 were
designed to assess four general education objectives. One prompt was a website report about a conference announcing
that human cloning would soon be available for therapeutic purposes. It talked about benefits and detriments of
human cloning. Students were asked to
define problems associated with human cloning based on the article. Answers were scored in two different
ways. First, we looked to see
whether the answer indicated the
student met the general education objective to define problems. Second, we looked to see whether the student
effectively communicated his or her ideas in writing.
A second prompt was a statement about Chief Illiniwek written by a University of Illinois alumnus. The statement focused on the controversy in Illinois about using Chief Illiniwek as a mascot for university athletic teams. The general education objective examined was students’ achievements in comparing and evaluating alternate solutions to a problem. Students were asked to write a paragraph to in which they compared and evaluated at least three of these alternatives for moving toward a solution to the problem.
The third prompt was
about the controversy in South Carolina over flying the Confederate flag above
the state capital. It examined
students’ achievements in explaining how information fits within an historical
context. Students were asked to write a
paragraph in which they explained to an international student not familiar with
it, the historical context of the South Carolina flag controversy.
We obtained data for the project from both the students’ scores on the prompts and the College’s student information management system. The dependent variable is the student’s achievement on the prompt. We also identified a number of demographic and academic variables to use as controls: age, gender, cumulative grade point average, cumulative college-level credits at the college, English composition proficiency, and mathematics proficiency. These same independent variables were used in both 1999 and 2000 as well. They are summarized in Appendix A.[5]
RESULTS
Table 1 shows that
overall, 64 percent of student responses were scored at a level to indicate
students met general education achievement levels, with 13 percent in the high
pass and 51 percent in the low pass categories. There were, however, disparities in pass rates among the
prompt/rubric combinations.
One in ten South
Carolina flag prompts was scored at a high pass, and slightly more than half
were scored as low pass answers.
Thirty-eight percent of student responses to this prompt did not meet
general education standards. The
largest proportion of high pass scores was on the Illiniwek prompt regarding
alternative solutions, where 25 percent of students received a high pass. However, barely one third of students
received a low pass on this prompt.
Thus the percent of students who met general education standards on the
Illiniwek prompt (high and low pass combined) was the lowest among the
four scored prompts. Like with the flag
prompt, one in ten cloning prompts scored for defining problems received a high
pass, and another 55 percent were scored as low passes. Thus just over one third of students did not
meet standards for the general education objective to define problems. Though the percentage of high pass scores
for the cloning prompt scored for writing was the lowest of the four prompts
(8%), the percent scored as low pass was the highest (64%). Overall, 72 percent of cloning prompts
received a passing score when evaluated for effective writing. This is in marked contrast to prior years,
when prompts scored for writing yielded the lowest percent of passing
scores.
Table 1
Results of General Education Assessment
by Prompt/Rubric
|
|
How information fits within an
historical context S. Carolina flag N = 467 |
Compare/ evaluate alternative solutions to a problem Illiniwek N=422 |
Define problems Clone-problems N= 437 |
Effectively communicate ideas
in writing Clone-write N = 437 |
||||
|
|
N |
Pct. |
N |
Pct. |
N |
Pct. |
N |
Pct. |
|
High pass |
48 |
10 |
106 |
25 |
44 |
10 |
34 |
8 |
|
Low pass |
241 |
52 |
142 |
34 |
241 |
55 |
280 |
64 |
|
No pass |
178 |
38 |
174 |
41 |
152 |
35 |
123 |
28 |
We next sought to
explore whether the independent variables were associated with students’
assessment scores. We used chi-square
analyses for each prompt, cross-tabulating students’ scores on the prompt with
each independent variable. We did so
first using the three-part scores (high pass, low pass, no pass) and then again
using a two-part score of pass, no pass.
Table 2 presents results when we used the two-part score of pass, no
pass.[6]
Gender was associated
with the achievements on putting information within an historical context. Sixty-nine percent of males but only 59
percent of females achieved standards.
Age was associated with achievement on two general education objectives:
information within an historical context and writing. Sixty-seven percent of students under age 25, but only 55 percent
of those age 25 or above, met the historical context standard. Seventy-six percent of younger students but
only 65 percent of older students met standards for effective writing. The results for age run counter what is
normally found at the college, where older students generally perform at higher
levels than younger ones.
Cumulative credits
earned at Oakton was not associated with achievements on any of the
prompts. Cumulative grade point average
was associated with achievements on two prompts. As would be expected, students who had higher grade point
averages were more likely to achieve standards. Two-thirds (67%) of students with cumulative GPAs of 3.0 or
higher met standards on comparing and evaluating alternate solutions, 55
percent with GPAs of 2-2.9 met standards, and 51 percent of students with GPAs
below 2.0 also met standards on this objective. Eighty-one percent of students with GPAs of 3.0 or higher met
standards for effective writing, two-thirds (65%) of students with cumulative
GPAs of 2-2.9 met standards, and 71 percent of students with GPAs below 2.0
also met standards for effective writing.
Students’ mathematics
competency was associated with scores on just one prompt, the one testing
students’ achievements in being able to fit information within an historical
context. Over two-thirds (68%) of
students with college-level mathematics competency met this standards, 57-58%
of students whose competency was at the level of intermediate algebra or for
whom there was no mathematics information met standards, and 52 percent of
students with remedial mathematics competency met standards. The finding of a
statistically significant association between mathematics competency and
meeting standards on this general education objective is interesting but
difficult to explain, because there does not seem to be a logical connection
among the content of prompt, what students were asked to write about to respond
to the prompt, and achievement in mathematics.
Finally, students’
writing competency was associated with their scores on all four prompts. The patterns were similar for all prompts as
well. Table 3 presents these data.
Betweeen 60 and 75 percent of students with college-level writing competency
met standards, between 60 and 70 percent of students for whom there was no
writing information met standards, and between 25 and 36 percent of students
with remedial-level writing competency met standards.
Table 2
Association between Independent Variables
and Results
General Education Assessment by Prompt/Rubric
Associations noted by 4
|
Prompt Variable
|
How information fits within an
historical context S. Carolina flag N = 467 |
Compare/evaluate alternative solutions to a problem Illiniwek N=422 |
Define problems Clone-problems N= 437 |
Effectively communicate ideas
in writing Clone-write N = 437 |
|
Gender |
4 |
|
|
|
|
Age |
4 |
|
|
4 |
|
CUMHRS |
|
|
|
|
|
CUMGPA |
|
4 |
|
4 |
|
MATH |
4 |
|
|
|
|
ENGLISH |
4 |
4 |
4 |