Assessing the Achievement of General Education Objectives:
Three Years of Assessment
September 2001
Trudy Bers
Office of Research
Oakton Community College
1600 E. Golf Rd.
Des Plaines, IL 60016
(847) 635-1894
Assessing the Achievement of General Education Objectives:
Three Years of Assessment
EXECUTIVE SUMMARY
In spring 2001 the Oakton Student Academic Achievement Team (SAAT) conducted the third year of its project to assess general education learning outcomes. A sample of 1183 students enrolled in spring 2001 was given and answered questions about prompts covering selected general education objectives. Trained faculty used rubrics developed for the prompts to score students’ work. A total of 1763 scores was obtained, because one prompt covered two different general education areas. Rubrics were based on a 3-level scoring system, with the top two levels indicating a student met standards for general education objectives on that prompt. These general education objectives were covered:
·
Explain how information fits within an
historical context
·
Compare/ evaluate alternative solutions to a
problem
·
Identify problems
·
Effectively
communicate ideas in writing
The last two objectives were based on the same prompt and student responses. Two different rubrics were used to score the students’ work, one rubric dealing with the objective of indentifying problems and the other dealing with the objective of effectively communicating ideas in writing.
Researchers examined the association between a number of student demographic and academic characteristics and students’ achievement levels on the prompts. Results were:
·
Overall, 64 percent of the students’ responses were
scored at a “passing” level; i.e., 64 percent of the responses indicated that
students met general education achievement levels. Thirteen percent of responses were scored high pass and 51
percent were scored low pass. The
number of responses scored for each prompt and percent of students who passed
is presented below:
|
|
How information fits within an
historical context S. Carolina flag N = 467 |
Compare/ evaluate alternative solutions to a problem Illiniwek N=422 |
Define problems Clone-problems N= 437 |
Effectively communicate ideas
in writing Clone-write N = 437 |
||||
|
|
N |
Pct. |
N |
Pct. |
N |
Pct. |
N |
Pct. |
|
High pass |
48 |
10 |
106 |
25 |
44 |
10 |
34 |
8 |
|
Low pass |
241 |
52 |
142 |
34 |
241 |
55 |
280 |
64 |
|
No pass |
178 |
38 |
174 |
41 |
152 |
35 |
123 |
28 |
·
Seventy-two percent of students received passing scores
on effectively communicating ideas in writing, a substantial increase from past
years.
·
Fifty-five percent of students whose work was scored
for both identifying problems and effectively communcating ideas in writing—the
same prompt was used for both objectives--received passing scores on both
rubrics applied to the prompt.
· Statistically significant associations[1] between independent variables and achievements on general education objectives were:
· Gender was associated with the achievements on fitting information within an historical context (69% of males achieved passing scores and 59% of females achieved passing scores).
· Age was associated with fitting information within an historical context and effective writing. On both prompts, a larger proportion of younger (below age 25) students achieved standards than older ones (students aged 25 or above).
· Cumulative credits earned at Oakton was not associated with achievements on any prompt.
· Cumulative grade point average was associated with comparing and evaluating alternative solutions to a problem and effective writing. Students with higher GPAs were more likely to achieve passing scores.
· Mathematics competency was associated with achievements on fitting information within an historical context, but there was no logical explanation for why this might be the case. Students with college-level mathematics competency were most likely to achieve a passing score and students with remedial-level competency were least likely to do so.
·
English competency was associated with achievements on
all four prompts. In all cases,
students with college-level writing competency and those for whom no
information was available were substantially more likely to achieve a passing
score than students at the remedial level.
· The percentage of students who achieved passing scores on the general education assessment varied in each of the three years of this general education assessment project (1999, 2000 and 2001). In 1999, 59 percent passed, in 2000 77 percent passed and in 2001 64 percent passed. In 2000 one prompt was answered in a multiple choice format and 95 percent of students achieved passing scores, skewing results for that year. Differences in the prompts, scoring rubrics, general education objectives, instructions to students and faculty reader procedures across the years make it inappropriate to compare achievements from one year to the next.
INTRODUCTION
This report describes the three years of the Oakton Student Academic Achievement Team (SAAT) project to assess general education learning outcomes. Results from three years are presented, with special emphasis and description of the year three project. More details about the assessments in 1999 and 2000 are available in several articles (Bers, 2000; Bers, 2001; Bers, Davis and Mittler, 2001).
PROJECT DESCRIPTION
In the spring semesters of 1999, 2000 and 2001 a subcommittee of the SAAT implemented a locally developed institutional approach for assessing students’ achievements of general education objectives. [2] While refinements to the project were made in the second and third years based on experiences of prior years, the overall process was essentially the same over the three-year span. The approach used prompts selected to cover a subset of general education objectives. Prompts were narrative or graphic documents such as editorial cartoons, newspaper articles, or graphs that faculty determined addressed the selected general education objective.[3] Each prompt was accompanied by a series of questions students were asked to answer. For example, in 2001 one prompt described a controversy in South Carolina about flying the confederate flag over the state’s capital building. Students were asked to write a paragraph in which they explained to an international student not familiar with it, the historical context of this controversy.
Different prompts were used in each year, and different general education objectives were assessed. Prompts were administered to students enrolled in some 90 Oakton classes during the week of April 23-28, 2001. Classes were chosen purposefully, with the intent of identifying courses enrolling large numbers of students with 30 or more college credits.[4] However, all students enrolled in the selected classes completed the assessment. Prompts were distributed randomly among students, with each participant completing just one prompt in a class. Students themselves were asked to select a different prompt if they had already completed a prompt in a different class.
Trained faculty members using scoring rubrics developed by subcommittee members scored student responses. Faculty readers received training at the beginning of each scoring session. Initially two readers scored each prompt. Students who received two scores of 3 were awarded a “high pass,” those who received two scores of 2 or scores of 2 and 3 were awarded a “low pass,” and those with two scores of 1 were awarded a “no pass.” Where readers differed on whether or not the student passed, a third reader scored the prompt. If the third reader scored the response as a pass, the student was given a “pass.” If the third reader scored the response as a “no pass,” then the student was given a “no pass.” Scores were not added together to compute a numeric total. Rather, the number of “high pass,” “pass” and “no pass” evaluations were used to determine the student’s final score.
Prompts in 2001 were
designed to assess four general education objectives. One prompt was a website report about a conference announcing
that human cloning would soon be available for therapeutic purposes. It talked about benefits and detriments of
human cloning. Students were asked to
define problems associated with human cloning based on the article. Answers were scored in two different
ways. First, we looked to see
whether the answer indicated the
student met the general education objective to define problems. Second, we looked to see whether the student
effectively communicated his or her ideas in writing.
A second prompt was a statement about Chief Illiniwek written by a University of Illinois alumnus. The statement focused on the controversy in Illinois about using Chief Illiniwek as a mascot for university athletic teams. The general education objective examined was students’ achievements in comparing and evaluating alternate solutions to a problem. Students were asked to write a paragraph to in which they compared and evaluated at least three of these alternatives for moving toward a solution to the problem.
The third prompt was
about the controversy in South Carolina over flying the Confederate flag above
the state capital. It examined
students’ achievements in explaining how information fits within an historical
context. Students were asked to write a
paragraph in which they explained to an international student not familiar with
it, the historical context of the South Carolina flag controversy.
We obtained data for the project from both the students’ scores on the prompts and the College’s student information management system. The dependent variable is the student’s achievement on the prompt. We also identified a number of demographic and academic variables to use as controls: age, gender, cumulative grade point average, cumulative college-level credits at the college, English composition proficiency, and mathematics proficiency. These same independent variables were used in both 1999 and 2000 as well. They are summarized in Appendix A.[5]
RESULTS
Table 1 shows that
overall, 64 percent of student responses were scored at a level to indicate
students met general education achievement levels, with 13 percent in the high
pass and 51 percent in the low pass categories. There were, however, disparities in pass rates among the
prompt/rubric combinations.
One in ten South
Carolina flag prompts was scored at a high pass, and slightly more than half
were scored as low pass answers.
Thirty-eight percent of student responses to this prompt did not meet
general education standards. The
largest proportion of high pass scores was on the Illiniwek prompt regarding
alternative solutions, where 25 percent of students received a high pass. However, barely one third of students
received a low pass on this prompt.
Thus the percent of students who met general education standards on the
Illiniwek prompt (high and low pass combined) was the lowest among the
four scored prompts. Like with the flag
prompt, one in ten cloning prompts scored for defining problems received a high
pass, and another 55 percent were scored as low passes. Thus just over one third of students did not
meet standards for the general education objective to define problems. Though the percentage of high pass scores
for the cloning prompt scored for writing was the lowest of the four prompts
(8%), the percent scored as low pass was the highest (64%). Overall, 72 percent of cloning prompts
received a passing score when evaluated for effective writing. This is in marked contrast to prior years,
when prompts scored for writing yielded the lowest percent of passing
scores.
Table 1
Results of General Education Assessment
by Prompt/Rubric
|
|
How information fits within an
historical context S. Carolina flag N = 467 |
Compare/ evaluate alternative solutions to a problem Illiniwek N=422 |
Define problems Clone-problems N= 437 |
Effectively communicate ideas
in writing Clone-write N = 437 |
||||
|
|
N |
Pct. |
N |
Pct. |
N |
Pct. |
N |
Pct. |
|
High pass |
48 |
10 |
106 |
25 |
44 |
10 |
34 |
8 |
|
Low pass |
241 |
52 |
142 |
34 |
241 |
55 |
280 |
64 |
|
No pass |
178 |
38 |
174 |
41 |
152 |
35 |
123 |
28 |
We next sought to
explore whether the independent variables were associated with students’
assessment scores. We used chi-square
analyses for each prompt, cross-tabulating students’ scores on the prompt with
each independent variable. We did so
first using the three-part scores (high pass, low pass, no pass) and then again
using a two-part score of pass, no pass.
Table 2 presents results when we used the two-part score of pass, no
pass.[6]
Gender was associated
with the achievements on putting information within an historical context. Sixty-nine percent of males but only 59
percent of females achieved standards.
Age was associated with achievement on two general education objectives:
information within an historical context and writing. Sixty-seven percent of students under age 25, but only 55 percent
of those age 25 or above, met the historical context standard. Seventy-six percent of younger students but
only 65 percent of older students met standards for effective writing. The results for age run counter what is
normally found at the college, where older students generally perform at higher
levels than younger ones.
Cumulative credits
earned at Oakton was not associated with achievements on any of the
prompts. Cumulative grade point average
was associated with achievements on two prompts. As would be expected, students who had higher grade point
averages were more likely to achieve standards. Two-thirds (67%) of students with cumulative GPAs of 3.0 or
higher met standards on comparing and evaluating alternate solutions, 55
percent with GPAs of 2-2.9 met standards, and 51 percent of students with GPAs
below 2.0 also met standards on this objective. Eighty-one percent of students with GPAs of 3.0 or higher met
standards for effective writing, two-thirds (65%) of students with cumulative
GPAs of 2-2.9 met standards, and 71 percent of students with GPAs below 2.0
also met standards for effective writing.
Students’ mathematics
competency was associated with scores on just one prompt, the one testing
students’ achievements in being able to fit information within an historical
context. Over two-thirds (68%) of
students with college-level mathematics competency met this standards, 57-58%
of students whose competency was at the level of intermediate algebra or for
whom there was no mathematics information met standards, and 52 percent of
students with remedial mathematics competency met standards. The finding of a
statistically significant association between mathematics competency and
meeting standards on this general education objective is interesting but
difficult to explain, because there does not seem to be a logical connection
among the content of prompt, what students were asked to write about to respond
to the prompt, and achievement in mathematics.
Finally, students’
writing competency was associated with their scores on all four prompts. The patterns were similar for all prompts as
well. Table 3 presents these data.
Betweeen 60 and 75 percent of students with college-level writing competency
met standards, between 60 and 70 percent of students for whom there was no
writing information met standards, and between 25 and 36 percent of students
with remedial-level writing competency met standards.
Table 2
Association between Independent Variables
and Results
General Education Assessment by Prompt/Rubric
Associations noted by 4
|
Prompt Variable
|
How information fits within an
historical context S. Carolina flag N = 467 |
Compare/evaluate alternative solutions to a problem Illiniwek N=422 |
Define problems Clone-problems N= 437 |
Effectively communicate ideas
in writing Clone-write N = 437 |
|
Gender |
4 |
|
|
|
|
Age |
4 |
|
|
4 |
|
CUMHRS |
|
|
|
|
|
CUMGPA |
|
4 |
|
4 |
|
MATH |
4 |
|
|
|
|
ENGLISH |
4 |
4 |
4 |
4 |
Table 3
Percentage of Students Meeting Standards
and Writing Competency
by Prompt/Rubric
|
Prompt Writing
Competency |
How information fits within an
historical context S. Carolina flag N = 467 |
Compare/evaluate alternative solutions to a problem Illiniwek N=422 |
Define problems Clone-problems N= 437 |
Effectively communicate ideas
in writing Clone-write N = 437 |
|
College-level |
64 |
60 |
69 |
75 |
|
Remedial |
25 |
23 |
25 |
36 |
|
No
information |
65 |
60 |
61 |
70 |
Oakton students for
whom there are no English or mathematics competency data may already possess a
bachelor’s degree or at least 24 credits of college level work at the C level
from other institutions, or have registered for fewer than 13 credits at
Oakton. They would, therefore, not have
been required to take placement tests unless they wished to register for
English composition or mathematics courses.
Thus, absence of data about
competencies in these areas should not be interpreted to mean the student lacks
competencies.
Finally,
we examined whether there was an association between students’ achievements on
the two scoring rubrics applied to the same prompt. Recall the same prompt was scored using a rubric for defining
problems and for communicating
ideas in writing. There were
424 prompts scored using both rubrics.
Sixty-six percent were scored “at standard” for identifying problems and
72 percent were scored at standard for writing. Only 46 students achieved standards identifying problems but
failed on the writing score, whereas 72 students achieved standards in writing
but failed on identifying problems.
Fifty-five percent of students achieved standards on both of the general
education objectives assessed on this prompt.
In all three years, faculty selected prompts, developed scoring rubrics, and scored students’ written responses. In the second year one prompt was answered with multiple choice questions only and did not require faculty involvement for scoring. Some 95% of students received passing scores on the multiple choice prompt. Faculty decided not to use a multiple choice answer format in the third year. In all years classes with high numbers of students who had earned 30 credits or more were selected for inclusion in the study, and the same six independent variables were used in the analysis of results. However, the actual prompts and the scoring rubrics changed in almost all cases from one year to the next, as did the general education objectives being assessed. Communicating ideas effectively in writing was assessed in all three years, and the scoring rubric for assessing writing remained the same in years two and three. In year one, students were not told their writing woult be evaluated. In both years two and three they were told this. Faculty scoring for writing achievement were given more explicit instructions in the second and third years as well. For example, they were urged to keep in mind that students’ writing samples were done in class in a limited period of time, and should be viewed as initial products, not final assignments.
Students were offered
the opportunity to receive their individual results in years two and
three. Fifty-seven percent of the 603
students who completed responses in year two wanted their results and 53
percent of the 1183 students who completed responses in year three requested
them. There is no way to determine
whether the availability of results might have affected students’ performances
on the assessments.
Changes in prompts, objectives, scoring rubrics and faculty reader instructions make direct comparisons from one year to the next impossible. Despite these reservations, some observations can be made about the general education assessment project findings overall.
Only some independent variables were associated with students’ achievements. Table 4 depicts results for the three years. Cumulative credits earned at the college, gender and age are not generally associated with students’ achievements on the general education assessment. Mathematics competency is more often associated with results. Cumulative grade point average and English writing competency are the two independent variables most often linked with students’ achievements in meeting general education objectives. Meeting objectives is operationally defined as a passing score on the general education assessment. As expected, a higher proportion of students with high grade point averages achieve standards as compared to students with lower grade point averages. The association of writing competency and meeting standards is more complex. Students whose competency is at college-level do better in meeting standards than students whose competency is at the remedial level, but about the same proportion of students for whom there are no writing competency data available meet standards as the proportion of students whose competency is college-level. Absence of data about writing competency is not an indicator of lack of competency per se.
Table 4
Association between Independent Variables
and Results
General Education Assessment by Prompt/Rubric
Associations noted by 4
|
|
Year 1 – 1999 |
Year 2 – 2000* |
Year 3 - 2001 |
||||||||
|
General education objective Independent Variable |
Humanities |
Social Science |
Math |
Communications |
Gather, analyze & interpret data |
Apply ethical principles to local,
national & global issues |
Effectively communicate ideas in writing |
Fit infor-mation in
historical context |
Iderntify & evaluate alternative solutions |
Define problems |
Effectively communicate ideas in writing |
|
Gender |
|
|
|
|
|
4 |
|
4 |
|
|
|
|
Age |
|
|
|
|
|
|
|
4 |
|
|
4 |
|
CUMHRS |
|
|
|
|
4 |
|
|
|
|
|
|
|
CUMGPA |
|
|
|
4 |
4 |
4 |
4 |
|
4 |
|
4 |
|
MATH |
|
|
4 |
|
4 |
|
4 |
4 |
|
|
|
|
ENGLISH |
|
4 |
4 |
|
4 |
4 |
4 |
4 |
4 |
4 |
4 |
· In Year 2 a fourth prompt assessed students’ abilities to differentiate between facts and opinions. Responses were multiple choice; 95 percent of students achieved standards rendering further analyses meaningless.
DISCUSSION
The project described above is a
unique approach to assessing general education. In considering the findings, these issues should be kept in mind.
·
Data about students have limitations. Many
students who have attended other colleges and universities do not transfer
their credits to Oakton, especially those who have already earned a bachelor’s
degree. Consequently data about credits
earned are incomplete because they do not include credits from other
institutions.
·
Students about whom there appear to be no data
about English placements/courses or mathematics placements/courses may in fact
have taken English or mathematics courses elsewhere.
·
It is still too early in the assessment project history
to judge whether the percentage of students achieving a passing score on the
general education objectives assessed in this project is a “good” or “poor”
percent. Since the project has now been
implemented for three years, it may be time for the Student Academic
Achievement Team, general education subcommittee and other faculty groups to
begin conversations about what a “good” or “poor” percent might be.
·
Different students participated in the project in the
three years; therefore student gains measured through a pretest-posttest design
cannot be assumed.
·
Oakton’s general education assessment project
has not been subjected to rigorous scrutiny with respect to validity and
reliability. We accord face validity to
the prompts and the scoring rubrics.
·
Like their colleagues in 1999 and 2000, faculty
members who selected prompts, developed scoring rubrics, and evaluated
students’ work on the prompts continued in 2001 to find the task of identifying
appropriate prompts and formulating clear questions for students to answer to
be a difficult one.
·
Students’ work on this assessment project was
not part of their regular coursework.
We can only speculate on whether asking students to do work which did
not matter for grades or graduation invited them not to take the work seriously.
·
The finding that writing competency is often
associated with general education assessment achievements, especially in the
third year, suggests that students’ abilities to write clearly and their
competencies on other general education objectives may either be closely linked
or that weaknesses in the general education assessment process and design may
be creating a situation in which writing skill confounds accurate assessments
of competencies on other objectives.
The Student Academic Achievement Team and the general education
subcommittee that leads the assessment project may wish to pay particular
attention to this concern in 2002.
REFERENCES
Bers, Trudy.
“Assessing the Achievement of General Education Objectives: Second Year
of a College-wide Approach and Lessons Learned.” Journal of Applied Research
in the Community College.
[Forthcoming].
Bers, Trudy.
“Assessing the Achievement of General Education Objectives: A
College-wide Approach.” Journal of
General Education. 49:3 (2000).
Bers, Trudy, Davis, B. Diane, and Mittler, Mary L. “Assessing the Achievement of General Education Objectives in the Community College: A Project Across the Disciplines. Assessment Update. May/June, 2001.
Appendix A
Variables
|
Variable |
Operational definition |
|
SCORE |
Score on prompt, using
rubric appropriate to assess achievement of specified general education
objectives: High Pass = two readers
gave scores of 3 Low Pass = scores of 2
and 3 by two readers, or scores of 2 by two readers No Pass = two readers
gave scores of 1 |
|
CUMHRS Cumulative college-level credits at the
College though fall 1998 semester (does not include transfer credits from
elsewhere) |
Four categories: 1 = no cumulative
credits 2 = 1-29 cumulative
credits 3 = 30-44 cumulative
credits 4 = 45+ cumulative
credits |
|
CUMGPA Cumulative grade point average at the
College through fall 1998 semester (A = 4; does not include grades earned
elsewhere) |
Four categories: 1 = no GPA 2 = GPA 0 – 1.9 3 = GPA 2.0 – 2.9 4 = GPA 3.0 – 4.0 |
|
AGE |
1 = below 25 2 = 25 and above |
|
GENDER |
Male or female |
|
ENGLISH Note: for analyses 3 categories used: 1 = College level 2 = Remedial level 3 = No information |
Highest level of English
composition, rank ordered as follows: 1 =
college level composition course with grade of C or better 2 = placed into or enrolled in college level composition course, but
did not earn grade of C or better 3 =
remedial composition course with grade of P (pass) – includes English as a
second language and remedial 4 =
placed into or enrolled in remedial composition course, but did not earn
grade of P – includes English as a second language and remedial 5 =
no information about English composition |
|
MATH Note: for analyses 4
categories used: 1 = College level (trig or above) 2 = College algebra 3 = Remedial 4 = No information |
Highest level of
mathematics, rank ordered as follows: 1
= college level (trigonometry or above) course with grade of C or better 2
= placed into or enrolled in college level (trigonometry or above) course,
but did not earn grade of C or better 3
= college algebra course with grade of C or better 4
= placed into or enrolled in college algebra, but did not earn grade of C or
better 5
= remedial mathematics course with grade of P (pass) 6
= placed into or enrolled in remedial mathematics course, but did not earn
grade of P 7
= no information about mathematics |