OutcomesAndAssessment.tex

% arara: pdflatex: {files: [MathSACpr2014]}
% !arara: indent: {overwrite: yes}
\chapter[Outcomes and Assessment]{Reflect on learning outcomes and assessment,
 teaching methodologies, and content in order to improve the quality of teaching,
 learning and student success.}\label{chap:outcomes}
\epigraph{Learning outcomes assessment is key to addressing both affordability and access issues}
{Knowing What Students Know and Can Do \cite{outcomesquote}}

\section[Course-level outcomes]{Identify and give examples of assessment-driven
 changes made to improve attainment of course-level student learning outcomes.
 Where key sequences exist, also include information about assessment-driven
changes to those sequences.}

The SAC has mostly concentrated on college level outcomes in
the last five years for the following two reasons:
\begin{enumerate}
	\item The Curriculum Committee currently requires an ``out there''
	(\cite{courseoutcomes}) focus for course-level learning outcomes, with no
	requirement that outcomes be assessable or measurable.
	\item The annual assessment reporting for the Learning Assessment Council (LAC)
	has focused on the College's Core Outcomes, not course outcomes.
\end{enumerate}

However, we are fortunate to have math faculty involved with the Learning
Assessment Council (LAC) and the Curriculum Committee. This involvement has kept
our SAC aware of the college's ongoing discussions regarding a possible future
change for the focus of course-level learning outcomes  (i.e., expectation of
measurability) and related accreditation standards (e.g., \cite[Standard
4.A.3]{NWCCU}).

Many of our current learning outcomes were developed to satisfy the requirements
of an ``out there'' focus and this has resulted in oddly worded or aspirational
outcomes.  Here are two examples from MTH 251 (Calculus 1):

\begin{quote}
	Appreciate derivatives and limit-related concepts that are encountered in the
	real world, [and] understand and be able to communicate the underlying
	mathematics involved to help another person gain insight into the situation.
\end{quote}
\begin{quote}
	Enjoy a life enriched by exposure to Calculus.
\end{quote}

While we hope that our students will be able to ``help another person gain
insight into the situation'' and that they will ``enjoy a life enriched by
exposure to Calculus,'' we recognize that outcomes like these are not easily
measured. Before our SAC can make assessment-driven changes to improve students'
attainment of outcomes, we need to first develop measurable outcomes that
represent the intent of our courses.

In 2012/13 the MTH 60/65 CCOG subcommittee decided to develop course-level
outcomes that were meaningful and assessable; this was done despite concerns
that Curriculum Committee might reject them because they lacked the required
``out there'' focus.  The Math SAC was largely supportive of the shift to
assessable outcomes, suggesting that we are supportive of the college
transitioning toward a culture assessing outcomes at all levels.

While crafting the new (\emph{draft}) course outcomes, the MTH 60/65 CCOG
subcommittee connected each proposed outcome to an actual assessment activity
used by a member of the committee in order to ensure that each outcome was truly
measurable.  What follows are a few of the resultant (\emph{draft}) outcomes.
\begin{description}
	\item[Argument Construction:] Construct and judge the validity of an argument.
	(e.g., Why does a particular symbolic representation match a particular
	graphical representation?)

	\item[Representational Fluency:] Demonstrate the ability to distinguish
	different meanings of `variable', e.g., a variable can represent a varying
	quantity in an expression or an unknown quantity in an equation.

	\item[Problem Solving:] Use appropriate (mathematical) tools in the context of
	problem solving, modeling, interpreting, etc. (Know what approach to take,
	what information you have, what information you need, what techniques you have
	to solve the problem, what the graph tells you, what the formula tells you,
	what model you can build).
\end{description}

The current layout of the CCOGs does not differentiate critical content from
less critical content.  To address this, the committee discussed the option of
formatting the content area of the CCOGs in a pyramid structure with the most
critical content highlighted at the bottom of the pyramid.

The current CCOGs do not align the content to the course outcomes. To help
make the CCOG a better communication tool, the committee discussed making
explicit connections between the course content and the course outcomes as well
as explicit connections between the course outcomes and the college's core
outcomes.  This work was in progress when we realized that this shift in
outcomes should not be done in isolation from other courses in the pre-college
math sequence; in response, we formed a DE Math Subcommittee with the goal of
creating vertically aligned outcomes that created a coherent progression from
MTH 20 through MTH 95.  Since its formation in June 2013 there has been a major
shift in how we are hoping to approach DE level math courses.  As the Math SAC
considers new math pathways for DE level students, the CCOGs for these new
courses will likewise be explicitly aligned and measurable.

In summary, we are interested in developing rich, meaningful, and measurable
outcomes that better represent the intent of our courses than our current
outcomes do. Since course outcomes are required in syllabi, properly
representing the intended focus for the course in the course outcomes is
critical. We hope that the conversations in the LAC and Curriculum Committee
concerning course-level learning outcomes will lead to Curriculum accepting more
flexible wording of course outcomes in the near future.

\recommendation[Learning Assessment Council, Curriculum Committee]{The college should adopt a new model
	for course-level learning outcomes that allow for outcomes that are more
	descriptive of the actual content in the course and that are also better suited
for student-level and course-level assessment.}

\section{Addressing College Core Outcomes}

\subsection[Outcomes in courses and the math program]{Describe how each of the
	College Core Outcomes are addressed in courses, and/or aligned with program
and/or course outcomes.}

The college's Core Outcomes may be found at \cite{coreoutcomes}.

\begin{description}
	\item[Communication] is stated in many of our CCOGs as a course outcome. We
	believe it is important for students to communicate ideas using mathematics in
	a meaningful manner through appropriate use of notation and concise, accurate
	statements.  Here are some excerpts from course outcomes in MTH CCOGs:
	\begin{aquote}{MTH 111--112, MTH 211--213, MTH 251--254}
		{\ldots}and then interpret and clearly communicate the results.
	\end{aquote}

	\begin{aquote}{MTH 243--244}
		{\ldots}and clearly interpret the results via written or oral communication.
	\end{aquote}

	\begin{aquote}{MTH 111--112, MTH 243--244, MTH 251--254}
		{\ldots}understand and be able to communicate the underlying mathematics
		involved to help another person gain insight into the situation.
	\end{aquote}

	\item[Community and Environmental Responsibility] is not directly addressed in
	our courses or outcomes. Our Social Justice Workgroup
	(see \cpageref{cur:sub:socialJustic} and \vref{app:sec:socialJustic}) discusses issues
	related to teaching with community and environmental responsibility in mind.
	Also, sections of MTH 111H (see \cpageref{cur:sub:111H}) have a related component
	involving tutoring math to someone in a student's community. Examples of
	Service Learning components used by some faculty are given on \cpageref{other:sec:servicelearning}.

	\item[Critical Thinking and Problem Solving]
	is fundamental to mathematics so, at first glance, it may seem that there would
	be easy agreement between mathematics faculty for how this Core Outcome is
	present in our curriculum. This is not the case.

	Some PCC math faculty believe that everything in a math course is a
	manifestation of this Core Outcome, including practice problems intended to
	develop procedural fluency in a non-contextual setting (e.g., solve the quadratic
	equation by the Zero Product Property). Indeed, one can certainly witness the
	critical analysis required within the realm of procedural fluency when students
	struggle with  mathematical abstraction. This happens at the most basic level,
	especially for students who are not detail-oriented or unable to easily
	generalize abstractly in a non-contextual manner. It is also present at higher
	levels of math when students are faced with skill-based problems that require
	integration of several previously learned procedures.

	Traditionally, the main focus of most undergraduate math programs is developing
	a students' algebraic procedural fluency (including our current curriculum).
	While procedural fluency is important, some PCC math faculty wish to narrow the
	focus of this Core Outcome in a way that goes beyond procedural fluency. Some
	examples include the development of students' ability to solve meaningful
	contextualized problems, explain concepts and justify results in a
	mathematically sound manner, analyze mathematical patterns, or create elegant
	proofs.

	Assessment work has made us aware of how differently we can view critical
	thinking and problem solving, and it has developed our thinking considerably.
	This will remain an ongoing conversation. At this time we have chosen to take
	the broad view of Critical Thinking and Problem Solving that includes
	procedural fluency.

	\item[Cultural Awareness] is not formally covered in any of our courses.  Some
	individual instructors may incorporate elements of cultural awareness into
	their classes.  Some examples of how cultural awareness could be addressed by
	individual math instructors instructors are:
	%  I was reading recently (in the book ``Lean In'' which as it happens Chris
	%  Chairsell gave me to read, though that book references other studies if you
	%  need the original research) that women are less likely than men to raise
	%  their hands in class, though they often know the answer as well as or better
	%  than the men in class.  Apparently this is also true of minority students.
	%  You may want to consider as a SAC creating a new paradigm in future classes
	%  banning hand-raising as a method of encouraging class participation, because
	%  if anyone self-suppresses then the practice is ineffective to its very
	%  purpose (of encouraging participation).  Bottom line:  there is more to
	%  cultural awareness than teaching students the history of math.
	\begin{itemize}
		\item differences in notations and procedures used by different cultures;
		\item similarities and differences with regard to the use of mathematics as a
		tool;
		\item attitudes toward mathematics in various cultures (e.g., math anxiety is
		not a worldwide issue);
		\item history of mathematical  development in various cultures and how it was
		influenced by dominant philosophical attitudes of the region;
		\item use of mathematics as an agent of social change by shining a light on
		inequity with regard to treatment of, and resource allocation for,
		marginalized populations as compared to the dominant culture; the social
		justice workgroup considers this---see \cpageref{cur:sub:socialJustic} and
		\vref{app:sec:socialJustic}.
	\end{itemize}

	\item[Professional Competence] is described by PCC as the ability to
	``demonstrate and apply the knowledge, skills and attitudes necessary to enter
	and succeed in a defined profession or advanced academic program.'' As a
	discipline that builds upon prior knowledge and skills, successful completion
	of any of our sequenced courses prepares students for the next math course. A
	certain level of mathematical competency is valued by PCC and is required for
	all of our degree-seeking students. Currently, most students show math
	competency by taking a math course at PCC. In this way, the Math SAC plays a
	critical role in the professional competence for students' educational goals
	whether it be for a defined profession or advanced academic program.
	Discussions about the necessary mathematical knowledge, skills and attitudes
	for the wide range of educational goals of PCC's students are ongoing.
	% Consider adding a whole new paragraph that maybe starts ``Additionally, we
	% hope to offer new new DE level courses that will focus on STEM applications
	% as an integral part of learning math.  This would improve student ...

	Additionally, we offer one sequence that directly addresses professional
	competence within math education:  Our  MTH 211--213 ``Foundations of Elementary
	Mathematics'' sequence of courses are taken by students interested in pursuing a
	career in teaching in the K--12 education system.  At minimum, two of the three
	courses are prerequisites for obtaining a degree in Education in Oregon.  Each
	course emphasizes specific topics of mathematical theory that are the basic
	building blocks of mathematics instruction in the K--12 system.  Some
	instructors require students to maintain a portfolio, learn multiple assessment
	techniques, do field observations of teachers in the K--12 system, learn about
	the Common Core Standards and trends in education both state and nation-wide,
	and are given guidance in areas such as preparing for the CBEST and PRAXIS as
	well as decision making in the various avenues for pursuing a degree in
	education.  This exposure to what the field of education actually entails
	helps students make sound career choices early on in their academic pursuits.
	\item[Self-Reflection]
	is an outcome that we believe students develop in the course of their
	educational experience. Many Math SAC faculty believe we can assist this
	development; others question if this should be a responsibility of faculty
	(or if this should even be a Core Outcome of the college). While the SAC
	needs more discussion to develop a shared understanding of what Self
	Reflection means in the context of a mathematics course and how to develop
	it, we believe that it is present in our curriculum.

	We are exploring aspects of Self Reflection; here are two examples:
	\begin{enumerate}
		\item
		Development of strong study skills will improving students' ability to self-reflect; consequently,
		the SAC has discussed adding study skills as formal components
		of MTH 20 (see \cpageref{cur:sub:mth20}).  See \cpageref{cur:sub:studyskills}
		for more information on study skills material that incorporates student
		self-reflection.
		\item
		In AY 2011/12 the Math Learning Assessment Subcommittee (Math LAS) explored
		Self-Regulated Learning where students were guided in a self-reflective
		process that helped them evaluate their depth of understanding (or lack
		thereof).  Although Self-Regulated Learning was not incorporated into the
		assessment activity (due to the complexity and the lack of time remaining),
		members of the Math LAS who explored this believe it is worthy of future
		consideration.
	\end{enumerate}
\end{description}

\subsection[Core Outcomes Mapping Matrix]{Update the Core Outcomes Mapping Matrix for your SAC as
appropriate.\footnote{\url{http://www.pcc.edu/resources/academic/core-outcomes/mapping-index.html}}}
At this time, the Learning Assessment Council expects each LDC SAC to address and assess at least four of the college's Core Outcomes somewhere in their curriculum. As shown in our current Core Outcomes Mapping Matrix (\vref{sec:app:coreoutcomes}), most of our courses address four Core Outcomes.  

To date, most of our outcome assessment activity has been focused in our
lower-level courses. Because of this, it is difficult for us to make truly
informed distinctions about variations in outcome levels through our vast array
of courses.  We offer many courses at the DE level as well as many courses at
the 100 and 200 levels.  While it is obvious that the skills assessed at these
various levels are quite different in scope, it is not as clear that the levels
of Core Outcome achievement vary in a similar way.

Consider level 4, which is described as ``Demonstrates thorough, effective
and/or sophisticated application
of knowledge and skills.''  Does that phrase come with an implied ``appropriate
to the course level being assessed''?  The assignment of levels is very
dependent upon the answer to this question.  There are other questions that can
be asked that could also greatly impact the way in which levels are assigned.
To wit:
\begin{itemize}
\item Pitched vs.\ Attained: Should levels be assigned based on ``the level to
  which the course is taught'' or ``the expected level of attainment for a
  student who passes with a C (or B or A)''?

\item Coverage vs.\ Depth: If level 3 or 4 is assigned, should the Core Outcome
  be a recurring theme in the course or can level 3 or 4 be reached if there is
  only one or two assignments that have ``depth''?

\item Prerequisite expectation: If a course has an expected prerequisite
  knowledge/skill for a particular Core Outcome, which is critical for success
  in the course, but not covered/developed in the course, what level should be
  assigned: 1 or 4?

\item Assessment: Is there an expectation for assessment for all courses based
  on the assigned level (e.g., perhaps SAC-level assessment is required for
  courses where a level 3 or 4 is assigned)?

\item Benchmark comparison: If we rate a course with a 200 designation as
  ``level 4'' for a particular outcome, is it appropriate to have a
  developmental education course also at level 4 or should it be at most level
  2?

\item Course-to-course comparison vs.\ stand-alone analysis: Generally, should
  a course-to-course comparison even be done? Perhaps given the various ways
  Core Outcomes might manifest, a course-to-course comparison is meaningless
  (i.e., just as Critical Thinking and Problem Solving could be level 4 for
  very different reasons across disciplines, it may be possible to have level 4
  for very different reasons within a single discipline).
\end{itemize}

\recommendation[LAC, Degrees and Certificates Committee]{We recommend that the
LAC compose a concise but thorough document that precisely specifies the
context in which levels should be determined when a SAC constructs its Core
Outcomes Mapping Matrix.} 

\section[Assessment of College Core Outcomes]{For Lower Division Collegiate
 (Transfer) and Developmental Education Disciplines:  Assessment of College Core
 Outcomes    (note:  Please include the full text of your annual reports as
 appendices, and summarize them here).}\label{ass:sec:coreoutcomes}

Our full annual Learning Assessment reports can be found at these links, and
are summarized in the pages that follow.
\begin{description}
	\item[2009/10:] \cite{annualLASreport2009}
	\item[2010/11:] \cite{annualLASreport2010}
	\item[2011/12:] \cite{annualLASreport2011}
	\item[2012/13:] \cite{annualLASreport2012}
\end{description}

\subsection[Core Outcome assessment design and process]{Describe the assessment design and processes that are used to
determine how well students are meeting the College Core Outcomes.}
We will discuss the evolution of the assessments by academic year, starting
from 2009/10.
\begin{description}
	\item [2009/10: Critical Thinking  \& Problem Solving]

	This was the first year the LAC asked SACs to assess a Core Outcome. Our
	assessment activity was developed by a small group of interested math faculty
	with minimal coordination with the full SAC. Student artifacts were collected
	from the interested faculty who volunteered to give the assessment activity in
	their courses, and so it was not a statistically sound sample.  The activity
	involved both direct and indirect assessment:
	\begin{itemize}
		\item The direct assessment involved finding and correcting conceptual,
		arithmetic, and formatting mistakes in expected procedural skills for MTH 65
		and MTH 95. Students must know how to do the problem before they are able to
		to identify and correct mistakes. This level of analysis is typically very
		difficult for students and is a high on Bloom's taxonomy.
		\item The indirect assessment involved asking students to respond to questions
		like, ``Do you feel this class has improved your critical thinking and problem
		solving skills?''
	\end{itemize}

	\item[2010/11:  Critical Thinking \& Problem Solving and Communication]

	At the beginning of Fall 2010, the Math SAC decided to create a standing
	committee to ensure assessment work was a high priority. The Math Learning
	Assessment Subcommittee (Math LAS) was born.  There were 14 members, most of
	whom were full-time faculty. This was a big improvement from last year's work
	where only a small group of faculty participated.

	Although our previous year's CT \& PS activity had been high on Bloom's
	Taxonomy, we decided that procedural skills did not fully capture how we want
	our student to think critically about mathematics.  Instead of continuing with
	the previous year's work, we decided to develop a new activity for CT \& PS.

	Still very new to assessment, we did not know what type of activity might
	generate the most useful information.  To help us decide, we developed three
	assessment activities and collected student artifacts for each activity.  The
	chosen activity was randomly given to 12 of the 72 sections of MTH 65 held in
	Winter 2011.  (Note: We also created an activity for MTH 244, but there was an
	error in one of the questions.  Although a portion of the artifacts were
	evaluated, we ultimately decided to abandon this attempt and focus our limited
	resources toward the MTH 65 analysis.)

	For the MTH 65 assessment, we collected 240 student artifacts.  To help ensure
	that data would be a true SAC-level assessment (vs.\ an evaluation of individual
	instructors), faculty members were instructed to remove identifying information
	from student work and submit their artifacts to an administrative assistant who
	tracked submission of work only.  Sixteen members of the SAC were normed to the
	rubric that had been developed by the Math LAS.  Two members of the LAC's
	Program Assessment for Learning (PAL) facilitated this work and guided faculty
	through a trend analysis.  (Note: Our process and involvement of Math SAC
	members was so impressive as compared to other large SACs that the Learning
	Assessment Council awarded the Math SAC an ``Oscar'' at their Spring circus
	event.)

	\item[2011/12: Self Reflection and Professional Competence]

	This year we sent a survey to all students enrolled in a math class in the first
	week of Spring 2012.  The Math LAS was awarded a LAC grant and we used these
	funds to hire a consultant, Una Chi, to help us refine the survey and evaluate
	the student responses to the survey.  We also discussed the wording of the
	questions with the DE Reading and Writing faculty members to help ensure the
	questions would be understood by all students. Approximately 2300 students
	responded to the survey, and the response rates for particular courses mirrored
	student enrollment in those courses and other demographic information.  The
	survey was an indirect measure of students' perceptions.

	For Self Reflection, we focused on questions that we felt would fit the
	following three areas:
	\begin{enumerate}
		\item Reflection---Core reflective thinking items; autonomy and relatedness
		aspects from self-determination theory
		\item Orientation---Mastery/performance, internal/external locus of control
		(hold self responsible vs. holding others responsible)
		\item Competency---Belief about self-ability to perform in math
	\end{enumerate}
	Sample Self Reflection items on the survey:
	\begin{quote}
		I know when I need help on a math concept.
	\end{quote}
	\begin{quote}
		When I get a math test back, my grade is what I expect it to be.
	\end{quote}
	\begin{quote}
		My feelings about math affect my learning of math.
	\end{quote}

	For Professional Competence, we used the suggestions of our LAC coach to craft
	questions about students' perceptions of math in terms of their future
	job and career goals.

	Sample Professional Competence items on the survey:
	\begin{quote}
		The skills I learn in a math class are not important to me or my future goals---I just need to pass the course.
	\end{quote}
	\begin{quote}
		In PCC math classes, what knowledge, skills, habits or ways of thinking have you
		practiced that might help you in the work place? [Choices included punctuality,
			problem solving, working in groups, self discipline, career specific math
		skills, interpret graphs/charts]
	\end{quote}
	\begin{quote}
		My career interest requires some mathematical knowledge.
	\end{quote}

	\item[2012/13:  Critical Thinking \& Problem Solving and Professional Competence]
	% On page 13 and 14 you discuss how individual course outcomes are not
	% addressed due to the LAC focus on college core outcomes.  But this year
	% y'all totally addressed individual course outcomes, in the context of the
	% CT & PS core outcome.  Plus it was doubly recognized with awards.  I think
	% you should pull out this years' work as specifically covering course
	% outcomes right upfront on page 13.

	This was our third investigation into Critical Thinking \& Problem Solving.  Our previous attempt had given
	us rich data, but this year we decided to investigate students' ability to solve
	nine math problems that specifically represent the topics covered in MTH 95 that
	we consider essential for success in MTH 111.  The assessment activity was
	administered to every face-to-face section of MTH 95 at all PCC campuses in the
	Winter of 2013.  We collected 677 student responses from 33 different sections
	across the college; all identifying information for both instructors and
	students were removed.

	The math problems included in the assessment were selected by faculty from our
	current MTH 95 textbook:
	\begin{itemize}
		\item For Critical Thinking \& Problem Solving: We incorporated problems that contain units and involve a
		real-world context.
		\item For Professional Competence:  We incorporated problems that emphasize the content needed to be
		successful in the next course, MTH 111.  For this activity, Professional
		Competence was interpreted as the, ``knowledge, skills and attitudes necessary
		to enter and succeed in a defined profession or advanced academic program''
		(\cite{coreoutcomes}).
	\end{itemize}
	During the LAC's summer peer review process, our report won awards in two
	categories: ``Assessment Design'' and ``Planned Improvements to Increase Student
	Attainment of Outcomes''.  The awards were announced at 2013 SAC Chair
	Inservice.\footnote{\awardsurl}
\end{description}

\subsection[Core Outcome assessment results]{Summarize the results of assessments of the Core Outcomes.}
As a reminder, the full reports for our assessment activity are available using
the links on \cpageref{ass:sec:coreoutcomes}.

\begin{description}
	\item[2009/10: Critical Thinking \& Problem Solving]
	We did not evaluate the student artifacts due to lack of time during the 2009/10
	academic year. We intended on completing the work during 2010/11; however, since
	we did not have a statistically valid sample and since we wished to try a
	different type of assessment during 2010/11, we decided not to evaluate these
	artifacts.  Even though we did not evaluate student learning, the faculty
	members gleaned valuable information about the process of assessment. The
	assessment artifacts have been saved in case the Math LAS wishes to review them
	to guide future work.
	\item[2010/11:  Critical Thinking \& Problem Solving and Communication]
	\Cref{ass:tab:201011scores} presents the results of the rubric scores for our
	2011 assessment of Critical Thinking \& Problem Solving and Communication in 13
	sections of MTH 65.
	\begin{table}[!htb]
		\centering
		\caption{2010/11 Assessment Scores}\label{ass:tab:201011scores}
		\begin{tabularx}{\linewidth}{Xlll}
			\toprule
			Rubric Score  & 1 or 2               & 3                 & 4 or 5                 \\
			              & (below expectations) & (met expectation) & (exceeded expectation) \\
			\midrule
			CT \& PS      & 55\%                 & 35\%              & 10\%                   \\
			Communication & 50\%                 & 28\%              & 22\%                   \\
			\bottomrule
		\end{tabularx}
	\end{table}

	We realized that the rubric scores did not tell us specific information (e.g.,
	\emph{why} did the artifact score ``below expectation'' or ``exceeded
	expectation''?).  The LAC Program Assessment for Learning  facilitators
	suggested that we do a trend analysis which produced more meaningful
	information. Here are some results of the trend analysis:
	\begin{itemize}
		\item Many students did not seem to realize that not all data are linear.
		\item Many students were not able to give a well-supported conclusion.
		\item Students typically do not represent equivalence correctly.
		\item Many students incorrectly applied the idea of percentage.
	\end{itemize}

	%%Alex is here

	\item[2011/12: Self Reflection and Professional Competence]
	For Self Reflection:
	\begin{itemize}
		\item Students are not self-critical enough.
		\item There was a significant group mean difference between self-reported grade
		and self-reflection behavior on all grade level differences.  Note: ``grade
		level'' is the self-reported grade (A, B, C, D, F, P, NP, other) for the
		student's previous math class.
		\item There is a clear difference in the reflective thinking ability of students
		in high-level math courses versus students in low-level math courses (like MTH
		20).
	\end{itemize}
	For Professional Competence:
	\begin{itemize}
		\item It was surprising that Engineering was chosen as ``my career interest'' by
		11\% of respondents (the plurality). Nursing and Business were both in second
		at 7\%.
		\item Presentation skills ranked low by students as helpful in the workplace.
		A good SAC discussion could center around this; we may not
		wish to ``force'' students to perform presentations in lower level courses where math
		anxiety is increased.
	\end{itemize}

	\item[2012/13:  Critical Thinking \& Problem Solving and Professional Competence]
	% FWIW, I think you should offer this evaluation again (though of course
	% modified based on what you've learned) if only to answer your own questions
	% about why they did so horribly.

	Our assessment consisted of nine math problems.  After all submissions were
	graded, the average score was approximately 3.8 out of 9.  On a class-to-class
	basis the average had a low of 1.7 out of 9 to a high of 5.4 out of 9.  We were
	unsure of why the average of all classes was so low, and we found it alarming.
	Did students not take the activity seriously since most instructors did not
	assign points to it?  Would it have been better to give it with the final exam
	when most students were prepared for a mathematics assessment?

	Data in more detail is presented graphically and with tables in section 3 of the
	full report, \cite{annualLASreport2012}. Some summary points follow:
	\begin{itemize}
		\item The problems that involve working with function notation were answered
		correctly at a much lower rate than we expected given the amount of time
		dedicated to that topic in MTH 95.
		\item A problem that involved linear equations was also answered correctly at a
		much lower rate than we expected, as that topic is covered in MTH 60, 65 and
		95.
		\item Given that time constraints made it difficult to discern a student's
		conceptual understanding of a topic and our decision to mark answers as
		correct or incorrect (with no ``partial credit''), we should consider altering
		this activity if used again.
	\end{itemize}
\end{description}

\subsection[Assessment-driven changes]{Identify and give examples of assessment-driven changes that have
been made to improve students' attainment of the Core Outcomes.}

Below we discuss the assessment-driven changes that came out of our annual
assessment projects in academic years 2009/10, 2010/11, 2011/12, and 2012/13:

\begin{description}
	\item[2009/10: Critical Thinking \& Problem Solving]

	We chose to start anew rather than analyze data that was not statistically
	significant.  No course-level assessment driven changes came from this years
	work, though the SAC did significantly modify its approach to assessment in
	following years due to this first year finding.

	\item[2010/11:  Critical Thinking \& Problem Solving and Communication]

	During 2010/11, it became clear that the Math LAS members would not be able to
	develop/conduct assessment \emph{and} lead the SAC with assessment-driven
	changes from the prior year's work.  At the end of this academic year, the SAC
	created another assessment standing committee, called the Action Subcommittee.
	This subcommittee will take the results and the recommendations from the
	previous year's Math LAS research and lead the SAC in deciding what should be
	changed and how to implement that change.  For 2010/11 work, only individual
	changes to instruction occurred from the assessment results.  (See section 1
	from the full report, \cite{annualLASreport2010}.)

	\item[2011/12: Self Reflection and Professional Competence]

	After brainstorming a list of actionable items from the 2012 research, the 
	Action Subcommittee decided to work on the following item: ``disseminate successful ideas
	already used by our faculty for improving self-reflection via study skills and
	student-centered learning.''  The goal was to create activities that faculty
	could easily incorporate into their classes that would help students develop
	self-reflection behaviors that would lead to better study skills.  Math SAC
	members were asked to submit activities that were already being used
	successfully, and the committee received 25 different activities.  During an
	all-day SAC meeting we split into breakout sessions and each group was asked to
	look over the activities and create a list of best practices for incorporating
	them into the classroom. These worksheets are available for instructors to
	download and incorporate into their classes at \cite{selfcenteredlearning}.
	Additionally, a faculty member created a series of self-reflection and study
	skills videos that are being used in a lot of developmental classes (see
	\cpageref{cur:sub:studyskills}).

	\item[2012/13:  Critical Thinking \& Problem Solving and Professional Competence]

	For the complete list of actionable items from this year's research, see section
	4 in the full report, \cite{annualLASreport2012}.
	\begin{itemize}
		\item Add to the CCOGs the expectation that students check the reasonableness of
		their results (e.g., a result of $-5$ or $1{,}000{,}496$ would not be a
		reasonable result for ``the number of hours driven on a weekend trip'').
		Ideally, students should be encouraged to develop a habit of verifying their
		results regardless of whether or not the problem is a contextualized problem
		or not.
		\item Create a minimum skills test for MTH 95.
		\item Discuss methods of course content delivery in a way that supports both
		full- and part-time faculty.
		\item Form a Developmental Math Committee that will research different ways we
		might be able to redesign our pre-college curriculum in order to better
		prepare students for college level math as well as better serve students in
		CTE programs.
	\end{itemize}

\end{description}

The Action Subcommittee is currently reviewing the 2012/13 assessment work and
may propose to the SAC other ideas for implementation.