Blog: Idea Exchange
Advancing Assessment at California Baptist University, Part 1: Automating Assessment
This is part one of a four-part series revolving around the Division of Online and Professional Studies (OPS) at California Baptist University (CBU), a private liberal arts university located in Southern California, and their journey in adopting new technology for the collection of assessment data on student achievement.
Throughout the series, Dr. Kathryn Norwood (Dean of Assessment and Accreditation) and Dr. Dirk Davis (Associate Vice-President of Academics) will present the eight steps of their change process using Kotter’s 8-Step Change Model, highlighting both their accomplishments, and some of the missteps that they wish they could redo like the second take in a movie shoot.
Step 1: Developing a sense of urgency around the need for change
We always begin with the end in mind. We began the process when we realized that, although we were way ahead of the game collecting academic assessment for student learning outcomes, we also had some challenges. First, some history. Within OPS, we design all of our courses in-house, using our full-time faculty. Every course has a “critical assignment” meaning that the final assignment for assessing student learning outcomes uses a rubric identifying the select, major outcomes for the course, and every course is assessed every time it is taught. The critical assignment rubric measures the select course level objectives, in addition to the applicable program level objectives, university student outcomes, regional accrediting core competencies, and professional accrediting core competencies. Prior to our Taskstream initiative, we were aggregating all of this data manually using our learning management system, spreadsheets, and much juggling.
We began noticing some challenges, beyond our juggling skills. As the challenges began to mount, we developed a sense of urgency around the need for change. We felt potential threats from validity challenges with data, reliability challenges with data collection, and sustainability challenges with an ever-increasing scope of work.
After significant research in examining the opportunities for vendors who could meet our needs, we identified only one vendor, Taskstream, that could help us to truly automate our assessment process. After getting approval to choose the vendor we thought best met our needs, we began frank discussions with department chairs, deans, and administration. In retrospect, more dialogue with faculty would have been beneficial. It would probably not have changed our overall direction or outcome, but it would have increased the awareness of faculty and their inclusion would have increased our social capital. We have certainly learned that those frank discussions with all constituency groups are imperative even at the onset of these types of initiatives. After extensive vendor exploration, we selected Taskstream as our partner in this automated assessment initiative. They immediately started working with our key personnel and training for the monumental task of personalizing the user interface, transferring all pertinent course and learning outcomes information, and training on how to build the standards within the application.
Step 2: Forming a powerful coalition
While we were building the skeleton of what assessment would look like in Taskstream, we were simultaneously forming a powerful coalition behind the scenes to a large extent. Ideally, these leaders and stakeholders within an institution of higher education include all faculty, discipline leads, department chairs, deans and administrators. We thought we captured this entire group, but subsequently discovered that the discipline leads were largely unaware of the preparations taking place for the roll-out of the new system. This was primarily due to a transition of multiple key personnel and the leadership missed the disconnection. Regardless, we strategically sought and acquired commitment from the other key stakeholders.
To start, we asked program faculty from a newly created program to pilot the new assessment tool. This group was the logical choice for a pilot due to the initial launch of their program, congruence of the curriculum, smaller volume of students, and because it seemed that all key stakeholders were on board with the new initiative. Retrospectively, this was a mistake due to us only getting feedback from one faculty member at a time, and the singleness of the curriculum led to unexpected challenges when we rolled out all the remaining programs. We had initially believed, after the pilot project, that we had addressed all potential challenges. We found out differently when we rolled out multiple programs simultaneously.
As we began to work on team building, we noticed that the organic composition of the team was department chairs and deans. So, at that basic level, ensuring a good team mix was not a challenge. As we began to critically look at team structure, we noticed the need for additional staff support at the dean level. These initial conversations ultimately led to the hiring of an academic support coordinator tasked with supporting all assessment initiatives. The dean of assessment and academic support coordinator subsequently were immersed in setting up our Taskstream structure. In hindsight, involving discipline and/or assessment leads in the development of templates and rubrics would have been more ideal. This may have led to a better understanding of the complexity of the programming involved with automating assessment. Our next steps were to begin training faculty in earnest, training the academic supporting coordinator, and fully involving faculty in rubric conversations. The goal here was to bring in all constituents so that they could gain an appreciation of the new assessment system. Again, bringing this group in earlier may have been more beneficial.
Part two of this series focused on Step #3: Creating a vision for change and Step #4: Communicating the newly formed vision is now available.
About California Baptist University
California Baptist University (CBU) is a private, faith-based, liberal arts university located in Southern California with a current enrollment of approximately 9,157 students. The Division of Online and Professional Studies (OPS) began in 2010 to service non-traditional students seeking a distance learning environment. Initially, offering eight programs to approximately 500 students, OPS has since grown to offering 37 programs that serve approximately 4,000 students each semester.
About the Authors
The authors have an extensive background in K-22 education. Both have served as K-12 public educators, full-time faculty in higher education, and have held administration positions that include student development, faculty development, assessment, and academic leadership at various levels. Dr. Kathryn Norwood currently serves as the Dean of Assessment and Accreditation for the Division of Online and Professional Studies at California Baptist University. Dr. Dirk Davis currently serves as the Associate Vice-President of Academics for the Division of Online and Professional Studies at California Baptist University.