At Taskstream, we’ve been working with colleges and universities since 2000 to support assessment for improvement. Our work usually involves partnering with assessment directors, accreditation coordinators, and other faculty and staff to help a single institution (or program) organize and improve how it assesses student learning.
More recently, we have also had the benefit of contributing to some very interesting initiatives to measure learning outcomes across institutions and state lines. These projects represent a paradigm shift in how learning is and will be assessed in this country to allow for better benchmarking across institutions and transfer between institutions. But they also include lessons for individual institutions working to engage their communities in meaningful learning assessment and improvement.
For us, partnering with the higher education associations and institutions involved in these projects has given us the opportunity to think more deeply about reliability and validity with experts in assessment design and data analysis. And thinking about how to scale a project to hundreds of faculty members and assessment directors at over a hundred two- and four-year institutions in twelve states has inspired us to focus more clearly on ease-of-use for all of the distinct users who are interacting with the system.
Reflecting on our involvement with these projects, I thought of a comment one of my colleagues made in a meeting the other day on another topic: in taking us up out of the context of a single institution and highlighting the importance of cross-cutting learning outcomes important for all students in our country, this work has helped us to “see the forest and the trees.”
The following infographic gives an overview of these projects. For those interested in learning more, the rest of the post below includes more of the details.
The Multi-State Collaborative to Advance Learning Outcomes Assessment
In 2014, the American Association of Colleges & Universities (AAC&U), the State Higher Education Executive Officers Association (SHEEO) and nine states selected Taskstream to be the technical partner for the Multi-State Collaborative to Advance Learning Outcomes Assessment (MSC).
The MSC seeks to create a scalable way to assess essential knowledge and skills (things like written communication, quantitative literacy, and critical thinking) based on actual student coursework scored by faculty using standard rubrics. Not standardized tests. Not surveys. The work students produce for their course assignments. Authentic evidence of student learning resulting from their work in college.
The initiative uses rubrics previously developed by over 100 faculty and other education professionals under the AAC&U VALUE Initiative to score samples of student work from participating institutions. Select faculty from these institutions attend an in-person scorer training workshop, led by AAC&U, prior to scoring.
“The MSC is designed to produce valid data summarizing faculty judgments of students’ own work, and also seeks to aggregate results in a way that allows for benchmarking across institutions and states. The primary goal of the initiative is to provide assessment data that will allow faculty and institution leaders to assess—and improve—the levels of student achievement on a set of cross-cutting outcomes important for all disciplines.” SHEEO Website
The project’s leaders knew from the beginning that technology was necessary in order to scale the work across institutions in different states. The previous VISION project in Massachusetts, led by former state commissioner Richard Freeland, laid the foundation for the MSC and tested many aspects of the process but was handled entirely on paper. Project leaders drove boxes of papers across the snow-covered state, faculty scorers wrote on paper rubrics, and then entered the scores into a spreadsheet. It was a cumbersome and arduous process, to say the least.
Although Taskstream already had a full-featured system for collecting, assessing, and reporting/analyzing student performance data that was being used in hundreds of institutions, we were in the process of developing a new product to simplify and streamline the direct assessment of student learning outcomes based on samples of student work.
We worked with the MSC project leads to make sure the details of their process were handled well through this new system, now known as Outcomes Assessment Projects (formerly Aqua). In December 2014, representatives from the participating campuses started uploading student work samples into Outcomes Assessment Projects (formerly Aqua) for distribution to faculty scorers. In March 2015, faculty from across the participating states and institutions began scoring the work, after attending a rubric calibration workshop in February, led by AAC&U, that also included a 30-minute demo and Q&A session on Outcomes Assessment Projects (formerly Aqua).
Pilot Study
The pilot study focused on three learning outcomes: Written Communication, Quantitative Literacy, and Critical Thinking. In that initial year, 176 faculty scored over 7,200 student artifacts and included 76 institutions. The study showed that student papers from different assignments – rather than standardized tests, indirect surveys, or common assignments – can be used to produce valid data for comparison between institutions and to improve teaching within an institution.
The study also demonstrated that Taskstream’s technical solution for uploading student work samples, scoring the work with the VALUE rubrics, and reporting on the project’s progress and assessment results not only works, but also was user-friendly. In a post-study survey conducted by Taskstream, 97% of the scorers who responded found Outcomes Assessment Projects (formerly Aqua), to be “easy” or “very easy” to use.
Demonstration Year
Building on the pilot’s success, the project has expanded to include 12 states and over a hundred institutions this year. The demonstration year also added a fourth (optional) learning outcome, civic engagement, and grew the number of calibrated scorers closer to a couple hundred faculty.
The year has not only expanded participation to include new states and institutions, but also has focused on improving validity and reliability, including testing online calibration activities for those faculty who scored last year and working more closely with recognized data experts to fine-tune the methodology for distributing work for additional rounds of scoring.
One of the most promising benefits of the MSC has been the conversations it has started on campuses all across the country. The process of identifying which assignment might yield appropriate samples of student work to demonstrate the learning outcomes and looking at how faculty at other institutions scored this work has generated rich discussions about assignment design – e.g., exploring questions such as, “what are we asking our students to do?” “are we providing enough guidance?” “does this prompt elicit the types of outcomes we are hoping students will achieve?”
In these two years, the MSC has made significant strides towards establishing a new model for cross-institutional learning assessment. State leaders, institutional representatives, and faculty have all expressed enthusiasm for the project and how it is helping guide and inform conversations about student learning within and across institutions. (If you’re interested in hearing what the project leaders have to say about the benefits and methodology of the project and the future of this work, I recommend these webinar recordings: “Assessment for Improvement: The Multi-State Collaborative to Advance Learning Outcomes Assessment/VALUE Initiative” and “A Look Ahead for the Multi-State Collaborative to Advance Learning Outcomes Assessment/VALUE Initiative.”)
The WICHE Interstate Passport Initiative
With its Interstate Passport Initiative, the Western Interstate Commission for Higher Education (WICHE) seeks to improve graduation rates, shorten time to degree, and save students’ time and money with a new framework for transfer based on student learning outcomes.
The Interstate Passport Initiative brought together faculty from 2- and 4-year colleges and universities from multiple states in the Western region to design a better way to help students who want to transfer to a new school. The first step was to develop a consensus set of learning outcomes in nine knowledge and skill areas that map to AAC&U’s LEAP Essential Learning Outcomes: oral communication, written communication, quantitative literacy, creative expression, human cultures, natural sciences, human society and the individual, critical thinking, and teamwork and value systems. The idea is that this set of outcomes will become the new currency for block transfer of lower-division general education.
With funding from the U.S. Department of Education, through its First in the World grant program, one aspect of the current phase of the project is concerned with engaging faculty at partner institutions to examine the types of evidence that are being used to determine student achievement of the outcomes and, therefore, eligibility for transfer. As described on the project website, “the goal is to expand faculty understanding and choices of critical assignments in designated courses and to share ideas in competency assessment among institutions.”
Faculty from 2-year and 4-year institutions in three states will use two new sets of rubrics – created in collaboration with the National Center for Higher Education Management Systems (NCHEMS) – to score the assignments and the responding student work in Outcomes Assessment Projects (formerly Aqua) by Taskstream. Much like the initial years of the MSC, the current phase of the Interstate Passport project is about laying the groundwork and proving the model for learning outcomes to become the new currency for transfer and comparisons between institutions.
The Result? Better Conversations About Student Learning.
It is incredibly rewarding when we work with institutions that are really able to take off in their assessment efforts and drive real change based on data they have organized and gathered more easily using our tools.
With these groundbreaking multi-state initiatives, we’re also excited to help broaden the reach of outcomes assessment beyond the walls of a single institution and open up meaningful conversations about what students know and are able to do. We’re honored to support these efforts and are excited by the promise they hold for higher education in our country.
The post Seeing the Forest and the Trees: Assessing Learning Outcomes Across Institutions appeared first on Watermark.
