The demand for data to demonstrate that educational funding is thoughtfully deployed and produces tangible benefits permeates higher education these days. This article is a brief for campus administrators about (1) campus-wide evaluation tools that produce some evaluative data about libraries, and (2) some of the library-specific assessment initiatives librarians have mounted to generate data for library management and to meet the accountability challenge.
 

The Trend

The accountability trend is clear in even a cursory review of the changes that have been made over the past few years in regional accreditation requirements. Looking at the six associations’ requirements for colleges and universities and for libraries reveals an unmistakable shift away from judging quality by describing inputs and outputs, and toward an expectation of routine assessment.* Moreover, the associations clearly expect educational quality to be demonstrated by a focus on using the assessment data for continual improvement. In its Accreditation Handbook, for example, the Western Association of Schools and Colleges describes this changing context for accreditation as, “A shift toward effectiveness and performance indicators beyond inputs and resources as the organizing basis for defining and evaluating quality.”

As institutions rush to learn the language and tools of this new trend, academic libraries are an excellent place to begin looking for information, and perhaps, inspiration. Traditionally, academic library quality has been expressed in terms of inputs such as collection size and square footage. Many institutions still celebrate the library’s acquisition of its millionth volume, for instance. But the dramatic, technology-driven reinvention of libraries over the past decade has resulted in demands for new methods of assessment.

Libraries now routinely supplement traditional collections and services with digital resources and Web-based service approaches. As librarians have reinvented academic libraries, we have also worked to create methods of assessment that reflect the reality of contemporary libraries and address the shift toward assessment and accountability in higher education. The North Central Association of Colleges and Schools offers the following perspective about libraries and learning resources:

In the same spirit, rather than providing a census of its collections, the Middle States Commission on Higher Education requires that “The institution provides sufficient learning and information resources and services to support the nature, scope, and level of the programs offered.” Colleges and universities are forced to ask, “What is ‘sufficient’?” The other regional associations have similarly broad statements, leaving librarians and institutional effectiveness staff to figure out a new approach. Library practitioners and scholars, therefore, have worked to create instruments and protocols for libraries that meet demands for accountability, measure service quality, and improve the data available to library administrators for effective library management.
 

New Measures for Libraries

Rather than focusing on inputs such as collection size and staffing, the first new library measures sought to describe what libraries do. That is, librarians began to report outputs such as the number of items borrowed or the number of reference questions answered. Those measures alone, however, still fell short of addressing whether library services were sufficient. As colleges and universities created student learning outcomes over the past few years, librarians also created measures that were based on outcomes — the extent to which student and faculty contact with libraries affected them and contributed to the mission of the university.

The increasing attention to measurement and assessment in libraries during the past decade might be described as a move beyond reporting what libraries do, to capturing the ways in which student learning is affected by library services and programs.

Since libraries contribute to the academic enterprise in a variety of ways including providing collections, creating research assignments, serving as an intellectual locus for the campus, and teaching bibliographic skills, there are several tools that may be used to evaluate one or more aspects of library sufficiency or quality. Most of these tools were first discussed in Library Issues by Stoffle and Phipps in March 2003.
 

NSSE

Institutions that participate in the National Survey of Student Engagement (NSSE) have an opportunity to use a portion of its scores as a library assessment tool. That this survey may be used for an additional purpose may be welcome news in Institutional Research offices across the country; the level of assessment activity required on campus has risen dramatically in recent years.

“The increasing attention to measurement and assessment in libraries during the past decade might be described as a move beyond reporting what libraries do, to capturing the ways in which student learning is affected by library services and programs.”

NSSE, administered by the Indiana University Center for Postsecondary Research, surveys first-year and senior students regarding their levels of engagement in the college experience. The key idea underlying the instrument is that student learning is tied to five areas of good practices in undergraduate education and may be analyzed through student engagement. The five areas measured are:

1. Level of academic challenge,

2. Active and collaborative learning,

3. Student-faculty interaction,

4. Enriching educational experiences, and

5. Supportive campus environment.

Increasingly popular, the most recent NSSE survey was conducted earlier this year at 538 four-year institutions and includes data from more than 660,000 randomly selected first-year and senior students.

Amy Mark and Polly Boruff-Jones have argued that bibliographic instruction, the library’s efforts to develop information literacy competencies in students, may be assessed through an analysis of specific NSSE results. Through case studies at the University of Mississippi and Indiana University-Purdue University Indianapolis, they have demonstrated that NSSE findings can be used as pre- and post-tests for assessing student progress in attaining Information Literacy Standards as defined by the Association of College and Research Libraries (ACRL). Since its launch in 2000, the NSSE data has produced statistically significant results and benchmarks for comparison groups.
 

SAILS: Standardized Assessment of Information Literacy Skills

If Information Literacy is a primary focus in your institution’s library, in addition to looking at NSSE, consider an instrument in development that is specifically intended to address information literacy programs. The SAILS project, coming out of Kent State University, has as its purpose, the creation of a standardized, valid, and reliable instrument for assessing information literacy programs. The instrument developed to date has 126 items that are intended to test the 5 standards, 22 performance indicators, 87 outcomes, and 138 objectives articulated in the Higher Education Information Literacy Standards adopted by ACRL.

One key difference between NSSE and SAILS is that the latter looks at information literacy among cohorts of students, rather than focusing on individuals. In practice, this means that institutions can see the specific institution average and the cross-institutional average, as well as the average by major, class standing, or other factors. What factors are the most important at your institution?

Both NSSE and SAILS assess libraries in relationship to the institution—their own institution, their students and faculty, tying them all together. The following programs compare libraries to libraries. This is still important, but it is not the same. Campus leadership should be aware of the difference; do not think that just because you are already involved in LibQUAL+^TM you don’t need to look at participating in the NSSE or SAILS programs. It’s apples and oranges.
 

LibQUAL+^TM

Overall service quality in libraries is increasingly being evaluated through LibQUAL+^TM, a library-specific instrument developed under the auspices of the Association of Research Libraries (ARL). While the practice of librarianship has been changing, it remains solidly based in a commitment to service. Librarians in all types of libraries work to ensure that their organizations provide high quality service to library users in support of the goals of the library’s parent institution. It would be rare indeed to discover an academic library, for example, that did not include the concept of service quality in its mission to support the teaching, learning, and research missions of the college or university in which it operates.

Librarians have sought improved methods of assessment to reflect the changes in libraries and provide pertinent information to library administrators for planning and management purposes. Consequently, library organizations such as ARL have engaged in a number of measurement projects including LibQUAL+^TM, which measures expectations and perceptions of library service quality.

The project has grown from 12 participating libraries in 2000, to 199 libraries in 2005. Libraries use the LibQUAL+^TM survey instrument to solicit and track users’ opinions of service quality. The survey has its origins in SERVQUAL, a service quality instrument that has been used widely in the commercial sector. Grounded in the gap theory of service quality, the instrument was developed for the for-profit sector in the 1980s.The gap theory of service quality measures three things.

1. Minimum level of service, or, the lowest level users will put up with,

2. Desired level of service, or, what they want,

3. Perceived level of service, or, how they think the agency or company is doing.

 

"Given the increasing investment in electronic information resources in libraries, assessing such services has become critical. A major goal of DigiQUAL is to identify best practices for digital library services."

Once the measures are collected, the gap between perceived and desired levels of service are defined and may be used as the basis for planning and implementing service improvements. LibQUAL+^TM survey data are analyzed and each participating library receives reports of the aggregate response data as well as its own institutional data so that they may compare local library performance with that of peer institutions. Librarians report that they have used the survey data to inform practical decision-making, identify best practices, analyze service deficits, and allocate resources.
 

Digital Library Assessment Project

The success of LibQUAL+^TM has spawned a related project that is working to modify the LibQUAL+^TM survey for assessing services provided by the National Science, Math, Engineering and Technology Education Digital Library (NSDL) program. Applying the LibQUAL+^TM protocol to NSDL has been dubbed, “DigiQUAL.” Given the increasing investment in electronic information resources in libraries, assessing such services has become critical. A major goal of DigiQUAL is to identify best practices for digital library services. Eventually, this project may result in the ability to assess service quality for digital libraries and generate data for librarians to use in improving the retrieval of relevant information and learner preparation for an information-rich society.
 

Another effort working toward determining the effectiveness of digital collections is dependent upon establishing measures. The E-Metrics project began in 2000 and was intended to explore defining and collecting data on the use of electronic resources. It became clear in the course of the E-Metrics project, however, that consistent standards for data did not exist. Without usable definitions, measurement and analysis is difficult and complex. Project COUNTER (Counting Online Usage of Networked Electronic Resources) began in 2002 to address the problem.

Project COUNTER is working to establish practices for consistently and accurately determining the usage of digital resources. Publishers have been invited to participate in the project, as they must agree to use the standards and protocols for defining and recording online usage. The current COUNTER Codes of Practice are published on the ARL website.

With standards in place, library administrators will be able to compare the cost per use of digital resources, and that is a powerful tool in the management of digital assets. Whether publishers accept and adhere to the COUNTER Code remains to be seen.
 

Knowing which digital resources are used, how they are used, and by whom becomes a very complex question in a networked environment. Based on an established method for determining the indirect costs of using digital resources for grant-funded activities, MINES differs from E-Metrics and COUNTER which quantify total usage. MINES for Libraries™ collects demographic information about those using electronic resources and the purpose for which they are using them, through a transaction-based survey. Data collected over a two-year period from 14 research libraries revealed, for example, that of the overall use of digital resources for all of the libraries 70 percent occurred outside the library and 30 percent inside the library.

Added to the ARL New Measures program in May 2003, MINES for Libraries ^TM was implemented in 16 Canadian institutions last year through a contract between ARL and the Ontario Council of University Libraries (OCUL).

This is an exciting and challenging time for academic libraries. The rapid development of information technologies has presented myriad opportunities and challenges. New technology-based tools have spurred librarians to reconsider and redefine collections, services, the skill sets required of library staff, and the attributes of library facilities. In some cases, old planning assumptions must be turned upside down—for instance, the assumptions that all users are in a building, and that the building contains all of the library’s collections. Clearly, contemporary libraries operate with different assumptions about the ways in which they might best carry out their responsibilities.

Assessment is evolving rapidly in education; from K-12 school report cards, to new accreditation requirements for colleges and universities. Librarians are committed to meeting the assessment challenge. We are finding new ways of measuring the quality of collections, services, and programs, and better ways of demonstrating the library’s contribution to student learning. For those libraries that have not yet moved away from merely calculating inputs and outputs to assessing their role and impact on teaching and learning—the time is now.
—Kathleen Hoeth can be reached at khoeth@fgcu.edu
 

References

Cook, C., Heath, F., & Thompson, B. (2003). “Zones of tolerance in perceptions of library service quality: A LibQUAL+.” Portal: Libraries and the Academy, 3(1), 113-123.

Gedeon, J., Radcliff, C. & O’Connor, L. Project SAILS: Facing the Challenges of Information Literacy Assessment, presented at the American Association for Higher Education 2004 Assessment Conference, June 14, 2004, Denver, Colorado.

Hernon, P. (2002). “Quality: New directions in the research.” The Journal of Academic Librarianship 28(4), 224-231.

Kyrillidou, M., Franklin, B. & Plum, T. Assessing the Value of Networked Electronic Services: The MINES survey. Panel Session at the ACRL Annual Conference, Minneapolis, MN, April 8, 2005. http://www.libqual.org/documents/admin/MINES%20Panel%20ACRL.ppt

Mark, A.E. & Boruff-Jones, P. D. (2003). “Information literacy and student engagement: What the National Survey of Student Engagement reveals about your campus.” College & Research Libraries, 64(6), 480-493.

OCUL MINES results for 2004-2005 http://www.arl.org/stats/mines/ocul/MINESFINAL2005.HTML.

Parasuraman, A., Berry, L. L. & Zeithaml, V.A. (1991). “Refinement and reassessment of the SERVQUAL Scale.” Journal of Retailing 67, 420-450.