Seminar: Information Artifact Evaluation and Comparison

To be held at the University of Zürich, Department of Informatics, Information Management Research Group in April/May of 2017.

Lecturer: Hans J Scholl, Professor, The Information School, University of Washington, Seattle, WA, USA

Dates: 24.4 from 08.00 - 12.00 and 28.4, 5.5, 12.5 from 12.15 -18.00 

Target group: Master students interested in IS and PhD students 

Grading: The participants will graded on the basis of a presentation (40%) and a paper written during the four weeks (60%). Both will be a group effort. 

Credits: 3 ECTS 

Language: English

Type: Seminar

Programm: Wirtschaftsinformatik, Informatik

Code: MINFS546

Course catalogue: Link

The evaluation of information systems or artifacts as “outcomes” of software engineering (SE) projects and new Information Systems (IS) implementations have been a focus of study in SE and information systems-related research for quite some time. In recent years, evaluating artifacts, for example, mobile applications or websites has become more important, since such artifacts play increasingly critical roles in generating revenues for businesses, and the degree of artifact effectiveness is seen as a competitive factor. With the TEDS framework/procedure a novel and comprehensive approach to systematic artifact evaluation and comparison has been presented a few years ago, whose effectiveness and analytical power in comprehensive and highly detailed artifact evaluations and comparisons was empirically shown. In this seminar the TEDS framework/procedure will be detailed, and the empirical evidence of its effectiveness will be discussed. However, despite its demonstrated capabilities and fine-grained analytical power the TEDS approach has not completely overcome the main barrier to usability and evaluative studies, which are high cost and extensive time and resource commitments. The seminar will also highlight and discuss avenues to overcome these particular barriers, since frequent and speedy information artifact evaluations and comparison have become a major factor in rapidly redesigning and improving the effectiveness of information systems.

At the end of this research seminar, participants will have been introduced in depth to the TEDS framework and procedure. Depending on the number of seminar attendance, the participants will be divided into two groups, which each will choose for themselves, which information artifact they want to evaluate and compare. Typically, the evaluation will entail the evaluation of a publicly available compound artifact, that is, a set of connected artifacts, for example, an organization’s official website, its official social media appearance, and its official mobile application. The evaluation will analyze each component part, the integration of the components in terms of functionality, consistency, and content.

Seminar participants will be trained as expert raters in the use of the TEDS framework and procedure. Next the participants will jointly perform the evaluation of an “anchor” or reference artifact. The raters will be trained in inter-rater consistency and reliability checking. In a second round the raters will then rate the component parts of the information artifact of their choosing. They will learn to document strengths and weaknesses of the component parts in fine granularity.

The numerical ratings will be reconciled again in an inter-rater consistency and reliability session. Narrative accounts and screenshots will be inspected an discussed. The detailed evaluation of the compound information artifact and its comparison to the anchor site will be documented for all personae, scenarios, and categories detailed in the TEDS framework.

The findings will be written up in the format of a research paper, which will be submitted for review and potential publication to an appropriate academic outlet.

Scholl, H. J., Eisenberg, M., Dirks, L., & Carlson, T. S. (2011). The TEDS Framework for Assessing Information Systems From a Human Actors’ Perspective: Extending and Repurposing Taylor’s Value-Added Model. Journal of the American Society for Information Science and Technology, 62(4), 789–804. doi:10.1002/asi.21500