Work Action Analysis to Structure Planning and Formative Evaluation of an Engineering Course Using a Course Management System

Abstract

Cognitive engineering, by identifying behavior-shaping constraints, provides methods for design and evaluation of complex socio-technical systems. However, traditional methods examine only one type of constraint, either cognitive or environmental. In learning service systems such as education, both cognitive and environmental constraints must be examined together. Improved methods of planning and formative evaluation are needed for engineering education and other learning service systems. Therefore, this dissertation develops a new cognitive engineering method, Work Action Analysis (WAA), that is able to capture cognitive and environmental constraints in a single model. The WAA model represents a learning service system on three dimensions: means-end decomposition, parts-whole decomposition, and roles of cognitive agents. WAA also provides methods for developing and using this model in planning and formative evaluation. The WAA method for planning evaluation explicitly represents the evaluators mental model of a learning service system and examines its alignment to guide its design. The WAA method for formative evaluation then takes the WAA model and interprets evaluation measures in the context of the model. As a demonstration, the methods for planning and formative evaluation are applied to a portion of an undergraduate engineering course. To provide measures for formative evaluation of a course, a centralized evaluation component that collects performance, perception, and process measures was added to an Internet-based course management system. The WAA methods provide insights to the design and operation of this learning service system, including recommendations that could be implemented during instruction. The theoretical implications of the WAA model of learning service systems, and further extensions of WAA, are also discussed.