Characterizing the Conceptual Coherence of Computing Applications Using Ontological Excavation
We are investigating metrics for measuring the usefulness of computing applications relative to a specific use context. We define "usefulness" as "the extent to which an application's features succeed in assisting a set of users to achieve a set of goals, relative to the amount of effort required to engage those features." We define a feature as a user-accessible behavior or service implemented by a computing application. Computing applications embody and operationalize a set of concepts that correlate to concepts in the domain of the user. The degree to which the application’s concepts and the user's concepts agree is its conceptual fitness. We believe that applications with high conceptual fitness to a particular use context will also be perceived as useful by the users in this context. We have chosen in this work to study the problem of measuring conceptual fitness from the application side using conceptual coherence as our unit of interest. Conceptual coherence is an attribute of conceptual integrity, described by Fred Brooks as the property of a system designed under a unified and coordinated set of design ideas. It is the property of a computing application that measures the degree to which that application's concepts are tightly related. In previous work, we established that applications have core concepts – concepts that are essential to defining that application's features. An application will have a low conceptual coherence if it possesses a disproportionate number of non-core (peripheral) concepts. We intend to show that the conceptual coherence of an application determines its perceived usefulness to its users, and features with only tangential relationships to an application are less likely to be used and reduce that application's conceptual coherence. The set of concepts and relationships contained in an application can be said to be its ontology. We have developed methods for the black-box reverse engineering (excavation) of a computing application's ontology from the user interface and use techniques from the user interface and use techniques from graph theory to identify the core concepts of an application and its teleons – tightly connected functional subgroups within the ontology. We have also developed a technique called use case silhouetting which measures the ontological coverage, the number of concepts activated by a use case or set of use cases, and the relative importance of a concept to a set of use cases as a first approximation of conceptual fitness. We have applied these techniques to four small applications: the Windows 95/98 CD Player, the Palm Pilot Scheduler, Microsoft Notepad, and the Protocol Calculator / Calendar. We propose to perform two exploratory studies and one confirmatory one. Our first exploratory study will excavate and analyze the ontologies from three large systems – Microsoft Powerpoint 2000, Microsoft Word 2000, and Yahoo Instant Messenger 5.5. Our second study will obtain use cases from an independently written instruction manual (the "for Dummies" series). We will use these to develop use case silhouettes on our excavated obtained from Dr. Joanna McGrenere in her study on adaptable interfaces. We will show that the user preferences expressed by her subjects correlate to data obtained from our ontological analysis and use case silhouettes of Word.