There has been much discussion and hope of an impending learning revolution led by computer technologies that promise the augment, or replace, the teacher’s role in the classroom with the likes of which educators and learners have never seen. Yet, despite many successes and exciting advances using computer software in education, there has not yet been a revolution to speak of. Nonetheless, the educational sector does recognise the importance of learning technologies and already embraces the use of technology in education to aid their teachers and facilitators. However, progress in computer technologies occurs quickly and there is an ever‐increasing market of educational software and courseware that promises to be the next mind‐blowing step toward a learning revolution; with each new release, computer software producers seem to ask to forget about the software of yesterday and invest the time, resources and finances on the improved software of today. Unfortunately, for every succeeding development in learning technology there have been a greater number of failing attempts ‐ the demise of which is often due to budgetary and time constraints; arguably economics are the biggest influencing factor dictating the successful development of educational software. When presented with a saturated market of educational software, and tight budgets for investing the necessary resources into purchasing or developing educational software, the need for effective and meaningful evaluation techniques and principles are becoming increasingly important.
Educational software, once described as “a field with no memory” (Morris, Ehrmann, Goldsmith, Howat, & Kumar, 1994), is one of the newest introductions to the education market. Furthermore, it becomes increasing hard to evaluate educational software products as they continue to profess a new or innovative approach to learning thus making precedent evaluation techniques, proven or not, difficult to apply. This seemingly developing field, compounded with the fact that the term educational software covers an extraordinarily wide scope – encompassing almost any type of learning that may occur on the computer, from word‐processors to specific multimedia applications ‐ makes it increasingly difficult to form effective evaluation techniques. So, although the process of evaluation itself seems to be a simple concept, in the field of educational software there are several complex issues arising from competing perspectives on the methods of how to best evaluate software and who should perform such evaluations. It is clear, from the discussions and research, that there is no one‐size‐fits‐all approach to evaluating and this paper will discuss some of those issues faced when evaluating educational software and derive a set of principles an educator may use to perform evaluation.
The scope of educational software is broad and often contains many poorly defined pseudonyms such as Computer Aided Learning (CAL), Computer Based Training (CBT), Electronic‐Learning (eLearning), etc., that have arisen depending on where and when the technologies have been developed. It is beyond the scope of this paper to identify the difference between different forms of educational software, however, we will assume to be discussing educational software in a broad sense, encompassing all forms mentioned above, in a Higher Education context that caters for a largely ICT (Information and Communication Technology) literate audience. Naturally, it may be necessary for educators to take into consideration other criteria not mentioned within this paper, especially in the context of a different learner cohort.
Discussing Reasons to Perform Evaluations of Educational Software
The use of educational software first requires educators to make decisions about the software they would like to use based on their student cohort and their teaching environment. The best choice of educational software may be a critical success factor to the success, or otherwise, of their learners and consequently making the right decision may be crucial for themselves and their students. Choosing the correct educational software would, of course, be simpler if existing evaluations of plausible software candidates existed. However, there are a number of reasons why existing evaluations may not be useful in the decision making process and why it may be necessary for educators to evaluate software.
Problems of existing evaluations exist at many levels. For example, the underlying business model of developing educational software may focus on marketing of a product, rather than evaluation, which may lead educators to assume that educational software products are effective based on the advertising that claims them to be so (Reeves, 1994). If, therefore, the developers of educational software may fail to accurately evaluate, in a meaningful way, the educational soundness of their products it lies on external evaluators and educators to do so. It should also be noted that third‐party evaluations may be useful when determining the value of software. Testimonials, reviews, awards and case studies can provide excellent insight into evaluating software, however, may need to be taken lightly for a number of reasons. Such evaluations may not be as objective as they seem, being reviewed by inexperienced or highly subjective groups or individuals, and can often be promotional materials in the guise of an evaluation. More simply put, such third party reviews may not take into consideration the appropriateness of the software in all contexts or give heed to criteria that is exceptionally important in the circumstance of the reader, such as the longevity, target audience, or interoperability of the product.
It may be that perfectly executed and well‐documented evaluations of a given educational technology may have been done as part of lengthy Government or institutionally funded assessments and reports. Such empirical research documents are naturally time consuming and costly to produce but may almost certainly be more accurate than a quick‐and‐dirty evaluation performed by a facilitator. Unfortunately these kinds of evaluations often read more like science reports and, consequently, sit unread on shelves (Oliver M. , 2000). This forms another reason for evaluators to perform their own assessments of education software by determining their own set of criteria aiding them in making their own decisions.
There are many proven methods of evaluation, each with their own strengths and weaknesses and clearly there is no single most important criteria or single ‘magic bullet’ method of evaluating software (Oliver, 2000, p. 21, citing Oliver & Conole, 1998). However there are guiding sets of principles, as outlined in this paper, by which educators may decide the most appropriate criteria and methods for evaluating educational software on a case‐by‐case basis. There have been attempts to create broad software evaluation templates that intend be used to universally and quickly evaluate software, many of which are based upon rating systems, such as White (1989, p. 67), or checklists. Whilst evaluations using such systems may provide a simple and time‐cost effective method of evaluating software they may be problematic for a number of reasons. Firstly, whilst checklists are quite useful for assessing one‐dimensional features, such as cost and required hardware, they may become problematic when assessing more complex data such as, the crucial matter of how effectively the software enhances learning (Tergan, 1998). It would seem that most evaluation tools omit, or only lightly weigh, the outcomes the learner experiences when using the tool. The may be because evaluating learner experiences and outcomes is usually done by longer‐term emperical studies; these outcomes, however, are logically one of the most important criteria. Such evaluations may benefit from a divergence of qualitative and quantitative methods of evaluation by adoping a untilisation approach to evaluation. Secondly, considering the broad scope of educational software and contexts, as discussed above, checklists may be inflexible and require the assessment of non‐applicable criteria, or weighing less relevant criteria more heavily. White (1989) for example, poses a series of questions into categories of: General Quality of Content; General Instructional Quality; General Technical Quality; Social Studies Knowledge; Social Studies Skill Development; Values in Social Studies. Tergan (1998) suggests more simplified categories: Subject‐Matter Content; Learners; Instructional Methods and; Technology/Media, with central focus on the learner. The issue of determining irrelevant categories and criteria may be addressed by allowing evaluation tools to be customised for each evaluation to better ensure that only relevant criteria are assessed. Of course, such tailoring would require the evaluator to be sufficiently skilled to determine the appropriate categories to assess. Whether through customisation, self‐determined criteria, or simply choosing the correct pre‐existing template, the evaluator’s expertise in choosing the correct evaluation method and effectively assess based on the chosen method, will be critical in determining the effective outcome of any evaluation.
Examining Principles of Evaluative Criteria
It is clear that there is no straightforward evaluation method, therefore before evaluation can take place, it is important to discern the reasons for why it is to be done. It may be that educational software is evaluated in some circumstances based on its ability to perform a specialised task, however, it would often be more useful that educational software facilitates interoperability and extensibility so that it might be used over a wide period of time across a greater scope of tasks. Also, for educational software to be viable long‐term it is important that a product can demonstrate an active production cycle. If software fails to cope with changes in operating environments or systems it may become outdated before it can be provide the necessary return on investment (Morris, Ehrmann, Goldsmith, Howat, & Kumar, 1994). Whilst short‐term or small‐scale implementations of educational software may not be adversely affected by active support cycles from the developers, for long‐term systems, intended for use institute wide over a number of years, a proven supported productions cycle will be critical to successful outcomes.
Long production life‐cycles can be difficult and expensive for delivering educational software on a variety of platforms and operating systems; this may contribute to the reasons why we are beginning to see more educational software developed for use on the internet. Internet technologies are now almost ubiquitously available on computers and mobile computing devices in educational institutions and in homes, and are largely system and platform independent. Furthermore, internet technologies aid the distribution of software since there is no need for physical media and all transfers of software, and associated material, can be done online. The ubiquitous nature of the internet and softwares available on it may give rise to previously unheard of amounts of information and content.
The use of the internet is one of the newest advances in educational software, however that in itself cannot be an improved measure of how the software works. Candy (2004) suggests the sheer amounts of information available online is likely to be overwhelming to learners and us unlikely to facilitate the kinds of deep‐level and constructivist learning that is desirable (Candy, 2004, p. 225). By extrapolation, this is also true for educational software – particularly if software is to be used independently by learners. It is important to acknowledge that online facilitation of learning is different to traditional methods, and as such, should demonstrate effective use of online pedagogies (Candy, 2004, p. 233). Of course, the use of the internet for educational software delivery has also opened up opportunities for facilitating and integrating synchronous and asynchronous communication between learners and facilitators. However, Oliver (2000, p. 27) notes how the introduction of certain group methodologies may detrimentally affect the success of online facilitation by “loss of spontaneity” and the “inability to take body language into consideration”. Technologically it may be advantageous for online educational software to allow for a combination synchronous and asynchronous exchange of information (Volery & Lord, 2000), particularly if the software is facilitating course delivery and, if present, the usefulness of such online communication is taken into account during an evaluation.
An increasing important and critically essential component of evaluating educational software is heuristics – a measure of the usability of a product’s user‐interface. The ten usability heuristics defined by Nielsen in 1994 are still widely used today and comprise of: Visibility of system status; Match between system and the real world; User control and freedom; Consistency and standards; Error prevention; Recognition rather than recall; Flexibility and efficiency of use; Aesthetic and minimalist design; Help users recognize, diagnose, and recover from errors; and Help and documentation. Interface design is becoming a more standardised practice and we can see the emergence of unified approaches, such as those employed by Apple Inc. in their iPhone and OSX UIs (User Interfaces), which use significantly less user documentation and more on standard interface elements that rely on aesthetic integrity. Regardless of which specific usability heuristics are used for evaluating software it is important to consider the how the usable the interface is for learners.
One of the potential advantages in using educational software, particularly the use of multimedia, to assist learning in ways conventional media may prove less effective is its increased ability to cater for different learning styles. There have been many papers produced on the effectiveness of using multimedia as educational software (Mayer, 1997; Mayer & Anderson, 1992; Mayer & Moreno, 2002). The empirical evidence of studies suggests that learners who are provided with animation alongside narration would generally perform better than learners that were given only narration without animation (Mayer & Anderson, 1992). Similar studies (Mayer & Moreno, 2002; Mayer, 1997) discuss how different learning styles benefit from a combination of multiple media (multimedia) as opposed to single or seperated media. Similarly to Candy (2004), Montgomery (1995) also discusses the advantages of an awareness of pedagogical needs for educational multimedia to best address the differing learning styles in students. It is therefore evident that the content of educational software must be evaluated for its ability to utilise effective online pedagogies that address the needs of multiple learning styles.
As alluded to earlier, one of the most critical factors for evaluating software is how effectively if facilitates learning. Whilst it may be possible for experienced educators to predict how effectively learning might take place it is almost cetainly more benefical to perform evaluations in conjunction with users. This may may be infeasibly difficut with educational software that takes place over extensive time frames and facilitates complex learning, however, where possible, such evaluations would provide useful insight. For this reason it seems useful to combine a two‐pronged evaluation approach. The first stage of which would evalutate the educational software from the facilitator’s perspective, or that of the institution they represent and the second evaluation would be performed from the learners’ perspective. This may be done by diverging away from qualitative and quantitative methods to consider using methods like Patton’s (2002) utilisation focussed checklist approach whereby an evaluation facilitator works closely with intended users to determine the focus of evalation, which will be discussed in more detail later.
Evaluations Principles
Taking into consideration the wide scope of evaluative criteria and existing evaluation techniques it is important to decide upon the most appropriate framework for determining evaluations. This paper has outlined a number of broad criteria to consider before evaluating an educational software product and there are many others, equally important, not discussed in this paper. The benefits of empirical studies compared to practitioner facilitated evaluations using more simplified methods, such as checklists, have also been discussed. Ultimately, it falls upon the evaluator, be they professionals in the field or facilitators needing to make an informed decision, being informed and educated enough to understand the need and establishing the best criteria for determining the best method for evaluating a given software. As time and cost are often major driving factors in forming evaluations it is usually preferred to perform checklist based evaluations, given the lengthy process of establishing the empirical evidence required to facilitate other kinds of evaluative reports. Whatever evaluation method employed by the evaluator, based on the discussions above, there are principally fundamental criteria that will need to be assessed.
History
The software being evaluated may have a history of involvement with previous evaluations. It is important to discern the extent to which these past evaluations may be useful, if it all.
- Existing evaluations were conducted by non‐biased evaluators and if so favour the educational software
- The developers have a positive track‐history
- If the software is closed‐source it demonstrates active productions cycles (see Technology)
Technology
If the technology does not fit with the current, or planned future, technologies of an institution then it may prove to be non‐viable in the long term. If the technology is closed source and is not supported by the developers over active production cycles it may fail to reach long‐term viability.
- The software is extensible and may incorporate new knowledge
- The software demonstrates modern instructional design
- There is technical support available for the product
- The developer may provide updates to the software when required
- The software can be used on multiple platforms
- The intended users have the hardware necessary to run the software
- Ordinary learners with baseline IT literacy can operate the product
- The software allows users to print print‐friendly information
- The software is mostly error‐free and contains errors that are useful to diagnose problems
Content
The content in the educational software should be suited to the cognitive needs of the learner and should demonstrate appropriate use of the technologies used.
- The content appropriately suits the requirements of the learner
- The content caters for multiple learning styles by using a variety of media
- The content is designed in such a way that it uses the medium/media in a way that is suitable to the needs of the learner
- The quality of media, including images, text, videos and sound, is high quality
- The content’s vocabulary and concepts are suited to the needs of the learner
- The source of the content is known / credible / reliable
- The content is current and up‐to‐date
- The content facilitates activities that allow for the construction of new knowledge by the learner
- The content allows learners to progress at their own pace
- The content has varying degrees of difficulty
Pedagogical Approaches & Instructional Design
Blending heuristics with online pedagogical approaches marries well considering the content and interface are closely related in educational software.
- The software demonstrates effective use of online pedagogies
- The software encourages active participation and construction of knowledge rather than passive assimilation of knowledge
- The software encourages deep level learning, rather than surface level learning
- The structure of the content is clear and easy to understand
- The content is clear and easy to understand
- Text information is enhanced by meaningful graphics, animation and videos
- The use of sound is not critical for the use of the product
- The speed of downloading or loading information is acceptable for the learners platform
- The software has an integrated search and/or help feature
- The screen design and learning materials are displayed in a consistent way
Involving learners
Evaluating the learners experience and what they have learned is perhaps one of the most essential criteria for which to assess educational software. As individual skills and pre‐ condition of learners is likely to effect the successful outcomes of educational software evaluations should be performed in conjunction with the learner. Ideally such evaluation should be performed with learners in such a way that their satisfaction and learning after using the product can be measured. Using some criteria from Patton’s (2002) utilisation‐ focused checklist it is important to ensure the product is ready for evaluation and that the participatory learners demonstrate the following qualities; they are: interested; knowledgeable; open; credible; teachable; available throughout the evaluation process. Once learners have been recruited determine evaluation criteria, that can be assessed and correlated, from the learners point of view rather than simple the evaluators. Such considerations may include the following:
- Consider how the software might be improved
- Consider how the software might generate knowledge
- Consider the knowledge of the learner after using the educational software compared to previously using the software
- Consider learners concerns of barriers that may effect the use of the software
- Consider how effectively the learner uses the software
- Consider how the learner copes with underlying skills and underpinning knowledge required by the software
- Does the software consider individual learners differences
- Does the software allow learners to construct and participate in activities that are meaningful to their needs
Conclusion
There is debate concerning educators’, as opposed to professional evaluators’, ability to effectively evaluate software (Oliver, 2000, p. 23). Clearly, there are circumstances in which it becomes necessary for facilitators to evaluate educational software despite pre‐existing evaluations. Furthermore, it can be said that there is a need for ongoing evaluation of educational software that requires a continiual reflective practice, even after an initial evaluation has taken place; such would require that practitioners develop the necessary skills to perform evaluation when the need is recognised to do so. Therefore, despite criticism, it becomes necessary for facilitators of educational software to become skilled evaluators. It is clear that further research into evaluation of educational software is needed and although there are many existing tools available for facilitators to perform their own evaluations there is no single tool that has been proven to evaluate all software in all cirecumstances and choosing the correct tool becomes a necessary skill. Although there are tools that support evaluators with no previous experience there is little evidence that these tools produce anything of real value. A process of evauating the evaluation tools and making an educated decision on how best to reach a decision is required by all evaluators.
Whilst evaluators should become skilled at determining the most relevant evaluation principles for valuable evaluation to take place, the tools they use should more often consider the measurable outcomes of how participants use and learn with the software, rather than simply considering the evaluator’s expectations. This requires taking the context of the use of the software, as well as the pedagogical design of the software, into deeper consideration than is typically allowed by standard checklist and rating based models. Improved evaluation techniques and tools that cover the technological requirements, pedagogical approaches and learners’ responses to educational software will ensure the increased effectiveness of software in educational contexts.
References
Candy, P. C. (2004). Linking Thinking: Selfdirected learning in the digital age. Commonwealth Government of Aust.
Mayer, R. (1997). Multimedia learning: Are we asking the right questions? Educational Psychologist , 32 (1), 1‐19.
Mayer, R., & Anderson, R. (1992). The Instructive Animation: Helping students build connections between words and pictures in multimedia learning. Journal of Educational Psychology , 84 (4), 444‐452.
Mayer, R., & Moreno, R. (2002). Animation as an Aid to Multimedia Learning. Educational Psychology Review , 14 (1), 87‐99.
Mongomery, S. (1995). Addressing Diverse Learning Styles Through the Use of Multimedia. Frontiers in Education Conference. University of Michigan.
Morris, P., Ehrmann, S. C., Goldsmith, R., Howat, K., & Kumar, V. (. (1994). Valuable, viable software in education: Cases and analysis. New York: McGraw‐Hill.
Oliver, M. (2000). An introduction to the Evaluation of Learning Technology. Educational Technology & Society , 3 (4), 20‐30.
Oliver, M., & Conole, G. (1998). The Evaluation of Learning Technology – an overview. London: University of North London Press.
Patton, M. (2002). Evaluation Checklists Project. Retrieved May 19, 2009 from West Michigan University: http://www.wmich.edu/evalctr/checklists/ufe.pdf
Reeves, T. (1994). Evaluating what really matters in computer‐based education. In M. Wild, & D. (. Kirkpatrick, Computer education: New Perspectives (pp. 219‐46). Perth: MASTEC.
Tergan, S.‐O. (1998). Checklists for the Evaluation of Educational Software: Critical Review and Prospects. Innovations in Education and Teaching International , 35 (1), 9‐20.
Volery, T., & Lord, D. (2000). Critical success factors in Online Education. The International Journal of Educaitonal Management , 14 (5), 216‐23.
White, C. S. (1989). Directing the Software Evaluation Process: A Guide for Evaluators. Social Education , 53 (1), 67‐8.