October 2006 Volume 9 Number 4

by user








October 2006 Volume 9 Number 4
October 2006
Volume 9 Number 4
Educational Technology & Society
An International Journal
Aims and Scope
Educational Technology & Society is a quarterly journal published in January, April, July and October. Educational Technology &
Society seeks academic articles on the issues affecting the developers of educational systems and educators who implement and manage such
systems. The articles should discuss the perspectives of both communities and their relation to each other:
• Educators aim to use technology to enhance individual learning as well as to achieve widespread education and expect the technology to
blend with their individual approach to instruction. However, most educators are not fully aware of the benefits that may be obtained by
proactively harnessing the available technologies and how they might be able to influence further developments through systematic
feedback and suggestions.
• Educational system developers and artificial intelligence (AI) researchers are sometimes unaware of the needs and requirements of typical
teachers, with a possible exception of those in the computer science domain. In transferring the notion of a 'user' from the humancomputer interaction studies and assigning it to the 'student', the educator's role as the 'implementer/ manager/ user' of the technology has
been forgotten.
The aim of the journal is to help them better understand each other's role in the overall process of education and how they may support
each other. The articles should be original, unpublished, and not in consideration for publication elsewhere at the time of submission to
Educational Technology & Society and three months thereafter.
The scope of the journal is broad. Following list of topics is considered to be within the scope of the journal:
Architectures for Educational Technology Systems, Computer-Mediated Communication, Cooperative/ Collaborative Learning and
Environments, Cultural Issues in Educational System development, Didactic/ Pedagogical Issues and Teaching/Learning Strategies, Distance
Education/Learning, Distance Learning Systems, Distributed Learning Environments, Educational Multimedia, Evaluation, HumanComputer Interface (HCI) Issues, Hypermedia Systems/ Applications, Intelligent Learning/ Tutoring Environments, Interactive Learning
Environments, Learning by Doing, Methodologies for Development of Educational Technology Systems, Multimedia Systems/ Applications,
Network-Based Learning Environments, Online Education, Simulations for Learning, Web Based Instruction/ Training
Kinshuk, Athabasca University, Canada; Demetrios G Sampson, University of Piraeus & ITI-CERTH, Greece; Ashok Patel, CAL
Research & Software Engineering Centre, UK; Reinhard Oppermann, Fraunhofer Institut Angewandte Informationstechnik, Germany.
Associate editors
Alexandra I. Cristea, Technical University Eindhoven, The Netherlands; John Eklund, Access Australia Co-operative Multimedia
Centre, Australia; Vladimir A Fomichov, K. E. Tsiolkovsky Russian State Tech Univ, Russia; Olga S Fomichova, Studio "Culture,
Ecology, and Foreign Languages", Russia; Piet Kommers, University of Twente, The Netherlands; Chul-Hwan Lee, Inchon National
University of Education, Korea; Brent Muirhead, University of Phoenix Online, USA; Erkki Sutinen, University of Joensuu, Finland;
Vladimir Uskov, Bradley University, USA.
Advisory board
Ignacio Aedo, Universidad Carlos III de Madrid, Spain; Sherman Alpert, IBM T.J. Watson Research Center, USA; Alfred Bork,
University of California, Irvine, USA; Rosa Maria Bottino, Consiglio Nazionale delle Ricerche, Italy; Mark Bullen, University of
British Columbia, Canada; Tak-Wai Chan, National Central University, Taiwan; Nian-Shing Chen, National Sun Yat-sen University,
Taiwan; Darina Dicheva, Winston-Salem State University, USA; Brian Garner, Deakin University, Australia; Roger Hartley, Leeds
University, UK; Harald Haugen, Høgskolen Stord/Haugesund, Norway; J R Isaac, National Institute of Information Technology,
India; Paul Kirschner, Open University of the Netherlands, The Netherlands; William Klemm, Texas A&M University, USA; Rob
Koper, Open University of the Netherlands, The Netherlands; Ruddy Lelouche, Universite Laval, Canada; Rory McGreal, Athabasca
University, Canada; David Merrill, Brigham Young University - Hawaii, USA; Marcelo Milrad, Växjö University, Sweden; Riichiro
Mizoguchi, Osaka University, Japan; Hiroaki Ogata, Tokushima University, Japan; Toshio Okamoto, The University of ElectroCommunications, Japan; Gilly Salmon, University of Leicester, United Kingdom; Timothy K. Shih, Tamkang University, Taiwan;
Yoshiaki Shindo, Nippon Institute of Technology, Japan; Brian K. Smith, Pennsylvania State University, USA; J. Michael Spector,
Florida State University, USA.
Assistant Editors
Sheng-Wen Hsieh, National Sun Yat-sen University, Taiwan; Taiyu Lin, Massey University, New Zealand; Kathleen Luchini,
University of Michigan, USA; Dorota Mularczyk, Independent Researcher & Web Designer; Carmen Padrón Nápoles, Universidad
Carlos III de Madrid, Spain; Ali Fawaz Shareef, Massey University, New Zealand; Jarkko Suhonen, University of Joensuu, Finland.
Executive peer-reviewers
Subscription Prices and Ordering Information
Institutions: NZ$ 120 (~ US$ 75) per year (four issues) including postage and handling.
Individuals (no school or libraries): NZ$ 100 (~ US$ 50) per year (four issues) including postage and handling.
Single issues (individuals only): NZ$ 35 (~ US$ 18) including postage and handling.
Subscription orders should be sent to The International Forum of Educational Technology & Society (IFETS), c/o Prof. Kinshuk, School of
Computing & Information Systems, Athabasca University, 1 University Drive, Athabasca, Alberta T9S 3A3, Canada. Tel: +1 780 675 6812.
Fax: +1 780 675 6148. E-mail: [email protected]
Educational Technology & Society accepts advertisement of products and services of direct interest and usefulness to the readers of the
journal, those involved in education and educational technology. Contact the editors at [email protected]
Abstracting and Indexing
Educational Technology & Society is abstracted/indexed in Social Science Citation Index, Current Contents/Social & Behavioral Sciences,
ISI Alerting Services, Social Scisearch, ACM Guide to Computing Literature, Australian DEST Register of Refereed Journals, Computing
Reviews, DBLP, Educational Administration Abstracts, Educational Research Abstracts, Educational Technology Abstracts, Elsevier
Bibliographic Databases, ERIC, Inspec, Technical Education & Training Abstracts, and VOCED.
© International
and 1176-3647
of Educational
© International
Forum of
& Educational
Society (IFETS).
The authors
& Society
and (IFETS).
the forumThe
the copyright
forum jointly
of the
of the
to make
or hard copies
to make
of digital
part or or
of this
or all of
or this
work for
is granted
or without
is granted
that without
copies are
made orthat
or or
for profit and
or commercial
that copies advantage
bear the full
that copies
on thebear
full Copyrights
citation on the
for components
first page. Copyrights
of this work
for owned
by others
of this
than IFETS
must with
be honoured.
credit is permitted.
is permitted.
to republish,
To copy
to otherwise,
post on servers,
to republish,
or to redistribute
to post ontoservers,
lists, requires
or to redistribute
prior specific
to lists,
prior a
fee. Request
and/or afrom
fee. the
at [email protected]
from the editors at [email protected]
Guidelines for authors
Submissions are invited in the following categories:
• Peer reviewed publications: a) Full length articles (4000 - 7000 words), b) Short articles, Critiques and Case studies (up to 3000 words)
• Book reviews
• Software reviews
• Website reviews
All peer review publications will be refereed in double-blind review process by at least two international reviewers with expertise in the
relevant subject area. Book, Software and Website Reviews will not be reviewed, but the editors reserve the right to refuse or edit review.
• Each peer review submission should have at least following items: ƒ title (up to 10 words), ƒ complete communication details of ALL
authors , ƒ an informative abstract (75-200 words) presenting the main points of the paper and the author's conclusions, ƒ four - five
descriptive keywords, ƒ main body of paper (in 10 point font), ƒ conclusion, ƒ references.
• Submissions should be single spaced.
• Footnotes and endnotes are not accepted, all such information should be included in main text.
• The paragraphs should not be indented. There should be one line space between consecutive paragraphs.
• There should be single space between full stop of previous sentence and first word of next sentence in a paragraph.
• The keywords (just after the abstract) should be separated by comma, and each keyword phrase should have initial caps (for example,
Internet based system, Distance learning).
• Do not use 'underline' to highlight text. Use 'italic' instead.
Articles should be subdivided into unnumbered sections, using short, meaningful sub-headings. Please use only two level headings as far
as possible. Use 'Heading 1' and 'Heading 2' styles of your word processor's template to indicate them. If that is not possible, use 12 point
bold for first level headings and 10 point bold for second level heading. If you must use third level headings, use 10 point italic for this
purpose. There should be one blank line after each heading and two blank lines before each heading (except when two headings are
consecutive, there should be one blank like between them).
Tables should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be
provided for every table (below the table) and must be referenced in the text.
Figures should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be
provided for every figure (below the figure) and must be referenced in the text. The figures must NOT be larger than 500 pixels in width.
Please also provide all figures separately (besides embedding them in the text).
• All references should be listed in alphabetical order at the end of the article under the heading 'References'.
• All references must be cited in the article using "authors (year)" style e.g. Merrill & Twitchell (1994)
or "(authors1, year1; authors2,
year2)" style e.g. (Merrill, 1999; Kommers et al., 1997).
Do not use numbering style to cite the reference in the text e.g. "this was done in this way and was found successful [23]."
It is important to provide complete information in references. Please follow the patterns below:
Journal article
Laszlo, A. & Castro, K. (1995). Technology and values: Interactive learning environments for future generations. Educational Technology,
35 (2), 7-13.
Newspaper article
Blunkett, D. (1998). Cash for Competence. Times Educational Supplement, July 24, 1998, 15.
Clark, E. (1999). There'll never be enough bandwidth. Personal Computer World, July 26, 1999, retrieved July 7, 2004, from
Book (authored or edited)
Brown, S. & McIntyre, D. (1993). Making sense of Teaching, Buckingham: Open University.
Chapter in book/proceedings
Malone, T. W. (1984). Toward a theory of intrinsically motivating instruction. In Walker, D. F. & Hess, R. D. (Eds.), Instructional
software: principles and perspectives for design and use, California: Wadsworth Publishing Company, 68-95.
Internet reference
Fulton, J. C. (1996). Writing assignment as windows, not walls: enlivening unboundedness through boundaries, retrieved July 7, 2004,
from http://leahi.kcc.hawaii.edu/org/tcc-conf96/fulton.html.
Submission procedure
Authors, submitting articles for a particular special issue, should send their submissions directly to the appropriate Guest Editor. Guest
Editors will advise the authors regarding submission procedure for the final version.
All submissions should be in electronic form. The editors will acknowledge the receipt of submission as soon as possible.
The preferred formats for submission are Word document and RTF, but editors will try their best for other formats too. For figures, GIF
and JPEG (JPG) are the preferred formats. Authors must supply separate figures in one of these formats besides embedding in text.
Please provide following details with each submission: ƒ Author(s) full name(s) including title(s), ƒ Name of corresponding author, ƒ Job
title(s), ƒ Organisation(s), ƒ Full contact details of ALL authors including email address, postal address, telephone and fax numbers.
The submissions should be uploaded at http://www.ifets.info/ets_journal/upload.php. In case of difficulties, they can also be sent via
email to (Subject: Submission for Educational Technology & Society journal): [email protected] In the email, please state clearly that the
manuscript is original material that has not been published, and is not being considered for publication elsewhere.
© International
and 1176-3647
of Educational
© International
Forum of
& Educational
Society (IFETS).
The authors
& Society
and (IFETS).
the forumThe
the copyright
forum jointly
of the
of the
to make
or hard copies
to make
of digital
part or or
of this
or all of
or this
work for
is granted
or without
is granted
that without
copies are
made orthat
or or
for profit and
or commercial
that copies advantage
bear the full
that copies
on thebear
full Copyrights
citation on the
for components
first page. Copyrights
of this work
for owned
by others
of this
than IFETS
must with
be honoured.
credit is permitted.
is permitted.
to republish,
To copy
to otherwise,
post on servers,
to republish,
or to redistribute
to post ontoservers,
lists, requires
or to redistribute
prior specific
to lists,
prior a
fee. Request
and/or afrom
fee. the
at [email protected]
from the editors at [email protected]
Journal of Educational Technology & Society
Volume 9 Number 4 2006
Table of contents
Special issue articles
Theme: E-learning and Human-Computer Interaction: Exploring Design Synergies for
more Effective Learning Experiences
Editorial: E-learning and Human-Computer Interaction: Exploring Design Synergies for more Effective
Learning Experiences
Alan Dix, Teresa Roselli and Erkki Sutinen
Automatically Producing Accessible Learning Objects
Angelo Di Iorio, Antonio Angelo Feliziani, Silvia Mirri, Paola Salomoni and Fabio Vitali
A Boosting Approach to eContent Development for Learners with Special Needs
Silvia Gabrielli, Valeria Mirabella, Stephen Kimani and Tiziana Catarci
Supporting Students with a Personal Advisor
Berardina De Carolis, Sebastiano Pizzutilo, Giovanni Cozzolongo, Pawel Drozda and Francesca
eLSE Methodology: a Systematic Approach to the e-Learning Systems Evaluation
Rosa Lanzilotti, Carmelo Ardito, Maria F. Costabile and Antonella De Angeli
Cooperative Project-based Learning in a Web-based Software Engineering Course
Nicola Piccinini and Giuseppe Scollo
Full length articles
Discussion Tool Effects on Collaborative Learning and Social Network Structure
Astrid Tomsic and Daniel D. Suthers
Institutional Potential for Online Learning: A Hong Kong Case Study
Sue L. Donoghue
Reuse- and Aspect-Oriented Courseware Development
Khaldoun Ateyeh and Peter C. Lockemann
Using Online Education Technologies to Support Studio Instruction
Diane M. Bender and Jon D. Vredevoogd
The Effect of Video Presentation in a CBT Environment
Ali Reza Montazemi
Web-based Tools for Designing and Developing Teaching Materials for Integration of Information
Technology into Instruction
Kuo-En Chang, Yao-Ting Sung and Huei-Tse Hou
Enhancing user support in open problem solving environments through Bayesian Network inference
Nikolaos Tselios, Adrian Stoica, Manolis Maragoudakis, Nikolaos Avouris and Vassilis Komis
All Aboard - Destination Unknown: A Sociological Discussion of Online Learning
Andra K. Goldberg and Frances Julia Riemer
A Systemic Plan of Technology Integration
Pi-Sui Hsu and Priya Sharma
Surviving the shipwreck: what makes online students stay online and learn?
Johannes C. Cronjé, Debbie E. Adendorff, Salome M. Meyer and Linda van Ryneveld
Innovative Web-based Professional Development for Teachers of At-Risk Preschool Children
Mable B. Kinzie, Stephen D. Whitaker, Kathy Neesen, Michael Kelley, Michael Matera and Robert
C. Pianta
ISSN 1436-4522
© International
and 1176-3647
Forum (print).
of Educational
© International
The authors
& Society
and the
forum The
forum jointly
of theretain
or hard copies
to make
of part
or all
work for
of part
or allorofclassroom
this work use
for is
fee provided
use is granted
that copies
made or that
for profit
not made
or commercial
or distributed
for profitand
or that
copies bear
the fulland
that copies
on the bear
first page.
the full
citation onfor
first page. Copyrights
of this workfor
by others
of than
this work
must by
IFETS mustwith
be honoured.
credit is permitted.
To with
copy credit
is permitted.
to republish,
To copy
to post
on servers,
to republish,
or to redistribute
to post on
to lists,
or to prior
lists, requires
fee. Request
a fee.
editors permissions
at [email protected]
from the editors at [email protected]
How Does Educational Technology Benefit Humanity? Five Years of Evidence
Pedro Hernández-Ramos
Interoperability for Individual Learner Centred Accessibility for Web-based Educational Systems
Liddy Nevile and Jutta Treviranus
Reengineering Framework for Systems in Education
Christophe Choquet and Alain Corbière
Book review(s)
Web-Based Intelligent e-Learning Systems: Technologies and Applications
Reviewer: Charalampos Karagiannidis
Visual Media and the Humanities: A Pedagogy of Representation
Reviewer: Lauren Glenn
© International
and 1176-3647
of Educational
© International
Forum of
& Educational
Society (IFETS).
The authors
& Society
and (IFETS).
the forumThe
the copyright
forum jointly
of the
of the
to make
or hard copies
to make
of digital
part or or
of this
or all of
or this
work for
is granted
or without
is granted
that without
copies are
made orthat
or or
for profit and
or commercial
that copies advantage
bear the full
that copies
on thebear
full Copyrights
citation on the
for components
first page. Copyrights
of this work
for owned
by others
of this
than IFETS
must with
be honoured.
credit is permitted.
is permitted.
to republish,
To copy
to otherwise,
post on servers,
to republish,
or to redistribute
to post ontoservers,
lists, requires
or to redistribute
prior specific
to lists,
prior a
fee. Request
and/or afrom
fee. the
at [email protected]
from the editors at [email protected]
Dix, A., Roselli, T., & Sutinen, E. (2006). E-learning and Human-Computer Interaction: Exploring Design Synergies for
more Effective Learning Experiences. Educational Technology & Society, 9 (4), 1-2.
E-learning and Human-Computer Interaction: Exploring Design Synergies
for more Effective Learning Experiences
Alan Dix
Computing Department, InfoLab21, South Drive, Lancaster University, LA1 4WA, Lancaster, UK
[email protected]
Teresa Roselli
Department of Computer Science, University of Bari, Via E. Orabona 4, 70125 Bari, Italy
[email protected]
Erkki Sutinen
Department of Computer Science, University of Joensuu, 80110 Joensuu, Finland
[email protected]
The current trend in developing e-learning systems is largely empirical anecdotal, while consolidated, evidencebased models ensuring systematic and pedagogically sound learning experiences are still lacking. Up to now, the
e-learning community has mainly focused on investigating the technical qualities of such systems, but has tended
rather to neglect their didactic effectiveness and usability. Thus the e-learning community needs to devise and
discuss new criteria for the design of more usable and innovative systems supporting creative learning, based on
strategies which can, on the one hand, guide the learner to make the most effective use of the didactic content,
and, on the other hand, refrain from being too intrusive in scaffolding the learning process.
A major challenge currently faced by e-learning systems’ designers is the development of improved tools better
able to engage novice learners and sustain their online learning activities any time and anywhere. HumanComputer Interaction (HCI) theories and methodologies can support the design of appropriate e-learning settings
responding to the complex and rapidly changing requirements of both the academic and business contexts of our
society. Basically, e-learning applications should become smart enough to adapt themselves to the students’
learning styles and to assure high standards of accessibility and usability, in order to make learners’ interaction
with the systems as natural and intuitive as possible.
Stronger synergies should be established between the design of e-learning experiences and the analysis of
learners’ preferred interactions with e-learning environments. To reach this objective an evolving learner-centred
design perspective should be adopted, taking into account also the typical learning styles shared within the
different cultural contexts. Future studies based on these assumptions could provide valuable results and inspire
interesting lines of thought for the intersection of HCI and e-learning.
To be successful, the synergy approach requires that the researchers design new tools for the users whose
feedback from concrete user scenarios is analyzed from the very beginning of and throughout the design process.
In this regard, special needs education provides researchers with a particularly beneficial context since diverse
learners force the designers to really listen to the users feedback to be able to create functional e-learning tools.
This special issue features the best papers presented at the workshop e-learning and Human-Computer
Interaction: Exploring Design Synergies for more Effective Learning Experiences part of the International
Conference on Human-Computer Interaction (INTERACT 2005) held on Rome in September 2005.
The workshop aimed at stimulating discussions about the latest advances in e-learning, based on application of
HCI approaches to distant education.
The overall acceptance rate of the two blind review processes was 30%. The five published papers cover several
key themes in the e-learning and HCI research area.
There are two papers in the area of accessibility of didactic resources. Di Iorio, Feliziani, Mirri, Salomoni and
Vitali present a learning object creation and management process based on common personal productivity tools,
which guarantees both content accessibility as well as universality and offers a simple and friendly interface to
authors. Gabrielli, Mirabella, Kimani and Catarci propose a design method for increasing the quality of elearning materials for learners with special needs and an authoring environment to support authors in their
development of didactic material matching those needs.
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
Adaptivity is one of the most important topics of e-learning research. The aim is to supply more and more
customized learning paths in order to meet the learners' needs and to achieve more effective learning. De Carolis,
Pizzutilo, Cozzolongo, Drozda and Muci present an architecture of an Embodied Conversational Agent (ECA)
designed to assist students by providing personalized suggestions related both to the fruition of didactic material
and to general orientation for student’s daily life.
Learner needs are the starting point of the Lanzilotti, Ardito, Costabile and De Angeli paper. The authors
highlight the lack of high-quality systems tailored to the needs of individual users and groups. They refine the
concept of quality of e-learning systems and propose a framework TICS (Technology, Interaction, Content,
Services), which focuses on the user-system interaction as one of the most important aspects to be considered
when designing or evaluating an e-learning system. Moreover they propose an evaluation methodology called
eLSE (e-Learning Systematic Evaluation).
The last paper Piccinni and Scollo presents and analyzes a case study in software engineering education,
spanning over a seven-year evolution, characterized by a blend of educational techniques: traditional classroom
lectures, textbook and lecture notes, as well as a web-based cooperation platform, supporting interaction and
self-organization of laboratory projects.
The special issue and workshop have played a major, "prime mover" role in fostering a greater sense among
HCI-oriented researchers of e-learning and tracking important directions in e-learning research for the coming
years. As Guest Editors we hope that this special issue will provide an overview of studies highlighting the
multiple relationships between technological and educational approaches to the design of e-learning
Special thanks to our reviewers for the relevant and detailed comments provided to the authors of all papers
submitted. We would like to thank all conference chairs, the reviewers, all the members of the workshop
secretariat and all those who helped to make the workshop and this resultant special issue a success.
Di Iorio, A., Feliziani, A. A., Mirri, S., Salomoni, P., & Vitali, F. (2006). Automatically Producing Accessible Learning
Objects. Educational Technology & Society, 9 (4), 3-16.
Automatically Producing Accessible Learning Objects
Angelo Di Iorio, Antonio Angelo Feliziani, Silvia Mirri, Paola Salomoni and Fabio
Department of Computer Science, Università di Bologna, Via Mura Anteo Zamboni 7, 40126 Bologna BO, Italy
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
The “Anywhere, Anytime, Anyway” slogan is frequently associated to e-learning with the aim to emphasize
the wide access offered by on-line education. Otherwise, learning materials are currently created to be used
with a specific technology or configuration, leaving out from the virtual classroom students who have
limited access capabilities and, in particular, students with special needs. On the contrary, accessibility of
learning materials is a key issue to ensure a whole inclusion of people with disability in instructional
process, and, consequently, to prevent risks of “digital exclusion”. The foremost explanation for the
creation of not inclusive materials is the assumption that creating accessible and universal Learning Objects
(LO) is a dismaying and lengthy task, partially supported by complicated ad-hoc tools. New simple
mechanisms that drive authors in creating accessible LO are needed to enlarge LO audience; they should
contemporary support students’ needs (while reading) and reduce authors’ efforts (while writing). In this
paper we present a LO creation and management process, based on common personal productivity tools,
which guarantees both content accessibility as well as universality and offers a simple and friendly interface
to authors.
Accessibility, Authoring, Standards compliance, Content creation
Tools ease of use for e-learning contents creation has been always paired with the continuously raising
expectations about the quality of the final result. Although nowadays many sophisticated tools to create elearning contents can be track down, authors’ job is increasingly complicated by new quality requirements. Two
main principles have contemporaneously to be addressed, which conjure to make e-learning authors’ job once
again difficult and full of technical issues to be tackled:
1. Accessibility: producing fully barrier-free learning contents is one of the key issues to meet the goal of an
inclusive “knowledge society”. Accessibility, with full and authoritative backing from standard bodies such
as the W3C and national governments through laws and norms, further raise the bar for web authors.
2. Portability: conformance to e-learning standards enhances contents portability and is recognized as one of
the fundamental aspects to preserve contents value. Specific e-learning standards guarantee that compliant
contents could be used in several learning systems.
Existing tools, because of several different reasons, still fall short in providing a full authoring environment, both
capable of dealing with all the appropriate technical aspects and, at the same time, as easy to use as normal
desktop applications such as word processors or presentation applications. As a consequence of such a lack, an
additional professional figure is required to step in between the educators and the final e-learning system, which
assists the authors in creating final learning objects t satisfying all modern technical requirements.
How should the ideal authoring tool be, in order to fully assist educators in creating accessible content for
modern Learning Management Systems? What kind of support could we build for users that do not really want
to deal with all the technical details of current e-learning technologies? We have identified at least seven
1. Ease of use: the tool should be at least as easy to use as existing word processors and presentation tools. In
particular, it should behave as similar as possible to their work so as to lower its learning curve.
2. Ease of re-use: the tool should help and assist authors in reusing and converting documents and materials
they have already prepared in past times by using any of a large number of desktop tools, to the new elearning platforms.
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain
the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work
owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists,
requires prior specific permission and/or a fee. Request permissions from the editors at [email protected]
Ease of editing and updating: whenever the authoring process starts with a conversion step, it oftentimes
becomes a one-way process that cannot be repeated at will, but it costs money and time and requires
expertise to be done several times. The conversion process raises difficulties in regularly editing and
updating the published material, and constrains users to directly edit the converted material rather than the
source documents which were originally used before the conversion.
Standards support: the tool should generate learning objects that can be read by a large variety of
commercial and open source e-learning platforms. It practically means that such a tool should produce
learning objects according to some major e-learning standards, defined by different standardization
Visual Homogeneity: the tool should produce content that easily undergoes platform- and site-specific styles
and look&feel, by fully and easily adapting any content to the templating and styling locally mechanism
Universality: the tool should generate content which under appropriate conditions (e.g., the choice of
different and independent templates), can be fully and at best quality displayed on a wide variety of
applications, including non-dominant versions of browsers and operating systems, older versions of
browsers and operating systems, new and emerging hardware devices (such as PDAs, cellular phones,
interactive TV sets, etc.).
Accessibility: the tool should create fully accessible content according to international standards and
national laws. It should fully assist the author, silently generating the accessible structures that can be
automatically deduced from existing content, and prompting the author in providing anything cannot be
automatically generated (e.g., the textual description of images).
Accessibility is of paramount importance, especially when dealing with e-learning applications: learners with
disabilities can benefit a great advantage from e-learning, not simply because it allows distant and flexible
learning activities, but mainly because it could support impaired students in overcoming barriers to resources
which would otherwise be hard to access (Salomoni et al., 2004). In particular, learners with difficulties in
accessing to printed materials (i.e. people with visual impairments) can take a great advantage of the integration
of digital materials into teaching practice. Across the world, laws are in place or under definition to ensure that
interactive/on-line services and, sometimes, specifically e-learning, are made accessible to citizen with
disabilities (Italian Parliament, 2004; U.S. Rehabilitation Act Amendments of 1998, 1998). Two main strategies,
combined together, are used in order to meet accessible requirements:
¾ Appropriate use of W3C standard compliant technologies, in order to enhance interoperability and
portability (World Wide Web Consortium, 2000; World Wide Web Consortium, 1999b; World Wide Web
Consortium, 2001b).
¾ Use of alternative versions of inaccessible content, in order to accommodate different users, e.g., students
using limited or obsolete devices and/or students with sensorial impairments (IMS Global Learning
Consortium, 2002b; World Wide Web Consortium, 1999a; World Wide Web Consortium, 1999c).
In this paper we present a methodology and some tools for the creation and management of accessible and
universal learning objects (LO) which considerably improves the process across the seven mentioned
dimensions. Our methodology involves different tools, from widespread text editors, which are used by authors
to easily produce content, till an ad-hoc application, called ISA-BeL, designed and implemented to support
automatic production of standard compliant e-learning materials. The process has been widely used to publish
about 40 learning modules which are currently in use by our University for several e-learning activities in a
number of subjects. Accessibility of the whole process and all the produced LOs have been verified on the field.
The remainder of the paper is organized as follows. Section 2 provides some background information about
authoring e-learning content platform. Section 3 introduces the creation and management process in e-learning
contexts using ISA-BeL, while section 4 fully details a use-case in producing accessible and standard e-learning
materials by using ISA-BeL. Finally, Section 5 provides some conclusions and suggestions for future works.
Systems and standards
Many recent efforts in the field of knowledge management address e-learning, i.e., distance learning based on
the use of personal computers. Systems providing e-learning services can be divided in two main categories:
LMSs (Learning Management Systems), which are web-based platforms by actually providing content to the
users and LCMSs (Learning Content Management System), the authoring environments used to create learning
objects. The main features of an LCMS are related to the content management, from the production to the
storage including reusability and distribution of content. On the other hand an LMS manages the administrative
functions (i.e. users’ activities logging and curriculum management), the distribution of contents to learners, and
the tracking of the learners’ experiences and assessments.
A relevant role is played by existing e-learning standards, in ensuring interoperability and reuse of didactical
materials. Main interoperability specifications have been developed by IEEE (Institute of Electrical and
Electronics Engineers), with a specific working group, the Learning Technology Standards Committee, which is
working on e-learning standardization (IEEE LTSC WG12, 2006) and IMS (Instructional Management System)
Global Learning Consortium (IMS Global Learning Consortium, 2006). A relevant role is also played by
Advanced Distributed Learning (ADL) initiative (Advanced Distributed Learning, 2006), which has developed a
de-facto standard called SCORM (Shareable Content Object Reference Model) (Advanced Distributed Learning,
2004c), based on some specifications previously defined by IEEE-LTSC and IMS. It is structured in modules
defining sub specifications. In this context let us introduce two SCORM modules:
1. The Content Aggregation Model (CAM) defining the structure of learning material, and describes it with
metadata (Advanced Distributed Learning, 2004a).
2. The Run-time Environment (RTE), a JavaScript Application Programming Interface (API) able to deliver
real time information about user actions (Advanced Distributed Learning, 2004b) to a LMS or a LCMS.
Learning Object Production
An interesting field in e-learning research is the simplification and automation of the learning objects production:
some works can be found in literature which are devoted to automatically produce LOs starting from available
didactical materials (created in many different formats), or by directly generating SCORM-compliant contents.
Several projects and products provide authoring tools for e-learning materials (4system, 2006; Macromedia,
2006; ReadyGo Inc., 2006; SumTotal Systems Inc., 2006). They commonly imply high-level technical skills in
order to be appropriately used by authors, which have typically to complete a preliminary training phase to
effectively use them (as they can do with well-known “productivity tools”) These products generally provide
authors with different interfaces and functionalities so that they can create e-learning materials, manage
resources, aggregate contents, add meta-data, and so on. Some of these tools are free, while some others are
commercial products. In many cases, the use of these new tools cannot be fully appreciated by authors of
e-learning materials, who may prefer to rely on well-known productivity tools, which could allow them some
savings in time and money. Products and platforms are designed by moving onto in this direction and they
generate e-learning course materials starting from well-known productivity tools, such as Microsoft Word
(Horizon Wimba, 2006; Serco, 2006). The main advantage in exploiting such products is that no learning and
training phases are needed. On one hand these tools provide a too rigid structure in drawing up created contents,
on the other hand they keep authors’ stylistic choices, instead of maintaining only designers’ ones, invalidating
accessibility and usability principles. In addition, they do not support useful mechanisms to manage DTD
configurations or allow the insertion of new structure style. One of these products (Horizon Wimba, 2006)
provides a partial support to accessibility of created contents, but, in some cases, generated LOs are not
compliant to international guidelines and laws, denying actual benefits to learners with disabilities.
Another interesting issue in e-learning course contents creation is the design o Generative Learning Objects
(GLOs) (Boyle et al., 2004; Bradley et al., 2004; Morales, 2005). The underlying idea is based on the division
into two different parts the LOs creation. The first one consists of building a Learning Object Template (LOT),
while the second one is devoted to adding the template a subject specific content. The LOT encloses the deep
general structure of the e-learning course. Once a template has been created, authors and/or tutors can add
different subject specific contents, i.e. the surface structure, so as to produce Learning Objects which fit the
specific fields of the discipline.
Finally, let us consider a new way to think intermediate data format that is promising to have an interesting
impact on LO production. Microformats (Microformats.org, 2006). Microformats are a set of simple open data
format standard which are developed and implemented for more/better structured web microcontent publishing.
Some already developed microformats specify for example formats for calendar and events data. Many other
microformats are currently in a developing phase. In (Downes, 2006) authors propose Microformats use in elearning, by conceiving it as a network phenomenon, so as to facilitate a personal e-learning centre design.
Accessible and personalized Learning Objects
Other academic projects and works are devoted to produce accessible e-learning contents and/or to personalize
them. One of these proposes the design of a prototype (Gabrielli et al., 2005; Mirabella et al., 2004) which drives
authors in creating accessible didactical materials. The authoring interface of this prototype is developed in Java
and its main aim is to support authors’ job with suggestions and examples. They are provided in order to show
and explain how accessible contents have to be produced by providing appropriate and equivalent alternatives to
learners and according to W3C Web Content Accessibility Guidelines (World Wide Web Consortium, 1999c). In
(Kelly et al., 2005), the authors propose a holistic approach for addressing e-learning accessibility which takes
into account e-learning usability of pedagogic issues, student learning styles and needs and some other
significant constrains which usually accessibility guidelines do not cover. The main aim of this work is showing
how the whole quality of e-learning contents can be assured by using this approach.
Open Adaptive Learning Environment (OPAL) 0 is an e-learning service, which considers the personalization of
e-learning course contents and is based on the delivery of personalized contents. The personalization is done on
the basis of students’ cognitive and presentation learning preferences by using aggregation models based on
SCORM (Advanced Distributed Learning, 2004c). Another interesting academic project is ELENA (Dolog et al.,
2003; Dolog et al., 2004), which supports personalized access to distributed learning repositories. The approach
to customization employed in this project takes advantage of semantic Web technologies and metadata
description standards, such as LOM (IEEE LTSC WG12, 2002) and IMS AccessForAll Meta-data (IMS Global
Learning Consortium, 2002a). In addition, it adapts and customizes access, delivery and consuming of learning
services and LOs on the basis of rule-based matching of contents and learners descriptions.
No one of the above described projects and products combines accessibility issues with methodologies to
automatically produce LOs, like the ones described in Section 2.2.
From content creation to e-learning delivery
Context: the A3 Project
Teaching basic computer knowledge is becoming a matter of big interest in a lot of fields, particularly in
Universities, where in every degree course is necessary to certify a minimal skill in computer knowledge. For
this reason, the Department of Computer Science at the University of Bologna has developed a project (called
A3, Accessible Learning Environment (University of Bologna, 2004), “Ambiente Accessibile d’Apprendimento”
in Italian language) for the creation and fruition of contents taking in a particular account the training structure
uniformity, a low management cost, and a little effort in resources and time consumption for the process startup
Main requirements related to LOs used in A³ are due to:
¾ Accessibility and Web standard compliance. The project was developed and carried out inside an Italian
University and it respects the Italian Law on Information Accessibility, the so called “Stanca Act” (Italian
Parliament, 2004). In particular, as requested by the Italian Law about Web accessibility: (i) all templates
completely separate presentation from content and structure, by producing XHTML Strict 1.0 and CSS 2.0
code, (ii) layout and font dimensions are relative, and (iii) foreground and background colors are compliant
to W3C requirements.
¾ Portability of LOs and e-learning standard compliance. Learning objects produced in A³ are package
SCORM 1.2 RTE compliant (Advanced Distributed Learning, 2004b), so that contents can be imported in
every LMS (Learning Management System) SCORM compliant.
The Italian Law about accessibility (Italian Parliament, 2004), analogously to the WCAG 1.0 (World Wide Web
Consortium, 1999c) requires contents and services to be provided independently from the use of scripts and
programming objects. This means that a complete use of SCORM is not possible in respect of the current set of
Italian (or W3C) requirements, and specifically, only the SCORM-CAM (Advanced Distributed Learning,
2004a) standard has a complete compatibility with accessibility specification. Otherwise the SCORM-RTE
(Advanced Distributed Learning, 2004b) module, devoted to Real Time activities, is completely based on the use
of Javascript and it is not compatible with the (current) accessibility requirements. Let us note that both the
Italian Law (near to adopt specific requirements on e-learning) and the WCAG (near to version 2.0) will be
updated in the next future to allow Javascript and it will permit a full use of SCORM (World Wide Web
Consortium, 2006).
Producing usable, accessible and universal content is a complex and time-consuming task that, in most cases,
still requires expertise to the authors. All constrains derived from A³ requirements presented in the previous
section exacerbate this condition. The basic idea to overcome authors difficulties, which is proposed by A³ was
based on developing a chain of tools, called ISA-BeL (Di Iorio et al., 2005), which allows users to easily create
accessible and portable learning objects (LOs) by mainly using a word processor (or, more generally, a
productivity tool). Thus, ISA-BeL does not require authors to know technical details of e-learning technologies
and languages, to master complex LCMSs or to use potentially complicated ad-hoc tools. ISA-BeL author writes
a document of raw content and he/she indicates the role of each fragment (by using styles according to a set of
given guidelines), and the conversion engine transforms each fragment in a proper element of the final learning
object. Actually, information about the whole output structure as well as some metadata are required, but
inserting such data is really simple and fast, as we will discuss later.
A three step workflow can be now defined, from knowledge creation to e-learning provision:
1. Authoring (Content creation), done by teachers using a word processor (or alternatively a different personal
productivity tool, such as a presentation application).The output of this phase is a set of documents in
common formats like rtf, doc, ppt, sxw, etc. containing accessible learning materials.
2. Producing (Content transformation), i.e. the process creating a LO from a set of documents produced during
phase 1. The output of this phase is a LO which has to maintain accessibility features embedded in original
3. Delivery (Content distribution), the real e-learning service, provided by a LMS which guests the LO which
is produced in phase 2. The LMS has to guarantee accessibility of content and service provisioning.
The whole process is depicted in Figure 1, which also shows the content production step performed by ISA-BeL
and described in the Section 3.4 with more details.
The output of ISA-BeL is not a simple set of common HTML pages, but a group of several alternative contents,
which are used to enhance portability and accessibility, such as:
1. Learning Objects, compliant to SCORM-CAM 1.2 or 1.3 (Advanced Distributed Learning, 2004a); this
version can be uploaded on a SCORM compliant LMS and provided to students throughout the Web.
2. Web-based materials, to be distributed on-line on the Web or on an optical storage device (CD or DVD) or
to be delivered through the Internet.
3. Printed materials in PDF format, obtained from the original contents by using XSL Formatting Objects
(World Wide Web Consortium, 2001a).
Figure 1. Authoring-management-provision of accessible e-learning by means of ISA-BeL
The ISA-BeL support for content creation consists of providing users an alternative and simplified way to
express all the data and content useful to create learning objects. Focusing on the resources packaging (even if
the whole project we are working on currently supports tracking, run-time monitoring and assessments
management too), a learning object can be defined as a set of structured resources supplied with a (SCORM)
manifest that describes them.
From this definition, we figured out alternative mechanisms to (i) indicate which pages compose the learning
object and which content elements compose each page, (ii) verify these content elements express all the required
information and (iii) add metadata associated to the learning object. We have provided many facilities to sustain
contents creation also by authors which are non-expert in technologies and standards concerning accessibility
and usability issues. In particular for accessibility, we supply authors with MSWord styles which are accessible
with a custom toolbar. Each style is meant for a specific role in the final document; the role could regard a
particular required accessibility constraint or a predefined semantic position in the text. The correct use of styles
according to their implicit semantics is managed by some macros which alert the author about lacks of
information. Moreover, the author can use a Word embedded form to fill in specific metadata.
An ISA-BeL author, in fact, writes his/her contents by using a word processor, and he/she assisted by three
different tools, specifically designed to cover these three aspects:
¾ An authoring toolbar which offers a fast access to him/her main activities such as defining presentational
aspects and structural elements, inserting accessibility related information and so on. A screenshot of the
ISA-BeL toolbar is depicted in the following Figure 2.
¾ A verification toolbar which runs automatic controls over the respect of accessibility and universality
constrains. For example, some exploited controls are: have any image an alternative description? Has any
table a summary? Are titles used according to the correct hierarchical structure? Other manual controls are
made by the post-production team (about foreign language words, acronyms, abbreviations, and so on); if
the LO does not respect such constrains then authors have to correct and control contents again, before
starting the production phase.
¾ A set of forms to collect SCORM Metadata. As expected, a key role is played by the SCORM metadata:
while some of them can be derived by the system (last-saved date, file size, language, version number and
so on), other information have necessarily to be provided by the authors.
A step-by-step description of author activities, from creation to production is detailed in the following Section 4.
Figure 2. The ISA-BeL authoring toolbar (in MS Word)
The production process is performed by an ad-hoc application, ISA-BeL which is composed of two modules:
¾ ISA (Immediate Site Activator): a conversion tool which actually transforms document from the word
processor format into an intermediate XML representation, enriched by all the necessary metadata. An
introduction about ISA architecture and functionalities can be found in (Vitali, 2003).
¾ BeL (Backed e-Learning): a stand-alone application which gathers all the information stored in the
intermediate XMLs, creates the SCORM structures (in particular the tracking scripts and the manifest file)
and merges the content into a single .ZIP file, by processing the output of ISA. BeL also integrates into the
LO a (multimedia) recorded accessible video lecture, which is automatically transcoded through a different
line of the LO production. More information about this transcoding process can be found in (Salomoni et al.,
The production process is based on a set of templates and configuration files which are used to define structural
aspects as well as layout and graphical aspects of the automatically produced LO.
The system has been used for the above mentioned A3 project. Contents which were produced by the automatic
process are loaded on the e-learning platform by means of standard (for SCORM compliant contents) import
procedures. Our choice has been made between the large number of open-source platforms and after a deep
testing period we have adopt ATutor platform (Adaptive Technology Resource Centre, University of Toronto,
2006). The choice has been driven by the built-in usability support even though it has been necessary modifying
the platform to adapt it to the Italian Law on accessibility (Italian Parliament, 2004) which is narrower than the
Canadian one. Accessibility of the adopted LMS partially guarantees that produced contents maintain
accessibility features and ensures accessibility and portability of services (chat, forum, news). Some
modifications were needed to completely meet the “Stanca Act” requirements, and particularly the constraint to
use Strict (X)HTML code.
The main delivery of A³ was through the LCMS, but ISA-BeL produces also a printable version and a fully
HTML one (which could be distributed both on the Internet and on a CD/DVD).
Simply creating an accessible LO
This section offers a detailed description of creation and production phases, by presenting a step-by-step
description of author activities. Let us consider a specific scenario inside the A³ Project, in which the author uses
MS Word to create contents that are written in Italian to introduce some basic IT skills. In particular, our author
has to structure a lecture on “Digital Documents” through 6 sub-modules about different aspects of the main
topic. Each sub-module is created by generating a .doc file which includes its contents. In order to complete the
LO production, the author has to:
¾ Prepare the 6 files, by creating the sub-modules content with MS Word. The module is completed with some
other .doc files containing scope, goals and references.
¾ Verify the structure of the produced contents, file by file.
¾ Add the SCORM Metadata.
¾ Obtain the .htm version of each file, by using the “Save as” option of the “File” menu which is provided by
MS Word.
¾ Use the ISA online tool to produce the set of XML intermediate files.
¾ Use the BeL online tool to compose and produce an accessible LO.
This sequence of activities is fully described in the following.
Creating contents
Let us suppose to work on the sub-module about “Digital Images” which starts with a general description of
vector and bitmap formats. At the beginning of his/her work our author writes the title of the first page of the
section (in Italian “Bitmap e Vettoriale”) and uses the “Title 1” style to format it. The page is based on a short
explanation text describing this specific topic and the author uses the authoring toolbar to add a suitable format
to it. In order to complete his/her exposition, the author adds a figure showing a bitmap and a vector image and
compares the two data representations. Such an image has to be formatted by using the “image” style, available
through the toolbar. The image needs a short description that will be used as a caption and a longer explanation
which completely describes the information embedded inside the image. This description will be provided to
blind students and to students that access to didactical contents through a non visual browser. Both the
descriptions are identified by using the appropriate styles, and specifically:
¾ the “short description” style, associated to a button on the toolbar, and
¾ the “long description”style, associated to another button on the toolbar.
The style “Short description” automatically appears after each use of the style “Image” in order to suggest the
author a correct use of styles. Analogously, the style “Long description” appears after each use of the style
“Short description”.
The page is closed by opening a new one, e.g. inserting a section break; this task could be performed by using
the appropriate MS Word menu option or by activating a related button. The final result is depicted in the
following Figure 3, showing the final layout of the page inside the MS Word editor.
The author completes the sub-module by creating a sequence of pages, similarly to that one described above.
Appropriate mechanisms to format and structure contents are also provided to insert tables, terms in foreign
languages, animations and all the other elements that can be included in an A³ module.
Figure 3. The page created with MS Word
Verifying contents
Some of the actions due to obtain a universal and accessible content could be forgotten by the author that
sometimes prefers to write a fast draft and improve it by adding additional information later on. The system
provides a verification toolbar, depicted in the following Figure 4, to support the author in controlling the
correctness and completeness of his contents.
Figure 4. The verification toolbar
With the verification toolbar, the author can control the syntactical respect of all the guidelines which guarantee
universality and accessibility of the learning materials. A typical control verifies the presence of the required
description of images; in this case, two different alerts are provided (as depicted in Figure 5):
¾ An error, every time a short description is missing. The short description corresponds to the alt attribute of
the img tag and is required to guarantee the accessibility of graphical information, thus the absence of a
short description it is considered as a heavy error (World Wide Web Consortium, 1999c).
¾ A warning, related to every missing long description. The long description is needed for complex graphical
contents (e.g. charts, graphs, etc.), which need additional descriptive information to be understood by blind
people or by people accessing with no-graphical browsers (World Wide Web Consortium, 1999c), e.g. Lynx
(Lynx Project, 2006).
Figure 5. Alert messages
The automatic control can catch only syntactical errors, such as a missing description. It is substantially
impossible to automatically ensure the description will be adequate to the image. Other constraints to the
structure and the accessibility of contents could be activated through the verification toolbar such as the
hierarchical use of nested headings and the presence of summaries for data tables.
Adding SCORM metadata
The interface for this step has been designed according to the above mentioned approach: giving users the
possibility of completing their tasks without having to learn new technologies and tools.
A form directly integrated into Microsoft Word allows users to add the required metadata. The SCORM
metadata (Advanced Distributed Learning, 2004a) have been classified into different groups, based on the author
competency and the importance of each single data. We have identified three main roles in the process of content
authoring: the lesson author (writing single units of content), the module author (collecting single units into a
compound) and the editor (assembling the final LO). We have selected the relevant metadata and we have
created specific interfaces for each of these roles. Moreover, we put the most important fields onto the main form
and the other relevant fields in an advanced form, so as to simplify the author’s work. An example of the
SCORM interface in ISA-BeL is shown in the following Figure 6.
Figure 6. The form for Meta-Data
Producing the LO
Once the SCORM meta-data were inserted, the creative work of the author is completed and the production
phase begins. In complex organizations (such as, for example, in the Project A³ staff) this phase is performed by
a post-production team. Three actions have to be performed in sequence:
1. The author has to save each file by using the “Save as” option of the “File” menu provided by MS Word to
obtain an .htm version. This phase depends from the used word processor and it is unnecessary for some
specific tools, i.e. Open Office Writer (OpenOffice, 2006).
2. Each of the .htm needs to be transformed into an intermediate XML format by using ISA represents
fragments of raw LO. This cleaning process transforms the presentation-centered HTML automatically
produced by MSWord in a valid XML file which correctly separates presentation and structure. Such a
XML includes some HTML tags used with their implicit semantics to structure its content. Each file
contains a portion of final LO content of the while presentational elements are completely removed. Figure
7 presents the ISA on line interface allowing the user to process single files or a .zip file containing all the
fragment that compose a LO.
3. Finally the set of files processed by ISA are sent to the BeL system which offers two main functions: (i) to
compose the LO by sequentializing the content defined by each single .xml file and (ii) to add all the
elements needed for the provisioning of a Web content (i.e. presentation and navigation). Figure 8 presents
the ISA on line interface that allows the user to process single files or a zip file containing all the fragments
composing a LO.
Figure 7. The ISA on line interface
Figure 8. The BeL on line interface
The authoring and producing phases are now completed. The LO has to be loaded onto a LCMS to be browsed.
In A³ scenarios such a platform is a personalized instance of ATutor (Adaptive Technology Resource Centre,
University of Toronto, 2006). Figure 9 shows the page created during the process we have detailed above stepby-step as it is rendered on the Web.
Accessibility evaluation
Accessibility of e-learning materials which has been produced by the above mentioned tools has been widely
tested by involving a group of people with disability in verifying on-line contents and services. Universality of
materials has been tested by using different browser running on different platforms and specifically:
¾ MS Internet Explorer 5.0 and later (Microsoft Corp., 2006),
¾ Mozilla Firefox 1.0 and later (Mozilla Project, 2006),
¾ Netscape Communicator 7.0 and later (Netscape Communications Corp., 2006),
¾ Lynx 2.8.4 rel. 1 (Lynx Project, 2006),
¾ IBM Home Page Reader 3.0 (IBM, 2006), and finally
¾ Apple Safari 1.0 (Apple Computer Inc., 2006).
There are two obvious considerations about such an evaluation. First of all, we have exclusively considered
accessibility from a technical point of view. Pedagogical issues, including those ones on learning styles, are out
of the scope of our tests. Secondly, all LOs were controlled once loaded onto the LCMS to be properly browsed.
Furthermore content produced by our process is compliant to all the constraints considered by the Italian Law on
Web Accessibility (Italian Parliament, 2004), meeting also WCAG 1.0 AA level (World Wide Web Consortium,
1999c) and Section 508 (U.S. Rehabilitation Act Amendments of 1998, 1998) requirements.
Figure 9. A page as it’s rendered by a LCMS
Finally, accessibility of the whole process has been verified by involving in authoring and reviewing tasks blind
people using the screen reader JAWS, versions 5.10 and later (Freedom Scientific, 2006). The accessibility of
the process itself is both a complex and a crucial objective, which creates the basis for a true inclusion of people
with disabilities inside LO production teams.
Conclusions and future works
The need for simplicity in knowledge creation and sharing has been a basic concern in the international elearning community. This paper presents a content creation and management process allowing authors to easily
produce accessible and portable LO by using personal productivity tools (e.g. a word processor).
The proposed system meets the seven main goals, listed in Section 1: it permits author to simply editing
contents, by starting from zero while reusing existing materials. The tool supports the best know conciliation
between e-learning standard and accessibility guidelines, so as to guarantee portability through LCMS together
with the respect of WCAG (AA level) and of the Italian Law on accessibility. Finally, such an high accessibility
level is itself an assurance that our process produces LOs which are compliant to main Web Standards based on
an uniform and simple presentation style.
Compared with literature, our approach combines the idea to automatically process contents to produce SCORM
compliant LOs (Advanced Distributed Learning, 2004c) with the growing need to meet accessibility
requirements in e-learning (IMS Global Learning Consortium, 2002b).
Main future work is devoted to extend the functionalities of the system, by offering supports in creating more
complex contents to the authors (such as rich media contents and conceptual maps) by ensuring accessibility and
universality of the results together with simplicity of usage.
This work was partially funded by MIUR (Italian Ministry of Education, University and Research) and was
supported by CRIAD (www.criad.unibo.it). Authors want to thank Lorenzo Donatiello, Simone Martini, Marco
Roccetti, Nelda Parisini and all the colleagues that supported this work with their precious suggestions. Finally
authors want to thank ASPHI Onlus Foundation (www.asphi.it) that supported tests with users.
4system (2006). WBTExpress 5.1, retrieved June, 2006 from: http://www.wbtexpress.com/.
Adaptive Technology Resource Centre, University of Toronto (2006). ATutor Learning Content Management
System, retrieved June, 2006 from: http://www.atutor.ca/.
Advanced Distributed Learning (2006). Retrieved June, 2006 from: http://www.adlnet.org/.
Advanced Distributed Learning (2004a). Content Aggregation Model (CAM), Sharable Content Object
Advanced Distributed Learning (2004b). Run Time Environment (RTE), Sharable Content Object Aggregation
Model (SCORM) Version 1.3, retrieved June, 2006 from: http://www.adlnet.gov/downloads/files/194.cfm.
Advanced Distributed Learning (2004c). Sharable Content Object Reference Model (SCORM) 2004 2nd Edition
Document Suite, retrieved June, 2006 from: http://www.adlnet.org/downloads/70.cfm .
Apple Computer, Inc. (2006). Safari
Boyle, T., Leeder, D.C., & Chase, H. (2004). To boldly GLO - Towards the next generation of learning objects.
In Proceedings of the Panel session at E-Learn 2004, November 1 - 5, 2004, Washington, DC.
Bradley, C., & Boyle, T. (2004). The design, development and use of multimedia learning objects. Journal of
Educational Multimedia and Hypermedia, 13 (4), 371-389.
Conlan, O., Dagger, D., & Wade, V. (2002). Towards a Standards-based Approach to e-Learning Personalization
using Reusable Learning Objects. In Proceedings of E-Learn 2002, October 15 - 19, 2002, Montreal, Canada.
Di Iorio, A., Feliziani, A. A., Mirri, S., Salomoni, P. & Vitali, F. (2005). Simply Creating Accessible Learning
Object. In Proceedings of eLearning and Human-Computer Interaction: Exploring Design Synergies for more
Effective Learning Experiences, INTERACT 2005 Workshop, 12-16 Sep 2005. Roma, Italy.
Dolog, P., & Nejdl, W. (2003). Personalisation in Elena: How to cope with personalisation in distributed
eLearning Networks. In Proceedings of International Conference on Worldwide Coherent Workforce, Satisfied
Users - New Services For Scientific Information, 17-19 September 2003 Oldenburg, Germany.
Dolog, P., Henze, N., Nejdl, W., & Sintek, M. (2004). Personalization in Distributed eLearning Environments. In
Proceedings of WWW2004, 17-122 May 2004, New York, USA.
Downes, S. (2006). E-learning 2.0 at the E-learning Forum. In Perspectives and Interactive Discussion around
eLearning 2.0, Elearning Forum Meeting, February 22, 2006, Menlo Park, CA.
Freedom Scientific (2006). Jaws for Windows Overview,
Gabrielli, S., Mirabella, V., Kimani, S., & Catarci, T. (2005). eLearning Accessibility: A Comprehensive
Approach to Content Development. In Proceedings of eLearning and Human-Computer Interaction: Exploring
Design Synergies for more Effective Learning Experiences, INTERACT 2005 Workshop, 12-16 Sep 2005. Roma,
IEEE LTSC WG12 (2006). IEEE Learning Technology Standards Committee (LTSC), retrieved June, 2006 from
IEEE LTSC WG12 (2006). Learning Object Metadata Standard Maintenance/Revision, retrieved June, 2006
from http://ltsc.ieee.org/news/20021210-LOM.html.
IMS Global Learning Consortium (2006). Retrieved June, 2006 from http://www.imsproject.org/.
IMS Global Learning Consortium (2002). IMS AccessForAll Meta-data Specification, retrieve June, 2006 from
IMS Global Learning Consortium (2002). IMS Guidelines for Developing Accessible Learning Applications,
retrieved June, 2006 from http://www.imsproject.org/accessibility/accessiblevers/index.html.
Italian Parliament (2004). Law nr. 4 – 01/09/2004, Official Journal nr. 13 – 01/17/2004.
Kelly, B., Phipps, L., & Howell, C. (2005). Implementing A Holistic Approach To E-Learning Accessibility. In
Proceedings of the 12th International Conference of the Association for Learning Technology, 6-8 September
2005, Manchester, England.
Lynx Project (2006). Lynx Home Page, retrieved June, 2006 from http://lynx.browser.org/.
Microformats.org (2006). Microformats.org Home Page, retrieved June, 2006 from http://microformats.org.
Microsoft Corp. (2006). MS Internet Explorer
Mirabella, V., Kimani, S., Gabrielli, S., & Catarci, T. (2004). Accessible e-learning material: A no-frills avenue
for didactical experts. New Review of Hypermedia and Multimedia, 10 (2), 165–180.
Morales, R., Leeder, D., & Boyle, T. (2005). A Case in the Design of Generative Learning Objects (GLOs):
Applied Statistical Methods. In Proceedings of Hypermedia and Telecommunications (EDMEDIA), June 27-July
2, 2005, Montreal, Canada.
Mozilla Project (2006). Mozilla Firefox Home Page, retrieved June, 2006 from: http://www.mozilla.org/.
Netscape Communications Corp. (2006). Netscape Browser home page, retrieved June, 2006 from
OpenOffice (2006). Writer, retrieved June, 2006 from http://www.openoffice.org/product/writer.html.
ReadyGo, Inc. (2006). ReadyGo Web Course Builder, retrieved June, 2006 from http://www.readygo.com/.
Salomoni, P., & Mirri, S. (2004). A multimedia broker for ubiquitous and accessible rich media content
transcoding. In Proceedings of 1st IEEE International Workshop on Net-working Issues in Multimedia
Entertainment – NIME (Globecom Satellite Workshop), November 29, 2004 Dallas, TX, USA.
Salomoni, P. & Mirri, S. (2005). Providing Accessible and Portable Video Lecture from Content transcoding. In
Proceedings of 11th Euromedia Conference, 11-13 April 2005, Toulouse, France.
Serco (2006). VirtualCampus, retrieved June, 2006 from http://www.teknical.com/products/virtual_campus.htm.
SumTotal Systems, Inc. (2006). Toolbook, retrieved June, 2006 from: http://www.toolbook.com/index.php.
U.S. Rehabilitation Act Amendments of 1998 (1998). Section 508, retrieved June, 2006 from
University of Bologna (2004). Ambiente Accessibile d’Apprendimento, retrieved June, 2006 from:
Vitali, F. (2003). Creating sophisticated web sites using well-known interfaces. In Proceedings of HCI
International 2003 Conference, June 22-27, 2003, Crete, Greece.
World Wide Web Consortium (1999a). Accessibility Features of SMIL, retrieved June, 2006 from:
World Wide Web Consortium (2001a). Extensible Stylesheet Language (XSL), Version 1.0, retrieved June, 2006
from http://www.w3.org/TR/xsl/.
World Wide Web Consortium (2000). Extensible HyperText Markup Language (XHTML) Version 1.0, retrieved
June, 2006 from http://www.w3.org/TR/xhtml1/.
World Wide Web Consortium (1999b). HyperText Markup Language (HTML), Version 4.01, retrieved June,
2006 from http://www.w3.org/TR/html4/.
World Wide Web Consortium (2001b). Synchronized Multimedia Integration Language (SMIL) 2.0, retrieved
June, 2006 from http://www.w3.org/TR/smil20/.
World Wide Web Consortium (1999c). Web Content Accessibility Guidelines 1.0. (WCAG), retrieved June,
2006 from http://www.w3.org/TR/WCAG10/.
World Wide Web Consortium (2006). Web Content Accessibility Guidelines 2.0 W3C Working Draft 27 April
2006, retrieved June, 2006 from http://www.w3.org/TR/WCAG20.
Gabrielli, S., Mirabella, V., Kimani, S., & Catarci, T. (2006). A Boosting Approach to eContent Development for
Learners with Special Needs. Educational Technology & Society, 9 (4), 17-26.
A Boosting Approach to eContent Development for Learners with Special
Silvia Gabrielli, Valeria Mirabella, Stephen Kimani and Tiziana Catarci
University of Rome “La Sapienza”, DIS (Dipartimento di Informatica e Sistemistica), via Salaria 113, 00198
Rome, Italy
Tel: +39-06-49918548
Fax: +39-06-85300849
[email protected]
[email protected]
[email protected]
[email protected]
Of late there has been a growing interest and effort toward meeting the requirements of persons with special
needs. However, most of the accessibility standards and guidelines that have been proposed have been
developed by adopting a domain independent and often ‘technical’ perspective. Such proposals are
therefore often not sufficient to achieve accessibility goals in specific application areas such as eLearning.
This paper presents a boosting approach/framework toward the development of more effective and usable
accessibility indications for authors of didactic content, which are currently being fed and tested within the
Italian context of the VICE project. This approach is intended to take into account the aforementioned issue
and to make a step forward with respect to existing accessibility proposals and approaches in the eLearning
domain. In particular, we discuss our design method for increasing the quality of eLearning materials for
learners with special needs and an authoring tool, aLearning, to support eLearning content authors in their
development of didactic material matching those needs.
eLearning, Special needs, Accessibility, Guidelines, Didactical experts
1. Introduction
eLearning systems typically rely on repositories of online materials that are made available to learners and
teachers for access, use or re-use. We have of late witnessed a growing commitment and effort toward providing
universal access to online contents for meeting the requirements of persons with special needs, the objective
being either to comply with national and international regulations on accessibility (PubbliAccesso, 2004; Section
508, 1973) or to expand the number of potential users that can take advantage of online resources.
However, most of standards and guidelines now available in the accessibility field (Caldwell et al., 2003; W3C,
2004) have been developed by adopting a domain independent perspective, which is sometimes insufficient to
achieve accessibility goals in specific application areas, such as in the case of e-learning and the creation of
eContent for people with special needs. Previous studies have shown that W3C guidelines are often more suited
to ensure ‘technical’ aspects of accessibility (i.e., that a visual content is readable by a blind person through the
support of assistive technologies, such as screen readers) rather than ‘conceptual’ ones, which are more related to
the usability and quality of experience the disabled user is provided with when accessing that specific content
(Di Blas et al., 2004).
In this paper we discuss a boosting approach aimed at supporting the usability and effectiveness of accessibility
guidelines that eLearning authors could follow to generate appropriate materials for users with special needs (as
well as to transform inaccessible contents into accessible one). This approach takes into account the issue
mentioned in the foregoing paragraph and contributes to progress from existing accessibility proposals in the
eLearning domain, by feeding and testing its ideas within a national project for the production and reuse of
didactic material targeted at higher education and industrial contexts (VICE project, Virtual Communities for
Education, CNR/MIUR Italy, http://www.progettovice.it). In particular, the contribution consists of: i) a design
method/framework for increasing the quality of eLearning materials for learners with special needs, ii) an
authoring tool, named aLearning, to investigate and fulfill usability requirements of eLearning content creators
in their development of didactic material for learners with special needs. While acknowledging that the support
accorded by the proposed authoring tool is not optimal for learners with special needs, the tool does provide a
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain
the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work
owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists,
requires prior specific permission and/or a fee. Request permissions from the editors at [email protected]
guiding and supporting environment through which authors can realize eLearning materials of higher quality for
those learners.
The rest of the paper is organized as follows: Section 2 comparatively introduces our approach by relating it to
previous work in the field. In Section 3 we describe the design process and the main functionalities of a tool for
supporting didactic authors in the creation of accessible eLearning content. In Section 4 we present our current
plan and ongoing work for evaluating the tool and methodology developed, both in terms of its usability for
authors and effectiveness in reaching accessibility goals. Section 5 contains our concluding remarks.
2. Related work
In recent years many recommendations for the design of accessible eLearning contents have been delivered by
international standard organizations, as well as private educational initiatives worldwide (W3C, 2004; ADL,
2005; IMS, 2004; Hodgins & Duval, 2002). Among these we refer in particular to the efforts of NCAM, IMS
Global Learning Consortium and California Community Colleges (Freed et al., 2003; IMS, 2002; CCC, 2000),
for providing indications on how to create accessible learning material and raising awareness about the most
appropriate approach to take when accomplishing this task. Their concern is not only that of achieving content
accessibility, but they aim at showing how to address and preserve the didactical properties of eLearning
resources when they are to be made accessible to learners with special needs.
However, it has previously been shown that indications in the form of guidelines can be very difficult to
incorporate into the everyday design or authoring practice, especially by educators who do not have prior
expertise on accessibility (Dix et al., 2003; Gabrielli et al., 2004). To tackle this issue, our recent work has been
addressed towards investigating effective methods and tools to make accessibility guidelines more useful and
usable to eLearning authors during the creation of accessible content (Mirabella et al., 2004; Gabrielli et al.,
2004). This work is based on the assumption that, compliance of a specific web content to guidelines, as it might
be assessed by the most commonly used accessibility checkers, e.g., Bobby, LiFT, A-Prompt (Bobby 2006;
LiFT, 2006; A-Prompt, 2006), is not sufficient for eLearning material, since these tools mainly perform a
syntactic assessment of web pages, but say very little about the adequacy of any equivalent-alternative contents
created, to enable effective use of this materials by learners with special needs.
The approach we have adopted is not dissimilar from other relevant proposals found in the literature that support
authors in creating directly the right (i.e., accessible) version of didactic material. An example is ATutor Content
Editor (ATRC, 2006), part of a Learning Content Management System meant to support educators in a quick
creation, reuse, packaging and distribution of web-based instructional materials. What is different in our work is
the specific interest and focus on approaching the design of this type of systems from an experimental
perspective. Specifically, we aim at developing an experimental prototype enabling a deeper analysis of authors
usability requirements during the process of creating accessible content for learners with special needs (including
different kinds of disabilities). By conducting a series of evaluation studies on the prototype use we intend to
discover, and also propose solutions, to usability issues raised by the deployment of accessibility guidelines in
the authoring practice. These findings will inform the current debate on how to effectively approach the design
of eLearning contents when a wide range of disabilities (not only sensorial ones) is taken into account.
In a previous work (Mirabella et al., 2004) we have presented a no-frills methodology to guide didactical experts
during the creation of accessible eLearning content, specifically customized for learners with special needs. This
methodology is based on a first categorization of the types of disabilities or impairments for the potential
learners. We also characterize the types of content that are critical when it comes to making the learning material
accessible. Such an undertaking would be resourceful especially to the creator of the didactical contents (e.g., a
lesson), who are usually not well acquainted with the issues or problems of accessibility. We then associate the
realized categories of disabilities with the various types of critical content. At the point where a particular
disability intersects with a particular critical content, we analyse the range of accessibility barriers and
opportunities presented. In the process, we are better positioned to develop indications that could be resourceful
especially to the creator of the didactical contents of the learning material. It is worth noting that such indications
can guide in specifying alternative content that is characterized by matching the following requirements of being
useful, appropriate and effective, where:
¾ Usefulness is closely related to the concept of necessity of the content. The objective is for an author to
eliminate the contents that are not necessary, in order to increase the overall usability of the didactic
modules, especially in the case in which alternative content types (or formats) have also been introduced.
Appropriateness is related to the selection of the right contents while taking into account the characteristics
and requirements of a certain type of disability.
Effectiveness is related to the capacity of the didactic content to enabling the achievement of the learning
In the rest of this section, we first discuss how we categorized the types of disabilities or impairments and the
critical content types considered so far, as well as the relations between them. We then exemplify some possible
applications of our proposed guidelines and no-frills methodology.
2.1 The disabilities, the contents and their critical relations
With critical learning resource types we refer to typical didactical contents (e.g., of an eLearning module) that
can affect accessibility. For example a graph (but also a diagram, image or table) is a critical didactical content
for people who present cognitive disabilities (such as dyslexia) because of problems with ordering and cluster
identification, thus it is not accessible as it is. Several lists of didactical contents are reported by some of the
most notable organizations involved in eLearning standardization and specifications.
The IEEE Learning Object Metadata (LOM) (LTSC, 2004) is one of the most comprehensive schemes
developed for the description of Learning Objects. In the Educational section of the LOM, there is a list of
Learning Resource Types including: exercise, simulation, questionnaire, diagram, figure, graph, index, slide,
table, narrative text, exam, experiment, problem statement, self-assessment, and lecture.
We have partially adopted this list, in agreement with the opinion that "LOM allows this (the Learning Resource
Types) element to be understood as designating types or formats of content, as well as potential uses of this
content. Also, the values recommended by the LOM exclude many important types of content, and also exclude
many educational applications of content" (CanCore, 2006). In our proposal we start with an investigation of
only the elements from the LOM list that are both critical from an accessibility perspective and that are format
We consider the following elements from the LOM proposal to be format dependent: exercise, simulation,
questionnaire, exam, experiment, problem statement and self-assessment. Their format, in fact, can vary
according to the different types of contents (images, text, tables, etc.) they may include. Moreover, we consider a
slide and a lecture to be non-digital content with respect to the LOM definition. So far, our approach translates
visual content to text and therefore, here, we do not list text and index as part of critical content.
Consequently, we focus on four types of critical didactical content from the LOM model: diagram, figure, graph,
and table. We also consider another set of critical learning resource types which is derived from a list provided
by the CPB/WGBH National Center for Accessible Media (NCAM). In (Freed et al., 2003), CPB/WGBH
presents eight guidelines in relation with the same number of learning content types: images, multimedia, forms,
tables, text-books, interactivity, graphs and math.
From the CPB/WGBH NCAM we investigate: multimedia and math/scientific expressions. We do not take into
account forms because the kind of accessibility issues they present are more related with the format rather than
with the semantic meaning of didactical content. We also exclude textbooks, since we assume they are in textual
form, and interactivity, since it is not a kind of content. We are aware that multimedia contents present complex
technical and pedagogical issues to be addressed for ensuring their accessibility. Initially, we have considered
different media as aggregated in a same category (multimedia) to refer to their semantic meaning more than to
their particular format. Our aim is to provide clear indications, for example, on how to create effective captions
and audio descriptions for video contents typically employed as eLearning materials. More detailed analysis of
different components of this category will be considered in our future work in compliance with related efforts in
the field e.g., Rich Media Accessibility (NCAM, 2006).
The six elements selected above are then mapped and analysed in relation to the following main types of
disabilities for potential learners: visual, hearing, motor, cognitive-language impairments. It is worth observing
within each of these categories there are many variations and degrees of impairment that require specific
strategies to be addressed. Also, there could be learners that exhibit more than one type of disability, like senior
citizens having both sensorial and cognitive impairments. However, our current efforts are oriented toward
providing didactic experts with a wide spectrum of indications supporting the removal of accessibility barriers
from teaching materials for the main categories of disabilities that might be present in our target user population.
As an example, to improve the accessibility of a figure for learners with cognitive disabilities, we indicate how
to provide significant alternative content that can properly explain that figure, possibly including a description of
the overall meaning and contribute provided by that figure within the teaching material in which it is embedded.
For the needs of learners with motor impairments, we raise awareness and provide support to authors on how to
make operations on mathematical and scientific expressions more easily accessible, since this type of learners
may have difficulties in using input modalities, such as the mouse, to insert and modify symbols or strings, as
well as to keep track of the content already inserted.
Table 1 which is adapted from (Mirabella et al., 2004) reports schematically the most relevant mappings
identified between disabilities and type of content considered so far, which are also of particular relevance to the
didactic materials currently dealt with in the VICE project.
Table 1: Relevant mappings between disability types and critical contents
Type of Disabilities
Type of Content
Math/scientific expression
Figure 1: Example of Guideline for type of Content Graph w.r.t. Visual Disabilities
2.2 The guidelines and their application through the no-frills methodology
Guidelines were developed for each relevant mappings identified between critical contents and types of
disabilities, as discussed in the previous section. They included detailed indications on how to create equivalentalternative versions of inaccessible contents, as well as hands-on examples for authors on how to proceed during
the repairing (Gabrielli et al., 2004; Mirabella et al., 2004). Fig. 1 below reports a selection of indications
contained in the guideline developed for the type of content Graph w.r.t. Visual disabilities.
In practice, the guidelines are expected to support authors as they apply the no-frills methodology (Mirabella et
al., 2004) in both cases in which: a) they are creating new didactic materials to be added to a content repository,
b) to reuse already existing materials and transform them into accessible ones for learners with special needs. In
particular, the no-frills methodology provides an avenue for guiding didactical experts in making the most of
their didactical experience when choosing the necessary and alternative didactical content that can fit the
requirements of disable learners. A brief description of the different actions an author should take while applying
the no-frill methodology is provided below.
1. Initially, the method requires the categorization of the potential learners, in terms of disabilities or
impairments for the specific didactical module considered.
2. The next step involves identifying the content types the didactical module is made of. For every type of
content, the methodology analyses its impact on each of the categories of the learners in terms of physical
accessibility (the learner can practically access the content) and logical accessibility (the learner can
effectively access the content). In the first case, the focus is on accessibility as intended in a rather rigid
sense, whereas in the second case, the focus is on accessibility related more to didactic effectiveness.
3. The third step involves exploiting opportunities for inserting alternative content that corresponds to the
critical content under consideration for some category of learners. In particular, we consider that the didactic
content may be considered as:
a) optional, in which case the content is not essential to the realization of the module’s objective;
b) mandatory, in which case the content is essential or relevant to the realization of the module’s
objective. In the mandatory case, the didactic expert may also specify whether the level of
accessibility is acceptable toward realizing the module’s objective. If the accessibility level is not
considered acceptable for accomplishing the objective, the didactic expert may decide whether to
translate the content or substitute it with some other relevant alternative content better able to
contribute to the realization of the same objective.
2.3 Main issues raised in the authoring practice
The guidelines and methodology proposed were initially tested with a small group of eLearning authors during a
formative evaluation study that we conducted to assess the principle on which our approach is based (Gabrielli et
al., 2004). The evaluation scenario involved five didactic experts who were asked to work individually at
inspecting the contents of a Learning Object of the type used in the VICE repository, which consisted of 4 basic
modules of material on computer science topics (European Computer Driving License material). These included
a series both text and the collection of critical contents of the types mentioned above: figures, diagrams, graphs,
tables, multimedia, and math/scientific expressions.
The main task didactical experts were asked to perform was to examine and to identify any content that might
have been critical for disable learners (according to the categories reported in Table 1) and to make it accessible
to them by following the accessibility guidelines and no-frills approach described above. These were made
available in the authoring environment as indications contained within HTML page whose links inspectors could
activate whenever required. Throughout the study, suggestions for alternative contents required were reported on
paper protocols by the didactic experts, then transformed into digital format and added to the LO by the study
moderator, so that subsequent participants had the opportunity of evaluating the appropriateness of the
alternative content previously created by other authors. Didactic experts were asked to think aloud during their
task performance (lasting approximately 30-45 minutes) and briefly interviewed at the end of the task to collect
their impressions about the guidelines and methodology used, as well as about the difficulties raised during their
application. As a result we observed that didactic experts found quite difficult and time demanding the
translation or substitution of critical content with appropriate alternatives and accessible versions of it, especially
when it involved the use of diagrams, graphs, tables and scientific-math expressions. Typically, experts used to
develop and apply a limited but consistent set of ‘translation’ strategies throughout the whole duration of their
activity (e.g., inserting textual description for visual contents), whilst elaboration of more original alternatives
was rarely found (although indications in the guidelines were specifically prompting them to undertake this
option). Although this finding might be related to experts lack of familiarity with accessibility issues, it also
highlighted the need for content creators to be better supported and trained, for instance by interface
functionalities, in the acquisition of a specific expertise on accessibility. However, we derived from observations
collected in this study that any accessibility support provided during such a task should not constrain authors’
activity in a very prescriptive way, but enable flexible and creative use of the indications made available.
To achieve the design of this kind of support, we decided to develop a user-friendly interface for eLearning
authors. This serves to present accessibility indications in a more interactive way and to enable authors a more
intuitive and easier application of our no-frills methodology during the creation of accessible contents.
3. aLearning: Usable Presentation of Accessibility Guidelines
The experimental prototype we have designed is an accessibility interface we named aLearning. The main
objective of the tool is to enable exploration of more usable presentation of accessibility guidelines in the context
of eLearning content creation. The tool (developed in Java) is integrated into an application server which is a
key component of the VICE architecture and provides also other functionalities, such as a metadata modeler and
editor (Fig. 2). The application server is responsible of ensuring that the eLearning contents retrieved from the
VICE repository are packetized into SCORM compliant LOs, to be then uploaded on Learning Management
Systems, such as Atutor (ATRC, 2006) for instance, and presented as web-based courses.
The support provided by aLearning is also available in the case where authors decide to retrieve materials not
originally created for users with special needs. This is not the approach generally recommended by the
accessibility literature for generating high quality didactic contents, however it could bring some important
advantages in specific eLearning contexts, such as costs and time savings in the production of the required
materials. Initial investigations conducted during the user requirements analysis of the VICE project also
reported authors increased willingness to work at transforming contents into accessibility ones in return of the
possibility of reusing existing materials for the design of university courses.
Figure 2: aLearning and the VICE architecture
A user-centred design approach was adopted to make aLearning interface intuitive to use even for authors
without a specific expertise on accessibility and/or without a technical background. A main characteristic of the
tool is to facilitate the creation of accessible and usable eLearning contents by making automatic or semiautomatic the application of any technical step of our no-frills methodology that would not benefit from human
intervention or expertise, such as the identification of critical contents within the eLearning material. However,
the tool would prompt, inform and capitalize on authors’ didactic knowledge and decision making when
addressing the non-technical steps of the method (e.g., creating an alternative-equivalent version of the content).
The design process went through a task analysis phase and a series of low-mid tech prototyping activities to
develop the interface features required, that we briefly describe in the following.
3.1 Overview of aLearning Prototype
aLearning interface mainly supports authors by:
¾ Automatically identifying and marking any critical and inaccessible content within the eLearning material.
For example, when a graphic file is detected (e.g., by detecting the file extension) the tool first asks the
author to classify it as a diagram, figure, graph etc., then it presents all relevant guidelines to ensure
accessibility of that content according to the category assigned by the author. In this way the authoring or
repairing process performed on eLearning material is led by the consideration of didactically relevant
categories of contents (such as diagrams, figures, graphs, ...), instead of the web related ones (like framesets,
scripts, links, ...) typically used by common tools for accessibility checking and repairing (Bobby, 2006;
LiFT, 2006; A-Prompt, 2006; NCAM, 2006).
¾ Enabling the author to select and start repairing a critical content (e.g., a graph) by choosing among three
alternative modalities: i) directly clicking on the critical content as it is marked within the didactic module’s
pages that are displayed, ii) selecting that content from a category list of learning resources reported on a
frame window, iii) starting a step-by-step repairing procedure for the whole sequence of inaccessible
contents identified. These different modalities are expected to provide authors with more flexibility on how
to complete the repairing process (also in terms of its timescale), differently from the file-by-file repairing
process typically supported by other tools.
¾ Explaining to the user why a specific content is inaccessible, prompting the user to classify it as optional or
mandatory for the objectives of the course (according to our no-frills methodology), supporting different
ways of creating alternative versions of the content if it is mandatory, providing links to more detailed
information (guidelines and examples) on how to create an appropriate alternative representation of it.
Fig. 3 below presents an example of how an author would be guided to check and repair the critical content
Graph according to the guideline reported in section 2.
Figure 3: Example of aLearning support to authors for repairing the type of content Graph
Beyond these basic functionalities of the tool, we are also investigating how aLearning interface could support
authors in the generation of appropriate metadata for the accessible contents created. By pursuing compliance
with existing standards and specifications (IMS AccessForAll, http://www.imsglobal.org/accessibility), as well
as by acknowledging current open issues in the field (e.g., adaptability of content), we are analysing the
opportunity for a possible extension of LOM categories that would specifically address accessibility properties
of eLearning content and enhance the retrieval of accessible resources from the VICE LOs repository. Moreover,
according to our design approach, any interface support for the editing of metadata should include clear and
detailed explanations of the metadata fields required, to make it easier for authors without previous expertise on
standards technicalities or vocabulary, a straightforward understanding of the metadata information to be
Our current efforts are addressed toward refining and improving the usability and learnability of the tool
developed so far, to speed up not only any authoring and repairing process performed by the user, but also
authors’ acquisition of expertise on accessibility by means of navigation through aLearning functionalities.
4. Evaluation Plan and Work in Progress
As we mentioned in Section 2, we have already conducted some formative evaluation activities to assess the
soundness of our approach and guidelines as a support to the eLearning authoring practice. However, we plan
now to conduct a more thorough evaluation on how effective the method and aLearning tool developed are in
supporting the creation of accessible contents for learners with special needs. This can be achieved by
performing comparative studies with other existing accessibility repairing tools and evaluation strategies.
Current efforts toward a detailed verification of web content accessibility suggest recur to a combination of
techniques and tools, such as:
¾ the use of more than one automatic check-repairing tool,
¾ a manual verification of content compliance with accessibility guidelines (usually performed by accessibility
¾ some testing phases directly involving users with special needs or accessibility experts in using the content
created (Lang, 2003; Mankoff et al., 2005; Bertini et al., 2005).
The main motivation for adopting this comprehensive strategy to accessibility verification is that the different
methods mentioned above provide useful information as well as possible limitations (e.g., in terms of resources
to be invested for the evaluation), thus their combination is often likely to produce better results at a lower cost.
Inspired by these considerations we are currently conducting the following activities to validate the quality of the
proposed approach to pursue accessibility in the eLearning domain:
¾ A comparative assessment of aLearning features with those provided by other well-known tools for the
authoring of accessible eLearning material, e.g., ATutor Content Editor (ATRC, 2006), or accessibility
check and repairing, e.g., A-Prompt and LiFT (A-Prompt, 2006; LiFT, 2006), in terms of their usability and
support provided to users. In the evaluation studies we observe authors in the activity of transforming
eContent (e.g., LOs resources retrieved from the VICE repository) into accessible one, by means of the
authoring and check-repairing tools mentioned above. Application of usability criteria (such as
effectiveness, efficiency and user satisfaction) steer the assessment process.
¾ The involvement of a number of accessibility experts in manually checking if a correct application of the
eLearning guidelines and no-frills methodology has been performed by authors in phase 1. This evaluation
is inspired to methodological studies typically carried out in the usability field to assess the validity of
inspection techniques, such as Heuristic Evaluation and Cognitive Walkthrough (Cockton et al., 2003). In
our case, the aim is mainly to verify which possible improvements to our guidelines are required to ease
their understandability and deployment.
¾ The direct involvement of learners with special needs in accessing the educational materials produced in
phase 1 (by using the assistive technologies they are most familiar with) to verify the quality of the
eLearning contents generated by authors, with special attention to the equivalent-alternative versions of
didactic material they have created.
Observations collected from these evaluation studies will enable us to feed any future refinement of our
accessibility approach, as well as to further inform the iterative design of aLearning.
5. Conclusion
In this paper, we have highlighted some main limitations of the existing accessibility guidelines for eLearning.
The methodological approach we have proposed contributes to improve usability and effectiveness of
accessibility indications provided to eLearning authors. This kind of support is particularly important to deliver
in contexts such as our VICE project (which is targeted at higher education and professional training
environments) where there is also the need to comply with more restrictive accessibility regulations
(PubbliAccesso, 2004) than the ones that have inspired the design of other existing accessibility tools, e.g.,
ATutor in Canada (ATRC, 2006). Specifically, in this paper we have presented motivations and current efforts
towards designing more usable and interactive tools for the presentation of accessibility guidelines as supports to
the authoring practice. One of the main objectives of the approach adopted so far has been to encourage the
acquisition of expertise on accessibility by creators of eLearning resources, no matter what their professional
expertise or previous technical abilities are. Another main focus has been to promote, through a correct
application of the methodology presented, that the eLearning contents and experience eventually delivered to
learners with special needs reach the same levels of quality and effectiveness of the ones provided to nondisabled learners. Our current work entails a validation of the approach proposed, by which we expect to inform
its future refinement as well as its possible extension to a larger set of disabilities and types of contents.
This work has been supported by the VICE project (Virtual Communities for Education, CNR/MIUR, Italy,
http://www.progettovice.it/). We would like to thank Loredana De Giovanni for her comments and support
provided during the preparation of this work.
ADL (2005). Advanced Distributed Learning, retrieved June 29, 2006 from http://www.adlnet.org/.
A-Prompt (2006). University
ATRC (2006). ATutor, retrieved June 29, 2006 from http://www.atutor.ca/atutor/index.php.
Bertini, E., Billi, M., Burzagli, L., Gabbanini, F., Gabrielli, S., Graziani, P., & Palchetti, E. (2005). Testing
Accessibility in Mobile Computing. International Conference on Human-Computer Interaction 2005, July 2005,
Las Vegas, USA.
Bobby (2006). Watchfire Corporation, Bobby, retrieved March 14, 2006 from http://bobby.watchfire.com/.
Caldwell, B., Chisholm, W., White, J., & Vanderheim, G. (2005). Web Content Accessibility Guidelines 2.0,
Working Draft 23 November 2005, retrieved June 29, 2006 from http://www.w3.org/TR/WCAG20/.
CanCore (2006). CanCore Guidelines Version 1.9: Educational Category, retrieved March 14, 2006 from
CCC (2000). Guidelines for Producing Instructional and Other Printed Materials in Alternate Media for Persons
Cockton, G., Lavery, D., & Woolrych, A. (2003). Inspection-based Evaluations. In Jacko, J. A. & Sears, A.
(Eds.), The HCI Handbook, Lawrence Erlbaum Assoc., 1118-1138.
Di Blas, N., Paolini, P., & Speroni, M. (2004). “Usable Accessibility” to the Web for Blind Users. In Adjunct
Proceedings of the 8th ERCIM Workshop on User Interfaces for All, June 2004, Vienna, Austria,
Dix, A., Finlay, J., Abowd, G., & Beale, R. (2003). Human-Computer Interaction, Hillsdale, NJ: Prentice Hall.
Freed, G., Rothberg, M., & Wlodkowski, T. (2003). Making Educational Software and Web Sites Accessible.
Design Guidelines Including Math and Science Solutions, retrieved March 14, 2006 from
Gabrielli, S., Mirabella, V., Kimani, S., & Catarci, T. (2004). Steering the Development of Accessible eLearning Content. In Proceedings of the 3rd ECEL Conference, Paris, France, 517-526.
Hodgings, W. & Duval, E. (2002). IEEE LTSC Learning Object Meta-data LOM 1484.12.1 Vl Final Draft,
retrieved March 14, 2006 from http://ltsc.ieee.org/wg12/files/.
IMS (2002). IMS Guidelines for Developing Accessible Learning Applications, Version 1.0, White Paper,
retrieved March 14, 2006 from http://www.imsproject.org/accessibility/.
Lang, T. (2003). Comparing website accessibility evaluation methods and learnings from usability evaluation
methods, retrieved March 14, 2006 from http://www.peakusability.com.au/pdf/website_accessibility.pdf.
LiFT (2006). UsableNet, LiFT, retrieved March 14, 2006 from http://www.usablenet.com/.
LTSC (2004). IEEE LTSC Learning Object Meta-data LOM_1484_12_1_v1_Final_Draft, November 2004,
retrieved March 14, 2006 from http://ltsc.ieee.org/wg12/files/.
MAGpie (2006). NCAM, MAGpie, retrieved March 14, 2006 from http://ncam.wgbh.org/webaccess/magpie/.
Mahemoff, M. J., & Johnston, L. J. (1998). Principles for a Usability-Oriented Pattern Language. In Proc. of
Australian Computer Human Interaction Conference OZCHI’98, Adelaide: IEEE Computer Societey, Los
Alamitos, 132–139.
Mankoff, J, Fait, H., & Tran, T. (2005). Is Your Web Page Accessible? A Comparative Study of Methods for
Assessing Web Page Accessibility for the Blind. Proc. of CHI 2005, April 2-7, 2005, Portland, USA, 41-50.
Mirabella, V., Kimani, S., Gabrielli, S., & Catarci, T. (2004). Accessible e-Learning Material: A No-Frills
Avenue for Didactical Experts. The New Review of Hypermedia and Multimedia, 10 (2), 1-16.
PubbliAccesso (2004). PubbliAccesso, Italian Law 9 January 2004, n. 4, retrieved March 14, 2006 from
Section 508 (1973). Section 508 of the Rehabilitation Act, amended 1998, retrieved March 14, 2006 from
Van Welie, M., Van der Veer, G. C., & Eliëns, A. (2000). Patterns as Tools for User Interface Design.
International Workshop on Tools for Working with Guidelines, 7-8 October 2000, Biarritz, France, 313-324.
W3C (2004). Web Content Accessibility Guidelines 2.0, W3C Recommendations, Working Draft 29, November
2004, retrieved March 14, 2006 from http://www.w3.org/TR/WCAG20/#overview-design-principles.
De Carolis, B., Pizzutilo, S., Cozzolongo, G., Drozda, P., & Muci, F. (2006). Supporting Students with a Personal Advisor.
Educational Technology & Society, 9 (4), 27-41.
Supporting Students with a Personal Advisor
Berardina De Carolis
Researcher ,Dipartimento di Informatica – Università di Bari, Via Orabona 4, 70126 Bari, Italy
[email protected]
Sebastiano Pizzutilo
Associated Professor, Dipartimento di Informatica – Università di Bari, Via Orabona 4, 70126 Bari, Italy
[email protected]
Giovanni Cozzolongo, Pawel Drozda and Francesca Muci
Phd Student, Dipartimento di Informatica – Università di Bari, Via Orabona 4, 70126 Bari, Italy
[email protected]
[email protected]
[email protected]
Recently, many national and international initiatives show an increasing interest in promoting distance
education interventions by Universities. These directives aim at supporting students during their studies
providing them with personalized solutions to their problems related not only to the fruition of on-line
courses but also to orientation issues. In this paper we present a general architecture of an Embodied
Conversational Agent (ECA) that has the main aim to assist students by providing personalized suggestions.
This ECA represents the student interface in interacting with a Virtual University environment able to
provide different types of services. In particular to support mobility, the ECA architecture has been
conceived in order to run also on the student handheld device. The paper discusses the design and technical
issues involved in developing this personal agent and the results of an evaluation study aiming at assessing
the impact of conversational agents on the effectiveness of the interaction.
Personal Advisor, Embodied Conversational Agents, M-learning, Multiagent-Systems, Virtual Universities
In this paper we present a general architecture of an Embodied Conversational Agent (ECA) that has the main
aim to assist students by providing personalized suggestions. These suggestions can be related not only to the
fruition of on-line courses but also to orientation issues regarding student daily life.
The motivations at the basis of this work come from the directive of the Italian Universities that aims at
promoting interventions for improving mentoring and orientation services. This directive aims at following
students during the entire course of study, giving them personalized suggestions about different topics and
helping them to take decisions about their formative process. In particular, taking as a reference the role of an
advisor in our Department, his/her main goal consists in assisting and orienting the students focusing on:
suggestions about orientation choices, personalization of the study curricula, removal of “obstacles” during the
course of study, identification of appropriate and interesting research fields for the thesis, suggestions on how to
make students more participative in their formative process. At the first year, each student is assigned to an
advisor, chosen between available professors by the Department Council. Each advisor follows a group of
students during the entire course of study. After a trial period, we noticed that this service was not fully exploited
by students and we investigated the reasons related to this phenomenon. In order to assess the motivations, we
made a user study consisting in a questionnaire (150 subjects in total) aiming at understanding which was the
user’s expectations about the advisor role and at assessing why students were not using this orientation service.
Analyzing the result of this survey, it came out that the main reason was related to the difficulty to find the
advisor always available when needed (professors teach, do research, are involved in meeting, etc.) and also to
the shyness of some student that are afraid of asking questions about their problems to professors.
Taking these motivation and the ubiquitous computing vision (Weiser, 1991) into account, we decided to
develop VU-MAS, a Virtual University Multi-Agent System (MAS) with the aim of supporting students during
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain
the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work
owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists,
requires prior specific permission and/or a fee. Request permissions from the editors at [email protected]
the entire course of study. This system is able to support students by providing a interface to access to several
services by combining knowledge stored in distributed sources whenever needed.
Each student can interact with VU-MAS using a personal agent, called MyCoach, represented as an ECA. The
main goal of this agent is to monitor the student activities, following his/her learning improvements, but also to
select useful material according to the recognized student’s goals and needs. The agent is also capable to
proactively provide the student with useful suggestions whenever it is needed. As it is designed to run on a
smart phone or a PDA, this agent combines e-learning capabilities with mobile computing, thus realizing an mlearning experience where the student can feel always in touch with his advisor.
The paper is structured as follows: after a brief overview on the related work, we will briefly illustrate the VUMAS architecture, focusing on the description of the interaction aspects of the system. Then we will describe
how we extended the Mind-Body architecture, developed in the context of the MagiCster project (de Rosis et al.,
2003), in order to implement MyCoach. In order to assess the impact of the conversational agent on the
effectiveness of the interaction we performed an evaluation study whose results are reported in the related
section. The paper ends with some concluding remarks and some pointers to future work.
Related Work
Developing more human-like systems seems to improve interaction by establishing a more engaging relation
with the agent (Dehn and van Mulken, 2000). In this role, ECAs offer to people the possibility to relate with
computer media at a social level (Reeves and Nass, 1996) and, therefore, they have potential as facilitators for
this type of interventions thus making the interaction more effective and enjoyable (Moreno et al., 2001).
According to research and evaluation studies in the field of intelligent interfaces, ECAs (Gratch et al.,2002) have
shown to be a good interaction metaphor when acting in the role of counselors (Marsella, Johnson, & LaBore
2003), personal trainers (Bickmore, 2003), or healthy living advisors (de Rosis et al., 2003). Indeed, ECAs have
the potential to involve users in a human-like conversation using verbal and non-verbal signals for providing
feedback, showing empathy and emotions in their behavior (Cassell, 2001). Due to these features, an ECA can
be successfully employed as interaction metaphor in the pedagogical domain (Johnson, LaBore & Chiu, 2004)
and in other domains where is important to settle long term relation with the user (Bickmore & Picard, 2005).
In particular, as far as the interaction with a computational environment supporting the learning process is
concerned, the use of conversational agents can be envisaged in the following roles (Roda, Angehrn and Nabeth,
2001): i) as advanced helping facilities ; ii) as personal coaches equipped with specific domain knowledge; iii) as
role-playing actors in simulated environments (i.e. 3D virtual worlds). When an ECA acts as a personal coach it
has to show a personality appropriate to its role, the ability to guide the user, to reason about user actions, to
provide useful suggestions and to communicate through a realistic, interactive interface.
Moreover, even if ECAs have shown to have a good impact on settling an emphatic relation with the user (de
Rosis et al., 2005; Cassell et al. 2000), involving them in a deeper and intimate interaction, it is difficult to
communicate with these agents whenever needed (i.e. when the user is not in front of a computer but he/she has
the need to get suggestions and advices). For example, DESIA (Johnson, LaBore and Chiu, 2004), is a step in
this direction. This agent, presented in Carmen’s Bright IDEAS, has been adapted for running successfully on a
handheld device in order to assist in a psychosocial intervention aimed at providing training in problem solving
skills (Marsella, Johnson & LaBore, 2003). Therefore, if we want to support student in a continuous way the
personal advisor should be available and accessible also on a mobile device. Our work represents a step in this
VU-MAS Architecture Overview
A Virtual University should be able to support students from different points of view, during their formative
process. It has to help students not only with online courses and material, but also with logistic information and
tuition support. (Cunha, Tavares & Ferreira, 2005). Following this vision, the Virtual University MAS is
composed by agents that provide information services of different kind.
VA-m = Virtual Advisor
SA-k = Service Agent
MyC–n = MyCoach Agent
Requests /
Response /
Figure 1. VU-MAS architecture
The agents in VU-MAS are grouped into three main categories: MyCoach, Virtual Advisor and Service Agents.
MyCoaches (MyC-s) are personal agents that manage the student profile and the interaction between the student
and his advisor, as we will show in the next section. In VU-Mas there will be a personal agent for each student.
Virtual Advisors (VA-s) have the role of their real world counterpart. In fact, they represent the teachers when
they are in the role of advisors. They can access a knowledge base (KB) and use a reasoning engine to decide
what to suggest. The KB contains information about the students and the strategy used to advise according to
student features. VA-s can store every activity of the students, when they are connected to VU-MAS, either
using MyC agents or a web applications, updating the students’ profiles accordingly.
Service Agents (SAs) are specialized in providing a specific service, such as administration office, a library or
even a teacher, and they can be either simple request/response agents, or more complex ones.
Every human advisor can access and update the KB using his/her VA. He/she is responsible for a set of students.
The students use their personal device to connect to VU-MAS. This is a FIPA compliant MAS, in which the
communication between MyC and VA agents is made using ACL messages (FIPA,2006). The SAs can be either
agents or webservices (Greenwood, Buhler & Reitbauer, 2005). They answer to requests using standard
protocols, such as SOAP and HTTP (W3C, 2004).
In VU-MAS, each student is associated to one advisor. The first time the student has to download his MyC agent
on his personal device, from the VU-MAS web site. Starting from that moment a Virtual Advisor agent is
associated to the student who will be able to interact using MyC ECA and, according to the student year of
course, passed exams or specialization level, a stereotype is assigned to him/her. This stereotype is used to set
the initial student background knowledge and interests. After the agent initialization, the body of the user
stereotype is copied in the student profile, which is updated according to the student interaction with VU-MAS.
The profile is used for adapting the information to the student situation and recording the interaction history with
the system.
The profile of the students contains personal data (name, birthday, etc.), information about the student
curriculum (courses, exams, marks, etc.), preferences, hobbies and interests. Each MyC agent running on the
personal device manages and stores the profile of the student in a xml structures, formalized using a common
representation language. We decided to use the situational statement language UserML from UbisWorld
(Heckmann et al., 2005), since it allows to represent the user model in a very detailed structure, introducing
information useful for the contextualization. Furthermore, as privacy can be a crucial issue in this kind of
systems, UserML provides some tags to set privacy retention constraints.
Whenever a MyC agent needs to answer to some questions from the student, if it can be connected to the VUMAS, it contacts the VA of his advisor. Then MyC sends a request and the student profile to the VA, triggering
its tasks. The VA answers sending back the information on the basis of the student profile received, attaching the
needed resources. When the student asks some information, the VA looks for information asking to SAs. For
example a SA can manage information about courses, such as name, teachers timetables and so on.
MyC agent is supposed to be always connected to VU-MAS. However, if it is not possible to be online, or if the
student decides to disconnect, MyC agent has to be able to give some suggestions. In this case, using the natural
language generation strategies that will be shown in the next section, it gives an answer reusing information
already downloaded on the device during previous sessions, combining also the cached animations and sounds.
If these resources are not enough to answer the request, this is stored and the agent will give an answer when
connected. Than, when offline, MyC agent can answer to questions that it has already answered to or that have
some related links to previous requests, using its cache. In this way the interaction is much less engaging,
because the agent cannot exploit new knowledge, but can be used to remember previous answers. This is an
issue which will be overcome as well as the computational power increases, or broadband wireless connection
become more spread. During the interaction MyCoach keeps tracks of all the questions and answers to update
the student profile.
MyCoach: a Personal Advisor Agent
All the interaction described in the previous section is handled by MyCoach agent. This is an Embodied
Conversation Agent, with the aim of giving the agent a more engaging look and feel. As mentioned in the
introduction, MyCoach agent has been developed as an extension of the architectural schema developed in the
context of the MagiCster project. In this project, the ECA was intended as an entity made up of two main
components, a ‘Mind’ and a ‘Body’, interfaced by an I/O language, so as to overcome integration problems and
to allow their independence and modularity.
During the interaction, the Agent's Mind has the purpose of deciding what to communicate, by considering the
dialogue history, the conversational context and the communicative goal related to the user request. The output
of the mind module is an APML (Affective Presentation Markup Language) specification of the meaning to be
communicated (De Carolis et al., 2003).
The Body has the purpose to interpret and to render this output at the surface level, according to the available
communicative channels: different bodies may have different expressive capabilities and therefore may use
different channels. This approach was mainly driven by the definition of communicative function in Poggi et al.
(Poggi et al., 2000). In their theory, a communicative function is a (meaning, signal) pair, where the meaning
item corresponds to the communicative value of the signal item. For instance, a smile can be the signal of a
“joy” emotion.
Therefore, the Mind should convey only the meanings associated with the act to be communicated and the Body
should interpret these meanings and render them into an expressive behaviour according to the interaction
context and the device used in the interaction.
In the first version of the architecture, the APML interpretation process was embedded in the Body Player (De
Carolis et al., 2003). But, since the Agent’s believability is strictly related to features such as its personality and
role and the cultural and social context, separating the way in which the Body will render that meaning enforces
adaptivity .
Mobile devices place limitations on the functionality of an embodied conversational agent since their
computational and display capabilities are limited. Let’s see now how this architecture structure has been
adapted for the development of an ECA that runs also on an handheld computer platform (Figure 2).
User Input: Communicative Goal
Student profile
Body Animation
Figure 2. Mind-Body Architecture
Agent’s Mind
The task of this module is to generate the specification of the Agent’s move at the meaning level. Planning is a
task hard to compute, then it is not feasible to have a planner running on an handheld computer, moreover
MyCoach has to run on and off-line. For this reason, we adopted a schema based approach.
Given a communicative goal, derived by an interpretation of the user input, the Agent’s Mind selects from its
plan library the schema or a combination of schemas that allow to satisfy that goal. The result is an XAPML
(eXtended APML) specification of meanings to be communicated.
APML is a markup language designed to represent the communicative functions (De Carolis et al., 2003). Each
tag of this language is used to add some affective accent to the content of a sentence. For instance the
<performative> tag is used to specify what is the communicative speech act of the sentence (Pelachaud and
Poggi, 1998).
The extension of APML includes the following information that were not present in the first version: i) besides
the specification of what the agent has to communicate it is possible to specify the information background
(access to more details, visualization of relevant domain objects, etc.); ii) the focus attribute of the
<performative> tag allows to establish which is the main topic under discussion and to update the student profile
with information about what has been communicated to the student; iii) the voice attribute has been added since,
in case the student interact through its handheld device, the voice output has to be generated on the server side
and then passed to the client for being played together with the body animation. These recipes have been
designed on the basis of the results of a survey, in which the students were asked to rate a set of information
about the department and student life. Figure 3 shows results of second year students.
Analysing the survey results, major interests seem to concern information about modalities for choosing optional
exams for completing the third year curricula, accessing student facilities, setting an examination test, get credits
and recover from formative debits. Therefore, we developed XAPML schemas for answering to these families of
questions and for getting more detailed answers once selected a topic question. These schema allows to organize
the sequencing of the answer and to manage the multimodality of the interaction, combining text, voices, and
expressions of the face.
Nr of student
formative credits
modality of
sport facilities
a - where's the teacher's room
b - information about department labs
c - information about a course
d - modality of examination
e - personalized curricula
f - where can i find the programs
g - examination timetable
h - date of degree session
i - bureaucracy about thesis
j - modality of degree sessions
k - modality of registration
l – deadlines
m - office's location
n – scholarship
o - modules and forms
p – stage
q - suggestions about studying
r - dates and rooms of seminars
s - how to acquire formative credits
t - how to recover from formative debts
u - how to record exams
v - information about EDISU
w - information about sport facilities
x - colleges and restaurants
y - student organized events
z - examination rooms
Figure 3. Results of the questionnaire of second year students
The following is a recipe for providing personalized suggestions about “optional courses”:
<XAPML-recipe goal=”suggestion-choose-course”>
[FORALL optional_course(topic-i) DO
IF NOT(HasInterest(S,topic-i)) DO
<performative type="inform" affect="sorry-for" voice="sorry.mp3"> I’m sorry for
you but</performative>
<performative type="suggest" focus=”optional_course(topic-i)” voice="coursei.mp3"> I have to suggest you to follow the course [topic-i]</performative>
<performative type="inform" voice="motivation-i.mp3"> even if I know you don’t like
[SuperClass(topic-i) OR Related(topic-i)]</performative>
IF (HasInterest(S,topic-i))
<performative type="inform" affect="happy-for" voice="happy.mp3"> I’m happy
<performative type="suggest" focus=”optional_course(topic-i)” voice="coursei.mp3"> to suggest you to add the course of [topic-i] to your curriculum
<performative type="inform" voice="motivation-i.mp3"> since I know that you like
[SuperClass(topic-i)OR Related(topic-i)]</performative>
[FORALL optional_course(topic-i) DO
IF detail <object type=”link” label="[topic-i]">[URI-topic-i]</object>
These schemas are applied by the Mind in order to generate the instantiated XAPML specification. In the
previous example, square parenthesis represent instructions to the XAPML generator that will substitute them
with the appropriate text by manipulating information in the Domain Knowledge Base. This transformation is
executed on the server side and not on the handheld computer where, as we will see later on, only instantiated
XAPML specification are interpreted and rendered at a surface level.
Agent’s Body
When the student downloads the agent from the Department Web Site she can choose its Body according to
his/her preferences. The selected body, obviously, influences the way in which the agent is able to express
Since our main aim was not to develop a new ECA player but to test the validity of our approach, at the moment
we have developed only two types of bodies (Table 2): a face (male professor and a young female professor
assistant) and a character (a funny ball). This last one could be used, for instance, for interacting with children.
Each ECA conveys the same meanings using signals that are typical of its body. For instance the two faces will
use as signal channels those typical of a face (eye, eyebrow, mouth, gaze, etc.) while the funny ball will use
movements, colors and changes in its dimensions as communicative channels. The XAPML specification is not
directly interpreted by the player that is the same for all the different bodies. This is possible since the proposed
architecture decouples meanings from signals.
Each body has a conditional meaning-signal table that allows to appropriately translate an XAPML tag into a
Signal Expression Markup Language (SEML). SEML tags define the expressions that can be performed on each
channel of the Body. Table 1 specifies how a body’s <act> can be specified in SEML along the appropriate
channels for the face (Gratch et al., 2002; Poggi et al., 2000) and the ball.
Table 1. SEML signals expression for two different bodies
Face Body
Funny Ball
<play channel="eye" value=" " />
<play channel="eyebrow" value=" "/>
<play channel="mouth" value=" "/>
<play channel="gaze" value=””/>
<play channel="voice" value=“ "/>
<play channel="text" value=""/>
<play channel="shape" value=""/>
<play channel="color" value=""/>
<play channel="size" value=""/>
<play channel="look" value=""/>
<play channel="sound" value=""/>
<play channel="text" value=" "/>
An example of SEML output for both the “professor” and the “funny ball“ bodies, deriving from a
transformation of a XAPML performative, is shown in Table 2.
Table 2. Example of transformation from XAPML to SEML
<performative type="inform" affect="sorry-for" voice="sorry.mp3”>I’m sorry for </performative>
The professor:
The funny ball:
<play channel="mouth" value="serious"/>
<play channel="shape" value="deflated"/>
<play channel="voice" value=“sorry.mp3"/>
<play channel="color" value="grey"/>
<play channel="text" value="I’m sorry!"/>
<play channel="size" value="1"/>
<play channel="sound" value="cry.mp3"/>
<play channel="text" value="I’m sorry!"/>
I’m sorry!
I’m sorry!
The first row shows the XAPML input, the second one the SEML specification for the two bodies and the third
one their rendering by the player. In this example, the face body will show a typical human-like expression of
sorry-for, while the funny-ball will express it by changing its shape (deflated) and color (grey instead of its
normal bright colors).
An Example of Interaction
VU-MAS allows to access different types of information and services and can be used as an interface for online
courses. Students can interact with it using MyCoach as an interface. According to the type of student request the
agent will adapt the presentation to the student’s profile, which contains his/her preferences, skill level, progress
in attending a course, etc.
In order to show an example of interaction, let us assume, as a scenario example, the following one:
Vincenzo, a second year student, needs some help in choosing some optional exams to organize his
study curriculum. His personal profile, besides his personal data, contains the information about
his student curriculum (given exams and relative marks), interests and preferences. In particular,
analysing these data, it is possible to infer that he is interested in Videogames and that he is
skilled in Programming while he is not very good in Mathematics.
Therefore, MyCoach has to suggest him the courses that, according to Vincenzo’s profile, are the most
appropriate to his skills and previous years curriculum.
Once the personal advisor is activated by the student, it greets him and prompts a list of tasks on which it could
help (Figure 4a). From available options the student chooses the course planning help, by clicking on the relative
link. The possible choices are planning of the official and personalized curricula. Let’s suppose that the student
selects the personalized curriculum option (Figure 4b). Obviously, the student can insert a free text question
using the keyboard icon if the list of options does not contain what he was looking for.
Figure 4. MyCoach a) greeting the student, b) proposing help about completing the third year curriculum
MyCoach will then contact the VA responsible for Vincenzo and will send to this agent the request and the
statements in the student profile that are relevant to this goal. In this scenario the triggered communicative goal
will be “Suggest(MyCoach,Vincenzo,Choose-Course)” and therefore the corresponding XAPML schema shown
in the previous Section. This schema will be instantiated then with a list of courses that fit the user profile. To
this aim, the VA asks to the SA responsible for curriculum planning the list of possible courses that will be
extracted from the related KB.
I year
II year
III year
NM for Computer Graphics
Interactive 3D Env
Video Games
Figure 5. Diagram describing the structure of the Courses KB.
Figure 5 illustrates a simple diagram representing the Courses KB. As we can see from the diagram, and
considering the information in the student profile, a list of possible courses is determined. Thereafter, MyCoach
suggests a list of possible courses to insert in the curriculum as shown in Figure 6.
In the given scenario, the course Numerical Methods for Computer Graphics is considered useful, because it is a
prerequisite for the course of Interactive 3DEnvironment, which is connected to the student interests in
Videogames. However, since the student has got a bad mark in Mathematics, therefore this is not one of his
preferred topics, the expression of the body of the agent for the first proposal is a sad face (Figure 6 a) and b)),
while the second suggestion is communicated in a happy mode (Figure 6 c) and d)). The instantiated XAPML
output generated for the illustrated example is the following:
<performative type="inform" affect="sorry-for" voice="sorry.mp3"> I’m sorry for
you but</performative>
<performative type="suggest" focus=”optional_course(Numeric Methods for Computer
Graphics)” voice="NMCG.mp3"> I have to suggest you to follow the course of Numeric
Methods for Computer Graphics </performative>
<performative type="inform" voice="motivation-NMCG.mp3"> even if I know that you
don’t like Mathematics</performative>
<performative type="inform" affect="happy-for" voice="sorry.mp3"> I’m
<performative type="suggest" focus=”optional_course(Interactive 3D Environment)”
voice="I3DE.mp3"> to suggest you to add Interactive 3D Environment to your
<performative type="inform" voice="motivation-I3DE.mp3"> since I know that you
really like VideoGames</performative>
<object type=”link” label="NM for Computer Graphics">LINK</object>
<object type=”link” label="Interactive 3D environments">LINK</object>
<object type=”link” label="Computer Graphics">LINK</object>
<object type=”link” label="More…">LINK</object>
Figure 6 shows how the schema is played by the agent’s body when suggesting these courses to the student.
Figure 6. a) and b) “happy-for” expression; c) and d) ”sorry-for” expression
The student can always access his profile to modify his preferences or updating his curriculum clicking on an
icon in the lower bar. When the user searches for details in a different page, it is always possible to go back and
talk to the agent by selecting the icon on the bottom status bar of the handheld application. Moreover it is
possible to change the attitude of the advisor, selecting different faces.
Recently embodied conversational agents have been widely used in e-learning projects, even if there are no
proved evidence that the use of an ECA can improve the learning experience. To support our proposed approach
we have developed a working prototype and we have conducted some evaluation studies with the main goal of
assess the effectiveness of the system, in term of level of believability and reliability, perceived by the students.
Moreover, this study was aiming also at assessing the student attitudes toward the agent in terms of how much
the presence of an embodied agent improved the effectiveness and engagement of interaction. This has been
evaluated through the use of a questionnaire aiming at understanding how much the student liked the character
and how much they found helpful MyCoach suggestions.
To this aim we have conducted a test with two groups of ten students of the second year of our Department. Both
groups had to execute the task of planning their courses for the third year.
Figure 7. User Profile interface
The first group was asked to do it using a PDA and MyCoach personal advisor as interface. The second group
was asked to perform the same task using the web pages of our Department. These are mainly a collection of
printable files, so they can be considered similar to a paper guide.
In order to give to MyCoach the capability to provide personalized suggestions we asked to the students
belonging to the first group first to create their profiles using a simple user interface (Figure 7) and then to
interact with MyCoach.
easy of
student guide
mean rating
Figure 8. Comparison of features between agent and students webpages
After the experiment, a post-test questionnaire (Appendix 1) was administered to measure the impact of the
suggestions about courses on participants’ subsequent choice. This comprised a range of different measures
concerning both informational and interaction aspects. To assess the effectiveness of the ‘information provision’
aspect of the message, we included ratings of satisfaction and ease of understanding (both on five-point unipolar
Likert scales). In addition for the first group, questions about reliability and believability of the ECA were
For comparing the data obtained from questionnaires we used t-test. As it could be seen from Figure 8, the level
of trustworthiness is lower for MyCoach when compared to webpages of the department. The difference between
means is statistically significant for p=0,05 where t-test value is t1(18)=-1,98. Moreover the ‘easy of
understanding’ feature is higher for written text in the website. It is confirmed by t-test with t2(18)=-1,5.
However difference between means becomes statistically significant for p=0,1. In our opinion this can be due to
the fact that subjects had more time for reading written information whereas MyCoach advices were spoken.
However, the grade of satisfaction and likelihood is higher for the agent. The t-test values are t3(18)=1,63 and
t4(18)=2,69 respectively. The first feature is statistically significant at p=0,1 level and the second at p=0,05 level.
In fact, the webpages of our department do not permit personalization, but contains a static list of course names
without any link to student preferences or attitudes.
As mentioned earlier, the questionnaire given to the first group of students contained a set of items to measure
participants’ perceptions towards the agent. Participants were requested to rate the agent along the following
dimensions: helpfulness, likeability, reliability and competency. All ratings were made using five-point unipolar
likert scales.
mean rating
Figure 9. Measure of some dimension of the agent behaviour
The graph (Figure 9) shows that the reliability level has been evaluated with a lower score, even if the students
considered the use of the personal agent helpful and likeable.
Students think that the agent is helpful even if they showed to trust more paper based information. Unfortunately
the reliability and trustworthiness of a system like this often relays on a long term experience. In our tests users
interacted with the agent for a short period of time, which perhaps was not enough to create this relationship. In
this view, improving the social and affective behaviour of the agent might lead to a higher level of reliability and
Implementation Issues
VU-MAS is a MAS developed using JADE, a FIPA compliant framework to build multi agent system in java
(JADE 2006), and its extension LEAP for handheld devices. MyCoach is one of the agents in this MAS. The
Service Agents can be either agents or web-services, which can be connected to the MAS using a gateway
(Greenwood, Buhler & Reitbauer, 2005).
MyCoach is implemented combining Macromedia Flash ActionScript with the .NET Compact Framework using
Visual Basic.
The interaction may be performed on or off-line; when the user is on-line, the computation of the MIND output
is performed server side and the rendering on the handheld (client-side). In this case, when the user connects to
the server and asks for specific information the related communicative goal is passed to the system that computes
and sends back the XAPML specification of the communication, the mp3 file corresponding to the voice output
generated by a TTS on the server side and the set of frames needed for that animation if not yet available on the
client side. When the agent is off-line uses pre-stored data to answer to questions, as described in the section
about the architecture.
On the client side, MyCoach composes and plays a combination of animation and sound resources loaded onto
the Pocket PC.
As in DESIA, with whom this system has a lot of similarities from the implementation point of view, we store
the different animations for each body channel. A scene is a collection of sequences of animation frames that are
composed in real time as needed according to the SEML specification. Obviously, one SEML specification is
then translated into a sequence of frames that are synchronized with the mp3 files by the SEML-Wrapper
MyCoach is a personal agent integrated into a Virtual University MAS. This is a system that has to support the
students along their studies by providing orientation suggestions and information useful to solve their problems.
The agents cooperate together to reach the main goal of finding useful information, which seems to be one of the
most important issues to improve the student study career.
MyCoach is an animated agent designed to be consulted on a handheld device. For this reason we decided to use
an extension of the Mind-Body architecture already used in the context of the MagiCster project. In particular,
the Mind, running on the server side, decides what to say and generates, through the use of plan “recipes”, a
specification (in XAPML) of the meaning to be communicated by the Body that can be used to generate the
agent behaviour on the handheld computer (client side). Since the architecture can support the use of different
“bodies”, that can express meanings using different signals according to their available channels, a meaningsignal table specifies how a specific body has to convey particular meanings. This is represented through the use
of another markup language SEML that specify which combination of signal to employ in correspondence of
each XAPML tag. Then, the SEML specification is interpreted by a Wrapper, developed in Flash for MyCoach
application, that is responsible for the synchronization and the rendering of the agent animations.
We are aware that MyCoach agent animations are very simple, but this was a first step in order to test how the
architecture could support this kind of interaction.
The evaluation studies results are promising, so we are planning to develop more characters with different bodies
and interaction attitudes in order to let a higher level of personalization. This is important not only for practical
reason, but also because the more the agent can be personalized the more engaging the interaction could be. In
particular, we plan to add to MyCoach mind the capability to use motivational and affective strategies to
encourage the student when needed.
Further evaluation studies will be done to investigate other features of the system. For example it could be
interesting to compare the suggestions of our personal agent with those of a real advisor.
Bickmore, T. (2003). Relational Agents: Effecting Change through Human-Computer Relationships. PhD
Thesis, Media Arts & Sciences, Massachusetts Institute of Technology.
Bickmore, T. W., & Picard, R. W. (2005). Establishing and maintaining long-term human-computer
relationships. ACM Transactions on Computer Human Interaction, 12 (2), 293-327.
Cassell, J., Sullivan, J., Prevost, S., & Churchill, E. (2000). Embodied Conversational Agents, Cambridge, MA:
MIT Press.
Cassell, J. (2001). Embodied conversational agents: representation and intelligence in user interfaces. AI
Magazine, 22 (4), 67-83.
Cunha, M. M., Tavares, A. J., & Ferriera, L. (2005). Infrastructures for the Virtual University, EUNIS 2005 Leadership and Strategy in a Cyber-Infrastructure World, 21-24 June 2005, Manchester, United Kingdom.
De Carolis, B., Pelachaud, C., Poggi, I., & Steedman, M. (2003). APML, a Markup Language for Believable
Behavior Generation. In H Prendinger and M Ishizuka (Eds), Life-like Characters. Tools, Affective Functions
and Applications, Berlin: Springer.
de Rosis, F., Cavalluzzi, A., Mazzotta, I., and Novielli, N. (2005). Can embodied conversational agents induce
empathy in users? AISB'05 Virtual Social Characters Symposium, 12-15 April 2005, Hatfield, United Kingdom.
de Rosis, F., De Carolis, B., Carofiglio, V.,& Pizzutilo, S. (2003). Shallow and inner forms of emotional
intelligence in advisory dialog simulation. In H Prendinger & M Ishizuka (Eds.), Life-like Characters. Tools,
Affective Functions and Applications, Berlin: Springer.
Dehn, D. M., & van Mulken, S. (2000). The impact of animated interface agents: a review of empirical research.
International Journal of Human Computer Studies, 52 (1), 1-22.
Ekman, P. (1993). Facial expression of emotion. American Psychologist, 48, 384-392.
FIPA Home Page (2006), http://www.fipa.org.
Gratch, J., Rickel, J., Andre, J., Badler, N., Cassell, J., & Petajan, E. (2002) Creating Interactive Virtual
Humans:Some Assembly Required. IEEE Intelligent Systems, July/August, 54-63.
Greenwood, D., Buhler, P., & Reitbauer, A. (2005). Web Service Discovery and Composition using the Web
Service Integration Gateway. Proceedings of the 2005 IEEE International Conference on e-Technology, eCommerce and e-Service, 29 March – 1 April 2005, Hong Kong.
Heckmann, D. (2003). Introducing Situational Statements as an integrating Data Structure for User Modeling,
Context-Awareness and ResourceAdaptive Computing, ABIS 2003, Karlsruhe, Germany, 283-286.
Heckmann, D., Schwartz, T., Brandherm, B., Kröner, A. (2005). Decentralized User Modeling with UserML and
GUMO. Proceedings of the Workshop on Decentralized, Agent Based and Social Approaches to User Modelling
(DASUM 2005), Edinburgh, Scotland, 61-65.
JADE (2006). Java Agent Development Framework, http://jade.cselt.it/.
Johnson, W. L., LaBore, L., & Chiu, Y. C. (2004). A Pedagogical Agent for Psychosocial Intervention on a
Handheld Computer. AAAI Fall Symposium on Dialogue Systems for Health Communication, October 22-24,
2004, Washington DC, USA.
Marsella, S. C., Johnson, W. L., & LaBore, C. M. (2003). Interactive pedagogical drama for health interventions.
In U. Hoppe et al. (Eds.), Artificial Intelligence in Education: Shaping the Future of Learning through Intelligent
Technologies, Amsterdam: IOS Press, 341-348.
Moreno, R., Mayer, R. E., Spires, H., & Lester, J. (2001).The case for social agency in computer-based teaching:
Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction,
19, 177-213.
Pelachaud, C., Carofiglio, V., De Carolis, B., de Rosis, F., & Poggi, I. (2002). Embodied Contextual Agents for
Information Delivery Applications. Proceedings of AAMAS'02, July 15-19, 2002, Bologna, Italy.
Pelachaud, C. (2005). Multimodal expressive embodied conversational agents. In Proceedings of the 13th
Annual ACM international Conference on Multimedia, ACM Press, New York, NY, 683-689.
Pelachaud, C., & Poggi, I. (1998). Multimodal communication between synthetic agents. In Proceedings of the
Working Conference on Advanced Visual interfaces, May 24-27, 1998, L'Aquila, Italy.
Poggi, I., Pelachaud, C., & de Rosis, F. (2000). Eye communication in a conversational 3D synthetic agent. AI
Communications, 13 (3), 169-181.
Reeves, B., & Nass, C. (1996). The Media Equation, New York: Cambridge University Press.
Roda, C., Angehrn, A., & Nabeth, T. (2001). Conversational Agents for Advanced Learning: Applications and
Research. Proceedings BotShow 2001, June 14, 2001, Paris, France.
W3C Web Services Architecture Working Group Note (2004). http://www.w3.org/TR/ws-arch/.
Weiser, M. (1991). The Computer for the 21st Century. Scientific American, September, 94-104.
Appendix 1
Personal Information:
a) Are you a student of : first year[ ]
b) Age: ____
c) Sex: ____
second year [ ]
third year [ ]
Please answer this questions:
1) On the basis of what you have just heard how likely are you to follow the recommendations for
planning your courses?
Not at all
2) How satisfied were you with the suggestion contained in the message?
Not at all
3) How easy did you think that the message was to understand?
Not at all
4) How trustworthy did you think the content of the message was?
Not at all
(Only for the group using the Agent Modality)
For each word below, please indicate how well it describes the computer character that you have
just seen. Note that you are now evaluating the character, NOT the message.
a) Helpful
Not at all
b) Likable
Not at all
c) Reliable
Not at all
d) Competent
Not at all
End of questionnaire.
Lanzilotti, R., Ardito, C., & Costabile, M. F., & De Angeli, A. (2006). eLSE Methodology: a Systematic Approach to the eLearning Systems Evaluation. Educational Technology & Society, 9 (4), 42-53.
eLSE Methodology: a Systematic Approach to the e-Learning Systems
Rosa Lanzilotti, Carmelo Ardito and Maria F. Costabile
Dipartimento di Informatica, Università di Bari, 70125, Bari, Italy
[email protected]
[email protected]
[email protected]
Antonella De Angeli
School of Informatics, University of Manchester, PO BOX 88, M601QD, UK
[email protected]
Quality of e-learning systems is one of the important topics that the researchers are investigating in the last
years. This paper refines the concept of quality of e-learning systems and proposes a new framework, called
TICS (Technology, Interaction, Content, Services), which focuses on the most important aspects to be
considered when designing or evaluating an e-learning system. Our proposal emphasizes user-system
interaction as one of such important aspects. Guidelines that address the TICS aspects and an evaluation
methodology, called eLSE (e-Learning Systematic Evaluation) have been derived. eLSE methodology
combines a specific inspection technique with user-testing. This inspection, called AT inspection, uses
evaluation patterns, called Abstract Tasks (ATs), that precisely describe the activities to be performed
during inspection. The results of an empirical validation of the AT inspection technique, carried out to
validate this technique, have shown an advantage of the AT inspection over the other two usability
evaluation methods, demonstrating that Abstract Tasks are effective and efficient tools to drive evaluators
and improve their performance.
E-learning systems, Quality, Evaluation, Controlled experiment
E-learning is becoming very important in fields where access to learning materials needs to be brought about
effectively and efficiently. Its “any time, any place” nature could be a winning strategy for particular needs, such
as decongestion of overcrowded education facilities, support for learners or lecturers who live far from schools
and universities, life-long education. When making remote data and tools available to users it is necessary to
consider their different characteristics, such as cultural background, technical experience, technological
equipment, and physical/cognitive abilities. In the e-learning context, a major challenge for designers and
Human-Computer Interaction (HCI) researchers is to develop software tools that can engage novice learners and
support their learning even at a distance. Towards this end, there should be a synergy between the learning
process and the learner’s interaction with the software. As for any interactive system, usability is a primary
requirement. If an e-learning system is not usable, the learner spend more time learning how to use the software
rather than learning the contents. Beside being usable, an e-learning system must be effective in meeting the
instructor’s pedagogical objectives. System evaluation should thus integrate an assessment of the educational
quality aspects of e-learning systems.
Despite the large number of e-learning systems now available, one of the barriers to successful deployment of
technology-based learning is the lack of high quality systems tailored to the needs of individual users and
groups. Quality, from the Latin qualis, which means a pleasant thing, is an abstract term that assumes specific
meanings according to the context in which it is used. From the end of the 1970s, in the software engineering
context, some factors have been introduced as measures of the software quality. McCall affirms that quality
factors represent attributes or characteristics of the software that a user or a client of the software couples with
the quality of the software (McCall, 1994). Details on the first studies on quality factors can be found in
(McCall, 1994; Boehm, 1978).
When speaking of quality, it is important to consider the regulations for quality certification. In particular, the
ISO/IEC 9126 establishes standards for ensuring the quality of a software product (ISO 9126, 1991),
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
emphasizing that the quality is an attribute that depends on the users, the context, the goal, and the cost of the
In the last few years, we have been involved in e-learning projects and faced several problems that impact
learner-centred design, which presents its peculiarities with respect to general user-centred design (Quintana et
al., 2001; Costabile et al., 2003; Ardito et al., 2004; Ardito et al., 2006). The evaluation of e-learning systems
deserves special attention, and evaluators need appropriate guidelines as well as effective evaluation
methodologies (Zaharias et al., 2002). Unfortunately, the number of studies addressing evaluation of e-learning
systems is relatively small and inadequate to the importance of the issue (Squires and Preece, 1999; Quinn et al.,
2005). Moreover, it is often the case that the evaluation criteria are only vaguely stated (Parlangeli et al., 1999;
Squires and Preece, 1999; Wong et al., 2003), so that an actual measurement of the system quality is left to
subjective interpretation.
In this paper, we propose a new framework for quality of e-learning systems, called TICS (Technology,
Interaction, Content, Services), which focuses on the most important aspects to be considered when designing or
evaluating an e-learning system. TICS makes explicit the importance of user-system interaction that is neglected
by other authors, who implicitly assume that the user interface of an e-learning system does not influence user
activities (Moore, 1989; Ewing and Miller, 2002). Actually, making the user interface ‘transparent’ is the
ambition of any interactive system. In this way, users could concentrate only on their tasks. People who have
experience in e-learning know the difficulties of users when interacting with an interface that is not usable. Many
e-learning systems provide various functionalities and services, but it is often difficult, if not impossible, for
users to find and use them in an effective and efficient way. Thus, our definition of e-learning systems quality
highlights the importance of this aspect in designing and/or evaluating the overall quality of these systems. If
designers are fully aware of the high value of the user interface, they will not neglect this aspect of e-learning
systems and will create products of better quality, which will contribute to the success of technology-based
TICS aims at highlighting the main aspects that contribute to the quality of e-learning systems, so that designers
and evaluators can focus on such aspects, thus providing what is required for good quality system. As a further
contribution we have derived guidelines that address the TICS aspects and an evaluation methodology, called
eLSE (e-Learning Systematic Evaluation). eLSE methodology combines a specific inspection technique with
user-testing. This inspection aims at allowing inspectors that may not have a wide experience in evaluating elearning systems to perform accurate evaluations. It is based on the use of evaluation patterns, called Abstract
Tasks (ATs), which precisely describe the activities to be performed during inspection. For this reason, it is
called AT inspection. An empirical validation of the AT inspection technique has been carried out: three groups
of novice inspectors evaluated a commercial e-learning system applying the AT inspection, the heuristic
inspection, or user-testing. Results have shown an advantage of the AT inspection over the other two usability
evaluation methods, demonstrating that Abstract Tasks are effective and efficient tools to drive evaluators and
improve their performance.
The paper has the following organization. First, contributions of some researchers in the domain of e-learning
systems evaluation are reported. Then, the TICS framework is illustrated, highlighting the most important
aspects to consider in designing and/or evaluating the overall quality of e-learning systems. Consequently, the
eLSE methodology is presented and the validation of the AT inspection with its results is reported. Finally, the
section Conclusion closes the paper.
Quality of e-learning systems
Various definitions and frameworks for the quality of e-learning systems are reported in literature, but the
identified solutions appear as a short blanket, able to cover only some of the multiple aspects that characterize
the complexity of the problem.
Quality is defined as “the totality of characteristics of an entity that bear on its ability to satisfy stated and
implied user needs” (ISO 9126, 1991). It includes a new quality model distinguishing between three different
approaches to product quality:
¾ external quality, which is measured by the dynamic properties of the code when executed (such as response
¾ internal quality, which is measured by the static properties of the code, typically by inspection (such as path
quality in use, which is measured by the extent to which the software meets the needs of the user in the
working environment (such as productivity).
External quality is the result of the combined behaviour of the software and the computer system, while quality
in use is the effectiveness, productivity and satisfaction of the user when carrying out representative tasks in a
realistic working environment. External measures can be used to validate the internal quality of the software.
Quality in use measures the degree of excellence and can be used to validate the extent to which the software
meets user needs. Appropriate internal attributes of the software are a pre-requisite for achieving the required
external behaviour, and appropriate external behaviour is a pre-requisite for achieving quality in use (ISO 9126,
Chua and Dyson propose the ISO/IEC 9126 Quality Model (ISO 9126, 1991) to evaluate e-learning systems
(Chua and Dyson, 2004). It provides an indication to educators and educational administrators of the quality of a
system they are considering buying and provides a basis for comparison of different systems. However, for
software developers without educational expertise, the ISO model alone would not be sufficient because it is a
general software quality model and does not specify the particular teaching and learning activities needed for
effective learning. Moreover, ISO/IEC 9126 presents some inherent weaknesses, particularly with regards to the
usability characteristic. The authors suggest that this characteristic should be extended to include more specific
factors such as consistency, simplicity, legibility. In addition, they propose the inclusion of user satisfaction as a
global characteristic to summarise the general impact of the system on the users in their specific educational
context (Chua and Dyson, 2004).
Some authors have highlighted that quality in e-learning needs to be envisaged from different perspectives.
However, they look at the quality primarily from a pedagogical point of view and emphasize the learner’s
perspective (Ehlers, 2004), while they do not analyze in depth aspects related to the design of interactive
software systems.
Accordingly, Ehlers (2004) suggests that subjective quality requirements be structured in 7 fields of quality (see
Figure 1):
¾ Quality Field (QF) 1 (Tutor Support) considers the preferences that learners have for communication and
cooperation with the tutor of an online course.
¾ QF 2 (Cooperation and Communication in the Course) contains quality requirements for the course that
learners express, that concern the communication and cooperation environment in which they work with
other learners in learning groups, with experts or the tutor.
¾ QF 3 (Technology) is also important to learners. If technical requirements are fulfilled they do not raise the
perceived quality very much - as they are taken for granted. Yet if the expected technical standards are not
met the learners quality assessment decreases.
¾ QF 4 (Costs-Expectations-Benefits) refers to the information possibilities learners have about a course or the
institution/organization which is offering the course.
¾ QF 5 (Information Transparency of Provider/Course) contains the provision of formal and standard
information as well as individualized counseling on course contents, learning methodology or technical
¾ QF 6 (Course structure) contains learners’ requirements concerning the structure of an e-learning course.
Learners’ quality preferences clearly show that the presence of lessons as part of an e-learning course
(blended learning) is of high importance to certain groups of learners, whereas others do not regard them as
¾ Finally, QF 7 (Didactics) covers aspects of content, learning goals, methods and materials. Experienced elearners are often very precise in their requirements concerning the didactical setting of an e-learning course.
The quality fields are subdivided into 30 dimensions, each of them represents a set of criteria of learners’
preferences that are clustered in a dimension on the basis of empirical evidence.
As Ehlers, Rovinskyi and Synytsa have suggested to perform multidimensional quality evaluation according to
the parameters identified as important by the learner (Rovinskyi and Synytsa, 2004). These parameters will be
hereafter named quality metadata and will actually be a part of metadata for a course or derived from the
metadata of its components. Quality metadata may be used to measure the correspondence of a course to the
needs and expectations of a learner, thus determining the course’s quality from a learner’s perspective. Quality
metadata fall into the following groups: learning goals (learning objectives and tasks, curricula and certificates);
instructors and experts (their duties, qualification, and way of interaction); target audience (required knowledge
and skills, prior education or professional background, learning groups and interaction); learning environment
(required hard and software, additional plug-ins, complexity of environment, navigation, help, assessment
instruments and feedback); learning resources (types of resources, availability on/off line, variety of
representation, composition and complexity, types of activity offered, learning material completeness).
Figure 1. Model of subjective quality requirements from [Ehlers, 2004]
Other researchers have proposed to consider technological as well as educational aspects in evaluating e-learning
systems. In (Herrington et al., 2001), a framework is presented as a checklist enumerating what are considered to
be critical elements of quality learning environments. The checklist is based on the determination of critical
elements within three main areas:
¾ the pedagogies area, referring to the activities that are fundamental for the learning process: the learning
activities, proposed by the e-learning systems, would involve tasks that reflect the way in which the
knowledge will be used in the real life settings; the learners would have the opportunity to work
collaboratively; the lecturer’s role would be considered as a coach rather than an instructor; and, finally,
authentic and integrated assessment tests should be used to evaluate learner’ achievement;
¾ the resources area, concerning the content and information provided to the learners: resources should be
organized in ways that make them easily accessed. They should be current and based on literature reviews
by lecturers and they should present various viewpoints to allow learners the opportunity to judge the merit
of different arguments;
¾ the delivery strategies area, involving issues associated with the ways in which the course is delivered to the
learners: the learning material should be accessible and available to all learners, including people with
special needs, such as people with physical disabilities, or geographically isolated learners. Moreover, the
learners should be count on the reliability of the technology: the e-learning system should allow learners to
navigate and download materials within a reasonable period of time, and learners should be able to use
various technologies to communicate and collaborate with each other and their teachers, and so on.
In a publication that has been approved and publicized by the Italian Ministry of Education, Montedoro and
Infante indicate three quality dimensions of e-learning systems: technology, content and services (Montedoro
and Infante, 2003) Concerning technology, the educational process has to be supported by an on-line platform
that manages the educational material. The platform has to be compatible with the international standards
(AICC, SCORM, etc.) and to manage functions such as: analysis of the user profiles, building of personalized
educational paths, the planning of the learning activities, the management of on-line courses and educational
materials, interaction and in particular, forums, chat, virtual classrooms, and communications among learners.
Regarding content, it is important to perform an accurate educational design that is learner-centered. It is
characterized by the following elements: user orientation, the ease of navigation and fruition of the educational
products and resources, interaction in all the phases if the educational process is enriched by specific and
continuous feedback, length of the didactic unit not exceeding twenty minutes, content modularity, evaluation of
the educational results. Finally, the services dimension increases the learner’s possibilities of contacting the
tutor. The tutor has the task of managing the relations with the learners and guaranteeing rapid answers.
Grutzner et al. state that the quality of e-learning systems is affected by four dimensions: the content of learning
materials; the presentation of these materials; the pedagogic content, i.e., the way in which materials are taught;
the overall functionality of the courseware. All four dimensions have to be considered at the same time and
continuously throughout the courseware life-cycle to ensure high quality of the final product and thus to
facilitate learning (Grutzner et al., 2004). The authors propose IntView, a systematic courseware development
process, designed to support and integrate all of the different perspectives and views that are involved in the
process of designing high quality courseware. The courseware engineering life-cycle model of the IntView
methodology is accompanied by four engineering methods that assure the quality of the output artefacts of each
phase. These methods include perspective-based inspections, prototyping, tests, and both formative and
summative evaluations. Thus, the whole life-cycle is encompassed by verification and validation tasks that are
assigned to different roles.
In one of the early papers in distance education, Moore explicitly speaks of user interaction as an important
aspect to be considered for the quality of e-learning systems and proposes a quality analysis model that considers
three types of interaction (Moore, 1989). First of all learner-content interaction is considered, since it is the
process of interacting with content that brings about changes in the learner's understanding and perspective.
Learner-lecturer interaction is also important, because lecturers seek to stimulate the learner’s interest and to
motivate the student to learn. Learner-learner interaction among members of a class is the third type of
interaction, since it is often an extremely valuable resource for learning. Like Moore, Fata affirms that for
evaluating the quality of e-learning systems it is important to consider three different types for interaction (Fata,
2003): interaction between the learners and didactic material, the educational material just be of good quality,
pleasant, easily usable, and equipped with an accurate and rich bibliography; interaction among learners, tutor,
and experts: the tutor has to favour discussions among the learners, to reduce the learners’ sense of isolation.
During the course, the learners must constantly be able to perceive the tutor’s presence; and during the
conclusion of the course, the tutor must stimulate meta-cognitive reflection; and finally, interaction among
learners: the communications among learners should take into account not only the discussions as group
activities but also the free communications, in order to encourage a greater familiarity among learners. Other
researchers have spoken of interaction as an important aspects for quality e-learning systems (Scanlon et al.,
2000; Muirhead, 2001; Ewing and Miller, 2002), but it is worth noticing that they focus on the interaction
between learner and educational content.
All these types of interaction are mediated by the software system through which the content is delivered. As
HCI researchers, we know the importance of designing the interface to allow a good interaction between the user
and the provided content, which is the basis for effective learning. We must suggest models and techniques that
support e-learning system designers in creating quality products.
TICS framework for the quality of e-learning systems
With respect to the other definitions of e-learning systems quality, we emphasize the interaction dimension and,
specifically, the interaction between the user (teacher or learner) and the overall system, not only its content (the
learning materials). On the basis of our experience, we believe that in the e-learning context, design
methodologies and techniques have to ensure a user-system interaction that facilitates the learning process. It is
not possible to neglect the influence of the user interface on the learner activities. Thus, the interface must be
designed and evaluated on the basis of well defined criteria and methods, specific for e-learning (Lanzilotti,
Our novel definition says that “e-learning systems quality is the extent with which technology, interaction,
content and offered services comply with expectations of learners and teachers by allowing them to learn/teach
with satisfaction”.
Figure 2. Quality aspects of e-learning systems
By considering the literature on the e-learning systems quality (briefly reported in the previous section), the
experience of human-computer interaction experts and the results of observing real users interacting with elearning systems, a new framework called TICS (Technology, Interaction, Content, Services), has been
developed. It focuses on the most important aspects to be considered when an e-learning system is designed or
evaluated (Figure 2). TICS primarily focuses on the quality dimensions recommended by the Italian Ministry of
Education in (Montedoro and Infante, 2003), integrating them with the interaction dimension that, as we have
discussed above, plays a crucial role in the fruition of the e-learning material. In the following each TICS aspect
is described.
The technology aspect is specialized in the dimension hypermedial technology, that refers to the technological
problems that can obstruct or make difficult the use of the e-learning system and to the hypermedia
characteristics of e-learning systems i.e., compatibility with different operating systems; performance; monitor
resolutions; hardware and software for system accessibility. By technology we also refer to the hypermedia
characteristics of e-learning systems, such as selecting and integrating the media made available by current
technology. In fact, when designing an e-learning system, we must accurately choose media that are really
complementary and may enrich the contents without overloading the user’s perceptual and cognitive systems.
The explicit inclusion of the interaction aspect is one of the novelty of our approach. We believe it is crucial for
technology-based learning. This aspect involves presentation of the educational material and of the provided
services and tools, and user activity performed during the interaction with the system. In particular, user errors
should be prevented as much as possible. If an error occurs, the e-learning system should provide appropriate
support to manage it. Learners not only interact with their educational material but they consider comments
provided by other users. In fact, according to other researchers (Ewing and Miller, 2002; Moore, 1989), a great
contribution to learning comes from interaction among learners and their lecturers. Researchers who neglect the
importance of user-system interaction implicitly assume that the e-learning system interface does not influence
user activities. People who have experience in e-learning know the problems that users have when interacting
with an interface that is not usable. In some studies performed with users (Ardito et al., 2006, Lanzilotti, 2006),
we have seen that if an e-learning system interface is not usable, learners spend more time learning how to use
the software rather than learning the content, which is their main goal.
The content aspect is more closely related to education. It refers to the educational process that happens during
the interaction with the e-learning system. It focuses on the appropriateness and quality of the educational
material that could be achieved through an accurate learner-centered design. This aspect also refers to the way
the material is taught and to the capability of the e-learning system to propose study activities to the learner, who
should also be free to autonomously choose his/her learning path (e.g. alternating moments of study, exercises,
and verifications). Besides, the learner must have the possibility to go more deeply into the course content and
the system must provide concrete, real, and helpful examples to facilitate understanding of the educational
The services aspect refers to the application proactivity, that involves the tools that facilitate and support the user
during the navigation through the system and the fruition of the educational material. It is necessary to provide
users with communication tools, auto-evaluation tools, help services, search engines, references, scaffolding, and
so on. Ease of learn and ease of use of such tools permit users to concentrate their efforts on the learning paths
without being required to spend too much time trying to understand the way the e-learning system works.
eLSE methodology
Designers and evaluators can focus on TICS dimensions, for providing what is required for good quality system.
As a further contribution we have derived guidelines that address the TICS aspects and an evaluation
methodology, called eLSE (e-Learning Systematic Evaluation), which prescribes a structured flow of activities.
The main idea of eLSE is that the most reliable evaluation results can be achieved by systematically combining
inspection with user-based evaluation. Several studies have outlined how two such methods are complementary
and can be effectively coupled to obtain a reliable evaluation process (Nielsen, 1993). In line with those studies,
eLSE suggests coupling inspection activities and user-testing, and precisely indicates how to combine them to
make evaluation more reliable and still cost-effective.
eLSE derives from a usability evaluation methodology, called SUE (Systematic Usability Evaluation), originally
developed for evaluating hypermedia systems (Matera et al., 2002). eLSE shares with SUE three important
characteristics. First, eLSE couples inspection and user testing, to make the evaluation more reliable and still
cost-effective. Each evaluation process starts by having evaluators inspecting the application and identifying
possible problems and troubles. User testing is then conducted, whenever necessary, to validate some inspection
findings with real users. Second, eLSE suggests to analyze an application along the TICS dimensions (i.e.
hypermedial technology, presentation, user activity, educational process, application proactivity) that address the
appropriateness of design with respect to the peculiar nature and purposes of the e-learning systems. Finally,
eLSE proposes an inspection technique based on the use of Abstract Tasks (ATs) that are specifically defined for
e-learning systems. For this reason, it is called AT inspection. ATs precisely describe which objects of the
application to look for and which actions the evaluator must perform in order to analyze such objects.
eLSE phases
According to eLSE methodology, the activities in the evaluation process are organized into a preparatory phase
and an execution phase. The preparatory phase is performed only once for each analysis dimension; its purpose
is to create a conceptual framework that will be used to carry out actual evaluations. The output of the
preparatory phase can be easily shared among different evaluators, or different evaluation laboratories that have
similar interests and evaluate such applications from similar points of view. The preparatory phase consists of
the identification of guidelines to be considered for the given dimensions and the definition of a library of ATs.
The execution phase is performed every time a specific application must be evaluated. It mainly consists of
inspection, performed by evaluators. If needed, inspection can be followed by user testing sessions, involving
real users. At the end of each evaluation session, the evaluators must provide designers and developers with an
organized evaluation feedback. The activities in the two phases are described in the following sections.
The preparatory phase
In the preparatory phase, a number of decisions must be taken and the definition of a specific set of Abstract
Tasks must be carried out.
Abstract Task formulation
By considering TICS dimensions, the literature on e-learning, the experience of human-computer interaction
experts, and the results of observing real users interacting with e-learning systems (Ardito et al., 2006), a number
of specific guidelines have been identified, to be taken into account during the initial design phase. Then, a set of
Abstract Tasks focusing on these guidelines is identified to support the evaluators in their inspections; an AT
addresses one or more TICS guidelines. In this way, the defined set of ATs allows inspectors to verify that all the
design aspects that must be considered for developing a good e-learning system are taken into account.
An AT is a description of what an evaluator has to do when inspecting an application. ATs guide the evaluator’s
activities by describing which elements of the application to look for, and which actions the evaluators must
perform in order to analyse such elements. In this way, even novice evaluators, with lack of expertise in usability
and/or application domain, are able to come out with more complete and precise results. We use the term
abstract task since it describes a task that an inspector performs when is evaluating an application, but this
description is provided in general terms and abstracts from a specific application to be evaluated.
ATs can be seen as evaluation patterns, making possible to maximize the reuse of evaluators’ know-how, by
capturing usability inspection expertise, and by expressing it in a precise and understandable form, so that it can
be easily reproduced, communicated, and exploited. They therefore allow evaluators to take “... advantage of any
of the efforts done in previous works, to reduce the effort needed to achieve a new one” (Nanard et al., 1998).
As stated above, ATs are precisely formulated by means of a template that provides a consistent format and
includes the following items:
¾ AT Classification Code and Title: they univocally identify the AT, and succinctly convey its essence.
¾ Focus of Action: it shortly describes the context, or focus, of the AT, by listing the application components
that are the evaluation entities.
¾ Intent: it describes the problem addressed by the AT and its rationale, trying to make clear which is the
specific goal to be achieved through the AT application.
¾ Activity Description: it describes in detail the activities to be performed during the AT application.
¾ Output: it describes the output of the fragment of the inspection the AT refers to.
Optionally, a comment is provided, with the aim of indicating further ATs to be applied in combination, or when
available, significant examples of inspection findings should be reported, to better clarify which situations the
evaluators should look for while applying the AT activity.
ATs are defined by expert evaluators on the basis of their experience, complemented with observations of other
experts performing various inspections. In this way, ATs are a means for capturing evaluators' expertise and for
reusing it; moreover, they provide information about application domain, tasks and users, that is reported in the
AT description. Being the heuristic evaluation based only on the use of heuristics, it lacks this information,
which is usually provided through some training given to the evaluators prior to the inspection.
As we already said, TICS is the framework to which eLSE also refers. Indeed, the defined ATs are classified in
two main categories: Content learnability and Quality in use. Specifically, content learnability addresses both
user activity and educational process. On the other hand, quality in use addresses hypermedial technology,
presentation, and application proactivity. The ATs refer to ISO standards (ISO 9126, 1991; ISO 9241, 1997),
thus supporting the evaluations of effectiveness, efficiency, security, productivity, and satisfaction.
Table 1 reports an example of AT, whose title is “Graphical interface elements”. The AT derives from two
guidelines referring to Presentation dimension, namely “Choose text dimension, font, and colour for having good
readability” and “Avoid possible forms of distraction (e.g. flashing, sliding inscriptions…)”.
Table 1. Example of AT
Focus of action: interface graphical elements
Intent: analyze the platform interface from the graphical viewpoint
Activity description: Analyze:
- the colours
- the use of flashing or sliding inscriptions
- the characters font and dimension
- the coherence of the LO pages.
Output: a list reporting if:
- there is an exaggerated use of different colours
- there is an exaggerated use of distraction form (flashing or sliding
- characters are not easily readable
- the different platform pages are not coherent among them.
Our approach proposes an important distinction between the e-learning platform and the educational modules.
The e-learning platform is the software environment that usually offers a number of integrated tools and services
for teaching, learning, communicating, and managing learning material. The educational modules, also called
Learning Objects, are the specific learning material provided through the platform. Design guidelines and ATs
defined for the platform differ from those ones defined for e-learning modules, since different features and
criteria need to be considered (Ardito et al., 2006, Lanzilotti, 2006).
Execution phase
Execution phase activities are carried out every time an e-learning system must be evaluated. They include two
major jobs: a systematic inspection and a user-based evaluation.
Systematic inspection is performed by evaluators. During the inspection, the evaluator uses the ATs to perform a
rigorous and systematic analysis and produces a report in which the discovered problems are described, as
suggested in the AT. The list of ATs provides a systematic guidance to the evaluator on how to inspect an
application. Most evaluators are very good in analysing certain features of interactive applications; however,
they often neglect some other features, strictly dependent on the specific application category. Exploiting a set of
ATs ready for use allows evaluators with limited experience in a particular domain to perform a more accurate
User-based evaluation is conducted only when there is disagreement among the evaluators on some inspection
findings, so that validation with real users becomes necessary. ATs are still useful since they indicate how to
define the Concrete Tasks (CTs for short), i.e. the actual tasks that users are required to perform during the test.
A CT is thus simply formulated by considering the activity description item of the AT whose application
provided contrasting findings; this description is not general as in the AT but it explicitly refers to the application
to be evaluated.
Since the AT activity description is a formulisation of the user tasks, starting from this it is immediately possible
to formulate experimental tasks which can guide users in the critical situations encountered by the evaluators
during inspection. CTs are therefore conceived as a means of actually verifying the impact, upon the users, of the
specific points of the application that are supposed to be critical for e-learning quality. In this sense, they make
user-based evaluation better focused, so optimizing exploitation of the users resources and helping to obtain a
more precise feedback for designers.
During evaluation execution, a sample of users is observed while they are executing CTs and relevant data are
collected (users’ actions, users’ errors, time for executing actions, etc.). The outcome of this is therefore a
collection of raw data. In the result summary, these data are coded and organized in a synthetic manner and then
The last activity of the execution phase aims at providing the designers and developers of the application with an
organised evaluation feedback. The result of this activity is an evaluation report describing the problems
detected, possibly revised in the light of the user testing outcome, using the terminology provided in the AT for
referring to system objects or interface elements, and for describing critical incidents. This standardised language
increases the precision of the report and decreases the risk of misunderstandings.
AT inspection validation
The advantage of the AT inspection for e-learning systems over other evaluation techniques has been
demonstrated by a controlled experiment. The study involved seventy-three senior students of a HumanComputer Interaction (HCI) class at the University of Bari in Italy. They were divided in three groups that were
asked to evaluate a commercial e-learning system by applying the AT inspection, or the traditional heuristic
evaluation, or a thinking aloud technique. Each group was assigned to one of the three experimental conditions.
The heuristic inspection group had to perform an heuristic evaluation exploiting the “learning with software”
heuristics (Squires and Preece, 1999). In the user testing group, every evaluator observed a student during the
interaction with the e-learning application using the thinking aloud technique. Finally, the AT inspection group
used the inspection technique with ATs proposed by eLSE methodology.
In addition, we recruited 25 students of another computer science class, who use the e-learning system, thus
acting as users for the user testing.
A week before the experiment, all participants were given a 1-hour demonstration of the application to be
evaluated. A few summary indications about the application content and the main functions were introduced,
without providing too many details. A couple of days before the experiment, a training session of about one hour
introduced participants with the conceptual tools to be used during the experiment. Each group participated in
their specific training session.
Data were collected in a group setting, but every participants worked individually. The study consisted of two
experimental sessions lasting three hours each. During the first session, participants evaluated the e-learning
system applying the technique they were assigned to. Participants of the three groups had to find usability
problems in the application, and to record them on a booklet. A day after, each evaluator was asked to type their
discovered problems in an electronic form. This was required in order to avoid readability problems during data
analysis. In the electronic booklet, each participant reported the description of the problem, where it occurred,
and how it was found. Finally, the evaluator gave a severity rating to the problem in the scale from 1 (I do not
agree that this is a usability problem at all) to 5 (usability catastrophe).
Two expert usability evaluators independently examined all the electronic forms in order to identify single and
unique usability problems. This analysis led to the identification of 247 problems and 49 non problems, or
statements which reported not understandable content or unverifiable information. Non-problems accounted for
7% of the statements written by the inspectors applying the heuristic inspection, and 3% of the problems written
by the inspectors applying the AT inspection or performing user testing.
On average, participants reported significantly more problems when applying the AT inspection techniques
(mean = 21), than when performing user testing (mean = 9.62) or the heuristic inspection (mean = 12.22). The
difference is significant as demonstrated by the results of an Anova with evaluation technique as betweensubjects factor (F(2,70) = 25.45, p < .001). Post-hoc tests applying the Bonferroni’s correction indicated that the
effect was due to the higher number of problems reported in the AT inspection condition, while there was no
significant difference between heuristic inspection and user testing.
An overall efficiency index was computed, which reflected the number of problems each participant found in 10
minutes. The Anova indicated that overall efficiency is affected by experimental condition, F(2,70) = 3.80, p < .05.
Post Hoc comparisons (Bonferroni’s correction) showed that eLSE was the most efficient technique (mean =
1.19 problems in 10 minutes, standard error = .08). There were no differences between heuristics and user
testing, in the order mean = .91, standard error = .08 and mean = .90, standard error = .08.
The study also revealed that different techniques addressed different type of problems. User testing and heuristic
inspection helped to highlight problems common to all interactive systems, whereas the AT inspection focused
also on specific problems of e-learning. These findings confirm the peculiar characteristics of AT that
specifically addresses the e-learning domain, provided that appropriate ATs are defined.
This paper has discussed the concept of the quality of e-learning systems. In particular, we have refined the elearning system quality by introducing the aspect of the user-system interaction that had been neglected by some
researchers. There is the ambition that the system user interface is “transparent” to the user. Unfortunately,
people, that have experience in e-learning, knows difficulties that user meets when s/he interacts with an
unusable interface. From this derives the need to emphasize the interaction aspect and the usability of the user
interface. Thus, we have defined a framework, called TICS, which focuses on the most important aspects to
consider when an e-learning system is designed or evaluated.
The paper have introduced an approach systematic to the evaluation of e-learning systems, called eLSE
methodology, that prescribes a structured flow of activities. eLSE proposes an evaluation technique, called AT
inspection. Such a technique falls into the category of the methods called “inspection” which do not involve
users, but expert evaluators only. This inspection technique uses evaluation patterns, called Abstract Tasks,
which make possible to maximize the reuse of the evaluator’s expertise and to express it in a precise and
understandable form, so that it can be easily reproduced, communicated, and exploited.
The results of the experiment performed to validate the AT inspection have shown that the inspection based on
the use of ATs is capable to address specific issues of e-learning better than other techniques such as a
specialised heuristic evaluation and user testing.
During the experiment, the students expressed positive opinions on the use of ATs. They said that the guidance
offered by ATs allows them to perform the inspection more accurately. Students perceived to have more control
on the inspection process and, consequently, they were confident to obtain good results. Furthermore, the two
experts that have examined the electronic forms filled by the inspectors noticed that ATs enforce standardization
and uniformity of inspection results, since all inspectors are guided by the output item writing their evaluation
We already defined a good number of ATs addressing the quality issues of e-learning systems. Other ATs are
currently under development that consider the overall user experience in the e-learning domain.
Ardito, C., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., Rossano, V., & Tersigli, M. (2004). Usability
of E-Learning Tools. Proceedings of AVI 2004, May 25-28, 2004, Gallipoli, Italy, 80-84.
Ardito, C., Costabile, M.F., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., & Rossano, V. (2006). An
Approach to Usability Evaluation of e-Learning Applications. Universal Access in the Information Society
International Journal, 4 (3), 270–283.
Boehm, B. (1978). Characteristics of Software Quality, New York: North Holland Publishing.
Chua, B. B., & Dyson, L. E. (2004). Applying the ISO 9126 Model to the Evaluation of an e-Learning System.
Proc. of ASCILITE 2004, December 5-8, Perth, Australia, 184-190.
Costabile, M.F., De Angeli, A., Roselli, T., Lanzilotti, R., & Plantamura, P. (2003). Evaluating the Educational
Impact of a Tutoring Hypermedia for Children. Information Technology in Childhood Educational Annual, 289308.
Ehlers, U. (2004). Quality in e-Learning form a Learner’s Perspective, European Journal of Open Distance and
Ewing, J., & Miller, D. (2002). A Framework for Evaluating Computer Supported Collaborative Learning.
Educational Technology & Society, 5 (1), 112-118.
Fata, A. (2003). La Qualita' nella Formazione a Distanza. Psico-Pratika, 4, retrieved April 2006 from
Grutzner, I., Weibelzahl, S., & Waterson, P. (2004). Improving Courseware Quality through Life-Cycle
Encompassing Quality Assurance. Proc. of Symposium on Applied Computing (SAC’04), March 14-17, 2004,
Nicosia, Cyprus, 946-951.
Herrington, A., Herrington, J., Oliver, R., Stoney, S., & Willis, J. (2001). Quality Guidelines for Online Courses:
The Development of an Instrument to Audit Online Units, Proc. of ASCILITE 2001, December 9-12, 2001,
Melbourne, Australia, 263-270.
ISO 9126 (1991). ISO/IEC 9126: Information technology - Software Product Evaluation - Quality characteristics
and guidelines for their use.
ISO 9241 (1997) ISO 9241: Ergonomics Requirements for Office Work with Visual Display Terminal (VDT) Parts 1-17.
Lanzilotti, R. (2006). A Holistic Approach to Designing and Evaluating e-Learning Quality: Usability and
Educational Effectiveness, PhD dissertation, Dip. Informatica, Università di Bari, Bari, Italy, 2006.
Matera, M., Costabile, M.F., Garzotto F., & Paolini P. (2002). SUE Inspection: an Effective Method for
Systematic Usability Evaluation of Hypermedia. IEEE Transactions on Systems, Man and Cybernetics - Part A,
32 (1), 93-103.
McCall, J. A. (1994). Quality factors. In John J. Marciniak, (Ed.), Encyclopaedia of Software Engineering, New
York: John Wiley, 958-969.
Montedoro, C., & Infante, V. (2003). Linee Guida per la Valutazione di Qualità del Software didattico nell'eLearning: ISFOL, I libri del Fondo Sociale Europeo, Roma.
Moore, M.G.(1989). Three Types of Interaction. The American Journal of Distance Education, 3 (2), 1–6.
Muirhead, B. (2001). Interactivity Research Studies. Educational Technology & Society, 4 (3), 108-112.
Nanard, M., Nanard, J., & Kahn, P. (1998). Pushing Reuse in Hypermedia Design: Golden Rules, Design
Patterns and Constructive Templates, Proc. of Hypertext ’98, the 9th ACM conference on Hypertext and
hypermedia: links, objects, time and space, June 20-24, 1998, Pittsburgh, PA, USA, 11-20.
Nielsen, J. (1993). Usability Engineering, Cambridge, MA: Academic Press.
Parlangeli, O., Marchigiani, E., & Bagnara, S. (1999). Multimedia System in Distance Education: Effects on
Usability. Interacting with Computers, 12, 37-49.
Quinn, C.N., Alem, L., & Eklund, J. (2005). A pragmatic evaluation methodology for an assessment of learning
Quintana, C., Carra, A., Krajcik, J., & Soloway, E. (2001) Learner-Centred Design: Reflections and New
Directions. In Carroll (Ed.), Human-Computer Interaction in the New Millennium, New York: Addison-Wesley,
Rovinskyi, D., & Synytsya, K. (2004). Distance Courses Quality: A Learner’s View. Proc. of Fourth IEEE
ICALT'04, August 30-September 1, 2004, Joensuu, Finland, 1080-1081.
Scalon, E., Jones, A., Barnard, J., Thompson, J., & Calder, J. (2000). Evaluating Information and
Communication Technologies for Learning. Educational Technology & Society, 3 (4), 101-107.
Squires, D., & Preece, J. (1999). Predicting quality in Educational Software: Evaluating for Learning, Usability,
and the Synergy between them. Interacting with Computers, 11 (5), 467-483.
Wong, B., Nguyen, T. T., Chang, E., & Jayaratna, N. (2203). Usability Metrics for E-Learning. Lecture Notes in
Computer Science, 2889, 235–252.
Zaharias, P., Vasslopoulou, K., & Poulymenakou, A. (2002). Designing On-Line Learning Courses:
Implications for Usability, retrieved July 2006, from http://www.japit.org/zaharias_etal02.pdf.
Piccinini, N., & Scollo, G. (2006). Cooperative Project-based Learning in a Web-based Software Engineering Course.
Educational Technology & Society, 9 (4), 54-62.
Cooperative Project-based Learning in a Web-based Software Engineering
Nicola Piccinini
Università di Verona, Dipartimento di Informatica, Ca' Vignal 2, Strada Le Grazie 15, I-37134 Verona, Italy
[email protected]
Giuseppe Scollo
Università di Catania, Dipartimento di Matematica e Informatica, Viale Andrea Doria, 6, I-95125 Catania, Italy
[email protected]
Even in self-organized project-based learning, the instructors’ role re-mains critical, especially in the initial
orientation provided to the students in order to grasp the educational goals and the various roles they may
undertake to achieve them. In this paper we survey a few questions proposed to that purpose in a web-based
software engineering course, together with relevant answers, we outline the project set-up methodology
aimed at providing students with that initial orientation in the laboratory part of the course, we collect a few
empirical data out of the latest seven-year history of the course and, finally, we put the presented work in
the context of current approaches to software engineering education and draw brief conclusions.
Problem-based learning, Project-based learning, Cooperative web-based learning, Self-organized learning,
Meta-cognitive reinforcement
Since seven years, the Software engineering course offered at the CS department in Verona relies upon the use of
a cooperative web-based organization, whereby laboratory project work as well the course organization itself
evolve following feedback and proposals made by the students. The educational approach thus matches the
profile of “problem-based learning, project-based learning, and collaborative problem solving” Nesbit & Winne
(2003). Furthermore, besides using a web-based cooperation platform BSCW (2005) to support interaction,
coordination and resource sharing, the web itself is used as a source of knowledge exploration and support to
collaborative inquiry learning (Chang et al., 2003; Salovaara, 2005), within a largely self-organized project
Nonetheless, the instructors’ role remains critical, especially in the initial orientation provided to the students in
order to grasp the educational goals settled from the outset, and the various roles and responsibilities they may
undertake to achieve those goals. This paper is aimed at corroborating this statement with empirical evidence
drawn from the subject course experience. This is presented according to the following organization of the paper.
In the next section, we survey a few questions presented to students in the initial part of the course, to the
aforementioned orientation purpose. In the subsequent section, a select blend of answers proposed to those
questions are reviewed and discussed, with reference to the main statement of the paper. We then proceed to
outline the project set-up methodology that is aimed at providing students with initial orientation in the
laboratory part of the course. Empirical data out of the latest seven-year history of the course laboratory are then
collected and discussed, again with reference to the main statement. We finally put the presented work in the
context of current approaches to software engineering education and draw brief conclusions. A more extensive,
albeit earlier (hence less up-to-date) report on this web-based laboratory learning experience is available (in
Italian) in Piccinini & Scollo (2005).
The following questions are excerpted (and translated from Italian) from the on-line lecture notes, which can be
consulted at Scollo (2005).
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
Q.1: Use of software engineering principles in web design
The notes start with an introductory lecture where 1) several analogies are pointed out between software design
and web design, and 2) traditional software engineering principles are presented and argued about. The very first
group of exercises then looks like as follows.
We propose a few examples of web design problems, all somehow connected to its rapid evolution
character (...) whereas we propose as exercises: 1) the identification of software engineering
principles, out of those introduced in the lecture notes, that appear relevant to each of the problems
proposed, and 2) the development of a rationale for their respective applications to evaluation of
solutions to those problems.
With websites whose management brings frequent changes, users find it useful to get information
about website updates, that is, answering the question: What’s new? Identify and evaluate at least
two different solutions to the problem of how to provide this service.
Q.2: Inquiry into ergonomic qualities of the course website
In a subsequent lecture, software engineering principles are put in the wider context of quality design rules, some
of which are drawn from ancient, philosophical tradition (such as Ockham’s Razor, Duns Scott’s Consistency
Rule, etc.). Use of these rules is then invited to be exercised through inquiry into ergonomic qualities of the
course website. To this purpose the following questions are proposed, among others:
Find aspects (of the structure and/or contents) of the website of this course which violate some
(which?) of the proposed quality design rules, and motivate your answer.
Find aspects (of the structure and/or contents) of the website of this course which satisfy some
(which?) of the proposed quality design rules, and motivate your answer.
These questions, as well as those relating to subsequent lectures, share a common premiss, which alludes at
expected students’ contributions as co-designers of the course website contents:
We do not propose to consider design qualities of the BSCW system as such, but only qualities
that be relevant to the use of this system for the educational purposes of this course. The following
may thus come into play:
aspects of the BSCW system as such, that prove relevant to the aforementioned use;
aspects of its use made by the designers of this workspace (instructors and tutors), who provide a
web-based educational service to the course participants, and so are both system users and service
providers at the same time;
aspects of the use of this system and of this workspace made by the users of the afore-mentioned
educational service, who, too, contribute to modify the structure and contents of the workspace,
hence also have their share of responsibility for the ergonomic quality of the website in question.
Q.3: Self-referential inquiry into workgroup organization
As a third, final case of interest we excerpt a couple of questions relating to workgroup organization, that are
based on a preliminary working assumption which takes the flavour of a self-referential case study:
Consider the following problem as a working assumption:
Organize (part of) the documentation that is produced by the participants to this edition of
the course, in the form of a website that would be suitable for use by the instructors and
participants to future editions of the same course.
With reference to such a working assumption:
1. propose working practices meant to help the efficiency of the production process;
2. characterize an organizational structure that would prove both adequate to the objectives and
compatible with the production context constraints;
3. etc.
Answers and reinforcement
A.1: Use of software engineering principles in web design
The first questions listed in the previous section were proposed since the 2002 edition of the course. Following
students’ feedback, viz. a first series of “systematically wrong” answers, their presentation is now endowed with
a supplement of meta-cognitive information, just aimed at clarification and reinforcement of the stated goal.
Here is what is added to this purpose, and why. The first answers only dealt with the proposed web design
problems, with no reference to software engineeering principles whatsoever. An example of such kind of
answers is the following:
The first method to handle such a situation consists in mailing an update report, after every update,
where the various changes made are described and documented.
The second proposal is to display a list of the recent changes as initial page.
This led to inclusion of the following premiss to the exercise:
The use of software engineering principles, such as those proposed in the lecture notes, proves
purposeful to identification as well as evaluation of solutions to web design problems. Each of the
proposed exercises asks to validate this statement in a concrete problem. What is thus required is
not so much to invent a solution to each of the proposed problems as, rather, to reason about the
use of the aforementioned principles to those purposes.
A.2: Inquiry into ergonomic qualities of the course website
What makes the following answers interesting, from an educational process viewpoint, is that they reveal a
limited inquiry into the proposed subject of investigation. This gives instructors the opportunity to point at
further inquiry directions and to interactively refine the students’ knowledge and understanding, again with a
meta-cognitive reinforcement side-effect. Here are two examples, respectively referring to the two questions
quoted in the previous section, where a student’s answer is excerpted, followed by the instructor’s reply.
A.: In my opinion an important functionality is lacking within the BSCW system, and this may
violate the completeness rule. The functionality which is not present (or I didn’t find) in the
BSCW is that which would enable one to download a whole folder, including its contents.
R.: The functionality is available, by means of the creation of a compressed archive of the folder,
both for download and for upload.
Note that the student is well aware that s/he may have failed to locate the desired function. The instructor’s reply
thus works as a solution advice to a student’s inquiry problem.
A.: The very structure of the BSCW system warrants conformance to the orthogonality rule. The
organization into folders and subfolders, each of them dealing with a particular subject, allows the
separation of distinct aspects and a quicker information retrieval by the user.
R.: Rather than “warrants” I should like to say “enables”, since it’s up to the content producers to
actually maintain separation of independent aspects by means of their placement into distinct
In this case, the instructor’s reply reinforces the alluded students’ responsibility as content producers.
A.3: Self-referential inquiry into workgroup organization
A seemingly surprising fact about the third group of questions recalled in the previous section is that none of
them has ever been answered directly so far. This can be explained as follows. Unlike the previous questions,
which are proposed at an early stage of the course, the subject questions occur at an advanced stage, when
laboratory project work is well beyond take-off. Answering the proposed questions is an optional assignment in
all cases, but with the first, early questions there’s no alternative assignment anyway. On the contrary, at the
advanced stage when the subject of group organization comes to be dealt with, practice on it may well take place
within the self-organized context of each project teamwork.
The proposed questions prove useful nonetheless, in that they inspire similar questions which groups do address
within their own self-regulated contribution to the laboratory production work, thereby answering those
questions indirectly, viz. with reference to a different, albeit related working assumption. Rather than working as
task assignments, thus, questions of this kind act as models for laboratory project set-up and organization, which
is the subject of the next section.
The lack of direct answers to the proposed questions, together with their proven usefulness in the development of
student projects are thus no failure but rather to be taken as a sign of success of a scaffolding strategy, whereby
instructors first model the desired working skills to be developed and then gradually shift responsibility to the
Laboratory project set-up
In order to accomplish the laboratory course tasks, students have got to put the concepts and methods learnt from
the lectures into practice. This is organized by way of simulation of a (fairly liberal) software production
environment, where project ideas and proposals are invited, project teams are dynamically created around them,
and team projects are viewed as mildly co-ordinated constituents of a somewhat larger “laboratory project”,
where industry-like control and co-ordination mechanisms are experimented.
A project idea proposal does not only involve a concise description of the functionality of an intended software
product, but is also required to exhibit a clear identification of the pursued educational objectives, mostly
specified in the form of a list of intended project deliverables. Here it is understood that the production of a
certain deliverable, say a requirements specification, a risk management plan, or a collection of architectural
models, is a task that is instrumental to learn how to apply concepts and methodological prescriptions relating to
that kind of task—this educational objective is thus designated by the corresponding deliverable.
The intended software product is by no means expected to be actually produced, only the planned deliverables so
are. However, in addition to setting educational objectives, a project idea proposal may include quality
objectives. These refer to the intended software product, are kept distinct from the educational objectives, and
denote the intent to evaluate and document the influence of desired quality characteristics of the specific product
on the production process to be planned.
Team projects are largely self-organized, in that all project activities, ranging from project idea formulation and
choice of educational objectives through project team composition, production organization and actual
execution, are left to the students. Quality of the organization work is a most relevant subject of the final
evaluation, which is in principle aimed at evaluating the production process maturity rather than quality of the
resulting products.
Despite full-range self-organization, instructors do play a key role of guidance and control, which shows up at
least on three subsequent stages of project evolution:
Initial explanation, of what we are summarizing in this section, in classroom and by means of on-line
documentation, in particular Piccinini (2005).
Team project set-up. Before starting each team project work, students are strongly encouraged to seek
instructors’ advice through a public on-line discussion about their project idea. Other students may of course
join the discussion. This starts with a project idea proposal, hopefully including the pursued educational
objectives, and sometimes also including a first identification of team members. The instructor may ask for
further clarification when any of the following is deemed to hold:
¾ the idea is not clearly expressed,
¾ the educational objectives are inaccurate, confuse or (as it often happens) missing,
¾ the quality objectives, if present, are inaccurate or confuse.
The discussion continues until no doubt is left and the team is formed.
Final exam evaluation. This relates to the whole course program, but draws upon the project work developed
in the laboratory.
The second out of the previous three points is the most important one in the laboratory course organization.
Since every discussion is publicly held on-line, it gives rise to an interesting case study for course students—
those of the current edition as well as those of future ones. Throughout the latest seven-year course history, those
discussions have come to build-up a fairly large information corpus that, besides comprising a rich bestiary of
potential project ideas (from pure fictionary ones to potential merchandisable products), proves amenable to
quantitative analysis, such as that which is presented in the next section.
Laboratory project evolution, collected data samples
The data sample sizes are displayed in Table 1: a total of almost 1000 students attended the seven course
# ideas
team set-up
Table 1. Data sample sizes
Table 2. Project idea discussions
% of ideas
The data relating to the current course edition, 2005–2006, should not be considered as final ones, since course
exams are on schedule for the rest of the year; students are not required to physically attend lectures, as they may
get the course materials as well as organize team projects on-line. They proposed 223 different project ideas;
most of these (209, nearly 94%) evolved to a concrete project, run by a working team with well defined
However, in the majority of the sample cases, the transition from project idea to actual project is not immediate.
Table 2 shows, in actual as well as percentual value, the incidence of the most frequent cases of delayed project
start-up, as found in the project idea discussions, where corrections or refinements of idea proposals were put
forward. This yields a quantitative measure of the relevance of the instructor's role to orientate the project startup along the desired educational path. The first row in Table 2 lists the number of project idea proposals, briefly
“ideas”, per course edition. Subsequent rows are labelled according to the following terminology.
correct are the ideas that appear sufficientely clear, accurate and complete since their first formulation. For
example, students who débute with:
Our idea is to design a on-line store of CD's, DVD's and, in general, consumable electronic
materials (printer cartdridge, etc.). The customer may consult the on-line catalogue and book the
chosen products. There will also be a forum where the customer may comment on the purchased
The project objectives are:
¾ quality control (functionality, reliability, usability, maintainability),
¾ feasibility study,
¾ risk management.
Group components: [omissis]
All is right and nothing is missing, hence the instructor replies:
Good job, go ahead!
clarified are the correct ideas that receive either additional suggestions from the instructor or requests for further
information from the instructor and/or the students.
refined are those ideas whose initial description is too vague, inaccurate, or even confuse, for example:
We intend to design a software for the management of the civil guard emergency units. Units coordination will be driven by the emergency type and by the skills and availability of staff
Although the use context of the target software is well described, an outline of the desidered functionality is
missing, and this makes it difficult to evaluate the idea.
edu.oriented are those ideas whose initial description either lacks educational objectives or has such a list
thereof, but this is inaccurate or unclear. Here is an example of the first kind:
The Romantik Tour travel agency has contacted our software house to produce a CD-ROM
catalogue that is to be distributed both to customers and to tour operators. Moreover, this catalogue
may be later upgraded by the agency.
The lack of educational orientation is pinpointed by the instructor:
The idea is briefly and well expressed, with respect to functionality. But what about the objectives
of your laboratory work?
As an example of the second kind, here is a proposed list of objectives:
feasibility study, risk management, quality objectives, production process architecture, usability
and accessibility, identification of resources and operational constraints, operational plan, time and
cost estimation.
The instructor replies:
The proposal is not clear because objectives and sub-objectives are listed with no logical structure,
could you please reorganize them? Moreover, I suggest to add workgroup planning to your
objectives, because the group is very large, hence it needs careful management.
The advice is not limited to point out a problem in the formulation of the educational objectives, but it also
contains an additonal suggestion aimed at getting the team started with proper attention to effectiveness of its
own organization. This example highlights the guidance role of the instructor in operational terms, besides
conceptual ones.
qual.oriented are ideas that feature quality objectives, but in an unclear manner.
team set-up are discussions that, besides clarifying, refining or orienting a project idea, are also exploited to
define the project team.
unfinished are those idea discussions which leave unanswered questions. For example, the aforementioned
discussion of the “RomantikTour” idea belongs to this category, as the students never satisfied the instructor's
For each column in Table 2, the sum of the discussion counts placed under the aforementioned categories need
not coincide with the total number of ideas, since those categories are not mutually exclusive, but also because a
few ideas (approximately 10% of the total number over the seven course editions) directly show up as started
projects, without going through a discussion between students and instructors.
After this due premiss, a quick look at the data in Table 2 immediately reveals that the by far most frequent
reason for discussion has to do with the educational orientation, whose lack or unclarity affects almost a half of
the ideas. Refinement of the functional content of the idea takes the second place, finally followed by orientation
with respect to quality objectives. This one featured a rapidly growing trend until the previous course edition,
which fact may be explained by the growing importance given by the instructors to quality objectives, in the
course of time. The exploitment of project idea discussions as an opportunity to build up the project team
exhibits a steadily growing trend throughout the whole sequence of course editions.
Table 3 allows us to evaluate other interesting data about the project start-up discussions, expressed as average
values. The first two columns respectively concern the size of the project team and the number of notes posted to
those discussions. Because these usually consist of a sequence of questions and answers between instructors and
students, we put forward this value as another quantitative indicator of the relevance of the instructors’ role.
team size
Table 3. Statistical averages
posted notes
start week
start week
start week
start week
Starting from the second course edition, for each idea discussion the smallest number of notes is 2, as it happens
when the idea is immediately judged to be correctly formulated, whereas more notes are to be expected when
problems affect its first formulation. In the first course edition the average value of the number of posted notes
falls below 2 because, unlike the subsequent course editions, all project idea proposals were posted to a single
forum, where they most often did not receive any feedback from the instructor—that was rather given
personally, during project discussion time in the lab. Several other changes have occurred in the course
organization over the years, testifying to the fact that this has been a learning experience for the instructors no
less than for the students.
The three rightmost columns provide one with a few indicators on how the various start-up discussions, added in
the course of time, come to build a sort of collective memory that helps newcomers to correctly formulate their
own idea. The term “start week” denotes the number of weeks spent before proposers of a project idea introduce
their idea; this number is counted starting from the day when the instructors give a classroom introduction to
how should team projects be set up.
The subsequent three columns report the same statistic, but only computed over the project ideas which fall
under one of the three aforementioned categories which have been shown to be the main reasons of discussions.
Except for one case (precisely refined ideas at the fifth course edition), the averages in these three columns are
lower than the general average; this fact clearly tells that ideas submitted later are more frequently found correct,
probably because proposers have learnt from the experience made by their preceding, eager-to-start colleagues.
It seems that no similar improvement can be ascribed to progress throughout subsequent course editions, for
reasons that are as yet to be fully understood. However, there appears to be a significant correlation between the
rate of correctness of initial formulation of ideas and average start week throughout different course editions; if
cy denotes the percentual number of correct ideas at course edition y, while wy denotes the average start week at
the same course edition, our data yield a value of 0.805 for the correlation coefficient between these two
Related work
There is no standard methodology for software engineering education. The basic rationale for including student
projects in the educational activities is to let students get a foretaste of what a professional in the field is
expected to do, and how. Not all software engineering courses include such a laboratory part. Even those which
do so, may largely differ in the way projects are set-up and organized, e.g. whether or not do they involve
collaborations with industrial sponsors, and, in either case, who is in charge of defining project objectives and
task assignments.
Because of the rapid pace of change in software engineering methods and technologies, industrial impact
through education is a persistently hot topic in specialized conferences on the subject, see for example the panel
CSEET (2006) at the forthcoming edition of CSEET.
The laboratory project set-up in the present case study is motivated by firm reasons of principle. The instructors
held a constructivist viewpoint, which may be traced back to Vygotsky's social interaction theory Vygotsky
(1978), takes due account of the educational value of training students to take up the responsibility of setting
their own learning objectives, and exploits the opportunities of web-based cooperation to implement a
scaffolding strategy to that take-up purpose.
While the debate on software engineering education has been most often centered on the specific traits of the
discipline so far, the importance of adopting proper educational theories is starting to be recognized—see for
example Basili & Basili (2006) at the aforementioned conference. Our approach seems consistent with that
In this paper we have presented and briefly analyzed a case study in software engineering education, spanning
over a seven-year evolution, characterized by a blend of educational techniques: traditional classroom lectures,
textbook and lecture notes, as well as a web-based cooperation platform, supporting interaction and selforganization of laboratory projects. The laboratory project set-up in the present case study is motivated not only
by reasons of principle, as recalled in the previous section, but also by practical constraints. The largely selforganized character of the laboratory project work turns out to be a necessity, since the bare number of students
enrolled in the course would make it impossible to provide a continued guidance and daily organization of all
projects by just one or two instructors. In other words, scarcety of human resources puts a severe limit on the
level of guidance that could be effectively provided under a different arrangement.
The following conclusions can be drawn from the case study summarized in the previous sections.
First, regardless of their “easiness”, the first questions proposed as exercises always turn out to be the most
difficult to get answered properly. The main difficulty is to be traced, in this case, to the novelty of the
methodological mindset wherein the initial subject—applicability of software engineering principles to web
design case studies—is framed.
Second, the issue of endowing project proposals with clearly stated educational objectives has persistently
turned out to be the most significant reason for delayed set-up of team projects, despite the provision of detailed
guidelines to this purpose in the laboratory edu-cational material. The reason for this difficulty is, again, the
novelty of a methodological mindset where metacognition is explicitly meant to play the most significant role.
Students are just not used to that, they are rather inclined to focus on the functional features of their intended
product than on their own educational gain. This is surely not surprising, at the initial project set-up stage, nor is
it necessarily a bad sign; it just highlights the usefulness of scaffolding strategies where the level of guidance
provided by instructors is significant at the early stages, and then progressively decreasing.
Third, regardless of the variety of alternative organization models proposed for group work, the vast majority of
students privilege the most established one, viz. that which appears to be most popular from the previous
editions of the course. This can be seen as an instance of the well-known “popularity is attractive” principle from
the theory of network evolution Dorogovtsev & Mendes (2003).
Our final, perhaps most interesting observation is that, even in self-organized project-based learning, the
instructors' role remains critical, especially in the initial orientation provided to the students in order to grasp the
educational goals and the various roles they may undertake to achieve them. This educational aim seems to be
most effectively achieved by prompting and exploiting all available opportunities for meta-cognitive
reinforcement, which fosters deeper inquiry capabilities, knowledge and insight.
Basili, V.R., & Basili, P.A. (2006). Software Engineering Instruction and Education Theory: a Dialogue,
retrieved July 24, 2006 from http://db-itm.cba.hawaii.edu/cseet2006/Basili.php.
BSCW (2005). Basic Support
CSEET (2006). CSEET 2006 Panel: Industrial Impact through Education – Lessons Learned from Barry
Chang, K.-E., Sung, Y.-T., & Lee, C.-L. (2003). Web-based collaborative inquiry learning. Journal of Computer
Assisted Learning, 19 (1), 56-69.
Dorogovtsev, S., & Mendes, J. (2003). Evolution of Networks, Oxford, England: OUP.
Nesbit, J., & Winne, P. (2003). Self-regulated inquiry with networked resources. Canadian Journal of Learning
and Technology, 29 (3), retrieved October 2, 2006 from http://www.cjlt.ca/content/vol29.3/cjlt29-3_art5.html.
Piccinini, N. (2005). Norme d'uso del BSCW. Integrazione per il lavoro di laboratorio, retrieved October 27,
2006 from http://amarena.sci.univr.it/edu/pub/bscw.cgi/2482086/NormeUsoBSCWLab-in.html.
Piccinini, N., & Scollo, G. (2005). Fare laboratorio in rete: tracce di un’esperienza di web-learning. DIPAV
Quaderni, 12/13, 143-164.
Salovaara, H. (2005). An exploration of students’ strategy use in inquiry-based computer-supported
collaborative learning. Journal of Computer Assisted Learning, 21 (1), 39-52.
Scollo, G. (2005). Architetture software e Ingegneria del software, Note delle Lezioni, retrieved October 9, 2006
from http://amarena.sci.univr.it/edu/pub/bscw.cgi/2481306/AIS0506-in.
Vygotsky, L. (1978). Mind in Society, Cambridge, MA: Harvard University Press.
Tomsic, A., & Suthers, D. D. (2006). Discussion Tool Effects on Collaborative Learning and Social Network Structure.
Educational Technology & Society, 9 (4), 63-77.
Discussion Tool Effects on Collaborative Learning and Social Network
Astrid Tomsic and Daniel D. Suthers
Department of Information and Computer Sciences, University of Hawai`i at Manoa, Honolulu, HI 96822, USA
Tel: 1-808-383-0411
[email protected]
[email protected]
This study investigated the social network structure of booking officers at the Honolulu Police Department
and how the introduction of an online discussion tool affected knowledge about operation of a booking
module. Baseline data provided evidence for collaboration among officers in the same district using e-mail,
telephone and face-to-face media but showed minimal collaboration between officers in different districts.
On average, knowledge of the booking module was low. After introduction of the online discussion tool the
social network structure changed, showing an increase in collaboration between different districts and an
increase in knowledge of the booking module, even though frequency of collaboration did not increase
significantly. The study suggests that the formation of new collaborative ties is more significant for learning
through information sharing in social networks than raw frequency of interaction. This work is framed by
theories of knowledge building, social architecture, and communities of practice.
Collaborative learning, Knowledge building, Social network analysis, Online discussion
This paper reports on a study that investigated how the introduction of a web-based discussion board changed the
social network of an organization, with concomitant changes in distribution of knowledge concerning the use of
a new tool that had been recently introduced into the organization. The study was situated in the Honolulu Police
Department (HPD), a multi-district and hence distributed organization. At the outset of our study, the
Department had recently introduced a client-server Records Management System (RMS), replacing a centralized
system requiring that paper-based forms be submitted from district sites. The first phase of the study documented
the social network structure between six different physical locations and distribution of officers’ knowledge of
the RMS. The second phase of the study introduced an online discussion tool into a subset of the districts that
allowed officers to collaborate with their colleagues in the participating districts. Social Network Analysis
(SNA) indicated that the introduction of the new software led to a change in the social network between districts.
The results of a second knowledge assessment showed that there was also an increase in knowledge among
booking officers using the online discussion tool. Social network structure, specifically information sharing
across distributed locations, appeared to be more important than frequency of interaction. The study contributes
to our understanding of the value of computer-mediated communication in distributed organizational contexts
where other forms of communication are available and continue to be used.
The paper begins with an introduction to relevant aspects of the organization and practice of the Honolulu Police
Department and the tool that mandated a change in practice. The next section provides theoretical background,
focusing on social architecture, knowledge building and communities of practice perspectives on learning in
organizations, and discusses the potential benefits of computer mediated communication for encouraging
collaboration among participants and supporting communities of practice. The remainder of the paper reports on
the methodology, results and implications of the study itself.
A Change of Practice in the Honolulu Police Department
The Honolulu Police Department (HPD) is responsible for police operations in the City and County of Honolulu,
which consists of the entire island of O`ahu. Units included four districts and two support units. This study
focused on booking officers, police officers who have the primary duty of processing arrestees as they are
brought into a district police station. Depending on the severity of the crime or availability of holding cells, the
booking officer may decide to transfer the arrestee to the main booking station, known as the Central Receiving
Division (CRD).
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
A new computerized Records Management System (RMS) was introduced by the Information Technology
Division (ITD) in August 2003. Prior to the implementation of the RMS, a standalone DOS-based booking
program was used to process and track arrestees. This booking program was only available in CRD. Officers at
other districts filled out paper-based forms and sent them to CRD for processing. The new RMS was clientserver based and contained an integrated booking module that made entering and processing of bookings
possible from any district on the island of O`ahu. With the new RMS came the need to train officers on how to
use the RMS to enter and search for information. Entering accurate data and being capable of searching criminal
history records is a mandatory and vital part of a police officers’ daily duty and essential to effective police
work. The degree of knowledge that an officer has about how to use the RMS system to locate pertinent
information has a direct bearing on the outcome of criminal cases.
At the time that the RMS was introduced, officers used a combination of e-mail, telephone, face-to-face
discussions and other means to collaborate with each other in order to locate information, raise issues and
propose more efficient ways to use the RMS. The relevant expertise was distributed throughout the organization:
the ITD staff had a good understanding of how to use the RMS in more general terms (e.g., search for a name),
while police officers had a better understanding of what information they needed to find and how it related to
business processes, but not necessarily the best way to find it. The existing collaboration process between police
officers and the ITD staff was inefficient because knowledge was spread over many types of media, and due to
the nature of these media needed to be repeated frequently to different parties at different times, was often lost
and could not be easily located and improved upon. Coupled with this, officers did not have a clear
understanding of who knew what and tended to address all information requests to a single ITD staff member.
This situation presented us with an opportunity to investigate how computer mediated collaboration (CMC),
specifically an asynchronous discussion tool, might improve knowledge sharing and knowledge building within
this distributed community of practice. Our intervention will be described in a subsequent section, but first we
consider theoretical motivations.
Learning in Organizations
An intervention intended to influence learning in an organization must consider how such learning might take
place. Although various social learning theories are available, the most relevant theories for the present study are
those that emphasize the community aspects of learning. We draw particularly upon knowledge building theory
(Scardamalia & Bereiter, 1996), Bogenrieder’s (2002) sociocognitive theory, and communities of practice theory
(Wenger, 1998; Wenger, McDermott & Snyder, 2002).
Knowledge Building
Knowledge building (Scardamalia & Bereiter, 1996) refers to the intentional pursuit of advances in collective
understanding undertaken by a community. According to Scardamalia & Bereiter (1996), it is “work on the
creation and improvement of ideas. The dynamic is social, resulting in the creation of public knowledge …
public knowledge can itself become an object of inquiry and the basis for further knowledge building.” At the
time of our study, departmental officers and staff were engaged in negotiating practices for use of the new
booking management system. The CMC technology that we introduced made statements of knowledge needs
and proposed solutions public in a persistent medium available to many other community members, and hence
offered the conditions for treating knowledge as an object of inquiry in this community.
Extensive research motivated by knowledge building theory has been conducted in settings for formal education
(schools and colleges). For example, Sha & Van Aalst (2003) conducted a study using server log data to explore
knowledge building in the classroom. The techniques of Social Network Analysis (SNA, Wasserman & Faust,
1995) were used to analyze student participation and interactivity in an online discussion database called
Knowledge Forum. SNA was used to map the structure of relationships, the major lines of communication and
patterns of interaction within the social network. It provided valuable insight into the effectiveness of the class’s
ongoing efforts to improve their knowledge building and the conditions under which knowledge building
occurred. The Sha & Van Aalst study sought to extend the framework for knowledge assessment. Despite a
widespread emphasis on social constructivism, assessment is still based on the individual student’s final
knowledge. Their study explored how server log data could be used to inform continued knowledge building. It
suggested that improving connectivity within a community might provide better support for knowledge building.
Similarly, our study used server log data to map the social network and examine the content of the posts for
different behaviors. It also explored how changes in the social network affected learning.
Social Architecture
Bogenrieder (2002) argues that organizational learning is a combination of both social-relational and cognitive
activity. According to Bogenrieder, the social-relational aspect consists of the social networks fostered by the
organization while socio-cognitive conflict (Doise & Mugny, 1984) is the vehicle for nurturing cognitive
activity. Socio-cognitive conflict has two conditions that must be met for learning to take place. The first
condition is that a social relationship exists between the participants and the second condition is cognitive
diversity. When participants contribute different ideas, these differences trigger learning through socio-cognitive
conflict (and, we would argue, other mechanisms of intersubjective meaning making: see Suthers, 2006). For
example, in a police department officers have different educational backgrounds and levels of experience in
police work, and varied expertise in certain types of crimes. If sharing of ideas between officers is encouraged,
these differences can be leveraged to produce more effective practices for crime reduction.
Bogenrieder (2002) suggests that social architecture can be used as an instrument to build relationships for
learning. As previously discussed, both cognitive diversity and contact between diverse people are necessary for
organizational learning. The specific design for this contact—the social network—depends on the goal
uncertainty and technical uncertainty that characterize the problem situation. Goal uncertainty is “ambiguity
about the preferences or goals the decision-maker aims to satisfy” and technical uncertainty is “uncertainty in
parameters, input data and initial states (resulting from 'inexactness' and 'conflicting evidence')” (Weber, 2000).
Where conditions of high goal and technical certainty exist, instrumental network structures are satisfying.
However, where goal certainty coexists with technical uncertainty, an effective network structure is sought based
on the distribution of knowledge. In the case of officers learning the new RMS, there is high goal certainty
(officers know what they want to do), but high technical uncertainty (officers don’t necessarily know how to do
it). In these types of networks, the network paths are not pre-determined: the structure of the network forms
based on participants’ seeking of the knowledge needed to resolve the technical uncertainty.
Laboratory studies have indicated that some types of network structures are more effective than others for
diffusing information throughout a group (Cummings & Cross, 2003). However, the task is usually pre-defined
by the researcher who establishes paths for the diffusion of information. In organizations, information flow
depends on the expertise of the group, as discussed above. Therefore, while certain network structures may be
more efficient for information diffusion in an experiment, these structures may not be effective for leveraging
group expertise in an organization. The literature suggests that from a cognitive perspective, network structures
with greater integration (connectivity) may be more effective for leveraging group expertise. Studies of
transactive memory suggest that groups benefit from knowledge of who knows what in the group (Liang et. al.,
1995, Moreland & Myaskovsky, 2000). Work in diffusion of innovations suggests that hierarchical network
structures are both inefficient and result in the degradation of information quality (Rogers, 1995; Valente, 1995).
Communities of Practice
The theory of Communities of Practice (CoP; Wenger, 1998; Wenger et al., 2002) provides further conceptual
tools for understanding how the collective knowledge of a community is sustained and adopted to new
situations. According to this theory, a CoP consists of people mutually engaged in ongoing participation,
oriented towards a common domain that is addressed through a shared repertoire: a “group of people who share a
concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise in this area
by interacting on an ongoing basis” (Wenger, et al. 2002). When members of a CoP are distributed throughout an
organization in different teams, the interaction between them required to sustain knowledge is at risk.
Information technology can be valuable for bringing community members together to share and resolve
problems, as discussed in the next section.
According to CoP theory (Wenger, 1998), the knowledge of an organization—the repertoire of its communities
of practice, such as the police officers—is sustained through the dynamic interplay of the duality of participation
and reification. A duality is not a pair of polar opposites, but rather a dynamic productive pair: increasing one
may well require increasing rather than decreasing the other. When discussing communities of practice,
participation refers to how we engage together in an activity, and through that engagement generate, affirm and
replicate patterns of engagement: our practices. Reification refers to ways in which these practices become
reflected in artifacts and other structures that help replicate the practices. For example, the practices of a police
department are reflected in the structure of its booking forms. This structure supports replication of those
practices in the moment by reminding participants of what they need to do, and in the long term by serving as a
guide through which new members of a CoP can be brought into its practice. Reifications of practice are not
restricted to physical artifacts and information technologies. Terminologies, ways of talking, conventions of
social greeting, etc. can also serve as reifications of practice. Importantly, participation that engages a reification
is not just controlled and constrained by that reification, but also reaffirms and sustains the meaning of the
reification, and can change the meaning of (reinterpret) the reification. Participation in an online discussion
results in reifications in the form of messages that can guide future participation or be reinterpreted through that
Computer Supported Social Networks as Socio-Technical Capital
Wellman (1996) claims that Computer Supported Social Networks (CSSN’s) support a focus on information
exchanges. “People can easily post a question or comment and receive information in return. Broadcasting
queries through CSSN’s increases the chances of finding information quickly and alters the distribution patterns
of information. It gives those working in small or distant sites better access to experienced, skilled people.”
Additionally, “online information flows spill over unexpectedly through message forwarding, providing access
to more people and new social circles, thus increasing the probability of finding those who can solve problems”
(Kraut & Attewell, 1993). For example, officers don't know who to ask for information, but a discussion board
works to their advantage because it doesn't require that officers know whom to ask: they need only post a
message for all to read. Computer mediated communication “turns communication into substance”, a reification
that can be accessed and elaborated on by others at different times and places in a manner not possible with
volatile media such as speech. Furthermore, the value of CSSNs is not limited to sharing information. The social
relationships supported by CSSN technology in conjunction with the affordances of that technology for
continuing those relationships in certain ways collective constitute socio-technical capital: “a resource that can
be accumulated and whose availability allows people to create value for themselves or others” (Resnick, 2002).
As soon as an officer receives a reply, not only has new information become available, but also his or her social
network has expanded. If the reply is helpful, the officer now knows who to ask for further information on that
particular topic, as do others who are observing the interaction, and obligations to be helpful in a reciprocal
manner are created. For these reasons, we expected that it would be beneficial for police officers and CRD and
ITD staff to collaborate in an online environment. This expectation was assessed by (1) studying the changes in
the network structure, (2) investigating the frequency and degree of collaboration, and (3) analyzing the content
of the posted messages.
Organizational learning is required when practices must change to reflect changing conditions, for example, the
introduction of a new tool such as the new booking system. Ideally the members of the organization engage in
knowledge building, which in CoP terms can be understood as the deliberate pursuit of extensions to the
collective repertoire to adapt to the new conditions. The role of a CSSN in this process is twofold, as suggested
by the participation/reification duality. From the perspective of participation, a CSSN extends mutual
engagement to a larger collection of people, in this case distributed across multiple sites. In an expanded
network, cognitive diversity increases and innovations are shared more widely, as the reifications resulting from
participation are shared more widely. From the perspective of reification, the persistent reifications generated by
participation in a CSSN can influence practice at other times and places, even the practice of those not involved
in the production of those reifications, and can also be reinterpreted by others. Thus, for reasons influencing both
participation and reification, we expected the introduction of an online discussion tool to improve the scope and
extent of mutual engagement in the knowledge building process of developing new practice around the new
booking system.
Study Design and Methods
The foregoing theoretical discussion concluded that a CSSN can increase knowledge building in an organization
by enabling and expanding interaction between a greater diversity of community members across space and
time, through networking and persistent representations, respectively. One way to determine whether this
happens in practice is to introduce a CSSN into an organization, observe whether that introduction expands the
social network of the organization (whether there is more interaction between more people), and observe
whether this change is responsible for advances in knowledge (those who interact increase their knowledge
concerning the topic about which they interact). Our study design applies this strategy to address the following
research questions.
Research Questions and Hypotheses
Research Question I. How will the introduction of an online discussion tool affect the existing social network
structure at the Honolulu Police Department for officers learning the booking module for a new Records
Management System (RMS)?
Hypothesis 1 (H1). There will be an increase in collaboration about the booking module of the RMS between
participants in different districts using the online discussion tool compared to participants who do not, using pre
and post interview instruments and social network analysis (SNA). Rationale: CMC expands the potential social
network over space and time, and officers will take advantage of this expanded network to seek the knowledge
needed to reduce technical uncertainty.
Hypothesis 2 (H2). There will be an increase in frequency of collaboration about the booking module of the
RMS between participants using the online discussion tool compared to participants who do not, using pre and
post interview instruments and SNA. Rationale: CMC, being asynchronous and persistent, makes collaboration
easier, and the need for knowledge building makes collaboration desirable.
Research Question II. How will use of the online discussion tool influence knowledge of the RMS booking
Hypothesis 3 (H3). There will be an increase in knowledge of the booking module of the RMS for participants
using the online discussion tool compared to participants who do not, using pre and post survey instruments.
Rationale: A CSSN enables sharing of information and encourages knowledge building by increasing cognitive
diversity and supporting reifications that make knowledge claims open to inspection and interpretation.
Booking officers were randomly selected from a pool of officers of different ages with different cultural and
academic backgrounds from four (4) different locations (districts) on O`ahu. Twenty (20) officers were selected
from districts 5 and 8; of these, fifteen (15) chose to participate. We call these officers Group A. Another twenty
(20) officers were selected from districts 2 and 3; of these, sixteen (16) chose to participate. We call these
officers Group B. Along with these thirty-one (31) police officers, four (4) CRD civilian booking staff and seven
(7) ITD staff participated in the study. Each district collaborated with the ITD staff and CRD staff. The districts
were at different physical locations than each other and the ITD and CRD staff. This physical separation enabled
us to minimize cross-collaboration between the two districts that were provided with access to an online
discussion tool (Group A) and those that were not (Group B).
Our intervention consisted of the introduction of an online discussion board dedicated to discussion of the use of
the RMS. This discussion board was made available to Group A and the CRD and ITD staff, but not to Group B,
enabling the foregoing hypotheses to be tested in a quasi-experimental design.
An online discussion board is a viable tool to support knowledge building in a distributed community of practice
because it can support spatially and temporally distributed interaction and it offers persistent representations
through which public knowledge becomes an object of inquiry. The HPD officers are distributed across physical
HPD districts, yet are faced with a similar set of issues that non-booking officers may not be familiar with. They
use the same software module in the RMS and are required to be familiar with specific procedures for booking.
They must also be familiar with basic troubleshooting procedures because the software support staff (ITD) are
separated from the officers who must put the knowledge to use. An online discussion board offers a shared and
persistent forum for booking officers to increase their awareness of system-wide issues and booking practices.
Other media such as face-to-face, e-mail and telephone lack persistence and are limiting for officers working in
different districts or different shifts. However, a discussion board is not without its limitations: the challenges for
any distributed community include building trust and maintaining online participation (Preece 2000; Wenger et
al., 2002).
The particular online discussion tool used in this study was Discus, available at www.discusware.org. The
authors are not associated with the Discus project. The research reported in this paper is concerned with the
potential impact of the genre of web tools represented by Discus, not with evaluation of Discus’ specific design
features. The first author chose this tool based on the following features. It used a standard and familiar threaded
format (see Figure 1). Being web-based, Discus was accessible from every computer that ran the booking
module for the Records Management System. Questions could be posted as soon as an issue occurred, and an
officer could access information anytime, not only when the ITD staff or CRD staff were available. Discus also
offered the ability for officers to see who posted a message. This is important in community building because if
an officer finds a particular post helpful, they can continue asking the author of that post for further information.
It also allows officers to become familiar with who knows what about the RMS and thus who to ask for
clarification or further explanation.
Figure 1. Layout of the threaded discussion format in the Discus tool
Procedure and Task
The study was conducted over a six-week period in March/April 2004. (At the end of this period, a departmentwide reassignment of officers changed the composition of the districts.) Participants were asked to complete two
(2) surveys (pre and post) and two (2) interviews (pre and post). The interview layout and content was adapted
from surveys created by Krackhardt & Haythornthwaite (1998). Both the surveys and interviews were piloted
with two (2) officers who did not participate in the final study and was revised according to the officers’
feedback and suggestions for improvement.
The pre-survey was used to determine prior knowledge of the booking module of the RMS system. It consisted
of five (5) questions with an estimated duration of ten (10) minutes. Each question consisted of typical search
tasks related to the booking module in the RMS that an officer is required to perform on a daily basis. For
example, “What is the report number for the FRAUD incident that occurred in sector 1 beat 150 on 10/19/2003?”
A pre-interview was conducted to determine the current social structure. It consisted of questions to determine
with whom the participant collaborated about the booking module of the RMS during the prior three (3) weeks.
It asked how well they knew this person, the frequency of these collaborations, and the media they used to
collaborate. In order to benchmark the initial social network structure, data on each of the different types of
communication media used was collected. This was necessary to determine what connections for knowledge
building existed beforehand and whether new connections developed as a result of the introduction of the online
discussion tool.
Participants in Group A (districts 5 and 8), the ITD staff and CRD staff were asked to contribute to an online
discussion tool. They were given introductory training on the online discussion tool and provided with a login
and password to record questions, comments, ideas and suggestions about the booking module. Posting activity
on the online discussion tool was initially slow, so the ITD staff posted what it thought to be useful tips on how
to use the booking module in order to generate activity. The ITD staff checked the board on a regular basis to
ensure that questions were answered in a timely manner. This was done to encourage participants from other
districts to post questions. Group B (districts 2 and 3) continued to collaborate with the ITD staff, CRD staff and
each other using the traditional face-to-face, e-mail and telephone methods of collaboration.
A post-survey consisting of five (5) questions with an estimated duration of ten (10) minutes was given to Group
A and Group B to determine whether any changes in knowledge had occurred. Similar to the pre-survey, the
post-survey contained search questions on the booking module in the RMS but did not have the same questions
as the pre-survey to avoid participants learning from the pre-survey. For example, “Who was the arresting officer
for the arrest with report number XXXXXX?”
A post-interview was conducted with the same questions as the pre-interview to determine whether any changes
had occurred in the social structure between the districts.
Measures and Analysis
Scardamalia (2000) developed twelve descriptors that represent the “best practices” of knowledge building.
Chan, Lee & Van Aalst (2001) used a modified subset to organize course evaluation and to scaffold knowledgebuilding advances. These four principles are: 1. Working at the cutting edge, 2. Progressive problem solving, 3.
Collaborative effort, and 4. Identifying high points in the discourse. Sha & Van Aalst (2003) conducted a study
where they focused on the pedagogical knowledge principal of collaborative effort. They analyzed the
characteristics of student’s social interaction by measuring participation (posts), reciprocity (replies),
connectivity (linked notes), social position (activity in the social network) and social interaction (reads).
Like Sha & Van Aalst (2003), the present study measures characteristics of individual social interaction relating
to collaborative effort, but groups these measurements into broader factors that investigate social interaction
between districts. These factors correspond to the three hypotheses:
Collaboration – defined as any new social ties between participants who had not previously
communicated. This study is particularly interested in collaboration between districts. It is measured by
the messages contributed to the online discussion tool and the interactions in other media reported in the
pre and post interviews. This includes messages posted, replied to and read.
Collaboration Frequency – defined as the number of interactions between participants in the social
network. It is measured by the number of messages contributed to the online discussion tool and the
number of interactions in other media reported in the pre and post interviews. This includes the number
of messages posted, replied to and read.
Knowledge – defined as the understanding, familiarity and proficiency of a participant with the booking
module of the RMS. It is measured by the results of the search tasks in the post-survey and the presurvey. Knowledge is rated on a scale of 0 for the lowest score with no tasks correct to 5 for the highest
score for all tasks correct.
SAS statistical software was used to tabulate and analyze the results of the knowledge scores. The UCINet suite
of programs (Borgatti, Everitt & Freedman, 1996) was used to create the data sets for SNA and a visual
depiction of the socio-centric social network in the form of social network analysis diagrams. Each participant is
represented as a circle or node in the network with their interactions displayed as lines. These lines have
arrowheads to show the direction of each collaborative interaction. The thickness of each line represents the
number of interactions or tie-strength between each node. The minimum tie-strength is zero (0) and the
maximum is ten (10). However, for the combined medium diagrams, the minimum tie-strength is zero (0) and
the maximum is forty (40). The district that each node belongs to is shown by the color of the node. The score
that each participant receives for the collaborative learning tasks is shown by the size of each node. The
minimum score for no tasks correct is four (4) and the maximum score for all correct is nine (9). The minimum
of four (4) was chosen instead of zero (0) so that each node is visible on the diagram. Three SNA diagrams will
be presented in the next section.
A content analysis was conducted to gain a more in-depth understanding of what types of interactions were
occurring on the online discussion tool. Each message posted on the discussion board was identified and
categorized using a list of behaviors suggested by Rubin & Goldberg (1992). They include basic communication
relations such as information seeking (IS) and information providing (IP). Other behaviors that Rubin &
Goldberg (1992) term “contractual relations” are used to indicate messages where one participant posts a
message in order to obtain a response. These are coded as confirming action (CA), seeking consensus (SC),
statement of problem (SP), and statement of solution (SS). Other behaviors that showed an outcome from the
discussion were coded as making a decision (MD), notifying the occurrence of an event (NE) and volunteering
assistance (VA).
Table 1 shows the knowledge scores for each district for the pre and post surveys. The two districts (2 and 3) that
did not use the online discussion tool received a lower score and the same score respectively on the post survey.
Three out of the four districts (8, ITD and CRD) that used the online discussion tool received higher scores on
the post-survey. There was a significant effect for knowledge, t(40) = 5.41, p < .05, with discus users receiving
higher scores than non-discus users. (In this and subsequent t-tests, we verified that variance probability for
equal variance applies; Kerlinger & Lee, 2000).
% change
Table 1. Knowledge scores by district (percent correct)
Control group
Discus users
Officer Group B
Officer Group A
n = 10
n = 10
n = 42
Table 2 shows the types of behaviors examined for content analysis. Table 3 shows the results for the content
analysis of the messages posted in the discussion tool. The most common behaviors were those of IS
(Information Seeking) and IP (Information Providing). The next most common actions were RA (Requesting
Action) and CA (Confirming Action). During content analysis, it was observed that different districts sometimes
disagreed on and discussed their booking procedures. Various formatting issues were negotiated, such as
whether to include the dashes in the social security number, how to format the state adult booking number, and
what additional fields to include in the booking. Officers were used to the prior booking procedure where
booking reports were typed onto pre-printed paper reports. Previously formatting was not an issue because a
limited number of people entered the data from the paper reports into the booking system.
Key to codes
Table 2. Types of content analysis behaviors
Types of behavior
Key to codes
Information providing
Making a decision
Information seeking
Volunteering assistance
Requesting action
Raising funds
Confirming action
Seeking funds
Seeking consensus
Providing funds
Statement of problem
Other people
Statement of solution
Notifying occurrence of event
Table 3. Breakdown of behaviors by district
2 3 5
ITD CRD Total per behavior
Total per district
Table 4 shows the number of posts, replies and reads in the discussion tool by district from the server logs. ITD
was the most active in posting and replying to posts. All of the districts that used the discussion tool (5, 8, ITD
and CRD) were more active in reading messages posted by others compared to posting. The ratio of reads to
posts was almost 41:1 in the case of ITD, 50:1 for district 5 and 22:1 for district 8.
Table 4. Number of posts, replies and reads by district in the online discussion tool
Figure 2 shows the initial social network with all media combined, and Figure 3 shows the social network after
the introduction of the online discussion tool with all media combined. These two diagrams allow comparison of
differences in the social network. The initial social network shows collaboration within each district but a lack of
collaboration between districts. The post social network shows new collaboration between the districts (5 and 8)
using the discussion tool. There was a significant effect for inter-district collaboration, t(40) = 2.57, p < .05, with
increased collaboration across districts following the introduction of the tool. However, there was no significant
effect for collaboration frequency, t(40) = -0.02, p > .05, with discus users receiving similar scores to non-discus
users. Therefore the null hypothesis for H2 cannot be rejected.
Figure 2. Social network before introduction of the online discussion tool. In this and subsequent figures, **
indicates districts that used discus
Figure 3. Social network after introduction of the online discussion tool
Figure 4. Social network of interactions using the online discussion tool
Figure 4 shows the collaborative interactions between the districts using the online discussion tool (connections
via other media are omitted from this diagram). Figure 4 shows that there was new collaboration using the online
discussion tool between district 5 and district 8 and that the overall knowledge scores (represented as size of the
nodes in the graph) had increased.
Officer Comments
Comments from officers indicated that Discus was particularly useful because it was available no matter what
booking computer they were assigned to. One user commented that the ability to post attachments such as
screenshots saved time by allowing other officers to refer to the screenshot, thereby reducing the amount of
textual description needed to explain a point.
The results of changes to the social network, including collaboration patterns between individual participants and
different districts and collaboration frequency will be discussed.
Initial Social Network and Knowledge
The results of the initial social network analysis (Figure 2) indicate that officers tended to collaborate within
their own districts for information and rarely collaborated between districts. Furthermore, Figure 2 shows that
there appeared to be significant collaboration between the individuals in the Information Technology Division
(ITD), and especially between two of the central individuals (3 and 16). There was significant collaboration
between individuals in the ITD and individuals in the Central Receiving Division (CRD). Each of the districts
appeared to have a unique network structure, with the one commonality being a central or liaison individual who
collaborated with others outside the district. This central individual was not necessarily the individual who has
the highest knowledge score (34, 15, 23). Similarly, some of the individuals (10, 42, 20) who are isolated
received the highest knowledge scores, indicating that these individuals are underutilized for their expertise. The
initial survey showed that the average knowledge score across all districts was 51.6% (Table 1). In addition, each
of the districts had a different social network structure (Figure 2). The districts with the highest scores were
district 5 with 64%, district 2 at 60% and CRD at 60% (Table 1). The similarity between the network structures
of these three districts is that each of the individuals collaborated with others in the same district and also with
individuals in other districts. The collaboration was two-way. The network structure of the districts that received
the lowest pretest scores, district 8 at 53%, ITD at 42.8% and district 3 at 30% (Table 1) tended to have one or
two officers collaborate outside the district and then share the information within the district. The collaboration
appeared to be mostly one-way. Interestingly, ITD staff had one of the lowest overall knowledge scores. Relating
this to Figure 2, it appears that two ITD staff members (3, 16) are over-utilized, creating a bottleneck in the
network between other ITD staff and other districts collaborating via these two individuals.
Collaboration Between Districts (H1)
The results from the social network analysis support H1, which states that there would be an increase in
collaboration between participants in different districts using the online discussion tool compared to participants
who did not. The results indicated that the online discussion tool encouraged new ties to be formed between
participants who had not previously collaborated. There was a significant increase in collaboration between
districts 5 and 8 (Figure 3) after the online discussion tool was introduced compared to the initial social network
(Figure 2). The two districts that did not have access to the online discussion tool (2 and 3) showed collaboration
within their own district but there was a lack of collaboration with any of the other districts (Figure 3).
Collaboration Frequency (H2)
H2 states that there would be an increase in collaboration frequency between participants using the online
discussion tool compared to participants who did not. The results from the social network analysis did not show
a significant difference in collaboration frequency between the group that used Discus and the group that did not:
the null hypothesis for H2 cannot be rejected. The only significant increase in tie-strength was between
participants 3 and 12. There was increased collaboration between officers in district 2 but this was mainly due to
increased face-to-face collaboration. The level of participation in the online discussion database may explain
why the frequency of collaboration between participants was low. To overcome this limitation, these results can
be analyzed again once the online discussion tool has been used over a longer time period.
Knowledge (H3)
The results from the second survey showed an increase in average knowledge scores across all districts from
51.6% to 59.9% (Table 1). Knowledge scores across districts showed that three of the four districts (district 8,
CRD and ITD) that participated in the online discussion tool showed increased knowledge scores, while the two
districts that were not given access to the online discussion tool showed either constant (district 3) or decreased
(district 2) overall knowledge scores. The statistical test of results from the second survey supports H3: there was
an increase in knowledge for participants using the online discussion tool compared to participants who did not.
This difference suggests that the changes in the social network structure (Figure 3), in addition to practice over
time, improved learning.
Relationship Between Social Network and Knowledge
The districts with the highest post-scores were CRD with the highest score at 100%, followed by district 8 at
88% and district 5 at 55%. It appears that CRD received the highest score due to the increase in knowledge by
two key individuals (28, 1). Staff member 28 was the only individual in CRD to contribute in the online
discussion tool. The social network of district 8 changed noticeably, with increased two-way collaboration
between officers 30, 14, 40 and 34 (Figure 3). These individuals were active participants in the online discussion
tool, communicating with ITD (Figure 4). The social network within district 5 remained essentially the same.
Officers 18, 21 and 26 increased their collaborative learning scores; however, of these only officer 26
participated in the online discussion tool. It appears that officers 18 and 21 had increased two-way collaboration
to other individuals (21 to 4, 11 and 18 to 12) who did participate in the online discussion tool (Figure 4). ITD
marginally increased knowledge scores. This can be explained by the lack of participation by ITD staff in the
online discussion tool. Only staff members 38, 3 and 16 chose to participate (Figure 4), with individual 38 being
the only one with an increase in knowledge score. As part of the study design, district 2 and district 3 did not
have access to the online discussion tool. District 2’s social network structure changed as a result of increased
face-to-face collaboration. Officers 5, 32, 15 and 24 marginally increased their knowledge scores. This may be
explained by the social connections that these officers have outside of their own district. District 3’s social
network structure remained the same, as did their overall knowledge score at 30%.
The results showed that there was an increase in knowledge scores for 10 out of the 14 individuals in the online
discussion tool, with two of these (3 and 16) receiving the maximum scores in both pre and post surveys (Figures
2 and 3). There was an increase in knowledge scores for 4 out of the 16 individuals who did not participate in the
online discussion tool. These four officers (all from district 2) have social connections outside of their own
district. This overall distribution of scores suggests that the gains may be attributed to the use of the online
discussion tool. Even though there was some variation in the social network structure of each district over time
(pre versus post social network), the overall knowledge scores indicate that those districts that used the
discussion tool improved their knowledge. An appropriate example is the change in the social network structure
in District 2. They did not use the online discussion tool, but despite the network changes using other media
(Figure 2), their overall knowledge score did not increase.
Content Analysis
Results from the content analysis of the messages posted using the online discussion tool indicated that all of the
districts that had access to the online discussion tool used the board mainly to seek and provide information (IS =
19, IP = 19 in Table 3). District 5 displayed the most information seeking (IS) behavior. They have the most
integrated network structure (Figure 3). It is interesting that all districts participated in information providing
(IP), not just ITD and CRD as expected. To a lesser extent, the districts used the board to request actions (RA =
10) and confirm completed actions (CA = 14). An example of this is where a participant would ask participants
in other districts to type data into the booking module in a certain way (see informal observations concerning the
negotiation of formatting, previous section).
Participation by Reading
The logs indicate that many of the participants chose not to post messages but were active in their reading of
messages posted by others. The ratio of reads to posts was almost 41:1 in the case of ITD, 50:1 for district 5 and
22:1 for district 8. This indicates that the frequency of collaboration by posting and replying to messages was
low, with most participants preferring to only read messages. Analysis of the knowledge results indicates that
despite a low level of posting, the participants who only read the posts of others increased their own knowledge
scores. Wenger et al. (2002) suggest that the learning of “lurkers” may be of significant advantage in some
online communities. Although not included in the original goals of this study, further analysis of the behavior
and learning of lurkers in a social network should be conducted.
Conclusions and Future Work
Findings from this study have indicated that the introduction of an online discussion tool had significant effects
on the social network and learning of officers working with a booking module of the RMS at the Honolulu
Police Department. Results supported the hypothesis that there would be an increase in collaboration between
participants in different districts in which the online discussion tool was available compared to participants in
districts where the tool was not available. The results did not support the hypothesis that there would be an
increase in collaboration frequency between participants. However, results supported the hypothesis that there
would be an increase in knowledge for participants who used the online discussion tool compared to participants
who did not use the tool. These results along with the lack of knowledge gain in a district that increased its
internal social networking without CMC (the exceptions are individuals who had outside contacts) suggest that
increased interaction within a group has less influence on knowledge gains than the expanded size of a social
network enabled by CMC. This finding might be explained by the greater cognitive diversity of a larger network.
The nature of the media must also be considered: while a quantitative social network analysis counts contact
through volatile media (such as face to face speech or telephone) the same as contact through persistent media
(such as a discussion board), persistent media can reach more people over time and support reflection on the
contributions. Content analysis and the finding that lurkers achieved knowledge gains also suggest that the
discussion tool supported information sharing and coordination of action over time and space. In general, CMC’s
value to an organization should not be judged merely based on whether interaction becomes more frequent:
benefits also lie in increased connectivity and persistence of information.
This study has shown that it is possible to change the social network structure from one that presents barriers to
knowledge sharing to one that promotes learning. However, it is clear that introduction of a collaboration tool is
not sufficient by itself. For example, it is necessary to maintain activity in the online discussion tool in order to
create value that encourages further participation, which in turn creates further value. Participants need support
in learning how to use the new collaboration technology, and how to adapt it to work processes and social
processes. Technology supported media can be used to enable collaboration between participants who would not
necessarily collaborate through any other type of media, but the CMC does not operate in isolation. Various
media support collaboration within the social network, and synergy with communication in other media may
amplify the impact of the CMC tool.
This study was conducted over a six-week period due to time constraints imposed in part by movement of the
participating police officers. It would be useful to evaluate the results of a similar study held over a longer period
of time. Further research could examine other independent variables, such as the introduction of different media
types, or dependent variables, such as how the nature of the interaction changes between participants in the
social network. Studies conducted in different types of organizations and environments and examining how
different media are used for different work and social process would be of value. In general, the objective is an
empirically grounded theory that can predict how the introduction of different media types and practices
surrounding their use would affect social networks and their implications for knowledge building within a
community of practice.
The authors gratefully acknowledge the cooperation of the officers and staff of the Honolulu Police Department.
The second author was supported by the National Science Foundation under award #0093505.
Bogenrieder, I. (2002). Social architecture as a prerequisite for organizational learning. Management Learning,
33 (2), 192-212.
Borgatti, S., Everitt, M. & Freeman, L. (1996). UCINET IV Version 1.64. Natick, MA: Analytic Technologies.
Chan, C., Lee, E. & Van Aalst, J. (2001). Accessing and fostering knowledge building inquiry and discourse.
Presented at the 9th Biennial Meeting of the European Association for Learning and Instruction, Switzerland.
Cummings, J. N. & Cross, R. (2003). Structural properties of work groups and their consequences for
performance. Social Networks, 25, 197-210.
Doise, W. & Mugny, G. (1984). The Social Development of the Intellect. Oxford: Pergamon Press.
Kerlinger, F. & Lee, H. (2000). Foundations of Behavioral Research. 4th ed. Harcourt College Publishers.
Krackhardt, D. & Haythornthwaite, C. (1998). A social network study of the growth of community among
distance learners. Information Research, 4 (1). Available at: http://informationr.net/ir/4-1/paper49.html.
Kraut, R. & Attewell, P. (1993). Electronic Mail and Organizational Knowledge. Working paper, Department of
Social and Decision Sciences, Carnegie Mellon University.
Liang, D., Moreland, R. & Argote, L. (1995). Group versus individual training and group performance: The
mediating role of transactive memory. Personality and Social Psychology Bulletin, 21, 384-393.
Moreland, R. L. & Myaskovsky, L. (2000). Exploring the performance benefits of group training: Transactive
memory or improved communication. Organization Behaviour and Human Decision Processes, 82, 117-133.
Preece, J. (2000). Online Communities: Designing Usability and Supporting Sociability. John Wiley & Sons.
Resnick, P. (2002). Beyond bowling together: SocioTechnical capital. In HCI in the New Millennium, edited by
John Carroll. (647-672.) New York: Adddison-Wesley.
Rogers, E. (1995). The Diffusion of Innovations. Free Press, New York.
Rubin, K. & Goldberg, A. (1992). Object behaviour analysis. Communications of the ACM, 35 (9), 48-62.
Scardamalia, M. (2000). Principles of Knowledge Building. Fourth Knowledge Building Institute, Toronto,
Ontario, August 9-12, 2000.
Scardamalia, M. & Bereiter, C. (1996). Rethinking learning. In Olson, D. R. & Torrance, N. (Eds.), The
Handbook of Education and Human Development: New Models of Learning, Teaching and Schooling,
Cambridge, MA: Basil Blackwell, 485-513.
Sha, L. & Van Aalst, J. (2003). An application of social network analysis to knowledge building. Paper
presented at the Annual Meeting of the American Educational Research Association, Chicago, April 21-25,
Suthers, D. (2006). Technology affordances for intersubjective meaning-making: A research agenda for CSCL.
To appear in International Journal of Computer Supported Collaborative Learning, 1 (3).
Valente, T. (1995). Network Models of Diffusion of Innovations. Hampton Press, Cresskill, NJ.
Wasserman, S. & Faust, K. (1995). Social Network Analysis: Methods and Applications. Cambridge: Cambridge
University Press.
Wellman, B. (1996). For A Social Network Analysis of Computer Networks: A Sociological Perspective on
Collaborative Work and Virtual Community. Center for Urban and Community Studies, University of Toronto,
Weber, M. (2000). Modeling Technological Change under Uncertainty in Integrated Assessment Models for
Climate Change, retrieved January 7, 2004 from http://www.mpimet.mpg.de/~weber.michael/modeling_tc.html.
Wenger, E. (1998). Communities of Practice: Learning, Meaning and Identity. Cambridge: Cambridge
University Press.
Wenger, E., McDermott, R. & Snyder, R. (2002). Cultivating Communities of Practice: a Guide to Managing
Knowledge. Boston: Harvard Business School Press..
Donoghue, S. L. (2006). Institutional Potential for Online Learning: A Hong Kong Case Study. Educational Technology &
Society, 9 (4), 78-94.
Institutional Potential for Online Learning: A Hong Kong Case Study
Sue L. Donoghue
17A Island View, Midvale Village, 21 Middle Lane, Discovery Bay, Lantau Island, Hong Kong
Tel/Fax: +852 28161607
[email protected]
Hong Kong’s tertiary education environment has changed dramatically in recent years with universities
now facing specific educational challenges in the areas of critical thinking, ‘life-long learning’ and English
language. The question exists as to what pedagogic developments will best allow the universities to address
these challenges. In this paper, the appropriateness of online pedagogy as one response to these challenges
is assessed through use of illustrative case-examples and post-course surveys. The potential of The
University of Hong Kong (HKU) to implement this pedagogy is examined through a systematic
consideration of the requirements for embedding online learning, specifically student and institutional
knowledge, culture, and resources.
The case examples provide empirical evidence to suggest that online environments are useful in addressing
these challenges, largely as a consequence of afforded flexibility in teaching and learning, support of more
student-centered learning approaches, and a high degree of student engagement. Opportunity for
international collaborative teaching, with associated benefits in curriculum extension, cost-spreading and
benchmarking, is also demonstrated. Within HKU, student factors appear to pose no major constraint to
such development, but there exist significant inconsistencies in terms of institutional culture, pedagogic
knowledge and non-hardware resources. The author concludes that small-scale online developments are
viable and the cost of establishing and maintaining these need not be prohibitive. While online learning
holds promise for HKU, the University will require internal institutional change to fully realize this
Online learning, E-learning, Pedagogy, Strategic planning, Higher education, The University of Hong Kong
The tertiary education environment of Hong Kong (HK) has been subject to significant flux over the last 15
years, as a consequence of evolving political and educational objectives. A number of challenges, including
massification of undergraduate student numbers and changing graduate skill requirements, require universities to
re-examine the nature of teaching and learning.
Given these challenges, and the pervasive and integral use of personal information technology in HK society, it
is timely to examine the potential of online learning as an appropriate response to the changing HK tertiary
environment. Much has been made of the synergistic potential between online learning and the growing
educational emphasis on ‘life-long learning’, with its intrinsic demands for flexibility (e.g. Ryan, Scott, Freeman,
& Patel, 2000, p. 10).
This paper examines, through empirical illustrative case study, the degree to which The University of Hong
Kong (HKU), Hong Kong’s pre-eminent tertiary institution, is positioned to embrace online learning. There are
two considerations to this: firstly, whether online learning environments, through their design, may offer
opportunities to help address the specific challenges faced, and secondly, what is tangibly required by both the
university and the student body to embed online learning environments.
The Hong Kong tertiary environment: educational challenges
Immediately prior, and following the 1989 Secretary for Education and Manpower education strategy, there was
a veritable ‘explosion’ of tertiary student numbers in HK (UGC, 1996). While many nations struggle to
accommodate slowly increasing numbers of tertiary students as a result of societal evolution toward service
economies and ‘qualification deflation’, such expansion was overshadowed in HK by active government policy
to rapidly and extensively increase the percentage of students continuing into the tertiary sector. First-year firstdegree places (full time equivalents) at University Grants Committee (UGC) funded institutions increased from
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
approx. 4000 to over 15000 in the decade from 1985/86 to 1995/96 (UGC, 1996). Further expansion is planned
to bring the total number to around 55000 (HKSAR, 2000, par. 66).
The HK Government is in the process of changing the educational structure from the current five-year junior
secondary, two-year senior secondary, and three-year university (‘5+2+3’ structure) to a ‘3+3+4’ structure
(HKSAR, 2004). This change in educational boundaries will replace the final year of secondary education with
an additional year in undergraduate tertiary programs, resulting in a further expansion of numbers with a large
influx of younger, less mature tertiary students.
Coinciding with this massification, the HK Government is seeking to limit public purse burden by reducing the
UGC recurrent grants to the tertiary sector. Between 1998 and 2004, the UGC realized a HKD 1.1 billion
reduction in funding (UGC, 2004).
Skill demands
Students entering tertiary undergraduate programs are almost exclusively a product of the local secondary
education system. This system is generally regarded as instilling a limiting teacher-directed, assessment-oriented
culture within the student population. It focuses heavily on the lower, convergent cognitive levels of Bloom’s
(1956) taxonomy of educational objectives (Bloom & Krathwohl, 1956), and fails to develop the higher,
divergent levels (analysis, synthesis, evaluation); “Hong Kong education has always been regarded as
examination-oriented and students were taught only to memorize and not to think.” (Fok, 2002, p.83).
The UGC-commissioned Preparation of Students for Tertiary Education Report (POSTE) in 1996, found that
“the aims of the Curriculum Development Council for inculcating creative thinking and rational and independent
decision making are unlikely to be realized.” (UGC, 1996, p.2). The HK Government has determined that this
represents a serious educational deficiency that needs to be addressed, and has accordingly attempted to
introduce resources and remedial initiatives (HKSAR, 2000). The development of a ‘liberal education’
promoting life-long learning and critical thinking presents a significant challenge to higher education
institutions, and their traditional ‘divorced lecture’ pedagogy.
English language proficiency also remains a concern for the HK Government, given its role as lingua franca in
both academic and commercial environments, and the recognition that adequate numbers of multilingual
graduates are of great importance to HK’s economy (UGC, 1996). The University of Hong Kong is officially an
English-language institution, and instruction remains significantly in this language.
The POSTE report (Cheng, Lai, Lam, Leung & Tsoi, 1996), concluded that “although English is the declared
medium of instruction in many secondary schools, few of them earnestly teach in English” and that “A-level
subjects are taught in Cantonese in the great majority of schools”. This language disjoint was amplified from the
commencement of the 1998 academic year when the majority (307) of government and government-aided
secondary schools in Hong Kong were officially converted to dominantly ‘mother-language’ teaching in
Cantonese, with only 114 ‘high-achievement’ schools being permitted to pursue English-medium instruction
(Lao & Krashen, 1999, p.1). The impact of this on general education levels remains a subject of debate, although
it is likely the average level of exposure to English has been significantly further reduced at secondary level.
Chen (2001, p.2), notes that ‘many students reach university unable to construct a single sentence correctly. It is
disastrous’. It is clear, even without examining internal institutional language use that at least the ‘input’ side of
the University language equation has changed dramatically. Cheng, Lai, Lam, Leung & Tsoi’s (1996) POSTE
Report provides a summation of this change in student profile:
The ‘cream’ which used to be admitted to local universities are [sic] now most likely attracted by overseas
institutions. In other words,
a) the students who are now in our universities are very different in nature from those who entered the
universities 15 years ago;
b) the universities are now facing a large number of eligible students whereas they used to face a small number
of elite.
The immediate cost of tertiary education on the individual student has greatly increased over the last decade.
Composition fees for a fully government-funded bachelor’s degree currently exceed $42,000 p.a. and are
scheduled to rise to $50,000 p.a. A result of this partial movement toward ‘user-pays’ education is a need for
many students to have regular term-time employment simultaneously with their studies, or alternatively to
contemplate part-time, rather than full-time, study. The need to work impacts tremendously on the students’
ability to participate in the full gambit of campus-based learning activities. Already, teachers are witnessing
lower attendance rates in lectures, receiving more requests from students to extend deadlines for submitting
coursework, and observing reduced student participation in campus-based activities (Donoghue, 2002).
Many students are now choosing to remain living in the family home, rather than the halls of residence. This
reduces costs, as well as providing refuge from the extra pressures (principally the social commitments) that life
in a university hall brings. Associated with this trend are longer travelling times (often 2-3 hours per day) and a
reluctance to come to campus unless absolutely necessary. HK students are now more ‘mobile’ than before, and
the teaching and learning environment must change to accommodate this trend (Donoghue, 2002)
The HK tertiary environment can thus be characterized by:
¾ massively increased tertiary student numbers over the early 1990s
¾ the likelihood of further rapid expansion and introduction of younger students
¾ demand to reduce per unit costs
¾ students with an instilled rote-learning philosophy of education
¾ students with poor English-language skills, and of a potentially declining standard
¾ students requiring time and location flexibility to allow for other demands such as paid employment and
residence location.
Online learning: a potential response to these challenges?
Online learning offers potential benefits in addressing the changing profile of the student body and the other
significant challenges to the Hong Kong tertiary education sector, especially given what is often reported as the
top three skill requirements of graduates – communicating in English, problem-solving, and information
technology (see, for example, Fok, 2002).
In terms of language, online learning offers second-language students opportunities for reviewing and re-reading
material for full comprehension, placing all material into an overall context, and actively interacting
asynchronously with teachers and peers, both local and foreign, in the second language.
Much has been written of the flexibility in teaching and learning (removing time/place constraints) and more
effective cognitive learning (and therefore ‘quality learning’) available through online learning. The integration
of face-to-face and online learning to create ‘blended’ learning contexts has been seen to provide the
independence and control required for students to develop critical thinking (Rosenberg, 2001; Stokes, 2000;
Garrison & Kanuka, 2004). Once established, online learning, with its capability for large groups,
internationalisation, and low capital item demands, has the potential for the high efficiency and low unit costs
being currently sought in the tertiary sector.
Scales of development
In the last decade, many larger universities embarked on grand global online learning schemes that attempted to
offer entirely online institutions (‘comprehensive virtual universities’), paralleling and competing with
traditional residential university teaching (e.g. UK eUniversities Worldwide [UKeU], NYU Online, Scottish
Knowledge, Fathom). These were often for-profit collaborations with commercial partners, with emphasis on
capturing a ‘ground floor market entry’ into a perceived imminent paradigm shift in education delivery.
‘First generation’ virtual universities were characterized by massive and often unexpected capital expenditure
requirements, poor pedagogy, and an indefinite marketplace - “mistakes from which the industry is still
struggling to recover; mistakes that had an undeniably negative impact on all key user groups in the e-learning
marketplace” (Williams, 2003, p.1). The recently reported ‘salvage’ restructuring of the UKeU, originally
backed by £55m of UK government funds, provides a typical example (see, for example, Naughton, 2004).
Equivalently spectacular ‘failures’ were seen in the USA, with the US$40 million inter-institutional ‘Fathom’
initiative headed by Columbia University, and the ₤9 million Open University’s foray into the American market
(MacLeod, 2004). Such initiatives are likely to remain white elephant developments in the near future, and
innovators in this area will suffer the significant costs of both product and market development.
Such comprehensive parallel virtual-institutional development is of questionable relevance to a traditional
residential campus university like HKU, whereas selected incorporation of online learning at course level within
the existing university structure may offer rewards. The Vice-Chancellor of Kingston University summarises:
It is not clear that universities are well equipped for it [establishing virtual university developments]…
but universities are reluctant to be relegated to the supply chain. We want to be on top [as
initiators/owners], not on tap [as providers]…There is a world of difference between developing oncampus, or near-campus, e-learning, and venturing into the global e-learning business. The first does
not challenge the core purposes of the university; the second may. (Scott, 2001, p.1).
There are two objectives to this exploratory examination. The first is to confirm through illustrative case study
that online learning environments, at course level, have the potential to address some of the education challenges
facing HKU. Of particular relevance are the development of skills required for life-long learning (especially with
regard to communication and critical thinking), and the provision of flexibility in teaching and learning in terms
of when and where teaching and learning takes place (in response to the changing student profile and numbers).
The second objective is to provide some empirically-based commentary as to where deficiencies exist that
impinge on HKU’s ability to embed online teaching and learning in response to these educational challenges.
The Case Examples
Illustrative case example online courses were established by the author to provide teacher and student experience
with online teaching and learning environments across a range of course contexts. These courses have now run
over several academic years.
Three courses in earth and environmental sciences have been developed, each with their own specific teaching
and learning objectives and requirements for student and teacher engagement in the online environment.
Importantly, in all courses, the use of technology is subservient to the pedagogical function. The online
component is intended to promote ‘student-centred’ learning, involving both independent learning and the
setting of learning goals by the student, and skills acquisition and development, specifically communication
(teacher–student and student-student interaction), critical thinking, synthesis and language skills. All courses
feature inbuilt synchronous (forum) and asynchronous (chat rooms) communication tools. Course assessment is
tied to measurable outcomes in these areas and across all course components.
Hong Kong’s Natural Environment (HKNE), is offered as a trans-faculty ‘broadening’ course, and was first
made available in 2002-2003. Broadening courses were introduced as a compulsory curriculum element to
provide students with non-specialist exposure to studies outside of their degree programme.
HKNE utilizes ‘blended’ teaching and learning environments, involving online and field-based study, supported
by classroom seminars. In this course, online study involves independent examination of web-provided course
content and parallel participation in group-based discussions using the course forum (bulletin board). The
discussion topics are set by the teachers and require that students use other resources to support their learning of
these. The teacher’s role in the online discussions is facilitative. An intensive 3-day field camp provides
experiential learning about the HKSAR environment and field skills development that is otherwise unavailable
by either traditional lecture-based or exclusively online pedagogies. The enrolment quota for this course has
been increased from and initial 45 students in 2002-2003 to 90 students in 2004-2005, reflecting teacher
confidence in the effectiveness of the modes of instruction.
Geohazards is a modular short-course developed through international teaching collaboration with Massey
University (NZ). It was first made available in 2000-2001 and has run annually. Since its inception, it has
undergone two revisions (of structure and content) based on teacher and student feedback.
Geohazards is offered to three distinct groups of students in a dominantly ‘distance education’ mode, with
physical resources for extension and support tutorials being made available through the teachers in their
respective institutions. The primary groups are final year Earth Science undergraduate students at HKU (30
students) and, separately, extramural (Diploma in Civil Defence) students at the collaborating institution
(Massey University, New Zealand). Given its accessibility in terms of both content and pedagogy, the course has
also been offered in past years to HKU medical students (MBBS degree; 25 to 30 students) as a Special Study
Module for completion over their summer vacation. In this course, the assessment methods have been carefully
chosen to allow examination of the learning that has taken place in the online environment. They include
participation in group-based discussions using the course forum (bulletin board), synthesis essays, and research
posters or companion website projects. For third year earth sciences students who regularly attend campus for
other courses, viva voce-type interview assessment is also used.
Volcanic Processes, the most recently developed course, is also a modular course developed through
international teaching collaboration with Greenwich University (UK). This online course was first made
available to HKU students (30 students) in 2003-2004. It utilizes both distance-education and classroom-based
teaching and learning modes. The theory content and exercises are made available online. Students engage in
independent learning of content and group-based discussions on set-topics using the course forum (bulletin
board) with the teacher (who resides at Greenwich University) facilitating discussions from a distance. At a later
date the teacher visits HKU to run a series of condensed workshops (assessed) with HKU students using the
web-course as the basis for content.
Access for all three courses is through a registered collective domain (www.e-HKU.net) hosted on an external
commercial (US) provider at a sparing cost of around USD10/month. This allows easy, common access and
equally rapid transmission for all three collaborating institutions.
Course development is not static, but followed a purposefully reflective iterative approach (Figure 1), with
subsequent course development and modification being driven by post-course reflection, significantly directed
by data obtained from the students. For most students involved, the author’s courses have provided them with
their first foray into online learning.
Figure 1. Reflective iterative development approach used
Post-course surveys
Data from students was obtained by the use of post-course surveys. Survey questions were categorised into six
specific areas of focus: student skills upon enrolment, technology availability, perceptions of different learning
environments, value-added attributes of an online learning environment, pedagogy, and future course
Given the mix of quantitative and decision/acceptance data being solicited, questions were largely presented
through a Likert-type format (Trochim, 2002) which allows easy interrogation of both types of data. The surveys
conclude with an open-ended comments request allowing free commentary on non-defined aspects.
Survey forms are sent to students by email, with the choice of return by email attachment or printing and
posting, should anonymity be desired.
Determination of requirements
The second objective requires a determination of the requirements for introducing online environments, and an
interpretation of the HKU context with respect to these.
The implementation of online learning is dependent upon three general determinants – knowledge, reluctance
(culture), and resources (see, for example, McCormack & Jones, 1998). Knowledge includes the three aspects of
subject content, pedagogy and technology. The cultural determinant refers to attitudes of participants, whilst
resources embraces both tangible (e.g. hardware) and intangible (e.g. time) requirements. By presenting these as
a matrix in which each parameter is examined in terms of both sides of the ‘education equation’ (the student
body and the institution dimensions), and its consistency with the other parameters, a framework that allows
systematic examination of HKU’s potential for online learning development (Table 1) can be produced.
Table 1. Determinants in the implementation of online learning environments
Institution and Faculty
Student Body
Subject content
Appropriate pedagogy
Technical competency
Technical competency
Teaching culture
Learning culture
For course development
and maintenance
For access and use
Relevant data from the student body dimension is obtained through the post-course surveys. Practitioner
experience and literature (including publicly available quantitative data on resources) provides the basis for
interpreting the institutional and faculty dimension.
Relevance of online learning to HKU educational challenges
Flexibility in teaching and learning
One distinct benefit of the online environment has been the ability to overcome time/place constraints on
teaching and learning. This allows the courses to be available to wider, and in some cases larger, groups of
students (across faculties), and involve teachers from other departments (as with HKNE) and institutions (as with
Geohazards and Volcanic Processes).
Characteristics of the online learning consistently rated as beneficial by the responding students across all
courses are: being able to progress through the course content at their own speed (i.e. intensity flexibility), (98%
MBBS, 82% HKNE, 80% Geohazards); being able to work from home (location flexibility), (100% MBBS, 84%
HKNE, 73% Geohazards); and having the whole content available at all times (overview and context and selfpaced study), (100% MBBS, 90% HKNE, 80% Geohazards). “The web-based teaching method allows me to
work in the most appropriate time and therefore I think it’s a good way to learn this course” [HKNE student,
It is good to have reading material on line since sometimes we don’t know what the lecturer is talking
about and just fall asleep or daydreaming [sic]. The materials on the web allow us to read it at own
speed and ask question through e-mail or ask the lecturer directly. Thus whether the lecturer is available
for consulting or not is very important in online courses. [Geohazards student, 2000-01]
The desire for this flexibility is demonstrated by the end-of-course surveys that indicate access is typically
dominated by the 6pm-midnight and midnight-6am periods and from home or off-campus locations. Only onefifth of access is during normal working hours (6am-6pm), (see Figure 2). The ‘chat room’ function
(synchronous communication), being inconsistent with time flexibility, is also less supported than the forum and
email functions (asynchronous communication). Not surprisingly, the majority of HKNE students (82%) realize
greater flexibility in course selection as a result of replacing rigidly time-tabled lectures with web-provided
I think the site is a good tool for us to study. Because I think as a University student, we have to be
more independent in learning! And the advantages of the site can be best seen recently. It is because it
was least affected or totally unaffected by the suspension of classes. Thank you. [HKNE student, 200203]
Figure 2. Illustrative access parameters for Hong Kong’s Natural Environment 2003-04
[Data suggest the utility of a flexible learning environment]
The advantage of the flexibility and accessibility afforded by online learning environments in the life-long
learning context is obvious. This was clearly demonstrated not only by the ability for Geohazards to continue
during the suspension of classes during the SARS outbreak in HK, but also the successful offering of this course
to HKU medical students over the summer break outside of the teaching year.
Language and Communication
HKU students no longer receive the same exposure to English language as a result of the change in HK
sovereignty and the consequent introduction of Chinese medium of instruction in secondary schooling in 1998.
The promotion of Putonghua as an alternative ‘second language’ in HK, and the reduction in first-language
English lecturing staff at HKU have also been influential in this regard.
In addressing English language skills of students, there are two related considerations –increasing engagement of
students in the use of English language and improving comprehension of subject content in English.
The traditional didactic teaching context provides little assistance in either aspect; it gives very limited
opportunity for active student engagement in the language, and the verbal lecturing presentation of specialist
content is often difficult for a second-language listener. The author’s post-course surveys indicate that a
significant percentage of students (40%) describe their comprehension of English in traditional verbal lectures as
“less than 80% understood”. From this case study, online courses demonstrated potential benefits in both
engagement and comprehension aspects.
The author’s experience is that for some students, communicating verbally in English is an uncomfortable
experience. They find expressing thoughts and phrasing questions verbally in English intimidating. Developing a
‘learning partnership’, even in the small-group classroom setting, with this type of student is difficult, as they
tend to remain reticent and reluctant to engage, relying on friends to phrase questions and seek responses on their
behalf. In the online environment, however, asynchronous communication tools allowed these students to engage
confidently. They could ‘prepare’ their questions and responses before posting these, helping them to build
confidence in communicating with the teacher and classmates.
Informal involvement (i.e. not prescribed) in online discussion was universal, and many students took advantage
of the real-time chat room to relate with the teacher. The benefits of this were immediate, with students who had
interacted through the course chat rooms the night before actively seeking out the teacher on campus for
continued discussion, in a semi-mentoring role.
In terms of comprehension, many students in their post-course surveys have remarked on the several benefits
that a web-based format allows over a traditional lecture presentation: an unforced speed of presentation (reading
rather than listening), an ability to repeat and review content when comprehension is not immediate, and an
ability to ‘overview’ and ‘context’ material through having the complete content available at all times.
The post-course surveys for Geohazards and Volcanic Processes (courses which involve international
collaboration, and in the case of Geohazards, course operation in full distance education mode) indicated a
surprising student willingness (ranging from 56-73%) to have work assessed by the overseas (English-language)
teacher. The potential also exists for asynchronous group interaction with first-language English students and
teachers operating in parallel at these institutions. Not only would these features support engagement (and
therefore learning) and language development, but they would also provide a subsidiary benefit in providing
external, international ‘quality-benchmarking’ of curricula and student performance.
I believe the web-based teaching gave me control over my learning, which was most important. I thought
there could be more interaction with overseas students/teachers, e.g. perhaps include some of them into
our small discussion groups. This is the first time I’ve done a course like this, and it’s been a very positive
experience for me. Thanks. [MBBS Geohazards student, 2002-2003]
Critical-thinking and problem-solving
Critical-thinking and problem-solving skills can also be developed in online learning environments, although the
extent to which this is supported depends largely on the design of the course, i.e. the specified learning
objectives and the assessment methods used to determine learning. This is also the case with traditional
classroom-based teaching.
All case example courses were designed to promote ‘enquiry-based’ learning. All required student participation
(assessed) in online peer-group discussions, necessitating self-directed study of online content and other learning
resources (e.g. DVD, periodicals) and the reporting of ‘synthesis’ results (that evidence acquisition and
application of new knowledge) via the course forum. This learning was supported by high degree of access to a
‘virtual community’ of learners and teachers, providing for targeted and timely feedback. “I think the course is a
well-organized one and I particularly like the way we can contact and obtained [sic] help from the teachers,
either using a forum or thru [sic] email” [MBBS Geohazards student, 2002-2003]. This accessibility provides for
a more student-centred approach to learning, which is integral to life-long learning, and which is not supported
by the traditional ‘divorced lecture’ pedagogy.
Critical-thinking and problem-solving skills are further developed in the ‘blended-learning’ environments of
HKNE and Volcanic Processes which have a high degree of student engagement in the learning process (through
field-based and classroom activities). In the HKNE course surveys, responding students rated field trips and the
online environment (84% and 76%, respectively) consistently above the traditional environments of lectures
(44%) and practical classes (36%) as the ‘teaching methods’ they felt were ‘helpful’ or ‘very helpful’ to their
Individual viva voce-type assessment was offered to third-year Geohazards students. Discussion with each
student sought to determine if the course learning objectives had been met through largely independent learning
and skills (e.g. critical-thinking) further developed. This type of assessment proved surprisingly popular,
especially given that, across all courses and all student groups, consistently 33% or less of students regard
themselves as confident users of oral English language. “Oral test is quite new to me. But I think it is a very
good experience. Before I took the oral test, I was very nervous. After I took it, I feel more comfortable and
think it is a good testing method!” [Geohazards student, 2003-04].
One distinct benefit of online courses is their scalability in terms of access, and therefore the ability to ‘teach’
increasingly larger class sizes with little additional development cost. Although scalability is an important
consideration in assessing cost-effectiveness of online course developments, the fixed costs for course
development have to be met regardless of class size.
The case example courses were developed within the author’s earth sciences specialism with its small student
numbers, ranging from 30 students in Geohazards to 90 for the interfaculty HKNE broadening course. By most
standards, these developments are small-scale and therefore are likely to present the least-favourable cost-benefit
circumstance. That said, all three courses were established ground up for a modest total expenditure of around
USD50,000 demonstrating that small-scale developments are viable and that the cost of establishing and
maintaining online environments need not be prohibitive.
The author’s department, the Department of Earth Sciences (DES), is the only tertiary department offering an
Earth Sciences degree programme in HK. With an annual freshman enrolment of less than 30 students, it is too
small to feasibly offer the desired range of specialist courses for an integrated and comprehensive curriculum
internally. In the case of both Geohazards and Volcanic Processes, international collaboration importantly
allowed for development of desired expert content at a slight fraction of the otherwise untenable cost of regular
internal employment of such specialists.
While both enrolment numbers and the centralized University end-of course ‘student evaluation of teaching’
(SET) scores demonstrated the popularity and acceptance of the example case courses, the author’s post-course
surveys indicate that this does not extend to a carte blanche affirmation of online learning as an alternative to
traditional lecturing.
There is, however, a consistency in students’ determination of those general categories of courses to which an
online learning component would be suited. Typically greater than 80% of students identify ‘broadening
courses’, ‘summer study modules’ and modular courses (like Geohazards) as suitable for online learning
components. It could be inferred that such support reflects the students’ perceptions of the inherent benefits in
terms of time and place constraints. ‘Language courses’, with their strong interpersonal and verbal components
are universally identified as inappropriate for online methods. ‘Interfaculty electives’ are more mixed, with
generally around 50% identifying these as appropriate for online learning development.
Full semester core courses as a category typically find favour with less than 30% of respondents. There is,
however, a great diversity in the nature of full semester core courses, and therefore perhaps a consequent
unwillingness to assign all such courses as collectively suitable or unsuitable. Interrogation by specific course
title yields a consistent picture. When asked to categorize a listing of other core Earth Science courses as to
appropriateness for online learning, third year Earth Science Geohazards students have, over the last three years,
consistently and dominantly (greater than 80% of respondents) graded the same three core courses as appropriate
for online learning development, whilst deeming the remaining almost universally as inappropriate. It may well
be a case of ‘customer knows best’, and it may be that student commentary on pedagogical design is undervalued and under-utilized.
Feasibility of online developments given HKU’s current institutional status
The extent to which the institution currently meets the requirements for the successful development and
implementation of online learning is examined in terms of the determinants identified in Figure 2, which are
knowledge (technical competency and pedagogy), reluctance or ‘culture’, i.e. the teaching and learning culture
within the institution, and resources (provision and access).
It is apparent that in HK there is a high level of computer literacy, and student knowledge does not hinder the use
of online learning. In the author’s post-course surveys, students reiterate that training is not required for them to
effectively engage in online learning. While there are minorities who have a ‘fascination with technology’ and
regard themselves as ‘advanced users’, the majority of students, typically around three-quarters, regard computer
technology only as a tool, and themselves as but ‘competent’ users of computers.
At the institutional level, knowledge of both technical and educational principles is required to construct an
online learning environment. This combination of skills is one that few teachers have or might be expected to
have. Presently, the three spheres of knowledge needed to create online learning environments (content,
technology and pedagogy) remain noticeably divorced from one another within the institution.
Hong Kong society has been an ‘early adopter’ of internet technology, and this acceptance and use has been
especially rapid and pervasive in the ‘educated’ sector of society. As early as 2002, a HK Government survey
indicated 93.4% of all students above the age of ten were ‘internet users’ (HKCSD, 2003). In Hong Kong, there
is clear acceptance, and even anticipation, of the growing role of internet technology in society (see ITU, 2003).
In terms of acceptance of online learning specifically, it is worth noting that while it is consistent with the
desired ‘educational culture shift’ toward student-centered, flexible, life-long learning based on critical thinking
and personal responsibility, it does represent a significant departure from the surface-learning heritage of passive
spoon-fed ‘threshold’ recall in the teacher-centered, examination-oriented secondary system (e.g. Yeung, 2005,
p. E3). Practices that focus on continuing assessment, development of non-recall skills, “discourse of
participation” (Li, 1999) and ‘out-of-the-box’ extension of concepts into non-familiar contexts can elicit some
reluctance by a minority who regard formal tertiary education as a series of heavily guided and detailed shortterm assessment and content-based thresholds.
The move to more student-centred learning approaches requires commensurate change in both the teachers’ and
students’ attitudes to learning. For some students, this adjustment is difficult because of the greater responsibility
placed on them for the learning that takes place. In all courses, there was an identified minority of students who
attempted to replicate the familiar teacher-centered approach. This was illustrated by, for example, their
attendance at course tutorials not to engage in discussion with teachers and peers, but in the hope they would
receive a ‘lecture’, and their requests for highly specific exemplar assessment material. Such students found the
flexibility of online learning a challenge to their personal time-management. “It is good to have the online
material available all the time but when there are lots of other assignment (e.g. tutorial work in other courses),
working online will be put at a later priority” [HKNE student, 2003-04].
For these students, these observations suggest a distinct preference for definite threshold instruction and a
dictated time-management environment when the ‘stakes are higher’ and a consequent reluctance to assume a
high-degree of personal responsibility for the learning that takes place. The motivating desire appears to be ‘not
to fail’ rather than ‘to learn’. For the majority, the ‘leap’ to more student-centred environments is welcome, with
feedback indicating they perceive this environment as beneficial to their personal and academic development.
At the institutional level, such developments require that teachers are knowledgeable in basic educational
principles, specifically how students best learn. They must be willing and able to invest time and effort in teacher
professional development and consequent development of practice. Teachers need to be supported by an
institutional culture that recognizes and rewards teaching as a primary role of the university.
The author’s experience, in her recent role as Faculty Teaching & Learning Network (TLN) Consultant, suggests
the dominant institutional culture at HKU regards ‘education’ as a subordinate university function. By way of
illustration, a TLN survey of the Faculty’s teachers identified that 38% believed teaching performance has no
impact on promotion prospects, and almost all (94%) of the respondents indicated that their teaching has not
been recognized in any way at Faculty or University level. Almost all those surveyed indicated they spent more
time on research than teaching roles. Not surprisingly then, less than 15% of Science teachers had engaged in
teacher professional development activities or programmes offered by the staff development unit during their
tenure at HKU. Less than 5% had prepared a teaching portfolio to document their development as a teacher.
Teacher knowledge of documentation intended to guide teaching development (i.e. Departmental Development
Plans [DDP’s] and the 2002 externally-reviewed Faculty Teaching and Learning Quality Process Review
[TLQPR] submissions) was negligible.
The development of online learning at HKU does not appear limited by technological knowledge or resources,
with required hardware and internet access being pervasively available, both on campus and within the
community as a whole.
In the University, the HKU/IBM Assured Access Mobile Computing Programme, which commenced in 1998,
seeks to provide subsidized purchase of IBM notebook computers and software to incoming students. It has
historically attracted around 80% of first-year full-time (FYFT) students (see Table 2), despite the fact that
almost all such students have existing computer resources already available (HKUCAUT, 2004).
Table 2. Computer hardware availability for first-year students at HKU. Data from HKUCAUT (2004)
Percentage of HKU FYFT students taking up HKU/IBM Student Notebook Computer Programme
Percentage of HKU first year students with computer ownership (by student or student’s family) before
In addition to this private-ownership program, the University, with a full time enrolment of around 12,000,
further records 5,630 ‘public’ computers (HKUERO, 2003).
In concert with this, The University of Hong Kong has invested extensively in network infrastructure, both on
campus and in the halls of residence, in order to achieve ‘ubiquitous’ network access. Hallnet ensures “all
student residential halls are connected to the HKU Campus Network with a network point provided for each hall
resident”, and the Access Everywhere Network (ACEnet), “provides extensive roaming network access for use
by staff and students” on the remainder of the campus (HKUCC, 2004). The External Relations Office
(HKUERO, 2003) records impressive figures of 22,220 network points, 10,320 PC connection points, 450
wireless network access points, and 308 Mbps total bandwidth for direct connection to the internet available on
the campus.
Access to courses must also be readily available off campus, and in this regard, HK is exceptionally wellresourced. In the wider off-campus community, the Office of the Telecommunications Authority (OFTA)
estimates there are in excess of 2.33 million registered internet accounts in Hong Kong (March 2004), excluding
users ‘who are not customers of the licensed ISPs, such as users of the campus networks in the universities’.
(OFTA, 2004a, p.3). This represents approximately one registered account for every three citizens.
Further to the availability of internet connection, the typical bandwidth in HK is very high. As a result of the
concentrated apartment block housing, over 95% of HK households have broadband (services with downloading
speed of 1 Mbps or above) potentially available. This is complemented by the highest international internet
bandwidth per capita in the Asia-Pacific region and near the lowest broadband connection fees, allowing pricing
to become among the cheapest in the region (ITU, 2003). There has been a consequent rapid expansion and
conversion to broadband in the HK market, with an 18-fold increase in broadband accounts in the 2000-2003
period (OFTA, 2004b). By 2003, 83.7% of household internet connections were broadband (HKCSD, 2003).
Such transition in bandwidth is mirrored by the ‘home’ connection shown in representative post-course surveys
over the last few years, with the 2003 MBBS cohort the first to report 100% broadband access.
In terms of comparative statistics with respect to internet resources, be it in terms of availability, bandwidth, or
pricing, Hong Kong holds a pre-eminent place internationally, and is accordingly developing into a ‘broadband
society’ where a pervasive range of societal functions (e.g. banking, travel, utilities, government) is becoming
available online.
With respect to the institution, HKU is extremely well-provisioned with the hardware and software infrastructure
components for the ‘consumption’ of online learning. It does, however, suffer in the provision of time, training
and support necessary to ‘implement’ such an initiative. A succession of short-lived central support centers, with
their often ill-defined financing and independent service role, has historically been poorly utilized by individual
teachers. Despite the significant cost of these centres on institutional resources, the development of individual
courses through these centres is not centrally supported, but requires independent funding by the teacher.
As early as 1999, the Secretary-General of the UGC questioned institutional commitment to the provision of
these less tangible resources toward online learning development: “are institutions prepared to invest in the
facilities and the training and staff development that will be required to enable faculty to make full use of the
benefits of the technology without being overwhelmed by it?” (French, 1999, p.2)
The empirical evidence provided by the successful and continuing operation of the case examples indicates that
online learning does in fact have a role in addressing some of the education challenges facing the HKU
The degree to which online teaching and learning is an appropriate pedagogy is course and content specific. For
some courses, it may be most beneficial when developed as one of a range of integrated teaching and learning
approaches (i.e. ‘blended-learning’ environments) as in HKNE and Volcanic Processes. In other courses, it may
be beneficial to utilize full ‘distance-education’ as in Geohazards. For some courses it may offer no specific
benefits and be an inappropriate pedagogy. Students with previous exposure to online learning environments can
usefully indicate where online learning may yield specific benefits.
Developing the potential
For HKU, online learning presents a potential consistency with the formally espoused strategic direction, with
the University’s mission and planning priority statements emphasizing the development of a comprehensive
education, life-long learning and internationalisation, and the need to deploy new technologies in advancing
teaching, learning and scholarship (HKUSPU, 2003).
For effective development and utilization of online learning environments, there must be a consistency between
the determinants of knowledge, culture, and resources (Figure 3). Many historical developments have failed to
deliver the benefits expected, largely as a result of major inconsistencies or ‘disconnects’ (Figure 3) between
these three parameters.
Figure 3. Consistency between parameters favourable for online development is represented by the shaded
region; other areas indicate inconsistency in one or more of the parameters
‘Knowledge’ inconsistencies arise when there is no clear mapping between pedagogical and technical
knowledge, or where either is insufficient to allow effective development within the constraints of the other
parameters. An example of this is when an institution’s online development is a response to the demand to be
‘up-with-the play’ rather than being driven by pedagogy. This can result in the application of online teaching to
courses where it is has little potential benefit, or use of the technology at an incorrect or superficial level. Such
initiatives are often characterized by replication, where the technology maintains an existing teacher-centric
pedagogy without change, or simple redundancy, where the online development is both supplementary and
superfluous to the existing pedagogy.
‘Cultural’ inconsistencies arise when the institution’s policies and practice do not support pedagogical
developments, or where either teachers or students are unaccepting of it, despite the availability of both
resources and knowledge. Many technically-robust and comprehensive ‘platform’ developments at universities
have failed to solicit the appropriate utilization by those teachers and courses that could potentially benefit from
these and do not establish ‘critical mass’ that might embed the pedagogy at an institutional level.
‘Resource’ inconsistencies arise either when an institution lacks the physical resources to realize the
developments possible given the available knowledge and culture, or where it is seduced by the “rapture of
technology” (Ehrmann, 2002), and sponsors initiatives that are too resource-demanding in their development,
evolution, and maintenance to have any hope of long-term continuation. These latter ‘bells and whistles’
initiatives are characterized by focus on advanced technology rather than education, and consequently often
evolve extended and convoluted structure, and large technology costs. Such initiatives require a high level of
continuing technical support beyond that immediately available by or to the teacher. While being eminent
‘institutional show-pieces’, they demonstrate excessive cost for limited benefit, and are typically either transient,
with a lifespan tied to funding availability, or static after their initial development.
At HKU, there appear no inconsistencies within the student body that present a major impediment to online
development. It is apparent that the required technical literacy and technological acceptance of internet-based
teaching is established. Students are quick to identify the advantages of this methodology and determine which
courses would lend themselves to it. The reluctance of some students to assume the challenge of student-centered
pedagogies may be expected to reduce with growing familiarity. Resource availability is exceptionally high.
The major hindrances to the development of online learning at HKU appear to lie largely in institutional
inconsistencies, particularly those of culture, pedagogic knowledge, and non-hardware resources.
Institutional culture
The ‘culture’ within an institution is significantly framed by its strategic and internal policies, and consequent
reward structures. HKU is currently pursuing market ‘niching’ through elitism and ‘excellence’ largely
dominated by quantified research publication, as have many ‘first-founded’ state universities worldwide. This
developing and unambiguous research focus has strengthened a perception that teaching and education aspects of
scholarship are not the primary focus in university career paths, dominantly rewarding discipline-based
Stoll and Fink (1996), provide a categorization of institutional educational cultures, based on a matrix using
continua of both outcomes achieved and the changes in educational process applied (see Figure 4). It could be
asserted that HKU traditionally is placed in the scope of a cruising institution, i.e. one that possesses a
historically high ability intake where the students have achieved largely in spite of teaching quality. Rankings
based on absolute achievement in alternative criteria, such as research publication lists or inter-institutional
advantage in securing state research monies, rather than ‘value added’ education, often give the illusory
appearance of overall institutional effectiveness.
Figure 4. Educational culture: modified from Stoll & Fink (1996, figure 6.1, p.85)
Online developments require a large initial investment of time and application by the teacher, and presently, a
consequent perceived cost in terms of career development.
I guess it will take a long time to convince the average junior lecturer or assistant professor that their
track record in research is not much more important for their career advancement than developing a truly
imaginative and exciting web-based learning module for one of their department’s courses (French, 1999,
Without a significant Kuhnian ‘paradigm-shift’ in institutional policy and culture to offer balance and promote
the concept of HKU as a value-adding educational institution, the future of teaching-based initiatives, including
online learning, will continue to be one of fragmentary, non-integrated and transient efforts by “pedagogical lone
rangers” (D. Johnson, pers. comm.) achieving “random acts of progress” (L. Horne, pers. comm.).
Pedagogic knowledge
It is imperative that teaching developments be founded in an appreciation of pedagogical advantage, rather than a
fascination with technology or academic politics. Lee (2004) points out the required direction of development,
stating “… the only way to get maximum benefit out of educational technology is to start with the pedagogy.”
The development of the internet has given academia an opportunity to reassess the pedagogy of teaching
in higher education. It has allowed us to return to some of the true principles of teaching, such as
interaction, reflection, good resources, and student-centered teaching. It is interesting that students on
online courses report that they are getting more human interaction than on any other type of course. (R.
Mason, professor of educational technology at The Open University, United Kingdom quoted in
Williams, 2003)
‘Pedagogic knowledge’ is not generally a formal pre-requisite for university teaching. It has been said that
university teaching is indeed the most eminent professional role that is placed in the hands of those formally
unqualified in that function (T. Morrison, pers. comm.). This lack of qualification in a primary role of the tertiary
education sector can be compounded when institutional policy and culture fails to effectively provide any
incentive for engagement in reflective practice, and active improvement or professional development in that
At HKU, human resource policy has largely failed to exact and reward commitment to professional teaching
practice. The recent ‘re-inventing’ of the institution as ‘research-led’ (Yeung, 2004, p. E3), with an unambiguous
focus on research publication, has aggravated the divide between teaching and research roles rather than
sponsoring effective integration.
Non-hardware resources
Currently, teaching development within HKU is being driven by small and disconnected groups of individuals
with funding through a ‘research-model’ of small discrete grants of limited tenure. There is little opportunity or
institutional endorsement for successful teaching developments to be funded or developed as ‘going concerns’,
with a regard for extension and integration into a university-wide context. Consequently, teaching initiatives,
including those involving online teaching, tend to be “a series of attempts at creating new and alternative wheels,
with little intent at placing the wheels on the cart to get it moving” (I. Bell, pers. comm.). Such a fragmentary
approach also results in a diversity of initiatives being repeated, and a consequent “sandstorm of confused
practice and promotion” (Moore, 1998, p.6).
Both institutional policy and resource provision must recognize the ongoing nature of teaching, and support both
the initial development and the subsequent continuation and potential extension.
Resources, including funding and personnel (both technical and, more importantly, educational specialism),
should be directed to embedding proven and sustainable pedagogies. The requirement for consistency, expertise,
and avoidance of expensive replication, suggests that this may be best achieved through an accountable
centralized unit, with focus on those courses identified as appropriate.
A fellowship programme, such as the University of New South Wales ITET initiative described by Lee (2004),
may offer peer-driven extension through the academic community and resulting embedding of pedagogic
initiatives within the institution. Importantly, the inclusion of such programmes within policy signals an
institutional commitment to recognising teaching and promoting a much needed shift in culture.
The last decade has seen HKU move into a context that demands the development of new teaching paradigms in
addition to the traditional ‘sage-on-a-stage’ one-way teacher-centric lecturing. In many aspects, the limitations of
traditional teaching are becoming significantly more pronounced with the rapid environmental changes in the
HK tertiary sector that have placed this institution in an unfamiliar educational environment. Online education
appears to hold tremendous promise for HKU in addressing many of the exceptional environmental factors that
demand a greater education focus. The availability and acceptance of the technology in HK make it a feasible
and viable development for the tertiary sector. Any policy toward online learning development should be
compatible with these demands and also with the institution’s espoused mission and goals, strengths and areas of
distinction (Lujan, 2002), and assessment of cost-benefits. In the case of HKU, it is clear that the institution
wishes to continue as a pre-eminent bricks and mortar institution within dominantly state-funding. The
‘university-replacement’ virtual institution, with its massive initial capital expenditure, is currently an
inappropriate course of action. Globally the pursuit of such policies is likely to remain both expensive and
unsuccessful in the immediate, and such developments are best left to institutions with an existing ‘open
university’ or ‘extramural university’ experience base. Importantly, for HKU, with its history and credibility in
society, online learning assumes the mantle of an evolutionary pedagogy with an established client base, rather
than a contested ‘paradigm shift’ seeking to establish a market.
Focused, niche initiatives, especially those leveraged by collaborative development amongst faculties and
institutions at undergraduate level, offer clear and unambiguous benefits in addressing the changing demands of
the peculiar HKU environment. In these situations, advantage may arise through either replacement or ‘blending’
with traditional teaching pedagogies.
Presently, however, HKU is not positioned to develop the online learning potential fully. While the level of
readiness by the student body appears high, the institution has notable inconsistencies between knowledge,
culture and resources. If HKU is to develop this latent potential in a manner meriting its espoused goals in
‘world-class education’ and ‘life-long learning’, its policies must tangibly aim at shifting institutional culture and
overcoming the inconsistencies identified.
The author acknowledges the teaching collaboration with Massey University, New Zealand and Greenwich
University, and the funding support provided by The University of Hong Kong Senate Teaching Quality
Committee and the University Grants Council (Teaching Development Grant). Thanks are extended to the
reviewers for their thoughtful reviews of the manuscript.
Bloom, B. S., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational
goals, Handbook I: Cognitive domain, London: Longman.
Chen, E. (2001). Towards quality tertiary education in Hong Kong: A summary of the remarks by guest speaker
Professor Edward Chen, President of Lingnan University. HKDF Newsletter, 18, retrieved March 14, 2006,
from, http://www.hkdf.org/newsletters/0107/0107_8.htm.
Cheng, K. M., Lai, A. Y. Y. W., Lam, C. C., Leung, K. S., & Tsoi, H. S. (1996). POSTE: Preparation of
students for tertiary education: Final report: A study commissioned by the University Grants Committee (4th
Ed.), Hong Kong: Hong Kong Government Printer.
Donoghue, S. L. (2002). Online Teaching - An international approach. Proceedings of the Association of
Southeast Asian Institutions of Higher Learning (ASAIHL) Seminar on Lifelong Learning, Nanyang
Technological University, Singapore, 221-228.
Ehrmann, S. C. (2002). Improving the outcomes of education: Learning from past mistakes. Educause,
January/February, 54-55.
Fok, S. C. (2002). Teaching critical thinking in a Hong Kong secondary school. Asia Pacific Education Review
2002, 3 (1), 83-91.
French, N. J. (1999). E-Education: Are Hong Kong’s higher education institutions rising to the challenge?
Keynote address paper presented at the 5th Hong Kong Web Symposium, October, Hong Kong.
Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher
education. The Internet and Higher Education, 7, 95-105.
HKCSD (The Hong Kong Census and Statistics Department) (2003). Hong Kong as an information society
(2003 Ed.), Hong Kong, China: Hong Kong Government Printer.
HKSAR (The Government of the Hong Kong Special Administrative Region) (2000). Chief Executive’s policy
address 2000, retrieved March 14, 2006, from, http://www.policyaddress.gov.hk/pa00/eindex.htm.
HKSAR (The Government of the Hong Kong Special Administrative Region) (2004). Chief Executive’s policy
address 2004, retrieved March 14, 2006, from, http://www.policyaddress.gov.hk/pa04/eng/index.htm.
HKUCAUT (The University of Hong Kong Centre for the Advancement of University Teaching) (2004). 20
Questions about the HKU/IBM student notebook computer programme, retrieved March 14, 2006, from,
HKUCC (The University of Hong Kong Computer Centre) (2004). HKU campus network, retrieved March 14,
2006, from, http://www.hku.hk/cc/home/networks/hkunet.htm.
HKUERO (The University of Hong Kong External Relations Office) (2003). The University of Hong Kong:
Quik stats, retrieved March 14, 2006, from, http://www.hku.hk/eroesite/html/pub/index_pub.htm.
HKUSPU (The University of Hong Kong Strategic Planning Unit) (2003). The University of Hong Kong:
Strategic development 2003-2008, retrieved March 14, 2006, from, http://www.hku.hk/strategicbooklet/pdf/strategic-development_e.pdf.
ITU (International Telecommunications Union) (2003). Broadband as a commodity: Hong Kong, China internet
case study, retrieved March 14, 2006, from, http://www.itu.int/ITU-D/ict/cs/hongkong/material/CS_HKG.pdf.
Lao, C. Y., & Krashen, S. (1999). Implementation of mother-tongue teaching in Hong Kong secondary schools:
Lee, A. (2004). Institutional approaches to e-learning: UNSW and the ITET fellowship. HERDSA News, 26 (1),
Li, M. S. (1999). Discourse and culture of learning - communication challenges. Paper presented at the joint
Lujan, H. D. (2002). Commonsense ideas from an online survivor. Educause, March/April, 29-32.
MacLeod, D. (2004, April 13). The online revolution, mark II. Education Guardian, retrieved March 14, 2006,
from, http://education.guardian.co.uk/elearning/story/0,10577,1190470,00.html.
McCormack, C., & Jones, D. (1998). Building a web-based education system, New York, USA: John Wiley &
Moore, M.G. (1998). Introduction. In C. Gibson (Ed.), Distance learners in higher education: Institutional
responses for quality outcomes, Madison, WI, USA: Attwood Publishing, 1-8.
Naughton, J. (2004, March 21). A little e-learning is a dangerous thing. The Observer, retrieved March 14, 2006,
from, http://education.guardian.co.uk/elearning/comment/0,10577,1174384,00.html.
OFTA (Office of the Telecommunications Authority, Hong Kong Government) (2004a). Statistics of customers
of licensed internet service providers in Hong Kong, retrieved March 14, 2006, from,
OFTA (Office of the Telecommunications Authority, Hong Kong Government) (2004b). The groundbreaking
decade: major developments in the Hong Kong telecom industry in the past 10 years, retrieved March 14, 2006,
from, http://www.ofta.gov.hk/brochure/pdf/decade_eng.pdf.
Rosenberg, M. J. (2001). E-Learning: Strategies for delivering knowledge in the digital age, New York, USA:
Ryan, S., Scott, B., Freeman, H., & Patel, D. (2000). The virtual university: The internet and resource-based
learning, London: Kogan Page.
Scott, P. (2001, November 13). Universities should resist e-imperialism. The Guardian, retrieved March 14,
2006, from, http://education.guardian.co.uk/higher/news/story/0,9830,592147,00.html.
Stokes, P. J., (2000). How e-learning will transform education. Education Week, 20 (2), retrieved March 14,
2006, from, http://www.edweek.org/ew/ewstory.cfm?slug=02stokes.h20.
Stoll, L., & Fink, D. (1996). Changing our schools: Linking school effectiveness and school improvement,
Buckingham, UK: Open University Press.
Trochim, W. M. (2002). Research Methods Knowledge Base (2nd Ed.): Likert Scaling, retrieved March 14, 2006,
from, http://www.socialresearchmethods.net/kb/scallik.htm.
UGC (University Grants Committee) (1996, October). Higher education in Hong Kong: A report by the
University Grants Committee, Hong Kong: Hong Kong Government Printer.
UGC (University Grants Committee) (2004). General statistics on higher education in Hong Kong: UGC-funded
Institutions, retrieved March 14, 2006, from, http://www.ugc.edu.hk/english/statistics/genstat.pdf.
Williams, D. (2003, March 22). Success still in the distance. The Guardian, retrieved March 14, 2006, from,
Yeung, L. (2004). HKU marks out its global standing strategy. The South China Morning Post, September 18,
2004, E3.
Yeung, L. (2005). Self-reliance key to online English learning. The South China Morning Post, April 30, E3.
Ateveh, K. & Lockemann, P. C. (2006). Reuse- and Aspect-Oriented Courseware Development. Educational
Technology & Society, 9 (4), 95-113.
Reuse- and Aspect-Oriented Courseware Development
Khaldoun Ateyeh and Peter C. Lockemann
Fakultät für Informatik, Universität Karlsruhe, Postfach 6980, 76128 Karlsruhe, Germany
[email protected]
[email protected]
No longer can courseware providers deal with one homogeneous target group, one learning form and
possibly one pedagogical approach. Instead they must develop a broad range of courseware, each serving its
specific target group, each adjusted to a specific learning and teaching form, each appealing to its own
learning and teaching scenario, and each incorporating its own pedagogical approach, and to do all this in a
cost-effective and timely fashion. The thesis of this paper is that only an approach that is much more
dictated by software engineering principles than what has been usual so far will meet these needs. Because
of the economical constraints, the overriding engineering principle should be component reuse, and if
several distinctive concerns become interwoven – above all content, didactics and technology – component
reuse should be augmented by aspect-oriented programming. The paper develops and details a novel
courseware engineering process that combines software reuse, component technology and aspect-oriented
Courseware engineering, Courseware reuse, Aspect-oriented development, Educational system development
The goal of modern instructional design and learning theory are curricula that no longer follow a standardized
pattern but are customized to the students' prior knowledge and learner type. But the goal also raises new
challenges to courseware providers. No longer can they deal with one homogenous target group, one learning
form and possibly one pedagogical approach. Instead they must develop a broad range of courseware, each
serving its specific target group, each adjusted to a specific learning and teaching form, each appealing to its own
learning and teaching scenario, and each incorporating its own pedagogical approach, and they must do all this in
a cost-effective and timely fashion.
This makes courseware development essentially an engineering challenge. It is a challenge, though, that has long
been known to software engineering. There, component reuse is considered the key technique. Our vision, then,
is to apply this technique to the creation of courseware that is highly adaptable and modular, and thus reusable in
many contexts and for many needs. It should be possible to easily adapt the same courseware so that it can be
used by students at universities and by employees in companies, by the student who prefers to learn online as
well as by the one who prefers the traditional way of learning by printing out a script, by an instructor in the
lecture hall or by one in a virtual class room. Stated differently, the way courseware components are to be reused
depends on further factors such as pedagogical, psychological, and ergonomic aspects. Observing such factors
during the development of component-base software has been known in software engineering as aspect-oriented
This paper demonstrates that despite the differences the engineering techniques of software reuse and aspectoriented programming can successfully and profitably be applied to courseware development, although they
need to be specialized for the purpose. The paper is organized as follows. After examining the facets of reuse and
aspect-orientation and the relevant state-of-the-art we develop the main ideas of a reuse-driven courseware
development process. We introduce a model development process that allows us to separately pursue the
different educational concerns in courseware. In particular we treat domain engineering and its specialization to
content and didactics, and discuss the weaving of the concerns (or aspects) into a complete courseware. The final
chapters cover the implementation and the conclusions.
Software engineering for courseware
The thesis of this paper is that courseware development can profit from certain principles of software
engineering, specifically software reuse and aspect-oriented programming. We now take a closer look at the
implications of these principles.
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
Software reuse has two dimensions, building a stock of software components and combining components into a
system with the desired properties. The first dimension is referred to as engineering for reuse and the second as
engineering with reuse. Correspondingly in courseware development, engineering for reuse would refer to the
design and development of learning assets (any reasonably delimited material which can contribute to a learning
object) so that they can be reused later on when a specific teaching or learning environment is to be served, and
engineering with reuse to synthesizing a particular courseware.
Now, suppose a repository of learning assets that has been built up over time. Imagine a teacher or learner who
needs instructional material on a certain subject. In all likelihood he/she will inspect the repository for assets that
cover the desired content. Suppose a suitable asset has been found. Then the particular circumstances of the
teacher or student come into play. For one, these determine the didactics to be pursued, that is, the content
should be organized to account for the expected background and experience, the desired learning form and
scenario, and the pedagogical approach. For another, the circumstances dictate how the material is to be accessed
and – above all – how it is to be technically presented, e.g., visually by static or animated graphics, video
sequences, audio, or several in combination, and what technical arrangement to choose. Content, didactics and
technology constitute what we call aspects. Courseware development must observe these aspects. It is the
combination of component reuse and aspect observation that seems to pose major particular challenges.
Existing contributions to the problem
Software engineering contributions
The literature distinguishes between two main categories of software reuse approaches, component-based reuse
and generation-based reuse (Biggerstaff & Perlis, 1989; Henninger, 1997; Szyperski, 2003a). A componentbased development process describes all activities in the context of a complete software life cycle on the basis of
components. A software component has contractually specified interfaces and explicit context dependencies, can
be deployed independently, and is subject to composition by third parties (Szyperski, 2003b). This matches our
goal of deploying entire learning objects and configuring them into larger courses, following, e.g., the SEI
reference model (SEI, 2004 ).
Component-based reuse, or more precisely systematic, reuse-based development of system families in a domain
rather than one-of-a-kind systems is the objective of Domain Engineering (DE). As defined in AOSD (2003),
domain engineering is the activity of collecting, organizing and storing past experience in building systems or
parts of systems in a particular domain in the form of reusable assets, as well as providing an adequate means for
reusing these assets when building new systems. What a domain is in a specific situation is left to the consensus
of the so-called stakeholders in that domain. The practical goal of the domain engineering process is to build a
reference architecture that can be easily reused across members of a system family or families in that domain.
Domain engineering appears particularly attractive to us, since its objective seems to concur with the engineering
for reuse phase.
Generation-based reuse takes a higher-level specification of a software artifact and produces its implementation
via a generator. The users of a generator see a system that allows them to go from a specification to a software
component without having to understand the internal details of the generator (Bell et al., 1994). A specific
representative of generation-based reuse, Aspect-Oriented Programming (AOP), seems particularly suited to our
problem of separating and integrating aspects. AOP deals with separation of concerns at the implementation
level and tries to provide linguistic mechanisms to factor out different aspects of a program, which can be
defined, understood, and evolved separately (Czarnecki & Eisenecker, 2000). The goal is to provide methods and
techniques for decomposing problems into a number of functional components as well as a number of aspects
that cut across functional components, and then compose these components and aspects to obtain a final system
implementation (IBM, 2004). Unfortunately, the definition sounds very abstract, and even the three steps of
aspectual decomposition, concern implementation and aspectual re-composition by weaving remain abstract
principles. Not surprisingly, then, AOP is more an open research agenda than an existing technology that one can
readily use. First approaches are subject-oriented programming (SOP) (IBM, 2004), Adaptive Programming
(AP) (Lieberherr, 1996), and Composition Filters (CF) (Aksit, 1989), all of them tailored to object-oriented
programming. So while the concepts sound attractive to our problem, we would need to invest into further
Systematic courseware development
Traditional methods of courseware development have a tendency towards courseware for specific learning
situations, learner characteristics, learning objectives, or learning/teaching strategies. Consequently, there has
been scant need for systems that are adaptable to varying needs. Rather the result is a monolithic structure that
mixes different aspects such as content, didactic, and technical aspects in inseparable ways, and where one of
these aspects dominates the others.
Take the didactic-oriented model that concentrates on the didactic aspect of courseware development. Most
prominent is the instructional design (ID) model that tends towards the design and development of so-called
tutorial and drill-and-practice systems (see, e.g., Tennyson & Rasch, 1995; Merrill et al., 1990). While
considered outdated by today‘s learning/teaching methods objectives, the model still is interesting because of its
strength in its methods for analyzing learning situations, characteristics of the learners, learning objectives, and
for designing suitable learning/teaching strategies. Or as another example, take technology-oriented models that
focus on the technical and engineering aspects of courseware under the assumption that the use of multimedia
and/or hypermedia as well as technical tools will significantly enhance the learning/teaching process. Typical
examples can be found in (Boles & Schlattmann, 1998; Garzooto et al.,1991; Isakowitz et al., 1995) or as
commercial products (Macromedia, 2004a; Macromedia, 2004b; ToolBook, 2004). What these models have to
offer to other approaches, though, are the sophisticated authoring environments and a structured development
process divided into phases of information objects production, authoring, and generation. Courseware
engineering models seek to combine the strengths of the two other models. They consider courseware
development as a mixture of software development along the line of a process of developing business software,
and instructional design as the process of developing instructional or didactic models. However, the main focus
of these models is to provide a systematic development process that allows one to manage the growing
complexity of courseware development. Little or no consideration is given to courseware reuse. Examples are
the Essener-Learn-Model (Pawlowski, 2001) and the IntView Lifecycle Model (Grützner et al., 2002) as well
some recent research results (Blumstengel, 1998; Klein, 2002).
Even a focused, monolithic system can find many users provided it is accessible from outside places. This has
been recognized over the past years by the eLearning community which has come up with various standards in
order to ensure reuse and interoperability. Relevant standards are the Learning Object Metadata (LOM) standard
(IEEE, 2003), the IMS Content Packaging (IMSCP) standard (IMS, 2004), the IMS Learning Design standard
(IMSLD, 2004), the Learning Technology Systems Architecture (LTSA) standard (IEEE, 2001), and the
Sharable Content Object Reference Model (SCORM) (ADL, 2004). The standards cover nearly all aspects of
learning such as didactic, content, learner profile, learning management system, and their use seems essential for
the support of reusability at an inter-community/inter-organization level and of interoperability among
courseware/learning management systems.
Clearly, any effort towards more flexible courseware development should take cognizance of these
standardization efforts even though their large number certainly is confusing. In addition, they make assumptions
on the development process that do not seem entirely realistic. One assumption is that a component can be
reused in new contexts just as it is. This seems unrealistic for the content and didactic aspects: In our experience
these seem highly interdependent, so that one cannot simply transfer a component into a different didactic
environment without adapting its content. Under IMSLD, even the instructional or didactic models are designed
with a specific learning context in mind so all one can do is reuse the corresponding components if their context
happens match the selected or designed instructional strategy.
There has been some work on generating courseware from components, given they fit the requirements. Merrill
introduces the concept of instructional transaction shell to organize existing so-called instructional transactions
and knowledge objects into new courseware (Merrill et al., 1990). On the other hand, little is being said about the
process of the development of such reusable elements, rather the focus is on the process of automatic generation
of instructions from existing reusable instructional transactions and knowledge objects.
In conclusion, we face a number of challenges. It should be possible to design courseware in small units, with a
clear separation of the various aspects, in our case predominantly contents and didactics. Their interdependence
should be taken into account when courseware is produced for a specific learning/teaching context.
Consequently, there should be a clear separation of the modeling phase – corresponding to engineering for reuse
– and the production phase – corresponding to engineering with reuse. The production phase should largely be
automated, somewhat in the sense of model-driven programming.
Courseware development process model
The challenges as discussed in the previous section seem to suggest a crude process model consisting of two
major processes, engineering for reuse and engineering with reuse. Figure 1 gives an overview. The two
processes are referred to as domain engineering and course engineering. Courseware engineering must start with
domain engineering because it is this process that lays the foundation for course families and, hence, for a
variety of courseware even for the same topic, where the differences reflect the situation to be served. Only when
a family has been established – at least in part – can individual family members be developed. In practice,
though, the processes are not strictly separated but iteratively performed. For example, if we assume a repository
to hold all developed assets the repository may not entirely satisfy all current needs so that one may decide to
ignore an asset and build a new, suitable one from scratch. The result will be stored in the repository and thus
become part of some other development.
Both processes are themselves iterative processes. The domain engineering process is relatively independent of
the course engineering process whereas the course engineering process strongly depends on the results of the
domain engineering process. Therefore, there is only limited potential for conducting the two in parallel. On the
other hand, new requirements that have a general character and have not yet been considered in the domain
engineering process but are first discovered in the course engineering process are fed back to the domain
engineering process.
Figure 1. An overview of the development process
Both processes are further divided into phases. The division according to Figure 1 is only effective if, beyond
associating clear milestones with each phase and allowing for collective development by the various experts, one
can clearly identify which phases from domain engineering contribute which results to which phases of the
course engineering process. The answer to this problem is the subject of the remaining chapters.
How do the software engineering techniques we plan to employ contribute to the courseware development
¾ Domain engineering is geared towards the development of families within a given knowledge domain.
¾ Aspect-oriented programming allows one to concentrate on different aspects one at a time (a principle often
also referred to as separation of concerns).
¾ Component technology allows one to assemble the learning assets into larger but still self-contained courses.
Domain engineering is the general principle that underlies our process model. Aspect-oriented programming
gives substance to the principle and will affect all three phases of the domain engineering process. We consider
three aspects:
¾ The contents of the course (content aspect): The instructor needs to decide on a syllabus of the course,
choosing which (sub-)topics to include, how much emphasis to put on each, and deciding on an order of
The didactic strategy and methods to use (didactic aspect): The instructor needs to decide which didactic
strategy is suited best to reach the objectives of the course.
The learning/teaching context or constraints under which the course will be taught (technical aspect): These
include the number of participants, the amount of time available, the room and the technical equipment that
will be used, whether the course will be face-to-face or distance learning, whether teaching will be
synchronous or asynchronous and so on. We will not discuss the technical aspect in detail but include it in
the didactic aspect whenever appropriate.
Figure 2. Influence of existing techniques on the development process
Finally, component technology is the basis for constructing the learning objects for a specific course from the
assets and should follow one or more of the established techniques for constructing components.
Figure 2 summarizes how the three techniques impact the domain engineering process and where, according to
the earlier discussion on systematic courseware development, instructional design and standards exert an
The domain engineering process
The gross structure of domain engineering consists of the three phases of domain analysis, domain design, and
domain implementation. All three are mainly dominated by considerations of the domain knowledge – the
contents – and of three independent aspects – content, didactics, and technology. Below we study the effects
mainly of the first two factors on each phase. A course on database systems will illustrate the development
Domain analysis
Content analysis
During content analysis the scope of the knowledge domain is selected and defined. It is the responsibility of a
larger community to agree on the general topics that should be covered by the domain model as well as those to
be left out. Figure 3 gives some examples. The main contributions come from material used by the community
members in conducting their own courses, from reference textbooks, from knowledge domain experts in the
community, and perhaps from available knowledge domain ontologies. As part of the discussion one should
refine the general concepts into more specific ones, assign terms to them and collect these terms in a vocabulary.
Didactics analysis
This phase considers both, content didactics and process didactics. The latter covers general properties such as
teaching/learning goals and contexts, potential audiences. The former focuses on the definition and selection of
the various didactic principles through which the learning materials will be submitted to the students. Figure 3
shows some examples.
Obviously, didactic principles are generic, i.e., independent of a specific knowledge domain. Consequently, they
can be specified just once and early on, and can then be applied to a broad range of knowledge domains.
Figure 3. Sample result of the domain analysis
Domain design
The purpose of the domain design phase is to develop a more formal courseware model from which one can
systematically derive an implementation.
Content design
Content design defines a shared content model for the content concern. The content model aims at the creation of
a shared understanding for the knowledge domain that is of interest to the members of the community
independent of its use in a specific learning context (Ateyeh et al., 2003). Basically one determines the
conceptual entities that form a common basis for the reuse and exchange of courseware content, where each unit
can be (re)used by each member of the community.
Ontologies appear well-suited for the purpose. An ontology is an explicit specification of a conceptualization,
where a conceptualization is an abstract, simplified view of the world that we wish to represent for some
purpose. The purpose of an ontology is the shared understanding of some domain of interest (Uschold &
Gruninger, 1996). It usually takes the form of a set of concepts (e.g. entities, attributes, processes, etc.), their
definitions, and their relationships.
The basis of a knowledge domain ontology are the concepts identified during analysis. These are organized into
a relational structure, with the relationships dictated by the purpose of courseware. Two types of relationships
seem sufficient:
¾ isSubTopicOf: Two terms A and B have relationship B isSubTopicOf A if A is more generic than B. E.g.,
“relational algebra” is more generic than “join operator”: join operator isSubTopicOf relational algebra.
isPrerequisiteFor: Two terms A and B have relationship A isPrerequisiteFor B when the knowledge of A is
needed to understand B. In our example, knowledge about functional dependencies is needed before tackling
normalization: functional dependency isPrerequisiteFor normalization.
Figure 4. Sample domain knowledge ontology (White arrow: isSubTopicOf, Black arrows: isPrerequisiteFor,
both relationships are transitive)
Designing an ontology is by no means trivial, nor are there commonly accepted standard methods for building
ontologies. We refer the reader to the work of Holsapple and Joshi (2002) on collaborative ontology design.
Figure 4 shows a very small part of an ontology for the domain ''data base systems'' that captures the topic of
normalization. The resulting ontology can then be implemented using one of the many ontology representation
languages such as RDF/RDFS, OWL, DAML, KIF, etc. The different representation languages provide different
levels of formality to capture different kinds of ontology characteristics and semantics. We found RDF to be
sufficient four our purpose.
We note that content analysis and content design have to be done just once for each knowledge domain. Ideally,
one could then defer the combination of ontology and didactic principles to the domain implementation phase.
Content didactics design
As pointed out earlier, content and didactics are interdependent so that one cannot simply transfer a component
into a different didactic environment without adapting its content. Indeed, we found it extremely difficult to
delay their combination past content didactics design. Instead we merge the ontology with the didactic
principles, i.e., the intended didactic usage of the content, into what we call learning atoms. Figure 5 gives a few
examples. Figure 5 also demonstrates that atom types may belong to terms on any level of the ontology.
Learning atoms that belong to the same term in the ontology are combined into a learning module. Hence, for
each term in the ontology there exists exactly one corresponding learning module with all learning atoms related
to that term. As we move upwards to higher levels of the ontology, more complex modules are constructed that
include modules from the lower levels. Consequently, the notion of learning module is recursively defined.
Figure 5 sketches an example.
Since there is a learning module for each term in the ontology, modules are already implicitly defined during the
creation of the ontology. By instantiating an atom from a certain atom type it automatically becomes part of the
appropriate module (and implicitly also of the higher-up modules through the isSubtopicOf relationship).
Process didactics design
We focus on two aspects of the so-called learning process didactics:
¾ The Learning/teaching strategy: Structures the learning/teaching process into one or more phases. Each
phase meets a specific didactic goal. For every learning/teaching phase a suitable didactic method is
The Didactic method: Is a system of didactic rules that instructs or guides students or teachers while
learning or teaching. A didactic method achieves certain learning or teaching goals under given
As an example for a learning/teaching strategy take the problem-based learning/teaching strategy of Merrill
rooted in the constructivism learning theory (Merrill, 2000). A second example and the one we use to illustrate
our approach is the three-step learning/teaching strategy of Figure 6 with three main phases:
¾ Introduction phase: It defines the starting point and the goals of the learning process.
Working phase: This phase consists of further sub-phases (didactic functions): setup, workthrough,
apply, transfer, assess and integrate, and should be supported by a suitably stimulating environment.
Completion phase: The goals of the learning process must be secured.
Independent of the didactic functions one can define didactic methods. These are shown in Figure 6 below the
solid line to indicate that, at least in principle, each may be associated with each didactic function.
In the literature, learning/teaching strategies and didactic methods are mostly (if at all) expressed in natural
language. Clearly this is not suitable in a cooperative environment that tries to increase courseware reuse and
that should be based on automatic processing of courseware entities. What is needed is some kind of formalism
that allows to capture the didactic concern and to easily adapt them to a given learning/teaching context. Figure 7
graphically illustrates our own didactic metamodel. The figure also indicates (shaded box) where the didactic
methods are connected to the didactic functions. Further, the figure demonstrates how to integrate the technical
aspect with the didactic aspect. The implementation has been done in XML.
Figure 5. Sample learning modules
Figure 6. Classical three-step learning/teaching strategy and associated didactic methods
Figure 7. The didactic method metamodel
Domain implementation
In this phase concrete reusable courseware entities (learning assets) for the models defined in the design phases
(content entities, didactic entities, software tools) are implemented and then kept in a repository for subsequent
use in the course engineering process. They serve as a starting kit for the courseware development in the course
engineering process.
Content implementation
Strictly speaking, what we defined in the design phase are learning atom types and module types. Atom types are
instantiated to one or more learning atoms: The content is developed in detail and perhaps with different foci
depending on the background of the learner and the intentions of the teacher, representations are chosen such as
powerpoint slides or pdf files, and both are combined into an executable unit in a form that can be reused in
different environments and situations (for an example, see Figure 8). The atoms are stored in a content
The repository is organized into containers. A container maintains the contents of a learning module.
Consequently, there exists a container for each term in the ontology. At a minimum, for a lowest-level term a
container consists solely of learning atoms. In general, a container may hold atoms as well as “smaller”
containers for the less generic modules. Taking the example of Figure 5, there are containers for functional
dependency (with atoms only), dependencies (it would include the functional dependency container),
normalization, etc.
Didactics implementation
In a sense, domain design “atomizes” the learning material into a combination of local contents and local
didactics. However, learners are to be offered self-contained courses. A course is subject to a content didactic
strategy. Such a strategy starts from a didactic function, determines the associated didactic method and defines in
the form of a procedure (Figure 7) the didactic principles through which to reach the students, and is represented
by a so-called didactic template. The central idea of our approach is to include in the repository a number of such
templates as assort of preplanning the potentially desired course structures. Figure 9 gives an example for a
didactic template that could be used for a presentation during the setup phase. Suppose we apply it to an SQL
course module. The template states that the course should begin with an overview followed by a motivation and
an activation atom. Subsequently, for each sub-topic of the course an explanation or a definition of the topic is
needed followed by an example. Finally the course should finish with a concluding example.
Since our goal is to automatically generate a course from the content repository and a given template, templates
should be formulated in a machine-processable script language. For the purpose, we developed DidaScript, a
typical script language with the usual constructs for variable declarations, assignment statements, conditional
and loop expressions. DidaScript includes built-in functions for the navigation in a course structure of learning
modules, for ordering all or selected atoms of a specified module, and for hiding atoms in a module (Ateyeh,
2004). The resulting course structure is represented via XML.
Figure 8. Sample learning atoms
Figure 9. The classical three-step learning/teaching strategy augmented by didactic methods and the association
of a content didactic template
Figure 10. A sample course specification
The course engineering process
Domain engineering is engineering for reuse and results in a host of learning assets and organizes them in an
organizational structure. Course engineering is engineering with reuse and employs the results of the domain
engineering process to construct from them, with moderate effort, context-specific courses in the form of
learning objects (We use the term ''course'' as a general notion that refers to any learning/teaching event such as
course, seminar, lesson, etc.).
We pursue two goals. One is to fuse the outputs of content engineering and didactics engineering (or in AOP
jargon, to weave them). The second goal is to do the fusion as much as possible in an automated fashion, by
drawing on the repository.
Requirements analysis
The goal of this phase is to analyze and define the (general) requirements for the course, such as the
learning/teaching topics, target group, objectives, learning form, time frame etc. In this phase both results from
the analysis and the design phase of the domain engineering process are used. The requirements are expressed
through so-called general specifications for the learning/teaching event (see Figure 10 for an example).
Course configuration
Didactics configuration
Based on our initial experience we recommend to start the course configuration with didactics. As a first step the
requirements are re-formulated as a learning strategy. Consider nested queries as a subtopic of an SQL course.
Using the classical three-step learning/teaching strategy we proceed as follows: In the introduction phase,
existing knowledge about SQL should be recapitulated, and it should be motivated why nested queries are
needed. In the working phase, the syntax of nested queries should be introduced (set-up), some examples should
be shown (work-through and apply), given natural language requests for specific information nested queries
should be formulated (transfer), nested queries should be compared to alternative ways to obtain the same
information (assess) and nested queries should be used together with other SQL constructs (integrate).
In the next step the learning strategy must be translated into an assignment of didactic methods. To return to our
example of nested queries, the introduction phase could require students to try and formulate queries with the
SQL constructs they already know, revealing the need for nested queries (active structuring). Then, a
presentation of the syntax of nested queries followed by the online presentation of some examples could cover
the set up and apply phases, the students could then individually formulate nested queries to gain information, a
collaborative discussion comparing nested queries to alternative formulations would be an appropriate method
for the assessing phase. A computer session, requiring the usage of nested queries together with other SQL
constructs could serve both to integrate and to secure the knowledge gained (Figure 9 is a good illustration).
Given these assignments, we choose an appropriate – hopefully predefined – didactic template from the
repository. The template in turn defines the didactic principles to be applied to the material. Together with the
content structure we now know which learning atoms and modules to select from the repository.
Content configuration
During content configuration we have to decide on the precise content (learning modules). The process of
didactics configuration gives numerous suggestions on how to organize the content. Therefore, didactic and
content configuration run somewhat in parallel. Remember that a learning module contains all types of material
from all participants that can be associated with the corresponding term in the ontology. Consequently, the
course that evolves from choosing the topics (modules) from the ontology reflects the structure predefined by the
isSubtopicOf relationship of the domain-specific ontology. The instructor can now adapt this structure to his
requirements. In particular, she/he selects from the predefined structure those sub-topics (modules) she/he
intends to address in her/his course, and defines structural relationships between the selected topics which define
the navigation structure that has to be used by the students and/or the instructor. The development platform may
support her/him by warning her/him to avoid a structure that would violate the relation isPrerequisiteFor defined
in the domain-specific ontology. To continue our earlier example, we would obtain for the part of the course that
deals with normalization the sequential structure of Figure 11. Comparing it with Figure 4 we note that all direct
sub-modules of normalization were selected, whereas for module normal forms the sub-module BCNF has been
Weaving a course
Given the template and the course structure, we can mechanically draw the necessary learning atoms and
modules from the repository and weave them into a course flow. Automation is in the form of a generator.
Concrete learning atoms can be drawn from the repository by the generator merging, in a stepwise fashion, the
learning atom type from the content structure and the didactic principle from the didactic template. Figure 12
illustrates the principle for our normalization example. The course generator selects from the root learning
module only those learning atoms that follow either the didactic principle of overview or motivation. These are
then ordered so that the overview atoms precede the motivation atoms. Further, the strategy states that for each
sub-module of the root module an explanation or a definition of the topic should be given followed by an
exercise or an example. Finally the course generator visits the root learning module again to select a concluding
example followed by an exercise and the solution for that exercise.
Figure 11. Sample course structure
Assembly, test and integration
The course modules resulting from the configuration phase are normally half-finished modules. E.g., for some
learning topics appropriate learning atoms may not be found in the repository. In this case, an appropriate
learning atom must be developed and added to the repository for future use. In this case, the instructor switches
back into his role as an author, develops the appropriate material and adds it to the repository. Furthermore, in
many cases it is necessary to add a transitional content between the different learning atoms as well as contextspecific atoms such as an overview, a motivation, and a summary for the whole course. In many cases it is also
necessary to adapt the layout of some of the learning atoms to meet a selected standard layout for the whole
We are now in a position to complete Figure 2 (Figure 13).
Proof of concept
Project background
The work underlying the paper evolved as part of the five-year program ViKar (Virtual University Karlsruhe)
that, among others, undertook to explore how to develop shared course material that could subsequently be
adapted to the needs of various schools of higher learning with different educational goals. Our focus was less on
content and more on methodical issues of how to prepare the individual courses. Hence, we simply used an
already existing suite of one-semester database courses that were of interest to all participating schools. The
student sample was relatively broad: Third-year computer science students of a university, second-year business
school students of a university, third-year students of a polytechnical school, and third-year students who
alternated between the school and work at industry. Consequently, there were considerable differences with
regard to both the selection and depth of the teaching material, and the didactic strategies. Our approach evolved
slowly as we tried to deal with the – sometimes sobering – experiences, often as mundane as differing notations
and terminology, or incompatible examples for illustration and exercises.
Figure 12. Weaving content and didactics: Example
Figure 13. Figure 2 refined
Much of the effort went into domain engineering. Over the years a complete ontology for database technology
was developed, driven both by standard textbooks and the powerpoint presentation of a complete 16-week
university course at 3 hours per week, augmented by course material from the other three schools. Once a
significant portion of the ontology was available we were able to systematically determine a large set of learning
objects (atoms and modules) for the university courses and selective sets for the other schools, and complete
them using much of the existing material.
We could thus demonstrate that working from an ontology is indeed a viable approach to domain engineering. In
particular, the approach is very well-structured and gives conciseness and unambiguity to the notion of learning
object. We are certain that the same approach can be applied to all disciplines with a well-structured domain,
such as the natural and engineering sciences.
Technical platform: SCORE
To support the experiments we developed a prototypical learning system, SCORE (System for COurseware
REuse) (SCORE, 2004). SCORE distinguished between different participants of the learning/teaching process
such as author, instructor, and student, and provides a suitable environment for each of these groups.
SCORE is based on standard technologies that support openness, portability, and reusability, particularly
relational/XML databases (Oracle9.1), Java, J2EE and XML, learning technology standards such as LOM and
IMS content packaging, and design patterns such as Three-Tier architecture and Model-View-Controller. Figure
14 shows the system architecture of SCORE with three layers:
Figure 14. SCORE architecture
The data base and data access layer (layer 1) manages all types of data needed or produced during the
processes of courseware development and reuse, learning, and teaching. In particular, this layer includes a
courseware repository that manages the different kinds of courseware entities such as learning objects
(learning atoms, modules, course module) and didactic entities (didactic methods, learning/teaching
strategies, didactic templates).
The basic services layer (Layer 2): As the core of the SCORE system the layer provides a modular
framework of common tools and services that are necessary for conducting different activities or tasks by
the different user types. These tools or services are reusable building blocks used to assemble or build
complete user-specific (author, instructor, and student) environments at layer 3 (see below). The services or
tools found in this layer can be classified into three categories:
¾ Courseware development tools. These tools are used in the course engineering process and help the
developer to conduct the different activities of the development process in an efficient way. Examples
for tools in this category are AtomBuilder, ModuleBuilder, DidacticBuilder, and CourseBuilder.
¾ Learning/teaching tools. These tools are used in the learning/teaching process. From the course
engineering point of view, these tools are considered reusable technology components that can be
combined to provide a suitable learning/teaching environment for a specific learning/teaching method
and content. Examples for tools in this category are CoursePresenter, Annotation, Chat/Mail.
¾ General tools. These are tools that are used by the different user types. Examples for such tools are the
OntoBrowser and ScoreSearch.
The user environments layer (layer 3): It includes specialized environments for each user type (author,
instructor, and student). We restrict our discussion to the author environment as the one most relevant to this
paper. The environment provides tools that allow one to effectively conduct the courseware development
process. The components are integrated within a tool we call ''Course Wizard (CW)''.
¾ The AtomBuilder is used to build a new learning atom. It provides a comfortable user interface for
describing a learning atom by SCORE metatadata and assigning it to a learning topic (module) in the
domain-specific ontology. For the production of the atoms themselves, the author is able to use any
available content production tools.
¾ The ModuleBuilder supports the course (content) engineering process. It is mainly used to deal with
learning modules and provides all functionalities necessary for transforming a (general) learning
module into a (learning context specific) course module. Therefore, the ModuleBuilder makes use of
other tools such as the OntoBrowser and ScoreSearch.
¾ The DidacticBuilder deals with the learning process didactic discussed. It is used to find, select, create,
and reuse learning/teaching strategies and methods.
¾ The CourseBuilder allows to build complete learning/teaching events (course, seminar, …). It is used to
merge and integrate the different components of the different concerns to a complete learning/teaching
¾ The TestBuilder supports the creation of tests for the students.
Figure 15 Course Wizard: Screenshot
Figure 15 shows a screen shot of the CW with, from right to left, the ModuleBuilder, the DidacticBuilder and the
CourseBuilder. The screen shot nicely shows how the user interface reflects the aspect separation of content and
didactic. A complete example of how to use the CW in practice can be found in (Ateyeh, 2004).
Evaluation and conclusions
After reviewing the state of the art, we set ourselves several goals. It should be possible to design courseware in
small units, with a clear separation of the various aspects, in our case predominantly contents and didactics.
Their interdependence should only be taken into account when courseware is produced for a specific
learning/teaching context. There should be a clear separation of the modeling phase – corresponding to
engineering for reuse – and the production phase – corresponding to engineering with reuse. The production
phase should largely be automated, somewhat in the sense of model-driven programming. Our approach was
modeled after the software engineering techniques of software reuse, component technology, and aspect-oriented
The combination of these techniques poses challenges even under ordinary circumstances and has never before
been applied to courseware engineering. As a first attempt in this direction we feel we have been quite
successful. Content is the driving force. Through a domain engineering approach that is based on a commonly
agreed ontology a first component structure is defined. Component adaptability is achieved by introducing the
concept of learning resource type. It allows refining an ontology-based component in a large variety of ways that
each suit a specific instructional intent. The resulting so-called learning atoms can then be implemented and
maintained in a repository.
We also managed to separate the aspects of contents and didactics. We could show that one can derive a host of
didactical methods for a given didactical strategy, and then detail the method in the form of a template that
prescribes how atoms from the repository should be composed and ordered to meet the instructor’s and/or
student’s objectives. A specific course can be generated almost automatically by a weaving process that takes a
template as a prescription for which atoms to draw from the repository and how to arrange them in order.
Nonetheless, the results are only a first and still modest step towards the flexible and economical production of
courseware for a large variety of settings. One weakness of our work is the dearth of empirical evidence on the
suitability of our approach. The approach evolved slowly in the course of a five-year project on a virtual
university. Originally the emphasis was on the content level by ontology engineering. During the course of the
work it became apparent that the desired flexibility could only be achieved if content and didactic were dealt
with separately. This resulted in the concepts of didactic template and aspect weaving by generation.
Unfortunately, the short remaining time of the program did not allow us to test the strategy on a larger scale.
As a consequence, more empirical evidence must be gathered to convince oneself that the separation of content
and didactic works under many conditions, and what these conditions are. Likewise, before undergoing the vast
effort of building an ontology criteria would have to be developed whether a subject area is well-structured
enough to justify the effort. As a first step, one should apply the approach to additional domains beyond the
subject area of database courses. Also, for reasons of economy, it seems incumbent to integrate the SCORE
courseware development and reuse environment into existing commercial learning platforms that support the
learning technology standards and provide (to some extent) a modular learning/teaching environment.
We were not entirely successful to separate the aspects of contents and didactics. Although we did better than
other current work, we still had to merge content and didactic principles – as judged from our goals –
prematurely during the domain design phase. Future work should concentrate on how to shift more of the
merging into course engineering.
Another characteristic of the approach is its heavy reliance on the existence of a suitable ontology. At least
initially ontologies will have to be developed quite often, and this requires a fairly large investment in time and
competent manpower and, therefore, there are considerable delays before the first course becomes available for a
particular subject area. Consequently, one should steadily observe the market for ontology development tools.
The authors gratefully acknowledge the collaborative and stimulating environments of ViKar with Wolffried
Stucky, Klaus Gremminger, Rudolf Krieger, Daniel Sommer, Jutta Mülle, and of SCORE with Birgitta KönigRies and Michael Klein.
ADL (Advanced Distributed Learning Initiative) (2004). Sharable Content Object Reference Model (SCORM):
SCORM 2004 3rd Edition, retrieved May, 30, 2006 from http://www.adlnet.gov/scorm/downloads/index.cfm.
AOSD (2003). Aspect-Oriented Software development, retrieved May, 30, 2006 from http://aosd.net.
Aksit, M. (1989). On the Design of the Object-Oriented Programming Language Sina. Ph.D. thesis, University of Twente.
Ateyeh, K. & Klein, M. & König-Ries, B. & Mülle, J. (2003). A Practical Strategy the Modularization of Courseware
Design. In: Professionelles Wissensmanagement – Erfahrungen und Visionen: Adaptive E-Learning and Metadata, Luzern.
Ateyeh, K. (2004). Reuse-Driven Courseware Engineering. Ph.D. thesis, Universität Karlsruhe. Shaker Verlag.
Bell, J., Bellegarde, F., Hook, J., Kieburtz, R. B., Kotov, A., Lewis, J., McKinney, L., Oliva, D. P., Sheard, T., Tong, L.,
Walton, L.& Zhou, T. (1994). Software Design for Reliability and Reuse: A Proof-of-Concept Demonstration. In
Proceedings of the Conference TRI-Ada.
Biggerstaff, T. J. & Perlis, A. J. (1989). Software Reusability, Concepts, and Models. Vol. 1. ACM Press. Addison-Wesley.
Blumstengel, A. (1998). Entwicklung hypermedialer Lernsysteme. (Development of Hypermedia Learning Systems, in
German) Ph.D. thesis, Universität Paderborn.
Boles, D. & Schlattmann, M. (1998). Multimedia-Autorensysteme: Grafisch-interaktive Werkzeuge zur Erstellung
multimedialer Anwendungen. (Multimedia Author Systems: Graphical-interactive Tools for the Development of Multimedia
Applications, in German) LOG IN 18 (1), 10-18.
Czarnecki, K. & Eisenecker, U. W. (2000). Generative Programming. Addison-Wesley.
Garzotto, P., Paolini, P. & Schwabe, D. (1991). HDM – a Model for the Design of Hypertext Applications. In Proceedings of
the 3rd Annual ACM Conference on Hypertext Systems, 313-328.
Grützner, I., Pfahl, D. & Ruhe, G. (2002). Systematic Courseware Development Using an Integrated Engineering Style
Method. In: Networked Learning NL2002, Berlin.
Henninger, S. (1997). An Evolutionary Approach to Constructing Effective Software Reuse Repositories. ACM Trans.
Software Engineering and Methodology (TOSEM) 6 (2), 111-140.
Holsapple, C. W. & Joshi, K. D. (2002). A Collaborative Approach Ontology Design. Comm. ACM 45 (2).
IBM (2004). Subject-Oriented Programming, retrieved May, 11, 2006 from http://www.research.ibm.com/sop/.
IEEE Learning Technology Standards Committee (LTSC) P1484.1/D9 (2001). Draft Standard for Learning Technology
IEEE Learning Technology Standards Committee (LTSC) WG12 (2003). LOM Meta Data Standard. URL:
IMS (2004). IMS Content Packaging Information Model Version 1.1.3 Final Specification, retrieved May, 30, 2006 from
IMSLD (2004), retrieved May, 11, 2006 from http://www.imsglobal.org/learningdesign/index.html.
Isakowitz, T., Stohr, E. A. & Balasubramanian, P. (1995). RMM: A Methodology for Structured Hypermedia Design. Comm.
ACM 38 (8).
Klein, M. (2002). Courseware Engineering: Ein Vorgehensmodell zur Erstellung von wiederverwendbaren hypermedialen
Kursen. (Courseware Engineering: A Procedural Model for the Development of Reusable Hypermedia Courses, in German)
Ph.D. thesis, Universität Karlsruhe.
Lieberherr, K. J. (1996). Adaptive Object-Oriented Software: The Demeter Method with Propagation Patterns. PWS
Publishing Company.
Macromedia (2004a). Authorware 7, retrieved May, 11, 2006 from http://www.macromedia.com/software/authorware/.
Macromedia (2004b). Director MX 2004, retrieved May, 11, 2006 from http://www.macromedia.com/software/director/.
Merrill, M. D., Li, Z. & Jones, M. K. (1990). Second Generation Instructional Design. Educational Technology 30 (2), 7-14.
Pawlowski, J. M. (2001). Das Essener-Lern-Modell (ELM): Ein Vorgehensmodell zur Entwicklung computerunterstützter
Lernumgebungen. (The Essener-Learning-Model (ELM): A Procedural Model for the Development of Computer-Supported
Learning Environments, in German) Ph.D. thesis, Universität Essen.
SCORE (2004). SCORE Metadata, retrieved May, 11, 2006 from http://www.ipd.uka.de/SCORE/xsd/score_v1.xsd.
SEI (Software Engineering Institute) (2004): Component-Based Software Development. retrieved May, 30, 2006 from
Szyperski, C. (2003a). Component Technology: What, where, and how? In Proceedings of the 25th International Conference
for Software Engineering, 684-693.
Szyperski, C. (2003b). Component Software: Beyond Object-Oriented Programming. 2nd ed. ACM Press.
Tennyson, R. D. & Rasch, M. (1995). Instructional System Development: The Fourth Generation. In: Automating
Instructional Design: Computer-Based Development and Delivery Tools. Springer, 33-78.
ToolBook (2004). ToolBook, retrieved May, 11, 2006 from http://www.toolbook.com/
Uschold, M. & Gruninger, M. (1996). Ontologies, Principles, Methods and Applications. Knowledge Engineering Review, 11
(2), 93-155.
Bender, D. M., & Vredevoogd, J. D. (2006). Using Online Education Technologies to Support Studio Instruction.
Educational Technology & Society, 9 (4), 114-122.
Using Online Education Technologies to Support Studio Instruction
Diane M. Bender
College of Design, Arizona State University, P.O. Box 872105, Tempe, AZ 85287-2105, USA
Tel: +1 480-965-8684
[email protected]
Jon D. Vredevoogd
School of Planning, Design, and Construction, Michigan State University, 309 Human Ecology, East Lansing,
MI 48824 USA
Tel: +1 517-353-3054
[email protected]
Technology is transforming the education and practice of architecture and design. The newest form of
education is blended learning, which combines personal interaction from live class sessions with online
education for greater learning flexibility (Abrams & Haefner, 2002). Reluctant to join the digital era are
educators teaching studio courses (Bender & Good, 2003), who may be unaware of the possibilities and
benefits of teaching with technology. The argument proposed in this study is that blended learning will
enhance studio courses. Studios are unique learning environments embedded in an historical context. This
article presents a process of infusing a traditional studio with online technologies. The result is a more
streamlined course that enhances student learning, provides targeted instruction to individual students,
serves a larger group of students than a traditional studio, and does not increase faculty workload.
Online education, Studio, Faculty workload
Technology is moving higher education from the traditional campus of “brick and mortar” to the electronic
classroom of “wire and chip.” The technology with the greatest potential for impact on architectural education is
the Internet. By the year 2006, it is estimated there will be 900 million computers and other web-based
appliances in use, providing universal access, multimedia resources, and interactive medium (Charp, 2000). The
Internet has revolutionized the design process by allowing architects and designers to research new products,
download specifications, access code information, transfer drawings, and even synchronously collaborate with
colleagues from around the world. Technology is also having an impact on the process and culture of
architectural education, as evident in previous research on digitally immersed classes and studios (Matthews &
Weigand, 2001). The use of digital media is a logical addition to the traditional design studio.
Technology has radically changed the way educators can exchange information with students. Higher education
support systems have adapted to changes in technological innovation, but the studio has remained disturbingly
constant. Pedagogical integration of digital media is critical to the success of future architecture and design
education. With the ever-increasing need to communicate globally, distance is no longer a barrier to education
(Matthews & Weigand, 2001). Based on several years of experimentation, the authors believe studio courses can
be enhanced with online technologies. A popular format for teaching both in the classroom and online is blended
learning. Blended learning involves both traditional face-to-face instruction, where both students and faculty are
present at the same time and place, supplemented with asynchronous and/or synchronous communication via the
Internet. In comparison to courses that are completely online, courses with a combination of online and face-toface interaction produce the same or better success rates, plus dropout rates are lower (Dziuban & Moskal,
2004). The authors are not advocating technology as a substitute for the existing process, but as a means to
enrich instruction in the design studio.
There are many benefits of using online technologies, such as the accuracy and consistency of data. All students
are guaranteed to receive the same presentation material and get the same view of the professor and material,
unlike the front row advantage in the traditional classroom. Students appreciate the on-demand access of online
classes, for it allows them flexibility in viewing course information at their convenience and as many times as
they wish. This information may include course syllabus and outline, staff profiles and contact information,
examples of projects, and other useful items typically provided to students in a traditional course. Electronic
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
collaboration allows students to propose, experiment, refine, and resubmit ideas. The following study examines
the impact of blended learning on the traditional studio, in particular the impact on student learning, accessibility
of course material, and faculty workload. The goal is to increase the quality of interaction without losing the
master-apprentice relationship established in the traditional studio.
The Traditional Design Studio
The concept of today’s architecture and design studios began with the French Royal Academy and continued
with the methodologies of the Ecole des Beaux-Arts. It became traditional for schools in the United States to
pattern their instruction in Beaux-Arts format, in which the studio is the central focus of the curriculum (see
Figure 1). Studios (where drawing, debate, and analysis of design take place) are considered more of an active
learning experience than a lecture-style classroom.
Figure 1: A Typical Studio Environment
The strength and assurance of the Beaux-Arts approach was influential on the creation of architecture and design
programs in the United States in the early 20th century (“Architectural Education”, 2000). Since that time, studio
instruction has essentially remained unchanged. Students then and now attend a studio where instruction is
delivered from master to apprentice within a small group setting. Faculty to student ratio can range from 12 to 24
students per instructor, with an average of 17 to 1 (Ochsner, 2000; Design Futures Council, 2005). Studio classes
may range from four to 12 hours per week, during which the instructor works with each student independently
for short periods of time. It may common for a student to wait almost three hours for a few minutes of insight
and direction. While waiting, a student may not have the opportunity to view and hear the critique addressing
the work of classmates. The more students present in the studio, the less time can be spent with each student.
Because of the long tradition of one-on-one instruction, faculties in the arts frequently generate significantly
more weekly contact hours than faculty in other disciplines (Lawn, 1998).
Perpetuated by past experience, design educators continue to teach with the Beaux-Arts method, even though
there is little evidence to suggest that it is an effective means of instruction (Rapoport, 1983, 1984). Since
architectural design is viewed as a subjective process, the defense of the studio instruction format has been very
feeble and unsuccessful (Wooley, 1991). The current model of teaching and learning is not used because it is the
“right way”, but because the method has worked for so long a time (Farrington, 1999).
The traditional studio is but one method of teaching architecture and design. Other forms of instruction are used
in combination with the studio, such as large lectures, small group sessions, and classes held in computer labs
(see Figure 2). The incorporation of technologies commonly used in online education into the traditional studio
can alter the interaction between the faculty and the student (or the master and apprentice). It is hypothesized that
blending technology with traditional instruction will impact typical studio problems of high student to faculty
ratio and high faculty workload.
Figure 2: Alternative Education Environments
The Modified Design Studio
This case study focused on an introductory course in Structural Systems for integration with online teaching
technologies. This course is offered at a large public state university and is offered to students in majors such as
building construction, engineering, interior design, landscape architecture, and packaging. After years of
teaching this course as a traditional design studio with multiple class sections, the authors sought to streamline
the teaching process. This course was a logical choice in which to add online technologies, as the students were
creating their projects with Computer-Aided Design (CAD) software. The existing six hours of studio times were
divided into a one-hour weekly lecture by the faculty and four hours of computer lab time supervised by
qualified teaching assistants. A course website was created to house course materials and the faculty members
were available outside of lecture via email and web-based conferencing.
The proposed model expands the educational opportunity to participate in the design critique to more students, in
more places, and in more ways than ever before. Blended learning is being used in various educational
environments outside architecture and design (Jonasson, 1997; Ravaglia, 2001). This form of mediated education
brings many benefits to the studio. One limitation is the need for both parties to be physically present in the same
location at the same time. The proposed model streamlines the interaction loop by incorporating technology in
the teaching and delivery of feedback to the entire class of students at one time. The instructor can post course
materials and students can access it from any place and at any time.
Typical studio interaction involves one student requesting feedback and one faculty providing it directly to that
student. Though individual feedback is beneficial for student learning, the lower faculty to student ratio typical
in a studio is impractical. As educational institutions continue to increase enrollments by expanding their
“territory” through online education, faculty must deal with larger courses, often as multiple sections of the same
studio course. A typical class size for this course is 100 students. Traditionally, this would equate to five sections
of 20 students with six contact hours each, for a total of 30 faculty contact hours per week. The new model pars
this down to one hour of lecture by the faculty, plus time spent training teaching assistants, answering email,
doing conference calls, and preparing online critiques, for a total of approximately five hours per week. Teaching
assistants cover 20 hours of computer lab time each week.
Figure 3: The Modified Design Studio Process
The new model involves 12 steps (see Figure 3). As in a traditional studio, each student works toward
completing one or more major projects for the semester. Each week, parts of the project are due and critiqued in
lecture. In a study of traditional studio interactions, students often can choose when to work in isolation and
when to be interact socially (James, 1996). In this course, computer lab time is available for students to work
independently or in small groups; lecture time is for interaction with the whole class.
Students draw their projects by hand and with CAD. Each student reports his weekly progress by scanning the
drawing into an image file, or by capturing the CAD drawing off the computer screen by depressing the Print
Screen button on his keyboard. The student then pastes the image onto a blank Microsoft PowerPoint© slide. He
may resize and add text as necessary. Assignments are saved to Adobe’s Portable Document File (PDF) format.
This allows the drawing file to be compressed, while retaining the layout and graphic presentation of the
submission. Using available file transferring software, he submits a copy of the image to the course website. The
goal is to use basic technology, which every student will have on his computer, such as the Print Screen key on a
keyboard, or free software to watch MP3 movies and view PDF files. All assignments are time and date stamped
when submitted electronically, keeping the students more accountable for submitting their work on time.
The process then transfers to the instructor or teaching assistant, who assembles all the image files into a single
collection representing student work for that week (see Figure 4). Student work is placed in alphabetical order by
last name. This allows the faculty to compare current work with previous work to ascertain progress (or lack of).
Confidentiality is maintained by removing all names from the slides.
Figure 4: Example of Summary Image File
Figure 5: Example of Creating Audio Critique
The instructor reviews each student’s submission to generate a critique in the form of a 20- to 30-minute MP3
audio file (see Figure 5). This audio critique can be recorded while speech recognition software is running in the
background, such as Nuance’s Dragon Naturally Speaking. This provides a written text of the critique as an
alternative form of feedback or to accommodate diverse student learning styles. Not every slide receives
comments each week, as some comments are redundant or have little impact on the class as a whole. A student
may receive comments on his individual project one week and not the following week. All comments are in one
audio file. Students hear comments directed at both their project and those of their classmates. The audio file and
summary image file are then uploaded to the course website, where they are archived and accessible throughout
the remainder of the semester. This format maintains a permanent record of student work and participation.
Finally, the student reviews both files prior to the next weekly lecture. He is able to note his individual progress
in comparison to his peers, which can be used as a measure of progress toward course goals (James, 1996).
Student Learning
Integrating blended learning with the traditional studio can increase student learning. Technology provides
several benefits to students. It can be used to present information in a variety of formats, accessible at all times,
and leaving live class time for the intellectual communications that only people can provide. The one-hour
weekly lecture focuses on interaction with and between the students, reviewing a specific project with the whole
class or doing problem solving in informal groups. The instructor uses this time to gauge the progress of the
whole class, which is an important component of other technology-enhanced courses (Gilbert, 1999). It was
believed that combining face-to-face meetings and online education would better provide access to instructional
materials, facilitate group work, enhance the quality of interaction, and provide a superior level of feedback to
Positive factors of learning with technology such as accuracy and consistency are hard to surpass. All students
viewing the lecture are guaranteed to receive the exact same lecture material, regardless of the instructional
model implemented (Charp, 2000). All students get the same audio and video materials, unlike the front row
advantage in conventional classroom situations (Romiszowski, 1988). Students who have difficulty grasping
concepts from printed materials may be more at ease with learning experiences that engage all of their senses.
Information technologies are predominantly visual rather than relying primarily on audio and text in the
traditional classroom. Because of this, complex content can be conveyed more effectively due to the integration
of multiple representations of material such a computer animations with audio and video. These alternative
technologies can provide an additional degree of motivation, allowing students to express themselves in new and
creative ways (Dias, 1999). In addition, students hear all critiques of all projects and benefit from feedback to
their peers. In a traditional studio, individual faculty comments are rarely shared with the entire class.
Faculty Workload
Many instructors are integrating computers into their architecture and design curriculums (Budd, Vanka, &
Runton, 1999; Andia, 2002). Like them, the authors believe a major advantage to adding online technologies to
the curriculum is the efficiency of instruction. In the traditional studio, it may be common for a student to wait
two or more hours for a few minutes of insight and direction. With blended learning, the student gets more
succinct feedback in less time. In addition, he learns to communicate a creative design solution with the latest
technologies used in today’s design offices. Becoming familiar with these technologies is an advantage to the
student as it helps prepare him to progress in professional practice offers career opportunities in digital
architecture fields (Proctor, 2003).
This model of interaction provides a number of advantages to the studio instructor. First, there is an opportunity
to see the range of responses before making comments. He can quickly ascertain the effectiveness of each
assignment and make any necessary adjustments in the instructional format. Tracking student work from week to
week can also be done easily. The history and process of design is clearly articulated through the layering of
information. This allows the faculty to watch the project evolve from concept to final presentation (Norman,
2001). Second, faculty fatigue is reduced because critique duplication and the amount of time necessary for
giving and receiving project feedback substantially decrease. This is a common problem in large classes with
multiple sections where critiques are limited by time and space (James, 1996). Third, the feedback is streamlined
because the instructor spends less time repeating the same information. He can increase the turn-around time for
feedback to larger groups of students. Several other advantages are provided in Table 1.
Table 1: A Comparison of the Two Instructional Models
Traditional Design Studio
Modified Design Studio
Assignments are introduced by the instructor and Assignments are introduced in the same manner but are
submitted by the students during class time.
submitted electronically 24-48 hours before class time.
Class size is typically 15-20 students to one instructor Class size remains the same but the instructor can handle
per section. The same instructor may oversee two or an increased number of sections with assistance.
more sections.
The individual critiques provided in class are seldom Students receive feedback via the audio critique and
shared with other class members. Therefore, the same group feedback during the weekly lecture period.
feedback may be repeated to several students within the
same class period.
The critic repeatedly corrects the same or similar student The critic needs to address student errors only one time.
The competitive nature of design classes limits the
sharing and nurturing of ideas and causes many students
to work independently.
Students have to be physically present in studio when a
critique is given. Unless recorded, it cannot be reviewed
later time.
Guest critics must travel to the class site during a
specified class period to participate in critiques.
Work done in the studio may turn into a collaborative
effort between the student and the instructor, making it
difficult to determine what part of the project the student
has done. The instructor must be cautious about doing
the assignment for the student.
Design skills are strengthened by the student’s ability to
build on the feedback of other students’ projects and
Students can review critiques on demand and can
“attend” the critique from remote locations.
Guest critics can review the projects and provide an
audio critique (independent or with others) at their
The instructor does not guide any single student through
the design process, but guides the class as a whole.
It is apparent this form of instruction has limitations. With the array of necessary technology, the instructor must
devote time to learning these tools. The authors of this study were self-taught. Likewise, time must be spent
teaching the tools to the students so that they feel comfortable using them. This is particularly true for students
and educators who are only knowledgeable of traditional studio instruction (Sagun, Demirkan, & Goktepe,
2001). Data from the latest NEA faculty survey indicate the level of technical support is an important factor in
determining whether a faculty member has positive or negative feelings toward various forms of distance
education (NEA, 2000). There is an obvious need for faculty development offered by the institution.
In the new model, the instructor is not able to spend much personal time with each individual student. Though
student interaction is high in the computer lab sessions, the traditional studio setting may provide the students
more opportunity to socialize and network than in this format. Depending on the submissions received, the
faculty may or may not address individual student work each week. Some students may have difficulty gleaning
insight from comments not made directly to them. In a national survey of online education faculty, the average
number of students in their courses is 26 students (Schifter, 2000). When much larger classes are accommodated
in this online format, a potential downside is that each student will receive less individualized attention from the
This process has been successful for introductory and advanced CAD courses. However, this approach may not
be appropriate for every course because online activities cannot replace hand-drawing and rendering exercises,
or activities requiring textual artistic media. In addition, educators may prefer to balance the lecture and lab
times in a different ratio to allow more personal time with students. While student expectations are consistent
between this model and traditional studio, assignment submission procedures are different. All assignments are
now submitted electronically, freeing the student from the physical restraints of time and place. There is also a
shift from instructor time devoted to teaching, to time spent on course development and management. The
development and administration of a new course using online technologies can take more time than traditional
studio methods (Bender, Wood, & Vredevoogd, 2004). Once initial course development is complete, faculty
workload decreases due to increased instructor experience (Visser, 2000).
Future Research
Several issues need further exploration. One issue is setting boundaries for implementing new technology into
curriculum. Similar to other findings (Visser, 2000), the development time for this course was higher than a
traditional course. The formal individual interaction time noted in this paper can also be misleading. Email is
exchanged between faculty, teaching assistants and students for additional communication throughout the week,
which has been found to contribute to a heavy instructor workload (Lehman, 1996). Unless carefully monitored,
the instructor can find himself available to students 24 hours a day, seven days a week, and spending more time
answering email, critiquing projects, and working with the teaching assistants than interacting in a traditional
studio setting. These concerns lead to a need for research in the area of course management strategies.
Issues of Acceptance
With the implementation of online technologies, the traditional studio environment and the instructor’s workload
have been altered with the proposed process. Though many design educators are reluctant to embrace teaching
with technology, the authors believe the addition of online technologies enhances studio education. The
proposed alternative approach to studio instruction may provide potential solutions to the unease and resistance
of these other faculty. In order to change the culture of the existing studio environment, digital media must
become a transparent tool for design inquiry. Instructors should give serious thought to the complex pedagogical
issues underlying new methods of teaching and learning before implementing them in the curriculum.
Though this form of blended learning provides many benefits to students and faculty, the personal component
should not be eliminated from studio education. To rely solely on virtual instruction poses a serious risk, as “live
instruction has to build upon the virtual part and vice versa” (Van Eijl & Pilot, 2003, p. 55). As rapid changes
occur in architectural practice, student populations, and the design process, continued research is necessary in
order to understand technology’s impact on architectural education.
This study demonstrates that blended learning can revolutionize instruction in the design studio. Student learning
can be enhanced by having pertinent course material available online whenever students wish to access it.
Project critiques can be delivered in both audio and text format. They can be reviewed at any time and as many
times as needed. All students can view and hear projects and comments of the entire class, which is something
that is often missing in studio.
This model also exemplifies the ability to serve a larger body of students without increasing faculty workload.
The student enrollments of architecture and design programs are increasing at a faster-than-average rate in
comparison to other disciplines. It is anticipated this trend will continue through 2010, exacerbating the problem
of increased faculty workload (ASID report, 2004). The use of technology in the studio allows faculty feedback
to be less repetitive and more streamlined. More students can be accommodated in this model than the traditional
studio. If blended learning can be used to positively impact student learning with the same amount of faculty
effort, the integration of technology into the studio environment will be viewed in a more favorable light.
Abrams, G. & Haefner, J. (2002). Blending online and traditional instruction in the mathematics classroom. The
Technology Source, retrieved 10 May 2006 from http://ts.mivu.org/default.asp?show=article&id=970.
Andia, A. (2002). Reconstructing the effects of computers on practice and education during the past three
decades. Journal of Architectural Education, 56 (2), 7-13.
Architectural Education, (2000). A brief history. Association of Collegiate Schools of Architecture. Retrieved
August 5, 2004 from https://www.acsa-arch.org/students/education.aspx.
ASID Report. (2004). The interior design profession: Facts and figures. Washington, DC: American Society of
Interior Designers.
Bender, D. M. & Good, L. (2003). Attitudes of higher education interior design faculty toward distance
education. Journal of Interior Design, 29 (1/2), 66-81.
Bender, D. M., Wood, B. J. & Vredevoogd, J. D. (2004). Teaching time: Distance education versus classroom
instruction. The American Journal of Distance Education, 18 (2), 103-114.
Budd, J. Vanka, S. & Runton, A.(1999). The ID-online asynchronous learning network: A ‘virtual studio’ for
interdisciplinary design collaboration. Digital Creativity, 10 (4), 205-214.
Charp, S. (2000). Internet usage in education. T.H.E. Journal, 27 (10), 12, 14.
Design Futures Council (DFC). (2005). 2005 Architecture & Design Schools Rankings Issue Overview, retrieved
May 6, 2006 from http://www.di.net/article.php?article_id=374.
Dias, L. B. (1999, November). Integrating technology: Some things you should know. Learning and Leading
with Technology, International Society for Technology in Education, 27 (3), 10-13.
Dziuban, C. & Moskal, P. (2001). Distributed learning program impact evaluation, retrieved 10 May 2006 from
Farrington, G.C. (1999 July/August). The new technologies and the future of residential undergraduate
Gilbert, J. (1999). But where is the teacher? Learning and Leading with Technology, 27 (2), 42-44.
James, J. (1996). The construction of learning and teaching in a sculpture studio class. Studies in Art Education,
37 (3), 145-159.
Jonasson, J. (1997 June). The Internet: The educational medium of today. Educational Media International, 34
(2), 88-93.
Lawn, R. J. (1998). Integrating the arts and technology. Educational Technology, 38 (6), 56-59.
Lehman, J. D. (1996). Computer networking in distance education (147-155). In Boschmann, E. (Ed.) The
electronic classroom: A handbook for education in the electronic environment, London: Learned Information.
Matthews, D. & Weigand, J. (2001). Collaborative design over the Internet. Journal of Interior Design, 27 (1),
National Education Association (NEA). (2002). A survey of traditional and distance learning higher education
members. Retrieved February 26, 2006 from http://www.nea.org/nr/nr000614.html.
Norman, F. (2001 April). Towards a paperless studio. Proceedings from the ARCC Spring Research Conference
at Virginia Tech, 85-91, retrieved 10 May 2006 from http://www.asu.edu/caed/arcc/articles/AAINDEX.PDF.
Ochsner, J. K. (2000). Behind the mask: A psychoanalytic perspective on interaction in the design studio.
Journal of Architectural Education, 53 (4), 194-206.
Proctor, G. (2003). Digital Foundations: Building a base for digital futures. In Bermudez, J. & Klinger, K.
(Eds.), Digital technology and architecture, Association for Computer Aided Design in Architecture, white
Rapoport, A. (1983 October). Studio questions. The Architects’ Journal, 55-57.
Rapoport, A. (1984 October). Architectural education. Architectural Record, 100-105.
Ravaglia, R. (2001 August/September). E-education: The case for blended e-learning. Education West, 12-13.
Romiszowski, A. J. (1988). The Selection and Use of Instructional Media (2nd Ed.). New York: Nichols.
Sagun, A., Demirkan, H. & Goktepe, M. (2001). A framework for the design studio in web-based education.
Journal of Art and Design Education, 20 (3), 332-342.
Schifter,C. C. (2000). Compensation models in distance education. Online Journal of Distance Learning
Administration, 3 (1), retrieved 10 May 2006 from http://www.westga.edu/~distance/schifter31.html.
Van Eijl, P. & Pilot, A. (2003). Using a virtual learning environment in collaborative learning: Criteria for
success. Educational Technology, 43 (2), 55.
Visser, J. A. (2000). Faculty work in developing and teaching web-based distance courses: A case study of time
and effort. The American Journal of Distance Education, 14 (3), 21-32.
Wooley, T. (1991). Why Studio? The Architects’ Journal, 20, 46-49.
Montazemi, A. R. (2006). The Effect of Video Presentation in a CBT Environment. Educational Technology & Society, 9 (4),
The Effect of Video Presentation in a CBT Environment
Ali Reza Montazemi
DeGroote School of Business, McMaster University, 1280 Main St. West, Hamilton, ON L8S 4M4, Canada
Tel: +1 905 529 7070 Ext. 27434
Fax: +1 905 521 8995
[email protected]
Multimedia is increasingly used to enhance human-computer interaction in a variety of formats (text,
graphics, audio, animation, and video). The challenge is how to use the most effective presentation format
that would result in performance improvement. This paper addresses the added value of video
presentations in a technology-mediated learning environment for a basic Management Information System
course at MBA level. We postulate two hypotheses that take into account students’ mastery learning and
satisfaction with regard to the usefulness of video presentation, moderated by students’ self-interest to learn
the subject matter (i.e., intrinsic motivation). Our analysis, on the basis of empirical research, shows that
the subjects find a learning environment supported with video presentations is more useful. Their intrinsic
motivation to learn the subject matter had a significantly positive effect on their satisfaction with the
availability of video presentations. Video presentations had no significant effect on their performance on
exams, however.
Video presentation, Multimedia, CBT
The most significant applications of multimedia are in (1) education and training, (2) communications,
coordination, and collaboration, and (3) entertainment. With the growth of the new market for multimedia, end
users, application developers, and content providers face the challenge of how to effectively manage, organize,
and access the vast amount of information they accumulate. The focus of this paper is on the effect of video
presentation for education and training in the context of technology-enabled learning media. There is an
increasing expectation that information technology will enhance the learning process (Benbunan-Fich, 2002).
This expectation is based on the notion that information technology, by supporting interactive instruction, will
encourage students to be more responsible for their own learning. In this endeavor, the design of information
technology should be based on well-grounded theories of learning processes (Alavi and Leidner, 2001; Hannafin
and Rieber, 1989; Leidner and Jarvenpaa, 1995). For example, Hannafin and Rieber (1989) caution, “The
emergence of computer-driven ‘hybrid’ technologies … has spawned unprecedented interest, yet advances in
technological capability alone no more improve instruction than sharpened pencils improve prose.” More than a
decade later, a research essay by Alavi and Leidner (2001) calls for greater depth and breadth of investigation
into technology-mediated learning. It seems that in spite of the large number of reported research findings, we
know little about the effective means of applying and managing information technology in support of the
learning process. This is mainly due to the intricacy of the task at hand, for the dynamic nature of the interaction
between the environmental factors (i.e., instructional strategy and information technology) that affect the
learning process is complex (Alavi and Leidner, 2001; Carroll, 1997; Leidner and Jarvenpaa, 1995).
For example, let us consider the usefulness of dynamic visuals such as video presentations when used in support
of learning. Video presentation is used routinely in support of classroom lectures to enhance the learning
process. Inclusion of video in support of learning is expensive, and this means that we must make sure that it
benefits the learner. This paper presents the findings from an empirical investigation into the effectiveness of
video in a computer-based tutoring environment.
Computer-based tutoring (CBT) systems have been used in diverse learning environments to enable the learner
to self-pace in acquiring the pertinent materials. In such systems, feedback is a critical part of effective learning,
and it is expected that active involvement will lead to more effective learning than will passive involvement.
CBT systems are generally based on the stimulus-response-feedback views of learning that are associated with
the objectivist model of learning (Leidner and Jarvenpaa, 1995; Montazemi and Wang, 1995-a; Retalis and
Papasalours, 2005; Stone, 2001). Thus, CBT can be viewed as enhancing the cognitive information processing
of learners by tailoring the learning process to individual needs (Leidner and Jarvenpaa, 1995; Montazemi and
Wang, 1995-b). The learner is guided by the CBT system so that he or she reaches the predefined learning
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
materials built into the system. Therefore, CBT systems can be developed for learning environments with a
well-defined structure, pre-defined learning objectives and possible solutions to the learner’s enquiries.
Multimedia technology can enrich the learning experience by making it possible to present the learning materials
in different formats. The question raised in this paper is whether inclusion of video presentations in a CBT
system is useful to the learner. To this end, the following two hypotheses are postulated and empirically tested:
HYPOTHESIS 1: Inclusion of video in a CBT system has no effect on the learner's perception regarding the
usefulness of the system.
HYPOTHESIS 2: Inclusion of video in a CBT system has no effect on the learner's mastery of the material to be
The remainder of paper is structured as follows: Section Two provides a review of research findings related to
the usefulness of dynamic visuals to the learning process. In Section Three, the basic theories on multimedia
learning and human perception are used to postulate above hypotheses in regard to the usefulness and
effectiveness of dynamic visuals in support of the learning process. A CBT system called E-Tutor is used to test
these hypotheses. Section Four outlines the methodology adopted in this investigation. Section Five presents
the results. A discussion and an overview of the implication for practice and future research close the paper.
On the Usefulness of Dynamic Visuals on Learning
Effective presentation of instructional materials, whether it be in the form of text, picture, charts, graphs,
diagrams, animation, videos, etc. is an important facet of the learning process. As a result, multimedia
educational systems are rapidly growing in popularity because they enable us to provide a variety of presentation
modes in support of the learning process (Anderson et al., 2000; Heller et al., 2001; Pilkington and Grierson,
1996). In particular, interactive multimedia systems with dynamic visual presentations are becoming popular
because they can provide an enriched learning environment by facilitating the acquisition of pertinent materials.
Dynamic visuals that have been embedded in CBT for improved learning can be classified as movies and
simulations. We can distinguish movies and simulations in the following way: animations, videos, time-lapse
photography and other motion pictures that are the same each time they are viewed are classified as movies; and
graphics, tables, or other output generated by running a computer program under student control are classified as
simulations (Pane, 1994). In the latter, the visual results are subject to any changes that the student makes to the
simulation program, and thus may be different each time they are viewed.
A number of investigators have examined the effect of dynamic visuals on learning (Pane et al., 1996; Park and
Hopkins, 1993; Rieber, 1990; Velayo, 2000). Park and Hopkins (1993) identified five important instructional
roles of dynamic visuals:
1. As an attention guide, the dynamic visual can serve to guide and direct the subject's attention.
2. As an aid for illustration, dynamic visuals can be used as an effective aid to represent the structural and
functional relations among components in a domain of knowledge.
3. As a representation of domain knowledge, movement and action can be used to effectively represent certain
domain knowledge.
4. As a device model for forming a mental image, dynamic visuals can be used to represent system structures
and functions that are not directly observable (e.g., blood flowing through the heart).
5. As a visual analogy or reasoning anchor for understanding abstract and symbolic concepts or processes,
dynamic visuals can make abstract and symbolic concepts (e.g., velocity) become more concrete and
directly observable.
The question arises whether dynamic visual has any effect on students' learning. To answer this, Park and
Hopkins (1993) produced a research summary of 25 studies and concluded that:
The research findings do not consistently support the superior effect of dynamic visual displays.
The conflicting findings seem to be related to the different theoretical rationales and
methodological approaches used in various studies (Park and Hopkins, 1993, p. 427).
Traditionally, designers of instructions have had two categories of outcomes in mind: those directed toward
cognitive goals, and those related to the attitude of the learner. Student achievement related to cognitive goals
has been the paramount objective of most instructional activities. However, it may also be important to recognize
the need for establishing attitudinal goals and for planning activities that are designed to facilitate the learning
process. Thus, one of the major goals of instruction involving media should be the development of positive
attitudes (Venkatesh, 1999; Simonson, 1985; Simonson and Maushak, 1995). Although the strength of the
relationship between attitudes and achievement is unclear (Zimbardo and Leippe, 1991), it seems logical that
students are more likely to remember information, seek new ideas, and continue studying when they react
favorably to an instructional situation or when they like a certain content area. Nonetheless, despite its
importance in the learning process, we know little about assessment of student perception on the usefulness of
dynamic visual media and their effect on learning. This vital research question is addressed in this paper.
The Objective of this Investigation
The basic question raised is "How do dynamic visuals in the form of videos embedded in a CBT system enhance
learning?" As mentioned above, previous investigations have contradictory findings concerning improvement in
students' learning outcomes (i.e., improved exam scores) when dynamic visuals were added to the CBT systems.
Missing from these studies is some consideration of the internal psychological processes through which learning
occurs (Alavi and Leidner, 2001). The contention of Alavi and Leidner in regard to technology-mediated
learning (TML) is that:
To be useful, TML research questions need to be formulated in terms of the way in which
technology features can engage psychological processes of learning that will in turn result in the
desired learning outcomes (Alavi and Leidner, 2001, p. 4).
Dynamic visuals can be embedded in CBT systems in the form of multimedia learning instructions. Multimedia
learning occurs when students use information presented in two or more formats – such as text, figures, and
videos. Figure 1 summarizes a dual-coding theory of multimedia learning (Mayer and Sims, 1994). It offers a
three-process account of how visually and verbally presented material might be integrated in the learner’s
working memory. On the top left portion of the figure, a verbal explanation, such as an oral narration, is
presented to the learner. Within the working memory the learner constructs a mental representation of the
system described in the verbal explanation. The cognitive process of going from an external to an internal
representation of the verbal material is called building a verbal representational connection (or verbal encoding)
(Mayer and Sims, 1994). On the bottom left portion of the figure, a visual explanation is presented to the
learner, such as video. Within the working memory the learner constructs a mental representation of the visually
presented system. The cognitive process of going from an external to an internal representation of visual
information is called building a visual representational connection (or visual encoding) and is indicated by the
second arrow. The third arrow denotes the construction of referential connections between the two mental
representations, that is, the mapping of structural relations between the two representations of the system. Longterm memory enables the learner to interpret and assimilate the content of working memory to interact with the
decision environment. It is expected that the added value of multimedia enhance the decision performance of the
learner. For example, in understanding an explanation of how intranet and extranet work, the learner can build
referential connections between visual and verbal representation of essential parts, actions, relations and
principals in the communication system.
Figure 1: A dual coding model of multimedia learning (Mayer and Sims, (1994)
This theory of multimedia learning has been the basis of previous investigations on the effectiveness of dynamic
visuals in support of the learners’ improved learning performance. However, as indicated earlier, missing from
these investigations is internal psychological processes of the learner. Students must be motivated to use
strategies to regulate their cognition and effort (Pintrich, 1988 & 1989). Research findings suggest that students
with a motivational orientation involving goals of mastery learning, as well as a belief that the task is interesting
and important, will engage in more metacognitive activity, more cognitive strategy use and more effective effort
management (Ames and Archer, 1988; Meece et al., 1988; Nolen, 1988). There are two main classes of
motivation: extrinsic and intrinsic (Davis et al., 1992). Extrinsic motivation relates to the drive to perform a
behavior to achieve specific goals/rewards, while intrinsic motivation relates to perception of pleasure and
satisfaction from performing the behavior.
In this research, our measurement for CBT system's learning outcome is the learner’s performance and affective
reactions to the usefulness of videos. We use exam scores as a measure of the student’s extrinsic motivation to
achieve mastery learning of the pertinent learning materials. A student’s affective reaction to the usefulness of
videos is based on the Technology Acceptance Model (TAM) (Davis, 1989; Davis et al., 1989, 1992; Taylor and
Todd, 1995). In TAM, extrinsic motivation and the associated instrumentality are captured by the perceived
usefulness construct (Davis et al., 1992). In this research, affect of dynamic visuals on extrinsic motivation of
the learner and subsequent learning outcome is of interest. To this end, based on previous research findings, the
following two null-hypotheses are postulated:
HYPOTHESIS 1: Inclusion of video in a CBT system has no effect on the learner's perception regarding the
usefulness of the system.
HYPOTHESIS 2: Inclusion of video in a CBT system has no effect on the learner's mastery of the material to be
An effective assessment of these hypotheses requires us to take into account the effect of the learner’s intrinsic
motivation and self-regulated learning behavior, in form of covariates, which are important aspects of learning
and academic performance (Corno and Mandinach, 1983; Corno and Rohrkemper, 1985). The theoretical
framework adopted in this research for conceptualizing student intrinsic motivation is an adaptation of a general
expectancy-value model of motivation (Pintrich, 1988, 1989; Pintrich and DeGroot, 1990).
The model
proposes that there are three motivational components that can be linked to the components of self-regulated
learning: (a) an expectancy component, which includes the student’s belief in his or her ability to perform a task
(i.e., self-efficacy), (b) a value component, which includes the student’s goals and beliefs about the importance
and interest of the task (i.e., intrinsic value), and (c) an affective component, which includes the student’s
emotional reactions to the task (i.e., test anxiety).
Self-regulated learning adopted in this research as covariate to assess the stated hypotheses includes the student’s
metacognitive strategies for planning, monitoring, and modifying cognition (Corno, 1986; Weinstein and Mayer,
1986; Zimmerman, 1988). Different cognitive strategies have been found to foster active cognitive engagement
in learning and to result in higher levels of achievement (Weinstein and Mayer, 1986). Self-regulation, on the
other hand, refers to the student’s ability to manage and control the handling classroom academic tasks. For
example, capable students who persist at a difficult task or block out distractions (i.e., noisy classmates) maintain
their cognitive engagement in the task, and so are able to perform better (Corno, 1986; Corno and Rohrkemper,
A CBT system called E-Tutor, developed in support of a Management Information Systems (MIS) textbook was
adopted to test the stated hypotheses (see O’Brien, 2001). The structure of E-Tutor was similar to the model
proposed by Retalis and Papasalours (2005). E-Tutor enables each student to test his or her knowledge of the
lessons to be learned. Textbook materials are embedded in the E-tutor in the form of hypermedia that presents
text, pictures, figures and videos. We developed two versions of E-Tutor referred here as “without-video” and
“with-video”: both versions had the basic textbook materials (i.e., text, figures, tables and pictures). However,
one version had 80 videos in form of AVI files, called here E-Tutor (with video), and the other without videos,
called here E-Tutor (without video). The videos that were embedded in the E-Tutor (with video), provided
further definition or examples to support various parts of the text of the lessons being studied. These videos are
expected to add value to the instructor’s explanation of the materials. Thus, McGraw Hill has included them in
the instructor's CD ROM that accompanies their MIS textbooks. The question posed through our hypotheses is
whether such learning materials in the form of videos are useful to the learning process when embedded in a
CBT system.
Experimental Design
We used a 2x2 factorial design with one between-subjects factor and two repeated trials to assess the stated
hypotheses. The between-subject factor was the presence or absence of the AVI videos embedded in a CBT
system. Two dependent variables used were the usefulness of the video presentations in support of the learning
materials and students’ test score gained from two exams. Students’ intrinsic motivation to learn the subject
matter and self-regulated learning behavior, assessed through two structured questionnaires, were used as
Fifty-one students registered in a basic Management Information Systems course at the MBA level were asked to
participate in this investigation by completing the following steps for 5% extra credit towards their final grade.
Student participation was optional and they could drop out at will. Forty-eight students completed the
experiments. None of the subjects were disabled.
There were four exams, three weeks apart, for the students to take. Each exam, which consisted of 50 multiple
choice questions with medium-level of difficulty, was to cover 3-4 chapters of the textbook materials. The
results of the first two exams were used to assess random assignment of the students to two groups (See
Appendix A). Assessment of the stated hypotheses was based on exams 3 and 4.
A 44-item questionnaire, adopted from Pintrich and DeGroot (1990), to assess the learner’s “motivation” and
“self-regulated learning” (Appendix B) was administered when the class had covered half the textbook and taken
the first two exams. By covering about half of the textbook materials and completing two exams, students were
expected to have a better understanding of their interest in the subject matter.
Next, students were randomly placed into two groups (Table 1). Each student in one group received a CD that
contained E-Tutor (with video), and each subject in the other group received a CD that contained E-Tutor
(without video). Students participating in this investigation were required to use E-Tutor to prepare for the two
scheduled exams 3 and 4. This repeated trials made it possible to assess the effects of videos embedded in the ETutor within and between groups. A hands-on demonstration of E-Tutor use was done to make sure that all
students were familiar with the way the system functioned.
Table 1: Assignment of the subjects to the two test groups
Trial 1
Subject with access to E-Tutor (with Video)
Subjects with access to E-Tutor (without video)
Trial 2
Trial 1: Subjects were required to complete the pertinent chapters in E-Tutor in order to prepare for the
scheduled third exam. In addition, subjects who had access to E-Tutor (with video) were required to view all the
pertinent videos. E-Tutor kept track of each student’s progress through the lessons and saved it in a “working
file”. To make sure that all the subjects completed the assigned tasks, they were required to upload their
working files to a site. This upload included captured data about the date that the assigned chapters were
completed, as well as the video files—for students with E-Tutor (with video)—that were viewed. Two weeks
after receiving the E-Tutor CDs, students took a scheduled exam that consisted of 50 multiple-choice questions
covering the assigned chapters of the textbook (Appendix C). Prior to viewing the questions, they completed a
questionnaire to assess the usefulness of E-Tutor in support of related learning materials for the exam (Appendix
Trial 2: The procedure followed in this trial was similar to trial 1, except that subjects who used E-Tutor (with
video) in trial 1 exchanged it for E-Tutor (without video). Those with E-Tutor (without video) exchanged it with
E-Tutor (with video). Two weeks after exchanging the E-Tutor CDs, students took the fourth exam. Prior to
viewing the questions on this exam, as in trial 1, subjects completed a questionnaire designed to assess their
perceived usefulness of videos.
We used MANOVA to assess within and between subject difference in regard to the E-Tutor’s usefulness with
and without videos in support of learning and improved performance. Students’ “motivation” and “selfregulated learning” were included in the analysis as covariates. Levene’s test of equality of error variance of the
dependent variables showed insignificant inequality across the groups (for usefulness F=0.49 (p=0.69) and for
the tests 3 & 4 F=0.93 (p=0.43)). In addition, Box’s test of equality of covariance matrices showed insignificant
inequality across groups (F=0.78 (p=0.63)). Hypothesis 1 states, “Inclusion of video in a CBT system has no
effect on the learner's perception regarding the usefulness of the system”. The average score for the E-Tutor
usefulness (Table 2) in the Trial 1 was: with video = 30.24 and without video = 27.64, and for the Trial 2: with
video = 30.97 and without video = 28.39. The results of MANOVA test demonstrated that there was a
significant difference (F=11.43), p<0.01) about the perceived usefulness of videos embedded in E-Tutor among
the groups (Table 3). Subsequent pairwise comparison of the variables showed that the group with access to
videos perceived E-tutor to be more useful than did the other group in both trial 1 and trial 2 (Table 4). There
was no significant difference in regard to the usefulness of E-Tutor when both groups had access to E-Tutor with
the same capability (i.e., both had access to E-Tutor with video or without videos). Analyses also showed that
subjects’ motivation to learn had a significant interaction effect on their perception of the E-Tutor usefulness to
enable them learn the pertinent reading materials (Table 3). However, subjects’ self-regulated learning had no
interaction effect on their perceived usefulness of E-Tutor.
The above analysis provides sufficient evidence that the difference in the perceived usefulness of the E-Tutor is
due to the presence or absence of videos. Therefore, based on the results of this experiment, the stated
hypothesis cannot be supported.
Table 2: Descriptive Statistics of the Research Variables
Subjects in Group 1
Subjects in Group 2
Mean, (Variance)
Mean, (Variance)
Min. – Max.
Min. – Max.
Exam 1
Exam 2
Exam 3
Exam 4
Motivational beliefs
Self-Regulated Learning Strategies
Usefulness of the E-Tutor for Trial 1
Usefulness of the E-Tutor for Trial 2
79.13%, (7.30%)
68% - 88%
78.80%, (10.02)
61% - 92%
85.92%, (5.64%)
74% - 96%
87.60%, (5.32%)
72% - 94%
91.44, (21.62)
44 - 119
102.96, (18.42)
65 - 127
30.24, (2.42)
25 - 33
28.39, (1.92)
25 - 33
77.04%, (6.30%)
68% - 82%
77.80%, (10.99%)
60% - 94%
85.82%, (4.82%)
72% - 96%
85.92%, (6.96%)
70% - 96%
93.35, (16.14)
63 - 127
101.65, (20.96)
51 - 133
27.64, (2.12)
25 - 30
30.91, (2.65)
25 - 35
Table 3: MANOVA Tests of Between-Subjects with and without Video Presentation
Dependent Variable
Score for exams 3&4
Score for Exams 3&4
Self-regulated learning
Score for Exams 3&4
Mean Square
Our hypothesis 2 states; “Inclusion of videos in a CBT system has no effect on the learner's mastery of the
material to be learned.” The results of MANOVA test (Table 3) showed that there was no significant difference
between the scores on exams 3 and 4 with and without video (F = 0.56, p = 0.64). This indicates that although
availability of video was perceived to be useful in students’ learning processes, it had no effect on their mastery
of the material to be learned based on their exam scores. Thus, hypothesis 2 cannot be rejected.
Table 4: Pairwise Comparisons of Usefulness of E-Tutor with and without Video
Dependent Variable: Usefulness
Mean Difference
Group with video in trial 1 vs. group without video in trial 1
Group with video in trial 2 vs. group without video in trial 2 test
Group with video in trial 1 vs. group with video in trial 2
Group without video in trial 1 vs. group without video in trial 2
Sig. (p)
Inspired by the advent of new information and communication technologies, use of technology-mediated
learning environments is on the rise. The number of online courses offered through academic and professional
institutions is evidence of this. Although research related to technology-based learning has been with us for
more than 30 years, we are still at an early stage of knowing how to effectively apply technology in support of
learning. In this endeavor, textual material is supplemented by still images, animation, video sequences, and/or
sound sequences. The objective is to produce an enhanced learning environment in which students will be able
to learn and understand more effectively. This research shows that students find a learning environment with
video presentation more useful. Furthermore, it shows that students’ intrinsic motivation to learn the subject
matter had a significant positive effect on their satisfaction with the availability of video presentation. Based on
this, we can conclude that students with higher interest in the subject matter would have found the added
information gleaned from video presentation more satisfying. However, added functionality of video
presentation had no significant effect on performance in exams. That is, the addition of the video presentation to
the text and pictures did not produce significant gains in students’ ability to answer questions on it. Of course,
these findings are limited by the characteristics of our environment. E-Tutor was designed to support text-based
explicit knowledge to improve student mastery learning of basic MIS materials. However, the effect of video
presentation on learning might be different when used in support of tacit knowledge (e.g., analysis of complex
case studies) (Lim and Benbasat, 2000). Tacit knowledge relate to experiential learning, whereas explicit
knowledge deals with objective and technical knowledge. Thus, future research should assess the effectiveness
of video in support of learners in need of acquiring tacit knowledge.
Future research should find it interesting to assess the long-term effect of video on students learning and student
motivation to pursue the subject matter further. Learners’ cognitive style may also affect effectiveness of video
presentation (Montazemi and Wang, 1989). In this endeavor, we need to develop methodologies to distinguish
students as visual-spatial learners or auditory-sequential learners: Visual-spatial learner is believed to think in
pictures and have a different brain organization than auditory-sequential learner (Silverman, 2002). It is possible
that visual-spatial learns benefit more from video presentation. We should also search for different means of
using dynamic visuals in support of improved learning performance. Learning performance, as depicted in
Figure 1, is contingent on internal (long-term memory) and external (e.g., video) mental representation of the
learner. Dynamic visual based on animation is a possible medium to satisfy the individual learning requirements
of the learner. We can exploit the rapidly evolving graphics capabilities of information systems to provide
learners with engaging three-dimensional worlds (Badler et al., 2002). For example, to learn procedural tasks, a
three-dimensional learning environment could enable students to perform the task directly in the environment.
Hence, rather than memorizing an abstract procedure, students learn by interacting with rich three-dimensional
models representing the subject matter. Lifelike pedagogical agents can play a key role in a central problem
posed by task-oriented three-dimensional learning environment; detecting and correcting misconception. While
the pertinent technologies are new, they are beginning to mature, and we have already begun to see the
emergence of techniques for real-time behavior sequencing in response to rapidly changing learning contexts
(Badler et al., 2002; Funge, 2000; Light and Maybury, 2002).
Information technology can play a significant role in support of learning if the embedded functionality of the
system is perceived to be useful by the learner. We believe, as others do (Alavi and Leidner, 2001; Carroll,
1997; Ohlsson, 1991; Tong and Angelides, 2000), that system “hacking” is not an effective means of improving
the state of the art in technology-mediated learning (TML). Instead, well-grounded learning theories need to be
empirically tested in TML environments. Only then will system designers be provided with the much-needed
theoretical foundation for implementing more perfect TML systems.
The Author would like to thank the three anonymous reviewers for their comments on the earlier version of this
manuscript. This research has been supported in part by grants from Indiana University Ameritech Fellow of
Information Technology Program, and Natural Sciences and Engineering Research Council of Canada.
Alavi, M. & Leidner, D. E., (2001). Research Commentary: Technology-Mediated Learning-A Call for Greater
Depth and Breath of Research. Information Systems Research, 12 (1), 1-10.
Ames, C. & Archer, J., (1988). Achievement Goals in the Classroom: Student Learning Strategies and
Motivation Processes. Journal of Educational Psychology, 80 (4), 260-267.
Anderson, A. H., Smallwood, L., MacDonald, R., Mullin, J., Annemarie, F. & O’Malley, C., (2000). Video Data
and Video Links in a Mediated Communication: What Do Users Value? International Journal of HumanComputer Studies. 52 (1), 165-187.
Badler, N. I., Erignac, C. A. & Liu, Y., (2002. Virtual Humans for Validating Maintenance Procedures.
Communications of the ACM. 45 (7), 57-63.
Benbunam-Fich, A, (2002). Improving Education and Training with IT,” Communications of the ACM. 45 (5),
Carroll, J. M., (1997). Human-Computer Interaction: Psychology as a Science of Design. International Journal
of Human-Computer Studies. 46 (4), 501-522.
Corno, L., (1986). The Metacognitive Control Components of Self-Regulated Learning. Contemporary
Educational Psychology, 11 (4), 333-46.
Corno, L. & Mandinach, E., (1983). The Role of Cognitive Engagement in Classroom Learning and Motivation.
Educational Psychologist, 18 (4), 88-100.
Corno, L. & Rohrkemper, M., (1985). The Intrinsic Motivation to Learn in Classrooms,” In Ames, C. & Ames,
R. (Eds.), Research in Motion, Vol. 2. The Classroom Milieu, Academic Press, New York, NY, 53-90.
Davis, F. D., (1989). Perceived Usefulness, Perceived Ease of Use and User Acceptance of Information
Technology. MIS Quarterly, 13 (3), 319-39.
Davis, F. D., Bagozzi, R. P. & Warshaw, P. R., (1992). Extrinsic and Intrinsic Motivation to Use Computers in
the workplace. Journal of Applied Social Psychology, 22 (4), 1111-32.
Davis, F. D., Bagozzi, R. P. & Warshaw, P. R., (1989). User Acceptance of Computer Technology: A
Comparison of Two Theoretical Models. Management Science, 35 (8), 982-1002.
Funge, J., (2000). Cognitive Modeling for Games and Animation. The Communications of the ACM. 43 (7), 4148.
Hannafin, M. J. & Rieber, L. P., (1989). Psychological foundations of instructional design for emerging
computer-based instructional technologies. Part II. Educational Technology Research and Development, 7, 10214.
Heller, R. C., Martin, C. D., Haneef, N. & Gievska-Krliu, S., (2001). Using a Theoretical Multimedia Taxonomy
Framework. ACM Journal of Educational Resources in Computing, 1 (1), 1-22.
Kerlinger, F. N., (1973). Foundations of Behavioral Research. Holt, Rinehart, and Winston Inc., Chicago, IL.
Leidner, D. E. & Jarvenpaa, S. L., (1995). The Use of Information Technology to Enhance Management School
Education: A Theoretical View. MIS Quarterly, 19 (3), 265-91.
Light, M. & Maybury, M. T., (2002). Personalized Multimedia Information Access. Communications of the
ACM, 45 (5), 54-59.
Lim, K. H. & Benbasat, I, (2000). The Effect of Multimedia on Perceived Equivocality and Perceived
Usefulness of Information Systems. MIS Quarterly, 24 (3), 449-471.
Mayer, R. E. & Sims, V. K., (1994). For Whom Is a Picture Worth a Thousand Word? Extensions of a DualCoding Theory of Multimedia Learning. Journal of Educational Psychology, 86 (3), 389-401.
Meece, J., Blumenfeld, P. & Hoyle, R., (1988). Students’ Goal Orientations and Cognitive Engagement in
Classroom Activities,” Journal of Educational Psychology, 80 (4), 514-23.
Montazemi, A. R. & Wang, F., (1995-a). CBT in Support of Mastery Learning. Journal of Educational Computing
Research, 13 (2), 185-205.
Montazemi, A.R. & Wang, F., (1995-b). On the effectiveness of Neural Network for Adaptive External Pacing.
Journal of Artificial Intelligence in Education, 6 (4) 379-404.
Montazemi, A. R. & Wang, S., (1989). The Effects of Modes of Presentation on Decision Making: A Review and
Meta-Analysis. Journal of Management Information Systems, 5 (3), 111-127.
Nolen, S. (1988). Reasons for Studying: Motivational Orientations and Study Strategies,” Cognition and
Instruction, 5 (4), 269-287.
O’Brien, J. A., (2001). Management Information Systems; Managing Information Technology in the E-Business
Enterprise. New York, NY, McGraw-Hill.
Ohlsson, S., (1991). System Hacking meets learning Theory: Reflection on the Goals and Standards of Research
in Artificial Intelligence and Education. Journal of Artificial Intelligence in Education, 2 (3), 5-18.
Pane, J. F., (1994). Assessment of the ACSE Science Learning Environment and the Impact of Movies and
Simulations,” Carnegie Mellon University, CMU-CS-94-162.
Pane, J. F., Corbett, T. A., & John, B. E., (1996). Assessing Dynamics in Computer-Based Instruction. In
Proceedings of the Conference on Human Factors and Computing Systems, April 13-18, Vancouver, Canada,
Park, O. & Hopkins, R., (1993). Instructional Conditions for Using Dynamic Visual Displays: A Review.
Instructional Science, 21 (6), 427-499.
Pintrich, P. R., (1988). A Process-Oriented View of Student Motivation and Cognition. In Stark, J. S. &Mets, L.
(Eds.), Improving Teaching and Learning Through Research. New Directions for Institutional Research, San
Francisco, CA, Jossey-Bass, 55-70.
Pintrich, P. R., (1989). The Dynamic Interplay of Student Motivation and Cognition in the College Classroom.
In Ames, C. & Maehr, M. (Eds.), Advances in Motivation and Achievement: Vol. 6. Motivation Enhancing
Environments, Greenwich, CT, JAI Press, 117-160,.
Pintrich, P. R. & DeGroot, V. E., (1990). Motivational and Self-Regulated Learning Components of Classroom
Academic Performance. Journal of Educational Psychology, 82 (1), 33-40.
Pilkington, R. & Grierson, A., (1996). Generating explanation in a simulation-Based Learning Environment.
International Journal of Human-Computer studies, 45 (5), 527-551.
Rieber, L. P., (1990). Animation in Computer-Based Instruction. Educational Technology Research and
Development, 38 (1), 77-86.
Retalis, S. & Papasalours, A., (2005). Designing and Generating Educational Adaptive Hypermedia
Applications, Educational Technology and Society, 8 (1), 69-79.
Silverman, L. K., (2002). Upside-down Brilliance: The Visual-Spatial Learner, DeLeon Publishing, Denver, CO.
Simonson, M., (1985). Persuasive Films: A Study of Techniques Used to Change Attitudes. Journal of Teaching
and Learning Technologies. 1 (2), 39-48.
Simonson, M. & Maushak, N., (1995). Situated Learning, Instructional Technology, and Attitude Change. In
McLellan, H. (Ed)., Perspectives on Situated Learning, Cliffs, NJ, Englewood, 46-79.
Stone, R., (2001). Virtual Reality for Interactive Training: An Industrial Practitioner’s View Point. International
Journal of Human-Computer Studies. 55 (4), 699-711.
Taylor, S. & Todd, P., (1995). Understanding Information Technology Usage: A Test of Competing Models.
Information Systems Research. 6 (2), 144-176.
Tong, A. K. Y. & Angelides, M. C. (2000) An Empirical Model for Tutoring Strategy Selection in Multimedia
Tutoring Systems. Decision Support Systems. 29 (1), 31-45.
Velayo, R. S., (2000). How do Presentation Modality and Strategy Use Influence Memory for Paired Concepts.
Journal of Instructional Psychology. 27 (2), 126-133.
Venkatesh, V., (1999). Creation of Favorable User Perceptions: Exploring the Role of Intrinsic Motivation. MIS
Quarterly, 23 (2), 239-260.
Weinstein, C. E. & Mayer, R. E., (1986). The Teaching of Learning Strategies. In Wittrock, M. (Ed.), Handbook
of Research on Teaching, Macmillan, New York, NY, 315-327.
Zimbardo, P. & Leippe, M., (1991). The Psychology of Attitude Change and Social Influence. Temple
University Press, Philadelphia, PA.
Zimmerman, B. & Pons, M., (1988). Construct Validation of a Strategy Model of Student Self-Regulated
Learning. Journal of Educational Psychology, 80 (3), 284-290.
Appendix A
Questionnaire Validity and Reliability
One measure of construct validity is the extent to which each item of a questionnaire correlates with the total
score (Kerlinger, 1973). The correlation between the score for each item (question) and the total usefulness
score was 0.65 to 0.84 (all significant at p<0.001) (Table 1A). Correlation between the overall score for
motivation with 22 pertinent items showed that items 1 and 22 were not significantly correlated (alpha >0.05).
We removed these two items from the subsequent analysis. Correlation between overall score for self-regulated
learning and the 22 pertinent items (i.e., items 23–44 as depicted in Appendix B) showed that three items 26, 27
and 37 were insignificant. Thus, these three items were removed from the subsequent analysis.
Usefulness for Trial 1
Usefulness for Trial 2
Self-regulated learning
Table 1A: Validity and Reliability of the Questionnaires
Correlation Coefficient
(all significant at alpha < 0.01)
0.65 – 0.84
0.68 – 0.90
0.51 - .88
0.50 – 0.77
Factor Analysis
(Item communality range)
0.84 – 0.88
0.86 – 0.95
0.65 – 0.89
0.59 – 0.76
Cronbach alpha
The data were further analyzed using factor analysis (principal component analysis using Varimax rotation
method with Kaiser normalization) to establish convergent and discriminant validity (Table 1A). The factor
analysis for usefulness questionnaire scores in exam 3 and exam 4 showed one orthogonal factor with
eigenvalues above 1.0 accounting for 97.18% and 99.18% respectively, with item communality ranging between
0.84 and 0.95 (Table 2A). The factor analysis for “motivation and self-regulated learning” showed two
orthogonal factors with eigenvalues above 1.0, together accounting for 76.4% of the variation, with item
communality ranging between 0.59 and 0.89.
The Cronbach alpha reliability score for the three questionnaires was 0.83 to 0.91 (Table 1A). These values are
well within the thresholds suggested by Nunnally (1978). Thus, the questionnaires can be used as acceptable
instruments for subsequent analysis.
Q1-Trial 1
Q1-Trial 2
Table 2A: Factor Analysis for Perceived Usefulness
Factor loading
Item Communality
Test for Random Assignment
To test for random assignment of subjects to the two groups, the following analysis was performed. The
ANOVA test was used to assess subject performance for the first and second multiple-choice exams (Table 4) to
find out if there were any differences between the two groups regarding their domain knowledge. The result of
the ANOVA test demonstrated no significant difference between the two groups: F = 1.104 (p = 0.299) for the
first exam and F = 0.648 (p = 0.425) for the second exam. We also used ANOVA to test possible differences
between the two groups in regard to the subjects’ “motivational beliefs” and “self-regulated learning strategies”.
The results of ANOVA test showed no significant difference between the two groups: F = 0.079 (p = 0.971) for
the motivational beliefs and F = 0.035 (p = 0.991) for the self regulated learning strategies.
Nunnally, J. C. (1978). Psychometric Theory, New York, NY: McGraw-Hill.
Appendix B
The following 44 items represent the Motivated Strategies for Learning Questionnaire (MSLQ) used in this
study to measure students’ motivational beliefs and self-regulated learning. The number next to the items
reflects the item’s actual position on the questionnaire.
All the items were assessed on the following 7-point Likert-type scale:
not at all true of me
Very true of me
Motivational Beliefs
Compared with other students in this class I expect to do well.
I’m certain I can understand the ideas taught in this course.
I expect to do very well in this class.
Compared with other students in this class, I think I am good student.
I am sure I can do an excellent job on the problems and tasks assigned for this class.
I think I will receive a good grade in this class.
My study skills are excellent compared with others in this class.
Compared with other students in this class I think I know a great deal about the subject.
I know that I will be able to learn the material for this class.
I prefer class work that is challenging so I can learn new things.
It is important for me to learn what is being taught in this class.
I like what I am learning in this class.
I think I will be able to use what I learn in this class in other classes.
I often choose paper topics I will learn something from even if they require more work.
Even when I do poorly on a test I try to learn from my mistakes.
I think that what I am learning in this class is useful for me to know.
I think that what we are learning in this class is interesting.
Understanding this subject is important to me.
I am so nervous during a test that I cannot remember facts I have learned.
I have an uneasy, upset feeling when I take a test.
I worry a great deal about test.
When I take a test I think about how poorly I am doing.
Self-Regulated Learning
When I study for a test, I try to put together the information from class and from the book.
When I do homework, I try to remember what the teacher said in class so I can answer the
questions correctly.
It is hard for me to decide what the main ideas are in what I read.
When I study I put important ideas into my own words.
I always try to understand what the teacher is saying even if it does not make sense.
When I study for a test I try to remember as many facts as I can.
When studying, I copy my notes over to help me remember material.
When I study for a test I practice saying the important facts over and over to myself.
I use what I have learned from old homework assignments and the textbook to do new assignments.
When I am studying a topic, I try to make everything fit together.
When I read material for this class, I say the words over and over to myself to help me
I outline the chapters in my book to help me study.
When reading I try to connect the things I am reading about with what I already know.
I ask myself questions to make sure I know the material I have been studying.
When work is hard I either give up or study only the easy parts.
I work on practice exercise and answer end of chapter questions even when I do not have to.
Even when study materials are dull and uninteresting, I keep working until I finish.
Before I begin studying I think about the things I will need to do to learn.
I often find that I have been reading for class but do not know what it is all about.
I find that when the teacher is talking I think of other things and do not really listen to what is
being said.
When I’m reading I stop once in a while and go over what I have read.
I work hard to get a good grade even when I do not like a class.
Appendix C
A Sample of Exam Questions used in Trials 1 & 2
End user resistance can be minimized through:
Proper end user education and training.
Improved communications with IS professionals.
End user involvement in the development and implementation of new systems.
All of the above.
Management involvement in information systems can take the form of:
Top management participation in an executive information technology committee.
Management participation in steering committees for systems development projects.
End user management participation in the systems development process and the management of
information systems.
All of the above.
Your text defines change management as managing the process of implementing major changes in information
technology, business processes, organizational structures, and job assignments to reduce the risks and costs of
change, and optimize its benefits. A number of factors (dimensions) of change management are illustrated and
discussed. Of these factors, which one is the major focus of organizational change management?
Knowledge management.
Your text states that the E-business planning process ahs three major components - strategy development,
resource management, and technology architecture. Strategy Development involves:
Developing E-business and E-commerce strategies that support a company's E-business vision that
focuses on customer and business value.
Developing strategic plans for managing or outsourcing a company's IT resources.
Developing strategic IT choices that reflect information technology architecture designed to support a
company's E-business and E-commerce initiatives.
None of the above applies to defining strategy development.
Prototyping involves:
The execution of the standard systems development cycle using CASE tools.
A rapid generation of systems by information systems professionals without the need for end user input.
The use of a fail-safe development process designed to ensure that an information system meets all user
requirements without revision.
An iterative and interactive development process with extensive end user involvement.
Organizations are increasingly using encryption of data and firewall computers as methods to protect computer
network resources. Firewalls are described as:
The transmission of data through telecommunications lines in "scrambled" form.
A "gatekeeper" system that protects a company's intranets and other computer networks from intrusion
by providing a filter and safe transfer point for access to and from the Internet and other networks.
The act of ensuring the accuracy, integrity, and safety of all E-business processes and resources.
None of the above apply.
A DBMS query language is designed to:
Support information systems professionals in the development of complex application software.
Support end users that wish to interrogate the database using English-like or natural language
Provide efficient processing of the database in the batch mode.
Specify the content, relationships, and structure of a database.
In the data planning and modeling process described in your text:
Database development starts with a bottom-up data planning process.
Subject area databases are consolidated to form the enterprise model.
User views are generated by identifying the key data elements in the subject area databases, which are
needed to perform specific business activities.
The enterprise model is mapped directly into the physical database design.
The logical structure of information in a database is contained in the
Data dictionary.
Data manipulation subsystem.
Data administration subsystem.
Data definition subsystem.
A network that contains one or more host computers that provide some type of service to the other computers in
a network is a:
Peer-to-peer network
Client/server network
Local area network.
Wide area network.
Appendix D
Perceived Usefulness Questionnaire
All the items were assessed on the following 7-point Likert-type scale:
|________|________|________|________|________|________|________| Unlikely
slightly neither slightly
Using E-Tutor enables me to learn the MIS course materials more quickly
Using E-Tutor improve my performance in the pertinent tests.
Using E-Tutor enhance my learning effectiveness
Using E-Tutor make it easier to learn the subject matter
I find E-Tutor useful in learning subject matter of MIS course
Chang, K.-E., Sung, Y.-T., & Hou, H.-T. (2006). Web-based Tools for Designing and Developing Teaching Materials for
Integration of Information Technology into Instruction. Educational Technology & Society, 9 (4), 139-149.
Web-based Tools for Designing and Developing Teaching Materials for
Integration of Information Technology into Instruction
Kuo-En Chang
Department of Information and Computer Education, Center of Research for Educational Evaluation and
Development, National Taiwan Normal University, Taipei, Taiwan
Tel: +886223622841
Fax: +886223512772
[email protected]
Yao-Ting Sung
Department of Educational Psychology and Counseling, Center of Research for Educational Evaluation and
Development, National Taiwan Normal University, Taipei, Taiwan
Tel: +886223952445
Fax: +886223413865
[email protected]
Huei-Tse Hou
Department of Information and Computer Education, National Taiwan Normal University, Taipei, Taiwan
Tel: +886223622841
Fax: +886223512772
[email protected]
Educational software for teachers is an important, yet usually ignored, link for integrating information
technology into classroom instruction. This study builds a web-based teaching material design and
development system. The process in the system is divided into four stages, analysis, design, development,
and practice. Eight junior high school history teachers participated in the evaluation of the system. Through
experts’ reviews and content analyses of their instructional materials and interviews, we found that
instructional materials produced using the system appear to be more coherent and systematic, provide
deeper and broader information for learning, apply more adequate teaching strategies, and lessen the design
and development load on teachers.
Integrated technology into instruction, web-based teaching material design and development, Instructional
The impact of information technology (includes web technology, software or hardware tools) on the overall
teaching environment is becoming more pronounced. The ability to incorporate information technology into
instruction has been deemed to be one of the most important professional competences of teachers (Fisher, 1997;
Scheffler & Logan, 1999). Indeed, when used properly, information technology not only allows diverse ways for
the presentation of materials (Chang, Sung, & Chiou, 2002; Roberts & Hsu, 2000), but also offers easy access to
the wealth of resources available on the Internet and is useful for developing learner-centered strategies and
activities (Chang, Chen, & Sung, 2001; Chang, Sung, & Lee, 2003; Linn, 2000).
Compared to the pace of advancements in information technology, however, the rate of its adoption in the
classroom has been slow. Studies reveal that teachers are often especially interested in such technology, but
because of a lack of confidence, are intimidated by it (OTA, 1995; Willis, Thompson & Sadera, 1999).
Researchers have identified several internal and external barriers to the adoption of technology, such as teachers’
attitudes and willingness, their workload, the availability and accessibility of hardware and software, staff
development, and institutional and technical support (Ertmer, 1999; Rogers, 2000).
All these factors are interrelated. Lawson and Comber (1999), for instance, have argued that one of the key
factors preventing teachers from applying information technology to teaching is their lack of computer literacy,
which would require time to learn in addition to the time required for course preparation. However, Topper
(1998) has observed that teacher training programs designed to facilitate technology literacy development often
fail to achieve the desired results when teachers are over burdened with too much additional work. Cuban,
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
Kirkpatrick, and Peck (2001) have also found that, although the increasing availability of computer equipment in
schools offers easy access to computer resources, many teachers fail to alter existing patterns of teaching.
The lack of software designed specifically for instruction might be one of the major reasons for the continued
lack of willingness to use information technology. The strong emphasis on the learner’s perspective in recent
years has led to many software products for teaching which focus on students’ learning needs instead of
teachers’ instructional design facilitation. Hence, teachers themselves have benefited relatively little from such
developments (Leu et al., 1998; Marx, Blumenfeld, Krajcik, & Soloway, 1998). The software for “integrating
technology into instruction” is therefore needed by teachers to leverage information technology with minimal
effort. Moreover, through such software, the burden of course preparation would be lightened, and thus teachers
would be better motivated to apply information technology in their teaching. As long as they have the will, they
are more likely to seek ways to integrate technology with teaching and try out new materials and methods
(Norum, Grabinger, & Duffield, 1999). If the ultimate goal of integrating instruction with technology is to make
the technology an integral part of how the classroom functions (International Society for Technology in
Education, 2000), then the design and application of software for instructional purposes is an essential step.
Most Web tools currently available for instructional design provide only assistance with website construction for
either asynchronous learning or distance learning. For instance, WebCT (http://www.webct.com) and eEducation (http://www.jonesknowledge.com) both provide tools for teachers to construct an online classroom to
place course content pages and to utilize other functions such as course schedule, discussion zone, online quiz,
and bulletin board. However, these web-based tools fail to address the difficulty of finding needed information
through Web searches, which limits the amount of supplementary material teachers can offer (Small, Sutton,
Miwa, Urfels, & Eisenberg, 1998). Moreover, the emphasis of these web-based tools on distance learning is
different from the functions emphasized in technology-incorporated teaching, such as the support of teaching
strategies for classroom teaching.
In short, a web-based teaching materials design and development system tailored to the needs of teachers who
have a lower level of computer literacy is helpful for teachers wishing to incorporate technology into teaching. It
is the goal of this study to develop a web-based environment that allows teachers to implement their instructional
procedures, for example by developing teaching plans, preparing course materials, and managing teaching
activities in an efficient and effective way. This system will focus more on each step of web-based teaching
material design (and will highlight the steps of Analysis, Design, Develop and Practice), which offer a series of
technological auxiliary tools for teaching. We will also look from the aspect of information-searching, providing
URL management modules, offering keyword indexing, and record browsing history. In order to evaluate the
system, eight junior high school history teachers were invited to participate in the evaluation. This paper uses
experts’ reviews and content analyses of teachers’ instructional materials and interviews, and concludes that
instructional materials produced by the system appear to be more coherent and systematic, provide deeper and
broader information for learning, apply more adequate teaching strategies, and lessen the design and
development load on teachers.
System outline
The system is divided into four stages: analysis, design, development, and practice. Table 1 shows the functions
for each stage. The portal page URL of the system is http://elearning.ice.ntnu.edu.tw. Below is a detailed
description of each.
Analysis stage
In this stage teachers analyze both learners’ characteristics and teaching content. Two functions are used for
analyzing the progress and characteristics of learners. The first function is students’ basic information analyzer
and manager, which records students’ basic information, including their past learning experience, family
background, aptitudes, and what students feedback to teacher throughout the learning process. The second
function is students’ score manager. Students’ score data accumulate over time with tests and exams as learning
progresses. The score manager helps teachers better analyze students’ learning status.
1. Analysis
2. Design
3. Development
4. Practice
Table 1. List of the functions for each stage
Learners’ characteristics analysis
Students’ basic information analyzer and manager
Students’ score manager
Teaching material analysis
Multimedia resource bank
Web browser
URL collector
URL browser recorder
URL resource bank
Developing course outline
Weekly course scheduler
Syllabus builder
Developing unit and lesson plans
Unit plan builder
Lesson plan builder
Preparing teaching materials and Web
Course page editor and manager
pages such as online handouts,
Personal webpage editor
assignments, learning worksheets,
Learning worksheet editor
Slide editor
demos, references, supplementary
information, test questions, and slides for Test editor & manager
classroom use.
Carrying out class teaching activities
Daily course bulletin
Online evaluation & communication
Online synchronous communicator
Online asynchronous communicator
Online assignment evaluator
With regard to teaching material analysis, teachers may utilize built-in databanks or conduct Web searches to
collect information for the analysis and design of teaching materials. According to Keller (1983), multimedia
presentations of materials usually increase cognition and arouse interest in learners. Therefore, the system
provides a multimedia resource bank for easy retrieval by teachers. This databank contains many multimedia
resources for teachers. Teachers can type a keywords related to the intended theme and relevant multimedia
resources will then pop up for selection. Teachers may integrate these with the teaching materials they are
It is also noted that because of the complexity of Internet hyperlink connections, teachers often find themselves
losing focus during Web searches (Park, 1991). In the other word, the hyperlinks around the Internet provide
many different forms of various information. It is difficult for instructors and learners to search or even to use
the resources in teaching or learning and it costs the learners too much time to filter the unrelated information.
Furthermore, instructors and learners may perhaps get lost in many hyperlinks paths. To solve this problem, the
system provides a notepad for teachers to record important information from the Web at any time during their
online search. Small, Sutton, Miwa, Urfels, & Eisenberg (1998) pointed out that the chances of a teacher locating
the right resources or adequate multimedia materials with just a few Web searches are slim. Hence, in addition to
a web browser, the system provides teachers with a URL collector, a URL browser recorder, and a URL
resource bank to assist teachers with online resource searching and gathering.
The URL collector allows teachers to add new websites that they frequently use, and provides keyword search
for URLs already stored in the system. The URL browser recorder helps record the names and URLs of websites
teachers have visited using the system. The URL resource bank stores all relevant and assorted URLs. Teachers
can easily access some other websites via Web links provided by the URLs in the databank and gather
information for teaching materials design. The databank stores two types of URLs: (1) literature of educational
or psychological theories; (2) course related information, such as lesson plans and examples of teaching activity
Design stage
During this stage, teachers produce their teaching outline and plan based on the analyses in the previous stage.
Teachers often draw on past experience and personal beliefs about teaching as they teach (Mannaz, 1999).
Personal styles and beliefs are also reflected in their choice of educational software (Niederhauser & Stoddart,
2001). The system thus aims to assist teachers to produce outlines and plans in their own style. The model also
provides a teaching plan and drafting tools to facilitate collation of lesson plans and self-reflection upon past
teaching performances.
There are four tools used in the design stage. Weekly course scheduler provides an academic calendar and
monthly planner to facilitate time management. Syllabus builder assists teachers in developing, editing, and
managing syllabuses, which is a course plan for the whole semester. Unit plan builder assists teachers in
developing, editing, and managing unit plans, each of which is a teaching plan for a specific unit. Lesson plan
builder assists teachers in developing, editing, and managing daily lesson plans, and includes a memo feature for
particular lessons or classes.
Development stage
During this stage teachers prepare supplementary materials according to the draft teaching plan prepared in the
design stage and the actual teaching needs, and then place the completed materials onto the website for students’
review before and after class. Most of the available tools for web-based teaching material development and
course management lack the needed support for instructional webpage production and fail to provide assistance
with the preparation of supplementary teaching materials such as learning worksheets and slides. The key task in
this stage is therefore to produce all the materials (learning worksheets, slides, etc.) necessary for instructional
webpage production and teaching activity implementation in a web-based support environment. To familiarize
teachers with the new operating environment, the system incorporates the standard Microsoft Office editing tools
(MS Office 2000), which the majority of teachers should be familiar with. The tools in this stage are as follows:
1. Course page editor and manager: This tool provides teachers with an environment for instructional webpage
editing and combines real-time on-line multimedia functions to facilitate the production of instructional
2. Personal webpage editor: This tool assists teachers in developing their own personal teaching pages through
which students can access the course pages designed by the teacher.
3. Learning worksheet editor: This tool helps teachers develop learning worksheets which are becoming more
popular with teachers.
4. Slide editor: This tool incorporates functions from Microsoft PowerPoint and assists teachers with slide
5. Test editor and manager: This tool incorporates functions from Microsoft Word and assists teachers with
test question editing. It also helps produce a test template according to the type and number of questions
teachers need in order to reduce the extra work with test paper formatting and layout.
Practice stage
During this stage teachers utilize the Internet to conduct various classroom and after hour teaching activities and
employ online tools for distance discussion and evaluation. In addition to synchronous and asynchronous
network communication tools, the system also helps teachers evaluate students’ papers online. The following
four functions are provided by this stage.
1. Daily course bulletin: This tool provides an online bulletin for announcements and course daily messages to
2. Online synchronous communicator: This function provides a discussion forum similar to online chat rooms
to allow teachers to provide counseling or respond to course related inquiries.
3. Online asynchronous communicator: This function provides an asynchronous discussion zone similar to a
message board to allow teachers and students to discuss course related problems.
4. Online assignment evaluator: This function allows teachers to evaluate and score students’ papers and
assignments on-line.
Eight history teachers from three junior high schools in Taipei participated in this study. The participants have
one to nine years of teaching experience with an average of six years. They had a basic understanding of
computer operations such as word processing and surfing the Internet, but had no experience in the use of webbased applications for the design or development of teaching materials.
Research design
The eight participating teachers were randomly divided into an experimental group (referred to as Teachers A, B,
C, and D) and a control group (Teachers E, F, G, and H). The subjects were requested to develop their teaching
materials for an assigned teaching unit. The experimental group was given the system, and the control group an
Internet browser and word processor.
Tools and materials
Course Content Material: The participants of this study were given the unit “Domestic Politics of the Ming
Dynasty” from the “Glories of the Late Imperial Era” section of the junior high school history book. The
participants were assigned the text of the unit along with the relevant parts in the teacher’s handbook as a
reference for designing the teaching materials and developing four hour teaching periods.
Scale for evaluating instructional materials: In order to analyze the differences between the materials developed
by the experimental and control groups, we developed a five-point Likert-type scale (1=strongly disagree to
5=strongly agree). Zero points were given when a desired item was missing from the teacher’s materials. The
scale comprises three categories of eight items (Table 2), including the quality of supplementary information
gathered in the analysis stage, the teaching plan developed in the design stage, and the teaching materials
produced in the development stage.
Questionnaire for interview: To obtain the opinions of the participating teachers in the experimental group, the
researcher developed a set of 21 structured questions for the interview. The questions are divided into several
categories such as background, opinions about the system interface, functions, guiding mechanism, system
usefulness, and the personal process of instructional design.
The procedures of the experimental group were implemented in three stages: (1) Giving instructions and
background interview (60 min.). During this stage, a teacher’s general background is probed during the
interview, including his or her past experience with the use of computers in teaching. The proposed system is
also explained to the teacher. (2) Planning, data searching, and preparing worksheet. This stage starts with the
actual analysis and design done by the participants (80 min.). Based on the textbook and the teacher’s handbook,
as well as information gathered from the Web, teachers then proceeded to produce the “teaching plan,” the
“lesson plan,” the “learning worksheet,” and the “detailed activity plan” with the help of the system tools. The
researcher also conducted interviews with the participants regarding this particular stage of the experiment (30
min.). (3) Developing teaching materials as Web pages. This stage starts with the participants executing the
development stage (80 min.). The participants decide on the content for their Web pages, which they may
develop with the system tools. After developing teaching materials, the researcher interviewed the participants in
this stage for 30 minutes.
For the control group, due to a lack of Web page production experience, it was difficult for the participants to
implement Web pages with other Web page tools like FrontPage. So, they were asked to complete the analysis
and design stages. The data in the development stage was not included in the comparison. This part of the
experiment was thus divided into the following two stages: (1) Giving instructions and background interview (60
min.). The content of the interviews was the same as for the experimental group. The functions of Microsoft
Word and Internet Explorer were also explained to the teachers. (2) Planning, data search, and preparation for
worksheet. During this stage teachers conducted their own analysis and design of the teaching unit (80 min.).
This includes the course content analysis, the use of computers to prepare and save supplementary materials
(using Word or Notepad), formulating a teaching plan, and finally, developing learning worksheets. After the
teachers completed their work, the researcher then conducted interviews on this part of the experiment (30 min.).
The finished instructional materials were evaluated by two junior high history teachers with 22 and 27 years of
experience. The materials used were the teaching plans, supplementary materials, and work sheets. The
evaluation results for eight teachers are shown in Table 3. The Spearman’s rank correlation coefficient indicates
a strong agreement between the two experts’ ratings on the quality of the materials, ρ =1.00 (p<.01). The average
of the two experts’ scores was then used as the basis for analysis. The results of the one-tailed Mann-Whitney U
test (Siegel & Castellan, 1988) on the experts’ mean scores for the eight measures of all materials are
summarized in Table 2.
Table 2. Average expert ratings for the two groups
Teacher in experimental group
Teacher in control group
MW-U test p
Supplementary material adequacy
Supplementary material depth
Supplementary material width
Teaching plan/strategy adequacy
Learning worksheet (handout) usability
Overall plan & material coordination
Overall material coherence
Overall material creativity
Total score
Expert 1
Expert 2
Table 3. Expert ratings of the eight instructional materials
Adequacy, width and depth of supplementary materials
With supplementary materials the results show that the experimental group received higher ratings than the
control group in measures of material adequacy, depth, and breadth, all reaching significance level (U=0,
p=.014). We conclude that given the same amount of time and teaching content, the supplementary material
gathered from the Web by the experimental group was of higher quality than that gathered by the control group.
Table 4 shows that with the help of the system, the teachers in the experimental group appear to provide deeper
and broader supplementary materials, teaching plans, learning worksheets, and instructional Web pages.
Furthermore, the anecdotes, data reasoning, historical site introduction, and relevant pictures contained in their
teaching materials design are all very helpful in making the given text more interesting.
Adequacy and usability of teaching plans and worksheets
As shown in Table 2, the group difference in teaching plan and teaching strategy fails to achieve statistical
significance (U = 4, p = 0.171). Regarding the usability of the learning worksheet (or handout), the experts’
ratings for the two groups reach a significant level (U = 0, p = 0.014). The learning worksheets developed by the
experimental group are more useful than those developed by the control group.
Although the evaluations of to-be-adopted instructional activities and strategies of the two groups were not
significantly different, we can still discern some differences in the materials. As Table 4 shows, teachers in the
experimental group tend to choose lecturing as their main strategy, and to supplement that with discussions and
worksheets to arouse the interest of students. Among them, teacher A selected a more life-like theme of the
“Ming’s Overseas Tour Guide of the Ming Dynasty” for teaching the assigned unit. Teacher B posed thought
provoking questions at appropriate times throughout the lesson to help students think and reason. Teacher C used
a map for illustration and comparison. Teacher D provided discussion questions so students could make
judgments of historical figures. These activities reflect teachers’ personal styles and originality, their ability to
utilize discussions and everyday life themes in teaching, and their effectiveness to provide supplementary
materials to stimulate thinking and reasoning among students and to encourage them to make comparisons and
offer criticisms. The teachers in the control group choose lecturing as their main strategy, with only key lecture
points of the unit listed on their teaching plans. No other methods are mentioned in their teaching plans, nor were
worksheets developed to supplement the lecture.
Coordination, coherence, and creativity among teaching plans and materials
With respect to the coordination between teaching plan and teaching materials, as well as the coherence of the
teaching materials, the experts’ ratings for the two groups show a marginal significance (U = 2, p = 0.057). This
suggests that the experimental group shows greater coordination between the supplementary information
collected, activity plans, learning worksheets, and Web pages developed using the system. For overall creativity
of teaching materials, experts’ ratings show a statistically significant difference (U = 0, p = 0.014), indicating
that the experimental group demonstrated more creativity than the control group when given the same amount of
preparation time and teaching content.
Table 5 lists the supplementary materials, the content of the teaching plan and learning worksheet, and the major
headings of the Web page produced by teacher B. As the table shows, teacher B adopted direct lecturing and
guided discussion as his main strategy in the teaching plan, which was developed by incorporating the
information gathered from the Web with the textbook content (e.g., a unit about Chen-Ho’s voyage to the west in
the textbook and the materials about Chen-Ho on the Web). The order and content of the lecture are closely
related to the supplementary material the teacher has prepared. Also, teacher B’s learning worksheet is
significantly related to the supplementary material and the progress of discussion and lecture. For example,
teacher B used the question “Did Chen Ho’s boats reach Africa?” which is an issue currently being debated by
historians. Other teachers in the experimental group showed similar features with their instructional materials.
The control group not only failed to produce learning worksheets, but the information they gathered show no
direct relation to the lecture outline in their teaching plans. As can be seen from Table 4, the data collected on
“public flogging at court with clubs,” a unique punishment during the Ming Dynasty, by teacher F are absent
from the key points in his outline. The supplementary material provided by teacher F is not organized for the
purpose of teaching, but rather is presented to students in its original form, which is complicated and somewhat
inconsistent with the entire teaching plan.
Table 4. Information collected and planned teaching strategies by participants
Information gathered from the Web
Activities or strategies suggested in the teaching
plan and worksheets
1. Six relevant pictures.
Uses lectures as the main thread and supplements
2. Supplementary information on the maritime expeditions it with a self-designed learning worksheet, and
of Cheng Ho (his life story, stories of his voyages,
employs a light, everyday theme of “Overseas
relevant websites, historical sites and information left
Tour Guide of the Ming Dynasty” to introduce the
from his journeys).
supplementary materials of the unit.
3. Introductory information on Islam.
1. Six relevant pictures.
Uses lectures as the main strategy, coupled with
2. Stories about the founder of the Ming Dynasty, Tai-tsu, much elaborate and broad-ranging supplementary
on his founding of a new dynasty, political contributions, information. Prior to the introduction of the theme
and problematic policies.
of the lesson, the teacher poses proper questions to
3. Supplementary information on the Jing Nan Rebellion
elicit students’ responses and stimulate their
and the contributions of the succeeding Emperor Chen
thinking as part of his effort to raise students’
after the rebellion.
motivation. The learning worksheet provides
4. Supplementary information on the reforms by Chief
students with questions that help them to review
Minister Chang Chu-cheng.
the lesson, and to think and reason.
5. Information on Tung-lin Partisan Disputes and the
analysis of causes for the decline of the Ming Dynasty.
6. Supplementary information, relevant websites, and
archeological materials on Cheng Ho’s voyages west.
1. Four relevant pictures.
Uses lectures as the main thread and supplements
2. Supplementary information on Cheng Ho’s voyages to
it with a map in the learning worksheet. The
the West (his life story, uniqueness of his expeditions).
teacher attempts to help students brainstorm for
ideas regarding Cheng Ho’s maritime voyages by
comparing the two great geographical discoveries
of the East and West in human history.
1. Two relevant pictures.
Uses lectures as the main thread and allows
2. The background of the founding Emperor Chu Yuanstudents to express their own views and judgments
chang, his rise and important measures at the beginning about historical figures before the conclusion of
of the dynasty.
3. Information on “public flogging at court with clubs.”
4. Introductory websites on Ming Founder Tai-tsu.
1. One picture of Cheng Ho.
2. Brief introduction to Cheng Ho’s life.
3. Brief introduction to the Ming Dynasty and a table of
Ming emperors.
1. Relevant information on “public flogging at court with
1. One picture of Cheng Ho.
2. Supplementary information on Cheng Ho’s maritime
expeditions (his life story and the uniqueness of his
1. Information on the founding of a dynasty by Tai-tsu and
his most important aid Liu Chi.
the lesson, and then introduces various judgments
by historians in order to offer students a broader
perspective. The learning worksheet also requests
students evaluate various historical figures.
Listing the headings for major talking points in the
teaching plan, no notes on any classroom
Listing the headings for major talking points in the
teaching plan, no notes on any classroom activities
or learning worksheet to supplement the lecture.
Listing the headings for major talking points in the
teaching plan, no notes on any classroom activities
or learning worksheet to supplement the lecture.
Listing the headings for major talking points in the
teaching plan, no notes on any classroom activities
or learning worksheet to supplement the lecture.
Table 5. Content analysis of teacher B’s instructional materials
Instructional materials
Titles of main items
Supplementary information
1. Pictures related to the teaching material.
collected from the Web
2. Stories about the founding of the Ming Dynasty by Emperor Tai-tsu.
3. Tai-tsu’s contributions and failures.
4. Jing Nan Rebellion and Emperor Chen’s achievements.
5. Information on the reforms of Chief Minister Chang Chu-cheng.
6. Analysis of causes of the decline of the Ming Dynasty.
7. Information on the Tung-lin Partisan Disputes.
8. General and archeological information on Cheng Ho’s voyages to the West.
Teaching plan
1. Based on the knowledge about emperors already learned, discuss what
(The teacher uses lecture and
family background or life experience it takes to become an emperor.
guided discussion as the main 2. Ask students if they can name any peasant-turned emperors and if they have
strategy in his teaching plan;
heard about Chu Yuan-chang.
the right column shows the
3. Introduce Chu Yuan-chang.
outlines for the lecture and
4. Discuss whether his background as a former peasant will make any
difference in the policies he makes and the regime he forms.
5. Introduce various political measures during Tai-tsu’s rule.
6. Discuss the Jing Nan Rebellion.
7. Discuss the possible political measures Emperor Chen would take to retain
power after he overthrew his predecessor.
8. Tell the class that the maritime industry was highly developed in the Ming
Dynasty and ask if they have heard of Cheng Ho’s expeditions.
9. Discuss the places Cheng Ho visited.
10. Lecture on the savior of the Ming Dynasty—Chang Chu-cheng.
11. Lecture on the causes and impact of the decline of the Ming Dynasty.
Learning worksheet
1. Who was the founding emperor of the Ming Dynasty? Name two peasant(The learning worksheet is
turned emperors in the history of China.
developed both for classroom 2. What are the three reforms that Tai-tsu instituted to ensure a better life for
and after-class use with an aim
his people? What are the potential problems with these measures? Give two
to help students think and
practice; the right column
3. What emperor was pulled off the throne during the Jing Nan Rebellion? And
shows the questions listed on
who succeeded him?
the sheet.)
4. Who contributed to the maritime expansion in the Ming Dynasty and thus
secured China’ status in Southeast Asia?
5. Did Ming’s sovereignty ever reach Africa?
6. What are the most important economic measures of Chief Minister Chang
7. What are the important causes of Ming’s downfall? When did it officially
end? Who is the last emperor that committed suicide?
Teaching material page
1. Opening up a new prospect—Chu Yuan-chang
(The headings and titles on the
teaching material page are
shown on the right, which
includes the pictures and URL
hyperlinks collected by the
teacher for students’ reference.)
2. Important measures at the beginning of the new dynasty
3. Cheng Ho’s voyages to the West
4. Chief Minister Chang Chu-cheng and his “integrated taxation” scheme
5. History champion (This section contains challenging questions that the
teacher poses on the Web.)
Analysis of interview content
The summarization and analysis of the interviews with participating teachers reveal information about
experiences and feelings caused by using different media for teaching material design. All four teachers in the
experimental group showed satisfaction with the friendly environment provided by the proposed system, and
agreed that the web editor and notepad are easy to use, thereby saving time during Web search. Regarding the
notepad, the teachers indicated that everything returned from any single search could be easily retrieved later for
the production of various instructional materials. As teacher A put it,
“…it’s quite a handy tool for editing. It’s just like writing your own lesson plans or reorganizing
various elements. In fact, the moment you start using the notepad, you feel its handiness…I never
thought it would be this easy because in the past when I used the Internet, I used to open a window for
Word on the side so that I could cut and paste along the way. Usually I had to do a lot of dragging and
pasting. After I have searched for a while I would tell myself that it is time to start editing…which I
usually spent an awful lot of time on, so by the time I finished editing and tried to get back to where I
left off searching, boy!...Where was I? What else is left for me to supplement with extra material? So I
find this tool really saves me a lot of time.”
Teachers also consider the URL databank helpful and convenient to their online search effort. As teacher C
“…I think in terms of teaching resource collection. This tool is more convenient because the
information in the databank is already sorted and so contains more relevant websites. That’s why it is
convenient. You can save a lot of time with this databank because you no longer need to weed out
totally irrelevant websites that pop up during your Web search. For example when you type in a few
keywords, sometimes it returns over 200 results, many of them mixed with English and Chinese. It is
very frustrating….”
Teachers further indicated that the proposed system helps them compare different units, which, in turn, promotes
coherent and integrated thinking, offers inspiration, and increases creativity for teaching material design during
their Web search. During their interview, teachers B and D related
Teacher B: “…one of its strengths is that you can store a series of notepads…and retrieve them later,
which makes comparing and connecting units easier.”
Teacher D: “…when gathering information becomes easier, you usually get more inspiration….”
The four teachers in the control group admitted during their interviews that searching the Web for teaching
resources is no easy task. One of the difficulties cited is that the search engines available often return many links
that are totally irrelevant to what is needed. As a result they have to spend a lot of time screening the links one
by one and sometimes have to visit a website to determine if the information is indeed useful. Many times their
search returns with information that is totally unexpected. Teachers in this group also agreed that computer
technology can provide a complement to conventional written texts because it allows concurrent display of text
and graphics.
Discussion and conclusions
The role of information technology is becoming more important in the instructional environment. However, the
lack of appropriate software for instructional purposes inhibits the process of integrating technology with
instruction. To compensate for this lack, we developed a web-based teaching material design and development
system and tested its effectiveness in helping in-service teachers develop their teaching plans and materials. We
hope that the facilities and friendly user interface of the system will be able to reduce teachers’ workloads and
increase the quality of teaching plans and materials designed on the Internet.
Comparing works designed using the system with those designed using an Internet browser and word processor,
we have some encouraging findings. For example, previous studies (Small et al., 1998) indicated that seeking
suitable materials from the Internet is time consuming and frustrating owing to the messiness of the databank,
which then limits the amount of supplementary materials teachers can offer. We addressed this difficulty by
providing a URL collector, a URL browser recorder, and a URL resource bank to assist teachers. Teachers in the
experimental group all successfully and without much difficulty completed many instructional materials with the
supporting functions. On the contrary, using the same of preparation time, teachers in the control group failed to
produce instructional materials of similar quality. Furthermore, Roberts et al. (2000) found that teaching
products made through computer-based technology were superior in design and required less time than
handmade ones, but that there was no significant difference in the creativity of the two kinds of products.
However, teachers in the experimental group not only completed better instructional materials, but also their
materials were more creative and diverse, and the overall plans and materials were more coordinated and better
Innovative instruction relies on both a teacher’s willingness to engage in designing innovative teaching plans and
on having access to facilities for helping achieve such innovation. Computer technology is one of the major tools
that should provide facilities for innovation. However, increased access to equipment and training neither led to
widespread teacher use nor altered existing patterns of teaching practice (Cuban, 2001; Havista & Lesgold,
1996). One possible reason for the failure to achieve the desired level of innovation despite the investment in
effort and resources is the lack of suitable software to optimize the functions of equipment and training. Only
when teachers believe that their workload can be reduced by using computers and quality of teaching can be
enhanced will they be willing to increase their use of computers for instruction. The results of this study also
support this assertion. Both the quantitative and qualitative data from our interviews demonstrate that teachers
using our system experienced a lower workload and higher efficiency. The result was that they then generated
more ideas and designed more activities reflecting their individual styles and originality.
We believe that the design of software for instructional purposes and the exposure to such software are the most
direct ways to link equipment and training, and to optimize the possible benefits of using information
technology. The design and application of our system may serve as a reference for related studies. Due to time
constraints and environmental factors however, this study did not conduct an observation and evaluation of the
use of the system in a real teaching context. Future studies will investigate the usefulness of the system in a
classroom context and other issues related to teaching material design on the Internet.
This research was supported by a grant from the National Science Council, Republic of China, under contract
number NSC93-2524-S-003-014.
Chang, K. E., Chen, S. F. & Sung, T. C. (2001) Learning through computer-based concept mapping with
scaffolding aid. Journal of Computer-Assisted Learning, 17 (1), 21-33.
Chang, K. E., Sung, Y. T. & Chiou, S. K. (2002). Use of hierarchical hyper concept map in Web-based courses.
Journal of Educational Computing Research, 27 (4), 333-351.
Chang, K. E., Sung, Y. T. & Lee, C. L. (2003). Web-based collaborative inquiry learning system. Journal of
Computer Assisted Learning, 19 (1), 56-69.
Cuban, L., Kirkpatrick, H. & Peck, C. (2001). High access and low use of technologies in high school
classrooms: Explaining an apparent paradox. American Educational Research Journal, 38, 813-834.
Ertmer, P. A. (1999). Addressing first- and second-order barriers to change: Strategies for technology
integration. Educational Technology Research and Development, 47, 47-61.
Fisher, M. M. (1997). The voice of experience: In-service teacher technology competency recommendations for
pre-service teacher preparation programs. Journal of Technology and Teacher Education, 5, 139-147.
Havista, N. & Lesgold, A. (1996). Situational effects in classroom technology implementations: Unfulfilled
expectations and unexpected outcomes. In Kerr, S. T. (Ed.), Technology and the future of schooling: Ninety-fifth
Yearbook of the National Society for the Study of Education, part 2 (131-171). Chicago, University of Chicago
International Society for Technology in Education (2000). National Educational Technology Standards for
Students: Connecting Curriculum and Technology. O. R. Eugene: Author.
Keller, J. M. (1983). Motivational design and instruction. In Reiguluth, C. M. (Ed.), Instructional design theories
and models: An overview of their current status (383-434). Hillsdale, NY: Lawrence Erlbaum Associates.
Lawson, T. & Comber, C. (1999). Superhighway technology: Personnel factors leading to successful integration
of information and communications technology in schools and colleges. Journal of Information Technology for
Teacher Education, 8, 41-53.
Leu, D. J., Hillinger, M., Loseby, P., Balcom, M., Dinkin, J., Eckels, M., Johnson, J., Mathews, K. & Raegler, R.
(1998). Grounding the design of new technologies for literacy and learning in teachers’ instructional needs. In
Reinking, D., McKenna, M., Labbo, L. D. & Kieffer, R. (Eds.), Handbook of literacy and technology:
Transformations in a post-typographic world (203-220). Mahwah, NJ: Erlbaum.
Linn, M. C. (2000). Designing the knowledge integration environment. International Journal of Science
Education, 22, 781-796.
Mannaz, M. (1999). An expert teacher’s thinking and teaching and instructional design models and principles:
An Ethnographic study. Educational Technology Research and Development, 46, 37-64.
Marx, R. W., Blumenfeld, P. C., Krajcik, J. S. & Soloway. E. (1998). New technologies for teacher professional
development. Teaching and Teacher Education, 14, 33-52.
Niederhauser, D. S. & Stoddart, T. (2001). Teachers’ instructional perspectives and use of educational software.
Teaching and Teacher Education, 17, 15-31.
Norum, K., Grabinger, R. S. & Duffield, J. A.(1999). Healing the universe is an inside job: Teacher’s views on
integrating technology. Journal of Technology and Teacher Education, 7, 187-203.
OTA (Office of Technology Assessment). (1995). Teachers and technology: Making the connection.
Washington, DC: Government Printing Office.
Park, O. (1991). Hypermedia: Functional features and powerful ideas. NewYork: Basic Books.
Roberts, S. K. & Hsu, Y. S. (2000). The tools of teacher education: Preservice teachers’ use of technology to
create instructional materials. Journal of Technology and Teacher Education, 8, 133-152.
Rogers, P. A. (2000). Barriers to adopting emerging technologies in education. Journal of Educational
Computing Research, 22, 455-472.
Scheffler, F. L. & Logan, J. P. (1999). Computer technology in schools: What teachers should know and be able
to do. Journal of Research on Computing in Education, 31, 305-326.
Siegel, S., & Castellan, N. J. (1988). NonparametricSstatistics for the Behavioral Sciences. New York: McGraw
Small, R. V., Sutton, S., Miwa, M., Urfels, C. & Eisenberg, M. (1998). Information seeking for instructional
planning: An exploratory study. Journal of Research on Computing in Education, 31, 204-215.
Topper, A. (1998). Co-constructing practices of teaching with technology: Working collaboratively with TE
instructors on technology adoption. Technology and Teacher Education Annual 1998, 74-748.
Willis, J., Thempson, A., & Sadera, W. (1999). Research on technology and teacher education: Current status
and future directions. Educational Technology Research and Development, 47, 29-45.
Tselios, N., Stoica, A., Maragoudakis, M., Avouris, N., & Komis, V. (2006). Enhancing user support in open problem
solving environments through Bayesian Network inference techniques. Educational Technology & Society, 9 (4), 150-165.
Enhancing user support in open problem solving environments through
Bayesian Network inference techniques
Nikolaos Tselios
Human-Computer Interaction Group, Electrical and Computer Engineering Dept and ICT in Education Group,
Dept of Educational Sciences and Early Childhood Education, University of Patras, Greece
[email protected]
Adrian Stoica
Human-Computer Interaction Group, Electrical and Computer Engineering Dept, University of Patras, Greece
[email protected]
Manolis Maragoudakis
Wireless Communication Laboratory, Electrical and Computer Engineering Dept, University of Patras, Greece
[email protected]
Nikolaos Avouris
Human-Computer Interaction Group, Electrical and Computer Engineering Dept, University of Patras, Greece
[email protected]
Vassilis Komis
ICT in Education Group, Dept of Educ Sciences and Early Childhood Education, University of Patras, Greece
[email protected]
During the last years, development of open learning environments that support effectively their users has
been a challenge for the research community of educational technologies. The open interactive nature of
these environments results in users experiencing difficulties in coping with the plethora of available
functions, especially during their initial efforts to use the system. In addition, -from the tutors’ perspectivethe problem solving strategies of the students are often particularly difficult to identify. In this paper, we
argue that such problems could be tackled using machine learning techniques such as Bayesian Networks.
We show how we can take advantage of log files obtained during field studies to build an adaptive help
system providing the most useful support to the student, according to the state of interaction. On the other
hand, we attempt to support the tutor, by automating the process of diagnosing students’ problem solving
strategies using Bayesian Networks. The presented approaches are discussed through examples of two
prototypes that have been developed and corresponding evaluation studies. These studies have shown that
the proposed approach can effectively support the tasks of students and tutors in such open learning
Bayesian Belief Networks, Open problem solving environments, Inference algorithms, On-line adaptation,
Adaptive help, Automated problem solving strategy identification.
During the last few years, a number of open problem-solving environments have been built that are based on the
constructivist approach. Open problem solving environments are computer-based learning environments that let
students actively explore certain concepts while they are engaged in problem solving, with emphasis in the
active, subjective and constructive character of learning (Luger and Stubblefield, 1998). Compared to traditional
learning environments the student’s activity cannot be reduced in a sequence of pre-defined tasks. In general,
through an open problem solving environment the student is engaged in processes such as model building,
investigation and reflection. As a result, students have the freedom to explore various entities in their own unique
way. The open nature of these environments, results in users’ activity taking various forms and the patterns of
use of the tools that are included in them not to be fully anticipated. Therefore, the complexity of such
environments and the open nature of the tasks can often lead to poor usability, which can be an obstacle to obtain
the expected pedagogical value from their use.
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
In general, through an open problem solving environment the student is engaged in processes such as model
building, investigation and reflection. The process of building such a model is usually carried out in three phases.
First, the system allows users to express their ideas, through ‘entities’ that are related to objects, corresponding
to their phenomenological status. These objects have properties, directly manipulated by the students, and
behaviour. At the second level, the students can relate explicitly or implicitly a number of entities to create a
more abstract entity, considered also as a ‘construct’, depicting an object from a group of uniform objects, that
take meaning in the context of a phenomenon, system, process or conjecture (Komis et al., 2001,
Dimitracopoulou and Komis, 2005). Finally, the students are able to ‘run’ the model to validate their hypothesis
and observe the function of their model, often aided by data visualisation tools such as equation plots, graphs, etc
(Komis et al., 2002).
Improving usability of these environments is an objective that could be achieved in various ways, such as.
applying user centred design techniques (Norman, 1986) at design time or by building user support components
for run time. User centred design propose advanced usability evaluation techniques during system design. For
instance, various usability evaluation techniques that take into account both the pedagogical value and the
usability of the environment have been proposed (Squires and Preece, 1999, Avouris et al., 2000). Furthermore,
more complex user task modelling approaches have been proposed for the design of such environments (Tselios
et al., 2002). On the other hand, user support at run time through adaptive systems is an approach that could
improve usability. It should be recognized that, overall, artificial intelligence techniques have not succeeded to
deliver the expected results in the educational field in the form of Intelligent Tutoring Systems, despite the
existence of some sporadic promising results (Woolf et al., 2001, see also Kinshuk and Russell, 2002, for a
classification of adaptable and adaptive systems). However, the premise of adaptive system behaviour through
which higher usability and increased system transparency can be obtained remains a valid scientific objective.
Our approach is compatible with the vision of Suthers et al. (2001) which stress the value of ‘Minimalist AI’ in
education: Instead of trying to build smart machines that teach -a rather optimistic goal demanding a great effort
of modeling knowledge in a particular subject, as well as pedagogy strategies and explanation mechanisms-, this
approach suggests providing machines with abilities to ‘respond to the semantics of student activities and
constructions, to test the educational value of these abilities, and add functionality as needed to address
deficiencies in the utility of the system’.
In the research reported in this paper, we attempt to investigate the usefulness of Bayesian Networks in tackling
two significant problems, common in open learning environments. These two problems are complementary in
nature and address difficulties of students and tutors when engaged in activities in open learning environments:
¾ First, from a student perspective, an adaptive help module is presented which automatically recognizes the
most probable next action and presents relevant help items. The need for such an adaptive help system is
significant, especially in the case of an open problem solving environment where the cognitive effort to deal
with the task is affected by concerns on how to handle the various tools that may be used for accomplishing
a certain task.
¾ From a tutor perspective, we attempt to present a practical method to classify effectively problem solving
strategies expressed by pupils while they were using such an open problem solving environment. Due to the
nature of such environments, problem solving strategies could be numerous. Posteriori analysis of log data
in order to identify the strategy expressed by the students could be a rather tedious and painstaking process.
Thus, a method to automatically classify the solutions presented to the users could substantially increase the
evaluation of the learning process.
In the presented cases, we try to implement a state-based AI approach instead of a rich knowledge-based
(Nathan, 1998). We attempt to identify important states, features and repeatable patterns in the students’ system
interaction cycle in order to infer about their intentions or adopted strategy during their problem solving activity,
while retaining at the same time the sense of self-activity and engagement. Bayesian networks are used as
modelling tools in this state based approach in both cases.
This paper is organized as follows: First, we present an overview of interesting applications of Bayesian Belief
Networks (BBN) in various problems in the field of educational technology. Then, we briefly present Bayesian
probabilistic theory together with a description of the notion of BBN’s and their implementation in educational
environments. Next, we present our approach to build an adaptive help system in the ModelsCreator open
problem solving environment. The proposed approach of adapting user support through a BBN is described. The
BBN was built using a large amount of log files of actual usage of the system. Next, a system testing and
evaluation experiment that took place in order to demonstrate the benefits of the proposed architecture is
presented, followed by a short presentation of the results obtained from our experience of student modelling with
BBN in open problem solving environments. Finally, we present our approach to automatically identify users’
strategies during their interactions with an open learning environment that is used for teaching geometrical
concepts to pupils aged 11-14 years. In the proposed approach, a BBN has been used to infer problem-solving
strategies that the pupils applied in order to solve a given problem from logs of activity of typical pupils. In this
case, identification of user’s strategies was done with the aim to facilitate classification of the solutions presented
by the students and aid the evaluation of the learning process, a typical tutor task.
Use of Bayesian Belief Networks in Educational Systems
In recent years, symbolic modelling approaches applied in traditional AI problems, have been replaced by
implicit models built from rich data sets through techniques proposed by Machine Learning and Knowledge
Discovery from typical data of a specific domain. These techniques have produced efficient algorithms during
the last years and found new areas of application. Among them, a technique that has certain advantages and has
received attention during the last few years is based on Bayesian Belief Networks (BBN’s, Niedermayer, 1998).
A Bayesian Belief Network (Stephenson, 2000) is a directed acyclic graph, where each node represents a random
variable of interest and each edge represents direct correlations between the variables.
Bayesian reasoning is based on formal probability theory and is used extensively in several current areas of
research, including pattern recognition, decision support and classification. Assuming a random sampling of
events, Bayesian theory supports the calculation of more complex probabilities from previously known results
(Luger and Stubblefield, 1998). The advantages of BBNs include the simple process for constructing
probabilistic networks even from a relatively limited amount of data, the efficient algorithms to evaluate degrees
of belief for instances of a node and the versatile knowledge representation which such networks provide.
Due to the underlying probabilistic model that describes the belief on the existence of a specific event, BBNs are
considered one of the most effective ways to represent uncertainty. By formulating a limited, approximate but
representative cognitive model of the users suitable for the specific problem, and then modeling uncertainty in
human computer interaction, future activity could be supported. This is done through the modeling of a user’s
activity, recorded in click streams of typical user behavior when interacting with a software system. This could
lead to interpretation of the nature of the cognitive processes involved and to more efficient ways to support
future users’ tasks. This approach is particularly suitable for learning applications, in which interactions are
complex and support is often needed.
So in recent years, various prototypes have been produced, demonstrating that BBNs are suitable for effective
modelling of student behaviour (Jameson et al., 1995). BBNs have been utilized in various ways to achieve
adaptability in educational environments in terms of determining student goals, determining feedback,
curriculum sequencing and fine-tuning of the pedagogical strategy to deliver knowledge. ANDES (Conati et al.,
1997), is a system that teaches physics problem solving techniques to college students, uses BBNs to identify the
current problem solving approach of the user. ANDES also use a BBN to determine what hints to provide to the
user by identifying how the student is solving a problem and how he has progressed down the solution path.
Extensions of the modeling process to cope with issues that arose in scaling up the model to a full-scale, field
evaluated application coupled with results of several evaluations of Andes which provide evidence on the
accuracy of the implemented models are presented by Conati et al., (2002). Bunt and Conati (2003) attempt a
generalization to the student modelling process in open problem environments. They argue in favour of using
such a technique, finding it beneficial even for those learners who do not already possess the skills relevant to
effective exploration. The scope of their model is twofold: To assess the effectiveness of a learner's exploratory
behaviour in an open learning environment and to monitor the learners' actions in the environment unobtrusively
in order to maintain the unrestricted nature of the interaction is one of the key assets of such environments.
In addition to reasoning at a lower interaction level about student actions it is possible to use a BBN in order to
reason at a higher level. Collins et al. (1996) constructed a BBN that represents a hierarchy of skills for
arithmetic, containing the test questions, the theory and the user’s skill in specific topics. The semantics implied
by the links, represent which question refers to a specific topic. The system considers the current estimate of the
student’s ability to compute the probability of responding correctly to each question contained in the database
schema. Then the system selects an item that the student has a near to 50% chance of answering correctly. With
regard to the selected pedagogical approach, the criteria of item selection could be altered appropriately. CAPIT,
(Mayo and Mitrovic, 2000), is a constraint-based tutor for English capitalization and punctuation that uses BBN
for long term student modeling. An evaluation of this system showed that a group of students using the adaptive
version learned the domain rules at a faster rate than the group that used the non-normative version of the same
Usage of Bayesian Networks is not limited to monitoring the student’s behaviour in an educational environment.
For example, Vomlel (2004) stresses the use of Bayesian Networks for educational testing. He presented a series
of case studies referring to operations that use fractions. He showed that a Bayesian network models relations
between required skills to carry out these operations and improve the process of student's evaluation. Also Xenos
(2004), presents a methodological approach based on Bayesian Networks for modelling the behaviour of the
students of a bachelor course in computers in an Open University that deploys distance educational methods.
This method offered an effective way to model past experience, which can significantly aid decision-making
regarding the educational procedure. According to Xenos (2004) this approach can also be used for assessment
purposes regarding the current state enabling tutors to identify best and worst practices.
As discussed in this section, probabilistic models, such as BBN have focused on traditional “drill and test”
systems to a great extent. However, in the research reported here, we attempt to show that applications of this
method are not necessarily constrained only in the aforementioned type of systems. In the following, we show
applications of BBNs in open learning environments with a much more complex and non linear nature of user
system interaction. As described in the rest of the paper, this class of educational systems could also benefit from
a BBN approach, in a variety of ways.
Bayesian Modeling
A Bayesian Belief Network (BBN) is a significant knowledge representation and reasoning tool, under
conditions of uncertainty (Mitchell, 1997). Given a set of variables D = <X1, X2…XN>, where each variable Xi
could take values from a set Val(Xi), a BBN describes the probability distribution over this set of variables. We
use capital letters as X,Y to denote variables and lower case letters as x,y to denote values taken by these
variables. Formally, a BBN is an annotated directed acyclic graph (DAG) that encodes a joint probability
distribution. We denote a network B as a pair B=<G,Θ>, (Pearl, 1988) where G is a DAG whose nodes
symbolize the variables of D, and Θ refers to the set of parameters that quantifies the network. G embeds the
following conditional independence assumption:
Each variable Xi is independent of its non-descendants given its parents.
Θ includes information about the probability distribution of a value xi of a variable Xi, given the values of its
immediate predecessors. The unique joint probability distribution over <X1, X2…XN> that a network B describes
can be computed using:
PB ( X 1 ... X N ) =
∏ P(x
| parents ( X i ))
i =1
Learning BBN from data
The process of efficiently detecting the most suitable network is not straightforward. Thus, a BBN should be
learned from the training data provided. Learning a BBN unifies two processes: learning the graphical structure
and learning the parameters Θ for that structure. In order to seek out the optimal parameters for a given corpus of
complete data, we directly use the empirical conditional frequencies extracted from the data (Cooper and
Herskovits, 1992). The selection of the variables that will constitute the data set is of great significance, since the
number of possible networks that could describe N variables equals to:
N ( N −1 )
where N is the number of variables (Jeffreys, 1939).We use the following equation along with Bayes theorem to
determine the relation r (or Bayes factor) of two candidate networks B1 and B2 respectively:
P( B1 | D)
P( B2 | D)
P(B| D) =
P(D| B)P(B)
P(B|D) is the probability of a network B given data D.
P(D|B) is the probability the network gives to data D.
P(D) is the ‘general’ probability of data.
P(B) is the probability of the network before seen the data.
Applying equation (1) to (2), we get:
P( D | B1 ) P( B1 )
P ( D | B2 ) P ( B 2 )
Having not seen the data, no prior knowledge is obtainable and thus no straightforward method of computing
P(B1) and P(B2) is feasible. A common way to deal with this is to assume that every network has the same
probability with all the others, so equation (3) becomes:
P( D | B1 )
P( D | B2 )
The probability the model gives to the data can be extracted using the following formula (Glymour and Cooper,
P( D | B) = ∏∏
i =1 j =1
+ N ij ) k =1
+ N ijk )
ri qi
ri qi
¾ Γ is the gamma function.
¾ n equals to the number of variables.
¾ ri denotes the number of values in i:th variable.
¾ qi denotes the number of possible different value combinations the parent variables can take.
¾ Nij depicts the number of rows in data that have j:th value combinations for parents of i:th variable.
¾ Nijk corresponds to the number of rows that have k:th value for the i:th variable and which also have j:th
value combinations for parents of i:th variable.
¾ Ξ is the equivalent sample size, a parameter that determines how readily we change our beliefs about the
quantitative nature of dependencies when we see the data. In our study, we follow a simple choice inspired
by Jeffreys (1939) prior. Ξ is equal to the average number of values variables have, divided by 2.
Given the great number of possible networks produced by the learning process, a search algorithm has to be
applied. We follow greedy search with one modification: instead of comparing all candidate networks, we
consider investigating the set that resembles the best current model the most.
In general, a BBN is capable of computing the probability distribution for any partial subset of variables, given
the values or distributions of any subset of the remaining variables. Note that the values have to be discretised,
and different discretisation size affects the network. As we shall discuss in the result section, BBNs constitute a
significant tool for knowledge representation, visualising the relationships between features and subsets of them.
This fact has a significant result on identifying which features actually affect the class variable, thus reducing
training data size without any significant impact in the performance.
However, we should stress the fact that transforming the Bayesian Belief Network elicitation process to a
tractable procedure is not enough. The most important step, while attempting to resolve a problem using the
application of such a technique, is the establishment of a deep understanding of the nature of the problem. This
understanding could lead to a manageable transformation of the problem domain, where applications of BBNs
could serve as an important inference tool. From our expertise obtained dealing with the adaptation problems
described in the following, attempting to re-express the problem to be modelled should first include an initial
understanding and clarification of the prominent variables of the domain which influence our reasoning about the
state or the goal of the student while executing a task. These variables, coupled with nodes indicating the issue
(the ‘class’ variable) that is needed to infer upon, are used to feed the BBN construction process. The states of
the variables represent actual user’s actions, neither at a high task level, nor in the lowest keystroke level of
interaction. The granularity of the abstraction is heavily depended by the nature of the problem and our aim. It
seems that different problems may need a slightly different approach. For example, while attempting to
recognise a student’s current goal, the temporal order of her recent actions significantly influences our inference
about her forthcoming action. When we attempt to infer general goals of the user from log files, it seems that
focusing upon the frequency of a tool’s usage is a preferable selection.
Using BBN to Construct an Adaptive Help System
This section presents an adaptive help system that was built using BBNs, for ModelsCreator (MC), an open
learning environment for modelling activities of young students. ModelsCreator allows users to create models
representing aspects and phenomena of the natural world and provides them with tools for testing their models
(simulations, graphs, tables of values, etc.). ModelsCreator design and implementation has been inspired by two
main design principles: (a) support of expression through different kinds of reasoning in a simplified and
synthetic mode and (b) model mechanisms that derive from different subject matters, which permit
interdisciplinary approaches (Komis et al., 2001, Dimitrakopoulou and Komis, 2005). Models can be built in
MC, using qualitative reasoning, as well as quantitative and semi-quantitative relations that can be used to link
various objects, representing primitive concepts. The system supports a high level of visualisation, combines
modeling tools with simulations and incorporates alternative and multiple forms of representation (Von
Glasersfeld, 1987). Visual representations of entities in a specific topic (i.e. food, water, photosynthesis)
constitute essential elements of the environment since they have the mediating role of providing students with a
‘concrete’ framework to reason on an abstract level (Figure 1).
Concerning the MC user interaction design, a direct manipulation space, suitable for young students has been
designed. A view of this space is shown in Figure 1, in which the modelling environment of MC can be seen. It
has been shown (Komis et al., 2002, Ergazaki et al., 2005) that this system can provide a rich and constructive
learning experience to young students. For example, in Figure 1, in the main modelling space a typical model of
a biological phenomenon (photosynthesis) is shown. Entities like a plant, the sun, the soil, the leaf of the plant
and the substances of the air are included in the model and are related through qualitative relations. On the left, a
library of entities is included. These can be dragged into the modelling space. On the right, the available relations
are shown. Finally, on the top, various tools for model manipulation, for creating various representations of the
model and running the model are included.
Figure 1. ModelsCreator’s modelling environment
Online help adaptation using Bayesian networks
ModelsCreator contains a rich help system. However, it was discovered that in its earlier form this help system
was particularly difficult to use by young students who had to spend a long time browsing in large areas of
information in order to receive support. So a BBN has been developed in the context of the reported research,
used in order to adapt the user support (help) system of ModelsCreator. The BBN was built using data collected
from typical real world user interaction with the MC environment. The model was based on the selected data, so
the approach we followed can be considered as a data-centric one. The data used to train the structure and the
probabilities of the Bayesian network was obtained from a quite large amount of log files produced by
ModelsCreator during previous experiments, involving 15 students aged 11-14 years. . Representative tasks were
given to the students to accomplish. An extract of the log files used can be seen in Figure 2. This is a history of
the sequence of operations of a user in the MC environment. Each operation might have associated attributes so
the first operation of the user was an insertion of an entity in the modelling space. The attribute of this specific
action was the kind of entity that was inserted (the “Plant” entity).
A log file pre-processing procedure took place and resulted in the building of a database containing some 2200
logged actions. Our effort was to construct a BBN in order to obtain a belief about the next most probable action
of the student with respect to the current state of the interaction flow. We assumed that by predicting the next
most probable operation, we can infer the user intention and thus relate the most relevant section of the help
system if the user sought support. The database that was built in this pre-processing phase contained the required
fields to predict the next action given the two previous actions. The field Property of the action has been used for
each action describing the particular nature of the action. For example, when a user selects a particular attribute
of an Object, the selected attribute is recorded in the property field. The fields of the database are: Previous
Action (Paction), Previous Property of the carried out Action (Pprop), Previous Action Time, Current Action
(Caction), Current Action Property (Cproperties), Current Action Time, Next Action (Naction), Next Property
and the Time difference between current and previous (CPTime_d). The database has this structure since the
temporal ordering of the actions has not been taken into consideration.
The user activity in MC is very unpredictable. Therefore, there is virtually no constraint in solutions that the user
might adopt and express during problem solving. Thus, an adaptation procedure in a higher cognitive goal level
(e.g., concerning specific goals which the user attempts to accomplish, or even a sub-goal as a part of a pupil’s
approach to solve a problem) could prove very difficult to be carried out. Instead, the proposed approach focuses
on general patterns that can occur in the stream of user interaction with the software.
The log file extract shows a representative student’s interaction with the system (Gudzial, 1993). First the user
attempts to enter three entities (Plant, Leaf, Sun) to the MC’s working space (Actions 1,3,4). Subsequently, she
selects specific attributes for each entity during her effort to build a model describing causal-effect relationships
between the members of the model (Actions 2 and 5-12). Finally she executes the model in order to observe the
behaviour of the model. To further examine the exact nature of the relationships between the entities, she creates
a barchart representing the evolution of the three selected entity’s attributes over time (Actions 13-18).
01. 00 : 08 : 14 InsertEntity
02. 00 : 08 : 19 ChooseAttribute
03. 00 : 08 : 37 InsertEntity
04. 00 : 08 : 40 InsertEntity
05. 00 : 08 : 46 ChooseAttribute
06. 00 : 08 : 49 ChooseAttribute
07. 00 : 09 : 07 InsertRelation
08. 00 : 09 : 07 ConnectRelation
09. 00 : 09 : 07 ConnectRelation
10. 00 : 09 : 48 InsertRelation
11. 00 : 09 : 48 ConnectRelation
12. 00 : 09 : 48 ConnectRelation
13. 00 : 10 : 00 RunModel
14. 00 : 11 : 19 BarChartActivation
15. 00 : 11 : 21 GraphChooseAttribute
16. 00 : 11 : 23 GraphChooseAttribute
17. 00 : 11 : 24 GraphChooseAttribute
18. 00 : 11 : 31 RunModel
Property of the action
Intensity of Light->Sun
Intensity of Light
Intensity of Light->Sun
Figure 2. Extract of the log file used to build the BBN
In the derived BBN (Figure 3), causal relationships between Previous action (PAction) and Current action
(CAction), and between Current action and Next action have been depicted. From an interaction flow
perspective, the structure is semantically meaningful and the relation between the previous and current action of
a student is an expectable empirical model of interaction. To find the probability of occurrence of an instance of
the next action variable, only the current instance of the current action is needed. The structure of our Bayesian
network is a tree structure denoting that it is suitable to be used even in real time training and adaptation as well,
since the time required to obtain such a structure from data is polynomial (Stephenson, 2000).
Figure 3. The final Bayesian network obtained. Previous action (PAction), Current action (Caction) and the time
difference between them (CPTime_d), influences network’s belief concerning the most probable next action
Environment architecture modification
The architecture of ModelsCreator was enriched with a new module that incorporated the developed BBN and
enhances the functionality of the User Support Module (Help System). The architecture of the new module
named Adaptive User Support Module (AUSM) is conceptually presented in Figure 4.
The role of the new module is twofold. First, it is to collect data from the user’s stream of actions and to write
them into a database. Subsequently, to identify the current interaction state using the collected data and to
provide at any time the most appropriate help topics according to the probabilistic estimation made using the
developed BBN. The database obtained is used as input for a Bayesian modelling tool that will process the data
in the database and will construct the BBN that models our problem. The resulting network is exported in Hugin
Lite file format. The module parses the file that contains the network and uses the conditional probability table
within to provide the most useful help topics to ModelsCreator by predicting the most probable actions of the
student to follow. This Bayesian network is actually used by a module that interacts with our open problemsolving environment, ModelsCreator and with a Bayesian modelling tool through files (it reads the BBN
description file and writes the database with the data to update the probabilities of the network).
Figure 4. Data exchange between modules in the frame of the Enhanced User Support System of ModelsCreator
At run time the developed module operates as follows: Every action of the user is sent to the AUSM module
through an appropriate method call. When the user asks for help the ModelsCreator will call another method of
AUSM that will return the most probable topics for the current state of the interaction. These actions are
obtained by instantiating in the Bayesian network the current action node and calculating the probabilities for the
next action instances. Each next action is related to a relevant help topic. When a new action is performed the
AUSM module writes in the database the data relating to the previous, current and next actions. Afterwards, it
sends the action performed further so that the module is able to instantiate the current action node with the
current action performed thus providing the adaptive version of user support.
An example of the actual behaviour of the AUSM module is showed in Figure 5. In order to connect two entities
with each other, the student has to enter those entities into the main space, select at least an attribute in each
entity, drag a relationship and finally connect the edges of the relationship with the desired attributes, by pointing
exactly at the area of the entity where the verbal representation of the attribute is presented. This procedure posed
significant problems to the students, and patterns of interaction showing repeated attempts to properly connect a
relationship were detected in the log files obtained. When the user asks for help, while she tries unsuccessfully to
connect two objects with a relation, MC’s extended version presents the help section proposing three help topics
suggested as relevant to the current interaction state. All help topics are presented in a task driven form to help
the user carry out specific tasks. That is, the help system is not constrained to a description of each element of
the user interface which is of limited help to the user, but it describes in detail how a representative task is
carried out using MC’s environment. In the specific example described, the three proposed help items explain
how to connect a relation (which is actually the desired help item), how to run a model and how to insert a
relation into the MC’s environment (Figure 5). This approach considerably facilitates the task of finding the
desired help item: Instead of selecting from a plethora of available help topics concerning a variety of actions,
the user is provided with the most probable three. If the desired help item is not successfully suggested by the
AUSM as in this example, the system alternatively permits viewing the help topics, sorted in alphabetical order
and grouped by subject.
Module evaluation
Evaluation of the developed system was carried out in three different ways. First, the predictive performance has
been measured using the training data set. Secondly, through inspection of the efficiency of the realised AUSM
system against log files obtained by another study, and finally with a user observation involving actual users
while executing representative tasks with the system.
User support
Figure 5.The user interface of the adaptive user support environment. Three options are provided to the user in
the help window, with the most probable one actually shown to the user
It is common practice to evaluate BBN predictive performance by testing the network against data that were not
used for training it. This is an advantage of such data-driven approach against efficiency-centric and expertcentric approaches for development of an adaptive system (Heckerman, 1996). The effectiveness of the Bayesian
Network obtained was validated by using the 10-fold cross validation method. With this method, the data
population is randomly divided to training and validation data sets in a 9:1 ratio, repeated ten times. The mean
performance of the developed network was found to be 88,43% (+-1,36%, p<0.05) which is considered
remarkably high considering the relatively limited amount of data used to train the network.
As described in the previous section, the Bayesian network structure and conditional probabilities were derived
from data collected from 15 log files produced by actual use of the ModelsCreator software. A new log file was
obtained during an experiment in which the activity given to the student to complete was of a different nature
than that of the tasks in the training data set. This log file contained 141 records of user actions and was used to
evaluate and test the adaptive module.
An agent has been implemented to simulate performing the actions contained in the log file and logged the
action and the help topics provided by the AUSM module. For every action performed, the module provides the
most probable three next actions. The help topics presented by the AUSM are directly related with the
predictions in the form: <most probable next action, second most probable action and third most probable
action>. Therefore, the quality of the help item presented to the user is directly related to the successful
prediction of the next action. The adaptive module inferred successfully the next action and designated as the
most probable action in 44,681% of the cases, as the second most probable in 24,113% of the cases and as the
third most probable action in 8,511% of the cases. So the overall result is that the adaptive module guessed
correctly the next action in 77,305% of the cases.
The results are of the same order with those produced by the mathematical evaluation of the BBN derived,
especially if we consider the fact that the task presented here was completely different from the tasks that
actually the BBN was constructed from. Extensive testing of several configurations and reformulations for the
data used in order to train the Bayesian network, showed that the best results were obtained when we kept track
of previous and current action and we tried to guess the next action. In addition to this, taking into account a
third action, led to a very small improvement to the obtained results.
Finally, a user study was conducted, involving 5 students (four male, 1 female), in order to further examine the
robustness of the Enhanced User Support Model (AUSM). Representative activities were given to the users to
carry out, such as creating a simple model, execution of the model and creation of alternative representations for
selected variables of interest of the model. During observation, the users were asked to externalise their thoughts
and retrieve help via the AUS Module whenever they had problems carrying out the task. In addition to this, they
were asked to evaluate the relevance of the proposed help items in a scale from 1 (not relevant) to 5 (very
relevant). 32 calls of help were recorded in total (6.4 per user). The relevance of the proposed help items was
3,358 on average (standard deviation 0,212). After the end of the session, a questionnaire was given to the
student, to evaluate effectiveness of the AUSM. The questions and the results of the questionnaire are
summarised in Table 1.
Table 1.Synopsis of the questionnaire referring to the AUSM, and the results obtained (Scale 1-5)
Mean Value
The AUSM helped you carry out specific tasks?
Did you find suitable the approach to present the three more relevant help items?
How frequently did you find the AUSM useful?
Did you find the help information presented understandable and in suitable structure?
To what extent did you find the AUSM useful to you?
The presentation of the three most relevant help items created confusion to you?
1,66 (4,33)
Is it difficult to find the desired information item using AUSM?
2,66 (3,33)
Overall Evaluation
In general, the users evaluated very positively the existence of the Adaptive Help Module, particularly in the
case where many alternative tools were available, leading to confusion and long searches for help. The
discussion with the users, revealed their preference for active exploration of the facilities of the software when
they use it for the first time. This statement and the observation that the users avoid reading handbooks prior to
direct experimentation with a new software environment has been recorded also by other researchers (Hellman,
Some specific observations of the user study are discussed next. In some cases, (for example (a) activity of
linking two objects in a model with a relation and (b) the selection of variables to create a graph), the provided
help was found to be critical for successful and uninterrupted task execution. This was discovered during the
user study. These problems were attributed to the fundamental gap between the designer’s conceptual model
(actual system design) and the user’s conceptual model (expectations of the users referring to the actual way to
carry out specific actions). On another occasion, the users experienced frustration when they repeatedly tried to
change the direction of a specific relation between two objects, an action that is not directly supported by the
MC. Finally, the users found the cut and paste tools and functions difficult to use. Concerning the AUSM’s
presentation of information, they found that the explanations are clear and well written, but they would like to
have more links to relative topics in the body of the help text.
Finally, the actions carried out during the usability evaluation of the AUSM were recorded, collected and
analyzed. They were analyzed against the relevance of the three help items presented to the user, concerning the
actual user’s next action. The results were in line with the preliminary evaluation of AUSM. The AUSM
presented first the desired help item in the 40,181% of analysed actions, as second in the 27,765% of actions and
as third most likely in the 5,869% of cases, with a total forecast success 73,815%. The analysis of the events
seems to be in line with the user’s subjective evaluation of the AUSM.
Using Bayesian Networks for automatic classification of problem solving strategies
In this section, development of a tool for automatic classification of problem solving strategies is described. The
source data for BBN construction were captured from user observation sessions of 30 high school students while
using the C.AR.ME microworld. This environment is an open interactive system using multiple representations
for learning concepts of geometry, i.e. equivalent shapes and surface measure methods, (Kordaki and Potari,
1998). The experiment concerned activities for solving two distinct problems: (a) transformation of a nonconvex polygon into a geometrically equivalent shape and (b) surface comparison of a non convex polygon with
a square. Students were asked to solve these two problems with all the possible approaches that might find
appropriate, using the tools offered by the microworld. The analysis of the problem solving strategies expressed
by the students is extensively described in the work of Kordaki and Potari, (1998). The different problem
solution strategies that the students used were classified in thirteen (13) categories. The most common strategy
was described as “Using the measurement function”, that was applied by 23.7% of the students, while the “Using
the automatic measurement function” was applied by 18.4% . These two strategies involved direct use of
provided tools. Other strategies involved direct manipulation of shapes and combination of tools with direct
manipulation and transformation of the drawn shapes.
114 log files have recorded the sequences of operations of the 30 students to solve the given problems. Each one
of them was related to a specific problem solving strategy. These log files were used to construct the BBN. In
most of these log files, extensive comments were associated, describing the strategy expressed by the student,
accompanied by a screenshot showing the final state of the microworld, at the end of the problem solving
process. These log files were analysed using the Usability Analyzer (Tselios et al., 2002), a tool supporting
analysis of log files captured from educational environments. In this tool, the evaluator can review the
interaction process, by observing simultaneously the captured log files, comments taken during the field study
and recorded screenshots showing the student’s interaction with the educational software.
The next phase was to store the log files’ information into a table. The fields used were the 20 different low level
student operations recorded and the frequency of usage for each one in every problem solving strategy. For
example, Cut and Paste operations depict editing actions, MeasuringAreas depicts usage of a tool to measure the
area of a specific area, and Triangle, or Rectangle operations depicts usage of automated tools to transform a
geometric shape to another type of equivalent area. The hypothesis used to build the BBN and transform
accordingly the interaction data, was the hypothesis that whenever a student adopts a specific strategy to solve a
problem, he carries out a set of specific operations with high frequency. This kind of activity could be traced in
her log file, and could help us classify automatically the problem solving strategy. Additionally information
stored, referred to the total number of interaction events (sum variable) and the type of strategy used as
recognized originally by the evaluator (class variable, see Figure 6).
ΒΒΝ Construction
The diagrammatic representation of the BBN obtained is shown in Figure 6. In general, a BBN is able to derive
the probability distribution for any given variable subset, for specific values of the remaining variables. From the
structure of the network it is shown that the startdraw and enddraw variables -which are the events created when
the student begins or completes a shape’s drawing accordingly- are depending the one from the other, but are not
connected with the main body of the network. This is not something unexpected, since these events indicate only
the start and the end of the user’s activity, but not the exact nature of the specific problem solving strategy. In
addition, four (4) more types of operations used in the log files, UnitIteration, Erase, Clear and Symmetry are
found to be independent from any other variable and they were excluded from the graph.
Figure 6. Bayesian network obtained to infer for the problem solving strategy expressed by the student from the
log files
Useful information can be extracted by observing the BBN graph. An example of the use of the network
obtained is presented in Figure 7. For specific relative high usage of triangle and square transformations
(Triangle and Square variables have been set to specific values, see Figure 7) the probability distribution in the
‘class’ variable heavily suggests that the most probable problem solving strategy followed by the student is the
C7, i.e. the one that involves usage of automatic measurement function (60% to 9.3% for the second most
probable strategy, C5).
Comparison with other methods
In order to validate the effectiveness of the network, that is the ability to correctly predict the problem solving
strategy originally expressed by the student, the ten fold cross validation method has been used. The
performance of the BBN was 84.07% (+- 6.75% for p<0.05) of correct estimations.
Subsequently, the experiment has been repeated using other popular machine learning techniques such as
Decision Tables, C4.5 and Naïve Bayes (Witten and Frank, 2005), which have been included in the automated
learning tool constructed by Holmes et al. (1994). The goal was to compare the effectiveness of the BBN method
in comparison with other widely accepted techniques. The results are summarized in Table 2. As shown in Table
2 the proposed method using BBN seems to perform better with respect to the other techniques. The only
disadvantage compared to Naïve Bayes method is that the latter is easier to implement and apply.
Figure 7. An example of use of the obtained BBN. For given values of variables ‘Rectangle’ and ‘Triangle’ the
probability the used followed the strategy C7 is very high (above right)
Table 2. Performance comparison of categorization techniques
Bayesian networks
Naive Bayes
Decision Tables
% success
As one can deduct from the above table, Bayesian networks outperform all other methodologies, both of similar
nature (such as the statistical-based Naïve Bayes which does not take into account any dependencies between the
variables) and the decision-tree based (such as C4.5 and Decision Tables). Regarding the former case, Bayesian
networks alleviate the over-restrictive assumption on the conditional independency on the attributes of the naïve
Bayesian classifier, which is also often unrealistic. Bayesian networks, can reason under conditions of
uncertainty, taking the semantic interrelation of attributes into account. Concerning the decision-tree based
algorithms, we could argue that information gain, a metric that is the core of the algorithms can provide a clear
view on the value of each attribute but can also lead to large trees, in which the task of pruning some redundant
nodes is difficult and may cause the stall of the algorithm in local maxima. On the other hand, Bayesian
networks are based on the probability of a structure over the given training set, which may be an NP-hard
problem but using intelligent search strategies can overcome the problem of local maxima.
This paper presents the application of Bayesian Networks techniques for tackling two important problems in the
frame of two open problem solving environments (MC and CARME), with very promising results. In the first
case, we discussed the design and implementation of the adaptive user support module AUSM. The performance
of the developed module during the evaluation experiment was very promising. The system, when fed with
interaction data of a new problem solving task was able to correctly predict 3 out of 4 cases for the next user
action and therefore to potentially provide the user with useful support in an efficient way. The proposed
approach serves as a scaffolding mechanism during modelling activities. The support system helps students
accomplish a complex task requiring specific actions, at critical points of their task. The proposed scaffolding
mechanism, realised with the use of Bayesian Networks, does not reduce the transparency of the system. Instead,
it provides the ability to the student to move ahead over potentially critical points in terms of tools operations. In
the scaffolding process adopted, the system provides help to the students on those elements of tasks that are
beyond their capacity, and allow them to concentrate upon the elements that are within their range of
competence. This is accomplished with identification of frequent interaction patterns using Bayesian Networks
and subsequent correlation with desired task accomplishment, thus enabling the system to infer the desired help
item. It provides help just when it is needed thus reducing the effort to search for relevant support, while
maintaining a focus on the modelling process without interruption and disorientation.
In the second case of automatic classification of problem solving strategies in an open problem solving
environment, the approach produced good results. The classification rate of the BBN algorithm is very high in
comparison with other popular algorithms, even with a small amount of data provided. However, the BBN could
be proven unsuccessful to recognize strategies with a low occurrence rate represented in the reference sample. A
future research goal is to estimate the lowest possible sample for each category related to the total, which leads
to unbiased and solid results. Various techniques have been proposed in the literature for tackling this problem
(e.g. Daskalaki et al. 2006), These could be particularly useful in our case, since often rare problem solving
strategies are of particular interest and their diagnosis is required by the tutors.
Another important advantage of the Bayesian Network approach is the derived graph which offers an easy to
understand representation of the variables showing the nature of the interaction. Study of the graph could lead to
usability improvements, as shown in the first study, by providing the most suitable help items or the possible
actions related to the interaction dialogue flow and status. From an educational perspective, automated problem
solving strategy recognition in open problem solving environments could substantially improve the learning
process and support the tutors, when dealing with large sets of student data. That is, the educator could rapidly
verify the adoption of specific problem solving strategies. In addition to this, real time implementation of the
proposed approach, in the case of distance learning systems could aid diagnosing and evaluating the learning
outcome and support teacher intervention in the form of adequate feedback.
In general, it is argued that a more efficient and adaptive user system increases indirectly the pedagogical value
of the environment because it contributes to the transparency of the tool A more intuitive flow of interaction
enhances the learnability of complex tools such as those included in open problem solving environments. Our
approach could be expanded to adapt the behaviour of the open educational environment in various aspects such
as goal recognition, adaptive assessment regarding the expertise of the user, and adaptation of the interface (for
example, adaptation of right click pull down menus, together with a constant set of commands). Further research
is needed in order to investigate these areas, together with a longitudinal evaluation of the effects of the
developed module on the way it affects student learning within such an environment.
Avouris, N., Tselios, N. & Tatakis E. C. (2000). Development and Evaluation of a Computer-based Laboratory
Teaching Tool, Journal Computer Applications in Engineering Education, 9 (1), 8-19.
Bunt, A. & Conati, C. (2003). Probabilistic Student Modelling to Improve Exploratory Behaviour. Journal of
User Modeling and User-Adapted Interaction, 13 (3), 269-309.
Collins, J., Greer, J. & Huang, S. (1996). Adaptive Assessment Using Granularity Hierarchies and Bayesian
Nets. In Proceedings of Intelligent Tutoring Systems, 569-577.
Conati, C., Gertner, A. S., VanLehn, K. & Druzdel, M. (1997). On-line Student Modeling for Coached Problem
Solving Using Bayesian Networks. In Proceedings of the 7th International Conference on User Modeling, 231242.
Conati, C., Gertner, A. & VanLehn, K. (2002). Using Bayesian Networks to Manage Uncertainty in Student
Modeling. Journal of User Modeling and User-Adapted Interaction, 12 (4), 371-417.
Cooper, J. & Herskovits, E. (1992). A Bayesian method for the induction of probabilistic networks from data.
Machine Learning, 9 (4), 309-347.
Daskalaki, S., Kopanas, I. & Avouris, N. (2006). Evaluation of Classifiers for an Uneven Class Distribution
Problem, Applied Artificial Intelligence, 20, 1-37.
Dimitracopoulou, A. & Komis, V. (2005). Design principles for the support of modelling and collaboration in a
technology-based learning environment. International Journal of Continuing Engineering Education and
Lifelong Learning, 15 (1/2), 30-55.
Ergazaki, M., Komis, V. & Zogza, V. (2005). High-school Students’ Reasoning while Constructing Plant
Growth Models in a Computer-Supported Educational Environment, International Journal of Science Education,
27 (2), 909-933.
Glymour, C. & Cooper, G. (eds.), (1999). Computation, Causation & Discovery. AAAI Press/The MIT Press.
Gudzial, M. J. (1993). Deriving Software Usage Patterns from Log Files. Georgia Institute of Technology. GVU
Center Technical Report. 93-41.
Heckerman, D. (1996). A Tutorial on Learning Bayesian Networks. Technical Report MSR-TR-95-06, Microsoft
Hellman, R. (1989). User Support: Revealing Structure Instead of Surface, Behaviour and Information
Technology, 8, (6), 417-435.
Holmes, G., Donkin, A. & Witten, I. H. (1994). Weka: a machine learning workbench. In Proceedings of the 2nd
Australian and New Zealand Conference on Intelligent Information Systems, Brisbane, Australia, 357-361.
Jameson, A. (1995). Numerical uncertainty management in User and Student Modeling: An Overview of
Systems and Issues. In User Modeling and User-Adapted Interaction, 5 (3/4), 193-251.
Jeffreys, H. (1939). Theory of Probability. Clarendon Press, Oxford.
Kinshuk, P. A. & Russell, D. (2002). Intelligent and Adaptive Systems. In Adelsberger, H., Collins, B &
Pawlowski, M. J. (Eds.) Handbook on Information Technologies for Education and Training, Germany:
Springler-Verlag, 79-92.
Komis, V., Dimitracopoulou, A., Politis, P. & Avouris, N., (2001). Expérimentations exploratoires sur
l’utilisation d’un environnement informatique de modélisation par petits groupes d’élèves, Sciences et
Techniques Educatives, 8, (1-2), 75-86.
Komis, V., Avouris, N. & Fidas, C. (2002). Computer Supported collaborative concept mapping: Study of
Interaction, Education and Information Technologies, 7 (2), 169-188.
Kordaki, M. & Potari, D. (1998). A learning environment for the conservation of area and its measurement: a
computer microworld. Computers and Education, 31, 405-422.
Luger, G. F. & Stubblefield, W. A. (1998). Artificial Inteligence – Structures and Strategies for Complex
Problem Solving, (3rd Ed), Addison Wesley.
Mayo, M. & Mitrovic, A. (2000). Optimizing ITS Behaviour with Bayesian Networks and Decision Theory. In
International Journal of Artificial Intelligence in Education, 12, 124-153.
Mitchell, T. (1997). Machine Learning, McGraw-Hill, New York.
Nathan, M. J. (1998). Knowledge and Situational Feedback in a Learning Environment for Algebra Sotry
Problem Solving. Interactive Learning Environments, 5 (1), 135-159.
Niedermayer, D. (1998). An Introduction to Bayesian Networks and Their Contemporary Applications,
University of Saskatchewan, Technical Report 184-3-54440 1998.
Norman, D. A. (1986). Cognitive Engineering in User-Centred System Design, Laurence Erlbaum Associates,
Hillsdale NJ, 1986.
Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Mateo,
Squires, D. & Preece, J. (1999). Predicting Quality in Educational Software: Evaluating for Learning, Usability
and the Synergy between them, Interacting with Computers, 11, 1999, 467-483.
Stephenson, T. A. (2000). An Introduction To Bayesian Network Theory And Usage , Institut Dalle Molle
d'Intelligence Artificielle Perceptive, Technical Report IDIAP-RR 00-03, 2000.
Tselios, N., Avouris, N. & Kordaki, M. (2002). Student Task Modeling in design and evaluation of open
problem-solving environments, Journal of Education and Information Technologies, 7 (1), 19-42.
Suthers, D., Connely, J., Lesgold, A., Paolucci, M., Toth, E. E., Toth, J. & Weiner, A. (2001). Representational
and Advisory Guidance for Students Learning Scientific Inquiry. In Forbus, K. and Feltovich, P. (Eds.), Smart
Machines in Education. The MIT Press. 2001, ISBN 0-262-56141-7
Von Glasersfeld, E. (1987). E. Learning as a constructive activity. In Janvier, C. (Ed.) Problems of
representation in teaching and learning of mathematics. London: Lawrence Erlbaum associates, 3-18.
Vomlel, J. (2004). Bayesian networks in educational testing. International Journal of Uncertainty, Fuzziness and
Knowledge-Based Systems, 12, 83-100.
Xenos, M. (2004). Prediction and assessment of student behaviour in open and distance education in computers
using Bayesian networks. Computers & Education, 43 (4), 345-359.
Witten, I. & Frank, E.(2005). Data Mining: Practical Machine Learning Tools and Techniques, 2nd Ed. Morgan
Kaufmann Publishers, San Francisco, USA.
Woolf, B. P., Beck, J., Eliot, C. & Stern, M. (2001). Growth and Maturity of Intelligent Tutoring Systems. In
Forbus, K. & Feltovich, P. (Eds.), Smart Machines in Education. The MIT Press. ISBN 0-262-56141-7.
Goldberg, A. K., & Riemer, F. J. (2006). All Aboard - Destination Unknown: A Sociological Discussion of Online
Learning. Educational Technology & Society, 9 (4), 166-172.
All Aboard - Destination Unknown: A Sociological Discussion of Online
Andra K. Goldberg
Doctoral student, College of Education, Northern Arizona University, USA
Tel: 928 692 3015
[email protected]
Frances Julia Riemer
College of Education, Northern Arizona University, PO Box 5774, Flagstaff AZ 86011-5774 USA
Tel: 928 523 0352
[email protected]
This paper is an attempt to describe the emergence and growing popularity of online distance education
over the past 30 years through changing sociological lenses. Examining the re-casting of the electronic
classroom through the euphoria of techno-positivism, the power-embedded analysis of Critical Theory of
Technology (CTT), and the critique of postmodernism, the paper addresses the implications suggested by
each theoretical interpretation. Using the metaphor of a high-speed train, we encourage administrators,
instructors and technicians to stop and reflect on the destination, rather than simply marvel at the speeds at
which we are traveling and the engine that powers our ride.
Online classes, Distance education, Techno-utopianism, Critical theory of technology, Postmodernism
Imagine trying to board a train that is already moving along the tracks. You manage to get on, but the directions
to your seat are written in a foreign language. You recognize the letters, but the words seem unintelligible. The
conductor is faceless yet clearly frustrated with your inability to sit and interact with your unseen fellow riders.
Worse yet, you must get to your destination on time or you will not be able to disembark.
This scenario is neither a nightmare nor a horror movie plot; it is instead analogous to a first experience in an
online class. Rest assured though that you are not alone in your discomfort. Both students and instructors
struggle to negotiate online instruction, a still somewhat new mode of teaching and learning. Although assuming
a far more removed stance, social theorists have also disparaged similar technological changes in modes of
instruction. Their denunciations date back to Plato's condemnation of the privileging of writing over dialog in
teaching – ironically while conducting his criticism in written form (Plato, 1961). Analyzing online education in
terms of social theory is challenging because like Plato, theorists criticize technology but find themselves
dependent on the very same tools, e-mail and computerized word-processing programs for example, that they
criticize (Nobel, 1997).
Mindful of this inherent paradox between message and medium, in this article we attempt to examine online
education, the most recent entwining of technology and education. We begin with the birth of the Internet arguably January 1, 1983 when TCP/IP (Transmission Control Protocol/Internet Protocol) was established as the
standard protocol for Internet transmissions (Cutshall, 2003). Access to the Internet enabled computers of all
types to interact with one another, resulting in the rapid growth of the World Wide Web and the beginning of a
new type of distance education, online courses. Defined as “any formal educational process that occurs with the
teacher and the student separated by either time or distance” (Davey, 1999, pp. 44-45), distance learning in an
online format combines synchronous and asynchronous interaction. This combination is then packaged and
delivered to anyone with a computer, Internet access, and the often hidden prerequisite of basic technical skills.
Embraced as “the educational pedagogy of the future” (O’Malley, 1999, p. 3), distance learning, enhanced by
web based technology, has made great strides since first offered in the form of correspondence courses, and later
as cable television courses and via interactive television. By 2000-2001 for example, 56 percent of all
postsecondary institutions offered some version of online courses, up from 34 percent in 1997. This growth is
particularly significant in the public sector, where 90 percent of all public, two-year and 89 percent of all public,
four-year institutions offer courses from a distance (Wirt, Choy, Rooney, Provasnik, & Sen, 2004). Recent data
reveals that secondary students also take advantage of this growth in course offerings. In the 2002-2003 school
year, thirty-six percent of all public schools had students enrolled in distance education courses, with students in
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
fifty-nine percent of those schools enrolled in online courses. Of particular relevance to this paper, in forty-eight
percent of those districts, post secondary institutions delivered the online courses. Online learning has become a
run-away train, racing forward so quickly that it appears as a blur to the pedestrian standing by the tracks.
Our examination of this new technological context of learning and teaching is sociological in nature. In an
attempt to understand the high-speed bullet train of changes technology has brought to education and the
strategies employed for coping with these changes, we focus on the portrayal of online learning in both public
and academic discourse. Our discussion of these modes of thought pivot around three widely divergent
theoretical perspectives: techno-utopianism, an incontestable belief in technology as progress regardless of any
difficulties that may arise (Lears, 2000), Critical Theory of Technology (CTT), a critique that equates digitizing
data to the “legitimiz[ation] of knowledge that fits into that structure” (Spitz, 2000, p. 9), and postmodernism, an
emergent theoretical perspective that focuses on the coupling of immediacy and isolation in the contemporary
world (Neal, 1999). By tracing the discourse that defined the emergence, growth, and use of online learning over
the past 20 years, we hope to make visible the gap that Agalianos (1996, p. 2) noted when writing, “The social
processes and the political decisions involved in the production and ‘consumption’ of IT in education are largely
Hopeful Beginnings
From the beginning of education’s courtship with technology, what has come to be known as techno-utopianism
has flourished. Within this techno-utopian stance, technology in the classroom has long been proclaimed to
“offer the greatest potential to right what’s wrong with our public schools” (California Educational Task Force,
cited in Oppenheimer, 1997, p. 53). In schools and campuses across the US, computer-assisted learning has been
deemed capable of great accomplishments that include both the transformation of teaching and the improvement
of students' academic and vocational performance (DeCastell, Bryson, & Jenson (2003). Like the general public,
teachers have come to see computers as essential to improving student achievement. When Michigan’s governor
announced her intention to cut funding for a laptop project that provides computers to sixth grade students, for
example, a middle school teacher insisted, “If the funding ends for this program, in three to four years, Bear
Lake schools will take a 5-year leap backward in education” (Wendland, 2005, p. 12). Such strong convictions
reflect an unquestioning acceptance of and growing dependence on technology that is manifested by teachers in
a wide range of school settings.
At its most basic, techno-utopianism is based on the belief that technology will revolutionize society. Actively
promoted by Wired magazine since the periodical’s inception in 1993, techno-utopians view the Internet, the
most recent example of digital technology, as a colorblind mechanism that fosters participation and involvement,
while freeing individuals from governmental constraints. Techno-utopians promote online instruction as both an
alternative to traditional class attendance and a mechanism for maximizing service to students at the lowest cost.
District personnel cited the availability to offer “courses not otherwise available at the school, …meeting the
needs of specific groups of students and offering Advanced Placement or college-level classes” as reasons for
having distance education courses in the district (Setzer, & Lewis, 2005, p. 14). In a 1998 advertising campaign,
community colleges, in the form of the Community College Distance Learning Network, were able to target a
new potential student base, the large immigrant populations in Chicago and Los Angeles, because of the
development of more than 500 courses delivered via the Internet (Chronicle of Higher Education, 1998).
According to techno-utopian texts, students in online courses have been divided into seven discrete groups: 1)
corporate learners pursuing education to maintain or upgrade work-related skills, 2) professional enhancement
learners seeking to advance or shift careers, 3) degree-completion adult learners working to complete a degree at
a later stage of their lives, 4) college experience learners preparing for life, 5) pre-college learners doing
postsecondary work prior to the completion of high school, 6) remediation and test preparation learners
preparing for an examination or enrollment in another program, and 7) recreational learners enrolled in classes
for personal enjoyment (Oblinger, Barone, & Hawkins, 2001). Regardless of specific rationale and motivation,
these learners are unable to attend traditional classes either because work or family situations restrict their
participation or because they do not live in geographic proximity to a college campus. As a public relations
article on Blackboard’s website asserts: “When course content and activities are provided online, students no
longer need to worry about accessing course materials. Students can complete assignments during their most
productive times. Busy students can choose to download readings or take practice exams whenever it is more
convenient, in the evening after kids are put to bed or at 4am during a bout of insomnia” (Blackboard, 2000, p.
In addition to convenience, proponents of online education described this new virtual context as both more fair
and less biased than traditional classrooms. As the context for discussion in college classrooms shifted to chat
rooms and other modes of electronic networks, techno-utopians reported that "groups of students at different
schools, even in different countries, work together on collaborative projects, comparing the results of
environment studies or cross-cultural surveys and thereby learning not only the subject at hand but also other
skills in social relationships just the kind of learning that the early critics of teaching machines were afraid
computers would stifle" (Starr, 1996, p. 55).
Others highlighted the benefit of learners’ disembodiment in a virtual learning environment. “At once, you may
say, we have eradicated the pathology of the classroom: learners will no longer feel their very presence has
generated an inscription on their bodies by others. Fat, thin, shy, squeaky-voiced, slow, boisterous, late, sleepy,
hairy–the whole Seven Dwarfs roll call–will be irrelevant in the new virtual learning environment” (Beckett
1998, p. 6). Online distance learning has been described by Starr (1996, p. 56) as “race-neutral, location-neutral,
age-neutral, income-neutral, disability-neutral and would be gender-neutral except for the clue of first names.”
Starr continues, “Student participation in the discussion is greater than in any of our face-to-face classes. Some
kinds of personal warmth appear to be more freely exchanged in the absence of bodies.”
The hopeful optimism surrounding online education has been so pervasive that both pundits and academics alike
have predicted its role in the demise of the residential university. MIT mathematician Seymour Papert (1984 as
cited in Cuban, 1996) claimed that computers would make schools obsolete. In a similar vein, management
pundit Peter Drucker (Lenzner & Johnson, 1997, pp. 7 - 8) predicted that:
Thirty years from now the big university campuses will be relics. Universities won’t survive. It’s
as large a change as when we first got the printed book …totally uncontrollable expenditures,
without any visible improvement in either the content or the quality of education mean that the
system is rapidly becoming untenable. Higher education is in deep crisis.…The college won’t
survive as a residential institution.
Conflicts Emerge
But as the marriage of education and technology moved beyond the honeymoon stage, tensions began to emerge
in the relationship. By the late 1990s, suspicions surfaced that the techno-utopianism that had been embraced by
educators and the general public alike was in fact a result of “technopositivism,” a heavily marketed ideology
that perpetuates a naïve faith in the promises of technology (Robertson, 2003b, p. 282). Critics claimed that
teachers and schools were "easy targets" for assertive marketing campaigns that linked computer-assisted
learning with issues of equity and access (Robertson, 2003a, p. 414). While techno-utopians proclaimed the cost
effectiveness of electronic delivery, others saw universities struggling to upgrade instructional television
systems, choosing between competing and costly course management software, and maintaining state of the art
equipment in computer labs, libraries, and other points of access (Carmean & Haefner, 2003).
The reality of online distance education also brought with it concerns about students’ actual access to
technology, an increasingly apparent gap between requisite and actual skill levels, faculty members’ complaints
about increased workloads, and a growing sense of isolation on the part of both instructors and students. While
public discourse continued to portray the Internet as available to all, able to transcend class distinctions, and free
of bias, research findings began to surface critiquing the expense of Internet access. According to a study
conducted by Vanderbilt University’s Project 2000, access to the Internet was not randomly distributed, but
correlated instead to income, education, and race (Novak & Hoffman, 1998). Conducted in January 1997 with a
sample of nearly 6000 respondents, the study’s authors concluded that “household income explains home
computer ownership; increasing levels of income correspond to an increased likelihood of owning a home
computer, regardless of race” (Novak & Hoffman, 1998, p, 13). A report released the following year by the
Benton Foundation (Goslee, 1998) attributed this gap to a lack of infrastructure, rather than to choices made by
poor families. Exposing a pattern of telephone and cable companies’ prioritization of wealthier suburbs when
wiring advanced systems, the report faulted the companies for neglecting to upgrade services in poor, inner-city
neighborhoods. The report cited the mechanics of computer sales as problematic, claiming that a lack of
financing options (i.e. credit cards, and checking accounts) severely limits the availability of Internet accounts to
those living in poverty. In addition to infrastructure and hardware, access to online classes was also found to be
dependent on the ability to afford a telephone, subscriber costs, and user fees.
Instructors also found that students lacked the requisite technical skills to participate in an online class and did
not always develop these skills as they progressed through the semester (Belcheir & Cucek, 2002; O'Brien &
Renner, 2002; van Schaik, Barker, & Beckstrand, 2003). To quote Ronald Walker, professor of African
American Studies at the University of Maryland, “It isn’t enough simply to have a personal computer – people
need to integrate personal computers into their lives” (Floyd 1996, p. A20). Doubts began to surface concerning
who actually benefited from online classes, students who would otherwise not have access to college classes, or
already advantaged students for whom online learning added another layer of convenience. Distance education
that required specialized software and Internet access may have been a boon for harassed professionals.
However, it became increasingly apparent that for the unemployed student who needed these classes to get work,
the requirements seemed to be yet another stumbling block (Gladieux & Swail, 1999).
Blacker (1994, p. 17) labeled these tentative anti-technology sentiments a “critical theory of technology” (CTT).
Driven by Marxists and neo-Marxists, critical theory asserts that rather than being neutral, technology
advantages and profits the privileged (Feenberg, 1991; Marcuse, 1964). From a critical theory perspective, the
much lauded cost-effectiveness of online learning has failed to address the additional burdens on faculty
members, who struggle with the expanded time commitment required to convert a class to an online format and
to attend to students who demand the immediate attention of faculty members to solve their technology-related
problems. College administrators have addressed the increase in workload by providing faculty stipends and
stressing the need for faculty buy-in (Natale, 2002; Serwatka, 2002; Thor, 1999). But according to critical
theorists, administrators have given little to no consideration to the displacement of teacher in an online
environment that has “a preference for substituting ‘delivery’ for ‘teaching’” (McWilliam & Taylor, 1998, p.29).
Faculty members, in turn, have voiced skepticism about the de-peopling of electronic classrooms and the
rendering redundant of their pedagogical skills. But in a climate in which market forces have already threatened
their academic tenure, only a few have mounted any credible, cogent response to this pedagogical dislocation
(Noble, 1998).
Online Education in a Postmodern World
The move from classroom-based to online education has also been viewed through a postmodern lens and
positioned as an artifact of postmodern life. As Krishan Kumar (2001) observed: “The new society is now
defined, and named, by its novel methods of acquiring, processing and distributing information…We do live in a
world saturated with information and communication” (p. 97, 110). For post-modernists, wireless access has
changed every aspect of our interaction. E-mail, instant messaging, online chat, and teleconferencing allow
constant and immediate communication. At the same time, lives have also become more separate, and in turn,
more isolated and private. Online classes reflect this dichotomy; students can access class at all hours of the day
or night, yet they are increasingly isolated from their peers by an “electronic curtain” that constrains the
development of a community of learning (Neal, 1999, p. 43).
Discourse among students in online chat rooms and virtual conference centers mirrors this separateness. Neal
(1999) describes student's online postings as “sequential position statements,” that simply refine their initial
position rather than address those of their peers. According to Neal (1999), “It is too easy, in an electronic
environment for students to escape the confrontations, challenges, and learning opportunities that are present in
the classroom” (p. 43). While students may in fact choose this "self-imposed exile from communal conversation
and action” when they register for online classes (Borgmann 1993, p. 2), university administrators have not fully
grappled with the emotional distancing students and faculty experience while working within this disembodied
context. In an unyielding campus climate of techno-utopianism, two mechanisms have become mainstays:
counseling to steer students who voice a need for interaction away from online learning, and training to help both
faculty develop online courses and students improve their technical skills. The presupposition here is two-fold.
Students who favor distance learning “differ in their perceived learning needs from students who choose face to
face formats” (Roblyer, 1999, p. 166). And that with support, anything can be taught and anyone can learn
At the same time, students’ expectations of constant communication, instantaneous information, and immediate
feedback have become painfully obvious to faculty members who teach in today’s instant culture. Instructors
find themselves barraged by students who assume they are tethered to the computer, poised day and night to
respond to e-mail requests or comment on discussion postings. Unlike scheduled office hours, online
availability, 24 hours seven days a week has become the norm in our postmodern world. Yet within a postmodern frame, these changes are neither all negative nor all positive. While online classes have the potential to
transform the teacher student relationship to one of service on demand, technology has also afforded faculty
members increased flexibility. They no longer need to be physically present for office hours to answer student
inquiries. As post-modernists proclaim, the collapsing of time and space, coupled with the disembodied nature of
online instruction, has the potential to re-define the academy in ways we cannot yet imagine (Robertson, 2003b).
Drastically recast from an initial techno-utopianism to the power-centered analysis of CTT and the inherent
contradictions of postmodernism, the cyber classroom is a manifestation of the pervasive changes engendered by
the birth of the Internet. Managing and critiquing this pedagogical transformation is like painting a landscape
from the window of le Train á Grande Vitesse (TGV). Feenberg (1999, p. 29) best describes the challenge when
Fortunately, how we design our new technologies is still an open question; the answer will
decide which benefits and which limitations we end up with… If we can resist simplistic appeals
to managerial efficiency and focus our efforts on sustaining the dialogue that has always been at
the heart of the educational experience, then technology holds great promise; if not, then we face
a great threat.
Never immune to our own entrenchment in society, we recognize our techno-utopianism when we continue to
hope that web-based technology holds great promise for learning and teaching.
In conclusion, we reflect back to the train metaphor used in the title of this paper. When we doubt our own
abilities to paint a landscape that has become a blur outside the window of our high-speed train, we contemplate
the possibility of getting off at the next stop and re-evaluating the train itself. Unfortunately however, we fear
that “full speed ahead” is almost always the dictum of the day. We encourage administrators, technicians, and
faculty alike to make the occasional forced stop and reflect on where they are going and why. They need to ask
the questions: What is the purpose and who profits? We hope they disembark, and take time to consider the
destination, not simply marvel at the engine that is powering this high-speed train called Online Education.
Agalianos, A. (1996). Towards a sociology of education computing. Paper presented at the annual meeting of the
American Educational Research Association. New York, NY.
Beckett, D. (1998, April). Disembodied learning: How flexible delivery shoots higher education in the foot.
Electronic Journal of Sociology, 3 (3), 5-10.
Belcheir, M. J. & Cucek, M. (2002). Faculty perceptions of teaching distance education courses. Research
Report 2002-02. Boise, Idaho: Boise State University.
Blacker, D. (1994). Philosophy of technology and education: An invitation to inquiry. Philosophy of Education
Blackboard. (2000). Educational benefits of online learning: A Blackboard tip sheet, retrieved May 30, 2006
from http://www.spokane.wsu.edu/net/help/content/BBdocs/Online_Learning_Benefits.pdf.
Borgmann, A. (1993). Crossing the postmodern divide. Chicago: University of Chicago Press.
Carmean, C. & Haefner, J. (2003). Next-generation course management systems. EDUCAUSE Quarterly, 1, 1013.
Chronicle of Higher Education (1998). Community colleges tout new distance-learning network. Edupage
Newsletter. The American Educom Consortium. July 10, 1998.
Cuban, L. (1996). Teachers and machines: The classroom use of technology since 1920. New York: Teachers
College Press.
Cutshall, S. (2003, March). Century schools, designing smarter, sleeker high-tech facilities. Techniques, 78 (3),
19-21 and 60-61.
Davey, K. (1999, Winter). Distance learning demystified. National Forum, 79 (1), 44-46.
DeCastell, S., Bryson, M., & Jenson, J. (2003). Object lessons: Critical visions of educational technology,
retrieved 30 May 2006 from http://educ.ubc.ca/faculty/bryson/gentech/objectlessons.htm.
Feenberg, A. (1999). Whither educational technology? Peer Review, 1 (4), retrieved 30 May 2006 from
Feenberg, A. (1991). Critical theory of technology. New York and Oxford: Oxford University Press.
Floyd, B. (1996). Program in Afro-American studies explores the racial gap in access to technology. The
Chronicle of Higher Education, 43 (17), A19-A20.
Gladieux, L., & Swail, W. (1999). The virtual university & educational opportunity: Issues of equity and access
for the next generation. New York: The College Board Publications.
Goslee, S. (1998). Losing ground bit by bit: Low income communities in the Information age. Washington DC:
The Benton Foundation, retrieved May 30, 2006 from http://www.benton.org/publibrary/losingground/home.html.
Kumar, K. (2001). Post-industrial to modern society: New theories of the contemporary world. In Halsey, A. H.,
Lauder, H., Brown, P. & Wells, A. S. (Eds.) Education: Culture, economy, and society, New York: Oxford
University Press, 96-112.
Lears, J. (2000). Techno-Utopia? Tikkun, 15 (1), 39.
Lenzner, R., & Johnson, S. (1997, March 10). Seeing things as they really are. Forbes, retrieved May 30, 2006
from http://www.forbes.com/forbes/1997/0310/5905122a.html.
Marcuse, H. (1964). One-dimensional man. Boston: Beacon Press.
McWilliam, E., & Taylor, P. G. (1998, November). Teacher im/material: Challenging the new pedagogies of
instructional design. Educational Researcher, 27 (8), 29-34.
Natale, R. D. I. (2002). Ensuring quality from a distance. Community College Week, 14 (2), 4.
Neal, E. (1999, Winter). Distance education: Prospects and problems. National Forum – Phi Kappa Phi Journal,
79 (1), 40-43.
Noble, D. F. (1997). Digital diploma mills: The automation of higher education. First Monday, 3 (1), retrieved
May 30, 2006 from http://www.firstmonday.org/issues/issue3_1/noble/.
Novak, T., & Hoffman, D. (1998). Bridging the digital divide: The impact of race on computer access Internet
Oblinger, D. G., Barone, C. A., & Hawkins, B. L. (2001). Distributed education and its challenges: An overview.
Washington DC: American Council on Education Center for Policy Analysis.
O'Brien, B. S. & Renner, A. L. (2002, June). Online student retention: Can it be done? Paper presented at the
ED-MEDIA 2002 World Conference on Educational Multimedia, Hypermedia & Telecommunications. Denver,
O’Malley, J. (1999). Students perceptions of distance learning, online learning and the traditional classroom.
Online Journal of Distance Learning Administration. 2 (4), retrieved May 30, 2006 from
Oppenheimer, T. (1997, July). The computer delusion. Atlantic Monthly, 280 (1), 45-62.
Plato (1961). Collected dialogues. New York: Pantheon Books.
Robertson, H. J. (2003a, January). Recycled promises. Phi Delta Kappan, 84 (5), 414-415.
Robertson, H. J. (2003b, September/October). Toward a theory of negativity: Teacher education and information
and communications technology. Journal of Teacher Education, 54 (4), 280-296.
Roblyer, M. D. (1999). Is choice important in distance learning? A study of student motives for taking internetbased courses at the high school and community college levels. Journal of Research on Computing in Education,
32 (1), 157-172.
Serwatka, J. A. (2002). Improving student performance in distance learning courses. THE Journal, 29 (9), 46-51.
Setzer, J. C. & Lewis, L. (2005). Distance education courses for public elementary and secondary school
students: 2002-03 (NCES 2005-010). U.S. Department of Education. Washington DC: National Center for
Education Statistics.
Spitz, R. (2000). Is the Internet for everyone? Art in search of a better connected society. Paper presented at the
10th International Symposium on Electronic Art (ISEA). Rio de Janeiro, Brazil. August 12.
Starr. P. (1996, July-August). Computing our way to educational reform. The American Prospect, 27, 50-60.
Thor, L. M. (1999). Keeping up with the Joneses. Community College Week, 11 (22), 4.
van Schaik, P., Barker, P., & Beckstrand, S. (2003). A comparison of on-campus and online course delivery
methods in Southern Nevada. Innovations in Education and Teaching International, 40 (1), 5-15.
Wendland, M. (2005). School laptop project at risk. Detroit Free Press, retrieved April 18, 2005 from
Wirt, J., Choy, S. P., Rooney, P., Provasnik, S., Sen, A. & Tobin, R. (2004). The condition of education, 2004.
National Center for Education Statistics. Washington, DC: US Department of Education.
Hsu, P.-S., & Sharma, P. (2006). A Systemic Plan of Technology Integration. Educational Technology & Society, 9 (4), 173184.
A Systemic Plan of Technology Integration
Pi-Sui Hsu
Department of Educational Technology, Research and Assessment, Northern Illinois University, USA
[email protected]
Tel: +1 815 7536025
Priya Sharma
Instructional Systems, Pennsylvania State University, USA
[email protected]
Tel: +1 814 8654374
The purpose of this article is to suggest a research-based systemic plan for educational researchers,
practitioners, and policymakers involved in the change process to implement successful technology
integration in the context of teacher education. This article provides a background about reform efforts in
science education in the United States in recent years. This article also addresses technology tools that are
responsive to reform efforts in the elementary science education and indicates the challenges of modifying
elementary preservice teacher education to meet technology reform needs. The technology integration
change process takes time and efforts; not all change can be sustained. Thus, a systemic plan that considers
three major components, including people, process activities, and systems, to ensure successful technology
integration is suggested.
Technology integration, Science education, Preservice teacher education, Systemic inquiry
This article is to suggest a research-based systemic plan for educational researchers, practitioners, and
policymakers involved in the change process to implement successful technology integration in their contexts.
We begin by describing the background of reform in science education in the United States, especially the
changes that led to the emphasis on teaching science as inquiry. Next, a number of major types of technology
tools that have the potential to support reform efforts are addressed. However, these technology tools are still
rarely seen at school nowadays, which reflects a lack of technology integration in elementary science teacher
education programs that play an important role of shaping school practice. The challenges of preparing
elementary preservice teachers to teach science with technology are described and we end by proposing a
systemic plan for technology integration in elementary science teacher education programs.
Reform in science education
In recent years, the United States has called for reform on science education in response to students’ poor
performance in science tests and a general lack of interest in science. A number of professional scientific
societies and organizations have taken the initiative in reforming science education. The National Research
Council with the assistance of the National Academy of Sciences developed the National Science Education
Standards (National Research Council, 1996), which addressed standards for teaching science and science
education programs. The goals addressed standards for teaching science and science education programs. Based
on the standards, it was recommended that science teaching take an inquiry-based approach and be adapted to
meet the interests, abilities, and experiences of students. In this case, inquiry-based referred to “approaches and
strategies for teaching and learning that enable learners to master scientific concepts as a result of carrying out
scientific investigation” (National Research Council, 2001a, p. 187). Science teachers were encouraged to create
an environment to enhance science understanding by encouraging students to communicate ideas with a
community of learners. In addition, the standards specified the roles of colleges and universities in preparing
teachers for implementing curricula that were consistent with the content standards.
In 2000 and 2001, the National Research Council provided detailed guidelines for implementing inquiry-based
science and classroom assessment (National Research Council, 2000, 2001b). The council suggested that in
inquiry-based science, teachers serve as facilitators who set up a number of conditions for students to explore
and investigate problems. Assessment becomes an integral part of science teaching and learning rather than a
stand-alone activity.
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
The basis of reform in science education
With the increasing awareness of the importance of teaching quality, the National Academies established the
Committee on Science and Mathematics Teacher Preparation (CSMTP) to identify critical issues in existing
practices and policies for K-12 teacher preparation in science and mathematics in 1998. In 2001, Educating
Teachers of Science, Mathematics, and Technology presented detailed recommendations for improving science
teacher education from a systemic view based on extensive research and the committee’s careful examinations of
practical experiences.
From these reform documents, statements, and policies, four distinct characteristics can be identified as forming
the basis of reformed science education:
(1) Science teaching should employ an inquiry-based approach;
(2) Assessment should attempt to ensure the quality of teaching and enhance the effectiveness of learning;
(3) Science should be integrated with other disciplines and,
(4) Applications of technology tools into science teaching and learning should be increased.
The notion of science inquiry is the major focus of reform views of science teaching and learning (National
Research Council, 1996, 2000). In the National Science Education Standards (National Research Council, 1996),
two elements of scientific inquiry for science learners have been emphasized: abilities to do science inquiry and
understandings about scientific inquiry. Some researchers proposed that science specific technology tools,
including data collection tools, simulations and modeling tools, have the potential to facilitate learners’
development from simple to sophisticated forms of scientific inquiry in recent years (Windschitl, 2000; ZembalSaul, Munford, & Friedrichsen, 2002). However, the science specific technology applications in science teaching
and learning are rarely seen or appropriately used in classrooms currently, particularly in elementary level. Most
research reports are limited to discussing the integration of generic technology tools such as programming
languages, spreadsheet, word processor, graphic programs, and the internet in elementary science teacher
education programs (Cavanaugh, 2003; Linn, 2003; Skinner & Preece, 2003). For example, at Northern Florida
University, preservice teachers were trained to use the internet as an instructional tool in an elementary science
methods course. They learned search techniques and methods for evaluating web resources critically. They were
then required to combine web resources with science content to create instructional units (Cavanaugh, 2003). In
the UK, elementary teachers used a web site dedicated to elementary science to foster students’ understanding of
science concepts and the use of concept-mapping software to stimulate discussion and assess students’
understanding (Skinner & Preece, 2003). While these are important aspects of technology use and integration
into science teaching, a more specific focus on the use of science-specific technological tools is missing and only
a few teacher education programs have begun to integrate inquiry-based technology into science curriculum. For
example, the University of Michigan has developed a science education program that prepares new teachers who
understand physical science and are able to create meaningful learning experiences (Krajcik, Blumenfeld, Starr,
Palincsar, Coopola, & Soloway, 1993). The Science Education Group at Penn State (Dana & Zembal-Saul, 2001;
Zembal-Saul et al., 2002) also provided elementary preservice teachers with opportunities to teach science with
technology and modeled the use of technology in lesson plans during the semester. These programs connect
science with pedagogical courses, and provide an early teaching apprenticeship where students can develop
lessons, discuss issues, and exchange ideas about using technology tools to enhance teaching. The next section
proposes a number of technology tools that have the potential to support reform efforts.
Technology tools for reforming inquiry-based science
With emerging technological innovations, the use of technology tools in inquiry-based science learning has
gained growing attention. Three types of inquiry-empowering technologies include data collection tools,
simulations and modeling tools, and online collaborative tools (Windschitl, 2000). These three types of
technologies assist elementary students in engaging in scientific phenomenon, conducting investigations, and
communicating, and developing products (Zembal-Saul et al., 2002).
Data collection tools
Data collection tools encourage students’ active engagement with phenomena in a manner similar to scientists’
engagement with scientific phenomena. The National Science Education Standards suggest that “a variety of
technologies, such as hand tools, measuring instruments, and calculators, should be an integral component of
scientific investigation. The use of computers for collection, analysis, and display of data is also part of this
standard” (1996, p. 175). For example, in microcomputer-based laboratories (MBLs), students can observe
graphs being produced by a microcomputer while an experiment is simultaneously being conducted. Nakhlen
(1994) defined MBLs as “software which uses an electronic probe to collect information about a physical system
and converts that information to a graphical system in approximately real time” (p. 368).
In microcomputer-based laboratories, students are shown to develop science concepts, collect and analyze real
time data, and visualize data much more quickly (Royuk & Brooks, 2003; Russel, Lucas, & McRobbie, 2003;
Trumper & Gelbman, 2001). As mentioned earlier, the ability to graph and collect data is one of the integral
practices of scientists in the course of their investigations. The microcomputer-based laboratories enable students
to conduct difficult or previously impossible investigations (Bannasch, 2001). In addition, the microcomputerbased laboratories purport to strengthen students’ science graphing process and problem solving skills because of
the capability of simultaneously collecting and graphing data. One example is probes, a microcomputer-based
tool that can detect temperature, voltage, light intensity, sound, distance, dissolved oxygen and so on. Probes are
now available for various hand-held devices, such as the Palm, through ImagiWorks, and the PASPORT System,
through PASCO, thus increasing the possibility for dissemination and use in schools.
Simulations tools
Simulations combine video, pictures, computer graphics, text and interactivity to present students with
phenomena that otherwise would be inaccessible, too hazardous, too time-consuming, or too expensive to
observe. Using simulations, students can manipulate variables that would otherwise be too unethical, difficult, or
impossible to do. For instance, Genscope (Horwitz & Christie, 2000) allows middle school students to breed
various types of dragons to see the effect of selective breeding on observable characteristics, or phenotypes, in
the offspring. Genscope allows students to ask “What if” questions and to do explorations that would otherwise
not be feasible. Moreover, the linking between the visual characteristics of the offspring and the invisible world
of chromosomes and DNA provides for unprecedented learning outcomes. Simulations allow students to ask
“What if” questions, make predictions, and test out their ideas (Windschitl, 2000).
Online collaborative tools
Spitulnik, Stratford, Krajcik, & Soloway (1998) suggested that “teachers need to provide environments which
support students’ inquiry, collaborating, and communicating” (p. 6). Such environments can help students to
generate scientific understandings, which can be built up through email, threaded discussions, and chat rooms
and other online communication tools.
In recent years, several projects have made substantive efforts to create collaborative environments for students
to get access to second-hand data and participate in discussions with people in the U.S and from around the
world (Bombaugh, Sparrow, & Mal, 2003; Butler, &MacGregor, 2003; Howland & Becker, 2002). For example,
the Global Learning and Observations to Benefit the Environment (GLOBE) Program (www.globe.gov),
initiated by former Vice President Al Gore on Earth Day, April 22, 1994, is a hands-on international
environmental science education program that establishes a partnership between students, their teachers, and the
scientific research community. GLOBE provides opportunities for children to communicate and collaborate with
scientists and other GLOBE students around the world. Students can participate in threaded discussions about a
variety of topics and investigations with scientists. GLOBE also supports students in data collection and
analysis. After data collection, students report their data through the Internet and compare their data to archived
data collected in previous years. This project has been joined by 3,800 schools in the U.S. and has drawn schools
from 50 countries to participate.
In the preceding sections, this article discussed science education reform and the technology tools that have the
potential to realize the goals of science education reform. In the next section we discuss the current challenges of
preservice science teacher education in the area of technology integration.
Challenges of elementary preservice science teacher education
In this section, we discuss a number of problems in technology integration in teacher education programs that are
equally evident for elementary science teacher education programs. A number of national reports (International
Society for Technology in Education, 2002, 2005) have indicated that although teachers have more resources
through technology, some of them have not received sufficient training in the effective use of technology. Most
elementary preservice teachers in different content areas and disciplines know little about the effective use of
technology and are not confident in using technology when they teach (Nonis & O’Bannon, 2002; US Congress,
1995). They also lack an understanding of their role in implementing lessons with technology. Whetstone and
Carr-Chellman (2001) conducted a survey, which revealed that elementary preservice teachers did not appear to
see the importance of their roles in integrating technology in classrooms.
In order to address these issues, research has shown that teacher education programs need to engage in
substantive efforts to prepare preservice teachers for teaching with technology. Successful integration of
technology in teacher education programs should provide preservice teachers with opportunities to use
technology to advance content area learning (Bull, Willis, & Bell, 2000), and promote authentic learning
experiences (Brush, Igoe, Brinkerhoff, Glazewski, Ku, & Smith, 2000).
Specifically, colleges and universities should be aware of their ability to address elementary preservice teachers’
competence in teaching with technology. Teacher education programs should consider all aspects of teacher
education and have the ability to provide adequate opportunities for preservice teachers to apply what they have
learned. Hargrave and Hsu (2000) presented results of a survey of instructional technology courses in preservice
teacher education programs and concluded that more emphasis was placed on integrating instructional
technology into the curriculum than on using technology for teacher productivity or personal use. However, most
universities and colleges offered only one three-credit instructional technology course to prepare preservice
teachers to teach with technology. In their survey study of investigating important factors in predicting
preservice teachers’ professional uses of technology with K-12 students in the classroom, Dexter and Riedel
(2003) found that setting high expectations for designing and delivering instruction using technology was
effective in getting preservice teachers to use technology during clinical experiences. Yet, the results also
indicated that preservice teachers wanted to have more access to technology and support and feedback from
cooperating teachers and university faculty during field experiences.
Systemic plan
In light of challenges of elementary science education programs, this article suggests a research-based systemic
plan to sustain the technology integration change process. The previous sections identified the reasons for
elementary preservice teachers being ill-prepared to teach science with technology, including poor design of
college-level courses and lack of integration of instructional technology courses into field experience. A
systemic plan to address these shortcomings considers three major components: people, process activities, and
systems to ensure successful technology integration.
1. Recruit key persons to form a leadership team.
The first way to sustain technology integration is to form a leadership team by recruiting a minimal number of
key faculty members or school teachers who possess expertise in elementary science education and share
different experiences in technology integration. The faculty members fulfill certain types of functions, each
focusing on design, administration, and establishing liaisons. Specifically, one leader focuses on the reconceptualization of the curriculum changes, monitors the progress in the implementation stage, and revises the
curriculum either by informal feedback or conducting research. One leader acts like a facilitator to facilitate the
changes by empowering other teachers to be receptive to changes within school by providing professional
development. The other leader is responsible for outreach to the school community and securing necessary
resources in the form of funding and human capital. It is suggested that these three major responsibilities be
shared among the leadership team, consisting of faculty members, principals, school teachers, or school
An example of leadership team is illustrated in the case of a team of university faculty in a science education
program that sustained successful technology integration at the university level (Hsu, 2004). In this case, the key
leadership team consisted of three faculty members, who had certain types of experience and adopted specific
roles. The three faculty members fulfilled specific types of functions, each focusing on design, administration,
and establishing liaisons. For example, one faculty member adopted the role of lead researcher in the context of
technology integration because of her past research and teaching interest in technology integration in elementary
science education. She, thus, conducted research with graduate students within the context of technology
integration and based on the findings, she became responsible for refining course design and revising the
technology integration process. The second faculty member assumed the role of coordinator with the university
and the software companies because of the length of his tenure at the department and his knowledge of how and
where to negotiate and secure resources. He, thus, ensured that the faculty members and graduate assistants
participated in professional development workshops and obtained technological support from software
companies. He also negotiated with administration to secure classrooms to set up technology tools and resources.
The third faculty member, a former science teacher in a local school district, adopted the role of liaison between
the science education program and various local school districts. She became responsible for providing adequate
hardware, software and conceptual understanding of technology integration when these resources or support
were unavailable in schools for integration into lessons by elementary preservice science teachers. Although
these three leaders were responsible for different aspects, they held meetings and talked to each other to ensure
that they were on the same track. This division of roles and responsibilities, in the light of individual abilities and
interests, was a strong support for the technology integration process.
A study of 13 Chicago elementary schools in an urban setting indicated another form of a school leadership team
(Spillane, Diamond, Walker, Halverson, & Jita, 2001). The leadership team consisted of the principal, the
science coordinator, and the assistant to the coordinator. The principal appointed a particular classroom teacher
as a science coordinator. Collaborating with an assistant funded by the school, the science coordinator took
advantage of the annual science fair and worked with other science teachers across all grade levels to integrate
more inquiry-based science instruction and develop performance assessment rubrics to assess students’ projects.
The principal doubled the weekly science instruction period, secured financial resources, established connections
to external resources, including local universities, colleges, science institutions, and external science consultants,
to empower change in science education. The leadership team showed a division among the functions of design,
administration, and liaisons although some of the functions might overlap. Another example of a leadership team
was identified in one elementary school in North Carolina (Franklin, 1993; Nesbit, DiBiase, Miller, & Wallace,
The joint effort of the leadership team members is a powerful force in sustaining change because leadership team
members can deal with different aspects in the technology integration change process and thus contribute their
2. Encourage the formation of a learning community.
A learning community is essential in sharing ownership of the innovation and involving other members in a
variety of activities. The formation of a learning community is important for several reasons: to generate a sense
of shared ownership among program participants as well as to foster a lifelong learning experience for the
One suggestion to form a learning community is to involve stakeholders in conducting action research. In Hsu’s
(2004) study, the learning community consisted of a faculty member and four graduate students who were either
course instructors or were interested in technology integration in elementary science. They conducted research
within the context of technology integration in the elementary science methods course and field experience
classrooms to examine practice critically. In the process, they developed a conceptual model for guiding
technology integration in a science methods course. They also examined and evaluated the organization and
content of the course assignments such as web-based portfolios to better assist elementary preservice science
teachers in reflecting on their technology integration experience critically. Accordingly, the framework for
guiding the web-based portfolio assignment underwent several revisions based on the group’s research for a
period spanning six years from 1997 to 2003 (Hsu, 2004; Avraamidou & Zembal-Saul, 2002). The formation of
the learning community helped the participants have shared ownership of the technology integration process,
leading to a deeper commitment to the change process.
The second activity for a learning community is to form study groups. Study groups read articles or books
together, discuss the implications of ideas and engage in reflective dialogues, which lead to better instructional
practice. For example, in addition to conducting research, the faculty member and the graduate students met
weekly to discuss the readings assigned from the previous week (Hsu, 2004). During the meeting, they
exchanged their ideas about obstacles encountered by their elementary preservice science teachers when they
integrated technology into their field experience classrooms. Through these discussions, the study group
participants were able to gain insight into specific problems and solve them by consulting each other in a short
period of time. Reflective dialogues in the study group provided a basis for the changes of the design of
curriculum in the following semester. The formation of the learning community sends a message to the
participants that learning as a group can be much more powerful in sustaining the technology integration change
process than learning individually.
In the FIRST (Fund for the Improvement and Reform of Schools and Teaching) project initiated in North
Carolina, teacher leaders and the other teachers used various activities to form a learning community to introduce
inquiry-based science instruction (Nesbit et al., 2001). The teacher leaders co-taught lessons with other teachers
in the school, which encouraged exchange of expertise between the teacher leaders and other teachers. The
leaders also led other teachers to reflect on the challenges faced in the implementation of inquiry-based science
instruction and solved the problems together. Teachers who showed resistance were invited to regular meetings
to share their expertise; this involvement resulted in reduced resistance because of a growing understanding of
the process and positive feedback from the teacher leaders. The teacher leaders also designed workshops where
trainee teachers participated in role-playing to increase their confidence in delivering inquiry-based science
lessons. In this case, the formation of the learning community creates a group learning environment, resulting in
the reduced resistance from the stakeholders.
At Ganado Primary School in Ganado, Arizona, the principal and teacher leaders built their learning community
by involving teachers and staff in discussion about curriculum and instructional practices on a regular basis and
by making time for collegial sharing (Cosner & Peterson, 2003). Other ways of building a learning community
in schools include distributing weekly staff newsletters highlighting efforts of teachers who adopted innovative
instructional strategies and communicating the expectations and vision with the teachers and staff.
It is suggested that careful planning of these activities, including conducting action research, forming study
groups, engaging in reflective dialogues, modeling teaching, collaborating on solving problems, and involving in
regular discussions of instructional practices, bolsters the formation of a learning community in order to sustain
technology integration.
Process activities
A variety of process activities are needed to ensure that the technology integration change process is able to
sustain over time. Process activities include pilot-testing of technology tools and adopting web-based portfolios
to scaffold teacher development.
3. Pilot-testing the use of technology tools.
Prior to any large-scale technology integration, it is important to pilot test the use of technology tools with a
small group of end users and stakeholders within the authentic learning environment and collect as much
feedback as possible from the users. It is also critical to allow some time to reflect on the feedback and to avoid
rushing into the implementation of technology integration. Hsu’s (2004) research indicated that a key faculty
member in the change process piloted different technology projects with a small group of elementary preservice
teachers for two years before implementing the technology tools with the entire group of 180 elementary
preservice teachers.
Including stakeholders such as course instructors, students, and researchers (Hsu, 2004) in pilot testing can
provide very important information. In the specific context of technology integration in an elementary science
education program, for example, course instructors were able to describe their feelings about the technology
tools and their appropriate integration into class and curriculum. Course instructors also provided assessment
data that supported the utility of technology tools in enhancing learning; they could identify students’ technical
and conceptual problems and suggest ways to improve student learning. Students could describe whether they
learned better by using technology tools and whether the technology tools were easy to operate. Researchers
could assess the project holistically in terms of technology integration and curriculum conceptualization, and
suggest appropriate design refinements. Such testing provided empirical support regarding the efficacy of the
proposed instruction in the elementary science methods course. Although pilot-testing appears to be a common
procedure to help instructional designers make informed decision of the use of technology in organizations
(White & Branch, 2001), it became an essential task to bring about the sustenance of the technology integration
change process in this particular context.
Pilot-testing appears to be a critical process activity in the technology integration change process. Although
resistance might occur, it is imperative to take time to pilot-testing the technology tools with a small group of
end users within the authentic learning environments.
4. Using practice and reflective activities to scaffold teacher development.
One important method for supporting pre-service teachers learning of technology-based science is the integration
of reflection as part of the process. According to Lin, Hmelo, Kinzer, and Secules (1999), reflective thinking
involves “actively monitoring, evaluating, and modifying one’s thinking and comparing it to both the expert’s
models and peers” (p. 43). It is critical for preservice teachers to draw connection between the coursework,
practice in clinical experience, and national standards. In addition, they need to adapt their knowledge and skills,
cope the issues in their clinical experience classrooms and develop their repertoire of integrating technology to
enhance teaching and learning. All of these skills require highly complex reflective thinking process. Therefore,
it is essential to provide support and scaffolds to assist preservice teachers in developing reflective and critical
thinking about these tasks in teacher education programs. While many options can support reflection, one
popular form is a portfolio that integrates practice and reflection meaningfully.
The notion of portfolio was proposed by the work of Lee Shulman (1988) and the creation of the National Board
for Professional Teaching Standards in 1989. In teacher preparation programs, portfolio development has been
demonstrated as being useful for preservice teachers in a variety of ways. First, portfolio development is a
powerful tool to engage teachers in reflecting on their experiences, interrogating their practices, understanding
their effects on students, and shaping their practices (Lyon, 1998; Schon, 1983). Second, portfolio development
facilitates preservice teachers’ ability to connect theories and practices (Morris & Buckland, 2000).
The portfolio has been represented in different media over time. Because of photocopying costs and storage
problems, hypermedia portfolios gradually replaced paper-based portfolios. Web-based portfolio, as one type of
hypermedia portfolio, is defined as “a user’s hypertextually linked set of electronic texts that have been created
for and placed on the World Wide Web” (Watkins, 1996, p. 219). A number of studies have shown that
hypermedia portfolios can promote students’ deep understandings of ideas in the development process
(McKinney, 1998; Morris & Buckland, 2000). A similar conclusion was reached by Glasson and McKenzie
(1999) in examining the development of multimedia portfolios for enhancing learning and assessment in an
elementary science methods course. They concluded that “developing a hypermedia presentation enabled
prospective teachers to construct and develop their ideas about teaching and learning. The portfolio documented
the progress of preservice teachers as they developed curriculum and taught children at a local school and in the
classroom” (p. 337).
Land and Zembal-Saul (2002) investigated the influence of scaffolds in the Progress Portfolio, a generalized tool
for articulation and reflection, on preservice teachers’ construction of scientific arguments within the context of
an innovative science course that aimed at providing preservice teachers with experiences learning science using
inquiry empowering technologies. The results indicated that the computer-based scaffolding supported
articulation and reflection of evidence-based explanations. The preservice teachers showed increasing
sophistication in their explanations, and the prompts within the Progress Portfolio seemed to stimulate preservice
teachers to become more precise in their explanations, to offer justification, and to connect evidence with claims.
They also suggested further research on varied scaffolding methods incorporated into computer-supported
learning environments to facilitate reflective thinking.
In the specific case of Hsu’s (2004) study, web-based portfolios were used to scaffold elementary preservice
science teachers’ development in a science methods course. Two faculty leaders conducted a study about the
progress of these teachers in developing web-based portfolios that included evidence and evidence-based
justification to reflect on their experience in integrating technology into their science teaching within one
semester in an elementary science methods course. The findings indicated that web-based portfolio development
can support preservice teachers’ metacognition by making connections between evidence and justification
explicit, and allowing them to express themselves creatively and save changes over time. Most importantly, the
findings indicated that web-based portfolio development could engage students in meaningful reflection in
technology integration in science teaching since these preservice teachers appeared to select more convincing
evidence and provided stronger justification as the semester progressed.
A hierarchy of Systems
Identifying the function of different levels of the educational systems allows adoption of approaches that can
strengthen synergy among different levels. In addition, foreknowledge of these levels allows one to address
problems that may arise from these systems before and during the technology integration change process. In
general, one can focus on the university level, the local community level, state level, and the broader public.
5. Increase the involvement of the local community.
At the local community level, it is essential to involve local school district teachers and students through the
change process. Example activities include involving local school community in university-held events and
delegating liaisons to regularly coordinate activities involving local community and local school district
personnel. During the early stages of the technology integration change process at a northeastern university, Hsu
(2004) found that in the first few years, barriers to technology integration included a lack of technology
resources in school and lack of support from mentor teachers. Due to these deficiencies, elementary preservice
teachers were unable to implement lessons with technology in their field experience classrooms. However, in the
later stages of technology integration, Hsu (2004) indicated that inviting local elementary school students and
teachers to visit the university appeared to be a successful intervention in sustaining the technology integration
change process. During such visits, elementary preservice teachers practiced teaching lessons with technology
with a small group of students and thereby gained opportunities to experience the effect of technology
integration on science learning. These visits also provided in-service teachers with opportunities to observe how
inquiry-based science lessons were taught using technology.
White (2001) suggested the importance of reaching out to different interest groups in the community that may
contribute to the enhancement in the areas of science and technology. These groups include the private sector,
government, educational institutions, and the science and technology community. Taylor and Wochenske (2001)
identified a number of strategies for a long-term business-school partnership that has enhanced science education
reform efforts in the San Diego school districts for the past ten years. These strategies included holding annual
science fair, inviting local scientists, university faculty and graduate students to schools, and engaging in-service
science teachers in discussion with the science research community in after-school seminars.
Professional development schools (PDS) are another method of establishing close relationships between
university-based researchers and school-based educators. Grove, Strudler, and Odell (2004) suggested that it is
critical to establish school district-university partnerships to assist student teachers in integrating technology in
field experiences by implementing frequent professional development sessions. Mentor teachers helped novice
teachers to build knowledge about how to teach in reform-minded ways with technology and how to mentor
student teachers to teach in ways consistent with science reform standards. Mentor teachers were introduced to
new practices and research in integrating technology with curriculum-based, student-centered activities by
university faculty, which exposed them to new models for teaching and learning and learn to encourage novice
teachers to teach in similar ways through modeling, practicing, and analyzing teaching together. Collaboration
between school teachers and university faculty appears to be critical in sustaining the technology integration
change process.
6. Actively identifying funding opportunities and standards from the state department of education.
At the state level, the department of education plays a critical role in the technology integration change process
because it dictates curriculum design and funding of resources to sustain the changes. In the specific case of
Hsu’s (2004) investigation of change in elementary science education, state standards greatly influenced the
technology integration process. State academic standards in science informed the selection of topics for longterm projects every semester and the choice of technology tools. As an example, instructors designed units of
lessons on watershed management to respond to the new standards of environmental ecology from the state
department of education.
In light of the economic hardship of state departments of education, a number of methods to secure external
funding need to be identified. Cooper and Bull (1997) indicated that funding appeared to be an essential factor in
technology integration because technology software and hardware need to be updated every year. They
suggested that deans and other leaders, individually or collectively, should educate university presidents,
provosts, state boards of education or professional practices boards and state legislatures on the importance of
adequate technology resources. Bauer (2001) specified four major funding sources, including foundation
funding, corporation grants, individual giving, and federal and state grants. Among these resources, federal and
state grants place the greatest impact on technology resources in educational settings. Thus, it is critical to
recognize the importance of funding in technology integration, to identify where the funds can be granted, and to
identify means to obtain the funds to sustain technology integration.
7. Identify means to collaborate with and gain recognition from national professional organizations and software
At the national level, it is essential to identify means to gain recognition from accreditation agencies, to attend
national contests or reviews and meetings of professional organizations, and to disseminate experience by
authoring articles in journals, and to develop partnerships with educational software companies for appropriate
support. Hsu (2004) identified that positive attention from an accreditation agency was critical in sustaining
technology integrations. The faculty members and course instructors within the science education program
gained confidence and momentum in technology integration because of recognition by National Council for
Accreditation of Teacher Education (NCATE), an accreditation organization. Published journal articles and
conference papers drew the attention and interest of people working in similar areas of research, which
reinforced the accomplishments of faculty members’ and course instructors’ efforts in technology integration.
Thus, in educational settings, rewards can accrue in a number of ways and it is not necessary to use money as a
form of reward.
Software companies can provide technical assistance and buy-in. They can also offer professional development
workshops for the potential users and learners. Cooper and Bull (1997) suggested the establishment of
partnerships between the educational programs and vendors of educational software to ensure that students
receive appropriate exposure to software. They further suggested the National Council for Accreditation of
Teacher Education (NCATE) and other organizations that establish technology standards for teacher education
standards could take the initiative to implement such arrangements with educational vendors. In Hsu’s (2004)
study, software companies were very supportive of the science education program and provided additional tools
and professional training for the faculty members and course instructors. The companies also worked diligently
to develop compatible technologies and more elementary-friendly tools based on faculty members’ and course
instructors’ feedback. You (2001) provided a number of practical suggestions for developing school-business
partnerships that promote educational reform in science and technology. These suggestions include aiming for a
long-term relationship, getting top-level commitment, deciding on goals, looking for partnership opportunities
that fit in school’s situation, being flexible, building on little success and publicizing your efforts, and assigning
a staff person or outreach coordinator to develop partnerships.
As discussed in the previous sections, the technology integration change process can involve different levels of
systems. It is essential to identify the impacts that different systems might have and to exercise strategies to
reinforce strengths of and tackle possible resistance from these systems.
Although we suggest a research-based systemic plan for integrating technology in elementary science education,
a number of areas remain to be answered. First, a leadership team has proved as a powerful force in sustaining
technology integration in elementary science education. However, it is important to further study methods for
recruiting key persons with expertise in science education and keeping them involved in the technology
integration change process. It is also important to identify and clarify mechanisms that allow for sharing
responsibilities in a leadership team.
Second, a learning community can be very powerful in sustaining the technology integration change process in
elementary science education. However, we need to identify how to designate experts with vision to serve as
catalysts for forming and leading the learning community. Additionally, research and practice must explore the
design of a variety of significant, motivating, and relevant activities in which to engage stakeholders. Also, for
busy teachers, it is important to identify methods to motivate them to engage regularly in aforementioned
Third, the technology integration change process cannot be sustained if support from different systems is
insufficient. We identified a number of systems that have influence on technology integration. It is important for
the change agents to recognize the resources and inhibitors within the systems and accordingly assign different
weights while evaluating the success of the implementation plan in different contexts.
This article provided the background of reform in science education in the United States in recent years. This
article also addressed major types of technology tools that are responsive to reform efforts. In addition, this
article indicated the challenges of elementary preservice teacher education and justified why the technology
integration change process is worthy of further examination. Change takes time and efforts; not all change can be
sustained. Thus, a systemic perspective was provided for educational researchers, practitioners, and
policymakers involved in the change process to plan for successful technology integration in their contexts.
Avraamidou, L. & Zembal-Saul, C. (2002). Making the case for the use of web-based portfolios in support of
learning to teach. The Journal of Interactive Online Learning, 1 (2), 1-19.
Bannasch, S. (2001). Educational innovations in portable technologies. In Tinker, R. F. & Krajcik, J. S. (Eds.)
Portable Technologies: Science Learning in Context, New York: Plenum, 121-166.
Bauer, D. G. (2001). Finding the money. Principal Leadership, 1 (9), 24-28.
Brush, T., Igoe, A., Brinkerhoff, A., Glazewski, K., Ku, H. Y. & Smith, T. C. (2000). Lessons from the field:
Integrating technology into preservice teacher education. Journal of Computing in Teacher Education, 17 (4),
Bombaugh, R., Sparrow, E. & Mal, T. (2003). Using GLOBE plant phenology protocols to meet the “National
Science Education Standards.” American Biology Teachers, 65 (4), 279-285.
Bull, G., Willis, J., & Bell, L. (2000). A user’s guide to the CITE journal. Contemporary Issues in Technology
and Teacher Education, 1 (1), 4–8.
Butler, D. M. & MacGregor, I. D. (2003). GLOBE: Science and education. Journal of Geoscience Education, 51
(1), 9-20.
Cavanaugh, C. (2003). Information age teacher education: Educational collaboration to prepare teacher for
today’s students. Tech Trend, 47 (2), 24-28.
Cooper, J. M., & Bull, G. (1997). Technology and teacher education: Past practice and recommended directions.
Action in Teacher Education, 19, 97-106.
Cosner, S, & Peterson, K. (2003). Building a learning community. Leadership, 32 (5), 12-15.
Dana, T. M. & Zembal-Saul, C. (2001, March). Learning to teach with technology. Paper presented at the
Annual Meeting of the Society for Information Technologies in Teacher Education (SITE), Orlando, FL.
Dexter, S. & Riedel, E. (2003). Why improving preservice teacher educational technology preparation must go
beyond the college’s walls. Journal of Teacher Education, 54 (4), 334-346.
Franklin, M. E. (1993). Statewide Improvement in Elementary Mathematics and Science Education through Peer
Teacher Training (Final Report of Project R 168D00258-92), Chapel Hill, NC: Mathematics and Science
Education Network, University of North Carolina at Chapel Hill.
Glasson, G. E. & McKenzie, W. L. (1999). The development of a multi-media portfolio for enhancing learning
and assessment in a K-8 science method class. Journal of Science Teacher Education, 10 (4), 335-344.
Grove, K., Strudler, N. & Odell, S. (2004). Mentoring toward technology use: Cooperating teacher practice in
supporting student teachers. Journal of Research on Technology in Education, 37 (1), 85-109.
Hargrave, C. P. & Hsu, Y. (2000). Survey of instructional technology courses for preservice teachers. Journal of
Technology and Teacher Education, 8 (4), 303-314.
Horwitz, P. & Christie, M. (2000). Computer-based manipulation for teaching scientific reasoning: An example.
In Jacobson, M. & Kozma, R. B.(Eds.) Innovations in Science and Mathematics Education: Advanced Designs
for Technologies and Learning, New York: Lawrence Erlbaum Associates.
Howland, D. & Becker, M. L. (2002). GLOBE- The science behind launching an international environmental
education program. Journal of Science Education and Technology, 11 (3), 199-210.
Hsu, P. (2004). A Case Study of the Change Process of Integrating Technology into An Elementary Science
Methods Course from 1997 to 2003, Unpublished doctoral dissertation, University Park, PA: The Pennsylvania
State University.
International Society for Technology in Education. (2002). NETS for teachers – Preparing teachers to use
technology, Eugene, OR: ISTE.
International Society for Technology in Education. (2005). The new national education technology plan,
Eugene, OR: ISTE.
Krajcik, J., Blumenfeld, P., Starr, M. L., Palincsar, A., Coppola, B. & Soloway, E. (1993). Integrating
knowledge bases: An upper-elementary teacher preparation program emphasizing the teaching of science. In
Rubba, P., Campbell, L. & Dana, T. (Eds.) Excellencing Educating Teachers of Science, Aubum, AL:
Association for the Education of Teachers in Science.
Land, S. M. & Zembal-Saul, C. (2003). Scaffolding reflection and articulation of scientific explanations in a
data-rich, project-based learning environment: An investigation of progress portfolio. Educational Technology
Research and Development, 51 (4), 65-84.
Lin, X., Hmelo, C., Kinzer, C. K., & Secules, T. J. (1999). Designing technology to support reflection.
Educational Technology Research and Development, 47 (3), 43-62.
Linn, M. C. (2003). Technology and science education: starting points, research programs, and trends.
International Journal of Science Education, 25 (6), 727-758.
Lyon, N. (1998). Portfolios and their consequences: developing as a reflective practitioner. In Lyons, N. (Ed.)
With Portfolio in Hand: Validating the New Teacher Professionalism, New York: Teachers College Press, 247264.
McKinney, M. (1998). Preservice teachers’ electronic portfolios: Integrating technology, self-assessment, and
reflection. Teacher Education Quarterly, 85-103.
Morris, J. & Buckland, H. (2000). Electronic portfolios for learning and assessment. Paper presented at the
Society for Information Technology and Teacher Education conference, San Diego, CA.
Nakhlen, M. B. (1994). A review of microcomputer-based labs: How have they affected science learning?
Journal of Computers in Mathematics and Science Teaching, 13 (4), 368-381.
National Research Council. (1996). National Science Education Standards, Washington, DC: National Academy
National Research Council. (2000). Inquiry and the National Science Education Standards, Washington, DC:
National Academy Press.
National Research Council. (2001a). Educating Teachers of Science, Mathematics, and Technology: New
Practices for the New Millennium, Washington, DC: National Academy Press.
National Research Council. (2001b). Classroom Assessment and the National Science Education Standards,
Washington, DC: National Academy Press.
Nesbit, C. R., DiBiase, W. J., Miller, A. S. & Wallace, J. D. (2001). In their own words: what science and
mathematics teacher leaders say are important aspects of professional development, Chapel Hill, NC:
Mathematics and Science Education Network, University of North Carolina at Chapel Hill.
Nonis, A. S., & O’Bannon, B. (2002). A field-based initiative for integrating technology in the content areas:
Using a team approach to preparing preservice teachers to use technology. Paper presented at the Society for
Information Technology and Teacher Education, Nashville, TN.
Royuk, B. & Brooks, D. W. (2003). Cookbook procedures in MBL physics exercises. Journal of Science
Education and Technology, 12 (3), 317-324.
Russel, D. W., Lucas, K. B. & McRobbie, C. (2003). The role of the microcomputer-based laboratory display in
supporting the construction of new understandings in kinematics. Research in Science Education, 33 (2), 217243.
Schon, D. A. (1983). The Reflective Practitioner, New York: Basic Books.
Shulman, L. S. (1988). A union of insufficiencies: strategies for teacher assessment in a period of education
reform. Educational Leadership, 46, 36-41.
Skinner, N. C. & Preece, P. F. (2003). The use of information and communications technology to support the
teaching of science in primary schools. International Journal of Science Education, 25 (2), 205-219.
Spillane, J. P. Diamond, J. B., Walker, L. J., Halverson, R. & Jita, L. (2001). Urban school leadership for
elementary science instruction: Identifying and activating resources in an undervalued school subject. Journal of
Research in Science Teaching, 38 (8), 918-940.
Spitulnik, M. W., Stratford, S. J., Krajcik, J. & Soloway, E. (1998). In Fraser, B. J., & Tobin, K. G. (Eds.)
International Handbook of Science Education, Great Britain: Kluwer, 363-381.
Taylor, N. & Wochenske, J. (2001). The San Diego Science Alliance. In Thorson, A. (Ed.), Partnerships with
Business and the Community, Columbus, OH: Eisenhower National Clearing House for Mathematics and
Science Education, 37-38.
Trumper, R. & Gelbman, M. (2001). A microcomputer-based contribution to scientific and technological
literacy. Journal of Science Education and Technology, 10 (3), 213-221.
US Congress. Office of Technology Assessment. (1995). Teachers and Technology: Making the Connections
(GPO # 052-003-01409-2), Government Printing Office, Washington, DC.
Watkins, S. (1996). World Wide Web authoring in the portfolio-assessed, (inter)networked composition course.
Computers and Composition, 13 (2), 219-230.
Whetstone, L. & Carr-Chellman, A. A. (2001). Preparing preservice teachers to use technology: Survey results.
Tech Trends, 45 (4), 11-17.
White, P. J. (2001). Finding Prospective Partners. In Thorson, A. (Ed.) Partnerships with Business and the
Community, Columbus: OH: Eisenhower National Clearing House for Mathematics and Science Education, 2729.
White, B. S. & Branch, R. M. (2001). Systematic pilot testing as a step in the instructional design process of
corporate training and development. Performance Improvement Quarterly, 14 (3), 75-94.
Windschitl, M. (2000). Supporting the development of science inquiry skills with special classes of software.
Educational Technology, Research and Development, 48 (2), 81-95.
You, A. (2001). Guidelines for effective partnerships. In Thorson, A.(Ed.) Partnerships with Business and the
Community, Columbus, OH: Eisenhower National Clearing House for Mathematics and Science Education, 2123.
Zembal-Saul, C., Munford, D. & Friedrichsen, P. (2002). Technology tools for supporting scientific inquiry: A
preservice science education course. Paper presented at the Annual Meeting of the Association for the Education
of Teachers of Science, Charlotte, NC.
Cronjé, J., Adendorff, D. E., Meyer, S. M., & van Ryneveld, L. (2006). Surviving the shipwreck: what makes online students
stay online and learn? Educational Technology & Society, 9 (4), 185-193.
Surviving the shipwreck: what makes online students stay online and learn?
Johannes C. Cronjé
Faculty of Education, Groenkloof Campus, Universtity of Pretoria, Leyds Street, Groenkloof, South Africa
[email protected]
Tel: +27 82 558 5311
Fax: +27 12 343 5065
Debbie E. Adendorff
Department of Curriculum Studies, Groenkloof Campus, University of Pretoria, 0002 South Africa
[email protected]
Tel: +27 825575295
Salome M. Meyer
Faculty of Medicine, University of Pretoria, 0002 Pretoria, South Africa
[email protected]
Tel: +27 828283380
Linda van Ryneveld
Deputy Director, Telematic Learning, Tshwane, University of Technology, Pretoria, 0001 South Africa
[email protected]
Tel: +27 828096379
Although much is written about reasons why students drop out of online courses, little is said about what
makes them stay. This article reports on an experiment whereby online students were exposed to a learning
experience modelled on the US television series Survivor. Twenty-four students were put into tribes and
allowed to vote one another off the island at the end of each week. Students who were voted out of their
tribes, were still on the course, but could no longer rely on the support of their peers. The course had a very
high dropout rate, and students reported that the experience was extremely stressful. Yet there were fifteen
students who completed the whole course. The question is why? This article identifies and discusses three
aspects that contributed to the success of those who completed: The game metaphor, the roles and
competencies of the facilitator, and the affective dimensions of peer support in a non-contact environment
Online learning, Peer support, Games, Metaphor, Facilitator
In his examination report of the first PhD that came out of this project Tom Reeves called it “a shipwreck of an
online course”. Although it was an apt pun on the Survivor metaphor that we used, it was disconcerting to think
of our project as a shipwreck. Yet, in reflection, we concluded that in a constructivist paradigm one has to learn
from one’s mistakes; and in order to do so, one has to make them in the first place.
In this article we reflect upon three PhD studies that investigated the Surfiver experiment. Twenty-four Masters’
students enrolled for a six-week module on Internet-based distance education, which was to be presented almost
entirely over the Internet. Instead of following a traditional approach whereby students would read and discuss
documents and then finally write a term paper or an online test, the course was presented following the metaphor
of a reality game show. Only fifteen students survived. The idea was to investigate the implications of using
role-play and other games with adult online students. Games and case studies are not unknown in contact
teaching of adults, particularly as icebreakers at training sessions, but there is little evidence of games being used
in formal education, and no evidence of games being used in online education of adults.
The question asked, was: “What are the conative implications of using games and a game-show metaphor to
motivate adult students to complete an online learning course?” (Conative means: “to do with staying power”).
We were specifically interested in what makes them stay, and not in what makes them drop out, as much
research has already been done on barriers to online learning (Galusha, 1997). This phase of an ongoing
development research project built on previous studies in the use of co-operative learning to improve online
interaction (Cronje, 1997; Cronje & Clarke, 1999), as well as an investigation into the use of metaphor in
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
maintaining the interest levels of online adult students (Cronje, 2001) and a small pilot study on affective aspects
of online learning (Rauscher and Cronje, 2005). The main contribution of this article to the project is its enquiry
on the implications of the game metaphor. Three classes of implications were identified, and each class formed
the basis of an individual doctoral enquiry into the same case – the Surfiver online learning game. The first
enquiry was thesis topic of the online facilitator, who investigated the implications of playing games with adult
students in an online learning community (Van Ryneveld, 2005:85). Adendorff (2004) investigated the roles and
competencies of an online facilitator, and the third study by Meyer (2005) considered how students coped with
the lack of physical contact in an online learning environment.
Course description
The learning event that forms the basis of this article is six weeks of coursework in partial fulfilment of the twoyear Master’s Degree program in Computer-Integrated Education presented by the University of Pretoria. It is an
optional course presented in the second year, but not as the final course, so that learners could enrol in another
course should they deselect this one. By its nature as a distance course, it tends to have a high dropout rate, and
students who drop out should have the opportunity to select another module before the end of the course. The
course has been presented since 1995 and, in the spirit of development research, each iteration is enriched by
what has been learnt from the previous one, while new facets are often introduced by way of an experiment. The
complete curriculum can be found at http://hagar.up.ac.za/catts/ole/oro/outcomes.html.
The students were highly motivated individuals who had already successfully completed more than two-thirds of
the course. The students ranged from 23 to 55 years of age. There were eight male and sixteen female students.
The course was presented in English; however, there were Afrikaans, Sotho, Xhosa, Zulu as well as English
first-language speakers on the course. Within the first two weeks, nine students withdrew from the course while
the remaining fifteen all completed successfully.
Figure 1: The Surfiver island early in the course
The course was supposed to be presented entirely online. No physical contact was allowed between students,
except for an initial briefing session and a final de-briefing session. It was necessary, however, to have an
emergency contact tribal council meeting early in the course to get floundering students on track.
At the beginning of the course a Web-based island was created and students were required to erect their own
shelters on the island, by way of modifying a graphic file in their home directories. Figure 1 shows the end result
of their collaborative effort. Respecting the copyright of the television game, our game was called CyberSurfiver;
a play on the original game show’s name and the fact that the participants surf the Internet. Five groups were
formed; four competing tribes and one for those voted off the “island”.
At the introductory meeting, the 24 self-selected participants were grouped into four groups according to their
self-perceived Internet literacy. The rules of the game were explained to them while they sat in their groups to
exchange email addresses, decide on a name for their group and to formulate a game plan. To stay true to the
Survivor game format, they were given games to play. These were in the form of individual reward challenges
and group immunity challenges. At the end of each week tribes had to vote one member off the team. The
members who were voted off were not dropped from the course, but were put in a separate tribe where they
would continue working, but they would no longer be eligible for the great prize, which in this case was not a
million dollars, but a weekend holiday for one family. The winner would be the last one remaining.
The tasks given to students included finding free online typing tutors, completing the tasks and sending in a
screenshot of their results. Other tasks included building online jigsaw puzzles as well as uploading sound files
containing their thoughts at a particular stage of the module. The final exam was a reflective essay on a topic
relating to Internet-based learning in an online community. Topics included motivation, affective considerations,
conflict management, etc.
The academic importance of Cybersurfiver
CyberSurfiver was an unusual experiment in Internet-based learning in higher education. Firstly, it is conducted
almost entirely online and secondly, it uses a game as a means to achieve the course’s objectives. This course
therefore had a wealth of information waiting to be extracted, analysed and, ultimately, published in order to
positively contribute to the academic fraternity.
The focus of this paper is to present the studies of three doctoral students in a manner which allows the reader to
obtain a holistic understanding of online learning in general and, in this specific case, particular focus is given to
the role that games play in adult online education. Firstly, the game metaphor is discussed. Thereafter, the roles
and competencies of an online facilitator are investigated. This is followed by a discussion on affective
considerations, as well as peer support, with regards to online learning. As this article covers the findings of
three doctoral theses it will necessarily be rather superficial. The individual theses are available online and are
listed in the references.
Research methods
The artefacts produced by the students, online discussions, individual emails to the facilitator and the final essays
formed the main data sources of the investigations for the doctoral students. The transcripts of the primary
briefing, the emergency tribal council meeting and the final council meeting also contributed to the research the
doctoral students conducted. In addition, an independent professional facilitator conducted two focus group
discussions and provided transcripts and analyses for triangulation. Two separate interview schedules (Creswell,
1998:124) were designed. An independent interviewer and independent moderator were used to conduct semistructured interviewing to avoid any bias during the interview sessions. The transcripts of the interviews were
used to corroborate evidence gathered during the focus groups. Verification methods for this study included:
Member checking, peer reviews and the crystallisation of various points of view. The three doctoral students
analysed the same basic data using different techniques to arrive at answers concerning the game metaphor,
facilitator roles and competencies, and affective considerations and peer support. Each doctoral student also
conducted an independent literature survey.
The game metaphor
Introducing games in adult learning can focus the adult’s attention. It can also garner positive attitudes,
contribute to the adult’s motivation and improve their concentration, all while they actually enjoy the activity
(Krasnor & Pepler 1980; Malone & Lepper 1987; Cheng & Van de Ven 1998; Cordova 1993; Garris Ahlers &
Driscell, 2002).
Focus group interviews with our students, as well as comments in their individual essays, showed that they were
motivated by the immunity challenges and reward challenges. An analysis of the artefacts produced by students
showed that even those who said that they did not like playing games still produced work of exceptional quality.
The interviews revealed that they were more motivated by the small, intangible rewards (in the form of digital
photos of prizes) that they received for the reward challenges than by the idea of being the overall winner of the
weekend away.
On the negative side, students felt that playing games was not central to their learning need. They perceived a
mismatch between attaining the course objectives and reaching the game objectives; although in most cases these
objectives were deliberately aligned with each other. A course objective, for instance, could be for students to
improve their keyboard skills and the game objective was for students to see who types the fastest. The winner of
this particular challenge, however, later admitted that she had asked her daughter to type for her, and sent in
those results.
Like Burke (2002), we also found that social interaction played a central role in stimulating students to
participate in and, ultimately, complete the course. They used Yahoo messenger, email and (even in breach of the
rules) the telephone to obtain help if they were stuck, or to share their joy in getting something right. The reward
challenges intensified the course to such an extent that some students reported a feeling of “flow”
(Csikszentmihalyi, 1990). This phenomenon is characterised by time going by unnoticed; ironically presenting a
problem to some students because local telephone calls (to Internet service providers) are billed per second and
as a result, excessive telephone charges are incurred.
Students’ responses to the challenges of the games corresponded to O’Houle’s (1998) three categories of
motivational styles: Goal, activity and learning orientation. An analysis of the students’ emails showed that goaloriented students disliked the games, as they were mainly interested in achieving high marks. Activity-oriented
students enjoyed getting to grips with new technology and reported enjoying activities such as participating in an
online demonstration of Interwise software building, online jig-saw puzzles or participating in a virtual photo
shoot. The creativity of the metaphor intrigued the learning-orientated participants who enjoyed the unusual
learning experience.
Malone and Lepper (1987) identify challenge, curiosity, control, and fantasy as requirements for intrinsically
motivating instruction. We found that in CyberSurfiver, challenge was a powerful motivating factor. Participants
often reported working late into the night to complete difficult and challenging tasks.
Participants received the topics of their assignments once a week. One student commented that she particularly
appreciated that all topics were not issued at the beginning of the module because it had raised her levels of
curiosity. The creative approach of the module ensured that students were keen to see what the next week’s
assignments entailed − especially since they had just survived yet another turbulent week.
Participants often indicated their despair at not being able to control their learning environment. They could not
always control their computers or their connectivity, and battled to work collaboratively with other students
whose learning styles, timetables and personalities differed from their own.
The games in CyberSurfiver stimulated students to conceptualise artificially created situations. They were on a
fantasy island. By engaging in the tribal and individual assignments and the reward and immunity challenges
students had to use their imaginations to meet challenges, exercise control and experience interpersonal
Even though the game metaphor of the course did stimulate and motivate the students to learn, they were still in
need of guidance and support from an authoritative figure. This was made evident by the study done on the roles
and competencies of an online facilitator.
The roles and competencies of an online facilitator
There is a misconception that teaching online is the same as teaching in the classroom (Broadbent & Legassie,
2002; Zorfass, Remz & Ethier, 1998). Although individual authors never list more than seven online facilitator
roles at a time, when synthesised, these roles amount to at least 23, but some overlap and could be integrated into
already mentioned roles. Choden (2001) suggests that the various roles could be divided amongst several people,
both in synchronous and asynchronous mode.
To derive facilitator roles and competencies, White & Weight (2000) suggest various messages that online
facilitators could write, while Anderson, Rourke, Garrison & Archer (2001:6-10) have a coding scheme for
instructional design and organisation to facilitate discourse and direct instruction. Our own classification follows
the Blignaut and Trollip (2003) taxonomy of facilitator messages, which distinguishes between those with
academic and non-academic content. Academic content relates to the intellectual messages with sub-headings
dealing with corrective, informative and Socratic messages. Non-academic content includes social,
administrative, affective and other matters.
Although there are several tasks for online facilitators (Full Circle Associates, 2002; Schuman, 2000), they still
do not identify the online facilitator competencies. Moreover, competencies alone do not tell us the roles played
by facilitators to make them “visible” in a web-based environment. Palloff & Pratt (1999) suggest that online
facilitators have both intellectual and social roles.
We attempted to isolate competencies that an online facilitator needs in the online environment. We selected the
Work Profiling System (WPS) tool from Saville and Holdsworth Ltd (SHL) as the conceptual framework. The
WPS has a cognitive component, (thinking competencies) an affective component, (people competencies) and a
psychomotor component (energy competencies). Our analysis focused on the ‘visible’ roles played by the online
facilitator and the identified competencies needed to fulfil in these roles. The data collected by the instruments
already listed was analysed both by hand and using Atlas.ti. An unstructured face-to-face interview with the
online facilitator was added.
The analyses following the Blignaut & Trollip (2003) taxonomy indicated that the online facilitator provided
comprehensive administrative information, supported the students in an affective manner and ensured that
students received individual feedback on their postings. There was still no indication, however, which roles
made the online facilitator “visible” to the students or how conflict is managed (White & Weight, 2000).
Rourke, Anderson, Garrison & Archer (1999) suggest five units of analysis for research on computer
conferencing: Proposition, sentence, paragraph, thematic and message units. We chose thematic units, defined as
a “single thought unit or idea unit that conveys a single item of information extracted from a segment of
information” (Rourke et al., 1999:60). It enabled us to capture the essence of each communication and more than
one code could be assigned to each posting. The specific theme was “visibility”. Blignaut & Trollip’s (2003)
categories were transformed into roles looking for indicators that would reflect “visibility” on the part of the
online facilitator. We attempted to isolate what steps the students and the online facilitator recommended to
improve the visibility of the online facilitator. Five roles of visibility (administrator, social supporter, instructor,
guide and mediator) were identified.
The administrator role conducts timely administration. The online facilitator divided students into groups,
established communication channels, supplied important administrative information, posted reminders and
provided support in terms of time management. The role of administrator involves course management, rather
than dealing with academic issues. For instance, the facilitator discovered that students did not have one
another’s email addresses. Students often complained about the high costs of online time and the amount of time
the assignments took to complete. The facilitator accommodated the students where possible with deadlines for
individual assignments, because she understood the constraints of work, family, health and other unforeseen
The social supporter role ensures participation by, and retention of, students. This is achieved by providing
emotional support and enhancing collaboration. Students need recognition, acknowledgement and positive
feedback. The facilitator understood the students’ frustrations with the workload and the challenges they faced in
terms of the technology and their fellow tribal members. She made a conscious effort to combat the feelings of
The instructor should not take for granted that students all hear, read and internalise important information. The
module started with a short contact session during which the nature of the Survivor metaphor was explained;
students were divided into heterogeneous tribes; and given detailed instructions of where to find the
CyberSurfiver: Introduction document as well as the first week’s assignments.
The facilitator explained that students would experience “what it is like to be an online student”. Most of the
students did not initially hear the information about the document. Others could not access the web-based
version of Yahoo Groups, so they could not read the rationale of game and the outcomes of the module. The
online facilitator, fulfilling the instructor role, undertook to assess all individual assignments halfway through the
course and again at the end of the module. Apart from this official means of feedback the online facilitator
stretched herself to provide the students with constant feedback online, both individually and for the group as a
The role of guide concerns building new knowledge. Just because students are not physically present does not
mean that instructors can forget about their students. The facilitator guided them through the process by
encouraging collaboration, keeping the students motivated, providing a challenge, putting them at ease and
working one-on-one with individual students to fulfil their potential. The diversity of student skills was
considered in the design of teaching strategies and curricula. Students with a strong Internet background used
PHP to design their websites, while novices preferred easy-to-use web-hosting services.
The mediator role ensures fair play. This module produced much conflict and some students missed out on the
collaborative, group formation that the game should offer. Some could not co-operate, others would not and
some never became involved, causing tremendous frustration for the rest. The facilitator managed the conflict
within the groups and dysfunctional teams. Two students did not discuss their roles and contributions with each
other, creating separate websites for one collaborative tribal assignment. Two students could not synchronise
their online times, making all their collaboration asynchronous, very difficult and time consuming. Clashing
individual personalities also affected the dynamics in the tribes.
The five online facilitator roles where put through the Work Profiling System Job Analysis Questionnaires
(JAQs) to determine people, thinking and energy competencies. The results indicated that the online facilitator
needed a total of 13 competencies to be visible in the five roles of an online facilitator.
The people competencies were: motivating and developing others, interpersonal sensitivity, teamwork and
building and maintaining relationships. People competencies were mostly associated with the role of social
supporter, where the facilitator established a friendly virtual environment. The thinking competencies were:
judgment, information gathering, problem analysis, written communication skills, technical skills and
competence. These were distributed fairly evenly across all roles, as were the energy competencies of selfconfidence, persuasiveness and oral communication skills.
Once the roles and competencies of the online facilitator were examined, the question surrounding the emotional,
or affective, responses of students who are being taught by a virtual lecturer were brought to the fore. What are
the effects of being an “online student” and not being taught by a lecturer who is physically present? This
problem was addressed; and the affective considerations of online learning as well as peer support were
Affective considerations and peer support
The data set generated by the methods already listed was analysed once again, but this time to determine how
students cope with the absence of physical contact. The themes or meaning units were identified by means of in
vivo coding (Graneheim & Lundman 2004:106; Holloway & Wheeler 2002:239,240). First-level coding was
done by paraphrasing the words of the participants and collecting them into themes. Incorporating the themes
into clusters (second level coding) and categories (third level coding) refined the themes (Holloway & Wheeler
2002:239,240). Three main categories were identified: Curative factors, the process of development and
inhibiting factors.
Three clusters of themes form the first category, curative factors. These clusters were altruism versus
individualism, communication, and internal drive or value system. Some themes indicated either altruistic or
individualistic behaviour; resulting from feelings such as fear, trust, distrust, safety, insecurity, joy, and stress.
Feelings of loneliness and isolation were experienced by a number of participants. Loneliness was a cause of
increased levels of uncertainty and anxiety; particularly late at night and aggravated when they had a problem
and requested assistance from their peers who would only read the email the next morning.
Participants experienced both positive and negative emotions, but not ambivalence. In their digital
communication they used emoticons to transfer their feelings, and compensate for the lack of personal contact.
Nevertheless, they felt that e-mail was indirect and clinical and did not allow for spontaneous reactions. Sharing
emotions (positive and negative) bound the participants together as a group. A feeling of closeness developed
between tribe members, so that one of the participants felt it would be wrong to get rid of a person by voting
her/him out, because they “came a long way together”.
Feeling scared of being exposed also related to the self-image of participants. Some felt they had an image to
uphold and should not be caught using English grammar incorrectly, as they were Master’s students. One
particular participant indicated that she typed her e-mail message in MSWord, spell checked it, and then pasted it
onto the e-mail composer.
Another three clusters of themes were put together to form the second category of meaning, namely process of
affective development. These phases included the initial phase: responding to requirements; the second phase:
valuing, commitment and organising; and the third phase: internalisation. The phases correspond well with
Krathwohl’s taxonomy of the affective domain (Huitt 2001).
In the initial phase participants felt insecure, vulnerable and exposed because they did not know the level of their
peers’ knowledge and skills:
You sit there, totally naked and struggle. You do not know what the others knew [sic], and you
know too little, but you do not know if they also know as little or less or more than you know.
The intense module requirements initially made participants struggle alone. After feelings of chaos and distrust
of their own abilities early in the module, they began to feel competent. During the second phase they took
charge of their situation accepting their inabilities and abilities. They took active responsibility for their
situations and motivated one another. During the third phase, participants made cognitive decisions regarding
their efforts and those of their peers. Participants who completed the module saw themselves as people who do
not give up, even though some considered doing that in the beginning.
The third category of meaning was called inhibiting factors, and contained nine clusters: Giving up/being voted
out; students’ lack of preparation; lack of technological support and knowledge, troublesome group selection,
language, overload, high financial costs; and telephone service provider problems. This category addresses the
intensity of the volition of the participants who stayed and completed the module regardless of experiencing
many inhibitors. Participants felt that there was a definite gap in their knowledge and skill levels at the onset of
the module. Participants experienced frustration due to technical problems and a lack of technical know-how,
aggravated by the use of unfamiliar software.
In summary, we found that the highly competitive reality game show metaphor caused excessive stress to the
students, to such an extent that one commented that “this was no game”. Nevertheless, those who survived
reported a high level of satisfaction with what they had achieved, indicating, possibly some relationship between
academic stress and a sense of achievement.
The reasons for the successful completion were identified as the game metaphor, the skill of the facilitator in
making herself “visible” over the Internet, and the emotional concern that the students showed for one another
regardless of the absence of physical contact.
The game metaphor allowed motivation through challenge, curiosity and fantasy. Students preferred intangible
and insignificant email rewards, which seemed more attainable to them than the final prize, the weekend away.
The downside of the metaphor was the perceived discrepancy between the goals of the games, and the learning
goals. The main cause of conflict was the lack of availability of team members, lack of commitment and active
participation, as well as contrasting personalities and strong individual wills.
The online facilitator fulfilled five roles to attain full “visibility”. As administrator, she conducted timely course
administration. She maintained social and emotional support. As an instructor she facilitated the learning
process, but failed in providing explicit logistical guidelines. This was overcome by a database of frequently
asked questions (FAQs). As a guide, she encouraged interactivity to foster the building of new knowledge. She
did not risk losing students by abandoning them in cyberspace. As a mediator, she ensured fair play. If problems
occurred, she intervened to resolve them. Filtering these competencies through a job profiling system produced
five people competencies, five thinking competencies and three energy competencies.
Three aspects were identified regarding affective considerations and peer support. These were curative factors,
including altruism versus individualism, communication, and internal drive or value system. Three phases were
identified in a process of affective development. These included an initial phase of responding to requirements; a
second phase of valuing, commitment and organising; and a third phase of internalisation. Nine inhibiting factors
were identified. These included the fear of dropping out, lack of preparation, lack of technical support and
knowledge, group dynamics, work overload, financial concerns and connectivity problems.
Students dropped out mainly because they could not work with the technology. Students stayed mainly because
they enjoyed the game, felt looked after by the facilitator, and supported one another.
More research should be conducted into the relationship between stress and a sense of achievement. Further
work could include the development of a checklist for learners to determine how likely they would be to survive
an online course, and an instrument to determine how likely a traditional facilitator would be to transfer to an
online environment.
We acknowledge with thanks the excellent co-supervision of Prof. Dr Irma Eloff, and the pain of the
participating students.
Adendorff, D. E., (2004). ‘An investigation into the roles and competencies of an online facilitator.’
Unpublished Ph.D. thesis. University of Pretoria. Available at http://upetd.up.ac.za/thesis/available/etd08232004-071632/
Anderson, T. Rourke, L. Garrison, D. R. & Archer, W. (2001). Assessing Teaching Presence in a Computer
Conferencing Context. Journal of Asynchronous Learning Networks, 5 (2), 1-17.
Blignaut, A. S. & Trollip, S. R. (2003). Developing a taxonomy of faculty participation in asynchronous learning
environments – an exploratory investigation. Computers & Education, 41 (2), 149-172.
Broadbent, B. & Legassie, R. (2002). How to facilitate e-learning courses, retrieved June 26, 2006 from
Burke, K. (2000). Consumers turn to video and computer games for social interaction, retrieved June 26, 2006
from http://www.games-advertising.com/assets/socialinteraction.htm.
Cheng, Y. T. & Van de Ven, A. H. (1996). Learning the Innovation Journey: Order out of Chaos? Organization
Science, 7, 593-614.
Choden, A. (2001). How to Succeed as an Online Facilitator, retrieved June 26, 2006 from
Cordova, D. I. & Lepper, M. R. (1993). Intrinsic motivation and the process of learning: Beneficial Effects of
Contextualisation, Personalisation, and Choice. Journal of Educational Psychology, 88 (4), 715-730.
Cordova, D. I. (1993). ‘The effects of personalisation and choice on students’ intrinsic motivation and learning.’
Ph.D dissertation, Stanford University.
Creswell, J. W. (1998). Qualitative Inquiry and Research Design Choosing Among Five Traditions, Thousand
Oaks, California: Sage Publications.
Cronje, J. C. (1997). Interactive Internet: Using the Internet to facilitate cooperative distance learning. SA
Journal of Higher Education, 11 (2), 149-156.
Cronje, J. C. (2001). Metaphors and Models in Internet-Based Learning. Computers & Education, 37, 241-256.
Cronje, J. C. & Clarke, P. A. (1999). Teaching 'Teaching on the Internet' on the Internet. SA Journal for Higher
Education, 13 (1), 213-226.
Csikszentmihalyi, M. (1990). Flow the psychology of optimal experience, New York: HarperPerennial.
Full Circle Associates. (2002). Facilitator Qualities and Skills, retrieved June 26, 2006 from
Galusha, J. M. (1997). Barriers to Learning in Distance Education, University of Southern Mississipi. Retrived
June 26, 2006 from http://www.infrastruction.com/articles.htm.
Garris, R. Ahlers, R. & Driskell, J. E. (2002). Games, motivation and learning: a research and practice model.
Simulation & Gaming, 33 (4), 441-467.
Graneheim, U. H. & Lundman, B. (2004). Qualitative content analysis in nursing research: Concepts,
procedures, and measures to achieve trustworthiness. Nurse Education Today, 24, 105-112.
Holloway, I. & Wheeler, S. (2002). Qualitative research in nursing, (2nd ed.) Oxford, UK: Blackwell.
Huitt, W. (2001). Krathwohl et al.’s taxonomy of the affective domain. Educational Psychology Interactive:
Taxonomy of the Affective Domain, Valdosta, GA: Valdosta State University, retrieved June 26, 2006 from
Krasnor, L. R. & Pepler, D. J. (1980). The study of children's play: Some future directions. In Rubin, K. H. (Ed.)
New directions for child development: Children's play, San Francisco: Jossey-Bass, 1545-1558.
Malone, T. W. & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning.
In Snow, R. E. & Farr, M. J. (Eds.) Aptitude, learning, and instruction: Vol. 3. Cognitive and affective process
analysis, Hillsdale, NJ: Erlbaum, 223-253.
Meyer, S. M., (2005). ‘An investigation into the affective experiences of students in an online learning
environment.’ Unpublished Ph.D. thesis. University of Pretoria.
O’Houle, C. (1988). The inquiring mind, Madison, WI: University of Wisconsin Press.
Palloff, R. M. & Pratt, K. (1999). Building Learning Communities in Cyberspace: Effective Strategies for the
Online Classroom, San Francisco: Jossey-Bass Publishers.
Rauscher, W. & Cronje. J. C.(2005). Online with Krathwohl: Affective aspects of learning in an online
environment. South African Journal of Higher Education, 19 (3), 512-526.
Rourke, L. Anderson, T. Garrison, D. & Archer, W. (1999). Assessing social presence in asynchronous, textbased computer conferencing. Journal of Distance Education, 14 (3), 51-70.
Saville & Holdsworth Ltd (SHL). (1998). The Work Profiling System – Technical Manual. Printed in-house.
Schuman, S. P. (2002). Facilitator Competencies from the Electronic Discussion on Group Facilitation,
retrieved June 26, 2006 from http://www.albany.edu/cpr/gf/resources/FacilitatorCompetencies.html.
Van Ryneveld, L., (2005). ‘Surviving the game: Interaction in an adult online learning community.’
Unpublished Ph.D. Thesis. University of Pretoria. Available at http://upetd.up.ac.za/thesis/available/etd03082005-220804/.
White, K. W. & Weight, B. H. (Eds). (2000). The Online Teaching Guide – A Handbook of Attitudes, Strategies,
and Techniques for the Virtual Classroom, Boston: Allyn & Bacon.
Zorfass, J., Remz, A. & Ethier, D. (1998). Illustrating the potential of an online workshop through a case study
example. Computer-Mediated Communication Magazine, 5 (2) retrieved June 26, 2006 from
Kinzie, M. B., Whitaker, S. D., Neesen, K., Kelley, M., Matera, M., & Pianta, R. C. (2006). Innovative Web-based
Professional Development for Teachers of At-Risk Preschool Children. Educational Technology & Society, 9 (4), 194-204.
Innovative Web-based Professional Development for Teachers of At-Risk
Preschool Children
Mable B. Kinzie, Stephen D. Whitaker, Kathy Neesen, Michael Kelley, Michael Matera
and Robert C. Pianta
Curry School of Education, University of Virginia, Charlottesville, VA 22904-4265 USA
Tel: +1 434-924-0835
Fax: +1 434-924-1384
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
Research suggests the importance of professional development for teachers of at-risk children, and while
educational technologies can be harnessed to help support teaching practice in innovative ways, such
programs are often limited in scale. My Teaching Partner (MTP) was designed to deliver a targeted
program of professional development for teaching quality, integrated with early childhood curricula, to
large numbers of teachers. It has been used by 235 teachers of at-risk preschool children across the
commonwealth of Virginia. MTP links curricula with layers of web-based support, including an on-line
consultant--a teaching expert who regularly observes, de-briefs, and extends teachers’ educational practice.
The research-based curricula support the needs of at-risk Pre-K students in language, literacy, and social
relationships; numerous video clips demonstrate effective teaching practice. In this paper, we describe the
user-centered design process used and the support services and curricula delivered. The dynamic nature of
the website is emphasized, including development practices which enable non-developers to easily
contribute and maintain website contents. We discuss the strengths and weaknesses of this approach, and
describe the field trial now underway, which includes examination of child outcomes, teaching quality, and
teacher reflectivity. Evaluation data from the project are also summarized.
Teacher professional development, Dynamic web design, Early childhood education, At-risk children
Introduction & Background
Needs of Early Childhood Teachers Working with At-Risk Students
By the age of five, children growing up in poverty or related social disadvantage are often lacking in the
language, literacy, and social relationship skills that are critical to school success (Pianta, 1999; Snow, Hemphill,
& Barnes, 1991; Vernon-Feagans, 1996). Although increasing numbers of state-supported pre-kindergarten
programs have been funded over the past decade, to meet the educational needs of these children (Blank,
Shulman, & Ewen, 1999), many Pre-K teachers are in need of training in language and literacy development and
ways to form positive relationships with children, foster emotional and social competence and self-regulation
(Bowman, Donvan & Burns, 2001). In addition, large scale studies suggest overall mediocrity and high
variability in classroom quality and practices, even when experienced, credentialed teachers use the same
curriculum (Bryant, Clifford, Pianta, Howes, & Burchinal, 2002). As a result, a key factor in ensuring program
quality, particularly at large scale, is the training and support of teachers in their implementation of
scientifically-based classroom practices.
Need for Innovation with Educational Technologies at Scale
Innovative applications of technology to support teaching and learning are not uncommon. Further, evaluation
outcomes for some of these initiatives have suggested their utility for supporting the learning and professional
development of teachers. Few of these projects, however, are undertaken “at scale,” with the potential of
addressing the needs of larger numbers of teachers and of making a significant impact on the practice of teaching
and learning.
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
Clifford and Maxwell (2002) have suggested that if a voluntary Pre-K program were offered to all four-year-olds
across the United States, over 200,000 teachers would be required, significantly more than the 27,000 qualified
teachers estimated as practicing in 2002. It is clear that if educational technologies are to help meet the training
and professional development needs of these teachers, these efforts must be at scale.
Innovative Technologies, At Scale, to Meet Teacher Needs
To address the professional development needs of a large number of Pre-K teachers working with at-risk
students, a team of five principal investigators at the University of Virginia’s Curry School of Education
designed and developed MyTeachingPartner (MTP). MTP is an innovative, technology-assisted program that
provides teachers with support from their own on-line consultant, a teaching expert who assists teachers by
regularly (about twice each month) observing, de-briefing, and extending teachers’ educational practice. MTP
also provides a set of web-based resources to support high quality teaching, including research-based educational
curricula designed specifically to support the development of early language, literacy and social relationships in
Pre-K students. The curricula are presented on the Web and illustrated with numerous video demonstrations of
effective practice taken from actual Pre-K classrooms. At present, 235 teachers across the commonwealth of
Virginia are participating in a two-year field trial of MTP, using our curricula every day in their classrooms.
Among our outcome measures is a two-year follow-up on the effects of MTP on children’s language, literacy,
and social development.
In this paper we describe the user-centered design process which led to the development of MTP, including
needs assessment and the design of on-line professional development support services and teaching curricula.
We demonstrate our methods for dynamically generating all pages within this website, a development practice
that enables non-developer content experts and administrative staff to quickly and easily contribute and update
website content. Next, we describe the current research underway as part of our field trial, including examination
of child outcomes (from baseline to two year follow-up for each child studied), effects on teaching quality, and
development of teacher reflectivity as a function of consultation. Finally, we will discuss the completed and ongoing evaluations of MTP, and some of the implications.
User-Centered Design Process
Needs Assessment & Iterative Prototype Development & Testing
From its inception, MTP has focused on the development of high quality Pre-K teaching practices and, through
them, on developing children’s early language, literacy, and social/emotional competence. This approach is
supported in naturalistic studies of child care and Pre-Kindergarten settings (Howe, et al., in press; NICHD
ECCRN, 2002). The design of its methods and materials has evolved over time.
Figure 1. Prototypic Website Flow Document
Although this work is based on a strong foundation of prior research and theory building, the design process still
began with a needs assessment. Ten graduate students spent seven weeks in an advanced instructional design
course conducting this assessment. They began by consulting with teachers, educational experts, and specialists
in emerging Internet-based communications technologies, among others. They went on to review models of
consultancy and literature on Pre-K education and child development. Project goals were then articulated to
describe outcomes for the Pre-K teachers, and a website prototype and initial design for the Internet-based
Consultancy for teachers were mocked up. Although the prototypes were early ones, their form predicted many
of the features of current MTP products. See Figure 1 for an example.
Figure 2. Weekly Lesson Plan Options
The prototypes, along with early drafts of the curriculum design, were evaluated by four Pre-K teachers in three
rounds of prototype testing, and the prototypes were revised.
Curriculum Design
MTP instructional activities focus upon language and literacy as well as social and emotional competence. The
MTP Language and Literacy Curriculum (Justice, Pullen, Hall, & Pianta, 2003) draws upon scientificallysupported practices for teaching phonological and written language awareness skills. The curriculum integrates a
child-driven, highly-contextualized approach with a concentrated teacher-guided and explicitly targeted
methodology for development of key language concepts. This is a model of best practice for early childhood
language intervention programs (Justice & Kaderavek, 2004). There are six targeted skill areas within the MTP
Language and Literacy Curriculum, which teachers address during 30 minute lessons, four times each week:
Alphabet Knowledge, Narrative, Phonological Awareness, Pragmatics and Social Language, Print Concepts, and
Vocabulary Linguistic Concepts. Teachers can either follow our sample weekly lesson plans or build their own
weekly plans in response to the specific needs of children in their classrooms (see Figure 2).
In its current field trial, MTP is also designed to support implementation of the PATHS (Promoting Alternative
THinking Strategies; Greenberg, Kusche, Cook, & Quamma, 1995) curriculum, which focuses on development
of healthy child-teacher relationships and classroom management. Teachers devote at least one 30 minute lesson
to PATHS each week. Other methods for building teacher-child relationships are also employed, such as Banking
Time (Pianta & Hamre, 2001). Thus, the MTP approach addresses major skill needs of Pre-K children, to
encourage school readiness (Pianta, 1999).
Figure 3. Video Demonstration
Website Design
The MTP website (www.myteachingpartner.net) offers a variety of supporting materials, including video
demonstrations of the activities and of quality teaching practice, sample weekly lesson plans, professional
development materials related to Quality Teaching (La Paro, Pianta, & Stuhlman, 2004) and Banking Time
techniques. Examples of the demonstration videos and professional development resources are given in Figures 3
and 4. As the website was being developed, we subjected it to two additional rounds of user testing with twelve
pre-service teachers, with accompanying revisions. In its final form, the MTP website provides both a set of
comprehensive instructional activities and resources to support high quality implementation of those activities.
Figure 4. Professional Development Resources for Quality Teaching
Consultancy Design
The purpose of the MTP Consultancy is to provide teachers with a more intense, individualized form of support
for high quality implementation--direct feedback on their teaching and opportunities to reflect on their practice.
It is based on a collaborative professional development relationship between a teacher and an MTP consultant, in
which systematic observation of classroom practice forms the basis of a partnership that works to support
teachers’ high quality implementation of MTP activities and promote quality teacher-child interactions. The
MTP consultancy process provides opportunities for weekly collaboration through a combination of technologies
that enable regular communication. One form of communication occurs through videoconferencing (and/or
telephone conversations, in the event of technical difficulties). Videoconferencing allows the consultant to see
and hear the teacher in real time. The consultant and the teacher view video clips and share insights about the
implementation of the activities. Another form of communication occurs through use of online journals and
email. Figure 5 displays a consultant-teacher conference, and Figure 6 depicts the consultancy process.
Figure 5. Teacher-Consultant Video Conference
Figure 6. The Consultancy Cycle
Dynamic Web Design
A key to our success with a limited staff and budget was to create a dynamic web infrastructure for the entire
project. Once we determined the functions that were needed, we crafted a framework for the project built upon a
PHP/MySQL server. This has allowed contents to be created and entered by various staff members, and has
enabled content to be updated quickly and easily at any time.
Access Portals Created for Project Development by Non-Programmers
We developed password-protected web interfaces for each type of MTP staff member, to assist them in
contributing contents and managing their branch of operations. We describe some of these access portals in
Table 1. Similar interfaces have been created for:
¾ Project Managers, to integrate all aspects of the project (e.g., when new teachers join the project, their data
automatically populates all databases)
Researchers, for review and coding videos from participating classrooms, and to review student assessment
Development/Technical Team, for production and review of demonstration videos (see Neesen, et al., 2005)
and for review of web usage statistics.
Table 1. Access Portals for Project Staff
Curriculum Developers use this interface
¾ Entering activity contents via a
structured interface that prompts for
each needed component (e.g., title,
curricular area, materials, etc.)
Formatting the activity display and
linking to related materials
¾ Editing descriptions of existing
Consultants have their own website through
which, they can:
¾ Select a teacher to work with
¾ Identify and upload a video clip for that
teacher’s current consulting cycle
¾ Provide prompts for teacher reflection
¾ Review teacher responses to those
¾ Summarize and code their on-line chat
¾ E-mail teachers any follow-ups
The Administrative website allows project
development/technical team to:
¾ Access teacher information, including
any technical support issues they
¾ Submit a new technical issue for a
given teacher
¾ Search the Frequently Asked Questions
(FAQs) database to assist a teacher
¾ Suggest a new possible FAQ
¾ Get a quick calculation of the average
time required to address technical
A similar interface was developed to
support the research functions of the project,
including delivery of measures to teachers
for data collection and systems for data
Benefits & Limitations of Dynamic Web Design
As noted in Table 1, a dynamic approach to our interface has a variety of benefits:
¾ Quick updating of the contents of our teaching activities,
¾ Easy modification of the interface (only the template needs to be modified, not every page displaying a
teaching activity),
Flexibly linking of materials throughout the website, without reproducing materials or re-coding for their
display (for instance, we use video clips extensively, both to demonstrate how to implement particular
activities and, packaged differently with related commentary, as examples of Quality Teaching practice),
Easier backup of site contents: each database is stored in just one file,
“Ownership” and responsibility for site contents is shared across a wider array of individuals (MTP is made
up of 11 full-time and 10 part-time faculty and staff, plus 6-8 graduate students at any point in time),
lessening the work load for the development team and speeding the development and maintenance
Different types of permissions assigned to different individuals, allowing some to have administrative access
to databases (e.g., the ability to create or delete records), and others to have “view only” access.
However, dynamic design does have attendant limitations:
¾ Initial setup of website and database is time-consuming and required more sophisticated development
expertise than static web page creation. We estimate spending six months total time of a systems developer
to build all of the structures needed, in addition to the time normally required to design a web site.
¾ While editing the contents of the website and revising the layout are greatly simplified with a dynamic
approach, it is less easy to add categories of information, as both the web page template and the database
need to be revised.
MTP Project Implementation & Outcomes
Current Research Design
In our current field trial, we are comparing the relative effects of different versions of the MTP professional
development model. Participating teachers attended a one-day summer workshop on MTP, during which they
received a laptop computer they have used to review MTP teaching materials. Teachers in each of the three
treatment groups also received:
¾ Materials Group: The 66 teachers in the Materials group received the MTP language and literacy
curriculum. The curriculum was provided via a “limited features” version of the MTP website.
¾ Web Group: The 89 teachers in the Web group received access to the full-featured MTP website, including
the curricula described above, plus video demonstrations, sample lesson plans, and professional
development activities, among other things.
¾ Consultancy Group: The 80 teachers in the Consultancy group received all of the above features and also
collaborated with an MTP consultant to improve their teaching practice (via online chat and by using the
MTP website to review their videos and respond to consultant-driven prompts)
Multidimensional data are being collected on the effects of MTP, including child, teacher, and classroom quality
Child Outcomes
In each of the two intervention years of our project, four children were randomly selected from each teacher’s
classroom and assessed (pre and post) on language and literacy. (With attrition due in large part to families
moving away from participating school districts, we are now following a total of 1,659 children.) Pre-K teachers
and videotaped classroom observations are providing information on children’s social relationships and selfregulation, in addition to literacy and language development. All children are being followed into kindergarten
and first grade with teacher questionnaires assessing language, literacy, and social relationships/regulation.
We hypothesize an additive effect for the Web and Consultancy treatments, when combined with the MTP
teaching materials, and anticipate that children will show progressively greater growth in child outcomes through
first grade.
Teacher & Classroom Quality Outcomes
Observations of teaching practice and classroom quality were collected via videotape during the two intervention
years and are now being collected in a third non-intervention follow-up year. Observations are coded using the
Classroom Assessment Scoring System (CLASS), a coding system for the 14 different dimensions of teaching
quality that are the focus of the MTP web-based resources. Studies involving over 2,000 pre-kindergarten and K1 classrooms have shown that higher ratings on the dimensions assessed by the CLASS predict higher
performance of children on standardized assessments of academic achievement and better social adjustment in
the early grades of school (La Paro, Pianta, & Stuhlman, 2004). The relationship between the different aspects of
this model is described in Figure 7.
Figure 7. Linking Professional Development with teacher and child outcomes through standardized observations
of classroom processes (Pianta, et al., 2004)
The child and teacher outcomes from this project will be the subject of forthcoming papers. In addition, we are
developing a methodology for evaluating the reflectiveness of teachers participating in the consultancy, and
expect to examine the evolution of teacher reflections over time concurrently with teaching quality.
Evaluation Outcomes
It should be noted that even with the best of design and implementation methods (we provided a 1.5 day
introductory teacher workshop, along with extensive on-line technical reference materials and FAQs), teachers
still require substantial technical support, and since teachers were from many different districts with different
computer skills, their problems tended to be diverse. Our approaches to technical support evolved over the first
year of the project, as support shifted away from being provided by teaching consultants to the technical staff
using a structured support system (see Kelly, Whitaker, Neesen, Kinzie, & Pianta, 2005, for an overview). This
resulted in a greater amount of support provided to teachers in the same amount of time, high levels of teacher
satisfaction, and no dropouts due to technical challenges.
There are obvious advantages to our technology-assisted process for content development and content review.
For instance, drawing upon video submitted from participating teachers and employing an on-line video
production and quality review process has meant that we can produce between five and ten demonstration videos
in as little as six hours, a significant advantage for MTP when we have hundreds of teaching activities we would
like to demonstrate (see Neesen, et al., 2005, for more on this approach to video production).
Our user-centered design process has yielded a website perceived to be easy to use and worthwhile, judging from
the feedback we received from teachers in our first implementation year: Overall, 92.7% of the 110 MTP
teachers who responded to our mid-year evaluation survey “agreed” or “strongly agreed” that the MTP website
was easy to use; 88.2% “felt that the site added value to their professional practice, and 87.2% indicated that
using the MTP website was worth the time spent on it. With data from the end-of-year survey of 203
participating teachers, we conducted analyses by treatment group (Materials [n = 45], Web [n = 79], Consultancy
[n = 79]). Teachers in the Web and Consultancy groups, who received higher “doses” of MTP, reported higher
perceptions of value and usefulness, and felt more strongly that MTP was worthwhile, than did the Materials
group teachers (Whitaker, Kinzie, Kraft-Sayre, Mashburn, & Pianta, 2005). Teachers receiving different
amounts of support also evidenced differing levels of participation (Whitaker, et al., 2005).
According to server log analysis of the on-line behavior of teachers in the first implementation year, teachers in
the Materials group [n = 50], whose version of the website contains only the curricula, visited the MTP site an
average of only 6.4 times during the school year, but they spent significantly more time during each visit
(average length of 10.9 minutes each time), perhaps because these teachers needed to print out the MTP
activities they wanted to use. These Materials group teachers also reported spending the least amount of time
preparing and delivering MTP activities. Web-group teachers [n = 79] accessed the site an average of 10 times
each, with an average length of 4.2 minutes per visit. Consultancy teachers [n = 84] accessed the site
significantly more often than the other two groups, an average of 40.8 times each, and they spent longer than
teachers in the Web group, with a mean visit of 7.62 minutes (Whitaker, et al., 2005).
Overall, teachers have been enthusiastic in their support of the program, making comments such as:
¾ “I would say this is the best professional development opportunity that I have had in my 23 years of
¾ “My children’s assessment scores have never been higher…this year I have seen scores in many cases
When asked about their favorite aspect of MyTeachingPartner, many teachers mentioned MTP professional
development features:
¾ “Ease of using the lesson plans on the Internet.”
¾ “My favorite part of MTP is the activities and suggestions given on the website.”
¾ “I like viewing the videos for presenting the books and using the MTP activities.”
¾ “Having a consultant made me more reflective and focused on what I need to do each day.”
These results support the adage: if you build it, they will come: Teachers voluntarily spent additional time
preparing to deliver a new curriculum and participating in the professional development that MTP provides, and
that they found it of significant value. With MTP, we were able to employ effective and efficient development
methods, in order to provide teaching materials and professional support that 235 teachers valued and spent time
using. Our future research will show the degree to which these efforts influence the development and school
readiness of children who are at-risk.
This research was supported by a grant from the National Institute of Child Health and Human Development
(NICHD), National Institutes of Health (NIH). A special thanks to the hard-working and talented staff of My
Teaching Partner and our participating Pre-Kindergarten teachers across the commonwealth of Virginia, who
make a very important difference to at-risk children.
Blank, H, Schulman, K. & Ewen, D. (1999). Seeds of Success, State Prekindergarten Initiatives, 1998-1999.
Washington, DC: Children’s Defense Fund.
Bowman, B. T., Donovan, M. S. & Burns, M. S. (2001). Eager to learn. Washington, DC: National Academy
Bryant, D., Clifford, R., Pianta, R. C., Howes, C. & Burchinal, M. (2002) (in press). Characteristics of prekindergarten programs in six-states: Children, teachers, and programs. Applied Developmental Science.
Clifford, D. & Maxwell, K. (2002). The need for highly qualified prekindergarten teachers. Paper presented at
the Preparing Highly Qualified Prekindergarten Teachers Symposium. Retrieved 31 August 2006, from
Greenberg, M. T., Kusche, C. A., Cook, E. T. & Quamma, J. P. (1995). Promoting emotional competence in
school-aged children: The effects of the PATHS curriculum. Development and Psychopathology, 7, 117-136.
Justice, L, Pullen, P., Hall, S., & Pianta, R. (2003). Curry School Curriculum for Early Literacy and Oral
Language Support. Unpublished. University of Virginia, Charlottesville.
Justice, L. M. & Kaderavek, J. (2004). Embedded-explicit emergent literacy I: Background and description of
approach. Language, Speech, and Hearing Services in Schools, 35, 201-211.
Howes, C., Burchinal, M., Pianta, R., Bryant, D., Early, D., Clifford, R. & Barbarin, O. (in press). Ready to
learn? Children's pre-academic achievement in pre-kindergarten programs. Developmental Psychology.
Kelley, M. A., Whitaker, S. D., Neesen, K., Kinzie, M. B. & Pianta, R. C. (2005). Tech support requirements for
large-scale technology innovation in teacher professional development. Paper presented at the annual meeting
of E-Learn, Vancouver.
La Paro, K., Pianta, R. & Stuhlman, M. (2004).Classroom Assessment Scoring System (CLASS): Findings from
the Pre-K Year. Elementary School Journal, 104 (5), 409-426.
Neesen, K., Kinzie, M. B., Whitaker, S. D., Funk, G. G., Hall, A. P. & Pianta, R. C. (2005). Educational video
production to support teacher reflection and professional development: A high quality, cost effective
collaboration between educators, researchers and educational technologists. Paper presented at the annual
meeting of E-Learn, Vancouver.
NICHD Early Child Care Research Network (ECCRN). (2002). Structure>Process>Outcome: direct and indirect
effects of caregiving quality on young children’s development. Psychological Science, 13, 199-206.
Pianta, R. & Hamre, B. (2001). Students, Teachers, and Relationship Support (STARS). Lutz, FL: Psychological
Assessment Resources, Inc. Available at www.parinc.com
Pianta, R. C. (1999). Enhancing relationships between children and teachers. Washington, DC: American
Psychological Association.
Pianta, R. C., Hall, A. P., Dudding, C., Whitaker, S., Kraft-Sayre, M. & Downer, J. (2004). MTP Procedural
Manual for Teachers. Center for Advanced Study of Teaching and Learning, University of Virginia.
Snow, C. E., Hamphill, L., & Barnes, W. S. (Eds.). (1991). Unfulfilled expectations: Home and school influences
on literacy. Cambridge, MA: Harvard University Press.
Vernon-Feagans, L. (1996). Children’s talk in communities and classrooms. Cambridge, MA: Blackwell.
Whitaker, S. D., Kinzie, M. B., Kraft-Sayre, M. & Pianta, R. C. (2005). Use and Evaluation of Web-based
Professional Development Services Across Level of Service and By Teacher/District Characteristics. Paper
presented at the annual meeting of E-Learn, Vancouver.
Hernández-Ramos, P. (2006). How Does Educational Technology Benefit Humanity? Five Years of Evidence. Educational
Technology & Society, 9 (4), 205-214.
How Does Educational Technology Benefit Humanity? Five Years of
Pedro Hernández-Ramos
Dept of Education & Center for Science, Tech, and Society, Santa Clara University, CA 95053 USA
Tel: +1 408 554-4131
Fax: +1 408 554-2392
[email protected]
This article presents a review of the 25 finalists (Laureates) in the Education category of the Technology
Benefiting Humanity Awards, which started in 2001. Most of the applicants can be considered social
entrepreneurs working to improve educational systems and the learning opportunities and experiences of
their intended beneficiaries. While the benefits to humanity derived from educational technology are often
taken for granted, the work of these 25 Laureates, working in countries around the world, provides an
opportunity to critically examine how such benefits are actually obtained. Based on this analytical review,
suggestions for further research and development in educational technology are presented.
Social entrepreneurs, Educational technology, Development, Awards
Two major reports published since 2000 highlight the importance of education for economic and social
advancement in less-developed countries. The United Nations’ Millenium Development Goals (MDGs) set down
on September 18, 2000 committed all 191 member states to meet eight major goals, with number two being:
“Achieve universal primary education—ensure that all boys and girls complete a full course of primary
schooling” (United Nations, 2005). Similarly, the World Education Forum that took place in Senegal also in
2000 established six goals (Basic Education Coalition, 2004) that, in a way, add detail to the UN’s Millenium
Development Goals regarding education:
1. Expanding and improving comprehensive early childhood care and education, especially for the most
vulnerable and disadvantaged children.
2. Ensuring that by 2015 all children, particularly girls, children in difficult circumstances, and those belonging
to ethnic minorities, have access to and complete free and compulsory primary education of good quality.
3. Ensuring that the learning needs of all young people and adults are met through equitable access to
appropriate learning and life skills programs.
4. Achieving a 50% improvement in levels of adult literacy by 2015, especially for women, and equitable
access to basic and continuing education for adults.
5. Eliminating gender disparities in primary and secondary education by 2005, and achieving gender equality
in education by 2015, with a focus on ensuring girls’ full and equal access to and achievement in basic
education of good quality.
6. Improving all aspects of the quality of education and ensuring excellence so that recognized and measurable
learning outcomes are achieved by all students, especially in literacy, numeracy, and essential life skills.
Given these goals, the question arises as to why should poor, less developed countries spend scarce resources
providing technology (usually computers and Internet access) to only a few of their schools when most other
schools lack even the basics of proper buildings, enough and properly trained teachers, books, and other learning
materials? Also, the fact that technology-related projects tend to be quite expensive and usually financed by
foreign loans that are open to graft and misuse (Easterly, 2001) gives some people reasons to advocate against
such projects.
In a developed world context like the U.S. Japan, Australia, or Europe, the reasons for introducing technology
into education seem obvious if they are rooted in goals like the overall improvement of teaching and learning:
introducing technology to schools helps the system to better prepare students for their future, when they’ll need
so-called 21st century skills (e.g., Fulton, 1997). In developing world contexts, however, where four fifths of the
world population lives, the desired benefits of educational technology often cannot be focused on improving on
existing practices as much as in creating practices where there may be none—or none worth holding on to. The
motivation often comes from a belief in the potential benefits of technology to allow education to reach unserved
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
or underserved populations. For example, the United Nations Development Program’s (2003) Human
Development Report argued that “Technological innovations advance human development in two ways—by
increasing productivity that raises household incomes (Goal 1 [of the MDGs]) and by providing solutions to
problems of disease, transport, energy, water supply, sanitation and information and communications technology
for education, all important for achieving Goals 2–7 [of the MDGs]” (p. 158).
On the other hand, there is also a body of literature that questions the commitment to technology in the future of
education (e.g., Cuban, 2001; Ferneding, 2003). Part of the challenge comes from understanding the profound
structural changes that the integration of technology requires of education systems: financial, social, cultural, and
of course pedagogical. The “failures” of past attempts to integrate technology into teachers’ daily practice (e.g.,
Cuban, 1986) also serve as an argument used by critics to request or demand a change in directions away from
technology investments (mainly computers and Internet access), and by extension, away from many of the
pedagogical practices associated with the use of technology for teaching and learning. The point made for
education specifically can be generalized for national economic and social development, as the Markle
Foundation’s report, Global Digital Opportunities (Tipson & Frittelli, 2003), argued by stating that, “Simply
increasing connectivity or distributing computers and software will not lead to development impacts unless the
range of other factors—enterprise, applications, human capacity, policy—converge to make that possible”
[emphasis in original] (p.8).
The fact that technologies by themselves are incapable of achieving widespread and deep changes in education is
the key argument offered by critics of educational technology. It happens to be, as well, a point easily
acknowledged by even the most ardent proponents of the use of technology in education (e.g., Fisher, Dwyer, &
Yocam, 1996; Osin, 1998). Tipson & Frittelli (2003) summarized it neatly by stating that
ICTs [Information and Communication Technologies] clearly have major contributions to make
in education, healthcare, gender equality, environmental sustainability and the other MDGs, but
to be successful, that contribution should always be viewed as dependent on a range of other
factors and subject to comparisons with other, less technology dependent approaches (p.13).
The balance of this article will focus on the work of the first 25 Laureates (2001–2005) in the Education category
of the Technology Benefiting Humanity Awards (http://www.techawards.org/), most of whom are operating in
less developed countries. A major goal of this awards program is to recognize and motivate the work of
innovators around the world who are working to benefit humanity (“social entrepreneurs”) through the
development or application of technology. This Awards program has quickly established itself as a global
positive force for change and innovation, attracting hundreds of applications. The five Education Laureates
selected each year come from among dozens of applicants (over 100 each year so far). An international judging
panel selects only one Education Laureate each year to receive a prize of US$50,000. A brief summary of their
work is presented next, followed by analysis of the patterns and trends that can be deduced from the range of
their activities. It will end with conclusions and recommendations for future actions and research.
The Education Laureates
Cardwell (1995) wrote that, “All that can be said is that it would be unwise to assume that all or even most
innovators are motivated by purely mercenary hopes. Personal satisfaction and the hope of social distinction are,
no doubt, factors in the lives of many technologists as well as in the lives of many ‘pure’ scientists” (p. 495).
This definition of an innovator certainly applies to social entrepreneurs, which in turn have been defined as,
“One who has created and leads an organization, whether for-profit or not, that is aimed at creating large scale,
lasting, and systemic change through the introduction of new ideas, methodologies, and changes in attitude”
(Kramer, 2005, p.6).
The five Education Laureates in each of the first five years certainly would fit that description, since their work
seems motivated more by the desire to improve the human condition than financial gain. Table 1 presents a
summary of the problem and solution(s) offered by each Laureate. (See also Hernández-Ramos, Soukup, &
McAnany, 2001; Soukup, 2002; Hernández-Ramos, 2003; Raphael, 2004; and Raphael, 2005.)
Table 1. Summary of Problems Identified and Solutions Offered by Education Laureates
Freeplay Foundation
Lack of access to education,
Windup radios and programming
health information; cost of
aimed at populations in critical
The Internet as an unorganized
resource for children’s learning
How to make best use of a
world-class telescope and
scientific facility for teaching
and learning in K-12 schools
How to disseminate thousands
of “famous and important
texts” in the public domain
Kids’ Space
Lewis Center for
Educational Research
Project Gutenberg
Schools Online
Barefoot College
Global SchoolNet
Center for Spoken
Language Research
Equal Access
Omar Dengo Foundation
Computers for Youth
How to motivate and engage
youth and their families in lowincome middle schools
Brij Kothari-IIM
Alexander McDonald,
How to help millions of people
in India with no access to
schools to learn to read
How to convey complex timebased data on the human
impact on the Earth, in
meaningful ways, to diverse
How to get schools in
developing countries, to benefit
from computers and the
How to offer educational
options for poor people in
remote parts of India
How to leverage computers and
the Internet to expand the
horizons of students
everywhere in the world
How to support learning for
profoundly deaf children and
others suffering from autism,
dyslexia, and other disorders.
How to best address the
learning and reading needs of
students with physical or
cognitive disabilities.
How to promote literacy,
preserve culture, and encourage
education among New Delhi’s
How to reach populations in
remote places of developing
countries (starting in Nepal) to
provide information and
How to reform the education
system while introducing
educational technology
situations in Africa
Moderated website with activities
specifically designed for schoolage children
Allow use of the radio-telescope
and provide teaching and learning
resources for science education and
Website has over 5,000 texts in
various electronic formats and
languages available for free
Create teacher training resources
and establish strategic partnerships
for effective local implementation.
Network of 20 campuses in 13
states trains people to create and
maintain technology infrastructures
for community benefit.
Create resources and projects that
teachers and students can use in
their classrooms and share around
the world.
Developed computer-based,
personalized, learning tools for
learners with speech learning
difficulties resulting from deafness,
autism spectrum disorders,
dyslexia, and others.
“Thinking Reader” technology
supports a wide range of learning
modalities and disabilities.
“Computer clubhouse,”
community access, teacher
training, education, and other
services for children and women
living in slums.
Use Digital Satellite Radio and
partner with local organizations to
create and disseminate relevant
content for hard-to-reach
Re-think teaching and learning to
better prepare students and support
teachers in the systematic
transformation of classroom and
community practices.
Provide a computer with Internet
access for each student, and
involve them in relevant activities
for the school and the community.
Add subtitles to popular television
shows to build the literacy skills of
Create the “Science on a Sphere”
3-D projection system that makes
it possible to visualize complex
data sets on a spherical surface.
How to help visually impaired
learners deal with visual
Baruch College Computer
Center for Visually
Impaired People and
Touch Graphics
Andrew E. Lieberman,
Asociación Ajb’atz’
Enlace Quiché
National Society for
Earthquake Technology
How to increase the knowledge
and earthquake awareness of
builders in Nepal
Gilberth Clark, Telescopes
in Education Foundation
Design that Matters, Inc.
How to offer students
opportunities to work with
world-class telescopes from
Teaching adults how to read in
areas with few books and no
Fahamu–Networks for
Social Justice
How to train staff of human
rights organizations working in
How to encourage the
development of literacy skills
among low-income students in
the U.S.
MIT Open CourseWare
How to share educational
resources in ways that protect
intellectual property yet
enhance the knowledge
Address the school funding
crisis in the U.S.
How to leverage the power of
computers and
telecommunications for
improved education and
How to counteract the decline
of indigenous cultures and
languages in Guatemala
Develop the Talking Tactile Tablet
that supports access to graphical
and multimedia applications.
Set up a website that matches
teacher-identified needs with
potential donors.
Online forums, project-based
learning, and other activities
connecting teachers and students
around the world.
Develop a series of multimedia and
web-based materials, and other
community services, that preserve
and enhance local indigenous
Created the “Shake Table” to
demonstrate the effects of
earthquakes on buildings erected
using inadequate vs. modern
building techniques.
Web-based access and control of
sophisticated telescopes in the
U.S., Australia, and Chile.
The Kinkajou projector relies on
solar batteries, high-efficiency
LEDs, low-cost optics in a
projector that shows print and
images from microfilm.
Created custom-developed courses
delivered via workshops, CDROMs, and email to serve their
Literacy program gives 5 books to
each participating child, and pairs
each student with an adult “pen
pal” via letters and email
Launched the MIT Open
CourseWare initiative in 2002,
where information and resources
for the majority of MIT courses are
available free of charge.
As can be deduced from the list in Table 1, there have been no corporations, only one government organization
(Alexander McDonald is affiliated with the U.S. National Oceanographic and Atmospheric Administration), and
only three individuals (Kothari and McDonald in 2003, Clark in 2005) so far among the Laureates. While the list
seems weighted toward organizations from the United States, many of them were being recognized for work
done outside of this country or with a deliberately global scope.
Whether or not the Laureates were aware of the United Nations’ MDGs and the six goals of the World Education
Forum, their work contributes to the achievement of those goals in measurable ways. From 2001, for example,
what FreePlay Foundation and Schools Online were doing addresses directly the need of providing access to
information and education through technology (wind-up radios for FreePlay, computers with Internet access for
Schools Online) for hard-to-reach populations.
It is also difficult—and to some extent meaningless—to attempt to categorize the work of the Laureates into
narrow niches, such as saying that solutions are hardware, software, or services since in most cases the work was
recognized precisely because it is not just a technology but a systemic, socio-technical approach that
encompasses other critical elements for success. A clear example of this is Costa Rica’s Omar Dengo Foundation
(2003 Laureate), which not only provides hardware and software for schools throughout the country but is also
in charge of teacher training, community outreach, and other essential services needed to make the introduction
of technology a success (Ringstaff & Kelly, 2002). The pattern makes clear that one of the criteria the Tech
Awards judges use is not just how impressive for its own sake the technology in use may be, but also the factors
surrounding its implementation in the intended contexts for the target audiences. There is an appreciation for the
value of designing systems around tools to accomplish complex objectives. Given the contexts in which many of
the Laureates work, simple technologies with good systems around them may be better than the latest
technologies but with less support. A very good example of this approach is Fahamu (2005 Laureate), which
relies on CD-ROMs, email, and in-person workshops to build capacity for human rights organizations given that
the Web is simply not as accessible or affordable as these older, technically simpler media.
Among these first 25 Laureates there are three outstanding examples of hardware and software technologies
developed to address the unique learning needs of people with disabilities, whether physical or cognitive: Center
for Spoken Language Research (2002 Laureate), Center for Applied Special Technology (CAST, 2002
Laureate), and Baruch College Computer Center for Visually Impaired People and Touch Graphics (2004
Laureate). Development of the hardware and/or software was only the starting point in all three cases. For
example, CAST created the Thinking Reader software as well as a broader framework, the Universal Design for
Learning, to help teachers differentiate their lessons so that the needs of all learners—including those with
disabilities in integrated classrooms—are successfully met. Thus, CAST provides a critical scaffold (the software
to support learners with reading difficulties) as well as the pedagogical framework within which the scaffolding
technology is best used.
Students with disabilities raise significant challenges for educators interested in affording them the same quality
of learning experiences as “normal” students. These solutions tend to be initially expensive, which limits their
dissemination, although it is not hard to imagine how all learners could eventually benefit from these advances.
The analogy to curb cuts in sidewalks, initially seen as a benefit for wheelchair-bound individuals, is also a help
to mothers pushing baby strollers, bicyclists, older people, and so on. This is an exciting prospect for education
around the world, given that if the pattern of rapidly decreasing costs of new technologies continues (as it is
likely to), disabled learners in countries other than those originating the technology (in these three cases, the
United States) are likely to eventually benefit. One strategy being pursued is to encourage Laureates to enter into
licensing agreements that extend the reach beyond the original developers to others who can include the
technology in their products.
Looking beyond from developed world contexts has the benefit of allowing us to see how practitioners in other
parts of the world define and then address common problems in education such as funding, training, access to
resources, and serving a wide variety of learners including those with physical or other learning disabilities.
Also, it allows us to consider both formal and informal education settings, since the educational infrastructure of
the formal systems (including inadequate funding, poor buildings, insufficiently trained teachers, resource
deficiencies, and so on) some times makes it more efficient to address the problem through alternative strategies.
The 20 Laureates share some distinguishing characteristics even across their widely different approaches to the
integration of technology in education and learning. Perhaps among the most salient is the recognition that
educational technology is not simply computers and the Internet, as much of the discourse in developed countries
seems to have focused, but should adopt a broader view of technological innovation. Such a view goes beyond
cutting-edge hardware and software to include older technologies and, critically, a holistic approach to the
implementation of complex systems that recognize the social dimensions of technology-based innovations. On
the other hand, those who have been recognized for truly innovative technological solutions have clearly
identified how their approach is better than any alternatives available to the populations they aim to serve (e.g.,
disabled learners).
From a learning theory perspective, it is striking to recognize how many of these projects are working—
explicitly or implicitly—from a constructivist perspective (e.g., Fosnot, 2005; Jonassen, Peck, & Wilson, 1999),
to afford learners meaningful experiences connected to or in the “real world” and giving them access to the
professional-level tools and resources of practicing scientists, researchers, and so on. Perhaps the best examples
among these 25 Laureates are the Lewis Center for Educational Research (2001 Laureate) and Gilbert Clark of
the Telescopes in Education Foundation (2005 Laureate). Each of these efforts allows teachers and their students
access to professional-level tools and resources (a radio-telescope through the Lewis Center, and several remotecontrolled telescopes through Telescopes in Education). It’s easy to imagine the excitement felt by those students
when given the chance to work in such a facilities, doing work as real scientists would. At another level, the
work by Nepal’s National Society for Earthquake Technology (2004 Laureate) exemplifies the belief that people
may learn best when they get to do and see and touch rather than just listen. Their Shake Table provides a
powerful and direct way to instruct builders on the consequences of inadequate building techniques and
materials, and thanks to the ability to simply and cheaply simulate how different structures fare in earthquakes of
different magnitudes, turn what may seem as abstract concepts into memorable lessons.
Considering first the approach to technology as meaning more than computers and the Internet, a closer look at
the work of the Freeplay Foundation, the Barefoot College, Equal Access, and Brij Kothari’s Same Language
Subtitling is rewarding. In each case, the technology used is not a breakthrough technology except in the context
where it was being introduced, as in the regions of Africa where the Freeplay Foundation’s windup radios
addressed a major problem of lack of access to information because the cost of batteries made operating a
transistor radio too expensive. The Barefoot College in India broadens the definition of technology to include
both hardware (solar panels for electricity generation) and software (work processes that empower local
communities through harnessing of natural resources). Equal Access, on the other hand, relies on state-of-the-art
digital satellite radio broadcast and receiving technology (including a custom-designed portable receiver
intended for personal or small group use) to reach populations that cannot have access to education and
information through any other media. And Brij Kothari’s elegantly simple idea to leverage subtitling of televised
images (mainly the very popular music variety shows broadcast in Indian television) as a way to expose illiterate
populations to the song lyrics—that is, the written languages they have had little or no exposure to in their
limited schooling experience—thus supporting beginning literacy.
Each of these four examples relied on technologies invented by someone else, but the Laureates made creative
use of them to address specific needs they identified. Cardwell’s (1995, p.493) valuable distinction between
machines (temporary, replaceable by better machines) and structures (designed to be permanent) is applicable
here, for even the Laureates designing mainly hardware clearly understand the need to have appropriate
structures around them that enhance the likelihood of successful, long-term implementation. The ripple effects of
their work on the communities they serve are broad and deep, in some cases leading to significant changes in
living standards and quality of life.
Cases like the Costa Rican national educational technology implementation, headed by the Omar Dengo
Foundation (2003 Laureate), highlight the importance of vision and leadership in change processes of this
magnitude. “National leaders must advance a vision of change that can steer a political course that somehow
balances rapid development with social cohesion and cultural integrity” (Tipson & Frittelli, 2003, p. 10), a
challenge that can be easily biased in the direction of political self-interest and preservation of special interests.
Costa Rica is, along with Chile (Hepp, Hinostroza, Laval, & Rehbein, 2004), one of the few success stories in
Latin America where changes of this magnitude to national education systems have been accomplished,
surviving political changes that destroy similar efforts in other countries. The process is not over, even after
more than 15 years of sustained work in Costa Rica, a fact that serves to reminds us that change—and the
benefits derived from it— can be painfully slow even under the best of circumstances.
It’s also worth noting that most of the Laureates are working toward improvement of conditions of people at the
bottom of the pyramid (Prahalad & Hammond, 2002). The systemic efforts to employ technology to address the
needs of populations in dire need is evident in the work of organizations such as the Barefoot College (2002
Laureate), Freeplay Foundation (2001 Laureate), Katha (2002 Laureate), Schools Online (2001 Laureate), Equal
Access (2003 Laureate), Computers for Youth (2003 Laureate), Andrew Lieberman of the Asociación Ajb’atz’
Enlace Quiché (2004 Laureate), and Design that Matters (2005 Laureate). In all these cases, the goal is both to
focus on a specific population and then try to reach as many of them as possible. The technologies employed by
these Laureates have been chosen wisely, to address the needs of the people served and enhance the
multiplicative effect of the introduction of technology in their environment. In all these cases, technology is
playing multiple roles, increasing individual opportunity and contributing to social cohesion and development.
An underlying and often unstated assumption about ICTs is that “when designed and phased in with attention to
each aspect of the strategic framework, ICTs should be able to trigger a form of development dynamic—a
multiplier or “network effect” that generates an overall impact greater than the sum of the separate inputs”
(Tipson & Frittelli, 2003, p.11).
The experiences of these Laureates give credence to this assumption. Barefoot College, started in 1972, now has
20 campuses in 13 states in India and is about to start training people from five other countries. Freeplay
Foundation has distributed over 50,000 radios reaching an estimated one million people every day. Schools
Online has installed computers with Internet access in over 400 schools in 35 countries, and trained hundreds of
teachers in those schools to make the most of their new technology resources. The radio programs developed in
cooperation with local partners by Equal Access, delivered through digital satellite radio, reach an estimated
audience of 9 million people in Nepal, while in Afghanistan a broadcast aimed at teachers reaches an estimated
3,500 teachers, benefiting the education of about 150,000 students. Katha in India and Asociación Ajb’atz
Enlace Quiché in Guatemala are both agents of change and of cultural survival for the communities they serve,
creating spaces and opportunities for people who would otherwise not know how to leverage the potential
benefits of ICTs to learn better, express their creativity, and preserve their cultural heritage.
Such “network effects” are even more in evidence when the technology in use is the Internet. Although based in
the United States, what programs like Kids’ Space (2001 Laureate), Global SchoolNet (2002 Laureate), Project
Gutenberg (2001 Laureate), and to some extent I*EARN (2004 Laureate) and MIT Open CourseWare (2005
Laureate) are doing was simply not possible in a pre-Internet era. The ideals sustaining their work argue that
ICTs are not meant to be merely “enablers” of outmoded but easily reproduced teaching and learning practices,
but rather a force or catalyst for structural changes in education. By enabling access to rich resources and making
possible communication and collaboration across time and space in well-structured virtual spaces, these
Laureates have created wonderful opportunities for students and teachers who no longer have to be limited to the
resources in their classroom and schools. Global SchoolNet and I*EARN, for example, each have thousands of
schools and teachers participating in collaborative projects from over 100 countries around the world, involving
more than one million students in learning experiences made all the more memorable because of the
international connections. The MIT Open CourseWare initiative has already spawned similar projects at several
other institutions in the U.S., Japan, and Spain, and also sparked a vital debate about the role of intellectual
property in the future of education. At a time when intellectual property regimes seem to be becoming more
restrictive, MIT’s attitude of generous information sharing has re-opened conversations about the academic
values of free knowledge dissemination for the benefit of humanity at large.
Concerns over the replicability, scalability, and sustainability of all these social benefit programs has led to the
creation of the Global Social Benefit Incubator (GSBI) program (Koch, Coppock, Guerra, & Bruno, 2004;
Mangan, 2004), which in addition to an intensive, two-week, on-campus workshop for selected attendees is
actively investigating the use of online environments to support ongoing access to education, information,
communication and other resources (Hernández-Ramos, Koch, Dommel, Guerra, & Bowker, 2005). The goal is
to leverage online technologies to sustain and advance the work of social entrepreneurs around the world. The
online environment would complement and extend the work that takes place during two-week summer
workshops at a university campus, and will allow access to individuals and organizations that cannot participate
in person.
Looking at the work of these first 25 Laureates of the Technology Benefiting Humanity Awards, it would be fair
to conclude that “educational technology” can, indeed, benefit humanity. However, this general answer requires
a deeper look at several common issues raised by this analysis and other experiences.
First among these is the fact that it is unproductive to speak of educational technology as a single entity, and in
particular to limit its meaning to computers and the Internet. Almost all of these Laureates used computer and
Internet technologies in their work, but they clearly went beyond providing access to computers and the Internet
to the recognition that the process of change in educational practices and systems requires systemic, long-term
A second question addresses the invisible demands of technology—the fact that extensive infrastructures may be
needed to allow computers and the Internet to function properly (or at all) in education settings around the world.
The world may be steadily moving toward a knowledge-based economy where networked computers and other
devices are critical tools, but increasingly this also means that electrical, telecommunications, educational, and
other infrastructures need to be in place before countries can hope to effectively participate in such an economy.
Projects like Schools Online, Barefoot College, and Computers for Youth highlight the need for—and the
benefits of—a systemic approach that aims to have all the pieces under their control in place, and work with
partners to address all other components.
Third, with the exception of the Omar Dengo Foundation in Costa Rica, these Laureates are working on the
periphery of education systems rather than at the core, and thus in some cases it is difficult to imagine them
being able to scale up to a point where their impacts and influence could be systemic and go beyond a fairly
narrow audience (Joyce, 2005). The positive impact of their mere presence should not be underestimated, but at
the same time, as organizations they face the typical challenges of growth and sustainability if they are to survive
and thrive for the long term. As mentioned above, the GSBI program intends to address this issue by developing
the skills and knowledge bases of key members of the Laureates’ organizations, initially through on-campus
training and later through an online environment.
While not a representative sample of the universe of technological innovation in education around the world, this
review does provide valuable information about possible directions for future research and development. These
Electrical power
The Freeplay Foundation’s development of the windup radio was a creative solution to vexing problems
surrounding access to information through electronic media, which require electrical power to function. When a
country’s electricity grid does not reach every community and household, and when portable power (batteries,
fuel-powered plants) is too expensive, alternative sources of power (mechanical, wind, solar) that can keep
equipment running for significant amounts of time, with little or no effort and at minimal cost, are a critical
strategy to pursue. Literacy programs are among the first to benefit when future readers have easier access to
electric lighting.
Despite the relentless drops in price for computer and networking equipment, computers and peripherals
designed to operate in industrialized office or home environments have a much shorter productive life when used
in less-than-ideal circumstances, and are still too expensive for the vast majority of people around the world.
Dust, heat, humidity, and unstable power sources are just some of the problems that may cause equipment to fail.
There is ample room for creativity and innovation in this area, particularly if design efforts involve potential
users from less-developed countries from the start as full-fledged co-designers, rather than merely as
consumers—and even that, as an afterthought.
Projects and programs from organizations like Schools Online, Global SchoolNet, I*EARN, the Lewis Center
for Educational Research, Katha, the Omar Dengo Foundation, Computers for Youth, Enlace Quiché, and
Fahamu spend considerable effort in preparing the intended beneficiaries to be successful over the long term.
Learning and instructional technologies are rarely self-evident in their use, even at the operational level, and
much less so when it comes to their integration into daily classroom (learning spaces) practice, highlighting the
need for thoughtful training and ongoing professional development processes.
While the Internet is, indeed, a wonderful resource for information and resources, much of what is available is in
English and was created with an industrialized world audience in mind. Project Gutenberg, Equal Access, Katha,
Brij Kothari’s subtitling technology, and Design that Matters’ Kinkajou project are examples of projects that
allow free access to valuable content, much of it developed in close cooperation with representatives from the
intended audiences. Affordable software tools that allow users with limited technical skills to create quality
content in a variety of media (text, audio, graphic/photo, video, and multimedia), in their own and multiple
languages, are needed to encourage and promote creativity as well as knowledge creation and dissemination at
all levels.
Structures and systems
Most of the projects highlighted here, but particularly those based in or focusing their work in less-developed
countries, would benefit from having ready access to communities of practice with whom to share information
and resources that would enhance their work. In contexts where the lack of infrastructure means that getting
anything done requires much greater effort, the opportunity to learn from the experiences of others to avoid
making the same mistakes, and to do things better with the benefit of another’s experiences, would be a great
benefit for all social entrepreneurs.
In addition to the content-creation tools mentioned above, new software development tools that allow users and
designers to easily create innovative simulations and other types of rich learning materials are needed. Just as
CAST, the Center for Spoken Language Research, and the partnership between Baruch College’s Computer
Center for Visually Impaired People and Touch Graphics have created wonderful software for learners with
special needs, similar efforts addressing the needs of all other types of learners—and again, in a wide variety of
learning contexts, in languages other than English only—are needed.
Innovative hardware, software, systems, and organizational structures have the potential to bring the benefits of
educational technology to the large majority of people around the world who don’t have access to even simple
telecommunications (over half of the world’s population still has not made a telephone call; UNDP, 2003), and
who live in countries where education and learning are mired in antiquated modes and structures that don’t serve
their needs adequately or at all.
As Tipson & Frittelli (2003) remarked, “Yet, enormous opportunities are being lost in the developing world from
failure to capitalize on what is available in anticipation of what is (realistically) not” (p.8). One can easily
imagine a utopian future where all social entrepreneurs are “out of business” because the world’s problems have
been solved. In the meantime, the work of the 25 Education Laureates of the Technology Benefiting Humanity
awards program must be recognized and encouraged.
The following Laureates were honored with the $50,000 prize: 2001—Freeplay Foundation; 2002—Barefoot
College; 2003—Equal Access; 2004—Andrew Liberman of the Asociación Ajb’ atz’ Enlace Quiche; 2005—
MIT Open CourseWare.
Basic Education Coalition. (2004). Teach a child. Transform a nation. Washington, DC: Author.
Cardwell, D. S. L. (1995). The Norton history of technology. New York: Norton.
Cuban, L. (1986). Teachers and machines. The classroom use of technology since 1920. New York: Teachers
College Press.
Cuban, L. (2001). Oversold and underused: Computers in the classroom. New York: Teachers College Press.
Easterly, W. (2001). The elusive quest for growth. Economists’ adventures and misadventures in the tropics.
Cambridge, MA and London: MIT Press.
Ferneding, K. A. (2003). Questioning technology: Electronic technologies and educational reform. New York:
Peter Lang Publishing.
Fisher, C., Dwyer, D. & Yocam, K. (Eds.). (1996). Education & technology. San Francisco: Jossey-Bass.
Fosnot, C. T. (Ed.) (2005). Constructivism: Theory, perspectives and practice. (2nd. Ed.) New York: Teachers
College Press.
Fulton, K. (1997). Learning in a digital age: Insights into the issues. The skills students need for technological
fluency. Los Angeles: Milken Exchange on Educational Technology.
Hepp, P., Hinostroza, E., Laval, E. & Rehbein, L. (2004). Technology in schools: Education, ICT, and the
knowledge society. Temuco, Chile: Instituto de Informática Educativa.
Hernández-Ramos, P. (2003). The NASDAQ Stock Market Education Award. STS Nexus 4 (1), 16-20.
Hernández-Ramos, P., Koch, J. L., Dommel, H. P., Guerra, P. & Bowker, G. (2005). Designing an online
collaboratory for the Global Social Benefit Incubator. Paper submitted to the 2005 E-Learn Conference,
Vancouver, BC, Canada.
Hernández-Ramos, P., Soukup, P. & McAnany, E. (2001). Educational technology in the service of global
leaders. NASDAQ Stock Market Education Award Finalists. STS Nexus (2) 1. Available online at:
Jonassen, D., Peck, K. & Wilson, B. (1999). Learning with technology: A constructivist perspective. Upper
Saddle River, NJ: Prentice Hall.
Joyce, M. V. (2005). Scaling up success: Technology-based educational innovations— An HGSE news
interview with Chris Dede and James Honan. HGSE News, May 1. Available at:
Koch, J. L., Coppock, K., Guerra, P. & Bruno, A. V. (2004). The global social benefit incubator: Toward a new
pedagogy for “scaling” in social benefit entrepreneurship. STS Nexus 5 (1). Available online at:
Kramer, M. R. (2005). Measuring innovation: Evaluation in the field of social entrepreneurship. Palo Alto, CA:
Skoll Foundation/Foundation Strategy Group.
Mangan. K. S. (2004). Teaching business skills to people with a social mission. Chronicle of Higher Education
51 (10), 40a. Available online at: http://chronicle.com/weekly/v51/i10/10a01001.htm
Osin, L. (1998). Computers in education in developing countries: Why and how. Education and Technology
Series, Vol. 1 (3). Washington, DC: The World Bank Human Development Network-Education GroupEducation and Technology Team.
Prahalad, C. K. & Hammond, A. (2002). What works: Serving the poor, profitably. New York: Markle
Foundation. Available online at: http://www.markle.org/news/What_works_Servingthepoorprofitably.pdf.
Raphael, C. (2004). The Microsoft Education Award. STS Nexus 5 (1). Available online at:
Raphael, C. (2005). The Microsoft Education Award. STS Nexus 6 (1), 28-34.
Ringstaff, C. & Kelly, L. (2002). The learning return on our educational technology investment: A review of
findings from research. San Francisco: WestEd.
Soukup, P. (2002). The NASDAQ Stock Market Education Award. STS Nexus 3 (1). Available online at:
Tipson, X. & Frittelli, C. (2003). Global Digital Opportunities. National Strategies for “ICT for Development.”
New York: Markle Foundation.
United Nations. (2005). UN Millenium Development
United Nations Development Programme (UNDP). (2003). Human development report 2003. Millenium
development goals: A compact among nations to end human poverty. New York and Oxford: Oxford University
Nevile, L., & Treviranus, J. (2006). Interoperability for Individual Learner Centred Accessibility for Web-based Educational
Systems. Educational Technology & Society, 9 (4), 215-227.
Interoperability for Individual Learner Centred Accessibility for Web-based
Educational Systems
Liddy Nevile
La Trobe University, Bundoora, Victoria, Australia
Tel: +61 4 1931 2902
[email protected]
Jutta Treviranus
University of Toronto, Toronto, Ontario, Canada
Tel: +1 416 978 5240
[email protected]
This paper describes the interoperability underpinning a new strategy for delivering accessible computerbased resources to individual learners based on their specified needs and preferences in the circumstances in
which they are operating. The new accessibility strategy, known as AccessForAll, augments the model of
universal accessibility of resources by engaging automated systems and builds upon the previous
development of libraries of suitable resources and components. It focuses on individual learners and their
particular accessibility needs and preferences. It fits within an inclusive framework for educational
accommodation that supports accessibility, mobility, cultural, language and location appropriateness and
increases educational flexibility. Its effectiveness will depend upon widespread use that will exploit the
‘network effect’ to increase the content available for accessibility and distribute the responsibility for the
availability of accessible resources across the globe. Widespread use will depend upon the interoperability
of AccessForAll implementations that, in turn, will depend on the success of the four major aspects of their
interoperability: structure, syntax, semantics and systemic adoption.
E-learning systems, Interoperability, Accessibility, AccessForAll, Learner profiles, Resource descriptions
This paper describes the effort to give effect to the interoperability goals of a new strategy for delivering
accessible computer-based resources to learners based on their immediate specific needs and preferences. There
are many reasons why learners have different needs and preferences with respect to their use of digital resources,
including because they have disabilities. Instead of classifying people by their disabilities, the ‘AccessForAll’
approach emphasizes the resulting needs in an information model for formal structured descriptions of those
needs and the resources available. It provides a common language for describing the needs and preferences of all
people, for whatever reason their accessibility is hampered, when they have special requirements. It may be
useful to those who are simply trying to gain access to information in a language other than one they
comprehend. It will be essential for those with very restricted access capabilities.
AccessForAll uses the same formal, structured information model for describing the needs and preferences of
users and the accessibility characteristics of resources. Both sets of descriptions are required so resources can be
matched to the needs and preferences of the learner. The goal is to enable systems to share the benefits and
burdens of making resources accessible. It must be easy to record the necessary information and it must be in a
form that will make it most useful and interoperable.
There is no doubt that an important aspect of achieving interoperability is the widespread adoption of common
solutions to problems. The new framework aims, where possible, to inherit this from extensively used
specifications and standards. In addition, it will depend on the widespread availability of suitable resources,
appropriately described. In the case of educational resources and services, there are many major communities
concerned with relevant aspects of descriptive standards and of those, a number have been engaged in the
development of the AccessForAll model. Cross-domain metadata also has well-established standards that have
been considered. The model is based on a set of principles that, when implemented in a variety of standard
languages or systems, should maintain their interoperability at structural, syntactic and semantic levels. It also
depends upon widespread systemic adoption to generate the required volume of information about available
accessible components.
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
The AccessForAll strategy complements work about how to make resources as accessible as possible
(universally accessible) done primarily by the World Wide Web Consortium Web Accessibility Initiative
(W3C/WAI). The focus of that work is technical specifications for the representation and encoding of content
and services, to ensure that they are simultaneously accessible to as many people as possible. W3C also develops
protocols and languages that become industry standards to promote interoperability for the creation, publication,
acquisition and rendering of resources.
Similarly, the strategy complements the work of organizations such as the Royal National Institute for the Blind
(RNIB), The Library of Congress National Library Service for the Blind and Physically Handicapped (NLS), the
Speech-to-Text Services Network and similar institutions in other parts of the world.
In the United Kingdom, the RNIB developed and maintains the National Union Catalogue of Alternative
Formats (NUCAF) for people with vision disabilities (Chapman, 2002). The American NLS maintains a Union
Catalogue (BPHP) and a file of In-Process Publications (BPHI) that can both be searched via the NLS website.
The Speech-to-Text Services Network (STSN) makes accessible content alternatives for content that cannot be
used by people with hearing disabilities. They offer three types of services based on the technology used: Steno
machine-based systems, commonly called CART (Communication Access Realtime Translation), Laptop-based
speed typing software systems (C-Print and TypeWell) and Laptop-based Automatic Speech Recognition
software systems (e.g., CaptionMic, iCommunicator).
Such organizations provide access to resources through catalogues but as most of them pre-dated the Web, and
certainly the widespread use of metadata as it is used today, their records are not in a common form and they do
not all contain the same information in the same structure or syntax. Their catalogues provide search facilities of
the type upon which the new AccessForAll strategy needs to be built.
The focus of AccessForAll is ensuring that the composition of resources, when delivered, is accessible from the
particular learner’s immediate perspective. It complements the W3C work by enabling a situation where a
particular suitable resource is discoverable and accessible to an individual learner even when it may not be
accessible to all learners. In some cases, this may mean discovery and provision of alternative, supplementary or
additional resource components to increase the accessibility of an original resource. In such a case, those
institutions that already have suitable resources may be able to lead to their discovery. The distinguishing feature
of AccessForAll is that it depends upon software to manage the process of determining if the resources are
suitable and, if not, replacing or augmenting them with the necessary resources. It assembles distributed,
sometimes cumulatively-created, content into accessible resources and so is not wholly dependent upon the
universal accessibility of the original resource and goes farther than mere discovery of a suitable resource.
The AccessForAll specifications, while initiated in the educational community, are suitable for any user in any
computer-mediated context. These contexts may include e-government, e-commerce, e-health and more. Their
use in education will be enhanced if they are adopted across a broad range of domains and used to describe the
accessibility of resources available to be used in education even if that was not their initial purpose. The
AccessForAll specifications can be used in a number of ways, including: to provide information about how to
configure workstations or software applications; to configure the display and control of on-line resources; to
search for and retrieve appropriate resources; to help evaluate the suitability of resources for a learner, and in the
sharing and aggregation of resources.
The AccessForAll specifications are designed to gain extra value from what is known as the ‘network effect’: the
more accessible content becomes available, the more likely accessible components will be available for
individual learners; the more people use the specifications, the more there will be opportunities for interchange
of resources or resource components, and the more opportunities there are, the more accessibility there will be
for learners.
Early versions of the specifications were developed by the Adaptive Technology Resource Centre (ATRC), at
the University of Toronto. They were developed further by collaboration between IMS Global Learning
Consortium; the Dublin Core Metadata Initiative Accessibility Working Group, and others. They have been
considered formally by CEN ISSS MMI-DC, CEN ISSS (WS-LT) and ISO JTC1 SC36 as well as at the national
level by government agencies in Canada, the UK, and Australia.
Provision of Networked Accessible Resources
There are a number of approaches to making networked resources accessible, whether on the Internet or on an
The first and most common approach is to aim to create a single resource (Web site, Web application) that meets
all the accessibility requirements. Such a resource is known as a universally accessible resource. While this
approach would work well in many situations, it is not often that authors succeed in making their resources
‘universally accessible’, especially when they contain interactive components. As well, it is possible to make a
resource that satisfies the guidelines but is, in fact, not accessible to everyone. There are also potential problems
due to lack of attention to usability principles that may account for lack of satisfactory access (Disability Rights
Commission (UK), 2004). The DRC rigorously tested sites that had passed the W3C accessibility tests by asking
people with disabilities to use them and, in this process, followed standard usability testing techniques.
In their report, the DRC said that:
2.2 Compliance with the Guidelines published by the Web Accessibility Initiative is a necessary
but not sufficient condition for ensuring that sites are practically accessible and usable by
disabled people. As many as 45% of the problems experienced by the user group were not a
violation of any Checkpoint, and would not have been detected without user testing.
FINDING 5: Nearly half (45%) of the problems encountered by disabled users when attempting
to navigate websites cannot be attributed to explicit violations of the Web Accessibility Initiative
Checkpoints. Although some of these arise from shortcomings in the assistive technology used,
most reflect the limitations of the Checkpoints themselves as a comprehensive interpretation of
the intent of the Guidelines.
(The authors of the guidelines, W3C, have recognized these short-comings and are in the process of producing a
revised set, Web Content Accessibility Guidelines Version 2 (W3C/WAI WCAG 2.0), that should better match
their intentions.)
Indeed, a resource may be accessible to everyone, but optimal for no one. Unfortunately, authors concerned
about accessibility often avoid resource components that may be very effective, entertaining or efficient for some
learners, for fear they will not be accessible to all learners (Macromedia’s Flash application is a typical
The second approach used by a number of educational content providers is to create two versions of a resource: a
media rich version and an “accessible version,” the latter being stripped of all media that may cause accessibility
problems. Typically these alternative versions are available as ‘text only’ versions for selection by a user.
UsableNet have developed a Lift Text Transcoder (UsableNet) that dynamically produces text versions of pages.
This was done to avoid the mismatch of maintenance of pages where the pages were not automatically generated
and content providers would up-date their media-rich pages but forget to do the same for their text-only versions.
In many cases, media-rich pages are made available as text-only pages by conversion of the text and thus a lot of
valuable content is simply not available to those who cannot access the media objects.
While the provision of text-only pages solves some of the problems with the first approach, it can also cause
other problems. In some cases, learners with disabilities only get an out-of-date view of the information but more
often, students who perhaps need more assistance get less because they are using an impoverished version of the
resource. The idea that learners with disabilities are a homogenous group that is well served by a single bland
version of a resource is flawed.
The third approach differs from the first two in a number of ways. Accessibility requirements are met not by a
single resource but by a resource system. Rather than a single resource or a choice between two resource
configurations, there can be as many configurations as there are learners. The ability of the computer mediated
environment to transform the presentation, change the method of control, to disaggregate and re-aggregate
resources and to supplement resources is capitalized upon to match resource presentation, organization, control
and content to the needs of each individual learner. The availability of automated services as described makes it
feasible for all learners to consider their needs and preferences and to record them for use by such a system. An
approach that is inclusive in this way significantly increases the multiplicity of presentations available from a
given source and thus shifts the responsibility for personalisation of presentations from the content provider, as
designer of presentations, to the user. It also decreases the effort required on the part of individual content
creators to anticipate the necessary combinations. It is an approach that lets the user select their desired content
rather than one that depends on the website designers’ best guesses at the options that may be required.
Extending the range of presentations of content as proposed by the third approach, the AccessForAll approach
literally means moving towards mass customization of content provision. It is not merely relevant in the context
of people with special needs, or disabilities, but for everyone. It is applicable in a range of contexts as it will lead
to highly personalized learning experiences, on demand learning and content assembly and publishing,
automated generation of content, and more. The availability of the information about the needs and preferences
of the user is the key feature here. Many existing systems provide descriptions of content, and many provide
customized assemblies, but the new approach provides for these to be user driven.
The notion of universality of personalization, of inclusiveness as a characteristic of learning resources for
everyone, differentiates the AccessForAll approach from the more traditional universal accessibility of resources
approach, described above. The goal is not new, and has been considered many times, but it is contended that the
new approach to user needs and preferences management actually makes it possible and, perhaps then, a
necessary response to the needs of learners if all learners are to be catered for equally.
The possibility is not of great value, however, unless the costs associated with this approach are clearly
appropriate. Given that accessibility is a requirement, not just a nice thing to do, there are already significant
costs to be borne by many content providers. The third approach lessens the need for all resources to be
developed in a strictly universally accessible way just in case they are required. It also allows for the resources to
be provided to individual users to be distributed, meaning that they can be shared amongst content providers.
This is not new in itself, but it is new that this sharing can be dynamically applied. If a content provider has not
made some content fully accessible, another can make available the necessary extra content and the system can
combine the components, so the authoring of accessible content becomes more of a distributed, cumulative
activity. Although there is no data to prove that this is more efficient and cheaper, there is reason to believe it
will be and research to test this can take place as soon as there are sufficient metadata and systems available for
such testing to be undertaken.
Describing Control and Display Requirements for interoperability
One aspect of interoperability is the ability to share the same kind of information with others using the same
systems and acting with the same goals. Another is to work across devices including using different hardware
and software without losing the necessary ‘look and feel’ that facilitates learner mobility between devices.
W3C has a Device Independence Working Group, another focused on the Mobile Web and a third working on
Evaluation and Repair. All three Working Groups produce specifications that are important to the
interoperability of AccessForAll. These groups do not work specifically on specifications for learners but
learners are included in the user groups of concern to them.
The Device Independence Working Group is responsible for a protocol known as Composite Capabilities and
Personal Preferences (W3C CC/PP). Its general aim is to enable the quick identification of the needs of a device,
such as a mobile phone or a desktop computer, so that user requests will initiate delivery of resource components
that are necessary for the correct functioning of the device. A typical example is that when a large display page
such as a newspaper page is to be presented on a tiny phone screen, it needs to be accompanied by a
transformation that will facilitate navigation through the content without relying on a global view of it. Another
important example is given when the mode of transmission of the content has to change to accommodate the
device or user. The Device Independence Working Group, like all other W3C groups, is directed to take into
account the accessibility aspects of this, and so Personal Preferences are included. The resulting CC/PP protocol
provides an interoperable way of automatically transmitting information to Web servers from devices. This is
what AccessForAll wants to do and so it is important that AccessForAll is in harmony with the CC/PP protocol.
The W3C Mobile Web Working Group wants to ensure that information about users is available for devices
wherever they are but is concerned that such information is not available for abuse by other people or agents.
This requirement is one that is shared by the AccessForAll: there is no need for a person to be identified with or
by the set of needs and preferences they choose to use (Nevile, 2005).
The W3C Evaluation and Reporting Working Group have developed a language for managing the evaluation of
content, typically the validation of it against formal test criteria. It enables the evaluations to be both distributed
and cumulative, and at least identified with their evaluator and the time and date of evaluation. AccessForAll
descriptions of learners’ needs and preferences, and resources’ accessibility characteristics, need to be trusted
and this information is therefore of interest in their case too.
Describing Content Needs and Preferences and Resources for Exchange
For a network delivery system to match individual learner needs with the appropriate configuration of a resource,
two kinds of descriptions are required: a description of the individual learner’s needs and preferences and a
description of the resource’s relevant characteristics. Experience has shown that by catering for needs, everyone
can benefit, as has been the case with curb-cuts that are now used by people in wheel-chairs, children on skateboards and men with heavy trolleys.
The AccessForAll approach involves specifications for describing learner needs and preferences that define a
functional description of how a learner prefers to have information presented, how they wish to control any
function in the application and what supplementary or alternative content they wish to have available. It was
found that these were the three main classes of such needs that should be in the descriptions. Specifically:
¾ display requirements usually include the use of screen readers or enhancers, tiny phone displays, reading
highlights, Braille, tactile displays, visual alerts and structural presentation;
¾ control requirements usually include the use of keyboard enhancements, onscreen keyboards, alternative
keyboards, phone keypads, mouse emulation, alternative pointing, voice recognition and coded input, and
¾ content requirements usually include the use of alternatives to each of the modes of display (auditory, visual,
tactile and what is classed as textual). It includes learner scaffolds, personal style sheets and extra time.
AccessForAll deals with the needs and preferences of learners. It might seem unnecessary to be concerned about
preferences at this level, but for some learners with disabilities, they have very limited means to access
resources, and it may be essential to them that they have their exact needs satisfied. It is necessary to distinguish
them from learners who have the capability to use other systems but prefer a particular set. It would be
inappropriate to limit the acquisition of resources for the more flexible learners just because their first preference
was not satisfied. There is also a third situation that needs to be considered; some learners can have dangerous
conditions induced by certain sets of features, such as when flashing content causes them to have epileptic fits.
(Many others are distracted by these and wish to avoid them, if possible.) For this reason, the three classes of
essential, preferred, and prohibited need to be available to qualify the requirements.
The AccessForAll approach requires finer than usual details with respect to embedded objects and for the
replacement of objects within resources where the originals are not suitable on a case-by-case basis. Embedded
objects include components such as images that appear integral to a Web page or resource. It is important that
there is no significant difference between the handling of embedded and distributed components. Distributed
components are those that form part of the Web page or resource but do not come from the same source as other
components. Knowing which component is being rearranged in the AccessForAll process and having general
information about the component will often be necessary when a system searches for an alternative to that
component. In some cases, the creator of the appropriate alternative will not know of the existence of the original
and certainly not have related the alternative to it in any way. This may mean that sometimes it will be necessary
to determine the source of a component so that alternatives can be found by reference to it. For example, when
looking for a video of signed language for use as an alternative to the sound track of a film of the play Hamlet, it
may be possible to find a sign-language presentation of the play. Managing such a discovery challenge will
benefit from interoperability.
In defining requirements, AccessForAll does not mention the reason for any of the requirements. In some cases,
learners with disabilities use assistive technologies to emulate other technologies, such as when a head-pointer is
used to emulate the standard mouse so that as far as the functioning of the computer and the resource is
concerned, there is no special accommodation. When the assistive technology impacts other technologies, as
happens when a section of a screen is used for an on-screen keyboard, there are often detailed requirements not
only for the display functions but also for the keyboard itself. Some learners need to specify the attributes of
their keyboards, such as the size and separation of the keys, and others want to take advantage of features of the
keyboard software they are using. Another kind of problem arises when a learner who has previously used a
resource on a desktop computer tries to continue to use the resource on a telephone screen.
The required interoperability happens at many levels.
Interoperability between Needs and Preferences and Resources
If learners are to be able to quickly configure their devices, they require their needs and preferences to be quickly
recognized and implemented by the device they are using. Also, if they are to search for appropriate resources
(including where their search for resources causes their system to search for accessible components from which
to make the resource they want), their needs and preferences descriptions have to be available to the search
engine for searching and matching with the resources and their components. Where this is happening across
collections of resources, a common way of describing the resources will be necessary and it will need to mirror
the descriptions of the resources. So interoperability between the two sets of descriptions is necessary so that,
even though one is concerned with the user’s needs and the other with a resource, they can both be used by
search engines. When a request is made for a resource, the search engine may need to look for both the user’s
needs and preferences and the resource. Both will be necessary to constrain the search results to appropriate
resources and their components. In effect, this means that for simplicity, the description of the learner’s needs
should be in the same format as the descriptions of the resources.
Interoperability between Devices
Typically, learners with special needs will be looking for resource components that are developed by specialists.
Usually, specialists who have not made the original resources produce closed captions, image descriptions and
video files of people signing. They are likely to know the standard assistive technologies and what they will
require and can do to use the special components. In automating the matching process for the learner, it is very
important that the standard triggers are available for the assistive technologies. This means that the resources
should be described in the way they can be understood by particular assistive technologies but also so that there
is a generic description specification that all the assistive technologies can be expected to refer to. For this
reason, care has been taken in AccessForAll to ensure that there is a seamless match and the established industry
terms are used. The implications for interoperability here are for exchange between systems known as ‘user
agents’, including browsers.
It is well known that browser developers pride themselves on the non-standard features they offer and that it is
not easy to satisfy all browser specifications simultaneously. Fortunately, assistive technology developers who
have a much smaller market are often more concerned to serve their customers and their industry associations.
Nevertheless, it is important to recognize their differences and allow for their use so the AccessForAll model has
to be capable of such flexibility. In fact it aims for some generic functions to be described in a common way
while allowing for extensions to accommodate custom functions or features.
Interoperability across Information Sectors
Many, but not all, resources used within educational institutions are described by or for the educational
community, usually according to standards designed for the educational community. Having worked with the
goal of sharing resources for some time now, the many educational communities have a number of ‘standards’,
the best-known being the LOM developed by IEEE (the IEEE 1484.12.1-2002 Standard for Learning Object
Metadata). Clearly, the accessibility characteristics of resources that are ‘learning objects’ need to be described
in a way that interoperates with all other aspects of LOM descriptions.
The IMS Global Learning Project, a consortium of educational organizations, has adopted the LOM and added
the Learner Information Profile (LIP) that describes the attributes of the learner. AccessForAll was an IMS
initiative and so it closely resembles their versions of the LOM and LIP in order to operate as an extension of
them. In fact, the IMS versions are known as AccLIP and AccMD (IMS Global Learning Consortium, 2005).
Often, however, educational activities involve learners using resources that have been developed and described
by other communities for their own purposes. For example, technical manuals are often used in Computer
Science courses but they are not usually written for this purpose. Government information is often used in
education, as are images of paintings and objects held in museums and galleries. The resources to be used by
learners then, do not always originate from the educational or even the same communities and their description
for discovery purposes can be very specific to the community from where they come. In order to discover
resources across communities, or disciplines then, the descriptions of the accessibility characteristics of
resources need to be consistent with descriptions used in those communities.
The simple Dublin Core Metadata Element Set (DCMES) is an ISO standard (ISO 15836) for core resource
description metadata. There is also a set of qualified Dublin Core elements with additional terms and extensions
(DC Metadata Terms). Dublin Core metadata is not domain specific. The Dublin Core Metadata Initiative
(DCMI) has worked on cross-disciplinary metadata (DC metadata) for resource description and provides a base
set of descriptive elements that support cross-domain discovery. Where domain specific communities want to
extend the Dublin Core set, they develop what are known as Application Profiles for their community (DCMI,
2003). Given that governments, museums and galleries, and other sectors use DC metadata for information
sharing, AccessForAll aims to take advantage of the interoperability of DC descriptions. DC metadata also has
the advantage that it is used in many countries for resources that are created in many different languages and so
can be used for cross-language discovery.
DC metadata is for describing resources and does not support rich descriptions of people but as people are not
described in the learner needs and preferences profile, this is not a problem. It would be necessary, however, for
there to be DC usable descriptions of learners’ needs and preferences but, as stated above, these need not be
Dublin Core descriptions although, as argued elsewhere, they could be DC-style descriptions and described as
DC resources to allow for their discovery (Nevile, 2005).
Interoperability across Information Models
DC Metadata and IEEE LOM and IMS LIP metadata belong to very different information models. DC metadata
has a ‘flat’ structure with a set of ‘tags’ that can be applied individually or collectively to resources. IEEE LOM,
and therefore IMS LIP, models are very different. IEEE LOM (and IMS LIP) structure is deeply hierarchical.
The difficulty of matching information from ‘flat’ to ‘heirarchical’ models, and the reverse, has separated the
two communities for some time. Recently there have been agreements to work towards a common structure in
the future, and some attempts to find suitable temporary solutions for the present.
The DC and IEEE LOM communities do share the use of standard syntax including HTML, XHTML,
eXtensible Markup Language (XML), in particular, and the more narrowly defined Resource Description
Framework XML (RDF). Both XML and RDF are extensible, however, so that they are used by two systems
does not necessarily mean very much in terms of functional interoperability. On the other hand, it is this
extensibility that makes it possible for the best attempts at interoperability.
In summary, AccessForAll needs to interoperate with a number of other relevant metadata specifications and
The History of the AccessForAll Interoperability Efforts
Originally, the AccessForAll descriptions were developed for an in-house application. The original prototype
Inclusive Learning Exchange (TILE), was a system that chose among a set of resource components according to
the learner’s needs and preferences. The resources were treated at the atomic level, each image and chunk of
text, such as text articles, captions for other rich media resources, and so on, being separately stored and
described. The learner’s needs and preferences information was set within the system and variable at any time
the system was being used. The descriptive information was related in a hierarchical decision-tree that allowed
for different levels of detail appropriate for the circumstances. The resource components in the learning system
were described so that their potential for matching was available to the application that chose and assembled the
components to be delivered as the resource. There was no need for interoperability with other systems in this
case as the system was in-house and contained all the resources, components and user profiles.
The original specifications for TILE were adopted by the IMS Global Learning Project with the aim that they
would become interoperable by extending the IEEE LOM and the IMS LIP in a consistent way. They were
dependent upon their common expression in an XML binding, inherited from TILE, for interoperability.
The IMS Accessibility Special Interest Group sought permission to work towards an open standard with
participation from others (including the DCMI Accessibility Working Group) in order to increase the
opportunities for widespread adoption and interoperability. This collaboration led to the deconstruction of the
original AccessForAll model and a new construction, without loss of particulars, as an abstract model (see
below). During this process, it was crucial to the success of the outcome that it would not compromise its
relationship with the LIP or the LOM.
Before the adoption by DCMI of AccessForAll as a recommendation to the DC users community, and in
anticipation of a revised, interoperable IMS version of the AccLIP and AccMD, the IMS specifications were
forwarded to the International Standards Organisation (ISO JTC1 SC36) for adoption as a multi-part
international standard. The ISO process involves the support and participation of a range of nations and helps
foster international adoption of the specifications. According to the agreement by which the specifications were
contributed by IMS, unlike many of the ISO standards, they will be free to those who want to use them.
Enabling Interoperability with Abstract Models
Between them, IEEE LOM and DC metadata describe many of the resources that are of interest in education.
Some educational systems use LOM-based metadata sets to describe their resources but others use DC-based sets
and others use a combination or different metadata. Although it is not clear how many do what, there has been
careful analysis of the use of IEEE LOM and it is clearly an important player as is DC metadata, particularly as
the latter is often used for government or document-type resources (Godby, 2004). It makes sense that these two
communities should be able to exchange metadata records about their resources so they can, in fact, share their
resources. To do this, they need to be able to transform metadata from one specification to the other. There is an
activity, started in 2001 by the production of a joint article (Duval et al, 2002) that aims to bring the two sets of
specifications into harmony. It cannot be done easily because LOM and DC metadata are based on very different
information models.
In 2004, to clarify the nature of DC metadata, the DCMI developed an abstract model of DC metadata (DCAM)
that was adopted in early 2005. The model defines the scope and functions of DC metadata that enables its
implementation using a range of syntaxes without loss. Where the syntax is standard, as where standard XML or
RDF is used, both the structure (defined in the abstract model) and the syntax will be interoperable across all DC
metadata applications. Such applications are formally known as application profiles (CWA14855).
Once abstract models are agreed, it is relatively easy to develop information models (instantiations of the
abstract model) and to create the syntax for representation of the information and then it is up to the community
to agree on the terms to be used (semantics) for complete technical interoperability. The DCMI community
developed their abstract model ten years after their information model, but fortunately for them, had a relatively
simple information model. The IEEE LOM information model is extensive and complex and so far there is no
corresponding abstract model for it.
DCMI introduces its abstract model as follows:
This document specifies an abstract model for DCMI metadata. The primary purpose of this
document is to provide a reference model against which particular DC encoding guidelines can
be compared. To function well, a reference model needs to be independent of any particular
encoding syntax. Such a reference model allows us to gain a better understanding of the kinds of
descriptions that we are trying to encode and facilitates the development of better mappings and
translations between different syntaxes. (DCMI, 2005)
In order to be recommended by the DC Usage Board for general adoption by the DC community, AccessForAll
metadata needed to be explained in DC terminology. In effect, by late 2005, this meant it had to comply with the
DC Abstract Model. As AccessForAll did not have an abstract model, developing one seemed like a good
exercise. In the end, it was a very constructive exercise.
The AccessForAll Abstract Model
Developing abstract models is not easy when what they are to represent is already defined but in the case of
AccessForAll, there was the opportunity to redefine the AccessForAll information model concurrently with
developing its abstract model. The main problem was finding an abstract way to define AccessForAll that would
maintain interoperability with all the relevant information models (structure), syntax (easier), semantics
(relatively simple), and encourage systemic adoption.
The AccessForAll Framework includes, significantly, both the user’s needs and preferences profile and the
resource profile.
In presenting its abstract model, DCMI provides a set of definitions and rules followed by a diagram expressed
in a formal graphical representation format known as Unified Modeling Language (UML). The AccessForAll
abstract model is now similarly described.
To simplify the scope of the AccessForAll and to increase interoperability with other metadata, only those things
that are necessary for the AccessForAll process are contained in its abstract model. Other descriptions should be
managed according to other relevant metadata specifications.
As unique definitions and rules, AccessForAll has the following:
The abstract model of the preference sets defined in the AccessForAll framework is as follows:
¾ Each person has zero or more needs and preferences.
¾ Each needs and preferences has zero or more adaptability statements.
¾ Each needs and preferences has zero or more contexts.
The abstract model of the resources defined in the AccessForAll framework is as follows:
¾ Each resource meets the needs and preferences (of a person).
¾ Each resource has zero or more adaptability statements.
¾ Each resource may be related to zero or more alternative resources.
The abstract model of the adaptability statements defined in the AccessForAll framework is as follows:
¾ Each adaptability statement may contain access mode information.
¾ Each adaptability statement may contain zero or more related descriptions. [DC-AWG, 2005]
Figure 1: A graphical representation of the AccessForAll abstract model.
Figure 1 shows that a user, a person (a learner) or agent (perhaps a device or application), has needs and
preferences that can be expressed as a resource, their Personal Needs and Preferences (PNP). This PNP is, or is
closely associated with, an adaptability statement. It should describe the access mode they need or prefer. It may
also have other descriptions (metadata) to make it easily discoverable and easily differentiated from other PNPs
they use. Often a user will refer to the PNP they want simply by using a name for the context in which they use
it. In some cases, the provision of needs and preferences information has already been tried on (Web-4-All)
smart cards that can be used to provide some of this information.
When the learner searches for a resource using the AccessForAll process, they want its adaptability statement, its
Digital Resource Description (DRD), matched to their PNP adaptability statement. When a resource does not
satisfy this rule, existing or alternative components or resources may need to be substituted, augmented or
transformed until it does. As with the PNP, the resources they seek may have other descriptions but these,
although useful, are not within the scope of the AccessForAll abstract model, except in as much as they may
provide the context description.
Adaptability statements can be very simple, merely showing a single attribute of the resource, or they can be
quite complex, using the refinement of a refinement model to capture hierarchical information.
Interoperability with DC Metadata
The AccessForAll abstract model now closely matches the Dublin Core Abstract model. Not everything that will
be useful to have as AccessForAll metadata is unique to the AccessForAll model so a significant amount of
information will be expressed using standard DC elements. Exactly how to do this will be described in what is
known as a DC Application Profile for which specific terminology (semantic values) will be defined. The value
of this work for DC users is that they will be able to express the AccessForAll metadata in DC compliant ways
so it will interoperate with other DC metadata. They will also be able to use standard DC applications without
significant modification.
Interoperability with IEEE LOM and IMS LIP Metadata
An encoding of AccessForAll metadata for use in an IEEE LOM Application is under construction by the CENISSS Learning Technologies Workshop (CEN/ISSS (WS-LT), Accessibility Properties for Learning Resources
Nilsson et al. (2005) have recently worked on developing what they call a future metadata standardization
framework. They say:
We have demonstrated that true metadata interoperability is still, to a large extent, only a vision,
and that metadata standards still live in relative isolation from each other. The modularity
envisioned in application profiles is severely hampered by the differences in abstract models
used by the different standards, and efforts to produce vocabularies often end up in the dead end
of a single framework. In order to enable automated processing of metadata, including
extensions and application profiles, the metadata will need to conform to formal metadata
To achieve this, there is a need for a radical restructuring of metadata standards, modularization
of metadata vocabularies, and formalization of abstract frameworks. RDF and the Semantic Web
provide an inspiringly fresh approach to metadata modeling: it remains to be seen whether that
framework will be reusable for learning object metadata standards.
This suggests that it may not be until there is a shared LOM/DC abstract model for education that there will be
perfect interoperability between DC and LOM resource descriptions but it is hoped to be possible sooner, in the
particular case of AccessForAll metadata, because it is based on a more interoperable abstract model.
Adoption of specifications and standards
The value of specifications and standards is only known some time after their release, when it becomes possible
to gauge how widely they have been adopted and how well they have solved identified problems for
organizations. Insurance that they will be adopted is not possible but open development of specifications,
consensus among a wide range of types of implementers, and points of contact that support dissemination of the
specifications are all known to help in the process.
In the case of AccessForAll, the needs and preferences have come from those who actually have them; people
who use technology in a wide range of situations, overcoming what often seem like insurmountable odds.
The needs and preferences are not new, they have been around since computers were first developed and are
tried and tested by their users. The provision of statements that describe needs and preferences is new. It is to be
done in a way that separates the user from the needs and preferences, while allowing them to have several sets.
Nevertheless, there is little doubt that this will not be a problem for many as already these settings have to be
entered into computer systems. In addition, there is more pressure on those providing education and training to
be more mindful of the special needs of their clientele, so they will be alerted to the need for such specifications
in the normal course of business, as well as by those specifically promoting these specifications. The only
difference for those already registering such needs is that they will be entered once and used many times. The
experience of several projects where such specifications have been stored on smart cards for re-use is wellknown in the field and has contributed to this work (Web-4-All).
The description of the accessibility characteristics of resources is new but it has been foreshadowed by the
development of applications that help organizations assess the accessibility of their resources, and to record their
characteristics in metadata. The difference is that rather than just do this for evaluation and penalty avoidance,
organizations can do this now to promote accessibility by preparing their collection of resources to interoperate
with others.
The last community that needs to be engaged with the specifications is the industry that develops the
technologies that are used for accessing resources. These players have been consulted and provide more or less
of the relevant features. They have indicated that what is proposed will work for them.
Finally, there is the question of who will use metadata anyway? If, as hoped, the distribution and accumulation
of accessibility solutions is enabled by the specifications, and those responsible for providing learning resources
are increasingly doing this through systems that work on metadata, it can only be hoped that they will be
adopted. Presentations, workshops, journal articles, and other forms of dissemination are already active but in the
end it will probably be good references from one person to another that will provide the final evaluation.
Future Work and Conclusions
In summary, the interoperability of the AccessForAll strategy is relevant in many circumstances. The process of
making AccessForAll has involved significant technical work and collaboration between representatives of
several communities, all with the goal of making education more accessible. The associated specifications show
how the AccessForAll strategy can be implemented. They are not prescriptive about the encoding that should be
used. Significantly, they are not prescriptive about what constitutes accessibility. The work that follows to
support the adoption and implementation of the AccessForAll approach will be the true test of its
interoperability: nothing is seriously interoperable unless people use it to interoperate. In addition, the network
effects, upon which ultimate success will depend, cannot be expected to show until there is widespread adoption
of the approach.
There are endless opportunities, given the model and strategy, to take further advantage of new technologies.
The Semantic Web offers one obvious technology that will be enabled by the AccessForAll approach. Already
the first AccessForAll specifications recommend using the Semantic Web Evaluation and Reporting Language
(EARL) so that the metadata will be as flexible and rich as possible. The range of other extensions includes
opportunities for valuable cross-lingual exchanges to suit learner needs as well as cross-disciplinary changes of
emphasis. Applications and Web services that transform resources or resource components to suit the needs of
users with cognitive disabilities is a huge area that has hitherto not received the attention it deserves. The
interoperability of specifications that purport to implement the new AccessForAll is clearly a major factor that
will determine its effectiveness.
The authors wish to contribute to the valuable work being done by others and welcome involvement in their own
work. They specially wish to acknowledge the contributions of collaborators Anastasia Cheetham, Martyn
Cooper, Andy Heath, Madeleine Rothberg and David Weinkauf.
AccessForAll: an Accessibility Framework: http://dublincore.org/accessibilitywiki/AccessForAllFramework.
CWA14855, CEN Workshop Agreement, Dublin Core Application Profile guidelines:
CEN/ISSS (MMI-DC), European Committee for Standardization, Meta-Data (Dublin Core) Workshop:
CEN/ISSS (WS-LT), Workshop on Learning Technologies:
DCAM, Dublin Core Metadata Initiative Abstract Model: http://dublincore.org/documents/abstract-model/.
DCMES, Dublin Core Metadata Element Set: http://dublincore.org/documents/dces/.
DC Metadata Terms: http://dublincore.org/documents/dcmi-terms/.
IEEE - 2002 Standard for Learning Object Metadata: http://ieeeltsc.org/.
IMS Global Learning Consortium: http://www.imsglobal.org/accessibility/.
ISO JTC1 SC36: http://jtc1sc36.org/.
NLS: http://lcweb.loc.gov/nls/.
Object Management Group, The Unified Modeling Language: http://www.uml.org/.
STSN: http://www.stsn.org/servicechart.html.
TILE, The Inclusive Learning Exchange: http://www.inclusivelearning.ca/.
UsableNet Lift text Transcoder:
W3C CC/PP, Composite Capabilities and Personal Preferences: http://www.w3.org/Mobile/CCPP/.
W3C EARL, Evaluation and Reporting Language: http://www.w3.org/TR/EARL10/.
W3C HTML, HyperText Markup Language: http://www.w3.org/TR/html4.
W3C RDF, Resource Description Framework: http://www.w3.org/RDF/.
W3C WAI, W3C Web Accessibility Initiative: http://www.w3.org/WAI/.
W3C WAI, Web Content Accessibility Guidelines for creating accessible Web pages:
W3C/WAI WCAG 2.0: http://www.w3.org/TR/WCAG20/.
W3C XHTML, Module-based XHTML: http://www.w3.org/TR/xhtml1.
W3C XML, Extensible Markup Language: http://www.w3.org/XML/.
Web-4-All: http://web4all.atrc.utoronto.ca/.
Chapman, A. (200). Library services for visually impaired people: a manual of best practice, Chapter 10,
retrieved 24 August 2006 from http://bpm.nlb-online.org/contents.html.
DC-AWG (2005). Dublin Core Accessibility Working Group, IMS Accessibility Working Group, ISO JTC1
SC36 WG7, AccessForAll: an Accessibility Framework retrieved 24 August 2006 from
DCMI, 2003. DCMI Usage Board Review of Application Profiles
retrieved 24 August 2006 from
DCMI (2005). Dublin Core Metadata Initiative Abstract Model, retrieved October 7, 2006 from
Disability Rights Commission (UK), (2004). The Web: Access and Inclusion for Disabled People, retrieved
October 7, 2006 from http://www.drc-gb.org/publicationsandreports/report.asp.
Duval, E, Hodgins, W. Sutton, S., & Weibel, S. L. (2002). Metadata Principles and Practicalities, in D-Lib
Magazine, 8 (4), retrieved 24 August 2006 from http://www.dlib.org/dlib/april02/weibel/04weibel.html.
Godby, J. (2004). What Do Application Profiles Reveal about the Learning Object Metadata Standard? retrieved
24 August 2006 from http://www.ariadne.ac.uk/issue41/godby/.
IMS Global Learning Consortium (2005). Accessibility for Learner Information Profile, and Accessibility Metadata, retrieved 24 August 2006 from http://www.imsglobal.org/accessibility/.
Nevile, L, (2005). Anonymous Dublin Core Profiles for Accessible User Relationships with Resources and
Services, Proceedings of the International Conference on Dublin Core and Metadata Applications, 67-78.
Madrid: Universidad Carlos III de Madrid.
Nilsson, M., Johnston, P., Naeve, A., & Powell, A. (2005). The Future of Learning Object Metadata
Interoperability. In Koohang, A. (Ed.), Principles and Practices of the Effective Use of Learning Objects,
Informing Science Press.
Choquet, C., & Corbière, A. (2006). Reengineering Framework for Systems in Education. Educational Technology &
Society, 9 (4), 228-241.
Reengineering Framework for Systems in Education
Christophe Choquet and Alain Corbière
LIUM Laboratory, University of Maine, IUT of Laval, 53020 Laval Cedex 9, France
[email protected]
[email protected]
Specifications recently proposed as standards in the domain of Technology Enhanced Learning (TEL),
question the designers of TEL systems on how to put them into practice. Recent studies in Model Driven
Engineering have highlighted the need for a framework which could formalize the use of these
specifications as well as enhance the quality of the developments. This paper deals with the opportunity for
the TEL community to adopt such a model to express and formalize the design organization and the
engineering process of a TEL system. This kind of model could provide a set of concepts allowing the
description of specifying, modeling and analyzing tasks one needs to perform for defining the negotiation
and the communication between actors in such a community. In a first part, we stress the need for the
instantiation of this framework so as to integrate recent results as well as taking into account the evolution
of software reengineering. In a second part, we propose two instances of this framework; the first concerns
the reverse engineering of a TEL system and the second its reengineering.
TEL engineering, TEL reengineering, TEL development process, Model driven engineering
The design of TEL (Technology Enhanced Learning) systems calls for the communication and sharing of
information about the users of these systems. In recent studies, the standards and specifications proposed by the
TEL community have provided interoperable languages for the specification of the learning session’s
progression, in order to enhance the communication between the actors of the development. These proposals are
mainly focused on the a priori specification of the system’s behavior. But (Wenger, 1987; Bruillard & Vivet,
1994; Hummel et al., 2004; Tchounikine et al., 2004) for instance, have shown that the design methodology of a
TEL system cannot be reduced to the specification but must also instrument the tasks of recording and analyzing
the uses of the system. Our objective is to merge TEL community results on standards and specifications, and
proposals made by the software engineering community. In this way, we aim to support the communication
between actors of development in a general reengineering framework, as Chikofsky (1990) for who
"reengineering, also known as both renovation and reclamation, is the examination and alteration of a subject
system to reconstitute it in a new form and the subsequent implementation of the new form".
Recent proposals of TEL standards and specifications lead to the adoption of new design practices and
methodologies, but they don’t provide designers with a set of concepts allowing the explicitation of the
organization and the implemented engineering. Some development solutions are provided by the Object Oriented
Software Engineering community, especially in the domain of software reengineering. With an empiric approach
first, the open source and free software communities have had to develop specific and effective tools which
support the structured communication around a design artifact. At the same time, the research community on
software engineering has provided methodological developments as well as a standardized and shared
vocabulary for software reengineering focused on the interoperability and reusability points of view. Chikofsky
(1990) for instance, proposed a first terminology dedicated to the definition of the tasks of reverse engineering
and reengineering. More recently, and based on this taxonomy, the European project ESPRIT 21975 (Bär et al.,
1999) has provided a set of patterns and pattern languages which define and standardize both methodological
concepts – such as “capture model details” or “negotiate a design” – and vocabulary – such as “process” or
The aim of this paper is to show how these results from the software engineering community could be used to
define a general framework, able to support a set of designers, considered as a community, involved in the
reengineering process of a TEL system. We propose to make an instance of the RM-ODP (Reference Model of
Open Distributed Processing) framework on the reengineering process of a TEL system. This reference model is
considered as a framework which coordinates and standardizes open distributed processes by the fine definition
of a set of concepts on engineering practices. Aims and motivations of this RM-ODP standard are detailed in the
document ISO/IEC 10746-1 (ISO/IEC 10746-1, 1998).
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
In a first part, we present the instantiation of the standard RM-ODP on the reengineering process of a TEL
system. Then, we provide two use cases of the model we have proposed. The first one highlights a reverse
engineering task where both an instructional designer and a software engineer are involved. The second one
shows how, during the reengineering of a TEL system, the communication between an instructional designer and
an analyst specialized in the interpretation of users logs could be supported.
A General Framework for the Reengineering of a TEL System
Reference Model of Open Distributed Processing Presentation
RM-ODP is defined by its authors as a generic framework aimed to support the modeling process of a complex
and distributed system. For this purpose, it stresses the need for the designers to refine within their own domain,
a set of generic concepts, such as object composition/decomposition, object state and behavior, viewpoints…
These concepts are declined following three modeling phases introduced by the model:
¾ the system specification, where designers classify and make up the different objects of the system;
¾ the system modeling which defines, for different levels of abstraction, the models of interaction between
¾ the system structuring, where the different structures which should be implemented in the system are
The aim assumed by RM-ODP is to help a community of designers to adopt a consensus (e.g. a standard)
regarding the designing process of their system. With this in mind, RM-ODP is structured into three
complementary documents. The first one, the “Overview” (ISO/IEC 10746-1, 1998) defines the model’s
application spectrum. The second one, “Foundations” (ISO/IEC 10746-2, 1996), defines the set of generic
concepts related to the specification, modeling and structuring phases. The last one, “Architecture” (ISO/IEC
10746-3, 1996) is dedicated to the definition of five viewpoints on the design of a system architecture, as well as
the definition of the different languages associated with these viewpoints. These are:
¾ The enterprise viewpoint: A viewpoint on the system and its environment that focuses on the purpose, scope
and policies for the system, represented with concepts such as community, actor, role, activity and tool.
¾ The information viewpoint: A viewpoint on the system and its environment that focuses on the semantics of
the information and information processing performed.
¾ The computational viewpoint, which is concerned with the functional decomposition of the system into a set
of objects that interact at interfaces – enabling system distribution.
¾ The engineering viewpoint: A viewpoint on the system and its environment that focuses on the mechanisms
and functions required to support distributed interaction between objects in the system.
¾ The technology viewpoint: A viewpoint on the system and its environment that focuses on the choice of
technology in that system.
Specifying an architecture in the RM-ODP framework allows designers to focus their attention with these five
viewpoints (see Figure 1). They are the main proposals of RM-ODP which allow the mastering of a complex
system specification..
Metamodel : Foundation
Enterprise viewpoint
computational viewpoint
information viewpoint
engineering viewpoint
Specification and
Modeling acts
Structuring acts
technology viewpoint
Figure 1. The five viewpoints of RM-ODP
TEL Standards, Specifications and Projects: Part of the Instantiation of RM-ODP
Recent works and projects on TEL engineering, conducted by several standardization committees, consortiums
and scientific and industrial communities, want to define formalisms, engineering methods and/or platforms
which focus on a specific aspect of a TEL system, whilst remaining neutral from a pedagogical point of view.
These approaches are close to the concept of viewpoint as defined by the RM-ODP standard, which allows the
specification of different aspects of a system within a same framework.
The information viewpoint. The LOM (Learning Object Metadata) standard of the IEEE (Institute of Electrical
and Electronics Engineers) (IEEE 1484.12.1, 2002) is a meta-descriptor for a learning object which allows
indexation in a distributed databank for reuse. The EMLs (Educational Modeling Languages) (Rawlings et al.,
2002), and especially the informational models proposed by the IMS Consortium, like Learning Design (IMSLD/IM, 2003), are languages which allow the definition of an instructional scenario by the association between
informational concepts such as roles or activities. The related constraints are defined by informational models in
several technical documents. These two main proposals could be considered as the instantiation of the
information viewpoint of RM-ODP on the TEL engineering: they allow a coherent description of both the
learning objects and the scenarios of a TEL system. The proposed formalisms bring designers to use a formal
syntax for their descriptions.
The computational viewpoint. One of the three documents provided by IMS for its Learning Design
Specification, named "Best Practice Implementation Guide" (IMS-LD/BP, 2003), describes the functional
decomposition of a TEL system through the specification of links, interfaces and interactions between objects.
Earlier, the LTSA (Learning Technology Systems Architecture) proposal (IEEE- P1484.1/D9, 2001) of IEEE
aimed to define an architectural model to highlight the functional components and the different flows existing in
a learning process. These two proposals could be considered as the instantiation of the computational viewpoint
of RM-ODP on the TEL engineering.
The engineering viewpoint. SCORM (Sharable Content Object Reference Model) (ADL, 2004) proposed by
ADL (Advanced Distributed Learning) and "Abstract Framework" (IMS-AF/WP, 2003) provided by IMS, could
be considered as the instantiation of the engineering viewpoint of RM-ODP on the TEL engineering. These two
proposals synthesize the specification of the technical framework of a TEL system. Freeware community
projects such as OpenUSS (2005) or industrial specifications such as E-Learning Framework (SUN, 2003) are
relevant of the consensus adopted in the IMS document. These proposals stress the need for the designers to
know and take into account the characteristics of the interfaces and behaviors of the different components of a
TEL system.
The technology viewpoint. We don't aim to elaborate here on the necessity for the TEL community to adopt
software approaches as Model Driven Architecture proposed by OMG (2001), but we think that the different
TEL projects which are relevant to this approach, like OpenUSS (Grob et al., 2004), FreeStyle Learning (Brocke,
2001), or Future Learning Environment (Leinonen et al., 2002) are good examples of the instantiation of the
technology viewpoint of RM-ODP in a TEL project.
This short panorama of the TEL community results allows us to focus on the shortcomings of these works. The
initial and main objective of these projects, especially the standards proposals, is to enhance the interoperability
between TEL systems and the reusability of their productions. As a consequence, all of these propositions lead to
the instrumentation the specification task, neglecting the iterative dimension of the development of an
educational system. The instantiation of RM-ODP on TEL engineering highlights the weakness of the enterprise
viewpoint in our TEL community. In particular, we want to underpin a meta reflection on the life cycle of a TEL
system, on languages used in each viewpoint, and on interrelations existing between all actors of the iterative
process of a TEL development. This is the reason we propose below an enterprise viewpoint for the
reengineering of a TEL system.
The Enterprise viewpoint for reengineering of a TEL System
The quality approach of RM-ODP concerns the organization of the actors, the documents and the phases of the
system life cycle. Quality could be defined here by the quantity of observed uses which have been taken into
account in a reengineering purpose, but also by the research of qualitative missing indicators (ISO/IEC 10746-1,
1998, Chapter 9.1). The enterprise viewpoint of RM-ODP allows the explicitation of the organizational
dimensions related to such an iterative process in a unique framework.
Pawlowski (2002) is a first approach for identifying and taking into account the specific enterprise aspects of a
TEL development. But it is restricted and only synthesizes the different projects in educational systems quality
insurance. Moreover, this work does not contribute to defining a unique framework including other approaches
such as, for instance, the existing standard proposals. Oubahssi et al. (2004) want to define a new meta descriptor
which allows the designers to describe each type of learning object (which here could be a web document, a
video,… but also a scenario, a service, a role, …) during all phases of the design, the use and the analysis of a
TEL system. But the enterprise viewpoint is not supported here: only the computational and information
viewpoints are taken into account.
The proposal of an enterprise viewpoint for the reengineering of TEL systems we make here, as shown in Figure
2, aims to integrate both the different constraints existing in a reengineering process and the specificities of the
use of TEL standards. We use the vocabulary described in (ISO/IEC-15414, 2002):
¾ Process: a collection of steps taking place in a prescribed manner and leading to an objective. A Process is a
set of tasks which operate and produce information for reaching and operational objective.
¾ Community: a collection of enterprise objects which share the same objectives. A Community is constituted
by humans, informative resources and/or tools which share the same objective into the system under
¾ Enterprise Object: a basic entity of the enterprise specification to fulfill one or more roles. An Enterprise
Object could be any component of the system specification which could assume at least one basic function
in the operational system.
¾ Role: abstraction of an enterprise object behavior or a community behavior. A Role emphasizes the
comportment of one entity or a set of them.
¾ Actor: an enterprise object that participates in the action. Thus, the term Actor qualifies the entities which
participate to a named interaction with other objects.
¾ Action: Action is a basic modeling concept to qualify each object within an ODP system. An Action allows
the definition of an interaction inside an object or between several objects.
¾ Objective: a practical advantage or intended effect, expressed as preferences about future states.
¾ Behavior: A behavior of an object is a collection of actions that the object may take part in, together with
the set of constraints on when those actions can occur.
¾ Artifact: an enterprise object that is referenced in the action, that is to say, that is necessary for performing
the action.
¾ Resource: an enterprise object which is essential to some behavior.
¾ Step: An abstraction of an action, used in a process that may leave unspecified objects that participate in that
action. This term allows the definition of actions prototypes.
Graphical conventions are taken from the ECA (Enterprise Collaboration Architecture) UML profile, as
proposed by the OMG consortium (OMG/ECA, 2004). This profile aims to support a model driven development
process, in an RM-ODP framework (Nagase et al., 2004).
The enterprise viewpoint we propose for the TEL systems reengineering is composed of six processes, described
here with the help of the vocabulary and rules proposed by RM-ODP.
The Design Process produces specifications guided by the main standardized informational models of TEL (e.g.
LOM and Learning Design). These specifications need to be adapted to existing resources (learning objects) and
learner profiles, and/or modified, taking into account information produced by the Analysis Process (uses
analysis, system comportment).
The Software Engineering Process interprets the designers' specification and produces a prototype of the TEL
system (we consider here each system as a prototype which could be modified by reengineering). This process
has to negotiate with the Design Process and the Analysis Process in order to define the observation needs of a
learning session and to develop and integrate the related software captors.
The Learning Process supports the learning sessions. It reifies the scenario specified by the Design Process and
implemented by the Software Engineering Process in the prototype. This scenario leads to the organization of
some actions, named activities in Learning Design, performed by the actors' roles of the learning session.
The Analysis Process operates computational and manual use analysis techniques. Its negotiation with the
Software Engineering Process could establish new needs: events to observe and/or new interpretations. Uses
analysis feedback is provided to the Design Process.
The Resources Management Process manages the learning objects resources.
The Learners Profiles Management Process manages the enterprises objects, named here audits, of the Analysis
and Design Processes of the TEL system reengineering community. These objects are mainly structured
knowledge, provided as, for example, described scenarios in the meaning of (Iksal & Choquet, 2005; Barré &
Choquet, 2005).
The processes take part in the actions of the TEL system reengineering process community. The definition of a
community (see above) allows us to consider each process itself as a community of enterprise objects. We will
describe the objectives, roles and processes of these communities with two examples, one on the reverse
engineering of a TEL system, the other on its reengineering.
TEL System Reengineering
execute a
produce an
produce a
analyze an
adapt a
produce a
use a
interpret a
Management Process
interpret a
store a
produce a
interpret an
provide a
analyze a
use a
search a
ask for an
produce an
make an
of an audit
Learners Profiles
Management Process
produce a
provide a
ask for an
interpret an
produce an
manage an
produce an
Figure 2: UML package of the enterprise viewpoint of the reengineering process of a TEL system
Use Cases
Presentation of the TEL system
For four years, we have been testing a TEL system with a reengineering approach using higher degree students
of a Computer Technology Institute and their teachers. Each year around fifty students participate to a two-hour
learning session. After each session, teachers – who are the designers – and ourselves – who play the roles of
software engineers and analysts, modify the system according to the uses analysis results. We will exemplify the
use cases with this test as described briefly below.
During a learning session, the students learn the main principles of the operationalization of a HTTP server with
the deployed educational environment. The session is a part of a well identified course within the cursus
provided by the Institute. The initial instructional scenario defined by designers was organized around 6
activities supported by, at least, one web resource.
¾ The activity "Introducing the Learning Unit" is supported by a video which presents the main characteristics
of a web server. This activity aims to present both the background knowledge mobilized by the learning
unit, and the set of know-how the learner will acquire.
The activity "Going into the Matter" is supported by a HTML document, composed of texts and figures
which describe the main web protocols a web server has to support, and the programming tools needed to
operate them.
The activity "Linking Concepts and Methods" is supported by two slide shows, one presenting the relevant
concepts, and the other describing, step by step, the execution of the Java code of a server.
The activity "Training to Understand" is supported by several exercises on the HTTP server programming.
The activity "Learning by Doing" is supported by a simulation where the learner, helped by a Java editor
and a compiler, should modify existing code in order to make a functional web server.
The activity "Evaluating the acquisition" is supported by two multiple choice questionnaires.
As we have defined before, the enterprise viewpoint of a TEL system is composed by six processes (design,
software engineering, learning, analysis, resources management and learners profiles management). Each of
them should use software such as the open software projects – referenced in the Table 1 – as we have used for
our test.
Table 1: Open Software Projects Used in the Test
Open software products
Design Process
Mozilla (Mozilla, 2005), OpenOffice (OpenOffice, 2005), OpenUSS
(OpenUSS, 2005), FreeStyle Learning (FSL, 2005)
Software Engineering Process
Eclipse (Eclipse, 2005), AndroMDA (AndroMDA, 2005),
PoseidonCE (PoseidonCE, 2005)
Learning Process
Mozilla, OpenUSS, FreeStyle Learning
Analysis Process
Weka (Weka, 2005), WUM (WUM, 2005)
Resources Management Process
GForge (GForge, 2005), CVS (CVS, 2005)
Learners Profiles Management Process
OpendLDAP (OpenLDAP, 2005)
The Use Case of a Reverse Engineering of an Educational System
On page 15 of Chikofsky (1990), the reverse engineering is defined as "the process of analyzing a subject system
with two goals in mind: 1) to identify the system's components and their interrelationships; and, 2) to create
representations of the system in another form or at a higher level of abstraction". We will here present a typical
use case of reverse engineering which explains how the RM-ODP framework could guide the software
engineering process when it tries to implement the specification (a representation of the TEL system from the
information viewpoint) provided the design process. The interaction between these two processes produces an
artifact which is a representation of the TEL system from the technology viewpoint. This use case is structured in
five steps, as shown in figure 3.
Enterprise viewpoint
Computational viewpoint
Information viewpoint
Engineering viewpoint
Technology viewpoint
Figure 3: The five steps of the Reverse Engineering Use Case
Step 1: Instantiation of the enterprise viewpoint. The reverse engineering of a TEL system needs a dialog
between the software engineering process and the design process. It is this interaction between these two
enterprise objects, grouped in a same community, which is modeled from the enterprise viewpoint, as shown in
Figure 4 with ECA notation.
Reverse Engineering
Engineering t
Designer / Engineer
Engineer / Designer
Figure 4: Instantiation of the Enterprise Viewpoint
Step 2: Instantiation of the information viewpoint. The role of the design process, instigator of the dialog, is to
produce a specification of the TEL system, from the information viewpoint. We have used the different
information formalisms provided by the IMS consortium for our test. Figure 5 is an excerpt from the
instructional scenario, modeled in IMS Learning Design and IMS Content Packaging. It shows the environment,
made up of two objects, chosen by the designers to support the activity "Going Into Matter". The object
"knowledge-object" is the HTML document "author_text1.html" which contains the pedagogical content. The
object "tool-object" is the Java component "manager.jar" which handles the diffusion of the document. In fact,
according to Koper (2001), when designers choose to present the pedagogical content ("knowledge-object"),
they explicit the need of the support object ("tool-object").
<imsld:environment identifier="ENV-textStudy-e1">
<imsld:learning-object identifier="LO-textStudy-e1" type="knowledge-object">
<imsld:item identifier="ITEM-textStudy-e1" identifierref="RES-textStudy-e1"/>
<imsld:learning-object identifier="LO-textStudyManagerFSL" type="tool-object">
<imsld:item identifier="ITEM-textStudyManager" identifierref="RES-textStudyFSL"/>
<imscp:resource identifier="RES-textStudy-e1" adlcp:scormtype="asset" type="webcontent"
<imscp:file ref="/textStudy/author_text1.html"/>
<imscp:resource identifier="RES-textStudyFSL" adlcp:scormtype="sco" type="">
<imscp:file ref="/learningUnitViewManagers/textStudy/manager.jar"/>
Figure 5: Excerpt of the Instructional Scenario
Step 3: Instantiation of the software engineering viewpoint. When the design process provides its specification,
the software engineering process needs to identify and characterize, in a modeling task, the states of the objects
introduced by the specification. Guided by the RM-ODP framework, a software engineer knows that he needs to
identify three schemas (ISO/IEC 10746-3, 1996, Chapter 6.1):
¾ the Invariant Schema is a set of predicates and relations on information objects which are always true;
¾ the Static Schema defines the state of the information objects at some point in time;
¾ the Dynamic Schema is a specification of the allowable state changes of the information objects, subject to
the constraints of an invariant schema.
Here, the static schema is the set of models defined by the proposals Learning Design and Content Packaging of
IMS and Content Package of ADLNet, to which are issued the information objects of the designers' specification
(imscp:file, imscp:resource, imsld:item, imsld:environment, imsld:learning-object and adlcp:scormtype). The
invariant schema defines the existing links between these different objects, as it is shown in the Figure 6, with
the help of the UML graphic language for a class diagram (OMG/UML, 2003).
Figure 6: Invariant Schema
The dynamic schema does not exist at this state of the design. Thus, the software engineering process needs to
identify and model new information objects, in order to establish the dynamic schema. One needs here to
characterize the behavior of these different objects.
Step 4: Instantiation of the computational viewpoint. To characterize the dynamic schema, the software
engineering process needs more information on the specification provided by designers. The pattern "Capturing
model details" proposed by Demeyer et al. (2002), supports such a reverse engineering approach. From the
computational viewpoint, the software engineering process wants to define a functional decomposition of the
system. As an example, the SCORM object proposed by the designers to support the activity "Going into the
Matter" (the HTML document "author_text1.html", see figure 5) is an "asset", which is indivisible. This type of
object is only provided to the learner, without any expected feedback. But a "SCO" SCORM object is linked to it
(see figure 5). This type of object contains a structure of internal objects which describes its functioning. The
RM-ODP framework proposes a typology of the interfaces (ISO/IEC 10746-3, 1996, Chapter 7.1) supporting the
interactions between these objects, in order to model them better:
¾ a Signal Interface, in which all the interactions are signals;
¾ an Operation Interface, in which all the interactions are operations;
¾ a Stream Interface, in which all the interactions are flows.
The focus of attention of the software engineering process on the signal interface leads it to adopt the event
programming paradigm as deployed by the SUN Java technology Swing (2005), for the implementation of the
SCORM object.
<<instance of>> HTMLDocument
XHTML 1.0 Extend Track
Figure 7: The enterprise artifact produced by the reverse engineering
Step 5: Instantiation of the technology viewpoint. From the computational viewpoint, the software engineering
process needs to identify the different states and behaviors of the objects specified by designers. From a
technology viewpoint, it means that the software engineering process needs to develop and integrate software
captors into these objects, in order to track the behaviors of the technology objects corresponding to the
information objects. In our example, it needs to identify what events (signals) could be exchanged and, for each
of these events, identify the emitter and the receptor. One of the design patterns widely adopted by the software
engineering community for this kind of problem is the MVC paradigm (Model/View/Control) described by
Buschmann et al. (1996). It is implemented in the Java library "swing/JFC". Each object proposed by the
designers and relevant to this paradigm could be a traceable object.
Conclusion. The reverse engineering of a TEL system described by this use case is initiated by the design
process but concerns mainly the software engineering process which, guided by the RM-ODP framework, aims
to define the dynamic schema of the TEL system from its information specification. The result of the reverse
engineering process is the artifact shown by figure 7. From the enterprise viewpoint, this artifact increases the
quality of the specification provided by designers: even if it wasn't specified, by asking to themselves questions
suggested by RM-ODP, the software engineers have enhanced the Learning Object in order to track its use. This
result could be shared with designers, and generate new kinds of instructional scenarios by taking into account
these tracking possibilities.
The Use Case of a Reengineering of a TEL System
With Chikofsky’s definition in mind (see before), one can say that reengineering a system is relevant to a model
driven approach, in an iterative development process. The main reengineering acts on a TEL system are the
modifications made by the designers on the informational model of the educational system, especially by taken
into account the uses observed during a learning session. We will present here a reengineering use case where
the analysis process provides information to the design process which is useful to understand better the real
sequence of learning objects that each learner performed. With these data, both the informational and the
software engineering viewpoints are modified. Figure 8 describes the four steps of the use case, one for each
participating role: software engineering process role, learning process role, analysis process role, design process
: software engineering
: produce a
: learning process
: produce an
: execute a
: analyze an
: design process
: interpret an
: analysis process
: produce an
Figure 8: UML Collaboration Diagram of the Reengineering Use Case
Step 1: the role of the software engineering process. Each reengineering of a system implies its reverse
engineering. In this use case, by applying the reverse engineering pattern (Demeyer et al., 2002), the software
engineering process has developed and integrated captors to the TEL system. This allows the reverse engineering
by producing information on the effects of the decisions made from the engineering viewpoint. Following the
RM-ODP guidelines, the engineering objects are grouped in clusters to reduce the complexity of their
manipulation. Figure 9 is an excerpt of the FreeStyle Learning deployment script on the learners’ computers.
One could see that, even, from an informational viewpoint, there are several objects with a same type (e.g. 3
exercises, 2 slide shows, 2 MCQ), the engineering objects are only the corresponding components (e.g. 1
exercise component, 1 slide show component, 1 MCQ components).
Step 2: the role of the learning process. From the computational viewpoint, these engineering objects are
SCORM components (SCO) provided to each client computer. They are observed and tracked by the LMS
(Learning Management System) following the SCORM technical specification. The transition/state UML
diagram of figure 10 shows the global interaction between the LMS and each SCO.
<jnlp spec="O.2 1.0"
<description>FreeStyle Learning</description>
<jar href="learningUnitViewAPI.jar"/>
<jar href="learningUnitViewManagers/introVideo/manager.jar"/>
<jar href="learningUnitViewManagers/textStudy/manager.jar"/>
<jar href="learningUnitViewManagers/slideShow/manager.jar"/>
<jar href="learningUnitViewManagers/exercise/manager.jar"/>
<jar href="learningUnitViewManagers/simulator/manager.jar"/>
<jar href="learningUnitViewManagers/MCQ/manager.jar"/>
<application-desc main-class="freestyleLearning.homeCore.FSLApp"/>
Figure 9: FreeStyle Learning Deployment Script
SCO is launched by
LMS and SCO has
found the API
calls Initialize()
Figure 10: Transition/State UML Diagram of a SCORM component (ADL, 2004)
Step 3: the role of the analysis process. The information provided by the TEL system to the analysis process is
structured in data as shown in figure 11: a session ID, a date, the relevant package of the engineering object and
the name of the observed event. The analysis process operates statistics with these data. It aims to discover new
scenarios rather than to validate the prescribed one. This data mining structures the observed scenarios as
sequences of resources, and the time spent, using the engineering viewpoint language. In our example, it leads to
the identification of strong links between resources, which could be grouped in clusters and finally interpreted by
designers as, for instance: “when they are expected to consult the video, many learners also accede to others
resources such as questionnaires”.
Step 4: the role of the design process. The design process, by its negotiation with the analysis process, aims to
adapt the information model. The two processes try to find a correlation between the instances of the engineering
viewpoint concepts (e.g. the new clusters defined by the analysis process) and those of the information viewpoint
concepts (e.g. possible new activities supported by a new combination of resources). Figure 12 shows how the
engineering and information viewpoints have been modified since the beginning of our test.
Conclusion. This reengineering use case involves the main roles of the enterprise processes of a “TEL system
reengineering community”. The interaction between these processes is guided by RM-ODP and leads to the
correlation of the different viewpoints on the system. In a more general approach, the results produced by this
community could be shared with other communities and increase the quality of the development of TEL
¾ the engineering objects developed (e.g. the five new clusters managers) as freeware,
¾ the use case itself as an example of instantiation of IMS Learning Design,
¾ the conformity rules found during the correlation process of the different viewpoints such as knowledge and
know-how which could be useful for a TEL system reengineering process.
<observer id="src2e05">
<element name="Format" value="ISO 8601"/>
<element name="Value" value="31/03/2005:10:45:42 +0407"/>
<element name="Package" value="freestyleLearningGroup.freestyleLearning.learningUnitViewManagers.questionnaire1"/>
<element name="Event" value="Initialize()"/>
Figure 11: Example of observed event provided by FreeStyle Learning
Before reengineering
the LO
Going into
the Matter
After reengineering
Text Study
Linking Concepts
and Methods
Text Study
Slide Show
Text Study
Slide Show
Discovering the
the LO
Slide Show
Training to
by Doing
Text Study
Evaluating the
Slide Show
Going into
the Matter
by Doing
Evaluating the
Figure 12: Information and Engineering Objects before and after the Reengineering
We have presented in this paper an unique framework allowing the highlighting and the description of the
transformation processes of the different models defined for the development of a TEL system. These models
involve formalisms and knowledge which is specific to the relevant communities. The comprehension of these
processes, their formalization and, looking forward, their instrumentation should be, for us, one of the crucial
research fields in the designing of TEL systems. By supporting these model transformations through and inside
different levels of abstraction and different viewpoints, the TEL community will increase its knowledge and
enhance its methods for designing higher quality systems.
With the instantiation of the RM-ODP framework we propose on TEL, we aim to bring the design community to
share and federate their experiences and expertise. Each experimentation, each deployment of a TEL system by
teachers, institutions or research teams could provide benefits to the whole community. The open software
community is a good example of the aims of our works, by promoting a design approach (Raymond, 2001), tools
(Bar & Fogel, 2003) and communication supports (SourceForge, 2005). The designer or the user member of
such a community works in a synergy with the other members by considering all the artifacts produced as
prototypes, always in progress.
This approach, guided by a framework as we have proposed, could be very valuable for structuring the TEL
community. With this resolution in mind, we think our proposal could be the background for the emergence of a
dynamic collaboration in the TEL community, in order to enhance the quality of the engineering and
reengineering processes of a TEL system.
ADL. (2004). Sharable Content Object Reference Model (SCORM), 2004, (2nd Ed), retrieved May 8, 2006 from
AndroMDA. (2005). retrieved May 8, 2006 from http://www.andromda.org/.
Bar, M. & Fogel, B. (2003). Open Source Development with CVS, (3rd Ed). Arizona: Paraglyph Press.
Bär, H., Bauer, M., Ciupke, O., Demeyer, S., Ducasse, S., Lanza, M., Marinescu, R., Nebbe, R., Nierstrasz, O.,
Przybilski, M., Richner, T., Rieger, M., Riva, C., Sassen, A.-M.., Schulz, B., Steyaert, P. & Tichelaar, S. (1999).
The FAMOOS Object-Oriented Reengineering Handbook, retrieved May 8, 2006 from
Barré, V. & Choquet, C. (2005). Language Independent Rules for Suggesting and Formalizing Observed Uses in
a Pedagogical Reengineering Context. In Goodyear, P., Sampson, D. G., Yang, D. J-T., Kinshuk, Okamoto, T.,
Hartley, R., Chen, N-S. (Eds.) Proceedings of the ICALT 2005 Conference, Los Alamitos, CA: IEEE Computer
Society Press, 550-554.
Brocke, J. (2001). Freestyle Learning - Concept, Platforms, and Applications for Individual Learning Scenarios.
In Heinrich Kern (Eds.) Proceedings of 46th International Scientific Colloquium, Ilmenau, Germany: Technical
University, 149-151.
Bruillard, E. & Vivet, M. (1994). Concevoir des EIAO pour des situations scolaires : approche méthodologique.
Recherche en Didactique des Mathématiques, 14 (1/2), 273-302.
Buschmann, F., Meunier, R., Rohnert, H., Sommerlad P. & Stal, M. (1996). Pattern-Oriented Software
Architecture, Volume 1: A System of Patterns. Chichester, WS: John Wiley & Sons.
Chikofsky, E. J. (1990). Reverse Engineering and Design Recovery: A Taxonomy. Software IEEE, 7, 13-17.
CVS. (2005). retrieved May 8, 2006 from http://ximbiot.com/cvs/.
Demeyer, S., Ducasse, S. & Nierstrasz, O. (2002). Object-Oriented Reengineering Patterns. San Francisco, CA:
Morgan Kaufmann Publishers.
Eclipse. (2005). retrieved May 8, 2006 from http://www.eclipse.org/.
FSL. (2005). retrieved May 8, 2006 from http://www.wi.uni-muenster.de/aw/fsl/.
GForge. (2005). retrieved May 8, 2006 from http://gforge.org.
Grob, H. L., Bensberg, F. & Dewanto, B. L. (2004). Developing, Deploying, Using and Evaluating an Open
Source Learning Management System. Journal of Computing and Information Technology, 12 (2), 127-134.
Hummel, H., Manderveld, J., Tattersall C. & Koper, R. (2004). Educational modelling language and learning
design: new opportunities for instructional reusability and personalised learning. International Journal of
Learning Technology, 1 (1), 111-126.
IEEE P1484.1/D9. (2001). Learning Technology Systems Architecture, Draft 9, retrieved May 8, 2006
IEEE 1484.12.1. (2002). Draft Standard for Learning Object Metadata, retrieved May 8, 2006 from
Iksal, S. & Choquet, C. (2005). An Open Architecture for Usage Analysis in a E-Learning Context. In Goodyear,
P., Sampson, D. G., Yang, D. J-T., Kinshuk, Okamoto, T., Hartley, R., Chen, N-S. (Eds.) Proceedings of the
ICALT 2005 Conference, Los Alamitos, CA: IEEE Computer Society Press, 177-181.
IMS-AF/WP. (2003). IMS Abstract Framework: White Paper, version 1.0, retrieved May 8, 2006 from
IMS-LD/BP. (2003). IMS Learning Design Best Practice and Implementation Guide, version 1.0, Final
Specification, retrieved May 8, 2006 from
IMS-LD/IM. (2003). IMS Learning Design Information Model, version 1.0, Final Specification, retrieved May 8,
2006 from http://www.imsglobal.org/learningdesign/ldv1p0/imsld_infov1p0.html.
ISO/IEC 15414. (2002). Information Technology : Open Distributed Processing Reference Model, Entreprise
Language, retrieved May 8, 2006 from http://www.joaquin.net/ODP/DIS_15414_X.911.pdf.
ISO/IEC 10746-1. (1998). Information Technology : Open Distributed Processing Reference Model, Part 1:
Overview, retrieved May 8, 2006 from
ISO/IEC 10746-2. (1996). Information Technology : Open Distributed Processing Reference Model, Part 2:
Foundations, retrieved May 8, 2006 from
ISO/IEC 10746-3. (1996). Information Technology : Open Distributed Processing Reference Model, Part 3:
Architecture, retrieved May 8, 2006 from
Koper, R. (2001). Modeling units of study from a pedagogical perspective: the pedagogical meta-model behind
EML, Educational Technology Expertise Center Open University of the Netherlands, version 2, retrieved May 8,
2006 from http://eml.ou.nl/introduction/docs/ped-metamodel.pdf.
Leinonen, T., Virtanen, O., Hakkarainen, K.& Kligyte, G. (2002), Collaborative Discovering of Key Ideas in
Knowledge Building. In Stahl, G (Eds.) Proceedings of the CSCL 2002 Conference, Hillsdale, NJ: Lawrence
Erlbaum Associates, 529-530.
Mozilla. (2005). retrieved May 8, 2006 from http://www.mozilla.org.
Nagase, Y., Hashimoto, D. & Sato, M. (2004). Applying Model-Driven Development to Business Systems using
RM-ODP en EDOC. In Vallecillo, A., Linington, P. & Wood, B. (Eds.), Proceedings of Workshop on ODP for
Enterprise Computing (WODPEC 2004), Málaga, Spain: University of Málaga, 36-42.
OMG. (2001). Model Driven Architecture (MDA), retrieved May 8, 2006 from http://www.omg.org/mda/.
OMG/ECA. (2004). Enterprise Collaboration Architecture (ECA) Specification, version 1.0, retrieved May 8,
2006, from http://www.omg.org/docs/formal/04-02-01.pdf.
OMG/UML. (2003). UML 2.0 Infrastructure
OpenLDAP05. (2005). retrieved May 8, 2006 from http://www.openldap.org/.
OpenOffice. (2005). retrieved May 8, 2006 from http://www.openoffice.org/.
OpenUSS. (2005). Open Source Software for Universities and Faculties (Open Source University Support
System), retrieved May 8, 2006 from http://openuss.sourceforge.net/openuss/.
Oubahssi, L., GrandBastien, M. & Claës, G. (2004). Ré-ingénierie d'une plate forme fondée sur la modélisation
d'un processus global de FOAD. In Moreau, C. (Eds.), Proceedings of the TICE'04 Conference, Compiègne,
France : Université de Technologie de Compiègne, 32-38.
Pawlowski, J. M. (2002). Report on Quality Assurance Standards / Proposal for Future Work: Project Team
Quality Assurance and Guidelines, CEN / ISSS Workshop on Learning Technologies, Working Draft, retrieved
May 8, 2006 from http://www.wip.uni-duisburg-essen.de/imperia/md/content/elm/qadraft20020328.doc.
PoseidonCE. (2005). retrieved May 8, 2006 from http://gentleware.com.
Rawlings, A., Rosmalen, P., Koper, R., Rodríguez-Artacho, M. & Lefrere, P. (2002). Survey of Educational
Modelling Languages (EMLs), CEN/ISSS WS/LT report, retrieved May 8, 2006 from
Raymond, E. S., (2001). The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental
Revolutionary, Sebastopol, CA: Edition O'Reilly.
SourceForge. (2005). retrieved May 8, 2006 from http://www.sourceforge.net.
SUN Microsystems (2003). E-Learning Framework: Technical White Paper, retrieved May 8, 2006 from
Swing. (2005). retrieved May 8, 2006 from http://java.sun.com/products/jfc/index.jsp.
Tchounikine, P., Baker, M., Balacheff, N., Baron, M., Derycke, A., Guin, D., Nicaud, J.-F. & Rabardel, P.
(2004). Platon-1: quelques dimensions pour l'analyse des travaux de recherche en conception d'EIAH. Paris,
France: STIC-CNRS.
Weka. (2005). retrieved May 8, 2006 from http://www.cs.waikato.ac.nz/ml/weka/.
Wenger, E. (1987). Artificial Intelligence and Tutoring Systems: Computational Cognitive Approaches to the
Communication of Knowledge, Los Altos, CA: Morgan Kaufmann.
Karagiannidis, C. (2006). Book review: Web-Based Intelligent e-Learning Systems: Technologies and Applications
(Zongmin Ma). Educational Technology & Society, 9 (4), 242-243.
Web-Based Intelligent e-Learning Systems: Technologies and Applications
(Book Review)
Charalampos Karagiannidis
Assistant Professor
Department of Special Education
University of Thessaly
Volos, Greece
[email protected]
Textbook Details:
Web-Based Intelligent e-Learning Systems: Technologies and Applications
Zongmin Ma (ed.)
Idea Group Inc.
ISBN: 1-59140-729-X (hardcover), 1-51940-730-3 (softcover), 1-59140-731-1 (ebook)
2006, 388 pages
Web-Based Intelligent e-Learning Environments (WILE) attract considerable attention worldwide, since they
bare the potential to improve the quality of e-learning applications and services: WILE can overcome the main
shortcoming of e-learning technologies (all learners receive the same learning material, activities, etc; “one-sizefits-all” approach), by facilitating personalized learning experiences, adapted to the particular characteristics of
each learner.
The book focuses on the technologies and applications of WILE, aiming to present latest research, development
and application results in the field.
Chapter Summary
The book includes 17 chapters, organized into two major sections. The first section discusses theories, key
technologies and designs of WILE, while the second section covers implementation and application issues.
In particular, Chapter 1 presents a framework of adaptive support for inductive reasoning ability in virtual
learning environments, based on research on cognitive science; the framework accommodates adaptive
navigational paths, adaptive content (amount, concreteness, structure) and adaptive information resources.
Chapter 2 discusses one of the most challenging issues in WILE: adaptive authoring; it then presents a solution
(MOT) based on the LAOS model, as well as a couple of WILE which have been created accordingly. Chapter 3
addresses the selection and sequencing of Learning Objects (LO); it presents a utility-based model which is
based on learning technologies specifications and standards, aiming to adaptively select and sequence LO,
according to the learners’ profile. Chapter 4 addresses virtual learning communities (VLC); it discusses the
essential criteria for their setup (social context, shared learning goal, technology and facilitation), as well as how
to build culturally-inclusive VLC. Chapter 5 presents a conceptual architecture for the development of
interactive educational multimedia; the architecture is open to further extensions, integration to other
frameworks and standards, and adaptations to particular learner needs. Chapter 6 focuses on students’ emotions
in learning environments; it reviews the state-of-the-art in this field, as well as the challenges and main
difficulties in accommodating students’ affects in learning environments. Chapter 7 introduces how to use a
web-based recommender system, developed with a collaborative bookmark management system approach; the
system is able to effectively filter relevant resources, taking advantage of the common interests of users. Chapter
8 presents MetaLinks, a domain-independent authoring tool and web server for adaptive textbooks; the tool
supports active reading, which is appropriate for novice and advanced readers, and can be coherently read from a
number of thematic perspectives. Chapter 9 discusses knowledge representation in ILE; they define the
requirements concerning all stages of the ILE’ lifecycle, different types of users, and different system modules.
Chapter 10 deals with adaptive tutoring for case-based ITS; it proposes a formal approach for the development
and evaluation of ITS with reusable components. Chapter 11 addresses the use of ontologies in WILE;
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
ontologies serve as shared knowledge representations that can be used to obtain enhanced learning object metadata records, towards the selection of LO according to learners’ profile. Chapter 12 addresses the role of learning
technologies specifications and standards in WILE; it provides an overview of standards, as well as their
utilization in existing systems. Chapter 13 presents AWLA - a writing e-learning alliance; AWLA is an
organised set of interactive web-based utilities that, when applied in a technologically enhanced learning
environment, allows learners to develop their writing skills and fulfil writing activities.
In section II, Chapter 14 describes a number of interactive virtual environments (IVE) which are developed at
the North Dakota State University; these IVE aim to increase student achievement and scientific problemsolving in a real-world context. Chapter 15 presents an integrated platform for educational virtual environments;
the platform aims to support educational communities, synchronous online courses in multi-user 3D
environments, and the creation and access of asynchronous courses through a learning content management
system. Chapter 16 describes a number web-based instructional tutors that support active and engaging learning;
these tutors reason about the students’ knowledge and teaching strategies, in order to adapt to them. Finally,
Chapter 17 introduces the concept (and discusses the implementation and evaluation) of special purpose elearning environments (SPELE), which aim to accommodate learners’ individual learning differences.
The book includes an interesting mix of chapters, addressing both theoretical and development issues, as well as
specific WILE systems and their use. Moreover, the chapters cover different aspects of WILE, including
adaptive authoring, adaptive selection of learning material and activities, knowledge representation and
reasoning for adaptive behaviour, etc.
On the other hand, some of the chapters are not directly related to WILE - readers focused only on WILE may
not be particularly interested in them. Also, the large-scale implementation and evaluation of WILE is not
thoroughly covered by the book chapters - this is a very challenging issue for WILE to become mainstream.
In summary, I would recommend this book to anyone who is interested in WILE. The research domain is not
new (research on ITS started in 80s), however a very limited number of WILE have been tested in real, largescale implementations. The ideas presented in the book can contribute towards a better quality of e-learning
systems, which can be a catalyst for the citizens of the knowledge society and knowledge-based economy.
Glenn, L. (2006). Book review: Visual Media and the Humanities: A Pedagogy of Representation (Kecia Driver
McBride). Educational Technology & Society, 9 (4), 244-245.
Visual Media and the Humanities: A Pedagogy of Representation
(Book Review)
Lauren Glenn
Texas A&M University – Commerce
Tel: +1 214 771-0447
[email protected]
Textbook Details:
Visual Media and the Humanities: A Pedagogy of Representation
Kecia Driver McBride, ed.
2004 Knoxville: University of Tennessee Press
ISBN: 1-57233-321-9
18 chapters, 452 pages, introduction by McBride, bibliography, index, animations and graphics
In recent scholarly discourse, the implementation of film into secondary or university courses has instigated
debates regarding the usefulness of new media in relation to shifting pedagogical styles. As audiences (and by
implication, students) change with the influx of visual imagery, many instructors believe that pedagogical styles
need to become more pliable in relation to new media. Kecia McBride’s Visual Media and the Humanities
brings together contributors from diverse university environments (Ivy League to community colleges) to
discuss the importance of teaching students to be critical and self-reflective when reading texts and images in
humanities courses that use new media, including film, internet, videoconferencing, and digitalized texts.
McBride’s collection includes several articles that explore the impact of new modes of visual representation on
pedagogical styles while emphasizing the importance of teaching students to incite reflective responses as they
read images and texts. In “‘Where can I get a camera?’: Documentary Film, Visual Literacy, and the Teaching
of Writing,” Dale Jacobs claims that visual literacy has been underestimated in the school environment, pointing
to his earliest encounter with visual imagery and literacy through Sesame Street episodes which taught him the
basic structure of visual literacy. He claims, “The unfortunate situation is that what I learned in front of that
television set on those countless mornings – the rudiments of visual literacy, or how to read and make meaning
from moving visual images – is often discounted or devalued in the school environment” (156). Although
contemporary Americans are bombarded with visual imagery from childhood on, Diane Negra and Walter Metz
point out, “Americans are used to articulating codes of pleasure but not codes of meaning in their texts” (229).
Negra and Metz present an example of using media in introductory college courses to combat the American
public’s lack of critical analysis when confronted with visual images in television and film. James W. Miles II
claims that the higher a student advances in academia (into secondary classes or college courses), the more the
“silent codex,” or written final product, is emphasized through book reports, essays, and the written discourse
(75). He argues that such practices may not hold their value as rhetorical demands evolve in society. The
articles included in this book do not argue for the abandonment of all traditional pedagogical methods, but rather
for a malleability of teaching conventions as time changes the emphasis on visual images in society.
Research in the twenty first century is changing rapidly, and nowhere is it more apparent than through the use of
the Internet. Thomas Crochunis’ article, “Visualizing Feminist Theater History on the Web,” discusses how
web-based working groups can offer scholarly support in areas of limited and on-going scholarship for students
and scholars. In “Simulation Machines, Media Boundaries, and the Re-expansion of Composition,” James W.
Miles II points to recent advancements in voice recognition technology and Internet communication to predict
that student writing will become radically more conversational in the future. He suggests that educators use this
change in technology and style to alter pedagogy in composition courses by comparing written and spoken
essays in class.
Contributors introduce topics such as culture, history, argument, foreign language, technology, and web-based
groups. When cultural or historical issues become distanced from students in the classroom, visual images may
be capable of influencing learning or instigating discussions in new ways. In “The World in a Frame:
Introducing Culture through Film,” Gerald Duchovnay approaches teaching a capstone university course that
initiates students into cultural issues. He provides a resourceful example demonstrating how students can be
introduced to cultural differences through film, guest lecturers, and supplementary materials. The article outlines
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected]
a sample course focus, textbooks, films, and assignments that would be beneficial to anyone wanting to broach
cultural issues in the classroom.
While recognizing the changing emphasis on visual literacy in contemporary society, many of the articles
provide analysis of specific films and their usefulness in classroom pedagogy. Maria Bachman, for example,
demonstrates how she uses Pleasantville in conjunction with Plato’s “Allegory of the Cave” to teach students
that higher knowledge consists of educators guiding students to unearth their own questioning abilities.
Likewise, in “An Imitation of Life: Deconstructing Racial Stereotypes in Popular Culture,” Stephanie Thompson
outlines her approach to initiating classroom discussion on the “perpetuation of racist ideology” by utilizing
Fannie Hurst’s novel Imitation of Life with two film adaptations (1934 and 1959) to demonstrate the evolving
racial issues in Western society (205) in her introduction to composition and women’s studies course.
From classic films to contemporary films, Internet groups to web-based writing styles, McBride’s collection
embraces various approaches to using new media to teach in an academic environment that relies heavily on
visual literacy. Anyone interested in accessible scholarly approaches aimed at utilizing new media as a learning
tool in diverse classroom settings will find this collection of course outlines, discussion of implementation, and
pedagogical examples of contemporary classroom issues to be an invaluable resource.
Fly UP