A Practice-Oriented Review of Learning Objects

Reusable learning objects support packaging of educational materials allowing their discovery and reuse. Open educational resources emphasize the need for open licensing and promote sharing and community involvement. For both teachers and learners, finding appropriate tried and tested resources on a topic of interest and being able to incorporate them within or alongside other learning materials can enrich provision and share best practice. Resources are made available by a number of general and subject-specific repositories, but there are also many educational resources residing outside these repositories which may provide useful additional materials. Potential users of materials need to be able to locate relevant material and to assess it with respect to a number of factors (such as suitability for purpose and license requirements). However, even such basic requirements can be less than straightforward to determine. This paper presents a view of the field from the user's perspective, bringing together themes from existing research relating to practice-oriented concerns including discoverability, reusability, and quality. It provides a background in this area, exploring current trends, controversies, and research findings. The discussion is also aligned with current provision and practice, indicating areas where further research, provision, and support would be useful.


INTRODUCTION
Over a decade ago, reusable learning objects (RLOs) were described by David Wiley as occupying the "position of technology of choice in the next generation of instructional design, development and delivery" [1].The concept of an RLO appears extremely attractive in the potential it offers for learning material to be packaged, shared and reused in many different contexts and by many different people.As reported by Friesen [2], significant resources have been invested in projects aimed at developing RLOs and in establishing learning object repositories (LORs) which can manage and make available searchable collections of RLOs.There now exist internationally many thousands of RLOs in many different repositories.In their 2009 study, Ochoa and Duval [3] point to the growing population of Learning Object Repositories, Learning Object Referatories, Open Courseware initiatives, Learning Management Systems and Institutional Repositories.We follow their usage of the word "repository" as covering all such systems.Some of these repositories are reasonably well known, but many other RLOs exist in more obscure environments and may not be easy to find..As noted by Metros [4] "learning objects are scattered across servers housed in academic departments and colleges, campus libraries, state and federal government agencies, community-based consortia, professional societies, and even commercial entities." As the benefits of sharing educational resources have become internationally recognized, so the importance of these resources being openly available for reuse and repurposing has received increasing emphasis (see for example the recent draft statement from the World Open Educational Resources Congress at UNESCO [5]).For many commentators, the focus of discussion has become the Open Educational Resource (OER) rather than the RLO.The extent to which these two perspectives overlap is another area of discussion, but they clearly have a common goal of supporting the sharing and reuse of digital materials for teaching and learning.A number of repositories have now been developed (or have been repurposed) to host open materials specifically and the topic of licensing is another crucial enabling factor.
The potential benefits of openly sharing educational resources are far reaching for learners (by opening educational possibil ities to learners around the world [6]), teachers (in enriching the materials they provide for students and improving teaching practice through sharing ideas [7]) and administrators (for example, by raising awareness and attracting applicants to traditional courses [6]).The Open Courseware initiative in particular seems to have been extremely successful in terms of both the number of high quality courses made available by universities worldwide and their popularity to users [8].However, despite the acknowledged benefits of sharing and reuse, when considering more general forms of learning materials (for example with granularity smaller than a full course or provenance other than a world-leading university) the position is somewhat different.Many studies (including [2], [3] and [4]) have noted that the actual extent of RLO contribution and reuse is somewhat disappointing.
Many reasons are cited for this, such as the seeming difficulty in defining what an RLO is, lack of awareness, deficiency in related standards and lack of firm evidence that RLOs provide educational benefit.Basic practical difficulties are also cited in this context, such as the limitation of current search mechanisms for finding appropriate RLOs.

Background
In 2010 the authors undertook a project sponsored by the UK Higher Education Academy's Subject Centre for Information and Computing Sciences to identify sources of computing-related RLOs from both well-established repositories and shared institutional provision, and to investigate the quality of materials found.To define the parameters for such an exercise it is necessary to revisit a number of fundamental questions relating to the nature and effectiveness of RLOs in general: what are they, where are they, how are they located, how do we identify "good" ones, on what basis can such judgments be made, and are there quality assurance measures that can be applied?The questions asked and issues raised by that exercise are likely to be those faced by any educator wishing to incorporate RLOs in their teaching and also by learners who seek suitable materials for their particular learning needs.
Much has been written on the topic of learning objects and there exist a number of survey papers covering specific aspects (such as listing and analysis of repositories).However, there is not to our knowledge any recent general survey of the field mapping the current state of the art, setting out a context for future enquiry within the field of RLOs and providing orientation within the subject for those not familiar with accessing and using shared digital learning material.This is the intention of the current paper.
In addition to identifying major themes in the literature, a discussion is presented of issues arising from them which continue to have implications for the use of RLOs in practice and for future research directions.One aspect which becomes very apparent when working with open shared resources is the "reality gap" between what users may expect and what currently exists.Issues arising from consideration of the practitioners' perspective are explored in the discussion and are used to ground and structure the presentation of the survey.

Scope and methodology
As a practice-oriented survey this review covers themes relating to the use in practice of shared learning resources.Relevant areas encompass: identification of factors noted as affecting users, practical studies relating to users' experiences with relation to those factors and studies reporting on different perspectives and ways of supporting or improving practice.Emergent themes include availability, location, discoverability, nature (for example, size and content), development methodology and quality of resources.Examination of these themes also necessitates discussion of factors which inform the debate, such as what RLOs are, how judgments on quality have been attempted and how subjectivity and user preference are taken into account.It is not the intention of this survey to catalogue and compare specific learning object technologies, architectures or implementations -rather, specific examples may be mentioned in relation to their affordances to the user or the advance they represent.So, for example, the importance of metadata is represented in relation to its functional roles in supporting user requirements (such as effective searching) and enabling interoperability.Studies concerning its effective use (or otherwise) are also referenced, but a detailed discussion of data models and comparison between them is re -garded as beyond the scope of the paper.
The methodology followed is that of a qualitative literature review.Sources were sought relating to learning objects and open educational resources dating from 2000 to 2011.There is a huge amount of literature published on these topics so an initial assessment was made to identify important themes relating to the user perspective (through consideration of key works in the field and database citation information).This was then used to provide input to more focused searches relating to each of the main identified issues.Journals and conferences relating to learning objects, open learning and educational technology in general were searched.In addition, a number of relevant books consisting of contributed research chapters have been published during this period and these were also included.A number of websites, initiatives and projects have also been mentioned where these serve to illustrate trends and provision in practice.It would not be possible to include all the many references relating to each theme.Our criterion for inclusion is that the reference represents a new idea or significant development related to that theme which is still of current interest or relevance.

Organization of paper
The rest of the paper is organized as follows.Section 2 briefly discusses the debate concerning the concept and definition of both "learning object" and "open educational resource".Whilst this is a well-rehearsed debate in the literature, the intention here is to emphasize the consequences of the resulting uncertainty for teachers and learners.Section 3 considers the user perspective and draws together the main areas of requirement and influencing factors that have been cited as being enablers or barriers to the successful use of RLOs in practice.Section 4 reviews where RLOs are currently found, referring to and commenting on existing surveys of LORs.The main questions from the user perspective appear to be discoverability, reusability and quality, and these issues are considered in Sections 5 to 7 respectively.Section 8 reviews the factors affecting reuse and examines the implications for how we view RLOs and what practical quality assessment judgments are influencial at the point of use.The final section discusses issues arising and future directions.

WHAT IS A REUSABLE LEARNING OBJECT?
It is clear from the publication record that 2000 was the year in which learning objects came to public prominence as an identified and defined area of interest and research in instructional technology.One notable development that year was the publication by the IEEE Learning Technology Standards Committee of a draft report on learning object metadata (later standardized in 2002) which included the definition: "a Learning Object is defined as any entity, digital or non-digital, which can be used, reused or referenced during technology supported learning" [9].Another key publication in 2000 was the collection, "The Instructional Use of Learning Objects" [10] in which Wiley was already noting that "the proliferation of definitions for the term 'learning object' makes communication confusing and difficult" [1].Since that time, learning objects have been a major focus for re -search and development in the reuse and repurposing of teaching and learning materials [1,10,11].The general idea of packaging learning content and making it available in a way which enables its reuse in different contexts and for different purposes appears intuitively useful and meaningful.However, attempts to clarify exactly what a "learning object" is have led to much debate within the learning object community and there are said to be "as many definitions of LOs as there are users" [12].

The variety of definitions
One area of variation is the level of exclusivity or inclusivity advocated.The IEEE definition given above is widely referenced as an extremely general and broadly-encompassing view of learning objects.However, a very loose characterization includes virtually any resource imaginable and risks making it "impossible to use the term Learning Object in a meaningful way" [12].
Attempts to be specific inevitably lead to criticisms that some resources are unnecessarily excluded.Proposed definitions have placed restrictions on various aspects including size, form, required content and necessary learning time, while others have prescribed features which may themselves need further clarification such as independence, flexibility, reusability, customizability and the need to be "designed with educational intent" [1,4,12,13,14].
Reflection from an educational perspective has focused on making a purposeful link between learning objects and pedagogic orientation [12,15,16], with the need to link learning object development to instructional design principles seen by many as a crucial factor in providing the "learning" in a learning object [1,17].From perhaps the opposite direction, some have urged the consideration of more technological aspects, suggesting that the term makes no sense unless considered along with the appropriate technology [18,19,20].Some authors have suggested that "learning object" is in fact an impossible term to define and that it is not meaningful to refer to anything other than plain "resources" [21].The confusion this debate causes has been noted by many [1,2,12,18].Several attempts have been made to avoid the problem by avoiding a single, head-on definition.Churchill uses a learning object categorization to provide a meaningful definition to the concept [21] and McDonald's recursive approach builds up from atomic "digital elements" using rules based on context-specific goals [22].
Definitions of the term "learning object" may or may not include explicit reference to a requirement of reusability, although this is seen as an implicit necessity by many.The phrase "reusable learning object" emphasizes the importance of this aspect and the primary requirement for resources to be made available in a way which allows them to be used by others, edited, placed in different contexts and combined with additional resources.For this to be possible in practice, other aspects such as RLO discovery, accessibility, and clear identification of associated information (such as objectives, level of study and prerequisites) become integral to the usefulness of a resource.

Reusable learning objects -an inherent contradiction
The very concept of an RLO is viewed by some as being controversial [2,23].An RLO aims to separate learning material from its context -and yet this divorces it from its pedagogic rationale and hence its value.Wiley's "Reusability Paradox" holds that "the more context-laden a given educational resource is, the more effectively it teaches but the more difficult it is to reuse in a novel context" [24,25].The problem has also been linked to the granularity of RLOs with small units seen as more reusable in one sense [12] but requiring more human intervention to render meaningful in a new context.A learning object approach may thus be seen as running counter to good instructional design practice since the developer should be targeting a specific educational context rather than publishing some multi-purpose, generic exposition [26].Some question the whole concept of reusability unless there is provision for major re-engineering [27].

Open Educational Resources
Increasing recognition of the educational value of freely available and reusable resources has led to the rise of the open educ ational resource (OER) movement.With OER, the emphasis of the discussion moves towards ensuring open access through clear and straightforward licensing and promoting sharing of practice and resources through widespread community involvement [28].A significant milestone was MIT's OpenCourseWare initiative [29] with large numbers of courses now being made freely available by many institutions worldwide [8].The benefits of accessibility and inclusivity are widely acknowledged [5,30] but the discussion is most often directed at the level of the course.In practice, however, the granularity of OERs is just as diverse as that of RLOs [31], and general repositories of OERs such as OERCommons offer a diversity of content.
The question then arises as to whether OERs are simply open RLOs.Any distinction seems to be one of ethos rather than definition [32] with social aspects and community-wide sharing emphasized.Lane and McAndrew [33] consider the similarities between the two concepts (including accessibility, discoverability, granularity and reusability) while also noting the differences in terms of social, informal and community involvement expectations for OERs.A recent JISC study on OERs [34] sets out recommendations to staff on how to improve teaching practice through engaging with OERs, with the emphasis on "seeking to share resources and ideas" rather than expecting a quick way to develop courses.Indeed, some commentators suggest that the sharing of practice is paramount, at the same time as questioning what "openness" really means and a concern that "we don't want to kill open practice dead by focusing too strictly on definitions of openness" [35].The importance of changing practice is reflected in the use of the term OEP (Open Educational Practice) and the call for greater institutional vision and commitment in encouraging OEP to flourish [36].

Resources in practice
Trying to pin down the discussion in terms of definitions is perhaps not possible.However, even with respect to practice the ac -tuality is somewhat confusing.Both RLOs and OERs may bring certain expectations (such as some sort of educational purposing or an interpretation of openness) but in practice, repositories offer many "resources" which have neither.The debate over definitions may appear to be very abstract and for practitioners the feeling is that "we just want to get on with it".However, the nature of an RLO is important in knowing what to expect, what its pedagogic potential is, how to find and best use the materials that are available, and how we can contribute as developers of resources.If we look for these objects to incorporate in our teaching, what will we encounter, how will we know what they contain and how can we judge their likely quality?In most of the following discussion the term RLO is used but it is viewed as a starting point for investigation of resource investigation rather than as an assumption of particular properties.
The above questions also underlie any attempt to survey and analyze the provision of RLOs.Should materials be discounted because they are not self-contained, or because they do not state learning objectives, or because there may be restrictions on their use?The problem is that whether features are classified as good or bad often depends on the user's perspective.A teacher may appreciate a carefully crafted, small RLO with well-defined objectives and an assessment.A learner may be happy to discover a full lecture course which has no explicit metadata but seems to cover the topic they require.Both resources may exist side by side in the same repository.Equally, they may not be in a repository at all.They may be just as useful in different ways.
These observations lead to a key principle that, from the practitioners' perspective, users need to be able to make an informed choice.Obviously, discovery is a vital first step, so the question of where and how to look is of major practical importance.The user then needs to know what each RLO contains and as much as possible about it, its educational context and its "quality" (perhaps through metadata, ratings, peer reports, user reviews, and so on).From a practical perspective, it seems best not to rule materials in or out because of detailed requirements on form or content, as long as there is easy access to relevant information.
Openness may be key for instructors who wish to reuse and repurpose teaching materials, and resources which are constrained to be used only in their original form due to the license situation cannot be thought of as fully reusable.However, the requirement may not be so important for an independent learner who simply wishes to use what is offered.Again, clear choice and the ability to select on these features may be most helpful in practice.

USING RLOS IN PRACTICE
The motivation for the current work comes from the perspective of instructors in Higher Education seeking to locate (and contribute to) tried and tested learning materials, and to incorporate these resources with their own, integrating and adapting them as necessary to fit a specific context.From the student perspective, it would also be useful to know how and where to locate learn-ing resources which could help supplement and elucidate those already provided by their instructors.In this section we introduce the general aspects of RLOs which render them useful and usable in practice and identify areas of concern which are then reviewed in subsequent sections.Most current reuse of resources is said to be internal to an institution [37].Principles as outlined below are of use internally but become even more necessary when the focus is on resources shared beyond institutional boundaries.

Discoverability
To be of greatest use, learning objects must be easily discoverable, openly available, of good quality, and fit for the intended learning outcomes.Discoverability includes aspects such as metadata and search facilities [38] but for the practitioner it also raises questions of whether materials are available, and if so, where.Currently there is no overall directory or way of knowing where, for any given subject of interest, resources reside, whether those resources are freely accessible, and if the resources provided are of good quality.There are some well-known (and some less well-known) repositories which provide fertile hunting grounds.However, if the user knows where to look, there are also a number of individual websites which make RLOs available.
For Higher Education in particular, various universities worldwide are happy to share some of their resources and these can be accessed and incorporated by other, external, users.However there is no comprehensive list of established repositories, yet alone any general index linking to other available resources judged to be open and repurposable.

Suitability
In identifying the factors which are critical in enabling effective reuse of learning components, Robson [39] places at the top of the list the location and the appropriate use of metadata which will enable discovery of relevant material.Simply finding what is available and where is a major hurdle, but further, if RLOs are not suitably presented and tagged then users quickly become frustrated in trying to discriminate between those which are useful and those which are not.Secondly, there may be issues surrounding the usage policies.Some LORs are set up as commercial ventures or do not have a suitable "open" license to allow sufficient flexibility of use.Thirdly, although something may be labeled as a reusable learning object, in practice, little thought may have gone into ensuring that it can be reused effectively.This relates to issues such as whether the RLO can be edited or not, or to whether educational context is evident.When considering the suitability of an RLO, these issues may be just as important as the subject area that the RLO covers.The granularity of an RLO is also seen as an influential factor [40] but may not be explicitly stated or not adequately described within the metadata framework [41].

Quality
Even when RLOs are located with suitable open access, are on the required subject, and have informative metadata, the question still remains as to how "good" the material is.Are there factors which give reasonable expectation of a good quality RLO, for example, the fact that it has been obtained from a known LOR, or that it has been produced using a particular methodology or conforming to a particular standard?This issue of quality is fundamental, but is also difficult to measure.If a user does not have confidence in finding RLOs which they feel are good enough to use then they are unlikely to persist in the search.Ways of achieving and assessing this "quality" must inevitably involve some degree of subjectivity, so again, from a user perspective, if any mark of quality is to form part of an informed choice it is important to know exactly what the basis of the judgment was.
The following sections survey existing work relating to the themes introduced in the previous paragraphs.

WHERE DO RLOS RESIDE?
Repositories which provide collections of RLOs (or links to them) can be an effective interface allowing authors to publish material and users to search for it.

The repository landscape
While some repositories hold RLOs directly, others act as a portal linking to RLOs held elsewhere.McGreal [42] classifies LORs according to three basic types: a centralized model with content stored on site, portals which mainly store links to material provided by others, and repositories with equal roles as content provider and portal.Within these types McGreal in 2008 lists 27, 20 and 14 LORs respectively.Amongst these, some cater for all subjects and levels of study while some are for a specific topic; some are already noted to be no longer active and some have issues of permissions and copyright.
There is a large variation in the number of RLOs provided by different repositories, ranging at the time of McGreal's review (2008) from just 8 RLOs in one centralized LOR (SOFIA, http://sofia.fhda.edu/) to 114,893 in a large portal (Intute, http://www.intute.ac.uk/).An influential factor in this is the granularity of the RLOs contained.SOFIA's 8 RLOs were complete courses whereas other repositories might hold many smaller learning components.It is interesting to note how LORs have developed, and MERLOT, for example, was recorded as containing 7408 RLOs in 2002, this had increased to 16,166 in 2008 and is currently (2012) over 38,000.Ochoa and Duval [3] provide further quantitative data and analysis on the growth and usage of LORs.One striking feature is that resources referred to as "learning object repositories" can vary greatly in nature, not just in terms of McGreal's type classification but also with respect to what is accepted as content.A repository such as MERLOT provides a searchable library of reusable RLOs with LOM-based metadata supporting a detailed description of the RLO's contents together with educational and access information.User rating and feedback are also recorded.In contrast some repositories consist simply of a web page with links to, for example, lecture slides.This variation relates to the question of what should be regarded as a reusable learning object and whether a resource stated (or classified by others) as being a learning object repository brings expectations beyond a collection of links or materials with no supporting information or educational metadata.

Where are the repositories?
Knowing which repositories are current and what each provides is not straightforward.Several published surveys have listed LORs and their attributes and provided some degree of analysis.In 2002 Neven and Duval's survey of repositories identified ten LORs containing RLOs with associated metadata [43].The LORs were compared according to factual and quantitative criteria such as the number of RLOs currently available and the metadata standard used (predominantly the IEEE Learning Object Metadata standard [9]).The survey does not attempt to assess the quality of RLOs or the suitability of matches returned when searching the LORs.It does however state that peer review would be a useful way to evaluate the RLOs.
Revisiting Neven and Duval's LORs today (ten years later) reveals that 4 of the 10 no longer exist.A further 2 have moved so that the stated links do not work and no redirection is provided.While some repositories become defunct, others are added to the scene.By the time of Tzikopoulos et al.'s 2007 survey [77], 59 LORs were found which were then examined with respect to three categories of characteristics.In this case, a "quality assurance" aspect is added.Although the survey is more recent, following up the LORs mentioned in it again reveals cases of websites no longer being maintained (as with Intute), links found to be broken, and references to repositories which are no longer operational (for example, CAREO, and Online Learning Net).This underlines the fact that, while survey exercises of this kind provide a useful snapshot of LOR provision at a given time, as with all digital resources they are likely to become out of date very quickly.For open (non commercial) repositories the main factor leading to decommissioning seems to be the difficulty in sustaining funding and hence effort, indicating the importance of sustainability [44,45].For some LORs, although there has been no explicit decommissioning and the link is accessible, there has been no evidence of activity for some years (no RLOs having been added or altered for over a decade) leading to doubts as to whether any maintenance is being provided.
As some LORs disappear, so new ones are established.However, finding out about these repositories can be surprisingly difficult.A Google search for the term "learning object repository" (conducted on 3 rd November 2011) offered (in the first 2 pages of results) just three extant repositories (Wisc-Online, MERLOT and VCILT), one which appears to have no content whatsoever (Informing Science Institute LOR), and one which is defunct (LORN).Even Wikipedia currently has no entry for "learning object repository" or "open educational repository".Several sites (such as Edutech wiki and a number of personal collections) attempt to list repositories.However, these clearly suffer from the same issues of outdatedness as the paper surveys.For example, of the 17 general repositories listed in Edutech wiki, 8 of the links did not work, and many well-known repositories are not included (as at 3 rd Nov 2011).Overall, a general searching process is somewhat frustrating and many LORs seem surprisingly elusive.LORs obviously share these difficulties with any other type of online resource, but it is a significant problem in an area where reuse is a defining factor.

Ownership and coverage
Amongst the LORs mentioned above, a number (including MERLOT, Jorum and Connexions) are both general in terms of content coverage and inclusive in terms of acceptance of material from those who wish to share.In addition to this type of repository, many universities (and some other organizations) make resources externally available but do not cater for hosting resources from outside.Provision may be promoted at an institution, faculty or departmental level or may be done informally by one or two enthusiastic members of staff.HKUST resources (http://www.cse.ust.hk/learning_objects/) and MIT Open Courseware (http://ocw.mit.edu/index.htm) are two very different examples of university sites making materials available.Not all such sites are associated with a single university as, for example, the UK Higher Education Academy's ICT resources (http://www.ics.heacademy.ac.uk/resources/rlos/rlo_repository.php) or the RLO-CETL collection (http://www.rlo-cetl.ac.uk/index.php).Some sites, such as the Edshare initiative at Southampton (http://www.edshare.soton.ac.uk/) promote sharing of local resources with additional restrictions on access for external users.
With thousands of universities and educational organizations worldwide it is likely that much very useful material exists, and indeed this is sometimes encountered when conducting a general Internet search.However, there is very little information on what is available at which institutions and what might be expected in terms of content or accompanying information.
When searching for RLOs for a particular purpose, subject specific repositories may also be of interest.A further distinguishing feature for repositories is the intended community of use with, for example, some aimed specifically at supporting sharing of resources and practice amongst school teachers.

FINDING RLOS
Many authors refer to the difficulty of retrieving appropriate RLOs even if the search is conducted within a single LOR.Ochoa [46] characterizes the indexing and retrieval facilities currently provided as still being in an immature stage and not yet embracing the advances of more general retrieval technologies.Najjar et al. [47] report the findings of an empirical study which indicate that the search tools are hard to use, do not organize information well and provide poor feedback (the study focused on the ARIADNE repository but the authors generalize these results to all major existing LORs).

Searching across repositories
The existence of many different repositories means that to find suitable RLOs the problem is multiplied by numerous separate searches each of which may vary in format.The "referatory" style of LOR brings together references to RLOs residing in differ -ing locations for the purpose of facilitating discovery.Using this type of LOR makes searching easier but means that provision may be less uniform, for example, there may be different terms of use or different approaches to metadata requirements for each RLO returned in a single search.The IMS Learning Object Discovery & Exchange (LODE) specification [48] is designed to facilitate the retrieval of RLOs across multiple repositories by providing a standard profile which brings together existing approaches within the learning domain.
A number of initiatives exist which bring together a "repository of repositories" although, again, the landscape changes quite rapidly.For example, the Learning Resource Exchange (http://lreforschools.eun.org/)provides access to multiple collections of resources for schools.Such collections may be specialized to a particular purpose, relate to certain included repositories only and may require user registration (for example, LORN).Further, some included resources may be commercial enterprises where payment is required.The difficulty of providing a general facility to allow searching and discovery of RLOs across repository boundaries is highlighted by Richards et al. [49].They introduce POOL (Portal for Online Objects in Learning) to harness the use of web browsing whilst making the actual location and distribution of resources transparent to the user.In this proposed architecture, a POOL is the top level of aggregation below which are arranged PONDs (community sites offering external interface) and SPLASH repositories (at the individual level).This is an interesting architecture which tries to encompass all levels from global accessibility to personal (possibly private) recording.However, it is somewhat different to the task of providing a general search mechanism across the worldwide population of web-accessible RLOs.In this case, different repositories with their own architectures and set-ups will be encountered in addition to many RLOs which may be accessible by web search but not via a specific repository.

Resource location by general searching
There is little published material relating directly to locating RLOs on the World Wide Web , and most research focuses on searches within LORs.Certainly, some RLOs may be found by a web search and some anecdotal evidence [50] suggests that this is sometimes a more efficient method of arriving at the same RLOs even when they exist in a repository.
Rosas et al. [51] study the strategies used by university teachers when trying to find relevant RLOs on the web by general Internet searching.They classify the problems encountered into three groups: problems common to all Internet searching, problems specific to RLOs, and problems related to language.The second category includes the issue of access, with individual portals proving difficult to search and some available to members only.Some require username and password entry, thus remaining opaque to web crawlers and search engines.With these problems in mind, Curlango-Rosas et al. [52] propose a Learning Object Search Tool Enhancer (LOBSTER).This provides an assisted searching mechanism which, in addition to topic, allows the user to search for specific digital format, preferred language and content structure.Search results are clustered for ease of locating the most relevant results The LOBSTER approach has the advantage of providing a familiar interface but with more support than a normal web search.However, it will still be prevented from finding material which is not immediately visible to a search engine.
There are many different proposals for providing effective federated searches and unifying cross-repository queries (for example, [53,54,55,56]).However, the great disparity between repository structures makes it difficult to achieve in practice.

REUSABILITY
For teachers and learners, reusability in practice is not simply a matter of discoverability, but is strongly related to quality and to the practicalities of being able to successfully use RLOs in different contexts.

Metadata
The appropriateness of the metadata is seen as a key factor as are the terms of reuse and ease of repurposing.Learning o bject metadata is information about a learning object and may include descriptive (e.g.subject), administrative (e.g. last updated) and structural (e.g.relationships between objects) elements [57].The IEEE Learning Object Metadata (LOM) standard [9] dates from 2002 and provides necessary context allowing learning objects to be selected, evaluated and used.Some LORs have standard requirements about metadata with LOM providing the obvious candidate of choice.However, some resources referred to as learning objects (either stand-alone or residing in a repository) are found to have little supporting information and certainly not in LOM form.
Sanz-Rodriguez et al. [58] have developed a measure of reusability based on assessments of cohesion, portability and size derived from the RLO's metadata.The results appear to be well aligned to manual ratings by users and experts, however the approach is limited by the accuracy and completeness of the metadata itself and complicated by the different types of metadata encountered.Najjar and Duval [59] study the way that users employ metadata when searching within the ARIADNE knowledge pool system.In addition to tracking the searches they study the usability of indexing tools.They conclude that such tools are generally better aligned to the metadata perspective than to the user (with terminology often copied directly from standards rather than being meaningful to the user).

Support for authors
It is perhaps understandable that materials, particularly open ones, are often not designed as RLOs and do not have useful metadata.In many cases they are teaching materials which have been successful in a particular context and which educators are making available through altruistic motives (or sometimes because of institutional policy).They may appear in LORs but have not been designed as learning objects (let alone reusable ones).Some repositories provide help and guidance for authors.A good ex-ample of this is Rice University's Connexions repository which provides a user-friendly 3-step guide for authors placed pro minently on the front page.Issues such as copyright are clearly addressed.MERLOT also provides a content builder tool while emphasizing support for a wide range of approaches and tools for creating learning materials.Many authors (for example, [60]) have acknowledged the need to assist authors further by automating the task of generating metadata.However, it is equally acknowledged that successfully generating good educational attributes automatically is very difficult.Hybrid systems for supporting partial automatic generation have also been suggested (such as [61]).However, in practice it is still the case that where existing materials are made available, often no educational metadata is added, either by hand or generated by tools.
One interesting initiative at the University of Southampton, EdShare [62], focuses on encouraging educators to share their material, making the task less daunting by reducing the requirements for metadata and content packaging.The philosophy is that the context provided by the system itself and by the user community will compensate for missing data.This leads to the perspective that "it is the pages generated by uploading resources that are analogous to Learning Objects, not the resources themselves" [62, p107].Providing support for authors' concerns such as required packaging effort and copyright issues gives them confidence to contribute.However, the in-house restriction option appears to be very popular and thus much of the material is not immediately available to the external viewer (but may be requested).

Evidence of reuse
The real test of the RLO concept is whether users can successfully access appropriate material and reuse it for their own needs.
There may be hundreds of thousands of RLOs available in general, but only a small number of those are likely to be relevant to a particular topic of interest at a particular level of study.Even fewer may contain high quality content which is free to be used and adapted.These need to be easily findable wherever they may reside.Elliot and Sweeney [63] report a case study in which RLOs were sought for incorporation into an online learning environment to provide resources for an inquiry project entitled "Diabetes: A waste of energy".RLOs were found by searching LORs and through general web searching.The experience seems to have been very positive in that of 77 RLOs returned, 37 were usable (with 36 requiring some modification).Only 2 additional ones were written to complete the resource set.The authors suggest that considerable savings in time were made as a result of this reuse.
The general picture seems to be less optimistic.Ochoa and Duval [64] conducted a study of reuse in which it was found that only around 20% of RLOs surveyed had been reused.The granularity of RLOs was noted but did not seem to have any effect on reuse (contrary to the view sometimes encountered that smaller grained RLOs are more likely to be reused).While 20% may seem a little disappointing, Ochoa and Duval conclude that this is no lower than reuse of other digital resources such as code in software libraries.Duncan [65] investigated the extent to which RLOs are reused within the Connexions repository (http://cnx.org/).One third of RLOs were never incorporated in other contexts, and a quarter of those reused were seen to be reused more than once.However, in nearly 80% of all cases of reuse, there was common authorship between the new module and the reused one.
A recent study by Clements and Pawalowski [66] surveyed teachers who had an established interest in using open educational resources.For simple reuse of objects the main barrier noted was that teachers wanted materials from their own country.
For repurposing and sharing activities, cultural and curriculum differences are again noted as a significant problem, with copyright issues also featuring as a major deterrent.This emphasis on the importance of location and culture is an interesting factor seemingly at odds with the idea of an RLO as an independent, reference-free unit of learning.This point seems to be the "Reusability Paradox" in a slightly different guise: the more contextualized and targeted the resource the more useful for teachers in a similar context but the more difficult to repurpose for others.

Licensing
The issue of copyright and terms of use has been mentioned in many studies.Elliot and Sweeney reported [63] that in creating a course from RLOs, the time taken to find, adapt and complete the set of RLOs (14 hours) was outweighed by the time taken to obtain permission for reuse (19 hours).In one case, commercial terms were requested.Further, "Many owners made specific requests, for example, as to how copyright statements should be worded, or to be given a ccess to the learning environment to see how their resource(s) was being used" ( [63] p141).This it places an additional burden on the developer to seek individual permissions and to ensure that all requirements are met and few users are prepared to go to this trouble.As noted by McGreal ([42 ] p6), "Licensing, subscriptions and other digital rights paraphernalia restrict the use of RLOs.The need for copyright clearances, contracts and licenses inhibit reuse.When access is hindered by these DRM devices, users just turn away.Open access overcomes these difficulties, facilitating access and the integration of content into lessons and courses."Connexions states that all its content is licensed under the Creative Commons Attribution License (CC-BY), but this still appears to be a difficult issue.MERLOT has obviously invested a lot of effort in considering and clarifying the conditions on acceptable use with a number of web pages (and a video) devoted to explanation, however, the situation is still difficult to understand.Submissions are required to fall under a Creative Commons by-nc-nd license, yet "the materials themselves" (which may be hosted elsewhere ) are not necessarily licensed in this way, individual RLOs come with both "Copyright" and "Creative Commons" metadata fields and both of these are very often given as "unsure".For example, a search on the term "computer science" returned 2,958 RLOS.Restricting the search so that only those resources with a definite "yes" for Creative Commons License are offered resulted in 766 RLOS.
The move towards clear and open licensing recognizes the great benefit to users of making the situation as transparent and as liberal as possible, while protecting contributors from misuse of material.However, some policies which are painstakingly set out (and presumably legally watertight) may be hard to interpret.Some repositories do not mandate such open access and, as indicated above, many different restrictions may be encountered.In some cases, payment to access RLOs is required.Further, repositories which promote an OER approach may still house resources which are not open.

Content models
Different formats not only complicate the location of suitable learning objects but also hinder the prospects for their aggregation and interoperability.Content models aim to provide a common basis for exchange and reuse.Content models describe units of learning resource with respect to granularity, content and architecture of compositionality.The shareable content object reference model (SCORM) [67] is a well-known and widely implemented set of specifications and standards.Self-contained SCORM learning objects (SCOs) may be aggregated to form higher level, compositional units.However, there are a variety of different models, with no common agreement on perspectives for granularity, content or aggregation description.
A comparison of nine content models is provided by Ververt and Duval [68] who also propose an ontology (ALOCOM) to support interoperability of learning content.Balatsoukas et al. [69] conduct a further survey in which they refer to the partial nature of existing ontologies and to the need for a unifying conceptualization for "defining holistically and exposing the nature and logical position of a learning object within the learning content hierarchy".
Although the difficulty of interoperability persists, much work has been done to build practical approaches and implementations from a particular content model perspective.SCORM is very commonly used in this respect and many recent publications continue the theme of SCORM-based development.

ASSESSING THE QUALITY OF RLOS
A notion of overall quality for RLOs involves a number of factors encompassing both pedagogic, technical and usability a spects.
A variety of different approaches exist which relate to quality issues and measures.

Designing for quality
One approach to ensuring high quality is to employ a methodology where RLOs are designed according to a set of design pri nciples.Krauss and Ally [26] use three strategies to develop and assess RLOs in order improve quality.Student engagement is seen as crucial and students are involved in providing early feedback on design and navigation issues, in reflecting on their interac -tion with the RLO, and later by reporting on their learning impact.Peer review is also recommended.This may produce very robust RLOs but seems to require a good deal of additional effort to achieve.Some authors have stressed the importance of good RLO design from a software engineering perspective to ensure properties such as cohesion and decoupling [70] and learning object patterns have been suggested as a way of capturing and reusing designs [71].Some design approaches seek to completely decouple learning content from pedagogical context [72].The Generative Learning Object approach views the pedagogic pattern at the heart of a learning object as being the key to reusability and educational value [73].A robust toolset and helpful documentation (www.glomaker.org)make this a practical as well as pedagogically attractive approach to developing RLOs from a newlydeveloped pattern or by modifying an existing one.
Effort put in at the development stage and adherence to principles of best practice in RLO design are likely to produce useful RLOs with good metadata.Some collections of RLOs have mandated the use of certain development methods or principles, for example, the Introductory Programming learning objects developed at London Metropolitan University provide detailed information on both the design approach and evaluation (www.londonmet.ac.uk/ltri/learningobjects).

Users' views
RLOs encountered in many repositories or by general searching may or may not have been constructed to conform to some pa rticular quality criteria, and it is therefore useful to have other measures to help assess their quality.One possibility mentioned by a number of authors is that of soliciting user evaluations which can be used to provide ratings or peer reviews to accompany the RLO itself [26,66].This requires both commitment from users to spend time providing evaluations and the capability from LORs to manage the information in the form of ratings, comments and reviews [43].Increasingly, repositories are providing this functionality (for example, Jorum, MERLOT and ARIADNE) and it appears to be a mechanism which users find very helpful.
Clements and Pawlowski's survey [66] found that users are inclined to put their trust in resources with high use-rates and good ratings and peer reviews.Davis et al. refer to the importance of an active community feeling ownership of a LOR which is then seen "not as an archive, but as a living online home for their materials" [62].
In addition to rankings and ratings many users have their own ideas about what they consider to be a "good" learning object.Some factors might be fairly straightforward (if not always easy to judge) such as general correctness of material and reputation of the organization providing the resources.However, users also link the idea of quality to "something they could not produce themselves" such as a clever animation or to resources which fit with the "lessons or curriculum of their country" [66].

Assessment instruments
Several different frameworks have been proposed within which RLOs can be assessed.These can be applied to existing RLOs and provide measures (in different forms) of quality based on a variety of aspects of the RLO's structure and content.

LORI
The Learning Object Review Instrument (LORI) (Vargo et al. [74], later refined in [75]) provides a common review format for making comparisons among RLOs in order to help users select for quality and suitability.Following LORI, reviewers evaluate each RLO by rating and commenting with respect to nine separate categories: content quality, learning goal alignment, feedback and adaptation, motivation, presentation, interaction usability, accessibility, reusability and standards compliance (Table 1).A rating is achieved (on a Likert scale of 1-5) for each category.These can be combined to provide an overall assessment for the RLO.Inevitably, ratings are subjective and there has been some criticism of the approach, but it provides a reasonable general measure of all-round quality and has been quite widely used.It has been recommended that LORI be used in conformance with a convergent participation model [7].In this approach, individual reviewers provide ratings which are then discussed communally and possibly modified before an integrated report is produced.In practice, it is often not possible to rate effectively on many categories as the information is so infrequently provided.Even if applied in "the approved manner" an instrument such as LORI cannot be unequivocal or objective.Design of visual and auditory information for enhanced learning and efficient mental processing Interaction Usability Ease of navigation, predictability of the user interface, and the quality of UI help features Accessibility Support for learners with disabilities Reusability Ability to port between different courses or learning contexts without modification Standards Compliance Adherence to international standards and specification

LOAM
The Learning Object Attribute Metric tool (LOAM) supports RLO evaluation with respect to assessment criteria derived from the IMS Learning Design framework [76].An interactive tool supports entry of evaluation and representation of results.For the three categories (environment (media type), learner role and activity) the evaluator assesses the proportion of the RLO or user activity which falls within various predefined possibilities.These three categories are subdivided with respect to pedagogic attributes which are each assessed on a Likert scale of 1-5 (note that interactivity, self-direction and richness are each shared between two categories) (Table 2).

Category Pedagogic attributes
Environment (media type) Richness, context, integration of media types, objective, interactivity Learner role Self-direction, feedback, support, prerequisites, richness Activity Interactivity, alignment, assessment, navigation, self-direction There are obvious similarities with the LORI approach within the pedagogic attributes.There are also some significant differences in the way that LOAM links to learning design and in the actual content and interpretation of the categories.The level descriptors for the LOAM categories appear to be more clearly defined and it would thus be expected that subjectivity would be reduced.On the other hand, LORI tries to capture some less easily defined ideas in the category of "content quality" which might be seen as the real point of the exercise and yet may be more subjective and which might also require detailed knowledge of the area to judge.

LOEM
The Learning Object Evaluation Metric (LOEM) is a multi-component model based on a methodical and comprehensive r eview of the literature on instructional design (Table 3).The reliability and validity of LOEM has been assessed using a large sample of students and teachers who selected a wide range of LOs from a variety of repositories [78].Again, there are similarities with LORI and LOAM, but LOEM has not yet been developed into a tool for assessing LOs, and there has been no standardization of the measurements used.

Other measures
Eguigure et al. [79] describe an evaluation approach encompassing the views of different stake-holders.Vuorikari et al. [80] consider 13 different evaluation approaches.They make the point that evaluations are often tailored to the specific needs and in -terests of a particular setting (such as a repository or location) making it extremely difficult for an end user to know exactly what is guaranteed by any given "evaluated" RLO.For work which looks at RLOs drawn from many different sources, the means used to assess and compare them has to be as general as possible.The categorizations provided by LORI or LOAM constitute a general framework for comparison which is more suited to this purpose.

Provenance
If RLOs are being sourced from an LOR, one obvious question is whether the LOR itself provides any measure of quality assurance.McGreal [42] reports that this is patchy -some repositories have various different measures in place, although these are not necessarily used, while others accept all entries.Tzikopoulos et al. [77] found that 64% of LORs follow some sort of quality policy for the RLOs they include, generally with guidelines provided to authors (although it is unclear how strictly the guidance is enforced).It was also reported that 43% of LORs have some evaluation or reviewing policy.So, even if an RLO is obtained from a known repository, it is not necessarily the case that it will have been subjected to any restrictions related to quality and in cases where policies do exist, the notion of "quality" may well differ between repositories.
Support for ratings and evaluations from experts and users have been mentioned above.Again, this is another area where provision varies greatly between repositories making it difficult to compare evaluations when sourcing RLOs across several repositories.There is thus a need to develop a framework to encompass the differences, such as a metadata model for evaluation information [80].
LORs which provide links to RLOs face the usual problem of ensuring liveness and quite often links are found to be broken.
Thus even the most basic quality issue of existence is a challenge.The issues concerning repositories noted in Section 5 mean that the long term future and sustainability of repositories themselves cannot be guaranteed and that the currency of material contained within some LORs is an issue.

Educational outcome
Some authors have questioned the effectiveness of judging RLOs outside the learning context in which they are used.Nurmi and Jaakkola state: "LOs per se are not necessarily good, bad or adequate, but the ways they are implemented, and the learning environments created around them, will determine their pedagogical value" [81].There has been some attempt to gauge the effect of RLOs embedded in a real educational context by assessing learning outcomes.This tends to be a more detailed undertaking as it requires tracking users of the resources and measuring the learning outcomes by assessing understanding of the topics studied.It is therefore difficult to conduct such studies with large numbers of learning objects.In [82 ], Akpinar assesses 8 RLOs by means of pre and post test scores.All RLOs were rated at broadly similar LORI levels.In all but one case test scores i ncreased -in the remaining instance students fared worse after working with the RLO.Further work by Akpinar [83] applies pre and post testing to 24 RLOs, using the results to compare with LORI evaluations.It was found that LORI scores did not provide a good indication of learning outcomes.

Aggregation
Comparisons so far seem to suggest that there is little correlation between the various forms of quality assessment.Sanz-Rodriguez et al. suggest that the different assessments of RLOs "constitute different views of their quality that might complement one another" [86].An aggregated approach which combines different measures might therefore provide a more rounded picture.
Their aggregated approach considers the MERLOT quality indicators but could be generalized to encompass the schemes from other LORs.The more repositories include their own mechanisms, the greater the need will be for comparative or aggregated approaches.In addition to different rating and evaluation systems being encountered in different repositories, it has been found that even within a single system the evaluation mechanisms may not be used consistently.For example, work with the MERLOT rating categories [84] suggests that different groups are in fact likely to be viewing rating categories in completely different ways.Consistency and alignment of measures remains an issue for research but standardization in some areas would certainly be of benefit to the user.

FACTORS AFFECTING REUSE IN PRACTICE
Efforts to map where RLOs can be found have usually been confined to surveys of general repositories, with focus on comparison across characteristics which may be general (such as size), technical (such as metadata standard used) or quality-related (for example, review policy).Even this is problematic, firstly because the changing nature of resources means that currency is difficult to maintain, and secondly, the lack of clarity in definitions makes decisions on inclusion somewhat difficult -is a page of links an LOR?The literature confirms that there has been very little work which casts the net wider to consider, for example, institution-hosted collections which may nevertheless provide high quality material.This may be particularly relevant for subject-specific provision, but locating such material is not easy.
In terms of assessing the quality of learning objects themselves interest has focused mainly on specific groups of RLOs, such as RLOs within one repository or RLOs specially selected (or indeed specially authored) for a particular project.Another feature of some of these studies (such as [82,76]) is that the RLOs have been preselected as being "good" in some sense (such as having a reasonable LORI score, having been designed using a rigorous method, or having been subjected to quality assurance tests).In general, when performing a search to find learning materials from general repositories or searching to incorporate in teaching, the RLOs encountered will be much more diverse.Overly stringent requirements (even to the extent of demanding a clearly de -fined learning objective) may leave very few resources to consider.
Following the ideas and uncertainties discussed above, and working from the perspective of an educator searching for materials which may be useful in practice, the following issues are highlighted as affecting the discovery of RLOs and judging their general quality.This gives a different focus from previous surveys and emphasizes the re-user's requirements in discovering materials which are fit for purpose.The first group of points relate to technical aspects, the second focus on quality.

Technical aspects
Open access.Users need to be able to find RLOs and, having found them, require clarity on the copyright and license situation.They might expect reasonable, educational reuse of the material to be permitted free of charge and for this permission to be clearly stated.Existing studies indicate that the question of ease of access is regarded as being an extremely important one for the purposes of assembling teaching materials.They also indicate that currently the user is likely to have to spend a considerable amount of time following up the situation on copyright and obtaining permissions.The move by several major repositories to insist on Creative Commons licensing is welcome, but the general position remains unclear.Many repositories allow users to specify different terms of use and even to ask for payment.
Passwords.Repositories may be password protected for entirely reasonable grounds such as the need for a website to protect itself from malicious users.Users may be allowed to register freely with no constraints other than the requirement to present a valid address and set up a password.This is a common requirement and users are generally happy to register for respected sites.However, password protection can make it more difficult for users to find materials with a general search.Web crawlers and search engines may not discover material for which login is required Clear information about the RLO.The user needs to know details about the RLO which will enable them to reject as soon as possible those that are not suitable.This is the task of the metadata.As well as subject and nature of content it is extremely useful to have a clear idea of the author's intended learning objectives, level of study, prerequisites, etc. Experience so far suggests that this is very patchy indeed.There is an acknowledged tension between onerous metadata requirements and encouraging contribution.However, the educational purpose and use of many "learning resources" currently encountered remains hidden.
Repurposing.Material may have to be adapted.This should be permitted and enabled by the format as well as the licensing where appropriate.Even in terms of technology, there is no "best way" to produce a multi-media object.However, most resources from general repositories have no multimedia element and many are at the level of a single image or a PDF document.
Context and cultural reference.To be reusable, learning materials should not be dependent on external references or resources which make it difficult to extract them to a new context.All RLOs will rely to some extent on external systems for de -livery and prerequisite knowledge at some level.They will also be coming from some context and some cultural perspective.
RLOs may thus be inaccessible to many if they rely on resources which are not freely and easily available or make specific references which may not be familiar to all students.However, user studies indicate that RLOs which relate to a particular context, culture or curriculum are found particularly useful -provided it is the right one.It seems that the best approach for the user is one of informed choice.If the user can clearly see that the material relates for example to a specific textbook or a particular examination then they can decide whether to inspect it further.
Granularity.Appropriateness will obviously vary according to the user's intent.Within the context of harnessing and putting together existing materials, "medium sized" units will probably be of most use, that is RLOs which are smaller than an existing full course.At one end of the scale, very fine-grained RLOs may be difficult to capture accurately with metadata and to put together in a seamless way.At the other extreme, full courses tend to be suitable for those who want a whole package rather than reuse and repurposing.Wiley's discussion [87] of the importance of context emphasises the need for metadata to give more information on internal context for larger RLOs, and more information on previous external contexts for smaller ones.There is no "right" or "wrong" size, but informed choice relies on the available information.
Basic sanity.Users do not want to be confronted with broken links and missing resources.It is frustrating to be presented with a "learning object" which turns out to have no content or not to relate at all to the subject it is supposed to cover.Surveys which pre-select material to assess will not encounter such problems, but in practice they arise very frequently.

Quality aspects
Of the aspects listed in Section 7 the most appropriate for general assessment by a user appear to be provenance and user ratings/evaluations.Use of a quality instrument may be applicable, but this is not something a user is likely to undertake at the point of searching for materials.A convergent model of application may also be required for validity.
If an RLO is found within an LOR then the repository's policy and provision will be relevant.For RLOs made available by an individual university or department one basic measure might be that the RLO has been developed and used as part of a validated course of study.Not only will the material have been used in practice but it should have passed the quality criteria of the institution.The content of the course (if not the RLO itself) will have been scrutinized by an external examiner or equivalent.
However, this information may not be readily available.Ratings and evaluations are unlikely to be found for RLOs outside major repositories.
Some categories in instruments such as LORI and LOAM relate to technical aspects as above.However, they also provide more specific measures of content quality.An additional consideration for quality is how current the materials are.For a subject such as Computer Science some materials may become out of date or of historic interest only.Surprisingly, many repositories do not provide users with information on when resources were added or when they were updated.Some LORs have a policy for suggested review time.However, this is generally indicative only and in practice materials may be left unrevised.The maturity of LORs and the fact that some of them have now been active for up to 20 years [77] mean that this is now becoming a significant issue.
Following the issues outlined above, we are currently conducting a study of sites containing open RLOs to map the general landscape and provide a qualitative assessment of materials found.

Discussion and Further Work
The debate over RLOs and LORs, terminology and meanings can leave the user somewhat confused.It seems to be hardly su rprising that access rates for learning objects are low in general and that the majority of the reuse is by the authors of the RLOs themselves.Discoverability is an issue for web resources in general, but finding LORs does appear particularly elusive for resources which exist to be findable and searchable.The issues of disappearing resources, broken links and resources with outdated content or indeed no content may be no greater than for other areas but, unless the results of perseverance are likely to be beneficial (with materials found more quickly than instructors can produce them themselves, or an innovative presentation) instructors are unlikely to persist.It is widely noted (for example [88]) that users of open RLOs need to have realistic expectations, looking for materials to enrich and supplement teaching rather than looking for a route to quick course development.Even within these limits, the reality of what is found in practice can be very disappointing, which can in turn discourage the user from becoming more involved and sharing their own practice and resources.
Despite the difficulties, there are very positive signs in a number of areas.Some of the well-known repositories ( Jorum, Connexions and MERLOT have already been mentioned) continue to grow and newer ones (such as Edshare and Sophia) are being introduced.The sites themselves are developing to provide a more useful and usable interface with support for authors and information for users, although there is still some way to go with this.Increasingly popular is the provision of peer review and user commenting.Provision is gradually moving towards an "LOR 2.0" perspective, encompassing aspects of Web 2.0 strategies as envisaged by Monge et al. [86].These strategies serve not just to provide useful information for users but also create a sense of ownership and involvement.Problems with resources such as errors, omissions and dated content can be policed by the whole community.As suggested by Davis et al. [62], improving the practice of sharing is key to the development of good repositories of resources.This applies not just to the authors, but also the users of materials who become involved and responsible through their feedback.A repository can thus become an organic expression of an educational community's efforts rather than a static shelving system for old material.
The increasing body of resources is another positive sign.Some have been created as part of dedicated projects and, as well as demonstrating high educational and presentational quality, focus on reuse and repurposing.Others are made available after being used by teachers in their own work and evidence the best principles of passing on tried and tested learning resources in reusable form.However, there are still problems both in locating RLOs and in the general quality of what is retrieved from open repositories.Many educational resources exist outside the well-known repositories and there is little help available in finding smaller repositories or materials available at a single institution.Even within some larger repositories, lack of indexing by search engines and no easy means of recommendation via hyperlinks can leave RLOs cut off from the wider Internet, There are also still many issues to do with copyright and licenses.Major repositories are well aware of this and the push towards Creative Commons licenses is to be welcomed.However, there is still a good deal of room for confusion even in repositories where the policy has been defined and set out in detail.Requirements are different from repository to repository and each needs to be checked individually.The intricacies of a policy may be difficult to understand and take some investigation to unravel.Further, even if there is a policy it may not apply to all materials located via the repository since it may well exclude everything which is linked to rather than held by the repository.The large number of resources found where the author has stated the licensing position as "unsure" indicates the level of uncertainty and creates a barrier for users.Checking individual RLOs, contacting authors to enquire terms and ensuring specific requirements are met does not encourage reuse.
Some repositories seem unnecessarily restrictive in the information available to users.For example, the date of entry is obviously known in some repositories but users are not offered an interface to search on this.Some repositories provide infrastructure for useful features (such as rating and commenting) but the features are not always well-used in practice.Further, there is little coordination between LORs and, despite a variety of medium-sized projects, there has been no umbrella initiative large enough to provide overall coordination for meaningful interoperation or standardization (this despite the fact that the LOM standard for Learning Object Metadata was the first e-learning standard [85]).New repositories seem to add to the divergence rather than work towards any unified interface.
Another issue apparent from this review is the rate at which repositories and collections become defunct.Smaller ones, perhaps developed as part of a project for which funding has come to an end, may find difficulty in ensuring continuity.This is common to many web resources and in the case of LORs may leave an abandoned or unmaintained collection of resources.It is now noticeable that there are dead links and unmaintained sites which once used to be well thought of repositories.What will happen to the resources housed in some of these is unclear.Further, the maturity of the area means that a number of repositories are old enough to include material which may be seriously out of date and misleading, particularly for subjects such as computing where the field changes rapidly.
This review covers some of the measures by which quality of an RLO may be assessed.However, the judgment of what does or does not make a "good" RLO will vary greatly depending on requirements and context.For example, although some defini-tions lean towards a particular granularity of object, the case can be made for the appropriateness of many different sized chunks in different circumstances.Only the user knows what is appropriate for their particular needs.Again, an RLO is often characterized as being "self-contained" and "stand-alone" yet users request more resources linked to a context -as long as it is their particular context.The difficulties posed by decontextualizing learning material have already been noted.It seems that the possibility of repurposing a resource and rendering it suitable for a particular situation or pedagogic strategy are basic requirements.As a minimum, this requires the ability and license to edit material.Again, rather than a static, resource-push model, the ability to repurpose and adapt can lead to a variety of related views of an original object which are more learner-focused.Many resources currently found in open repositories have no information to provide context and their fitness for discoverability and reuse is very limited.
Another user perspective on what makes a good RLO is that it should be something they could not easily achieve for themself.This includes both content (for instance, novel examples or approaches) and presentation.In terms of presentation, users very much appreciate the use of different media and the employment of appealing and engaging visual displays and interactions.
It is also a measure of quality in instruments such as LORI that a multimedia approach is taken (although currently only a very small proportion of RLOs appear to offer anything other than a linear, textual presentation).This raises one very basic problem for the authors of RLOs: how should the materials be produced?For some time, Adobe Flash has provided a standard platform for web based materials and this has often been used to create RLOs.However, with the limitations of Flash and its future now in question there is uncertainty over the best tools to use.While HTML5 provides the core technology, standard tools have yet to emerge.This is a gap that will doubtless be filled in due course but currently poses a problem to authors.A general assessment of the quality of an RLO is likely to involve categorization of a number of aspects which are, in themselves, neither good nor bad.From a user perspective, an informed choice of material can only be made if the necessary information is available.A primary need is therefore to have information (metadata) clearly defined and easily accessible.Currently, materials are very often found with very little information and issues such as learning objectives and level of study become a matter of detective work.
Of course, there is tension between the usefulness to a user and the burden placed upon the author.Growth in the number of materials published but lack of reuse by users other than the original authors may suggest that at the moment the reusability aspects are not being well enough addressed.
Our findings indicate that for those seeking reusable learning materials some of the most basic requirements are still the most elusive.These include: 1) knowing where to find the materials; 2) finding the license/copyright position clearly stated (and preferably a recognized open use policy); 3) finding at least some basic educational metadata; 4) LORs enforcing basic filters before RLOs are accepted (for example, ensuring that there is some content, that copyright stated, etc.); 5) LORs providing mechanisms for users to report broken links and support rating/feedback; 6) LORs providing a good search mechanism.
Obvious as these points may seem, they are not currently as widespread as would be hoped.
Whilst the infrastructure is undoubtedly vital for the effective use of RLOs, in practice the benefits of RLOs are also limited by what is available for the subject of interest.We are currently engaged in on-going work to map the provision of resources and to assess the general quality of available materials.
LORs such as Jorum (http://www.Jorum.ac.uk/) have championed the publication of RLOs with open licenses (such as Creative Commons) which allow their reuse free of financial and other restrictions.JorumOpen provided an initial collection of open resources created and shared by the UK Further and Higher Education community, and, since August 2011, Jorum has moved to make all resources open.