ART2

Knowledge Organization
An exploratory study of user perception in visual search

interfaces based on SKOS
Journal: Knowledge Organization
Manuscript ID KO-2015-0089.R3
Manuscript Type: Article

Fo
r Re
vi
ew
On
ly
https://mc04.manuscriptcentral.com/jisko
Page 1 of 41 Knowledge Organization
1
2
3 An exploratory study of user perception in visual search interfaces based on SKOS
4
5
6
Abstract
7
8 Repositories are web portals that provide access to learning objects. Resources can be easily
9
located through the use of metadata, an important factor to increase the ease of searching for
10
11 digital resources in repositories. However, there are as yet no similarly effective methods in
12 order to increase access to learning objects. The main goal of this paper is to offer an
13 alternative search system to improve access to academic learning objects and publications for
14 several repositories through the use of information visualisation and Simple Knowledge
15 Organization Schemes (SKOS). To this end we have developed a visual framework and have
16
used the Organic.Edunet and AGRIS as case studies in order to access academic and scientific
17
18 publications resources, respectively. In this paper we present the results of our work through a
Fo
19 test aimed at evaluating the whole visual framework, and offer recommendations on how to
20 integrate this type of visual search into academic repositories based on SKOS.
21
22 Article Classification:
r
23
24 Research Paper
Re
25
26
27 Keywords: information visualisation, learning objects, semantic web, simple knowledge
28 organization system, technology acceptance model.
vi
29
30
31
ew
32 1. Introduction
33
34 The development of repositories is one of the most widespread initiatives for centralising
35 search processes in the field of learning objects. The purpose of repositories is to aggregate
36 learning resources from different content providers while enabling them to update their own
On
37 resources (Tatnall 2009) and to store educational materials with the objective of optimising
38
the management and search processes. The evolution of digital repositories has enabled
39
40 management centralisation and provided access to thousands of digital resources. However,
41 the rapid development of educational materials has posed a challenge to the creators of digital
ly
42 repositories that aim to organise, classify and manage content. Over the last few years, one of
43 the most widely accepted strategies to address these needs has been the use of classification
44 tools based on knowledge representation schemes.
45
46 The large number of digital resources has generated limitations that, in some cases, affect the
47 use of repositories for accessing educational materials. These limitations have led to the
48
development and use of alternative solutions based on the implementation of markup
49
50 languages (semantic languages) and knowledge representation schemes which are used to
51 facilitate classification, categorisation, linking and content management. The use of these
52 strategies based on Simple Knowledge Organization Systems (SKOS) has led to a solid
53 technological structure, for linked to a series of semantic enrichment strategies (Rajabi, Sicilia,
54 and Sanchez-Alonso 2015), which allow certain management and administration activities to
55
creators and developers of repositories.
56
57
58
59
60
Knowledge Organization Page 2 of 41
1
2
3 Currently most digital repositories include SKOS like ontologies, thesauri, etc., in order to offer
4 creators and expert’s alternatives to manage, sort and organize learning objects within a
5 repository. These alternatives, on the other hand, facilitate and make possible the use of the
6 repository to help users displaying additional information, identify relationships, categories,
7
8
areas of expertise, etc. However, these classification schemes are sometimes not used and
9 exploited to the extent that would be desirable because for common users like students,
10 teachers and researchers the use of such educational repositories is: i) not always easy due to
11 their interfaces do not always offer adequate search strategies, ii) common users are not
12 familiar with the complex systems of knowledge representation, which experts and creators of
13 repositories repository creators are used to classifying resources, and iii) the current user
14
15 interfaces and search mechanisms do not always properly provide the essential functionalities
16 demanded by the users for the location of digital resources. The difficulty of operating these
17 repositories can hamper learning processes and consequently lead to the eventual
18 abandonment of these tools.
Fo
19
20 Previous related studies has found that navigation problems in repositories may occur when
21 users try to return to a previously accessed record (Jeng 2005), beside limitations related to
22 the optimal combination of navigation and search methods (Hartson, Shivakumar, and Pérez-
r
23 Quiñones 2004) that do not allow displaying (in at-a-glance way) the materials available in the
24
repository, as far as the thematic coverage is concerned (Hitchcock et al. 2003), (Tsakonas and
Re
25
26 Papatheodorou 2008) and (Rho 2014). Therefore, it is difficult to judge whether it is worth
27 searching for materials in the repository, or if it is more convenient and effective to use some
28 other external search strategy (e.g. using general purpose search engines like Google).
vi
29 However still, there are no effective methods to design visual search interfaces through the
30 use of knowledge representation schemes, and more specific the few studies of visualisation
31
ew
32
tools focus on search engines to access resources trough only one repository. Moreover, the
33 majority of these efforts do not include knowledge organization systems, as simple method to
34 integrate in navigational search interfaces, in order to access resources in several repositories.
35
36 The use of graphical sintax is a method used in several studies (Smart et al. 2008), (Russell et
On
37 al. 2009) and (Silvis and Alexander 2014), our purpose is i) to exploit this graphical-capabilities
38 based on knowledge area through the use of knowledge organization system, in order to
39 search relevance resources hosted in several repositories, and ii) analyse results according user
40 perception and performance of visual search interfaces. In consequence, this paper
41
ly
investigates whether through information visualisation techniques, we can help creators of

42
43 digital repositories to provide better services for their users in order to: i) integrate several
44 repositories trough a unique point of service in order to search resources according to a
45 navigational knowledge area structure, ii) locate materials in a more effective and precise
46 manner in very-large collections of several repositories, iii) help users to improve the location
47 of materials according to a thematic structure; and, finally iv) identify effective visual search
48
interfaces according to the context and criteria search for performing browsing and searching
49
50 processes over several repositories. This paper offers an alternative method of access to the
51 creators of repositories in order to improve search outcomes for learning objects in several
52 repositories. This alternative relies on the use of visual search interfaces classified according to
53 Simple Knowledge Organization System - SKOS. We use Organic.Edunet (Manouselis et al.
54 2009) and VOA3R/AGRIS (Šimek et al. 2012) as case study repositories. For the purposes of this
55
56
paper we focus on the current capacity of repositories to integrate effective data visualisation.
57 We propose a formal framework for the effective visual searching of learning objects that will
58 satisfy the basic needs of repositories.
59
60
1
2
3 The second section of this paper will provide background information on repositories to carry
4 out the access of learning objects with respect to i) Organic.Edunet and VOA3R/AGRIS as
5 metadata repositories and knowledge representation schemes, ii) the limitations of metadata
6 and visual search interfaces in repositories, and iii) related work in the field. In the third
7
8
section we describe the methodology for developing the proposed visual framework, including
9 the details of evaluation of this visual framework used for our case study. Section 4 presents
10 the results of this evaluation, giving special attention to user satisfaction with regard to the
11 visual framework and suitable visual interfaces. Section 5 discusses these results. The sixth and
12 final section presents conclusions and outlines the authors’ intention to integrate this type of
13 visual search into learning repositories in future work.
14
15 2. Background
16
17 In general, the user interface of digital repository search systems provides at least two search
18 alternatives: simple (also known as basic) and advanced. The basic search does not require the
Fo
19 user to have a deep knowledge of the system and/or the search process. It allows the user to
20
perform a quick search but restricts them to the use of keywords and precise queries. The
21
22 basic search mode is considered easy to understand and use without prior experience. On the
r
23 other hand advanced search normally are related with additional mechanisms and options that
24 users select in order to obtain a refined search based on to user needs. e.g. boolean search, an
Re
25 strategy defined from logical search filters (AND/OR) by associating keywords and some
26 selection criteria in order to search resources based on title, author, year, etc. Faceted search
27
(Tunkelang 2009), a visual method for search based on classification strategies of digital
28
resources like: keywords, location, language, country, type of resource, etc. More knowledge
vi
29
30 and skills are required to use the advanced method. Understandably, then, basic searches are
31 the most popular. However, a search for learning objects in repositories could return a huge
ew
32 list of results if definitions of essential elements that help the refinement of the search are not
33 provided.
34
35 Previous studies have found that some search interfaces do not completely meet the needs of
36 the final users (Tenopir 2003). Often, the results displayed are not relevant to the user-defined
On
37
criteria (Nash 2005). Moreover, navigation problems may occur when users try to return to
38
39 previously consulted records (Jeng 2005). Regarding this (Hee Kim and Ho Kim 2008) found a
40 number of problems associated with the design of the interface of an institutional repository in
41 Korea. The study showed that the topics of interest queried by users were not sufficiently
ly
42 visible, because the navigation menu was too small and too dark. Thus, the design of an
43 interface is of a paramount importance as it can greatly facilitate the search process and
44
significantly improve user satisfaction (Shneiderman et al. 2005).
45
46 Below, we provide brief background information on repositories, using the Organic.Edunet and
47
AGRIS repositories as examples. We also outline the importance of knowledge representation
48
49 schemes, some of the obstacles to accessing learning objects, and finally related work.
50
2.1 Educational repositories
51
52 Organic.Edunet is a pan-European service (www.organic-edunet.eu) that facilitates the access,
53
use and exploration of learning objects related to organic agriculture and agro-ecology
54
55 (Manouselis et al. 2009). It pretend to displays a federation of multilingual repositories for
56 quality learning objects in order to facilitate the search and access of digital resources hosted
57 in different repositories. Actually, this repository contains already more than 12,000 learning
58 objects in the form of images, text and videos.
59
60
1
2
3 VOA3R (Šimek et al. 2012) is a pan-European online service developed in a project funded by
4 the European Commission. It is a service provider for the integration of existing open access
5 repositories and libraries, sharing scientific and open access research associated with
6 agriculture, food and the aquaculture environment. VOA3R systematically creates
7
8
relationships between metadata and learning objects through the use of the AGROVOC
9 thesaurus keywords (Agrovoc 2016), thus relating research topics in the VOA3R/AGRIS
10 repositories. The main purpose of the VOA3R platform is to reuse mature metadata based on
11 the assessment of metadata quality of digital resources which include inconsistency
12 (Palavitsinis, Manouselis, and Karagiannidis 2013), (Sanz-Rodriguez, Dodero, and Sanchez-
13 Alonso 2011), (Kumar, Nesbit, and Han 2005) and (Shreeves et al. 2005), re-dundancy
14
15 (Palavitsinis, Manouselis, and Karagiannidis 2013), (Ochoa and Duval 2009), (Barton, Currier,
16 and Hey 2003) and ambiguity (Cechinel, Sánchez-Alonso, and Sicilia 2009), (Paulo Alonso
17 Gaona-García, Sánchez-Alonso, and Montenegro Marín 2014), (Lytras and Sicilia 2007), (Park
18 2009). This conditions would improve the establishment of a robust community focused on its
Fo
19 services (e.g. sharing and increase the quality of learning objects) and to retrieve relevant open
20 content from scientific publications. In general the use of mature metadata improve the
21
22 development of repositories in order to: i) classifying and indexing of educational materials
r
23 (Xavier Ochoa and Duval 2006), (Stuckenschmidt and Van Harmelen 2004); ii) reusability of
24 educational materials in open repositories (Cervera et al. 2009), (Sanz-Rodriguez, Dodero, and
Re
25 Sanchez-Alonso 2011); and iii) the location of relevant materials (Russell et al. 2009), (Cechinel
26 et al. 2012), (Ochoa and Duval 2009).
27
28 Both Organic.Edunet and VOA3R/AGRIS use a knowledge organisation system (KOS) based on a
vi
29 controlled vocabulary to classify their resources. Organic.Edunet’s learning materials are

30 organised based on an organic agriculture and agro-ecology (OA-AE) ontology. This OA-AE
31
ew
32
ontology is stored and published by a Web tool (Mooki-Tool 2016). AGRIS, on the other hand,
33 uses AGROVOC terms linked to many datasets of the linked open data LOD (Bizer, Heath, and
34 Berners-Lee 2009) (e.g., DBpedia), which allows users to take advantage of a web of linked
35 data.
36
On
37 The OA-EA ontology also uses mappings of OA-AE concepts onto terms defined in AGROVOC
38 (Sánchez-Alonso and Sicilia 2009). AGROVOC is a mature thesaurus that provides a rich
39 vocabulary of terminology associated with agriculture, forestry, food and related domains. For
40 this reason, we use the same KOS based on the recommendation by (Martín-Moncunill et al.
41
ly
2015), which is linked to both repositories in order to access learning objects (Organic-Lingua
42
43 2016) and scientific publications (VOA3R/AGRIS) (VOA3R-Agris 2016) related to Organic
44 Agriculture and Agroecology.
45
46 2.2 Knowledge representation schemes
47
Some repositories integrate a knowledge representation scheme, such as a thesaurus or
48
49 ontology, to better represent and classify learning objects. The representation scheme
50 includes data structures that are defined by the use of tables, trees, links, graphs, etc. Each
51 data structure has advantages and disadvantages when it comes to the representation of
52 different types of knowledge. Advantage e.g. i) (Uddin and Janecek 2007) found that through
53 the use of multimentional taxonomies users could to improve the location of resources, ii)
54
(Stafford et al. 2008) evaluated a bilingual version of a thesaurus-based GUI, and found that
55
56 the integration of search and navigation capabilities were useful for access to digital resources.
57 Meanwhile one of the most representative disadvantages is associated with the rigidity of the
58 scheme and levels of reasoning (Gašević, Djurić, and Devedžić 2009). Other studies have
59
60
1
2
3 identified errors associated with the definition of an ontology taxonomy (Gómez-Perez 1999),
4 (Noshairwan, Qadir, and Fahad 2007). According to Gašević and Devedžić (2009) there is no
5 generic method for knowledge representation that could serve as the standard for structuring
6 data in all cases. In order to obtain an overview of these knowledge representation structures,
7
8
each user need to navigate through the interface, an essential attribute in the field of
9 information visualisation (Graham, Kennedy, and Benyon 2000) to carry out an evaluation
10 process of user-interface.
11
12 In some cases knowledge is represented as raw data, often stored as complex structures
13 according to logical sequences, rules, trees, semantic graphs and other forms of representation
14 (Flouris, Plexousakis, and Antoniou 2003). These structures correspond to a formal
15 classification, and, in the best case, to semantic relationships of the data at the level of
16 dependence, association, affinity, etc. Each knowledge representation technique requires a
17
form of notation that include aspects of metadata records related to the subject area, level of
18
Fo
19 affinity, links and forms of association, etc. To be effective in the context of learning
20 environment, a knowledge representation scheme must be consistent. It must be detailed as
21 possible in order to represent a subject area and its connections to other subject areas
22 (Chrysafiadi and Virvou 2013). This form of representation allows, among other things, the
r
23 expansion or limitation of the knowledge offered to users of the scheme. The tools most often
24
used to design knowledge representation schemes for repositories include taxonomies,
Re
25
26 ontologies, thesauri, graphs and mind maps.
27
2.3 Limitations of repositories
28
vi
29 The use of knowledge representation schemes in repositories is to facilitate organization and

30
search processes related to specific topics or knowledge areas. However, a search for learning
31
ew
32 objects could return an unmanageably long list of results if the creations of essential elements
33 for refining the search are not provided. If indicators for assessing the quality of recorded
34 information are not provided, searching for learning objects may be seen as entailing a
35 considerable waste of time and effort (Kumar, Nesbit, and Han 2005).
36
On
37 Recent studies of usability in repositories have also revealed limitations in the use of
38 interfaces. These studies have identified a series of limitations affecting some libraries and
39 digital repositories. (Tsakonas and Papatheodorou 2007) for example, conducted a usability
40
analysis of the digital library E-LIS in which they found it difficult to perform certain search
41
ly
42 processes. Certain search tasks such as knowledge domains or topics by collections proved to
43 be excessively time-consuming to carry out and great effort was required to understand the
44 user interface. Given that these strategies of search were not sufficiently robust and flexible on
45 their own, the authors emphasised the importance of complementing these strategies with
46 others in order to better facilitate the location and accessing of digital resources in the
47
collection. Studies by (Buchanan and Salako 2009), and (Petrelli 2008) similarly concluded that
48
49 search processes based on a limited list of filtered criteria were highly time-consuming for
50 users. Awareness of the limitations discussed here influenced the design and implementation
51 of our visual framework tool, based on the use of knowledge representation schemes such as
52 the OA-AE ontology (API-Organic.Edunet 2016) to access learning objects and scientific
53 publications in both Organic.Edunet and VOA3R/AGRIS. The next section will discuss the
54
methodology and the details of the design and implementation of our visual framework tool.
55
56 2.4 Related work
57
58
59
60
1
2
3 The use of visual search methods has emerged only recently in the field of digital repositories.
4 Some methods focus on the adequate level of access to digital resources. The MACE project
5 (Stefaner et al. 2007) proposes several alternatives of visual search (semantic, social and
6 contextual) for accessing digital resources in the field of “design and architecture” through
7
8
classification strategies involving: keywords, location, competence, social area and facets
9 (Wolpers, Memmel, and Stefaner 2010). These studies focus on perspectives that use various
10 navigation paths combined with social labelling. Studies on the use of MACE showed that the
11 principles of multifaceted navigation facilitate immersion processes through activities of
12 collaborative tagging (Stefaner et al. 2007); and, that the definition of metadata is crucial for
13 the improvement of search processes based on contextual-search strategies.
14
15 Other visual methods used in search interfaces is called facet browsing or facet search
16 (Tunkelang 2009). Related studies reveal that this method is highly recommended to i)
17
organize and browse document collections (Ferré 2008), (Stefaner, Urban, and Seefelder
18
Fo
19 2008), ii) for relevance similarity of documents like books, blogs and photos via position on a
20 base map (Dörk, Carpendale, and Williamson 2012), iii) for heterogeneous data with explicit
21 semantics (Polowinski 2009). An example of this method is Fluid views (Dörk, Carpendale, and
22 Williamson 2012), a tool that integrate dynamic queries, semantic zooming, and dual layers in
r
23 order to explore collections like books or photos in digital libraries. PivotPaths (Dork et al.
24
2012) is another example of tool to explore information for strolling through faceted
Re
25
26 information spaces. The topics are depicted in a row and are connected to facets via
27 links. Weighted faceted browsing (Voigt et al. 2012) is other tool that provides a sophisticated
28 relevance ranking of the result set based on the distinction between mandatory and weighted
vi
29 optional search criteria. Video Lens (Matejka, Grossman, and Fitzmaurice 2014), a framework
30 which allows users to visualize and interactively explore large collections of videos and
31
ew
32
associated metadata.
33 3. Methodology, design and evaluation
34
35 We define three phases in order to make our study. These aspects are related to the analysis
36 of i) connect learning objects of both Organic.Edunet and VOA3R repositories to a visual
On
37
framework (section 3.1), ii) design of visual framework (section 3.2), and finally iii) the
38
39 evaluation of visual search tool based on usability aspects in order to analyze the use of
40 interfaces through the development of a case study based on Human Computer Interaction
41 (HCI) perspective (section 3.3).
ly
42
43 3.1 Connecting learning objects to a visual framework tool
44
45 The OA-EA ontology provides instances to connect academic resources through the use of an
46 API search (API-Organic.Edunet 2016). As well, to access scientific publications we use mapping
47 of vocabularies defined by a SKOS format. For learning resources, the OA-EA ontology provides
48 concepts and usefulness classification associated with agriculture and agro-ecology. For
49 scientific publications, the OA-EA ontology includes instances in order to connect
50
bibliographical references associated with SKOS format through the use of LOD. This interface
51
52 provides data in SKOS format, built in agreement with the syntax provided by the AGROVOC, in
53 order to maintain compliance with the other main organic agriculture repositories. Figure 1
54 illustrates the model of connection with resources used in both repositories (Organic.Edunet
55 and VOA3R/AGRIS).
56
57
58
59
60
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Fo
19
20
21
22
r
23
24
Re
25
26
Fig. 1. Model of process to connect learning objects in repositories
27
28 Figure 1 presents the model of the process to connect resources to visual framework tool for
vi
29
the two repositories. The first step was implemented in order to connect educational and
30
31 scientific resources based on OA-AE ontology that are stored and published by a central tool
ew
32 called MoKi tool (Mooki-Tool 2016). In the second step we transform the KOS of OA-AE
33 ontology to JavaScript Object Notation JSON format in order to define the navigational
34 structure of the search. The third step we establish a connection to the learning objects of
35 both educational and scientific publications through the use of Organic.Edunet and AGROVOC
36
vocabularies. We then integrate three visual search interfaces (tree layout, indented tree and
On
37
38 flow tree layout) based on the D3js library (D3js 2016). Finally, we develop a visual strategy to
39 display the found learning objects according to the topics selected by the search interface
40 selected of visual framework tool. In the next section we describe the development and
41 integration of the visual search interfaces.
ly
42
43 3.2 Design of visual framework
44
45 There are a large variety of visual techniques discussed by (Draper, Livnat, and Riesenfeld
46 2009), (Gleicher et al. 2011), and also by (Ren et al. 2010) in order to help end-users build
47 interactive information visualisation. For the design of the visual search interfaces, we used a
48 library called D3js made by (Bostock and Heer 2009) a project for manipulating documents
49
based on data through the use of JavaScript, Scalable Vector Graphics (SVG) and Cascading
50
51 Style Sheets (CSS); and selected three navigational search interfaces: tree layout, indented tree
52 and flow tree layout (an adaptation of indented tree by (White 2016)).
53
54 These navigational interfaces were selected based on: i) characteristics that relate to the
55 visualisation techniques defined by (Draper, Livnat, and Riesenfeld 2009), (Gleicher et al. 2011)
56 and (Graham, Kennedy, and Benyon 2000), which include classification and hierarchical depth,
57 ii) related studies based on taxonomy classification interfaces (Wang, Chaudhry, and Khoo
58
59
60
1
2
3 2008), (Dinkla et al. 2011), (Paulo Alonso Gaona-García et al. 2014), (Paulo Alonso Gaona-
4 García, Sánchez-Alonso, and Montenegro Marín 2014), and iii) the use of navigation
5 prototypes for hierarchical structures (Merčun and Žumer 2010), (Merčun, Žumer, and Aalberg
6 2012).
7
8 To carry out the model of process we use a three-layer model to define a visual framework
9 tool. Figure 2 illustrates our framework proposal for defining a strategy of connecting and
10
11
linking learning objects.
12
13
14
15
16
17
18
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33 Fig. 2. Visual framework proposal
34
The three layers of the models as follows:
35
36 Presentation layer: The presentation layer is implemented for the management and display of
On
37
data to the user. This layer treats the whole visual representation of the data, including both
38
39 the user side and the drivers and graphics libraries (D3js, JavaScript, CSS3 and HTML5). The
40 latter are responsible for processing the data to provide a unique representative data format,
41 in this case converted implemented from SKOS format to JSON format.
ly
42
43 Service layer: This layer is used for the consumption of services through web services, where
44 the information (metadata) is obtained that needs to be processed and presented in the
45 presentation layer. In this case we consume these services through the use of API
46 Organic.Edunet (API-Organic.Edunet 2016) and uses mappings to AGROVOC concepts for
47
access to (VOA3R/AGRIS).
48
49 Data storage layer: This layer translates all requests and processes performed by the user in
50
the presentation layer (topics of navigational search) into language understandable to
51
52 repositories (OA-EA ontology). E.g. all metadata of learning objects (title, description,
53 keywords, language, etc.), and mapping to topics or knowledge area to the Organic.Edunet and
54 VOA3R/AGRIS repositories.
55
56 Figure 3 present a mockup of the design of visual search tool that we defined, in order to
57 integrate visualisation interfaces and repositories.
58
59
60
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Fo
19
20
21
22
r
23 Fig. 3. Design of visual search tool

24
Based on principles of Human Computer Interaction (HCI), and aspects defined by the design
Re
25
26 of interfaces proposed by (White and Roth 2009), (Russell et al. 2015), (Hearst 2011), and
27 according to the purpose of the research to integrate several visual search interfaces in both
28 academic and scientific repositories, we focus in four basic aspects for the design of visual
vi
29 search tool: i) selection of visual search interfaces, ii) use of search topic in navigational
30
interface in order to apply principles of exploratory search, iii) search topic in a traditional
31
ew
32 search box method, and iv) unfold taxonomy of thematic selected by visual search interface.
33 Figure 3 depicts a representation of the navigational structure of the visual search interface.
34 (1) Users can select a navigational view: tree, bars, and folders. (2) It presents the navigational
35 search for the tree interface. (3) The user selected the concept in a traditional search box
36 method or visual search interface, and then is displayed the taxonomy classification of the
On
37
topic selected (4).
38
39 3.3 Evaluation
40
41 We used the Technology Acceptance Model (TAM) to conduct a questionnaire survey in order
ly
42 to evaluate the visual framework tool. TAM has been used in many digital libraries (Fuhr et al.
43
2007), (Jeong 2011), (Park et al. 2009), (Thong, Hong, and Tam 2002) and repositories (Hee Kim
44
45 and Ho Kim 2008), (Tsakonas and Papatheodorou 2008), (Zhang, Maron, and Charles 2013).
46 The survey obtained subjective impressions of the visual framework and demonstrated the
47 potential value of the approach for improving access to learning and scientific resources in the
48 two repositories (i.e., to academic learning objects associated with the Organic.Edunet
49 repository and scientific publications associated with the VOA3R/AGRIS repository). The
50
purpose of this evaluation was to obtain values with which to analyse the usability, usefulness
51
52 and performance of the visual framework. These three traits are predominant aspects of these
53 types of studies to evaluate the performance of applications in information systems (Griffiths,
54 Johnson, and Hartley 2007). In addition to these traits, other significant attributes that have
55 been used for evaluation in related research in the field of digital libraries are: ease of use,
56 navigation, relevance, learnability, terminology, reliability, response time and aesthetic
57
(Buchanan and Salako 2009), (Jeng 2005), (Tsakonas and Papatheodorou 2006), (Tsakonas and
58
59
60
1
2
3 Papatheodorou 2008). Also notable are the attributes coverage, precision and hierarchical
4 taxonomy defined by Gaona et al. (2014), which have been used to evaluate subjective
5 impressions of digital repositories.
6
7 According to the objectives of the study and the recommendations commonly used for this
8 type of study (Nielsen 1994), 74 participants were selected for the test. Participants included
9 26 students from the Agriculture University of Athens, 32 researchers and 16 librarians. All
10
participants are involved in the use of academic and scientific publications in the field of
11
12 agriculture and agro-ecology, respectively.
13
The questionnaire was developed in two different parts: (Athens-Greece and Madrid-Spain)
14
15 and is comprised of two parts. The first part (preparation phase, section 3.3.1) is a training
16 phase to introduce participants to the visual framework and learning objects associated with
17 repositories. The second phase (evaluation phase, section 3.3.2) had two objectives: i) analyse
18 the whole visual search tool in order to assess the integration of the visual navigational search
Fo
19 with learning resources found in both repositories, and ii) analyse the user satisfaction of
20
interfaces to identify the most suitable visual search interface.
21
22 3.3.1 Preparation phase
r
23
24 This phase served to introduce the Organic.Edunet ontology and explain the concepts of, and
Re
25 differences between, academic and scientific learning resources. The preparation phase
26
purposely did not involve and real interaction with the visual search tool so that the data
27
28 collected would illustrate the existing wants and preconceptions of the participants.
vi
29
The preparation phase consisted of a training session with participants in a face-to-face
30
31 setting. The organiser of the training session gave a short talk on similar tools related (e.g.,
ew
32 navigational search of Organic.Edunet (Organic-Lingua 2016) and VOA3R (VOA3R-Agris 2016)

33 repositories, explained the historical context). Following the talk a questionnaire was
34 distributed to the participants to collect information on their user profiles, experience and
35 basic demographic data (annex 1).
36
On
37 3.3.2 Evaluation phase

38
39 The second phase consisted of 15-minute sessions in which participants interacted with the
40 visual search tool. Participants used visual interfaces to search learning objects based on topics
41
ly
relating to agriculture and agro-ecology defined in the OA-EA ontology. Following the session,
42
a questionnaire was given to participants to gather information about perception of user
43
44 satisfaction and perceived utility of the visual framework as a whole. Follow in figure 4
45 presents the tool designed to carry out the evaluation of visual interfaces.
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Fo
19
20 Fig. 4. Evaluation of visual search interfaces
21
22 A Technology-Acceptance-Model questionnaire was adapted to the purpose of evaluation
r
23 visual search tool, enabling participants to describe their experience in concrete terms and
24
make suggestions for additional features. The results of this questionnaire were then
Re
25
26 compared to analyse the potential and limitations of visual search.
27
28
We adapted the TAM questionnaire according to the recommendations of Tsakonas and
Papatheodoru (2008) and attributes of user perceptions based on (Heradio et al. 2012), using
vi
29
30 26 questions based on three attributes (usefulness, usability and performance) (annex 2).
31 However, we designed the questions to further evaluate two aspects: the precision and
ew
32 understandability of visual search interfaces. We examined the whole visual framework in

33 order to assess aspects of the visual navigational search and precision of learning objects
34
35
found in both repositories. We designed the questions to evaluate the visual search interfaces
36 in order to identify the most suitable visual search interface. In figure 5, we present an
On
37 example of the use of one navigational search (tree interface). Figure 4(A) illustrates the
38 integration of visual search interfaces, in this case the tree interface, and figure 4(B) presents
39 the results of a search of educational and scientific publications associated with the concept
40 “agricultural method”.
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57 Fig. 5. (A) Tree navigational search (B) Learning objects found in both repositories
58
59
60
1
2
3 Figure 5(A) depicts a representation of the navigational structure of the tree interface. (1)
4 Users can select a navigational view: tree (tree layout), bars (indented tree) and folders (flow
5 tree layout). (2) It presents the navigational search for the tree interface. (3) The user selected
6 the concept “agricultural method” and displayed the number of materials, 489, found in
7
8
repositories for both educational and scientific publications of learning objects. Figure 5(B)
9 presents the results—the learning objects found in both repositories. Item (4) shows the
10 concept selected, “agricultural method”. In item (5), it presents the learning objects for
11 educational publications, 429, and scientific publications, 30. Item (6) presents the metadata
12 used to display learning objects, and, finally, item (7) presents a pagination system for
13 navigating through all found learning objects.
14
15 4. Results
16
17 The ﬁrst part of the questionnaire (11 questions) covers the usefulness, usability and
18 performance attributes of the whole visual framework, including traits for resources such as
Fo
19 relevance, coverage, taxonomic hierarchy, learnability, terminology, response time and
20
precision. The second part of the questionnaire (15 questions) evaluates the same attributes as
21
22 the first, only this time in relation to the visual search interfaces rather than the whole visual
r
23 framework. The second part of the questionnaire examines additional traits such as ease of
24 use, aesthetic, navigation, reliability, and efficiency.
Re
25
26 The next section presents the results of the evaluation. We first present the results related to
27 the whole visual framework (section 4.1) associated with navigational search and precision of
28 learning objects with topic selected for both repositories. We then analyse the aspects of the
vi
29 evaluation that relate to the visual search interfaces (section 4.2).

30
31 4.1 Visual framework as a whole
ew
32
33 This section presents the results of the first part of the questionnaire, which addresses aspects
34 of usefulness, usability and performance. Table 1 shows the participant survey results from our
35 study, also using a five-point scale (1=strongly disagree; 2=disagree; 3=neutral; 4=agree;
36
5=strongly agree). The five scale values represent the subjective satisfaction of the users with
On
37
38 regard to the visualisation framework and its functionality for accessing academic learning
39 objects and scientific publications.
40
41
ly
42
Table 1. Descriptive statistical analysis for visual framework (1=disagree, 5=agree)
43
44 According to visual perception by all participants, the whole visual framework is considered a
45 generally good tool for processing the search and access of learning objects. However, there
46
47 still exist some general issues in the use of these visual interfaces according to the results by
48 user profiles presented in table 2.
49
50
51 Table 2. Summary for interactions according to user profile (1 = low satisfaction, 5 = high
52
satisfaction). Source: Own elaboration
53
54 Next section we describe in general the results according to the attributes of usefulness,
55
usability and performance.
56
57 4.1.1 Usefulness results
58
59
60
1
2
3 According to the table 1 in the usefulness category, relevance received a high level of
4 agreement (66.7%). This result indicates that a high percentage of participants felt that the
5 visual framework provided good levels of helpful information (e.g., abstracts, descriptions,
6 formats, etc.) for search tasks. In the coverage category, a large number of participants
7
8
(58.3%) agreed that the visual framework provided a high number of learning objects covering
9 all topics of the knowledge representation scheme used for the visual search. However in table
10 2, researchers indicated that the information provided by the visual search tool for searching
11 scientific publications was not completely relevant (mean=3.75; SD=0.707) because there were
12 few options for refining searches (e.g. by title, type of publication, language, etc.).
13 Undergraduate students, meanwhile, were completely satisfied with the quality of the
14
15 retrieved content (mean=4.17, SD=0.389). The category of taxonomy showed similar results for
16 both undergraduate students (mean=4.17; SD=0.577) and researchers (mean=4.125;
17 SD=0.640). Librarians, however, demonstrated more neutral opinions (mean=3.75; SD=0.957),
18 particularly because the majority of them were not familiar with knowledge representation
Fo
19 schemes. We will discuss this further in the next section.
20
21 4.1.2 Usability results
22
r
23 Although the navigation structure had a high rate of agreement for usefulness, participants
24 agreed that it was necessary to have prior knowledge of the OA-EA ontology in order to
Re
25 perform exploration and search processes within the navigation structure. Researchers felt it
26 necessary to have more time to understand and learn the visual framework in order to search
27
learning objects (mean=3.75; SD=0.707). However, undergraduate students (mean=4.166;
28
SD=0.573) and librarians found the visual search tool easy to learn (learnability) in the time
vi
29
30 allotted (mean=4.25; SD=0.957). The aesthetic attributes of the visual search tool were rated
31 highly across researchers (mean=4.5; SD=0.535), librarians (mean=4.5; SD=0.577) and
ew
32 undergraduate students (mean=4.5; SD=0.522). A small percentage of participants (20.8%)

33 were not completely familiar with the terminology used by the OA-EA ontology for searching
34
learning objects. Although through the use of navigation taxonomy, participants later obtained
35
36 good results as presented in section 4.2.
On
37
4.1.2 Performance results
38
39 In terms of performance, participants’ subjective perception of response-time was positive
40
(mean=4.17; SD=0.702) according to table 1. Participants similarly indicated that the visual
41
ly
42 search tool presented results that were very precisely (precision) related to the topic or
43 knowledge area selected in the knowledge representation scheme for both educational and
44 scientific publications (mean=4.04; SD=0.609). However, there were results for visual
45 interfaces where users obtained an increase of response time for retrieval learning objects.
46 These results are presented in the next section.
47
48 4.2 Suitable visual search interface
49
50 Building on the results of the visual search tool survey questions, we here evaluate user
51 perceptions of the suitable visual search interface for accessing learning objects. Table 3
52 displays these results.
53
54
55
56 Table 3. Perception of visual search interfaces
57
58
59
60
1
2
3 As shown in table 3, the tree interface was generally rated as the most suitable visual search
4 interface on all attributes. The attributes scored as follows: reliability (mean=4.6667;
5 SD=0.4815), ease of use (mean=4.5833; SD=0.5036), aesthetic (mean=4.7500; SD=0.4423),
6 navigation (mean=4.6667; SD=0.4815) and effectiveness (mean=4.6667; SD=0.4815). Table 4
7
8
present results of perception by user profile.
9
10
11 Table 4. Summary of user perception by user profile (1 = low satisfaction, 5 = high satisfaction).
12 Source: Own elaboration
13
14 According to the user profile shown in table 4, researchers preferred the bars interface
15 (mean=4.104; SD=0.501) to the tree interface (mean=3.845; SD=0.451) because the bar
16
interface allowed the display of more terms and concepts associated with one narrower search
17
18 term (the root term of the structure). Nerveless for undergraduate students, tree interface had
Fo
19 a better ease of use (mean=4.583; SD=0.514) that bars interface by librarians who had a low
20 validation (mean=3.705; SD=0.957).
21
22 Finally, as shown in table 5, we studied correlation in order to determine if the visual
r
23 perception of some attributes influenced the assessment of other evaluation criteria in the use
24 of visual search interfaces.
Re
25
26
27
28 Table 5. Pearson correlation of attributes for visual search interfaces
vi
29
30 According to the Pearson correlation of table 3, there are highly significant correlations
31 between reliability and navigation (r=0.814) and aesthetic and navigation (r=0.708). These
ew
32 correlations were highly moderate and considerably positive; this means that participants in
33 general were influenced by reliability and aesthetic attributes in their evaluation of the
34 navigation of visual search interfaces.
35
36 5. Discussion and implications
On
37
38 According to Gaona et al. (2014) visual search interfaces generally allow users to build on their
39 knowledge, without deliberate effort, to carry out search processes through the use of a
40 knowledge representation scheme. However, based on the results of this research some
41
ly
problems still exist in the capacity for exploring concepts within a simple knowledge
42
representation scheme. These problems include i) terminologies associated with related
43
44 terms, or non-preferred terms in different hierarchical categories; ii) classification schemes --
45 sometimes it is complicated for end-users to understand the context of a classification scheme
46 concept and the relationships that link the terms. Therefore it is necessary to carry out
47 support/tutorial within the visual framework to allow the end-user to have a better
48 understanding of the knowledge representation scheme that is being used; iii) mapping of
49
50
knowledge representation scheme-- to improve the access of several repositories through the
51 use of a simple knowledge representation scheme it is necessary to carry out a complete
52 mapping of concepts and terms of all SKOS formats. If there are limitations to these mapping
53 vocabularies, the visual framework cannot visualise resources by external repositories. Such
54 problems may affect the system’s usability, if visual interfaces and the creators of repositories
55 cannot provide methods to guide participants within a navigation structure.
56
57
58
59
60
1
2
3 Although our research has proven the effectiveness of visual interfaces through visual
4 perception of participants, it is necessary to validate these findings by employing various
5 complementary research techniques and by experimenting with adaptations of this promising
6 visual search tool. It is necessary, then, to carry out usability studies with specialised
7
8
techniques such as eye-tracking (Rosch and Vogel-Walcutt 2013) and holistic models (Zhang
9 2010), (Bertot et al. 2006), (Heradio et al. 2012) to obtain a careful user analysis through
10 observation methods. These studies will facilitate the design of new easy-to-learn
11 functionalities, such as providing controls where the user expects them to be and reducing
12 external instructions or inconsistencies resulting from unnecessary additional functions. It is
13 also necessary, according to (Heradio et al. 2012), to establish a consensus on standard
14
15 definitions and methods for carrying out complete usability and usefulness studies in
16 repositories.
17
In relation to the effectiveness of visual search interfaces it is important to note that, if the
18
Fo
19 description fields of digital objects in repositories are not well described, this limitation in the
20 metadata has a very negative impact on the search process for digital resources (Cechinel,
21 Sánchez-Alonso, and Sicilia 2009), (Park 2009). High-quality resources cannot be located easily
22 by end-users if they are not well defined. The exclusion of the use of metadata in repositories
r
23 would not facilitate the search processes for learning objects (e.g., within a specific thematic
24
area defined in a knowledge representation scheme), and therefore the efforts to integrate
Re
25
26 visualisation techniques to repositories would be unsuccessful.
27
In summary, this may indicate that information visualisation allows rapid location of resources
28
within a taxonomic structure, by using search criteria based on knowledge representation
vi
29
30 schemes. However, in order to be able to understand the use of visual interfaces, it is
31 important to note that the learning curve is one of the factors, if not the only, which plays an
ew
32 important role to perform appropriate use of visual search interfaces. Therefore, it is essential
33 to define usability studies which combine the use of knowledge representation schemes and
34
effective search interfaces aimed at promoting resource exploration processes within a digital
35
36 library or repository according to the purposes thereof.
On
37
6. Conclusions and recommendations
38
39 Results presented usefulness visual search interfaces with a consistent terminology thanks to
40
the use of a mature knowledge representation scheme by using OA-EA ontology. It is
41
ly
42 important to remark the design of elements and controls placed in familiar locations in order
43 to provide a usable visual framework to search learning objects. In this direction and according
44 to the results of our study, information visualisation in digital repositories could improve
45 access and location to learning objects in both academic and scientific repositories; given the
46 increase of users, who are demanding for better service and functionalities in repositories.
47
Therefore, visual search interfaces based on knowledge representation schemes allow users,
48
49 with minimal effort, to build on their knowledge to search learning objects.
50
Based on our study results, it is clear that digital repositories should work on strategies to
51
52 facilitate the interoperability and re-use of digital resources such as semantic enrichment
53 defined by metadata. However, creators of repositories focus further efforts on facilitating
54 access to large collections of digital resources by linking learning objects based on
55 interoperability standards such as Linked Data. There are several studies based on structure
56 linked to a series of semantic enrichment strategies over educational resources (Rajabi, Sicilia,
57
and Sanchez-Alonso 2015), (Ren et al. 2010), (Zhang 2014). These strategies of linking would
58
59
60
1
2
3 facilitate the management and maintenance of digital resources through good design practices
4 and by linking learning objects to the areas of knowledge stored in external repositories.
5
6 The successful use of knowledge representation schemes in visual search interfaces depends
7 largely on several aspects defined by (Gaona-García, Martín-Moncunill, and Montenegro
8 Marín, in press) which has not been fully included in this study, but is necessary to mention the
9 most important factors in order to improve the conditions of access to digital resources e.g.
10
Firstly, creators should include a usefulness knowledge representation scheme. Visual
11
12 frameworks should have several knowledge representation schemes based on user profiles
13 (secondary students, undergraduate students, professionals or researchers). These tools could
14 facilitate the use of navigational search interfaces and improve the learning of end-users
15 seeking to understand complex relationships according to the term/concept selected.
16 Secondly, effectiveness visual search interfaces should be considered, this selection depends
17
on complete usability studies related to knowledge representation scheme selected that
18
Fo
19 including: e.g., taxonomy classification, levels of depth of KOS, relations and mappings by
20 terms and concepts related to other KOS (ontologies, thesauri), response time of queries made
21 at different classification levels and terms/concepts. These aspects are necessary conditions in
22 an educational scenario for creating a collaborative work environment in order to favour
r
23 processes to find educational material and categorically cover knowledge areas of interest by
24
end users like students or teachers.
Re
25
26 Future work should involve the analysis and visualisation evaluation of learning objects
27
through the use of services. Firstly, future projects should use multilingual knowledge
28
representation schemes. These schemes define visualisation strategies for the deployment of
vi
29
30 learning objects in several languages from the same SKOS format. Secondly, future projects
31 should include visual analytics of links to learning objects that are mapped within the
ew
32 knowledge representation scheme through external repositories. Thirdly, visual analytics of

33 the frequency of access to academic repositories and a history of the learning objects queried
34
should be available to provide relevant resources to end-users. Finally, the inclusion of the
35
36 User Interface as a Service UlaaS (Sherchan et al. 2012) in order to obtain better search results
On
37 and generate context-aware interface services like the multilingual services, favourite learning
38 objects, rankings in the use of repositories, and enrich learning objects through the use of
39 peer-reviewers.
40
41 References
ly
42
43 Agrovoc. 2016. “AGROVOC Multilingual Agricultural Thesaurus.” Accessed February 26
44 http://aims.fao.org/vest-registry/vocabularies/agrovoc-multilingual-agricultural-thesaurus.
45
46 API-Organic.Edunet. 2016. “Ontology Service API.” Accessed February 26
47 http://wiki.organic-lingua.eu/APIs#Ontology_Service_API.
48
49 Barton, J, S Currier, and J M Hey. 2003. “Building Quality Assurance into Metadata
50 Creation: An Analysis Based on the Learning Objects and E-Prints Communities of
51 Practice.” Proceedings of the 2003 International Conference on Dublin Core and Metadata
52 Applications: Supporting Communities of Discourse and Practice-Metadata Research and
53
Applications, Seattle, Washington, 39–48.
54
55 Bertot, J C, J T Snead, P T Jaeger, and C R McClure. 2006. “Functionality, Usability, and
56
Accessibility: Iterative User-Centered Evaluation Strategies for Digital Libraries.”
57
58 Performance Measurement and Metrics 7: 17–28. doi:10.1108/14678040610654828.
59
60
1
2
3 Bizer, C, T Heath, and T Berners-Lee. 2009. “Linked Data - The Story so Far.” International
4 Journal on Semantic Web and Information Systems 5: 1–22.
5 doi:10.4018/jswis.2009081901.
6
7 Bostock, M, and J Heer. 2009. “Protovis: A Graphical Toolkit for Visualization.” IEEE
8 Transactions on Visualization and Computer Graphics 15: 1121–28.
9 doi:10.1109/TVCG.2009.174.
10
11 Buchanan, S, and A Salako. 2009. “Evaluating the Usability and Usefulness of a Digital
12 Library.” Library Review 58: 638–51. doi:10.1108/00242530910997928.
13
14 Cechinel, C, S. S. Da Camargo, X Ochoa, S.S. Alonso, and M.A. Sicilia. 2012. “Populating
15 Learning Object Repositories with Hidden Internal Quality Information.” In Proceedings of
16
the 2nd Workshop on Recommender Systems for Technology Enhanced Learning
17
18 (RecSysTEL 2012). Manouselis, N., Draschler, H., Verber, K., and Santos, OC (Eds.).
Fo
19 Published by CEUR Workshop Proceedings, 896:11–22.
20
21 Cechinel, C, S Sánchez-Alonso, and M A Sicilia. 2009. “Empirical Analysis of Errors on
22 Human-Generated Learning Objects Metadata.” Communications in Computer and
r
23 Information Science 46: 60–70. doi:10.1007/978-3-642-04590-5_6.

24
Re
25 Cervera, J F, M G López-López, C Fernández, and S Sánchez-Alonso. 2009. “Quality Metrics

26 in Learning Objects.” Metadata and Semantics. Springer US, 135-141. doi:10.1007/978-0-
27 387-77745-0_13.
28
vi
29 Chrysafiadi, K, and M Virvou. 2013. “A Knowledge Representation Approach Using Fuzzy

30 Cognitive Maps for Better Navigation Support in an Adaptive Learning System.”
31 SpringerPlus 2: 1–13. doi:10.1186/2193-1801-2-81.
ew
32
33 D3js. 2016. “D3 Data-Driven Documents.” Accessed February 26 http://d3js.org/.
34
35 Dinkla, K, M A Westenberg, H M Timmerman, S.A.F.T. Van Hijum, and J J Van Wijk. 2011.
36 “Comparison of Multiple Weighted Hierarchies: Visual Analytics for Microbe Community
On
37 Profiling.” Computer Graphics Forum 30: 1141–50. doi:10.1111/j.1467-8659.2011.01963.x.

38
39 Dörk, M, S Carpendale, and C Williamson. 2012. “Fluid Views: A Zoomable Search
40 Environment.” In Proceedings of the International Working Conference on Advanced Visual
41
ly
Interfaces, Capri Island, Italy, 233–40. doi:10.1145/2254556.2254599.

42
43 Dork, M, N Henry Riche, G Ramos, and S Dumais. 2012. “Pivot Paths: Strolling through
44 Faceted Information Spaces.” IEEE Transactions on Visualization and Computer Graphics
45
18: 2709–18. doi:10.1109/TVCG.2012.252.
46
47 Draper, G M, Y Livnat, and R F Riesenfeld. 2009. “A Survey of Radial Methods for
48
Information Visualization.” IEEE Transactions on Visualization and Computer Graphics 15:
49
50 759–76. doi:10.1109/TVCG.2009.23.
51
Ferré, S. 2008. “Agile Browsing of a Document Collection with Dynamic Taxonomies.” In
52
53 Proceedings - International Workshop on Database and Expert Systems Applications, DEXA,
54 Turin, September 01-05, 377–81. doi:10.1109/DEXA.2008.28.
55
56 Flouris, G, D Plexousakis, and G Antoniou. 2003. “Describing Knowledge Representation
57 Schemes: A Formal Account.” In Institute of Computer Science. Foundation for Research
58 and Technology (ICS-FORTH), Crete, Greece, Technical Rep. No. TR-320.
59
60
1
2
3 Fuhr, N, G Tsakonas, T Aalberg, M Agosti, P Hansen, S Kapidakis, C.-P. Klas, et al. 2007.
4 “Evaluation of Digital Libraries.” International Journal on Digital Libraries 8: 21–38.
5 doi:10.1007/s00799-007-0011-z.
6
7 Gaona-García, P. A., David Martín-Moncunill, and Carlos Montenegro Marín. In press.
8 “Trends and Challenges of Visual Search Interfaces in Digital Libraries and Repositories.”
9 The Electronic Library.
10
11 Gaona-García, Paulo Alonso, David Martín-Moncunill, Salvador Sánchez-Alonso, and Ana
12 Fermoso García. 2014. “A Usability Study of Taxonomy Visualisation User Interfaces in
13
Digital Repositories.” Online Information Review 38:284–304. doi:10.1108/OIR-03-2013-
14
15 0051.
16
Gaona-García, Paulo Alonso, Salvador Sánchez-Alonso, and Carlos E. Montenegro Marín.
17
18 2014. “Visualization of information: a proposal to improve the search and access to digital
Fo
19 resources in repositories.” Revista Ingenieria e Investigacion 34: 83–89.
20
21 Gašević, D, D Djurić, and V Devedžić. 2009. “Model Driven Engineering and Ontology
22 Development.” Model Driven Engineering and Ontology Development. doi:10.1007/978-3-
r
23 642-00282-3.
24
Re
25 Gleicher, M, D Albers, R Walker, I Jusufi, C D Hansen, and J C Roberts. 2011. “Visual

26 Comparison for Information Visualization.” Information Visualization 10: 289–309.
27 doi:10.1177/1473871611416549.
28
vi
29 Gomez-Perez, A. 1999. “Evaluation of Taxonomic Knowledge in Ontologies and Knowledge

30 Bases.” Proceedings of the 12th Workshop for Knowledge Acquisition, Modeling and
31 Management (KAW’99), Banff, Canada.
ew
32
33 Graham, M, J Kennedy, and D Benyon. 2000. “Towards a Methodology for Developing
34 Visualizations.” International Journal of Human Computer Studies 53: 789–807.
35
doi:10.1006/ijhc.2000.0415.
36
On
37 Griffiths, J R, F Johnson, and R J Hartley. 2007. “User Satisfaction as a Measure of System

38 Performance.” Journal of Librarianship and Information Science 39: 142–52.
39
40 doi:10.1177/0961000607080417.
41
ly
Hartson, H R, P Shivakumar, and M A Pérez-Quiñones. 2004. “Usability Inspection of Digital

42
Libraries: A Case Study.” International Journal on Digital Libraries 4: 108–23.
43
44 doi:10.1007/s00799-003-0074-4.
45
Hearst, M. 2006. “Design Recommendations for Hierarchical Faceted Search Interfaces.” In
46
47 Andrei Z. Broder and Yoelle S. Maarek, editors ACM SIGIR Workshop on Faceted Search,
48 August), 26-30.
49
50 Hee Kim, H, and Y Ho Kim. 2008. “Usability Study of Digital Institutional Repositories.”
51 Electronic Library 26: 863–81. doi:10.1108/02640470810921637.
52
53 Heradio, R, D Fernández-Amorós, F J Cabrerizo, and E Herrera-Viedma. 2012. “A Review of
54 Quality Evaluation of Digital Libraries Based on Users’ Perceptions.” Journal of Information
55 Science 38: 269–83. doi:10.1177/0165551512438359.
56
57 Hitchcock, S, A Woukeu, T Brody, L Carr, W Hall, and S Harnad. 2003. “Evaluating Citebase,
58 An Open Access Web-Based Citation-Ranked Search and Impact Discovery Service.”
59
60
1
2
3 Accessed February 26 http://opcit.eprints.org/evaluation/Citebase-evaluation/evaluation-
4 report.html.
5
6 Jeng, J. 2005. “Usability Assessment of Academic Digital Libraries: Effectiveness, Efficiency,
7 Satisfaction, and Learnability.” Libri 55: 96–121.
8
9 Jeong, H. 2011. “An Investigation of User Perceptions and Behavioral Intentions towards
10 the E-Library.” Library Collections, Acquisition and Technical Services 35: 45–60.
11 doi:10.1016/j.lcats.2011.03.018.
12
13 Kumar, V, J Nesbit, and K Han. 2005. “Rating Learning Object Quality with Distributed
14 Bayesian Belief Networks: The Why and the How.” In Proceedings - 5th IEEE International
15 Conference on Advanced Learning Technologies, ICALT 2005, 685–87.
16
doi:10.1109/ICALT.2005.230.
17
18 Lytras, Miltiadis D, and Miguel-Angel Sicilia. 2007. “Where Is the Value in Metadata?”
Fo
19
International Journal of Metadata, Semantics and Ontologie 2: 235–41.
20
21 Manouselis, N, K Kastrantas, S Sanchez-Alonso, J Cáceres, H Ebner, M Palmer, and A Naeve.
22
2009. “Architecture of the Organic.Edunet Web Portal.” International Journal of Web
r
23
24 Portals 1: 71–91.
Re
25
Martín-Moncunill, D, E García-Barriocanal, M.-A. Sicilia, and S Sánchez-Alonso. 2015.
26
27 “Evaluating the Practical Applicability of Thesaurus-Based Keyphrase Extraction in the
28 Agricultural Domain: Insights from the VOA3R Projectt.” Knowledge Organization 42: 76–
vi
29 89.
30
31 Matejka, J, T Grossman, and G Fitzmaurice. 2014. “Video Lens: Rapid Playback and
ew
32 Exploration of Large Video Collections and Associated Metadata.” In UIST 2014 -

33 Proceedings of the 27th Annual ACM Symposium on User Interface Software and
34 Technology, 541–50. doi:10.1145/2642918.2647366.
35
36 Merčun, T, and M Žumer. 2010. “Visualizing for Explorations and Discovery.” Libraries in
On
37 the Digital Age, 104–15.

38
39 Merčun, T, M Žumer, and T Aalberg. 2012. “FrbrVis: An Information Visualization Approach
40 to Presenting FRBR Work Families.” Lecture Notes in Computer Science (Including Subseries
41
ly
Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 7489: 504–7.
42
doi:10.1007/978-3-642-33290-6_60.
43
44 Mooki-Tool. 2016. “Moki the Modelling Wiki.” Accessed February 26
45
https://dkmtools.fbk.eu/moki/multilingual/organiclingua/.
46
47 Nash, S S. 2005. “Learning Objects, Learning Object Repositories, and Learning Theory:
48
Preliminary Best Practices for Online Courses.” Interdisciplinary Journal of Knowledge and
49
50 Learning Objects 1: 217–28.
51
Nielsen, Jakob. 1994. “Usability Engineering.” San Francisco: Morgan Kaufmann Publishers
52
53 Inc.
54
Noshairwan, W, M A Qadir, and M Fahad. 2007. “Sufficient Knowledge Omission Error and
55
56 Redundant Disjoint Relation in Ontology.” Advances in Soft Computing 43: 260–65.
57 doi:10.1007/978-3-540-72575-6_42.
58
59
60
1
2
3 Ochoa, X, and E Duval. 2009. “Automatic Evaluation of Metadata Quality in Digital
4 Repositories.” International Journal on Digital Libraries 10: 67–91. doi:10.1007/s00799-
5 009-0054-4.
6
7 Ochoa, Xavier, and Erick. Duval. 2006. “Quality Metrics for Learning Object Metadata.” In
8 Paper Presented at the Proceedings of World Conference on Educational Multimedia,
9 Hypermedia and Telecommunications, Orlando, 1004-1011.
10
11 Organic-Lingua. 2016. “Organic.Lingua Portal.” Accessed February 26 http://www.organic-
12 edunet.eu/.
13
14 Palavitsinis, N, N Manouselis, and C Karagiannidis. 2013. “Applying the Nominal Group
15 Technique for Metadata Training of Domain Experts.” Communications in Computer and
16
Information Science 390: 111–22. doi:10.1007/978-3-319-03437-9_12.
17
18 Park, J.-R. 2009. “Metadata Quality in Digital Repositories: A Survey of the Current State of
Fo
19
the Art.” Cataloging and Classification Quarterly 47: 213–28.
20
21 doi:10.1080/01639370902737240.
22
Park, N, R Roman, S Lee, and J E Chung. 2009. “User Acceptance of a Digital Library System
r
23
24 in Developing Countries: An Application of the Technology Acceptance Model.”
Re
25 International Journal of Information Management 29: 196–209.

26 doi:10.1016/j.ijinfomgt.2008.07.001.
27
28 Petrelli, D. 2008. “On the Role of User-Centred Evaluation in the Advancement of
vi
29 Interactive Information Retrieval.” Information Processing and Management 44: 22–38.

30 doi:10.1016/j.ipm.2007.01.024.
31
ew
32 Polowinski, J. 2009. “Widgets for Faceted Browsing.” Lecture Notes in Computer Science
33 (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in
34 Bioinformatics) 5617: 601–10. doi:10.1007/978-3-642-02556-3_68.
35
36 Rajabi, E, M.-A. Sicilia, and S Sanchez-Alonso. 2015. “Interlinking Educational Resources to
On
37 Web of Data through IEEE LOM.” Computer Science and Information Systems 12: 233–55.
38 doi:10.2298/CSIS140330088R.
39
40 Ren, L, F Tian, X Zhang, and L Zhang. 2010. “DaisyViz: A Model-Based User Interface Toolkit
41
ly
for Interactive Information Visualization Systems.” Journal of Visual Languages and

42
Computing 21: 209–29. doi:10.1016/j.jvlc.2010.05.003.
43
44 Rho, Jee-Hyun. 2014. “Limits and Problems of the Current OPACs for Digital Kids.” Journal
45
of Korean Library and Information Science Society 45: 337–64.
46
47 http://www.dbpia.co.kr/Article/3565718.
48
Rosch, J L, and J J Vogel-Walcutt. 2013. “A Review of Eye-Tracking Applications as Tools for
49
50 Training.” Cognition, Technology and Work 15: 313–27. doi:10.1007/s10111-012-0234-7.
51
Russell, A, P R Smart, D Braines, and N R Shadbolt. 2009. “NITELIGHT: A Graphical Tool for
52
53 Semantic Query Construction.” In CEUR Workshop Proceedings. Vol. 543.
54
Russell, D M, J Teevan, M R Morris, M Hearst, and E H Chi. 2015. “Design for Searching &
55
56 Finding.” In Conference on Human Factors in Computing Systems - Proceedings, 2457–58.
57 doi:10.1145/2702613.2706692.
58
59
60
1
2
3 Sánchez-Alonso, S, and M.-A. Sicilia. 2009. “Using an AGROVOC-Based Ontology for the
4 Description of Learning Resources on Organic Agriculture.” Metadata and Semantics.
5 doi:10.1007/978-0-387-77745-0_47.
6
7 Sanz-Rodriguez, J, J M Dodero, and S Sanchez-Alonso. 2011. “Metrics-Based Evaluation of
8 Learning Object Reusability.” Software Quality Journal 19: 121–40. doi:10.1007/s11219-
9 010-9108-5.
10
11 Sherchan, W, S Nepal, A Bouguettaya, and S Chen. 2012. “Context-Sensitive User
12 Interfaces for Semantic Services.” ACM Transactions on Internet Technology, no. 3.
13
doi:10.1145/2078316.2078322.
14
15 Shneiderman, Ben, Plaisant, and Catherine. 2005. “Designing the User Interface.”
16
Strategies for Effective Human-Computer Interaction, 1.
17
18 Shreeves, Sarah L, Ellen M Knutson, Besiki Stvilia, Carole L Palmer, Michael B Twidale, and
Fo
19
Timothy W Cole. 2005. “Is ‘quality’ Metadata ‘shareable’ Metadata? The Implications of
20
21 Local Metadata Practices for Federated Collections H.A.” In Proceedings of the Twelfth
22 National Conference of the Association of College and Research Libraries, Association of
r
23 College and Research Libraries, edited by Thompson, Chicago: Association of College and
24 Research Libraries, 223–37.
Re
25
26 Silvis, E, and P M Alexander. 2014. “A Study Using a Graphical Syntax for Actor-Network
27 Theory.” Information Technology and People 27: 110–28. doi:10.1108/ITP-06-2013-0101.
28
vi
29 Šimek, P, J Vaněk, V Očenásek, M Stočes, and T Vogeltanzová. 2012. “Using Metadata

30 Description for Agriculture and Aquaculture Papers.” Agris On-Line Papers in Economics
31 and Informatics 4: 79–90.
ew
32
33 Smart, P R, A Russell, D Braines, Y Kalfoglou, J Bao, and N R Shadbolt. 2008. “A Visual
34 Approach to Semantic Query Design Using a Web-Based Graphical Query Designer.”
35
Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial
36
Intelligence and Lecture Notes in Bioinformatics) 5268: 275–91. doi:10.1007/978-3-540-
On
37
38 87696-0-25.
39
40 Stafford, A, A Shiri, S Ruecker, M Bouchard, P Mehta, K Anvik, and X Rossello. 2008.
41 “Searchling: User-Centered Evaluation of a Visual Thesaurus-Enhanced Interface for
ly
42 Bilingual Digital Libraries.” Lecture Notes in Computer Science (Including Subseries Lecture
43 Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5173: 117–21.
44 doi:10.1007/978-3-540-87599-4_13.
45
46 Stefaner, M, T Urban, and M Seefelder. 2008. “Elastic Lists for Facet Browsing and
47 Resource Analysis in the Enterprise.” In Proceedings - International Workshop on Database
48 and Expert Systems Applications, DEXA, 397–401. doi:10.1109/DEXA.2008.108.
49
50 Stefaner, M, E D Vecchia, M Condotta, M Wolpers, M Specht, S Apelt, and E Duval. 2007.
51 “MACE - Enriching Architectural Learning Objects for Experience Multiplication.” Lecture
52
Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and
53
54 Lecture Notes in Bioinformatics) 4753: 322–36.
55
Stuckenschmidt, H, and F Van Harmelen. 2004. “Generating and Managing Metadata for
56
57 Web-Based Information Systems.” Knowledge-Based Systems 17: 201–6.
58 doi:10.1016/j.knosys.2004.04.004.
59
60
1
2
3 Tatnall, A. 2009. “Gateways to Portals Research.” International Journal of Web Portals 1:
4 1–15.
5
6 Tenopir, Carol. 2003. “Use and Users of Electronic Library Resources: An Overview and
7 Analysis of Recent Research Studies.” In Paper Presented at the Council on Library and
8 Information Resources,. Washington.
9
10 Thong, J Y L, W Hong, and K.-Y. Tam. 2002. “Understanding User Acceptance of Digital
11 Libraries: What Are the Roles of Interface Characteristics, Organizational Context, and
12 Individual Differences?” International Journal of Human Computer Studies 57: 215–42.
13
doi:10.1016/S1071-5819(02)91024-4.
14
15 Tsakonas, G, and Papatheodorou, C. 2006. “Analysing and Evaluating Usefulness and
16
Usability in Electronic Information Services.” Journal of Information Science 32: 400–419.
17
18 doi:10.1177/0165551506065934.
Fo
19
Tsakonas, G, and Papatheodorou, C. 2007. “Critical Constructs of Digital Library
20
21 Interaction.” 11th Panhellenic Conference on Informatics (PCI 2007).
22
Tsakonas, G, and Papatheodorou, C. 2008. “Exploring Usefulness and Usability in the
r
23
24 Evaluation of Open Access Digital Libraries.” Information Processing and Management 44:
Re
25 1234–50. doi:10.1016/j.ipm.2007.07.008.
26
27 Tunkelang, D. 2009. “Faceted Search (Synthesis Lectures on Information Concepts,
28 Retrieval, and Services).” Morgan and Claypool Publishers.
vi
29
30 Uddin, M N, and P Janecek. 2007. “Performance and Usability Testing of Multidimensional
31 Taxonomy in Web Site Search and Navigation.” Performance Measurement and Metrics 8:
ew
32 18–33. doi:10.1108/14678040710748058.
33
34 VOA3R-Agris. 2016. “Virtual Open Access Agriculture & Aquaculture Repository.” Accessed
35 February 26 http://voa3r.eu/.
36
On
37 Voigt, M, A Werstler, J Polowinski, and K Meißner. 2012. “Weighted Faceted Browsing for
38 Characteristics-Based Visualization Selection through End Users.” In EICS’12 - Proceedings
39 of the 2012 ACM SIGCHI Symposium on Engineering Interactive Computing Systems, 151–
40 56. doi:10.1145/2305484.2305509.
41
ly
42 Wang, Z, A S Chaudhry, and C S G Khoo. 2008. “Using Classification Schemes and Thesauri
43 to Build an Organizational Taxonomy for Organizing Content and Aiding Navigation.”
44 Journal of Documentation 64: 842–76. doi:10.1108/00220410810912424.
45
46 White, B. 2016. “D3 Wrapping Flow Tree Layout.” Accessed February 26.
47 http://www.billdwhite.com/wordpress/2013/10/11/d3-flow-tree-layout/.
48
49 White, R W, and R A Roth. 2009. “Exploratory Search: Beyond the Query-Response
50 Paradigm.” Exploratory Search: Beyond the Query-Response Paradigm.
51
52 Wolpers, M, M Memmel, and M Stefaner. 2010. “Supporting Architecture Education Using
53 the MACE System.” International Journal of Technology Enhanced Learning 2: 132–44.
54
doi:10.1504/IJTEL.2010.031264.
55
56 Zhang, L. 2014. “Linking Information through Function.” Journal of the Association for
57
Information Science and Technology 65: 2293–2305. doi:10.1002/asi.23123.
58
59
60
1
2
3 Zhang, T, D J Maron, and C C Charles. 2013. “Usability Evaluation of a Research Repository
4 and Collaboration Web Site.” Journal of Web Librarianship 7: 58–82.
5 doi:10.1080/19322909.2013.739041.
6
7 Zhang, Y. 2010. “Developing a Holistic Model for Digital Library Evaluation.” Journal of the
8 American Society for Information Science and Technology 61: 88–110.
9 doi:10.1002/asi.21220.
10
11
12
13
14
15
16
17
18
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33 254x191mm (72 x 72 DPI)
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39 106x100mm (149 x 149 DPI)
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Fo
19
20
21
22
r
23
24
Re
25 250x127mm (72 x 72 DPI)

26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Fo
19
20
21
22
r
23
24
Re
25
26
134x73mm (149 x 149 DPI)
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Fo
19
20
21 486x188mm (72 x 72 DPI)
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4 Standard
5 1 2 3 4 5 Mean Deviation
6 (SD)
7
(%) (%) (%) (%) (%) (%)
8
9 Usefulness
10 Relevance 0 0 16.7 66.7 16.7 4.00 0.590
11 Coverage 0 0 8.3 33.3 58.3 4.50 0.659
12 Taxonomy 0 0 8.3 66.7 25.0 4.08 0.654
13
14 Usability
15 Learnability 0 0 25.0 50.0 25.0 4.04 0.722
16 Terminology 0 0 16.7 50.0 33.3 4.17 0.702
17 Aesthetics 0 0 0 50.0 50.0 4.50 0.511
18
Fo
Performance
19
20 Response time 0 0 8.3 50.0 41.7 4.17 0.702
21 Precision 0 0 16.7 58.3 25.0 4.04 0.690
22
r
23
24 Table 1. Descriptive statistics on the visual framework (1=disagree, 5=agree)
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3 Usefulness
4
5 User Profile Relevance Coverage Taxonomy Learnability
6
Researcher Mean 3.75 4.375 4.125 3.75
7
8 SD 0.707 0.744 0.64 0.707
9 Librarian Mean 4 4.75 3.75 4.25
10 SD 0.816 0.5 0.957 0.957
11 Undergraduate
12 student Mean 4.166 4.5 4.166 4.166
13
14 SD 0.389 0.674 0.577 0.5773
15 Total Mean 4.00 4.50 4.08 4.04
16 SD .589 .659 .653 .690
17
18
Fo
19 Table 2. Summary of interaction effects according to user profile (1=low satisfaction, 5=high satisfacti
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3 Usability Performance
4 Response
5 Terminology Aesthetics Precision
Time
6
3.875 4.5 3.875 4
7
8 0.834 0.534 0.834 0.755
9 4.5 4.5 4.25 4
10 0.577 0.577 0.5 0.816
11
12 4.166 4.5 4.333 4.083
13
14 0.717 0.522 0.651 0.668
15 4.13 4.50 4.17 4.04
16 .740 .510 .701 .690
17
18
Fo
ow19
satisfaction, 5=high satisfaction). Source: Own elaboration
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3 Usefulness Usability
4 Reliability Aesthetics
Ease of Use Navigation
5
Mean 4.6667 4.5833 4.7500 4.6667
6
7 Tree SD 0.4815 0.5036 0.4423 0.4815
8 Mean 3.9167 4.2500 3.9167 4.1667
9 SD 0.5036 0.7372 0.6539 0.5647
Bars
10
Mean 3.3333 3.0000 3.1667 2.9167
11
12 Folder SD 0.9631 0.9325 0.8165 0.8805
13 Mean 3.9722 3.9444 3.9444 3.9167
14 SD 0.8717 1.0055 0.9176 0.9894
TOTAL
15
16 Notes: 1=low satisfaction, 5=high satisfaction
17
18 Table 3. Perception of visual search interfaces
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3 Performance
4 Effectiveness
5
4.6667
6
7 0.4815
8 4.0833
9 0.6539
10
3.6667
11
12 0.6370
13 4.1389
14 0.7181
15
16
17
18
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4 Tree interface
5
User Profile Reliability Ease of Use Navigation Aesthetics Effectiveness
6
7 Researcher Mean 4.5000 4.5000 4.5000 4.7500 4.5000
8 SD .53452 .53452 .53452 .46291 .53452
9 Librarian Mean 5.0000 4.7500 5.0000 4.7500 5.0000
10
11 SD 0.00000 .50000 0.00000 .50000 0.00000
12 Undergraduate student Mean 4.6667 4.5833 4.6667 4.7500 4.6667
13 SD .49237 .51493 .49237 .45227 .49237
14 Total Mean 4.6667 4.5833 4.6667 4.7500 4.6667
15
16 SD .48154 .50361 .48154 .44233 .48154
17
18 Table 4. Summary of user perception by user profile (1=low satisfaction, 5=high satisfaction). Source: Own elab
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4 Bars interface Folder interface
5
Reliability Ease of Use Navigation Aesthetics Effectiveness Reliability Ease of Use Navigation Aesthetics
6
7 4.0000 4.5000 4.1250 3.8750 3.8750 3.2500 3.2500 3.1250 3.2500
8 .53452 .53452 .64087 .64087 .64087 1.03510 1.03510 .99103 .88641
9
3.7500 3.7500 4.2500 4.0000 4.5000 3.5000 2.5000 2.5000 3.0000
10
11 .50000 .95743 .50000 .81650 .57735 1.00000 .57735 .57735 .81650
12 3.9167 4.2500 4.1667 3.9167 4.0833 3.3333 3.0000 2.9167 3.1667
13 .51493 .75378 .57735 .66856 .66856 .98473 .95346 .90034 .83485
14
3.9167 4.2500 4.1667 3.9167 4.0833 3.3333 3.0000 2.9167 3.1667
15
16 .50361 .73721 .56466 .65386 .65386 .96309 .93250 .88055 .81650
17
18
tisfaction). Source: Own elaboration
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4
5
Effectiveness
6
7 3.5000
8 .53452
9
4.0000
10
11 .81650
12 3.6667
13 .65134
14
3.6667
15
16 .63702
17
18
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3
4 Reliability Ease of Use Navigation Aesthetics Effectiveness
5
Reliability 1
6
7 Ease of Use 0,352** 1
8 Navigation 0,814 **
0,618** 1
9 Aesthetics 0,562 **
0,577 **
0,708** 1
10 ** ** **
11
Effectiveness 0,591 0,401 0,611 0,568** 1
12 ** The correlation is significant at a level of 0.01 (bilateral).
13
14 Table 5. Pearson correlations for attributes of visual search interfaces
15
16
17
18
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1
2
3 Annex 1- Questionnaire 1
4
5
6 This questionnaire helps us to define your user profile, please be sincere in your answers. No personal
7 information will be stored, remember we are testing the software, not you .
8
9 Are you familiar with these terms? ( ) Metadata ( ) Semantic ( ) Thesaurus ( ) Ontology
10
11
12 Profession: ____________________________ Working at: ___________________________________
13
14 Country: ___________________________
15
16
17
18 What do you know about these searching interfaces?
Fo
19 Textual Search
20
21 Never used ( )
22
r
23 I’ve used it, but I don’t know when using this

24 searching method is more useful than using
Re
25 others. ( )
26
27 I’ve used it, know how it Works and when to use
28 it instead other searching methods. ( )
vi
29 Directory / Categories Browsing

30
31 Never used ( )
ew
32
35 others. ( )
36
I’ve used it, know how it Works and when to use
On
37
39 Tags Search
40
41 Never used ( )
ly
42
45 others. ( )
46
49
50
51
52
53
54
55
56
57
58
59
60
1
1
2
3 Semantic Search
4
5 Never used ( )
6
9 others. ( )
10
13
14
15
16 Please provide us an email in case we’ve to contact you for further information about this trial:
17 EMAIL: _____________________________
18
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
2
1
2
3 Annex 2 – Questionnaire 2
4
5
6 This online questionnaire serves research aims of the AGROKNOW Technologies, laboratory of Viticulture at
7 Agricultural University of Athens and Laboratory IERU, Dpt. of Computer Science, University of Alcalá, Spain. More
specific it aims to gather data (opinions) for the evaluation of VISUAL SEARCH INTERFACES through the aspects
8
of usefulness and ease of use.
9
10 Please reply to the following statements by providing your rate of agreement. The scale employed runs from one (1)
11 to five (5), directed from negative to positive.
12
13 The conductors of this research guarantee the safekeeping and anonymity of the gathered data. For any enquiries
14 please contact us through mail at xxxx@xxxx.com
15
Thank you very much for your interest and your participation.
16
17
18 General aspects of VISUAL FRAMEWORK
Fo
19
20 I believe that the VISUAL FRAMEWORK provides all levels
21 of information (e.g. abstracts, descriptions, etc.) that I need Disagree 1 2 3 4 5 Agree
22 for my information seeking tasks.
r
23 I believe that is easy to see, the number of digital resources

24 in the VISUAL FRAMEWORK (It refers to the number of Disagree 1 2 3 4 5 Agree
Re
25 digital resources of each concept)

26 I believe that the sources in the VISUAL FRAMEWORK are
reliable to support my works tasks. Disagree 1 2 3 4 5 Agree
27 (sources refers to the resources of content providers)
28 I believe that the formats of sources in the VISUAL
vi
29 FRAMEWORK are suitable for my work tasks. Disagree 1 2 3 4 5 Agree

30 (formats: text, sound, image, videos)
31 In general I find the VISUAL FRAMEWORK as a useful Disagree 1 2 3 4 5 Agree
ew
32 system for my work tasks.

33 I believe that the VISUAL FRAMEWORK has a pleasant Disagree 1 2 3 4 5 Agree
34 aesthetic appearance
35 I believe that the VISUAL FRAMEWORK offers easy Disagree 1 2 3 4 5 Agree
methods to navigate in the system
36
I believe that the VISUAL FRAMEWORK uses
On
37 understandable terminology. Disagree 1 2 3 4 5 Agree

38 (Terminology refers to concepts and terms in navigation
39 structure)
40 I believe that the VISUAL FRAMEWORK is a learnable Disagree 1 2 3 4 5 Agree
41 system
ly
42 I believe in general that the VISUAL FRAMEWORK Disagree 1 2 3 4 5 Agree

43 responds very quickly my search
44 In general I find the VISUAL FRAMEWORK as a well Disagree 1 2 3 4 5 Agree
performing system for my work tasks
45
46
About NAVIGATION
47 Please mark one (1) to five (5) from negative to positive aspects related to usability attributes
48 Tree Disagree 1 2 3 4 5 Agree
49 Bars Disagree 1 2 3 4 5 Agree
50 Folders Disagree 1 2 3 4 5 Agree
51
52 About USEFULNESS
57
About EASY OF USE
58
59
60
1
1
2
7
8 About PERFORMANCE
Folders Disagree 1 2 3 4 5 Agree
12
13
About AESTHETIC
18
Fo
19
20
21
22
r
23
24
Re
25
26
27
28
vi
29
30
31
ew
32
33
34
35
36
On
37
38
39
40
41
ly
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
2

ART2

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

ART2

Diunggah oleh

Hak Cipta:

Format Tersedia

Knowledge Organization

An exploratory study of user perception in visual search

Journal: Knowledge Organization

Manuscript Type: Article

investigates whether through information visualisation techniques, we can help creators of

29 controlled vocabulary to classify their resources. Organic.Edunet’s learning materials are

29 The use of knowledge representation schemes in repositories is to facilitate organization and

23 Fig. 3. Design of visual search tool

32 navigational search of Organic.Edunet (Organic-Lingua 2016) and VOA3R (VOA3R-Agris 2016)

37 3.3.2 Evaluation phase

32 understandability of visual search interfaces. We examined the whole visual framework in

29 evaluation that relate to the visual search interfaces (section 4.2).

32 undergraduate students (mean=4.5; SD=0.522). A small percentage of participants (20.8%)

32 knowledge representation scheme through external repositories. Thirdly, visual analytics of

23 Information Science 46: 60–70. doi:10.1007/978-3-642-04590-5_6.

25 Cervera, J F, M G López-López, C Fernández, and S Sánchez-Alonso. 2009. “Quality Metrics

29 Chrysafiadi, K, and M Virvou. 2013. “A Knowledge Representation Approach Using Fuzzy

37 Profiling.” Computer Graphics Forum 30: 1141–50. doi:10.1111/j.1467-8659.2011.01963.x.

Interfaces, Capri Island, Italy, 233–40. doi:10.1145/2254556.2254599.

25 Gleicher, M, D Albers, R Walker, I Jusufi, C D Hansen, and J C Roberts. 2011. “Visual

29 Gomez-Perez, A. 1999. “Evaluation of Taxonomic Knowledge in Ontologies and Knowledge

37 Griffiths, J R, F Johnson, and R J Hartley. 2007. “User Satisfaction as a Measure of System

Hartson, H R, P Shivakumar, and M A Pérez-Quiñones. 2004. “Usability Inspection of Digital

32 Exploration of Large Video Collections and Associated Metadata.” In UIST 2014 -

37 the Digital Age, 104–15.

25 International Journal of Information Management 29: 196–209.

29 Interactive Information Retrieval.” Information Processing and Management 44: 22–38.

for Interactive Information Visualization Systems.” Journal of Visual Languages and

29 Šimek, P, J Vaněk, V Očenásek, M Stočes, and T Vogeltanzová. 2012. “Using Metadata

25 250x127mm (72 x 72 DPI)

23 I’ve used it, but I don’t know when using this

29 Directory / Categories Browsing

23 I believe that is easy to see, the number of digital resources

25 digital resources of each concept)

29 FRAMEWORK are suitable for my work tasks. Disagree 1 2 3 4 5 Agree

32 system for my work tasks.

37 understandable terminology. Disagree 1 2 3 4 5 Agree

42 I believe in general that the VISUAL FRAMEWORK Disagree 1 2 3 4 5 Agree

Anda mungkin juga menyukai