Malmö University Publications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Human Factors in Interactive Online Machine Learning
Malmö University, Faculty of Technology and Society (TS), Department of Computer Science and Media Technology (DVMT). Malmö University, Internet of Things and People (IOTAP). Swedish Defense Research Agency (FOI), Stockholm, Sweden.ORCID iD: 0000-0002-3155-8408
Malmö University, Faculty of Technology and Society (TS), Department of Computer Science and Media Technology (DVMT). Malmö University, Internet of Things and People (IOTAP).ORCID iD: 0000-0003-0998-6585
Malmö University, Internet of Things and People (IOTAP). Malmö University, Faculty of Technology and Society (TS), Department of Computer Science and Media Technology (DVMT).ORCID iD: 0000-0002-9471-8405
2023 (English)In: HHAI 2023: Augmenting Human Intellect / [ed] Paul Lukowicz; Sven Mayer; Janin Koch; John Shawe-Taylor; Ilaria Tiddi, IOS Press, 2023, p. 33-45Conference paper, Published paper (Refereed)
Abstract [en]

Interactive machine learning (ML) adds a human-in-the-loop aspect to a ML system. Even though the input from human users to the system is a central part of the concept, the uncertainty caused by the human feedback is often not considered in interactive ML. The assumption that the human user is expected to always provide correct feedback, typically does not hold in real-world scenarios. This is especially important for when the cognitive workload of the human is high, for instance in online learning from streaming data where there are time constraints for providing the feedback. We present experiments of interactive online ML with human participants, and compare the results to simulated experiments where humans are always correct. We found combining the two interactive learning paradigms, active learning and machine teaching, resulted in better performance compared to machine teaching alone. The results also showed an increased discrepancy between the experiments with human participants and the simulated experiments when the cognitive workload was increased. The findings suggest the importance of taking uncertainty caused by human factors into consideration in interactive ML, especially in situations which requires a high cognitive workload for the human.

Place, publisher, year, edition, pages
IOS Press, 2023. p. 33-45
Series
Frontiers in Artificial Intelligence and Application, ISSN 0922-6389, E-ISSN 1879-8314 ; 368
Keywords [en]
interactive machine learning, online learning, human factors
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:mau:diva-61687DOI: 10.3233/faia230073ISI: 001150361600003Scopus ID: 2-s2.0-85171485242ISBN: 978-1-64368-394-2 (print)ISBN: 978-1-64368-395-9 (electronic)OAI: oai:DiVA.org:mau-61687DiVA, id: diva2:1780702
Conference
HHAI 2023, the 2nd International Conference on Hybrid Human-Artificial Intelligence, 26-30 June 2023, Munich, Germany
Available from: 2023-07-06 Created: 2023-07-06 Last updated: 2024-02-26Bibliographically approved
In thesis
1. Interactive Online Machine Learning
Open this publication in new window or tab >>Interactive Online Machine Learning
2022 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

With the Internet of Things paradigm, the data generated by the rapidly increasing number of connected devices lead to new possibilities, such as using machine learning for activity recognition in smart environments. However, it also introduces several challenges. The sensors of different devices might be mobile and of different types, i.e. there is a need to handle streaming data from a dynamic and heterogeneous set of sensors. In machine learning, the performance is often linked to the availability and quality of annotated data. Annotating data is in general costly, but it can be even more challenging if there is not any, or a very small amount of, annotated data to train the model on at the start of learning. To handle these issues, we implement interactive and adaptive systems. By including human-in-the-loop, which we refer to as interactive machine learning, the input from users can be utilized to build the model. The type of input used in interactive machine learning is typically annotations of the data, i.e. correctly labelled data points. Generally, it is assumed that the user always provides correct labels in accordance with the chosen interactive learning strategy. In many real-world applications these assumptions are not realistic however, as users might provide incorrect labels or not provide labels at all in line with the chosen strategy.

In this thesis we explore which interactive learning strategy types are possible in the given scenario and how they affect performance, as well as the effect of machine learning algorithms on the performance. We also study how a user who is not always reliable, i.e. who does not always provide a correct label when expected to, can affect performance. We propose a taxonomy of interactive online machine learning strategies and test how the different strategies affect performance through experiments on multiple datasets. Simulated experiments are compared to experiments with human participants, to verify the results. The findings show that the overall best performing interactive learning strategy is one where the user provides labels when current estimations are incorrect, but that the best performing machine learning algorithm depends on the problem scenario. The experiments also show that a decreased reliability of the user leads to decreased performance, especially when there is a limited amount of labelled data. The robustness of the machine learning algorithms differs, where e.g. Naïve Bayes classifier is better at handling a lower reliability of the user. We also present a systematic literature review on machine teaching, a subfield of interactive machine learning where the human is proactive in the interaction. The study shows that the area of machine teaching is rapidly evolving with an increased number of publications in recent years. However, as it is still maturing, there exists several open challenges that would benefit from further exploration, e.g. how human factors can affect performance.

Place, publisher, year, edition, pages
Malmö: Malmö universitet, 2022. p. 209
Series
Studies in Computer Science ; 18
Keywords
Interactive Machine Learning, Active Learning, Machine Teaching, Online Learning
National Category
Computer Sciences
Identifiers
urn:nbn:se:mau:diva-51987 (URN)10.24834/isbn.9789178772810 (DOI)978-91-7877-280-3 (ISBN)978-91-7877-281-0 (ISBN)
Public defence
2022-06-23, HS aula samt livestramas, Jan Waldenströms gata 25, Malmö, 10:00 (English)
Opponent
Supervisors
Note

In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of Malmö University's products or services. Internal or personal use of this material is permitted.

Paper VI and VII appear in dissertation as manuscripts.

Available from: 2022-06-03 Created: 2022-06-02 Last updated: 2023-09-05Bibliographically approved

Open Access in DiVA

fulltext(818 kB)104 downloads
File information
File name FULLTEXT01.pdfFile size 818 kBChecksum SHA-512
c727be4ba55ac9bb523bd7322dc7ae49f3b6ca54ccdaab5e13dc87b6bd086fc75f27c65117c7b429006de2ecb088c8d8d8155a05ce4449996bd32f91f255df2c
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Tegen, AgnesDavidsson, PaulPersson, Jan A.

Search in DiVA

By author/editor
Tegen, AgnesDavidsson, PaulPersson, Jan A.
By organisation
Department of Computer Science and Media Technology (DVMT)Internet of Things and People (IOTAP)
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 104 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 119 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf