Meet-AI.
24 February 2022 — 3 minutes read
Artificial Intelligence is increasingly making its way into our lives, but how is it affecting the User Experience (UX) of products and services? What qualities are end-users looking for in AI-infused products? How can we measure those qualities?
These products enable human-computer interactions that we have never experienced before; this is why they are game-changing. Nevertheless, the design discipline seems to have overlooked the impact of the introduction of AI systems in industrial products so far. The Meet-AI project is one of the first attempts to frame these issues through a structured methodology that unfolded with three major steps:
How is the UX currently assessed?
The idea of evaluating the user experience dates back to the early 2000s, and it developed through different orientation shifts. The first part of the research activity specifically focused on tracking all the methods available so far. We identified and analysed 129 methods to understand their suitability for AI-Infused products and the qualities they assess. As a result, we created a list of methods that is now freely available to the scientific community.
Fig. 1. UX qualities for Pragmatic dimension retrieved from literature review.
How do end-users perceive them?
The analysis of the literature regarding Artificial Intelligence and Design shows that new qualities should be considered. We verified this hypothesis through a questionnaire submitted to 50 expert users.
The results highlight that the responders do not consider AI-infused products as traditional products. Aesthetics qualities, ease of use, ergonomics, and pleasantness of use are no longer the only elements impacting user experience.
The end-users are interested in nested intelligence. They want to enter an empathetic relationship with them. They want to trust those objects and be sure they are not betraying their commitment. In other words, they are looking for assistants rather than products.
Fig. 2. Most relevant UX dimensions assessed by expert users.
So, what are the new qualities these products should have?
Moving from the questionnaire results, we built long lists of qualities that users attach to AI-infused products, organised according to eight main dimensions. Through a structured confrontation with the literature and within the group of researchers, we hypothesised a shortlist including those aspects we think users care about the most in AI-infused systems. They include transparency in showing which data are collected, how they are used, the reliability of the system, and its accuracy and usefulness. We also included qualities such as the system’s customizability and adaptability and, naturally, the capacity to listen to the user and provide answers using the voice.
Fig. 3. Organisation of the qualities from the questionnaire – UX dimension: Intelligence.
Are they the right qualities? How to measure them?
To verify our hypothesis, we created a questionnaire including numerous questions. We then submitted it to a significant number of respondents (about 650), asking them to assess their smart speaker. In scientific jargon, this is exploratory factor analysis. It allowed to skim less relevant qualities and recognise those that end-users consider significant for their user experience.
To assess the correctness of the identified qualities, we asked other 700 users to answer the revised questionnaire. This activity (confirmatory factor analysis) allowed us to skim the questions and reach the final questionnaire.
Fig. 4. Preparatory activity for defining the relevant qualities and related questions.
And now?
We have a validated questionnaire that assesses the user experience of systems that integrate AI capabilities. It includes 33 questions and can be used by students, designers, and companies to evaluate how their concept or system performs under different qualities. It is the first of its kind, and in the next months, we will continue to refine the system, adding tools to facilitate its adoption.