Survey Results and Analysis of Usability Evaluation Practices of Use in Wearable Robotics

Surveys, when done and analyzed properly, reveal both what is the current state and what is next for the exoskeleton industry. They can be a window into the thoughts of both producers and consumers, and a great learning opportunity. The survey’s ultimate aim is to make recommendations that tackle the technology acceptance limitations. Two years ago, the Exoskeleton Report had a call for action, asking our readers to contribute to a Survey: How do you evaluate the usability of wearable robots?

The results from the survey are back! The information has been carefully analyzed and published in a multi-page journal paper:

An analysis of usability evaluation practices and contexts of use in wearable robotics

The principle authors from ETH Zurich recorded a total of 286 responses to the online survey, but only 102 were fully completed. The responders used wearable robotics devices (WRD) for Augmentation (16.5%), Assistance (38.0%) Therapy (39.8%), and Other (5.7%). As expected, WRD for therapy and assistance were expected to function with supervision, and those for augmentation, mostly without (see Figure 1, below). The responders were primarily associated with lower body and hand/wrist wearable robotics. Use environments were split between leisure (consumer), work, laboratory, and clinic environment, with a notable and surprising dominance of home-use devices. The medical WRD were targeted primarily for stroke, spinal cord injury, and musculoskeletal disorders. This aligns perfectly with what the more established and well-funded exoskeleton producers and robotic orthotics developers are marketing their devices for.

Not surprisingly for emerging technology, the majority of responders felt that the technology readiness level of wearable robotic devices was somewhere in the “prototype” stage, and one quarter (1/4) see them as a “product.” The survey report focuses on how the users are evaluating the WRD and what are they looking for. Likert and Visual Analog scales, together with free text fields were dominant, but closely folwed by a plethora (12+) of established indexes and usability scales. Evaluated attributes were focused on functionality, ease of use, performance, safety, and comfort (surprisingly, safety was in 4th place).

After all the evaluation work on WRD was done, more than half of all the participants felt that they were not able to compare their evaluation data with state-of-the-art benchmarks. Similarly, the majority of participants were not able to easily find standardized measures for their context of use. This also led to a lack of confidence that the usability testing was sufficiently thorough. While the participants were not able to compare their evaluation data or match it to industry-accepted standardized measurement, almost 3/4 of the participants felt that the usability evaluation data improved the device development.

The survey authors extrapolate that end-users are involved in the WRD development pipeline relatively late, likely due to financial constraints. The current focus is testing device effectiveness, rather than efficiency and user satisfaction, which will be the ultimate deciding factors if an exoskeleton would be used continuously or abandoned. The authors also note that:

This appears to be a chicken-and-egg problem: WRD developers can’t find validated and standardized tools for their specific context of use and thus start creating their own, customized measures. At the same time, the usage of such non-standardized and non-validated metrics inhibits the emergence of evaluation standards and benchmarks...
...What the WRD field therefore might need, are not only guidelines and benchmarks for usability evaluation practice, but an agreement among peers to also value qualitative research as scientific practice worth publishing.

Recommendations:

In addition to the previously stated recommendations above to include real-world users sooner in the evaluation process and focus on usability and satisfaction, the authors have two additional take-home messages. First, usability evaluation should report more qualitative data that calls for more personal interactions with the target users (trust the people). Second, the authors would like to see a shift away from custom usability evaluation and more reproducible, standardized measurements that can lead to establishing evaluation benchmarks.

The survey was also used to guide the creation of an Interactive Usability Toolbox, which aims to support WRD evaluation by addressing the identified limitations. Follow the link to the Interactive Usability Toolbox: https://www.usabilitytoolbox.ch (note that it is not optimized for mobile devices as of this publication.)

New Survey is Open:

The Medical Practioner Exoskeleton Technology Survey by ASTM International is currently open and looking for additional participants. The survey specifically targets healthcare professionals and their use of medical exoskeletons. If you or your company works with physical therapists or other relevant professionals, please share with them the survey link: https://www.etcoe.org/

Explore the full survey paper and forward the medical survey:

• Meyer, J.T., Gassert, R. & Lambercy, O. An analysis of usability evaluation practices and contexts of use in wearable robotics. J NeuroEngineering Rehabil 18, 170 (2021). https://doi.org/10.1186/s12984-021-00963-8
• Global Survey of Healthcare Professionals using Medical Exoskeletons, ASTM Exo Technology Center of Excellence, https://survey.sogosurvey.com/survey1.aspx?k=RTsQYVYXXsWsPsPsP&lang=0