How are exoskeletons tested and evaluated numerically? The principle engineers from Ekso Bionics, one of the oldest companies in the exoskeleton industry, set out to shed some light on this question. Deciding on a method for testing wearable robots can have a broad impact on multiple projects and companies.
Special thanks to the Wearable Robotics Association and Ekso Bionics for putting together the October 4th, 2016, Lunch & Learn event. The presenters were Dr. Kurt Amundson, Sr. Director at Ekso Labs and Health and Dr. Katherine Strausser, Sr. Controls Engineer. Both have tremendous expertise with exoskeleton systems and over a decade worth of experience in the industry.
If you were not able to attend, below is a recap of the talk mixed in with a heavy dose of commentary.
How companies approach testing wearable robots seems to relate to their background and personal experience. There is a wide range of what testing means. To make matters more complicated, there are different types of devices with completely different goals. For example, a medical rehabilitation exoskeleton needs to facilitate interactive, user-controlled proper gait. In contrast, a tool holding exoskeleton needs to support or redistribute the weight of an instrument.
Sometimes it becomes difficult to distinguish between devices. For instance, a powered assistive exoskeleton will supply energy to complete a movement (for example walk up a flight of stairs) while a powered augmentative exoskeleton needs to provide an augmentative force while carrying a heavy load. On the outside, however, the two devices could appear identical, but they have diverging design goals and their testing needs to reflect that. Regardless, there is a lot of testing that applies to all wearable robots.
Exoskeleton testing ranges on a spectrum with the robot on one end and the person (user/pilot) on the other. In other words, classical engineering testing (motor power consumption, heat dissipation,…) vs physiological testing. But really, the exoskeleton industry and wearable robots are a whole new world which falls right in the middle.
Example: Power transfer between exo and human.
Initially, the Warrior Web program was focused on injury prevention “The Warrior Web program seeks to develop the technologies required to prevent and reduce musculoskeletal injuries caused by dynamic events typically found in the warfighter’s environment.” The testing of the 15 different devices submitted over the years became a “before and after model.” Record the performance of a soldier with and without the suit in identical conditions and then compare. Any difference in performance is attributed to the wearable robot.
Initially, all of the actuators of the robots were tested independently. But how to test the wearables with the human inside and get numerical, objective results? One approach was to check for oxygen consumption and heart rate while carrying a load, or the metabolic cost. But this is done on a treadmill over several hours in order to cut through the noise of the data. But what does walking on a treadmill have to do with injury prevention or “dynamic events found in the warfighter’s environment”? The design of the testing parameters had changed the entire program from safety to reducing energy consumption!
So now because of the testing what we are looking for is just energy and we no longer consider safety or injury prevention. Some exoskeleton entries for the program had to be completely redesigned. With these test conditions, we want the smallest possible exo that can provide the greatest energy. Devices such as the HULC and the WYSS Soft Exosuit (and many others) were able to achieve this. Soldiers wearing the exoskeletons did demonstrate a reduction in metabolic cost while walking on a treadmill. But if that is the only benefit from a military exoskeleton is the wearable robot worth the cost?
Bringing the cost of testing down:
How could a project be roped back to physiological testing? Ideally, a thousand prototypes would be given to the army and used over two years and the results collected. But this will be too slow and costly. One idea the presenters had was to take advantage of virtual modeling such as the OpenSim human modeling by Stanford (could AnyBody Simulation also be used?).
Another suggested approach was to utilize a slightly less scientific but more relatable testing, such as two people completing the same task with and without an exo side by side. It is less scientifically accurate because now there are two different users. Could one of them be working faster because they are more motivated? How can one prove the exact contribution of the wearable? An example was made using the marketing video for the ZeroG (G4 Arm). In it a worker that uses a spring loaded arm completes his assigned task much faster and dramatically sits down to drink a lemonade while the “regular” worker is lagging behind:
What about medical?
It is the most important, but how safe is safe? What about quality of life vs safety? There are already policies and guidelines on the subject from ISO, OSHA, and the FDA. However, regular robot testing remains much easier than when there is a human interacting with the machine. It is up to the medical robotics community to pay attention in shaping these policies as each company can have an impact on the regulators.
Does it do what we want the device to do? This can be tested with weights or a dummy to simulate a person wearing the device. Does the wearable robot move properly when loaded, can it execute commands such as walking, standing or squatting?
This is the tricky one, are we testing the correct outcomes? If a rehabilitation exoskeleton focused on just one parameter, such as the number of steps taken, is that enough to judge the efficacy of the rehabilitation it provides? This becomes a “surrogate measurement.” We try to equate something that can be numerically measured, in this case the number or steps, and we relate it to successful gait rehabilitation.
Unfortunately, it turns out that this is not accurate and just measuring the number of steps is meaningless if other conditions are not met. For rehabilitation to work the patient has to be fully engaged through variable assistance, visual or haptic feedback. The user has to want to perform the motion, otherwise, they are just going along for the ride. Even worse, the natural response for unwarranted motion is to try to move in the reverse direction. The past has shown that surrogate measurements can have a negative impact not only on the company that developed the wearable robot testing but on the entire industry.
Another way to look at the danger of focusing on just a few surrogate measurements is by imagining buying a vehicle solely on gas mileage. If we considered gas mileage and nothing else, then we will all be driving mopeds, even when a moped would clearly not be the correct choice. Need to move furniture, use your moped. Someone needs to be driven to the hospital, just throw them on your moped. Rain or shine, the moped has the best gas mileage but is clearly not always the best vehicle depending on the task at hand.
But wait a second, researchers from various rehabilitation centers have done a good job quantifying and measuring the performance of medical exoskeletons. So why is it made to sound so difficult? Part of the difficulty comes that those are specialized medical centers that take years for their research. This is outside of the scope of exoskeleton companies which, are at their core, engineering businesses that need to test wearable robots faster and more economically.
We are all in this together.
What is the take home message? That the exoskeleton industry is really one big community. The decision on how to test one wearable robot could have an impact on other projects and the industry’s reputation. The wearable robotics community is responsible for considering biases when making their studies. Consider the environment, test protocol, define metrics, and find what is acceptable.
There is a need to better identify and record confounds – who can’t use the exoskeletons is just as important as who can. Ensure the primary test goal is linked to the primary project goal. Finally, verify that the data being collected is accurate. When designing tests for wearable robots we have to be weary that we are testing for the correct thing, in the correct context.
Warrior Web, http://www.darpa.mil/program/warrior-web