Human-robot Interaction and Human Digital Twins


Continuous monitoring of human movement in the real world using wearable sensing technologies

Student: Abigail Rafter

Human-robot collaboration is becoming increasingly prevalent in many fields, especially manufacturing. In this research, we present a standard framework and methodology to develop digital twins (DTs) of humans for human-robot collaboration in manufacturing environments. Digital twins are purpose-driven digital replicas of physical systems. A standard framework and methodology to develop DTs are necessary for their efficient implementation. This type of framework and methodology are proposed for DTs in manufacturing to enforce requirements of scalability, reusability, interoperability, interchangeability, and extensibility. The second step in this research is to understand how to coordinate human and robot DTs during human-robot collaboration.

Flowchart

Human Motion Forecasting for Dynamic Human-Robot Collaboration

Students: Leo Bringer, Joey Wilson

Industry 5.0 has laid the necessity to relocate the human at the center of the manufacturing cycle. This implies one to redefine the human-robot collaboration, making it not only safe but also more inclusive for the operator. Nowadays, this goal is viable thanks to the integration of sensors in the work cell. The sensors provide the robot with a better understanding of its surrounding environment, allowing more organic cooperation through the use of advanced control strategies.

This project focuses on developing a neural network-based method for Human Motion Forecasting. The idea is to perform Human Pose Estimation in real-time using a deep learning model and understand the intrinsics of human motion in the context of Dynamic Human-Robot Collaboration to predict the future position of the human operator. These predictions will then be used to adapt the trajectory of the robot to avoid collisions with the human or to perform some even more complex collaborative tasks (pick and place, surface polishing, etc.). The network architecture relies on Graph Convolutional Networks, Self-Attention Layers, and LSTMs. The code, pre-trained models, and further explanations are available on our GitHub page.

Human motion capture diagram

Continuous monitoring of human movement in the real world using wearable sensing technologies

Student: Loubna Baroudi

Graphs

Walking speed is a strong health indicator for various health issues. However, measures of walking speed are often carried in clinical settings, where individuals walking behavior might not be representative of their natural behavior. Wearable sensors offer the opportunity to monitor human movement in the real world over extended periods. There are existing methods to estimate walking speed using Inertial Measurement Units (IMUs). However, these sensors typically have limited battery lives and can be cumbersome for long wear. As such, in this project we answered the following research question: Can we accurately estimate walking speed in the real world using a single-point accelerometer? We leveraged existing IMU methods to validate our estimation method.


Characterizing Walking Speed Variability Using An Accelerometer

Student: Loubna Baroudi

In the project described above, we developed and validated a model for the estimation of stride speed using an accelerometer. However, there is a large variability in the walking speed individuals use in their daily lives. As such, it is difficult to know how to interpret the data collected over extended periods in the real world. In this project, we introduced the notion of walking periods: a walking period starts when walking is detected and stops when an individual seat or stand still for more than a minute. Walking periods capture the natural start-and-stop nature of real-world walking. For instance, walks before and after stopping at a pedestrian light would be grouped together in a walking period. Then, we explored whether period duration and continuity (how much standing there is in a walking period) explain some of the variability in walking speed observed in the real world.

Graphs