Brokoslaw Laschowski - Academia.edu (original) (raw)

Academic Publications by Brokoslaw Laschowski

BioMedical Engineering OnLine, 2024

Human–robot walking with prosthetic legs and exoskeletons, especially over complex terrains, such... more Human–robot walking with prosthetic legs and exoskeletons, especially over complex terrains, such as stairs, remains a significant challenge. Egocentric vision has the unique potential to detect the walking environment prior to physical interactions, which can improve transitions to and from stairs. This motivated us to develop the StairNet initiative to support the development of new deep learning models for visual perception of real-world stair environments. In this study, we present a comprehensive overview of the StairNet initiative and key research to date. First, we summarize the development of our large-scale data set with over 515,000 manually labeled images. We then provide a summary and detailed comparison of the performances achieved with different algorithms (i.e., 2D and 3D CNN, hybrid CNN and LSTM, and ViT networks), training methods (i.e., supervised learning with and without temporal data, and semi-supervised learning with unlabeled images), and deployment methods (i.e., mobile and embedded computing), using the StairNet data set. Finally, we discuss the challenges and future directions. To date, our StairNet models have consistently achieved high classification accuracy (i.e., up to 98.8%) with different designs, offering trade-offs between model accuracy and size. When deployed on mobile devices with GPU and NPU accelerators, our deep learning models achieved inference speeds up to 2.8 ms. In comparison, when deployed on our custom-designed CPU-powered smart glasses, our models yielded slower inference speeds of 1.5 s, presenting a trade-off between human-centered design and performance. Overall, the results of numerous experiments presented herein provide consistent evidence that StairNet can be an effective platform to develop and study new deep learning models for visual perception of human–robot walking environments, with an emphasis on stair recognition. This research aims to support the development of next-generation vision-based control systems for robotic prosthetic legs, exoskeletons, and other mobility assistive technologies.

Journal of Computational and Nonlinear Dynamics, 2023

Robotic leg prostheses and exoskeletons have traditionally been designed using highly-geared moto... more Robotic leg prostheses and exoskeletons have traditionally been designed using highly-geared motor-transmission systems that minimally exploit the passive dynamics of human locomotion, resulting in inefficient actuators that require significant energy consumption and thus provide limited battery-powered operation or require large onboard batteries. Here we review two of the leading energy-efficient actuator design principles for legged and wearable robotic systems: series elasticity and backdrivability. As shown by inverse dynamic simulations of walking, there are periods of negative joint mechanical work that can be used to increase efficiency by recycling some of the otherwise dissipated energy using series elastic actuators and/or backdriveable actuators with energy regeneration. Series elastic actuators can improve shock tolerance during foot-ground impacts and reduce the peak power and energy consumption of the electric motor via mechanical energy storage and return. However, actuators with series elasticity tend to have lower output torque, increased mass and architecture complexity due to the added physical spring, and limited force and torque control bandwidth. High torque density motors with low-ratio transmissions, known as quasi-direct drives, can likewise achieve low output impedance and high backdrivability, allowing for safe and compliant human-robot physical interactions, in addition to energy regeneration. However, torque-dense motors tend to have higher Joule heating losses, greater motor mass and inertia, and require specialized motor drivers for real-time control. While each actuator design has advantages and drawbacks, designers should consider the energy-efficiency of robotic leg prostheses and exoskeletons beyond steady-state level-ground walking.

Frontiers in Neurorobotics, 2022

Robotic leg prostheses and exoskeletons can provide powered locomotor assistance to older adults ... more Robotic leg prostheses and exoskeletons can provide powered locomotor assistance to older adults and/or persons with physical disabilities. However, the current locomotion mode recognition systems being developed for automated high-level control and decision-making rely on mechanical, inertial, and/or neuromuscular sensors, which inherently have limited prediction horizons (i.e., analogous to walking blindfolded). Inspired by the human vision-locomotor control system, we developed an environment classification system powered by computer vision and deep learning to predict the oncoming walking environments prior to physical interaction, therein allowing for more accurate and robust high-level control decisions. In this study, we first reviewed the development of our “ExoNet” database—the largest and most diverse open-source dataset of wearable camera images of indoor and outdoor real-world walking environments, which were annotated using a hierarchical labeling architecture. We then trained and tested over a dozen state-of-the-art deep convolutional neural networks (CNNs) on the ExoNet database for image classification and automatic feature engineering, including: EfficientNetB0, InceptionV3, MobileNet, MobileNetV2, VGG16, VGG19, Xception, ResNet50, ResNet101, ResNet152, DenseNet121, DenseNet169, and DenseNet201. Finally, we quantitatively compared the benchmarked CNN architectures and their environment classification predictions using an operational metric called “NetScore,” which balances the image classification accuracy with the computational and memory storage requirements (i.e., important for onboard real-time inference with mobile computing devices). Our comparative analyses showed that the EfficientNetB0 network achieves the highest test accuracy; VGG16 the fastest inference time; and MobileNetV2 the best NetScore, which can inform the optimal architecture design or selection depending on the desired performance. Overall, this study provides a large-scale benchmark and reference for next-generation environment classification systems for robotic leg prostheses and exoskeletons.

Conference Publications by Brokoslaw Laschowski

IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), 2024

Environment sensing and recognition can allow hu-mans and/or robotic systems to dynamically adapt... more Environment sensing and recognition can allow hu-mans and/or robotic systems to dynamically adapt to different walking terrains. However., fast yet accurate visual perception is challenging., especially on embedded systems with limited computational resources. The purpose of this study was to develop and prototype a new pair of integrated AI-powered smart glasses for onboard sensing and recognition of human-robot walking en-vironments with high accuracy and low latency. Our system in-cludes a Raspberry Pi Pico micro controller and an ArduCam low-power camera., both of which interface with commercial eye-glass frames via 3D-printed mounts that we custom-designed. We trained and optimized a lightweight and efficient convolutional neural network using a MobileN etVI backbone to classify real-world walking terrains as either indoor surfaces., outdoor surfaces (grass and dirt)., or outdoor surfaces (paved) using over 62,500 egocentric images that we adapted and manually labelled from the Meta Eg04D dataset. We compiled and deployed our deep learning model using TensorFlow Lite micro and post-training quantization to create a minimized byte array model of size 0.31MB. Our system was able to accurately classify complex walking environments with 93.6% accuracy and an embedded inference speed of 1.5 seconds during online experiments. These AI-powered smart glasses open new opportunities for visual per-ception of human-robot walking environments where real-time embedded computing is desired. Future research will focus on improving the onboard inference speed and further miniaturization of the mechatronic components.

IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), 2024

Robotic prosthetic legs and exoskeletons require real-time and accurate predictions of the walkin... more Robotic prosthetic legs and exoskeletons require real-time and accurate predictions of the walking environment for smooth transitions between different locomotion mode controllers. However, previous studies have mainly been limited to static image classification, therein ignoring the temporal dynamics of human-robot locomotion. Motivated by these limitations, here we developed and tested a number of state-of-the-art temporal neural networks to compare the performance between using static vs. sequential images for environment classification (i.e., level-ground terrain, incline stairs, and transitions to and from stairs). Using our large-scale image dataset, we trained several 2D encoder networks such as MobileNetV2 and MobileViT, each coupled with a temporal long short-term memory (LSTM) backbone. We also trained MoViNet, a new 3D video classification model, to further compare the performance between 2D and 3D temporal neural networks. The 3D network outperformed the 2D encoder networks with LSTM backbones and a 2D CNN baseline model in terms of image classification accuracy, suggesting that the network architecture can play an important role. However, although the 3D neural network achieved the highest image classification accuracy (98.3%), it had disproportionally higher computational and memory storage requirements, which has practical implications for real-time embedded computing for control of robotic leg prostheses and exoskeletons.

IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), 2024

Control of robotic leg prostheses and exoskeletons is an open challenge. Computer modeling and si... more Control of robotic leg prostheses and exoskeletons is an open challenge. Computer modeling and simulation can be used to study the dynamics and control of human walking and extract principles that can be programmed into robotic legs to behave similar to biological legs. In this study, we present the development of an efficient two-layer Q-learning algorithm, with k-d trees, that operates over continuous action spaces and a reward model that estimates the degree of muscle activation similarity between the agent and human state-to-action pairs and state-to-action sequences. We used a human musculoskeletal model acting in a high-dimensional physics-based simulation environment to train and evaluate our algorithm to simulate biomimetic walking. We used imitation learning and artificial biomechanics data to accelerate training via “expert” demonstrations and used experimental human data to compare and validate our predictive simulations, achieving 79% accuracy. Moreover, when compared to the previous state-of-the-art that used deep deterministic policy gradient, our algorithm was significantly more efficient with lower computational and memory storage requirements (i.e., requiring 7 times less RAM and 87 times less CPU compute), which can benefit real-time embedded computing for robot control. Overall, our new two-layer Qlearning algorithm using sequential data for continuous imitation of human locomotion serves as a first step towards the development of bioinspired controllers for robotic prosthetic legs and exoskeletons. Future work will focus on improving the prediction accuracy compared to experimental data and expanding our simulations to other locomotor activities.

IEEE International Conference on Robotics and Automation (ICRA), 2023

Unlike traditional hierarchical controllers for robotic leg prostheses and exoskeletons, continuo... more Unlike traditional hierarchical controllers for robotic leg prostheses and exoskeletons, continuous systems could allow persons with mobility impairments to walk more naturally in real-world environments without requiring high-level switching between locomotion modes. To support these next-generation controllers, we developed a new system called KIFNet (Kinematics and Image Fusing Network) that uses lightweight and efficient deep learning models to continuously predict the leg kinematics during walking. We tested different sensor fusion methods to combine kinematics data from inertial sensors and computer vision data from smart glasses and found that adaptive instance normalization achieved the lowest RMSE predictions for knee and ankle joint kinematics. We also deployed our model on an embedded device. Without inference optimization, our model was 20 times faster than the previous state-of-the-art and achieved 20% higher prediction accuracies, and during some locomotor activities like stair descent, decreased RMSE up to 300%. With inference optimization, our best model achieved 125 FPS on an NVIDIA Jetson Nano. These results demonstrate the potential to build fast and accurate deep learning models for continuous prediction of leg kinematics during walking based on sensor fusion and embedded computing, therein providing a foundation for real-time continuous controllers for robotic leg prostheses and exoskeletons.

2022 9th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), 2022

Backdriveable actuators with energy regeneration can improve the efficiency and extend the batter... more Backdriveable actuators with energy regeneration can improve the efficiency and extend the battery-powered operating times of robotic lower-limb exoskeletons by converting some of the otherwise dissipated energy during negative mechanical work into electrical energy. However, previous related studies have focused on steady-state level-ground walking. To better encompass real-world community mobility, here we developed a feedforward human-exoskeleton energy regeneration system model to simulate energy regeneration and storage during other daily locomotor activities. Data from inverse dynamics analyses of 10 healthy young adults walking at variable speeds and slopes were used to calculate the negative joint mechanical power and work (i.e., the mechanical energy theoretically available for electrical energy regeneration). These human joint mechanical energetics were then used to simulate backdriving a robotic exoskeleton and regenerating energy. An empirical characterization of the exoskeleton device was carried out using a joint dynamometer system and an electromechanical motor model to calculate the actuator efficiency and to simulate energy regeneration. Our performance calculations showed that regenerating energy at slower walking speeds and decline slopes could significantly extend the battery-powered operating times of robotic lower-limb exoskeletons (i.e., up to 99% increase in total number of steps), therein improving locomotor efficiency.

2022 International Conference on Rehabilitation Robotics (ICORR), 2022

Computer vision can be used in robotic exoskeleton control to improve transitions between differe... more Computer vision can be used in robotic exoskeleton control to improve transitions between different locomotion modes through the prediction of future environmental states. Here we present the development of a large-scale automated stair recognition system powered by convolutional neural networks to recognize indoor and outdoor real-world stair environments. Building on the ExoNet database- the largest and most diverse open-source dataset of wearable camera images of walking environments-we designed a new computer vision dataset, called StairNet, specifically for stair recognition with over 515,000 images. We then developed and optimized an efficient deep learning model for automatic feature engineering and image classification. Our system was able to accurately predict complex stair environments with 98.4% classification accuracy. These promising results present an opportunity to increase the autonomy and safety of human-exoskeleton locomotion for real-world community mobility. Future work will explore the mobile deployment of our automated stair recognition system for onboard real-time inference.

Conference Presentations by Brokoslaw Laschowski

Toronto Robotics Conference, 2024

Muscle activations measured using surface electromyography (EMG) can be used for direct neural co... more Muscle activations measured using surface electromyography (EMG) can be used for direct neural control of robot actuators. This method can improve control and human-robot interaction, in addition to supporting non-cyclical activities such as seated leg movements. However, recognizing user intent in real-time using a myoelectric interface requires an accurate, responsive, and intuitive controller. Figure 2: System diagram of our new proportional myoelectric controller for robot actuators. Discussion Our new myoelectric controller for robotic actuators achieved state-of-the-art performance in terms of processing speed (8ms), exceeding the 20ms threshold perceivable by humans. Furthermore, despite the previous state-of-the-art having more advanced sensors, we achieved relatively comparable accuracy (10.5° vs. 6.7° RMSE) with unsupervised learning. Our controller also demonstrated robust performance across a wide range of movement speeds. This study serves as a proof-of-concept for real-time neural control of robot actuators using a myoelectric interface.

Toronto Robotics Conference, 2024

Environment sensing and recognition can allow robotic systems to dynamically adapt to different w... more Environment sensing and recognition can allow robotic systems to dynamically adapt to different walking terrains. However, fast and accurate visual perception is very challenging, especially on embedded devices with limited computational resources. Most egocentric vision systems like the Project Aria smart glasses by Meta have been limited to off-device inferencing via external machines and cloud computing, which motivated our design of an integrated system with embodied intelligence. Our smart glasses included a Raspberry Pi Pico microcontroller and ArduCam camera. We designed a semi-permanent mounting system to allow our mechatronics to be transferable to a wide range of commercial eyeglass frames. We 3D-printed mounting brackets for the camera and microcontroller.

Toronto Robotics Conference, 2024

Table 1: Average classification accuracy across all activities and subjects for each deep neural ... more Table 1: Average classification accuracy across all activities and subjects for each deep neural network. Deep neural network Classification accuracy Transformer 83% Stacked LSTM 77% CNN + BiLSTM 76% CNN + LSTM 74% ConvLSTM 74% LSTM 71% BiLSTM 69% The transformer network achieved the highest performance with 83% accuracy across all 101 subjects. We then fine-tuned the model using data from each of the five datasets, resulting in classification accuracies of 80%, 85%, 92%, 99%, and 94%, on each dataset, respectively.

Toronto Robotics Conference, 2024

BioMedical Engineering OnLine, 2024

Journal of Computational and Nonlinear Dynamics, 2023

Frontiers in Neurorobotics, 2022

IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), 2024

IEEE International Conference on Robotics and Automation (ICRA), 2023

2022 9th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), 2022

2022 International Conference on Rehabilitation Robotics (ICORR), 2022

Toronto Robotics Conference, 2024

Table 1: Average classification accuracy across all six locomotor activities and 101 subjects for... more Table 1: Average classification accuracy across all six locomotor activities and 101 subjects for each of the seven deep neural networks. The 3D-CNN MoViNet network outperformed all of the 2D-CNN encoders with LSTM backbones and our 2D-CNN baseline model in terms of accuracy, suggesting that the network architecture can play an important role in performance beyond merely the incorporation of temporal information. We also studied the impact of sequence outputs (e.g., many-tomany training). Our results showed an increase in performance using many-to-one classification.

Toronto Robotics Conference, 2024

bioRxiv, 2024

Here we present our development of a novel real-time neural controller based on an EMG-driven mus... more Here we present our development of a novel real-time neural controller based on an EMG-driven musculoskeletal model, designed for volitional control of robots and computers. Our controller uniquely enables motion control during both isometric and non-isometric muscle contractions. We address several key challenges in EMG control system design, including accuracy, latency, and robustness. Our approach combines EMG signal processing, neural activation dynamics, and Hill-type muscle modeling to translate neural commands into muscle forces, which can enhance robustness against electrode variability and signal noise. Additionally, we integrate muscle activation dynamics with impedance control, inspired by the human motor control system, for smooth and adaptive interactions. As an initial proof of concept, we demonstrated that our system could control a robot actuator across a range of movements, both static and dynamic, and at different operating speeds, achieving high reference tracking performance and state-of-the-art processing times of 2.9 ms, important for real-time embedded computing. This research helps lay the groundwork for next-generation neural-machine interfaces that are fast, accurate, and adaptable to diverse users and control applications.

bioRxiv, 2024

Accurate neural decoding of brain dynamics remains a significant and open challenge in brain-mach... more Accurate neural decoding of brain dynamics remains a significant and open challenge in brain-machine interfaces. While various signal processing, feature extraction, and classification algorithms have been proposed, a systematic comparison of these is lacking. Accordingly, here we conducted one of the largest comparative studies evaluating different combinations of state-of-the-art algorithms for motor neural decoding to find the optimal combination. We studied three signal processing methods (i.e., artifact subspace reconstruction, surface Laplacian filtering, and data normalization), four feature extractors (i.e., common spatial patterns, independent component analysis, short-time Fourier transform, and no feature extraction), and four machine learning classifiers (i.e., support vector machine, linear discriminant analysis, convolutional neural networks, and long short-term memory networks). Using a large-scale EEG dataset, we optimized each combination for individual subjects (i.e., resulting in 672 total experiments) and evaluated performance based on classification accuracy. We also compared the computational and memory storage requirements, which are important for real-time embedded computing. Our comparative analysis provides novel insights that help inform the design of next-generation neural decoding algorithms for brain-machine interfaces used to interact with and control robots and computers.

bioRxiv, 2024

Large language models can provide a more detailed contextual understanding of a scene beyond what... more Large language models can provide a more detailed contextual understanding of a scene beyond what computer vision alone can provide, which have implications for robotics and embodied intelligence. In this study, we developed a novel multimodal vision-language system for egocentric visual perception, with an initial focus on real-world walking environments. We trained a number of state-of-the-art transformer-based vision-language models that use causal language modelling on our custom dataset of 43,055 image-text pairs for few-shot image captioning. We then designed a new speech synthesis model and a user interface to convert the generated image captions into speech for audio feedback to users. Our system also uniquely allows for feedforward user prompts to personalize the generated image captions. Our system is able to generate detailed captions with an average length of 10 words while achieving a high ROUGE-L score of 43.9% and a low word error rate of 28.1% with an end-to-end processing time of 2.2 seconds. Overall, our new multimodal vision-language system can generate accurate and detailed descriptions of natural scenes, which can be further augmented by user prompts. This innovative feature allows our image captions to be personalized to the individual and immediate needs and preferences of the user, thus optimizing the closed-loop interactions between the human and generative AI models for understanding and navigating of real-world environments.

bioRxiv, 2023

Control of robotic leg prostheses and exoskeletons is an open challenge. Computer modeling and si... more Control of robotic leg prostheses and exoskeletons is an open challenge. Computer modeling and simulation can be used to study the dynamics and control of human walking and extract principles that can be programmed into robotic legs to behave similar to biological legs. In this study, we present the development of an efficient two-layer Q-learning algorithm, with k-d trees, that operates over continuous action spaces and a reward model that estimates the degree of muscle activation similarity between the agent and human state-to-action pairs and state-to-action sequences. We used a human musculoskeletal model acting in a high-dimensional, physics-based simulation environment to train and evaluate our algorithm to simulate biomimetic walking. We used imitation learning and artificial biomechanics data to accelerate training via expert demonstrations and used experimental human data to compare and validate our predictive simulations, achieving 79% accuracy. Also, when compared to the previous state-of-the-art that used deep deterministic policy gradient, our algorithm was significantly more efficient with lower computational and memory storage requirements (i.e., requiring 7 times less RAM and 87 times less CPU compute), which can benefit real-time embedded computing. Overall, our new two-layer Q-learning algorithm using sequential data for continuous imitation of human locomotion serves as a first step towards the development of bioinspired controllers for robotic prosthetic legs and exoskeletons. Future work will focus on improving the prediction accuracy compared to experimental data and expanding our simulations to other locomotor activities.

bioRxiv, 2023

Robotic prosthetic legs and exoskeletons require real-time and accurate estimation of the walking... more Robotic prosthetic legs and exoskeletons require real-time and accurate estimation of the walking environment for smooth transitions between different locomotion mode controllers. However, previous studies have mainly been limited to static image classification, therein ignoring the temporal dynamics of human-robot locomotion. Motivated by these limitations, here we developed several state-of-the-art temporal convolutional neural networks (CNNs) to compare the performances between static vs. sequential image classification of real-world walking environments (i.e., level-ground terrain, incline stairs, and transitions to and from stairs). Using our large-scale image dataset, we trained a number of encoder networks such as VGG, MobileNetV2, ViT, and MobileViT, each coupled with a temporal long short-term memory (LSTM) backbone. We also trained MoViNet, a new video classification model designed for mobile and embedded devices, to further compare the performances between 2D and 3D temporal deep learning models. Our 3D network outperformed all the hybrid 2D encoders with LSTM backbones and the 2D CNN baseline model in terms of classification accuracy, suggesting that network architecture can play an important role in performance. However, although our 3D neural network achieved the highest classification accuracy, it had disproportionally higher computational and memory storage requirements, which can be disadvantageous for real-time control of robotic leg prostheses and exoskeletons with limited onboard resources.

arxiv, 2023

Human-robot walking with prosthetic legs and exoskeletons, especially over complex terrains such ... more Human-robot walking with prosthetic legs and exoskeletons, especially over complex terrains such as stairs, remains a significant challenge. Egocentric vision has the unique potential to detect the walking environment prior to physical interactions, which can improve transitions to and from stairs. This motivated us to create the StairNet initiative to support the development of new deep learning models for visual sensing and recognition of stairs, with an emphasis on lightweight and efficient neural networks for onboard real-time inference. In this study, we present an overview of the development of our large-scale dataset with over 515,000 manually labeled images, as well as our development of different deep learning models (e.g., 2D and 3D CNN, hybrid CNN and LSTM, and ViT networks) and training methods (e.g., supervised learning with temporal data and semi-supervised learning with unlabeled images) using our new dataset. We consistently achieved high classification accuracy (i.e., up to 98.8%) with different designs, offering trade-offs between model accuracy and size. When deployed on mobile devices with GPU and NPU accelerators, our deep learning models achieved inference speeds up to 2.8 ms. We also deployed our models on custom-designed CPU-powered smart glasses. However, limitations in the embedded hardware yielded slower inference speeds of 1.5 seconds, presenting a trade-off between human-centered design and performance. Overall, we showed that StairNet can be an effective platform to develop and study new visual perception systems for human-robot locomotion with applications in exoskeleton and prosthetic leg control.

bioRxiv, 2023

Environment sensing and recognition can allow humans and robots to dynamically adapt to different... more Environment sensing and recognition can allow humans and robots to dynamically adapt to different walking terrains. However, fast and accurate visual perception is challenging, especially on embedded devices with limited computational resources. The purpose of this study was to develop a novel pair of AI-powered smart glasses for onboard sensing and recognition of human-robot walking environments with high accuracy and low latency. We used a Raspberry Pi Pico microcontroller and an ArduCam HM0360 low-power camera, both of which interface with the eyeglass frames using 3D-printed mounts that we custom-designed. We trained and optimized a lightweight and efficient convolutional neural network using a MobileNetV1 backbone to classify the walking terrain as either indoor surfaces, outdoor surfaces (grass and dirt), or outdoor surfaces (paved) using over 62,500 egocentric images that we adapted and manually labelled from the Meta Ego4D dataset. We then compiled and deployed our deep learning model using TensorFlow Lite Micro and post-training quantization to create a minimized byte array model of size 0.31MB. Our system was able to accurately predict complex walking environments with 93.6% classification accuracy and had an embedded inference speed of 1.5 seconds during online experiments using the integrated camera and microcontroller. Our AI-powered smart glasses open new opportunities for visual perception of human-robot walking environments where embedded inference and a low form factor is required. Future research will focus on improving the onboard inference speed and miniaturization of the mechatronic components.

bioRxiv, 2023

Convolutional neural networks trained using supervised learning can improve visual perception for... more Convolutional neural networks trained using supervised learning can improve visual perception for human-robot walking. These advances have been possible due to large-scale datasets like ExoNet and StairNet - the largest open-source image datasets of real-world walking environments. However, these datasets require vast amounts of manually annotated data, the development of which is time consuming and labor intensive. Here we present a novel semi-supervised learning system (ExoNet-SSL) that uses over 1.2 million unlabelled images from ExoNet to improve training efficiency. We developed a deep learning model based on mobile vision transformers and trained the model using semi-supervised learning for image classification. Compared to standard supervised learning (98.4%), our ExoNet-SSL system was able to maintain high prediction accuracy (98.8%) when tested on previously unseen environments, while requiring 35% fewer labelled images during training. These results show that semi-supervised learning can improve training efficiency by leveraging large amounts of unlabelled data and minimize the size requirements for manually annotated images. Future research will focus on model deployment for onboard real-time inference and control of human-robot walking.

bioRxiv, 2023

Robotic exoskeletons can provide powered locomotor assistance and rehabilitation to persons with ... more Robotic exoskeletons can provide powered locomotor assistance and rehabilitation to persons with mobility impairments due to aging and/or physical disabilities. Here we present the preliminary development and systems integration of T-BLUE - a modular, bilateral robotic hip-knee exoskeleton with 3D-printed backdriveable actuators. We retrofitted commercially available passive postoperative orthoses with open-source 3D-printed actuators to minimize cost and improve accessibility. The actuators are of quasi-direct drive design with high-torque density brushless DC motors and low gearing (15:1 transmission ratio) for low output impedance and high backdrivability, therein allowing for energy-efficient and dynamic human-robot physical interaction and legged locomotion. The modular design allows the exoskeleton to be customized and adapted to different users (e.g., persons with lateral vs. bilateral mobility impairments) and different hip-knee joint configurations. The goals of this preliminary study were to describe our experience with regards to the repeatability of the open-source 3D-printed actuators in engineering practice and the feasibility of integrating the actuators into wearable robotics hardware. This qualitative research serves as a first step towards using the robotic exoskeleton to support the development and testing of novel controller designs and rehabilitation protocols for different locomotor activities of daily living. We are especially interested in populations that could benefit from partial locomotor assistance such as older adults and/or persons with osteoarthritis. Future research will involve benchtop testing to quantitatively evaluate the actuator performance in terms of dynamics and energy-efficiency.

bioRxiv, 2023

Environment sensing and recognition can improve the safety and autonomy of human-robot locomotion... more Environment sensing and recognition can improve the safety and autonomy of human-robot locomotion, especially during transitions between environmental states such as walking to and from stairs. However, accurate and real-time perception on edge devices with limited computational resources is an open problem. Here we present the development and mobile deployment of StairNet - a vision-based automated stair recognition system powered by deep learning. Building on ExoNet, the largest open-source dataset of egocentric images of real-world walking environments, we designed a new dataset specifically for stair recognition with over 515,000 images. We then developed a lightweight and efficient convolutional neural network for image classification, which accurately predicted complex stair environments with 98.4% accuracy. We also studied different model compression and optimization methods and deployed our system on several mobile devices running a custom-designed iOS application with onboard accelerators using CPU, GPU, and/or NPU backend computing. Of the designs that we tested, our highest performing system showed negligible reductions in classification accuracy due to the model conversion for mobile deployment and achieved an inference time of 2.75 ms on an iPhone 11. The high speed and accuracy of the StairNet system on edge devices opens new opportunities for autonomous control and planning of robotic prosthetic legs, exoskeletons, and other assistive technologies for human locomotion

bioRxiv, 2023

bioRxiv, 2022

Computer vision can be used in robotic exoskeleton control to improve transitions between differe... more Computer vision can be used in robotic exoskeleton control to improve transitions between different locomotion modes through the prediction of future environmental states. Here we present the development of a large-scale automated stair recognition system powered by convolutional neural networks to recognize indoor and outdoor real-world stair environments. Building on the ExoNet database – the largest and most diverse open-source dataset of wearable camera images of walking environments – we designed a new computer vision dataset, called StairNet, specifically for stair recognition with over 515,000 images. We then developed and optimized an efficient deep learning model for automatic feature engineering and image classification. Our system was able to accurately predict complex stair environments with 98.4% classification accuracy. These promising results present an opportunity to increase the autonomy and safety of human-exoskeleton locomotion for real-world community mobility. Future work will explore the mobile deployment of our automated stair recognition system for onboard real-time inference.

bioRxiv, 2021

Robotic leg prostheses and exoskeletons can provide powered locomotor assistance to older adults ... more Robotic leg prostheses and exoskeletons can provide powered locomotor assistance to older adults and/or persons with physical disabilities. However, the current locomotion mode recognition systems being developed for intelligent high-level control and decision-making use mechanical, inertial, and/or neuromuscular data, which inherently have limited prediction horizons (i.e., analogous to walking blindfolded). Inspired by the human vision-locomotor control system, we designed and evaluated an advanced environment classification system that uses computer vision and deep learning to forward predict the oncoming walking environments prior to physical interaction, therein allowing for more accurate and robust locomotion mode transitions. In this study, we first reviewed the development of the ExoNet database – the largest and most diverse open-source dataset of wearable camera images of indoor and outdoor real-world walking environments, which were annotated using a hierarchical labelling architecture. We then trained and tested over a dozen state-of-the-art deep convolutional neural networks (CNNs) on the ExoNet database for large-scale image classification of the walking environments, including: EfficientNetB0, InceptionV3, MobileNet, MobileNetV2, VGG16, VGG19, Xception, ResNet50, ResNet101, ResNet152, DenseNet121, DenseNet169, and DenseNet201. Lastly, we quantitatively compared the benchmarked CNN architectures and their environment classification predictions using an operational metric called NetScore, which balances the image classification accuracy with the computational and memory storage requirements (i.e., important for onboard real-time inference). Although we designed this environment classification system to support the development of next-generation environment-adaptive locomotor control systems for robotic prostheses and exoskeletons, applications could extend to humanoids, autonomous legged robots, powered wheelchairs, and assistive devices for persons with visual impairments.