Thomas Eiszler
Wearable cognitive assistance applications are a new class of applications that combine computer vision and edge computing and are designed to aid in the completion of complex or involved tasks. At Carnegie Mellon University, we have developed the Gabriel platform for building wearable cognitive assistance applications. Below are some of the various applications we have created including links to the source code, docker containers encapsulating the backends, and demonstration videos of the assistants.
Gabriel Platform
The Gabriel repository contains the core backend code as well as implementations for several clients, including an Android client (available on the PlayStore), a Python client, and Microsoft Hololens client.
Lego
Lego was the very first wearable cognitive assistant ever developed. It uses a special backboard to detect the color/shape of various Lego pieces and helps the user to construct a specific Lego kit.
Sandwich
The idea behind sandwich was to develop a wearable cognitive assistant to help with cooking. The application gives instructions to the user on the order in which they should assemble the components of the sandwich. For this demo, a Microsoft Hololens client was also created which shows a holographic representation where the next item needs to be placed.
Ikea
A universally understood challenge is building furniture. To address this challenge, a wearable cognitive assistant was designed for an Ikea lamp. Through short video clips and audible instructions, users can quickly put together the lamp.
DiskTray
In cooperation with inWin , we developed an assistant to aid in the assembly of a hard disk tray prior to incorporating it into the computer chassis. Instead of reading written instructions, the user wears a pair of smart glasses (ODG-R7) and is given step by step video and audio guidance.
RibLoc
RibLoc is a Gabriel application designed to assist for surgical repair of ribs. The RibLoc system is made by AcuteInnovations, Inc . Today, this training is given to a doctor by an on-site technician. This wearable cognitive assistant illustrates how this training could be delivered more efficiently.