Human-Machine Collaborative Systems

Professor Gregory Hager of the Johns Hopkins University worked as a Hans Fischer Senior Fellow in the Focus Group Human-Machine Collaborative Systems hosted by TUM professor Nassir Navab.

The mediation of human activity through devices means that it is possible to gather data and develop human activity models with a scale and complexity that was heretofore not possible. For example, there are now nearly 500,000 robotic surgeries performed per year with the da Vinci surgical robot. Prior work of Professor Gregory Hager’s group at The Johns Hopkins University (JHU) and by Professor Nassir Navab’s group at TUM suggest that, in principle, it is possible to record and process each of these to assess both what was done and how well it was done. This anticipates a day when patient outcomes can be quantitatively related back to surgical performance and individualized surgical training can be optimized to improve the practice of surgery.

It is precisely the convergence of intelligent machines, humans, and quantitative performance data in medicine that we intend to focus on during Professor Hager’s stay at TUM-IAS. With the support of a TUM-IAS Hans Fischer Senior Fellowship, we propose to explore the modeling of data on human task performance as a path toward creating intelligent assistants that will enhance human abilities through individualized training and augmentation. We will focus our work on applications in surgery and related interventional care that demand high technical and cognitive skills. TUM and JHU are unique in that they are two of very few research institutions in the world that contains the concentration of expertise and resources to advance these ideas. Our long-term goal is to use the Hans Senior Fischer Fellowship as “seed funding” to develop a sustainable international collaborative center of excellence devoted to the advancement of medical technologies, and specifically medical devices and systems.

TUM-IAS funded doctoral candidate:
Christian Rupprecht, Computer Aided Medical Procedures & Augmented Reality

Publications by the Focus Group

2017

  • Coskun, Huseyin; Achilles, Felix; DiPietro, Robert; Navab, Nassir; Tombari, Federico: Long Short-Term Memory Kalman Filters: Recurrent Neural Estimators for Pose Regularization. 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, 2017 mehr…
  • Krishnan, Sanjay; Garg, Animesh; Patil, Sachin; Lea, Colin; Hager, Gregory; Abbeel, Pieter; Goldberg, Ken: Transition State Clustering: Unsupervised Surgical Trajectory Segmentation for Robot Learning. In: Springer Proceedings in Advanced Robotics. Springer International Publishing, 2017 mehr…
  • Rupprecht, Christian; Laina, Iro; DiPietro, Robert; Baust, Maximilian: Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypotheses. 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, 2017 mehr…

2016

  • Chi Li; Bohren, Jonathan; Carlson, Eric; Hager, Gregory D.: Hierarchical semantic parsing for object pose estimation in densely cluttered scenes. 2016 IEEE International Conference on Robotics and Automation (ICRA), Institute of Electrical and Electronics Engineers (IEEE), 2016 mehr…
  • Chi Li; Bohren, Jonathan; Carlson, Eric; Hager, Gregory D.: Hierarchical semantic parsing for object pose estimation in densely cluttered scenes. 2016 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2016 mehr…
  • DiPietro, Robert; Lea, Colin; Malpani, Anand; Ahmidi, Narges; Vedula, S. Swaroop; Lee, Gyusung I.; Lee, Mija R.; Hager, Gregory D.: Recognizing Surgical Activities with Recurrent Neural Networks. In: Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016. Springer International Publishing, 2016 mehr…
  • Lea, Colin; Vidal, Rene; Hager, Gregory D.: Learning convolutional action primitives for fine-grained action recognition. 2016 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2016 mehr…
  • Rupprecht, Christian; Lea, Colin; Tombari, Federico; Navab, Nassir; Hager, Gregory D.: Sensor substitution for video-based action recognition. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2016 mehr…

2015

  • Belagiannis, Vasileios; Rupprecht, Christian; Carneiro, Gustavo; Navab, Nassir: Robust Optimization for Deep Regression. 2015 IEEE International Conference on Computer Vision (ICCV), IEEE, 2015 mehr…
  • Ghalamzan E., Amir M.; Paxton, Chris; Hager, Gregory D.; Bascetta, Luca: An incremental approach to learning generalizable robot tasks from human demonstration. 2015 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2015 mehr…
  • Lea, Colin; Hager, Gregory D.; Vidal, Rene: An Improved Model for Segmentation and Recognition of Fine-Grained Activities with Application to Surgical Training Tasks. 2015 IEEE Winter Conference on Applications of Computer Vision, IEEE, 2015 mehr…
  • Rupprecht, Christian; Peter, Loic; Navab, Nassir: Image segmentation in Twenty Questions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2015 mehr…