Raphael Memmesheimer

I am a Postdoctoral Researcher at the Autonomous Intelligent Systems (AIS) group with Prof. Dr. Sven Behnke at the University of Bonn, where I re-initiated and lead the NimbRo@Home team. Our focus is on developing domestic service robots that integrate perception, human-robot interaction, task-planning and mobile manipulation to assist in real-world environments. My long-term research goal is to develop autonomous service robots that learn from human behavior using multi-modal sensory information and deploy them in real-world environments.

I completed my PhD in the Active Vision Group under Prof. Dr. Dietrich Paulus at the University of Koblenz. My thesis explored advanced techniques for on multi-modal action recognition in time-series data, including videos, inertial measurement units (IMUs), Wi-Fi CSI fingerprints, and skeleton sequences. By combining these modalities, I aim to enable robots to understand and respond to human activities more robustly and naturally.

I have a proven track record of leading award-winning robotics teams in international competitions. As a team leader, I have guided teams at different universities to multiple championship titles, including:

• RoboCup@Home World Championship (five-time winner, with three consecutive victories).
• Top placements in the European Robotics League and World Robot Summit.

I further contributed to the NimbRo Avatar team, winning the ANA Avatar XPrize challenge with a prize award of 5 million USD.

My experience spans designing, implementing, and optimizing robotic systems for real-world challenges, with a focus on autonomy, human-robot interaction, and multi-modal perception. These competitions have honed my ability to innovate under pressure, and push the boundaries of what service robots can achieve.

Contact

University of Bonn
Autonomous Intelligent Systems
Friedrich-Hirzebruch-Allee 8
53115 Bonn
Room 0.055
Phone: +49 228-73-60805


Travel direction

News

• 07/2025: We ended up second at RoboCup World Champtionship in Salvador (Brazil). More information here.
• 03/2025: Our team NimbRo won the RoboCup German Open 2025 in the @Home League. Here you find more information.
• 12/2024: The scientific magazine Bild der Wissenschaft wrote an article about our robot competition activities.
• 09/2024: We were featured by the local broadcasting channel WDR in their news Lokalzeit.
• 07/2024: We won the RoboCup@Home World Championship in the Open Platform League! Find more Info here.
• 07/2024: Open Vocabulary 6D pose estimation project funded.
• 04/2024: Won the RoboCup German Open in the @Home League.

Links

• Google Scholar
• Youtube
• Github

Content

• Publications
• Talks
• Awards

Publications

2026

• 
  OMCL: Open-vocabulary Monte Carlo Localization
  Kruzhkov, Evgenii; Memmesheimer, Raphael; Behnke, Sven
  IEEE Robotics and Automation Letters (RAL) (Accepted for Publication) (2026)
  [url]
• 
  EL3DD: Extended Latent 3D Diffusion for Language Conditioned Multitask Manipulation
  Bode, Jonas; Memmesheimer, Raphael; Behnke, Sven
  Accepted for European Robotics Forum (ERF), Stavanger, Norway, March 2026 (2026)
  
• 
  Anticipating Human Behavior for Safe Navigation and Efficient Collaborative Manipulation with Mobile Service Robots
  Bultmann, Simon; Memmesheimer, Raphael; Nogga, Jan; Hau, Julian; Behnke, Sven
  IEEE/SICE International Symposium on System Integration, SII 2026 (Accepted for publication) (2026)
  

2025

• 
  Integration of the TIAGo Robot into Isaac Sim with Mecanum Drive Modeling and Learned S-Curve Velocity Profiles
  Schönbach, Vincent; Wiedemann, Marvin; Memmesheimer, Raphael; Mosbach, Malte; Behnke, Sven
  2025 IEEE 21st International Conference on Automation Science and Engineering (CASE) IEEE (2025)
  [url]
• 
  LIAM: Multimodal Transformer for Language Instructions, Images, Actions and Semantic Maps
  Wang, Yihao; Memmesheimer, Raphael; Behnke, Sven
  19th International Conference on Intelligent Autonomous Systems (IAS) (Accepted for publication) (2025)
  
• 
  Anticipating Human Behavior for Safe and Efficient Collaborative Mobile Manipulation
  Bultmann, Simon; Memmesheimer, Raphael; Nogga, Jan; Hau, Julian; Behnke, Sven
  Proceedings of the 1st German Robotics Conference (GRC) (Accepted for publication) (2025)
  
• 
  Person Segmentation and Action Classification for Multi-Channel Hemisphere Field of View LiDAR Sensors
  Seliunina, Svetlana; Otelepko, Artem; Memmesheimer, Raphael; Behnke, Sven
  IEEE/SICE International Symposium on System Integration, SII 2025, Munich, Germany, January 21-24, 2025 IEEE (2025)
  [url]
• 
  NimbRo Wins ANA Avatar XPRIZE Immersive Telepresence Competition: Human-Centric Evaluation and Lessons Learned
  Lenz, Christian; Schwarz, Max; Rochow, Andre; Paetzold, Bastian; Memmesheimer, Raphael; Schreiber, Michael; Behnke, Sven
  Int. J. Soc. Robotics (2025)
  [url]

2024

• 
  RoboCup@Home 2024 OPL Winner NimbRo: Anthropomorphic Service Robots Using Foundation Models for Perception and Planning
  Memmesheimer, Raphael; Nogga, Jan; Paetzold, Bastian; Kruzhkov, Evgenii; Bultmann, Simon; Schreiber, Michael; Bode, Jonas; Karacora, Bertan; Park, Juhui; Savinykh, Alena; Behnke, Sven
  RoboCup 2024: Robot World Cup XXVII Springer (2024)
  [url]
• 
  A Comparison of Prompt Engineering Techniques for Task Planning and Execution in Service Robotics
  Bode, Jonas; Paetzold, Bastian; Memmesheimer, Raphael; Behnke, Sven
  23rd IEEE-RAS International Conference on Humanoid Robots, Humanoids 2024, Nancy, France, November 22-24, 2024 IEEE (2024)
  [url]
• 
  Self-centering 3-DOF feet controller for hands-free locomotion control in telepresence and virtual reality
  Memmesheimer, Raphael; Lenz, Christian; Schwarz, Max; Schreiber, Michael; Behnke, Sven
  2024 IEEE Conference on Telepresence (2024)
  [url]
• 
  Cleaning Robots in Public Spaces: A Survey and Proposal for Benchmarking Based on Stakeholders Interviews
  Memmesheimer, Raphael; Overbeck, Martina; Kral, Björn; Steffen, Lea; Behnke, Sven; Gersch, Martin; Roennau, Arne
  RoboCup 2024: Robot World Cup XXVII Springer (2024)
  [url]

2023

• 
  Audio-Based Roughness Sensing and Tactile Feedback for Haptic Perception in Telepresence
  Paetzold, Bastian; Rochow, Andre; Schreiber, Michael; Memmesheimer, Raphael; Lenz, Christian; Schwarz, Max; Behnke, Sven
  IEEE International Conference on Systems, Man, and Cybernetics, SMC 2023, Honolulu, Oahu, HI, USA, October 1-4, 2023 IEEE (2023)
  [url]
• 
  Robust Immersive Telepresence and Mobile Telemanipulation: NimbRo wins ANA Avatar XPRIZE Finals
  Schwarz, Max; Lenz, Christian; Memmesheimer, Raphael; Paetzold, Bastian; Rochow, Andre; Schreiber, Michael; Behnke, Sven
  22nd IEEE-RAS International Conference on Humanoid Robots, Humanoids 2023, Austin, TX, USA, December 12-14, 2023 IEEE (2023)
  [url]
• 
  External Camera-Based Mobile Robot Pose Estimation for Collaborative Perception with Smart Edge Sensors
  Bultmann, Simon; Memmesheimer, Raphael; Behnke, Sven
  IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023 IEEE (2023)
  [video][arxiv][url]
• 
  RoboCup 2022 AdultSize Winner NimbRo: Upgraded Perception, Capture Steps Gait and Phase-Based In-Walk Kicks
  Pavlichenko, Dmytro; Ficht, Grzegorz; Amini, Arash; Hosseini, Mojtaba; Memmesheimer, Raphael; Villar-Corrales, Angel; Schulz, Stefan M.; Missura, Marcell; Bennewitz, Maren; Behnke, Sven
  RoboCup 2022: Robot World Cup XXV [Bangkok, Thailand, July 11-17, 2022] Springer (2023)
  [video][arxiv][url]
• 
  On the recognition of human activities and the evaluation of its imitation by robotic systems
  Memmesheimer, Raphael
  (2023)
  PhD thesis[url]
• 
  Reduced Skeleton Representation for Action Recognition on Graph Convolutional Neural Networks
  Germann, Ida; Memmesheimer, Raphael; Paulus, Dietrich
  IEEE/SICE International Symposium on System Integration, SII 2023, Atlanta, GA, USA, January 17-20, 2023 IEEE (2023)
  [url]
• 
  Classification of pathological and healthy individuals for computer-aided physical rehabilitation
  Kramer, Ivanna; Memmesheimer, Raphael; Paulus, Dietrich
  IEEE/SICE International Symposium on System Integration, SII 2023, Atlanta, GA, USA, January 17-20, 2023 IEEE (2023)
  [url]

2022

• 
  Data augmentation for training a neural network for image reconstruction in MPI
  von Gladiss, Anselm; Kramer, Ivanna; Theisen, Nick; Memmesheimer, Raphael; Bakenecker, Anna C.; Buzug, Thorsten M.; Paulus, Dietrich
  International Workshop on Magnetic Particle Imaging (2022)
  [url]
• 
  Reconstruction of 1D Images with a Neural Network for Magnetic Particle Imaging
  von Gladiss, Anselm; Memmesheimer, Raphael; Theisen, Nick; Bakenecker, Anna C.; Buzug, Thorsten M.; Paulus, Dietrich
  Bildverarbeitung f\"ur die Medizin 2022 - Proceedings, German Workshop on Medical Image Computing, Heidelberg, June 26-28, 2022 Springer (2022)
  [url]
• 
  Skeleton-DML: Deep Metric Learning for Skeleton-Based One-Shot Action Recognition
  Memmesheimer, Raphael; Haering, Simon; Theisen, Nick; Paulus, Dietrich
  IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022 IEEE (2022)
  [arxiv][video][github][url]

2021

• 
  Fusion-GCN: Multimodal Action Recognition Using Graph Convolutional Networks
  Duhme, Michael; Memmesheimer, Raphael; Paulus, Dietrich
  Pattern Recognition - 43rd DAGM German Conference, DAGM GCPR 2021, Bonn, Germany, September 28 - October 1, 2021, Proceedings Springer (2021)

  Abstract:

  In this paper we present Fusion-GCN, an approach for multimodal action recognition using Graph Convolutional Network (GCNs). Action recognition methods based around Graph Convolutional Network (GCNs) recently yielded state-of-the-art performance for skeleton-based action recognition. With Fusion-GCN, we propose to integrate various sensor data modalities into a graph that is trained using a GCN model for multi-modal action recognition. Additional sensor measurements are incorporated into the graph representation either on a channel dimension (introducing additional node attributes) or spatial dimension (introducing new nodes). Fusion-GCN was evaluated on two publicly available datasets, the UTD-MHAD- and MMACT datasets, and demonstrates flexible fusion of RGB sequences, inertial measurements and skeleton sequences. Our approach gets comparable results on the UTD-MHAD dataset and improves the baseline on the large-scale MMACT dataset by a significant margin of up to 12.37{\%} (F1-Measure) with the fusion of skeleton estimates and accelerometer measurements.

  
  [arxiv][video][github][presentation-video][url]
• 
  Action Segmentation on Representations of Skeleton Sequences Using Transformer Networks
  Haering, Simon; Memmesheimer, Raphael; Paulus, Dietrich
  2021 IEEE International Conference on Image Processing, ICIP 2021, Anchorage, AK, USA, September 19-22, 2021 IEEE (2021)

  Abstract:

  We propose an approach for action segmentation by representing motions as images. A transformer object detection network is used to segment the sequences from the representation images. We examine different encoding approaches, normalization strategies and skeleton joint orders in an extensive experiment study. Our approach is evaluated on skeleton sequences from the PKU-MMD dataset. We successfully apply transformer networks for action segmentation on skeleton sequences. Our proposed approach achieves high class accuracies, while start and end-time estimation of the action segments are subject to further improvement.

  
  [video][url]
• 
  Customer Interaction of a Future Convenience Store with a Mobile Manipulation Service Robot
  Kramer, Ivanna; Memmesheimer, Raphael; Paulus, Dietrich
  2021 IEEE International Conference on Omni-Layer Intelligent Systems, COINS 2021, Barcelona, Spain, August 23-25, 2021 IEEE (2021)

  Abstract:

  We present an approach on how to integrate mobile manipulation service robots to support customer interaction of the future convenience store. Customers are identified by their gestures and served e.g., by carrying a shopping bag, guiding the customer to shelves of interest and bringing coffee. An integration into a widely spread telecommunication application is proposed for invoicing payment and further serves as an interface to the customer. Statistics and store layouts can be requested in the application as well as an extendable list of services provided by the robot. Further, a smart shopping bag is proposed which recognizes products stored into it by their barcode. This bag handles the invoicing and allows communication to i.e. the messenger app integration. Furthermore, our proposed system gives hints in how the customers can be profiled by their encoded faces as observed by the robot. We also warn about the possibilities of creating customer profiles without being transparent for the customers. The presented task was ranked 3rd for the Future Convenience Store Challenge at the World Robot Summit 2018.

  
  [url]
• 
  Mobile Manipulation Hackathon: Moving into Real World Applications
  Roa, M\'aximo A.; Dogar, Mehmet Remzi; Pag\`es, Jordi; Vivas, Carlos; Morales, Antonio; Correll, Nikolaus; Görner, Michael; Rosell, Jan; Foix, Sergi; Memmesheimer, Raphael; Ferro, Francesco
  IEEE Robotics Autom. Mag. (2021)

  Abstract:

  The Mobile Manipulation Hackathon was held in late 2018 during the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) to showcase the latest applications of wheeled robotic manipulators. The challenge had an open format, where teams developed an application using simulation tools and integrated it into a robotic platform. This article presents the competition and analyzes the results, with information gathered during the event and from a survey circulated among the finalist teams. We provide an overview of the mobile manipulation field, identify key areas required for further development to facilitate the implementation of mobile manipulators in real applications, and discuss ideas about how to structure future hackathon-style competitions to enhance their impact on the scientific and industrial communities.

  
  [url]
• 
  Next-Best-View Estimation based on Deep Reinforcement Learning for Active Object Classification
  Korbach, Christian; Solbach, Markus D; Memmesheimer, Raphael; Paulus, Dietrich; Tsotsos, John K
  arXiv preprint arXiv:2110.06766 (2021)
  

2020

• 
  SL-DML: Signal Level Deep Metric Learning for Multimodal One-Shot Action Recognition
  Memmesheimer, Raphael; Theisen, Nick; Paulus, Dietrich
  25th International Conference on Pattern Recognition, ICPR 2020, Virtual Event / Milan, Italy, January 10-15, 2021 IEEE (2020)

  Abstract:

  Recognizing an activity with a single reference sample using metric learning approaches is a promising research field. The majority of few-shot methods focus on object recognition or face-identification. We propose a metric learning approach to reduce the action recognition problem to a nearest neighbor search in embedding space. We encode signals into images and extract features using a deep residual CNN. Using triplet loss, we learn a feature embedding. The resulting encoder transforms features into an embedding space in which closer distances encode similar actions while higher distances encode different actions. Our approach is based on a signal level formulation and remains flexible across a variety of modalities. It further outperforms the baseline on the large scale NTU RGB+D 120 dataset for the One-Shot action recognition protocol by 5.6\%. With just 60\% of the training data, our approach still outperforms the baseline approach by 3.7\%. With 40\% of the training data, our approach performs comparably well to the second follow up. Further, we show that our approach generalizes well in experiments on the UTD-MHAD dataset for inertial, skeleton and fused data and the Simitate dataset for motion capturing data. Furthermore, our inter-joint and inter-sensor experiments suggest good capabilities on previously unseen setups.

  
  [arxiv][video][url]
• 
  Gimme Signals: Discriminative signal encoding for multimodal activity recognition
  Memmesheimer, Raphael; Theisen, Nick; Paulus, Dietrich
  IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020, Las Vegas, NV, USA, October 24, 2020 - January 24, 2021 IEEE (2020)

  Abstract:

  We present a simple, yet effective and flexible method for action recognition supporting multiple sensor modalities. Multivariate signal sequences are encoded in an image and are then classified using a recently proposed EfficientNet CNN architecture. Our focus was to find an approach that generalizes well across different sensor modalities without specific adaptions while still achieving good results. We apply our method to 4 action recognition datasets containing skeleton sequences, inertial and motion capturing measurements as well as \wifi fingerprints that range up to 120 action classes. Our method defines the current best CNN-based approach on the NTU RGB+D 120 dataset, lifts the state of the art on the ARIL Wi-Fi dataset by +6.78\%, improves the UTD-MHAD inertial baseline by +14.4\%, the UTD-MHAD skeleton baseline by 1.13\% and achieves 96.11\% on the Simitate motion capturing data (80/20 split). We further demonstrate experiments on both, modality fusion on a signal level and signal reduction to prevent the representation from overloading.

  
  [arxiv][video][url]
• 
  Robotic Imitation by Markerless Visual Observation and Semantic Associations
  Memmesheimer, Raphael; Kramer, Ivanna; Seib, Viktor; Theisen, Nick; Paulus, Dietrich
  2020 IEEE International Conference on Autonomous Robot Systems and Competitions, ICARSC 2020, Ponta Delgada, Portugal, April 15-17, 2020 IEEE (2020)

  Abstract:

  In this paper we present an approach for learning to imitate human behavior on a semantic level by markerless visual observation. We analyze a set of spatial constraints on human pose data extracted using convolutional pose machines and object information extracted from 2D image sequences. A scene analysis, based on an ontology of objects and affordances, is combined with continuous human pose estimation and spatial object relations. Using a set of constraints we associate the observed human actions with a set of executable robot commands. We demonstrate our approach in a kitchen task, where the robot learns to prepare a meal.

  
  [url]

2019

• 
  Simitate: A Hybrid Imitation Learning Benchmark
  Memmesheimer, Raphael; Kramer, Ivanna; Seib, Viktor; Paulus, Dietrich
  2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019 IEEE (2019)

  Abstract:

  We present Simitate — a hybrid benchmarking suite targeting the evaluation of approaches for imitation learning. A dataset containing 1938 sequences where humans perform daily activities in a realistic environment is presented. The dataset is strongly coupled with an integration into a simulator. RGB and depth streams with a resolution of $960 times 540$ at 30Hz and accurate ground truth poses for the demonstrator’s hand, as well as the object in 6 DOF at 120Hz are provided. Along with our dataset we provide the 3D model of the used environment, labeled object images and pre-trained models. A benchmarking suite that aims at fostering comparability and reproducibility supports the development of imitation learning approaches. Further, we propose and integrate evaluation metrics on assessing the quality of effect and trajectory of the imitation performed in simulation. Simitate is available on our project website: https://agas.uni-koblenz.de/simitate/.

  
  [arxiv][video][project-page][url]
• 
  Scratchy: A Lightweight Modular Autonomous Robot for Robotic Competitions
  Memmesheimer, Raphael; Kuhlmann, Isabelle; Mints, Mark; Schmidt, Patrik; Korbach, Christian; Germann, Ida; Paulus, Dietrich
  2019 IEEE International Conference on Autonomous Robot Systems and Competitions, ICARSC 2019, Porto, Portugal, April 24-26, 2019 IEEE (2019)

  Abstract:

  We present Scratchy-a modular, lightweight robot built for low budget competition attendances. Its base is mainly built with standard 4040 aluminium profiles and the robot is driven by four mecanum wheels on brushless DC motors. In combination with a laser range finder we use estimated odometry - which is calculated by encoders - for creating maps using a particle filter. A RGB-D camera is utilized for object detection and pose estimation. Additionally, there is the option to use a 6-DOF arm to grip objects from an estimated pose or generally for manipulation tasks. The robot can be assembled in less than one hour and fits into two pieces of hand luggage or one bigger suitcase. Therefore, it provides a huge advantage for student teams that participate in robot competitions like the European Robotics League or RoboCup. Thus, this keeps the funding required for participation, which is often a big hurdle for student teams to overcome, low. The software and additional hardware descriptions are available under: https://github.com/homer-robotics/scratchy.

  
  [arxiv][github][url]
• 
  Adaptive Learning Methods for Autonomous Mobile Manipulation in RoboCup@Home
  Memmesheimer, Raphael; Seib, Viktor; Evers, Tobias; Müller, Daniel; Paulus, Dietrich
  RoboCup 2019: Robot World Cup XXIII [Sydney, NSW, Australia, July 8, 2019] Springer (2019)

  Abstract:

  Team homer@UniKoblenz has become an integral part of the RoboCup@Home community. As such we would like to share our experience gained during the competitions with new teams. In this paper we describe our approaches with a special focus on our demonstration of this year’s finals. This includes semantic exploration, adaptive programming by demonstration and touch enforcing manipulation. We believe that these demonstrations have a potential to influence the design of future RoboCup@Home tasks. We also present our current research efforts in benchmarking imitation learning tasks, gesture recognition and a low cost autonomous robot platform. Our software can be found on GitHub at https://github.com/homer- robotics.

  
  [url]
• 
  Evaluation Of Physical Therapy Through Analysis Of Depth Images
  Kramer, Ivanna; Schmidt, Niko; Memmesheimer, Raphael; Paulus, Dietrich
  28th IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2019, New Delhi, India, October 14-18, 2019 IEEE (2019)

  Abstract:

  The support through robots in orthopaedic rehabilitation is an opportunity to relieve physiotherapists. However, to be able to provide a control in the robot-patient cooperation in the therapy process a certain standard in interpreting the exercise has to be established. In this paper we present an evaluation approach of the health subject performance in a tibiofemoral rehabilitation on the example of squat exercises. The proposed method utilizes only depth images for the performance evaluation and any human- robot interaction system for the performance correction. Thus, this method can be easily applied to a mobile service robot in the robot-aided physical therapy. The patient is observed while performing the exercise and the motion is evaluated and segmented using Motion History Images. Concrete, depth images are used to monitor local points of interest on the performer during the exercise. The proposed approach was evaluated on custom image sequences with a multitude of varying subjects and shows the suitable performance for assisting in the correctness of the exercise execution.

  
  [url]
• 
  Gesture Recognition in RGB Videos Using Human Body Keypoints and Dynamic Time Warping
  Schneider, Pascal; Memmesheimer, Raphael; Kramer, Ivanna; Paulus, Dietrich
  RoboCup 2019: Robot World Cup XXIII [Sydney, NSW, Australia, July 8, 2019] Springer (2019)

  Abstract:

  Gesture recognition opens up new ways for humans to intuitively interact with machines. Especially for service robots, gestures can be a valuable addition to the means of communication to, for example, draw the robot’s attention to someone or something. Extracting a gesture from video data and classifying it is a challenging task and a variety of approaches have been proposed throughout the years. This paper presents a method for gesture recognition in RGB videos using OpenPose to extract the pose of a person and Dynamic Time Warping (DTW) in conjunction with One-Nearest- Neighbor (1NN) for time-series classification. The main features of this approach are the independence of any specific hardware and high flexibility, because new gestures can be added to the classifier by adding only a few examples of it. We utilize the robustness of the Deep Learning-based OpenPose framework while avoiding the data-intensive task of training a neural network ourselves. We demonstrate the classification performance of our method using a public dataset.

  
  [arxiv][github][url]

2018

• 
  homer@UniKoblenz: Winning Team of the RoboCup@Home Open Platform League 2018
  Memmesheimer, Raphael; Mykhalchyshyna, Ivanna; Seib, Viktor; Evers, Tobias; Paulus, Dietrich
  RoboCup 2018: Robot World Cup XXII [Montreal, QC, Canada, June 18-22, 2018] Springer (2018)

  Abstract:

  We won this year’s RoboCup@Home track in the Open Platform League in Montreal (Canada). The approaches as used for the competition are briefly described in this paper. The robotic hardware of our custom built robot Lisa and the PAL Robotics TIAGo, both running the same methods, are presented. New approaches for object recognition, especially the preprocessed segment augmentation, effort based gripping, gesture recognition and approaches for visual imitation learning based on continuous spatial observations between a demonstrator and the interacting objects are presented. Further, we present the current state of research of our Imitation Learning approaches, where we propose a hybrid benchmark and methods for bootstrapping actions. Furthermore, our research on point cloud based object recognition is presented.

  
  [url]
• 
  Gesture Recognition On Human Pose Features Of Single Images
  Memmesheimer, Raphael; Mykhalchyshyna, Ivanna; Paulus, Dietrich
  9th IEEE International Conference on Intelligent Systems, IS 2018, Funchal, Madeira, Portugal, September 25-27, 2018 IEEE (2018)

  Abstract:

  Enabling robots to read intentions of humans i.e. by detecting a waving person in a restaurant or by pointing to something in a supermarket gives a huge set of possibilities for human robot interactions. Gesture recognition is a challenging problem because of the complexity and the wide variety of human gestures. In this paper, we propose a method for multi-person gesture classification for five gestures and one neutral body posture based on human pose features extracted on 2D images. Comparison between supervised machine learning methods for gesture classification are given. The results showed, that the proposed approach achieves good results on our own validation dataset and generalizes well on a public dataset.

  
  [url]
• 
  RoboCup@Home: Summarizing achievements in over eleven years of competition
  Matamoros, Mauricio; Seib, Viktor; Memmesheimer, Raphael; Paulus, Dietrich
  2018 IEEE International Conference on Autonomous Robot Systems and Competitions, ICARSC 2018, Torres Vedras, Portugal, April 25-27, 2018 IEEE (2018)

  Abstract:

  Scientific competitions are important in robotics because they foster knowledge exchange and allow teams to test their research in unstandardized scenarios and compare result. In the field of service robotics its role becomes crucial. Competitions like RoboCup@Home bring robots to people, a fundamental step to integrate them into society. In this paper we summarize and discuss the differences between the achievements claimed by teams in their team description papers, and the results observed during the competition1 from a qualitative perspective. We conclude with a set of important challenges to be conquered first in order to take robots to people's homes. We believe that competitions are also an excellent opportunity to collect data of direct and unbiased interactions for further research.

  
  [url]

2017

• 
  homer@UniKoblenz: Winning Team of the RoboCup@Home Open Platform League 2017
  Memmesheimer, Raphael; Seib, Viktor; Paulus, Dietrich
  RoboCup 2017: Robot World Cup XXI [Nagoya, Japan, July 27-31, 2017] Springer (2017)

  Abstract:

  In this paper we present the approaches that we used for this year's RoboCup@Home participation in the Open Platform League. A special focus was put on team collaboration by handing over objects between two robots of different teams that were not connected by network. The robots communicated using natural language (speech synthesis, speech recognition), a typical human-robot ``interface'' that was adapted to robot-robot interaction. Furthermore, we integrated new approaches for online tracking and learning of an operator, have shown a novel approach for teaching a robot new commands by describing them using natural language and a set of previously known commands. Parameters of these commands are still interchangeable. Finally, we integrated deep neural networks for person detection and recognition, human pose estimation, gender classification and object recognition.

  
  [url]
• 
  Joint operator detection and tracking for person following from mobile platforms
  Wojke, Nicolai; Memmesheimer, Raphael; Paulus, Dietrich
  20th International Conference on Information Fusion, FUSION 2017, Xi'an, China, July 10-13, 2017 IEEE (2017)

  Abstract:

  In this paper, we propose an integrated system to detect and track a single operator that can switch off and on when it leaves and (re-)enters the scene. Our method is based on a set-valued Bayes-optimal state estimator that integrates RGB-D detections and image-based classification to improve tracking results in severe clutter and under long-term occlusion. The classifier is trained in two stages: First, we train a deep convolutional neural network to obtain a feature representation for person re-identification. Then, we bootstrap a classifier that discriminates the operator from remaining people on the output of the state-estimator. We evaluate the approach on a publicly available multi-target tracking dataset as well as custom datasets that are specific to our problem formulation. Experimental results suggest reliable tracking accuracy in crowded scenes and robust re-detection after long-term occlusion.

  
  [url]
• 
  Benchmarking Speech Understanding in Service Robotics
  Vanzo, Andrea; Iocchi, Luca; Nardi, Daniele; Memmesheimer, Raphael; Paulus, Dietrich; Ivanovska, Iryna; Kraetzschmar, Gerhard K.
  Proceedings of the 4th Italian Workshop on Artificial Intelligence and Robotics A workshop of the XVI International Conference of the Italian Association for Artificial Intelligence (AI*IA 2017), Bari, Italy, November 14-15, 2017 CEUR-WS.org (2017)
  [url]

2016

• 
  A ROS-based System for an Autonomous Service Robot
  Seib, Viktor; Memmesheimer, Raphael; Paulus, Dietrich
  Robot Operating System (ROS): The Complete Reference (Volume 1) Springer (2016)

  Abstract:

  The Active Vision Group (AGAS) has gained plenty of experience in robotics over the past years. This contribution focuses on the area of service robotics. We present several important components that are crucial for a service robot system: mapping and navigation, object recognition, speech synthesis and speech recognition. A detailed tutorial on each of these packages is given in the presented chapter. All of the presented components are published on our ROS package repository: http:// wiki.ros.org/agas-ros-pkg.

  
  [url]

2015

• 
  Team Homer@UniKoblenz - Approaches and Contributions to the RoboCup@Home Competition
  Seib, Viktor; Manthe, Stephan; Memmesheimer, Raphael; Polster, Florian; Paulus, Dietrich
  RoboCup 2015: Robot World Cup XIX [papers from the 19th Annual RoboCup International Symposium, Hefei, China, July 23, 2015] Springer (2015)

  Abstract:

  In this paper we present the approaches and contributions of team homer@UniKoblenz that were developed for and applied during the RoboCup@Home competitions. In particular, we highlight the different abstraction layers of our software architecture that allows for rapid application development based on the ROS actionlib. This architectural design enables us to focus on the development of new algorithms and approaches and significantly helped us in winning the RoboCup@Home competition in 2015. We further give an outlook on recently published open-source software for service robots that can be downloaded from our ROS package repository on http://wiki.ros.org/agas-ros-pkg .

  
  [url]
• 
  Ensemble classifier for joint object instance and category recognition on RGB-D data
  Seib, Viktor; Memmesheimer, Raphael; Paulus, Dietrich
  2015 IEEE International Conference on Image Processing, ICIP 2015, Quebec City, QC, Canada, September 27-30, 2015 IEEE (2015)

  Abstract:

  Sensors for RGB-D data have gained high popularity in the computer vision community. We present an efficient ensemble classifier that combines visual and depth data and achieves higher recognition rates than the individual classifiers or a classifier exploiting visual and depth data at the same time. The presented approach was evaluated in practice on a mobile robot during the RoCKIn robotics challenge in 2014.

  
  [url]

Talks

• Advancing Autonomous Service Robots: From Human Intelligence to Embodied Articifial Intelligence in Domestic Environments (Scientific Keynote)
  RimA Summer School 2025 Karlsruhe, Germany
  2025
  
• NimbRo@Home: Autonomous service robots performing domestic tasks
  RIG STAGE - Demo Pitches Automatica, Munich, Germany
  2025
  
• Panel-Member for Round Table “Benchmarking and Challenges”
  RIG Panel Automatica, Munich, Germany
  2025
  
• Advancing Autonomous Service Robots: From Human Intelligence to Embodied Intelligence
  Campus Visit University of Technology Nuremberg (UTN), Nuremberg, Germany
  2025
  
• Embodied AI: Bridging Human Motion Analysis and Autonomous Robotic Actions
  Campus Visit Technical University of Braunschweig
  2025
  
• From Multimodal Few-Shot Action Recognition towards Imitation in Embodied AI
  Campus Visit Dresden (Online)
  2024
  
• Multimodal Few-Shot Action Recognition for Embodied AI
  Campus Visit Leipzig University, Leipzig, Germany
  2024
  
• Multimodal Few-Shot Action Recognition for Embodied AI
  Campus Visit University of Lübeck, Lübeck, Germany
  2024
  
• Embodied AI: Towards improved human robot interaction by learning from human motion
  Research Workshop: Control of Movement in Biology and Robotics University Bayreuth, Campus Kulmbach, Germany
  2023
  
• On the recognition and imitation of human activities on robotic platforms
  Advanced Lecture Series on Artificial Intelligence and Autonomous Systems Bonn Rhein Sieg, Germany (online)
  2022
  
• Servicerobotik - Wettbewerbe und Anwendungen
  MN-Seminar Darmstadt, Germany (online)
  2022
  
• Gesture and activity challenge result presentation of AcRec@UniKoblenz
  HEART-MET workshop Sankt Augustin, Germany (online)
  2021
  
• Golem - Faszination Künstliche Intelligenz
  Kultursalon Koblenz Koblenz, Germany
  2021
  
• Towards adaptive learning in robot competitions
  European Robotics Forum Malaga, Spain
  2020
  
• Roboter und künstliche Intelligenz
  Guestlecture: Intelligenz, Denken und Problemlösen / Roboter - KI oder Bauplan für eine Seele Koblenz, Germany
  2020
  
• Künstliche Intelligenz in Haushalt- und Servicerobotik
  VDI Mittelrhein Jahresversammlung Koblenz, Germany
  2019
  
• homer@Uni Koblenz
  Tsotsos Lab Toronto, Canada
  2018
  
• Lernen durch Demonstration
  Guestlecture: Intelligenz, Denken und Problemlösen Koblenz, Germany
  2018
  
• RoboCup 2017 / Where is HRI and where can it go to in RoboCup? RO-MAN Workshop: HRI for Service Robots in RoboCup@Home
  IEEE International Symposium on Robot and Human Interactive Communication Lisbon, Portugal
  2017
  
• Report from ERL Service Robots RO-MAN Workshop: HRI for Service Robots in RoboCup@Home
  IEEE International Symposium on Robot and Human Interactive Communication Lisbon, Portugal
  2017
  
• Team's experience in ERL - Service Robots European robotics competitions and challenges: status quo and lessons learned
  European Robotics Forum Edinburgh, Scotland
  2017
  
• Navigation and Mapping of Team homer@UniKoblenz
  European Robotics League Peccioli, Italy
  2017
  

Awards

• Placed 2nd in @Home Open Platform League, Best Poster Award
  RoboCup World Cup Salvador, Brazil 2025
  [url]
• Placed 1st in @Home League
  RoboCup German Open Nuremberg 2025
  [url][press]
• Placed 1st in @Home Open Platform League (World Champion), Waitress Captain (Best in Restaurant Test)
  RoboCup World Cup Eindhoven, Netherlands 2024
  [url][press]
• Placed 1st in @Home League
  RoboCup German Open Kassel 2024
  [url][press]
• 1st Place (NimbRo)
  ANA Avatar XPRIZE Challenge Las Vegas, USA 2022
  [url][press]
• 1st in AdultSize Soccer Competition / 1st in AdultSize Technical Challenge / 1st in AdultSize Drop-In Challenge / Best Humanoid Award
  RoboCup World Cup Bangkok, Thailand 2022
  [url][press]
• Placed 3rd in the Customer Interaction Category of the Future Convenience Store Challenge (private attendance)
  World Robot Summit Nagoya, Japan 2021
  
• Limbo (private built robot was awarded)
  Region56+ Award Koblenz, Germany 2021
  [url][press]
• Placed 1st in the Heart-Met Gesture Recognition Challenge / Placed 1st in the Heart-Met Action Recognition Challenge (AcRec@UniKoblenz)
  Metrics Project Online 2021
  
• Placed 1st in @Home Open Platform League (World Champion)
  RoboCup World Cup Sydney, Australia 2019
  [url][press]
• Best in Professional Service Robots League
  European Robotics League Bonn (Germany), Koblenz (Germany) 2019
  [press]
• Best in Consumer Service Robots League
  European Robotics League Lisbon, Portugal 2019
  [press]
• Placed 3rd in @Home League
  RoboCup German Open Magdeburg 2019
  [press]
• Placed 1st in @Home Open Platform League (World Champion) / Best Poster Award Open Platform League
  RoboCup World Cup Montreal, Canada 2018
  [url][press]
• Placed 3rd in the Customer Interaction Category of the Future Convenience Store Challenge
  World Robot Summit Tokyo, Japan 2018
  [url][press]
• Best in TBM1: Getting to know my home / Best in TBM2: Welcoming Visitors / Best in TBM3: Catering for Grannie Annie’s Comfort / Best in TBM5: GPSR (some of them shared equally with other teams)
  European Robotics League Lisbon (Portugal), Edinburgh (Scotland), Barcelona (Spain) 2018
  [url][press]
• Placed 1st in @Home League
  RoboCup German Open Magdeburg 2018
  [url][press]
• Placed 1st in @Home Open Platform League (World Champion)
  RoboCup World Cup Nagoya, Japan 2017
  [url][press]
• Lehrpreis Sommersemester 2017 | Projekt- und Forschungspraktikum: Robbie
  Lehrpreis der Hochschuldidaktischen Arbeitsstelle Koblenz, Germany 2017
  
• Finalist
  ICRA 2017 DJI RoboMaster Mobile Manipulation Challenge Singapore, Singapore 2017
  
• Placed 1st in @Home League
  RoboCup German Open Magdeburg 2017
  [url]
• Placed 2nd
  DJI RoboMaster Technical Challenge Shenzhen, China 2017
  [press]
• Best in TBM1: Getting to know my home, Best in TBM4: Visit my home (Navigation), Best in TBM5: GPSR
  European Robotics League Lisbon (Portugal), Peccioli (Italy) 2017
  [url][press]
• Placed 2nd in @Home League
  RoboCup European Open Eindhoven, Netherlands 2016
  [url][press]
• Finalist in @Home League
  RoboCup World Cup Leipzig, Germany 2016
  
• Placed 1st in @Home League (World Champion) / Placed 1st in Speech Recognition and Audio Detection / Best Looking Robot Award
  RoboCup World Cup Hefei, China 2015
  [url][press]
• 1st in overall ranking (together with team SocRob) / Best Team award
  RoCKIn Lisboa, Portugal 2015
  
• Best Demonstration in RoCKIn@Home track
  RoCKIn Camp Peccioli, Italy 2015
  
• Placed 2nd in @Home League
  RoboCup German Open Magdeburg 2015
  
• 1st in the @Home Track / 2nd in Object Recognition
  RoCKIn Toulouse, France 2014
  
• Finalist in the @Home League
  RoboCup German Open Magdeburg 2014
  
• Best Final Demonstration in the RoCKIn@Home track
  RoCKIn Camp Rome 2014
  
• Finalist in the @Home League
  RoboCup WorldCup Eindhoven, Netherlands 2013
  
• Placed 3rd in the @Home League
  RoboCup German Open Magdeburg 2013