| Beschreibung |
The growing capabilities of large language models and text-to-speech technology are expanding the use of conversational agents. In virtual environments, these agents can be represented as animated avatars, leading to more lively and engaging VR experiences, e.g. when using them as guides in virtual museums. However, communication with these agents in VR relies entirely on speech, as traditional text input is no longer an option. In this project, we aim to extend previous work of our research group to multi-user settings (i.e. social virtual reality), making interactions more natural and intuitive while integrating multiple agents into virtual worlds. The key challenge is to ensure smooth communication between multiple VR users and multiple AI agents without overwhelming or confusing conversations. We will investigate how users can initiate interactions with agents without explicit selection of a specific agent and how they can receive additional information in an intuitive way. To explore these aspects, you will build on our existing integrations of OpenAI’s language models and animated avatars in Unity3D. You will investigate different modalities such as pointing, looking, and speaking to select agents and experiment with ways to playfully engage VR users in immersive experiences. During this project, you will learn to design multi-user virtual reality applications and deepen your understanding of virtual reality development. Furthermore, you will explore advanced multi-modal interaction concepts, gain practical experience using OpenAI’s API and enhance your proficiency with Unity3D and C#. Prior experience with C#, Unity3D, and multi-user VR is strongly advised. Additionally, familiarity with rigging and character animations will be beneficial. Equipment for VR development will be provided. |
