Built a demo around integrating Gemini Live with Stream's Video API for agent use-cases. In this example, I'm having the LLM provide feedback to players as they try to improve their mini-golf swing.
On the backend, it uses the Python AI SDK to capture the WebRTC frames from the player, convert them, and then feed them to the Gemini Live API. Once we have a response from Gemini, the audio output is encoded and sent directly to the call, where the user can hear and respond.
Is anyone else building apps around AI and real-time voice/video? Would be curious to share notes. If anyone is interested in trying for themselves:
On the backend, it uses the Python AI SDK to capture the WebRTC frames from the player, convert them, and then feed them to the Gemini Live API. Once we have a response from Gemini, the audio output is encoded and sent directly to the call, where the user can hear and respond.
Is anyone else building apps around AI and real-time voice/video? Would be curious to share notes. If anyone is interested in trying for themselves:
Python SDK docs: https://getstream.io/video/docs/python-ai/basics/quickstart/ Github: https://github.com/GetStream/stream-py/tree/webrtc