Vercel Enhances AI Gateway with Session Tracking

What happened

Vercel has introduced a new feature in its AI Gateway that allows users to monitor how requests are grouped into sessions and how session affinity directs each request. This feature provides visibility into the routing process, enabling users to understand how the system manages interactions.

The update enables users to open individual sessions to see metrics such as cache hit rate, session duration, and the specific path each request takes. Each turn within a session is detailed with the model used and the timing of the request.

Why it matters

This enhancement is significant for developers and users who rely on efficient and predictable performance from AI services. By tracking session affinity, the system can ensure that each turn is processed by the same provider, which improves caching and reduces latency.

Key details

Sessions display the provider responsible for serving the request, along with any previous providers involved in the session. This information helps users understand the flow of data and the involvement of different services.

Users can filter sessions based on various criteria, including turns, client, model, provider, authentication, and cost. This flexibility allows for targeted analysis and management of AI interactions.