{"author":"mich5632","children":[{"author":"mich5632","children":[],"created_at":"2025-06-13T16:58:21.000Z","created_at_i":1749833901,"id":44270215,"options":[],"parent_id":44270214,"points":null,"story_id":44270214,"text":"We wrote a rust py03 client for OpenAI embeddings compatible servers (openai.com, or infinity, TEI, vllm, sglang). \nMost server-side ML infrastructure auto-scales based on the workload. On embedding workloads, this is no longer the bottleneck and has shifted to the client. In Python, the client is blocked by the global interpreter lock. \nWith the performance package, we release the gil during requests, so you have available resources to query your VectorDB again.","title":null,"type":"comment","url":null}],"created_at":"2025-06-13T16:58:21.000Z","created_at_i":1749833901,"id":44270214,"options":[],"parent_id":null,"points":7,"story_id":44270214,"text":null,"title":"High performance client for Baseten.co","type":"story","url":"https://github.com/basetenlabs/truss/tree/main/baseten-performance-client"}