{"exhaustive":{"nbHits":true,"typo":true},"exhaustiveNbHits":true,"exhaustiveTypo":true,"hits":[{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"jojo_sugar"},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"<em>Nebuly</em> \u2013 The LLM Analytics Platform"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"https://www.<em>nebuly</em>.com/"}},"_tags":["story","author_jojo_sugar","story_37802533"],"author":"jojo_sugar","children":[37802569],"created_at":"2023-10-07T15:27:43Z","created_at_i":1696692463,"num_comments":1,"objectID":"37802533","points":1,"story_id":37802533,"title":"Nebuly \u2013 The LLM Analytics Platform","updated_at":"2024-09-20T15:18:26Z","url":"https://www.nebuly.com/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"mskierkowski"},"story_text":{"matchLevel":"none","matchedWords":[],"value":""},"title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Show HN: <em>Nebuly</em> - SharePoint without the pain"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"http://www.<em>nebuly</em>.net/"}},"_tags":["story","author_mskierkowski","story_3044458","show_hn"],"author":"mskierkowski","created_at":"2011-09-27T18:03:30Z","created_at_i":1317146610,"num_comments":0,"objectID":"3044458","points":1,"story_id":3044458,"story_text":"","title":"Show HN: Nebuly - SharePoint without the pain","updated_at":"2024-09-19T18:00:57Z","url":"http://www.nebuly.net/"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"emilec___"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Training always takes too long. If it takes an hour, it would be better if it took 30 minutes, or maybe 15 minutes... or just 1 minute, why not? And if you want to speed up training, the techs available usually require to increase the complexity of the training process, whether it's making trade-off in terms of accuracy or time for the developer to learn a new framework. Often times it's trial and error, playing with parameters, training recipes, or switching framework/model. That's definitely not ideal.<p>\u201cFast &amp; easy-to-use\u201d These were keywords that motivated me to work on a new way of doing training, the library nebulgym, which now is open-source (https://github.com/<em>nebuly</em>-ai/nebulgym).<p>FAST<p>Training should be fast, period. Wouldn't it be great if in the near future you could train a GPT3 from scratch on your laptop? Or a large EfficientNet in a fraction of a minute? Nebulgym was built to try to bring developers closer to that future. This open-source optimizes the full training computation stack, from efficient data loading to faster forward and backward passes and earlier convergence. For example, by saving data samples in the cache on the first data read, it speeds up the full data loading process and eliminates what can become the bottleneck for the training process. Nebulgym also leverages techniques such as partial compilation of some calculations and smart sparse gradients to speed up forward and backward gradient propagations. And many more features will be implemented soon. And please let me know / open issues if you have ideas for making nebulgym even faster :)<p>EASY-TO-USE<p>&quot;Not another framework, please, there're already 1000&quot;. That's a call for help from many developers, so nebulgym has been developed with this in mind. Nebulgym let you use the training setup you've always used, and works &quot;on top&quot;. This is made possible with the use of class decorators (like Java's annotations). In short, you can just add these decorators before defining the model classes, and nebulgym will make sure that you use your computing resources to the fullest.<p>Here's a snippet of training with nebulgym decorators (@accelerate_dataset and @accelerate_model)<p>--------------------------<p>import torch<p>from nebulgym.decorators.torch_decorators import \naccelerate_model, accelerate_dataset<p>@accelerate_dataset()<p>class MyDataset(torch.utils.data.Dataset):\n   #Your Dataset definition<p>@accelerate_model()<p>class MyModel(torch.nn.Module):\n   # Your model definition<p># Train your model as you usually do<p>--------------------------<p>And that's it. Give it a try, and leave a star , it's a little contribution to show some love for open-source projects :) Also feedback would be super appreciated!<p>https://github.com/<em>nebuly</em>-ai/nebulgym"},"title":{"matchLevel":"none","matchedWords":[],"value":"Nebulgym, a new open-source that accelerates AI training (~1.5-2x)"}},"_tags":["story","author_emilec___","story_31666602","ask_hn"],"author":"emilec___","children":[31666702],"created_at":"2022-06-08T12:59:16Z","created_at_i":1654693156,"num_comments":1,"objectID":"31666602","points":3,"story_id":31666602,"story_text":"Training always takes too long. If it takes an hour, it would be better if it took 30 minutes, or maybe 15 minutes... or just 1 minute, why not? And if you want to speed up training, the techs available usually require to increase the complexity of the training process, whether it&#x27;s making trade-off in terms of accuracy or time for the developer to learn a new framework. Often times it&#x27;s trial and error, playing with parameters, training recipes, or switching framework&#x2F;model. That&#x27;s definitely not ideal.<p>\u201cFast &amp; easy-to-use\u201d These were keywords that motivated me to work on a new way of doing training, the library nebulgym, which now is open-source (https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebulgym).<p>FAST<p>Training should be fast, period. Wouldn&#x27;t it be great if in the near future you could train a GPT3 from scratch on your laptop? Or a large EfficientNet in a fraction of a minute? Nebulgym was built to try to bring developers closer to that future. This open-source optimizes the full training computation stack, from efficient data loading to faster forward and backward passes and earlier convergence. For example, by saving data samples in the cache on the first data read, it speeds up the full data loading process and eliminates what can become the bottleneck for the training process. Nebulgym also leverages techniques such as partial compilation of some calculations and smart sparse gradients to speed up forward and backward gradient propagations. And many more features will be implemented soon. And please let me know &#x2F; open issues if you have ideas for making nebulgym even faster :)<p>EASY-TO-USE<p>&quot;Not another framework, please, there&#x27;re already 1000&quot;. That&#x27;s a call for help from many developers, so nebulgym has been developed with this in mind. Nebulgym let you use the training setup you&#x27;ve always used, and works &quot;on top&quot;. This is made possible with the use of class decorators (like Java&#x27;s annotations). In short, you can just add these decorators before defining the model classes, and nebulgym will make sure that you use your computing resources to the fullest.<p>Here&#x27;s a snippet of training with nebulgym decorators (@accelerate_dataset and @accelerate_model)<p>--------------------------<p>import torch<p>from nebulgym.decorators.torch_decorators import \naccelerate_model, accelerate_dataset<p>@accelerate_dataset()<p>class MyDataset(torch.utils.data.Dataset):\n   #Your Dataset definition<p>@accelerate_model()<p>class MyModel(torch.nn.Module):\n   # Your model definition<p># Train your model as you usually do<p>--------------------------<p>And that&#x27;s it. Give it a try, and leave a star , it&#x27;s a little contribution to show some love for open-source projects :) Also feedback would be super appreciated!<p>https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebulgym","title":"Nebulgym, a new open-source that accelerates AI training (~1.5-2x)","updated_at":"2024-09-20T11:20:15Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"emilec___"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"The situation is fairly simple. Your model could run 10 times faster by adding a few lines to your code, but you weren't aware of it. Let me expand on that.<p>AI applications are multiplying like mushrooms, which is awesome<p>As a result, more and more people are turning to the dark side, joining the AI world, as I did<p>The problem? Developers focus only on AI, cleaning up datasets and training their models. Almost no one has a background in hardware, compilers, computing, cloud, etc<p>The result? Developers spend a lot of hours improving the accuracy and performance of their software, and all their hard work risks being undone by the wrong choice of hardware-software coupling<p>This problem bothered me for a long time, so with a couple of buddies at <em>Nebuly</em> (all ex MIT, ETH and EPFL), we put a lot of energy into an open-source library called nebullvm to make DL compiler technology accessible to any developer, even for those who know nothing about hardware, as I did.<p>How does it work? It speeds up your DL models by ~5-20x by testing the best DL compilers out there and selecting the optimal one to best couple your AI model with your machine (GPU, CPU, etc.). All this in just a few lines of code.<p>The library is open source and you can find it here https://github.com/<em>nebuly</em>-ai/nebullvm.<p>Please leave a star on GitHub for the hard work in building the library :) It's a simple act for you, a big smile for us. Thank you, and don't hesitate to contribute to the library!"},"title":{"matchLevel":"none","matchedWords":[],"value":"Almost no one knows how easily you can optimize your AI models"}},"_tags":["story","author_emilec___","story_30768105","ask_hn"],"author":"emilec___","created_at":"2022-03-22T16:26:21Z","created_at_i":1647966381,"num_comments":0,"objectID":"30768105","points":3,"story_id":30768105,"story_text":"The situation is fairly simple. Your model could run 10 times faster by adding a few lines to your code, but you weren&#x27;t aware of it. Let me expand on that.<p>AI applications are multiplying like mushrooms, which is awesome<p>As a result, more and more people are turning to the dark side, joining the AI world, as I did<p>The problem? Developers focus only on AI, cleaning up datasets and training their models. Almost no one has a background in hardware, compilers, computing, cloud, etc<p>The result? Developers spend a lot of hours improving the accuracy and performance of their software, and all their hard work risks being undone by the wrong choice of hardware-software coupling<p>This problem bothered me for a long time, so with a couple of buddies at Nebuly (all ex MIT, ETH and EPFL), we put a lot of energy into an open-source library called nebullvm to make DL compiler technology accessible to any developer, even for those who know nothing about hardware, as I did.<p>How does it work? It speeds up your DL models by ~5-20x by testing the best DL compilers out there and selecting the optimal one to best couple your AI model with your machine (GPU, CPU, etc.). All this in just a few lines of code.<p>The library is open source and you can find it here https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm.<p>Please leave a star on GitHub for the hard work in building the library :) It&#x27;s a simple act for you, a big smile for us. Thank you, and don&#x27;t hesitate to contribute to the library!","title":"Almost no one knows how easily you can optimize your AI models","updated_at":"2024-09-20T10:47:49Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"emilec___"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Hi everyone!!!<p>I recently launched an open-source library on github (https://github.com/<em>nebuly</em>-ai/nebullvm). It's growing super fast (700+ stars), but I can't get much feedback from github, and I'd like to ask for some feedback here.<p>Nebullvm is open-source library takes your AI model as input and outputs an optimized version that runs 5-20 times faster on your machine. Nebullvm tests multiple deep learning compilers to identify the best possible way to execute your model on your specific hardware, without impacting the accuracy of your model.<p>The goal of nebullvm is to let any developer benefit from deep learning (DL) compilers without having to spend tons of hours understanding, installing, testing and debugging this powerful technology.<p>You can find more info https://github.com/<em>nebuly</em>-ai/nebullvm. And please let me know what you think of the project and if you find any ways to improve it! Big thanks :)"},"title":{"matchLevel":"none","matchedWords":[],"value":"Nebullvm open-source accelerator of AI inference. Feedback?"}},"_tags":["story","author_emilec___","story_30788885","ask_hn"],"author":"emilec___","children":[30864020],"created_at":"2022-03-24T12:34:38Z","created_at_i":1648125278,"num_comments":1,"objectID":"30788885","points":2,"story_id":30788885,"story_text":"Hi everyone!!!<p>I recently launched an open-source library on github (https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm). It&#x27;s growing super fast (700+ stars), but I can&#x27;t get much feedback from github, and I&#x27;d like to ask for some feedback here.<p>Nebullvm is open-source library takes your AI model as input and outputs an optimized version that runs 5-20 times faster on your machine. Nebullvm tests multiple deep learning compilers to identify the best possible way to execute your model on your specific hardware, without impacting the accuracy of your model.<p>The goal of nebullvm is to let any developer benefit from deep learning (DL) compilers without having to spend tons of hours understanding, installing, testing and debugging this powerful technology.<p>You can find more info https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm. And please let me know what you think of the project and if you find any ways to improve it! Big thanks :)","title":"Nebullvm open-source accelerator of AI inference. Feedback?","updated_at":"2024-09-20T10:49:33Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"Telemaco019"},"story_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Hi HN! I\u2019m Michele Zanotti and today I\u2019m releasing nos, an open-source module to efficiently run GPU workloads on Kubernetes!<p>Nos is meant to increase GPU utilization and cut down infrastructure and operational costs providing 2 main features:<p>1. Dynamic GPU Partitioning: you can think of this as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia's MPS and MIG [1,2], finally making them dynamic.<p>2. Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them.<p>Let me know your thoughts on the project in the comments :)<p>Nos addresses some key challenges of Kubernetes tied to the fact that Kubernetes was not designed to support GPU and AI / machine learning workloads. In Kubernetes, GPUs are managed with [3] Nvidia k8s Device Plugin, which has a few major downsides. First, it requires the allocation of an integer number of GPUs per workload, not allowing workloads to request only fractions of GPU. Second, when enabling GPU shared access either with time-slicing or MIG, the device plugin advertises to Kubernetes a fixed set of GPU resources that do not dynamically adapt to the requests of the Pods at each time.<p>This often leads to both underutilized GPUs and pending Pods, and/or the cluster admin having to spend a lot of time looking for workarounds to make the best use of GPUs.<p>For example, consider a company with a k8s cluster with 20 GPUs, where 3 of these GPUs has been reserved to the data science team using Resource Quota objects. In most cases, the workloads of data scientists (notebooks, scripts, etc.) require much less memory/compute resources than those of an entire GPU, yet Kubernetes will force each container to consume an entire GPU. Also, if the team once needs to run a heavy workload, it may want to use as many resources as possible. However, the Resource Quota over their namespace would constrain the team to use at most the 3 GPUs reserved for them, even if the company cluster may be full of unused GPUs!<p>Instead, with nos the data science team would use nos Dynamic GPU Partitioning to request GPU slices so that many workloads can share the same GPU. Also, Elastic Resource Quotas would allow the team to consume more than the 3 reserved GPUs, borrowing quotas from other teams that are not using them. To recap, the team would be able to launch more Pods and the company would likely need fewer nodes. All this with minimal effort required by the cluster admin, who only has to set up nos.<p>Let me know what you think of nos, feedback would be very helpful! :) And please leave a star on GitHub if you like this opensource <a href=\"https://github.com/nebuly-ai/nos\">https://github.com/<em>nebuly</em>-ai/nos</a><p>Here are some other links that may be useful<p>- Tutorial on how to use Dynamic GPU Partitioning with Nvidia MIG <a href=\"https://towardsdatascience.com/dynamic-mig-partitioning-in-kubernetes-89db6cdde7a3\" rel=\"nofollow\">https://towardsdatascience.com/dynamic-mig-partitioning-in-k...</a><p>- Pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https://docs.nebuly.com/nos/dynamic-gpu-partitioning/partitioning-modes-comparison/\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/dynamic-gpu-partitioning/partiti...</a><p>- Nos documentation <a href=\"https://docs.nebuly.com/nos/overview/\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/overview/</a><p>References<p>[1] Multi-Process Service (MPS) <a href=\"https://docs.nvidia.com/deploy/mps/index.html\" rel=\"nofollow\">https://docs.nvidia.com/deploy/mps/index.html</a><p>[2] Multi-Instance GPU (MIG) <a href=\"https://docs.nvidia.com/datacenter/tesla/mig-user-guide/\" rel=\"nofollow\">https://docs.nvidia.com/datacenter/tesla/mig-user-guide/</a><p>[3] NVIDIA k8s Device Plugin <a href=\"https://github.com/NVIDIA/k8s-device-plugin\">https://github.com/NVIDIA/k8s-device-plugin</a>"},"title":{"matchLevel":"none","matchedWords":[],"value":"Show HN: Nos \u2013 Open-Source to Maximize GPU Utilization in Kubernetes"},"url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"https://github.com/<em>nebuly</em>-ai/nos"}},"_tags":["story","author_Telemaco019","story_34579425","show_hn"],"author":"Telemaco019","created_at":"2023-01-30T13:41:46Z","created_at_i":1675086106,"num_comments":0,"objectID":"34579425","points":2,"story_id":34579425,"story_text":"Hi HN! I\u2019m Michele Zanotti and today I\u2019m releasing nos, an open-source module to efficiently run GPU workloads on Kubernetes!<p>Nos is meant to increase GPU utilization and cut down infrastructure and operational costs providing 2 main features:<p>1. Dynamic GPU Partitioning: you can think of this as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia&#x27;s MPS and MIG [1,2], finally making them dynamic.<p>2. Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them.<p>Let me know your thoughts on the project in the comments :)<p>Nos addresses some key challenges of Kubernetes tied to the fact that Kubernetes was not designed to support GPU and AI &#x2F; machine learning workloads. In Kubernetes, GPUs are managed with [3] Nvidia k8s Device Plugin, which has a few major downsides. First, it requires the allocation of an integer number of GPUs per workload, not allowing workloads to request only fractions of GPU. Second, when enabling GPU shared access either with time-slicing or MIG, the device plugin advertises to Kubernetes a fixed set of GPU resources that do not dynamically adapt to the requests of the Pods at each time.<p>This often leads to both underutilized GPUs and pending Pods, and&#x2F;or the cluster admin having to spend a lot of time looking for workarounds to make the best use of GPUs.<p>For example, consider a company with a k8s cluster with 20 GPUs, where 3 of these GPUs has been reserved to the data science team using Resource Quota objects. In most cases, the workloads of data scientists (notebooks, scripts, etc.) require much less memory&#x2F;compute resources than those of an entire GPU, yet Kubernetes will force each container to consume an entire GPU. Also, if the team once needs to run a heavy workload, it may want to use as many resources as possible. However, the Resource Quota over their namespace would constrain the team to use at most the 3 GPUs reserved for them, even if the company cluster may be full of unused GPUs!<p>Instead, with nos the data science team would use nos Dynamic GPU Partitioning to request GPU slices so that many workloads can share the same GPU. Also, Elastic Resource Quotas would allow the team to consume more than the 3 reserved GPUs, borrowing quotas from other teams that are not using them. To recap, the team would be able to launch more Pods and the company would likely need fewer nodes. All this with minimal effort required by the cluster admin, who only has to set up nos.<p>Let me know what you think of nos, feedback would be very helpful! :) And please leave a star on GitHub if you like this opensource <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos</a><p>Here are some other links that may be useful<p>- Tutorial on how to use Dynamic GPU Partitioning with Nvidia MIG <a href=\"https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-kubernetes-89db6cdde7a3\" rel=\"nofollow\">https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-k...</a><p>- Pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partitioning-modes-comparison&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partiti...</a><p>- Nos documentation <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;</a><p>References<p>[1] Multi-Process Service (MPS) <a href=\"https:&#x2F;&#x2F;docs.nvidia.com&#x2F;deploy&#x2F;mps&#x2F;index.html\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nvidia.com&#x2F;deploy&#x2F;mps&#x2F;index.html</a><p>[2] Multi-Instance GPU (MIG) <a href=\"https:&#x2F;&#x2F;docs.nvidia.com&#x2F;datacenter&#x2F;tesla&#x2F;mig-user-guide&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nvidia.com&#x2F;datacenter&#x2F;tesla&#x2F;mig-user-guide&#x2F;</a><p>[3] NVIDIA k8s Device Plugin <a href=\"https:&#x2F;&#x2F;github.com&#x2F;NVIDIA&#x2F;k8s-device-plugin\">https:&#x2F;&#x2F;github.com&#x2F;NVIDIA&#x2F;k8s-device-plugin</a>","title":"Show HN: Nos \u2013 Open-Source to Maximize GPU Utilization in Kubernetes","updated_at":"2024-09-20T13:15:56Z","url":"https://github.com/nebuly-ai/nos"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"R21M1214"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Im not sure which ones are Otel compliant. Im only aware of 3 that are Otel compliant:<p>1. Traceloop Otel\n2. Langtrace.ai Otel\n3. OpenLIT Otel\n4. Portkey\n5. Langfuse\n6. Arize LLM\n7. Phoniex SDK\n8. Truera LLM\n9. Truelens\n10. Context\n11. Braintrust\n12. Parea\n13. Context AI\n14. openlayer.com\n15. Deepchecks\n16. langsmith\n17. Confident AI\n18. Helicone\n19. Langwatch.ai\n20. Arthur\n21. Aporia\n22. scale.com\n23. Whylabs\n24. gentrace.ai\n25. humanloop.com\n26. fixpoint.co\n27. W n B Traces\n28. Langtail\n29. Fiddler\n30. Evidently Ai\n31. Superwise\n32. Exxa\n33. Honeyhive\n34. Flowstack\n35. Log10\n36. Giskard\n37. Raga AI\n38. AgentOps\n39. Patronus AI\n40. Mona\n41. Bricks Ai\n42. Sentify\n43. LogSpend\n44. <em>Nebuly</em>\n45. Autoblocks\n46. Radar / Langcheck\n47. Dokulabs\n48. Missing studio\n49. Lunary.ai\n50. Censius.ai\n51. ML flow\n52. Galileo\n53. trubrics\n54. Prompt Layer\n55. Athina\n56. getnomos.com\n57. c3.ai\n58. baselime.io\n59. Honeycomb llm"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Launch HN: Traceloop (YC W23) \u2013 Detecting LLM Hallucinations with OpenTelemetry"}},"_tags":["comment","author_R21M1214","story_40985609"],"author":"R21M1214","children":[40988087,40994185,41076159],"comment_text":"Im not sure which ones are Otel compliant. Im only aware of 3 that are Otel compliant:<p>1. Traceloop Otel\n2. Langtrace.ai Otel\n3. OpenLIT Otel\n4. Portkey\n5. Langfuse\n6. Arize LLM\n7. Phoniex SDK\n8. Truera LLM\n9. Truelens\n10. Context\n11. Braintrust\n12. Parea\n13. Context AI\n14. openlayer.com\n15. Deepchecks\n16. langsmith\n17. Confident AI\n18. Helicone\n19. Langwatch.ai\n20. Arthur\n21. Aporia\n22. scale.com\n23. Whylabs\n24. gentrace.ai\n25. humanloop.com\n26. fixpoint.co\n27. W n B Traces\n28. Langtail\n29. Fiddler\n30. Evidently Ai\n31. Superwise\n32. Exxa\n33. Honeyhive\n34. Flowstack\n35. Log10\n36. Giskard\n37. Raga AI\n38. AgentOps\n39. Patronus AI\n40. Mona\n41. Bricks Ai\n42. Sentify\n43. LogSpend\n44. Nebuly\n45. Autoblocks\n46. Radar &#x2F; Langcheck\n47. Dokulabs\n48. Missing studio\n49. Lunary.ai\n50. Censius.ai\n51. ML flow\n52. Galileo\n53. trubrics\n54. Prompt Layer\n55. Athina\n56. getnomos.com\n57. c3.ai\n58. baselime.io\n59. Honeycomb llm","created_at":"2024-07-17T16:48:34Z","created_at_i":1721234914,"objectID":"40987794","parent_id":40987116,"story_id":40985609,"story_title":"Launch HN: Traceloop (YC W23) \u2013 Detecting LLM Hallucinations with OpenTelemetry","updated_at":"2026-11-16T08:36:12Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"jojo_sugar"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Hello everyone,<p><em>Nebuly</em> is a user analytics platform for LLMs. The platform helps you understand what LLM users want and why, how they interact with LLMs, and how satisfied they are. It also uncovers the topics discussed, the most frequently asked questions, so you can better understand what users want and ensure that your LLM-based products deliver value to users.<p>The platform is currently in early access and you can request a demo via the website or contact one of the founders on LinkedIn to get access."},"story_title":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"<em>Nebuly</em> \u2013 The LLM Analytics Platform"},"story_url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"https://www.<em>nebuly</em>.com/"}},"_tags":["comment","author_jojo_sugar","story_37802533"],"author":"jojo_sugar","comment_text":"Hello everyone,<p>Nebuly is a user analytics platform for LLMs. The platform helps you understand what LLM users want and why, how they interact with LLMs, and how satisfied they are. It also uncovers the topics discussed, the most frequently asked questions, so you can better understand what users want and ensure that your LLM-based products deliver value to users.<p>The platform is currently in early access and you can request a demo via the website or contact one of the founders on LinkedIn to get access.","created_at":"2023-10-07T15:33:29Z","created_at_i":1696692809,"objectID":"37802569","parent_id":37802533,"story_id":37802533,"story_title":"Nebuly \u2013 The LLM Analytics Platform","story_url":"https://www.nebuly.com/","updated_at":"2024-09-20T15:18:26Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"0xebo"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Langfuse is great ! Also check <a href=\"https://www.nebuly.com/\" rel=\"nofollow noreferrer\">https://www.<em>nebuly</em>.com/</a>"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Ask HN: Any tools or frameworks to monitor the usage of OpenAI API keys?"}},"_tags":["comment","author_0xebo","story_37377844"],"author":"0xebo","comment_text":"Langfuse is great ! Also check <a href=\"https:&#x2F;&#x2F;www.nebuly.com&#x2F;\" rel=\"nofollow noreferrer\">https:&#x2F;&#x2F;www.nebuly.com&#x2F;</a>","created_at":"2023-09-04T08:48:05Z","created_at_i":1693817285,"objectID":"37378501","parent_id":37377844,"story_id":37377844,"story_title":"Ask HN: Any tools or frameworks to monitor the usage of OpenAI API keys?","updated_at":"2024-09-20T15:04:47Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"mikie92"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"How does it differentiate from the original ChatLLaMA? <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/chatllama\">https://github.com/<em>nebuly</em>-ai/nebullvm/tree/main/apps/acceler...</a>"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Show HN: ChatLLaMA \u2013 A ChatGPT style chatbot for Facebook's LLaMA"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://chatllama.baseten.co/"}},"_tags":["comment","author_mikie92","story_35258553"],"author":"mikie92","comment_text":"How does it differentiate from the original ChatLLaMA? <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;tree&#x2F;main&#x2F;apps&#x2F;accelerate&#x2F;chatllama\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;tree&#x2F;main&#x2F;apps&#x2F;acceler...</a>","created_at":"2023-03-22T17:45:35Z","created_at_i":1679507135,"objectID":"35264560","parent_id":35258553,"story_id":35258553,"story_title":"Show HN: ChatLLaMA \u2013 A ChatGPT style chatbot for Facebook's LLaMA","story_url":"https://chatllama.baseten.co/","updated_at":"2024-09-20T13:40:00Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"minxomat"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Is this related in any way to the actual ChatLLama project that is developed here: <a href=\"https://github.com/nebuly-ai/nebullvm\">https://github.com/<em>nebuly</em>-ai/nebullvm</a> ?"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Show HN: ChatLLaMA \u2013 A ChatGPT style chatbot for Facebook's LLaMA"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://chatllama.baseten.co/"}},"_tags":["comment","author_minxomat","story_35258553"],"author":"minxomat","children":[35264960],"comment_text":"Is this related in any way to the actual ChatLLama project that is developed here: <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm</a> ?","created_at":"2023-03-22T09:43:31Z","created_at_i":1679478211,"objectID":"35258768","parent_id":35258553,"story_id":35258553,"story_title":"Show HN: ChatLLaMA \u2013 A ChatGPT style chatbot for Facebook's LLaMA","story_url":"https://chatllama.baseten.co/","updated_at":"2024-09-20T13:39:23Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"yarapavan"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Some of the available modules include:<p>Speedster: Automatically apply the best set of SOTA optimization techniques to achieve the maximum inference speed-up on your hardware. <a href=\"https://github.com/nebuly-ai/nebullvm/blob/main/apps/accelerate/speedster\">https://github.com/<em>nebuly</em>-ai/nebullvm/blob/main/apps/acceler...</a><p>Nos: Automatically maximize the utilization of GPU resources in a Kubernetes cluster through real-time dynamic partitioning and elastic quotas. <a href=\"https://github.com/nebuly-ai/nos\">https://github.com/<em>nebuly</em>-ai/nos</a><p>ChatLLaMA: Build faster and cheaper ChatGPT-like training process based on LLaMA architectures. <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/chatllama\">https://github.com/<em>nebuly</em>-ai/nebullvm/tree/main/apps/acceler...</a><p>OpenAlphaTensor: Increase the computational performances of an AI model with custom-generated matrix multiplication algorithm fine-tuned for your specific hardware. <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/open_alpha_tensor\">https://github.com/<em>nebuly</em>-ai/nebullvm/tree/main/apps/acceler...</a><p>Forward-Forward: The Forward Forward algorithm is a method for training deep neural networks that replaces the backpropagation forward and backward passes with two forward passes. <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/forward_forward\">https://github.com/<em>nebuly</em>-ai/nebullvm/tree/main/apps/acceler...</a>"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Nebullvm: Plug and play modules to optimize the performances of your AI systems"},"story_url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"https://github.com/<em>nebuly</em>-ai/nebullvm"}},"_tags":["comment","author_yarapavan","story_34981910"],"author":"yarapavan","comment_text":"Some of the available modules include:<p>Speedster: Automatically apply the best set of SOTA optimization techniques to achieve the maximum inference speed-up on your hardware. <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;blob&#x2F;main&#x2F;apps&#x2F;accelerate&#x2F;speedster\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;blob&#x2F;main&#x2F;apps&#x2F;acceler...</a><p>Nos: Automatically maximize the utilization of GPU resources in a Kubernetes cluster through real-time dynamic partitioning and elastic quotas. <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos</a><p>ChatLLaMA: Build faster and cheaper ChatGPT-like training process based on LLaMA architectures. <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;tree&#x2F;main&#x2F;apps&#x2F;accelerate&#x2F;chatllama\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;tree&#x2F;main&#x2F;apps&#x2F;acceler...</a><p>OpenAlphaTensor: Increase the computational performances of an AI model with custom-generated matrix multiplication algorithm fine-tuned for your specific hardware. <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;tree&#x2F;main&#x2F;apps&#x2F;accelerate&#x2F;open_alpha_tensor\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;tree&#x2F;main&#x2F;apps&#x2F;acceler...</a><p>Forward-Forward: The Forward Forward algorithm is a method for training deep neural networks that replaces the backpropagation forward and backward passes with two forward passes. <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;tree&#x2F;main&#x2F;apps&#x2F;accelerate&#x2F;forward_forward\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;tree&#x2F;main&#x2F;apps&#x2F;acceler...</a>","created_at":"2023-03-01T14:00:28Z","created_at_i":1677679228,"objectID":"34982459","parent_id":34981910,"story_id":34981910,"story_title":"Nebullvm: Plug and play modules to optimize the performances of your AI systems","story_url":"https://github.com/nebuly-ai/nebullvm","updated_at":"2024-09-20T13:30:44Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"dwheeler"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"The nebullvm projrct and specifically chatllama, by themselves, are clearly OSS. They're licensed under Apache-2.0:\n<a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/chatllama\">https://github.com/<em>nebuly</em>-ai/nebullvm/tree/main/apps/acceler...</a><p>However, many models including LLaMA are <i>not</i> OSS (they have terms such as &quot;non-commercial use only&quot;). And you need models.<p>It can be confusing, it's important to try to make that clear."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Open source implementation for LLaMA-based ChatGPT"},"story_url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"https://github.com/<em>nebuly</em>-ai/nebullvm/tree/main/apps/accelerate/chatllama"}},"_tags":["comment","author_dwheeler","story_34956807"],"author":"dwheeler","comment_text":"The nebullvm projrct and specifically chatllama, by themselves, are clearly OSS. They&#x27;re licensed under Apache-2.0:\n<a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;tree&#x2F;main&#x2F;apps&#x2F;accelerate&#x2F;chatllama\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nebullvm&#x2F;tree&#x2F;main&#x2F;apps&#x2F;acceler...</a><p>However, many models including LLaMA are <i>not</i> OSS (they have terms such as &quot;non-commercial use only&quot;). And you need models.<p>It can be confusing, it&#x27;s important to try to make that clear.","created_at":"2023-02-27T20:20:42Z","created_at_i":1677529242,"objectID":"34961372","parent_id":34957423,"story_id":34956807,"story_title":"Open source implementation for LLaMA-based ChatGPT","story_url":"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/chatllama","updated_at":"2024-09-20T13:28:24Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"georgehill"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"i am not the author of the repo, i came across this on github.<p><a href=\"https://twitter.com/nebuly_ai\" rel=\"nofollow\">https://twitter.com/<em>nebuly</em>_ai</a> is the author of the repo."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Open source implementation for LLaMA-based ChatGPT"},"story_url":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"https://github.com/<em>nebuly</em>-ai/nebullvm/tree/main/apps/accelerate/chatllama"}},"_tags":["comment","author_georgehill","story_34956807"],"author":"georgehill","comment_text":"i am not the author of the repo, i came across this on github.<p><a href=\"https:&#x2F;&#x2F;twitter.com&#x2F;nebuly_ai\" rel=\"nofollow\">https:&#x2F;&#x2F;twitter.com&#x2F;nebuly_ai</a> is the author of the repo.","created_at":"2023-02-27T17:40:31Z","created_at_i":1677519631,"objectID":"34959438","parent_id":34958709,"story_id":34956807,"story_title":"Open source implementation for LLaMA-based ChatGPT","story_url":"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/chatllama","updated_at":"2024-09-20T13:28:11Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"michele_zanotti"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Hi HN! I\u2019m Michele Zanotti and today I\u2019m releasing nos, an open-source module to efficiently run GPU workloads on Kubernetes!<p>Nos is meant to increase GPU utilization and cut down infrastructure and operational costs providing 2 main features:<p>1. Dynamic GPU Partitioning: you can think of this as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia's MPS and MIG [1,2], finally making them dynamic.<p>2. Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them.<p><a href=\"https://github.com/nebuly-ai/nos\">https://github.com/<em>nebuly</em>-ai/nos</a><p>What is your thoughts on this project? :)<p>Nos addresses some key challenges of Kubernetes tied to the fact that Kubernetes was not designed to support GPU and AI / machine learning workloads. In Kubernetes, GPUs are managed with [3] Nvidia k8s Device Plugin, which has a few major downsides. First, it requires the allocation of an integer number of GPUs per workload, not allowing workloads to request only fractions of GPU. Second, when enabling GPU shared access either with time-slicing or MIG, the device plugin advertises to Kubernetes a fixed set of GPU resources that do not dynamically adapt to the requests of the Pods at each time.<p>This often leads to both underutilized GPUs and pending Pods, and/or the cluster admin having to spend a lot of time looking for workarounds to make the best use of GPUs.<p>For example, consider a company with a k8s cluster with 20 GPUs, where 3 of these GPUs have been reserved for  the data science team using Resource Quota objects. In most cases, the workloads of data scientists (notebooks, scripts, etc.) require much less memory/compute resources than those of an entire GPU, yet Kubernetes will force each container to consume an entire GPU. Also, if the team once needs to run a heavy workload, it may want to use as many resources as possible. However, the Resource Quota over their namespace would constrain the team to use at most the 3 GPUs reserved for them, even if the company cluster may be full of unused GPUs!<p>Instead, with nos the data science team would use nos Dynamic GPU Partitioning to request GPU slices so that many workloads can share the same GPU. Also, Elastic Resource Quotas would allow the team to consume more than the 3 reserved GPUs, borrowing quotas from other teams that are not using them. To recap, the team would be able to launch more Pods and the company would likely need fewer nodes. All this with minimal effort required by the cluster admin, who only has to set up nos.<p>Let me know what you think of nos, feedback would be very helpful! :) And please leave a star on GitHub if you like this opensource <a href=\"https://github.com/nebuly-ai/nos\">https://github.com/<em>nebuly</em>-ai/nos</a><p>Here are some other links that may be useful<p>- Tutorial on how to use Dynamic GPU Partitioning with Nvidia MIG <a href=\"https://towardsdatascience.com/dynamic-mig-partitioning-in-kubernetes-89db6cdde7a3\" rel=\"nofollow\">https://towardsdatascience.com/dynamic-mig-partitioning-in-k...</a><p>- Pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https://docs.nebuly.com/nos/dynamic-gpu-partitioning/partitioning-modes-comparison/\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/dynamic-gpu-partitioning/partiti...</a><p>- Nos documentation <a href=\"https://docs.nebuly.com/nos/overview/\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/overview/</a><p>References<p>[1] Multi-Process Service (MPS) <a href=\"https://docs.nvidia.com/deploy/mps/index.html\" rel=\"nofollow\">https://docs.nvidia.com/deploy/mps/index.html</a><p>[2] Multi-Instance GPU (MIG) <a href=\"https://docs.nvidia.com/datacenter/tesla/mig-user-guide/\" rel=\"nofollow\">https://docs.nvidia.com/datacenter/tesla/mig-user-guide/</a><p>[3] NVIDIA k8s Device Plugin <a href=\"https://github.com/NVIDIA/k8s-device-plugin\">https://github.com/NVIDIA/k8s-device-plugin</a>"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"[dead]"}},"_tags":["comment","author_michele_zanotti","story_34724025"],"author":"michele_zanotti","comment_text":"Hi HN! I\u2019m Michele Zanotti and today I\u2019m releasing nos, an open-source module to efficiently run GPU workloads on Kubernetes!<p>Nos is meant to increase GPU utilization and cut down infrastructure and operational costs providing 2 main features:<p>1. Dynamic GPU Partitioning: you can think of this as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia&#x27;s MPS and MIG [1,2], finally making them dynamic.<p>2. Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them.<p><a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos</a><p>What is your thoughts on this project? :)<p>Nos addresses some key challenges of Kubernetes tied to the fact that Kubernetes was not designed to support GPU and AI &#x2F; machine learning workloads. In Kubernetes, GPUs are managed with [3] Nvidia k8s Device Plugin, which has a few major downsides. First, it requires the allocation of an integer number of GPUs per workload, not allowing workloads to request only fractions of GPU. Second, when enabling GPU shared access either with time-slicing or MIG, the device plugin advertises to Kubernetes a fixed set of GPU resources that do not dynamically adapt to the requests of the Pods at each time.<p>This often leads to both underutilized GPUs and pending Pods, and&#x2F;or the cluster admin having to spend a lot of time looking for workarounds to make the best use of GPUs.<p>For example, consider a company with a k8s cluster with 20 GPUs, where 3 of these GPUs have been reserved for  the data science team using Resource Quota objects. In most cases, the workloads of data scientists (notebooks, scripts, etc.) require much less memory&#x2F;compute resources than those of an entire GPU, yet Kubernetes will force each container to consume an entire GPU. Also, if the team once needs to run a heavy workload, it may want to use as many resources as possible. However, the Resource Quota over their namespace would constrain the team to use at most the 3 GPUs reserved for them, even if the company cluster may be full of unused GPUs!<p>Instead, with nos the data science team would use nos Dynamic GPU Partitioning to request GPU slices so that many workloads can share the same GPU. Also, Elastic Resource Quotas would allow the team to consume more than the 3 reserved GPUs, borrowing quotas from other teams that are not using them. To recap, the team would be able to launch more Pods and the company would likely need fewer nodes. All this with minimal effort required by the cluster admin, who only has to set up nos.<p>Let me know what you think of nos, feedback would be very helpful! :) And please leave a star on GitHub if you like this opensource <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos</a><p>Here are some other links that may be useful<p>- Tutorial on how to use Dynamic GPU Partitioning with Nvidia MIG <a href=\"https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-kubernetes-89db6cdde7a3\" rel=\"nofollow\">https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-k...</a><p>- Pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partitioning-modes-comparison&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partiti...</a><p>- Nos documentation <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;</a><p>References<p>[1] Multi-Process Service (MPS) <a href=\"https:&#x2F;&#x2F;docs.nvidia.com&#x2F;deploy&#x2F;mps&#x2F;index.html\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nvidia.com&#x2F;deploy&#x2F;mps&#x2F;index.html</a><p>[2] Multi-Instance GPU (MIG) <a href=\"https:&#x2F;&#x2F;docs.nvidia.com&#x2F;datacenter&#x2F;tesla&#x2F;mig-user-guide&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nvidia.com&#x2F;datacenter&#x2F;tesla&#x2F;mig-user-guide&#x2F;</a><p>[3] NVIDIA k8s Device Plugin <a href=\"https:&#x2F;&#x2F;github.com&#x2F;NVIDIA&#x2F;k8s-device-plugin\">https:&#x2F;&#x2F;github.com&#x2F;NVIDIA&#x2F;k8s-device-plugin</a>","created_at":"2023-02-09T14:10:07Z","created_at_i":1675951807,"objectID":"34724085","parent_id":34724025,"story_id":34724025,"story_title":"[dead]","updated_at":"2024-09-20T13:12:31Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"michele_zanotti"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Hi HN, I'm excited to share my latest Towards Data Science article on how to take advantage of Dynamic MIG Partitioning to efficiently run AI workloads on Kubernetes<p>Excited to share my latest Towards Data Science article on how to take advantage of Dynamic MIG Partitioning to efficiently run AI workloads on Kubernetes<p>MIG Partitioning is a way to divide GPU resources into smaller slices. This allows Pods to be scheduled only on the memory/compute resources they actually need, thus increasing GPU utilization and reducing infrastructure costs in Kubernetes clusters.<p>In this article, I discuss how the NVIDIA GPU Operator supports Multi-Instance GPU (MIG) in Kubernetes by exposing isolated GPU partitions as resources. I then present the limitations of static MIG configurations that can lead to inefficiencies and impracticality for cluster admins.<p>Finally, the article introduces Dynamic MIG Partitioning as a more effective solution to manage MIG configurations, improving GPU utilization while reducing the burden on cluster admins.<p>Dynamic GPU partitioning is a component of nos,  an opensource I developed to maximize GPU utilization by dynamically partitioning GPUs with NVIDIA MIG and MPS and leveraging Elastic Quotas.\n<a href=\"https://github.com/nebuly-ai/nos\">https://github.com/<em>nebuly</em>-ai/nos</a><p>I hope this will be helpful to you as well. And let me know your feedback on this solution!"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"[dead]"}},"_tags":["comment","author_michele_zanotti","story_34681544"],"author":"michele_zanotti","comment_text":"Hi HN, I&#x27;m excited to share my latest Towards Data Science article on how to take advantage of Dynamic MIG Partitioning to efficiently run AI workloads on Kubernetes<p>Excited to share my latest Towards Data Science article on how to take advantage of Dynamic MIG Partitioning to efficiently run AI workloads on Kubernetes<p>MIG Partitioning is a way to divide GPU resources into smaller slices. This allows Pods to be scheduled only on the memory&#x2F;compute resources they actually need, thus increasing GPU utilization and reducing infrastructure costs in Kubernetes clusters.<p>In this article, I discuss how the NVIDIA GPU Operator supports Multi-Instance GPU (MIG) in Kubernetes by exposing isolated GPU partitions as resources. I then present the limitations of static MIG configurations that can lead to inefficiencies and impracticality for cluster admins.<p>Finally, the article introduces Dynamic MIG Partitioning as a more effective solution to manage MIG configurations, improving GPU utilization while reducing the burden on cluster admins.<p>Dynamic GPU partitioning is a component of nos,  an opensource I developed to maximize GPU utilization by dynamically partitioning GPUs with NVIDIA MIG and MPS and leveraging Elastic Quotas.\n<a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos</a><p>I hope this will be helpful to you as well. And let me know your feedback on this solution!","created_at":"2023-02-06T18:10:17Z","created_at_i":1675707017,"objectID":"34681545","parent_id":34681544,"story_id":34681544,"story_title":"[dead]","updated_at":"2024-09-20T13:18:27Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"zanotti"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Hi HN, my name is Michele Zanotti and I\u2019ve just released an open-source called nos <a href=\"https://github.com/nebuly-ai/nos\">https://github.com/<em>nebuly</em>-ai/nos</a> that enables to finally make GPU partitioning dynamic on Kubernetes! (and much more) You can think of the Dynamic GPU Partitioning as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia MIG, as well as the less known MPS, finally making them dynamic.\nOn Towards Data Science you can find a tutorial on how to use the Dynamic GPU Partitioning <a href=\"https://towardsdatascience.com/dynamic-mig-partitioning-in-k\" rel=\"nofollow\">https://towardsdatascience.com/dynamic-mig-partitioning-in-k</a>...<p>I also wrote a review of the pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https://docs.nebuly.com/nos/dynamic-gpu-partitioning/partiti\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/dynamic-gpu-partitioning/partiti</a>...<p>In addition, this open source also has other components to increase GPU utilization even more. It\u2019s called Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them. It\u2019s better described in nos documentation <a href=\"https://docs.nebuly.com/nos/overview/\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/overview/</a><p>Let me know your thoughts on the project!"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"[dead]"}},"_tags":["comment","author_zanotti","story_34608733"],"author":"zanotti","comment_text":"Hi HN, my name is Michele Zanotti and I\u2019ve just released an open-source called nos <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos</a> that enables to finally make GPU partitioning dynamic on Kubernetes! (and much more) You can think of the Dynamic GPU Partitioning as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia MIG, as well as the less known MPS, finally making them dynamic.\nOn Towards Data Science you can find a tutorial on how to use the Dynamic GPU Partitioning <a href=\"https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-k\" rel=\"nofollow\">https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-k</a>...<p>I also wrote a review of the pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partiti\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partiti</a>...<p>In addition, this open source also has other components to increase GPU utilization even more. It\u2019s called Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them. It\u2019s better described in nos documentation <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;</a><p>Let me know your thoughts on the project!","created_at":"2023-02-01T10:31:35Z","created_at_i":1675247495,"objectID":"34608734","parent_id":34608733,"story_id":34608733,"story_title":"[dead]","updated_at":"2024-09-20T13:10:39Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"michele_zanotti"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Hi HN, my name is Michele Zanotti and I\u2019ve just released an open-source called nos <a href=\"https://github.com/nebuly-ai/nos\">https://github.com/<em>nebuly</em>-ai/nos</a> that enables to finally make GPU partitioning dynamic on Kubernetes! (and much more)\nYou can think of the Dynamic GPU Partitioning as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia MIG, as well as the less known MPS, finally making them dynamic.<p>On Towards Data Science you can find a tutorial on how to use the Dynamic GPU Partitioning <a href=\"https://towardsdatascience.com/dynamic-mig-partitioning-in-k\" rel=\"nofollow\">https://towardsdatascience.com/dynamic-mig-partitioning-in-k</a>...<p>I also wrote a review of the pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https://docs.nebuly.com/nos/dynamic-gpu-partitioning/partiti\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/dynamic-gpu-partitioning/partiti</a>...<p>In addition, this open source also has other components to increase GPU utilization even more. It\u2019s called Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them. It\u2019s better described in nos documentation <a href=\"https://docs.nebuly.com/nos/overview/\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/overview/</a><p>Let me know your thoughts on the project!"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"[dead]"}},"_tags":["comment","author_michele_zanotti","story_34597137"],"author":"michele_zanotti","comment_text":"Hi HN, my name is Michele Zanotti and I\u2019ve just released an open-source called nos <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos</a> that enables to finally make GPU partitioning dynamic on Kubernetes! (and much more)\nYou can think of the Dynamic GPU Partitioning as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia MIG, as well as the less known MPS, finally making them dynamic.<p>On Towards Data Science you can find a tutorial on how to use the Dynamic GPU Partitioning <a href=\"https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-k\" rel=\"nofollow\">https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-k</a>...<p>I also wrote a review of the pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partiti\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partiti</a>...<p>In addition, this open source also has other components to increase GPU utilization even more. It\u2019s called Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them. It\u2019s better described in nos documentation <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;</a><p>Let me know your thoughts on the project!","created_at":"2023-01-31T16:19:56Z","created_at_i":1675181996,"objectID":"34597138","parent_id":34597137,"story_id":34597137,"story_title":"[dead]","updated_at":"2024-09-20T13:17:39Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"michele_zanotti"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Hi HN, my name is Michele Zanotti and I\u2019ve just released an open-source called nos <a href=\"https://github.com/nebuly-ai/nos\">https://github.com/<em>nebuly</em>-ai/nos</a> that enables to finally make GPU partitioning dynamic on Kubernetes! (and much more)<p>You can think of the Dynamic GPU Partitioning as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia MIG, as well as the less known MPS, finally making them dynamic.<p>On Towards Data Science you can find a tutorial on how to use the Dynamic GPU Partitioning\n<a href=\"https://towardsdatascience.com/dynamic-mig-partitioning-in-kubernetes-89db6cdde7a3\" rel=\"nofollow\">https://towardsdatascience.com/dynamic-mig-partitioning-in-k...</a><p>I also wrote a review of the pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https://docs.nebuly.com/nos/dynamic-gpu-partitioning/partitioning-modes-comparison/\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/dynamic-gpu-partitioning/partiti...</a><p>In addition, this open source also has other components to increase GPU utilization even more. It\u2019s called Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them. It\u2019s better described in nos documentation <a href=\"https://docs.nebuly.com/nos/overview/\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/overview/</a><p>Let me know your thoughts on the project!"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"[dead]"}},"_tags":["comment","author_michele_zanotti","story_34593875"],"author":"michele_zanotti","comment_text":"Hi HN, my name is Michele Zanotti and I\u2019ve just released an open-source called nos <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos</a> that enables to finally make GPU partitioning dynamic on Kubernetes! (and much more)<p>You can think of the Dynamic GPU Partitioning as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia MIG, as well as the less known MPS, finally making them dynamic.<p>On Towards Data Science you can find a tutorial on how to use the Dynamic GPU Partitioning\n<a href=\"https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-kubernetes-89db6cdde7a3\" rel=\"nofollow\">https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-k...</a><p>I also wrote a review of the pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partitioning-modes-comparison&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partiti...</a><p>In addition, this open source also has other components to increase GPU utilization even more. It\u2019s called Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them. It\u2019s better described in nos documentation <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;</a><p>Let me know your thoughts on the project!","created_at":"2023-01-31T11:36:05Z","created_at_i":1675164965,"objectID":"34593876","parent_id":34593875,"story_id":34593875,"story_title":"[dead]","updated_at":"2024-09-20T13:17:17Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"emilecourth"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["nebuly"],"value":"Hi HN, my name is Michele Zanotti and I\u2019ve just released an open-source called nos <a href=\"https://github.com/nebuly-ai/nos\">https://github.com/<em>nebuly</em>-ai/nos</a> that enables to finally make GPU partitioning dynamic on Kubernetes! (and much more)<p>You can think of the Dynamic GPU Partitioning as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia MIG, as well as the less known MPS, finally making them dynamic.<p>On Towards Data Science you can find a tutorial on how to use the Dynamic GPU Partitioning\n<a href=\"https://towardsdatascience.com/dynamic-mig-partitioning-in-kubernetes-89db6cdde7a3\" rel=\"nofollow\">https://towardsdatascience.com/dynamic-mig-partitioning-in-k...</a><p>I also wrote a review of the pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https://docs.nebuly.com/nos/dynamic-gpu-partitioning/partitioning-modes-comparison/\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/dynamic-gpu-partitioning/partiti...</a><p>In addition, this open source also has other components to increase GPU utilization even more. It\u2019s called Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them. It\u2019s better described in nos documentation <a href=\"https://docs.nebuly.com/nos/overview/\" rel=\"nofollow\">https://docs.<em>nebuly</em>.com/nos/overview/</a><p>Let me know your thoughts on the project!"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"[dead]"}},"_tags":["comment","author_emilecourth","story_34593849"],"author":"emilecourth","comment_text":"Hi HN, my name is Michele Zanotti and I\u2019ve just released an open-source called nos <a href=\"https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos\">https:&#x2F;&#x2F;github.com&#x2F;nebuly-ai&#x2F;nos</a> that enables to finally make GPU partitioning dynamic on Kubernetes! (and much more)<p>You can think of the Dynamic GPU Partitioning as a cluster autoscaler for GPUs. Instead of scaling up the number of nodes and GPUs, it dynamically partitions them into smaller \u201cGPU slices\u201d. This ensures that each workload only uses the GPU resources it actually needs, resulting in spare GPU capacity that could be used for other workloads. To partition GPUs, nos leverages Nvidia MIG, as well as the less known MPS, finally making them dynamic.<p>On Towards Data Science you can find a tutorial on how to use the Dynamic GPU Partitioning\n<a href=\"https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-kubernetes-89db6cdde7a3\" rel=\"nofollow\">https:&#x2F;&#x2F;towardsdatascience.com&#x2F;dynamic-mig-partitioning-in-k...</a><p>I also wrote a review of the pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partitioning-modes-comparison&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;dynamic-gpu-partitioning&#x2F;partiti...</a><p>In addition, this open source also has other components to increase GPU utilization even more. It\u2019s called Elastic Resource Quota management: it allows to increase the number of Pods running on the cluster by allowing teams (namespaces) to borrow quotas of reserved resources from other teams as long as they are not using them. It\u2019s better described in nos documentation <a href=\"https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;\" rel=\"nofollow\">https:&#x2F;&#x2F;docs.nebuly.com&#x2F;nos&#x2F;overview&#x2F;</a><p>Let me know your thoughts on the project!","created_at":"2023-01-31T11:31:49Z","created_at_i":1675164709,"objectID":"34593850","parent_id":34593849,"story_id":34593849,"story_title":"[dead]","updated_at":"2024-09-20T13:17:17Z"}],"hitsPerPage":20,"nbHits":15588,"nbPages":50,"page":0,"params":"query=nebuly&advancedSyntax=true&analyticsTags=backend","processingTimeMS":16,"processingTimingsMS":{"_request":{"roundTrip":23},"afterFetch":{"format":{"highlighting":2,"total":2},"merge":{"mergeLoop":{"prepareNextHit":11,"total":11},"total":11},"total":11},"fetch":{"query":1,"scanning":1,"total":3},"total":16},"query":"nebuly","serverTimeMS":19}
