The other thing we've encountered is that vectors take up a lot of storage space compared to the normal columns (easily a couple kb per row). You can fill up a db really quickly, especially if you're embedding really small chunks","title":null,"type":"comment","url":null}],"created_at":"2024-06-02T14:31:30.000Z","created_at_i":1717338690,"id":40554472,"options":[],"parent_id":40554171,"points":null,"story_id":40522400,"text":"I have been playing with Milvus but as my use case evolves, I think PGVector may be a better fit . I currently store a lot of enriched data in PG and embeddings in Milvus. Consolidating them into one DB makes sense to me.","title":null,"type":"comment","url":null},{"author":"menacingly","children":[],"created_at":"2024-06-02T14:36:32.000Z","created_at_i":1717338992,"id":40554532,"options":[],"parent_id":40554171,"points":null,"story_id":40522400,"text":"definitely overhyped, but not useless. Consider one of your examples, Elastic. It's often employed in situations where the db could handle what it's doing just fine, but it survives, Largely because of optimizations it is free to make knowing it is targeted at a narrow set of tasks.","title":null,"type":"comment","url":null},{"author":"mewpmewp2","children":[],"created_at":"2024-06-02T15:34:50.000Z","created_at_i":1717342490,"id":40554972,"options":[],"parent_id":40554171,"points":null,"story_id":40522400,"text":"Scaling, accuracy and search quickness is very important for vector dbs. Do other general purpose databases scale as well as specialized ones?
Because ideally they hold massive, well optimised indexes in their memory to be able to search quickly and not miss any vectors.","title":null,"type":"comment","url":null},{"author":"sa-code","children":[{"author":"ukuina","children":[{"author":"sa-code","children":[],"created_at":"2024-06-13T14:47:56.000Z","created_at_i":1718290076,"id":40670388,"options":[],"parent_id":40556881,"points":null,"story_id":40522400,"text":"Disagree on using a managed vector db. That's just the same thing except you're paying someone else money? "Traditional datastore" could mean anything. Info retrieval and search have very established players like the Lucene ecosystem.","title":null,"type":"comment","url":null}],"created_at":"2024-06-02T20:08:48.000Z","created_at_i":1717358928,"id":40556881,"options":[],"parent_id":40555610,"points":null,"story_id":40522400,"text":"This is the right answer. When it's time to make a product, you either use a managed VectorDB or ditch it for a more traditional datastore with careful indexing.","title":null,"type":"comment","url":null}],"created_at":"2024-06-02T17:04:21.000Z","created_at_i":1717347861,"id":40555610,"options":[],"parent_id":40554171,"points":null,"story_id":40522400,"text":"Vector databases are just the easiest way to make a search engine in a demo.","title":null,"type":"comment","url":null}],"created_at":"2024-06-02T13:59:08.000Z","created_at_i":1717336748,"id":40554171,"options":[],"parent_id":40522400,"points":null,"story_id":40522400,"text":"Maybe I don\u2019t have enough \u2018AI\u2019 experience to understand, but I\u2019m not getting the future of vector databases. 90% of the use cases I\u2019ve encountered also benefit from keyword search, faceting, etc. and therefore a more traditional search engine like Elastic, Meilisearch, or even Postgres makes more sense than something that is purely focused on the vector index. At this point every search engine has a solid vector and hybrid search implementation.","title":null,"type":"comment","url":null},{"author":"syntaxfree","children":[{"author":"gkapur","children":[{"author":"mhuffman","children":[{"author":"Kydlaw","children":[],"created_at":"2024-06-02T19:09:20.000Z","created_at_i":1717355360,"id":40556505,"options":[],"parent_id":40556425,"points":null,"story_id":40522400,"text":"They do have an extension for vector similarity search (https://duckdb.org/docs/extensions/vss).
But Milvius might propose more features, as they have been in this specific space for longer.","title":null,"type":"comment","url":null}],"created_at":"2024-06-02T18:55:07.000Z","created_at_i":1717354507,"id":40556425,"options":[],"parent_id":40556260,"points":null,"story_id":40522400,"text":"Doesn't DuckDB already do vector search?","title":null,"type":"comment","url":null},{"author":"valstu","children":[],"created_at":"2024-06-03T08:06:36.000Z","created_at_i":1717401996,"id":40560353,"options":[],"parent_id":40556260,"points":null,"story_id":40522400,"text":"Chroma used DuckDB at some point, might not be the case anymore though","title":null,"type":"comment","url":null}],"created_at":"2024-06-02T18:32:02.000Z","created_at_i":1717353122,"id":40556260,"options":[],"parent_id":40554902,"points":null,"story_id":40522400,"text":"Not really. This is more like SQLite or DuckDB for vector databases (on disk.) Chroma is more like redis for vector databases (in memory.)
We have seen similar products in the olap space, as well, ie. Clickhouse local.","title":null,"type":"comment","url":null}],"created_at":"2024-06-02T15:25:30.000Z","created_at_i":1717341930,"id":40554902,"options":[],"parent_id":40522400,"points":null,"story_id":40522400,"text":"So like chromadb","title":null,"type":"comment","url":null},{"author":"mikl","children":[],"created_at":"2024-06-02T16:11:14.000Z","created_at_i":1717344674,"id":40555217,"options":[],"parent_id":40522400,"points":null,"story_id":40522400,"text":"It\u2019ll be nice when the AI hype settles down a bit, so many of these \u201cre-invent the wheel, with more AI sprinkles\u201d projects popping up.
So many existing DBs can already do vector search, do we really need one dedicated to just that?","title":null,"type":"comment","url":null},{"author":"nutanc","children":[],"created_at":"2024-06-02T17:33:40.000Z","created_at_i":1717349620,"id":40555829,"options":[],"parent_id":40522400,"points":null,"story_id":40522400,"text":"This is awesome. Does Milvus lite also support binary embeddings?","title":null,"type":"comment","url":null}],"created_at":"2024-05-30T11:17:52.000Z","created_at_i":1717067872,"id":40522400,"options":[],"parent_id":null,"points":56,"story_id":40522400,"text":null,"title":"Milvus Lite: The Lightweight Version of Milvus","type":"story","url":"https://milvus.io/blog/introducing-milvus-lite.md"}