[1] https://arxiv.org/pdf/2308.15996v1.pdf\n[2] https://www.adept.ai/blog/fuyu-8b\n[3] If you don't want to search for the figures I made a tiny post about it on my weblog: https://weblog.snats.xyz/posts/2024/02/16/","title":null,"type":"comment","url":null},{"author":"vissidarte_choi","children":[],"created_at":"2024-04-02T04:45:24.000Z","created_at_i":1712033124,"id":39902470,"options":[],"parent_id":39897959,"points":null,"story_id":39896923,"text":"RAGFlow uses Yolov8 for its OCR/layout recognition/TSR(table structure recognition). And RAGFlow uses large amount private data to train these models for them to perform well in some specialized scenarios.","title":null,"type":"comment","url":null},{"author":"yingfeng","children":[],"created_at":"2024-04-02T05:25:58.000Z","created_at_i":1712035558,"id":39902662,"options":[],"parent_id":39897959,"points":null,"story_id":39896923,"text":"We've used YOLOv8 as the object detection model, and use some public datasets, such as PubTable, CDLA, together with some private data to train the model. The model on Huggingface is the one trained using public dataset, and we would open this work later. We use YOLOv8 just because we want to let the document parser run without GPU, I think you could also try any other object detection models such as Detectron, and use the public datasets to train the model as well. We've not used transfomers for this task, because given limited data, it could not outperform traditional CNN based models.","title":null,"type":"comment","url":null}],"created_at":"2024-04-01T19:14:19.000Z","created_at_i":1711998859,"id":39897959,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"Apparently "deep document understanding" refers to OCR and structured document parsing: https://github.com/infiniflow/ragflow/blob/main/deepdoc/READ...
Since "deep document understanding" is not a term of art, I would have just said "OCR and document parsing".
How well does it work? Please include benchmarks. You may be interested in
https://paperswithcode.com/sota/optical-character-recognitio...
https://paperswithcode.com/task/document-layout-analysis
The models seem to be closed source, hosted here: https://huggingface.co/InfiniFlow/deepdoc","title":null,"type":"comment","url":null},{"author":"gardenfelder","children":[{"author":"shekhar101","children":[{"author":"gardenfelder","children":[],"created_at":"2024-04-02T14:42:12.000Z","created_at_i":1712068932,"id":39906277,"options":[],"parent_id":39898516,"points":null,"story_id":39896923,"text":"https://github.com/BerriAI/liteLLM-proxy","title":null,"type":"comment","url":null}],"created_at":"2024-04-01T20:02:19.000Z","created_at_i":1712001739,"id":39898516,"options":[],"parent_id":39898012,"points":null,"story_id":39896923,"text":"It's trivial to run a proxy server that routes all OpenAi calls to another LLM, even local ones. See litellm-proxy.","title":null,"type":"comment","url":null},{"author":"_akhe","children":[{"author":"gardenfelder","children":[{"author":"_akhe","children":[],"created_at":"2024-04-03T19:47:59.000Z","created_at_i":1712173679,"id":39922164,"options":[],"parent_id":39906414,"points":null,"story_id":39896923,"text":"https://github.com/infiniflow/ragflow/blob/main/rag/llm/chat...","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T14:53:57.000Z","created_at_i":1712069637,"id":39906414,"options":[],"parent_id":39899361,"points":null,"story_id":39896923,"text":"where do you see that?","title":null,"type":"comment","url":null}],"created_at":"2024-04-01T21:08:36.000Z","created_at_i":1712005716,"id":39899361,"options":[],"parent_id":39898012,"points":null,"story_id":39896923,"text":"I see a `LocalLLM` chat model where it looks like you can pass a host/port (for example, ollama's)","title":null,"type":"comment","url":null},{"author":"rosspackard","children":[],"created_at":"2024-04-01T23:08:31.000Z","created_at_i":1712012911,"id":39900457,"options":[],"parent_id":39898012,"points":null,"story_id":39896923,"text":"Looks like they do but aren't really documented yet:
https://github.com/infiniflow/ragflow/pull/119","title":null,"type":"comment","url":null},{"author":"vissidarte_choi","children":[],"created_at":"2024-04-02T04:48:08.000Z","created_at_i":1712033288,"id":39902485,"options":[],"parent_id":39898012,"points":null,"story_id":39896923,"text":"RAGFlow will support more LLMs, including locally deployed LLMs.","title":null,"type":"comment","url":null}],"created_at":"2024-04-01T19:20:12.000Z","created_at_i":1711999212,"id":39898012,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"It seems to be limited to certain LLM servers, on of which is OpenAI, none of which includes e.g. Mystral and popular OSS LLMs.
I wonder if that will change - eventually.
Discord channels are named in Chinese, though there are English posts.","title":null,"type":"comment","url":null},{"author":"_akhe","children":[{"author":"vissidarte_choi","children":[],"created_at":"2024-04-02T02:45:43.000Z","created_at_i":1712025943,"id":39901893,"options":[],"parent_id":39899351,"points":null,"story_id":39896923,"text":"Hi bschmidt1, This is a good feature. We do plan to support it soon. Please stay tuned. If you have further suggesions, welcome to file an issue with us.","title":null,"type":"comment","url":null}],"created_at":"2024-04-01T21:07:48.000Z","created_at_i":1712005668,"id":39899351,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"Is there a JavaScript library? Both LlamaIndex and Langchain have nice JS/TS packages on npm. Could thinly wrap a JS client around this Python API but the community aspect of having an official library is nice.
Also might be helpful to have a simple example on the README showing how to fetch a document and start querying it. I would try it!","title":null,"type":"comment","url":null},{"author":"NKosmatos","children":[{"author":"rosspackard","children":[],"created_at":"2024-04-01T23:08:14.000Z","created_at_i":1712012894,"id":39900451,"options":[],"parent_id":39900324,"points":null,"story_id":39896923,"text":"Looks like they do but aren't really documented yet:
https://github.com/infiniflow/ragflow/pull/119","title":null,"type":"comment","url":null},{"author":"vissidarte_choi","children":[],"created_at":"2024-04-02T04:51:53.000Z","created_at_i":1712033513,"id":39902496,"options":[],"parent_id":39900324,"points":null,"story_id":39896923,"text":"To be honest, RAGFlow already supports this but has not documented this local deployment process yet, as we are still working on simplifying this process, and will release this feature soon. Please keep tuned!","title":null,"type":"comment","url":null}],"created_at":"2024-04-01T22:50:56.000Z","created_at_i":1712011856,"id":39900324,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"If only they supported local LLMs out of the box. I have a very specific use case buy it needs to run locally offline only.\nAny suggestions/recommendations from fellow HN users are more than welcomed :-)","title":null,"type":"comment","url":null},{"author":"mpeg","children":[{"author":"shekhar101","children":[{"author":"mpeg","children":[{"author":"shekhar101","children":[],"created_at":"2024-04-02T01:44:00.000Z","created_at_i":1712022240,"id":39901507,"options":[],"parent_id":39901031,"points":null,"story_id":39896923,"text":"Thank you very much!","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T00:23:39.000Z","created_at_i":1712017419,"id":39901031,"options":[],"parent_id":39900759,"points":null,"story_id":39896923,"text":"it's https://huggingface.co/InfiniFlow/deepdoc and the code for usage is in https://github.com/infiniflow/ragflow/blob/main/deepdoc/READ... \u2013 it took me a bit of trial and error to get it working
It seems to be a YOLOv8 fine-tune, I only did a couple tests but results were decent. Another model that is supposed to be fine tuned for borderless is https://huggingface.co/keremberke/yolov8m-table-extraction but I haven't had great results myself with it, but maybe worth a try for you.","title":null,"type":"comment","url":null},{"author":"thegeomaster","children":[],"created_at":"2024-04-02T22:57:20.000Z","created_at_i":1712098640,"id":39911760,"options":[],"parent_id":39900759,"points":null,"story_id":39896923,"text":"Here's a quick test to run: if you have Windows and MS Office, File->Open your PDF and report the results. You might be surprised at the layout extraction quality.","title":null,"type":"comment","url":null}],"created_at":"2024-04-01T23:44:46.000Z","created_at_i":1712015086,"id":39900759,"options":[],"parent_id":39900338,"points":null,"story_id":39896923,"text":"What's the name of the layout recorgniser model? I did not have a good experience extracting layout from tables, especially those without column boundaries (space instead of lines to demarcate boundaries)","title":null,"type":"comment","url":null},{"author":"vissidarte_choi","children":[],"created_at":"2024-04-02T05:07:11.000Z","created_at_i":1712034431,"id":39902572,"options":[],"parent_id":39900338,"points":null,"story_id":39896923,"text":"This is because PDF has so many different versions. A third-party tools like pdfplumber won't fit it all. For example, using pdfplumber to parse some PDFs will cause the system to raise exceptions. Sometimes fitz works in situations where pdfplumber won't handle well. It looks a bit complicated, but RAGFlow is using multiple parsing tools to handle different types of PDFs.","title":null,"type":"comment","url":null}],"created_at":"2024-04-01T22:52:13.000Z","created_at_i":1712011933,"id":39900338,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"Took me some time to figure out how to run it, but the layout recogniser model hosted on huggingface is pretty good!
It correctly identifies tables that even paid models like the AWS Textract Document Analysis API fails to \u2013 for instance tables with one column which often confuse AWS even if they have a clear header and are labelled "Table" in the text.
I would however love to know broadly what kind of document it was trained on, as my results could be pure luck, hard to say without a proper benchmark
Very nice layout recognition, although I can't quite comment on the RAG performance itself \u2013 I think some of the architecture decisions are odd, it mixes a bunch of different PDF parsers for example which will all result in different quality and it's not clear to me which one it defaults to as it seems to be different in different places in the code (the simple parser defaults to pypdf2 which is not a great option)","title":null,"type":"comment","url":null},{"author":"cetra3","children":[{"author":"viraptor","children":[],"created_at":"2024-04-01T23:29:49.000Z","created_at_i":1712014189,"id":39900621,"options":[],"parent_id":39900598,"points":null,"story_id":39896923,"text":"It only depends on the interface. There's a lot of projects which present the openai interface to whatever you want.","title":null,"type":"comment","url":null},{"author":"vissidarte_choi","children":[{"author":"_akhe","children":[],"created_at":"2024-04-03T19:49:41.000Z","created_at_i":1712173781,"id":39922180,"options":[],"parent_id":39902443,"points":null,"story_id":39896923,"text":"Just link them to https://github.com/infiniflow/ragflow/blob/main/rag/llm/chat... :)","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T04:40:28.000Z","created_at_i":1712032828,"id":39902443,"options":[],"parent_id":39900598,"points":null,"story_id":39896923,"text":"Not quite certain about your meaning. Could you be more specific? RAGFlow does not have its own LLM model or souce code. RAGFlow supports API calling from third-party large language model providers, as well as local deployment of these large models. RAGFlow has open-sourced these two parts of codes already.","title":null,"type":"comment","url":null}],"created_at":"2024-04-01T23:26:46.000Z","created_at_i":1712014006,"id":39900598,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"I don't consider this strictly open source if components it depends on (I.e, the LLM) is closed source. I've seen a lot of these Fauxpen source style projects around","title":null,"type":"comment","url":null},{"author":"yding","children":[],"created_at":"2024-04-02T01:20:26.000Z","created_at_i":1712020826,"id":39901356,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"This is really cool! Starred and look forward to seeing how this develops further!","title":null,"type":"comment","url":null},{"author":"zzleeper","children":[{"author":"trenchgun","children":[{"author":"jtr101","children":[],"created_at":"2024-04-02T16:51:37.000Z","created_at_i":1712076697,"id":39907943,"options":[],"parent_id":39903459,"points":null,"story_id":39896923,"text":"Love this counterpoint to "OSS means I can get everybody else to do work for me for free" => "allows you to do the work yourself and share buddy"\nPR or it didn't happen","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T08:20:22.000Z","created_at_i":1712046022,"id":39903459,"options":[],"parent_id":39901380,"points":null,"story_id":39896923,"text":"It is open-source though. Just rip it off and make that PDF() class.","title":null,"type":"comment","url":null},{"author":"demilich","children":[],"created_at":"2024-04-02T09:28:53.000Z","created_at_i":1712050133,"id":39903810,"options":[],"parent_id":39901380,"points":null,"story_id":39896923,"text":"Each project has its own detailed requirements and scenarios, and we cannot demand that each project use same library to implement similar functions","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T01:24:09.000Z","created_at_i":1712021049,"id":39901380,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"I'm partly sad at the approach this and other engines take: reimplement each part (PDF parser, etc etc) in a way where they are pretty much useless except in their specific engine.
If instead we had a PDF() class that did what RAGFlow is doing (dealing with all the different trade-offs of the different python PDF engines such as pdfplumber), then we could easily adapt it and improve it, and it can be useful for other projects as well.","title":null,"type":"comment","url":null},{"author":"bgun","children":[{"author":"scrollaway","children":[{"author":"bgun","children":[{"author":"kaliqt","children":[],"created_at":"2024-04-02T03:25:03.000Z","created_at_i":1712028303,"id":39902080,"options":[],"parent_id":39901725,"points":null,"story_id":39896923,"text":"As a native English speaker I would not dock points from this name for the gender related points as that doesn't even cross my mind, it's not relevant.
Keep gender out of engineering.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T02:18:45.000Z","created_at_i":1712024325,"id":39901725,"options":[],"parent_id":39901601,"points":null,"story_id":39896923,"text":"I didn't point it out because it was funny-ha-ha, but because it's a teachable example of a linguistic mis-step that could be easily resolved by diversifying your reviewers - in this case, since you mention it, perhaps a native English speaker?
I'm not even suggesting that this project needs a new name - although I think if they were naming a consumer-facing product or company, someone in marketing would push back on the name almost immediately.","title":null,"type":"comment","url":null},{"author":"kadoban","children":[],"created_at":"2024-04-02T03:08:19.000Z","created_at_i":1712027299,"id":39902001,"options":[],"parent_id":39901601,"points":null,"story_id":39896923,"text":"> Also you might find this normal but I cannot imagine my female colleagues being amused at me calling one of them up asking to \u201cconsult\u201d about a name solely because she\u2019s a woman.
I'd certainly hope it wouldn't be "solely" because she's a woman. There's no people you'd be interested in input from that happen to be women?
A project name is something you'd want to throw around to a few people (ideally with different perspectives) and make sure it conveys what you intend and has the right tone and such.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T02:00:33.000Z","created_at_i":1712023233,"id":39901601,"options":[],"parent_id":39901514,"points":null,"story_id":39896923,"text":"The authors are not native English speakers (and what you\u2019re seeing in the name is not something most non native English speakers would spot).
I understand it\u2019s funny to point and laugh \u201cha ha dumb dudes didn\u2019t think about something female related because they\u2019re dudes\u201d, but it\u2019s worth remembering that not everyone is from the anglosphere.
Also you might find this normal but I cannot imagine my female colleagues being amused at me calling one of them up asking to \u201cconsult\u201d about a name solely because she\u2019s a woman.","title":null,"type":"comment","url":null},{"author":"hitekker","children":[{"author":"bgun","children":[],"created_at":"2024-04-02T02:31:32.000Z","created_at_i":1712025092,"id":39901812,"options":[],"parent_id":39901737,"points":null,"story_id":39896923,"text":"Not sure if sarcasm but 'about as odd as "Nintendo Wii"' kind of makes my point:
https://web.archive.org/web/20130623080716/http://www.forbes...
http://news.bbc.co.uk/2/hi/technology/4953650.stm
https://www.cbc.ca/news/entertainment/why-wii-ask-nintendo-1...","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T02:20:25.000Z","created_at_i":1712024425,"id":39901737,"options":[],"parent_id":39901514,"points":null,"story_id":39896923,"text":"The assumption that author didn't talk to a woman sounds pretty knee-jerk to me. Can you explain what you mean?
"On the rag" is a semi-common slang for menstruation, but "rag" has way more meanings than just menstrual pads. IMO, as a marketing term, "RAGflow" sounds about as odd as "Nintendo Wii".","title":null,"type":"comment","url":null},{"author":"nmfisher","children":[],"created_at":"2024-04-02T03:00:01.000Z","created_at_i":1712026801,"id":39901956,"options":[],"parent_id":39901514,"points":null,"story_id":39896923,"text":"People said the same thing about the iPad.","title":null,"type":"comment","url":null},{"author":"yingfeng","children":[],"created_at":"2024-04-02T04:40:40.000Z","created_at_i":1712032840,"id":39902447,"options":[],"parent_id":39901514,"points":null,"story_id":39896923,"text":"Hi, buddy, I'm sorry to make you feel like we don't take women's voices into account. We came up with this name because RAG is already a consensus, as an acronym for retrieval augmented generation, RAG is used in many places, and we think it would actually be a standard for LLM oriented B-side scenarios, so that's why we adopted this name.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T01:45:24.000Z","created_at_i":1712022324,"id":39901514,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"I know this is just an open source project but it\u2019s a good example of why you might want to consult a woman before naming things.","title":null,"type":"comment","url":null},{"author":"constantinum","children":[{"author":"TuringNYC","children":[],"created_at":"2024-04-02T04:29:26.000Z","created_at_i":1712032166,"id":39902392,"options":[],"parent_id":39901877,"points":null,"story_id":39896923,"text":"Do you have any recommendations for extraction from Powerpoint documents? Those seem like the worst since each of the layouts tend to be unique (unlike the AMEX Statements in the example)","title":null,"type":"comment","url":null},{"author":"demilich","children":[],"created_at":"2024-04-02T09:30:13.000Z","created_at_i":1712050213,"id":39903817,"options":[],"parent_id":39901877,"points":null,"story_id":39896923,"text":"Cool! This is really helpful.","title":null,"type":"comment","url":null},{"author":"yingfeng","children":[],"created_at":"2024-04-02T09:48:22.000Z","created_at_i":1712051302,"id":39903903,"options":[],"parent_id":39901877,"points":null,"story_id":39896923,"text":"Actually we've tried almost all lof existing open source models for document processing, and none of them performs well for complex documents, especially those having complicated tables, such as tables cells without borders, cells need to be combined,...,etc. Although adopting LLMs to perform such document understanding tasks is more scalable, it requires much more data and computation power to achieve similar results. That's why we design such models start from scratch.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T02:43:00.000Z","created_at_i":1712025780,"id":39901877,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"Document processing is getting better and better with new tools leveraging LLMs. \nIf anyone is interested in exploring this space, try another similar tool LLMWhisperer (https://llmwhisperer.unstract.com/). It is a part of Unstract, an open-source document processing tool (https://github.com/Zipstack/unstract)","title":null,"type":"comment","url":null},{"author":"demilich","children":[],"created_at":"2024-04-02T07:11:14.000Z","created_at_i":1712041874,"id":39903108,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"The recognition of file layout to parse file content is indeed a very innovative idea. Compared to many open-source projects we have seen before, this provides us with new ideas for using RAG to solve problems in the future. I hope the author of this project will continue to update it, so that more people can benefit from it.","title":null,"type":"comment","url":null},{"author":"forrest2","children":[{"author":"yingfeng","children":[],"created_at":"2024-04-02T08:08:27.000Z","created_at_i":1712045307,"id":39903392,"options":[],"parent_id":39903304,"points":null,"story_id":39896923,"text":"Thanks for your nice suggestion. We train the model using YOLO, but during inference, the model is converted into ONNX and we use ONNXRuntime for the model inference. As a result, YOLO itself is not included in the software package. \nWe will open the training code in the repo soon.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T07:52:42.000Z","created_at_i":1712044362,"id":39903304,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"A lot of the yolo stuff from ultralytics is AGPL3 fyi. Recommend caution depending on what code or models / model lineage are used","title":null,"type":"comment","url":null},{"author":"StrayHardball","children":[{"author":"zingiscool","children":[],"created_at":"2024-04-02T08:38:22.000Z","created_at_i":1712047102,"id":39903536,"options":[],"parent_id":39903439,"points":null,"story_id":39896923,"text":"Hello friend,
I would like to recommend the FigJam feature in Figma to you, a common tool used by high-tech companies for creating Information Architecture (IA). Our company always adheres to the tool usage standards of high-tech companies.
For guidance on how to use the FigJam feature in Figma, you can refer to this link: https://youtu.be/axDzyLEfYgU?si=V6tqO_tEUKYuLxrL (or search for FigJam on YouTube).
Here are three quick tips on how to efficiently create Information Architecture\uff08Also called System Architecture\uff09:
Start with the Sketch method by drawing your product's Information architecture on paper. Communicate with your PM, design, and development team to validate its effectiveness.
Open FigJam in Figma to turn it into an electronic version. This step is crucial as design and development will follow the version in FigJam for collaborative work.
When creating your Information Architecture (IA), first identify the core functionalities of your product, represented by one color. Next, determine what sub-functions each core functionality can be divided into, marked by a second color. Finally, decide what detailed functionalities compose each sub-function, indicated by a third color. Continue in this manner until you complete the entire IA construction.
It\u2019s important to note that perfecting the IA is not a linear process; it will go through multiple iterations and modifications. Every great product undergoes this process. Also, initially, you can focus on creating an IA for just one core functional module (usually the innovative feature with the highest user pain point) without defining the entire scope. In essence, flexibly establishing the IA to achieve company goals is the primary task.","title":null,"type":"comment","url":null}],"created_at":"2024-04-02T08:16:18.000Z","created_at_i":1712045778,"id":39903439,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"Does anyone know what software was used to create the "System Architecture" diagram?","title":null,"type":"comment","url":null},{"author":"gardenfelder","children":[],"created_at":"2024-04-02T15:00:55.000Z","created_at_i":1712070055,"id":39906512,"options":[],"parent_id":39896923,"points":null,"story_id":39896923,"text":"I am curious: if I pass a pdf file such as a research report or an open source text book, what will be the result. Does it create a knowledge graph? triples? Thanks in advance for comments.","title":null,"type":"comment","url":null}],"created_at":"2024-04-01T17:50:54.000Z","created_at_i":1711993854,"id":39896923,"options":[],"parent_id":null,"points":230,"story_id":39896923,"text":null,"title":"RAGFlow is an open-source RAG engine based on OCR and document parsing","type":"story","url":"https://github.com/infiniflow/ragflow"}