Replace the `qn("w:ins")` in the example with `qn("w:hyperlink")` and that should hopefully work?","title":null,"type":"comment","url":null}],"created_at":"2023-07-06T20:35:29.000Z","created_at_i":1688675729,"id":36622564,"options":[],"parent_id":36622326,"points":null,"story_id":36616799,"text":"This sounds amazing! Thanks for sharing it, I will try it to see if I can replace it with the main python-docx. For my use case it suffices to have full text of each paragraph (even if it includes a hyperlink) and heading but also be able to have each of them separated when needed.","title":null,"type":"comment","url":null},{"author":"convivialdingo","children":[],"created_at":"2023-07-06T22:11:36.000Z","created_at_i":1688681496,"id":36623886,"options":[],"parent_id":36622326,"points":null,"story_id":36616799,"text":"Hey, that's fantastic. I'll definitely check that out.","title":null,"type":"comment","url":null}],"created_at":"2023-07-06T20:24:03.000Z","created_at_i":1688675043,"id":36622326,"options":[],"parent_id":36621124,"points":null,"story_id":36616799,"text":"Heh, I got a bit into hacking on python-docx last year (the original author seems to be focusing on other things than python-docx now) - I have a fork/branch where I tried to more properly implement external hyperlink functionality (https://github.com/icegreentea/python-docx/pull/7)
I realize now staring at this, that I might have broken API a little. You can't do "text = paragraph.text" anymore, but you can do "text = ''.join([run.text for run in paragraph.runs])" instead.
If you're curious at all why it breaks, it's because in the OOXML spec paragraphs are made up of a ordered list of runs or hyperlinks (and hyperlinks can then contain additional runs). The master branch just implements paragraphs as ordered list of runs (and ignores all hyperlinks).","title":null,"type":"comment","url":null}],"created_at":"2023-07-06T19:03:40.000Z","created_at_i":1688670220,"id":36621124,"options":[],"parent_id":36620904,"points":null,"story_id":36616799,"text":"I tried python-docx with a bunch of docx files (downloaded from Google Docs). It returns empty strings for hyperlinks and I couldn't manage to fix this. So if there is a sentence like "This is an important link to another doc or url." and the "link" is a hyperlink, python-docx returns "This is an important to another doc or url."","title":null,"type":"comment","url":null}],"created_at":"2023-07-06T18:50:07.000Z","created_at_i":1688669407,"id":36620904,"options":[],"parent_id":36618948,"points":null,"story_id":36616799,"text":"I've had good luck with python-docx for reading word documents (typically specifications). Tables are supported - but it's not obvious where the table comes from in the document and I had to come up with a hack way to read image captions.
PDF has been hit or miss, but pypdf has improved in the last couple of years. Depending on the document you'll sometimes get random spaces or nospacesatall.","title":null,"type":"comment","url":null}],"created_at":"2023-07-06T16:52:36.000Z","created_at_i":1688662356,"id":36618948,"options":[],"parent_id":36618877,"points":null,"story_id":36616799,"text":"Do you have any suggestions for Python libraries (other than what's mentioned in the post)?","title":null,"type":"comment","url":null}],"created_at":"2023-07-06T16:48:58.000Z","created_at_i":1688662138,"id":36618877,"options":[],"parent_id":36618056,"points":null,"story_id":36616799,"text":"I second this suggestion. I tested numerous Python tools to extract text - nothing matches Tika for general extraction of just about any data format.
However - if you can expect a certain format beforehand - then Python is better since you can extract higher-quality data (tables, lists) with the appropriate tool.","title":null,"type":"comment","url":null},{"author":"mcswell","children":[],"created_at":"2023-07-06T23:08:46.000Z","created_at_i":1688684926,"id":36624564,"options":[],"parent_id":36618056,"points":null,"story_id":36616799,"text":"Tika can be used as a library in Python: https://pypi.org/project/tika/","title":null,"type":"comment","url":null},{"author":"ramraj07","children":[{"author":"mbwgh","children":[{"author":"mcswell","children":[],"created_at":"2023-07-07T15:49:43.000Z","created_at_i":1688744983,"id":36633533,"options":[],"parent_id":36629257,"points":null,"story_id":36616799,"text":"There's a book called "Tika in action" which I found useful.","title":null,"type":"comment","url":null}],"created_at":"2023-07-07T09:28:31.000Z","created_at_i":1688722111,"id":36629257,"options":[],"parent_id":36627438,"points":null,"story_id":36616799,"text":"I second this, there is absolutely no easily discoverable entry point to the documentation.\nIn the end if you want to get a feeling of what this is you search for "tika tutorial" and get a rough idea via (in my case) some medium article I guess.","title":null,"type":"comment","url":null},{"author":"mcswell","children":[],"created_at":"2023-07-07T15:48:12.000Z","created_at_i":1688744892,"id":36633523,"options":[],"parent_id":36627438,"points":null,"story_id":36616799,"text":"I can't speak to the Apache documentation, but I once had the task of extracting plain text from many different document formats: Word, spreadsheets, PDFs, the EXIF information in JPEGs, and so on for a long list. I had written code with calls to extractor libraries for several of these formats, when I can across tika. Out when my if..then..elif..elif..elif.. code, to be replaced with a single (Python) call to tika.
I can't answer your question about pandas, though.","title":null,"type":"comment","url":null}],"created_at":"2023-07-07T05:11:10.000Z","created_at_i":1688706670,"id":36627438,"options":[],"parent_id":36618056,"points":null,"story_id":36616799,"text":"As is customary for all of Apache, I have no clue what I\u2019m looking at after trying to read through the links in that page for ten minutes. Like who is this tool for? When should I use this vs any other competing tools? No clue. I suppose it can read documents of any type and give it out as a dictionary? Why would I use this vs pandas?","title":null,"type":"comment","url":null},{"author":"jghn","children":[],"created_at":"2023-07-07T05:49:28.000Z","created_at_i":1688708968,"id":36627711,"options":[],"parent_id":36618056,"points":null,"story_id":36616799,"text":"I found myself today trying to parse a TSV and substituting a few fields with a different value, then writing the new file out.
Something that perl would excel at, although I used Python. Because Perl isn't as maintainable as Python
I was intrigued by this comment. A JVM solution would also be viable in my tech stack. Would Tika be easier than line processing compiled regexes in Python? I tried looking at the Usage examples but it wasn't clear.","title":null,"type":"comment","url":null}],"created_at":"2023-07-06T16:01:27.000Z","created_at_i":1688659287,"id":36618056,"options":[],"parent_id":36616799,"points":null,"story_id":36616799,"text":"If you accept running Java, the Apache Tika is extremely good at parsing content (https://tika.apache.org/)","title":null,"type":"comment","url":null},{"author":"CShorten","children":[],"created_at":"2023-07-06T16:10:10.000Z","created_at_i":1688659810,"id":36618184,"options":[],"parent_id":36616799,"points":null,"story_id":36616799,"text":"Awesome!","title":null,"type":"comment","url":null},{"author":"oersted","children":[],"created_at":"2023-07-06T16:34:20.000Z","created_at_i":1688661260,"id":36618587,"options":[],"parent_id":36616799,"points":null,"story_id":36616799,"text":"I have been using it extensively during the last few weeks. I've very thankful for such a clean and practical API, and I think it will become the central solution for ingesting heterogeneous text in the Python ecosystem.
However, I'm afraid it is not there yet. Other libraries like PDFMiner give higher quality outputs and specialized libraries like Camelot are still needed to extract tables as reasonably well formatted text. It also needs a lot of extra tooling for web scraping. Sure it can read plain HTML from a URL, but it cannot run JavaScript, or control things like User Agent. It could be argued that such features are not within the scope, but it is rather bothersome for a library that presents a magic `partition` function for most standard text sources.
I'm sure it will get there soon though. It shouldn't be hard to integrate with state-of-the-art parsers and tooling, and the simple API undoubtedly brings a lot of peace of mind.","title":null,"type":"comment","url":null},{"author":"froggychairs","children":[],"created_at":"2023-07-06T18:32:17.000Z","created_at_i":1688668337,"id":36620649,"options":[],"parent_id":36616799,"points":null,"story_id":36616799,"text":"API looks very clean :) I\u2019ve also been avoiding LangChain since it just seems too big for my tastes. Will give this a shot","title":null,"type":"comment","url":null},{"author":"raphman","children":[],"created_at":"2023-07-07T02:59:44.000Z","created_at_i":1688698784,"id":36626533,"options":[],"parent_id":36616799,"points":null,"story_id":36616799,"text":"Another way to parse Markdown, HTML, or docx files would be pandoc [1]:
pandoc --to json file.docx\n import json\n from sh import pandoc\n doc = json.loads( pandoc("file.docx", to="json").stdout )\n {'pandoc-api-version': [1, 22, 2],\n 'meta': {'title': {'t': 'MetaInlines',\n 'c': [{'t': 'Str', 'c': 'The'}, {'t': 'Space'}, {'t': 'Str', 'c': 'Title'}]}},\n 'blocks': [{'t': 'Header',\n 'c': [1,\n ['first-chapter', [], []],\n [{'t': 'Str', 'c': 'First'}, {'t': 'Space'}, {'t': 'Str', 'c': 'Chapter'}]]},\n {'t': 'Para',\n 'c': [{'t': 'Str', 'c': 'I'}, {'t': 'Space'}, {'t': 'Str', 'c': 'like'}, {'t': 'Space'},\n {'t': 'Emph', 'c': [{'t': 'Str', 'c': 'cursive'}]}, {'t': 'Space'}, {'t': 'Str', 'c': 'or'}, \n {'t': 'Space'}, {'t': 'Strong', 'c': [{'t': 'Str', 'c': 'bold'}]}, {'t': 'Space'},\n {'t': 'Str', 'c': 'text.'}]},\n {'t': 'Para',\n 'c': [{'t': 'Str', 'c': 'Here'}, {'t': 'Space'}, {'t': 'Str', 'c': 'is'}, {'t': 'Space'},\n {'t': 'Str', 'c': 'a'}, {'t': 'Space'}, {'t': 'Link',\n 'c': [['', [], []], [{'t': 'Str', 'c': 'link'}], ['https://ix.de/', '']]},\n {'t': 'Str', 'c': '.'}]},\n {'t': 'BulletList',\n 'c': [[{'t': 'Para', 'c': [{'t': 'Str', 'c': 'Item'}, {'t': 'Space'}, 't': 'Str', 'c': '1'}]}],\n [{'t': 'Para', 'c': [{'t': 'Str', 'c': 'Item'}, {'t': 'Space'}, {'t': 'Str', 'c': '2'}]}]]}]}\n