Automated-Vietnamese-Phrase-Detecter

Did this phrase exists in this pdf? Hmmm, how bout their synonyms? OHHH, 100 files? How can I check all of them for the existence of my wanted words and phrases? ...

If these are ever your questions, here I have a solution for you!!

Automated-word-matcher can fully automate the process of reading pdf files, finding phrases and words before answering you whether a word/phrase does exist in a given pdf file.

How to use it?

Note: This code only runs on Jupyternotebook. The reason is some pdf blocks access of google colab, making it necessary to download the file. By running code in Jupyternotebook, you do not have to upload the pdf again to google colab

Setup

• pip install: PyMuPDF, transformers==4.50.3, torch==2.6.0, pytesseract, pdf2image, opencv-python
• Download packages manually from github before importing:

  • vws: https://github.com/Sudo-VP/Vietnamese-Word-Segmentation-Python

    • click code -> download zip
    • after unzipping, take out folder vws
    • navigate anaconda3 -> Lib -> site-packages
    • drop vws folder into this site-packages

  • tesseract: https://github.com/UB-Mannheim/tesseract/wiki

    • this will download as .exe file, please run it

    • then this appear click ok

    • then comes some setup steps which are not important to the functionality of this code, so you can choose whatever and click next

    • then this appears Please choose:

      • "Vietnamses script" for "Additional script data (download)"
      • "Vietnamses" for "Additional langauge data"

    • Choose the default path of Tesseract-OCR Or you can change the code below to the path that you want your Tesseract-OCR to be in (making sure the Tesseract-OCR file stay in the same location as dictated in the below code)

    • Now click install (the folder Tesseract-OCR will be placed at location you dictated)

Usage

• Prepare a folder locally packed with pdf files in need of scanning

• An excel file full of words that you want to extact from the pdf files Note: you have to put all phrases and words only in column A, under a title "criteria" as i do in the image above (that means maximum 1048575 words) for the code to run properly

• Change the path to pdf folder and excel file accordingly to your own files' name The output path name is arbitrary, and it will appear on "Home" tab of Jupyternotebook once the code finishes

• Change similarity score to match your desire, with higher score means stricter demand for a phrase/word to be seen as synonymous

• Now run the code and wait for your result!!

DISCLAIMER:

• this is open-source code and you do not have to obtain my permission for use
• the code, however, contain code from other sources too (links of them will be pasted below), so please consult their policy too when using my code for purposes other than non-profit or educational
• i will not be legally responsible for any misuse of this code

Reference

I'm very grateful of the people posting materials freely to help me finish this code and my science project, which require reading many pdf files to detect the existence of specific phrases:

• Sudo-VP for https://github.com/Sudo-VP/Vietnamese-Word-Segmentation-Python
• UB-Mannheim for https://github.com/UB-Mannheim/tesseract/wiki
• VinAIResearch for their phoBERT model https://github.com/VinAIResearch/PhoBERT