Hello, this script is designed to carry out topic modeling using large language models specifically for short texts. While it can technically be applied to longer texts as well, limitations on token counts from OpenAI and other large language models make it less practical for that purpose.
I have compared the results of topic modeling using LLMs with LDA, and human classification. The results are pretty promising. In addition, the findings suggest that the sample size may not influence the performance as we expect. You may see the excel file for the human classification. The csv file for GPT classification. The R script for LDA topic modelling. The colab file is for topic modeling by GPT. For claude, just upload the text and use relevant prompts. With GPT 4, you may now directly use the following GPT developed by me to conduct topic modeling. It supports up to 128k tokens. https://gptstore.ai/gpts/mrgtVPSEgp-topic-modeler-pro
If you have any questions, you can contact me shubin.yu@bi.no Visit http://GAIforResearch.com for more tutorials.