I'm trying to accelerate Pandas df.apply(), and also get a progress bar. The problem is, p_map is orders of magnitude slower than plain multiprocess.Pool.map() for a job where most of the processing is done by nltk.sentiment.vader.SentimentIntensityAnalyzer().
This notebook is self-explanatory:
https://github.com/FlorinAndrei/misc/blob/master/p_tqdm_bug_1.ipynb
p_map() is orders of magnitude slower.
However, the same function seems to work fine, fast enough, for another task - reading 25k files off the disk.
Windows 10, Python 3.8.8, Jupyter Notebook
I'm trying to accelerate Pandas
df.apply(), and also get a progress bar. The problem is,p_mapis orders of magnitude slower than plainmultiprocess.Pool.map()for a job where most of the processing is done bynltk.sentiment.vader.SentimentIntensityAnalyzer().This notebook is self-explanatory:
https://github.com/FlorinAndrei/misc/blob/master/p_tqdm_bug_1.ipynb
p_map()is orders of magnitude slower.However, the same function seems to work fine, fast enough, for another task - reading 25k files off the disk.
Windows 10, Python 3.8.8, Jupyter Notebook