Before the transcripts could be used for qualitative analysis, a preprocessing pipeline was developed in Python to organize and collate them. This was necessary primarily because of sheer the number of transcripts (insert approx number) and the way they were structured. The folders containing the documents were heavily nested: all documents were contained inside zipped folders, which in turn contained further zipped or unzipped folders, often several levels deep. An additional factor that required attention was the variability in file formats. The transcripts were distributed across six formats, namely Microsoft Word (.docx), Microsoft Word 97-2003 (.doc), Rich Text Format (.rtf), plain text (.txt), OpenDocument Text (.odt) and Portable Document Format (.pdf).
The combination of the nested folder organization and the variability of file formats made parsing and extracting key information very time intensive. The preprocessing pipeline therefore addressed both issues in sequence. It first restructured the file organization, recursively unpacking all archives until all transcripts inside a given folder were easily accessible and at the top level itself i.e., no more nested organization. Documents that did not form part of the transcript material were excluded at this stage (i.e., meta data).
Then, for each parent folder, the pipeline converted all the contained files into the PDF format while preserving the text structures (pagination, tables, alignment) within each file. Files that were already in the PDF format were left as is. Once this was done, all converted PDF files were merged into a single PDF file. This was done because for larger text files, the PDF format provides the fastest parsing speeds. Additionally, the title of each individual file that was merged into the pdf was extracted and was used to denote the position of the file in the merged document. This was done to verify if all files were part of the final merged document and to ease keyword based searching and parsing. Finally, if the total page count of the resulting merged PDF was greater than 2000 pages, it was split into consecutive parts of at most 1000 pages. This was done to improve the speed at which each document could be parsed using keyword based searching methods.