Skip to content

Code overview

Jump to bottom
Aurelie Herbelot edited this page Jun 16, 2015 · 6 revisions

PeARS uses a computational linguistics technique called distributional semantics (DS) -- there is a lightweight intro to the topic here: http://aurelieherbelot.net/research/distributional-semantics-intro/. DS analyses large bodies of text to learn conceptual representations in terms of points in a vector space. In the case of PeARS, we have a 300-dimension vector space, so basically each concept (e.g. 'cat', 'dog', 'red', 'contribute') is represented by 300 numbers which express its 'meaning'. The vectors PeARS knows about are in the wikiwoods.dm file (10,000 of them). Those were pre-computed from Wikipedia.

Indexing

It is possible to index pages by feeding a list of urls to indexPages.py. The script retrieves pages using the lynx browser and performs the following operations on the files:

  1. part-of-speech tagging using the textblob_aptagger module (written by Matthew Honnibal): this gives each word a 'part-of-speech' (POS), i.e. finds out whether the word is a noun, an adjective, a verb, etc (this is run by local/runTextBlobTagger.py);
  2. get a DS representation for the document by adding up the vectors for each word occurring on the page -- so the 'meaning' of a document ends up being a list of 300 numbers, as for the single words (local/runDistSem.py);
  3. make a positional index for the document (I explain briefly what a positional index is in the course of this blog entry: http://aurelieherbelot.net/copyright-vs-privacy/). It basically tells you which words appear where on the page (local/mkPositionalIndex.py);
  4. update the 'signature' of the pear (I use 'pear' to refer to peers in the network :-) ) by summing the DS representations of all new documents and adding them to the existing signature (which is again a list of 300 numbers, contained in the profile.txt file of each pear)
  5. make a word cloud of the most important words in each document, to be displayed in search results (we are not doing snippets at the moment: see why in the blog entry mentioned before (http://aurelieherbelot.net/copyright-vs-privacy/): local/mkWordClouds.py.

Four of the files produced have to be shared on some website to constitute a pear:

  • doc.dists.txt = the DS representations of the documents indexed by that pear
  • profile.txt = the pear's profile
  • urls.dict.txt = a dictionary mapping file identifiers to urls
  • wordclouds.txt = the produced wordclouds for each indexed document

You can look at these files in e.g. web/demo1pear/.

At query time, we will need to know which pears are on the network, but also their signatures. This is contained in the file shared_pears_ids.txt. This file has to be shared by all pears on the network, and updated regularly.

Querying

mkQueryPage.py is responsible for displaying the search page at localhost:8080. We have a two-step process for search. Step 1 decides which pears on the network might be useful to answer the query (so that we don't have to query thousands of nodes). Step 2 retrieves some relevant files from the pears identified in Step 1 to get out URLs of actual web pages which might answer the query:

Step 1: the query is converted into a DS representation by adding up the vectors of the words it contains. If I search for 'python programming', it adds up the vectors for 'python' and 'programming', ending up with 300 numbers that represent that query. This representation is compared with the signatures of all pears in the shared_pears_ids.txt file, using the cosine distance measure (by web/findBestPears.py). The 'best' pears are recorded.

Step 2: The best pears are then accessed, i.e. their doc.dists.txt files are retrieved, and the query is again compared to the DS representations of each document on the pear (by web/scorePages.py). The best documents are then displayed on the results page, together with their word clouds (as read from the file wordsclouds.txt). On the right-hand side of the results page, there is a box supposed to contain the 'profile picture' of each pear, which we can hover over to get a little greetings message. The idea is that people might want to personalise their pear – this is not very important for the moment.

Clone this wiki locally