-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathLSH.minhashing.html
More file actions
74 lines (71 loc) · 4.48 KB
/
Copy pathLSH.minhashing.html
File metadata and controls
74 lines (71 loc) · 4.48 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html><head><title>Python: module minhashing</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head><body bgcolor="#f0f0f8">
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="heading">
<tr bgcolor="#7799ee">
<td valign=bottom> <br>
<font color="#ffffff" face="helvetica, arial"> <br><big><big><strong>minhashing</strong></big></big></font></td
><td align=right valign=bottom
><font color="#ffffff" face="helvetica, arial"><a href=".">index</a><br><a href="file:/home/divakar/Downloads/IR/LSH/minhashing.py">/home/divakar/Downloads/IR/LSH/minhashing.py</a></font></td></tr></table>
<p><tt>Reduce the size of shingle incidence matrix by converting docs to signatures<br>
<br>
This module calculates the signature of each document using minhashing <br>
technique by utilizing a number of hash functions<br>
<br>
This module contatins following functions:<br>
* generate_signature_matrix - to generate signature matrix from incidence <br>
matrix</tt></p>
<p>
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="section">
<tr bgcolor="#aa55cc">
<td colspan=3 valign=bottom> <br>
<font color="#ffffff" face="helvetica, arial"><big><strong>Modules</strong></big></font></td></tr>
<tr><td bgcolor="#aa55cc"><tt> </tt></td><td> </td>
<td width="100%"><table width="100%" summary="list"><tr><td width="25%" valign=top><a href="numpy.html">numpy</a><br>
<a href="os.html">os</a><br>
</td><td width="25%" valign=top><a href="pandas.html">pandas</a><br>
<a href="random.html">random</a><br>
</td><td width="25%" valign=top><a href="sys.html">sys</a><br>
</td><td width="25%" valign=top></td></tr></table></td></tr></table><p>
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="section">
<tr bgcolor="#eeaa77">
<td colspan=3 valign=bottom> <br>
<font color="#ffffff" face="helvetica, arial"><big><strong>Functions</strong></big></font></td></tr>
<tr><td bgcolor="#eeaa77"><tt> </tt></td><td> </td>
<td width="100%"><dl><dt><a name="-floor"><strong>floor</strong></a>(x, /)</dt><dd><tt>Return the floor of x as an Integral.<br>
<br>
This is the largest integer <= x.</tt></dd></dl>
<dl><dt><a name="-generate_hash_functions"><strong>generate_hash_functions</strong></a>(rows, no_of_hash_functions)</dt><dd><tt>This function generates parameters for given no of hash functions<br>
<br>
Parameters<br>
----------<br>
rows: int<br>
no of shingles in corpus, a.k.a no of rows in shingle matrix<br>
no_of_hash_functions: int, optional<br>
no of hash functions to generate for minhashing<br>
Default: 100<br>
<br>
Returns<br>
-------<br>
list<br>
list of functions which can be used as hashes[i](x)</tt></dd></dl>
<dl><dt><a name="-generate_signature_matrix"><strong>generate_signature_matrix</strong></a>(incidence_matrix, no_of_hash_functions)</dt><dd><tt>This function generates the signature matrix for whole corpus<br>
<br>
if a already generated pickle file named sig_mat.pickle exists,<br>
this function will load it instead<br>
<br>
Parameters<br>
----------<br>
incidence_matrix: pandas.DataFrame<br>
incidence index generated after shingling of similar process<br>
no_of_hash_functions: int, optional<br>
no of hash functions to use to generate document signatures.<br>
Default: 100<br>
<br>
Returns<br>
-------<br>
pandas.DataFrame<br>
dataframe containing signatures of each document</tt></dd></dl>
</td></tr></table>
</body></html>