diff --git a/README.md b/README.md index ebc7eec..cd09bd4 100644 --- a/README.md +++ b/README.md @@ -16,20 +16,20 @@ Taxonomies and document classification -------------------------------------- Clade taxonomies have a tree structure, with a single top-level category (e.g. -in the example data, "Social Psychology"). There is no distinction between +in the example data, "Social Psychology"). There is no distinction between parent and child nodes (except that the former has children) and the hierachical structure of the taxonomy is completely orthogonal from the node data. The structure may be freely edited. Each node represents a category, which is represented by a set of "keywords" -(words or phrases) which should be present in a document belonging to that -category. Not all the keywords have to be present - they are joined with -Boolean OR rather than AND. A document may belong to multiple categories, +(words or phrases) which should be present in a document belonging to that +category. Not all the keywords have to be present - they are joined with +Boolean OR rather than AND. A document may belong to multiple categories, which are ranked according to standard Solr (TF-IDF) scoring. It is also possible to exclude certain keywords from categories. Clade will also suggest keywords to add to a category, based on the content of -the documents already in the category. This feature is currently slow as it +the documents already in the category. This feature is currently slow as it uses the standard Solr MoreLikeThis component to analyse a large number of documents. We plan to improve this for a future release by writing a custom Solr plugin. @@ -43,20 +43,20 @@ export the document categorisation to an XML or CSV file. Installation prerequisites -------------------------- -- Java 6 or 7 +- Java 6 or later -- Python 2.6 or 2.7 +- Python 3 - Solr 4.6 (also tested with 4.8.1) http://www.apache.org/dyn/closer.cgi/lucene/solr/4.6.2 Other versions may also work, but we have not tested them with Clade. - Python modules - sunburnt: http://opensource.timetric.com/sunburnt/installation.html + scorched: https://github.com/lugensa/scorched lxml: http://pypi.python.org/pypi/lxml/ - httplib2: http://code.google.com/p/httplib2/downloads/list - -On Windows you can download binaries of httplib2 from http://www.lfd.uci.edu/~gohlke/pythonlibs/ + pyjq: https://github.com/doloopwhile/pyjq + httplib2 + Download the latest versions from these sites and follow the installation instructions. The system has been developed on Linux and the following instructions @@ -69,32 +69,32 @@ as appropriate): $ export CFLAGS=-Qunused-arguments $ export CPPFLAGS=-Qunused-arguments - $ sudo -E pip install lxml + $ sudo -E pip install lxml ------------------- Windows quick start ------------------- On Windows, you can use the demo.bat file to run all the various parts of Clade with the -supplied example data. You will need to edit paths.bat to match your configuration first. +supplied example data. You will need to edit paths.bat to match your configuration first. -The demo_noclassify.bat file also runs Clade, but doesn't repeat the initial classification +The demo_noclassify.bat file also runs Clade, but doesn't repeat the initial classification to avoid duplicates. If you need it, cleanup.bat clears all the indexes. ------------ Example data ------------ -The Clade distribution includes an example taxonomy and documents derived -from Wikipedia and on the topic of Social Psychology. The taxonomy is +The Clade distribution includes an example taxonomy and documents derived +from Wikipedia and on the topic of Social Psychology. The taxonomy is provided as a CSV file to illustrate how to import existing taxonomies. To import the taxonomy, run on the command line: $ python classify.py import data/socpsy.csv - + on Windows: - + C:\> python classify.py import data\socpsy.csv This will create a Python data structure, and pickle it as data/tax. @@ -103,16 +103,16 @@ To import the documents, first copy the Clade Solr configuration files into your Solr home, e.g.: $ cp -f clade/solr-conf/* apache-solr-4.6.2/example/solr/collection1/conf - + on Windows: - + C:\> copy clade\solr-conf\*.* apache-solr-4.6.2\example\solr\collection1\conf Then [re]start Solr, e.g.: - $ cd apache-solr-4.6.2/example + $ cd apache-solr-4.6.2/example $ java -jar start.jar & - + on Windows: C:\> cd apache-solr-4.6.2\example @@ -120,25 +120,25 @@ Then [re]start Solr, e.g.: Now start the Stanford Named Entity Recognition server, which is used to pull names, places etc. out of the source data: - - $ cd stanford-ner-2011-09-14 + + $ cd stanford-ner-2018-10-16 $ ./server.sh & - + or on Windows: - - C:\ cd stanford-ner-2011-09-14 + + C:\ cd stanford-ner-2018-10-16 C:\ server.bat Finally, add the example documents (which are provided as plaintext files): $ python classify.py textdir data/socpsy-pages - + on Windows: - + C:\> python classify.py textdir data\socpsy-pages - -This will output the name of each file as it is processed, and will take a -few minutes to complete. + +This will output the name of each file as it is processed, and will take a +few minutes to complete. -------------- @@ -146,18 +146,18 @@ Running the UI -------------- The Clade UI is implemented as a web application. To start it, run: - + $ python server.py - + on Windows: - + C:\> python server.py -Then point a browser at +Then point a browser at http://localhost:8080/ - - + + ---------------------- Taxonomy import/export ---------------------- @@ -169,9 +169,9 @@ optionally the document categorisations) to CSV or XML. To import from CSV: $ python classify.py import taxonomy.csv - + on Windows: - + C:\> python classify.py import taxonomy.csv To import from XML, use the .xml file extension. The extension must be either .csv @@ -180,22 +180,22 @@ or .xml and is used to determine the type of file read. To export to CSV: $ python classify.py export taxonomy.csv - + on Windows: - + C:\> python classify.py export taxonomy.csv - + To export to XML, use the .xml file extension. Again, the extension determines the type of file written. In the case of exported to XML, document classifications may be written to the file by specifying the number of document ids that should be exported per category, e.g.: $ python classify.py export taxonomy.xml 100 - + on Windows: - + C:\> python classify.py export taxonomy.xml 100 - + ----------- UI Controls @@ -206,14 +206,14 @@ The Clade UI has two modes: Taxonomy and Document. It starts up in the former. Taxonomy mode ------------- -The page is divided into two halves: the taxonomy tree on the left, and +The page is divided into two halves: the taxonomy tree on the left, and information about the current selected category on the right. Clade can support multiple taxonomies, which can be selected from the drop-down in the upper left. Selecting a taxonomy will load it into the tree view, where nodes can be expanded or collapsed, selected and manipulated. -The taxonomy tree display has three small icons in the upper right. The + -icon causes a new category to be added as a child of the currently selected +The taxonomy tree display has three small icons in the upper right. The + +icon causes a new category to be added as a child of the currently selected node. The "pen" icon allows the current node name to be edited, and the X icon deletes the current node (with no warning!) @@ -240,7 +240,7 @@ category, showing the document ID, the title, the current rank and the previous rank (before changing the active keywords). To view a document, click the title, which will switch the UI into document mode. -To add a new taxonomy, make sure the taxonomy drop-down has no taxonomy +To add a new taxonomy, make sure the taxonomy drop-down has no taxonomy selected, then click the Create button. Enter a name for the new taxonomy and click OK. You can then create the taxonomy by adding nodes to the root node. All changes will be immediately saved to the data/tax file. @@ -262,12 +262,11 @@ Document mode ------------- Again, in document mode the page has a left section and a right section. On the -left is a list of taxonomy categories which match the document, ranked in -decreasing score order. The document ID and full text are displayed on the +left is a list of taxonomy categories which match the document, ranked in +decreasing score order. The document ID and full text are displayed on the right, together with a list of keywords from the selected category. The document text is highlighted for matching keywords. To go back to the taxonomy display, click the Back button in the upper right. -Don't use your browser back button - it won't work, as the UI is implemented +Don't use your browser back button - it won't work, as the UI is implemented in Javascript/AJAX. - diff --git a/classify.py b/classify.py index 1de776b..147de81 100644 --- a/classify.py +++ b/classify.py @@ -19,7 +19,7 @@ import os import re -import cPickle +import pickle import csv import time @@ -28,7 +28,8 @@ from lib import taxonomy from lib import ner from lib import csv_unicode -import sunburnt + +import scorched import httplib2 import settings @@ -44,29 +45,29 @@ def process_dir(solr, dir): """ Process the .docx files in dir, using the supplied taxonomies to classify them, and storing the results in the xapian index. - + """ def f(arg, dirname, names): for name in names: if name.lower().endswith('.docx'): process_file(solr, os.path.join(dirname, name)) - + os.path.walk(dir, f, None) solr.commit() def process_dir_xml(solr, dir): """ Process the .xml files in dir, using the supplied taxonomies to classify them, and storing the results in the xapian index. - + """ def f(arg, dirname, names): for name in names: if name.lower().endswith('.xml'): process_xml_file(solr, os.path.join(dirname, name)) - + os.path.walk(dir, f, None) solr.commit() - + def process_dir_text(solr, dir): """ Process the text files in dir, etc. """ @@ -75,41 +76,49 @@ def process_dir_text(solr, dir): if name[0] != '.': with open(os.path.join(dir, name)) as f: text = f.read() - name = name.decode('utf8', 'ignore') + #print('naame:',name) + #print(text) + #name = name.decode('utf8', 'ignore') # dep Getting a message this doesn't exist + #text = text.decode('utf8') docid = '%s.%s' % (docidbase, count) - update_solr(solr, docid, name, text.decode('utf8')) - + print(docid) + update_solr(solr, docid, name, text) + solr.commit() def process_file(solr, file): _, _, text, title = docx.parse_docx(file) filename = os.path.basename(file) - update_solr(solr, filename, title or filename, text) + update_solr(solr, filename, title or filename, text) solr.commit() def process_xml_file(solr, file): _, _, text, title = xmldoc.parse_xml(file) filename = os.path.basename(file) - update_solr(solr, filename, title or filename, text) + update_solr(solr, filename, title or filename, text) solr.commit() def process_csv(solr, path): with open(path) as f: for line in csv_unicode.UnicodeReader(f): text = "%s\n\n%s" % (line[0], line[1]) - update_solr(solr, line[2], line[6] or 'no title', text) + update_solr(solr, line[2], line[6] or 'no title', text) solr.commit() def update_solr(solr, unique_term, title, text): - print 'updating', unique_term, title + print ('updating', unique_term, title) + # not needed? 'id': '1', doc = { 'title': title, 'text': text, 'doc_id': unique_term } doc['entity'] = tuple(set(x for x in iter_entity_terms(text))) - solr.add(doc) - + #solr.add([doc]) #pysolr + #print(doc) + solr.add(doc) #scoched/sunburnt + def iter_entity_terms(text): for term in ner.get_entities(settings.ner_host, settings.ner_port, text): if len(term) < 50: - yield unicode(term, "utf-8") + #yield unicode(term, "utf-8") DEP + yield term def iter_text_terms(text): phrase = [] @@ -144,17 +153,18 @@ def iter_text_terms(text): def get_doctext(solr, did): """ Return the text for a document ID. - + """ return solr.query(doc_id=did).execute()[0]["text"] - + if __name__ == '__main__': import sys - import cPickle - + #import cPickle + import pickle + # FIXME - help message - + def _parse(path): with open(path) as f: if path.endswith('.xml'): @@ -162,20 +172,21 @@ def _parse(path): elif path.endswith('.csv'): return taxonomy.parse_csv(f) else: - print 'unrecognized input file extension (use .xml or .csv)' + print ('unrecognized input file extension (use .xml or .csv)') sys.exit(1) - + if sys.argv[1] == 'import': - with open(settings.taxonomy_path, 'w') as f: + with open(settings.taxonomy_path, 'wb') as f: taxes = [_parse(x) for x in sys.argv[2:]] - cPickle.dump(taxes, f) - print 'imported', len(taxes), 'taxonomies' + pickle.dump(taxes, f) + print ('imported', len(taxes), 'taxonomies') else: - with open(settings.taxonomy_path) as f: - taxes = cPickle.load(f) + with open(settings.taxonomy_path, 'rb') as f: + taxes = pickle.load(f) - h = httplib2.Http(cache=settings.http_cache) - _solr = sunburnt.SolrInterface(settings.solr_url, http_connection=h) + #h = httplib2.Http(cache=settings.http_cache) + #_solr = pysolr.Solr(settings.solr_url, timeout=100) + _solr = scorched.SolrInterface(settings.solr_url) #// dep http_cnnection?? if sys.argv[1] == 'export': with open(sys.argv[2], 'w') as f: @@ -185,7 +196,7 @@ def _parse(path): try: rows = int(sys.argv[3]) except ValueError: - print 'rows value must be integer' + print ('rows value must be integer') sys.exit(1) else: solr = None @@ -194,37 +205,37 @@ def _parse(path): elif sys.argv[2].endswith('.csv'): taxonomy.write_csv(f, taxes) else: - print 'unrecognized output file extension (use .xml or .csv)' + print ('unrecognized output file extension (use .xml or .csv)') elif sys.argv[1] == 'lscat': tax = taxonomy.term_for_path(taxes, sys.argv[2]) if tax: - print 'tax:', tax + print ('tax:', tax) for doc in taxonomy.get_docs_for_category(_solr, tax): - print doc + print (doc) else: - print 'no matching taxonomy' + print ('no matching taxonomy') elif sys.argv[1] == 'classify': with open(sys.argv[2], 'w') as f: taxonomy.write_classification(_solr, f, taxes) elif sys.argv[1] == 'doctext': - print get_doctext(_solr, int(sys.argv[2])) + print (get_doctext(_solr, int(sys.argv[2]))) elif sys.argv[1] == 'suggest': for term in taxonomy.suggest_keywords(_solr, sys.argv[2:], [], 7): - print term + print (term) elif sys.argv[1] == 'docdir': process_dir(_solr, sys.argv[2]) elif sys.argv[1] == 'xmldocdir': process_dir_xml(_solr, sys.argv[2]) - + elif sys.argv[1] == 'docfile': process_file(_solr, sys.argv[2]) - + elif sys.argv[1] == 'csvfile': process_csv(_solr, sys.argv[2]) diff --git a/delete-all.sh b/delete-all.sh index 5f9b2db..8069f3d 100755 --- a/delete-all.sh +++ b/delete-all.sh @@ -1,4 +1,4 @@ #!/bin/sh -URL=http://localhost:8983/solr/update +URL=http://localhost:8983/solr/update?commit=true curl $URL --data-binary '*:*' -H 'Content-type:application/xml' echo diff --git a/go2.bat b/go2.bat index 6751533..417d42a 100644 --- a/go2.bat +++ b/go2.bat @@ -1,8 +1,8 @@ rem go2.bat -rem This is one of the three batch files to be run to set up Clade on Windows. +rem This is one of the three batch files to be run to set up Clade on Windows. rem Each must run in its own command line shell call paths.bat -cd stanford-ner-2011-09-14 +cd stanford-ner-2018-10-16 server.bat diff --git a/lib/csv_unicode.py b/lib/csv_unicode.py index 48274a2..726b8ab 100644 --- a/lib/csv_unicode.py +++ b/lib/csv_unicode.py @@ -1,4 +1,5 @@ -import csv, codecs, cStringIO +import csv, codecs +from io import StringIO class UTF8Recoder: """ @@ -58,4 +59,3 @@ def writerow(self, row): def writerows(self, rows): for row in rows: self.writerow(row) - diff --git a/lib/docx.py b/lib/docx.py index 90fd6d1..5e7b370 100644 --- a/lib/docx.py +++ b/lib/docx.py @@ -1,11 +1,11 @@ # Copyright 2012 Lemur Consulting Limited -# +# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at -# +# # http://www.apache.org/licenses/LICENSE-2.0 -# +# # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -13,7 +13,7 @@ # limitations under the License. import zipfile -from StringIO import StringIO +from io import StringIO from lxml import etree import sys @@ -22,12 +22,12 @@ def parse_docx(path): """Return ([analysts], service number, document text, document title). - + """ analysts = [] service_number = None - mydoc = zipfile.ZipFile(path) + mydoc = zipfile.ZipFile(path) doc = etree.parse(mydoc.open('word/settings.xml')) els = doc.xpath("//w:docVar[@w:name='dvMetaData']", namespaces={'w':NSMAIN}) if els: @@ -38,7 +38,7 @@ def parse_docx(path): els = doc.xpath("//analysts") if els: analysts = [x.strip() for x in els[0].text.split(';')] - + els = doc.xpath("//service_number") if els: service_number = int(els[0].text) @@ -52,7 +52,7 @@ def parse_docx(path): for para in el: paratext=[] for el2 in para.iter(): - if (el2.tag == '{%s}instrText' % NSMAIN and + if (el2.tag == '{%s}instrText' % NSMAIN and 'DocTitle' in el2.text): doctitle = True @@ -71,9 +71,9 @@ def parse_docx(path): return analysts, service_number, '\n\n'.join(paratextlist), doctitle - + if __name__ == '__main__': import os for f in os.listdir(sys.argv[1]): if f.endswith('.docx'): - print f, parse_docx(os.path.join(sys.argv[1], f))[3] + print (f, parse_docx(os.path.join(sys.argv[1], f))[3]) diff --git a/lib/ner.py b/lib/ner.py index b09ff00..406a454 100644 --- a/lib/ner.py +++ b/lib/ner.py @@ -1,11 +1,11 @@ # Copyright 2012 Lemur Consulting Limited -# +# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at -# +# # http://www.apache.org/licenses/LICENSE-2.0 -# +# # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -16,20 +16,32 @@ def get_entities(host, port, text): - if isinstance(text, unicode): - text = text.encode('utf-8', 'ignore') + #if isinstance(text, unicode): # dep python 3 everything is unicode + #text = text.encode('utf-8', 'ignore') + #text = text.encode('utf-8', 'ignore') s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((host, port)) - s.send(text.replace('\n', ' ') + '\n') + txt = text.replace('\n', ' ') + txt = txt + '\n' + s.send(txt.encode()) data = [] while True: d = s.recv(1024) if not d: break - data.append(d) + data.append(d.decode()) s.close() - + + print('raw data', data) + print(len(data[0])) + print(data[0]) + + #print(data[0].decode()) + #print('decoded') + #data = data.decode() + #print(data) + lasttag = '' stack = [] for word in (x.split('/') for x in ''.join(data).split(' ')): @@ -45,14 +57,14 @@ def get_entities(host, port, text): yield ' '.join(stack) stack = [] yield word[0].strip() - + lasttag = word[1] if __name__ == '__main__': - test = """The fate of Lehman Brothers, the beleaguered investment bank, - hung in the balance on Sunday as Federal Reserve officials and the leaders - of major financial institutions continued to gather in emergency meetings + test = """The fate of Lehman Brothers, the beleaguered investment bank, + hung in the balance on Sunday as Federal Reserve officials and the leaders + of major financial institutions continued to gather in emergency meetings trying to complete a plan to rescue the stricken bank. John said.""" - print list(get_entities('localhost', 9000, test)) \ No newline at end of file + print (list(get_entities('localhost', 9000, test))) diff --git a/lib/taxonomy.py b/lib/taxonomy.py index 5d57978..bcc6317 100644 --- a/lib/taxonomy.py +++ b/lib/taxonomy.py @@ -1,11 +1,11 @@ # Copyright 2012 Lemur Consulting Limited -# +# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at -# +# # http://www.apache.org/licenses/LICENSE-2.0 -# +# # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -15,6 +15,8 @@ import csv import re import string +import json +import pyjq from xml.etree.ElementTree import ElementTree from xml.etree.ElementTree import TreeBuilder from lxml import etree @@ -53,48 +55,49 @@ def __init__(self, name=None, level=0, clues=None, uid=None): def add_child(self, child): self.children.append(child) child.parent = self - + def create_child(self, name): child = Term(name, self.level + 1, None) self.add_child(child) return child - + def iter_clues(self): - for clue in self._clues.iteritems(): + #for clue in self._clues.iteritems(): + for clue in self._clues.items(): yield clue - + def set_clue(self, clue, positive=True): self._clues[clue] = positive def toggle_clue(self, clue): self._clues[clue] = not self._clues[clue] - + def delete_clue(self, clue): if clue in self._clues: del self._clues[clue] - + def display(self): """Display this Term and all its descendants. - + """ - print '%s%s: %s' % (' ' * self.level, self.name, self.clues) + print ('%s%s: %s' % (' ' * self.level, self.name, self.clues)) for c in self.children: c.display() - + def __str__(self): """Return the path to this Term from the root as a string. - + """ l = [] while self: l.append(self.name) - self = self.parent + self = self.parent return '/'.join(l[::-1]) - + def walk(self, f_pre=None, f_post=None): """ Walk the tree, calling f_pre(self) and f_post(self) before and after calling on each child. - + """ if f_pre is not None: f_pre(self) @@ -102,7 +105,7 @@ def walk(self, f_pre=None, f_post=None): c.walk(f_pre, f_post) if f_post is not None: f_post(self) - + def for_jtree(self, solr): count = get_category_doc_count(solr, self) word = "document" if count == 1 else "documents" @@ -133,13 +136,13 @@ def f_clue(x): if x[0] == '-': return (x[1:], False) return (x, True) - + stack = [None] * 100 # should be plenty for line in csv.reader(f): # get level - for level in xrange(100): + for level in range(100): if line[level]: break - + name = line[level] clues = [] cluestr = '' @@ -186,13 +189,13 @@ def parse_xml(f): def write_csv(f, taxes): """ Export a CSV file for the given taxonomies to the (open) file handle specified. - + """ write_csv_file.level = 0 def f_clue(clue, positive): sign = '' if positive else '-' return '%s%s' % (sign, clue) - + def f_pre(term): for _ in xrange(write_csv_file.level): f.write(',') @@ -201,24 +204,24 @@ def f_pre(term): f.write(' '.join([f_clue(clue, positive) for clue, positive in term.iter_clues()])) f.write('"\r\n') write_csv_file.level += 1 - + def f_post(term): write_csv_file.level -= 1 - + for term in taxes: term.walk(f_pre, f_post) def write_xml(f, taxes, _solr=None, rows=0): """ Export an XML file for the given taxonomies to the (open) file handle specified. - + If a SOLR connection is supplied, then include document elements for each category. - + """ x = TreeBuilder() x.start("taxonomy", {}) - + def f_pre(term): x.start("category", { "id": str(term.uid) }) x.start("name", {}) @@ -242,26 +245,26 @@ def f_pre(term): x.data(title) x.end("name") x.end("doc") - + def f_post(term): x.end("category") - + for term in taxes: term.walk(f_pre, f_post) x.end("taxonomy") - + xml = ElementTree(x.close()) xml.write(f, xml_declaration=True, encoding="utf-8") def classify_doc(text, tax): """ Classify some document text using the supplied taxonomy tree(s). Return a list of (Term, score) ordered by score. - + """ results = [] parsed_text = parse_text(text) - + def calc(term): count = 0 total = 0.0 @@ -270,24 +273,24 @@ def calc(term): count += 1 m = 1 if p else -1 #FIXME: is this correct? total += m * calc_score(parsed_text, parsed_clue) - + total = total / count # normalisation(?) results.append((term, total)) - + if isinstance(tax, Term): tax.walk(calc) # walk a single taxonomy else: for t in tax: # walk several taxonomies t.walk(calc) - + return sorted((x for x in results if x[1]), # remove zero scores key=lambda x: x[1], reverse=True) def write_classification(solr, f, taxes): """ Export document classifications in CSV format: - + doc id, doc title, category id, category name - + """ out = csv.writer(f) first = 0 @@ -307,24 +310,24 @@ def parse_text(text): """Return a dict of word (lowercase) -> [positions]. Where there are periods, add 10 to the position so that phrases do not match across sentences etc (crude but should work). - + """ pos = 0 ret = {} - + for tok in RE_WORD_DELIM.findall(text): if tok in DELIMS: pos += 10 else: pos += 1 ret.setdefault(tok.lower(), []).append(pos) - + return ret def calc_score(parsed_text, clue): """Calculate a score for text parsed with parse_text() and a clue (as an iterable). - + """ # match whole phrases, score len(phrase) x tf @@ -338,13 +341,15 @@ def calc_score(parsed_text, clue): if pos + 1 in newmatches: tmp.append(pos + 1) matches = tmp - + return float(len(matches) * len(clue)) / len(parsed_text) - + def _category_doc_query(_solr, term): - """ Return a sunburnt query for category documents. - + """ Return a Fed query for category documents. + """ + print("_category_doc_query", term) + assert isinstance(term, Term) pos = None @@ -365,7 +370,7 @@ def _category_doc_query(_solr, term): neg = q else: neg |= q - + if pos is not None: if neg is not None: query = pos & ~neg @@ -381,7 +386,7 @@ def _category_doc_query(_solr, term): def get_category_doc_count(_solr, term): """ Return the number of documents in the category. - + """ query = _category_doc_query(_solr, term) return query.paginate(rows=0).execute().result.numFound @@ -390,17 +395,17 @@ def get_docs_for_category(_solr, term, rows=10): """ Return a tuple (numFound, docs) where numFound is the number of documents found in a category, and docs is a list of tuples (docid, title, score) for each document in the category. - + """ query = _category_doc_query(_solr, term) if query is None: return (0, []) - results = query.field_limit(["doc_id", "title"], score=True).paginate(rows=rows).execute() + results = query.field_limit(["doc_id", "title"], score=True).paginate(rows=int(rows)).execute() return results.result.numFound, [(doc["doc_id"], doc["title"], doc["score"]) for doc in results] def get_doc_ids_for_category(_solr, term): """ Return doc ids for each document in a category. - + """ query = _category_doc_query(_solr, term) if query is None: @@ -417,7 +422,7 @@ def num_docs(solr): def suggest_keywords(solr, dids, exclude, count): """ Return suggested terms for a set of document IDs. - + """ if len(dids) == 0: return [] @@ -430,35 +435,56 @@ def suggest_keywords(solr, dids, exclude, count): field = "text_mlt" opts = { "q": q, "rows": 100, "fl": "_", "tv": True, "tv.fl": field, "tv.df": True, "tv.tf": True } r = solr.search(**opts) - root = etree.fromstring(r.original_xml) + #root = etree.fromstring(r.original_xml) + orig_json = r.original_json + root = json.loads(orig_json) + + # The below should work, but we get a compile error from jq. + # result = pyjq.all('.termVectors | to_entries | group_by(.key/4 | floor) | map(map(.value))[][] | select(type == "array") | {docid:.[1],terms:(.[3]|to_entries|group_by(.key/2|floor) | map(map(.value)))} | {docid,terms:[.terms[]|{term:.[0],tf:.[1][1],df:.[1][3]}]}',root) + result = pyjq.all('.termVectors | to_entries | group_by(.key/4 | floor) | map(map(.value))[][] | select(type == "array") | {docid:.[1],terms:(.[3]|to_entries|group_by(.key/2|floor) | map(map(.value)))}',root) + + # Manually parse out the last bits since the above full jq doesn't work down below. + tvEl = root['termVectors'][0] # this should be the primary key # find the term vectors info - tvEl = root.xpath('/response/lst[@name="termVectors"]')[0] + #tvEl = root.xpath('/response/lst[@name="termVectors"]')[0] # echo any warnings - for warnEl in tvEl.xpath('lst[@name="warnings"]/arr'): - msg = warnEl.attrib["name"] - field = warnEl.xpath('str')[0].text - print "WARNING: '%s' for field '%s'" % (msg, field) - + print('checking warnings not supported ') + field = root['termVectors'][1] + #for warnEl in tvEl.xpath('lst[@name="warnings"]/arr'): + # msg = warnEl.attrib["name"] + # field = warnEl.xpath('str')[0].text + # print ("WARNING: '%s' for field '%s'" % (msg, field)) + N = num_docs(solr) - R = int(root.xpath('/response/result[@name="response"]')[0].attrib["numFound"]) + #R = int(root.xpath('/response/result[@name="response"]')[0].attrib["numFound"]) + R = int(root['response']['numFound']) # for each doc element... keywords = {} totalL = 0 - for docEl in tvEl.xpath('lst[@name!="warnings"]'): - doc_id = docEl.xpath('str[@name="uniqueKey"]')[0].text - keywordList = docEl.xpath('lst[@name="%s"]/lst' % field) + for docEl in result: +# for docEl in tvEl.xpath('lst[@name!="warnings"]'): + #doc_id = docEl.xpath('str[@name="uniqueKey"]')[0].text + doc_id = docEl['docid'] + #keywordList = docEl.xpath('lst[@name="%s"]/lst' % field) + keywordList = docEl['terms'] L = len(keywordList) totalL += L for keywordEl in keywordList: - keyword = keywordEl.attrib["name"] + #keyword = keywordEl.attrib["name"] + keyword = keywordEl[0] + # print('keyword',keyword) if keyword in exclude or len(keyword) < 5: continue - n = int(keywordEl.xpath('int[@name="df"]')[0].text) - wdf = int(keywordEl.xpath('int[@name="tf"]')[0].text) + #n = int(keywordEl.xpath('int[@name="df"]')[0].text) + n = int(keywordEl[1][3]) + #wdf = int(keywordEl.xpath('int[@name="tf"]')[0].text) + wdf = int(keywordEl[1][1]) + # print("n ", n, ' wdf', wdf) if keyword not in keywords: keywords[keyword] = [n, []] keywords[keyword][1].append((wdf, L)) - keywords = [(keyword, n, m) for keyword, (n, m) in keywords.iteritems()] + + keywords = [(keyword, n, m) for keyword, (n, m) in keywords.items()] avgL = float(totalL) / R def wt(n, m): r = len(m) @@ -468,6 +494,5 @@ def wt(n, m): v += 2 * wdf / (Ld + wdf) return v * float(r) * log((r+0.5)*(N-R-n+r+0.5)/((R-r+0.5)*(n-r+0.5))) keywords.sort(key=lambda k: wt(k[1], k[2]), reverse=True) - #print keywords + #print ('Keywords:',keywords) return [k[0] for k in keywords[:count]] - diff --git a/lib/web/__init__.py b/lib/web/__init__.py index 5fd0295..58c1425 100644 --- a/lib/web/__init__.py +++ b/lib/web/__init__.py @@ -3,7 +3,7 @@ from __future__ import generators -__version__ = "0.36" +__version__ = "0.40-dev1" __author__ = [ "Aaron Swartz ", "Anand Chitipothu " @@ -11,23 +11,23 @@ __license__ = "public domain" __contributors__ = "see http://webpy.org/changes" -import utils, db, net, wsgi, http, webapi, httpserver, debugerror -import template, form +from . import utils, db, net, wsgi, http, webapi, httpserver, debugerror +from . import template, form -import session +from . import session -from utils import * -from db import * -from net import * -from wsgi import * -from http import * -from webapi import * -from httpserver import * -from debugerror import * -from application import * -from browser import * +from .utils import * +from .db import * +from .net import * +from .wsgi import * +from .http import * +from .webapi import * +from .httpserver import * +from .debugerror import * +from .application import * +#from browser import * try: - import webopenid as openid + from . import webopenid as openid except ImportError: pass # requires openid module diff --git a/lib/web/application.py b/lib/web/application.py index b2ec7c3..2c95f0f 100644 --- a/lib/web/application.py +++ b/lib/web/application.py @@ -2,10 +2,14 @@ Web application (from web.py) """ -import webapi as web -import webapi, wsgi, utils -import debugerror -from utils import lstrips, safeunicode +from __future__ import print_function + +from . import webapi as web +from . import wsgi, utils, browser +from .debugerror import debugerror +from . import httpserver +from .utils import lstrips, safeunicode +from .py3helpers import iteritems, string_types, is_iter, PY2, text_type import sys import urllib @@ -13,16 +17,29 @@ import itertools import os import types -from exceptions import SystemExit +from inspect import isclass + +import wsgiref.handlers try: - import wsgiref.handlers + from urllib.parse import splitquery, urlencode, quote, unquote except ImportError: - pass # don't break people with old Pythons + from urllib import splitquery, urlencode, quote, unquote + +try: + from importlib import reload #Since Py 3.4 reload is in importlib +except ImportError: + try: + from imp import reload #Since Py 3.0 and before 3.4 reload is in imp + except ImportError: + pass #Before Py 3.0 reload is a global function + +from io import BytesIO + __all__ = [ "application", "auto_application", - "subdir_application", "subdomain_application", + "subdir_application", "subdomain_application", "loadhook", "unloadhook", "autodelegate" ] @@ -30,14 +47,14 @@ class application: """ Application to delegate requests based on path. - + >>> urls = ("/hello", "hello") >>> app = application(urls, globals()) >>> class hello: ... def GET(self): return "hello" >>> >>> app.request("/hello").data - 'hello' + b'hello' """ def __init__(self, mapping=(), fvars={}, autoreload=None): if autoreload is None: @@ -45,10 +62,10 @@ def __init__(self, mapping=(), fvars={}, autoreload=None): self.init_mapping(mapping) self.fvars = fvars self.processors = [] - + self.add_processor(loadhook(self._load)) self.add_processor(unloadhook(self._unload)) - + if autoreload: def main_module_name(): mod = sys.modules['__main__'] @@ -62,14 +79,14 @@ def modname(fvars): return None if name == '__main__': - # Since the __main__ module can't be reloaded, the module has - # to be imported using its file name. + # Since the __main__ module can't be reloaded, the module has + # to be imported using its file name. name = main_module_name() return name - + mapping_name = utils.dictfind(fvars, mapping) module_name = modname(fvars) - + def reload_mapping(): """loadhook to reload mapping and fvars.""" mod = __import__(module_name, None, None, ['']) @@ -88,13 +105,13 @@ def reload_mapping(): __import__(main_module_name()) except ImportError: pass - + def _load(self): web.ctx.app_stack.append(self) - + def _unload(self): web.ctx.app_stack = web.ctx.app_stack[:-1] - + if web.ctx.app_stack: # this is a sub-application, revert ctx to earlier state. oldctx = web.ctx.get('_oldctx') @@ -103,7 +120,7 @@ def _unload(self): web.ctx.homepath = oldctx.homepath web.ctx.path = oldctx.path web.ctx.fullpath = oldctx.fullpath - + def _cleanup(self): # Threads can be recycled by WSGI servers. # Clearing up all thread-local state to avoid interefereing with subsequent requests. @@ -117,8 +134,8 @@ def add_mapping(self, pattern, classname): def add_processor(self, processor): """ - Adds a processor to the application. - + Adds a processor to the application. + >>> urls = ("/(.*)", "echo") >>> app = application(urls, globals()) >>> class echo: @@ -129,7 +146,7 @@ def add_processor(self, processor): ... >>> app.add_processor(hello) >>> app.request("/web.py").data - 'hello, web.py' + b'hello, web.py' """ self.processors.append(processor) @@ -141,13 +158,13 @@ def request(self, localpart='/', method='GET', data=None, >>> urls = ("/hello", "hello") >>> app = application(urls, globals()) >>> class hello: - ... def GET(self): + ... def GET(self): ... web.header('Content-Type', 'text/plain') ... return "hello" ... >>> response = app.request("/hello") >>> response.data - 'hello' + b'hello' >>> response.status '200 OK' >>> response.headers['Content-Type'] @@ -174,17 +191,17 @@ def request(self, localpart='/', method='GET', data=None, >>> class uaprinter: ... def GET(self): ... return 'your user-agent is ' + web.ctx.env['HTTP_USER_AGENT'] - ... + ... >>> app = application(urls, globals()) >>> app.request('/ua', headers = { ... 'User-Agent': 'a small jumping bean/1.0 (compatible)' ... }).data - 'your user-agent is a small jumping bean/1.0 (compatible)' + b'your user-agent is a small jumping bean/1.0 (compatible)' """ - path, maybe_query = urllib.splitquery(localpart) + path, maybe_query = splitquery(localpart) query = maybe_query or "" - + if 'env' in kw: env = kw['env'] else: @@ -201,32 +218,35 @@ def request(self, localpart='/', method='GET', data=None, if 'HTTP_CONTENT_TYPE' in env: env['CONTENT_TYPE'] = env.pop('HTTP_CONTENT_TYPE') - if method in ["POST", "PUT"]: + if method not in ["HEAD", "GET"]: data = data or '' - import StringIO + if isinstance(data, dict): - q = urllib.urlencode(data) + q = urlencode(data) else: q = data - env['wsgi.input'] = StringIO.StringIO(q) - if not env.get('CONTENT_TYPE', '').lower().startswith('multipart/') and 'CONTENT_LENGTH' not in env: + + env['wsgi.input'] = BytesIO(q.encode('utf-8')) + if 'CONTENT_LENGTH' not in env: + #if not env.get('CONTENT_TYPE', '').lower().startswith('multipart/') and 'CONTENT_LENGTH' not in env: env['CONTENT_LENGTH'] = len(q) response = web.storage() def start_response(status, headers): response.status = status response.headers = dict(headers) response.header_items = headers - response.data = "".join(self.wsgifunc()(env, start_response)) + + data = self.wsgifunc()(env, start_response) + response.data = b"".join(data) return response def browser(self): - import browser return browser.AppBrowser(self) def handle(self): fn, args = self._match(self.mapping, web.ctx.path) return self._delegate(fn, self.fvars, args) - + def handle_with_processors(self): def process(processors): try: @@ -240,12 +260,12 @@ def process(processors): except (KeyboardInterrupt, SystemExit): raise except: - print >> web.debug, traceback.format_exc() + print(traceback.format_exc(), file=web.debug) raise self.internalerror() - + # processors must be applied in the resvere order. (??) return process(self.processors) - + def wsgifunc(self, *middleware): """Returns a WSGI-compatible function for this application.""" def peep(iterator): @@ -256,14 +276,12 @@ def peep(iterator): # so we need to do an iteration # and save the result for later try: - firstchunk = iterator.next() + firstchunk = next(iterator) except StopIteration: firstchunk = '' - return itertools.chain([firstchunk], iterator) - - def is_generator(x): return x and hasattr(x, 'next') - + return itertools.chain([firstchunk], iterator) + def wsgi(env, start_resp): # clear threadlocal to avoid inteference of previous requests self._cleanup() @@ -275,25 +293,37 @@ def wsgi(env, start_resp): raise web.nomethod() result = self.handle_with_processors() - if is_generator(result): + if is_iter(result): result = peep(result) else: result = [result] - except web.HTTPError, e: + except web.HTTPError as e: result = [e.data] - result = web.safestr(iter(result)) + def build_result(result): + for r in result: + if PY2: + yield utils.safestr(r) + else: + if isinstance(r, bytes): + yield r + elif isinstance(r, string_types): + yield r.encode('utf-8') + else: + yield str(r).encode('utf-8') + + result = build_result(result) status, headers = web.ctx.status, web.ctx.headers start_resp(status, headers) - + def cleanup(): self._cleanup() - yield '' # force this function to be a generator - + yield b'' # force this function to be a generator + return itertools.chain(result, cleanup()) - for m in middleware: + for m in middleware: wsgi = m(wsgi) return wsgi @@ -304,17 +334,24 @@ def run(self, *middleware): that protocol. If called from the command line, it will start an HTTP server on the port named in the first command line argument, or, if there is no argument, on port 8080. - + `middleware` is a list of WSGI middleware which is applied to the resulting WSGI function. """ return wsgi.runwsgi(self.wsgifunc(*middleware)) - + + def stop(self): + """Stops the http server started by run. + """ + if httpserver.server: + httpserver.server.stop() + httpserver.server = None + def cgirun(self, *middleware): """ Return a CGI handler. This is mostly useful with Google App Engine. There you can just do: - + main = app.cgirun() """ wsgiapp = self.wsgifunc(*middleware) @@ -325,7 +362,49 @@ def cgirun(self, *middleware): except ImportError: # we're not running from within Google App Engine return wsgiref.handlers.CGIHandler().run(wsgiapp) - + + def gaerun(self, *middleware): + """ + Starts the program in a way that will work with Google app engine, + no matter which version you are using (2.5 / 2.7) + + If it is 2.5, just normally start it with app.gaerun() + + If it is 2.7, make sure to change the app.yaml handler to point to the + global variable that contains the result of app.gaerun() + + For example: + + in app.yaml (where code.py is where the main code is located) + + handlers: + - url: /.* + script: code.app + + Make sure that the app variable is globally accessible + """ + wsgiapp = self.wsgifunc(*middleware) + try: + # check what version of python is running + version = sys.version_info[:2] + major = version[0] + minor = version[1] + + if major != 2: + raise EnvironmentError("Google App Engine only supports python 2.5 and 2.7") + + # if 2.7, return a function that can be run by gae + if minor == 7: + return wsgiapp + # if 2.5, use run_wsgi_app + elif minor == 5: + from google.appengine.ext.webapp.util import run_wsgi_app + return run_wsgi_app(wsgiapp) + else: + raise EnvironmentError("Not a supported platform, use python 2.5 or 2.7") + except ImportError: + return wsgiref.handlers.CGIHandler().run(wsgiapp) + def load(self, env): """Initializes ctx using env.""" ctx = web.ctx @@ -354,9 +433,9 @@ def load(self, env): # http://trac.lighttpd.net/trac/ticket/406 requires: if env.get('SERVER_SOFTWARE', '').startswith('lighttpd/'): ctx.path = lstrips(env.get('REQUEST_URI').split('?')[0], ctx.homepath) - # Apache and CherryPy webservers unquote the url but lighttpd doesn't. + # Apache and CherryPy webservers unquote the url but lighttpd doesn't. # unquote explicitly for lighttpd to make ctx.path uniform across all servers. - ctx.path = urllib.unquote(ctx.path) + ctx.path = unquote(ctx.path) if env.get('QUERY_STRING'): ctx.query = '?' + env.get('QUERY_STRING', '') @@ -364,14 +443,16 @@ def load(self, env): ctx.query = '' ctx.fullpath = ctx.path + ctx.query - - for k, v in ctx.iteritems(): - if isinstance(v, str): - ctx[k] = safeunicode(v) + + for k, v in iteritems(ctx): + # convert all string values to unicode values and replace + # malformed data with a suitable replacement marker. + if isinstance(v, bytes): + ctx[k] = v.decode('utf-8', 'replace') # status must always be str ctx.status = '200 OK' - + ctx.app_stack = [] def _delegate(self, f, fvars, args=[]): @@ -383,16 +464,14 @@ def handle_class(cls): raise web.nomethod(cls) tocall = getattr(cls(), meth) return tocall(*args) - - def is_class(o): return isinstance(o, (types.ClassType, type)) - + if f is None: raise web.notfound() elif isinstance(f, application): return f.handle_with_processors() - elif is_class(f): + elif isclass(f): return handle_class(f) - elif isinstance(f, basestring): + elif isinstance(f, string_types): if f.startswith('redirect '): url = f.split(' ', 1)[1] if web.ctx.method == "GET": @@ -420,20 +499,20 @@ def _match(self, mapping, value): return f, None else: continue - elif isinstance(what, basestring): - what, result = utils.re_subm('^' + pat + '$', what, value) + elif isinstance(what, string_types): + what, result = utils.re_subm(r'^%s\Z' % (pat,), what, value) else: - result = utils.re_compile('^' + pat + '$').match(value) - + result = utils.re_compile(r'^%s\Z' % (pat,)).match(value) + if result: # it's a match return what, [x for x in result.groups()] return None, None - + def _delegate_sub_application(self, dir, app): """Deletes request to sub application `app` rooted at the directory `dir`. The home, homepath, path and fullpath values in web.ctx are updated to mimic request - to the subapp and are restored after it is handled. - + to the subapp and are restored after it is handled. + @@Any issues with when used with yield? """ web.ctx._oldctx = web.storage(web.ctx) @@ -442,13 +521,13 @@ def _delegate_sub_application(self, dir, app): web.ctx.path = web.ctx.path[len(dir):] web.ctx.fullpath = web.ctx.fullpath[len(dir):] return app.handle_with_processors() - + def get_parent_app(self): if self in web.ctx.app_stack: index = web.ctx.app_stack.index(self) if index > 0: return web.ctx.app_stack[index-1] - + def notfound(self): """Returns HTTPError with '404 not found' message""" parent = self.get_parent_app() @@ -456,21 +535,29 @@ def notfound(self): return parent.notfound() else: return web._NotFound() - + def internalerror(self): """Returns HTTPError with '500 internal error' message""" parent = self.get_parent_app() if parent: return parent.internalerror() elif web.config.get('debug'): - import debugerror - return debugerror.debugerror() + return debugerror() else: return web._InternalError() +def with_metaclass(mcls): + def decorator(cls): + body = vars(cls).copy() + # clean out class body + body.pop('__dict__', None) + body.pop('__weakref__', None) + return mcls(cls.__name__, cls.__bases__, body) + return decorator + class auto_application(application): - """Application similar to `application` but urls are constructed - automatiacally using metaclass. + """Application similar to `application` but urls are constructed + automatically using metaclass. >>> app = auto_application() >>> class hello(app.page): @@ -480,9 +567,9 @@ class auto_application(application): ... path = '/foo/.*' ... def GET(self): return "foo" >>> app.request("/hello").data - 'hello, world' + b'hello, world' >>> app.request('/foo/bar').data - 'foo' + b'foo' """ def __init__(self): application.__init__(self) @@ -497,15 +584,16 @@ def __init__(klass, name, bases, attrs): if path is not None: self.add_mapping(path, klass) - class page: + + @with_metaclass(metapage) #little hack needed or Py2 and Py3 compatibility + class page(): path = None - __metaclass__ = metapage self.page = page # The application class already has the required functionality of subdir_application subdir_application = application - + class subdomain_application(application): """ Application to delegate requests based on the host. @@ -518,21 +606,21 @@ class subdomain_application(application): >>> mapping = (r"hello\.example\.com", app) >>> app2 = subdomain_application(mapping) >>> app2.request("/hello", host="hello.example.com").data - 'hello' + b'hello' >>> response = app2.request("/hello", host="something.example.com") >>> response.status '404 Not Found' >>> response.data - 'not found' + b'not found' """ def handle(self): host = web.ctx.host.split(':')[0] #strip port fn, args = self._match(self.mapping, host) return self._delegate(fn, self.fvars, args) - + def _match(self, mapping, value): for pat, what in mapping: - if isinstance(what, basestring): + if isinstance(what, string_types): what, result = utils.re_subm('^' + pat + '$', what, value) else: result = utils.re_compile('^' + pat + '$').match(value) @@ -540,11 +628,11 @@ def _match(self, mapping, value): if result: # it's a match return what, [x for x in result.groups()] return None, None - + def loadhook(h): """ Converts a load hook into an application processor. - + >>> app = auto_application() >>> def f(): "something done before handling request" ... @@ -553,37 +641,37 @@ def loadhook(h): def processor(handler): h() return handler() - + return processor - + def unloadhook(h): """ Converts an unload hook into an application processor. - + >>> app = auto_application() >>> def f(): "something done after handling request" ... - >>> app.add_processor(unloadhook(f)) + >>> app.add_processor(unloadhook(f)) """ def processor(handler): try: result = handler() - is_generator = result and hasattr(result, 'next') + is_gen = is_iter(result) except: # run the hook even when handler raises some exception h() raise - if is_generator: + if is_gen: return wrap(result) else: h() return result - + def wrap(result): - def next(): + def next_hook(): try: - return result.next() + return next(result) except: # call the hook at the and of iterator h() @@ -591,8 +679,11 @@ def next(): result = iter(result) while True: - yield next() - + try: + yield next_hook() + except StopIteration: + return + return processor def autodelegate(prefix=''): @@ -607,9 +698,9 @@ class prefs: def GET_password(self): pass def GET_privacy(self): pass - `GET_password` would get called for `/prefs/password` while `GET_privacy` for + `GET_password` would get called for `/prefs/password` while `GET_privacy` for `GET_privacy` gets called for `/prefs/privacy`. - + If a user visits `/prefs/password/change` then `GET_password(self, '/change')` is called. """ @@ -621,7 +712,7 @@ def internal(self, arg): else: func = prefix + arg args = [] - + if hasattr(self, func): try: return getattr(self, func)(*args) @@ -632,12 +723,15 @@ def internal(self, arg): return internal class Reloader: - """Checks to see if any loaded modules have changed on disk and, + """Checks to see if any loaded modules have changed on disk and, if so, reloads them. """ - SUFFIX = '$py.class' if sys.platform.startswith('java') else '.pyc' """File suffix of compiled modules.""" + if sys.platform.startswith('java'): + SUFFIX = '$py.class' + else: + SUFFIX = '.pyc' def __init__(self): self.mtimes = {} @@ -652,22 +746,22 @@ def check(self, mod): if not (mod and hasattr(mod, '__file__') and mod.__file__): return - try: + try: mtime = os.stat(mod.__file__).st_mtime except (OSError, IOError): return if mod.__file__.endswith(self.__class__.SUFFIX) and os.path.exists(mod.__file__[:-1]): mtime = max(os.stat(mod.__file__[:-1]).st_mtime, mtime) - + if mod not in self.mtimes: self.mtimes[mod] = mtime elif self.mtimes[mod] < mtime: - try: + try: reload(mod) self.mtimes[mod] = mtime - except ImportError: + except ImportError: pass - + if __name__ == "__main__": import doctest doctest.testmod() diff --git a/lib/web/browser.py b/lib/web/browser.py index 66d859e..8fe2392 100644 --- a/lib/web/browser.py +++ b/lib/web/browser.py @@ -1,12 +1,41 @@ """Browser to test web applications. (from web.py) """ -from utils import re_compile -from net import htmlunquote - -import httplib, urllib, urllib2 -import copy -from StringIO import StringIO +import os +import webbrowser +from io import BytesIO + +from .net import htmlunquote +from .py3helpers import PY2, text_type +from .utils import re_compile + +try: # Py3 + from http.client import HTTPMessage + from urllib.request import HTTPHandler, HTTPCookieProcessor, Request, HTTPError + from urllib.request import build_opener as urllib_build_opener + from urllib.parse import urljoin + from http.cookiejar import CookieJar + from urllib.response import addinfourl +except ImportError: # Py2 + from httplib import HTTPMessage + from urllib import addinfourl + from urllib2 import HTTPHandler, HTTPCookieProcessor, Request, HTTPError + from urllib2 import build_opener as urllib_build_opener + from cookielib import CookieJar + from urlparse import urljoin + +# Welcome to the Py2->Py3 httplib/urllib reorganization nightmare. + +if PY2: + get_selector = lambda x: x.get_selector() + get_host = lambda x: x.get_host() + get_data = lambda x: x.get_data() + get_type = lambda x: x.get_type() +else: + get_selector = lambda x: x.selector + get_host = lambda x: x.host + get_data = lambda x: x.data + get_type = lambda x: x.type DEBUG = False @@ -19,53 +48,59 @@ class BrowserError(Exception): pass -class Browser: +class Browser(object): def __init__(self): - import cookielib - self.cookiejar = cookielib.CookieJar() - self._cookie_processor = urllib2.HTTPCookieProcessor(self.cookiejar) + self.cookiejar = CookieJar() + self._cookie_processor = HTTPCookieProcessor(self.cookiejar) self.form = None self.url = "http://0.0.0.0:8080/" self.path = "/" - + self.status = None self.data = None self._response = None self._forms = None + @property + def text(self): + return self.data.decode('utf-8') + def reset(self): """Clears all cookies and history.""" self.cookiejar.clear() def build_opener(self): - """Builds the opener using urllib2.build_opener. + """Builds the opener using (urllib2/urllib.request).build_opener. Subclasses can override this function to prodive custom openers. """ - return urllib2.build_opener() + return urllib_build_opener() def do_request(self, req): if DEBUG: - print 'requesting', req.get_method(), req.get_full_url() + print('requesting', req.get_method(), req.get_full_url()) + opener = self.build_opener() opener.add_handler(self._cookie_processor) try: self._response = opener.open(req) - except urllib2.HTTPError, e: + except HTTPError as e: self._response = e self.url = self._response.geturl() - self.path = urllib2.Request(self.url).get_selector() + self.path = get_selector(Request(self.url)) self.data = self._response.read() self.status = self._response.code self._forms = None self.form = None + return self.get_response() def open(self, url, data=None, headers={}): """Opens the specified url.""" - url = urllib.basejoin(self.url, url) - req = urllib2.Request(url, data, headers) + url = urljoin(self.url, url) + req = Request(url, data, headers) + return self.do_request(req) def show(self): @@ -74,13 +109,12 @@ def show(self): f.write(self.data) f.close() - import webbrowser, os url = 'file://' + os.path.abspath('page.html') webbrowser.open(url) def get_response(self): """Returns a copy of the current response.""" - return urllib.addinfourl(StringIO(self.data), self._response.info(), self._response.geturl()) + return addinfourl(BytesIO(self.data), self._response.info(), self._response.geturl()) def get_soup(self): """Returns beautiful soup of the current document.""" @@ -90,12 +124,13 @@ def get_soup(self): def get_text(self, e=None): """Returns content of e or the current document as plain text.""" e = e or self.get_soup() - return ''.join([htmlunquote(c) for c in e.recursiveChildGenerator() if isinstance(c, unicode)]) + return ''.join([htmlunquote(c) for c in e.recursiveChildGenerator() + if isinstance(c, text_type)]) def _get_links(self): soup = self.get_soup() return [a for a in soup.findAll(name='a')] - + def get_links(self, text=None, text_regex=None, url=None, url_regex=None, predicate=None): """Returns all links in the document.""" return self._filter_links(self._get_links(), @@ -106,18 +141,18 @@ def follow_link(self, link=None, text=None, text_regex=None, url=None, url_regex links = self._filter_links(self.get_links(), text=text, text_regex=text_regex, url=url, url_regex=url_regex, predicate=predicate) link = links and links[0] - + if link: return self.open(link['href']) else: raise BrowserError("No link found") - + def find_link(self, text=None, text_regex=None, url=None, url_regex=None, predicate=None): - links = self._filter_links(self.get_links(), + links = self._filter_links(self.get_links(), text=text, text_regex=text_regex, url=url, url_regex=url_regex, predicate=predicate) return links and links[0] or None - - def _filter_links(self, links, + + def _filter_links(self, links, text=None, text_regex=None, url=None, url_regex=None, predicate=None): @@ -158,13 +193,13 @@ def select_form(self, name=None, predicate=None, index=0): forms = [f for f in forms if f.name == name] if predicate: forms = [f for f in forms if predicate(f)] - + if forms: self.form = forms[index] return self.form else: raise BrowserError("No form selected.") - + def submit(self, **kw): """submits the currently selected form.""" if self.form is None: @@ -180,11 +215,11 @@ def __setitem__(self, key, value): class AppBrowser(Browser): """Browser interface to test web.py apps. - + b = AppBrowser(app) b.open('/') b.follow_link(text='Login') - + b.select_form(name='login') b['username'] = 'joe' b['password'] = 'secret' @@ -198,9 +233,9 @@ def __init__(self, app): self.app = app def build_opener(self): - return urllib2.build_opener(AppHandler(self.app)) + return urllib_build_opener(AppHandler(self.app)) -class AppHandler(urllib2.HTTPHandler): +class AppHandler(HTTPHandler): """urllib2 handler to handle requests using web.py application.""" handler_order = 100 @@ -209,28 +244,35 @@ def __init__(self, app): def http_open(self, req): result = self.app.request( - localpart=req.get_selector(), + localpart=get_selector(req), method=req.get_method(), - host=req.get_host(), - data=req.get_data(), + host=get_host(req), + data=get_data(req), headers=dict(req.header_items()), - https=req.get_type() == "https" + https=get_type(req) == "https" ) return self._make_response(result, req.get_full_url()) def https_open(self, req): return self.http_open(req) - + try: - https_request = urllib2.HTTPHandler.do_request_ + https_request = HTTPHandler.do_request_ except AttributeError: # for python 2.3 pass def _make_response(self, result, url): + data = "\r\n".join(["%s: %s" % (k, v) for k, v in result.header_items]) - headers = httplib.HTTPMessage(StringIO(data)) - response = urllib.addinfourl(StringIO(result.data), headers, url) + + if PY2: + headers = HTTPMessage(BytesIO(data)) + else: + import email + headers = email.message_from_string(data) + + response = addinfourl(BytesIO(result.data), headers, url) code, msg = result.status.split(None, 1) response.code, response.msg = int(code), msg return response diff --git a/lib/web/contrib/template.py b/lib/web/contrib/template.py index 7495d39..b2480e3 100644 --- a/lib/web/contrib/template.py +++ b/lib/web/contrib/template.py @@ -5,7 +5,7 @@ __all__ = [ "render_cheetah", "render_genshi", "render_mako", - "cache", + "cache", ] class render_cheetah: @@ -24,13 +24,13 @@ def __init__(self, path): def __getattr__(self, name): from Cheetah.Template import Template path = os.path.join(self.path, name + ".html") - + def template(**kw): t = Template(file=path, searchList=[kw]) return t.respond() return template - + class render_genshi: """Rendering interface genshi templates. Example: @@ -75,7 +75,7 @@ def template(**kw): class render_jinja: """Rendering interface to Jinja2 Templates - + Example: render= render_jinja('templates') @@ -88,13 +88,13 @@ def __init__(self, *a, **kwargs): from jinja2 import Environment,FileSystemLoader self._lookup = Environment(loader=FileSystemLoader(*a, **kwargs), extensions=extensions) self._lookup.globals.update(globals) - + def __getattr__(self, name): # Assuming all templates end with .html path = name + '.html' t = self._lookup.get_template(path) return t.render - + class render_mako: """Rendering interface to Mako Templates. @@ -115,7 +115,7 @@ def __getattr__(self, name): class cache: """Cache for any rendering interface. - + Example: render = cache(render_cheetah("templates/")) diff --git a/lib/web/db.py b/lib/web/db.py index f0af55b..268601a 100644 --- a/lib/web/db.py +++ b/lib/web/db.py @@ -2,40 +2,54 @@ Database API (part of web.py) """ +from __future__ import print_function -__all__ = [ - "UnknownParamstyle", "UnknownDB", "TransactionError", - "sqllist", "sqlors", "reparam", "sqlquote", - "SQLQuery", "SQLParam", "sqlparam", - "SQLLiteral", "sqlliteral", - "database", 'DB', -] - +import datetime +import os +import re import time + +from .py3helpers import PY2, iteritems, numeric_types, string_types, text_type +from .utils import (iterbetter, iters, safestr, safeunicode, storage, + threadeddict) + try: - import datetime + from urllib import parse as urlparse + from urllib.parse import unquote except ImportError: - datetime = None + import urlparse + from urllib import unquote -try: set -except NameError: - from sets import Set as set - -from utils import threadeddict, storage, iters, iterbetter, safestr, safeunicode +try: + import ast +except ImportError: + ast = None try: # db module can work independent of web.py - from webapi import debug, config -except: + from .webapi import debug, config +except ImportError: import sys debug = sys.stderr config = storage() +__all__ = [ + "UnknownParamstyle", "UnknownDB", "TransactionError", + "sqllist", "sqlors", "reparam", "sqlquote", + "SQLQuery", "SQLParam", "sqlparam", + "SQLLiteral", "sqlliteral", + "database", 'DB', +] + +TOKEN = '[ \\f\\t]*(\\\\\\r?\\n[ \\f\\t]*)*(#[^\\r\\n]*)?(((\\d+[jJ]|((\\d+\\.\\d*|\\.\\d+)([eE][-+]?\\d+)?|\\d+[eE][-+]?\\d+)[jJ])|((\\d+\\.\\d*|\\.\\d+)([eE][-+]?\\d+)?|\\d+[eE][-+]?\\d+)|(0[xX][\\da-fA-F]+[lL]?|0[bB][01]+[lL]?|(0[oO][0-7]+)|(0[0-7]*)[lL]?|[1-9]\\d*[lL]?))|((\\*\\*=?|>>=?|<<=?|<>|!=|//=?|[+\\-*/%&|^=<>]=?|~)|[][(){}]|(\\r?\\n|[:;.,`@]))|([uUbB]?[rR]?\'[^\\n\'\\\\]*(?:\\\\.[^\\n\'\\\\]*)*\'|[uUbB]?[rR]?"[^\\n"\\\\]*(?:\\\\.[^\\n"\\\\]*)*")|[a-zA-Z_]\\w*)' + +tokenprog = re.compile(TOKEN) + class UnknownDB(Exception): """raised for unsupported dbms""" pass -class _ItplError(ValueError): +class _ItplError(ValueError): def __init__(self, text, pos): ValueError.__init__(self) self.text = text @@ -46,18 +60,18 @@ def __str__(self): class TransactionError(Exception): pass -class UnknownParamstyle(Exception): +class UnknownParamstyle(Exception): """ raised for unsupported db paramstyles (currently supported: qmark, numeric, format, pyformat) """ pass - + class SQLParam(object): """ Parameter in SQLQuery. - + >>> q = SQLQuery(["SELECT * FROM test WHERE name=", SQLParam("joe")]) >>> q @@ -70,7 +84,7 @@ class SQLParam(object): def __init__(self, value): self.value = value - + def get_marker(self, paramstyle='pyformat'): if paramstyle == 'qmark': return '?' @@ -78,20 +92,23 @@ def get_marker(self, paramstyle='pyformat'): return ':1' elif paramstyle is None or paramstyle in ['format', 'pyformat']: return '%s' - raise UnknownParamstyle, paramstyle - - def sqlquery(self): + raise UnknownParamstyle(paramstyle) + + def sqlquery(self): return SQLQuery([self]) - + def __add__(self, other): return self.sqlquery() + other - + def __radd__(self, other): - return other + self.sqlquery() - - def __str__(self): + return other + self.sqlquery() + + def __str__(self): return str(self.value) - + + def __eq__(self, other): + return isinstance(other, SQLParam) and other.value == self.value + def __repr__(self): return '' % repr(self.value) @@ -111,7 +128,7 @@ class SQLQuery(object): # tested in sqlquote's docstring def __init__(self, items=None): r"""Creates a new SQLQuery. - + >>> SQLQuery("x") >>> q = SQLQuery(['SELECT * FROM ', 'test', ' WHERE x=', SQLParam(1)]) @@ -132,7 +149,7 @@ def __init__(self, items=None): self.items = list(items.items) else: self.items = [items] - + # Take care of SQLLiterals for i, item in enumerate(self.items): if isinstance(item, SQLParam) and isinstance(item.value, SQLLiteral): @@ -142,7 +159,7 @@ def append(self, value): self.items.append(value) def __add__(self, other): - if isinstance(other, basestring): + if isinstance(other, string_types): items = [other] elif isinstance(other, SQLQuery): items = other.items @@ -151,15 +168,16 @@ def __add__(self, other): return SQLQuery(self.items + items) def __radd__(self, other): - if isinstance(other, basestring): + if isinstance(other, string_types): items = [other] + elif isinstance(other, SQLQuery): + items = other.items else: return NotImplemented - return SQLQuery(items + self.items) def __iadd__(self, other): - if isinstance(other, (basestring, SQLParam)): + if isinstance(other, (string_types, SQLParam)): self.items.append(other) elif isinstance(other, SQLQuery): self.items.extend(other.items) @@ -169,7 +187,10 @@ def __iadd__(self, other): def __len__(self): return len(self.query()) - + + def __eq__(self, other): + return isinstance(other, SQLQuery) and other.items == self.items + def query(self, paramstyle=None): """ Returns the query part of the sql query. @@ -193,7 +214,7 @@ def query(self, paramstyle=None): x = x.replace('%', '%%') s.append(x) return "".join(s) - + def values(self): """ Returns the values of the parameters used in the sql query. @@ -202,11 +223,11 @@ def values(self): ['joe'] """ return [i.value for i in self.items if isinstance(i, SQLParam)] - + def join(items, sep=' ', prefix=None, suffix=None, target=None): """ Joins multiple queries. - + >>> SQLQuery.join(['a', 'b'], ', ') @@ -226,35 +247,37 @@ def join(items, sep=' ', prefix=None, suffix=None, target=None): target_items.append(prefix) for i, item in enumerate(items): - if i != 0: + if i != 0 and sep != "": target_items.append(sep) if isinstance(item, SQLQuery): target_items.extend(item.items) + elif item == "": # joins with empty strings + continue else: target_items.append(item) if suffix: target_items.append(suffix) return target - + join = staticmethod(join) - + def _str(self): try: - return self.query() % tuple([sqlify(x) for x in self.values()]) + return self.query() % tuple([sqlify(x) for x in self.values()]) except (ValueError, TypeError): return self.query() - + def __str__(self): return safestr(self._str()) - + def __unicode__(self): return safeunicode(self._str()) def __repr__(self): return '' % repr(str(self)) -class SQLLiteral: +class SQLLiteral: """ Protects a string from `sqlquote`. @@ -263,11 +286,11 @@ class SQLLiteral: >>> sqlquote(SQLLiteral('NOW()')) """ - def __init__(self, v): + def __init__(self, v): self.v = v - def __repr__(self): - return self.v + def __repr__(self): + return "" % self.v sqlliteral = SQLLiteral @@ -285,7 +308,7 @@ def _sqllist(values): items.append(')') return SQLQuery(items) -def reparam(string_, dictionary): +def reparam(string_, dictionary): """ Takes a string and a dictionary and interpolates the string using values from the dictionary. Returns an `SQLQuery` for the result. @@ -295,18 +318,21 @@ def reparam(string_, dictionary): >>> reparam("s IN $s", dict(s=[1, 2])) """ + return SafeEval().safeeval(string_, dictionary) + dictionary = dictionary.copy() # eval mucks with it - vals = [] + # disable builtins to avoid risk for remote code exection. + dictionary['__builtins__'] = object() result = [] for live, chunk in _interpolate(string_): if live: v = eval(chunk, dictionary) result.append(sqlquote(v)) - else: + else: result.append(chunk) return SQLQuery.join(result, '') -def sqlify(obj): +def sqlify(obj): """ converts `obj` to its proper SQL version @@ -326,31 +352,33 @@ def sqlify(obj): return "'t'" elif obj is False: return "'f'" - elif datetime and isinstance(obj, datetime.datetime): + elif isinstance(obj, numeric_types): + return str(obj) + elif isinstance(obj, datetime.datetime): return repr(obj.isoformat()) else: - if isinstance(obj, unicode): obj = obj.encode('utf8') + if PY2 and isinstance(obj, text_type): # Strings are always UTF8 in Py3 + obj = obj.encode('utf8') + return repr(obj) -def sqllist(lst): +def sqllist(lst): """ Converts the arguments for use in something like a WHERE clause. - + >>> sqllist(['a', 'b']) 'a, b' >>> sqllist('a') 'a' - >>> sqllist(u'abc') - u'abc' """ - if isinstance(lst, basestring): + if isinstance(lst, string_types): return lst else: return ', '.join(lst) def sqlors(left, lst): """ - `left is a SQL clause like `tablename.arg = ` + `left is a SQL clause like `tablename.arg = ` and `lst` is a list of values. Returns a reparam-style pair featuring the SQL that ORs together the clause for each item in the lst. @@ -373,27 +401,28 @@ def sqlors(left, lst): lst = lst[0] if isinstance(lst, iters): - return SQLQuery(['('] + + return SQLQuery(['('] + sum([[left, sqlparam(x), ' OR '] for x in lst], []) + ['1=2)'] ) else: return left + sqlparam(lst) - -def sqlwhere(dictionary, grouping=' AND '): + +def sqlwhere(data, grouping=' AND '): """ - Converts a `dictionary` to an SQL WHERE clause `SQLQuery`. - - >>> sqlwhere({'cust_id': 2, 'order_id':3}) - - >>> sqlwhere({'cust_id': 2, 'order_id':3}, grouping=', ') + Converts a two-tuple (key, value) iterable `data` to an SQL WHERE clause `SQLQuery`. + + >>> sqlwhere((('cust_id', 2), ('order_id',3))) + + >>> sqlwhere((('order_id', 3), ('cust_id', 2)), grouping=', ') - >>> sqlwhere({'a': 'a', 'b': 'b'}).query() + >>> sqlwhere((('a', 'a'), ('b', 'b'))).query() 'a = %s AND b = %s' """ - return SQLQuery.join([k + ' = ' + sqlparam(v) for k, v in dictionary.items()], grouping) -def sqlquote(a): + return SQLQuery.join([k + ' = ' + sqlparam(v) for k, v in data], grouping) + +def sqlquote(a): """ Ensures `a` is quoted properly for use in a SQL query. @@ -440,7 +469,7 @@ def do_rollback(self): self.query('ROLLBACK TO SAVEPOINT webpy_sp_%s') class dummy_engine: - """Transaction Engine used instead of subtransaction_engine + """Transaction Engine used instead of subtransaction_engine when sub transactions are not supported.""" do_transact = do_commit = do_rollback = lambda self: None @@ -475,7 +504,7 @@ def rollback(self): self.engine.do_rollback() self.ctx.transactions = self.ctx.transactions[:self.transaction_count] -class DB: +class DB: """Database""" def __init__(self, db_module, keywords): """Creates a database. @@ -491,77 +520,77 @@ def __init__(self, db_module, keywords): # flag to enable/disable printing queries self.printing = config.get('debug_sql', config.get('debug', False)) self.supports_multiple_insert = False - + try: import DBUtils # enable pooling if DBUtils module is available. self.has_pooling = True except ImportError: self.has_pooling = False - + # Pooling can be disabled by passing pooling=False in the keywords. self.has_pooling = self.keywords.pop('pooling', True) and self.has_pooling - - def _getctx(self): + + def _getctx(self): if not self._ctx.get('db'): self._load_context(self._ctx) return self._ctx ctx = property(_getctx) - + def _load_context(self, ctx): ctx.dbq_count = 0 ctx.transactions = [] # stack of transactions - + if self.has_pooling: ctx.db = self._connect_with_pooling(self.keywords) else: ctx.db = self._connect(self.keywords) ctx.db_execute = self._db_execute - + if not hasattr(ctx.db, 'commit'): ctx.db.commit = lambda: None if not hasattr(ctx.db, 'rollback'): ctx.db.rollback = lambda: None - + def commit(unload=True): - # do db commit and release the connection if pooling is enabled. + # do db commit and release the connection if pooling is enabled. ctx.db.commit() if unload and self.has_pooling: self._unload_context(self._ctx) - + def rollback(): # do db rollback and release the connection if pooling is enabled. ctx.db.rollback() if self.has_pooling: self._unload_context(self._ctx) - + ctx.commit = commit ctx.rollback = rollback - + def _unload_context(self, ctx): del ctx.db - + def _connect(self, keywords): return self.db_module.connect(**keywords) - + def _connect_with_pooling(self, keywords): def get_pooled_db(): from DBUtils import PooledDB # In DBUtils 0.9.3, `dbapi` argument is renamed as `creator` # see Bug#122112 - + if PooledDB.__version__.split('.') < '0.9.3'.split('.'): return PooledDB.PooledDB(dbapi=self.db_module, **keywords) else: return PooledDB.PooledDB(creator=self.db_module, **keywords) - + if getattr(self, '_pooleddb', None) is None: self._pooleddb = get_pooled_db() - + return self._pooleddb.connection() - + def _db_cursor(self): return self.ctx.db.cursor() @@ -575,12 +604,12 @@ def _param_marker(self): return ':1' elif style in ['format', 'pyformat']: return '%s' - raise UnknownParamstyle, style + raise UnknownParamstyle(style) - def _db_execute(self, cur, sql_query): + def _db_execute(self, cur, sql_query): """executes an sql query""" self.ctx.dbq_count += 1 - + try: a = time.time() query, params = self._process_query(sql_query) @@ -588,7 +617,7 @@ def _db_execute(self, cur, sql_query): b = time.time() except: if self.printing: - print >> debug, 'ERR:', str(sql_query) + print('ERR:', str(sql_query), file=debug) if self.ctx.transactions: self.ctx.transactions[-1].rollback() else: @@ -596,7 +625,7 @@ def _db_execute(self, cur, sql_query): raise if self.printing: - print >> debug, '%s (%s): %s' % (round(b-a, 2), self.ctx.dbq_count, str(sql_query)) + print('%s (%s): %s' % (round(b-a, 2), self.ctx.dbq_count, str(sql_query)), file=debug) return out def _process_query(self, sql_query): @@ -606,25 +635,37 @@ def _process_query(self, sql_query): query = sql_query.query(paramstyle) params = sql_query.values() return query, params - - def _where(self, where, vars): - if isinstance(where, (int, long)): + + def _where(self, where, vars): + if isinstance(where, numeric_types): where = "id = " + sqlparam(where) #@@@ for backward-compatibility elif isinstance(where, (list, tuple)) and len(where) == 2: where = SQLQuery(where[0], where[1]) + elif isinstance(where, dict): + where = self._where_dict(where) elif isinstance(where, SQLQuery): pass else: - where = reparam(where, vars) + where = reparam(where, vars) return where - - def query(self, sql_query, vars=None, processed=False, _test=False): + + def _where_dict(self, where): + where_clauses = [] + + for k, v in sorted(iteritems(where), key= lambda t:t[0]): + where_clauses.append(k + ' = ' + sqlquote(v)) + if where_clauses: + return SQLQuery.join(where_clauses, " AND ") + else: + return None + + def query(self, sql_query, vars=None, processed=False, _test=False): """ Execute SQL query `sql_query` using dictionary `vars` to interpolate it. - If `processed=True`, `vars` is a `reparam`-style list to use + If `processed=True`, `vars` is a `reparam`-style list to use instead of interpolating. - + >>> db = DB(None, {}) >>> db.query("SELECT * FROM foo", _test=True) @@ -634,15 +675,15 @@ def query(self, sql_query, vars=None, processed=False, _test=False): """ if vars is None: vars = {} - + if not processed and not isinstance(sql_query, SQLQuery): sql_query = reparam(sql_query, vars) - + if _test: return sql_query - + db_cursor = self._db_cursor() self._db_execute(db_cursor, sql_query) - + if db_cursor.description: names = [x[0] for x in db_cursor.description] def iterwrapper(): @@ -656,23 +697,25 @@ def iterwrapper(): for x in db_cursor.fetchall()] else: out = db_cursor.rowcount - - if not self.ctx.transactions: + + if not self.ctx.transactions: self.ctx.commit() return out - - def select(self, tables, vars=None, what='*', where=None, order=None, group=None, - limit=None, offset=None, _test=False): + + def select(self, tables, vars=None, what='*', where=None, order=None, group=None, + limit=None, offset=None, _test=False): """ - Selects `what` from `tables` with clauses `where`, `order`, - `group`, `limit`, and `offset`. Uses vars to interpolate. + Selects `what` from `tables` with clauses `where`, `order`, + `group`, `limit`, and `offset`. Uses vars to interpolate. Otherwise, each clause can be a SQLQuery. - + >>> db = DB(None, {}) >>> db.select('foo', _test=True) >>> db.select(['foo', 'bar'], where="foo.bar_id = bar.id", limit=5, _test=True) + >>> db.select('foo', where={'id': 5}, _test=True) + """ if vars is None: vars = {} sql_clauses = self.sql_clauses(what, tables, where, group, order, limit, offset) @@ -680,45 +723,40 @@ def select(self, tables, vars=None, what='*', where=None, order=None, group=None qout = SQLQuery.join(clauses) if _test: return qout return self.query(qout, processed=True) - - def where(self, table, what='*', order=None, group=None, limit=None, + + def where(self, table, what='*', order=None, group=None, limit=None, offset=None, _test=False, **kwargs): """ Selects from `table` where keys are equal to values in `kwargs`. - + >>> db = DB(None, {}) >>> db.where('foo', bar_id=3, _test=True) >>> db.where('foo', source=2, crust='dewey', _test=True) - + >>> db.where('foo', _test=True) """ - where_clauses = [] - for k, v in kwargs.iteritems(): - where_clauses.append(k + ' = ' + sqlquote(v)) - - if where_clauses: - where = SQLQuery.join(where_clauses, " AND ") - else: - where = None - - return self.select(table, what=what, order=order, - group=group, limit=limit, offset=offset, _test=_test, + where = self._where_dict(kwargs) + return self.select(table, what=what, order=order, + group=group, limit=limit, offset=offset, _test=_test, where=where) - - def sql_clauses(self, what, tables, where, group, order, limit, offset): + + def sql_clauses(self, what, tables, where, group, order, limit, offset): return ( ('SELECT', what), ('FROM', sqllist(tables)), ('WHERE', where), ('GROUP BY', group), ('ORDER BY', order), - ('LIMIT', limit), - ('OFFSET', offset)) - - def gen_clause(self, sql, val, vars): - if isinstance(val, (int, long)): + # The limit and offset could be the values provided by + # the end-user and are potentially unsafe. + # Using them as parameters to avoid any risk. + ('LIMIT', limit and SQLParam(limit).sqlquery()), + ('OFFSET', offset and SQLParam(offset).sqlquery())) + + def gen_clause(self, sql, val, vars): + if isinstance(val, numeric_types): if sql == 'WHERE': nout = 'id = ' + sqlquote(val) else: @@ -726,6 +764,8 @@ def gen_clause(self, sql, val, vars): #@@@ elif isinstance(val, (list, tuple)) and len(val) == 2: nout = SQLQuery(val[0], val[1]) # backwards-compatibility + elif sql == 'WHERE' and isinstance(val, dict): + nout = self._where_dict(val) elif isinstance(val, SQLQuery): nout = val else: @@ -737,38 +777,42 @@ def xjoin(a, b): return xjoin(sql, nout) - def insert(self, tablename, seqname=None, _test=False, **values): + def insert(self, tablename, seqname=None, _test=False, **values): """ Inserts `values` into `tablename`. Returns current sequence ID. Set `seqname` to the ID if it's not the default, or to `False` if there isn't one. - + >>> db = DB(None, {}) >>> q = db.insert('foo', name='bob', age=2, created=SQLLiteral('NOW()'), _test=True) >>> q - + >>> q.query() - 'INSERT INTO foo (age, name, created) VALUES (%s, %s, NOW())' + 'INSERT INTO foo (age, created, name) VALUES (%s, NOW(), %s)' >>> q.values() [2, 'bob'] """ def q(x): return "(" + x + ")" - + if values: - _keys = SQLQuery.join(values.keys(), ', ') - _values = SQLQuery.join([sqlparam(v) for v in values.values()], ', ') + #needed for Py3 compatibility with the above doctests + sorted_values = sorted(values.items(), key=lambda t: t[0]) + + _keys = SQLQuery.join(map(lambda t: t[0], sorted_values), ', ') + _values = SQLQuery.join([sqlparam(v) for v in map(lambda t: t[1], sorted_values)], ', ') sql_query = "INSERT INTO %s " % tablename + q(_keys) + ' VALUES ' + q(_values) else: sql_query = SQLQuery(self._get_insert_default_values_query(tablename)) if _test: return sql_query - + db_cursor = self._db_cursor() - if seqname is not False: + if seqname is not False: sql_query = self._process_insert_query(sql_query, tablename, seqname) + if isinstance(sql_query, tuple): - # for some databases, a separate query has to be made to find + # for some databases, a separate query has to be made to find # the id of the inserted row. q1, q2 = sql_query self._db_execute(db_cursor, q1) @@ -776,49 +820,52 @@ def q(x): return "(" + x + ")" else: self._db_execute(db_cursor, sql_query) - try: + try: out = db_cursor.fetchone()[0] - except Exception: + except Exception: out = None - - if not self.ctx.transactions: + + + if not self.ctx.transactions: self.ctx.commit() + return out - + def _get_insert_default_values_query(self, table): return "INSERT INTO %s DEFAULT VALUES" % table def multiple_insert(self, tablename, values, seqname=None, _test=False): """ - Inserts multiple rows into `tablename`. The `values` must be a list of dictioanries, + Inserts multiple rows into `tablename`. The `values` must be a list of dictionaries, one for each row to be inserted, each with the same set of keys. - Returns the list of ids of the inserted rows. + Returns the list of ids of the inserted rows. Set `seqname` to the ID if it's not the default, or to `False` if there isn't one. - + >>> db = DB(None, {}) >>> db.supports_multiple_insert = True >>> values = [{"name": "foo", "email": "foo@example.com"}, {"name": "bar", "email": "bar@example.com"}] >>> db.multiple_insert('person', values=values, _test=True) - - """ + + """ if not values: return [] - + if not self.supports_multiple_insert: out = [self.insert(tablename, seqname=seqname, _test=_test, **v) for v in values] if seqname is False: return None else: return out - + keys = values[0].keys() #@@ make sure all keys are valid - # make sure all rows have same keys. for v in values: if v.keys() != keys: - raise ValueError, 'Bad data' + raise ValueError('Not all rows have the same keys') + + keys = sorted(keys) #enforce query order for the above doctest compatibility with Py3 sql_query = SQLQuery('INSERT INTO %s (%s) VALUES ' % (tablename, ', '.join(keys))) @@ -826,15 +873,15 @@ def multiple_insert(self, tablename, values, seqname=None, _test=False): if i != 0: sql_query.append(", ") SQLQuery.join([SQLParam(row[k]) for k in keys], sep=", ", target=sql_query, prefix="(", suffix=")") - + if _test: return sql_query db_cursor = self._db_cursor() - if seqname is not False: + if seqname is not False: sql_query = self._process_insert_query(sql_query, tablename, seqname) if isinstance(sql_query, tuple): - # for some databases, a separate query has to be made to find + # for some databases, a separate query has to be made to find # the id of the inserted row. q1, q2 = sql_query self._db_execute(db_cursor, q1) @@ -842,18 +889,18 @@ def multiple_insert(self, tablename, values, seqname=None, _test=False): else: self._db_execute(db_cursor, sql_query) - try: + try: out = db_cursor.fetchone()[0] - out = range(out-len(values)+1, out+1) - except Exception: + out = range(out-len(values)+1, out+1) + except Exception: out = None - if not self.ctx.transactions: + if not self.ctx.transactions: self.ctx.commit() return out - - def update(self, tables, where, vars=None, _test=False, **values): + + def update(self, tables, where, vars=None, _test=False, **values): """ Update `tables` with clause `where` (interpolated using `vars`) and setting `values`. @@ -863,29 +910,31 @@ def update(self, tables, where, vars=None, _test=False, **values): >>> q = db.update('foo', where='name = $name', name='bob', age=2, ... created=SQLLiteral('NOW()'), vars=locals(), _test=True) >>> q - + >>> q.query() - 'UPDATE foo SET age = %s, name = %s, created = NOW() WHERE name = %s' + 'UPDATE foo SET age = %s, created = NOW(), name = %s WHERE name = %s' >>> q.values() [2, 'bob', 'Joseph'] """ if vars is None: vars = {} where = self._where(where, vars) + values = sorted(values.items(), key=lambda t: t[0]) + query = ( - "UPDATE " + sqllist(tables) + - " SET " + sqlwhere(values, ', ') + + "UPDATE " + sqllist(tables) + + " SET " + sqlwhere(values, ', ') + " WHERE " + where) if _test: return query - + db_cursor = self._db_cursor() self._db_execute(db_cursor, query) - if not self.ctx.transactions: + if not self.ctx.transactions: self.ctx.commit() return db_cursor.rowcount - - def delete(self, table, where, using=None, vars=None, _test=False): + + def delete(self, table, where, using=None, vars=None, _test=False): """ Deletes from `table` with clauses `where` and `using`. @@ -905,50 +954,52 @@ def delete(self, table, where, using=None, vars=None, _test=False): db_cursor = self._db_cursor() self._db_execute(db_cursor, q) - if not self.ctx.transactions: + if not self.ctx.transactions: self.ctx.commit() return db_cursor.rowcount def _process_insert_query(self, query, tablename, seqname): return query - def transaction(self): + def transaction(self): """Start a transaction.""" return Transaction(self.ctx) - -class PostgresDB(DB): + +class PostgresDB(DB): """Postgres driver.""" def __init__(self, **keywords): if 'pw' in keywords: keywords['password'] = keywords.pop('pw') - + db_module = import_driver(["psycopg2", "psycopg", "pgdb"], preferred=keywords.pop('driver', None)) if db_module.__name__ == "psycopg2": import psycopg2.extensions psycopg2.extensions.register_type(psycopg2.extensions.UNICODE) + if db_module.__name__ == "pgdb" and 'port' in keywords: + keywords["host"] += ":" + str(keywords.pop('port')) # if db is not provided postgres driver will take it from PGDATABASE environment variable if 'db' in keywords: keywords['database'] = keywords.pop('db') - + self.dbname = "postgres" self.paramstyle = db_module.paramstyle DB.__init__(self, db_module, keywords) self.supports_multiple_insert = True self._sequences = None - + def _process_insert_query(self, query, tablename, seqname): if seqname is None: # when seqname is not provided guess the seqname and make sure it exists seqname = tablename + "_id_seq" if seqname not in self._get_all_sequences(): seqname = None - + if seqname: query += "; SELECT currval('%s')" % seqname - + return query - + def _get_all_sequences(self): """Query postgres to find names of all sequences used in this database.""" if self._sequences is None: @@ -964,18 +1015,29 @@ def _connect(self, keywords): # fallback for pgdb driver conn.cursor().execute("set client_encoding to 'UTF-8'") return conn - + def _connect_with_pooling(self, keywords): conn = DB._connect_with_pooling(self, keywords) conn._con._con.set_client_encoding('UTF8') return conn -class MySQLDB(DB): +class MySQLDB(DB): def __init__(self, **keywords): - import MySQLdb as db - if 'pw' in keywords: - keywords['passwd'] = keywords['pw'] - del keywords['pw'] + + db = import_driver(["MySQLdb", "pymysql","mysql.connector"], preferred=keywords.pop('driver', None)) + if db.__name__ == "MySQLdb": + if 'pw' in keywords: + keywords['passwd'] = keywords['pw'] + del keywords['pw'] + if db.__name__ == "pymysql": + if 'pw' in keywords: + keywords['password'] = keywords['pw'] + del keywords['pw'] + if db.__name__ == "mysql.connector": + if 'pw' in keywords: + keywords['password'] = keywords['pw'] + del keywords['pw'] + if 'charset' not in keywords: keywords['charset'] = 'utf8' @@ -986,10 +1048,10 @@ def __init__(self, **keywords): self.dbname = "mysql" DB.__init__(self, db, keywords) self.supports_multiple_insert = True - + def _process_insert_query(self, query, tablename, seqname): return query, SQLQuery('SELECT last_insert_id();') - + def _get_insert_default_values_query(self, table): return "INSERT INTO %s () VALUES()" % table @@ -1006,25 +1068,26 @@ def import_driver(drivers, preferred=None): pass raise ImportError("Unable to import " + " or ".join(drivers)) -class SqliteDB(DB): +class SqliteDB(DB): def __init__(self, **keywords): db = import_driver(["sqlite3", "pysqlite2.dbapi2", "sqlite"], preferred=keywords.pop('driver', None)) if db.__name__ in ["sqlite3", "pysqlite2.dbapi2"]: db.paramstyle = 'qmark' - + # sqlite driver doesn't create datatime objects for timestamp columns unless `detect_types` option is passed. # It seems to be supported in sqlite3 and pysqlite2 drivers, not surte about sqlite. keywords.setdefault('detect_types', db.PARSE_DECLTYPES) self.paramstyle = db.paramstyle keywords['database'] = keywords.pop('db') - self.dbname = "sqlite" + keywords['pooling'] = False # sqlite don't allows connections to be shared by threads + self.dbname = "sqlite" DB.__init__(self, db, keywords) def _process_insert_query(self, query, tablename, seqname): return query, SQLQuery('SELECT last_insert_rowid();') - + def query(self, *a, **kw): out = DB.query(self, *a, **kw) if isinstance(out, iterbetter): @@ -1041,12 +1104,13 @@ def __init__(self, **keywords): db = None pass if 'pw' in keywords: - keywords['passwd'] = keywords['pw'] - del keywords['pw'] - keywords['database'] = keywords['db'] - del keywords['db'] + keywords['password'] = keywords.pop('pw') + keywords['database'] = keywords.pop('db') + + self.paramstyle = db.paramstyle + DB.__init__(self, db, keywords) - + def delete(self, table, where=None, using=None, vars=None, _test=False): # firebird doesn't support using clause using=None @@ -1066,7 +1130,7 @@ def sql_clauses(self, what, tables, where, group, order, limit, offset): class MSSQLDB(DB): def __init__(self, **keywords): - import pymssql as db + import pymssql as db if 'pw' in keywords: keywords['password'] = keywords.pop('pw') keywords['database'] = keywords.pop('db') @@ -1076,14 +1140,14 @@ def __init__(self, **keywords): def _process_query(self, sql_query): """Takes the SQLQuery object and returns query string and parameters. """ - # MSSQLDB expects params to be a tuple. + # MSSQLDB expects params to be a tuple. # Overwriting the default implementation to convert params to tuple. paramstyle = getattr(self, 'paramstyle', 'pyformat') query = sql_query.query(paramstyle) params = sql_query.values() return query, tuple(params) - def sql_clauses(self, what, tables, where, group, order, limit, offset): + def sql_clauses(self, what, tables, where, group, order, limit, offset): return ( ('SELECT', what), ('TOP', limit), @@ -1092,14 +1156,14 @@ def sql_clauses(self, what, tables, where, group, order, limit, offset): ('GROUP BY', group), ('ORDER BY', order), ('OFFSET', offset)) - + def _test(self): """Test LIMIT. Fake presence of pymssql module for running tests. >>> import sys >>> sys.modules['pymssql'] = sys.modules['sys'] - + MSSQL has TOP clause instead of LIMIT clause. >>> db = MSSQLDB(db='test', user='joe', pw='secret') >>> db.select('foo', limit=4, _test=True) @@ -1107,52 +1171,80 @@ def _test(self): """ pass -class OracleDB(DB): - def __init__(self, **keywords): - import cx_Oracle as db - if 'pw' in keywords: - keywords['password'] = keywords.pop('pw') +class OracleDB(DB): + def __init__(self, **keywords): + import cx_Oracle as db + if 'pw' in keywords: + keywords['password'] = keywords.pop('pw') - #@@ TODO: use db.makedsn if host, port is specified - keywords['dsn'] = keywords.pop('db') - self.dbname = 'oracle' - db.paramstyle = 'numeric' + #@@ TODO: use db.makedsn if host, port is specified + keywords['dsn'] = keywords.pop('db') + self.dbname = 'oracle' + db.paramstyle = 'numeric' self.paramstyle = db.paramstyle - # oracle doesn't support pooling - keywords.pop('pooling', None) - DB.__init__(self, db, keywords) + # oracle doesn't support pooling + keywords.pop('pooling', None) + DB.__init__(self, db, keywords) - def _process_insert_query(self, query, tablename, seqname): - if seqname is None: + def _process_insert_query(self, query, tablename, seqname): + if seqname is None: # It is not possible to get seq name from table name in Oracle return query else: - return query + "; SELECT %s.currval FROM dual" % seqname + return query + "; SELECT %s.currval FROM dual" % seqname + +def dburl2dict(url): + """ + Takes a URL to a database and parses it into an equivalent dictionary. + + >>> dburl2dict('postgres:///mygreatdb') == {'pw': None, 'dbn': 'postgres', 'db': 'mygreatdb', 'host': None, 'user': None, 'port': None} + True + >>> dburl2dict('postgres://james:day@serverfarm.example.net:5432/mygreatdb') == {'pw': 'day', 'dbn': 'postgres', 'db': 'mygreatdb', 'host': 'serverfarm.example.net', 'user': 'james', 'port': 5432} + True + >>> dburl2dict('postgres://james:day@serverfarm.example.net/mygreatdb') == {'pw': 'day', 'dbn': 'postgres', 'db': 'mygreatdb', 'host': 'serverfarm.example.net', 'user': 'james', 'port': None} + True + >>> dburl2dict('postgres://james:d%40y@serverfarm.example.net/mygreatdb') == {'pw': 'd@y', 'dbn': 'postgres', 'db': 'mygreatdb', 'host': 'serverfarm.example.net', 'user': 'james', 'port': None} + True + >>> dburl2dict('mysql://james:d%40y@serverfarm.example.net/mygreatdb') == {'pw': 'd@y', 'dbn': 'mysql', 'db': 'mygreatdb', 'host': 'serverfarm.example.net', 'user': 'james', 'port': None} + True + """ + parts = urlparse.urlparse(unquote(url)) + + return {'dbn': parts.scheme, + 'user': parts.username, + 'pw': parts.password, + 'db': parts.path[1:], + 'host': parts.hostname, + 'port': parts.port} _databases = {} def database(dburl=None, **params): """Creates appropriate database using params. - - Pooling will be enabled if DBUtils module is available. + + Pooling will be enabled if DBUtils module is available. Pooling can be disabled by passing pooling=False in params. """ + if not dburl and not params: + dburl = os.environ['DATABASE_URL'] + if dburl: + params = dburl2dict(dburl) dbn = params.pop('dbn') if dbn in _databases: return _databases[dbn](**params) else: - raise UnknownDB, dbn + raise UnknownDB(dbn) def register_database(name, clazz): """ Register a database. - >>> class LegacyDB(DB): - ... def __init__(self, **params): - ... pass + >>> class LegacyDB(DB): + ... def __init__(self, **params): + ... pass ... >>> register_database('legacy', LegacyDB) - >>> db = database(dbn='legacy', db='test', user='joe', passwd='secret') + >>> db = database(dbn='legacy', db='test', user='joe', passwd='secret') """ _databases[name] = clazz @@ -1163,7 +1255,7 @@ def register_database(name, clazz): register_database('mssql', MSSQLDB) register_database('oracle', OracleDB) -def _interpolate(format): +def _interpolate(format): """ Takes a format string and returns a list of 2-tuples of the form (boolean, string) where boolean says whether string should be evaled @@ -1171,8 +1263,6 @@ def _interpolate(format): from (public domain, Ka-Ping Yee) """ - from tokenize import tokenprog - def matchorfail(text, pos): match = tokenprog.match(text, pos) if match is None: @@ -1186,7 +1276,7 @@ def matchorfail(text, pos): while 1: dollar = format.find("$", pos) - if dollar < 0: + if dollar < 0: break nextchar = format[dollar + 1] @@ -1197,9 +1287,9 @@ def matchorfail(text, pos): match, pos = matchorfail(format, pos) tstart, tend = match.regs[3] token = format[tstart:tend] - if token == "{": + if token == "{": level = level + 1 - elif token == "}": + elif token == "}": level = level - 1 chunks.append((1, format[dollar + 2:pos - 1])) @@ -1216,21 +1306,190 @@ def matchorfail(text, pos): match, pos = matchorfail(format, pos) tstart, tend = match.regs[3] token = format[tstart:tend] - if token[0] in "([": + if token[0] in "([": level = level + 1 - elif token[0] in ")]": + elif token[0] in ")]": level = level - 1 - else: + else: break chunks.append((1, format[dollar + 1:pos])) else: chunks.append((0, format[pos:dollar + 1])) pos = dollar + 1 + (nextchar == "$") - if pos < len(format): + if pos < len(format): chunks.append((0, format[pos:])) return chunks +class _Node(object): + def __init__(self, type, first, second=None): + self.type = type + self.first = first + self.second = second + + def __eq__(self, other): + return (isinstance(other, _Node) + and self.type == other.type + and self.first == other.first + and self.second == other.second) + + def __repr__(self): + return "Node(%r, %r, %r)" % (self.type, self.first, self.second) + +class Parser: + """Parser to parse string templates like "Hello $name". + + Loosely based on (public domain, Ka-Ping Yee) + """ + namechars = "abcdefghijklmnopqrstuvwxyz" \ + "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_" + + def __init__(self): + self.reset() + + def reset(self): + self.pos = 0 + self.level = 0 + self.text = "" + + def parse(self, text): + """Parses the given text and returns a parse tree. + """ + self.reset() + self.text = text + return self.parse_all() + + def parse_all(self): + while True: + dollar = self.text.find("$", self.pos) + if dollar < 0: + break + nextchar = self.text[dollar + 1] + if nextchar in self.namechars: + yield _Node("text", self.text[self.pos:dollar]) + self.pos = dollar+1 + yield self.parse_expr() + + # for supporting ${x.id}, for backward compataility + elif nextchar == '{': + saved_pos = self.pos + self.pos = dollar+2 # skip "${" + expr = self.parse_expr() + if self.text[self.pos] == '}': + self.pos += 1 + yield _Node("text", self.text[self.pos:dollar]) + yield expr + else: + self.pos = saved_pos + break + else: + yield _Node("text", self.text[self.pos:dollar+1]) + self.pos = dollar + 1 + # $$ is used to escape $ + if nextchar == "$": + self.pos += 1 + + if self.pos < len(self.text): + yield _Node("text", self.text[self.pos:]) + + def match(self): + match = tokenprog.match(self.text, self.pos) + if match is None: + raise _ItplError(self.text, self.pos) + return match, match.end() + + def is_literal(self, text): + return text and text[0] in "0123456789\"'" + + def parse_expr(self): + match, pos = self.match() + if self.is_literal(match.group()): + expr = _Node("literal", match.group()) + else: + expr = _Node("param", self.text[self.pos:pos]) + self.pos = pos + while self.pos < len(self.text): + if self.text[self.pos] == "." and \ + self.pos + 1 < len(self.text) and self.text[self.pos + 1] in self.namechars: + self.pos += 1 + match, pos = self.match() + attr = match.group() + expr = _Node("getattr", expr, attr) + self.pos = pos + elif self.text[self.pos] == "[": + saved_pos = self.pos + self.pos += 1 + key = self.parse_expr() + if self.text[self.pos] == ']': + self.pos += 1 + expr = _Node("getitem", expr, key) + else: + self.pos = saved_pos + break + else: + break + return expr + +class SafeEval(object): + """Safe evaluator for binding params to db queries. + """ + def safeeval(self, text, mapping): + nodes = Parser().parse(text) + return SQLQuery.join([self.eval_node(node, mapping) for node in nodes], "") + + def eval_node(self, node, mapping): + if node.type == "text": + return node.first + else: + return sqlquote(self.eval_expr(node, mapping)) + + def eval_expr(self, node, mapping): + if node.type == "literal": + return ast.literal_eval(node.first) + elif node.type == "getattr": + return getattr(self.eval_expr(node.first, mapping), node.second) + elif node.type == "getitem": + return self.eval_expr(node.first, mapping)[self.eval_expr(node.second, mapping)] + elif node.type == "param": + return mapping[node.first] + +def test_parser(): + def f(text, expected): + p = Parser() + nodes = list(p.parse(text)) + print(repr(text), nodes) + assert nodes == expected, "Expected %r" % expected + + f("Hello", [_Node("text", "Hello")]) + f("Hello $name", [_Node("text", "Hello "), _Node("param", "name")]) + f("Hello $name.foo", [ + _Node("text", "Hello "), + _Node("getattr", + _Node("param", "name"), + "foo")]) + f("WHERE id=$self.id LIMIT 1", [ + _Node("text", "WHERE id="), + _Node('getattr', + _Node('param', 'self', None), + 'id'), + _Node("text", " LIMIT 1")]) + + f("WHERE id=$self['id'] LIMIT 1", [ + _Node("text", "WHERE id="), + _Node('getitem', + _Node('param', 'self', None), + _Node('literal', "'id'")), + _Node("text", " LIMIT 1")]) + +def test_safeeval(): + def f(q, vars): + return SafeEval().safeeval(q, vars) + + print(f("WHERE id=$id", {"id": 1}).items) + assert f("WHERE id=$id", {"id": 1}).items == ["WHERE id=", sqlparam(1)] + if __name__ == "__main__": import doctest doctest.testmod() + test_parser() + test_safeeval() diff --git a/lib/web/debugerror.py b/lib/web/debugerror.py index 5ee5ddf..40e23b5 100644 --- a/lib/web/debugerror.py +++ b/lib/web/debugerror.py @@ -2,7 +2,7 @@ pretty debug errors (part of web.py) -portions adapted from Django +portions adapted from Django Copyright (c) 2005, the Lawrence Journal-World Used under the modified BSD license: http://www.xfree86.org/3.3.6/COPYRIGHT2.html#5 @@ -10,11 +10,20 @@ __all__ = ["debugerror", "djangoerror", "emailerrors"] -import sys, urlparse, pprint, traceback -from template import Template -from net import websafe -from utils import sendmail, safestr -import webapi as web +import sys, pprint, traceback +from .template import Template +from .net import websafe +from .utils import sendmail, safestr +from . import webapi as web +from .py3helpers import urljoin, PY2 + +if PY2: + def update_globals_template(t, globals): + t.t.func_globals.update(globals) +else: + def update_globals_template(t, globals): + t.t.__globals__.update(globals) + import os, os.path whereami = os.path.join(os.getcwd(), __file__) @@ -38,11 +47,11 @@ h2 span { font-size:80%; color:#666; font-weight:normal; } h3 { margin:1em 0 .5em 0; } h4 { margin:0 0 .5em 0; font-weight: normal; } - table { + table { border:1px solid #ccc; border-collapse: collapse; background:white; } tbody td, tbody th { vertical-align:top; padding:2px 3px; } - thead th { - padding:1px 6px 1px 3px; background:#fefefe; text-align:left; + thead th { + padding:1px 6px 1px 3px; background:#fefefe; text-align:left; font-weight:normal; font-size:11px; border:1px solid #ddd; } tbody th { text-align:right; color:#666; padding-right:.5em; } table.vars { margin:5px 0 2px 40px; } @@ -50,14 +59,14 @@ table td.code { width:100%;} table td.code div { overflow:hidden; } table.source th { color:#666; } - table.source td { + table.source td { font-family:monospace; white-space:pre; border-bottom:1px solid #eee; } ul.traceback { list-style-type:none; } ul.traceback li.frame { margin-bottom:1em; } div.context { margin: 10px 0; } - div.context ol { + div.context ol { padding-left:30px; margin:0 10px; list-style-position: inside; } - div.context ol li { + div.context ol li { font-family:monospace; white-space:pre; color:#666; cursor:pointer; } div.context ol.context-line li { color:black; background-color:#ccc; } div.context ol.context-line li span { float: right; } @@ -79,7 +88,7 @@ + + + +$def dicttable (d, kls='req', id=None): + $ items = d and d.items() or [] + $items.sort() + $:dicttable_items(items, kls, id) + +$def dicttable_items(items, kls='req', id=None): + $if items: + + + $for k, v in items: + + +
VariableValue
$k
$prettify(v)
+ $else: +

No data.

+ +
+

$exception_type at $ctx.path

+

$exception_value

+ + + + + + +
Python$frames[0].filename in $frames[0].function, line $frames[0].lineno
Web$ctx.method $ctx.home$ctx.path
+
+
+

Traceback (innermost first)

+
    +$for frame in frames: +
  • + $frame.filename in $frame.function + $if frame.context_line is not None: +
    + $if frame.pre_context: +
      + $for line in frame.pre_context: +
    1. $line
      2. +
    +
    1. $frame.context_line ...
    + $if frame.post_context: +
      + $for line in frame.post_context: +
    1. $line
      2. +
    +
    + + $if frame.vars: +
    + ▶ Local vars + $# $inspect.formatargvalues(*inspect.getargvalues(frame['tb'].tb_frame)) +
    + $:dicttable(frame.vars, kls='vars', id=('v' + str(frame.id))) +
    • +
+
+ +
+$if ctx.output or ctx.headers: +

Response so far

+

HEADERS

+ $:dicttable_items(ctx.headers) + +

BODY

+

+ $ctx.output +

+ +

Request information

+ +

INPUT

+$:dicttable(web.input(_unicode=False)) + + +$:dicttable(web.cookies()) + +

META

+$ newctx = [(k, v) for (k, v) in ctx.iteritems() if not k.startswith('_') and not isinstance(v, dict)] +$:dicttable(dict(newctx)) + +

ENVIRONMENT

+$:dicttable(ctx.env) +
+ +
+

+ You're seeing this error because you have web.config.debug + set to True. Set that to False if you don't to see this. +

+
+ + + +""" + +djangoerror_r = None + +def djangoerror(): + def _get_lines_from_file(filename, lineno, context_lines): + """ + Returns context_lines before and after lineno from file. + Returns (pre_context_lineno, pre_context, context_line, post_context). + """ + try: + source = open(filename).readlines() + lower_bound = max(0, lineno - context_lines) + upper_bound = lineno + context_lines + + pre_context = \ + [line.strip('\n') for line in source[lower_bound:lineno]] + context_line = source[lineno].strip('\n') + post_context = \ + [line.strip('\n') for line in source[lineno + 1:upper_bound]] + + return lower_bound, pre_context, context_line, post_context + except (OSError, IOError, IndexError): + return None, [], None, [] + + exception_type, exception_value, tback = sys.exc_info() + frames = [] + while tback is not None: + filename = tback.tb_frame.f_code.co_filename + function = tback.tb_frame.f_code.co_name + lineno = tback.tb_lineno - 1 + + # hack to get correct line number for templates + lineno += tback.tb_frame.f_locals.get("__lineoffset__", 0) + + pre_context_lineno, pre_context, context_line, post_context = \ + _get_lines_from_file(filename, lineno, 7) + + if '__hidetraceback__' not in tback.tb_frame.f_locals: + frames.append(web.storage({ + 'tback': tback, + 'filename': filename, + 'function': function, + 'lineno': lineno, + 'vars': tback.tb_frame.f_locals, + 'id': id(tback), + 'pre_context': pre_context, + 'context_line': context_line, + 'post_context': post_context, + 'pre_context_lineno': pre_context_lineno, + })) + tback = tback.tb_next + frames.reverse() + urljoin = urlparse.urljoin + def prettify(x): + try: + out = pprint.pformat(x) + except Exception, e: + out = '[could not display: <' + e.__class__.__name__ + \ + ': '+str(e)+'>]' + return out + + global djangoerror_r + if djangoerror_r is None: + djangoerror_r = Template(djangoerror_t, filename=__file__, filter=websafe) + + t = djangoerror_r + globals = {'ctx': web.ctx, 'web':web, 'dict':dict, 'str':str, 'prettify': prettify} + t.t.func_globals.update(globals) + return t(exception_type, exception_value, frames) + +def debugerror(): + """ + A replacement for `internalerror` that presents a nice page with lots + of debug information for the programmer. + + (Based on the beautiful 500 page from [Django](http://djangoproject.com/), + designed by [Wilson Miner](http://wilsonminer.com/).) + """ + return web._InternalError(djangoerror()) + +def emailerrors(to_address, olderror, from_address=None): + """ + Wraps the old `internalerror` handler (pass as `olderror`) to + additionally email all errors to `to_address`, to aid in + debugging production websites. + + Emails contain a normal text traceback as well as an + attachment containing the nice `debugerror` page. + """ + from_address = from_address or to_address + + def emailerrors_internal(): + error = olderror() + tb = sys.exc_info() + error_name = tb[0] + error_value = tb[1] + tb_txt = ''.join(traceback.format_exception(*tb)) + path = web.ctx.path + request = web.ctx.method + ' ' + web.ctx.home + web.ctx.fullpath + + message = "\n%s\n\n%s\n\n" % (request, tb_txt) + + sendmail( + "your buggy site <%s>" % from_address, + "the bugfixer <%s>" % to_address, + "bug: %(error_name)s: %(error_value)s (%(path)s)" % locals(), + message, + attachments=[ + dict(filename="bug.html", content=safestr(djangoerror())) + ], + ) + return error + + return emailerrors_internal + +if __name__ == "__main__": + urls = ( + '/', 'index' + ) + from application import application + app = application(urls, globals()) + app.internalerror = debugerror + + class index: + def GET(self): + thisdoesnotexist + + app.run() diff --git a/web/form.py b/web/form.py new file mode 100644 index 0000000..18e7341 --- /dev/null +++ b/web/form.py @@ -0,0 +1,370 @@ +""" +HTML forms +(part of web.py) +""" + +import copy, re +import webapi as web +import utils, net + +def attrget(obj, attr, value=None): + if hasattr(obj, 'has_key') and obj.has_key(attr): return obj[attr] + if hasattr(obj, attr): return getattr(obj, attr) + return value + +class Form(object): + r""" + HTML form. + + >>> f = Form(Textbox("x")) + >>> f.render() + '\n \n
' + """ + def __init__(self, *inputs, **kw): + self.inputs = inputs + self.valid = True + self.note = None + self.validators = kw.pop('validators', []) + + def __call__(self, x=None): + o = copy.deepcopy(self) + if x: o.validates(x) + return o + + def render(self): + out = '' + out += self.rendernote(self.note) + out += '\n' + + for i in self.inputs: + html = utils.safeunicode(i.pre) + i.render() + self.rendernote(i.note) + utils.safeunicode(i.post) + if i.is_hidden(): + out += ' \n' % (html) + else: + out += ' \n' % (i.id, net.websafe(i.description), html) + out += "
%s
%s
" + return out + + def render_css(self): + out = [] + out.append(self.rendernote(self.note)) + for i in self.inputs: + if not i.is_hidden(): + out.append('' % (i.id, net.websafe(i.description))) + out.append(i.pre) + out.append(i.render()) + out.append(self.rendernote(i.note)) + out.append(i.post) + out.append('\n') + return ''.join(out) + + def rendernote(self, note): + if note: return '%s' % net.websafe(note) + else: return "" + + def validates(self, source=None, _validate=True, **kw): + source = source or kw or web.input() + out = True + for i in self.inputs: + v = attrget(source, i.name) + if _validate: + out = i.validate(v) and out + else: + i.set_value(v) + if _validate: + out = out and self._validate(source) + self.valid = out + return out + + def _validate(self, value): + self.value = value + for v in self.validators: + if not v.valid(value): + self.note = v.msg + return False + return True + + def fill(self, source=None, **kw): + return self.validates(source, _validate=False, **kw) + + def __getitem__(self, i): + for x in self.inputs: + if x.name == i: return x + raise KeyError, i + + def __getattr__(self, name): + # don't interfere with deepcopy + inputs = self.__dict__.get('inputs') or [] + for x in inputs: + if x.name == name: return x + raise AttributeError, name + + def get(self, i, default=None): + try: + return self[i] + except KeyError: + return default + + def _get_d(self): #@@ should really be form.attr, no? + return utils.storage([(i.name, i.get_value()) for i in self.inputs]) + d = property(_get_d) + +class Input(object): + def __init__(self, name, *validators, **attrs): + self.name = name + self.validators = validators + self.attrs = attrs = AttributeList(attrs) + + self.description = attrs.pop('description', name) + self.value = attrs.pop('value', None) + self.pre = attrs.pop('pre', "") + self.post = attrs.pop('post', "") + self.note = None + + self.id = attrs.setdefault('id', self.get_default_id()) + + if 'class_' in attrs: + attrs['class'] = attrs['class_'] + del attrs['class_'] + + def is_hidden(self): + return False + + def get_type(self): + raise NotImplementedError + + def get_default_id(self): + return self.name + + def validate(self, value): + self.set_value(value) + + for v in self.validators: + if not v.valid(value): + self.note = v.msg + return False + return True + + def set_value(self, value): + self.value = value + + def get_value(self): + return self.value + + def render(self): + attrs = self.attrs.copy() + attrs['type'] = self.get_type() + if self.value is not None: + attrs['value'] = self.value + attrs['name'] = self.name + return '' % attrs + + def rendernote(self, note): + if note: return '%s' % net.websafe(note) + else: return "" + + def addatts(self): + # add leading space for backward-compatibility + return " " + str(self.attrs) + +class AttributeList(dict): + """List of atributes of input. + + >>> a = AttributeList(type='text', name='x', value=20) + >>> a + + """ + def copy(self): + return AttributeList(self) + + def __str__(self): + return " ".join(['%s="%s"' % (k, net.websafe(v)) for k, v in self.items()]) + + def __repr__(self): + return '' % repr(str(self)) + +class Textbox(Input): + """Textbox input. + + >>> Textbox(name='foo', value='bar').render() + '' + >>> Textbox(name='foo', value=0).render() + '' + """ + def get_type(self): + return 'text' + +class Password(Input): + """Password input. + + >>> Password(name='password', value='secret').render() + '' + """ + + def get_type(self): + return 'password' + +class Textarea(Input): + """Textarea input. + + >>> Textarea(name='foo', value='bar').render() + '' + """ + def render(self): + attrs = self.attrs.copy() + attrs['name'] = self.name + value = net.websafe(self.value or '') + return '' % (attrs, value) + +class Dropdown(Input): + r"""Dropdown/select input. + + >>> Dropdown(name='foo', args=['a', 'b', 'c'], value='b').render() + '\n' + >>> Dropdown(name='foo', args=[('a', 'aa'), ('b', 'bb'), ('c', 'cc')], value='b').render() + '\n' + """ + def __init__(self, name, args, *validators, **attrs): + self.args = args + super(Dropdown, self).__init__(name, *validators, **attrs) + + def render(self): + attrs = self.attrs.copy() + attrs['name'] = self.name + + x = '\n' + return x + +class Radio(Input): + def __init__(self, name, args, *validators, **attrs): + self.args = args + super(Radio, self).__init__(name, *validators, **attrs) + + def render(self): + x = '' + for arg in self.args: + if isinstance(arg, (tuple, list)): + value, desc= arg + else: + value, desc = arg, arg + attrs = self.attrs.copy() + attrs['name'] = self.name + attrs['type'] = 'radio' + attrs['value'] = value + if self.value == value: + attrs['checked'] = 'checked' + x += ' %s' % (attrs, net.websafe(desc)) + x += '' + return x + +class Checkbox(Input): + """Checkbox input. + + >>> Checkbox('foo', value='bar', checked=True).render() + '' + >>> Checkbox('foo', value='bar').render() + '' + >>> c = Checkbox('foo', value='bar') + >>> c.validate('on') + True + >>> c.render() + '' + """ + def __init__(self, name, *validators, **attrs): + self.checked = attrs.pop('checked', False) + Input.__init__(self, name, *validators, **attrs) + + def get_default_id(self): + value = utils.safestr(self.value or "") + return self.name + '_' + value.replace(' ', '_') + + def render(self): + attrs = self.attrs.copy() + attrs['type'] = 'checkbox' + attrs['name'] = self.name + attrs['value'] = self.value + + if self.checked: + attrs['checked'] = 'checked' + return '' % attrs + + def set_value(self, value): + self.checked = bool(value) + + def get_value(self): + return self.checked + +class Button(Input): + """HTML Button. + + >>> Button("save").render() + '' + >>> Button("action", value="save", html="Save Changes").render() + '' + """ + def __init__(self, name, *validators, **attrs): + super(Button, self).__init__(name, *validators, **attrs) + self.description = "" + + def render(self): + attrs = self.attrs.copy() + attrs['name'] = self.name + if self.value is not None: + attrs['value'] = self.value + html = attrs.pop('html', None) or net.websafe(self.name) + return '' % (attrs, html) + +class Hidden(Input): + """Hidden Input. + + >>> Hidden(name='foo', value='bar').render() + '' + """ + def is_hidden(self): + return True + + def get_type(self): + return 'hidden' + +class File(Input): + """File input. + + >>> File(name='f').render() + '' + """ + def get_type(self): + return 'file' + +class Validator: + def __deepcopy__(self, memo): return copy.copy(self) + def __init__(self, msg, test, jstest=None): utils.autoassign(self, locals()) + def valid(self, value): + try: return self.test(value) + except: return False + +notnull = Validator("Required", bool) + +class regexp(Validator): + def __init__(self, rexp, msg): + self.rexp = re.compile(rexp) + self.msg = msg + + def valid(self, value): + return bool(self.rexp.match(value)) + +if __name__ == "__main__": + import doctest + doctest.testmod() diff --git a/web/http.py b/web/http.py new file mode 100644 index 0000000..b6030fa --- /dev/null +++ b/web/http.py @@ -0,0 +1,150 @@ +""" +HTTP Utilities +(from web.py) +""" + +__all__ = [ + "expires", "lastmodified", + "prefixurl", "modified", + "changequery", "url", + "profiler", +] + +import sys, os, threading, urllib, urlparse +try: import datetime +except ImportError: pass +import net, utils, webapi as web + +def prefixurl(base=''): + """ + Sorry, this function is really difficult to explain. + Maybe some other time. + """ + url = web.ctx.path.lstrip('/') + for i in xrange(url.count('/')): + base += '../' + if not base: + base = './' + return base + +def expires(delta): + """ + Outputs an `Expires` header for `delta` from now. + `delta` is a `timedelta` object or a number of seconds. + """ + if isinstance(delta, (int, long)): + delta = datetime.timedelta(seconds=delta) + date_obj = datetime.datetime.utcnow() + delta + web.header('Expires', net.httpdate(date_obj)) + +def lastmodified(date_obj): + """Outputs a `Last-Modified` header for `datetime`.""" + web.header('Last-Modified', net.httpdate(date_obj)) + +def modified(date=None, etag=None): + """ + Checks to see if the page has been modified since the version in the + requester's cache. + + When you publish pages, you can include `Last-Modified` and `ETag` + with the date the page was last modified and an opaque token for + the particular version, respectively. When readers reload the page, + the browser sends along the modification date and etag value for + the version it has in its cache. If the page hasn't changed, + the server can just return `304 Not Modified` and not have to + send the whole page again. + + This function takes the last-modified date `date` and the ETag `etag` + and checks the headers to see if they match. If they do, it returns + `True`, or otherwise it raises NotModified error. It also sets + `Last-Modified` and `ETag` output headers. + """ + try: + from __builtin__ import set + except ImportError: + # for python 2.3 + from sets import Set as set + + n = set([x.strip('" ') for x in web.ctx.env.get('HTTP_IF_NONE_MATCH', '').split(',')]) + m = net.parsehttpdate(web.ctx.env.get('HTTP_IF_MODIFIED_SINCE', '').split(';')[0]) + validate = False + if etag: + if '*' in n or etag in n: + validate = True + if date and m: + # we subtract a second because + # HTTP dates don't have sub-second precision + if date-datetime.timedelta(seconds=1) <= m: + validate = True + + if date: lastmodified(date) + if etag: web.header('ETag', '"' + etag + '"') + if validate: + raise web.notmodified() + else: + return True + +def urlencode(query, doseq=0): + """ + Same as urllib.urlencode, but supports unicode strings. + + >>> urlencode({'text':'foo bar'}) + 'text=foo+bar' + >>> urlencode({'x': [1, 2]}, doseq=True) + 'x=1&x=2' + """ + def convert(value, doseq=False): + if doseq and isinstance(value, list): + return [convert(v) for v in value] + else: + return utils.safestr(value) + + query = dict([(k, convert(v, doseq)) for k, v in query.items()]) + return urllib.urlencode(query, doseq=doseq) + +def changequery(query=None, **kw): + """ + Imagine you're at `/foo?a=1&b=2`. Then `changequery(a=3)` will return + `/foo?a=3&b=2` -- the same URL but with the arguments you requested + changed. + """ + if query is None: + query = web.rawinput(method='get') + for k, v in kw.iteritems(): + if v is None: + query.pop(k, None) + else: + query[k] = v + out = web.ctx.path + if query: + out += '?' + urlencode(query, doseq=True) + return out + +def url(path=None, doseq=False, **kw): + """ + Makes url by concatinating web.ctx.homepath and path and the + query string created using the arguments. + """ + if path is None: + path = web.ctx.path + if path.startswith("/"): + out = web.ctx.homepath + path + else: + out = path + + if kw: + out += '?' + urlencode(kw, doseq=doseq) + + return out + +def profiler(app): + """Outputs basic profiling information at the bottom of each response.""" + from utils import profile + def profile_internal(e, o): + out, result = profile(app)(e, o) + return list(out) + ['
' + net.websafe(result) + '
'] + return profile_internal + +if __name__ == "__main__": + import doctest + doctest.testmod() diff --git a/web/httpserver.py b/web/httpserver.py new file mode 100644 index 0000000..aafd546 --- /dev/null +++ b/web/httpserver.py @@ -0,0 +1,283 @@ +__all__ = ["runsimple"] + +import sys, os +from SimpleHTTPServer import SimpleHTTPRequestHandler +import urllib +import posixpath + +import webapi as web +import net +import utils + +def runbasic(func, server_address=("0.0.0.0", 8080)): + """ + Runs a simple HTTP server hosting WSGI app `func`. The directory `static/` + is hosted statically. + + Based on [WsgiServer][ws] from [Colin Stewart][cs]. + + [ws]: http://www.owlfish.com/software/wsgiutils/documentation/wsgi-server-api.html + [cs]: http://www.owlfish.com/ + """ + # Copyright (c) 2004 Colin Stewart (http://www.owlfish.com/) + # Modified somewhat for simplicity + # Used under the modified BSD license: + # http://www.xfree86.org/3.3.6/COPYRIGHT2.html#5 + + import SimpleHTTPServer, SocketServer, BaseHTTPServer, urlparse + import socket, errno + import traceback + + class WSGIHandler(SimpleHTTPServer.SimpleHTTPRequestHandler): + def run_wsgi_app(self): + protocol, host, path, parameters, query, fragment = \ + urlparse.urlparse('http://dummyhost%s' % self.path) + + # we only use path, query + env = {'wsgi.version': (1, 0) + ,'wsgi.url_scheme': 'http' + ,'wsgi.input': self.rfile + ,'wsgi.errors': sys.stderr + ,'wsgi.multithread': 1 + ,'wsgi.multiprocess': 0 + ,'wsgi.run_once': 0 + ,'REQUEST_METHOD': self.command + ,'REQUEST_URI': self.path + ,'PATH_INFO': path + ,'QUERY_STRING': query + ,'CONTENT_TYPE': self.headers.get('Content-Type', '') + ,'CONTENT_LENGTH': self.headers.get('Content-Length', '') + ,'REMOTE_ADDR': self.client_address[0] + ,'SERVER_NAME': self.server.server_address[0] + ,'SERVER_PORT': str(self.server.server_address[1]) + ,'SERVER_PROTOCOL': self.request_version + } + + for http_header, http_value in self.headers.items(): + env ['HTTP_%s' % http_header.replace('-', '_').upper()] = \ + http_value + + # Setup the state + self.wsgi_sent_headers = 0 + self.wsgi_headers = [] + + try: + # We have there environment, now invoke the application + result = self.server.app(env, self.wsgi_start_response) + try: + try: + for data in result: + if data: + self.wsgi_write_data(data) + finally: + if hasattr(result, 'close'): + result.close() + except socket.error, socket_err: + # Catch common network errors and suppress them + if (socket_err.args[0] in \ + (errno.ECONNABORTED, errno.EPIPE)): + return + except socket.timeout, socket_timeout: + return + except: + print >> web.debug, traceback.format_exc(), + + if (not self.wsgi_sent_headers): + # We must write out something! + self.wsgi_write_data(" ") + return + + do_POST = run_wsgi_app + do_PUT = run_wsgi_app + do_DELETE = run_wsgi_app + + def do_GET(self): + if self.path.startswith('/static/'): + SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self) + else: + self.run_wsgi_app() + + def wsgi_start_response(self, response_status, response_headers, + exc_info=None): + if (self.wsgi_sent_headers): + raise Exception \ + ("Headers already sent and start_response called again!") + # Should really take a copy to avoid changes in the application.... + self.wsgi_headers = (response_status, response_headers) + return self.wsgi_write_data + + def wsgi_write_data(self, data): + if (not self.wsgi_sent_headers): + status, headers = self.wsgi_headers + # Need to send header prior to data + status_code = status[:status.find(' ')] + status_msg = status[status.find(' ') + 1:] + self.send_response(int(status_code), status_msg) + for header, value in headers: + self.send_header(header, value) + self.end_headers() + self.wsgi_sent_headers = 1 + # Send the data + self.wfile.write(data) + + class WSGIServer(SocketServer.ThreadingMixIn, BaseHTTPServer.HTTPServer): + def __init__(self, func, server_address): + BaseHTTPServer.HTTPServer.__init__(self, + server_address, + WSGIHandler) + self.app = func + self.serverShuttingDown = 0 + + print "http://%s:%d/" % server_address + WSGIServer(func, server_address).serve_forever() + +def runsimple(func, server_address=("0.0.0.0", 8080)): + """ + Runs [CherryPy][cp] WSGI server hosting WSGI app `func`. + The directory `static/` is hosted statically. + + [cp]: http://www.cherrypy.org + """ + func = StaticMiddleware(func) + func = LogMiddleware(func) + + server = WSGIServer(server_address, func) + + print "http://%s:%d/" % server_address + try: + server.start() + except KeyboardInterrupt: + server.stop() + +def WSGIServer(server_address, wsgi_app): + """Creates CherryPy WSGI server listening at `server_address` to serve `wsgi_app`. + This function can be overwritten to customize the webserver or use a different webserver. + """ + import wsgiserver + + # Default values of wsgiserver.ssl_adapters uses cheerypy.wsgiserver + # prefix. Overwriting it make it work with web.wsgiserver. + wsgiserver.ssl_adapters = { + 'builtin': 'web.wsgiserver.ssl_builtin.BuiltinSSLAdapter', + 'pyopenssl': 'web.wsgiserver.ssl_pyopenssl.pyOpenSSLAdapter', + } + + server = wsgiserver.CherryPyWSGIServer(server_address, wsgi_app, server_name="localhost") + server.nodelay = not sys.platform.startswith('java') # TCP_NODELAY isn't supported on the JVM + return server + +class StaticApp(SimpleHTTPRequestHandler): + """WSGI application for serving static files.""" + def __init__(self, environ, start_response): + self.headers = [] + self.environ = environ + self.start_response = start_response + + def send_response(self, status, msg=""): + self.status = str(status) + " " + msg + + def send_header(self, name, value): + self.headers.append((name, value)) + + def end_headers(self): + pass + + def log_message(*a): pass + + def __iter__(self): + environ = self.environ + + self.path = environ.get('PATH_INFO', '') + self.client_address = environ.get('REMOTE_ADDR','-'), \ + environ.get('REMOTE_PORT','-') + self.command = environ.get('REQUEST_METHOD', '-') + + from cStringIO import StringIO + self.wfile = StringIO() # for capturing error + + try: + path = self.translate_path(self.path) + etag = '"%s"' % os.path.getmtime(path) + client_etag = environ.get('HTTP_IF_NONE_MATCH') + self.send_header('ETag', etag) + if etag == client_etag: + self.send_response(304, "Not Modified") + self.start_response(self.status, self.headers) + raise StopIteration + except OSError: + pass # Probably a 404 + + f = self.send_head() + self.start_response(self.status, self.headers) + + if f: + block_size = 16 * 1024 + while True: + buf = f.read(block_size) + if not buf: + break + yield buf + f.close() + else: + value = self.wfile.getvalue() + yield value + +class StaticMiddleware: + """WSGI middleware for serving static files.""" + def __init__(self, app, prefix='/static/'): + self.app = app + self.prefix = prefix + + def __call__(self, environ, start_response): + path = environ.get('PATH_INFO', '') + path = self.normpath(path) + + if path.startswith(self.prefix): + return StaticApp(environ, start_response) + else: + return self.app(environ, start_response) + + def normpath(self, path): + path2 = posixpath.normpath(urllib.unquote(path)) + if path.endswith("/"): + path2 += "/" + return path2 + + +class LogMiddleware: + """WSGI middleware for logging the status.""" + def __init__(self, app): + self.app = app + self.format = '%s - - [%s] "%s %s %s" - %s' + + from BaseHTTPServer import BaseHTTPRequestHandler + import StringIO + f = StringIO.StringIO() + + class FakeSocket: + def makefile(self, *a): + return f + + # take log_date_time_string method from BaseHTTPRequestHandler + self.log_date_time_string = BaseHTTPRequestHandler(FakeSocket(), None, None).log_date_time_string + + def __call__(self, environ, start_response): + def xstart_response(status, response_headers, *args): + out = start_response(status, response_headers, *args) + self.log(status, environ) + return out + + return self.app(environ, xstart_response) + + def log(self, status, environ): + outfile = environ.get('wsgi.errors', web.debug) + req = environ.get('PATH_INFO', '_') + protocol = environ.get('ACTUAL_SERVER_PROTOCOL', '-') + method = environ.get('REQUEST_METHOD', '-') + host = "%s:%s" % (environ.get('REMOTE_ADDR','-'), + environ.get('REMOTE_PORT','-')) + + time = self.log_date_time_string() + + msg = self.format % (host, time, protocol, method, req, status) + print >> outfile, utils.safestr(msg) diff --git a/web/net.py b/web/net.py new file mode 100644 index 0000000..40ff197 --- /dev/null +++ b/web/net.py @@ -0,0 +1,193 @@ +""" +Network Utilities +(from web.py) +""" + +__all__ = [ + "validipaddr", "validipport", "validip", "validaddr", + "urlquote", + "httpdate", "parsehttpdate", + "htmlquote", "htmlunquote", "websafe", +] + +import urllib, time +try: import datetime +except ImportError: pass + +def validipaddr(address): + """ + Returns True if `address` is a valid IPv4 address. + + >>> validipaddr('192.168.1.1') + True + >>> validipaddr('192.168.1.800') + False + >>> validipaddr('192.168.1') + False + """ + try: + octets = address.split('.') + if len(octets) != 4: + return False + for x in octets: + if not (0 <= int(x) <= 255): + return False + except ValueError: + return False + return True + +def validipport(port): + """ + Returns True if `port` is a valid IPv4 port. + + >>> validipport('9000') + True + >>> validipport('foo') + False + >>> validipport('1000000') + False + """ + try: + if not (0 <= int(port) <= 65535): + return False + except ValueError: + return False + return True + +def validip(ip, defaultaddr="0.0.0.0", defaultport=8080): + """Returns `(ip_address, port)` from string `ip_addr_port`""" + addr = defaultaddr + port = defaultport + + ip = ip.split(":", 1) + if len(ip) == 1: + if not ip[0]: + pass + elif validipaddr(ip[0]): + addr = ip[0] + elif validipport(ip[0]): + port = int(ip[0]) + else: + raise ValueError, ':'.join(ip) + ' is not a valid IP address/port' + elif len(ip) == 2: + addr, port = ip + if not validipaddr(addr) and validipport(port): + raise ValueError, ':'.join(ip) + ' is not a valid IP address/port' + port = int(port) + else: + raise ValueError, ':'.join(ip) + ' is not a valid IP address/port' + return (addr, port) + +def validaddr(string_): + """ + Returns either (ip_address, port) or "/path/to/socket" from string_ + + >>> validaddr('/path/to/socket') + '/path/to/socket' + >>> validaddr('8000') + ('0.0.0.0', 8000) + >>> validaddr('127.0.0.1') + ('127.0.0.1', 8080) + >>> validaddr('127.0.0.1:8000') + ('127.0.0.1', 8000) + >>> validaddr('fff') + Traceback (most recent call last): + ... + ValueError: fff is not a valid IP address/port + """ + if '/' in string_: + return string_ + else: + return validip(string_) + +def urlquote(val): + """ + Quotes a string for use in a URL. + + >>> urlquote('://?f=1&j=1') + '%3A//%3Ff%3D1%26j%3D1' + >>> urlquote(None) + '' + >>> urlquote(u'\u203d') + '%E2%80%BD' + """ + if val is None: return '' + if not isinstance(val, unicode): val = str(val) + else: val = val.encode('utf-8') + return urllib.quote(val) + +def httpdate(date_obj): + """ + Formats a datetime object for use in HTTP headers. + + >>> import datetime + >>> httpdate(datetime.datetime(1970, 1, 1, 1, 1, 1)) + 'Thu, 01 Jan 1970 01:01:01 GMT' + """ + return date_obj.strftime("%a, %d %b %Y %H:%M:%S GMT") + +def parsehttpdate(string_): + """ + Parses an HTTP date into a datetime object. + + >>> parsehttpdate('Thu, 01 Jan 1970 01:01:01 GMT') + datetime.datetime(1970, 1, 1, 1, 1, 1) + """ + try: + t = time.strptime(string_, "%a, %d %b %Y %H:%M:%S %Z") + except ValueError: + return None + return datetime.datetime(*t[:6]) + +def htmlquote(text): + r""" + Encodes `text` for raw use in HTML. + + >>> htmlquote(u"<'&\">") + u'<'&">' + """ + text = text.replace(u"&", u"&") # Must be done first! + text = text.replace(u"<", u"<") + text = text.replace(u">", u">") + text = text.replace(u"'", u"'") + text = text.replace(u'"', u""") + return text + +def htmlunquote(text): + r""" + Decodes `text` that's HTML quoted. + + >>> htmlunquote(u'<'&">') + u'<\'&">' + """ + text = text.replace(u""", u'"') + text = text.replace(u"'", u"'") + text = text.replace(u">", u">") + text = text.replace(u"<", u"<") + text = text.replace(u"&", u"&") # Must be done last! + return text + +def websafe(val): + r"""Converts `val` so that it is safe for use in Unicode HTML. + + >>> websafe("<'&\">") + u'<'&">' + >>> websafe(None) + u'' + >>> websafe(u'\u203d') + u'\u203d' + >>> websafe('\xe2\x80\xbd') + u'\u203d' + """ + if val is None: + return u'' + elif isinstance(val, str): + val = val.decode('utf-8') + elif not isinstance(val, unicode): + val = unicode(val) + + return htmlquote(val) + +if __name__ == "__main__": + import doctest + doctest.testmod() diff --git a/lib/web/python23.py b/web/python23.py similarity index 100% rename from lib/web/python23.py rename to web/python23.py diff --git a/web/session.py b/web/session.py new file mode 100644 index 0000000..fb30a23 --- /dev/null +++ b/web/session.py @@ -0,0 +1,354 @@ +""" +Session Management +(from web.py) +""" + +import os, time, datetime, random, base64 +import os.path +try: + import cPickle as pickle +except ImportError: + import pickle +try: + import hashlib + sha1 = hashlib.sha1 +except ImportError: + import sha + sha1 = sha.new + +import utils +import webapi as web + +__all__ = [ + 'Session', 'SessionExpired', + 'Store', 'DiskStore', 'DBStore', +] + +web.config.session_parameters = utils.storage({ + 'cookie_name': 'webpy_session_id', + 'cookie_domain': None, + 'timeout': 86400, #24 * 60 * 60, # 24 hours in seconds + 'ignore_expiry': True, + 'ignore_change_ip': True, + 'secret_key': 'fLjUfxqXtfNoIldA0A0J', + 'expired_message': 'Session expired', + 'httponly': True, + 'secure': False +}) + +class SessionExpired(web.HTTPError): + def __init__(self, message): + web.HTTPError.__init__(self, '200 OK', {}, data=message) + +class Session(object): + """Session management for web.py + """ + __slots__ = [ + "store", "_initializer", "_last_cleanup_time", "_config", "_data", + "__getitem__", "__setitem__", "__delitem__" + ] + + def __init__(self, app, store, initializer=None): + self.store = store + self._initializer = initializer + self._last_cleanup_time = 0 + self._config = utils.storage(web.config.session_parameters) + self._data = utils.threadeddict() + + self.__getitem__ = self._data.__getitem__ + self.__setitem__ = self._data.__setitem__ + self.__delitem__ = self._data.__delitem__ + + if app: + app.add_processor(self._processor) + + def __contains__(self, name): + return name in self._data + + def __getattr__(self, name): + return getattr(self._data, name) + + def __setattr__(self, name, value): + if name in self.__slots__: + object.__setattr__(self, name, value) + else: + setattr(self._data, name, value) + + def __delattr__(self, name): + delattr(self._data, name) + + def _processor(self, handler): + """Application processor to setup session for every request""" + self._cleanup() + self._load() + + try: + return handler() + finally: + self._save() + + def _load(self): + """Load the session from the store, by the id from cookie""" + cookie_name = self._config.cookie_name + cookie_domain = self._config.cookie_domain + httponly = self._config.httponly + self.session_id = web.cookies().get(cookie_name) + + # protection against session_id tampering + if self.session_id and not self._valid_session_id(self.session_id): + self.session_id = None + + self._check_expiry() + if self.session_id: + d = self.store[self.session_id] + self.update(d) + self._validate_ip() + + if not self.session_id: + self.session_id = self._generate_session_id() + + if self._initializer: + if isinstance(self._initializer, dict): + self.update(self._initializer) + elif hasattr(self._initializer, '__call__'): + self._initializer() + + self.ip = web.ctx.ip + + def _check_expiry(self): + # check for expiry + if self.session_id and self.session_id not in self.store: + if self._config.ignore_expiry: + self.session_id = None + else: + return self.expired() + + def _validate_ip(self): + # check for change of IP + if self.session_id and self.get('ip', None) != web.ctx.ip: + if not self._config.ignore_change_ip: + return self.expired() + + def _save(self): + if not self.get('_killed'): + self._setcookie(self.session_id) + self.store[self.session_id] = dict(self._data) + else: + self._setcookie(self.session_id, expires=-1) + + def _setcookie(self, session_id, expires='', **kw): + cookie_name = self._config.cookie_name + cookie_domain = self._config.cookie_domain + httponly = self._config.httponly + secure = self._config.secure + web.setcookie(cookie_name, session_id, expires=expires, domain=cookie_domain, httponly=httponly, secure=secure) + + def _generate_session_id(self): + """Generate a random id for session""" + + while True: + rand = os.urandom(16) + now = time.time() + secret_key = self._config.secret_key + session_id = sha1("%s%s%s%s" %(rand, now, utils.safestr(web.ctx.ip), secret_key)) + session_id = session_id.hexdigest() + if session_id not in self.store: + break + return session_id + + def _valid_session_id(self, session_id): + rx = utils.re_compile('^[0-9a-fA-F]+$') + return rx.match(session_id) + + def _cleanup(self): + """Cleanup the stored sessions""" + current_time = time.time() + timeout = self._config.timeout + if current_time - self._last_cleanup_time > timeout: + self.store.cleanup(timeout) + self._last_cleanup_time = current_time + + def expired(self): + """Called when an expired session is atime""" + self._killed = True + self._save() + raise SessionExpired(self._config.expired_message) + + def kill(self): + """Kill the session, make it no longer available""" + del self.store[self.session_id] + self._killed = True + +class Store: + """Base class for session stores""" + + def __contains__(self, key): + raise NotImplementedError + + def __getitem__(self, key): + raise NotImplementedError + + def __setitem__(self, key, value): + raise NotImplementedError + + def cleanup(self, timeout): + """removes all the expired sessions""" + raise NotImplementedError + + def encode(self, session_dict): + """encodes session dict as a string""" + pickled = pickle.dumps(session_dict) + return base64.encodestring(pickled) + + def decode(self, session_data): + """decodes the data to get back the session dict """ + pickled = base64.decodestring(session_data) + return pickle.loads(pickled) + +class DiskStore(Store): + """ + Store for saving a session on disk. + + >>> import tempfile + >>> root = tempfile.mkdtemp() + >>> s = DiskStore(root) + >>> s['a'] = 'foo' + >>> s['a'] + 'foo' + >>> time.sleep(0.01) + >>> s.cleanup(0.01) + >>> s['a'] + Traceback (most recent call last): + ... + KeyError: 'a' + """ + def __init__(self, root): + # if the storage root doesn't exists, create it. + if not os.path.exists(root): + os.makedirs( + os.path.abspath(root) + ) + self.root = root + + def _get_path(self, key): + if os.path.sep in key: + raise ValueError, "Bad key: %s" % repr(key) + return os.path.join(self.root, key) + + def __contains__(self, key): + path = self._get_path(key) + return os.path.exists(path) + + def __getitem__(self, key): + path = self._get_path(key) + if os.path.exists(path): + pickled = open(path).read() + return self.decode(pickled) + else: + raise KeyError, key + + def __setitem__(self, key, value): + path = self._get_path(key) + pickled = self.encode(value) + try: + f = open(path, 'w') + try: + f.write(pickled) + finally: + f.close() + except IOError: + pass + + def __delitem__(self, key): + path = self._get_path(key) + if os.path.exists(path): + os.remove(path) + + def cleanup(self, timeout): + now = time.time() + for f in os.listdir(self.root): + path = self._get_path(f) + atime = os.stat(path).st_atime + if now - atime > timeout : + os.remove(path) + +class DBStore(Store): + """Store for saving a session in database + Needs a table with the following columns: + + session_id CHAR(128) UNIQUE NOT NULL, + atime DATETIME NOT NULL default current_timestamp, + data TEXT + """ + def __init__(self, db, table_name): + self.db = db + self.table = table_name + + def __contains__(self, key): + data = self.db.select(self.table, where="session_id=$key", vars=locals()) + return bool(list(data)) + + def __getitem__(self, key): + now = datetime.datetime.now() + try: + s = self.db.select(self.table, where="session_id=$key", vars=locals())[0] + self.db.update(self.table, where="session_id=$key", atime=now, vars=locals()) + except IndexError: + raise KeyError + else: + return self.decode(s.data) + + def __setitem__(self, key, value): + pickled = self.encode(value) + now = datetime.datetime.now() + if key in self: + self.db.update(self.table, where="session_id=$key", data=pickled, vars=locals()) + else: + self.db.insert(self.table, False, session_id=key, data=pickled ) + + def __delitem__(self, key): + self.db.delete(self.table, where="session_id=$key", vars=locals()) + + def cleanup(self, timeout): + timeout = datetime.timedelta(timeout/(24.0*60*60)) #timedelta takes numdays as arg + last_allowed_time = datetime.datetime.now() - timeout + self.db.delete(self.table, where="$last_allowed_time > atime", vars=locals()) + +class ShelfStore: + """Store for saving session using `shelve` module. + + import shelve + store = ShelfStore(shelve.open('session.shelf')) + + XXX: is shelve thread-safe? + """ + def __init__(self, shelf): + self.shelf = shelf + + def __contains__(self, key): + return key in self.shelf + + def __getitem__(self, key): + atime, v = self.shelf[key] + self[key] = v # update atime + return v + + def __setitem__(self, key, value): + self.shelf[key] = time.time(), value + + def __delitem__(self, key): + try: + del self.shelf[key] + except KeyError: + pass + + def cleanup(self, timeout): + now = time.time() + for k in self.shelf.keys(): + atime, v = self.shelf[k] + if now - atime > timeout : + del self[k] + +if __name__ == '__main__' : + import doctest + doctest.testmod() diff --git a/web/template.py b/web/template.py new file mode 100644 index 0000000..6b45f7b --- /dev/null +++ b/web/template.py @@ -0,0 +1,1512 @@ +""" +simple, elegant templating +(part of web.py) + +Template design: + +Template string is split into tokens and the tokens are combined into nodes. +Parse tree is a nodelist. TextNode and ExpressionNode are simple nodes and +for-loop, if-loop etc are block nodes, which contain multiple child nodes. + +Each node can emit some python string. python string emitted by the +root node is validated for safeeval and executed using python in the given environment. + +Enough care is taken to make sure the generated code and the template has line to line match, +so that the error messages can point to exact line number in template. (It doesn't work in some cases still.) + +Grammar: + + template -> defwith sections + defwith -> '$def with (' arguments ')' | '' + sections -> section* + section -> block | assignment | line + + assignment -> '$ ' + line -> (text|expr)* + text -> + expr -> '$' pyexpr | '$(' pyexpr ')' | '${' pyexpr '}' + pyexpr -> +""" + +__all__ = [ + "Template", + "Render", "render", "frender", + "ParseError", "SecurityError", + "test" +] + +import tokenize +import os +import sys +import glob +import re +from UserDict import DictMixin + +from utils import storage, safeunicode, safestr, re_compile +from webapi import config +from net import websafe + +def splitline(text): + r""" + Splits the given text at newline. + + >>> splitline('foo\nbar') + ('foo\n', 'bar') + >>> splitline('foo') + ('foo', '') + >>> splitline('') + ('', '') + """ + index = text.find('\n') + 1 + if index: + return text[:index], text[index:] + else: + return text, '' + +class Parser: + """Parser Base. + """ + def __init__(self): + self.statement_nodes = STATEMENT_NODES + self.keywords = KEYWORDS + + def parse(self, text, name="