Enhancements for generating sitemaps

[msnoigrs]

This PR contains three enhancements of generating sitemaps.

1. Avoiding making the same robotparser over and over again while generating sitemaps. It lead to performance improvement.
2. ROBOTS_EXCLUSIONS can set non-ascii strings without crashing in Python2.
3. If robots.txt already exists, read that file and prefer it to ROBOTS_EXCLUSIONS. This is the same as robots task.

[da2x]

Need to output a warning when using copied resource robots.txt and 1) it doesn’t contain "Sitemap: /sitemapindex.xml" or 2) doesn’t contain every entry in ROBOTS_EXCLUSIONS.

[msnoigrs]

A warning is already outputed in nikola/plugins/task/robots.py.

1. It seems that generating sitemaps have nothing to do with "Sitemap:" lines. It's same as the current code.
2. no. It surely contain every entry.

    rules = []
    for rule in exclusions:
        rules.append('Disallow: {0}'.format(rule))

[msnoigrs]

It works for me. on both python2 and python3.

[msnoigrs]

I wrote the simple code.
Once The robotparser is created, it can parse only once. The parse method can accept multiple lines at once as follows.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import urllib.robotparser

def main():
    r = urllib.robotparser.RobotFileParser()
    r.parse(["User-Agent: *", "Disallow: /a.html", "Disallow: /b.html"])
    print(r.can_fetch("*", '/a.html')) # False
    print(r.can_fetch("*", '/b.html')) # False
    print(r.can_fetch("*", '/c.html')) # True

    r = urllib.robotparser.RobotFileParser()
    r.parse(["User-Agent: *", "Disallow: /a.html"])
    print(r.can_fetch("*", '/a.html')) # False
    print(r.can_fetch("*", '/b.html')) # True
    print(r.can_fetch("*", '/c.html')) # True
    r.parse(["User-Agent: *", "Disallow: /b.html"]) # Ignored
    print(r.can_fetch("*", '/a.html')) # False
    print(r.can_fetch("*", '/b.html')) # True
    print(r.can_fetch("*", '/c.html')) # True

if __name__ == '__main__':

    args = main()

[ralsina]

I am not familiar with this code, but this PR looks ok for me ad it has sat here for way too long... any opinions @Kwpolska ?