Skip to content
 
 

Latest commit

 

History

127 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

HCF (HubStorage Frontier) Backend for Frontera

When used with scrapy, use it with Scrapy Scheduler provided by scrapy-frontera. Scrapy scheduler provided by Frontera is not supported

See usage instructions at module and class docstrings at backend.py.

Package also installs a convenient command line tool for hubstorage frontier handling and manipulation: hcfpal.py. It supports dumping, count, deletion, moving, listing, etc.

Another tool provided is hcfmanager.py. It facilitates the scheduling of consumer spider jobs.

About

Crawl Frontier HCF backend

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages