krawla_tech

Here is a list of technologies used for the Krawla Project (in no particular order):

python

python doc: https://docs.python.org

zeromq

brokerless messaging
PUSH and PULL should not be used over the internet
zeromq guide for python: http://zguide.zeromq.org/py:all
doc for pyzmq: https://pyzmq.readthedocs.io/en/latest/

mongo

mongo doc: https://docs.mongodb.com/
pymongo doc: https://api.mongodb.com/python/current/

json

json doc: https://www.json.org/

msgpack

msgpack doc: https://msgpack.org/index.html
binary serialization protocol
faster than json
good for data that needs to be sent over the wire
bad for files that a human needs to read

sqlite

sqlite doc: https://www.sqlite.org/index.html
python sqlite: https://docs.python.org/3/library/sqlite3.html

flask

flask doc: http://flask.pocoo.org/

beautifulsoup

beautifulsoup doc: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

css selectors

css selectors doc: https://www.w3schools.com/cssref/css_selectors.asp

regular expressions (regex)

python doc: https://docs.python.org/3/library/re.html

requests

requests: http://docs.python-requests.org/en/master/

firefox

firefox download: https://www.mozilla.org/en-US/firefox/new/

selenium

selenium: https://github.com/SeleniumHQ/selenium

stem

stem: https://stem.torproject.org/

git

git
git doc: https://git-scm.com/docs

virtualenv

virtualenv

ssh

ssh

pandas

pandas
official documentation: https://pandas.pydata.org/pandas-docs/stable/
youtube channel: https://www.youtube.com/user/dataschool/playlists