krawla_tech
Here is a list of technologies used for the Krawla Project (in no particular order):
python
- python doc: https://docs.python.org
zeromq
- brokerless messaging
- PUSH and PULL should not be used over the internet
- zeromq guide for python: http://zguide.zeromq.org/py:all
- doc for pyzmq: https://pyzmq.readthedocs.io/en/latest/
mongo
- mongo doc: https://docs.mongodb.com/
- pymongo doc: https://api.mongodb.com/python/current/
json
- json doc: https://www.json.org/
msgpack
- msgpack doc: https://msgpack.org/index.html
- binary serialization protocol
- faster than json
- good for data that needs to be sent over the wire
- bad for files that a human needs to read
sqlite
- sqlite doc: https://www.sqlite.org/index.html
- python sqlite: https://docs.python.org/3/library/sqlite3.html
flask
- flask doc: http://flask.pocoo.org/
beautifulsoup
- beautifulsoup doc: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
css selectors
- css selectors doc: https://www.w3schools.com/cssref/css_selectors.asp
regular expressions (regex)
- python doc: https://docs.python.org/3/library/re.html
requests
firefox
- firefox download: https://www.mozilla.org/en-US/firefox/new/
selenium
- selenium: https://github.com/SeleniumHQ/selenium
stem
git
- git doc: https://git-scm.com/docs
virtualenv
- virtualenv
ssh
- ssh
pandas
- official documentation: https://pandas.pydata.org/pandas-docs/stable/
- youtube channel: https://www.youtube.com/user/dataschool/playlists
krawla_tech.txt · Last modified: 2024/04/11 14:23 by 127.0.0.1