I note down here things that are used to import into the report engine. the scripts live in the Report Engine Git in the 'julius' branch.
Tasks are: exporting intscores, visual attributes, krawla startpages and importing to report engine.
Scripts are deployed on dev01 and dev02, here comes the list:
script to upload int scores to osa data mongo:
/home/picalike/v5/v5_sim_api/scripts/push_intscores_mongo.py
→ called per cron
script to upload visual attributes to osa data mongo:
/home/picalike/v5/scripts/push_attrs_mongo.py
→ called per cron
HINT: logs in /home/picalike/import_mongo/logs
script to import feed shops into report engine (twice a week at the moment):
/home/picalike/import_mongo/feed_import.sh
→ called per cron
script to import krawla shops that finished during the last 24 hours (SCB):
/home/picalike/import_mongo/daily_krawla_import.sh
→ called per cron
script to get startpage data for krawla shops:
/home/picalike/import_mongo/daily_startpage_import.sh
→ called per cron
script to generate a bash script to import shops that were not imported this week:
/home/picalike/import_mongo/old_shops.py
→ trigger manually (takes ~5 minutes) (and remove the bash script after using it)