====== Daily QA/Operations Task for the V5/PSQL ====== Cron output at dev02 is sent to the local inbox and can be viewed with mutt. The machine should at least be checked two times:
    * At the begin of your workday * In the early afternoon around 15:00
to verify that there is no need for any actions. Mails should be deleted as soon as they are analyzed and at the end of the work day, there should be no 'busy' mails in the inbox. **There should be always a QA trello ticket and it should contain all issues that need further investigation** **TODO**: add the missing steps you do every day ==== Daily ==== Check for processes that have been blocked (not started) due to database locking (mutt). === refresh daily live/prelive === * import times, num prods in db, obvious failures * errors: less /var/mail/picalike | grep ERROR * warnings: less /var/mail/picalike | grep WARNING

→ list shops with: many missing cat maps, truncated prices, …

=== v5 extractor === there is a script for error checking: /home/picalike/v5/scripts/check_extractor_logs.sh check that the v5_extractor is up & running and that were recent enrichments for any shop: docker logs -t --tail 10000 v5_extractor_container 2>&1 | grep "times: img/net/store" important check: docker logs -t --tail 30000 v5_extractor_container 2>&1 | grep "error rate above" check the summary of all shop enrichments: curl --silent "http://localhost:1115/list_history?non_empty=1"| python3 -m json.tool if a shop has errors » 0 or processed_images « total_images, check the logs: docker logs -t --tail 20000 v5_extractor_container 2>&1 | grep SHOP_ID === Image Blocking === Recently it happened that shops block the image cloud, despite a user-agent, to check for shops for which this happened, check the tiny proxy log: ssh picalike@v22019026221283999.supersrv.de grep ": CONNECT" /home/picalike/proxy/tinyproxy.log |cut -d':' -f5|cut -d' ' -f3|sort -u grep "Dec 21" /home/picalike/proxy/tinyproxy.log | grep Timeout | wc -l For an overview time-outs per domain use dump_timeouts.py: https://git.picalike.corpex-kunden.de/incubator/swiss-army-knife/-/blob/master/tinyproxy/dump_timeouts.py A domain is listed there if at least one image had status_code 403 (forbidden/access denied). After this happened, all further requests for this domain are routed through the proxy! === Misc === * push_attrs: check that there are any uploads and the times are no super high * upload_int_scores: check that uploads have been performed to reports engine (multiple times per day) and also to pre-live (at least once a day) * grep for ERRORs and WARNINGs in refresh_daily_[pre]live logs to see if there are new (not in weekly logs) problems Keywords: devops v5 operations qa daily mutt import ops