Picalike Dokuwiki Archive

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Sync Project ======

Due to many workaround solutions on the report-engine speed problems, Sebastian created many collections besides the mains ones (products_with_history, trends_data_v4).

Due to this fact, scripts was created to fill this new collections. The main problem with this is that don't exist a control of how import or a full reimport of the system due to changes on the mapping service.

So the sync project was created for this, to gather all the important scripts to fill these collections and to monitor them

Due to development phases, there are two versions of the Sync Project, the Old and New ones


===== Old Sync =====

The Old Sync uses a library called [[https://rocketry.readthedocs.io/en/stable/|Rocketry]] to run the function in the desired time range, run function after other being finished

This library uses another library from the same creator, called RedBird to register the calls in a LOG system. In this case, we use CSV to register the calls and to know if was successful or not

This version tend to be deprecated due to the fact that a better sync would be by shop, not the whole system every day. The reason for the Old Sync still being up is due to some tasks that are better to still be ran as whole day


===== New Sync =====

The new sync uses only FastAPI to receive Shop Conveyor Belt calls. To run each function by each shop, to the updates being incremented by shop


===== Services =====

The services are


  * create_agg_workers (To generate workers packages on the aggregation_tasks inside the report-engine mongo) [New]
  * run_agg_workers (To grab the workers packages and create the products_with_history_agg mongo collection incrementally, the agg collection is for almost all endpoint in the report-engine) [New]
  * color_v4 (to fill the OSA mongo DB collection trend_data_v4, to calculate the trends of the shops) [Old | New]
  * unwinded_products (To fill the collection products_unwinded in the report-engine mongo to be used on the search endpoint only) [Old | New]
  * trends_agg (this is to speed up the trending colors for the get_trend_overview endpoint) [Old | New]
  * trends_by_metadata (this is to speed up the search of trends for the get_trend_overview) [Old | New]
  * attr_filter (This is for endpoints in the report-engine that deals with attributes, to filter by categories and shops) [Old | New]
  * recommedations (Generate all the counselor data to all users and shops) [Old]


===== Dependencies =====


  * Rocketry
  * FastAPI
  * Python
  * pipenv
  * motor


===== Incident Log =====

None registered at the moment


===== Authors =====


  * Marcus Pantoja
  * Wellington Moraes
  * Otavio Mascarenhas