[SNE #1] Add ci cd

What does this MR do and why?

This MR intends to develp the CI/CD pipeline for dbt. The goal is to lint via sqlfluff our SQL queries and make sure the syntax is correct. Then we install the dependencies to run ssh commands to deploy our code on the remote instance. Finally, we run our models and tests on the postgres machine.

Screenshots or screen recordings

How to set up and validate locally

In order for this CI/CD pipeline to work properly, we have set up the VM with the following:

  1. Create a gitlabci user on the linux server and create the ssh_keys in /home/gitlabci/.ssh/ to be used.
  2. Create a /var/dbt folder in the linux server to host dbt related files
  3. Create a virtualenv dbt in /var/dbt/.virtualenv/dbt/
  4. Add the $SSH_PRIVATE_KEY in gitlab Variables, along with the $SSH_USER and $VM_IPADDRESS
  5. Add the SSH public key as a deploy key for the repo
  6. Git clone the repo in /var/dbt with username and password git clone https://abarakat:<PERSONAL_TOKEN>@gitlab-forge.din.developpement-durable.gouv.fr/dgaln/mission-numerique/dbt-sne-mission-numerique.git
  7. Add in the VM the env variables to connect to PostgresSQL in /etc/profile.d/postgres.sh and the url proxies in /etc/profile.d/http_proxy.sh
  8. Make sure that the gitlabci user has the right access to the /var/dbt/dbt-sne-mission-numerique folder: sudo chown -R gitlabci /var/dbt/dbt-sne-mission-numerique and sudo chown -R gitlabci:root /var/dbt/.virtualenv/

MR acceptance checklist

Edited by andy.barakat

Merge request reports

Loading