[SNE #1] Add ci cd
What does this MR do and why?
This MR intends to develp the CI/CD pipeline for dbt. The goal is to lint via sqlfluff our SQL queries and make sure the syntax is correct. Then we install the dependencies to run ssh commands to deploy our code on the remote instance. Finally, we run our models and tests on the postgres machine.
Screenshots or screen recordings
How to set up and validate locally
In order for this CI/CD pipeline to work properly, we have set up the VM with the following:
- Create a
gitlabciuser on the linux server and create the ssh_keys in/home/gitlabci/.ssh/to be used. - Create a
/var/dbtfolder in the linux server to host dbt related files - Create a virtualenv
dbtin/var/dbt/.virtualenv/dbt/ - Add the $SSH_PRIVATE_KEY in gitlab Variables, along with the $SSH_USER and $VM_IPADDRESS
- Add the SSH public key as a deploy key for the repo
- Git clone the repo in
/var/dbtwith username and passwordgit clone https://abarakat:<PERSONAL_TOKEN>@gitlab-forge.din.developpement-durable.gouv.fr/dgaln/mission-numerique/dbt-sne-mission-numerique.git - Add in the VM the env variables to connect to PostgresSQL in
/etc/profile.d/postgres.shand the url proxies in/etc/profile.d/http_proxy.sh - Make sure that the
gitlabciuser has the right access to the/var/dbt/dbt-sne-mission-numeriquefolder:sudo chown -R gitlabci /var/dbt/dbt-sne-mission-numeriqueandsudo chown -R gitlabci:root /var/dbt/.virtualenv/
MR acceptance checklist
-
I have evaluated the MR acceptance checklist for this MR.
Edited by andy.barakat