airflow_installation_guide.txt

Airflow Installation Process

Install Postgres
- "sudo apt update && sudo apt-get install postgresql-14" (14 is version number)
- "sudo -u postgres psql"
- 
Create Airflow database;
- "CREATE DATABASE airflow_db;"
- "CREATE USER airflow_user WITH PASSWORD 'choose_your_password';"
- "GRANT ALL PRIVILEGES ON DATABASE airflow_db TO airflow_user;"

Create Linux user and grant sudo privileges 
- sudo adduser lamisplus 
- sudo usermod -aG sudo lamisplus
- cd /home/lamisplus

Install Airflow on Linux Server
- "sudo apt update && sudo apt install python3 && sudo apt install python3-pip && sudo apt-get install pkg-config && sudo apt-get install libmysqlclient-dev"
- "sudo apt-get update && sudo apt-get install -y libkrb5-dev && sudo apt-get install libpq-dev && sudo apt-get install libldap2-dev libsasl2-dev"
- "pip install python-ldap"
- create airflow directory with "mkdir airflow"
- Change ownership of the directory with "sudo chown -R lamisplus:lamisplus /home/lamisplus/airflow"
- navigate into airflow directory with "cd airflow"
- create virtual environment with "pip install virtualenv && python3 -m venv airflow_env && source airflow_env/bin/activate"
- Activate virtual environment with "source airflow_env/bin/activate"
- Run "pip install apache-airflow==2.10.5 && pip install pandas && pip install numpy && pip install psycopg2-binary" (do this in the virtual environment)
- Confirm successful installation with "airflow version"
- Initialize Airflow with "airflow_env/bin/airflow db migrate"
- If initialization is successful, open airflow.cfg and update with "nano airflow.cfg"
# Set the metadata database connection string
sql_alchemy_conn = postgresql+psycopg2://airflow_user:D3ng1ne3r@localhost/airflow_db
# HIDE Example DAGs;
load_examples = False
 
- Create Airflow credentials with the following

 airflow users create \
          --username admin \
          --firstname Admin \
          --lastname Lamisplus \
          --role Admin \
          --email admin@localhost \
		  --password choose_your_password

- Test Whether Airflow is running 
- Start Airflow webserver and scheduler with "airflow webserver --port 8351 && airflow scheduler"
- navigate to "http://external_IP:8351"


CREATE AIRFLOW WEBSERVER AND SCHEDULER AS SYSTEMCTL SERVICES
- cd into systemctl services folder with "cd /etc/systemctl/system"

- create airflow-webserver.service file with "nano airflow-webserver.service", fill it with the info below and save with CTRL+X and hit Y
[Unit]
Description=Airflow Webserver daemon
After=network.target postgresql.service
Wants=postgresql.service
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
User=lamisplus
Group=lamisplus
Type=simple
ExecStart=/home/lamisplus/airflow_env/bin/python /home/lamisplus/airflow_env/bin/airflow webserver --port 8351
Restart=on-failure
RestartSec=5s
PrivateTmp=true
[Install]
WantedBy=multi-user.target

- create airflow-scheduler.webservice file with "nano airflow-scheduler.service", fill it with the info below and save with CTRL+X and hit Y 
[Unit]
Description=Airflow Scheduler daemon
After=network.target postgresql.service
Wants=postgresql.service
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
User=lamisplus
Group=lamisplus
Type=simple
ExecStart=/home/lamisplus/airflow_env/bin/python /home/lamisplus/airflow_env/bin/airflow scheduler
Restart=on-failure
RestartSec=5s
PrivateTmp=true
[Install]
WantedBy=multi-user.target

- To start both services, run "sudo systemctl daemon-reload && sudo systemctl start airflow-webserver && sudo systemctl start airflow-scheduler && sudo systemctl enable airflow-webserver && sudo systemctl enable airflow-scheduler"
- Confirm if both services are running with "sudo systemctl status airflow-webserver" and "sudo systemctl status airflow-scheduler"

Monitor Airflow Logs
sudo journalctl -u airflow-webserver.service --since "1 hour ago" | tail -n 20
sudo journalctl -u airflow-webserver

Accessing the Web UI:
Once the webserver is running, you can access the Airflow Web UI by navigating to http://external_IP:8351 in a web browser.