From 655252ba27831636e3842d61741e407640dda069 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Thu, 27 Oct 2022 14:14:29 -0500 Subject: [PATCH 01/35] Update README.md --- README.md | 54 ++++++++++++++++++++++++++---------------------------- 1 file changed, 26 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index f614606..3511b0f 100644 --- a/README.md +++ b/README.md @@ -1,66 +1,64 @@ -# ansol +# Autoclock RPC -### machine setting -* this process works best on latitude machines - * because the initial state of the machine is cleaner +### Optimal Machine Settings +* Our Latitude.sh s3.large.x86 server starts with the settings below, which we prefer because: + * the initial state of the machine is cleaner than others that we have tried * disks are named consistently (nvme01, nvme0n2) * ubuntu installed (preferably ubuntu 20.04, 22.04) - this won't work with centos etc since they don't use aptitude by default * the login user being ubuntu helps (all the solana operations are done using the solana user that the ansible playbook creates) - * ubuntu should be in the sudoers list - * clean unmounted disks. if your root is on one of partitions and you pass it as an argument, this could potentially be disastrous. + * ubuntu is in the sudoer's list + * unmounted disks are clean - if your root is on one of partitions and you pass it as an argument, this could be disastrous. -* all the above are satisfied by a fresh latitude launch +* all the above are satisfied by a fresh s3.large.x86 launch here https://www.latitude.sh/pricing +* Zen3 AMD Epyc’s such as the 7443p are considered some of the most performant nodes for keeping up with the tip of the chain at the moment, and support large amounts of RAM. -* you can launch latitude machines here https://www.latitude.sh/pricing -* recommend the s3.large.x86 - it is one of the most performant nodes for staying at tip - -* Specs +* Recommended RPC Specs * 24 cores or more - * 512 GB ram if you want to use ramdisk/tmpfs and store the accounts db in RAM (we use 300 GB for ram disk). without tmpfs, the ram requirement can be significantly lower. ~256 GB - * 3-4 TB (multiple disks is ok i.e. 2x 1.9TB because the ansible playbook stripes them together) + * 512 GB RAM if you want to use ramdisk/tmpfs and store the accounts db in RAM (we use 300 GB for ram disk). without tmpfs, the ram requirement can be significantly lower (~256 GB) + * 3-4 TB (multiple disks is okay i.e. 2x 1.9TB because the ansible playbook stripes them together) -### step 1: ssh into your machine +### Step 1: SSH into your machine -### step 2: start a screen session +### Step 2: Start a screen session ``` screen -S sol ``` -### step 3: install ansible +### Step 3: Install ansible ``` sudo apt-get install ansible -y ``` -### step 4: clone the anssol repository +### Step 4: Clone the autoclock-rpc repository ``` -git clone https://github.com/dubbelosix/ansol.git +git clone https://github.com/overclock-validator/autoclock-rpc.git ``` -### step 5: cd into the ansol folder +### Step 5: cd into the autoclock-rpc folder ``` -cd ansol +cd autoclock-rpc ``` -### step 6: run the ansible command +### Step 6: Run the ansible command * this command can take between 10-20 minutes based on the specs of the machine * it takes long because it does everything necessary to start the validator (format disks, checkout the solana repo and build, download the latest snapshot etc) ``` time ansible-playbook runner.yaml --extra-vars='{"solana_version": "v1.13.4", "swap_mb":100000,"raw_disk_list":["/dev/nvme0n1","/dev/nvme1n1"],"setup_disks":true,"download_snapshot":true,"ramdisk_size":300}' ``` -#### params explained -* solana_version: which version of solana do we want to run -* swap_mb: megabytes of swap. can set this to 50% of RAM or even lower. 100 GB is fine on a 512 GB RAM machine (variable value is in MB so 100000) +#### ~ Parameters explained ~ +* solana_version: the version of solana that we want to run. Check the Solana Tech discord’s mb-announcements channel for the recommended version. +* swap_mb: megabytes of swap. This can be set this to 50% of RAM or even lower. 100 GB is fine on a 512 GB RAM machine (variable value is in MB so 100000) * raw_disk_list: the list of currently unmounted disks that will be wiped, raided, formatted with ext4 and then mounted to /mnt * ramdisk_size: this is optional and only necessary if you want to use ramdisk for the validator - carves out a large portion of the RAM to store the accountsdb. On a 512 GB RAM instance, this can be set to 300 GB (variable value is in GB so 300) -### step 7: after ansible finishes +### Step 7: Once ansible finishes: switch to the solana user with ``` sudo su - solana ``` -### step 8: and check the validator status with +### Step 8: Check the status ``` /mnt/solana/target/release/solana-validator --ledger /mnt/solana-ledger monitor ledger monitor @@ -68,11 +66,11 @@ Ledger location: /mnt/solana-ledger ⠉ Validator startup: SearchingForRpcService... ``` -### Initially the monitor should just show. this will last for a few minutes and is normal +### Initially the monitor should just show the below message which will last for a few minutes and is normal: ``` ⠉ Validator startup: SearchingForRpcService... ``` -### after a while, the message at the terminal should change to +### After a while, the message at the terminal should change to: ``` ⠐ 00:08:26 | Processed Slot: 156831951 | Confirmed Slot: 156831951 | Finalized Slot: 156831917 | Full Snapshot Slot: 156813730 | ``` From 8a8a699025a64121187a8f17675351635926dae4 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Thu, 27 Oct 2022 14:15:11 -0500 Subject: [PATCH 02/35] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3511b0f..c6cc007 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ * ubuntu is in the sudoer's list * unmounted disks are clean - if your root is on one of partitions and you pass it as an argument, this could be disastrous. -* all the above are satisfied by a fresh s3.large.x86 launch here https://www.latitude.sh/pricing +* All the above are satisfied by a fresh s3.large.x86 launch here https://www.latitude.sh/pricing * Zen3 AMD Epyc’s such as the 7443p are considered some of the most performant nodes for keeping up with the tip of the chain at the moment, and support large amounts of RAM. * Recommended RPC Specs From f0cebe7172dc68e302b8cb133ee28fb5172e46d0 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Fri, 28 Oct 2022 23:48:46 -0500 Subject: [PATCH 03/35] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c6cc007..2a04c2a 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ * 3-4 TB (multiple disks is okay i.e. 2x 1.9TB because the ansible playbook stripes them together) ### Step 1: SSH into your machine - + ### Step 2: Start a screen session ``` From d91a957345e9050b6ad3fa7099ee946449577562 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Sat, 29 Oct 2022 00:07:34 -0500 Subject: [PATCH 04/35] Update README.md --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 2a04c2a..51bc3fb 100644 --- a/README.md +++ b/README.md @@ -4,18 +4,18 @@ * Our Latitude.sh s3.large.x86 server starts with the settings below, which we prefer because: * the initial state of the machine is cleaner than others that we have tried * disks are named consistently (nvme01, nvme0n2) - * ubuntu installed (preferably ubuntu 20.04, 22.04) - this won't work with centos etc since they don't use aptitude by default + * ubuntu installed (preferably ubuntu 20.04, 22.04) - this won't work with centos, etc. since they don't use aptitude by default * the login user being ubuntu helps (all the solana operations are done using the solana user that the ansible playbook creates) * ubuntu is in the sudoer's list - * unmounted disks are clean - if your root is on one of partitions and you pass it as an argument, this could be disastrous. + * unmounted disks are clean - if your root is on one of partitions and you pass it as an argument, this could be disastrous -* All the above are satisfied by a fresh s3.large.x86 launch here https://www.latitude.sh/pricing +* All the above are satisfied by a fresh s3.large.x86 launch found here: https://www.latitude.sh/pricing * Zen3 AMD Epyc’s such as the 7443p are considered some of the most performant nodes for keeping up with the tip of the chain at the moment, and support large amounts of RAM. * Recommended RPC Specs * 24 cores or more * 512 GB RAM if you want to use ramdisk/tmpfs and store the accounts db in RAM (we use 300 GB for ram disk). without tmpfs, the ram requirement can be significantly lower (~256 GB) - * 3-4 TB (multiple disks is okay i.e. 2x 1.9TB because the ansible playbook stripes them together) + * 3-4 TB (multiple disks is okay - i.e. 2x 1.9TB - because the ansible playbook stripes them together) ### Step 1: SSH into your machine @@ -42,7 +42,7 @@ cd autoclock-rpc ### Step 6: Run the ansible command * this command can take between 10-20 minutes based on the specs of the machine -* it takes long because it does everything necessary to start the validator (format disks, checkout the solana repo and build, download the latest snapshot etc) +* it takes long because it does everything necessary to start the validator (format disks, checkout the solana repo and build it, download the latest snapshot, etc.) ``` time ansible-playbook runner.yaml --extra-vars='{"solana_version": "v1.13.4", "swap_mb":100000,"raw_disk_list":["/dev/nvme0n1","/dev/nvme1n1"],"setup_disks":true,"download_snapshot":true,"ramdisk_size":300}' ``` @@ -75,5 +75,5 @@ Ledger location: /mnt/solana-ledger ⠐ 00:08:26 | Processed Slot: 156831951 | Confirmed Slot: 156831951 | Finalized Slot: 156831917 | Full Snapshot Slot: 156813730 | ``` -If you see the message above, then everything is working fine! Gratz. you have a new RPC server and you can visit the URL at http://xx.xx.xx.xx:8899/ +If you see the message above, then everything is working fine! Gratz. You have a new RPC server and you can visit the URL at http://xx.xx.xx.xx:8899/ From f0a8326232f64aafb80c549ec635dfaec25105c2 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Sat, 29 Oct 2022 00:32:02 -0500 Subject: [PATCH 05/35] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 51bc3fb..0c041eb 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ cd autoclock-rpc ### Step 6: Run the ansible command * this command can take between 10-20 minutes based on the specs of the machine * it takes long because it does everything necessary to start the validator (format disks, checkout the solana repo and build it, download the latest snapshot, etc.) +* make sure that the solana_version is up to date (see below) ``` time ansible-playbook runner.yaml --extra-vars='{"solana_version": "v1.13.4", "swap_mb":100000,"raw_disk_list":["/dev/nvme0n1","/dev/nvme1n1"],"setup_disks":true,"download_snapshot":true,"ramdisk_size":300}' ``` From 4b143b569b506d25af24a94fccb5e14c9da497da Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Sat, 29 Oct 2022 00:33:37 -0500 Subject: [PATCH 06/35] Update README.md --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 0c041eb..ed9737a 100644 --- a/README.md +++ b/README.md @@ -54,8 +54,7 @@ time ansible-playbook runner.yaml --extra-vars='{"solana_version": "v1.13.4", "s * raw_disk_list: the list of currently unmounted disks that will be wiped, raided, formatted with ext4 and then mounted to /mnt * ramdisk_size: this is optional and only necessary if you want to use ramdisk for the validator - carves out a large portion of the RAM to store the accountsdb. On a 512 GB RAM instance, this can be set to 300 GB (variable value is in GB so 300) -### Step 7: Once ansible finishes: -switch to the solana user with +### Step 7: Once ansible finishes, switch to the solana user with: ``` sudo su - solana ``` From 8c959291a1318f3acc536e50772c80766717787c Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Sat, 29 Oct 2022 00:35:16 -0500 Subject: [PATCH 07/35] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ed9737a..6ba6b3e 100644 --- a/README.md +++ b/README.md @@ -66,11 +66,11 @@ Ledger location: /mnt/solana-ledger ⠉ Validator startup: SearchingForRpcService... ``` -### Initially the monitor should just show the below message which will last for a few minutes and is normal: +#### Initially the monitor should just show the below message which will last for a few minutes and is normal: ``` ⠉ Validator startup: SearchingForRpcService... ``` -### After a while, the message at the terminal should change to: +#### After a while, the message at the terminal should change to: ``` ⠐ 00:08:26 | Processed Slot: 156831951 | Confirmed Slot: 156831951 | Finalized Slot: 156831917 | Full Snapshot Slot: 156813730 | ``` From 0bb54e178d5358ec966134c96db7e774e784af03 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Sat, 29 Oct 2022 00:36:30 -0500 Subject: [PATCH 08/35] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6ba6b3e..16f3166 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ Ledger location: /mnt/solana-ledger ``` ⠉ Validator startup: SearchingForRpcService... ``` -#### After a while, the message at the terminal should change to: +#### After a while, the message at the terminal should change to something like this: ``` ⠐ 00:08:26 | Processed Slot: 156831951 | Confirmed Slot: 156831951 | Finalized Slot: 156831917 | Full Snapshot Slot: 156813730 | ``` From 60c4bf4786248213d491f76cb266ec4ae4cc2214 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Sat, 29 Oct 2022 00:36:48 -0500 Subject: [PATCH 09/35] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 16f3166..128b5a6 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ Ledger location: /mnt/solana-ledger ``` ⠉ Validator startup: SearchingForRpcService... ``` -#### After a while, the message at the terminal should change to something like this: +#### After a while, the message at the terminal should change to something similar to this: ``` ⠐ 00:08:26 | Processed Slot: 156831951 | Confirmed Slot: 156831951 | Finalized Slot: 156831917 | Full Snapshot Slot: 156813730 | ``` From 76cc1989133b32d2c34e82237395678f7ca5cb05 Mon Sep 17 00:00:00 2001 From: Flavio Meyer Date: Sat, 29 Oct 2022 23:44:51 +0400 Subject: [PATCH 10/35] Restructure Ansible role --- defaults/main.yml | 8 +++ restart.sh => files/restart.sh | 0 snapcheck.py => files/snapcheck.py | 0 .../snapshot-finder.py | 0 sol.service => files/sol.service | 0 runner.yaml | 58 ------------------- deps.yaml => tasks/deps.yaml | 0 dirs.yaml => tasks/dirs.yaml | 0 disks.yaml => tasks/disks.yaml | 0 file_setup.yaml => tasks/file_setup.yaml | 0 git.yaml => tasks/git.yaml | 0 keygen.yaml => tasks/keygen.yaml | 0 tasks/main.yaml | 52 +++++++++++++++++ ramdisk.yaml => tasks/ramdisk.yaml | 0 rotate.yaml => tasks/rotate.yaml | 0 .../snapshot_downloader.yaml | 0 swap.yaml => tasks/swap.yaml | 0 user.yaml => tasks/user.yaml | 0 var_check.yaml => tasks/var_check.yaml | 1 - validator.j2 => templates/validator.j2 | 0 20 files changed, 60 insertions(+), 59 deletions(-) create mode 100644 defaults/main.yml rename restart.sh => files/restart.sh (100%) rename snapcheck.py => files/snapcheck.py (100%) rename snapshot-finder.py => files/snapshot-finder.py (100%) rename sol.service => files/sol.service (100%) delete mode 100644 runner.yaml rename deps.yaml => tasks/deps.yaml (100%) rename dirs.yaml => tasks/dirs.yaml (100%) rename disks.yaml => tasks/disks.yaml (100%) rename file_setup.yaml => tasks/file_setup.yaml (100%) rename git.yaml => tasks/git.yaml (100%) rename keygen.yaml => tasks/keygen.yaml (100%) create mode 100644 tasks/main.yaml rename ramdisk.yaml => tasks/ramdisk.yaml (100%) rename rotate.yaml => tasks/rotate.yaml (100%) rename snapshot_downloader.yaml => tasks/snapshot_downloader.yaml (100%) rename swap.yaml => tasks/swap.yaml (100%) rename user.yaml => tasks/user.yaml (100%) rename var_check.yaml => tasks/var_check.yaml (99%) rename validator.j2 => templates/validator.j2 (100%) diff --git a/defaults/main.yml b/defaults/main.yml new file mode 100644 index 0000000..54a1093 --- /dev/null +++ b/defaults/main.yml @@ -0,0 +1,8 @@ +--- +# defaults file for Solana RPC +solana_version: "v1.13.4" +raw_disk_list: "/dev/nvme0n1","/dev/nvme1n1" +setup_disks: "true" +download_snapshot: "true" +ramdisk_size: "300" +swap_mb: "100000" diff --git a/restart.sh b/files/restart.sh similarity index 100% rename from restart.sh rename to files/restart.sh diff --git a/snapcheck.py b/files/snapcheck.py similarity index 100% rename from snapcheck.py rename to files/snapcheck.py diff --git a/snapshot-finder.py b/files/snapshot-finder.py similarity index 100% rename from snapshot-finder.py rename to files/snapshot-finder.py diff --git a/sol.service b/files/sol.service similarity index 100% rename from sol.service rename to files/sol.service diff --git a/runner.yaml b/runner.yaml deleted file mode 100644 index 2fcfd78..0000000 --- a/runner.yaml +++ /dev/null @@ -1,58 +0,0 @@ ---- - - name: "playbook runner" - hosts: localhost - connection: local - tasks: - - - name: var check - ansible.builtin.import_tasks: var_check.yaml - - - name: setup disks - ansible.builtin.import_tasks: disks.yaml - when: setup_disks|default(false)|bool == true - - - name: create user - ansible.builtin.import_tasks: user.yaml - - - name: install dependencies - ansible.builtin.import_tasks: deps.yaml - - - name: folders - ansible.builtin.import_tasks: dirs.yaml - - - name: swap - ansible.builtin.import_tasks: swap.yaml - - - name: ramdisk - ansible.builtin.import_tasks: ramdisk.yaml - when: ramdisk_size is defined - - - name: logrotate - ansible.builtin.import_tasks: rotate.yaml - - - name: git - ansible.builtin.import_tasks: git.yaml - - - name: solana keygen - ansible.builtin.import_tasks: keygen.yaml - - - name: file setup - ansible.builtin.import_tasks: file_setup.yaml - - - name: snapshot download - ansible.builtin.import_tasks: snapshot_downloader.yaml - when: download_snapshot|default(true)|bool == true - - - name: restart without waiting - become: true - become_user: root - shell: /home/solana/restart.sh 1 - when: download_snapshot|default(true)|bool == true - - - name: restart with waiting - become: true - become_user: root - shell: /home/solana/restart.sh - when: download_snapshot|default(true)|bool == false - - diff --git a/deps.yaml b/tasks/deps.yaml similarity index 100% rename from deps.yaml rename to tasks/deps.yaml diff --git a/dirs.yaml b/tasks/dirs.yaml similarity index 100% rename from dirs.yaml rename to tasks/dirs.yaml diff --git a/disks.yaml b/tasks/disks.yaml similarity index 100% rename from disks.yaml rename to tasks/disks.yaml diff --git a/file_setup.yaml b/tasks/file_setup.yaml similarity index 100% rename from file_setup.yaml rename to tasks/file_setup.yaml diff --git a/git.yaml b/tasks/git.yaml similarity index 100% rename from git.yaml rename to tasks/git.yaml diff --git a/keygen.yaml b/tasks/keygen.yaml similarity index 100% rename from keygen.yaml rename to tasks/keygen.yaml diff --git a/tasks/main.yaml b/tasks/main.yaml new file mode 100644 index 0000000..3977ff9 --- /dev/null +++ b/tasks/main.yaml @@ -0,0 +1,52 @@ +--- +# tasks file for Solana RPC +- name: var check + include_tasks: var_check.yaml + +- name: setup disks + include_tasks: disks.yaml + when: setup_disks|default(false)|bool == true + +- name: create user + include_tasks: user.yaml + +- name: install dependencies + include_tasks: deps.yaml + +- name: folders + include_tasks: dirs.yaml + +- name: swap + include_tasks: swap.yaml + +- name: ramdisk + include_tasks: ramdisk.yaml + when: ramdisk_size is defined + +- name: logrotate + include_tasks: rotate.yaml + +- name: git + include_tasks: git.yaml + +- name: solana keygen + include_tasks: keygen.yaml + +- name: file setup + include_tasks: file_setup.yaml + +- name: snapshot download + include_tasks: snapshot_downloader.yaml + when: download_snapshot|default(true)|bool == true + +- name: restart without waiting + become: true + become_user: root + shell: /home/solana/restart.sh 1 + when: download_snapshot|default(true)|bool == true + +- name: restart with waiting + become: true + become_user: root + shell: /home/solana/restart.sh + when: download_snapshot|default(true)|bool == false diff --git a/ramdisk.yaml b/tasks/ramdisk.yaml similarity index 100% rename from ramdisk.yaml rename to tasks/ramdisk.yaml diff --git a/rotate.yaml b/tasks/rotate.yaml similarity index 100% rename from rotate.yaml rename to tasks/rotate.yaml diff --git a/snapshot_downloader.yaml b/tasks/snapshot_downloader.yaml similarity index 100% rename from snapshot_downloader.yaml rename to tasks/snapshot_downloader.yaml diff --git a/swap.yaml b/tasks/swap.yaml similarity index 100% rename from swap.yaml rename to tasks/swap.yaml diff --git a/user.yaml b/tasks/user.yaml similarity index 100% rename from user.yaml rename to tasks/user.yaml diff --git a/var_check.yaml b/tasks/var_check.yaml similarity index 99% rename from var_check.yaml rename to tasks/var_check.yaml index caa59a8..99b129f 100644 --- a/var_check.yaml +++ b/tasks/var_check.yaml @@ -11,4 +11,3 @@ with_items: - "{{ raw_disk_list }}" when: setup_disks is defined and (setup_disks | bool) - diff --git a/validator.j2 b/templates/validator.j2 similarity index 100% rename from validator.j2 rename to templates/validator.j2 From 7abcd63d48561265c925a3df23bba7fa04e55838 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Wed, 2 Nov 2022 23:02:48 -0500 Subject: [PATCH 11/35] Update README.md --- README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.md b/README.md index 128b5a6..06423aa 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,8 @@ # Autoclock RPC +### What is it good for? +*The goal of the Autoclock RPC ansible playbook is to have you caught up with the Solana blockchain within 15 minutes, once you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For an in depth guide on possible RPC configurations once your RPC is good to go, refer to https://github.com/rpcpool/solana-rpc-ansible + ### Optimal Machine Settings * Our Latitude.sh s3.large.x86 server starts with the settings below, which we prefer because: * the initial state of the machine is cleaner than others that we have tried @@ -75,5 +78,10 @@ Ledger location: /mnt/solana-ledger ⠐ 00:08:26 | Processed Slot: 156831951 | Confirmed Slot: 156831951 | Finalized Slot: 156831917 | Full Snapshot Slot: 156813730 | ``` +#### Check whether the RPC is caught up with the rest of the cluster with: +``` +solana catchup --our-localhost +``` + If you see the message above, then everything is working fine! Gratz. You have a new RPC server and you can visit the URL at http://xx.xx.xx.xx:8899/ From 27f73b99035263c95171604c3f23763522917e37 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Thu, 3 Nov 2022 15:33:17 -0500 Subject: [PATCH 12/35] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 06423aa..c205404 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Autoclock RPC ### What is it good for? -*The goal of the Autoclock RPC ansible playbook is to have you caught up with the Solana blockchain within 15 minutes, once you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For an in depth guide on possible RPC configurations once your RPC is good to go, refer to https://github.com/rpcpool/solana-rpc-ansible +The goal of the Autoclock RPC ansible playbook is to have you caught up with the Solana blockchain within 15 minutes, once you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For an in depth guide on possible RPC configurations once your RPC is good to go, refer to https://github.com/rpcpool/solana-rpc-ansible ### Optimal Machine Settings * Our Latitude.sh s3.large.x86 server starts with the settings below, which we prefer because: From d8b379492f5d56a452acb4478e907b77cddf2581 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Thu, 3 Nov 2022 15:34:17 -0500 Subject: [PATCH 13/35] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c205404..dfcfcd0 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Autoclock RPC ### What is it good for? -The goal of the Autoclock RPC ansible playbook is to have you caught up with the Solana blockchain within 15 minutes, once you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For an in depth guide on possible RPC configurations once your RPC is good to go, refer to https://github.com/rpcpool/solana-rpc-ansible +The goal of the Autoclock RPC ansible playbook is to have you caught up on the Solana blockchain within 15 minutes, assuming you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For an in depth guide on possible RPC configurations once your RPC is good to go, refer to https://github.com/rpcpool/solana-rpc-ansible ### Optimal Machine Settings * Our Latitude.sh s3.large.x86 server starts with the settings below, which we prefer because: From 253e4f7363a43b711010004d5398c79c9e80eb57 Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Fri, 4 Nov 2022 10:05:26 -0500 Subject: [PATCH 14/35] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index dfcfcd0..e1642b6 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Autoclock RPC ### What is it good for? -The goal of the Autoclock RPC ansible playbook is to have you caught up on the Solana blockchain within 15 minutes, assuming you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For an in depth guide on possible RPC configurations once your RPC is good to go, refer to https://github.com/rpcpool/solana-rpc-ansible +The goal of the Autoclock RPC ansible playbook is to have you caught up on the Solana blockchain within 15 minutes, assuming you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For an in depth guide on RPC's refer to https://github.com/rpcpool/solana-rpc-ansible ### Optimal Machine Settings * Our Latitude.sh s3.large.x86 server starts with the settings below, which we prefer because: From bec05c7ed2b17da48e58608ceb0ceb29d47d54ec Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Fri, 4 Nov 2022 10:06:41 -0500 Subject: [PATCH 15/35] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e1642b6..8ce5481 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Autoclock RPC ### What is it good for? -The goal of the Autoclock RPC ansible playbook is to have you caught up on the Solana blockchain within 15 minutes, assuming you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For an in depth guide on RPC's refer to https://github.com/rpcpool/solana-rpc-ansible +The goal of the Autoclock RPC ansible playbook is to have you caught up on the Solana blockchain within 15 minutes, assuming you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For a more catch-all ansible playbook and in depth guide on RPC's refer to https://github.com/rpcpool/solana-rpc-ansible ### Optimal Machine Settings * Our Latitude.sh s3.large.x86 server starts with the settings below, which we prefer because: From 69816ddb482681d68a867bd29ff213fcf627552b Mon Sep 17 00:00:00 2001 From: Overclock Validator <116830051+overclock-validator@users.noreply.github.com> Date: Sat, 5 Nov 2022 12:08:34 -0500 Subject: [PATCH 16/35] Update snapshot-finder.py --- snapshot-finder.py | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/snapshot-finder.py b/snapshot-finder.py index 3bbd0aa..6a76e5e 100644 --- a/snapshot-finder.py +++ b/snapshot-finder.py @@ -28,6 +28,7 @@ parser.add_argument('--max_download_speed', type=int, help='Maximum snapshot download speed in megabytes - https://github.com/c29r3/solana-snapshot-finder/issues/11. Example: --max_download_speed 192') parser.add_argument('--max_latency', default=40, type=int, help='The maximum value of latency (milliseconds). If latency > max_latency --> skip') +parser.add_argument('--version', type=str, help='version of the snapshot required') parser.add_argument('--with_private_rpc', action="store_true", help='Enable adding and checking RPCs with the --private-rpc option.This slow down checking and searching but potentially increases' ' the number of RPCs from which snapshots can be downloaded.') parser.add_argument('--measurement_time', default=7, type=int, help='Time in seconds during which the script will measure the download speed') @@ -51,6 +52,7 @@ MAX_DOWNLOAD_SPEED_MB = args.max_download_speed SPEED_MEASURE_TIME_SEC = args.measurement_time MAX_LATENCY = args.max_latency +VERSION = args.version SNAPSHOT_PATH = args.snapshot_path if args.snapshot_path[-1] != '/' else args.snapshot_path[:-1] NUM_OF_MAX_ATTEMPTS = args.num_of_retries SLEEP_BEFORE_RETRY = args.sleep @@ -106,6 +108,18 @@ def convert_size(size_bytes): return "%s %s" % (s, size_name[i]) +def version_check(url: str, version: str) -> bool: + data = {"jsonrpc":"2.0","id":1, "method":"getVersion"} + headers = {"Content-Type": "application/json"} + res = requests.post("http://%s"%url, json=data) + parsed_res = json.loads(res.text) + rpc_ver = parsed_res["result"]["solana-core"] + logger.info("version: %s"%rpc_ver) + if rpc_ver != version: + return False + return True + + def measure_speed(url: str, measure_time: int) -> float: logging.debug('measure_speed()') url = f'http://{url}/snapshot.tar.bz2' @@ -379,6 +393,7 @@ def main_worker(): rpc_nodes_inc_sorted = [] logger.info("TRYING TO DOWNLOADING FILES") for i, rpc_node in enumerate(json_data["rpc_nodes"], start=1): + logger.info("checking node: %s %s"%(i,rpc_node)) # filter blacklisted snapshots if BLACKLIST != ['']: if any(i in str(rpc_node["files_to_download"]) for i in BLACKLIST): @@ -389,7 +404,17 @@ def main_worker(): if rpc_node["snapshot_address"] in unsuitable_servers: logger.info(f'Rpc node already in unsuitable list --> skip {rpc_node["snapshot_address"]}') continue - + + logger.info("checking version") + try: + if not version_check(rpc_node["snapshot_address"], VERSION): + logger.info("version check failed") + continue + except: + import traceback + print(traceback.format_exc()) + print("broken") + logger.info("version check succeeded") down_speed_bytes = measure_speed(url=rpc_node["snapshot_address"], measure_time=SPEED_MEASURE_TIME_SEC) down_speed_mb = convert_size(down_speed_bytes) if down_speed_bytes < MIN_DOWNLOAD_SPEED_MB * 1e6: @@ -494,4 +519,4 @@ def main_worker(): sys.exit() logger.info(f"Sleeping {SLEEP_BEFORE_RETRY} seconds before next try") - time.sleep(SLEEP_BEFORE_RETRY) \ No newline at end of file + time.sleep(SLEEP_BEFORE_RETRY) From 13dbcca63cd9040bce4879ccdd38e2f78ca521a3 Mon Sep 17 00:00:00 2001 From: vovkman Date: Thu, 24 Nov 2022 14:44:34 -0800 Subject: [PATCH 17/35] Remove systuner and format README file --- README.md | 65 ++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 8ce5481..2a32d3b 100644 --- a/README.md +++ b/README.md @@ -1,67 +1,79 @@ # Autoclock RPC ### What is it good for? + The goal of the Autoclock RPC ansible playbook is to have you caught up on the Solana blockchain within 15 minutes, assuming you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For a more catch-all ansible playbook and in depth guide on RPC's refer to https://github.com/rpcpool/solana-rpc-ansible ### Optimal Machine Settings -* Our Latitude.sh s3.large.x86 server starts with the settings below, which we prefer because: - * the initial state of the machine is cleaner than others that we have tried - * disks are named consistently (nvme01, nvme0n2) - * ubuntu installed (preferably ubuntu 20.04, 22.04) - this won't work with centos, etc. since they don't use aptitude by default - * the login user being ubuntu helps (all the solana operations are done using the solana user that the ansible playbook creates) - * ubuntu is in the sudoer's list - * unmounted disks are clean - if your root is on one of partitions and you pass it as an argument, this could be disastrous - -* All the above are satisfied by a fresh s3.large.x86 launch found here: https://www.latitude.sh/pricing -* Zen3 AMD Epyc’s such as the 7443p are considered some of the most performant nodes for keeping up with the tip of the chain at the moment, and support large amounts of RAM. - -* Recommended RPC Specs - * 24 cores or more - * 512 GB RAM if you want to use ramdisk/tmpfs and store the accounts db in RAM (we use 300 GB for ram disk). without tmpfs, the ram requirement can be significantly lower (~256 GB) - * 3-4 TB (multiple disks is okay - i.e. 2x 1.9TB - because the ansible playbook stripes them together) + +- Our Latitude.sh s3.large.x86 server starts with the settings below, which we prefer because: + + - the initial state of the machine is cleaner than others that we have tried + - disks are named consistently (nvme01, nvme0n2) + - ubuntu installed (preferably ubuntu 20.04, 22.04) - this won't work with centos, etc. since they don't use aptitude by default + - the login user being ubuntu helps (all the solana operations are done using the solana user that the ansible playbook creates) + - ubuntu is in the sudoer's list + - unmounted disks are clean - if your root is on one of partitions and you pass it as an argument, this could be disastrous + +- All the above are satisfied by a fresh s3.large.x86 launch found here: https://www.latitude.sh/pricing +- Zen3 AMD Epyc’s such as the 7443p are considered some of the most performant nodes for keeping up with the tip of the chain at the moment, and support large amounts of RAM. + +- Recommended RPC Specs + - 24 cores or more + - 512 GB RAM if you want to use ramdisk/tmpfs and store the accounts db in RAM (we use 300 GB for ram disk). without tmpfs, the ram requirement can be significantly lower (~256 GB) + - 3-4 TB (multiple disks is okay - i.e. 2x 1.9TB - because the ansible playbook stripes them together) ### Step 1: SSH into your machine - ### Step 2: Start a screen session + ``` screen -S sol ``` ### Step 3: Install ansible + ``` sudo apt-get install ansible -y ``` ### Step 4: Clone the autoclock-rpc repository + ``` git clone https://github.com/overclock-validator/autoclock-rpc.git ``` ### Step 5: cd into the autoclock-rpc folder + ``` cd autoclock-rpc ``` ### Step 6: Run the ansible command -* this command can take between 10-20 minutes based on the specs of the machine -* it takes long because it does everything necessary to start the validator (format disks, checkout the solana repo and build it, download the latest snapshot, etc.) -* make sure that the solana_version is up to date (see below) + +- this command can take between 10-20 minutes based on the specs of the machine +- it takes long because it does everything necessary to start the validator (format disks, checkout the solana repo and build it, download the latest snapshot, etc.) +- make sure that the solana_version is up to date (see below) + ``` time ansible-playbook runner.yaml --extra-vars='{"solana_version": "v1.13.4", "swap_mb":100000,"raw_disk_list":["/dev/nvme0n1","/dev/nvme1n1"],"setup_disks":true,"download_snapshot":true,"ramdisk_size":300}' ``` #### ~ Parameters explained ~ -* solana_version: the version of solana that we want to run. Check the Solana Tech discord’s mb-announcements channel for the recommended version. -* swap_mb: megabytes of swap. This can be set this to 50% of RAM or even lower. 100 GB is fine on a 512 GB RAM machine (variable value is in MB so 100000) -* raw_disk_list: the list of currently unmounted disks that will be wiped, raided, formatted with ext4 and then mounted to /mnt -* ramdisk_size: this is optional and only necessary if you want to use ramdisk for the validator - carves out a large portion of the RAM to store the accountsdb. On a 512 GB RAM instance, this can be set to 300 GB (variable value is in GB so 300) + +- solana_version: the version of solana that we want to run. Check the Solana Tech discord’s mb-announcements channel for the recommended version. +- swap_mb: megabytes of swap. This can be set this to 50% of RAM or even lower. 100 GB is fine on a 512 GB RAM machine (variable value is in MB so 100000) +- raw_disk_list: the list of currently unmounted disks that will be wiped, raided, formatted with ext4 and then mounted to /mnt +- ramdisk_size: this is optional and only necessary if you want to use ramdisk for the validator - carves out a large portion of the RAM to store the accountsdb. On a 512 GB RAM instance, this can be set to 300 GB (variable value is in GB so 300) ### Step 7: Once ansible finishes, switch to the solana user with: + ``` sudo su - solana ``` + ### Step 8: Check the status + ``` /mnt/solana/target/release/solana-validator --ledger /mnt/solana-ledger monitor ledger monitor @@ -69,19 +81,22 @@ Ledger location: /mnt/solana-ledger ⠉ Validator startup: SearchingForRpcService... ``` -#### Initially the monitor should just show the below message which will last for a few minutes and is normal: +#### Initially the monitor should just show the below message which will last for a few minutes and is normal: + ``` ⠉ Validator startup: SearchingForRpcService... ``` + #### After a while, the message at the terminal should change to something similar to this: + ``` ⠐ 00:08:26 | Processed Slot: 156831951 | Confirmed Slot: 156831951 | Finalized Slot: 156831917 | Full Snapshot Slot: 156813730 | ``` #### Check whether the RPC is caught up with the rest of the cluster with: + ``` solana catchup --our-localhost ``` If you see the message above, then everything is working fine! Gratz. You have a new RPC server and you can visit the URL at http://xx.xx.xx.xx:8899/ - From 2eeff57fb05fb146e7c43543cc4fdb223a1b21b7 Mon Sep 17 00:00:00 2001 From: vovkman Date: Sat, 26 Nov 2022 11:06:07 -0800 Subject: [PATCH 18/35] update tasks to work properly --- README.md | 2 +- defaults/main.yml | 10 +- files/restart.sh | 9 - files/snapcheck.py | 36 --- files/snapshot-finder.py | 520 --------------------------------- files/sol.service | 18 -- montip.py | 41 --- tasks/deps.yaml | 65 ++--- tasks/dirs.yaml | 2 +- tasks/file_setup.yaml | 8 +- tasks/git.yaml | 4 +- tasks/keygen.yaml | 2 +- tasks/main.yaml | 4 + tasks/snapshot_downloader.yaml | 2 +- templates/validator.j2 | 35 ++- test_runner.yaml | 14 +- 16 files changed, 88 insertions(+), 684 deletions(-) delete mode 100644 files/restart.sh delete mode 100644 files/snapcheck.py delete mode 100644 files/snapshot-finder.py delete mode 100644 files/sol.service delete mode 100644 montip.py diff --git a/README.md b/README.md index 2a32d3b..512caee 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ cd autoclock-rpc - make sure that the solana_version is up to date (see below) ``` -time ansible-playbook runner.yaml --extra-vars='{"solana_version": "v1.13.4", "swap_mb":100000,"raw_disk_list":["/dev/nvme0n1","/dev/nvme1n1"],"setup_disks":true,"download_snapshot":true,"ramdisk_size":300}' +time ansible-playbook tasks/main.yaml --extra-vars='{"solana_version": "v1.13.4", "swap_mb":100000,"raw_disk_list":["/dev/nvme0n1","/dev/nvme1n1"],"setup_disks":true,"download_snapshot":true,"ramdisk_size":300}' ``` #### ~ Parameters explained ~ diff --git a/defaults/main.yml b/defaults/main.yml index 54a1093..789e117 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -1,8 +1,10 @@ --- # defaults file for Solana RPC -solana_version: "v1.13.4" -raw_disk_list: "/dev/nvme0n1","/dev/nvme1n1" -setup_disks: "true" +solana_version: "1.13.5" +raw_disk_list: ["/dev/nvme0n1", "/dev/nvme1n1"] +# setup_disks: "true" +setup_disks: "false" download_snapshot: "true" -ramdisk_size: "300" +# ramdisk_size: "300" swap_mb: "100000" +solana_installer: "true" diff --git a/files/restart.sh b/files/restart.sh deleted file mode 100644 index 5e139e3..0000000 --- a/files/restart.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/bin/bash -if [ $# -eq 0 ] - then - python3 /mnt/snapcheck.py -fi -sudo systemctl stop sol.service -sudo pkill -9 -f solana-sys-tuner -sudo $(command -v /mnt/solana/target/release/solana-sys-tuner) --user $(whoami) > sys-tuner.log 2>&1 & -sudo systemctl start sol.service diff --git a/files/snapcheck.py b/files/snapcheck.py deleted file mode 100644 index 086c549..0000000 --- a/files/snapcheck.py +++ /dev/null @@ -1,36 +0,0 @@ -import urllib.request -import threading -import json -import time - -LOCAL = "http://localhost:8899" -PAYLOAD = {"jsonrpc":"2.0","id":1,"method":"getHighestSnapshotSlot"} - -def get_snap_slot(req, jsondata): - response = urllib.request.urlopen(req, jsondata) - res = response.read() - slotnum = json.loads(res)["result"]["incremental"] - return slotnum - -if __name__ == "__main__": - jsondata = json.dumps(PAYLOAD) - jsondataasbytes = jsondata.encode('utf-8') - content_len = len(jsondataasbytes) - latest_snap = None - while True: - req = urllib.request.Request(LOCAL) - req.add_header('Content-Type', 'application/json; charset=utf-8') - req.add_header('Content-Length', content_len) - req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36') - if latest_snap is None: - latest_snap = get_snap_slot(req,jsondataasbytes) - print("current incr snapshot slot: %s\nsleeping..."%(latest_snap)) - stime = int(time.time()) - time.sleep(1) - latest_incr = get_snap_slot(req,jsondataasbytes) - if latest_incr == latest_snap: - time.sleep(1) - else: - etime = int(time.time()) - stime - break -print("latest incr snapshot slot: %s\nslept: %s seconds"%(latest_incr,etime)) diff --git a/files/snapshot-finder.py b/files/snapshot-finder.py deleted file mode 100644 index 29bc0b0..0000000 --- a/files/snapshot-finder.py +++ /dev/null @@ -1,520 +0,0 @@ -from distutils.log import debug -import os -import glob -import requests -import time -import math -import json -import sys -import argparse -import logging -import subprocess -from pathlib import Path -from requests import ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError -from tqdm import tqdm -from multiprocessing.dummy import Pool as ThreadPool -import statistics - -parser = argparse.ArgumentParser(description='Solana snapshot finder') -parser.add_argument('-t', '--threads-count', default=1000, type=int, - help='the number of concurrently running threads that check snapshots for rpc nodes') -parser.add_argument('-r', '--rpc_address', - default='https://api.mainnet-beta.solana.com', type=str, - help='RPC address of the node from which the current slot number will be taken\n' - 'https://api.mainnet-beta.solana.com') - -parser.add_argument('--max_snapshot_age', default=1300, type=int, help='How many slots ago the snapshot was created (in slots)') -parser.add_argument('--min_download_speed', default=60, type=int, help='Minimum average snapshot download speed in megabytes') -parser.add_argument('--max_download_speed', type=int, -help='Maximum snapshot download speed in megabytes - https://github.com/c29r3/solana-snapshot-finder/issues/11. Example: --max_download_speed 192') -parser.add_argument('--max_latency', default=40, type=int, help='The maximum value of latency (milliseconds). If latency > max_latency --> skip') -parser.add_argument('--version', type=str, help='version of the snapshot required') -parser.add_argument('--with_private_rpc', action="store_true", help='Enable adding and checking RPCs with the --private-rpc option.This slow down checking and searching but potentially increases' - ' the number of RPCs from which snapshots can be downloaded.') -parser.add_argument('--measurement_time', default=7, type=int, help='Time in seconds during which the script will measure the download speed') -parser.add_argument('--snapshot_path', type=str, default=".", help='The location where the snapshot will be downloaded (absolute path).' - ' Example: /home/ubuntu/solana/validator-ledger') -parser.add_argument('--num_of_retries', default=5, type=int, help='The number of retries if a suitable server for downloading the snapshot was not found') -parser.add_argument('--sleep', default=30, type=int, help='Sleep before next retry (seconds)') -parser.add_argument('--sort_order', default='slots_diff', type=str, help='Priority way to sort the found servers. latency or slots_diff') -parser.add_argument('-b', '--blacklist', default='', type=str, help='If the same corrupted archive is constantly downloaded, you can exclude it.' - ' Specify either the number of the slot you want to exclude, or the hash of the archive name. ' - 'You can specify several, separated by commas. Example: -b 135501350,135501360 or --blacklist 135501350,some_hash') -parser.add_argument("-v", "--verbose", help="increase output verbosity to DEBUG", action="store_true") -args = parser.parse_args() - -DEFAULT_HEADERS = {"Content-Type": "application/json"} -RPC = args.rpc_address -WITH_PRIVATE_RPC = args.with_private_rpc -MAX_SNAPSHOT_AGE_IN_SLOTS = args.max_snapshot_age -THREADS_COUNT = args.threads_count -MIN_DOWNLOAD_SPEED_MB = args.min_download_speed -MAX_DOWNLOAD_SPEED_MB = args.max_download_speed -SPEED_MEASURE_TIME_SEC = args.measurement_time -MAX_LATENCY = args.max_latency -VERSION = args.version -SNAPSHOT_PATH = args.snapshot_path if args.snapshot_path[-1] != '/' else args.snapshot_path[:-1] -NUM_OF_MAX_ATTEMPTS = args.num_of_retries -SLEEP_BEFORE_RETRY = args.sleep -NUM_OF_ATTEMPTS = 1 -SORT_ORDER = args.sort_order -BLACKLIST = str(args.blacklist).split(",") -AVERAGE_SNAPSHOT_FILE_SIZE_MB = 2500.0 -AVERAGE_INCREMENT_FILE_SIZE_MB = 200.0 -AVERAGE_CATCHUP_SPEED = 2.0 -FULL_LOCAL_SNAP_SLOT = 0 - -current_slot = 0 -DISCARDED_BY_ARCHIVE_TYPE = 0 -DISCARDED_BY_LATENCY = 0 -DISCARDED_BY_SLOT = 0 -DISCARDED_BY_UNKNW_ERR = 0 -DISCARDED_BY_TIMEOUT = 0 -FULL_LOCAL_SNAPSHOTS = [] -# skip servers that do not fit the filters so as not to check them again -unsuitable_servers = set() -# Configure Logging -logging.getLogger('urllib3').setLevel(logging.WARNING) -if args.verbose: - logging.basicConfig( - level=logging.DEBUG, - format="%(asctime)s [%(levelname)s] %(message)s", - handlers=[ - logging.FileHandler(f'{SNAPSHOT_PATH}/snapshot-finder.log'), - logging.StreamHandler(sys.stdout), - ] - ) - -else: - # logging.basicConfig(stream=sys.stdout, encoding='utf-8', level=logging.INFO, format='|%(asctime)s| %(message)s') - logging.basicConfig( - level=logging.INFO, - format="%(asctime)s [%(levelname)s] %(message)s", - handlers=[ - logging.FileHandler(f'{SNAPSHOT_PATH}/snapshot-finder.log'), - logging.StreamHandler(sys.stdout), - ] - ) -logger = logging.getLogger(__name__) - - -def convert_size(size_bytes): - if size_bytes == 0: - return "0B" - size_name = ("B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB") - i = int(math.floor(math.log(size_bytes, 1024))) - p = math.pow(1024, i) - s = round(size_bytes / p, 2) - return "%s %s" % (s, size_name[i]) - - -def version_check(url: str, version: str) -> bool: - data = {"jsonrpc":"2.0","id":1, "method":"getVersion"} - headers = {"Content-Type": "application/json"} - res = requests.post("http://%s"%url, json=data) - parsed_res = json.loads(res.text) - rpc_ver = parsed_res["result"]["solana-core"] - logger.info("version: %s"%rpc_ver) - if rpc_ver != version: - return False - return True - -def measure_speed(url: str, measure_time: int) -> float: - logging.debug('measure_speed()') - url = f'http://{url}/snapshot.tar.bz2' - r = requests.get(url, stream=True, timeout=measure_time+2) - r.raise_for_status() - start_time = time.monotonic_ns() - last_time = start_time - loaded = 0 - speeds = [] - for chunk in r.iter_content(chunk_size=81920): - curtime = time.monotonic_ns() - - worktime = (curtime - start_time) / 1000000000 - if worktime >= measure_time: - break - - delta = (curtime - last_time) / 1000000000 - loaded += len(chunk) - if delta > 1: - estimated_bytes_per_second = loaded * (1 / delta) - # print(f'{len(chunk)}:{delta} : {estimated_bytes_per_second}') - speeds.append(estimated_bytes_per_second) - - last_time = curtime - loaded = 0 - - return statistics.median(speeds) - - -def do_request(url_: str, method_: str = 'GET', data_: str = '', timeout_: int = 3, - headers_: dict = None): - global DISCARDED_BY_UNKNW_ERR - global DISCARDED_BY_TIMEOUT - r = '' - if headers_ is None: - headers_ = DEFAULT_HEADERS - - try: - if method_.lower() == 'get': - r = requests.get(url_, headers=headers_, timeout=(timeout_, timeout_)) - elif method_.lower() == 'post': - r = requests.post(url_, headers=headers_, data=data_, timeout=(timeout_, timeout_)) - elif method_.lower() == 'head': - r = requests.head(url_, headers=headers_, timeout=(timeout_, timeout_)) - # print(f'{r.content, r.status_code, r.text}') - return r - - except (ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError) as reqErr: - # logger.debug(f'error in do_request(): {reqErr=}') - DISCARDED_BY_TIMEOUT += 1 - return f'error in do_request(): {reqErr}' - - except Exception as unknwErr: - DISCARDED_BY_UNKNW_ERR += 1 - # logger.debug(f'error in do_request(): {unknwErr=}') - return f'error in do_request(): {reqErr}' - - -def get_current_slot(): - logger.debug("get_current_slot()") - d = '{"jsonrpc":"2.0","id":1, "method":"getSlot"}' - try: - r = do_request(url_=RPC, method_='post', data_=d, timeout_=25) - if 'result' in str(r.text): - return r.json()["result"] - else: - logger.error(f'Can\'t get current slot') - return None - except: - logger.error(f'Can\'t get current slot') - return None - - -def get_all_rpc_ips(): - logger.debug("get_all_rpc_ips()") - d = '{"jsonrpc":"2.0", "id":1, "method":"getClusterNodes"}' - r = do_request(url_=RPC, method_='post', data_=d, timeout_=25) - if 'result' in str(r.text): - if WITH_PRIVATE_RPC is True: - rpc_ips = [] - for node in r.json()["result"]: - if node["rpc"] is not None: - rpc_ips.append(node["rpc"]) - else: - gossip_ip = node["gossip"].split(":")[0] - rpc_ips.append(f'{gossip_ip}:8899') - - else: - rpc_ips = [rpc["rpc"] for rpc in r.json()["result"] if rpc["rpc"] is not None] - - rpc_ips = list(set(rpc_ips)) - return rpc_ips - - else: - logger.error(f'Can\'t get RPC ip addresses {r.text}') - sys.exit() - - -def get_snapshot_slot(rpc_address: str): - global FULL_LOCAL_SNAP_SLOT - global DISCARDED_BY_ARCHIVE_TYPE - global DISCARDED_BY_LATENCY - global DISCARDED_BY_SLOT - - pbar.update(1) - url = f'http://{rpc_address}/snapshot.tar.bz2' - inc_url = f'http://{rpc_address}/incremental-snapshot.tar.bz2' - # d = '{"jsonrpc":"2.0","id":1,"method":"getHighestSnapshotSlot"}' - try: - r = do_request(url_=inc_url, method_='head', timeout_=1) - if 'location' in str(r.headers) and 'error' not in str(r.text) and r.elapsed.total_seconds() * 1000 > MAX_LATENCY: - DISCARDED_BY_LATENCY += 1 - return None - - - if 'location' in str(r.headers) and 'error' not in str(r.text): - snap_location_ = r.headers["location"] - if snap_location_.endswith('tar') is True: - DISCARDED_BY_ARCHIVE_TYPE += 1 - return None - incremental_snap_slot = int(snap_location_.split("-")[2]) - snap_slot_ = int(snap_location_.split("-")[3]) - slots_diff = current_slot - snap_slot_ - - if slots_diff < -100: - logger.error(f'Something wrong with this snapshot\\rpc_node - {slots_diff=}. This node will be skipped {rpc_address=}') - DISCARDED_BY_SLOT += 1 - return - - if slots_diff > MAX_SNAPSHOT_AGE_IN_SLOTS: - DISCARDED_BY_SLOT += 1 - return - - if FULL_LOCAL_SNAP_SLOT == incremental_snap_slot: - json_data["rpc_nodes"].append({ - "snapshot_address": rpc_address, - "slots_diff": slots_diff, - "latency": r.elapsed.total_seconds() * 1000, - "files_to_download": [snap_location_], - "cost": AVERAGE_INCREMENT_FILE_SIZE_MB / MIN_DOWNLOAD_SPEED_MB + slots_diff / AVERAGE_CATCHUP_SPEED - }) - return - - r2 = do_request(url_=url, method_='head', timeout_=1) - if 'location' in str(r.headers) and 'error' not in str(r.text): - json_data["rpc_nodes"].append({ - "snapshot_address": rpc_address, - "slots_diff": slots_diff, - "latency": r.elapsed.total_seconds() * 1000, - "files_to_download": [r.headers["location"], r2.headers['location']], - "cost": (AVERAGE_SNAPSHOT_FILE_SIZE_MB + AVERAGE_INCREMENT_FILE_SIZE_MB) / MIN_DOWNLOAD_SPEED_MB + slots_diff / AVERAGE_CATCHUP_SPEED - }) - return - - r = do_request(url_=url, method_='head', timeout_=1) - if 'location' in str(r.headers) and 'error' not in str(r.text): - snap_location_ = r.headers["location"] - # filtering uncompressed archives - if snap_location_.endswith('tar') is True: - DISCARDED_BY_ARCHIVE_TYPE += 1 - return None - full_snap_slot_ = int(snap_location_.split("-")[1]) - slots_diff_full = current_slot - full_snap_slot_ - if slots_diff_full <= MAX_SNAPSHOT_AGE_IN_SLOTS and r.elapsed.total_seconds() * 1000 <= MAX_LATENCY: - # print(f'{rpc_address=} | {slots_diff=}') - json_data["rpc_nodes"].append({ - "snapshot_address": rpc_address, - "slots_diff": slots_diff_full, - "latency": r.elapsed.total_seconds() * 1000, - "files_to_download": [snap_location_], - "cost": AVERAGE_SNAPSHOT_FILE_SIZE_MB / MIN_DOWNLOAD_SPEED_MB + slots_diff_full / AVERAGE_CATCHUP_SPEED - }) - return - return None - - except Exception as getSnapErr_: - return None - - -def download(url: str): - fname = url[url.rfind('/'):].replace("/", "") - temp_fname = f'{SNAPSHOT_PATH}/tmp-{fname}' - # try: - # resp = requests.get(url, stream=True) - # total = int(resp.headers.get('content-length', 0)) - # with open(temp_fname, 'wb') as file, tqdm( - # desc=fname, - # total=total, - # unit='iB', - # unit_scale=True, - # unit_divisor=1024, - # ) as bar: - # for data in resp.iter_content(chunk_size=1024): - # size = file.write(data) - # bar.update(size) - - # logger.info(f'Rename the downloaded file {temp_fname} --> {fname}') - # os.rename(temp_fname, f'{SNAPSHOT_PATH}/{fname}') - - # except (ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError) as downlErr: - # logger.error(f'Exception in download() func\n {downlErr}') - - try: - # dirty trick with wget. Details here - https://github.com/c29r3/solana-snapshot-finder/issues/11 - if MAX_DOWNLOAD_SPEED_MB is not None: - process = subprocess.run(['/usr/bin/wget', f'--limit-rate={MAX_DOWNLOAD_SPEED_MB}M', '--trust-server-names', url, f'-O{temp_fname}'], - stdout=subprocess.PIPE, - universal_newlines=True) - else: - process = subprocess.run(['/usr/bin/wget', '--trust-server-names', url, f'-O{temp_fname}'], - stdout=subprocess.PIPE, - universal_newlines=True) - - logger.info(f'Rename the downloaded file {temp_fname} --> {fname}') - os.rename(temp_fname, f'{SNAPSHOT_PATH}/{fname}') - - except Exception as unknwErr: - logger.error(f'Exception in download() func. Make sure wget is installed\n{unknwErr}') - - -def main_worker(): - try: - global FULL_LOCAL_SNAP_SLOT - rpc_nodes = list(set(get_all_rpc_ips())) - global pbar - pbar = tqdm(total=len(rpc_nodes)) - logger.info(f'RPC servers in total: {len(rpc_nodes)} | Current slot number: {current_slot}\n') - - # Search for full local snapshots. - # If such a snapshot is found and it is not too old, then the script will try to find and download an incremental snapshot - FULL_LOCAL_SNAPSHOTS = glob.glob(f'{SNAPSHOT_PATH}/snapshot-*tar*') - if len(FULL_LOCAL_SNAPSHOTS) > 0: - FULL_LOCAL_SNAPSHOTS.sort(reverse=True) - FULL_LOCAL_SNAP_SLOT = FULL_LOCAL_SNAPSHOTS[0].replace(SNAPSHOT_PATH, "").split("-")[1] - logger.info(f'Found full local snapshot {FULL_LOCAL_SNAPSHOTS[0]} | {FULL_LOCAL_SNAP_SLOT=}') - - else: - logger.info(f'Can\'t find any full local snapshots in this path {SNAPSHOT_PATH} --> the search will be carried out on full snapshots') - - print(f'Searching information about snapshots on all found RPCs') - pool = ThreadPool() - pool.map(get_snapshot_slot, rpc_nodes) - logger.info(f'Found suitable RPCs: {len(json_data["rpc_nodes"])}') - logger.info(f'The following information shows for what reason and how many RPCs were skipped.' - f'Timeout most probably mean, that node RPC port does not respond (port is closed)\n' - f'{DISCARDED_BY_ARCHIVE_TYPE=} | {DISCARDED_BY_LATENCY=} |' - f' {DISCARDED_BY_SLOT=} | {DISCARDED_BY_TIMEOUT=} | {DISCARDED_BY_UNKNW_ERR=}') - - if len(json_data["rpc_nodes"]) == 0: - logger.info(f'No snapshot nodes were found matching the given parameters: {args.max_snapshot_age=}') - sys.exit() - - # sort list of rpc node by SORT_ORDER (latency) - rpc_nodes_sorted = sorted(json_data["rpc_nodes"], key=lambda k: k[SORT_ORDER]) - - json_data.update({ - "last_update_at": time.time(), - "last_update_slot": current_slot, - "total_rpc_nodes": len(rpc_nodes), - "rpc_nodes_with_actual_snapshot": len(json_data["rpc_nodes"]), - "rpc_nodes": rpc_nodes_sorted - }) - - with open(f'{SNAPSHOT_PATH}/snapshot.json', "w") as result_f: - json.dump(json_data, result_f, indent=2) - logger.info(f'All data is saved to json file - {SNAPSHOT_PATH}/snapshot.json') - - best_snapshot_node = {} - num_of_rpc_to_check = 15 - - rpc_nodes_inc_sorted = [] - logger.info("TRYING TO DOWNLOADING FILES") - for i, rpc_node in enumerate(json_data["rpc_nodes"], start=1): - logger.info("checking node: %s %s"%(i,rpc_node)) - # filter blacklisted snapshots - if BLACKLIST != ['']: - if any(i in str(rpc_node["files_to_download"]) for i in BLACKLIST): - logger.info(f'{i}\\{len(json_data["rpc_nodes"])} BLACKLISTED --> {rpc_node}') - continue - - logger.info(f'{i}\\{len(json_data["rpc_nodes"])} checking the speed {rpc_node}') - if rpc_node["snapshot_address"] in unsuitable_servers: - logger.info(f'Rpc node already in unsuitable list --> skip {rpc_node["snapshot_address"]}') - continue - logger.info("checking version") - try: - if not version_check(rpc_node["snapshot_address"], VERSION): - logger.info("version check failed") - continue - except: - import traceback - print(traceback.format_exc()) - print("broken") - logger.info("version check succeeded") - down_speed_bytes = measure_speed(url=rpc_node["snapshot_address"], measure_time=SPEED_MEASURE_TIME_SEC) - down_speed_mb = convert_size(down_speed_bytes) - if down_speed_bytes < MIN_DOWNLOAD_SPEED_MB * 1e6: - logger.info(f'Too slow: {rpc_node=} {down_speed_mb=}') - unsuitable_servers.add(rpc_node["snapshot_address"]) - continue - - elif down_speed_bytes >= MIN_DOWNLOAD_SPEED_MB * 1e6: - logger.info(f'Suitable snapshot server found: {rpc_node=} {down_speed_mb=}') - for path in reversed(rpc_node["files_to_download"]): - # do not download full snapshot if it already exists locally - if str(path).startswith("/snapshot-"): - full_snap_slot__ = path.split("-")[1] - if full_snap_slot__ == FULL_LOCAL_SNAP_SLOT: - continue - - - if 'incremental' in path: - r = do_request(f'http://{rpc_node["snapshot_address"]}/incremental-snapshot.tar.bz2', method_='head', timeout_=2) - if 'location' in str(r.headers) and 'error' not in str(r.text): - best_snapshot_node = f'http://{rpc_node["snapshot_address"]}{r.headers["location"]}' - else: - best_snapshot_node = f'http://{rpc_node["snapshot_address"]}{path}' - - else: - best_snapshot_node = f'http://{rpc_node["snapshot_address"]}{path}' - logger.info(f'Downloading {best_snapshot_node} snapshot to {SNAPSHOT_PATH}') - download(url=best_snapshot_node) - return 0 - - elif i > num_of_rpc_to_check: - logger.info(f'The limit on the number of RPC nodes from' - ' which we measure the speed has been reached {num_of_rpc_to_check=}\n') - break - - else: - logger.info(f'{down_speed_mb=} < {MIN_DOWNLOAD_SPEED_MB=}') - - if best_snapshot_node is {}: - logger.error(f'No snapshot nodes were found matching the given parameters:{args.min_download_speed=}' - f'\nTry restarting the script with --with_private_rpc' - f'RETRY #{NUM_OF_ATTEMPTS}\\{NUM_OF_MAX_ATTEMPTS}') - return 1 - - - - except KeyboardInterrupt: - sys.exit('\nKeyboardInterrupt - ctrl + c') - - except: - return 1 - - -logger.info("Version: 0.3.3") -logger.info("https://github.com/c29r3/solana-snapshot-finder\n\n") -logger.info(f'{RPC=}\n' - f'{MAX_SNAPSHOT_AGE_IN_SLOTS=}\n' - f'{MIN_DOWNLOAD_SPEED_MB=}\n' - f'{MAX_DOWNLOAD_SPEED_MB=}\n' - f'{SNAPSHOT_PATH=}\n' - f'{THREADS_COUNT=}\n' - f'{NUM_OF_MAX_ATTEMPTS=}\n' - f'{WITH_PRIVATE_RPC=}\n' - f'{SORT_ORDER=}') - -try: - f_ = open(f'{SNAPSHOT_PATH}/write_perm_test', 'w') - f_.close() - os.remove(f'{SNAPSHOT_PATH}/write_perm_test') -except IOError: - logger.error(f'\nCheck {SNAPSHOT_PATH=} and permissions') - Path(SNAPSHOT_PATH).mkdir(parents=True, exist_ok=True) - -json_data = ({"last_update_at": 0.0, - "last_update_slot": 0, - "total_rpc_nodes": 0, - "rpc_nodes_with_actual_snapshot": 0, - "rpc_nodes": [] - }) - - -while NUM_OF_ATTEMPTS <= NUM_OF_MAX_ATTEMPTS: - current_slot = get_current_slot() - logger.info(f'Attempt number: {NUM_OF_ATTEMPTS}. Total attempts: {NUM_OF_MAX_ATTEMPTS}') - NUM_OF_ATTEMPTS += 1 - - if current_slot is None: - continue - - worker_result = main_worker() - - if worker_result == 0: - logger.info("Done") - exit(0) - - if worker_result != 0: - logger.info("Now trying with flag --with_private_rpc") - WITH_PRIVATE_RPC = True - - if NUM_OF_ATTEMPTS >= NUM_OF_MAX_ATTEMPTS: - logger.error(f'Could not find a suitable snapshot --> exit') - sys.exit() - - logger.info(f"Sleeping {SLEEP_BEFORE_RETRY} seconds before next try") - time.sleep(SLEEP_BEFORE_RETRY) diff --git a/files/sol.service b/files/sol.service deleted file mode 100644 index b583333..0000000 --- a/files/sol.service +++ /dev/null @@ -1,18 +0,0 @@ -[Unit] -Description=Solana Validator -After=network.target -StartLimitIntervalSec=0 - -[Service] -Type=simple -Restart=always -RestartSec=1 -User=solana -LimitNOFILE=1000000 -LogRateLimitIntervalSec=0 -Environment="PATH=/mnt/solana/target/release/:/mnt/solana/target/release/:/home/solana/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/snap/bin:/home/solana/.local/bin/:/home/solana/.local/bin/:/home/solana/.local/bin/" -ExecStart=/home/solana/validator.sh - -[Install] -WantedBy=multi-user.target - diff --git a/montip.py b/montip.py deleted file mode 100644 index 00b0227..0000000 --- a/montip.py +++ /dev/null @@ -1,41 +0,0 @@ -import urllib.request -import threading -import json -import time - -GENESYSGO = "https://ssc-dao.genesysgo.net/" -LOCAL = "http://localhost:8899" -PAYLOAD = {"jsonrpc":"2.0","id":1, "method":"getSlot", "params":[{"commitment":"processed"}]} - -def get_slot(req, jsondata, result, idx): - response = urllib.request.urlopen(req, jsondata) - res = response.read() - slotnum = json.loads(res)["result"] - result[idx] = slotnum - - -if __name__ == "__main__": - jsondata = json.dumps(PAYLOAD) - jsondataasbytes = jsondata.encode('utf-8') - content_len = len(jsondataasbytes) - while True: - tlist = [] - reqlist = [] - resultlist = [0,0] - for c,u in enumerate([LOCAL, GENESYSGO]): - req = urllib.request.Request(u) - req.add_header('Content-Type', 'application/json; charset=utf-8') - req.add_header('Content-Length', content_len) - req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36') - reqlist.append((req, jsondataasbytes, resultlist, c)) - for r in reqlist: - tlist.append(threading.Thread(target = get_slot, args = r)) - for t in tlist: - t.start() - for t in tlist: - t.join() - print("reference : %s"%(resultlist[1])) - print("my node : %s"%(resultlist[0])) - print("slots behind reference : %s"%(resultlist[1] - resultlist[0])) - print("=====\n") - time.sleep(10) diff --git a/tasks/deps.yaml b/tasks/deps.yaml index 0802707..d876f43 100644 --- a/tasks/deps.yaml +++ b/tasks/deps.yaml @@ -4,49 +4,40 @@ apt: update_cache: yes pkg: - - build-essential - - pkg-config - - libudev-dev - - cmake - - libclang-dev - - libssl-dev - - gparted - - nload - - python3-pip - - net-tools - - logind - - curl - - git - - chrony - - htop - - acl - -- name: check if cargo is installed + - build-essential + - pkg-config + - libudev-dev + - cmake + - libclang-dev + - libssl-dev + - gparted + - nload + - python3-pip + - net-tools + - logind + - curl + - git + - chrony + - htop + - acl + - cpufrequtils + +- name: Upgrade packages become: true - become_user: solana - become_method: sudo - shell: test -f /home/solana/.cargo/env && source /home/solana/.cargo/env && command -v cargo - args: - executable: /bin/bash - register: cargo_exists - ignore_errors: yes + become_user: root + apt: + upgrade: dist -- name: Download Installer - when: cargo_exists is failed +- name: install solana become: true become_user: solana - become_method: sudo - get_url: - url: https://sh.rustup.rs - dest: /tmp/sh.rustup.rs - mode: '0755' - force: 'yes' + shell: sh -c "$(curl -sSfL https://release.solana.com/{{ solana_version }}/install)" + args: + chdir: /home/solana -- name: install rust/cargo +- name: add solana deps to path become: true become_user: solana - become_method: sudo - when: cargo_exists is failed - shell: /tmp/sh.rustup.rs -y + shell: export PATH="/home/solana/.local/share/solana/install/active_release/bin:$PATH" args: executable: /bin/bash diff --git a/tasks/dirs.yaml b/tasks/dirs.yaml index 1bf7689..fe0f159 100644 --- a/tasks/dirs.yaml +++ b/tasks/dirs.yaml @@ -4,7 +4,7 @@ state: directory owner: solana group: solana - mode: '0777' + mode: "0777" become: true become_user: root diff --git a/tasks/file_setup.yaml b/tasks/file_setup.yaml index 0a05794..5065010 100644 --- a/tasks/file_setup.yaml +++ b/tasks/file_setup.yaml @@ -6,7 +6,7 @@ dest: /home/solana/validator.sh owner: solana group: solana - mode: '0755' + mode: "0755" - name: setup sol.service become: true @@ -16,17 +16,17 @@ dest: /etc/systemd/system/sol.service owner: root group: root - mode: '0755' + mode: "0755" - name: copy restart.sh become: true become_user: solana template: - src: restart.sh + src: restart.sh dest: /home/solana/restart.sh owner: solana group: solana - mode: '0755' + mode: "0755" - name: copy snapcheck.py become: true diff --git a/tasks/git.yaml b/tasks/git.yaml index 3c921e3..ac5c10f 100644 --- a/tasks/git.yaml +++ b/tasks/git.yaml @@ -2,9 +2,9 @@ become: true become_user: solana git: - repo: 'https://github.com/solana-labs/solana.git' + repo: "https://github.com/solana-labs/solana.git" dest: /mnt/solana - version: tags/{{ solana_version }} + version: tags/v{{ solana_version }} - name: build solana become: true diff --git a/tasks/keygen.yaml b/tasks/keygen.yaml index d13ebf1..2af4bac 100644 --- a/tasks/keygen.yaml +++ b/tasks/keygen.yaml @@ -1,6 +1,6 @@ - name: create identity key become: true become_user: solana - shell: /mnt/solana/target/release/solana-keygen new --no-bip39-passphrase -o /home/solana/rpc_node.json + shell: /home/solana/.local/share/solana/install/active_release/bin/solana-keygen new --no-bip39-passphrase -o /home/solana/rpc_node.json args: creates: /home/solana/rpc_node.json diff --git a/tasks/main.yaml b/tasks/main.yaml index 3977ff9..6904b22 100644 --- a/tasks/main.yaml +++ b/tasks/main.yaml @@ -28,6 +28,7 @@ - name: git include_tasks: git.yaml + when: solana_installer|default(false)|bool == false - name: solana keygen include_tasks: keygen.yaml @@ -35,6 +36,9 @@ - name: file setup include_tasks: file_setup.yaml +- name: tune host + include_tasks: tuner.yaml + - name: snapshot download include_tasks: snapshot_downloader.yaml when: download_snapshot|default(true)|bool == true diff --git a/tasks/snapshot_downloader.yaml b/tasks/snapshot_downloader.yaml index b43199e..037a51c 100644 --- a/tasks/snapshot_downloader.yaml +++ b/tasks/snapshot_downloader.yaml @@ -10,5 +10,5 @@ - name: download latest snapshot become: true become_user: solana - shell: python3 /mnt/snapshot-finder.py --snapshot_path /mnt/solana-snapshots --max_latency 1000 + shell: python3 /mnt/snapshot-finder.py --snapshot_path /mnt/solana-snapshots --version {{ solana_version }} diff --git a/templates/validator.j2 b/templates/validator.j2 index e1c1c0d..1cae8eb 100644 --- a/templates/validator.j2 +++ b/templates/validator.j2 @@ -1,2 +1,35 @@ #!/bin/bash -/mnt/solana/target/release/solana-validator --identity /home/solana/rpc_node.json --rpc-port 8899 --full-rpc-api --entrypoint entrypoint.mainnet-beta.solana.com:8001 --limit-ledger-size --log /mnt/logs/solana-validator.log --accounts /mnt/solana-accounts --ledger /mnt/solana-ledger --snapshots /mnt/solana-snapshots --no-snapshot-fetch +PATH=/home/solana/.local/share/solana/install/active_release/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin +# export RUST_BACKTRACE=1 +# export RUST_LOG=error + +# Start solana rpc node, intended to be used with journalctl/systemctl. Logs will go to journalctl. +exec solana-validator \ + --identity /home/solana/rpc_node.json \ + --entrypoint entrypoint.mainnet-beta.solana.com:8001 \ + --entrypoint entrypoint2.mainnet-beta.solana.com:8001 \ + --entrypoint entrypoint3.mainnet-beta.solana.com:8001 \ + --entrypoint entrypoint4.mainnet-beta.solana.com:8001 \ + --entrypoint entrypoint5.mainnet-beta.solana.com:8001 \ + --accounts /mnt/solana-accounts \ + --ledger /mnt/solana-ledger \ + --snapshots /mnt/solana-snapshots \ + --log /mnt/logs/solana-validator.log \ + --gossip-port 8001 \ + --rpc-port 8899 \ + --rpc-bind-address 0.0.0.0 \ + --dynamic-port-range 8002-8099 \ + --wal-recovery-mode skip_any_corrupted_record \ + --limit-ledger-size 50000000 \ + --private-rpc \ + --no-snapshot-fetch \ + --no-port-check \ + --no-voting \ + --full-rpc-api \ + --tpu-use-quic \ + --enable-cpi-and-log-storage \ + --enable-rpc-transaction-history \ + --account-index program-id spl-token-mint spl-token-owner \ + --accounts-index-memory-limit-mb 800000 \ + --account-index-exclude-key TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA \ + --account-index-exclude-key kinXdEcpDQeHPEuQnqmUgtYykqKGVFq6CeVX5iAHJq6 \ No newline at end of file diff --git a/test_runner.yaml b/test_runner.yaml index 3923bfd..987066b 100644 --- a/test_runner.yaml +++ b/test_runner.yaml @@ -1,9 +1,7 @@ --- - - name: "playbook runner" - hosts: localhost - connection: local - tasks: - - - name: var checks - ansible.builtin.import_tasks: var_check.yaml - +- name: "playbook runner" + hosts: localhost + connection: local + tasks: + - name: var checks + ansible.builtin.import_tasks: var_check.yaml From 3e2b300f2e3b5a4904af30fcfa8af38237262c45 Mon Sep 17 00:00:00 2001 From: vovkman Date: Sat, 26 Nov 2022 11:15:50 -0800 Subject: [PATCH 19/35] update readme --- README.md | 8 ++++++-- tasks/deps.yaml | 4 +++- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 512caee..8c2b756 100644 --- a/README.md +++ b/README.md @@ -54,9 +54,11 @@ cd autoclock-rpc - this command can take between 10-20 minutes based on the specs of the machine - it takes long because it does everything necessary to start the validator (format disks, checkout the solana repo and build it, download the latest snapshot, etc.) - make sure that the solana_version is up to date (see below) +- you should create a playbook file, you can look at `test_runner.yaml` for inspiration +- check the values set in `defaults/main.yml` and update to the values you want ``` -time ansible-playbook tasks/main.yaml --extra-vars='{"solana_version": "v1.13.4", "swap_mb":100000,"raw_disk_list":["/dev/nvme0n1","/dev/nvme1n1"],"setup_disks":true,"download_snapshot":true,"ramdisk_size":300}' +time ansible-playbook ``` #### ~ Parameters explained ~ @@ -65,6 +67,7 @@ time ansible-playbook tasks/main.yaml --extra-vars='{"solana_version": "v1.13.4" - swap_mb: megabytes of swap. This can be set this to 50% of RAM or even lower. 100 GB is fine on a 512 GB RAM machine (variable value is in MB so 100000) - raw_disk_list: the list of currently unmounted disks that will be wiped, raided, formatted with ext4 and then mounted to /mnt - ramdisk_size: this is optional and only necessary if you want to use ramdisk for the validator - carves out a large portion of the RAM to store the accountsdb. On a 512 GB RAM instance, this can be set to 300 GB (variable value is in GB so 300) +- solana_installer: whether to install solana from the installer. If set to false it will build solana cli from the solana github ### Step 7: Once ansible finishes, switch to the solana user with: @@ -75,7 +78,8 @@ sudo su - solana ### Step 8: Check the status ``` -/mnt/solana/target/release/solana-validator --ledger /mnt/solana-ledger monitor +source ~/.profile +solana-validator --ledger /mnt/solana-ledger monitor ledger monitor Ledger location: /mnt/solana-ledger ⠉ Validator startup: SearchingForRpcService... diff --git a/tasks/deps.yaml b/tasks/deps.yaml index d876f43..2c0e544 100644 --- a/tasks/deps.yaml +++ b/tasks/deps.yaml @@ -31,9 +31,10 @@ - name: install solana become: true become_user: solana - shell: sh -c "$(curl -sSfL https://release.solana.com/{{ solana_version }}/install)" + shell: sh -c "$(curl -sSfL https://release.solana.com/v{{ solana_version }}/install)" args: chdir: /home/solana + when: solana_installer|default(true)|bool == true - name: add solana deps to path become: true @@ -41,3 +42,4 @@ shell: export PATH="/home/solana/.local/share/solana/install/active_release/bin:$PATH" args: executable: /bin/bash + when: solana_installer|default(true)|bool == true From ba460edd986d724fc9e2785817c715879870ca16 Mon Sep 17 00:00:00 2001 From: vovkman Date: Sat, 26 Nov 2022 11:16:05 -0800 Subject: [PATCH 20/35] add new files --- runner.yaml | 6 + tasks/tuner.yaml | 68 +++++ templates/montip.py | 41 +++ templates/restart.sh | 7 + templates/snapcheck.py | 36 +++ templates/snapshot-finder.py | 520 +++++++++++++++++++++++++++++++++++ templates/sol.service | 18 ++ 7 files changed, 696 insertions(+) create mode 100644 runner.yaml create mode 100644 tasks/tuner.yaml create mode 100644 templates/montip.py create mode 100644 templates/restart.sh create mode 100644 templates/snapcheck.py create mode 100644 templates/snapshot-finder.py create mode 100644 templates/sol.service diff --git a/runner.yaml b/runner.yaml new file mode 100644 index 0000000..75dd414 --- /dev/null +++ b/runner.yaml @@ -0,0 +1,6 @@ +--- +- name: "playbook runner" + hosts: localhost + connection: local + roles: + - role: "./" diff --git a/tasks/tuner.yaml b/tasks/tuner.yaml new file mode 100644 index 0000000..748d4e9 --- /dev/null +++ b/tasks/tuner.yaml @@ -0,0 +1,68 @@ +- name: Set sysctl performance variables + become: true + become_user: root + shell: + # Note: this overwrite the file, if you want to append replace EOM with EOF + cmd: | + bash -c 'cat >> /etc/sysctl.conf <<- EOM + # set minimum, default, and maximum tcp buffer sizes (10k, 87.38k (linux default), 12M resp) + net.ipv4.tcp_rmem=10240 87380 12582912 + net.ipv4.tcp_wmem=10240 87380 12582912 + # Enable TCP westwood for kernels greater than or equal to 2.6.13 + net.ipv4.tcp_congestion_control=westwood + net.ipv4.tcp_fastopen=3 + net.ipv4.tcp_timestamps=0 + net.ipv4.tcp_sack=1 + net.ipv4.tcp_low_latency=1 + # Enable fast recycling TIME-WAIT sockets + # net.ipv4.tcp_tw_recycle = 1 this is in solana tuner, but fails, changing to below, see - https://djangocas.dev/blog/troubleshooting-tcp_tw_recycle-no-such-file-or-directory/ + net.ipv4.tcp_tw_reuse = 1 + # dont cache ssthresh from previous connection + net.ipv4.tcp_no_metrics_save = 1 + net.ipv4.tcp_moderate_rcvbuf = 1 + + # kernel Tunes + kernel.timer_migration=0 + kernel.hung_task_timeout_secs=30 + # A suggested value for pid_max is 1024 * <# of cpu cores/threads in system> + kernel.pid_max=49152 + + # vm.tuning + vm.swappiness=30 + vm.max_map_count=2000000 + vm.stat_interval=10 + vm.dirty_ratio=40 + vm.dirty_background_ratio=10 + vm.min_free_kbytes = 3000000 + vm.dirty_expire_centisecs=36000 + vm.dirty_writeback_centisecs=3000 + vm.dirtytime_expire_seconds=43200 + + # solana systuner + net.core.rmem_max=134217728 + net.core.rmem_default=134217728 + net.core.wmem_max=134217728 + net.core.wmem_default=134217728 + EOM' + args: + executable: /bin/bash + +- name: Reload sysctl + become: true + become_user: root + shell: sysctl -p + +- name: Set performance governor + become: true + become_user: root + shell: | + echo 'GOVERNOR="performance"' | tee /etc/default/cpufrequtils + systemctl disable ondemand + args: + executable: /bin/bash + +- name: Reboot host + become: true + become_user: root + reboot: + msg: Rebooting host... diff --git a/templates/montip.py b/templates/montip.py new file mode 100644 index 0000000..a557eb4 --- /dev/null +++ b/templates/montip.py @@ -0,0 +1,41 @@ +import urllib.request +import threading +import json +import time + +GENESYSGO = "https://api.mainnet-beta.solana.com" +LOCAL = "http://localhost:8899" +PAYLOAD = {"jsonrpc":"2.0","id":1, "method":"getSlot", "params":[{"commitment":"processed"}]} + +def get_slot(req, jsondata, result, idx): + response = urllib.request.urlopen(req, jsondata) + res = response.read() + slotnum = json.loads(res)["result"] + result[idx] = slotnum + + +if __name__ == "__main__": + jsondata = json.dumps(PAYLOAD) + jsondataasbytes = jsondata.encode('utf-8') + content_len = len(jsondataasbytes) + while True: + tlist = [] + reqlist = [] + resultlist = [0,0] + for c,u in enumerate([LOCAL, GENESYSGO]): + req = urllib.request.Request(u) + req.add_header('Content-Type', 'application/json; charset=utf-8') + req.add_header('Content-Length', content_len) + req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36') + reqlist.append((req, jsondataasbytes, resultlist, c)) + for r in reqlist: + tlist.append(threading.Thread(target = get_slot, args = r)) + for t in tlist: + t.start() + for t in tlist: + t.join() + print("reference : %s"%(resultlist[1])) + print("my node : %s"%(resultlist[0])) + print("slots behind reference : %s"%(resultlist[1] - resultlist[0])) + print("=====\n") + time.sleep(10) diff --git a/templates/restart.sh b/templates/restart.sh new file mode 100644 index 0000000..4be5120 --- /dev/null +++ b/templates/restart.sh @@ -0,0 +1,7 @@ +#!/bin/bash +if [ $# -eq 0 ] + then + python3 /mnt/snapcheck.py +fi +sudo systemctl stop sol.service +sudo systemctl start sol.service diff --git a/templates/snapcheck.py b/templates/snapcheck.py new file mode 100644 index 0000000..086c549 --- /dev/null +++ b/templates/snapcheck.py @@ -0,0 +1,36 @@ +import urllib.request +import threading +import json +import time + +LOCAL = "http://localhost:8899" +PAYLOAD = {"jsonrpc":"2.0","id":1,"method":"getHighestSnapshotSlot"} + +def get_snap_slot(req, jsondata): + response = urllib.request.urlopen(req, jsondata) + res = response.read() + slotnum = json.loads(res)["result"]["incremental"] + return slotnum + +if __name__ == "__main__": + jsondata = json.dumps(PAYLOAD) + jsondataasbytes = jsondata.encode('utf-8') + content_len = len(jsondataasbytes) + latest_snap = None + while True: + req = urllib.request.Request(LOCAL) + req.add_header('Content-Type', 'application/json; charset=utf-8') + req.add_header('Content-Length', content_len) + req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36') + if latest_snap is None: + latest_snap = get_snap_slot(req,jsondataasbytes) + print("current incr snapshot slot: %s\nsleeping..."%(latest_snap)) + stime = int(time.time()) + time.sleep(1) + latest_incr = get_snap_slot(req,jsondataasbytes) + if latest_incr == latest_snap: + time.sleep(1) + else: + etime = int(time.time()) - stime + break +print("latest incr snapshot slot: %s\nslept: %s seconds"%(latest_incr,etime)) diff --git a/templates/snapshot-finder.py b/templates/snapshot-finder.py new file mode 100644 index 0000000..29bc0b0 --- /dev/null +++ b/templates/snapshot-finder.py @@ -0,0 +1,520 @@ +from distutils.log import debug +import os +import glob +import requests +import time +import math +import json +import sys +import argparse +import logging +import subprocess +from pathlib import Path +from requests import ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError +from tqdm import tqdm +from multiprocessing.dummy import Pool as ThreadPool +import statistics + +parser = argparse.ArgumentParser(description='Solana snapshot finder') +parser.add_argument('-t', '--threads-count', default=1000, type=int, + help='the number of concurrently running threads that check snapshots for rpc nodes') +parser.add_argument('-r', '--rpc_address', + default='https://api.mainnet-beta.solana.com', type=str, + help='RPC address of the node from which the current slot number will be taken\n' + 'https://api.mainnet-beta.solana.com') + +parser.add_argument('--max_snapshot_age', default=1300, type=int, help='How many slots ago the snapshot was created (in slots)') +parser.add_argument('--min_download_speed', default=60, type=int, help='Minimum average snapshot download speed in megabytes') +parser.add_argument('--max_download_speed', type=int, +help='Maximum snapshot download speed in megabytes - https://github.com/c29r3/solana-snapshot-finder/issues/11. Example: --max_download_speed 192') +parser.add_argument('--max_latency', default=40, type=int, help='The maximum value of latency (milliseconds). If latency > max_latency --> skip') +parser.add_argument('--version', type=str, help='version of the snapshot required') +parser.add_argument('--with_private_rpc', action="store_true", help='Enable adding and checking RPCs with the --private-rpc option.This slow down checking and searching but potentially increases' + ' the number of RPCs from which snapshots can be downloaded.') +parser.add_argument('--measurement_time', default=7, type=int, help='Time in seconds during which the script will measure the download speed') +parser.add_argument('--snapshot_path', type=str, default=".", help='The location where the snapshot will be downloaded (absolute path).' + ' Example: /home/ubuntu/solana/validator-ledger') +parser.add_argument('--num_of_retries', default=5, type=int, help='The number of retries if a suitable server for downloading the snapshot was not found') +parser.add_argument('--sleep', default=30, type=int, help='Sleep before next retry (seconds)') +parser.add_argument('--sort_order', default='slots_diff', type=str, help='Priority way to sort the found servers. latency or slots_diff') +parser.add_argument('-b', '--blacklist', default='', type=str, help='If the same corrupted archive is constantly downloaded, you can exclude it.' + ' Specify either the number of the slot you want to exclude, or the hash of the archive name. ' + 'You can specify several, separated by commas. Example: -b 135501350,135501360 or --blacklist 135501350,some_hash') +parser.add_argument("-v", "--verbose", help="increase output verbosity to DEBUG", action="store_true") +args = parser.parse_args() + +DEFAULT_HEADERS = {"Content-Type": "application/json"} +RPC = args.rpc_address +WITH_PRIVATE_RPC = args.with_private_rpc +MAX_SNAPSHOT_AGE_IN_SLOTS = args.max_snapshot_age +THREADS_COUNT = args.threads_count +MIN_DOWNLOAD_SPEED_MB = args.min_download_speed +MAX_DOWNLOAD_SPEED_MB = args.max_download_speed +SPEED_MEASURE_TIME_SEC = args.measurement_time +MAX_LATENCY = args.max_latency +VERSION = args.version +SNAPSHOT_PATH = args.snapshot_path if args.snapshot_path[-1] != '/' else args.snapshot_path[:-1] +NUM_OF_MAX_ATTEMPTS = args.num_of_retries +SLEEP_BEFORE_RETRY = args.sleep +NUM_OF_ATTEMPTS = 1 +SORT_ORDER = args.sort_order +BLACKLIST = str(args.blacklist).split(",") +AVERAGE_SNAPSHOT_FILE_SIZE_MB = 2500.0 +AVERAGE_INCREMENT_FILE_SIZE_MB = 200.0 +AVERAGE_CATCHUP_SPEED = 2.0 +FULL_LOCAL_SNAP_SLOT = 0 + +current_slot = 0 +DISCARDED_BY_ARCHIVE_TYPE = 0 +DISCARDED_BY_LATENCY = 0 +DISCARDED_BY_SLOT = 0 +DISCARDED_BY_UNKNW_ERR = 0 +DISCARDED_BY_TIMEOUT = 0 +FULL_LOCAL_SNAPSHOTS = [] +# skip servers that do not fit the filters so as not to check them again +unsuitable_servers = set() +# Configure Logging +logging.getLogger('urllib3').setLevel(logging.WARNING) +if args.verbose: + logging.basicConfig( + level=logging.DEBUG, + format="%(asctime)s [%(levelname)s] %(message)s", + handlers=[ + logging.FileHandler(f'{SNAPSHOT_PATH}/snapshot-finder.log'), + logging.StreamHandler(sys.stdout), + ] + ) + +else: + # logging.basicConfig(stream=sys.stdout, encoding='utf-8', level=logging.INFO, format='|%(asctime)s| %(message)s') + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + handlers=[ + logging.FileHandler(f'{SNAPSHOT_PATH}/snapshot-finder.log'), + logging.StreamHandler(sys.stdout), + ] + ) +logger = logging.getLogger(__name__) + + +def convert_size(size_bytes): + if size_bytes == 0: + return "0B" + size_name = ("B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB") + i = int(math.floor(math.log(size_bytes, 1024))) + p = math.pow(1024, i) + s = round(size_bytes / p, 2) + return "%s %s" % (s, size_name[i]) + + +def version_check(url: str, version: str) -> bool: + data = {"jsonrpc":"2.0","id":1, "method":"getVersion"} + headers = {"Content-Type": "application/json"} + res = requests.post("http://%s"%url, json=data) + parsed_res = json.loads(res.text) + rpc_ver = parsed_res["result"]["solana-core"] + logger.info("version: %s"%rpc_ver) + if rpc_ver != version: + return False + return True + +def measure_speed(url: str, measure_time: int) -> float: + logging.debug('measure_speed()') + url = f'http://{url}/snapshot.tar.bz2' + r = requests.get(url, stream=True, timeout=measure_time+2) + r.raise_for_status() + start_time = time.monotonic_ns() + last_time = start_time + loaded = 0 + speeds = [] + for chunk in r.iter_content(chunk_size=81920): + curtime = time.monotonic_ns() + + worktime = (curtime - start_time) / 1000000000 + if worktime >= measure_time: + break + + delta = (curtime - last_time) / 1000000000 + loaded += len(chunk) + if delta > 1: + estimated_bytes_per_second = loaded * (1 / delta) + # print(f'{len(chunk)}:{delta} : {estimated_bytes_per_second}') + speeds.append(estimated_bytes_per_second) + + last_time = curtime + loaded = 0 + + return statistics.median(speeds) + + +def do_request(url_: str, method_: str = 'GET', data_: str = '', timeout_: int = 3, + headers_: dict = None): + global DISCARDED_BY_UNKNW_ERR + global DISCARDED_BY_TIMEOUT + r = '' + if headers_ is None: + headers_ = DEFAULT_HEADERS + + try: + if method_.lower() == 'get': + r = requests.get(url_, headers=headers_, timeout=(timeout_, timeout_)) + elif method_.lower() == 'post': + r = requests.post(url_, headers=headers_, data=data_, timeout=(timeout_, timeout_)) + elif method_.lower() == 'head': + r = requests.head(url_, headers=headers_, timeout=(timeout_, timeout_)) + # print(f'{r.content, r.status_code, r.text}') + return r + + except (ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError) as reqErr: + # logger.debug(f'error in do_request(): {reqErr=}') + DISCARDED_BY_TIMEOUT += 1 + return f'error in do_request(): {reqErr}' + + except Exception as unknwErr: + DISCARDED_BY_UNKNW_ERR += 1 + # logger.debug(f'error in do_request(): {unknwErr=}') + return f'error in do_request(): {reqErr}' + + +def get_current_slot(): + logger.debug("get_current_slot()") + d = '{"jsonrpc":"2.0","id":1, "method":"getSlot"}' + try: + r = do_request(url_=RPC, method_='post', data_=d, timeout_=25) + if 'result' in str(r.text): + return r.json()["result"] + else: + logger.error(f'Can\'t get current slot') + return None + except: + logger.error(f'Can\'t get current slot') + return None + + +def get_all_rpc_ips(): + logger.debug("get_all_rpc_ips()") + d = '{"jsonrpc":"2.0", "id":1, "method":"getClusterNodes"}' + r = do_request(url_=RPC, method_='post', data_=d, timeout_=25) + if 'result' in str(r.text): + if WITH_PRIVATE_RPC is True: + rpc_ips = [] + for node in r.json()["result"]: + if node["rpc"] is not None: + rpc_ips.append(node["rpc"]) + else: + gossip_ip = node["gossip"].split(":")[0] + rpc_ips.append(f'{gossip_ip}:8899') + + else: + rpc_ips = [rpc["rpc"] for rpc in r.json()["result"] if rpc["rpc"] is not None] + + rpc_ips = list(set(rpc_ips)) + return rpc_ips + + else: + logger.error(f'Can\'t get RPC ip addresses {r.text}') + sys.exit() + + +def get_snapshot_slot(rpc_address: str): + global FULL_LOCAL_SNAP_SLOT + global DISCARDED_BY_ARCHIVE_TYPE + global DISCARDED_BY_LATENCY + global DISCARDED_BY_SLOT + + pbar.update(1) + url = f'http://{rpc_address}/snapshot.tar.bz2' + inc_url = f'http://{rpc_address}/incremental-snapshot.tar.bz2' + # d = '{"jsonrpc":"2.0","id":1,"method":"getHighestSnapshotSlot"}' + try: + r = do_request(url_=inc_url, method_='head', timeout_=1) + if 'location' in str(r.headers) and 'error' not in str(r.text) and r.elapsed.total_seconds() * 1000 > MAX_LATENCY: + DISCARDED_BY_LATENCY += 1 + return None + + + if 'location' in str(r.headers) and 'error' not in str(r.text): + snap_location_ = r.headers["location"] + if snap_location_.endswith('tar') is True: + DISCARDED_BY_ARCHIVE_TYPE += 1 + return None + incremental_snap_slot = int(snap_location_.split("-")[2]) + snap_slot_ = int(snap_location_.split("-")[3]) + slots_diff = current_slot - snap_slot_ + + if slots_diff < -100: + logger.error(f'Something wrong with this snapshot\\rpc_node - {slots_diff=}. This node will be skipped {rpc_address=}') + DISCARDED_BY_SLOT += 1 + return + + if slots_diff > MAX_SNAPSHOT_AGE_IN_SLOTS: + DISCARDED_BY_SLOT += 1 + return + + if FULL_LOCAL_SNAP_SLOT == incremental_snap_slot: + json_data["rpc_nodes"].append({ + "snapshot_address": rpc_address, + "slots_diff": slots_diff, + "latency": r.elapsed.total_seconds() * 1000, + "files_to_download": [snap_location_], + "cost": AVERAGE_INCREMENT_FILE_SIZE_MB / MIN_DOWNLOAD_SPEED_MB + slots_diff / AVERAGE_CATCHUP_SPEED + }) + return + + r2 = do_request(url_=url, method_='head', timeout_=1) + if 'location' in str(r.headers) and 'error' not in str(r.text): + json_data["rpc_nodes"].append({ + "snapshot_address": rpc_address, + "slots_diff": slots_diff, + "latency": r.elapsed.total_seconds() * 1000, + "files_to_download": [r.headers["location"], r2.headers['location']], + "cost": (AVERAGE_SNAPSHOT_FILE_SIZE_MB + AVERAGE_INCREMENT_FILE_SIZE_MB) / MIN_DOWNLOAD_SPEED_MB + slots_diff / AVERAGE_CATCHUP_SPEED + }) + return + + r = do_request(url_=url, method_='head', timeout_=1) + if 'location' in str(r.headers) and 'error' not in str(r.text): + snap_location_ = r.headers["location"] + # filtering uncompressed archives + if snap_location_.endswith('tar') is True: + DISCARDED_BY_ARCHIVE_TYPE += 1 + return None + full_snap_slot_ = int(snap_location_.split("-")[1]) + slots_diff_full = current_slot - full_snap_slot_ + if slots_diff_full <= MAX_SNAPSHOT_AGE_IN_SLOTS and r.elapsed.total_seconds() * 1000 <= MAX_LATENCY: + # print(f'{rpc_address=} | {slots_diff=}') + json_data["rpc_nodes"].append({ + "snapshot_address": rpc_address, + "slots_diff": slots_diff_full, + "latency": r.elapsed.total_seconds() * 1000, + "files_to_download": [snap_location_], + "cost": AVERAGE_SNAPSHOT_FILE_SIZE_MB / MIN_DOWNLOAD_SPEED_MB + slots_diff_full / AVERAGE_CATCHUP_SPEED + }) + return + return None + + except Exception as getSnapErr_: + return None + + +def download(url: str): + fname = url[url.rfind('/'):].replace("/", "") + temp_fname = f'{SNAPSHOT_PATH}/tmp-{fname}' + # try: + # resp = requests.get(url, stream=True) + # total = int(resp.headers.get('content-length', 0)) + # with open(temp_fname, 'wb') as file, tqdm( + # desc=fname, + # total=total, + # unit='iB', + # unit_scale=True, + # unit_divisor=1024, + # ) as bar: + # for data in resp.iter_content(chunk_size=1024): + # size = file.write(data) + # bar.update(size) + + # logger.info(f'Rename the downloaded file {temp_fname} --> {fname}') + # os.rename(temp_fname, f'{SNAPSHOT_PATH}/{fname}') + + # except (ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError) as downlErr: + # logger.error(f'Exception in download() func\n {downlErr}') + + try: + # dirty trick with wget. Details here - https://github.com/c29r3/solana-snapshot-finder/issues/11 + if MAX_DOWNLOAD_SPEED_MB is not None: + process = subprocess.run(['/usr/bin/wget', f'--limit-rate={MAX_DOWNLOAD_SPEED_MB}M', '--trust-server-names', url, f'-O{temp_fname}'], + stdout=subprocess.PIPE, + universal_newlines=True) + else: + process = subprocess.run(['/usr/bin/wget', '--trust-server-names', url, f'-O{temp_fname}'], + stdout=subprocess.PIPE, + universal_newlines=True) + + logger.info(f'Rename the downloaded file {temp_fname} --> {fname}') + os.rename(temp_fname, f'{SNAPSHOT_PATH}/{fname}') + + except Exception as unknwErr: + logger.error(f'Exception in download() func. Make sure wget is installed\n{unknwErr}') + + +def main_worker(): + try: + global FULL_LOCAL_SNAP_SLOT + rpc_nodes = list(set(get_all_rpc_ips())) + global pbar + pbar = tqdm(total=len(rpc_nodes)) + logger.info(f'RPC servers in total: {len(rpc_nodes)} | Current slot number: {current_slot}\n') + + # Search for full local snapshots. + # If such a snapshot is found and it is not too old, then the script will try to find and download an incremental snapshot + FULL_LOCAL_SNAPSHOTS = glob.glob(f'{SNAPSHOT_PATH}/snapshot-*tar*') + if len(FULL_LOCAL_SNAPSHOTS) > 0: + FULL_LOCAL_SNAPSHOTS.sort(reverse=True) + FULL_LOCAL_SNAP_SLOT = FULL_LOCAL_SNAPSHOTS[0].replace(SNAPSHOT_PATH, "").split("-")[1] + logger.info(f'Found full local snapshot {FULL_LOCAL_SNAPSHOTS[0]} | {FULL_LOCAL_SNAP_SLOT=}') + + else: + logger.info(f'Can\'t find any full local snapshots in this path {SNAPSHOT_PATH} --> the search will be carried out on full snapshots') + + print(f'Searching information about snapshots on all found RPCs') + pool = ThreadPool() + pool.map(get_snapshot_slot, rpc_nodes) + logger.info(f'Found suitable RPCs: {len(json_data["rpc_nodes"])}') + logger.info(f'The following information shows for what reason and how many RPCs were skipped.' + f'Timeout most probably mean, that node RPC port does not respond (port is closed)\n' + f'{DISCARDED_BY_ARCHIVE_TYPE=} | {DISCARDED_BY_LATENCY=} |' + f' {DISCARDED_BY_SLOT=} | {DISCARDED_BY_TIMEOUT=} | {DISCARDED_BY_UNKNW_ERR=}') + + if len(json_data["rpc_nodes"]) == 0: + logger.info(f'No snapshot nodes were found matching the given parameters: {args.max_snapshot_age=}') + sys.exit() + + # sort list of rpc node by SORT_ORDER (latency) + rpc_nodes_sorted = sorted(json_data["rpc_nodes"], key=lambda k: k[SORT_ORDER]) + + json_data.update({ + "last_update_at": time.time(), + "last_update_slot": current_slot, + "total_rpc_nodes": len(rpc_nodes), + "rpc_nodes_with_actual_snapshot": len(json_data["rpc_nodes"]), + "rpc_nodes": rpc_nodes_sorted + }) + + with open(f'{SNAPSHOT_PATH}/snapshot.json', "w") as result_f: + json.dump(json_data, result_f, indent=2) + logger.info(f'All data is saved to json file - {SNAPSHOT_PATH}/snapshot.json') + + best_snapshot_node = {} + num_of_rpc_to_check = 15 + + rpc_nodes_inc_sorted = [] + logger.info("TRYING TO DOWNLOADING FILES") + for i, rpc_node in enumerate(json_data["rpc_nodes"], start=1): + logger.info("checking node: %s %s"%(i,rpc_node)) + # filter blacklisted snapshots + if BLACKLIST != ['']: + if any(i in str(rpc_node["files_to_download"]) for i in BLACKLIST): + logger.info(f'{i}\\{len(json_data["rpc_nodes"])} BLACKLISTED --> {rpc_node}') + continue + + logger.info(f'{i}\\{len(json_data["rpc_nodes"])} checking the speed {rpc_node}') + if rpc_node["snapshot_address"] in unsuitable_servers: + logger.info(f'Rpc node already in unsuitable list --> skip {rpc_node["snapshot_address"]}') + continue + logger.info("checking version") + try: + if not version_check(rpc_node["snapshot_address"], VERSION): + logger.info("version check failed") + continue + except: + import traceback + print(traceback.format_exc()) + print("broken") + logger.info("version check succeeded") + down_speed_bytes = measure_speed(url=rpc_node["snapshot_address"], measure_time=SPEED_MEASURE_TIME_SEC) + down_speed_mb = convert_size(down_speed_bytes) + if down_speed_bytes < MIN_DOWNLOAD_SPEED_MB * 1e6: + logger.info(f'Too slow: {rpc_node=} {down_speed_mb=}') + unsuitable_servers.add(rpc_node["snapshot_address"]) + continue + + elif down_speed_bytes >= MIN_DOWNLOAD_SPEED_MB * 1e6: + logger.info(f'Suitable snapshot server found: {rpc_node=} {down_speed_mb=}') + for path in reversed(rpc_node["files_to_download"]): + # do not download full snapshot if it already exists locally + if str(path).startswith("/snapshot-"): + full_snap_slot__ = path.split("-")[1] + if full_snap_slot__ == FULL_LOCAL_SNAP_SLOT: + continue + + + if 'incremental' in path: + r = do_request(f'http://{rpc_node["snapshot_address"]}/incremental-snapshot.tar.bz2', method_='head', timeout_=2) + if 'location' in str(r.headers) and 'error' not in str(r.text): + best_snapshot_node = f'http://{rpc_node["snapshot_address"]}{r.headers["location"]}' + else: + best_snapshot_node = f'http://{rpc_node["snapshot_address"]}{path}' + + else: + best_snapshot_node = f'http://{rpc_node["snapshot_address"]}{path}' + logger.info(f'Downloading {best_snapshot_node} snapshot to {SNAPSHOT_PATH}') + download(url=best_snapshot_node) + return 0 + + elif i > num_of_rpc_to_check: + logger.info(f'The limit on the number of RPC nodes from' + ' which we measure the speed has been reached {num_of_rpc_to_check=}\n') + break + + else: + logger.info(f'{down_speed_mb=} < {MIN_DOWNLOAD_SPEED_MB=}') + + if best_snapshot_node is {}: + logger.error(f'No snapshot nodes were found matching the given parameters:{args.min_download_speed=}' + f'\nTry restarting the script with --with_private_rpc' + f'RETRY #{NUM_OF_ATTEMPTS}\\{NUM_OF_MAX_ATTEMPTS}') + return 1 + + + + except KeyboardInterrupt: + sys.exit('\nKeyboardInterrupt - ctrl + c') + + except: + return 1 + + +logger.info("Version: 0.3.3") +logger.info("https://github.com/c29r3/solana-snapshot-finder\n\n") +logger.info(f'{RPC=}\n' + f'{MAX_SNAPSHOT_AGE_IN_SLOTS=}\n' + f'{MIN_DOWNLOAD_SPEED_MB=}\n' + f'{MAX_DOWNLOAD_SPEED_MB=}\n' + f'{SNAPSHOT_PATH=}\n' + f'{THREADS_COUNT=}\n' + f'{NUM_OF_MAX_ATTEMPTS=}\n' + f'{WITH_PRIVATE_RPC=}\n' + f'{SORT_ORDER=}') + +try: + f_ = open(f'{SNAPSHOT_PATH}/write_perm_test', 'w') + f_.close() + os.remove(f'{SNAPSHOT_PATH}/write_perm_test') +except IOError: + logger.error(f'\nCheck {SNAPSHOT_PATH=} and permissions') + Path(SNAPSHOT_PATH).mkdir(parents=True, exist_ok=True) + +json_data = ({"last_update_at": 0.0, + "last_update_slot": 0, + "total_rpc_nodes": 0, + "rpc_nodes_with_actual_snapshot": 0, + "rpc_nodes": [] + }) + + +while NUM_OF_ATTEMPTS <= NUM_OF_MAX_ATTEMPTS: + current_slot = get_current_slot() + logger.info(f'Attempt number: {NUM_OF_ATTEMPTS}. Total attempts: {NUM_OF_MAX_ATTEMPTS}') + NUM_OF_ATTEMPTS += 1 + + if current_slot is None: + continue + + worker_result = main_worker() + + if worker_result == 0: + logger.info("Done") + exit(0) + + if worker_result != 0: + logger.info("Now trying with flag --with_private_rpc") + WITH_PRIVATE_RPC = True + + if NUM_OF_ATTEMPTS >= NUM_OF_MAX_ATTEMPTS: + logger.error(f'Could not find a suitable snapshot --> exit') + sys.exit() + + logger.info(f"Sleeping {SLEEP_BEFORE_RETRY} seconds before next try") + time.sleep(SLEEP_BEFORE_RETRY) diff --git a/templates/sol.service b/templates/sol.service new file mode 100644 index 0000000..3d8a2c5 --- /dev/null +++ b/templates/sol.service @@ -0,0 +1,18 @@ +[Unit] +Description=Solana Validator +After=network.target +StartLimitIntervalSec=0 + +[Service] +Type=simple +Restart=on-failure +RestartSec=1 +LimitNOFILE=1000000 +LogRateLimitIntervalSec=0 +User=solana +Environment=PATH=/bin:/usr/bin:/home/solana/.local/share/solana/install/active_release/bin +Environment=SOLANA_METRICS_CONFIG=host=https://metrics.solana.com:8086,db=mainnet-beta,u=mainnet-beta_write,p=password +ExecStart=/home/solana/validator.sh + +[Install] +WantedBy=multi-user.target \ No newline at end of file From 8b7b8f5b5f8913d58e188ae0bd588e57745dc15a Mon Sep 17 00:00:00 2001 From: vovkman Date: Sat, 26 Nov 2022 11:18:05 -0800 Subject: [PATCH 21/35] update defaults --- defaults/main.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/defaults/main.yml b/defaults/main.yml index 789e117..b0e7045 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -2,8 +2,7 @@ # defaults file for Solana RPC solana_version: "1.13.5" raw_disk_list: ["/dev/nvme0n1", "/dev/nvme1n1"] -# setup_disks: "true" -setup_disks: "false" +setup_disks: "true" download_snapshot: "true" # ramdisk_size: "300" swap_mb: "100000" From 25d33eae17c56420d40bd73313f73c8b3e6d0360 Mon Sep 17 00:00:00 2001 From: vovkman Date: Sat, 26 Nov 2022 11:20:24 -0800 Subject: [PATCH 22/35] update docs --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 8c2b756..d675d56 100644 --- a/README.md +++ b/README.md @@ -54,11 +54,10 @@ cd autoclock-rpc - this command can take between 10-20 minutes based on the specs of the machine - it takes long because it does everything necessary to start the validator (format disks, checkout the solana repo and build it, download the latest snapshot, etc.) - make sure that the solana_version is up to date (see below) -- you should create a playbook file, you can look at `test_runner.yaml` for inspiration - check the values set in `defaults/main.yml` and update to the values you want ``` -time ansible-playbook +time ansible-playbook runner.yaml ``` #### ~ Parameters explained ~ From d2c73a2b2b3a5d1702e269011ef95c3ce1a551e6 Mon Sep 17 00:00:00 2001 From: vovkman Date: Sat, 26 Nov 2022 11:21:33 -0800 Subject: [PATCH 23/35] remove genesysgo --- templates/montip.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/templates/montip.py b/templates/montip.py index a557eb4..a2433c9 100644 --- a/templates/montip.py +++ b/templates/montip.py @@ -3,7 +3,7 @@ import json import time -GENESYSGO = "https://api.mainnet-beta.solana.com" +MAINNET = "https://api.mainnet-beta.solana.com" LOCAL = "http://localhost:8899" PAYLOAD = {"jsonrpc":"2.0","id":1, "method":"getSlot", "params":[{"commitment":"processed"}]} @@ -22,7 +22,7 @@ def get_slot(req, jsondata, result, idx): tlist = [] reqlist = [] resultlist = [0,0] - for c,u in enumerate([LOCAL, GENESYSGO]): + for c,u in enumerate([LOCAL, MAINNET]): req = urllib.request.Request(u) req.add_header('Content-Type', 'application/json; charset=utf-8') req.add_header('Content-Length', content_len) From 88d55b4313a25b880b910b5067316e490e16d79d Mon Sep 17 00:00:00 2001 From: vovkman Date: Sat, 26 Nov 2022 11:23:22 -0800 Subject: [PATCH 24/35] update validator script --- templates/validator.j2 | 30 +----------------------------- 1 file changed, 1 insertion(+), 29 deletions(-) diff --git a/templates/validator.j2 b/templates/validator.j2 index 1cae8eb..8216133 100644 --- a/templates/validator.j2 +++ b/templates/validator.j2 @@ -4,32 +4,4 @@ PATH=/home/solana/.local/share/solana/install/active_release/bin:/usr/sbin:/usr/ # export RUST_LOG=error # Start solana rpc node, intended to be used with journalctl/systemctl. Logs will go to journalctl. -exec solana-validator \ - --identity /home/solana/rpc_node.json \ - --entrypoint entrypoint.mainnet-beta.solana.com:8001 \ - --entrypoint entrypoint2.mainnet-beta.solana.com:8001 \ - --entrypoint entrypoint3.mainnet-beta.solana.com:8001 \ - --entrypoint entrypoint4.mainnet-beta.solana.com:8001 \ - --entrypoint entrypoint5.mainnet-beta.solana.com:8001 \ - --accounts /mnt/solana-accounts \ - --ledger /mnt/solana-ledger \ - --snapshots /mnt/solana-snapshots \ - --log /mnt/logs/solana-validator.log \ - --gossip-port 8001 \ - --rpc-port 8899 \ - --rpc-bind-address 0.0.0.0 \ - --dynamic-port-range 8002-8099 \ - --wal-recovery-mode skip_any_corrupted_record \ - --limit-ledger-size 50000000 \ - --private-rpc \ - --no-snapshot-fetch \ - --no-port-check \ - --no-voting \ - --full-rpc-api \ - --tpu-use-quic \ - --enable-cpi-and-log-storage \ - --enable-rpc-transaction-history \ - --account-index program-id spl-token-mint spl-token-owner \ - --accounts-index-memory-limit-mb 800000 \ - --account-index-exclude-key TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA \ - --account-index-exclude-key kinXdEcpDQeHPEuQnqmUgtYykqKGVFq6CeVX5iAHJq6 \ No newline at end of file +exec solana-validator --identity /home/solana/rpc_node.json --rpc-port 8899 --full-rpc-api --entrypoint entrypoint.mainnet-beta.solana.com:8001 --limit-ledger-size --log /mnt/logs/solana-validator.log --accounts /mnt/solana-accounts --ledger /mnt/solana-ledger --snapshots /mnt/solana-snapshots --no-snapshot-fetch \ No newline at end of file From dca7de21067f88700b2a067338513bd88d51f824 Mon Sep 17 00:00:00 2001 From: vovkman Date: Sun, 27 Nov 2022 13:07:18 -0800 Subject: [PATCH 25/35] add configurable base path --- defaults/main.yml | 4 +++- fstab.yaml | 4 ++-- tasks/dirs.yaml | 10 +++++----- tasks/disks.yaml | 5 ++--- tasks/file_setup.yaml | 4 ++-- tasks/git.yaml | 4 ++-- tasks/ramdisk.yaml | 6 +++--- tasks/rotate.yaml | 4 ++-- tasks/snapshot_downloader.yaml | 5 ++--- tasks/swap.yaml | 33 +++++++++++++++++---------------- templates/restart.sh | 2 +- templates/validator.j2 | 2 +- 12 files changed, 42 insertions(+), 41 deletions(-) diff --git a/defaults/main.yml b/defaults/main.yml index b0e7045..73bfb9b 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -2,8 +2,10 @@ # defaults file for Solana RPC solana_version: "1.13.5" raw_disk_list: ["/dev/nvme0n1", "/dev/nvme1n1"] -setup_disks: "true" +# setup_disks: "true" +setup_disks: "false" download_snapshot: "true" # ramdisk_size: "300" swap_mb: "100000" solana_installer: "true" +base_path: /home/mnt diff --git a/fstab.yaml b/fstab.yaml index 3af44b2..d0a9aec 100644 --- a/fstab.yaml +++ b/fstab.yaml @@ -3,5 +3,5 @@ path: /etc/fstab state: present block: | - /mnt/swapfile swap swap defaults 0 0 - /dev/md0 /mnt ext4 defaults,nofail,discard 0 0 + {{ base_path }}/swapfile swap swap defaults 0 0 + /dev/md0 {{ base_path }} ext4 defaults,nofail,discard 0 0 diff --git a/tasks/dirs.yaml b/tasks/dirs.yaml index fe0f159..a339124 100644 --- a/tasks/dirs.yaml +++ b/tasks/dirs.yaml @@ -1,6 +1,6 @@ - name: mnt folder file: - path: /mnt + path: "{{ base_path }}" state: directory owner: solana group: solana @@ -10,7 +10,7 @@ - name: solana snapshots file: - path: /mnt/solana-snapshots + path: "{{ base_path }}/solana-snapshots" state: directory owner: solana group: solana @@ -19,7 +19,7 @@ - name: solana ledger file: - path: /mnt/solana-ledger + path: "{{ base_path }}/solana-ledger" state: directory owner: solana group: solana @@ -28,7 +28,7 @@ - name: solana accounts file: - path: /mnt/solana-accounts + path: "{{ base_path }}/solana-accounts" state: directory owner: solana group: solana @@ -39,7 +39,7 @@ become: true become_user: root file: - path: /mnt/logs + path: "{{ base_path }}/logs" state: directory owner: solana group: solana diff --git a/tasks/disks.yaml b/tasks/disks.yaml index d065a14..2dd8528 100644 --- a/tasks/disks.yaml +++ b/tasks/disks.yaml @@ -4,7 +4,7 @@ apt: update_cache: yes pkg: - - mdadm + - mdadm - name: check raid already exists become: true @@ -50,6 +50,5 @@ - name: mount disks become: true become_user: root - shell: mount /dev/{{ raid_name.stdout }} /mnt + shell: mount /dev/{{ raid_name.stdout }} {{ base_path }} when: mount_mnt.rc != 0 - diff --git a/tasks/file_setup.yaml b/tasks/file_setup.yaml index 5065010..910ce13 100644 --- a/tasks/file_setup.yaml +++ b/tasks/file_setup.yaml @@ -33,7 +33,7 @@ become_user: solana template: src: snapcheck.py - dest: /mnt/snapcheck.py + dest: "{{ base_path }}/snapcheck.py" owner: solana group: solana @@ -42,6 +42,6 @@ become_user: solana template: src: snapshot-finder.py - dest: /mnt/snapshot-finder.py + dest: "{{ base_path }}/snapshot-finder.py" owner: solana group: solana diff --git a/tasks/git.yaml b/tasks/git.yaml index ac5c10f..ddd256b 100644 --- a/tasks/git.yaml +++ b/tasks/git.yaml @@ -3,7 +3,7 @@ become_user: solana git: repo: "https://github.com/solana-labs/solana.git" - dest: /mnt/solana + dest: {{ base_path }}/solana version: tags/v{{ solana_version }} - name: build solana @@ -11,5 +11,5 @@ become_user: solana shell: source /home/solana/.cargo/env && cargo build --release args: - chdir: /mnt/solana + chdir: {{ base_path }}/solana executable: /bin/bash diff --git a/tasks/ramdisk.yaml b/tasks/ramdisk.yaml index 3cf7ec8..f028431 100644 --- a/tasks/ramdisk.yaml +++ b/tasks/ramdisk.yaml @@ -1,14 +1,14 @@ - name: setup ramdisk for accountsdb become: true become_user: root - shell: mount -t tmpfs -o size={{ ramdisk_size }}G tmpfs /mnt/solana-accounts + shell: mount -t tmpfs -o size={{ ramdisk_size }}G tmpfs {{ base_path }}/solana-accounts when: ansible_memtotal_mb >= 1500*ramdisk_size -- name: add ramdisk to fstab +- name: add ramdisk to fstab become: true become_user: root lineinfile: dest: /etc/fstab state: present - line: 'tmpfs /mnt/solana-accounts tmpfs rw,size={{ ramdisk_size }}G,user=solana 0 0' + line: "tmpfs {{ base_path }}/solana-accounts tmpfs rw,size={{ ramdisk_size }}G,user=solana 0 0" when: ansible_memtotal_mb >= 1500*ramdisk_size diff --git a/tasks/rotate.yaml b/tasks/rotate.yaml index fd15643..de91269 100644 --- a/tasks/rotate.yaml +++ b/tasks/rotate.yaml @@ -4,7 +4,7 @@ file: path: /etc/logrotate.d/sol state: touch - mode: '0644' + mode: "0644" owner: root group: root modification_time: preserve @@ -16,7 +16,7 @@ blockinfile: path: /etc/logrotate.d/sol block: | - /mnt/logs/solana-validator.log { + {{ base_path }}/logs/solana-validator.log { rotate 7 daily missingok diff --git a/tasks/snapshot_downloader.yaml b/tasks/snapshot_downloader.yaml index 037a51c..ffea30b 100644 --- a/tasks/snapshot_downloader.yaml +++ b/tasks/snapshot_downloader.yaml @@ -3,12 +3,11 @@ become_user: solana ansible.builtin.pip: name: - - tqdm + - tqdm - requests executable: pip3 - name: download latest snapshot become: true become_user: solana - shell: python3 /mnt/snapshot-finder.py --snapshot_path /mnt/solana-snapshots --version {{ solana_version }} - + shell: python3 {{ base_path }}/snapshot-finder.py --snapshot_path {{ base_path }}/solana-snapshots --version {{ solana_version }} diff --git a/tasks/swap.yaml b/tasks/swap.yaml index b260994..305fa5d 100644 --- a/tasks/swap.yaml +++ b/tasks/swap.yaml @@ -1,49 +1,50 @@ - name: Create swap file become: true become_user: root - command: dd if=/dev/zero of=/mnt/swapfile bs=1024 count={{ swap_mb }}k - creates="/mnt/swapfile" + command: + dd if=/dev/zero of={{ base_path }}/swapfile bs=1024 count={{ swap_mb }}k + creates="{{ base_path }}/swapfile" - name: Change swap file permissions become: true become_user: root - file: path="/mnt/swapfile" - owner=root - group=root - mode=0600 + file: path="{{ base_path }}/swapfile" + owner=root + group=root + mode=0600 - name: Check swap file type become: true become_user: root - command: file /mnt/swapfile + command: file {{ base_path }}/swapfile register: swapfile - name: Make swap file become: true become_user: root - command: "sudo mkswap /mnt/swapfile" + command: "sudo mkswap {{ base_path }}/swapfile" when: swapfile.stdout.find('swap file') == -1 - name: Write swap entry in fstab become: true become_user: root mount: name=none - src=/mnt/swapfile - fstype=swap - opts=sw - passno=0 - dump=0 - state=present + src={{ base_path }}/swapfile + fstype=swap + opts=sw + passno=0 + dump=0 + state=present - name: swap check become: true become_user: root ignore_errors: yes - shell: swapon -show | grep /mnt/swapfile + shell: swapon -show | grep {{ base_path }}/swapfile register: swap_check - name: Mount swap become: true become_user: root - command: "swapon /mnt/swapfile" + command: "swapon {{ base_path }}/swapfile" when: swap_check.rc != 0 diff --git a/templates/restart.sh b/templates/restart.sh index 4be5120..bad8069 100644 --- a/templates/restart.sh +++ b/templates/restart.sh @@ -1,7 +1,7 @@ #!/bin/bash if [ $# -eq 0 ] then - python3 /mnt/snapcheck.py + python3 {{ base_path }}/snapcheck.py fi sudo systemctl stop sol.service sudo systemctl start sol.service diff --git a/templates/validator.j2 b/templates/validator.j2 index 8216133..f6f3437 100644 --- a/templates/validator.j2 +++ b/templates/validator.j2 @@ -4,4 +4,4 @@ PATH=/home/solana/.local/share/solana/install/active_release/bin:/usr/sbin:/usr/ # export RUST_LOG=error # Start solana rpc node, intended to be used with journalctl/systemctl. Logs will go to journalctl. -exec solana-validator --identity /home/solana/rpc_node.json --rpc-port 8899 --full-rpc-api --entrypoint entrypoint.mainnet-beta.solana.com:8001 --limit-ledger-size --log /mnt/logs/solana-validator.log --accounts /mnt/solana-accounts --ledger /mnt/solana-ledger --snapshots /mnt/solana-snapshots --no-snapshot-fetch \ No newline at end of file +exec solana-validator --identity /home/solana/rpc_node.json --rpc-port 8899 --full-rpc-api --entrypoint entrypoint.mainnet-beta.solana.com:8001 --limit-ledger-size --log {{ base_path }}/logs/solana-validator.log --accounts {{ base_path }}/solana-accounts --ledger {{ base_path }}/solana-ledger --snapshots {{ base_path }}/solana-snapshots --no-snapshot-fetch \ No newline at end of file From 3811948a79c98ce0e9abb6b49fcdaa9afb49cb93 Mon Sep 17 00:00:00 2001 From: vovkman Date: Sun, 27 Nov 2022 13:08:50 -0800 Subject: [PATCH 26/35] update defaults --- defaults/main.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/defaults/main.yml b/defaults/main.yml index 73bfb9b..104d857 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -2,10 +2,9 @@ # defaults file for Solana RPC solana_version: "1.13.5" raw_disk_list: ["/dev/nvme0n1", "/dev/nvme1n1"] -# setup_disks: "true" -setup_disks: "false" +setup_disks: "true" download_snapshot: "true" # ramdisk_size: "300" swap_mb: "100000" solana_installer: "true" -base_path: /home/mnt +base_path: /mnt From 6c16c72918bb092151bbda8f9f909806f3a14226 Mon Sep 17 00:00:00 2001 From: dubbelosix Date: Sat, 14 Jan 2023 20:11:30 +0530 Subject: [PATCH 27/35] merge ansol. start simple --- defaults/main.yml | 7 +++---- {templates => files}/montip.py | 0 {templates => files}/restart.sh | 2 +- {templates => files}/snapcheck.py | 0 {templates => files}/snapshot-finder.py | 24 ++++++++++++------------ files/sol.service | 17 +++++++++++++++++ files/validator.sh | 16 ++++++++++++++++ fstab.yaml | 4 ++-- tasks/dirs.yaml | 10 +++++----- tasks/disks.yaml | 4 ++-- tasks/file_setup.yaml | 14 +++++++------- tasks/git.yaml | 6 +++--- tasks/keygen.yaml | 4 ++-- tasks/main.yaml | 1 - tasks/ramdisk.yaml | 4 ++-- tasks/rotate.yaml | 2 +- tasks/snapshot_downloader.yaml | 2 +- tasks/swap.yaml | 17 +++++++++-------- tasks/tuner.yaml | 8 +------- templates/sol.service | 18 ------------------ templates/validator.j2 | 7 ------- 21 files changed, 84 insertions(+), 83 deletions(-) rename {templates => files}/montip.py (100%) rename {templates => files}/restart.sh (71%) rename {templates => files}/snapcheck.py (100%) rename {templates => files}/snapshot-finder.py (97%) create mode 100644 files/sol.service create mode 100644 files/validator.sh delete mode 100644 templates/sol.service delete mode 100644 templates/validator.j2 diff --git a/defaults/main.yml b/defaults/main.yml index 104d857..a305749 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -1,10 +1,9 @@ --- # defaults file for Solana RPC solana_version: "1.13.5" -raw_disk_list: ["/dev/nvme0n1", "/dev/nvme1n1"] +raw_disk_list: ["/dev/nvme1n1", "/dev/nvme2n1"] setup_disks: "true" download_snapshot: "true" -# ramdisk_size: "300" +ramdisk_size: "300" swap_mb: "100000" -solana_installer: "true" -base_path: /mnt + diff --git a/templates/montip.py b/files/montip.py similarity index 100% rename from templates/montip.py rename to files/montip.py diff --git a/templates/restart.sh b/files/restart.sh similarity index 71% rename from templates/restart.sh rename to files/restart.sh index bad8069..4be5120 100644 --- a/templates/restart.sh +++ b/files/restart.sh @@ -1,7 +1,7 @@ #!/bin/bash if [ $# -eq 0 ] then - python3 {{ base_path }}/snapcheck.py + python3 /mnt/snapcheck.py fi sudo systemctl stop sol.service sudo systemctl start sol.service diff --git a/templates/snapcheck.py b/files/snapcheck.py similarity index 100% rename from templates/snapcheck.py rename to files/snapcheck.py diff --git a/templates/snapshot-finder.py b/files/snapshot-finder.py similarity index 97% rename from templates/snapshot-finder.py rename to files/snapshot-finder.py index 29bc0b0..6c6e7c3 100644 --- a/templates/snapshot-finder.py +++ b/files/snapshot-finder.py @@ -28,7 +28,7 @@ parser.add_argument('--max_download_speed', type=int, help='Maximum snapshot download speed in megabytes - https://github.com/c29r3/solana-snapshot-finder/issues/11. Example: --max_download_speed 192') parser.add_argument('--max_latency', default=40, type=int, help='The maximum value of latency (milliseconds). If latency > max_latency --> skip') -parser.add_argument('--version', type=str, help='version of the snapshot required') +# parser.add_argument('--version', type=str, help='version of the snapshot required') parser.add_argument('--with_private_rpc', action="store_true", help='Enable adding and checking RPCs with the --private-rpc option.This slow down checking and searching but potentially increases' ' the number of RPCs from which snapshots can be downloaded.') parser.add_argument('--measurement_time', default=7, type=int, help='Time in seconds during which the script will measure the download speed') @@ -52,7 +52,7 @@ MAX_DOWNLOAD_SPEED_MB = args.max_download_speed SPEED_MEASURE_TIME_SEC = args.measurement_time MAX_LATENCY = args.max_latency -VERSION = args.version +# VERSION = args.version SNAPSHOT_PATH = args.snapshot_path if args.snapshot_path[-1] != '/' else args.snapshot_path[:-1] NUM_OF_MAX_ATTEMPTS = args.num_of_retries SLEEP_BEFORE_RETRY = args.sleep @@ -403,16 +403,16 @@ def main_worker(): if rpc_node["snapshot_address"] in unsuitable_servers: logger.info(f'Rpc node already in unsuitable list --> skip {rpc_node["snapshot_address"]}') continue - logger.info("checking version") - try: - if not version_check(rpc_node["snapshot_address"], VERSION): - logger.info("version check failed") - continue - except: - import traceback - print(traceback.format_exc()) - print("broken") - logger.info("version check succeeded") + # logger.info("checking version") + # try: + # if not version_check(rpc_node["snapshot_address"], VERSION): + # logger.info("version check failed") + # continue + # except: + # import traceback + # print(traceback.format_exc()) + # print("broken") + # logger.info("version check succeeded") down_speed_bytes = measure_speed(url=rpc_node["snapshot_address"], measure_time=SPEED_MEASURE_TIME_SEC) down_speed_mb = convert_size(down_speed_bytes) if down_speed_bytes < MIN_DOWNLOAD_SPEED_MB * 1e6: diff --git a/files/sol.service b/files/sol.service new file mode 100644 index 0000000..a8c3e95 --- /dev/null +++ b/files/sol.service @@ -0,0 +1,17 @@ +[Unit] +Description=Solana Validator +After=network.target +StartLimitIntervalSec=0 + +[Service] +Type=simple +Restart=always +RestartSec=1 +User=solana +LimitNOFILE=1000000 +LogRateLimitIntervalSec=0 +Environment="PATH=/mnt/solana/target/release/:/home/solana/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/snap/bin:/home/solana/.local/bin/:/home/solana/.local/bin/:/home/solana/.local/bin/" +ExecStart=/home/solana/validator.sh + +[Install] +WantedBy=multi-user.target diff --git a/files/validator.sh b/files/validator.sh new file mode 100644 index 0000000..f541508 --- /dev/null +++ b/files/validator.sh @@ -0,0 +1,16 @@ +#!/bin/bash +export SOLANA_METRICS_CONFIG="host=https://metrics.solana.com:8086,db=mainnet-beta,u=mainnet-beta_write,p=password" +exec /mnt/solana/target/release/solana-validator --identity /home/solana/rpc_node.json \ +--full-rpc-api \ +--rpc-port 8899 \ +--entrypoint entrypoint.mainnet-beta.solana.com:8001 \ +--limit-ledger-size \ +--log /mnt/logs/solana-validator.log \ +--accounts /mnt/solana-accounts \ +--ledger /mnt/solana-ledger \ +--snapshots /mnt/solana-snapshots \ +--no-snapshot-fetch \ +--known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 \ +--known-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ \ +--known-validator DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ \ +--known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S \ No newline at end of file diff --git a/fstab.yaml b/fstab.yaml index d0a9aec..71ca16d 100644 --- a/fstab.yaml +++ b/fstab.yaml @@ -3,5 +3,5 @@ path: /etc/fstab state: present block: | - {{ base_path }}/swapfile swap swap defaults 0 0 - /dev/md0 {{ base_path }} ext4 defaults,nofail,discard 0 0 + /dev/md0 /mnt ext4 defaults,nofail,discard 0 0 + /mnt/swapfile swap swap defaults 0 0 diff --git a/tasks/dirs.yaml b/tasks/dirs.yaml index a339124..fe0f159 100644 --- a/tasks/dirs.yaml +++ b/tasks/dirs.yaml @@ -1,6 +1,6 @@ - name: mnt folder file: - path: "{{ base_path }}" + path: /mnt state: directory owner: solana group: solana @@ -10,7 +10,7 @@ - name: solana snapshots file: - path: "{{ base_path }}/solana-snapshots" + path: /mnt/solana-snapshots state: directory owner: solana group: solana @@ -19,7 +19,7 @@ - name: solana ledger file: - path: "{{ base_path }}/solana-ledger" + path: /mnt/solana-ledger state: directory owner: solana group: solana @@ -28,7 +28,7 @@ - name: solana accounts file: - path: "{{ base_path }}/solana-accounts" + path: /mnt/solana-accounts state: directory owner: solana group: solana @@ -39,7 +39,7 @@ become: true become_user: root file: - path: "{{ base_path }}/logs" + path: /mnt/logs state: directory owner: solana group: solana diff --git a/tasks/disks.yaml b/tasks/disks.yaml index 2dd8528..093f81a 100644 --- a/tasks/disks.yaml +++ b/tasks/disks.yaml @@ -9,7 +9,7 @@ - name: check raid already exists become: true become_user: root - shell: fdisk -l | grep -e 'md0' -e 'md127' + shell: fdisk -l | grep -e 'md' ignore_errors: yes register: raid_exists @@ -50,5 +50,5 @@ - name: mount disks become: true become_user: root - shell: mount /dev/{{ raid_name.stdout }} {{ base_path }} + shell: mount /dev/{{ raid_name.stdout }} /mnt when: mount_mnt.rc != 0 diff --git a/tasks/file_setup.yaml b/tasks/file_setup.yaml index 910ce13..f9a8cf7 100644 --- a/tasks/file_setup.yaml +++ b/tasks/file_setup.yaml @@ -1,8 +1,8 @@ - name: sol validator setup become: true become_user: solana - template: - src: validator.j2 + copy: + src: validator.sh dest: /home/solana/validator.sh owner: solana group: solana @@ -11,7 +11,7 @@ - name: setup sol.service become: true become_user: root - template: + copy: src: sol.service dest: /etc/systemd/system/sol.service owner: root @@ -21,7 +21,7 @@ - name: copy restart.sh become: true become_user: solana - template: + copy: src: restart.sh dest: /home/solana/restart.sh owner: solana @@ -31,9 +31,9 @@ - name: copy snapcheck.py become: true become_user: solana - template: + copy: src: snapcheck.py - dest: "{{ base_path }}/snapcheck.py" + dest: /mnt/snapcheck.py owner: solana group: solana @@ -42,6 +42,6 @@ become_user: solana template: src: snapshot-finder.py - dest: "{{ base_path }}/snapshot-finder.py" + dest: /mnt/snapshot-finder.py owner: solana group: solana diff --git a/tasks/git.yaml b/tasks/git.yaml index ddd256b..bfddee7 100644 --- a/tasks/git.yaml +++ b/tasks/git.yaml @@ -3,13 +3,13 @@ become_user: solana git: repo: "https://github.com/solana-labs/solana.git" - dest: {{ base_path }}/solana + dest: /mnt/solana version: tags/v{{ solana_version }} - name: build solana become: true become_user: solana - shell: source /home/solana/.cargo/env && cargo build --release + shell: source /home/solana/.cargo/env && ./cargo build --release args: - chdir: {{ base_path }}/solana + chdir: /mnt/solana executable: /bin/bash diff --git a/tasks/keygen.yaml b/tasks/keygen.yaml index 2af4bac..28cf1e5 100644 --- a/tasks/keygen.yaml +++ b/tasks/keygen.yaml @@ -1,6 +1,6 @@ - name: create identity key become: true become_user: solana - shell: /home/solana/.local/share/solana/install/active_release/bin/solana-keygen new --no-bip39-passphrase -o /home/solana/rpc_node.json + shell: /mnt/solana/target/release/solana-keygen new --no-bip39-passphrase -o /home/solana/rpc_node.json args: - creates: /home/solana/rpc_node.json + creates: /home/solana/rpc_node.json \ No newline at end of file diff --git a/tasks/main.yaml b/tasks/main.yaml index 6904b22..76ad0bb 100644 --- a/tasks/main.yaml +++ b/tasks/main.yaml @@ -28,7 +28,6 @@ - name: git include_tasks: git.yaml - when: solana_installer|default(false)|bool == false - name: solana keygen include_tasks: keygen.yaml diff --git a/tasks/ramdisk.yaml b/tasks/ramdisk.yaml index f028431..5365bfb 100644 --- a/tasks/ramdisk.yaml +++ b/tasks/ramdisk.yaml @@ -1,7 +1,7 @@ - name: setup ramdisk for accountsdb become: true become_user: root - shell: mount -t tmpfs -o size={{ ramdisk_size }}G tmpfs {{ base_path }}/solana-accounts + shell: mount -t tmpfs -o size={{ ramdisk_size }}G tmpfs /mnt/solana-accounts when: ansible_memtotal_mb >= 1500*ramdisk_size - name: add ramdisk to fstab @@ -10,5 +10,5 @@ lineinfile: dest: /etc/fstab state: present - line: "tmpfs {{ base_path }}/solana-accounts tmpfs rw,size={{ ramdisk_size }}G,user=solana 0 0" + line: "tmpfs /mnt/solana-accounts tmpfs rw,size={{ ramdisk_size }}G,user=solana 0 0" when: ansible_memtotal_mb >= 1500*ramdisk_size diff --git a/tasks/rotate.yaml b/tasks/rotate.yaml index de91269..97ffbeb 100644 --- a/tasks/rotate.yaml +++ b/tasks/rotate.yaml @@ -16,7 +16,7 @@ blockinfile: path: /etc/logrotate.d/sol block: | - {{ base_path }}/logs/solana-validator.log { + /mnt/logs/solana-validator.log { rotate 7 daily missingok diff --git a/tasks/snapshot_downloader.yaml b/tasks/snapshot_downloader.yaml index ffea30b..8fc787b 100644 --- a/tasks/snapshot_downloader.yaml +++ b/tasks/snapshot_downloader.yaml @@ -10,4 +10,4 @@ - name: download latest snapshot become: true become_user: solana - shell: python3 {{ base_path }}/snapshot-finder.py --snapshot_path {{ base_path }}/solana-snapshots --version {{ solana_version }} + shell: python3 /mnt/snapshot-finder.py --snapshot_path /mnt/solana-snapshots diff --git a/tasks/swap.yaml b/tasks/swap.yaml index 305fa5d..a8799cf 100644 --- a/tasks/swap.yaml +++ b/tasks/swap.yaml @@ -2,13 +2,14 @@ become: true become_user: root command: - dd if=/dev/zero of={{ base_path }}/swapfile bs=1024 count={{ swap_mb }}k - creates="{{ base_path }}/swapfile" + dd if=/dev/zero of=/mnt/swapfile bs=1024 count={{ swap_mb }}k + creates="/mnt/swapfile" - name: Change swap file permissions become: true become_user: root - file: path="{{ base_path }}/swapfile" + file: + path=/mnt/swapfile owner=root group=root mode=0600 @@ -16,20 +17,20 @@ - name: Check swap file type become: true become_user: root - command: file {{ base_path }}/swapfile + command: file /mnt/swapfile register: swapfile - name: Make swap file become: true become_user: root - command: "sudo mkswap {{ base_path }}/swapfile" + command: "sudo mkswap /mnt/swapfile" when: swapfile.stdout.find('swap file') == -1 - name: Write swap entry in fstab become: true become_user: root mount: name=none - src={{ base_path }}/swapfile + src=/mnt/swapfile fstype=swap opts=sw passno=0 @@ -40,11 +41,11 @@ become: true become_user: root ignore_errors: yes - shell: swapon -show | grep {{ base_path }}/swapfile + shell: swapon -show | grep /mnt/swapfile register: swap_check - name: Mount swap become: true become_user: root - command: "swapon {{ base_path }}/swapfile" + command: "swapon /mnt/swapfile" when: swap_check.rc != 0 diff --git a/tasks/tuner.yaml b/tasks/tuner.yaml index 748d4e9..460dc25 100644 --- a/tasks/tuner.yaml +++ b/tasks/tuner.yaml @@ -57,12 +57,6 @@ become_user: root shell: | echo 'GOVERNOR="performance"' | tee /etc/default/cpufrequtils - systemctl disable ondemand + echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor args: executable: /bin/bash - -- name: Reboot host - become: true - become_user: root - reboot: - msg: Rebooting host... diff --git a/templates/sol.service b/templates/sol.service deleted file mode 100644 index 3d8a2c5..0000000 --- a/templates/sol.service +++ /dev/null @@ -1,18 +0,0 @@ -[Unit] -Description=Solana Validator -After=network.target -StartLimitIntervalSec=0 - -[Service] -Type=simple -Restart=on-failure -RestartSec=1 -LimitNOFILE=1000000 -LogRateLimitIntervalSec=0 -User=solana -Environment=PATH=/bin:/usr/bin:/home/solana/.local/share/solana/install/active_release/bin -Environment=SOLANA_METRICS_CONFIG=host=https://metrics.solana.com:8086,db=mainnet-beta,u=mainnet-beta_write,p=password -ExecStart=/home/solana/validator.sh - -[Install] -WantedBy=multi-user.target \ No newline at end of file diff --git a/templates/validator.j2 b/templates/validator.j2 deleted file mode 100644 index f6f3437..0000000 --- a/templates/validator.j2 +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash -PATH=/home/solana/.local/share/solana/install/active_release/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin -# export RUST_BACKTRACE=1 -# export RUST_LOG=error - -# Start solana rpc node, intended to be used with journalctl/systemctl. Logs will go to journalctl. -exec solana-validator --identity /home/solana/rpc_node.json --rpc-port 8899 --full-rpc-api --entrypoint entrypoint.mainnet-beta.solana.com:8001 --limit-ledger-size --log {{ base_path }}/logs/solana-validator.log --accounts {{ base_path }}/solana-accounts --ledger {{ base_path }}/solana-ledger --snapshots {{ base_path }}/solana-snapshots --no-snapshot-fetch \ No newline at end of file From a5f4bad01bb10983d022ef9ae6c8eeb31185dc6a Mon Sep 17 00:00:00 2001 From: dubbelosix Date: Sat, 14 Jan 2023 20:38:57 +0530 Subject: [PATCH 28/35] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d675d56..d304b09 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ screen -S sol ### Step 3: Install ansible ``` -sudo apt-get install ansible -y +sudo apt-get update && sudo apt-get install ansible -y ``` ### Step 4: Clone the autoclock-rpc repository From d08122e36d0c790b15f7e96a40f4b4e4f23abc4d Mon Sep 17 00:00:00 2001 From: dubbelosix Date: Sat, 14 Jan 2023 20:51:46 +0530 Subject: [PATCH 29/35] fix --- tasks/deps.yaml | 69 +++++++++++++++++++++++++++---------------------- tasks/git.yaml | 8 +++--- 2 files changed, 42 insertions(+), 35 deletions(-) diff --git a/tasks/deps.yaml b/tasks/deps.yaml index 2c0e544..0368ec6 100644 --- a/tasks/deps.yaml +++ b/tasks/deps.yaml @@ -4,42 +4,49 @@ apt: update_cache: yes pkg: - - build-essential - - pkg-config - - libudev-dev - - cmake - - libclang-dev - - libssl-dev - - gparted - - nload - - python3-pip - - net-tools - - logind - - curl - - git - - chrony - - htop - - acl - - cpufrequtils - -- name: Upgrade packages + - build-essential + - pkg-config + - libudev-dev + - cmake + - libclang-dev + - libssl-dev + - gparted + - nload + - python3-pip + - net-tools + - logind + - curl + - git + - chrony + - htop + - acl + +- name: check if cargo is installed become: true - become_user: root - apt: - upgrade: dist + become_user: solana + become_method: sudo + shell: test -f /home/solana/.cargo/env && source /home/solana/.cargo/env && command -v cargo + args: + executable: /bin/bash + register: cargo_exists + ignore_errors: yes -- name: install solana +- name: Download Installer + when: cargo_exists is failed become: true become_user: solana - shell: sh -c "$(curl -sSfL https://release.solana.com/v{{ solana_version }}/install)" - args: - chdir: /home/solana - when: solana_installer|default(true)|bool == true + become_method: sudo + get_url: + url: https://sh.rustup.rs + dest: /tmp/sh.rustup.rs + mode: '0755' + force: 'yes' -- name: add solana deps to path +- name: install rust/cargo become: true become_user: solana - shell: export PATH="/home/solana/.local/share/solana/install/active_release/bin:$PATH" + become_method: sudo + when: cargo_exists is failed + shell: /tmp/sh.rustup.rs -y args: - executable: /bin/bash - when: solana_installer|default(true)|bool == true + executable: /bin/bash \ No newline at end of file diff --git a/tasks/git.yaml b/tasks/git.yaml index bfddee7..a15c480 100644 --- a/tasks/git.yaml +++ b/tasks/git.yaml @@ -2,14 +2,14 @@ become: true become_user: solana git: - repo: "https://github.com/solana-labs/solana.git" + repo: 'https://github.com/solana-labs/solana.git' dest: /mnt/solana - version: tags/v{{ solana_version }} + version: tags/{{ solana_version }} - name: build solana become: true become_user: solana - shell: source /home/solana/.cargo/env && ./cargo build --release + shell: source /home/solana/.cargo/env && /mnt/solana/cargo build --release args: chdir: /mnt/solana - executable: /bin/bash + executable: /bin/bash \ No newline at end of file From 0cbf3c8cd625983037cb099a6e15ec9f92059d9b Mon Sep 17 00:00:00 2001 From: dubbelosix Date: Sat, 14 Jan 2023 20:55:09 +0530 Subject: [PATCH 30/35] prepend v --- defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/defaults/main.yml b/defaults/main.yml index a305749..d94735e 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -1,6 +1,6 @@ --- # defaults file for Solana RPC -solana_version: "1.13.5" +solana_version: "v1.13.5" raw_disk_list: ["/dev/nvme1n1", "/dev/nvme2n1"] setup_disks: "true" download_snapshot: "true" From b068d7e70fa5b06264d8089c855bea2eb218b868 Mon Sep 17 00:00:00 2001 From: dubbelosix Date: Sat, 14 Jan 2023 22:09:10 +0530 Subject: [PATCH 31/35] some changes --- defaults/main.yml | 2 +- tasks/tuner.yaml | 9 +++++++++ tasks/var_check.yaml | 6 ++++++ 3 files changed, 16 insertions(+), 1 deletion(-) diff --git a/defaults/main.yml b/defaults/main.yml index d94735e..d0100bb 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -4,6 +4,6 @@ solana_version: "v1.13.5" raw_disk_list: ["/dev/nvme1n1", "/dev/nvme2n1"] setup_disks: "true" download_snapshot: "true" -ramdisk_size: "300" +ramdisk_size: 300 swap_mb: "100000" diff --git a/tasks/tuner.yaml b/tasks/tuner.yaml index 460dc25..b82ad9f 100644 --- a/tasks/tuner.yaml +++ b/tasks/tuner.yaml @@ -60,3 +60,12 @@ echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor args: executable: /bin/bash + +- name: Set performance governor bare metal + become: true + become_user: root + shell: | + echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor + args: + executable: /bin/bash + ignore_errors: True diff --git a/tasks/var_check.yaml b/tasks/var_check.yaml index 99b129f..37a0384 100644 --- a/tasks/var_check.yaml +++ b/tasks/var_check.yaml @@ -11,3 +11,9 @@ with_items: - "{{ raw_disk_list }}" when: setup_disks is defined and (setup_disks | bool) + +- name: validate some variables + assert: + that: + - ((ramdisk_size is defined) and (ansible_memtotal_mb >= 1500*ramdisk_size) or ramdisk_size is not defined) + msg: machine ram is too low for ram diskq. comment out ramdisk_size in defaults/main.yml \ No newline at end of file From c028424b78f4c3362e928efd543b9d822d84c6fa Mon Sep 17 00:00:00 2001 From: dubbelosix Date: Sat, 14 Jan 2023 23:52:31 +0530 Subject: [PATCH 32/35] some fstab --- fstab.yaml | 7 ------- tasks/disks.yaml | 14 ++++++++++++++ tasks/file_setup.yaml | 25 ++++++++++++++++++++++--- tasks/user.yaml | 7 +++++++ 4 files changed, 43 insertions(+), 10 deletions(-) delete mode 100644 fstab.yaml diff --git a/fstab.yaml b/fstab.yaml deleted file mode 100644 index 71ca16d..0000000 --- a/fstab.yaml +++ /dev/null @@ -1,7 +0,0 @@ -- name: Add the below lines - blockinfile: - path: /etc/fstab - state: present - block: | - /dev/md0 /mnt ext4 defaults,nofail,discard 0 0 - /mnt/swapfile swap swap defaults 0 0 diff --git a/tasks/disks.yaml b/tasks/disks.yaml index 093f81a..b5de48b 100644 --- a/tasks/disks.yaml +++ b/tasks/disks.yaml @@ -52,3 +52,17 @@ become_user: root shell: mount /dev/{{ raid_name.stdout }} /mnt when: mount_mnt.rc != 0 + +- name: extract raid params + shell: sudo blkid | grep md | grep -oh \".*\" | sed -e 's/"//g' | awk '{print "UUID="$1" /mnt ext4 defaults 1 1"}' + register: raid_config + +- name: add raid to fstab + become: true + become_user: root + lineinfile: + dest: /etc/fstab + state: present + line: "{{ raid_config.stdout_lines[0] }}" + insertbefore: 'mnt' + firstmatch: yes diff --git a/tasks/file_setup.yaml b/tasks/file_setup.yaml index f9a8cf7..dc7e3c0 100644 --- a/tasks/file_setup.yaml +++ b/tasks/file_setup.yaml @@ -37,11 +37,30 @@ owner: solana group: solana -- name: copy snapshot-finder.py +- name: copy snapcheck.py + become: true + become_user: solana + copy: + src: snapcheck.py + dest: /mnt/snapcheck.py + owner: solana + group: solana + +- name: copy montip.py become: true become_user: solana template: - src: snapshot-finder.py - dest: /mnt/snapshot-finder.py + src: montip.py + dest: /mnt/montip.py owner: solana group: solana + +- name: solana profile + become: true + become_user: root + copy: + src: zsolpath.sh + dest: /etc/profile.d/zsolpath.sh + owner: root + group: root + mode: "0755" diff --git a/tasks/user.yaml b/tasks/user.yaml index 7ad27d8..bb1a511 100644 --- a/tasks/user.yaml +++ b/tasks/user.yaml @@ -23,3 +23,10 @@ regexp: '^solana' line: 'solana ALL=(ALL) NOPASSWD: ALL' validate: 'visudo -cf %s' + +- name: Ensure solana build is in path for solana user + become: true + become_user: root + lineinfile: + path: /home/solana/.bashrc + line: 'export PATH=/mnt/solana/target/release:$PATH' From 06f5ff6948bf507cc6967ad0ec1c1164af81d871 Mon Sep 17 00:00:00 2001 From: dubbelosix Date: Sat, 14 Jan 2023 23:55:55 +0530 Subject: [PATCH 33/35] some more changes --- tasks/file_setup.yaml | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/tasks/file_setup.yaml b/tasks/file_setup.yaml index dc7e3c0..f173149 100644 --- a/tasks/file_setup.yaml +++ b/tasks/file_setup.yaml @@ -37,30 +37,21 @@ owner: solana group: solana -- name: copy snapcheck.py +- name: copy snapshot-finder.py become: true become_user: solana copy: - src: snapcheck.py - dest: /mnt/snapcheck.py + src: snapshot-finder.py + dest: /mnt/snapshot-finder.py owner: solana group: solana - name: copy montip.py become: true become_user: solana - template: + copy: src: montip.py dest: /mnt/montip.py owner: solana group: solana -- name: solana profile - become: true - become_user: root - copy: - src: zsolpath.sh - dest: /etc/profile.d/zsolpath.sh - owner: root - group: root - mode: "0755" From 86576e22dc071cfa1015923b8b4202ee07179a21 Mon Sep 17 00:00:00 2001 From: dubbelosix Date: Sun, 15 Jan 2023 00:04:51 +0530 Subject: [PATCH 34/35] download_start.sh --- files/download_start.sh | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 files/download_start.sh diff --git a/files/download_start.sh b/files/download_start.sh new file mode 100644 index 0000000..fc33020 --- /dev/null +++ b/files/download_start.sh @@ -0,0 +1,4 @@ +#!/bin/bash +sudo systemctl stop sol.service +python3 /mnt/snapshot-finder.py --snapshot_path /mnt/solana-snapshots +sudo systemctl start sol.service \ No newline at end of file From b555fa45f5db7815158ee1ed736b9a8a25da76fe Mon Sep 17 00:00:00 2001 From: dubbelosix Date: Sun, 15 Jan 2023 00:05:34 +0530 Subject: [PATCH 35/35] ansible options --- tasks/file_setup.yaml | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tasks/file_setup.yaml b/tasks/file_setup.yaml index f173149..0fb5b90 100644 --- a/tasks/file_setup.yaml +++ b/tasks/file_setup.yaml @@ -27,6 +27,16 @@ owner: solana group: solana mode: "0755" + +- name: copy restart.sh + become: true + become_user: solana + copy: + src: download_restart.sh + dest: /home/solana/download_restart.sh + owner: solana + group: solana + mode: "0755" - name: copy snapcheck.py become: true