diff --git a/README.md b/README.md index c1529c08..eae9ad0b 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # heavy_script +## Website + +[HeavySetup - Further In-Depth Explanation](https://heavysetup.info/scripts/heavyscript/about/) + ## Table of contents: * [Arguments](#arguments) * [Examples](#examples) @@ -12,23 +16,24 @@ ## Arguments -| Flag | Example | Parameter | Description | -|----------------- |------------------------ |----------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| --self-update | --self-update | None | Updates HeavyScript prior to running it
_You no longer need to git pull_ | -| --delete-backup | --delete-backup | None | Opens a menu to delete backups
_Useful if you need to delete old system backups or backups from other scripts_ | -| --restore | --restore | None | Restore HeavyScript specific `ix-applications dataset` snapshot | -| --mount | --mount | None | Initiates mounting feature
Choose between unmounting and mounting PVC data | -| --dns | --dns | None | list all of your applications DNS names and their web ports | -| -U | -U | None | Update applications, ignoring major version changes | -| -u | -u | None | Update applications, do NOT update if there was a major version change | -| -b | -b 14 | Integer | Backup `ix-appliactions` dataset
_Creates backups up to the number you've chosen_ | -| -i | -i nextcloud -i sonarr | String | Applications listed will be ignored during updating
_List one application after another as shown in the example_ | -| (-R\|-r) | -r | None | Monitors applications after they update
If the app does not become "ACTIVE" after either:
The custom Timeout, or Default Timeout,
rollback the application.
__Warning: deprecating `-R` please begin using `-r` instead__ | -| -v | -v | None | Verbose Output
_Look at the bottom of this page for an example_ | -| -S | -S | None | Shutdown the application prior to updating it | -| -t | -t 150 | Integer | Set a custom timeout to be used with either:
`-m`
_Time the script will wait for application to be "STOPPED"_
or
`-(u\|U)`
_Time the script will wait for application to be either "STOPPED" or "ACTIVE"_ | -| -s | -s | None | Sync Catalogs prior to updating | -| -p | -p | None | Prune old/unused docker images | +| Flag | Example | Parameter | Description | +|----------------- |------------------------ |----------------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| NULL | NULL | NULL | If you choose not to supply an option, it will open the menu for easier access to the utilities | +| --self-update | --self-update | None | Updates HeavyScript prior to running it
_You no longer need to git pull_ | +| --delete-backup | --delete-backup | None | Opens a menu to delete backups
_Useful if you need to delete old system backups or backups from other scripts_ | +| --restore | --restore | None | Restore HeavyScript specific `ix-applications dataset` snapshot | +| --mount | --mount | None | Initiates mounting feature
Choose between unmounting and mounting PVC data | +| --dns | --dns | None | list all of your applications DNS names and their web ports | +| -U | -U
-U 5 | None or Integer | Update applications, ignoring major version changes
_Optionally, you can supply a number after the argument to update multiple applications at once_ | +| -u | -u
-u 5 | None or Integer | Update applications, do NOT update if there was a major version change
_Optionally, you can supply a number after the argument to update multiple applications at once_ | +| -b | -b 14 | Integer | Backup `ix-appliactions` dataset
_Creates backups up to the number you've chosen_ | +| -i | -i nextcloud -i sonarr | String | Applications listed will be ignored during updating
_List one application after another as shown in the example_ | +| -r | -r | None | Monitors applications after they update
If the app does not become "ACTIVE" after either:
The custom Timeout, or Default Timeout,
rollback the application. | +| -v | -v | None | Verbose Output
_Look at the bottom of this page for an example_ | +| -S | -S | None | Shutdown the application prior to updating it | +| -t | -t 150 | Integer | Set a custom timeout to be used with either:
`-m`
_Time the script will wait for application to be "STOPPED"_
or
`-(u\|U)`
_Time the script will wait for application to be either "STOPPED" or "ACTIVE"_ | +| -s | -s | None | Sync Catalogs prior to updating | +| -p | -p | None | Prune old/unused docker images |
@@ -37,7 +42,9 @@ ### Examples #### Typical Cron Job ``` -bash heavy_script.sh --self-update -b 14 -i portainer -i arch -i sonarr -i radarr -t 600 -rsup + +bash heavy_script.sh --self-update -b 14 -i portainer -i arch -i sonarr -i radarr -t 600 -rsp -u 5 + ``` > `-b` is set to 14. Up to 14 snapshots of your ix-applications dataset will be saved @@ -50,10 +57,13 @@ bash heavy_script.sh --self-update -b 14 -i portainer -i arch -i sonarr -i radar > `-s` will just sync the repositories, ensuring you are downloading the latest updates. -> `-u` update applications as long as the major version has absolutely no change, if it does have a change it will ask the user to update manually. - > `-p` Prune docker images. +> `-u` update applications as long as the major version has absolutely no change, if it does have a change it will ask the user to update manually. +>> The `5` after the `-u` means up to 5 applications will be updating and monitored at one time + +> `--self-update` Will update the script prior to running anything else. + > `--self-update` Will update the script prior to running anything else. #### Mounting PVC Data @@ -82,7 +92,9 @@ bash /mnt/tank/scripts/heavy_script/heavy_script.sh --dns #### My personal Cron Job ``` -bash /mnt/speed/scripts/heavy_script/heavy_script.sh --self-update -b 14 -rsup + +bash /mnt/speed/scripts/heavy_script/heavy_script.sh --self-update -b 14 -rsp -u 10 + ```
@@ -180,4 +192,3 @@ bash heavyscript.sh --self-update -b 14 -supr | ![image](https://user-images.githubusercontent.com/20793231/167971188-07f71d02-8da3-4e0c-b9a0-cd26e7f63613.png) | ![image](https://user-images.githubusercontent.com/20793231/167972033-dc8d4ab4-4fb2-4c8a-b7dc-b9311ae55cf8.png) | - diff --git a/functions/backup.sh b/functions/backup.sh new file mode 100644 index 00000000..9561b5d6 --- /dev/null +++ b/functions/backup.sh @@ -0,0 +1,176 @@ +#!/bin/bash + + +backup(){ +echo_backup+=("\nšŸ„± šŸ„° šŸ„² šŸ„ŗ šŸ…„ šŸ„æ šŸ…‚") +echo_backup+=("Number of backups was set to $number_of_backups") +date=$(date '+%Y_%m_%d_%H_%M_%S') +[[ "$verbose" == "true" ]] && cli -c 'app kubernetes backup_chart_releases backup_name=''"'HeavyScript_"$date"'"' &> /dev/null && echo_backup+=(HeavyScript_"$date") +[[ -z "$verbose" ]] && echo_backup+=("\nNew Backup Name:") && cli -c 'app kubernetes backup_chart_releases backup_name=''"'HeavyScript_"$date"'"' | tail -n 1 &> /dev/null && echo_backup+=(HeavyScript_"$date") +mapfile -t list_backups < <(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -Vr -k2,7 | awk -F '|' '{print $2}'| tr -d " \t\r") +if [[ ${#list_backups[@]} -gt "$number_of_backups" ]]; then + echo_backup+=("\nDeleted the oldest backup(s) for exceeding limit:") + overflow=$(( ${#list_backups[@]} - "$number_of_backups" )) + mapfile -t list_overflow < <(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -V -k2,7 | awk -F '|' '{print $2}'| tr -d " \t\r" | head -n "$overflow") + for i in "${list_overflow[@]}" + do + cli -c 'app kubernetes delete_backup backup_name=''"'"$i"'"' &> /dev/null || echo_backup+=("Failed to delete $i") + echo_backup+=("$i") + done +fi + +#Dump the echo_array, ensures all output is in a neat order. +for i in "${echo_backup[@]}" +do + echo -e "$i" +done +} +export -f backup + + + +deleteBackup(){ +while true +do + clear -x && echo "pulling all restore points.." + list_backups=$(cli -c 'app kubernetes list_backups' | sort -t '_' -Vr -k2,7 | tr -d " \t\r" | awk -F '|' '{print $2}' | nl -s ") " | column -t) + if [[ -z "$list_backups" ]]; then + echo "No restore points available" + exit + fi + while true + do + clear -x + title + echo -e "Choose a Restore Point to Delete\nThese may be out of order if they are not HeavyScript backups" + echo "$list_backups" + echo + echo "0) Exit" + read -rt 120 -p "Please type a number: " selection + restore_point=$(echo "$list_backups" | grep ^"$selection)" | awk '{print $2}') + if [[ $selection == 0 ]]; then + echo "Exiting.." + exit + elif [[ -z "$selection" ]]; then + echo "Your selection cannot be empty" + sleep 3 + continue + elif [[ -z "$restore_point" ]]; then + echo "Invalid Selection: $selection, was not an option" + sleep 3 + continue + fi + break + done + while true + do + clear -x + echo -e "\nWARNING:\nYou CANNOT go back after deleting your restore point" + echo -e "\n\nYou have chosen:\n$restore_point\n\nWould you like to continue?" + echo -e "1) Yes\n2) Exit\n" + read -rt 120 -p "Please type a number: " yesno + case $yesno in + 1) + echo -e "\nDeleting $restore_point" + cli -c 'app kubernetes delete_backup backup_name=''"'"$restore_point"'"' &>/dev/null || { echo "Failed to delete backup.."; exit; } + echo "Sucessfully deleted" + break + ;; + 2) + echo "Exiting" + exit + ;; + *) + echo "That was not an option, try again" + sleep 3 + continue + ;; + esac + done + while true + do + echo "Delete more?" + echo "1) Yes" + echo "2) No" + read -rt 120 -p "Please type a number: " yesno + case $yesno in + 1) + break + ;; + 2) + exit + ;; + *) + echo "$yesno was not an option, try again" + sleep 2 + continue + ;; + + esac + + done +done +} +export -f deleteBackup + + +restore(){ +while true +do + clear -x && echo "pulling restore points.." + list_backups=$(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -Vr -k2,7 | tr -d " \t\r" | awk -F '|' '{print $2}' | nl -s ") " | column -t) + if [[ -z "$list_backups" ]]; then + echo "No HeavyScript restore points available" + exit + fi + while true + do + clear -x + title + echo "Choose a Restore Point" + echo "$list_backups" + echo + echo "0) Exit" + read -rt 120 -p "Please type a number: " selection + restore_point=$(echo "$list_backups" | grep ^"$selection)" | awk '{print $2}') + if [[ $selection == 0 ]]; then + echo "Exiting.." + exit + elif [[ -z "$selection" ]]; then + echo "Your selection cannot be empty" + sleep 3 + continue + elif [[ -z "$restore_point" ]]; then + echo "Invalid Selection: $selection, was not an option" + sleep 3 + continue + fi + break + done + while true + do + clear -x + echo -e "\nWARNING:\nThis is NOT guranteed to work\nThis is ONLY supposed to be used as a LAST RESORT\nConsider rolling back your applications instead if possible" + echo -e "\n\nYou have chosen:\n$restore_point\n\nWould you like to continue?" + echo -e "1) Yes\n2) Exit\n" + read -rt 120 -p "Please type a number: " yesno + case $yesno in + 1) + echo -e "\nStarting Backup, this will take a LONG time." + cli -c 'app kubernetes restore_backup backup_name=''"'"$restore_point"'"' || { echo "Failed to delete backup.."; exit; } + exit + ;; + 2) + echo "Exiting" + exit + ;; + *) + echo "That was not an option, try again" + sleep 3 + continue + ;; + esac + done +done +} +export -f restore \ No newline at end of file diff --git a/functions/dns.sh b/functions/dns.sh new file mode 100644 index 00000000..c4316139 --- /dev/null +++ b/functions/dns.sh @@ -0,0 +1,27 @@ +#!/bin/bash + + +dns(){ +clear -x +echo "Generating DNS Names.." +#ignored dependency pods, may need to add more in the future. +dep_ignore="\-cronjob\-|^kube-system|\ssvclb|NAME|\-memcached\-.[^custom\-app]|\-postgresql\-.[^custom\-app]|\-redis\-.[^custom\-app]|\-mariadb\-.[^custom\-app]|\-promtail\-.[^custom\-app]" + +# Pulling pod names +mapfile -t main < <(k3s kubectl get pods -A | grep -Ev "$dep_ignore" | sort) + +# Pulling all ports +all_ports=$(k3s kubectl get service -A) + +clear -x +count=0 +for i in "${main[@]}" +do + [[ count -le 0 ]] && echo -e "\n" && ((count++)) + appName=$(echo "$i" | awk '{print $2}' | sed 's/-[^-]*-[^-]*$//' | sed 's/-0//') + ixName=$(echo "$i" | awk '{print $1}') + port=$(echo "$all_ports" | grep -E "\s$appName\s" | awk '{print $6}' | grep -Eo "^[[:digit:]]+{1}") + echo -e "$appName.$ixName.svc.cluster.local $port" +done | uniq | nl -b t | sed 's/\s\s\s$/- -------- ----/' | column -t -R 1 -N "#,DNS_Name,Port" -L +} +export -f dns \ No newline at end of file diff --git a/functions/menu.sh b/functions/menu.sh new file mode 100644 index 00000000..8330cc33 --- /dev/null +++ b/functions/menu.sh @@ -0,0 +1,218 @@ +#!/bin/bash + +menu(){ +script=$(readlink -f "$0") +script_path=$(dirname "$script") +script_name="heavy_script.sh" +cd "$script_path" || exit +clear -x +title +echo "1) Help" +echo "2) List DNS Names" +echo "3) Mount and Unmount PVC storage" +echo "4) Create a Backup" +echo "5) Restore a Backup" +echo "6) Delete a Backup" +echo "7) Update HeavyScript" +echo "8) Update Applications" +echo +echo "0) Exit" +read -rt 600 -p "Please select an option by number: " selection + +case $selection in + 0) + exit + ;; + 1) + help="true" + ;; + 2) + dns="true" + ;; + 3) + mount="true" + ;; + 4) + read -rt 600 -p "What is the maximun number of backups you would like?: " number_of_backups + backup="true" + ;; + 5) + restore="true" + ;; + 6) + deleteBackup="true" + ;; + 7) + self_update="true" + ;; + 8) + while true + do + clear -x + title + echo "Choose Your Update Type" + echo "-----------------------" + echo "1) -U | Update all applications, ignores versions" + echo "2) -u | Update all applications, does not update Major releases" + echo + echo "0) Exit" + echo + read -rt 600 -p "Please type the number associated with the flag above: " current_selection + if [[ $current_selection == 1 ]]; then + while true + do + echo -e "\nHow many applications do you want updating at the same time?" + read -rt 600 -p "Please type an integer greater than 0: " up_async + if [[ $up_async == 0 ]]; then + echo "Error: \"$up_async\" is less than 1" + echo "NOT adding it to the list" + sleep 5 + continue + elif ! [[ $up_async =~ ^[0-9]+$ ]]; then + echo "Error: \"$up_async\" is invalid, it needs to be an integer" + echo "NOT adding it to the list" + sleep 5 + continue + else + update_selection+=("-U" "$up_async") + break + fi + done + break + elif [[ $current_selection == 2 ]]; then + while true + do + echo -e "\nHow many applications do you want updating at the same time?" + read -rt 600 -p "Please type an integer greater than 0: " up_async + if [[ $up_async == 0 ]]; then + echo "Error: \"$up_async\" is less than 1" + echo "NOT adding it to the list" + sleep 5 + continue + elif ! [[ $up_async =~ ^[0-9]+$ ]]; then + echo "Error: \"$up_async\" is invalid, it needs to be an integer" + echo "NOT adding it to the list" + sleep 5 + continue + else + update_selection+=("-u" "$up_async") + break + fi + done + break + elif [[ $current_selection == 0 ]]; then + echo "Exiting.." + exit + else + echo "$current_selection was not an option, try again" && sleep 5 + continue + fi + done + while true + do + clear -x + title + echo "Choose Your Update Options" + echo "--------------------------" + echo "1) -b | Back-up your ix-applications dataset, specify a number after -b" + echo "2) -i | Add application to ignore list, one by one, see example below." + echo "3) -r | Roll-back applications if they fail to update" + echo "4) -S | Shutdown applications prior to updating" + echo "5) -v | verbose output" + echo "6) -t | Set a custom timeout in seconds when checking if either an App or Mountpoint correctly Started, Stopped or (un)Mounted. Defaults to 500 seconds" + echo "7) -s | sync catalog" + echo "8) -p | Prune unused/old docker images" + echo "9) --self-update | Updates HeavyScript prior to running any other commands" + echo + echo "99) Remove Update Options, Restart" + echo "00) Done making selections, proceed with update" + echo + echo "0) Exit" + echo + echo "Current Choices" + echo "---------------" + echo "bash heavy_script.sh ${update_selection[*]}" + echo + read -rt 600 -p "Type the Number OR Flag: " current_selection + case $current_selection in + 0) + echo "Exiting.." + exit + ;; + 00) + clear -x + echo "Running \"bash heavy_script.sh ${update_selection[*]}\"" + echo + exec bash "$script_name" "${update_selection[@]}" + exit + ;; + 1 | -b) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-b" && echo -e "\"-b\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it + echo "Up to how many backups should we keep?" + read -rt 600 -p "Please type an integer: " up_backups + ! [[ $up_backups =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_backups\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 5 && continue + [[ $up_backups == 0 ]] && echo -e "Error: Number of backups cannot be 0\nNOT adding it to the list" && sleep 5 && continue + update_selection+=("-b" "$up_backups") + ;; + 2 | -i) + read -rt 600 -p "What is the name of the application we should ignore?: " up_ignore + ! [[ $up_ignore =~ ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ ]] && echo -e "Error: \"$up_ignore\" is not a possible option for an application name" && sleep 5 && continue + update_selection+=("-i" "$up_ignore") + ;; + 3 | -r) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-r" && echo -e "\"-r\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it + update_selection+=("-r") + + ;; + 4 | -S) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-S" && echo -e "\"-S\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it + update_selection+=("-S") + ;; + 5 | -v) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-v" && echo -e "\"-v\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it + update_selection+=("-v") + ;; + 6 | -t) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-t" && echo -e "\"-t\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it + echo "What do you want your timeout to be?" + read -rt 600 -p "Please type an integer: " up_timeout + ! [[ $up_timeout =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_timeout\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 5 && continue + update_selection+=("-t" "$up_timeout") + ;; + 7 | -s) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-s" && echo -e "\"-s\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it + update_selection+=("-s") + ;; + 8 | -p) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-p" && echo -e "\"-p\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it + update_selection+=("-p") + ;; + 9 | --self-update ) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "--self-update" && echo -e "\"--self-update\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it + update_selection+=("--self-update") + ;; + 99) + count=2 + echo "restarting" + for i in "${update_selection[@]:2}" + do + unset "update_selection[$count]" + echo "$i removed" + ((count++)) + done + sleep 5 + continue + ;; + *) + echo "\"$current_selection\" was not an option, try again" && sleep 5 && continue + ;; + esac + done + ;; + *) + echo "\"$selection\" was not an option, please try agian" && sleep 5 && menu + ;; +esac +echo +} +export -f menu \ No newline at end of file diff --git a/functions/misc.sh b/functions/misc.sh new file mode 100644 index 00000000..880de232 --- /dev/null +++ b/functions/misc.sh @@ -0,0 +1,81 @@ +#!/bin/bash + + +sync(){ +echo_sync+=("\n\nšŸ…‚ šŸ…ˆ šŸ„½ šŸ„²") +cli -c 'app catalog sync_all' &> /dev/null && echo_sync+=("Catalog sync complete") + +#Dump the echo_array, ensures all output is in a neat order. +for i in "${echo_sync[@]}" +do + echo -e "$i" +done +} +export -f sync + +prune(){ +echo -e "\n\nšŸ„æ šŸ… šŸ…„ šŸ„½ šŸ„“" +echo "Pruned Docker Images" +docker image prune -af | grep "^Total" || echo "Failed to Prune Docker Images" +} +export -f prune + +title(){ +echo ' _ _ _____ _ _ ' +echo '| | | | / ___| (_) | | ' +echo '| |_| | ___ __ ___ ___ _\ `--. ___ _ __ _ _ __ | |_' +echo "| _ |/ _ \/ _\` \ \ / / | | |\`--. \/ __| '__| | '_ \| __|" +echo '| | | | __/ (_| |\ V /| |_| /\__/ / (__| | | | |_) | |_ ' +echo '\_| |_/\___|\__,_| \_/ \__, \____/ \___|_| |_| .__/ \__|' +echo ' __/ | | | ' +echo ' |___/ |_| ' +echo +} +export -f title + +help(){ +[[ $help == "true" ]] && clear -x + +echo "Access the HeavyScript Menu" +echo "---------------------------" +echo "(Just dont send any other argument)" +echo "bash heavy_script.sh" +echo +echo "Basic Utilities" +echo "---------------" +echo "--mount | Initiates mounting feature, choose between unmounting and mounting PVC data" +echo "--restore | Opens a menu to restore a \"heavy_script\" backup that was taken on your \"ix-applications\" dataset" +echo "--delete-backup | Opens a menu to delete backups on your system" +echo "--dns | list all of your applications DNS names and their web ports" +echo +echo "Update Options" +echo "--------------" +echo "-U | Update all applications, ignores versions" +echo "-U 5 | Same as above, but updates 5 applications at one time" +echo "-u | Update all applications, does not update Major releases" +echo "-u 5 | Same as above, but updates 5 applications at one time" +echo +echo "Additional Update Options" +echo "-------------------------" +echo "-b 14 | Back-up your ix-applications dataset, specify a number after -b" +echo "-i | Add application to ignore list, one by one, see example below." +echo "-r | Roll-back applications if they fail to update" +echo "-S | Shutdown applications prior to updating" +echo "-v | verbose output" +echo "-t 500| Set a custom timeout in seconds when checking if either an App or Mountpoint correctly Started, Stopped or (un)Mounted. Defaults to 500 seconds" +echo "-s | sync catalog" +echo "-p | Prune unused/old docker images" +echo +echo "Examples" +echo "--------" +echo "bash heavy_script.sh" +echo "bash heavy_script.sh -b 14 -i portainer -i arch -i sonarr -t 600 -vrsUp" +echo "bash heavy_script.sh -b 14 -i portainer -i arch -i sonarr -t 600 -vrsp -U 10" +echo "bash /mnt/tank/scripts/heavy_script.sh -t 150 --mount" +echo "bash /mnt/tank/scripts/heavy_script.sh --dns" +echo "bash heavy_script.sh --restore" +echo "bash /mnt/tank/scripts/heavy_script.sh --delete-backup" +echo +exit +} +export -f help \ No newline at end of file diff --git a/functions/mount.sh b/functions/mount.sh new file mode 100644 index 00000000..e574fcac --- /dev/null +++ b/functions/mount.sh @@ -0,0 +1,114 @@ +#!/bin/bash + +mount(){ +while true +do + clear -x + title + echo "1) Mount" + echo "2) Unmount All" + echo + echo "0) Exit" + read -rt 120 -p "Unmount All Please type a number: " selection + case $selection in + 0) + echo "Exiting.." + exit + ;; + 1) + call=$(k3s kubectl get pvc -A | sort -u | awk '{print $1 "\t" $2 "\t" $4}' | sed "s/^0/ /") + mount_list=$(echo "$call" | sed 1d | nl -s ") ") + mount_title=$(echo "$call" | head -n 1) + list=$(echo -e "# $mount_title\n$mount_list" | column -t) + while true + do + clear -x + title + echo "$list" + echo + echo "0) Exit" + read -rt 120 -p "Please type a number: " selection + [[ $selection == 0 ]] && echo "Exiting.." && exit + app=$(echo -e "$list" | grep ^"$selection)" | awk '{print $2}' | cut -c 4- ) + [[ -z "$app" ]] && echo "Invalid Selection: $selection, was not an option" && sleep 3 && continue #Check for valid selection. If none, contiue + pvc=$(echo -e "$list" | grep ^"$selection)") + status=$(cli -m csv -c 'app chart_release query name,status' | grep -E "^$app\b" | awk -F ',' '{print $2}'| tr -d " \t\n\r") + if [[ "$status" != "STOPPED" ]]; then + [[ -z $timeout ]] && echo -e "\nDefault Timeout: 500" && timeout=500 || echo -e "\nCustom Timeout: $timeout" + SECONDS=0 && echo -e "\nScaling down $app" && midclt call chart.release.scale "$app" '{"replica_count": 0}' &> /dev/null + else + echo -e "\n$app is already stopped" + fi + while [[ "$SECONDS" -le "$timeout" && "$status" != "STOPPED" ]] + do + status=$(cli -m csv -c 'app chart_release query name,status' | grep -E "^$app\b" | awk -F ',' '{print $2}'| tr -d " \t\n\r") + echo -e "Waiting $((timeout-SECONDS)) more seconds for $app to be STOPPED" && sleep 5 + done + data_name=$(echo "$pvc" | awk '{print $3}') + mount=$(echo "$pvc" | awk '{print $4}') + volume_name=$(echo "$pvc" | awk '{print $4}') + mapfile -t full_path < <(zfs list | grep "$volume_name" | awk '{print $1}') + if [[ "${#full_path[@]}" -gt 1 ]]; then #if there is another app with the same name on another pool, use the current pools application, since the other instance is probably old, or unused, or a backup. + echo "$app is using the same volume identifier on more than one pool.. attempting to use your current kubernetes apps pool" + pool=$(cli -c 'app kubernetes config' | grep -E "dataset\s\|" | awk -F '|' '{print $3}' | awk -F '/' '{print $1}' | tr -d " \t\n\r") + full_path=$(zfs list | grep "$volume_name" | grep "$pool" | awk '{print $1}') + fi + echo -e "\nMounting\n$full_path\nTo\n/mnt/heavyscript/$data_name" + zfs set mountpoint=/heavyscript/"$data_name" "$full_path" || echo "Failed to mount $app" + echo -e "Mounted\n\nUnmount with:\nzfs set mountpoint=legacy $full_path && rmdir /mnt/heavyscript/$data_name\n\nOr use the Unmount All option\n" + while true + do + echo -e "\nWould you like to mount anything else?" + echo "1) Yes" + echo "2) No" + read -rt 120 -p "Please type a number: " yesno + case $yesno in + 1) + clear -x + title + break + ;; + 2) + exit + ;; + *) + echo "Invalid selection \"$yesno\" was not an option" + sleep 2 + continue + ;; + esac + done + done + ;; + 2) + mapfile -t unmount_array < <(basename -a /mnt/heavyscript/* | sed "s/*//") + [[ -z ${unmount_array[*]} ]] && echo "Theres nothing to unmount" && sleep 3 && continue + for i in "${unmount_array[@]}" + do + main=$(k3s kubectl get pvc -A | grep -E "\s$i\s" | awk '{print $1, $2, $4}') + app=$(echo "$main" | awk '{print $1}' | cut -c 4-) + pvc=$(echo "$main" | awk '{print $3}') + mapfile -t path < <(find /mnt/*/ix-applications/releases/"$app"/volumes/ -maxdepth 0 | cut -c 6-) + if [[ "${#path[@]}" -gt 1 ]]; then #if there is another app with the same name on another pool, use the current pools application, since the other instance is probably old, or unused, or a backup. + echo "$i is a name used on more than one pool.. attempting to use your current kubernetes apps pool" + pool=$(cli -c 'app kubernetes config' | grep -E "dataset\s\|" | awk -F '|' '{print $3}' | awk -F '/' '{print $1}' | tr -d " \t\n\r") + full_path=$(find /mnt/"$pool"/ix-applications/releases/"$app"/volumes/ -maxdepth 0 | cut -c 6-) + zfs set mountpoint=legacy "$full_path""$pvc" + echo "$i unmounted" && rmdir /mnt/heavyscript/"$i" || echo "failed to unmount $i" + else + zfs set mountpoint=legacy "$path""$pvc" + echo "$i unmounted" && rmdir /mnt/heavyscript/"$i" || echo "failed to unmount $i" + fi + done + rmdir /mnt/heavyscript + sleep 2 + ;; + *) + echo "Invalid selection, \"$selection\" was not an option" + sleep 2 + continue + ;; + esac +done +} +export -f mount \ No newline at end of file diff --git a/functions/self_update.sh b/functions/self_update.sh new file mode 100644 index 00000000..586cadc9 --- /dev/null +++ b/functions/self_update.sh @@ -0,0 +1,28 @@ +#!/bin/bash + +args=("$@") +self_update() { +git fetch &> /dev/null +echo "šŸ…‚ šŸ„“ šŸ„» šŸ„µ šŸ…„ šŸ„æ šŸ„³ šŸ„° šŸ…ƒ šŸ„“" +# TODO: change beta to main once testing is complete +if git diff --name-only origin/beta | grep -qs ".sh" ; then + echo "Found a new version of HeavyScript, updating myself..." + git reset --hard -q + git pull --force -q + count=0 + for i in "${args[@]}" + do + [[ "$i" == "--self-update" ]] && unset "args[$count]" && break + ((count++)) + done + [[ -z ${args[*]} ]] && echo -e "No more arguments, exiting..\n" && exit + echo -e "Running the new version...\n" + sleep 5 + exec bash "$script_name" "${args[@]}" + # Now exit this old instance + exit +else + echo -e "HeavyScript is already the latest version\n" +fi +} +export -f self_update \ No newline at end of file diff --git a/functions/update_apps.sh b/functions/update_apps.sh new file mode 100644 index 00000000..be33626f --- /dev/null +++ b/functions/update_apps.sh @@ -0,0 +1,170 @@ +#!/bin/bash + + +commander(){ +mapfile -t array < <(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,container_images_update_available,status' | tr -d " \t\r" | grep -E ",true($|,)" | sort) +echo -e "\n\nšŸ…„ šŸ„æ šŸ„³ šŸ„° šŸ…ƒ šŸ„“ šŸ…‚" +[[ -z ${array[*]} ]] && echo "There are no updates available" && return 0 || echo "Update(s) Available: ${#array[@]}" +echo "Asynchronous Updates: $update_limit" +[[ -z $timeout ]] && echo "Default Timeout: 500" && timeout=500 || echo "Custom Timeout: $timeout" +[[ "$timeout" -le 120 ]] && echo "Warning: Your timeout is set low and may lead to premature rollbacks or skips" + + +it=0 +while true +do + while_status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status') + echo "$while_status" > temp.txt + proc_count=${#processes[@]} + count=0 + for proc in "${processes[@]}" + do + kill -0 "$proc" &> /dev/null || { unset "processes[$count]"; ((proc_count--)); } + ((count++)) + done + if [[ "$proc_count" -ge "$update_limit" ]]; then + sleep 3 + elif [[ $it -lt ${#array[@]} ]]; then + update_apps "${array[$it]}" & + processes+=($!) + ((it++)) + elif [[ $proc_count != 0 ]]; then # Wait for all processes to finish + sleep 3 + else # All processes must be completed, break out of loop + break + fi +done +rm temp.txt + +} +export -f commander + + +update_apps(){ +app_name=$(echo "${array[$it]}" | awk -F ',' '{print $1}') #print out first catagory, name. +printf '%s\0' "${ignore[@]}" | grep -iFxqz "${app_name}" && echo -e "\n$app_name\nIgnored, skipping" && return 0 #If application is on ignore list, skip +old_app_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #previous/current Application MAJOR Version +new_app_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #new Application MAJOR Version +old_chart_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}' | awk -F '.' '{print $1}') # Old Chart MAJOR version +new_chart_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}' | awk -F '_' '{print $2}' | awk -F '.' '{print $1}') # New Chart MAJOR version +startstatus=$(echo "${array[$it]}" | awk -F ',' '{print $2}') #status of the app: STOPPED / DEPLOYING / ACTIVE +diff_app=$(diff <(echo "$old_app_ver") <(echo "$new_app_ver")) #caluclating difference in major app versions +diff_chart=$(diff <(echo "$old_chart_ver") <(echo "$new_chart_ver")) #caluclating difference in Chart versions +old_full_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}') #Upgraded From +new_full_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}') #Upraded To +rollback_version=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}') + if [[ "$diff_app" == "$diff_chart" || "$update_all_apps" == "true" ]]; then #continue to update + if [[ $stop_before_update == "true" ]]; then # Check to see if user is using -S or not + if [[ "$startstatus" == "STOPPED" ]]; then # if status is already stopped, skip while loop + echo_array+=("\n$app_name") + [[ "$verbose" == "true" ]] && echo_array+=("Updating..") + cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("FAILED") + return 0 + else # if status was not STOPPED, stop the app prior to updating + echo_array+=("\n$app_name") + [[ "$verbose" == "true" ]] && echo_array+=("Stopping prior to update..") + midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && SECONDS=0 || echo_array+=("FAILED") + while [[ "$status" != "STOPPED" ]] + do + status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}') + if [[ "$status" == "STOPPED" ]]; then + echo_array+=("Stopped") + [[ "$verbose" == "true" ]] && echo_array+=("Updating..") + cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("Failed to update") + break + elif [[ "$SECONDS" -ge "$timeout" ]]; then + echo_array+=("Error: Run Time($SECONDS) has exceeded Timeout($timeout)") + break + elif [[ "$status" != "STOPPED" ]]; then + [[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be STOPPED") + sleep 10 + fi + done + fi + else #user must not be using -S, just update + echo_array+=("\n$app_name") + [[ "$verbose" == "true" ]] && echo_array+=("Updating..") + cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("FAILED") + fi + else + echo_array+=("\n$app_name\nMajor Release, update manually") + return 0 + fi +} +export -f update_apps + + +after_update_actions(){ +SECONDS=0 +count=0 +if [[ $rollback == "true" ]]; then + while true + do + (( count++ )) + status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}') + if [[ "$status" == "ACTIVE" && "$startstatus" == "STOPPED" ]]; then + [[ "$verbose" == "true" ]] && echo_array+=("Returing to STOPPED state..") + midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo_array+=("Stopped")|| echo_array+=("FAILED") + break + elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" != "true" ]]; then + echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nIf this is a slow starting application, set a higher timeout with -t\nIf this applicaion is always DEPLOYING, you can disable all probes under the Healthcheck Probes Liveness section in the edit configuration\nReverting update..") + midclt call chart.release.rollback "$app_name" "{\"item_version\": \"$rollback_version\"}" &> /dev/null + [[ "$startstatus" == "STOPPED" ]] && failed="true" && after_update_actions && unset failed #run back after_update_actions function if the app was stopped prior to update + break + elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" == "true" ]]; then + echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nThe application failed to be ACTIVE even after a rollback,\nManual intervention is required\nAbandoning") + break + elif [[ "$status" == "STOPPED" ]]; then + [[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Stopped..") && sleep 15 && continue #if reports stopped on FIRST time through loop, double check + [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check + echo_array+=("Stopped") + break #if reports stopped any time after the first loop, assume its extermal services. + elif [[ "$status" == "ACTIVE" ]]; then + [[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Active..") && sleep 15 && continue #if reports active on FIRST time through loop, double check + [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check + echo_array+=("Active") + break #if reports active any time after the first loop, assume actually active. + else + [[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE") + sleep 15 + continue + fi + done +else + if [[ "$startstatus" == "STOPPED" ]]; then + while true #using a constant while loop, then breaking out of the loop with break commands below. + do + (( count++ )) + status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}') + if [[ "$status" == "STOPPED" ]]; then + [[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Stopped..") && sleep 15 && continue #if reports stopped on FIRST time through loop, double check + [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check + echo_array+=("Stopped") #assume actually stopped anytime AFTER the first loop + break + elif [[ "$status" == "ACTIVE" ]]; then + [[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Active..") && sleep 15 && continue #if reports active on FIRST time through loop, double check + [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check + [[ "$verbose" == "true" ]] && echo_array+=("Returing to STOPPED state..") + midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo_array+=("Stopped")|| echo_array+=("FAILED") + break + elif [[ "$SECONDS" -ge "$timeout" ]]; then + echo_array+=("Error: Run Time($SECONDS) has exceeded Timeout($timeout)") + break + else + [[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE") + sleep 10 + continue + fi + done + fi +fi + +#Dump the echo_array, ensures all output is in a neat order. +for i in "${echo_array[@]}" +do + echo -e "$i" +done + + +} +export -f after_update_actions \ No newline at end of file diff --git a/heavy_script.sh b/heavy_script.sh index 8f38bfad..6de0e9de 100644 --- a/heavy_script.sh +++ b/heavy_script.sh @@ -1,375 +1,38 @@ #!/bin/bash -#If no argument is passed, kill the script. -[[ -z "$*" || "-" == "$*" || "--" == "$*" ]] && echo "This script requires an argument, use --help for help" && exit -args=("$@") - -self_update() { +# cd to script, this ensures the script can find the source scripts below, even when ran from a seperate directory script=$(readlink -f "$0") script_path=$(dirname "$script") script_name="heavy_script.sh" -cd "$script_path" || exit -git fetch &> /dev/null - -if git diff --name-only origin/main | grep -q "$script_name" ; then - echo "Found a new version of HeavyScript, updating myself..." - git reset --hard -q - git pull --force -q - echo -e "Running the new version...\n" - count=0 - for i in "${args[@]}" - do - [[ "$i" == "--self-update" ]] && unset "args[$count]" && break - ((count++)) - done - sleep 5 - exec bash "$script_name" "${args[@]}" - - # Now exit this old instance - exit -else - echo -e "HeavyScript is already the latest version\n" -fi -} +cd "$script_path" || { echo "Error: Failed to change to script directory" ; exit ; } -help(){ -[[ $help == "true" ]] && clear -x -echo "Basic Utilities" -echo "--mount | Initiates mounting feature, choose between unmounting and mounting PVC data" -echo "--restore | Opens a menu to restore a \"heavy_script\" backup that was taken on your \"ix-applications\" dataset" -echo "--delete-backup | Opens a menu to delete backups on your system" -echo "--dns | list all of your applications DNS names and their web ports" -echo -echo "Update Options" -echo "-U | Update all applications, ignores versions" -echo "-u | Update all applications, does not update Major releases" -echo "-b | Back-up your ix-applications dataset, specify a number after -b" -echo "-i | Add application to ignore list, one by one, see example below." -echo "-R | THIS OPTION WILL DEPRICATE SOON, USE \"-r\" instead. Roll-back applications if they fail to update" -echo "-r | Roll-back applications if they fail to update" -echo "-S | Shutdown applications prior to updating" -echo "-v | verbose output" -echo "-t | Set a custom timeout in seconds when checking if either an App or Mountpoint correctly Started, Stopped or (un)Mounted. Defaults to 500 seconds" -echo "-s | sync catalog" -echo "-p | Prune unused/old docker images" -echo -echo "Examples" -echo "bash heavy_script.sh -b 14 -i portainer -i arch -i sonarr -i radarr -t 600 -vrsUp" -echo "bash /mnt/tank/scripts/heavy_script.sh -t 150 --mount" -echo "bash /mnt/tank/scripts/heavy_script.sh --dns" -echo "bash heavy_script.sh --restore" -echo "bash /mnt/tank/scripts/heavy_script.sh --delete-backup" -echo -exit -} -export -f help +# shellcheck source=functions/backup.sh +source functions/backup.sh +# shellcheck source=functions/dns.sh +source functions/dns.sh +# shellcheck source=functions/menu.sh +source functions/menu.sh +# shellcheck source=functions/misc.sh +source functions/misc.sh +# shellcheck source=functions/mount.sh +source functions/mount.sh +# shellcheck source=functions/self_update.sh +source functions/self_update.sh +# shellcheck source=functions/update_apps.sh +source functions/update_apps.sh -deleteBackup(){ -clear -x && echo "pulling all restore points.." -list_backups=$(cli -c 'app kubernetes list_backups' | sort -t '_' -Vr -k2,7 | tr -d " \t\r" | awk -F '|' '{print $2}' | nl | column -t) -clear -x -[[ -z "$list_backups" ]] && echo "No restore points available" && exit || { title; echo -e "Choose a restore point to delete\nThese may be out of order if they are not HeavyScript backups" ; } -echo "$list_backups" && read -t 600 -p "Please type a number: " selection && restore_point=$(echo "$list_backups" | grep ^"$selection " | awk '{print $2}') -[[ -z "$selection" ]] && echo "Your selection cannot be empty" && exit #Check for valid selection. If none, kill script -[[ -z "$restore_point" ]] && echo "Invalid Selection: $selection, was not an option" && exit #Check for valid selection. If none, kill script -echo -e "\nWARNING:\nYou CANNOT go back after deleting your restore point" || { echo "FAILED"; exit; } -echo -e "\n\nYou have chosen:\n$restore_point\n\nWould you like to continue?" && echo -e "1 Yes\n2 No" && read -t 120 -p "Please type a number: " yesno || { echo "FAILED"; exit; } -if [[ $yesno == "1" ]]; then - echo -e "\nDeleting $restore_point" && cli -c 'app kubernetes delete_backup backup_name=''"'"$restore_point"'"' &>/dev/null && echo "Sucessfully deleted" || echo "Deletion Failed" -elif [[ $yesno == "2" ]]; then - echo "You've chosen NO, killing script." -else - echo "Invalid Selection" -fi -} -export -f deleteBackup -backup(){ -echo -e "\nNumber of backups was set to $number_of_backups" -date=$(date '+%Y_%m_%d_%H_%M_%S') -[[ "$verbose" == "true" ]] && cli -c 'app kubernetes backup_chart_releases backup_name=''"'HeavyScript_"$date"'"' -[[ -z "$verbose" ]] && echo -e "\nNew Backup Name:" && cli -c 'app kubernetes backup_chart_releases backup_name=''"'HeavyScript_"$date"'"' | tail -n 1 -mapfile -t list_backups < <(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -Vr -k2,7 | awk -F '|' '{print $2}'| tr -d " \t\r") -if [[ ${#list_backups[@]} -gt "number_of_backups" ]]; then - echo -e "\nDeleting the oldest backup(s) for exceeding limit:" - overflow=$(( ${#list_backups[@]} - "$number_of_backups" )) - mapfile -t list_overflow < <(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -V -k2,7 | awk -F '|' '{print $2}'| tr -d " \t\r" | head -n "$overflow") - for i in "${list_overflow[@]}" - do - cli -c 'app kubernetes delete_backup backup_name=''"'"$i"'"' &> /dev/null || echo "Failed to delete $i" - echo "$i" - done -fi -} -export -f backup +#If no argument is passed, kill the script. +[[ -z "$*" || "-" == "$*" || "--" == "$*" ]] && menu -restore(){ -clear -x && echo "pulling restore points.." -list_backups=$(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -Vr -k2,7 | tr -d " \t\r" | awk -F '|' '{print $2}' | nl | column -t) -clear -x -[[ -z "$list_backups" ]] && echo "No HeavyScript restore points available" && exit || { title; echo "Choose a restore point" ; } -echo "$list_backups" && read -t 600 -p "Please type a number: " selection && restore_point=$(echo "$list_backups" | grep ^"$selection " | awk '{print $2}') -[[ -z "$selection" ]] && echo "Your selection cannot be empty" && exit #Check for valid selection. If none, kill script -[[ -z "$restore_point" ]] && echo "Invalid Selection: $selection, was not an option" && exit #Check for valid selection. If none, kill script -echo -e "\nWARNING:\nThis is NOT guranteed to work\nThis is ONLY supposed to be used as a LAST RESORT\nConsider rolling back your applications instead if possible" || { echo "FAILED"; exit; } -echo -e "\n\nYou have chosen:\n$restore_point\n\nWould you like to continue?" && echo -e "1 Yes\n2 No" && read -t 120 -p "Please type a number: " yesno || { echo "FAILED"; exit; } -if [[ $yesno == "1" ]]; then - echo -e "\nStarting Backup, this will take a LONG time." && cli -c 'app kubernetes restore_backup backup_name=''"'"$restore_point"'"' || echo "Restore FAILED" -elif [[ $yesno == "2" ]]; then - echo "You've chosen NO, killing script. Good luck." -else - echo "Invalid Selection" -fi -} -export -f restore - - -dns(){ -clear -x -echo "Generating DNS Names.." -#ignored dependency pods, may need to add more in the future. -dep_ignore="\-cronjob\-|^kube-system|\ssvclb|NAME|\-memcached\-.[^custom\-app]|\-postgresql\-.[^custom\-app]|\-redis\-.[^custom\-app]|\-mariadb\-.[^custom\-app]|\-promtail\-.[^custom\-app]" - -# Pulling pod names -mapfile -t main < <(k3s kubectl get pods -A | grep -Ev "$dep_ignore" | sort) - -# Pulling all ports -all_ports=$(k3s kubectl get service -A) - -clear -x -count=0 -for i in "${main[@]}" -do - [[ count -le 0 ]] && echo -e "\n" && ((count++)) - appName=$(echo "$i" | awk '{print $2}' | sed 's/-[^-]*-[^-]*$//' | sed 's/-0//') - ixName=$(echo "$i" | awk '{print $1}') - port=$(echo "$all_ports" | grep -E "\s$appName\s" | awk '{print $6}' | grep -Eo "^[[:digit:]]+{1}") - echo -e "$appName.$ixName.svc.cluster.local $port" -done | uniq | nl -b t | sed 's/\s\s\s$/- -------- ----/' | column -t -R 1 -N "#,DNS_Name,Port" -L -} -export -f dns - - -mount(){ -clear -x -title -echo -e "1 Mount\n2 Unmount All" && read -t 600 -p "Please type a number: " selection -[[ -z "$selection" ]] && echo "Your selection cannot be empty" && exit #Check for valid selection. If none, kill script -if [[ $selection == "1" ]]; then - list=$(k3s kubectl get pvc -A | sort -u | awk '{print NR-1, "\t" $1 "\t" $2 "\t" $4}' | column -t | sed "s/^0/ /") - echo "$list" && read -t 120 -p "Please type a number: " selection - [[ -z "$selection" ]] && echo "Your selection cannot be empty" && exit #Check for valid selection. If none, kill script - app=$(echo -e "$list" | grep ^"$selection " | awk '{print $2}' | cut -c 4- ) - [[ -z "$app" ]] && echo "Invalid Selection: $selection, was not an option" && exit #Check for valid selection. If none, kill script - pvc=$(echo -e "$list" | grep ^"$selection ") - status=$(cli -m csv -c 'app chart_release query name,status' | grep -E "^$app\b" | awk -F ',' '{print $2}'| tr -d " \t\n\r") - if [[ "$status" != "STOPPED" ]]; then - [[ -z $timeout ]] && echo -e "\nDefault Timeout: 500" && timeout=500 || echo -e "\nCustom Timeout: $timeout" - SECONDS=0 && echo -e "\nScaling down $app" && midclt call chart.release.scale "$app" '{"replica_count": 0}' &> /dev/null - else - echo -e "\n$app is already stopped" - fi - while [[ "$SECONDS" -le "$timeout" && "$status" != "STOPPED" ]] - do - status=$(cli -m csv -c 'app chart_release query name,status' | grep -E "^$app\b" | awk -F ',' '{print $2}'| tr -d " \t\n\r") - echo -e "Waiting $((timeout-SECONDS)) more seconds for $app to be STOPPED" && sleep 5 - done - data_name=$(echo "$pvc" | awk '{print $3}') - mount=$(echo "$pvc" | awk '{print $4}') - volume_name=$(echo "$pvc" | awk '{print $4}') - full_path=$(zfs list | grep "$volume_name" | awk '{print $1}') - echo -e "\nMounting\n$full_path\nTo\n/mnt/heavyscript/$data_name" && zfs set mountpoint=/heavyscript/"$data_name" "$full_path" && echo -e "Mounted\n\nUnmount with:\nzfs set mountpoint=legacy "$full_path" && rmdir /mnt/heavyscript/"$data_name"\n\nOr use the Unmount All option\n" - exit -elif [[ $selection == "2" ]]; then - mapfile -t unmount_array < <(basename -a /mnt/heavyscript/* | sed "s/*//") - [[ -z $unmount_array ]] && echo "Theres nothing to unmount" && exit - for i in "${unmount_array[@]}" - do - main=$(k3s kubectl get pvc -A | grep -E "\s$i\s" | awk '{print $1, $2, $4}') - app=$(echo "$main" | awk '{print $1}' | cut -c 4-) - pvc=$(echo "$main" | awk '{print $3}') - mapfile -t path < <(find /mnt/*/ix-applications/releases/"$app"/volumes/ -maxdepth 0 | cut -c 6-) - if [[ "${#path[@]}" -gt 1 ]]; then #if there is another app with the same name on another pool, use the current pools application, since the other instance is probably old, or unused. - echo "$i is a name used on more than one pool.. attempting to use your current kubernetes apps pool" - pool=$(cli -c 'app kubernetes config' | grep -E "dataset\s\|" | awk -F '|' '{print $3}' | awk -F '/' '{print $1}' | tr -d " \t\n\r") - full_path=$(find /mnt/"$pool"/ix-applications/releases/"$app"/volumes/ -maxdepth 0 | cut -c 6-) - zfs set mountpoint=legacy "$full_path""$pvc" && echo "$i unmounted" && rmdir /mnt/heavyscript/"$i" || echo "failed to unmount $i" - else - zfs set mountpoint=legacy "$path""$pvc" && echo "$i unmounted" && rmdir /mnt/heavyscript/"$i" || echo "failed to unmount $i" - fi - done - rmdir /mnt/heavyscript -else - echo "Invalid selection, \"$selection\" was not an option" -fi -} -export -f mount - - -sync(){ -echo -e "\nSyncing all catalogs, please wait.." && cli -c 'app catalog sync_all' &> /dev/null && echo -e "Catalog sync complete" -} -export -f sync - - -update_apps(){ -# Replace with line below after testing -# cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,container_images_update_available,status' | tr -d " \t\r" | grep -E ",true($|,)" | sort -mapfile -t array < <(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,container_images_update_available,status' | grep -E ",true(,|\b)" | sort) -[[ -z $array ]] && echo -e "\nThere are no updates available" && return 0 || echo -e "\n${#array[@]} update(s) available" -[[ -z $timeout ]] && echo -e "\nDefault Timeout: 500" && timeout=500 || echo -e "\nCustom Timeout: $timeout" -[[ "$timeout" -le 120 ]] && echo "Warning: Your timeout is set low and may lead to premature rollbacks or skips" -for i in "${array[@]}" -do - app_name=$(echo "$i" | awk -F ',' '{print $1}') #print out first catagory, name. - old_app_ver=$(echo "$i" | awk -F ',' '{print $4}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #previous/current Application MAJOR Version - new_app_ver=$(echo "$i" | awk -F ',' '{print $5}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #new Application MAJOR Version - old_chart_ver=$(echo "$i" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}' | awk -F '.' '{print $1}') # Old Chart MAJOR version - new_chart_ver=$(echo "$i" | awk -F ',' '{print $5}' | awk -F '_' '{print $2}' | awk -F '.' '{print $1}') # New Chart MAJOR version - status=$(echo "$i" | awk -F ',' '{print $2}') #status of the app: STOPPED / DEPLOYING / ACTIVE - startstatus=$status - diff_app=$(diff <(echo "$old_app_ver") <(echo "$new_app_ver")) #caluclating difference in major app versions - diff_chart=$(diff <(echo "$old_chart_ver") <(echo "$new_chart_ver")) #caluclating difference in Chart versions - old_full_ver=$(echo "$i" | awk -F ',' '{print $4}') #Upgraded From - new_full_ver=$(echo "$i" | awk -F ',' '{print $5}') #Upraded To - rollback_version=$(echo "$i" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}') - printf '%s\0' "${ignore[@]}" | grep -iFxqz "${app_name}" && echo -e "\n$app_name\nIgnored, skipping" && continue #If application is on ignore list, skip - if [[ "$diff_app" == "$diff_chart" || "$update_all_apps" == "true" ]]; then #continue to update - if [[ $stop_before_update == "true" ]]; then # Check to see if user is using -S or not - if [[ "$status" == "STOPPED" ]]; then # if status is already stopped, skip while loop - echo -e "\n$app_name" - [[ "$verbose" == "true" ]] && echo "Updating.." - cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo -e "Updated\n$old_full_ver\n$new_full_ver" && after_update_actions || echo "FAILED" - continue - else # if status was not STOPPED, stop the app prior to updating - echo -e "\n$app_name" - [[ "$verbose" == "true" ]] && echo "Stopping prior to update.." - midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && SECONDS=0 || echo -e "FAILED" - while [[ "$status" != "STOPPED" ]] - do - status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status' | grep "^$app_name," | awk -F ',' '{print $2}') - if [[ "$status" == "STOPPED" ]]; then - echo "Stopped" - [[ "$verbose" == "true" ]] && echo "Updating.." - cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo -e "Updated\n$old_full_ver\n$new_full_ver" && after_update_actions || echo "Failed to update" - break - elif [[ "$SECONDS" -ge "$timeout" ]]; then - echo "Error: Run Time($SECONDS) has exceeded Timeout($timeout)" - break - elif [[ "$status" != "STOPPED" ]]; then - [[ "$verbose" == "true" ]] && echo "Waiting $((timeout-SECONDS)) more seconds for $app_name to be STOPPED" - sleep 10 - continue - fi - done - fi - else #user must not be using -S, just update - echo -e "\n$app_name" - [[ "$verbose" == "true" ]] && echo "Updating.." - cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo -e "Updated\n$old_full_ver\n$new_full_ver" && after_update_actions || echo "FAILED" - fi - else - echo -e "\n$app_name\nMajor Release, update manually" - continue - fi -done -} -export -f update_apps - - -after_update_actions(){ -SECONDS=0 -count=0 -if [[ $rollback == "true" ]]; then - while [[ "0" != "1" ]] - do - (( count++ )) - status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status' | grep "^$app_name," | awk -F ',' '{print $2}') - if [[ "$status" == "ACTIVE" && "$startstatus" == "STOPPED" ]]; then - [[ "$verbose" == "true" ]] && echo "Returing to STOPPED state.." - midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo "Stopped"|| echo "FAILED" - break - elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" != "true" ]]; then - echo -e "Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nIf this is a slow starting application, set a higher timeout with -t\nIf this applicaion is always DEPLOYING, you can disable all probes under the Healthcheck Probes Liveness section in the edit configuration\nReverting update.." - midclt call chart.release.rollback "$app_name" "{\"item_version\": \"$rollback_version\"}" &> /dev/null - [[ "$startstatus" == "STOPPED" ]] && failed="true" && after_update_actions && unset failed #run back after_update_actions function if the app was stopped prior to update - break - elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" == "true" ]]; then - echo -e "Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nThe application failed to be ACTIVE even after a rollback,\nManual intervention is required\nAbandoning" - break - elif [[ "$status" == "STOPPED" ]]; then - [[ "$count" -le 1 && "$verbose" == "true" ]] && echo "Verifying Stopped.." && sleep 15 && continue #if reports stopped on FIRST time through loop, double check - [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check - echo "Stopped" && break #if reports stopped any time after the first loop, assume its extermal services. - elif [[ "$status" == "ACTIVE" ]]; then - [[ "$count" -le 1 && "$verbose" == "true" ]] && echo "Verifying Active.." && sleep 15 && continue #if reports active on FIRST time through loop, double check - [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check - echo "Active" && break #if reports active any time after the first loop, assume actually active. - else - [[ "$verbose" == "true" ]] && echo "Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE" - sleep 15 - continue - fi - done -else - if [[ "$startstatus" == "STOPPED" ]]; then - while [[ "0" != "1" ]] #using a constant while loop, then breaking out of the loop with break commands below. - do - (( count++ )) - status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status' | grep "^$app_name," | awk -F ',' '{print $2}') - if [[ "$status" == "STOPPED" ]]; then - [[ "$count" -le 1 && "$verbose" == "true" ]] && echo "Verifying Stopped.." && sleep 15 && continue #if reports stopped on FIRST time through loop, double check - [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check - echo "Stopped" && break #assume actually stopped anytime AFTER the first loop - break - elif [[ "$status" == "ACTIVE" ]]; then - [[ "$count" -le 1 && "$verbose" == "true" ]] && echo "Verifying Active.." && sleep 15 && continue #if reports active on FIRST time through loop, double check - [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check - [[ "$verbose" == "true" ]] && echo "Returing to STOPPED state.." - midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo "Stopped"|| echo "FAILED" - break - elif [[ "$SECONDS" -ge "$timeout" ]]; then - echo "Error: Run Time($SECONDS) has exceeded Timeout($timeout)" - break - else - [[ "$verbose" == "true" ]] && echo "Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE" - sleep 10 - continue - fi - done - fi -fi -} -export -f after_update_actions - - -prune(){ -echo -e "\nPruning Docker Images" && docker image prune -af | grep "^Total" || echo "Failed to Prune Docker Images" -} -export -f prune - - -title(){ -echo ' _ _ _____ _ _ ' -echo '| | | | / ___| (_) | | ' -echo '| |_| | ___ __ ___ ___ _\ `--. ___ _ __ _ _ __ | |_' -echo "| _ |/ _ \/ _\` \ \ / / | | |\`--. \/ __| '__| | '_ \| __|" -echo '| | | | __/ (_| |\ V /| |_| /\__/ / (__| | | | |_) | |_ ' -echo '\_| |_/\___|\__,_| \_/ \__, \____/ \___|_| |_| .__/ \__|' -echo ' __/ | | | ' -echo ' |___/ |_| ' -echo -} -export -f title - # Parse script options -while getopts ":si:rb:t:uUpSRv-:" opt +while getopts ":sirb:t:uUpSRv-:" opt do case $opt in -) @@ -393,44 +56,62 @@ do deleteBackup="true" ;; *) - echo -e "Invalid Option \"--$OPTARG\"\n" && help - exit + echo -e "Invalid Option \"--$OPTARG\"\n" + help ;; esac ;; - \?) - echo -e "Invalid Option \"-$OPTARG\"\n" && help - exit - ;; :) - echo -e "Option: \"-$OPTARG\" requires an argument\n" && help - exit - ;; + echo -e "Option: \"-$OPTARG\" requires an argument\n" + help + ;; b) - re='^[0-9]+$' number_of_backups=$OPTARG - ! [[ $OPTARG =~ $re ]] && echo -e "Error: -b needs to be assigned an interger\n\"""$number_of_backups""\" is not an interger" >&2 && exit + ! [[ $OPTARG =~ ^[0-9]+$ ]] && echo -e "Error: -b needs to be assigned an interger\n\"""$number_of_backups""\" is not an interger" >&2 && exit [[ "$number_of_backups" -le 0 ]] && echo "Error: Number of backups is required to be at least 1" && exit + backup="true" ;; r) rollback="true" ;; i) - ignore+=("$OPTARG") + if ! [[ $OPTARG =~ ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ ]]; then # Using case insensitive version of the regex used by Truenas Scale + echo -e "Error: \"$OPTARG\" is not a possible option for an application name" + exit + else + ignore+=("$OPTARG") + fi ;; t) - re='^[0-9]+$' timeout=$OPTARG - ! [[ $timeout =~ $re ]] && echo -e "Error: -t needs to be assigned an interger\n\"""$timeout""\" is not an interger" >&2 && exit + ! [[ $timeout =~ ^[0-9]+$ ]] && echo -e "Error: -t needs to be assigned an interger\n\"""$timeout""\" is not an interger" >&2 && exit ;; s) sync="true" ;; U) update_all_apps="true" + # Check next positional parameter + eval nextopt=${!OPTIND} + # existing or starting with dash? + if [[ -n $nextopt && $nextopt != -* ]] ; then + OPTIND=$((OPTIND + 1)) + update_limit="$nextopt" + else + update_limit=1 + fi ;; u) update_apps="true" + # Check next positional parameter + eval nextopt=${!OPTIND} + # existing or starting with dash? + if [[ -n $nextopt && $nextopt != -* ]] ; then + OPTIND=$((OPTIND + 1)) + update_limit="$nextopt" + else + update_limit=1 + fi ;; S) stop_before_update="true" @@ -438,22 +119,23 @@ do p) prune="true" ;; - R) - rollback="true" - echo "WARNING: -R is being transisitioned to -r, this is due to a refactor in the script. Please Make the change ASAP!" - ;; v) verbose="true" ;; + \?) + echo -e "Invalid Option \"-$OPTARG\"\n" + help + ;; *) - echo -e "Invalid Option \"--$OPTARG\"\n" && help - exit + echo -e "Invalid Option \"-$OPTARG\"\n" + help ;; esac done -#exit if incompatable functions are called + +#exit if incompatable functions are called [[ "$update_all_apps" == "true" && "$update_apps" == "true" ]] && echo -e "-U and -u cannot BOTH be called" && exit #Continue to call functions in specific order @@ -463,7 +145,16 @@ done [[ "$dns" == "true" ]] && dns && exit [[ "$restore" == "true" ]] && restore && exit [[ "$mount" == "true" ]] && mount && exit -[[ "$number_of_backups" -ge 1 ]] && backup -[[ "$sync" == "true" ]] && sync -[[ "$update_all_apps" == "true" || "$update_apps" == "true" ]] && update_apps -[[ "$prune" == "true" ]] && prune \ No newline at end of file +if [[ "$backup" == "true" && "$sync" == "true" ]]; then # Run backup and sync at the same time + echo "šŸ…ƒ šŸ„° šŸ…‚ šŸ„ŗ šŸ…‚ :" + echo -e "-Backing up ix-applications dataset\n-Syncing catalog(s)" + echo -e "This can take a LONG time, please wait for both output..\n" + backup & + sync & + wait +fi +[[ "$backup" == "true" && -z "$sync" ]] && echo "Backing up \"ix-applications\" dataset, please wait.." && backup +[[ "$sync" == "true" && -z "$backup" ]] && echo "Syncing catalogs, this takes a LONG time, please wait.." && sync +[[ "$update_all_apps" == "true" || "$update_apps" == "true" ]] && commander +[[ "$prune" == "true" ]] && prune +