julian/scale-catalog
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #7 from Heavybullets8/beta

Browse Source 
Merge Major changes
This commit is contained in:
Heavybullets8 2022-07-29 13:14:58 -06:00 committed by GitHub
commit d6478da3eb
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
9 changed files with 919 additions and 403 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
README.md
View File

@ -1,5 +1,9 @@
# heavy_script # heavy_script
## Website
[HeavySetup - Further In-Depth Explanation](https://heavysetup.info/scripts/heavyscript/about/)
## Table of contents: ## Table of contents:
* [Arguments](#arguments) * [Arguments](#arguments)
* [Examples](#examples) * [Examples](#examples)
@ -13,17 +17,18 @@
## Arguments ## Arguments
| Flag | Example | Parameter | Description | | Flag | Example | Parameter | Description |
|----------------- |------------------------ |----------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |----------------- |------------------------ |----------------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| NULL | NULL | NULL | If you choose not to supply an option, it will open the menu for easier access to the utilities |
| --self-update | --self-update | None | Updates HeavyScript prior to running it<br>_You no longer need to git pull_ | | --self-update | --self-update | None | Updates HeavyScript prior to running it<br>_You no longer need to git pull_ |
| --delete-backup | --delete-backup | None | Opens a menu to delete backups<br>_Useful if you need to delete old system backups or backups from other scripts_ | | --delete-backup | --delete-backup | None | Opens a menu to delete backups<br>_Useful if you need to delete old system backups or backups from other scripts_ |
| --restore | --restore | None | Restore HeavyScript specific `ix-applications dataset` snapshot | | --restore | --restore | None | Restore HeavyScript specific `ix-applications dataset` snapshot |
| --mount | --mount | None | Initiates mounting feature<br>Choose between unmounting and mounting PVC data | | --mount | --mount | None | Initiates mounting feature<br>Choose between unmounting and mounting PVC data |
| --dns | --dns | None | list all of your applications DNS names and their web ports | | --dns | --dns | None | list all of your applications DNS names and their web ports |
| -U | -U | None | Update applications, ignoring major version changes | | -U | -U <br>-U 5 | None or Integer | Update applications, ignoring major version changes<br>_Optionally, you can supply a number after the argument to update multiple applications at once_ |
| -u | -u | None | Update applications, do NOT update if there was a major version change | | -u | -u<br>-u 5 | None or Integer | Update applications, do NOT update if there was a major version change<br>_Optionally, you can supply a number after the argument to update multiple applications at once_ |
| -b | -b 14 | Integer | Backup `ix-appliactions` dataset<br>_Creates backups up to the number you've chosen_ | | -b | -b 14 | Integer | Backup `ix-appliactions` dataset<br>_Creates backups up to the number you've chosen_ |
| -i | -i nextcloud -i sonarr | String | Applications listed will be ignored during updating<br>_List one application after another as shown in the example_ | | -i | -i nextcloud -i sonarr | String | Applications listed will be ignored during updating<br>_List one application after another as shown in the example_ |
| (-R\|-r) | -r | None | Monitors applications after they update<br>If the app does not become "ACTIVE" after either:<br>The custom Timeout, or Default Timeout,<br>rollback the application.<br>__Warning: deprecating `-R` please begin using `-r` instead__ | | -r | -r | None | Monitors applications after they update<br>If the app does not become "ACTIVE" after either:<br>The custom Timeout, or Default Timeout,<br>rollback the application. |
| -v | -v | None | Verbose Output<br>_Look at the bottom of this page for an example_ | | -v | -v | None | Verbose Output<br>_Look at the bottom of this page for an example_ |
| -S | -S | None | Shutdown the application prior to updating it | | -S | -S | None | Shutdown the application prior to updating it |
| -t | -t 150 | Integer | Set a custom timeout to be used with either:<br>`-m` <br>_Time the script will wait for application to be "STOPPED"_<br>or<br>`-(u\|U)` <br>_Time the script will wait for application to be either "STOPPED" or "ACTIVE"_ | | -t | -t 150 | Integer | Set a custom timeout to be used with either:<br>`-m` <br>_Time the script will wait for application to be "STOPPED"_<br>or<br>`-(u\|U)` <br>_Time the script will wait for application to be either "STOPPED" or "ACTIVE"_ |
@ -37,7 +42,9 @@
### Examples ### Examples
#### Typical Cron Job #### Typical Cron Job
``` ```
bash heavy_script.sh --self-update -b 14 -i portainer -i arch -i sonarr -i radarr -t 600 -rsup
bash heavy_script.sh --self-update -b 14 -i portainer -i arch -i sonarr -i radarr -t 600 -rsp -u 5
``` ```
> `-b` is set to 14. Up to 14 snapshots of your ix-applications dataset will be saved > `-b` is set to 14. Up to 14 snapshots of your ix-applications dataset will be saved
@ -50,10 +57,13 @@ bash heavy_script.sh --self-update -b 14 -i portainer -i arch -i sonarr -i radar
> `-s` will just sync the repositories, ensuring you are downloading the latest updates. > `-s` will just sync the repositories, ensuring you are downloading the latest updates.
> `-u` update applications as long as the major version has absolutely no change, if it does have a change it will ask the user to update manually.
> `-p` Prune docker images. > `-p` Prune docker images.
> `-u` update applications as long as the major version has absolutely no change, if it does have a change it will ask the user to update manually.
>> The `5` after the `-u` means up to 5 applications will be updating and monitored at one time
> `--self-update` Will update the script prior to running anything else.
> `--self-update` Will update the script prior to running anything else. > `--self-update` Will update the script prior to running anything else.
#### Mounting PVC Data #### Mounting PVC Data
@ -82,7 +92,9 @@ bash /mnt/tank/scripts/heavy_script/heavy_script.sh --dns
#### My personal Cron Job #### My personal Cron Job
``` ```
bash /mnt/speed/scripts/heavy_script/heavy_script.sh --self-update -b 14 -rsup
bash /mnt/speed/scripts/heavy_script/heavy_script.sh --self-update -b 14 -rsp -u 10
``` ```
<br> <br>
@ -180,4 +192,3 @@ bash heavyscript.sh --self-update -b 14 -supr
| ![image](https://user-images.githubusercontent.com/20793231/167971188-07f71d02-8da3-4e0c-b9a0-cd26e7f63613.png) | ![image](https://user-images.githubusercontent.com/20793231/167972033-dc8d4ab4-4fb2-4c8a-b7dc-b9311ae55cf8.png) | | ![image](https://user-images.githubusercontent.com/20793231/167971188-07f71d02-8da3-4e0c-b9a0-cd26e7f63613.png) | ![image](https://user-images.githubusercontent.com/20793231/167972033-dc8d4ab4-4fb2-4c8a-b7dc-b9311ae55cf8.png) |

176
functions/backup.sh Normal file
View File

@ -0,0 +1,176 @@
#!/bin/bash
backup(){
echo_backup+=("\n🄱 🄰 🄲 🄺 🅄 🄿 🅂")
echo_backup+=("Number of backups was set to $number_of_backups")
date=$(date '+%Y_%m_%d_%H_%M_%S')
[[ "$verbose" == "true" ]] && cli -c 'app kubernetes backup_chart_releases backup_name=''"'HeavyScript_"$date"'"' &> /dev/null && echo_backup+=(HeavyScript_"$date")
[[ -z "$verbose" ]] && echo_backup+=("\nNew Backup Name:") && cli -c 'app kubernetes backup_chart_releases backup_name=''"'HeavyScript_"$date"'"' | tail -n 1 &> /dev/null && echo_backup+=(HeavyScript_"$date")
mapfile -t list_backups < <(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -Vr -k2,7 | awk -F '|' '{print $2}'| tr -d " \t\r")
if [[ ${#list_backups[@]} -gt "$number_of_backups" ]]; then
echo_backup+=("\nDeleted the oldest backup(s) for exceeding limit:")
overflow=$(( ${#list_backups[@]} - "$number_of_backups" ))
mapfile -t list_overflow < <(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -V -k2,7 | awk -F '|' '{print $2}'| tr -d " \t\r" | head -n "$overflow")
for i in "${list_overflow[@]}"
do
cli -c 'app kubernetes delete_backup backup_name=''"'"$i"'"' &> /dev/null || echo_backup+=("Failed to delete $i")
echo_backup+=("$i")
done
fi
#Dump the echo_array, ensures all output is in a neat order.
for i in "${echo_backup[@]}"
do
echo -e "$i"
done
}
export -f backup
deleteBackup(){
while true
do
clear -x && echo "pulling all restore points.."
list_backups=$(cli -c 'app kubernetes list_backups' | sort -t '_' -Vr -k2,7 | tr -d " \t\r" | awk -F '|' '{print $2}' | nl -s ") " | column -t)
if [[ -z "$list_backups" ]]; then
echo "No restore points available"
exit
fi
while true
do
clear -x
title
echo -e "Choose a Restore Point to Delete\nThese may be out of order if they are not HeavyScript backups"
echo "$list_backups"
echo
echo "0) Exit"
read -rt 120 -p "Please type a number: " selection
restore_point=$(echo "$list_backups" | grep ^"$selection)" | awk '{print $2}')
if [[ $selection == 0 ]]; then
echo "Exiting.."
exit
elif [[ -z "$selection" ]]; then
echo "Your selection cannot be empty"
sleep 3
continue
elif [[ -z "$restore_point" ]]; then
echo "Invalid Selection: $selection, was not an option"
sleep 3
continue
fi
break
done
while true
do
clear -x
echo -e "\nWARNING:\nYou CANNOT go back after deleting your restore point"
echo -e "\n\nYou have chosen:\n$restore_point\n\nWould you like to continue?"
echo -e "1) Yes\n2) Exit\n"
read -rt 120 -p "Please type a number: " yesno
case $yesno in
1)
echo -e "\nDeleting $restore_point"
cli -c 'app kubernetes delete_backup backup_name=''"'"$restore_point"'"' &>/dev/null || { echo "Failed to delete backup.."; exit; }
echo "Sucessfully deleted"
break
;;
2)
echo "Exiting"
exit
;;
*)
echo "That was not an option, try again"
sleep 3
continue
;;
esac
done
while true
do
echo "Delete more?"
echo "1) Yes"
echo "2) No"
read -rt 120 -p "Please type a number: " yesno
case $yesno in
1)
break
;;
2)
exit
;;
*)
echo "$yesno was not an option, try again"
sleep 2
continue
;;
esac
done
done
}
export -f deleteBackup
restore(){
while true
do
clear -x && echo "pulling restore points.."
list_backups=$(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -Vr -k2,7 | tr -d " \t\r" | awk -F '|' '{print $2}' | nl -s ") " | column -t)
if [[ -z "$list_backups" ]]; then
echo "No HeavyScript restore points available"
exit
fi
while true
do
clear -x
title
echo "Choose a Restore Point"
echo "$list_backups"
echo
echo "0) Exit"
read -rt 120 -p "Please type a number: " selection
restore_point=$(echo "$list_backups" | grep ^"$selection)" | awk '{print $2}')
if [[ $selection == 0 ]]; then
echo "Exiting.."
exit
elif [[ -z "$selection" ]]; then
echo "Your selection cannot be empty"
sleep 3
continue
elif [[ -z "$restore_point" ]]; then
echo "Invalid Selection: $selection, was not an option"
sleep 3
continue
fi
break
done
while true
do
clear -x
echo -e "\nWARNING:\nThis is NOT guranteed to work\nThis is ONLY supposed to be used as a LAST RESORT\nConsider rolling back your applications instead if possible"
echo -e "\n\nYou have chosen:\n$restore_point\n\nWould you like to continue?"
echo -e "1) Yes\n2) Exit\n"
read -rt 120 -p "Please type a number: " yesno
case $yesno in
1)
echo -e "\nStarting Backup, this will take a LONG time."
cli -c 'app kubernetes restore_backup backup_name=''"'"$restore_point"'"' || { echo "Failed to delete backup.."; exit; }
exit
;;
2)
echo "Exiting"
exit
;;
*)
echo "That was not an option, try again"
sleep 3
continue
;;
esac
done
done
}
export -f restore

27
functions/dns.sh Normal file
View File

@ -0,0 +1,27 @@
#!/bin/bash
dns(){
clear -x
echo "Generating DNS Names.."
#ignored dependency pods, may need to add more in the future.
dep_ignore="\-cronjob\-|^kube-system|\ssvclb|NAME|\-memcached\-.[^custom\-app]|\-postgresql\-.[^custom\-app]|\-redis\-.[^custom\-app]|\-mariadb\-.[^custom\-app]|\-promtail\-.[^custom\-app]"
# Pulling pod names
mapfile -t main < <(k3s kubectl get pods -A | grep -Ev "$dep_ignore" | sort)
# Pulling all ports
all_ports=$(k3s kubectl get service -A)
clear -x
count=0
for i in "${main[@]}"
do
[[ count -le 0 ]] && echo -e "\n" && ((count++))
appName=$(echo "$i" | awk '{print $2}' | sed 's/-[^-]*-[^-]*$//' | sed 's/-0//')
ixName=$(echo "$i" | awk '{print $1}')
port=$(echo "$all_ports" | grep -E "\s$appName\s" | awk '{print $6}' | grep -Eo "^[[:digit:]]+{1}")
echo -e "$appName.$ixName.svc.cluster.local $port"
done | uniq | nl -b t | sed 's/\s\s\s$/- -------- ----/' | column -t -R 1 -N "#,DNS_Name,Port" -L
}
export -f dns

218
functions/menu.sh Normal file
View File

@ -0,0 +1,218 @@
#!/bin/bash
menu(){
script=$(readlink -f "$0")
script_path=$(dirname "$script")
script_name="heavy_script.sh"
cd "$script_path" || exit
clear -x
title
echo "1) Help"
echo "2) List DNS Names"
echo "3) Mount and Unmount PVC storage"
echo "4) Create a Backup"
echo "5) Restore a Backup"
echo "6) Delete a Backup"
echo "7) Update HeavyScript"
echo "8) Update Applications"
echo
echo "0) Exit"
read -rt 600 -p "Please select an option by number: " selection
case $selection in
0)
exit
;;
1)
help="true"
;;
2)
dns="true"
;;
3)
mount="true"
;;
4)
read -rt 600 -p "What is the maximun number of backups you would like?: " number_of_backups
backup="true"
;;
5)
restore="true"
;;
6)
deleteBackup="true"
;;
7)
self_update="true"
;;
8)
while true
do
clear -x
title
echo "Choose Your Update Type"
echo "-----------------------"
echo "1) -U | Update all applications, ignores versions"
echo "2) -u | Update all applications, does not update Major releases"
echo
echo "0) Exit"
echo
read -rt 600 -p "Please type the number associated with the flag above: " current_selection
if [[ $current_selection == 1 ]]; then
while true
do
echo -e "\nHow many applications do you want updating at the same time?"
read -rt 600 -p "Please type an integer greater than 0: " up_async
if [[ $up_async == 0 ]]; then
echo "Error: \"$up_async\" is less than 1"
echo "NOT adding it to the list"
sleep 5
continue
elif ! [[ $up_async =~ ^[0-9]+$ ]]; then
echo "Error: \"$up_async\" is invalid, it needs to be an integer"
echo "NOT adding it to the list"
sleep 5
continue
else
update_selection+=("-U" "$up_async")
break
fi
done
break
elif [[ $current_selection == 2 ]]; then
while true
do
echo -e "\nHow many applications do you want updating at the same time?"
read -rt 600 -p "Please type an integer greater than 0: " up_async
if [[ $up_async == 0 ]]; then
echo "Error: \"$up_async\" is less than 1"
echo "NOT adding it to the list"
sleep 5
continue
elif ! [[ $up_async =~ ^[0-9]+$ ]]; then
echo "Error: \"$up_async\" is invalid, it needs to be an integer"
echo "NOT adding it to the list"
sleep 5
continue
else
update_selection+=("-u" "$up_async")
break
fi
done
break
elif [[ $current_selection == 0 ]]; then
echo "Exiting.."
exit
else
echo "$current_selection was not an option, try again" && sleep 5
continue
fi
done
while true
do
clear -x
title
echo "Choose Your Update Options"
echo "--------------------------"
echo "1) -b | Back-up your ix-applications dataset, specify a number after -b"
echo "2) -i | Add application to ignore list, one by one, see example below."
echo "3) -r | Roll-back applications if they fail to update"
echo "4) -S | Shutdown applications prior to updating"
echo "5) -v | verbose output"
echo "6) -t | Set a custom timeout in seconds when checking if either an App or Mountpoint correctly Started, Stopped or (un)Mounted. Defaults to 500 seconds"
echo "7) -s | sync catalog"
echo "8) -p | Prune unused/old docker images"
echo "9) --self-update | Updates HeavyScript prior to running any other commands"
echo
echo "99) Remove Update Options, Restart"
echo "00) Done making selections, proceed with update"
echo
echo "0) Exit"
echo
echo "Current Choices"
echo "---------------"
echo "bash heavy_script.sh ${update_selection[*]}"
echo
read -rt 600 -p "Type the Number OR Flag: " current_selection
case $current_selection in
0)
echo "Exiting.."
exit
;;
00)
clear -x
echo "Running \"bash heavy_script.sh ${update_selection[*]}\""
echo
exec bash "$script_name" "${update_selection[@]}"
exit
;;
1 | -b)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-b" && echo -e "\"-b\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it
echo "Up to how many backups should we keep?"
read -rt 600 -p "Please type an integer: " up_backups
! [[ $up_backups =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_backups\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 5 && continue
[[ $up_backups == 0 ]] && echo -e "Error: Number of backups cannot be 0\nNOT adding it to the list" && sleep 5 && continue
update_selection+=("-b" "$up_backups")
;;
2 | -i)
read -rt 600 -p "What is the name of the application we should ignore?: " up_ignore
! [[ $up_ignore =~ ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ ]] && echo -e "Error: \"$up_ignore\" is not a possible option for an application name" && sleep 5 && continue
update_selection+=("-i" "$up_ignore")
;;
3 | -r)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-r" && echo -e "\"-r\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it
update_selection+=("-r")
;;
4 | -S)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-S" && echo -e "\"-S\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it
update_selection+=("-S")
;;
5 | -v)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-v" && echo -e "\"-v\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it
update_selection+=("-v")
;;
6 | -t)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-t" && echo -e "\"-t\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it
echo "What do you want your timeout to be?"
read -rt 600 -p "Please type an integer: " up_timeout
! [[ $up_timeout =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_timeout\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 5 && continue
update_selection+=("-t" "$up_timeout")
;;
7 | -s)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-s" && echo -e "\"-s\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it
update_selection+=("-s")
;;
8 | -p)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-p" && echo -e "\"-p\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it
update_selection+=("-p")
;;
9 | --self-update )
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "--self-update" && echo -e "\"--self-update\" is already on here, skipping" && sleep 5 && continue #If option is already on there, skip it
update_selection+=("--self-update")
;;
99)
count=2
echo "restarting"
for i in "${update_selection[@]:2}"
do
unset "update_selection[$count]"
echo "$i removed"
((count++))
done
sleep 5
continue
;;
*)
echo "\"$current_selection\" was not an option, try again" && sleep 5 && continue
;;
esac
done
;;
*)
echo "\"$selection\" was not an option, please try agian" && sleep 5 && menu
;;
esac
echo
}
export -f menu

81
functions/misc.sh Normal file
View File

@ -0,0 +1,81 @@
#!/bin/bash
sync(){
echo_sync+=("\n\n🅂 🅈 🄽 🄲")
cli -c 'app catalog sync_all' &> /dev/null && echo_sync+=("Catalog sync complete")
#Dump the echo_array, ensures all output is in a neat order.
for i in "${echo_sync[@]}"
do
echo -e "$i"
done
}
export -f sync
prune(){
echo -e "\n\n🄿 🅁 🅄 🄽 🄴"
echo "Pruned Docker Images"
docker image prune -af | grep "^Total" || echo "Failed to Prune Docker Images"
}
export -f prune
title(){
echo ' _ _ _____ _ _ '
echo '| | | | / ___| (_) | | '
echo '| |_| | ___ __ ___ ___ _\ `--. ___ _ __ _ _ __ | |_'
echo "| _ |/ _ \/ _\` \ \ / / | | |\`--. \/ __| '__| | '_ \| __|"
echo '| | | | __/ (_| |\ V /| |_| /\__/ / (__| | | | |_) | |_ '
echo '\_| |_/\___|\__,_| \_/ \__, \____/ \___|_| |_| .__/ \__|'
echo ' __/ | | | '
echo ' |___/ |_| '
echo
}
export -f title
help(){
[[ $help == "true" ]] && clear -x
echo "Access the HeavyScript Menu"
echo "---------------------------"
echo "(Just dont send any other argument)"
echo "bash heavy_script.sh"
echo
echo "Basic Utilities"
echo "---------------"
echo "--mount | Initiates mounting feature, choose between unmounting and mounting PVC data"
echo "--restore | Opens a menu to restore a \"heavy_script\" backup that was taken on your \"ix-applications\" dataset"
echo "--delete-backup | Opens a menu to delete backups on your system"
echo "--dns | list all of your applications DNS names and their web ports"
echo
echo "Update Options"
echo "--------------"
echo "-U | Update all applications, ignores versions"
echo "-U 5 | Same as above, but updates 5 applications at one time"
echo "-u | Update all applications, does not update Major releases"
echo "-u 5 | Same as above, but updates 5 applications at one time"
echo
echo "Additional Update Options"
echo "-------------------------"
echo "-b 14 | Back-up your ix-applications dataset, specify a number after -b"
echo "-i | Add application to ignore list, one by one, see example below."
echo "-r | Roll-back applications if they fail to update"
echo "-S | Shutdown applications prior to updating"
echo "-v | verbose output"
echo "-t 500| Set a custom timeout in seconds when checking if either an App or Mountpoint correctly Started, Stopped or (un)Mounted. Defaults to 500 seconds"
echo "-s | sync catalog"
echo "-p | Prune unused/old docker images"
echo
echo "Examples"
echo "--------"
echo "bash heavy_script.sh"
echo "bash heavy_script.sh -b 14 -i portainer -i arch -i sonarr -t 600 -vrsUp"
echo "bash heavy_script.sh -b 14 -i portainer -i arch -i sonarr -t 600 -vrsp -U 10"
echo "bash /mnt/tank/scripts/heavy_script.sh -t 150 --mount"
echo "bash /mnt/tank/scripts/heavy_script.sh --dns"
echo "bash heavy_script.sh --restore"
echo "bash /mnt/tank/scripts/heavy_script.sh --delete-backup"
echo
exit
}
export -f help

114
functions/mount.sh Normal file
View File

@ -0,0 +1,114 @@
#!/bin/bash
mount(){
while true
do
clear -x
title
echo "1) Mount"
echo "2) Unmount All"
echo
echo "0) Exit"
read -rt 120 -p "Unmount All Please type a number: " selection
case $selection in
0)
echo "Exiting.."
exit
;;
1)
call=$(k3s kubectl get pvc -A | sort -u | awk '{print $1 "\t" $2 "\t" $4}' | sed "s/^0/ /")
mount_list=$(echo "$call" | sed 1d | nl -s ") ")
mount_title=$(echo "$call" | head -n 1)
list=$(echo -e "# $mount_title\n$mount_list" | column -t)
while true
do
clear -x
title
echo "$list"
echo
echo "0) Exit"
read -rt 120 -p "Please type a number: " selection
[[ $selection == 0 ]] && echo "Exiting.." && exit
app=$(echo -e "$list" | grep ^"$selection)" | awk '{print $2}' | cut -c 4- )
[[ -z "$app" ]] && echo "Invalid Selection: $selection, was not an option" && sleep 3 && continue #Check for valid selection. If none, contiue
pvc=$(echo -e "$list" | grep ^"$selection)")
status=$(cli -m csv -c 'app chart_release query name,status' | grep -E "^$app\b" | awk -F ',' '{print $2}'| tr -d " \t\n\r")
if [[ "$status" != "STOPPED" ]]; then
[[ -z $timeout ]] && echo -e "\nDefault Timeout: 500" && timeout=500 || echo -e "\nCustom Timeout: $timeout"
SECONDS=0 && echo -e "\nScaling down $app" && midclt call chart.release.scale "$app" '{"replica_count": 0}' &> /dev/null
else
echo -e "\n$app is already stopped"
fi
while [[ "$SECONDS" -le "$timeout" && "$status" != "STOPPED" ]]
do
status=$(cli -m csv -c 'app chart_release query name,status' | grep -E "^$app\b" | awk -F ',' '{print $2}'| tr -d " \t\n\r")
echo -e "Waiting $((timeout-SECONDS)) more seconds for $app to be STOPPED" && sleep 5
done
data_name=$(echo "$pvc" | awk '{print $3}')
mount=$(echo "$pvc" | awk '{print $4}')
volume_name=$(echo "$pvc" | awk '{print $4}')
mapfile -t full_path < <(zfs list | grep "$volume_name" | awk '{print $1}')
if [[ "${#full_path[@]}" -gt 1 ]]; then #if there is another app with the same name on another pool, use the current pools application, since the other instance is probably old, or unused, or a backup.
echo "$app is using the same volume identifier on more than one pool.. attempting to use your current kubernetes apps pool"
pool=$(cli -c 'app kubernetes config' | grep -E "dataset\s\|" | awk -F '|' '{print $3}' | awk -F '/' '{print $1}' | tr -d " \t\n\r")
full_path=$(zfs list | grep "$volume_name" | grep "$pool" | awk '{print $1}')
fi
echo -e "\nMounting\n$full_path\nTo\n/mnt/heavyscript/$data_name"
zfs set mountpoint=/heavyscript/"$data_name" "$full_path" || echo "Failed to mount $app"
echo -e "Mounted\n\nUnmount with:\nzfs set mountpoint=legacy $full_path && rmdir /mnt/heavyscript/$data_name\n\nOr use the Unmount All option\n"
while true
do
echo -e "\nWould you like to mount anything else?"
echo "1) Yes"
echo "2) No"
read -rt 120 -p "Please type a number: " yesno
case $yesno in
1)
clear -x
title
break
;;
2)
exit
;;
*)
echo "Invalid selection \"$yesno\" was not an option"
sleep 2
continue
;;
esac
done
done
;;
2)
mapfile -t unmount_array < <(basename -a /mnt/heavyscript/* | sed "s/*//")
[[ -z ${unmount_array[*]} ]] && echo "Theres nothing to unmount" && sleep 3 && continue
for i in "${unmount_array[@]}"
do
main=$(k3s kubectl get pvc -A | grep -E "\s$i\s" | awk '{print $1, $2, $4}')
app=$(echo "$main" | awk '{print $1}' | cut -c 4-)
pvc=$(echo "$main" | awk '{print $3}')
mapfile -t path < <(find /mnt/*/ix-applications/releases/"$app"/volumes/ -maxdepth 0 | cut -c 6-)
if [[ "${#path[@]}" -gt 1 ]]; then #if there is another app with the same name on another pool, use the current pools application, since the other instance is probably old, or unused, or a backup.
echo "$i is a name used on more than one pool.. attempting to use your current kubernetes apps pool"
pool=$(cli -c 'app kubernetes config' | grep -E "dataset\s\|" | awk -F '|' '{print $3}' | awk -F '/' '{print $1}' | tr -d " \t\n\r")
full_path=$(find /mnt/"$pool"/ix-applications/releases/"$app"/volumes/ -maxdepth 0 | cut -c 6-)
zfs set mountpoint=legacy "$full_path""$pvc"
echo "$i unmounted" && rmdir /mnt/heavyscript/"$i" || echo "failed to unmount $i"
else
zfs set mountpoint=legacy "$path""$pvc"
echo "$i unmounted" && rmdir /mnt/heavyscript/"$i" || echo "failed to unmount $i"
fi
done
rmdir /mnt/heavyscript
sleep 2
;;
*)
echo "Invalid selection, \"$selection\" was not an option"
sleep 2
continue
;;
esac
done
}
export -f mount

28
functions/self_update.sh Normal file
View File

@ -0,0 +1,28 @@
#!/bin/bash
args=("$@")
self_update() {
git fetch &> /dev/null
echo "🅂 🄴 🄻 🄵 🅄 🄿 🄳 🄰 🅃 🄴"
# TODO: change beta to main once testing is complete
if git diff --name-only origin/beta | grep -qs ".sh" ; then
echo "Found a new version of HeavyScript, updating myself..."
git reset --hard -q
git pull --force -q
count=0
for i in "${args[@]}"
do
[[ "$i" == "--self-update" ]] && unset "args[$count]" && break
((count++))
done
[[ -z ${args[*]} ]] && echo -e "No more arguments, exiting..\n" && exit
echo -e "Running the new version...\n"
sleep 5
exec bash "$script_name" "${args[@]}"
# Now exit this old instance
exit
else
echo -e "HeavyScript is already the latest version\n"
fi
}
export -f self_update

170
functions/update_apps.sh Normal file
View File

@ -0,0 +1,170 @@
#!/bin/bash
commander(){
mapfile -t array < <(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,container_images_update_available,status' | tr -d " \t\r" | grep -E ",true($|,)" | sort)
echo -e "\n\n🅄 🄿 🄳 🄰 🅃 🄴 🅂"
[[ -z ${array[*]} ]] && echo "There are no updates available" && return 0 || echo "Update(s) Available: ${#array[@]}"
echo "Asynchronous Updates: $update_limit"
[[ -z $timeout ]] && echo "Default Timeout: 500" && timeout=500 || echo "Custom Timeout: $timeout"
[[ "$timeout" -le 120 ]] && echo "Warning: Your timeout is set low and may lead to premature rollbacks or skips"
it=0
while true
do
while_status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status')
echo "$while_status" > temp.txt
proc_count=${#processes[@]}
count=0
for proc in "${processes[@]}"
do
kill -0 "$proc" &> /dev/null || { unset "processes[$count]"; ((proc_count--)); }
((count++))
done
if [[ "$proc_count" -ge "$update_limit" ]]; then
sleep 3
elif [[ $it -lt ${#array[@]} ]]; then
update_apps "${array[$it]}" &
processes+=($!)
((it++))
elif [[ $proc_count != 0 ]]; then # Wait for all processes to finish
sleep 3
else # All processes must be completed, break out of loop
break
fi
done
rm temp.txt
}
export -f commander
update_apps(){
app_name=$(echo "${array[$it]}" | awk -F ',' '{print $1}') #print out first catagory, name.
printf '%s\0' "${ignore[@]}" | grep -iFxqz "${app_name}" && echo -e "\n$app_name\nIgnored, skipping" && return 0 #If application is on ignore list, skip
old_app_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #previous/current Application MAJOR Version
new_app_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #new Application MAJOR Version
old_chart_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}' | awk -F '.' '{print $1}') # Old Chart MAJOR version
new_chart_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}' | awk -F '_' '{print $2}' | awk -F '.' '{print $1}') # New Chart MAJOR version
startstatus=$(echo "${array[$it]}" | awk -F ',' '{print $2}') #status of the app: STOPPED / DEPLOYING / ACTIVE
diff_app=$(diff <(echo "$old_app_ver") <(echo "$new_app_ver")) #caluclating difference in major app versions
diff_chart=$(diff <(echo "$old_chart_ver") <(echo "$new_chart_ver")) #caluclating difference in Chart versions
old_full_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}') #Upgraded From
new_full_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}') #Upraded To
rollback_version=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}')
if [[ "$diff_app" == "$diff_chart" || "$update_all_apps" == "true" ]]; then #continue to update
if [[ $stop_before_update == "true" ]]; then # Check to see if user is using -S or not
if [[ "$startstatus" == "STOPPED" ]]; then # if status is already stopped, skip while loop
echo_array+=("\n$app_name")
[[ "$verbose" == "true" ]] && echo_array+=("Updating..")
cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("FAILED")
return 0
else # if status was not STOPPED, stop the app prior to updating
echo_array+=("\n$app_name")
[[ "$verbose" == "true" ]] && echo_array+=("Stopping prior to update..")
midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && SECONDS=0 || echo_array+=("FAILED")
while [[ "$status" != "STOPPED" ]]
do
status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}')
if [[ "$status" == "STOPPED" ]]; then
echo_array+=("Stopped")
[[ "$verbose" == "true" ]] && echo_array+=("Updating..")
cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("Failed to update")
break
elif [[ "$SECONDS" -ge "$timeout" ]]; then
echo_array+=("Error: Run Time($SECONDS) has exceeded Timeout($timeout)")
break
elif [[ "$status" != "STOPPED" ]]; then
[[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be STOPPED")
sleep 10
fi
done
fi
else #user must not be using -S, just update
echo_array+=("\n$app_name")
[[ "$verbose" == "true" ]] && echo_array+=("Updating..")
cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("FAILED")
fi
else
echo_array+=("\n$app_name\nMajor Release, update manually")
return 0
fi
}
export -f update_apps
after_update_actions(){
SECONDS=0
count=0
if [[ $rollback == "true" ]]; then
while true
do
(( count++ ))
status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}')
if [[ "$status" == "ACTIVE" && "$startstatus" == "STOPPED" ]]; then
[[ "$verbose" == "true" ]] && echo_array+=("Returing to STOPPED state..")
midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo_array+=("Stopped")|| echo_array+=("FAILED")
break
elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" != "true" ]]; then
echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nIf this is a slow starting application, set a higher timeout with -t\nIf this applicaion is always DEPLOYING, you can disable all probes under the Healthcheck Probes Liveness section in the edit configuration\nReverting update..")
midclt call chart.release.rollback "$app_name" "{\"item_version\": \"$rollback_version\"}" &> /dev/null
[[ "$startstatus" == "STOPPED" ]] && failed="true" && after_update_actions && unset failed #run back after_update_actions function if the app was stopped prior to update
break
elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" == "true" ]]; then
echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nThe application failed to be ACTIVE even after a rollback,\nManual intervention is required\nAbandoning")
break
elif [[ "$status" == "STOPPED" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Stopped..") && sleep 15 && continue #if reports stopped on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check
echo_array+=("Stopped")
break #if reports stopped any time after the first loop, assume its extermal services.
elif [[ "$status" == "ACTIVE" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Active..") && sleep 15 && continue #if reports active on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check
echo_array+=("Active")
break #if reports active any time after the first loop, assume actually active.
else
[[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE")
sleep 15
continue
fi
done
else
if [[ "$startstatus" == "STOPPED" ]]; then
while true #using a constant while loop, then breaking out of the loop with break commands below.
do
(( count++ ))
status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}')
if [[ "$status" == "STOPPED" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Stopped..") && sleep 15 && continue #if reports stopped on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check
echo_array+=("Stopped") #assume actually stopped anytime AFTER the first loop
break
elif [[ "$status" == "ACTIVE" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Active..") && sleep 15 && continue #if reports active on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check
[[ "$verbose" == "true" ]] && echo_array+=("Returing to STOPPED state..")
midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo_array+=("Stopped")|| echo_array+=("FAILED")
break
elif [[ "$SECONDS" -ge "$timeout" ]]; then
echo_array+=("Error: Run Time($SECONDS) has exceeded Timeout($timeout)")
break
else
[[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE")
sleep 10
continue
fi
done
fi
fi
#Dump the echo_array, ensures all output is in a neat order.
for i in "${echo_array[@]}"
do
echo -e "$i"
done
}
export -f after_update_actions

445
heavy_script.sh
View File

@ -1,375 +1,38 @@
#!/bin/bash #!/bin/bash
#If no argument is passed, kill the script.
[[ -z "$*" || "-" == "$*" || "--" == "$*" ]] && echo "This script requires an argument, use --help for help" && exit
args=("$@") # cd to script, this ensures the script can find the source scripts below, even when ran from a seperate directory
self_update() {
script=$(readlink -f "$0") script=$(readlink -f "$0")
script_path=$(dirname "$script") script_path=$(dirname "$script")
script_name="heavy_script.sh" script_name="heavy_script.sh"
cd "$script_path" || exit cd "$script_path" || { echo "Error: Failed to change to script directory" ; exit ; }
git fetch &> /dev/null
if git diff --name-only origin/main | grep -q "$script_name" ; then
echo "Found a new version of HeavyScript, updating myself..."
git reset --hard -q
git pull --force -q
echo -e "Running the new version...\n"
count=0
for i in "${args[@]}"
do
[[ "$i" == "--self-update" ]] && unset "args[$count]" && break
((count++))
done
sleep 5
exec bash "$script_name" "${args[@]}"
# Now exit this old instance
exit
else
echo -e "HeavyScript is already the latest version\n"
fi
}
help(){ # shellcheck source=functions/backup.sh
[[ $help == "true" ]] && clear -x source functions/backup.sh
echo "Basic Utilities" # shellcheck source=functions/dns.sh
echo "--mount | Initiates mounting feature, choose between unmounting and mounting PVC data" source functions/dns.sh
echo "--restore | Opens a menu to restore a \"heavy_script\" backup that was taken on your \"ix-applications\" dataset" # shellcheck source=functions/menu.sh
echo "--delete-backup | Opens a menu to delete backups on your system" source functions/menu.sh
echo "--dns | list all of your applications DNS names and their web ports" # shellcheck source=functions/misc.sh
echo source functions/misc.sh
echo "Update Options" # shellcheck source=functions/mount.sh
echo "-U | Update all applications, ignores versions" source functions/mount.sh
echo "-u | Update all applications, does not update Major releases" # shellcheck source=functions/self_update.sh
echo "-b | Back-up your ix-applications dataset, specify a number after -b" source functions/self_update.sh
echo "-i | Add application to ignore list, one by one, see example below." # shellcheck source=functions/update_apps.sh
echo "-R | THIS OPTION WILL DEPRICATE SOON, USE \"-r\" instead. Roll-back applications if they fail to update" source functions/update_apps.sh
echo "-r | Roll-back applications if they fail to update"
echo "-S | Shutdown applications prior to updating"
echo "-v | verbose output"
echo "-t | Set a custom timeout in seconds when checking if either an App or Mountpoint correctly Started, Stopped or (un)Mounted. Defaults to 500 seconds"
echo "-s | sync catalog"
echo "-p | Prune unused/old docker images"
echo
echo "Examples"
echo "bash heavy_script.sh -b 14 -i portainer -i arch -i sonarr -i radarr -t 600 -vrsUp"
echo "bash /mnt/tank/scripts/heavy_script.sh -t 150 --mount"
echo "bash /mnt/tank/scripts/heavy_script.sh --dns"
echo "bash heavy_script.sh --restore"
echo "bash /mnt/tank/scripts/heavy_script.sh --delete-backup"
echo
exit
}
export -f help
deleteBackup(){
clear -x && echo "pulling all restore points.."
list_backups=$(cli -c 'app kubernetes list_backups' | sort -t '_' -Vr -k2,7 | tr -d " \t\r" | awk -F '|' '{print $2}' | nl | column -t)
clear -x
[[ -z "$list_backups" ]] && echo "No restore points available" && exit || { title; echo -e "Choose a restore point to delete\nThese may be out of order if they are not HeavyScript backups" ; }
echo "$list_backups" && read -t 600 -p "Please type a number: " selection && restore_point=$(echo "$list_backups" | grep ^"$selection " | awk '{print $2}')
[[ -z "$selection" ]] && echo "Your selection cannot be empty" && exit #Check for valid selection. If none, kill script
[[ -z "$restore_point" ]] && echo "Invalid Selection: $selection, was not an option" && exit #Check for valid selection. If none, kill script
echo -e "\nWARNING:\nYou CANNOT go back after deleting your restore point" || { echo "FAILED"; exit; }
echo -e "\n\nYou have chosen:\n$restore_point\n\nWould you like to continue?" && echo -e "1 Yes\n2 No" && read -t 120 -p "Please type a number: " yesno || { echo "FAILED"; exit; }
if [[ $yesno == "1" ]]; then
echo -e "\nDeleting $restore_point" && cli -c 'app kubernetes delete_backup backup_name=''"'"$restore_point"'"' &>/dev/null && echo "Sucessfully deleted" || echo "Deletion Failed"
elif [[ $yesno == "2" ]]; then
echo "You've chosen NO, killing script."
else
echo "Invalid Selection"
fi
}
export -f deleteBackup
backup(){ #If no argument is passed, kill the script.
echo -e "\nNumber of backups was set to $number_of_backups" [[ -z "$*" || "-" == "$*" || "--" == "$*" ]] && menu
date=$(date '+%Y_%m_%d_%H_%M_%S')
[[ "$verbose" == "true" ]] && cli -c 'app kubernetes backup_chart_releases backup_name=''"'HeavyScript_"$date"'"'
[[ -z "$verbose" ]] && echo -e "\nNew Backup Name:" && cli -c 'app kubernetes backup_chart_releases backup_name=''"'HeavyScript_"$date"'"' | tail -n 1
mapfile -t list_backups < <(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -Vr -k2,7 | awk -F '|' '{print $2}'| tr -d " \t\r")
if [[ ${#list_backups[@]} -gt "number_of_backups" ]]; then
echo -e "\nDeleting the oldest backup(s) for exceeding limit:"
overflow=$(( ${#list_backups[@]} - "$number_of_backups" ))
mapfile -t list_overflow < <(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -V -k2,7 | awk -F '|' '{print $2}'| tr -d " \t\r" | head -n "$overflow")
for i in "${list_overflow[@]}"
do
cli -c 'app kubernetes delete_backup backup_name=''"'"$i"'"' &> /dev/null || echo "Failed to delete $i"
echo "$i"
done
fi
}
export -f backup
restore(){
clear -x && echo "pulling restore points.."
list_backups=$(cli -c 'app kubernetes list_backups' | grep "HeavyScript_" | sort -t '_' -Vr -k2,7 | tr -d " \t\r" | awk -F '|' '{print $2}' | nl | column -t)
clear -x
[[ -z "$list_backups" ]] && echo "No HeavyScript restore points available" && exit || { title; echo "Choose a restore point" ; }
echo "$list_backups" && read -t 600 -p "Please type a number: " selection && restore_point=$(echo "$list_backups" | grep ^"$selection " | awk '{print $2}')
[[ -z "$selection" ]] && echo "Your selection cannot be empty" && exit #Check for valid selection. If none, kill script
[[ -z "$restore_point" ]] && echo "Invalid Selection: $selection, was not an option" && exit #Check for valid selection. If none, kill script
echo -e "\nWARNING:\nThis is NOT guranteed to work\nThis is ONLY supposed to be used as a LAST RESORT\nConsider rolling back your applications instead if possible" || { echo "FAILED"; exit; }
echo -e "\n\nYou have chosen:\n$restore_point\n\nWould you like to continue?" && echo -e "1 Yes\n2 No" && read -t 120 -p "Please type a number: " yesno || { echo "FAILED"; exit; }
if [[ $yesno == "1" ]]; then
echo -e "\nStarting Backup, this will take a LONG time." && cli -c 'app kubernetes restore_backup backup_name=''"'"$restore_point"'"' || echo "Restore FAILED"
elif [[ $yesno == "2" ]]; then
echo "You've chosen NO, killing script. Good luck."
else
echo "Invalid Selection"
fi
}
export -f restore
dns(){
clear -x
echo "Generating DNS Names.."
#ignored dependency pods, may need to add more in the future.
dep_ignore="\-cronjob\-|^kube-system|\ssvclb|NAME|\-memcached\-.[^custom\-app]|\-postgresql\-.[^custom\-app]|\-redis\-.[^custom\-app]|\-mariadb\-.[^custom\-app]|\-promtail\-.[^custom\-app]"
# Pulling pod names
mapfile -t main < <(k3s kubectl get pods -A | grep -Ev "$dep_ignore" | sort)
# Pulling all ports
all_ports=$(k3s kubectl get service -A)
clear -x
count=0
for i in "${main[@]}"
do
[[ count -le 0 ]] && echo -e "\n" && ((count++))
appName=$(echo "$i" | awk '{print $2}' | sed 's/-[^-]*-[^-]*$//' | sed 's/-0//')
ixName=$(echo "$i" | awk '{print $1}')
port=$(echo "$all_ports" | grep -E "\s$appName\s" | awk '{print $6}' | grep -Eo "^[[:digit:]]+{1}")
echo -e "$appName.$ixName.svc.cluster.local $port"
done | uniq | nl -b t | sed 's/\s\s\s$/- -------- ----/' | column -t -R 1 -N "#,DNS_Name,Port" -L
}
export -f dns
mount(){
clear -x
title
echo -e "1 Mount\n2 Unmount All" && read -t 600 -p "Please type a number: " selection
[[ -z "$selection" ]] && echo "Your selection cannot be empty" && exit #Check for valid selection. If none, kill script
if [[ $selection == "1" ]]; then
list=$(k3s kubectl get pvc -A | sort -u | awk '{print NR-1, "\t" $1 "\t" $2 "\t" $4}' | column -t | sed "s/^0/ /")
echo "$list" && read -t 120 -p "Please type a number: " selection
[[ -z "$selection" ]] && echo "Your selection cannot be empty" && exit #Check for valid selection. If none, kill script
app=$(echo -e "$list" | grep ^"$selection " | awk '{print $2}' | cut -c 4- )
[[ -z "$app" ]] && echo "Invalid Selection: $selection, was not an option" && exit #Check for valid selection. If none, kill script
pvc=$(echo -e "$list" | grep ^"$selection ")
status=$(cli -m csv -c 'app chart_release query name,status' | grep -E "^$app\b" | awk -F ',' '{print $2}'| tr -d " \t\n\r")
if [[ "$status" != "STOPPED" ]]; then
[[ -z $timeout ]] && echo -e "\nDefault Timeout: 500" && timeout=500 || echo -e "\nCustom Timeout: $timeout"
SECONDS=0 && echo -e "\nScaling down $app" && midclt call chart.release.scale "$app" '{"replica_count": 0}' &> /dev/null
else
echo -e "\n$app is already stopped"
fi
while [[ "$SECONDS" -le "$timeout" && "$status" != "STOPPED" ]]
do
status=$(cli -m csv -c 'app chart_release query name,status' | grep -E "^$app\b" | awk -F ',' '{print $2}'| tr -d " \t\n\r")
echo -e "Waiting $((timeout-SECONDS)) more seconds for $app to be STOPPED" && sleep 5
done
data_name=$(echo "$pvc" | awk '{print $3}')
mount=$(echo "$pvc" | awk '{print $4}')
volume_name=$(echo "$pvc" | awk '{print $4}')
full_path=$(zfs list | grep "$volume_name" | awk '{print $1}')
echo -e "\nMounting\n$full_path\nTo\n/mnt/heavyscript/$data_name" && zfs set mountpoint=/heavyscript/"$data_name" "$full_path" && echo -e "Mounted\n\nUnmount with:\nzfs set mountpoint=legacy "$full_path" && rmdir /mnt/heavyscript/"$data_name"\n\nOr use the Unmount All option\n"
exit
elif [[ $selection == "2" ]]; then
mapfile -t unmount_array < <(basename -a /mnt/heavyscript/* | sed "s/*//")
[[ -z $unmount_array ]] && echo "Theres nothing to unmount" && exit
for i in "${unmount_array[@]}"
do
main=$(k3s kubectl get pvc -A | grep -E "\s$i\s" | awk '{print $1, $2, $4}')
app=$(echo "$main" | awk '{print $1}' | cut -c 4-)
pvc=$(echo "$main" | awk '{print $3}')
mapfile -t path < <(find /mnt/*/ix-applications/releases/"$app"/volumes/ -maxdepth 0 | cut -c 6-)
if [[ "${#path[@]}" -gt 1 ]]; then #if there is another app with the same name on another pool, use the current pools application, since the other instance is probably old, or unused.
echo "$i is a name used on more than one pool.. attempting to use your current kubernetes apps pool"
pool=$(cli -c 'app kubernetes config' | grep -E "dataset\s\|" | awk -F '|' '{print $3}' | awk -F '/' '{print $1}' | tr -d " \t\n\r")
full_path=$(find /mnt/"$pool"/ix-applications/releases/"$app"/volumes/ -maxdepth 0 | cut -c 6-)
zfs set mountpoint=legacy "$full_path""$pvc" && echo "$i unmounted" && rmdir /mnt/heavyscript/"$i" || echo "failed to unmount $i"
else
zfs set mountpoint=legacy "$path""$pvc" && echo "$i unmounted" && rmdir /mnt/heavyscript/"$i" || echo "failed to unmount $i"
fi
done
rmdir /mnt/heavyscript
else
echo "Invalid selection, \"$selection\" was not an option"
fi
}
export -f mount
sync(){
echo -e "\nSyncing all catalogs, please wait.." && cli -c 'app catalog sync_all' &> /dev/null && echo -e "Catalog sync complete"
}
export -f sync
update_apps(){
# Replace with line below after testing
# cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,container_images_update_available,status' | tr -d " \t\r" | grep -E ",true($|,)" | sort
mapfile -t array < <(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,container_images_update_available,status' | grep -E ",true(,|\b)" | sort)
[[ -z $array ]] && echo -e "\nThere are no updates available" && return 0 || echo -e "\n${#array[@]} update(s) available"
[[ -z $timeout ]] && echo -e "\nDefault Timeout: 500" && timeout=500 || echo -e "\nCustom Timeout: $timeout"
[[ "$timeout" -le 120 ]] && echo "Warning: Your timeout is set low and may lead to premature rollbacks or skips"
for i in "${array[@]}"
do
app_name=$(echo "$i" | awk -F ',' '{print $1}') #print out first catagory, name.
old_app_ver=$(echo "$i" | awk -F ',' '{print $4}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #previous/current Application MAJOR Version
new_app_ver=$(echo "$i" | awk -F ',' '{print $5}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #new Application MAJOR Version
old_chart_ver=$(echo "$i" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}' | awk -F '.' '{print $1}') # Old Chart MAJOR version
new_chart_ver=$(echo "$i" | awk -F ',' '{print $5}' | awk -F '_' '{print $2}' | awk -F '.' '{print $1}') # New Chart MAJOR version
status=$(echo "$i" | awk -F ',' '{print $2}') #status of the app: STOPPED / DEPLOYING / ACTIVE
startstatus=$status
diff_app=$(diff <(echo "$old_app_ver") <(echo "$new_app_ver")) #caluclating difference in major app versions
diff_chart=$(diff <(echo "$old_chart_ver") <(echo "$new_chart_ver")) #caluclating difference in Chart versions
old_full_ver=$(echo "$i" | awk -F ',' '{print $4}') #Upgraded From
new_full_ver=$(echo "$i" | awk -F ',' '{print $5}') #Upraded To
rollback_version=$(echo "$i" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}')
printf '%s\0' "${ignore[@]}" | grep -iFxqz "${app_name}" && echo -e "\n$app_name\nIgnored, skipping" && continue #If application is on ignore list, skip
if [[ "$diff_app" == "$diff_chart" || "$update_all_apps" == "true" ]]; then #continue to update
if [[ $stop_before_update == "true" ]]; then # Check to see if user is using -S or not
if [[ "$status" == "STOPPED" ]]; then # if status is already stopped, skip while loop
echo -e "\n$app_name"
[[ "$verbose" == "true" ]] && echo "Updating.."
cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo -e "Updated\n$old_full_ver\n$new_full_ver" && after_update_actions || echo "FAILED"
continue
else # if status was not STOPPED, stop the app prior to updating
echo -e "\n$app_name"
[[ "$verbose" == "true" ]] && echo "Stopping prior to update.."
midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && SECONDS=0 || echo -e "FAILED"
while [[ "$status" != "STOPPED" ]]
do
status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status' | grep "^$app_name," | awk -F ',' '{print $2}')
if [[ "$status" == "STOPPED" ]]; then
echo "Stopped"
[[ "$verbose" == "true" ]] && echo "Updating.."
cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo -e "Updated\n$old_full_ver\n$new_full_ver" && after_update_actions || echo "Failed to update"
break
elif [[ "$SECONDS" -ge "$timeout" ]]; then
echo "Error: Run Time($SECONDS) has exceeded Timeout($timeout)"
break
elif [[ "$status" != "STOPPED" ]]; then
[[ "$verbose" == "true" ]] && echo "Waiting $((timeout-SECONDS)) more seconds for $app_name to be STOPPED"
sleep 10
continue
fi
done
fi
else #user must not be using -S, just update
echo -e "\n$app_name"
[[ "$verbose" == "true" ]] && echo "Updating.."
cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo -e "Updated\n$old_full_ver\n$new_full_ver" && after_update_actions || echo "FAILED"
fi
else
echo -e "\n$app_name\nMajor Release, update manually"
continue
fi
done
}
export -f update_apps
after_update_actions(){
SECONDS=0
count=0
if [[ $rollback == "true" ]]; then
while [[ "0" != "1" ]]
do
(( count++ ))
status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status' | grep "^$app_name," | awk -F ',' '{print $2}')
if [[ "$status" == "ACTIVE" && "$startstatus" == "STOPPED" ]]; then
[[ "$verbose" == "true" ]] && echo "Returing to STOPPED state.."
midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo "Stopped"|| echo "FAILED"
break
elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" != "true" ]]; then
echo -e "Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nIf this is a slow starting application, set a higher timeout with -t\nIf this applicaion is always DEPLOYING, you can disable all probes under the Healthcheck Probes Liveness section in the edit configuration\nReverting update.."
midclt call chart.release.rollback "$app_name" "{\"item_version\": \"$rollback_version\"}" &> /dev/null
[[ "$startstatus" == "STOPPED" ]] && failed="true" && after_update_actions && unset failed #run back after_update_actions function if the app was stopped prior to update
break
elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" == "true" ]]; then
echo -e "Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nThe application failed to be ACTIVE even after a rollback,\nManual intervention is required\nAbandoning"
break
elif [[ "$status" == "STOPPED" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo "Verifying Stopped.." && sleep 15 && continue #if reports stopped on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check
echo "Stopped" && break #if reports stopped any time after the first loop, assume its extermal services.
elif [[ "$status" == "ACTIVE" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo "Verifying Active.." && sleep 15 && continue #if reports active on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check
echo "Active" && break #if reports active any time after the first loop, assume actually active.
else
[[ "$verbose" == "true" ]] && echo "Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE"
sleep 15
continue
fi
done
else
if [[ "$startstatus" == "STOPPED" ]]; then
while [[ "0" != "1" ]] #using a constant while loop, then breaking out of the loop with break commands below.
do
(( count++ ))
status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status' | grep "^$app_name," | awk -F ',' '{print $2}')
if [[ "$status" == "STOPPED" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo "Verifying Stopped.." && sleep 15 && continue #if reports stopped on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check
echo "Stopped" && break #assume actually stopped anytime AFTER the first loop
break
elif [[ "$status" == "ACTIVE" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo "Verifying Active.." && sleep 15 && continue #if reports active on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check
[[ "$verbose" == "true" ]] && echo "Returing to STOPPED state.."
midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo "Stopped"|| echo "FAILED"
break
elif [[ "$SECONDS" -ge "$timeout" ]]; then
echo "Error: Run Time($SECONDS) has exceeded Timeout($timeout)"
break
else
[[ "$verbose" == "true" ]] && echo "Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE"
sleep 10
continue
fi
done
fi
fi
}
export -f after_update_actions
prune(){
echo -e "\nPruning Docker Images" && docker image prune -af | grep "^Total" || echo "Failed to Prune Docker Images"
}
export -f prune
title(){
echo ' _ _ _____ _ _ '
echo '| | | | / ___| (_) | | '
echo '| |_| | ___ __ ___ ___ _\ `--. ___ _ __ _ _ __ | |_'
echo "| _ |/ _ \/ _\` \ \ / / | | |\`--. \/ __| '__| | '_ \| __|"
echo '| | | | __/ (_| |\ V /| |_| /\__/ / (__| | | | |_) | |_ '
echo '\_| |_/\___|\__,_| \_/ \__, \____/ \___|_| |_| .__/ \__|'
echo ' __/ | | | '
echo ' |___/ |_| '
echo
}
export -f title
# Parse script options # Parse script options
while getopts ":si:rb:t:uUpSRv-:" opt while getopts ":sirb:t:uUpSRv-:" opt
do do
case $opt in case $opt in
-) -)
@ -393,44 +56,62 @@ do
deleteBackup="true" deleteBackup="true"
;; ;;
*) *)
echo -e "Invalid Option \"--$OPTARG\"\n" && help echo -e "Invalid Option \"--$OPTARG\"\n"
exit help
;; ;;
esac esac
;; ;;
\?)
echo -e "Invalid Option \"-$OPTARG\"\n" && help
exit
;;
:) :)
echo -e "Option: \"-$OPTARG\" requires an argument\n" && help echo -e "Option: \"-$OPTARG\" requires an argument\n"
exit help
;; ;;
b) b)
re='^[0-9]+$'
number_of_backups=$OPTARG number_of_backups=$OPTARG
! [[ $OPTARG =~ $re ]] && echo -e "Error: -b needs to be assigned an interger\n\"""$number_of_backups""\" is not an interger" >&2 && exit ! [[ $OPTARG =~ ^[0-9]+$ ]] && echo -e "Error: -b needs to be assigned an interger\n\"""$number_of_backups""\" is not an interger" >&2 && exit
[[ "$number_of_backups" -le 0 ]] && echo "Error: Number of backups is required to be at least 1" && exit [[ "$number_of_backups" -le 0 ]] && echo "Error: Number of backups is required to be at least 1" && exit
backup="true"
;; ;;
r) r)
rollback="true" rollback="true"
;; ;;
i) i)
if ! [[ $OPTARG =~ ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ ]]; then # Using case insensitive version of the regex used by Truenas Scale
echo -e "Error: \"$OPTARG\" is not a possible option for an application name"
exit
else
ignore+=("$OPTARG") ignore+=("$OPTARG")
fi
;; ;;
t) t)
re='^[0-9]+$'
timeout=$OPTARG timeout=$OPTARG
! [[ $timeout =~ $re ]] && echo -e "Error: -t needs to be assigned an interger\n\"""$timeout""\" is not an interger" >&2 && exit ! [[ $timeout =~ ^[0-9]+$ ]] && echo -e "Error: -t needs to be assigned an interger\n\"""$timeout""\" is not an interger" >&2 && exit
;; ;;
s) s)
sync="true" sync="true"
;; ;;
U) U)
update_all_apps="true" update_all_apps="true"
# Check next positional parameter
eval nextopt=${!OPTIND}
# existing or starting with dash?
if [[ -n $nextopt && $nextopt != -* ]] ; then
OPTIND=$((OPTIND + 1))
update_limit="$nextopt"
else
update_limit=1
fi
;; ;;
u) u)
update_apps="true" update_apps="true"
# Check next positional parameter
eval nextopt=${!OPTIND}
# existing or starting with dash?
if [[ -n $nextopt && $nextopt != -* ]] ; then
OPTIND=$((OPTIND + 1))
update_limit="$nextopt"
else
update_limit=1
fi
;; ;;
S) S)
stop_before_update="true" stop_before_update="true"
@ -438,21 +119,22 @@ do
p) p)
prune="true" prune="true"
;; ;;
R)
rollback="true"
echo "WARNING: -R is being transisitioned to -r, this is due to a refactor in the script. Please Make the change ASAP!"
;;
v) v)
verbose="true" verbose="true"
;; ;;
\?)
echo -e "Invalid Option \"-$OPTARG\"\n"
help
;;
*) *)
echo -e "Invalid Option \"--$OPTARG\"\n" && help echo -e "Invalid Option \"-$OPTARG\"\n"
exit help
;; ;;
esac esac
done done
#exit if incompatable functions are called #exit if incompatable functions are called
[[ "$update_all_apps" == "true" && "$update_apps" == "true" ]] && echo -e "-U and -u cannot BOTH be called" && exit [[ "$update_all_apps" == "true" && "$update_apps" == "true" ]] && echo -e "-U and -u cannot BOTH be called" && exit
@ -463,7 +145,16 @@ done
[[ "$dns" == "true" ]] && dns && exit [[ "$dns" == "true" ]] && dns && exit
[[ "$restore" == "true" ]] && restore && exit [[ "$restore" == "true" ]] && restore && exit
[[ "$mount" == "true" ]] && mount && exit [[ "$mount" == "true" ]] && mount && exit
[[ "$number_of_backups" -ge 1 ]] && backup if [[ "$backup" == "true" && "$sync" == "true" ]]; then # Run backup and sync at the same time
[[ "$sync" == "true" ]] && sync echo "🅃 🄰 🅂 🄺 🅂 :"
[[ "$update_all_apps" == "true" || "$update_apps" == "true" ]] && update_apps echo -e "-Backing up ix-applications dataset\n-Syncing catalog(s)"
echo -e "This can take a LONG time, please wait for both output..\n"
backup &
sync &
wait
fi
[[ "$backup" == "true" && -z "$sync" ]] && echo "Backing up \"ix-applications\" dataset, please wait.." && backup
[[ "$sync" == "true" && -z "$backup" ]] && echo "Syncing catalogs, this takes a LONG time, please wait.." && sync
[[ "$update_all_apps" == "true" || "$update_apps" == "true" ]] && commander
[[ "$prune" == "true" ]] && prune [[ "$prune" == "true" ]] && prune