diff --git a/README.md b/README.md index 7199c34e..ce8d669c 100644 --- a/README.md +++ b/README.md @@ -14,32 +14,31 @@
-## Arguments +## The Menu -| Flag | Example | Parameter | Description | -|----------------- |------------------------ |----------------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| NULL | NULL | NULL | If you choose not to supply an option, it will open the menu for easier access to the utilities | -| --self-update | --self-update | None | Updates HeavyScript prior to running it
_You no longer need to git pull_ | -| --delete-backup | --delete-backup | None | Opens a menu to delete backups
_Useful if you need to delete old system backups or backups from other scripts_ | -| --restore | --restore | None | Restore HeavyScript specific `ix-applications dataset` snapshot | -| --mount | --mount | None | Initiates mounting feature
Choose between unmounting and mounting PVC data | -| --dns | --dns | None | list all of your applications DNS names and their web ports | -| -U | -U
-U 5 | None or Integer | Update applications, ignoring major version changes
_Optionally, you can supply a number after the argument to update multiple applications at once_ | -| -u | -u
-u 5 | None or Integer | Update applications, do NOT update if there was a major version change
_Optionally, you can supply a number after the argument to update multiple applications at once_ | -| -b | -b 10 | Integer | Backup `ix-appliactions` dataset
_Creates backups up to the number you've chosen_ | -| -i | -i nextcloud -i sonarr | String | Applications listed will be ignored during updating
_List one application after another as shown in the example_ | -| -r | -r | None | Monitors applications after they update
If the app does not become "ACTIVE" after either:
The custom Timeout, or Default Timeout,
rollback the application. | -| -v | -v | None | Verbose Output
_Look at the bottom of this page for an example_ | -| -S | -S | None | Shutdown the application prior to updating it | -| -t | -t 150 | Integer | Set a custom timeout to be used with either:
`-m`
_Time the script will wait for application to be "STOPPED"_
or
`-(u\|U)`
_Time the script will wait for application to be either "STOPPED" or "ACTIVE"_ | -| -s | -s | None | Sync Catalogs prior to updating | -| -p | -p | None | Prune old/unused docker images | +![image](https://user-images.githubusercontent.com/20793231/185020236-7b389499-8081-407d-b653-10dffd70de8c.png) +> Access this with `bash heavy_script.sh` + +
+
+ +## Update Arguments +| Flag | Example | Parameter | Description | +|--------------- |------------------------ |----------------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| -U | -U
-U 5 | None or Integer | Update applications, ignoring major version changes
_Optionally, you can supply a number after the argument to update multiple applications at once_ | +| -u | -u
-u 5 | None or Integer | Update applications, do NOT update if there was a major version change
_Optionally, you can supply a number after the argument to update multiple applications at once_ | +| -b | -b 14 | Integer | Backup `ix-appliactions` dataset
_Creates backups up to the number you've chosen_ | +| -i | -i nextcloud -i sonarr | String | Applications listed will be ignored during updating
_List one application after another as shown in the example_ | +| -r | -r | None | Monitors applications after they update
If the app does not become "ACTIVE" after either:
The custom Timeout, or Default Timeout,
rollback the application. | +| -v | -v | None | Verbose Output
_Look at the bottom of this page for an example_ | +| -S | -S | None | Shutdown the application prior to updating it | +| -t | -t 150 | Integer | Set a custom timeout to be used with either:
`-m`
_Time the script will wait for application to be "STOPPED"_
or
`-(u\|U)`
_Time the script will wait for application to be either "STOPPED" or "ACTIVE"_ | +| -s | -s | None | Sync Catalogs prior to updating | +| -p | -p | None | Prune old/unused docker images | +| --self-update | --self-update | None | Updates HeavyScript prior to running any other commands | -
-
- -### Examples +### Example #### Typical Cron Job ``` bash heavy_script.sh --self-update -b 10 -i portainer -i arch -i sonarr -i radarr -t 600 -rsp -u 5 @@ -62,7 +61,22 @@ bash heavy_script.sh --self-update -b 10 -i portainer -i arch -i sonarr -i radar > `--self-update` Will update the script prior to running anything else. +
+
+## Other Utilities +> All of these can ALSO be accessed with the HeavyScript menu, that you can access simply by not providing an argument `bash heavy_script.sh` + +| Flag | Description | +|-----------------|----------------------------------------------------------------------------------------------| +| --mount | Initiates mounting feature, choose between unmounting and mounting PVC data | +| --restore | Opens a menu to restore a heavy_script backup that was taken on your ix-applications dataset | +| --delete-backup | Opens a menu to delete backups on your system | +| --dns | list all of your applications DNS names and their web ports | +| --cmd | Open a shell for one of your applications | + + +### Examples #### Mounting PVC Data ``` @@ -87,10 +101,10 @@ bash /mnt/tank/scripts/heavy_script/heavy_script.sh --delete-backup bash /mnt/tank/scripts/heavy_script/heavy_script.sh --dns ``` -#### My personal Cron Job +#### Open a Containers Shell ``` -bash /mnt/speed/scripts/heavy_script/heavy_script.sh --self-update -b 10 -rsp -u 10 +bash /mnt/speed/scripts/heavy_script/heavy_script.sh --cmd ```
@@ -165,7 +179,7 @@ bash heavyscript.sh --self-update -b 10 -supr | Name | Value | Reason | |------------------------ |------------------------------------------------------------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `Description` | HeavyScript git pull and Update apps | This is up to you, put whatever you think is a good description in here | -| `Command` | `bash /PATH/TO/HEAVY_SCRIPT_DIRECTORY/heavy_script.sh --self-update -b 10 -rsup` | This is the command you will be running on your schedule I personally use: `bash /mnt/speed/scripts/heavy_script/heavy_script.sh --self-update -b 10 -rsup` | +| `Command` | `bash /PATH/TO/HEAVY_SCRIPT_DIRECTORY/heavy_script.sh --self-update -b 10 -rsp -u 10` | This is the command you will be running on your schedule I personally use: `bash /mnt/speed/scripts/heavy_script/heavy_script.sh --self-update -b 10 -rsp -u 10` | | `Run As User` | `root` | Running the script as `root` is REQUIRED. You cannot access all of the kubernetes functions without this user. | | `Schedule` | Up to you, I run mine everyday at `0400` | Again up to you | | `Hide Standard Output` | `False` or Unticked | I like to receive an email report of how the script ran, what apps updated etc. | diff --git a/functions/cmd_to_container.sh b/functions/cmd_to_container.sh new file mode 100644 index 00000000..dfbc5215 --- /dev/null +++ b/functions/cmd_to_container.sh @@ -0,0 +1,118 @@ +#!/bin/bash + + +cmd_to_container(){ +app_name=$(k3s crictl pods -s ready --namespace ix | sed -E 's/[[:space:]]([0-9]*|About)[a-z0-9 ]{5,12}ago[[:space:]]//' | sed '1d' | awk '{print $4}' | cut -c4- | sort -u | nl -s ") " | column -t) +while true +do + clear -x + title + echo "$app_name" + echo + echo "0) Exit" + read -rt 120 -p "Please type a number: " selection || { echo -e "\nFailed to make a selection in time" ; exit; } + if [[ $selection == 0 ]]; then + echo "Exiting.." + exit + elif ! echo -e "$app_name" | grep -qs ^"$selection)" ; then + echo "Error: \"$selection\" was not an option.. Try again" + sleep 3 + continue + else + break + fi +done +app_name=$(echo -e "$app_name" | grep ^"$selection)" | awk '{print $2}') +mapfile -t pod_id < <(k3s crictl pods -s ready --namespace ix | grep -E "[[:space:]]$app_name([[:space:]]|-([-[:alnum:]])*[[:space:]])" | awk '{print $1}') +search=$(k3s crictl ps -a -s running | sed -E 's/[[:space:]]([0-9]*|About)[a-z0-9 ]{5,12}ago[[:space:]]//') +for pod in "${pod_id[@]}" +do + if [[ $(echo "$search" | grep "$pod" | awk '{print $4}' | tr -d " \t\r " | wc -l) -gt 1 ]]; then + readarray -t containers <<<"$(echo "$search" | grep "$pod" | awk '{print $4}' | tr -d " \t\r ")" + continue + fi + printf '%s\0' "${containers[@]}" | grep -Fxqz -- "$(echo "$search" | grep "$pod" | awk '{print $4}' | tr -d " \t\r ")" && continue + containers+=("$(echo "$search" | grep "$pod" | awk '{print $4}' | tr -d " \t\r ")") + +done +case "${#containers[@]}" in + 0) + echo -e "No containers available\nAre you sure the application in running?" + exit + ;; + 1) + container=$(echo "$search" | grep "${pod_id[0]}" | awk '{print $4}') + container_id=$(echo "$search" | grep -E "[[:space:]]${container}[[:space:]]" | awk '{print $1}') + ;; + + *) + while true + do + clear -x + title + cont_search=$( + for i in "${containers[@]}" + do + echo "$i" + done | nl -s ") " | column -t + ) + echo "$cont_search" + echo + echo "0) Exit" + read -rt 120 -p "Choose a container by number: " selection || { echo -e "\nFailed to make a selection in time" ; exit; } + if [[ $selection == 0 ]]; then + echo "Exiting.." + exit + elif ! echo -e "$cont_search}" | grep -qs ^"$selection)" ; then + echo "Error: \"$selection\" was not an option.. Try again" + sleep 3 + continue + else + break + fi + done + container=$(echo "$cont_search" | grep ^"$selection)" | awk '{print $2}') + container_id=$(echo "$search" | grep -E "[[:space:]]${container}[[:space:]]" | awk '{print $1}') + ;; +esac +while true +do + clear -x + title + echo "App Name: $app_name" + echo "Container: $container" + echo + echo "1) Run a single command" + echo "2) Open Shell" + echo + echo "0) Exit" + read -rt 120 -p "Please choose an option: " selection || { echo -e "\nFailed to make a selection in time" ; exit; } + case $selection in + 0) + echo "Exiting.." + exit + ;; + 1) + clear -x + title + read -rt 500 -p "What command do you want to run?: " command || { echo -e "\nFailed to make a selection in time" ; exit; } + k3s crictl exec -it "$container_id" $command + break + ;; + 2) + clear -x + title + if ! k3s crictl exec -it "$container_id" /bin/bash 2>/dev/null; then + k3s crictl exec -it "$container_id" /bin/sh 2>/dev/null || echo "This container does not accept shell access, try a different one." + fi + break + ;; + *) + echo "That was not an option.. Try again" + sleep 3 + ;; + esac +done + +} +export -f cmd_to_container \ No newline at end of file diff --git a/functions/menu.sh b/functions/menu.sh index 22e4ac76..5b86ca5f 100644 --- a/functions/menu.sh +++ b/functions/menu.sh @@ -11,206 +11,43 @@ echo "5) Restore a Backup" echo "6) Delete a Backup" echo "7) Update HeavyScript" echo "8) Update Applications" +echo "9) Command to Container" echo echo "0) Exit" -read -rt 120 -p "Please select an option by number: " selection +read -rt 120 -p "Please select an option by number: " selection || { echo -e "\nFailed to make a selection in time" ; exit; } case $selection in 0) exit ;; 1) - help="true" + help ;; 2) - dns="true" + dns ;; 3) - mount="true" + mount ;; 4) - read -rt 120 -p "What is the maximun number of backups you would like?: " number_of_backups || echo "Failed to make a selection" + read -rt 120 -p "What is the maximun number of backups you would like?: " number_of_backups || { echo -e "\nFailed to make a selection in time" ; exit; } backup="true" ;; 5) - restore="true" + restore ;; 6) - deleteBackup="true" + deleteBackup ;; 7) - self_update="true" + self_update ;; 8) - while true - do - clear -x - title - echo "Choose Your Update Type" - echo "-----------------------" - echo "1) -U | Update all applications, ignores versions" - echo "2) -u | Update all applications, does not update Major releases" - echo - echo "0) Exit" - echo - read -rt 120 -p "Type the Number or Flag: " current_selection || { echo -e "\nFailed to make a selection in time" ; exit; } - case $current_selection in - 0 | [Ee][Xx][Ii][Tt]) - echo "Exiting.." - exit - ;; - 1 | -U) - while true - do - echo -e "\nHow many applications do you want updating at the same time?" - read -rt 120 -p "Please type an integer greater than 0: " up_async || { echo -e "\nFailed to make a selection in time" ; exit; } - if [[ $up_async == 0 ]]; then - echo "Error: \"$up_async\" is less than 1" - echo "NOT adding it to the list" - sleep 3 - continue - elif ! [[ $up_async =~ ^[0-9]+$ ]]; then - echo "Error: \"$up_async\" is invalid, it needs to be an integer" - echo "NOT adding it to the list" - sleep 3 - continue - else - update_selection+=("-U" "$up_async") - break - fi - done - break - ;; - 2 | -u) - while true - do - echo -e "\nHow many applications do you want updating at the same time?" - read -rt 120 -p "Please type an integer greater than 0: " up_async || { echo -e "\nFailed to make a selection in time" ; exit; } - if [[ $up_async == 0 ]]; then - echo "Error: \"$up_async\" is less than 1" - echo "NOT adding it to the list" - sleep 3 - continue - elif ! [[ $up_async =~ ^[0-9]+$ ]]; then - echo "Error: \"$up_async\" is invalid, it needs to be an integer" - echo "NOT adding it to the list" - sleep 3 - continue - else - update_selection+=("-u" "$up_async") - break - fi - done - break - ;; - *) - echo "$current_selection was not an option, try again" && sleep 3 - continue - ;; - esac - done - while true - do - clear -x - title - echo "Update Options" - echo "--------------" - echo "1) -r | Roll-back applications if they fail to update" - echo "2) -i | Add application to ignore list" - echo "3) -S | Shutdown applications prior to updating" - echo "4) -v | verbose output" - echo "5) -t | Set a custom timeout in seconds when checking if either an App or Mountpoint correctly Started, Stopped or (un)Mounted. Defaults to 500 seconds" - echo - echo "Additional Options" - echo "------------------" - echo "6) -b | Back-up your ix-applications dataset, specify a number after -b" - echo "7) -s | sync catalog" - echo "8) -p | Prune unused/old docker images" - echo "9) --self-update | Updates HeavyScript prior to running any other commands" - echo - echo "99) Remove Update Options, Restart" - echo "00) Done making selections, proceed with update" - echo - echo "0) Exit" - echo - echo "Current Choices" - echo "---------------" - echo "bash heavy_script.sh ${update_selection[*]}" - echo - read -rt 600 -p "Type the Number or Flag: " current_selection || { echo -e "\nFailed to make a selection in time" ; exit; } - case $current_selection in - 0 | [Ee][Xx][Ii][Tt]) - echo "Exiting.." - exit - ;; - 00) - clear -x - echo "Running \"bash heavy_script.sh ${update_selection[*]}\"" - echo - exec bash "$script_name" "${update_selection[@]}" - exit - ;; - 1 | -r) - printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-r" && echo -e "\"-r\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it - update_selection+=("-r") - ;; - 2 | -i) - read -rt 120 -p "What is the name of the application we should ignore?: " up_ignore || { echo -e "\nFailed to make a selection in time" ; exit; } - ! [[ $up_ignore =~ ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ ]] && echo -e "Error: \"$up_ignore\" is not a possible option for an application name" && sleep 3 && continue - update_selection+=("-i" "$up_ignore") - ;; - 3 | -S) - printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-S" && echo -e "\"-S\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it - update_selection+=("-S") - ;; - 4 | -v) - printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-v" && echo -e "\"-v\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it - update_selection+=("-v") - ;; - 5 | -t) - printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-t" && echo -e "\"-t\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it - echo "What do you want your timeout to be?" - read -rt 120 -p "Please type an integer: " up_timeout || { echo -e "\nFailed to make a selection in time" ; exit; } - ! [[ $up_timeout =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_timeout\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 3 && continue - update_selection+=("-t" "$up_timeout") - ;; - 6 | -b) - printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-b" && echo -e "\"-b\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it - echo "Up to how many backups should we keep?" - read -rt 120 -p "Please type an integer: " up_backups || { echo -e "\nFailed to make a selection in time" ; exit; } - ! [[ $up_backups =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_backups\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 3 && continue - [[ $up_backups == 0 ]] && echo -e "Error: Number of backups cannot be 0\nNOT adding it to the list" && sleep 3 && continue - update_selection+=("-b" "$up_backups") - ;; - 7 | -s) - printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-s" && echo -e "\"-s\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it - update_selection+=("-s") - ;; - 8 | -p) - printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-p" && echo -e "\"-p\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it - update_selection+=("-p") - ;; - 9 | --self-update ) - printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "--self-update" && echo -e "\"--self-update\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it - update_selection+=("--self-update") - ;; - 99) - count=2 - echo "restarting" - for i in "${update_selection[@]:2}" - do - unset "update_selection[$count]" - echo "$i removed" - ((count++)) - done - sleep 3 - continue - ;; - *) - echo "\"$current_selection\" was not an option, try again" && sleep 3 && continue - ;; - esac - done + script_create + ;; + + 9) + cmd_to_container ;; *) echo "\"$selection\" was not an option, please try agian" && sleep 3 && menu diff --git a/functions/misc.sh b/functions/misc.sh index 050055d8..87278b4b 100644 --- a/functions/misc.sh +++ b/functions/misc.sh @@ -48,6 +48,7 @@ echo "--mount | Initiates mounting feature, choose between unmounting an echo "--restore | Opens a menu to restore a \"heavy_script\" backup that was taken on your \"ix-applications\" dataset" echo "--delete-backup | Opens a menu to delete backups on your system" echo "--dns | list all of your applications DNS names and their web ports" +echo "--cmd | Open a shell for one of your applications" echo echo "Update Types" echo "------------" diff --git a/functions/script_create.sh b/functions/script_create.sh new file mode 100644 index 00000000..b1448218 --- /dev/null +++ b/functions/script_create.sh @@ -0,0 +1,174 @@ +#!/bin/bash + +script_create(){ +while true +do + clear -x + title + echo "Choose Your Update Type" + echo "-----------------------" + echo "1) -U | Update all applications, ignores versions" + echo "2) -u | Update all applications, does not update Major releases" + echo + echo "0) Exit" + echo + read -rt 120 -p "Type the Number or Flag: " current_selection || { echo -e "\nFailed to make a selection in time" ; exit; } + case $current_selection in + 0 | [Ee][Xx][Ii][Tt]) + echo "Exiting.." + exit + ;; + 1 | -U) + while true + do + echo -e "\nHow many applications do you want updating at the same time?" + read -rt 120 -p "Please type an integer greater than 0: " up_async || { echo -e "\nFailed to make a selection in time" ; exit; } + if [[ $up_async == 0 ]]; then + echo "Error: \"$up_async\" is less than 1" + echo "NOT adding it to the list" + sleep 3 + continue + elif ! [[ $up_async =~ ^[0-9]+$ ]]; then + echo "Error: \"$up_async\" is invalid, it needs to be an integer" + echo "NOT adding it to the list" + sleep 3 + continue + else + update_selection+=("-U" "$up_async") + break + fi + done + break + ;; + 2 | -u) + while true + do + echo -e "\nHow many applications do you want updating at the same time?" + read -rt 120 -p "Please type an integer greater than 0: " up_async || { echo -e "\nFailed to make a selection in time" ; exit; } + if [[ $up_async == 0 ]]; then + echo "Error: \"$up_async\" is less than 1" + echo "NOT adding it to the list" + sleep 3 + continue + elif ! [[ $up_async =~ ^[0-9]+$ ]]; then + echo "Error: \"$up_async\" is invalid, it needs to be an integer" + echo "NOT adding it to the list" + sleep 3 + continue + else + update_selection+=("-u" "$up_async") + break + fi + done + break + ;; + *) + echo "$current_selection was not an option, try again" && sleep 3 + continue + ;; + esac +done +while true +do + clear -x + title + echo "Update Options" + echo "--------------" + echo "1) -r | Roll-back applications if they fail to update" + echo "2) -i | Add application to ignore list" + echo "3) -S | Shutdown applications prior to updating" + echo "4) -v | verbose output" + echo "5) -t | Set a custom timeout in seconds when checking if either an App or Mountpoint correctly Started, Stopped or (un)Mounted. Defaults to 500 seconds" + echo + echo "Additional Options" + echo "------------------" + echo "6) -b | Back-up your ix-applications dataset, specify a number after -b" + echo "7) -s | sync catalog" + echo "8) -p | Prune unused/old docker images" + echo "9) --self-update | Updates HeavyScript prior to running any other commands" + echo + echo "99) Remove Update Options, Restart" + echo "00) Done making selections, proceed with update" + echo + echo "0) Exit" + echo + echo "Current Choices" + echo "---------------" + echo "bash heavy_script.sh ${update_selection[*]}" + echo + read -rt 600 -p "Type the Number or Flag: " current_selection || { echo -e "\nFailed to make a selection in time" ; exit; } + case $current_selection in + 0 | [Ee][Xx][Ii][Tt]) + echo "Exiting.." + exit + ;; + 00) + clear -x + echo "Running \"bash heavy_script.sh ${update_selection[*]}\"" + echo + exec bash "$script_name" "${update_selection[@]}" + exit + ;; + 1 | -r) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-r" && echo -e "\"-r\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it + update_selection+=("-r") + ;; + 2 | -i) + read -rt 120 -p "What is the name of the application we should ignore?: " up_ignore || { echo -e "\nFailed to make a selection in time" ; exit; } + ! [[ $up_ignore =~ ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ ]] && echo -e "Error: \"$up_ignore\" is not a possible option for an application name" && sleep 3 && continue + update_selection+=("-i" "$up_ignore") + ;; + 3 | -S) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-S" && echo -e "\"-S\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it + update_selection+=("-S") + ;; + 4 | -v) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-v" && echo -e "\"-v\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it + update_selection+=("-v") + ;; + 5 | -t) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-t" && echo -e "\"-t\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it + echo "What do you want your timeout to be?" + read -rt 120 -p "Please type an integer: " up_timeout || { echo -e "\nFailed to make a selection in time" ; exit; } + ! [[ $up_timeout =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_timeout\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 3 && continue + update_selection+=("-t" "$up_timeout") + ;; + 6 | -b) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-b" && echo -e "\"-b\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it + echo "Up to how many backups should we keep?" + read -rt 120 -p "Please type an integer: " up_backups || { echo -e "\nFailed to make a selection in time" ; exit; } + ! [[ $up_backups =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_backups\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 3 && continue + [[ $up_backups == 0 ]] && echo -e "Error: Number of backups cannot be 0\nNOT adding it to the list" && sleep 3 && continue + update_selection+=("-b" "$up_backups") + ;; + 7 | -s) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-s" && echo -e "\"-s\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it + update_selection+=("-s") + ;; + 8 | -p) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-p" && echo -e "\"-p\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it + update_selection+=("-p") + ;; + 9 | --self-update ) + printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "--self-update" && echo -e "\"--self-update\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it + update_selection+=("--self-update") + ;; + 99) + count=2 + echo "restarting" + for i in "${update_selection[@]:2}" + do + unset "update_selection[$count]" + echo "$i removed" + ((count++)) + done + sleep 3 + continue + ;; + *) + echo "\"$current_selection\" was not an option, try again" && sleep 3 && continue + ;; + esac +done +} +export -f script_create \ No newline at end of file diff --git a/functions/self_update.sh b/functions/self_update.sh index 98a3e19e..26b2fd56 100644 --- a/functions/self_update.sh +++ b/functions/self_update.sh @@ -2,10 +2,11 @@ args=("$@") self_update() { +branch="main" git fetch &> /dev/null echo "🅂 🄴 🄻 🄵" echo "🅄 🄿 🄳 🄰 🅃 🄴" -if git diff --name-only origin/main | grep -qs ".sh" ; then +if git diff --name-only origin/$branch | grep -qs ".sh" ; then echo "Found a new version of HeavyScript, updating myself..." git reset --hard -q git pull --force -q diff --git a/functions/update_apps.sh b/functions/update_apps.sh index 9989a64e..a32dc4a9 100644 --- a/functions/update_apps.sh +++ b/functions/update_apps.sh @@ -8,13 +8,29 @@ echo -e "🅄 🄿 🄳 🄰 🅃 🄴 🅂" echo "Asynchronous Updates: $update_limit" [[ -z $timeout ]] && echo "Default Timeout: 500" && timeout=500 || echo "Custom Timeout: $timeout" [[ "$timeout" -le 120 ]] && echo "Warning: Your timeout is set low and may lead to premature rollbacks or skips" - +pool=$(cli -c 'app kubernetes config' | grep -E "dataset\s\|" | awk -F '|' '{print $3}' | awk -F '/' '{print $1}' | tr -d " \t\n\r") it=0 +while_count=0 +rm deploying 2>/dev/null while true do - while_status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status') - echo "$while_status" > temp.txt + if while_status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status' 2>/dev/null) ; then + ((while_count++)) + [[ -z $while_status ]] && continue || echo -e "$while_count\n$while_status" > all_app_status + mapfile -t deploying_check < <(grep ",DEPLOYING," all_app_status) + for i in "${deploying_check[@]}" + do + app_name=$(echo "$i" | awk -F ',' '{print $1}') + [[ ! -e deploying ]] && touch deploying + if ! grep -qs "$app_name,DEPLOYING" deploying; then + echo "$app_name,DEPLOYING" >> deploying + fi + done + else + echo "Middlewared timed out. Consider setting a lower number for async applications" + continue + fi proc_count=${#processes[@]} count=0 for proc in "${processes[@]}" @@ -25,23 +41,29 @@ do if [[ "$proc_count" -ge "$update_limit" ]]; then sleep 3 elif [[ $it -lt ${#array[@]} ]]; then - update_apps "${array[$it]}" & + # loop=0 + # until [[ $loop -ge 2 || $it -ge ${#array[@]} ]]; + # do + pre_process "${array[$it]}" & processes+=($!) ((it++)) + # ((loop++)) + # done elif [[ $proc_count != 0 ]]; then # Wait for all processes to finish sleep 3 else # All processes must be completed, break out of loop break fi done -rm temp.txt +rm all_app_status 2>/dev/null +rm deploying 2>/dev/null echo echo } export -f commander -update_apps(){ +pre_process(){ app_name=$(echo "${array[$it]}" | awk -F ',' '{print $1}') #print out first catagory, name. printf '%s\0' "${ignore[@]}" | grep -iFxqz "${app_name}" && echo -e "\n$app_name\nIgnored, skipping" && return 0 #If application is on ignore list, skip old_app_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #previous/current Application MAJOR Version @@ -51,119 +73,190 @@ new_chart_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}' | awk -F '_' '{pr startstatus=$(echo "${array[$it]}" | awk -F ',' '{print $2}') #status of the app: STOPPED / DEPLOYING / ACTIVE diff_app=$(diff <(echo "$old_app_ver") <(echo "$new_app_ver")) #caluclating difference in major app versions diff_chart=$(diff <(echo "$old_chart_ver") <(echo "$new_chart_ver")) #caluclating difference in Chart versions +[[ "$diff_app" != "$diff_chart" && $update_apps == "true" ]] && echo -e "\n$app_name\nMajor Release, update manually" && return old_full_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}') #Upgraded From new_full_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}') #Upraded To rollback_version=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}') -if [[ "$diff_app" == "$diff_chart" || "$update_all_apps" == "true" ]]; then #continue to update - if [[ $stop_before_update == "true" ]]; then # Check to see if user is using -S or not - if [[ "$startstatus" == "STOPPED" ]]; then # if status is already stopped, skip while loop - echo_array+=("\n$app_name") - [[ "$verbose" == "true" ]] && echo_array+=("Updating..") - cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("FAILED") - return 0 - else # if status was not STOPPED, stop the app prior to updating - echo_array+=("\n$app_name") - [[ "$verbose" == "true" ]] && echo_array+=("Stopping prior to update..") - midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && SECONDS=0 || echo_array+=("FAILED") - while [[ "$status" != "STOPPED" ]] - do - status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}') - if [[ "$status" == "STOPPED" ]]; then - echo_array+=("Stopped") - [[ "$verbose" == "true" ]] && echo_array+=("Updating..") - cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("Failed to update") - break - elif [[ "$SECONDS" -ge "$timeout" ]]; then - echo_array+=("Error: Run Time($SECONDS) has exceeded Timeout($timeout)") - break - elif [[ "$status" != "STOPPED" ]]; then - [[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be STOPPED") - sleep 10 - fi - done - fi - else #user must not be using -S, just update - echo_array+=("\n$app_name") - [[ "$verbose" == "true" ]] && echo_array+=("Updating..") - cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("FAILED") +if grep -qs "^$app_name," failed 2>/dev/null; then + failed_ver=$(grep "^$app_name," failed | awk -F ',' '{print $2}') + if [[ "$failed_ver" == "$new_full_ver" ]] ; then + echo -e "\n$app_name" + echo -e "Skipping previously failed version:\n$new_full_ver" + return 0 + else + sed -i /"$app_name",/d failed fi +fi + +[[ ! -e external_services ]] && touch external_services +if ! grep -qs "^$app_name," external_services ; then + if ! grep qs "/external-service" /mnt/"$pool"/ix-applications/releases/"$app_name"/charts/"$(find /mnt/"$pool"/ix-applications/releases/"$app_name"/charts/ -maxdepth 1 -type d -printf '%P\n' | sort -r | head -n 1)"/Chart.yaml 2>/dev/null; then + echo "$app_name,false" >> external_services + else + echo "$app_name,true" >> external_services + fi +fi + +echo_array+=("\n$app_name") +if [[ $stop_before_update == "true" && "$startstatus" != "STOPPED" ]]; then # Check to see if user is using -S or not + [[ "$verbose" == "true" ]] && echo_array+=("Stopping prior to update..") + if stop_app ; then + echo_array+=("Stopped") + else + echo_array+=("Error: Failed to stop $app_name") + echo_array + return 1 + fi +fi +[[ "$verbose" == "true" ]] && echo_array+=("Updating..") +if update_app ;then + echo_array+=("Updated\n$old_full_ver\n$new_full_ver") else - echo_array+=("\n$app_name\nMajor Release, update manually") - return 0 + echo_array+=("Failed to update") + echo_array + return +fi +if grep -qs "^$app_name,true" external_services ; then + echo_array + return +else + after_update_actions fi } -export -f update_apps +export -f pre_process + + +update_app(){ +current_loop=0 +while true +do + update_avail=$(grep "^$app_name," all_app_status | awk -F ',' '{print $3}') + if [[ $update_avail == "true" ]]; then + if ! cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null ; then + before_loop=$(head -n 1 all_app_status) + current_loop=0 + until [[ "$(grep "^$app_name," all_app_status | awk -F ',' '{print $3}')" != "$update_avail" ]] # Wait for a specific change to app status, or 3 refreshes of the file to go by. + do + if [[ $current_loop -gt 2 ]]; then + cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null || return 1 # After waiting, attempt an update once more, if fails, return error code + elif ! echo -e "$(head -n 1 all_app_status)" | grep -qs ^"$before_loop" ; then # The file has been updated, but nothing changed specifically for the app. + before_loop=$(head -n 1 all_app_status) + ((current_loop++)) + fi + sleep 1 + done + fi + break + elif [[ $update_avail == "false" ]]; then + break + else + sleep 3 + fi +done +} +export -f update_app + + +stop_app(){ +count=0 +while [[ "$status" != "STOPPED" ]] +do + status=$( grep "^$app_name," all_app_status | awk -F ',' '{print $2}') + if [[ $count -gt 2 ]]; then # If failed to stop app 3 times, return failure to parent shell + return 1 + elif ! cli -c 'app chart_release scale release_name='\""$app_name"\"\ 'scale_options={"replica_count": 0}' &> /dev/null ; then + before_loop=$(head -n 1 all_app_status) + ((count++)) + until [[ $(head -n 1 all_app_status) != "$before_loop" ]] # Upon failure, wait for status update before continuing + do + sleep 1 + done + else + break + fi +done +} +export -f stop_app after_update_actions(){ SECONDS=0 count=0 -if [[ $rollback == "true" ]]; then +if [[ $rollback == "true" || "$startstatus" == "STOPPED" ]]; then while true do + status=$(grep "^$app_name," all_app_status | awk -F ',' '{print $2}') + if [[ $count -lt 1 && $status == "ACTIVE" && "$(grep "^$app_name," deploying 2>/dev/null | awk -F ',' '{print $2}')" != "DEPLOYING" ]]; then # If status shows up as Active or Stopped on the first check, verify that. Otherwise it may be a false report.. + [[ "$verbose" == "true" ]] && echo_array+=("Verifying $status..") + before_loop=$(head -n 1 all_app_status) + current_loop=0 + until [[ "$status" != "ACTIVE" || $current_loop -gt 4 ]] # Wait for a specific change to app status, or 3 refreshes of the file to go by. + do + status=$(grep "^$app_name," all_app_status | awk -F ',' '{print $2}') + sleep 1 + if ! echo -e "$(head -n 1 all_app_status)" | grep -qs ^"$before_loop" ; then + before_loop=$(head -n 1 all_app_status) + ((current_loop++)) + fi + done + fi (( count++ )) - status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}') - if [[ "$status" == "ACTIVE" && "$startstatus" == "STOPPED" ]]; then - [[ "$verbose" == "true" ]] && echo_array+=("Returing to STOPPED state..") - midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo_array+=("Stopped")|| echo_array+=("FAILED") - break - elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" != "true" ]]; then - echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nIf this is a slow starting application, set a higher timeout with -t\nIf this applicaion is always DEPLOYING, you can disable all probes under the Healthcheck Probes Liveness section in the edit configuration\nReverting update..") - midclt call chart.release.rollback "$app_name" "{\"item_version\": \"$rollback_version\"}" &> /dev/null - [[ "$startstatus" == "STOPPED" ]] && failed="true" && after_update_actions && unset failed #run back after_update_actions function if the app was stopped prior to update - break - elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" == "true" ]]; then - echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nThe application failed to be ACTIVE even after a rollback,\nManual intervention is required\nAbandoning") - break - elif [[ "$status" == "STOPPED" ]]; then - [[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Stopped..") && sleep 15 && continue #if reports stopped on FIRST time through loop, double check - [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check - echo_array+=("Stopped") - break #if reports stopped any time after the first loop, assume its extermal services. - elif [[ "$status" == "ACTIVE" ]]; then - [[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Active..") && sleep 15 && continue #if reports active on FIRST time through loop, double check - [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check - echo_array+=("Active") - break #if reports active any time after the first loop, assume actually active. + if [[ "$status" == "ACTIVE" ]]; then + if [[ "$startstatus" == "STOPPED" ]]; then + [[ "$verbose" == "true" ]] && echo_array+=("Returing to STOPPED state..") + if stop_app ; then + echo_array+=("Stopped") + else + echo_array+=("Error: Failed to stop $app_name") + echo_array + return 1 + fi + break + else + echo_array+=("Active") + break + fi + elif [[ "$SECONDS" -ge "$timeout" ]]; then + if [[ $rollback == "true" ]]; then + if [[ "$failed" != "true" ]]; then + echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)") + echo_array+=("If this is a slow starting application, set a higher timeout with -t") + echo_array+=("If this applicaion is always DEPLOYING, you can disable all probes under the Healthcheck Probes Liveness section in the edit configuration") + echo_array+=("Reverting update..") + midclt call chart.release.rollback "$app_name" "{\"item_version\": \"$rollback_version\"}" &> /dev/null || { echo_array+=("Error: Failed to rollback $app_name") ; break ; } + [[ "$startstatus" == "STOPPED" ]] && failed="true" && after_update_actions #run back after_update_actions function if the app was stopped prior to update + echo "$app_name,$new_full_ver" >> failed + break + else + echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)") + echo_array+=("The application failed to be ACTIVE even after a rollback") + echo_array+=("Manual intervention is required\nAbandoning") + break + fi + else + echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)") + echo_array+=("If this is a slow starting application, set a higher timeout with -t") + echo_array+=("If this applicaion is always DEPLOYING, you can disable all probes under the Healthcheck Probes Liveness section in the edit configuration") + break + fi else [[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE") - sleep 15 + sleep 5 continue fi done -else - if [[ "$startstatus" == "STOPPED" ]]; then - while true #using a constant while loop, then breaking out of the loop with break commands below. - do - (( count++ )) - status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}') - if [[ "$status" == "STOPPED" ]]; then - [[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Stopped..") && sleep 15 && continue #if reports stopped on FIRST time through loop, double check - [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check - echo_array+=("Stopped") #assume actually stopped anytime AFTER the first loop - break - elif [[ "$status" == "ACTIVE" ]]; then - [[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Active..") && sleep 15 && continue #if reports active on FIRST time through loop, double check - [[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check - [[ "$verbose" == "true" ]] && echo_array+=("Returing to STOPPED state..") - midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo_array+=("Stopped")|| echo_array+=("FAILED") - break - elif [[ "$SECONDS" -ge "$timeout" ]]; then - echo_array+=("Error: Run Time($SECONDS) has exceeded Timeout($timeout)") - break - else - [[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE") - sleep 10 - continue - fi - done - fi fi +echo_array +} +export -f after_update_actions + +echo_array(){ #Dump the echo_array, ensures all output is in a neat order. for i in "${echo_array[@]}" do echo -e "$i" done + } -export -f after_update_actions \ No newline at end of file +export -f echo_array diff --git a/heavy_script.sh b/heavy_script.sh index efecf333..fad37d0c 100644 --- a/heavy_script.sh +++ b/heavy_script.sh @@ -8,20 +8,15 @@ script_name="heavy_script.sh" cd "$script_path" || { echo "Error: Failed to change to script directory" ; exit ; } -# shellcheck source=functions/backup.sh source functions/backup.sh -# shellcheck source=functions/dns.sh source functions/dns.sh -# shellcheck source=functions/menu.sh source functions/menu.sh -# shellcheck source=functions/misc.sh source functions/misc.sh -# shellcheck source=functions/mount.sh source functions/mount.sh -# shellcheck source=functions/self_update.sh source functions/self_update.sh -# shellcheck source=functions/update_apps.sh source functions/update_apps.sh +source functions/cmd_to_container.sh +source functions/script_create.sh @@ -46,6 +41,9 @@ do dns) dns="true" ;; + cmd) + cmd="true" + ;; restore) restore="true" ;; @@ -139,8 +137,9 @@ done [[ "$update_all_apps" == "true" && "$update_apps" == "true" ]] && echo -e "-U and -u cannot BOTH be called" && exit #Continue to call functions in specific order -[[ "$help" == "true" ]] && help [[ "$self_update" == "true" ]] && self_update +[[ "$help" == "true" ]] && help +[[ "$cmd" == "true" ]] && cmd_to_container && exit [[ "$deleteBackup" == "true" ]] && deleteBackup && exit [[ "$dns" == "true" ]] && dns && exit [[ "$restore" == "true" ]] && restore && exit