julian/scale-catalog
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #8 from Heavybullets8/ignore-file

Browse Source 
Major Change
This commit is contained in:
Heavybullets8
2022-08-19 03:01:22 +00:00
committed by GitHub
parent 5007ab574f 1db4c18d88
commit 85a8529744
8 changed files with 543 additions and 306 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
README.md
View File

@@ -14,19 +14,20 @@
<br> <br>
## Arguments ## The Menu
![image](https://user-images.githubusercontent.com/20793231/185020236-7b389499-8081-407d-b653-10dffd70de8c.png)
> Access this with `bash heavy_script.sh`
<br >
<br >
## Update Arguments
| Flag | Example | Parameter | Description | | Flag | Example | Parameter | Description |
|----------------- |------------------------ |----------------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |--------------- |------------------------ |----------------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| NULL | NULL | NULL | If you choose not to supply an option, it will open the menu for easier access to the utilities |
| --self-update | --self-update | None | Updates HeavyScript prior to running it<br>_You no longer need to git pull_ |
| --delete-backup | --delete-backup | None | Opens a menu to delete backups<br>_Useful if you need to delete old system backups or backups from other scripts_ |
| --restore | --restore | None | Restore HeavyScript specific `ix-applications dataset` snapshot |
| --mount | --mount | None | Initiates mounting feature<br>Choose between unmounting and mounting PVC data |
| --dns | --dns | None | list all of your applications DNS names and their web ports |
| -U | -U <br>-U 5 | None or Integer | Update applications, ignoring major version changes<br>_Optionally, you can supply a number after the argument to update multiple applications at once_ | | -U | -U <br>-U 5 | None or Integer | Update applications, ignoring major version changes<br>_Optionally, you can supply a number after the argument to update multiple applications at once_ |
| -u | -u<br>-u 5 | None or Integer | Update applications, do NOT update if there was a major version change<br>_Optionally, you can supply a number after the argument to update multiple applications at once_ | | -u | -u<br>-u 5 | None or Integer | Update applications, do NOT update if there was a major version change<br>_Optionally, you can supply a number after the argument to update multiple applications at once_ |
| -b | -b 10 | Integer | Backup `ix-appliactions` dataset<br>_Creates backups up to the number you've chosen_ | | -b | -b 14 | Integer | Backup `ix-appliactions` dataset<br>_Creates backups up to the number you've chosen_ |
| -i | -i nextcloud -i sonarr | String | Applications listed will be ignored during updating<br>_List one application after another as shown in the example_ | | -i | -i nextcloud -i sonarr | String | Applications listed will be ignored during updating<br>_List one application after another as shown in the example_ |
| -r | -r | None | Monitors applications after they update<br>If the app does not become "ACTIVE" after either:<br>The custom Timeout, or Default Timeout,<br>rollback the application. | | -r | -r | None | Monitors applications after they update<br>If the app does not become "ACTIVE" after either:<br>The custom Timeout, or Default Timeout,<br>rollback the application. |
| -v | -v | None | Verbose Output<br>_Look at the bottom of this page for an example_ | | -v | -v | None | Verbose Output<br>_Look at the bottom of this page for an example_ |
@@ -34,12 +35,10 @@
| -t | -t 150 | Integer | Set a custom timeout to be used with either:<br>`-m` <br>_Time the script will wait for application to be "STOPPED"_<br>or<br>`-(u\|U)` <br>_Time the script will wait for application to be either "STOPPED" or "ACTIVE"_ | | -t | -t 150 | Integer | Set a custom timeout to be used with either:<br>`-m` <br>_Time the script will wait for application to be "STOPPED"_<br>or<br>`-(u\|U)` <br>_Time the script will wait for application to be either "STOPPED" or "ACTIVE"_ |
| -s | -s | None | Sync Catalogs prior to updating | | -s | -s | None | Sync Catalogs prior to updating |
| -p | -p | None | Prune old/unused docker images | | -p | -p | None | Prune old/unused docker images |
| --self-update | --self-update | None | Updates HeavyScript prior to running any other commands |
<br> ### Example
<br>
### Examples
#### Typical Cron Job #### Typical Cron Job
``` ```
bash heavy_script.sh --self-update -b 10 -i portainer -i arch -i sonarr -i radarr -t 600 -rsp -u 5 bash heavy_script.sh --self-update -b 10 -i portainer -i arch -i sonarr -i radarr -t 600 -rsp -u 5
@@ -62,7 +61,22 @@ bash heavy_script.sh --self-update -b 10 -i portainer -i arch -i sonarr -i radar
> `--self-update` Will update the script prior to running anything else. > `--self-update` Will update the script prior to running anything else.
<br >
<br>
## Other Utilities
> All of these can ALSO be accessed with the HeavyScript menu, that you can access simply by not providing an argument `bash heavy_script.sh`
| Flag | Description |
|-----------------|----------------------------------------------------------------------------------------------|
| --mount | Initiates mounting feature, choose between unmounting and mounting PVC data |
| --restore | Opens a menu to restore a heavy_script backup that was taken on your ix-applications dataset |
| --delete-backup | Opens a menu to delete backups on your system |
| --dns | list all of your applications DNS names and their web ports |
| --cmd | Open a shell for one of your applications |
### Examples
#### Mounting PVC Data #### Mounting PVC Data
``` ```
@@ -87,10 +101,10 @@ bash /mnt/tank/scripts/heavy_script/heavy_script.sh --delete-backup
bash /mnt/tank/scripts/heavy_script/heavy_script.sh --dns bash /mnt/tank/scripts/heavy_script/heavy_script.sh --dns
``` ```
#### My personal Cron Job #### Open a Containers Shell
``` ```
bash /mnt/speed/scripts/heavy_script/heavy_script.sh --self-update -b 10 -rsp -u 10 bash /mnt/speed/scripts/heavy_script/heavy_script.sh --cmd
``` ```
<br> <br>
@@ -165,7 +179,7 @@ bash heavyscript.sh --self-update -b 10 -supr
| Name | Value | Reason | | Name | Value | Reason |
|------------------------ |------------------------------------------------------------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |------------------------ |------------------------------------------------------------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `Description` | HeavyScript git pull and Update apps | This is up to you, put whatever you think is a good description in here | | `Description` | HeavyScript git pull and Update apps | This is up to you, put whatever you think is a good description in here |
| `Command` | `bash /PATH/TO/HEAVY_SCRIPT_DIRECTORY/heavy_script.sh --self-update -b 10 -rsup` | This is the command you will be running on your schedule I personally use: `bash /mnt/speed/scripts/heavy_script/heavy_script.sh --self-update -b 10 -rsup` | | `Command` | `bash /PATH/TO/HEAVY_SCRIPT_DIRECTORY/heavy_script.sh --self-update -b 10 -rsp -u 10` | This is the command you will be running on your schedule I personally use: `bash /mnt/speed/scripts/heavy_script/heavy_script.sh --self-update -b 10 -rsp -u 10` |
| `Run As User` | `root` | Running the script as `root` is REQUIRED. You cannot access all of the kubernetes functions without this user. | | `Run As User` | `root` | Running the script as `root` is REQUIRED. You cannot access all of the kubernetes functions without this user. |
| `Schedule` | Up to you, I run mine everyday at `0400` | Again up to you | | `Schedule` | Up to you, I run mine everyday at `0400` | Again up to you |
| `Hide Standard Output` | `False` or Unticked | I like to receive an email report of how the script ran, what apps updated etc. | | `Hide Standard Output` | `False` or Unticked | I like to receive an email report of how the script ran, what apps updated etc. |

118
functions/cmd_to_container.sh Normal file
View File

@@ -0,0 +1,118 @@
#!/bin/bash
cmd_to_container(){
app_name=$(k3s crictl pods -s ready --namespace ix | sed -E 's/[[:space:]]([0-9]*|About)[a-z0-9 ]{5,12}ago[[:space:]]//' | sed '1d' | awk '{print $4}' | cut -c4- | sort -u | nl -s ") " | column -t)
while true
do
clear -x
title
echo "$app_name"
echo
echo "0) Exit"
read -rt 120 -p "Please type a number: " selection || { echo -e "\nFailed to make a selection in time" ; exit; }
if [[ $selection == 0 ]]; then
echo "Exiting.."
exit
elif ! echo -e "$app_name" | grep -qs ^"$selection)" ; then
echo "Error: \"$selection\" was not an option.. Try again"
sleep 3
continue
else
break
fi
done
app_name=$(echo -e "$app_name" | grep ^"$selection)" | awk '{print $2}')
mapfile -t pod_id < <(k3s crictl pods -s ready --namespace ix | grep -E "[[:space:]]$app_name([[:space:]]|-([-[:alnum:]])*[[:space:]])" | awk '{print $1}')
search=$(k3s crictl ps -a -s running | sed -E 's/[[:space:]]([0-9]*|About)[a-z0-9 ]{5,12}ago[[:space:]]//')
for pod in "${pod_id[@]}"
do
if [[ $(echo "$search" | grep "$pod" | awk '{print $4}' | tr -d " \t\r " | wc -l) -gt 1 ]]; then
readarray -t containers <<<"$(echo "$search" | grep "$pod" | awk '{print $4}' | tr -d " \t\r ")"
continue
fi
printf '%s\0' "${containers[@]}" | grep -Fxqz -- "$(echo "$search" | grep "$pod" | awk '{print $4}' | tr -d " \t\r ")" && continue
containers+=("$(echo "$search" | grep "$pod" | awk '{print $4}' | tr -d " \t\r ")")
done
case "${#containers[@]}" in
0)
echo -e "No containers available\nAre you sure the application in running?"
exit
;;
1)
container=$(echo "$search" | grep "${pod_id[0]}" | awk '{print $4}')
container_id=$(echo "$search" | grep -E "[[:space:]]${container}[[:space:]]" | awk '{print $1}')
;;
*)
while true
do
clear -x
title
cont_search=$(
for i in "${containers[@]}"
do
echo "$i"
done | nl -s ") " | column -t
)
echo "$cont_search"
echo
echo "0) Exit"
read -rt 120 -p "Choose a container by number: " selection || { echo -e "\nFailed to make a selection in time" ; exit; }
if [[ $selection == 0 ]]; then
echo "Exiting.."
exit
elif ! echo -e "$cont_search}" | grep -qs ^"$selection)" ; then
echo "Error: \"$selection\" was not an option.. Try again"
sleep 3
continue
else
break
fi
done
container=$(echo "$cont_search" | grep ^"$selection)" | awk '{print $2}')
container_id=$(echo "$search" | grep -E "[[:space:]]${container}[[:space:]]" | awk '{print $1}')
;;
esac
while true
do
clear -x
title
echo "App Name: $app_name"
echo "Container: $container"
echo
echo "1) Run a single command"
echo "2) Open Shell"
echo
echo "0) Exit"
read -rt 120 -p "Please choose an option: " selection || { echo -e "\nFailed to make a selection in time" ; exit; }
case $selection in
0)
echo "Exiting.."
exit
;;
1)
clear -x
title
read -rt 500 -p "What command do you want to run?: " command || { echo -e "\nFailed to make a selection in time" ; exit; }
k3s crictl exec -it "$container_id" $command
break
;;
2)
clear -x
title
if ! k3s crictl exec -it "$container_id" /bin/bash 2>/dev/null; then
k3s crictl exec -it "$container_id" /bin/sh 2>/dev/null || echo "This container does not accept shell access, try a different one."
fi
break
;;
*)
echo "That was not an option.. Try again"
sleep 3
;;
esac
done
}
export -f cmd_to_container

189
functions/menu.sh
View File

@@ -11,206 +11,43 @@ echo "5) Restore a Backup"
echo "6) Delete a Backup" echo "6) Delete a Backup"
echo "7) Update HeavyScript" echo "7) Update HeavyScript"
echo "8) Update Applications" echo "8) Update Applications"
echo "9) Command to Container"
echo echo
echo "0) Exit" echo "0) Exit"
read -rt 120 -p "Please select an option by number: " selection read -rt 120 -p "Please select an option by number: " selection || { echo -e "\nFailed to make a selection in time" ; exit; }
case $selection in case $selection in
0) 0)
exit exit
;; ;;
1) 1)
help="true" help
;; ;;
2) 2)
dns="true" dns
;; ;;
3) 3)
mount="true" mount
;; ;;
4) 4)
read -rt 120 -p "What is the maximun number of backups you would like?: " number_of_backups || echo "Failed to make a selection" read -rt 120 -p "What is the maximun number of backups you would like?: " number_of_backups || { echo -e "\nFailed to make a selection in time" ; exit; }
backup="true" backup="true"
;; ;;
5) 5)
restore="true" restore
;; ;;
6) 6)
deleteBackup="true" deleteBackup
;; ;;
7) 7)
self_update="true" self_update
;; ;;
8) 8)
while true script_create
do
clear -x
title
echo "Choose Your Update Type"
echo "-----------------------"
echo "1) -U | Update all applications, ignores versions"
echo "2) -u | Update all applications, does not update Major releases"
echo
echo "0) Exit"
echo
read -rt 120 -p "Type the Number or Flag: " current_selection || { echo -e "\nFailed to make a selection in time" ; exit; }
case $current_selection in
0 | [Ee][Xx][Ii][Tt])
echo "Exiting.."
exit
;; ;;
1 | -U)
while true 9)
do cmd_to_container
echo -e "\nHow many applications do you want updating at the same time?"
read -rt 120 -p "Please type an integer greater than 0: " up_async || { echo -e "\nFailed to make a selection in time" ; exit; }
if [[ $up_async == 0 ]]; then
echo "Error: \"$up_async\" is less than 1"
echo "NOT adding it to the list"
sleep 3
continue
elif ! [[ $up_async =~ ^[0-9]+$ ]]; then
echo "Error: \"$up_async\" is invalid, it needs to be an integer"
echo "NOT adding it to the list"
sleep 3
continue
else
update_selection+=("-U" "$up_async")
break
fi
done
break
;;
2 | -u)
while true
do
echo -e "\nHow many applications do you want updating at the same time?"
read -rt 120 -p "Please type an integer greater than 0: " up_async || { echo -e "\nFailed to make a selection in time" ; exit; }
if [[ $up_async == 0 ]]; then
echo "Error: \"$up_async\" is less than 1"
echo "NOT adding it to the list"
sleep 3
continue
elif ! [[ $up_async =~ ^[0-9]+$ ]]; then
echo "Error: \"$up_async\" is invalid, it needs to be an integer"
echo "NOT adding it to the list"
sleep 3
continue
else
update_selection+=("-u" "$up_async")
break
fi
done
break
;;
*)
echo "$current_selection was not an option, try again" && sleep 3
continue
;;
esac
done
while true
do
clear -x
title
echo "Update Options"
echo "--------------"
echo "1) -r | Roll-back applications if they fail to update"
echo "2) -i | Add application to ignore list"
echo "3) -S | Shutdown applications prior to updating"
echo "4) -v | verbose output"
echo "5) -t | Set a custom timeout in seconds when checking if either an App or Mountpoint correctly Started, Stopped or (un)Mounted. Defaults to 500 seconds"
echo
echo "Additional Options"
echo "------------------"
echo "6) -b | Back-up your ix-applications dataset, specify a number after -b"
echo "7) -s | sync catalog"
echo "8) -p | Prune unused/old docker images"
echo "9) --self-update | Updates HeavyScript prior to running any other commands"
echo
echo "99) Remove Update Options, Restart"
echo "00) Done making selections, proceed with update"
echo
echo "0) Exit"
echo
echo "Current Choices"
echo "---------------"
echo "bash heavy_script.sh ${update_selection[*]}"
echo
read -rt 600 -p "Type the Number or Flag: " current_selection || { echo -e "\nFailed to make a selection in time" ; exit; }
case $current_selection in
0 | [Ee][Xx][Ii][Tt])
echo "Exiting.."
exit
;;
00)
clear -x
echo "Running \"bash heavy_script.sh ${update_selection[*]}\""
echo
exec bash "$script_name" "${update_selection[@]}"
exit
;;
1 | -r)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-r" && echo -e "\"-r\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("-r")
;;
2 | -i)
read -rt 120 -p "What is the name of the application we should ignore?: " up_ignore || { echo -e "\nFailed to make a selection in time" ; exit; }
! [[ $up_ignore =~ ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ ]] && echo -e "Error: \"$up_ignore\" is not a possible option for an application name" && sleep 3 && continue
update_selection+=("-i" "$up_ignore")
;;
3 | -S)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-S" && echo -e "\"-S\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("-S")
;;
4 | -v)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-v" && echo -e "\"-v\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("-v")
;;
5 | -t)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-t" && echo -e "\"-t\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
echo "What do you want your timeout to be?"
read -rt 120 -p "Please type an integer: " up_timeout || { echo -e "\nFailed to make a selection in time" ; exit; }
! [[ $up_timeout =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_timeout\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 3 && continue
update_selection+=("-t" "$up_timeout")
;;
6 | -b)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-b" && echo -e "\"-b\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
echo "Up to how many backups should we keep?"
read -rt 120 -p "Please type an integer: " up_backups || { echo -e "\nFailed to make a selection in time" ; exit; }
! [[ $up_backups =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_backups\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 3 && continue
[[ $up_backups == 0 ]] && echo -e "Error: Number of backups cannot be 0\nNOT adding it to the list" && sleep 3 && continue
update_selection+=("-b" "$up_backups")
;;
7 | -s)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-s" && echo -e "\"-s\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("-s")
;;
8 | -p)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-p" && echo -e "\"-p\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("-p")
;;
9 | --self-update )
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "--self-update" && echo -e "\"--self-update\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("--self-update")
;;
99)
count=2
echo "restarting"
for i in "${update_selection[@]:2}"
do
unset "update_selection[$count]"
echo "$i removed"
((count++))
done
sleep 3
continue
;;
*)
echo "\"$current_selection\" was not an option, try again" && sleep 3 && continue
;;
esac
done
;; ;;
*) *)
echo "\"$selection\" was not an option, please try agian" && sleep 3 && menu echo "\"$selection\" was not an option, please try agian" && sleep 3 && menu

1
functions/misc.sh
View File

@@ -48,6 +48,7 @@ echo "--mount | Initiates mounting feature, choose between unmounting an
echo "--restore | Opens a menu to restore a \"heavy_script\" backup that was taken on your \"ix-applications\" dataset" echo "--restore | Opens a menu to restore a \"heavy_script\" backup that was taken on your \"ix-applications\" dataset"
echo "--delete-backup | Opens a menu to delete backups on your system" echo "--delete-backup | Opens a menu to delete backups on your system"
echo "--dns | list all of your applications DNS names and their web ports" echo "--dns | list all of your applications DNS names and their web ports"
echo "--cmd | Open a shell for one of your applications"
echo echo
echo "Update Types" echo "Update Types"
echo "------------" echo "------------"

174
functions/script_create.sh Normal file
View File

@@ -0,0 +1,174 @@
#!/bin/bash
script_create(){
while true
do
clear -x
title
echo "Choose Your Update Type"
echo "-----------------------"
echo "1) -U | Update all applications, ignores versions"
echo "2) -u | Update all applications, does not update Major releases"
echo
echo "0) Exit"
echo
read -rt 120 -p "Type the Number or Flag: " current_selection || { echo -e "\nFailed to make a selection in time" ; exit; }
case $current_selection in
0 | [Ee][Xx][Ii][Tt])
echo "Exiting.."
exit
;;
1 | -U)
while true
do
echo -e "\nHow many applications do you want updating at the same time?"
read -rt 120 -p "Please type an integer greater than 0: " up_async || { echo -e "\nFailed to make a selection in time" ; exit; }
if [[ $up_async == 0 ]]; then
echo "Error: \"$up_async\" is less than 1"
echo "NOT adding it to the list"
sleep 3
continue
elif ! [[ $up_async =~ ^[0-9]+$ ]]; then
echo "Error: \"$up_async\" is invalid, it needs to be an integer"
echo "NOT adding it to the list"
sleep 3
continue
else
update_selection+=("-U" "$up_async")
break
fi
done
break
;;
2 | -u)
while true
do
echo -e "\nHow many applications do you want updating at the same time?"
read -rt 120 -p "Please type an integer greater than 0: " up_async || { echo -e "\nFailed to make a selection in time" ; exit; }
if [[ $up_async == 0 ]]; then
echo "Error: \"$up_async\" is less than 1"
echo "NOT adding it to the list"
sleep 3
continue
elif ! [[ $up_async =~ ^[0-9]+$ ]]; then
echo "Error: \"$up_async\" is invalid, it needs to be an integer"
echo "NOT adding it to the list"
sleep 3
continue
else
update_selection+=("-u" "$up_async")
break
fi
done
break
;;
*)
echo "$current_selection was not an option, try again" && sleep 3
continue
;;
esac
done
while true
do
clear -x
title
echo "Update Options"
echo "--------------"
echo "1) -r | Roll-back applications if they fail to update"
echo "2) -i | Add application to ignore list"
echo "3) -S | Shutdown applications prior to updating"
echo "4) -v | verbose output"
echo "5) -t | Set a custom timeout in seconds when checking if either an App or Mountpoint correctly Started, Stopped or (un)Mounted. Defaults to 500 seconds"
echo
echo "Additional Options"
echo "------------------"
echo "6) -b | Back-up your ix-applications dataset, specify a number after -b"
echo "7) -s | sync catalog"
echo "8) -p | Prune unused/old docker images"
echo "9) --self-update | Updates HeavyScript prior to running any other commands"
echo
echo "99) Remove Update Options, Restart"
echo "00) Done making selections, proceed with update"
echo
echo "0) Exit"
echo
echo "Current Choices"
echo "---------------"
echo "bash heavy_script.sh ${update_selection[*]}"
echo
read -rt 600 -p "Type the Number or Flag: " current_selection || { echo -e "\nFailed to make a selection in time" ; exit; }
case $current_selection in
0 | [Ee][Xx][Ii][Tt])
echo "Exiting.."
exit
;;
00)
clear -x
echo "Running \"bash heavy_script.sh ${update_selection[*]}\""
echo
exec bash "$script_name" "${update_selection[@]}"
exit
;;
1 | -r)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-r" && echo -e "\"-r\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("-r")
;;
2 | -i)
read -rt 120 -p "What is the name of the application we should ignore?: " up_ignore || { echo -e "\nFailed to make a selection in time" ; exit; }
! [[ $up_ignore =~ ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ ]] && echo -e "Error: \"$up_ignore\" is not a possible option for an application name" && sleep 3 && continue
update_selection+=("-i" "$up_ignore")
;;
3 | -S)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-S" && echo -e "\"-S\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("-S")
;;
4 | -v)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-v" && echo -e "\"-v\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("-v")
;;
5 | -t)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-t" && echo -e "\"-t\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
echo "What do you want your timeout to be?"
read -rt 120 -p "Please type an integer: " up_timeout || { echo -e "\nFailed to make a selection in time" ; exit; }
! [[ $up_timeout =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_timeout\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 3 && continue
update_selection+=("-t" "$up_timeout")
;;
6 | -b)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-b" && echo -e "\"-b\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
echo "Up to how many backups should we keep?"
read -rt 120 -p "Please type an integer: " up_backups || { echo -e "\nFailed to make a selection in time" ; exit; }
! [[ $up_backups =~ ^[0-9]+$ ]] && echo -e "Error: \"$up_backups\" is invalid, it needs to be an integer\nNOT adding it to the list" && sleep 3 && continue
[[ $up_backups == 0 ]] && echo -e "Error: Number of backups cannot be 0\nNOT adding it to the list" && sleep 3 && continue
update_selection+=("-b" "$up_backups")
;;
7 | -s)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-s" && echo -e "\"-s\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("-s")
;;
8 | -p)
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "-p" && echo -e "\"-p\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("-p")
;;
9 | --self-update )
printf '%s\0' "${update_selection[@]}" | grep -Fxqz -- "--self-update" && echo -e "\"--self-update\" is already on here, skipping" && sleep 3 && continue #If option is already on there, skip it
update_selection+=("--self-update")
;;
99)
count=2
echo "restarting"
for i in "${update_selection[@]:2}"
do
unset "update_selection[$count]"
echo "$i removed"
((count++))
done
sleep 3
continue
;;
*)
echo "\"$current_selection\" was not an option, try again" && sleep 3 && continue
;;
esac
done
}
export -f script_create

3
functions/self_update.sh
View File

@@ -2,10 +2,11 @@
args=("$@") args=("$@")
self_update() { self_update() {
branch="main"
git fetch &> /dev/null git fetch &> /dev/null
echo "🅂 🄴 🄻 🄵" echo "🅂 🄴 🄻 🄵"
echo "🅄 🄿 🄳 🄰 🅃 🄴" echo "🅄 🄿 🄳 🄰 🅃 🄴"
if git diff --name-only origin/main | grep -qs ".sh" ; then if git diff --name-only origin/$branch | grep -qs ".sh" ; then
echo "Found a new version of HeavyScript, updating myself..." echo "Found a new version of HeavyScript, updating myself..."
git reset --hard -q git reset --hard -q
git pull --force -q git pull --force -q

279
functions/update_apps.sh
View File

@@ -8,13 +8,29 @@ echo -e "🅄 🄿 🄳 🄰 🅃 🄴 🅂"
echo "Asynchronous Updates: $update_limit" echo "Asynchronous Updates: $update_limit"
[[ -z $timeout ]] && echo "Default Timeout: 500" && timeout=500 || echo "Custom Timeout: $timeout" [[ -z $timeout ]] && echo "Default Timeout: 500" && timeout=500 || echo "Custom Timeout: $timeout"
[[ "$timeout" -le 120 ]] && echo "Warning: Your timeout is set low and may lead to premature rollbacks or skips" [[ "$timeout" -le 120 ]] && echo "Warning: Your timeout is set low and may lead to premature rollbacks or skips"
pool=$(cli -c 'app kubernetes config' | grep -E "dataset\s\|" | awk -F '|' '{print $3}' | awk -F '/' '{print $1}' | tr -d " \t\n\r")
it=0 it=0
while_count=0
rm deploying 2>/dev/null
while true while true
do do
while_status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status') if while_status=$(cli -m csv -c 'app chart_release query name,update_available,human_version,human_latest_version,status' 2>/dev/null) ; then
echo "$while_status" > temp.txt ((while_count++))
[[ -z $while_status ]] && continue || echo -e "$while_count\n$while_status" > all_app_status
mapfile -t deploying_check < <(grep ",DEPLOYING," all_app_status)
for i in "${deploying_check[@]}"
do
app_name=$(echo "$i" | awk -F ',' '{print $1}')
[[ ! -e deploying ]] && touch deploying
if ! grep -qs "$app_name,DEPLOYING" deploying; then
echo "$app_name,DEPLOYING" >> deploying
fi
done
else
echo "Middlewared timed out. Consider setting a lower number for async applications"
continue
fi
proc_count=${#processes[@]} proc_count=${#processes[@]}
count=0 count=0
for proc in "${processes[@]}" for proc in "${processes[@]}"
@@ -25,23 +41,29 @@ do
if [[ "$proc_count" -ge "$update_limit" ]]; then if [[ "$proc_count" -ge "$update_limit" ]]; then
sleep 3 sleep 3
elif [[ $it -lt ${#array[@]} ]]; then elif [[ $it -lt ${#array[@]} ]]; then
update_apps "${array[$it]}" & # loop=0
# until [[ $loop -ge 2 || $it -ge ${#array[@]} ]];
# do
pre_process "${array[$it]}" &
processes+=($!) processes+=($!)
((it++)) ((it++))
# ((loop++))
# done
elif [[ $proc_count != 0 ]]; then # Wait for all processes to finish elif [[ $proc_count != 0 ]]; then # Wait for all processes to finish
sleep 3 sleep 3
else # All processes must be completed, break out of loop else # All processes must be completed, break out of loop
break break
fi fi
done done
rm temp.txt rm all_app_status 2>/dev/null
rm deploying 2>/dev/null
echo echo
echo echo
} }
export -f commander export -f commander
update_apps(){ pre_process(){
app_name=$(echo "${array[$it]}" | awk -F ',' '{print $1}') #print out first catagory, name. app_name=$(echo "${array[$it]}" | awk -F ',' '{print $1}') #print out first catagory, name.
printf '%s\0' "${ignore[@]}" | grep -iFxqz "${app_name}" && echo -e "\n$app_name\nIgnored, skipping" && return 0 #If application is on ignore list, skip printf '%s\0' "${ignore[@]}" | grep -iFxqz "${app_name}" && echo -e "\n$app_name\nIgnored, skipping" && return 0 #If application is on ignore list, skip
old_app_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #previous/current Application MAJOR Version old_app_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $1}' | awk -F '.' '{print $1}') #previous/current Application MAJOR Version
@@ -51,119 +73,190 @@ new_chart_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}' | awk -F '_' '{pr
startstatus=$(echo "${array[$it]}" | awk -F ',' '{print $2}') #status of the app: STOPPED / DEPLOYING / ACTIVE startstatus=$(echo "${array[$it]}" | awk -F ',' '{print $2}') #status of the app: STOPPED / DEPLOYING / ACTIVE
diff_app=$(diff <(echo "$old_app_ver") <(echo "$new_app_ver")) #caluclating difference in major app versions diff_app=$(diff <(echo "$old_app_ver") <(echo "$new_app_ver")) #caluclating difference in major app versions
diff_chart=$(diff <(echo "$old_chart_ver") <(echo "$new_chart_ver")) #caluclating difference in Chart versions diff_chart=$(diff <(echo "$old_chart_ver") <(echo "$new_chart_ver")) #caluclating difference in Chart versions
[[ "$diff_app" != "$diff_chart" && $update_apps == "true" ]] && echo -e "\n$app_name\nMajor Release, update manually" && return
old_full_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}') #Upgraded From old_full_ver=$(echo "${array[$it]}" | awk -F ',' '{print $4}') #Upgraded From
new_full_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}') #Upraded To new_full_ver=$(echo "${array[$it]}" | awk -F ',' '{print $5}') #Upraded To
rollback_version=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}') rollback_version=$(echo "${array[$it]}" | awk -F ',' '{print $4}' | awk -F '_' '{print $2}')
if [[ "$diff_app" == "$diff_chart" || "$update_all_apps" == "true" ]]; then #continue to update if grep -qs "^$app_name," failed 2>/dev/null; then
if [[ $stop_before_update == "true" ]]; then # Check to see if user is using -S or not failed_ver=$(grep "^$app_name," failed | awk -F ',' '{print $2}')
if [[ "$startstatus" == "STOPPED" ]]; then # if status is already stopped, skip while loop if [[ "$failed_ver" == "$new_full_ver" ]] ; then
echo_array+=("\n$app_name") echo -e "\n$app_name"
[[ "$verbose" == "true" ]] && echo_array+=("Updating..") echo -e "Skipping previously failed version:\n$new_full_ver"
cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("FAILED")
return 0 return 0
else # if status was not STOPPED, stop the app prior to updating else
echo_array+=("\n$app_name") sed -i /"$app_name",/d failed
fi
fi
[[ ! -e external_services ]] && touch external_services
if ! grep -qs "^$app_name," external_services ; then
if ! grep qs "/external-service" /mnt/"$pool"/ix-applications/releases/"$app_name"/charts/"$(find /mnt/"$pool"/ix-applications/releases/"$app_name"/charts/ -maxdepth 1 -type d -printf '%P\n' | sort -r | head -n 1)"/Chart.yaml 2>/dev/null; then
echo "$app_name,false" >> external_services
else
echo "$app_name,true" >> external_services
fi
fi
echo_array+=("\n$app_name")
if [[ $stop_before_update == "true" && "$startstatus" != "STOPPED" ]]; then # Check to see if user is using -S or not
[[ "$verbose" == "true" ]] && echo_array+=("Stopping prior to update..") [[ "$verbose" == "true" ]] && echo_array+=("Stopping prior to update..")
midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && SECONDS=0 || echo_array+=("FAILED") if stop_app ; then
while [[ "$status" != "STOPPED" ]]
do
status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}')
if [[ "$status" == "STOPPED" ]]; then
echo_array+=("Stopped") echo_array+=("Stopped")
[[ "$verbose" == "true" ]] && echo_array+=("Updating..") else
cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("Failed to update") echo_array+=("Error: Failed to stop $app_name")
break echo_array
elif [[ "$SECONDS" -ge "$timeout" ]]; then return 1
echo_array+=("Error: Run Time($SECONDS) has exceeded Timeout($timeout)")
break
elif [[ "$status" != "STOPPED" ]]; then
[[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be STOPPED")
sleep 10
fi
done
fi
else #user must not be using -S, just update
echo_array+=("\n$app_name")
[[ "$verbose" == "true" ]] && echo_array+=("Updating..")
cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null && echo_array+=("Updated\n$old_full_ver\n$new_full_ver") && after_update_actions || echo_array+=("FAILED")
fi fi
fi
[[ "$verbose" == "true" ]] && echo_array+=("Updating..")
if update_app ;then
echo_array+=("Updated\n$old_full_ver\n$new_full_ver")
else else
echo_array+=("\n$app_name\nMajor Release, update manually") echo_array+=("Failed to update")
return 0 echo_array
return
fi
if grep -qs "^$app_name,true" external_services ; then
echo_array
return
else
after_update_actions
fi fi
} }
export -f update_apps export -f pre_process
update_app(){
current_loop=0
while true
do
update_avail=$(grep "^$app_name," all_app_status | awk -F ',' '{print $3}')
if [[ $update_avail == "true" ]]; then
if ! cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null ; then
before_loop=$(head -n 1 all_app_status)
current_loop=0
until [[ "$(grep "^$app_name," all_app_status | awk -F ',' '{print $3}')" != "$update_avail" ]] # Wait for a specific change to app status, or 3 refreshes of the file to go by.
do
if [[ $current_loop -gt 2 ]]; then
cli -c 'app chart_release upgrade release_name=''"'"$app_name"'"' &> /dev/null || return 1 # After waiting, attempt an update once more, if fails, return error code
elif ! echo -e "$(head -n 1 all_app_status)" | grep -qs ^"$before_loop" ; then # The file has been updated, but nothing changed specifically for the app.
before_loop=$(head -n 1 all_app_status)
((current_loop++))
fi
sleep 1
done
fi
break
elif [[ $update_avail == "false" ]]; then
break
else
sleep 3
fi
done
}
export -f update_app
stop_app(){
count=0
while [[ "$status" != "STOPPED" ]]
do
status=$( grep "^$app_name," all_app_status | awk -F ',' '{print $2}')
if [[ $count -gt 2 ]]; then # If failed to stop app 3 times, return failure to parent shell
return 1
elif ! cli -c 'app chart_release scale release_name='\""$app_name"\"\ 'scale_options={"replica_count": 0}' &> /dev/null ; then
before_loop=$(head -n 1 all_app_status)
((count++))
until [[ $(head -n 1 all_app_status) != "$before_loop" ]] # Upon failure, wait for status update before continuing
do
sleep 1
done
else
break
fi
done
}
export -f stop_app
after_update_actions(){ after_update_actions(){
SECONDS=0 SECONDS=0
count=0 count=0
if [[ $rollback == "true" ]]; then if [[ $rollback == "true" || "$startstatus" == "STOPPED" ]]; then
while true while true
do do
(( count++ )) status=$(grep "^$app_name," all_app_status | awk -F ',' '{print $2}')
status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}') if [[ $count -lt 1 && $status == "ACTIVE" && "$(grep "^$app_name," deploying 2>/dev/null | awk -F ',' '{print $2}')" != "DEPLOYING" ]]; then # If status shows up as Active or Stopped on the first check, verify that. Otherwise it may be a false report..
if [[ "$status" == "ACTIVE" && "$startstatus" == "STOPPED" ]]; then [[ "$verbose" == "true" ]] && echo_array+=("Verifying $status..")
[[ "$verbose" == "true" ]] && echo_array+=("Returing to STOPPED state..") before_loop=$(head -n 1 all_app_status)
midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo_array+=("Stopped")|| echo_array+=("FAILED") current_loop=0
break until [[ "$status" != "ACTIVE" || $current_loop -gt 4 ]] # Wait for a specific change to app status, or 3 refreshes of the file to go by.
elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" != "true" ]]; then
echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nIf this is a slow starting application, set a higher timeout with -t\nIf this applicaion is always DEPLOYING, you can disable all probes under the Healthcheck Probes Liveness section in the edit configuration\nReverting update..")
midclt call chart.release.rollback "$app_name" "{\"item_version\": \"$rollback_version\"}" &> /dev/null
[[ "$startstatus" == "STOPPED" ]] && failed="true" && after_update_actions && unset failed #run back after_update_actions function if the app was stopped prior to update
break
elif [[ "$SECONDS" -ge "$timeout" && "$status" == "DEPLOYING" && "$failed" == "true" ]]; then
echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)\nThe application failed to be ACTIVE even after a rollback,\nManual intervention is required\nAbandoning")
break
elif [[ "$status" == "STOPPED" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Stopped..") && sleep 15 && continue #if reports stopped on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check
echo_array+=("Stopped")
break #if reports stopped any time after the first loop, assume its extermal services.
elif [[ "$status" == "ACTIVE" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Active..") && sleep 15 && continue #if reports active on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check
echo_array+=("Active")
break #if reports active any time after the first loop, assume actually active.
else
[[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE")
sleep 15
continue
fi
done
else
if [[ "$startstatus" == "STOPPED" ]]; then
while true #using a constant while loop, then breaking out of the loop with break commands below.
do do
(( count++ )) status=$(grep "^$app_name," all_app_status | awk -F ',' '{print $2}')
status=$( grep "^$app_name," temp.txt | awk -F ',' '{print $2}') sleep 1
if [[ "$status" == "STOPPED" ]]; then if ! echo -e "$(head -n 1 all_app_status)" | grep -qs ^"$before_loop" ; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Stopped..") && sleep 15 && continue #if reports stopped on FIRST time through loop, double check before_loop=$(head -n 1 all_app_status)
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports stopped on FIRST time through loop, double check ((current_loop++))
echo_array+=("Stopped") #assume actually stopped anytime AFTER the first loop
break
elif [[ "$status" == "ACTIVE" ]]; then
[[ "$count" -le 1 && "$verbose" == "true" ]] && echo_array+=("Verifying Active..") && sleep 15 && continue #if reports active on FIRST time through loop, double check
[[ "$count" -le 1 && -z "$verbose" ]] && sleep 15 && continue #if reports active on FIRST time through loop, double check
[[ "$verbose" == "true" ]] && echo_array+=("Returing to STOPPED state..")
midclt call chart.release.scale "$app_name" '{"replica_count": 0}' &> /dev/null && echo_array+=("Stopped")|| echo_array+=("FAILED")
break
elif [[ "$SECONDS" -ge "$timeout" ]]; then
echo_array+=("Error: Run Time($SECONDS) has exceeded Timeout($timeout)")
break
else
[[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE")
sleep 10
continue
fi fi
done done
fi fi
(( count++ ))
if [[ "$status" == "ACTIVE" ]]; then
if [[ "$startstatus" == "STOPPED" ]]; then
[[ "$verbose" == "true" ]] && echo_array+=("Returing to STOPPED state..")
if stop_app ; then
echo_array+=("Stopped")
else
echo_array+=("Error: Failed to stop $app_name")
echo_array
return 1
fi
break
else
echo_array+=("Active")
break
fi
elif [[ "$SECONDS" -ge "$timeout" ]]; then
if [[ $rollback == "true" ]]; then
if [[ "$failed" != "true" ]]; then
echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)")
echo_array+=("If this is a slow starting application, set a higher timeout with -t")
echo_array+=("If this applicaion is always DEPLOYING, you can disable all probes under the Healthcheck Probes Liveness section in the edit configuration")
echo_array+=("Reverting update..")
midclt call chart.release.rollback "$app_name" "{\"item_version\": \"$rollback_version\"}" &> /dev/null || { echo_array+=("Error: Failed to rollback $app_name") ; break ; }
[[ "$startstatus" == "STOPPED" ]] && failed="true" && after_update_actions #run back after_update_actions function if the app was stopped prior to update
echo "$app_name,$new_full_ver" >> failed
break
else
echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)")
echo_array+=("The application failed to be ACTIVE even after a rollback")
echo_array+=("Manual intervention is required\nAbandoning")
break
fi
else
echo_array+=("Error: Run Time($SECONDS) for $app_name has exceeded Timeout($timeout)")
echo_array+=("If this is a slow starting application, set a higher timeout with -t")
echo_array+=("If this applicaion is always DEPLOYING, you can disable all probes under the Healthcheck Probes Liveness section in the edit configuration")
break
fi
else
[[ "$verbose" == "true" ]] && echo_array+=("Waiting $((timeout-SECONDS)) more seconds for $app_name to be ACTIVE")
sleep 5
continue
fi
done
fi fi
echo_array
}
export -f after_update_actions
echo_array(){
#Dump the echo_array, ensures all output is in a neat order. #Dump the echo_array, ensures all output is in a neat order.
for i in "${echo_array[@]}" for i in "${echo_array[@]}"
do do
echo -e "$i" echo -e "$i"
done done
} }
export -f after_update_actions export -f echo_array

15
heavy_script.sh
View File

@@ -8,20 +8,15 @@ script_name="heavy_script.sh"
cd "$script_path" || { echo "Error: Failed to change to script directory" ; exit ; } cd "$script_path" || { echo "Error: Failed to change to script directory" ; exit ; }
# shellcheck source=functions/backup.sh
source functions/backup.sh source functions/backup.sh
# shellcheck source=functions/dns.sh
source functions/dns.sh source functions/dns.sh
# shellcheck source=functions/menu.sh
source functions/menu.sh source functions/menu.sh
# shellcheck source=functions/misc.sh
source functions/misc.sh source functions/misc.sh
# shellcheck source=functions/mount.sh
source functions/mount.sh source functions/mount.sh
# shellcheck source=functions/self_update.sh
source functions/self_update.sh source functions/self_update.sh
# shellcheck source=functions/update_apps.sh
source functions/update_apps.sh source functions/update_apps.sh
source functions/cmd_to_container.sh
source functions/script_create.sh
@@ -46,6 +41,9 @@ do
dns) dns)
dns="true" dns="true"
;; ;;
cmd)
cmd="true"
;;
restore) restore)
restore="true" restore="true"
;; ;;
@@ -139,8 +137,9 @@ done
[[ "$update_all_apps" == "true" && "$update_apps" == "true" ]] && echo -e "-U and -u cannot BOTH be called" && exit [[ "$update_all_apps" == "true" && "$update_apps" == "true" ]] && echo -e "-U and -u cannot BOTH be called" && exit
#Continue to call functions in specific order #Continue to call functions in specific order
[[ "$help" == "true" ]] && help
[[ "$self_update" == "true" ]] && self_update [[ "$self_update" == "true" ]] && self_update
[[ "$help" == "true" ]] && help
[[ "$cmd" == "true" ]] && cmd_to_container && exit
[[ "$deleteBackup" == "true" ]] && deleteBackup && exit [[ "$deleteBackup" == "true" ]] && deleteBackup && exit
[[ "$dns" == "true" ]] && dns && exit [[ "$dns" == "true" ]] && dns && exit
[[ "$restore" == "true" ]] && restore && exit [[ "$restore" == "true" ]] && restore && exit