Knowledge Base

Browse our knowledge base for free solutions to common problems

AWX Docker Migration From Host to Host

Created On: 23 February 2023
Written by: Ben

Introduction

Following a recent job we did for one of our customers we decided to document the process we used to move docker-based AWX from one host to another.

Assumptions:

  1. Your AWX instance is running inside of Docker.
  2. Migration is from CentOS 7 / RHEL 7 to CentOS 7 / RHEL 7
  3. AWX git clone is located in /root/
  4. AWX Postgres is running as a docker container

Old AWX Node Actions

So we can start the migration process ensure that all of the AWX containers are running inside of docker:

[root@ansible ~]# docker ps
CONTAINER ID   IMAGE                COMMAND                  CREATED       STATUS       PORTS                                                                            NAMES
f3f578b706a3   ansible/awx:17.1.0   "/usr/bin/tini -- /u…"   5 hours ago   Up 5 hours   8052/tcp                                                                         awx_task
cda90e08eb47   ansible/awx:17.1.0   "/usr/bin/tini -- /b…"   5 hours ago   Up 5 hours   0.0.0.0:80->8052/tcp, :::80->8052/tcp, 0.0.0.0:443->8053/tcp, :::443->8053/tcp   awx_web
0c40a5cabea4   postgres:12          "docker-entrypoint.s…"   5 hours ago   Up 5 hours   5432/tcp                                                                         awx_postgres
5f4c84bd0185   redis                "docker-entrypoint.s…"   5 hours ago   Up 5 hours   6379/tcp                                                                         awx_redis

Ensure as a minimum you have the following running:

  • awx_task
  • awx_web
  • awx_redis

Sometimes with docker  based  awx_postgres container may be missing, this is normally due to the database being hosted away from the container locally or externally, more on that further down.

If any of the containers are down you should start them with the following:

docker start awx_web awx_redis awx_postgres awx_task

Create a new directory for the config backups we are going to generate with:

mkdir /root/awx_config_backup
cd /root/awx_config_backup

Now check for the presence of awx-cli using the following command:

which awx-cli

If nothing is returned then the likelihood is that awx-cli is not installed on the machine. The awx-cli comes packaged with ansible-tower-cli pip module, the easiest way to install this is with the following command:

pip install ansible-tower-cli --upgrade

or (depending on python version)

pip3 install ansible-tower-cli --upgrade

Sometimes even after install it maybe excluded missing from your environment variables, in this case locate it using the following:

find / -type f -name "awx-cli"

Example:

[root@ansible ~]# find / -type f -name "awx-cli"
/usr/local/bin/awx-cli
[root@ansible ~]#

If the case is that awx-cli is missing as an environment variable then make sure you call it using it's full path. If not feel free to use just awx-cli.

Disable Verify SSL Option

Sometimes verify SSL option can cause issues for us when exporting our data, to prevent this use the following:

/usr/local/bin/awx-cli config verify_ssl False

Export AWX Data Into JSON Format

The data from the old docker AWX instance is going to be dumped into JSON files. We can generate these with the following commands:

/usr/local/bin/awx-cli receive -u USERNAME_HERE -p 'PASSWORD_HERE' -h https://my-awx-domain-or-ip.com --credential all > /root/awx_config_backup/credential.json
/usr/local/bin/awx-cli receive -u USERNAME_HERE -p 'PASSWORD_HERE' -h https://my-awx-domain-or-ip.com --project all > /root/awx_config_backup/project.json
/usr/local/bin/awx-cli receive -u USERNAME_HERE -p 'PASSWORD_HERE' -h https://my-awx-domain-or-ip.com --inventory all > /root/awx_config_backup/inventory.json
/usr/local/bin/awx-cli receive -u USERNAME_HERE -p 'PASSWORD_HERE' -h https://my-awx-domain-or-ip.com --job_template all > /root/awx_config_backup/job_template.json
/usr/local/bin/awx-cli receive -u USERNAME_HERE -p 'PASSWORD_HERE' -h https://my-awx-domain-or-ip.com --workflow all > /root/awx_config_backup/workflow.json

Ensure that the following are replaced to align with the AWX instance you are migrating from:

  • USERNAME_HERE - This should be the username used to login to AWX instance via the browser
  • PASSWORD_HERE - This should be the password for the user
  • https://my-awx-domain-or-ip.com - This should be the URL / Hostname or IP Address of the AWX instance

If you experience issues then try using http rather than https. As a rule you should use whatever you access your AWX instance with from within the browser.

Once all of the files have been generated add the original installer inventory file to the directory with:

cp -a /root/awx/installer/inventory /root/awx_config_backup/

Now tar.gz them up with the following:

tar -zcvf /root/awx_config_backup.tar.gz /root/awx_config_backup/

Shutdown AWX On Old Host

We have finished on the old server for the time being, time to stop the AWX containers:

docker stop awx_web awx_redis awx_postgres awx_task

Confirm they are stopped with the following:

[root@ansible~]# docker ps
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
[root@ansible ~]#

Transfer Config Backups

Finally transfer the configuration backups from the old server to the new server:

scp /root/awx_config_backup.tar.gz USERNAME_HERE@NEW_SERVER_IP_HOSTNAME_HERE:awx_config_backup.tar.gz

If you are using key based authentication with a custom port the command might look something like this:

scp -P CUSTOM_PORT -i PATH_2_PRIVATE_KEY /root/awx_config_backup.tar.gz USERNAME_HERE@NEW_SERVER_IP:awx_config_backup.tar.gz

Obviously replace the following with your specific details:

  • USERNAME_HERE - With the SSH username for the new server
  • NEW_SERVER_IP - With the IP address of the new AWX server
  • CUSTOM_PORT - With the custom SSH port active on the new AWX server
  • PATH_2_PRIVATE_KEY - With the path to the locally stored private key to authenticate against the new AWX server

DNS Actions

As downtime was agreed with our customer at this stage the A and AAAA record were updated to represent the new IP address of the AWX server.

This would ensure that when things were up and running they should also work inside of DNS. It will also allow us to create a LetsEncrypt SSL for them to use on the AWX interface when we got things up and running.

New AWX Node Actions

Install Required Packages & Repos

To prepare the new AWX server additional packages are to be installed by running the following in sequence:

yum install epel-release -y
yum clean all
yum makecache fast
yum install git gcc gcc-c++ ansible nodejs gettext device-mapper-persistent-data lvm2 bzip2 python3-pip python-docker wget nano yum-util screen certbot -y

We also need to add the docker repo and the tower CLI repo. The docker repo will be used to install docker and docker-compose, the tower cli repo will be used to install awx-cli. Add the repos with the following:

wget https://download.docker.com/linux/centos/docker-ce.repo --directory-prefix /etc/yum.repos.d/
wget https://releases.ansible.com/ansible-tower/cli/ansible-tower-cli-centos7.repo --directory-prefix /etc/yum.repos.d/

Now install docker and docker-compose:

yum install docker-ce -y

Now install ansible tower cli:

yum install ansible-tower-cli

Sometimes installing docker using the repo misses docker-compose, if this is the case and nothing is returned when running which docker-compose then install compose with pip3:

pip3 install docker-compose

Even now it may be missing from the environment paths, if nothing is returned when running which docker-compose even after installing via pip3 then manually add it to path with the following command:

sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

Start & Enable Packages

First start our services:

systemctl start docker

Now enable the services so they start on boot:

systemctl enable docker

Open Firewall Ports

We are running firewalld and for AWX to function we need to open two different ports:

firewall-cmd --permanent --add-port=80/tcp
firewall-cmd --permanent --add-port=443/tcp
firewall-cmd --reload

Allowing these ports should mean we can access the AWX interface after migration / install.

Create LetsEncrypt SSL

Run the following to trigger certificate generation via certbot:

certbot certonly
  1. When asked how you would like to generate the certificate use "1: Spin up a temporary webserver (standalone)"
  2. Enter the email address to receive renewal notifications
  3. Read and agree to any terms which are printed in the terminal
  4. Enter the DNS name for the AWX instance you are deploying. E.g ansible.mydomain.com. Please note this DNS name must resolve to the server IP.
  5. Certificate should be generated

Example output with domain omitted:

[root@ansible ~]# certbot certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator standalone, Installer None
Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): [email protected]
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Account registered.
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
to cancel): ansible.mydomain.com
Requesting a certificate for ansible.mydomain.com
Performing the following challenges:
http-01 challenge for ansible.mydomain.com
Waiting for verification...
Cleaning up challenges
Subscribe to the EFF mailing list (email: [email protected]).
Starting new HTTPS connection (1): supporters.eff.org
We were unable to subscribe you the EFF mailing list because your e-mail address appears to be invalid. You can try again later by visiting https://act.eff.org.

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/ansible.mydomain.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/ansible.mydomain.com/privkey.pem
   Your certificate will expire on 2023-05-22. To obtain a new or
   tweaked version of this certificate in the future, simply run
   certbot again. To non-interactively renew *all* of your
   certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

The certificate is now generated and ready to be used on our new AWX instance.

Clone AWX Via Git

Ensure that you move into the root directory before cloning AWX via git:

cd /root/

Now clone the AWX repo with the following command:

git clone -b 17.1.0 https://github.com/ansible/awx.git

Example output:

[root@ansible ~]# git clone -b 17.1.0 https://github.com/ansible/awx.git
Cloning into 'awx'...
remote: Enumerating objects: 309800, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 309800 (delta 1), reused 2 (delta 0), pack-reused 309792
Receiving objects: 100% (309800/309800), 273.77 MiB | 11.61 MiB/s, done.
Resolving deltas: 100% (239219/239219), done.
Note: checking out 'c1ab815c80cac96508d9779d92bc1280d0347627'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

Extract Old Configs

As part of the next step we will be modifying the AWX inventory file before installing AWX. For this we need to check the old inventory file that was used to install AWX on the old server. Now is a good time to extract the tar.gz file which we previously transferred as this contains our configurations:

cd /root/
tar -xf awx_config_backup.tar.gz

The configuration files should now be extracted and available in /root/awx_config_backup/

Modify AWX Inventory File

The inventory file inside of the cloned git is used for AWX install. We need to ensure that all of the properties are populated correctly inside of there before proceeding with the install.

The likelihood is that you can replace:

/root/awx/installer/inventory

with

/root/awx_config_backup/inventory

However just to be certain we will modify only the properties we need to within the new /root/awx/installer/inventory file.

Open the file with nano:

nano /root/awx/installer/inventory

AWX Hostname Config Change

Look for the following line:

awx_task_hostname=awx
awx_web_hostname=awxweb

Change them to what you want them to be, in our case:

awx_task_hostname=ansible
awx_web_hostname=ansible.mydomain.com

AWX Port & LetsEncrypt SSL Config Change

Now locate host_port and host_port_ssl and make sure they match the following:

host_port=80
host_port_ssl=443

Also add the following two lines so that your previously generated SSL certificate is loaded:

ssl_certificate=/etc/letsencrypt/live/ansible.mydomain.com/fullchain.pem
ssl_certificate_key=/etc/letsencrypt/live/ansible.mydomain.com/privkey.pem

AWX Postgres Config Change

As we are going to be using the AWX docker based version of postgres in the next step we need to ensure pg_hostname is commented and assign a random 10 character password for pg_password. Example:

# Set pg_hostname if you have an external postgres server, otherwise
# a new postgres service will be created
# pg_hostname=postgresql
pg_username=awx
# pg_password should be random 10 character alphanumeric string, when postgresql is running on kubernetes
# NB: it's a limitation of the "official" postgres helm chart
pg_password=Y7Ui98Vx6Y
pg_database=awx
pg_port=5432
#pg_sslmode=require

AWX Admin User Config Change

If you want to remove the default username next you need to modify admin_user with the following:

admin_user=admin

Change to whatever username you like, for example:

admin_user=ictu

Add a password for the user by uncommenting and modifying admin_password:

# admin_password=password

Example:

admin_password=NC@J7e60HZcE

AWX Disable Pre-Load Data Config

Locate the following line:

create_preload_data=True

Change this to:

create_preload_data=False

This is important because we will be loading our old data once the AWX build is complete.

AWX Secret Key Config Change

The value for secret_key is important as this allows you to decrypt credentials. For this you need to get the value from the old inventory config. Do do this run the following command first:

cat /root/awx_config_backup/inventory | grep secret_key

Example:

[root@ansible ~]# cat /root/awx_config_backup/inventory | grep secret_key
secret_key=zQje1SaMd7GXtA6ch7DzWYLRHu812RaARujFY7mb

You need to copy the full line and replace it inside of the new configuration.

AWX Alternative DNS Config Change

Locate the following line:

#awx_alternate_dns_servers="10.1.2.3,10.2.3.4"

Replace it with:

awx_alternate_dns_servers="8.8.8.8,8.8.4.4"

AWX Local Projects Directory

If you want to load local projects its a good idea to uncomment the following line:

#project_data_dir=/var/lib/awx/projects

In our case we did not require this option as the customer used remote git to manage repositories containing projects / playbooks.

AWX Example Config

After making the above changing this is an example of how our config looks:

localhost ansible_connection=local ansible_python_interpreter="/usr/bin/env python3"

[all:vars]

# Remove these lines if you want to run a local image build # Otherwise the setup playbook will install the official Ansible images. Versions may # be selected based on: latest, 1, 1.0, 1.0.0, 1.0.0.123 # by default the base will be used to search for ansible/awx dockerhub_base=ansible # Openshift Install # Will need to set -e openshift_password=developer -e docker_registry_password=$(oc whoami -t) # or set -e openshift_token=TOKEN # openshift_host=127.0.0.1:8443 # openshift_project=awx # openshift_user=developer # openshift_skip_tls_verify=False # openshift_pg_emptydir=True # Kubernetes Install # kubernetes_context=test-cluster # kubernetes_namespace=awx # kubernetes_web_svc_type=NodePort # Optional Kubernetes Variables # pg_image_registry=docker.io # pg_serviceaccount=awx # pg_volume_capacity=5 # pg_persistence_storageClass=StorageClassName # pg_persistence_existingclaim=postgres_pvc # pg_cpu_limit=1000 # pg_mem_limit=2 # Kubernetes Ingress Configuration # You can use the variables below to configure Kubernetes Ingress # Set hostname # kubernetes_ingress_hostname=awx.example.org # Add annotations. The example below shows an annotation to be used with Traefik but other Ingress controllers are also supported # kubernetes_ingress_annotations={'kubernetes.io/ingress.class': 'traefik', 'traefik.ingress.kubernetes.io/redirect-entry-point': 'https'} # Specify a secret for TLS termination # kubernetes_ingress_tls_secret=awx-cert # Kubernetes and Openshift Install Resource Requests # These are the request and limit values for a pod's container for task/web/redis/management. # The total amount of requested resources for a pod is the sum of all # resources requested by all containers in the pod # A cpu_request of 1500 is 1.5 cores for the container to start out with. # A cpu_limit defines the maximum cores that that container can reserve. # A mem_request of 2 is for 2 gigabytes of memory for the container # A mem_limit defines the maximum memory that that container can reserve. # Default values for these entries can be found in ./roles/kubernetes/defaults/main.yml # task_cpu_request=1500 # task_mem_request=2 # task_cpu_limit=2000 # task_mem_limit=4 # web_cpu_limit=1000 # web_mem_limit=2 # redis_cpu_limit=1000 # redis_mem_limit=3 # management_cpu_limit=2000 # management_mem_limit=2 # Common Docker parameters awx_task_hostname=ansible awx_web_hostname=ansible.mydomain.com # Local directory that is mounted in the awx_postgres docker container to place the db in postgres_data_dir="~/.awx/pgdocker" host_port=80 host_port_ssl=443 ssl_certificate=/etc/letsencrypt/live/ansible.mydomain.com/fullchain.pem # Optional key file ssl_certificate_key=/etc/letsencrypt/live/ansible.mydomain.com/privkey.pem docker_compose_dir="~/.awx/awxcompose" # Required for Openshift when building the image on your own # Optional for Openshift if using Dockerhub or another prebuilt registry # Required for Docker Compose Install if building the image on your own # Optional for Docker Compose Install if using Dockerhub or another prebuilt registry # Define if you want the image pushed to a registry. The container definition will also use these images # docker_registry=172.30.1.1:5000 # docker_registry_repository=awx # docker_registry_username=developer # Set pg_hostname if you have an external postgres server, otherwise # a new postgres service will be created # pg_hostname=postgresql pg_username=awx # pg_password should be random 10 character alphanumeric string, when postgresql is running on kubernetes # NB: it's a limitation of the "official" postgres helm chart pg_password=Y7Ui98Vx6Y pg_database=awx pg_port=5432 #pg_sslmode=require # If requiring SSL communication (e.g. pg_sslmode='verify-full') with Postgres # and using a self-signed certificate or a certificate signed by a custom CA # set pg_root_ca_file to a file containing the self-signed certificate or the # root CA certificate chain. # pg_root_ca_file='example_root_ca.crt' # The following variable is only required when using the provided # containerized postgres deployment on OpenShift # pg_admin_password=postgrespass # This will create or update a default admin (superuser) account in AWX, if not provided # then these default values are used admin_user=ictu admin_password=NC@J7e60HZcE # Whether or not to create preload data for demonstration purposes create_preload_data=False # AWX Secret key # It's *very* important that this stay the same between upgrades or you will lose the ability to decrypt # your credentials secret_key=zQje1SaMd7GXtA6ch7DzWYLRHu812RaARujFY7mb # By default a broadcast websocket secret will be generated. # If you would like to *rerun the playbook*, you need to set a unique password. # Otherwise it would generate a new one every playbook run. # broadcast_websocket_secret= # Build AWX with official logos # Requires cloning awx-logos repo as a sibling of this project. # Review the trademark guidelines at https://github.com/ansible/awx-logos/blob/master/TRADEMARKS.md # awx_official=false # Proxy #http_proxy=http://proxy:3128 #https_proxy=http://proxy:3128 #no_proxy=mycorp.org # Container networking configuration # Set the awx_task and awx_web containers' search domain(s) #awx_container_search_domains=example.com,ansible.com # Alternate DNS servers awx_alternate_dns_servers="8.8.8.8,8.8.4.4" # AWX project data folder. If you need access to the location where AWX stores the projects # it manages from the docker host, you can set this to turn it into a volume for the container. project_data_dir=/var/lib/awx/projects # AWX custom virtual environment folder. Only usable for local install. #custom_venv_dir=/opt/my-envs/ # CA Trust directory. If you need to provide custom CA certificates, supplying # this variable causes this directory on the host to be bind mounted over # /etc/pki/ca-trust in the awx_task and awx_web containers. # If you are deploying on openshift or kubernetes, set the variable to /etc/pki/ca-trust instead, # as the awx_web and awx_task containers will not run the `update-ca-trust` command. #ca_trust_dir=/etc/pki/ca-trust/source/anchors # Include /etc/nginx/awx_extra.conf # Note the use of glob pattern for nginx # which makes include "optional" - i.e. not fail # if file is absent #extra_nginx_include="/etc/nginx/awx_extra[.]conf" # Docker compose explicit subnet. Set to avoid overlapping your existing LAN networks. #docker_compose_subnet="172.17.0.1/16" # # Allow for different docker logging drivers # By Default; the logger will be json-file, however you can override # that by uncommenting the docker_logger below. # Be aware that journald may rate limit your log messages if you choose it. # See: https://docs.docker.com/config/containers/logging/configure/ # docker_logger=journald # # Add extra hosts to docker compose file. This might be necessary to # sneak in servernames. For example for DMZ self-signed CA certificates. # Equivialent to using the --add-host parameter with "docker run". #docker_compose_extra_hosts="otherserver.local:192.168.0.1,ldap-server.local:192.168.0.2"

Build / Install AWX

At this stage we are ready to build AWX using ansible. To do this go to the installer directory with the following:

cd /root/awx/installer

Now run the following command to begin the installation:

ansible-playbook -i inventory install.yml

The installation may take a while, the output should look similar to this:

[root@ansible installer]# ansible-playbook -i inventory install.yml

PLAY [Build and deploy AWX] **********************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************
ok: [localhost]

TASK [check_vars : admin_password should be defined] *********************************************************************************************
ok: [localhost] => {
    "changed": false, 
    "msg": "All assertions passed"
}

TASK [check_vars : include_tasks] ****************************************************************************************************************
skipping: [localhost]

TASK [check_vars : include_tasks] ****************************************************************************************************************
included: /root/awx/installer/roles/check_vars/tasks/check_docker.yml for localhost

TASK [check_vars : postgres_data_dir should be defined] ******************************************************************************************
ok: [localhost] => {
    "changed": false, 
    "msg": "All assertions passed"
}

TASK [image_build : Set global version if not provided] ******************************************************************************************
skipping: [localhost]

TASK [image_build : Verify awx-logos directory exists for official install] **********************************************************************
skipping: [localhost]

TASK [image_build : Copy logos for inclusion in sdist] *******************************************************************************************
skipping: [localhost]

TASK [image_build : Set awx image name] **********************************************************************************************************
skipping: [localhost]

TASK [image_build : Render Dockerfile] ***********************************************************************************************************
skipping: [localhost]

TASK [image_build : Build AWX image] *************************************************************************************************************
skipping: [localhost]

TASK [image_build : Tag awx images as latest] ****************************************************************************************************
skipping: [localhost] => (item=awx) 

TASK [image_push : Authenticate with Docker registry if registry password given] *****************************************************************
skipping: [localhost]

TASK [image_push : Remove awx image] *************************************************************************************************************
skipping: [localhost]

TASK [image_push : Tag and push awx image to registry] *******************************************************************************************
skipping: [localhost]

TASK [image_push : Set full image path for Registry] *********************************************************************************************
skipping: [localhost]

TASK [kubernetes : Generate broadcast websocket secret] ******************************************************************************************
skipping: [localhost]

TASK [kubernetes : fail] *************************************************************************************************************************
skipping: [localhost]

TASK [kubernetes : include_tasks] ****************************************************************************************************************
skipping: [localhost] => (item=openshift_auth.yml) 
skipping: [localhost] => (item=openshift.yml) 

TASK [kubernetes : include_tasks] ****************************************************************************************************************
skipping: [localhost] => (item=kubernetes_auth.yml) 
skipping: [localhost] => (item=kubernetes.yml) 

TASK [kubernetes : Use kubectl or oc] ************************************************************************************************************
skipping: [localhost]

TASK [kubernetes : set_fact] *********************************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Record deployment size] *******************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Set expected post-deployment Replicas value] **********************************************************************************
skipping: [localhost]

TASK [kubernetes : Delete existing Deployment (or StatefulSet)] **********************************************************************************
skipping: [localhost]

TASK [kubernetes : Get Postgres Service Detail] **************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Template PostgreSQL Deployment (OpenShift)] ***********************************************************************************
skipping: [localhost]

TASK [kubernetes : Deploy and Activate Postgres (OpenShift)] *************************************************************************************
skipping: [localhost]

TASK [kubernetes : Create Temporary Values File (Kubernetes)] ************************************************************************************
skipping: [localhost]

TASK [kubernetes : Populate Temporary Values File (Kubernetes)] **********************************************************************************
skipping: [localhost]

TASK [kubernetes : Deploy and Activate Postgres (Kubernetes)] ************************************************************************************
skipping: [localhost]

TASK [kubernetes : Remove tempfile] **************************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Set postgresql hostname to helm package service (Kubernetes)] *****************************************************************
skipping: [localhost]

TASK [kubernetes : Wait for Postgres to activate] ************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Check if Postgres 10 is being used] *******************************************************************************************
skipping: [localhost]

TASK [kubernetes : Set new pg image] *************************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Wait for change to take affect] ***********************************************************************************************
skipping: [localhost]

TASK [kubernetes : Set env var for pg upgrade] ***************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Wait for change to take affect] ***********************************************************************************************
skipping: [localhost]

TASK [kubernetes : Set env var for new pg version] ***********************************************************************************************
skipping: [localhost]

TASK [kubernetes : Wait for Postgres to redeploy] ************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Wait for Postgres to finish upgrading] ****************************************************************************************
skipping: [localhost]

TASK [kubernetes : Unset upgrade env var] ********************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Wait for Postgres to redeploy] ************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Set awx image name] ***********************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Determine Deployment api version] *********************************************************************************************
skipping: [localhost]

TASK [kubernetes : Get Root CA file contents] ****************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Render Root CA template] ******************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Apply Root CA template] *******************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Set Root CA file name] ********************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Set Root CA file location] ****************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Render deployment templates] **************************************************************************************************
skipping: [localhost] => (item=None) 
skipping: [localhost] => (item=None) 
skipping: [localhost] => (item=None) 
skipping: [localhost] => (item=None) 
skipping: [localhost]

TASK [kubernetes : Apply Deployment] *************************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Delete any existing management pod] *******************************************************************************************
skipping: [localhost]

TASK [kubernetes : Template management pod] ******************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Create management pod] ********************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Wait for management pod to start] *********************************************************************************************
skipping: [localhost]

TASK [kubernetes : Migrate database] *************************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Check for Tower Super users] **************************************************************************************************
skipping: [localhost]

TASK [kubernetes : create django super user if it does not exist] ********************************************************************************
skipping: [localhost]

TASK [kubernetes : update django super user password] ********************************************************************************************
skipping: [localhost]

TASK [kubernetes : Create the default organization if it is needed.] *****************************************************************************
skipping: [localhost]

TASK [kubernetes : Delete management pod] ********************************************************************************************************
skipping: [localhost]

TASK [kubernetes : Scale up deployment] **********************************************************************************************************
skipping: [localhost]

TASK [local_docker : Generate broadcast websocket secret] ****************************************************************************************
ok: [localhost]

TASK [local_docker : Create ~/.awx/pgdocker directory] *******************************************************************************************
changed: [localhost]

TASK [local_docker : Get full path of postgres data dir] *****************************************************************************************
changed: [localhost]

TASK [local_docker : Register temporary docker container] ****************************************************************************************
ok: [localhost]

TASK [local_docker : Check for existing Postgres data (run from inside the container for access to file)] ****************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "docker run --rm -v '/root/.awx/pgdocker:/var/lib/postgresql' centos:8 bash -c  \"[[ -f /var/lib/postgresql/10/data/PG_VERSION ]] && echo 'exists'\"\n", "delta": "0:00:17.694051", "end": "2023-02-21 19:34:24.780688", "msg": "non-zero return code", "rc": 1, "start": "2023-02-21 19:34:07.086637", "stderr": "Unable to find image 'centos:8' locally\n8: Pulling from library/centos\na1d0c7532777: Pulling fs layer\na1d0c7532777: Verifying Checksum\na1d0c7532777: Download complete\na1d0c7532777: Pull complete\nDigest: sha256:a27fd8080b517143cbbbab9dfb7c8571c40d67d534bbdee55bd6c473f432b177\nStatus: Downloaded newer image for centos:8", "stderr_lines": ["Unable to find image 'centos:8' locally", "8: Pulling from library/centos", "a1d0c7532777: Pulling fs layer", "a1d0c7532777: Verifying Checksum", "a1d0c7532777: Download complete", "a1d0c7532777: Pull complete", "Digest: sha256:a27fd8080b517143cbbbab9dfb7c8571c40d67d534bbdee55bd6c473f432b177", "Status: Downloaded newer image for centos:8"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [local_docker : Record Postgres version] ****************************************************************************************************
skipping: [localhost]

TASK [local_docker : Determine whether to upgrade postgres] **************************************************************************************
skipping: [localhost]

TASK [local_docker : Set up new postgres paths pre-upgrade] **************************************************************************************
skipping: [localhost]

TASK [local_docker : Stop AWX before upgrading postgres] *****************************************************************************************
skipping: [localhost]

TASK [local_docker : Upgrade Postgres] ***********************************************************************************************************
skipping: [localhost]

TASK [local_docker : Copy old pg_hba.conf] *******************************************************************************************************
skipping: [localhost]

TASK [local_docker : Remove old data directory] **************************************************************************************************
skipping: [localhost]

TASK [local_docker : Export Docker awx image if it isnt local and there isnt a registry defined] *************************************************
skipping: [localhost]

TASK [local_docker : Set docker base path] *******************************************************************************************************
skipping: [localhost]

TASK [local_docker : Ensure directory exists] ****************************************************************************************************
skipping: [localhost]

TASK [local_docker : Copy awx image to docker execution] *****************************************************************************************
skipping: [localhost]

TASK [local_docker : Load awx image] *************************************************************************************************************
skipping: [localhost]

TASK [local_docker : Set full image path for local install] **************************************************************************************
skipping: [localhost]

TASK [local_docker : Set DockerHub Image Paths] **************************************************************************************************
ok: [localhost]

TASK [local_docker : Create ~/.awx/awxcompose directory] *****************************************************************************************
changed: [localhost]

TASK [local_docker : Create Redis socket directory] **********************************************************************************************
changed: [localhost]

TASK [local_docker : Create Docker Compose Configuration] ****************************************************************************************
changed: [localhost] => (item={u'mode': u'0600', u'file': u'environment.sh'})
changed: [localhost] => (item={u'mode': u'0600', u'file': u'credentials.py'})
changed: [localhost] => (item={u'mode': u'0600', u'file': u'docker-compose.yml'})
changed: [localhost] => (item={u'mode': u'0600', u'file': u'nginx.conf'})
changed: [localhost] => (item={u'mode': u'0664', u'file': u'redis.conf'})

TASK [local_docker : Render SECRET_KEY file] *****************************************************************************************************
changed: [localhost]

TASK [local_docker : Remove AWX containers before migrating postgres so that the old postgres container does not get used] ***********************
ok: [localhost]

TASK [local_docker : Run migrations in task container] *******************************************************************************************
changed: [localhost]

TASK [local_docker : Start the containers] *******************************************************************************************************
changed: [localhost]

TASK [local_docker : Update CA trust in awx_web container] ***************************************************************************************
changed: [localhost]

TASK [local_docker : Update CA trust in awx_task container] **************************************************************************************
changed: [localhost]

TASK [local_docker : Wait for launch script to create user] **************************************************************************************
ok: [localhost -> localhost]


PLAY RECAP ***************************************************************************************************************************************
localhost                  : ok=20   changed=11   unreachable=0    failed=1    skipped=73   rescued=0    ignored=0

After install you should now see the interface at:

  • https://ansible.yourdomain.com/
  • http://ansible.yourdomain.com/

You can verify that the containers are now running using docker ps, for example:

[root@ansible installer]# docker ps
CONTAINER ID   IMAGE                COMMAND                  CREATED        STATUS        PORTS                                                                            NAMES
f3f578b706a3   ansible/awx:17.1.0   "/usr/bin/tini -- /u…"   46 hours ago   Up 32 hours   8052/tcp                                                                         awx_task
cda90e08eb47   ansible/awx:17.1.0   "/usr/bin/tini -- /b…"   46 hours ago   Up 32 hours   0.0.0.0:80->8052/tcp, :::80->8052/tcp, 0.0.0.0:443->8053/tcp, :::443->8053/tcp   awx_web
0c40a5cabea4   postgres:12          "docker-entrypoint.s…"   46 hours ago   Up 32 hours   5432/tcp                                                                         awx_postgres
5f4c84bd0185   redis                "docker-entrypoint.s…"   46 hours ago   Up 32 hours   6379/tcp                                                                         awx_redis
[root@ansible installer]#

If you wish to login you can do so using the username and password that you originally provided inside of the inventory file used to build AWX.

We are now in a position to import the old AWX configuration.

Restoring Old AWX Data

Check for presense of awx-cli

Check for the presence of awx-cli using the following command:

which awx-cli

If nothing is returned then the likelihood is that awx-cli is not installed on the machine. The awx-cli comes packaged with ansible-tower-cli pip module, the easiest way to install this is with the following command:

pip install ansible-tower-cli --upgrade

or (depending on python version)

pip3 install ansible-tower-cli --upgrade

Sometimes even after install it maybe excluded missing from your environment variables, in this case locate it using the following:

find / -type f -name "awx-cli"

Example:

[root@ansible ~]# find / -type f -name "awx-cli"
/usr/local/bin/awx-cli
[root@ansible ~]#

If the case is that awx-cli is missing as an environment variable then make sure you call it using it's full path. If not feel free to use just awx-cli.

Disable Verify SSL Option

Sometimes verify SSL option can cause issues for us when importing our data back into AWX, to prevent this use the following:

/usr/local/bin/awx-cli config verify_ssl False

Restore data into new AWX instance

Now to restore our data we need to run the following:

/usr/local/bin/awx-cli send -u USERNAME_HERE -p 'PASSWORD_HERE' -h https://my-awx-domain-or-ip.com /root/awx_config_backup/credential.json
/usr/local/bin/awx-cli send -u USERNAME_HERE -p 'PASSWORD_HERE' -h https://my-awx-domain-or-ip.com /root/awx_config_backup/project.json
/usr/local/bin/awx-cli send -u USERNAME_HERE -p 'PASSWORD_HERE' -h https://my-awx-domain-or-ip.com /root/awx_config_backup/inventory.json
/usr/local/bin/awx-cli send -u USERNAME_HERE -p 'PASSWORD_HERE' -h https://my-awx-domain-or-ip.com /root/awx_config_backup/job_template.json 
/usr/local/bin/awx-cli send -u USERNAME_HERE -p 'PASSWORD_HERE' -h https://my-awx-domain-or-ip.com /root/awx_config_backup/workflow.json

Ensure that the following are replaced to align with the AWX instance you are migrating from:

  • USERNAME_HERE - This should be the username used to login to AWX instance via the browser
  • PASSWORD_HERE - This should be the password for the user
  • https://my-awx-domain-or-ip.com - This should be the URL / Hostname or IP Address of the AWX instance

If you experience issues then try using http rather than https. As a rule you should use whatever you access your AWX instance with from within the browser.

Login and view the restore

You can now login to AWX to find all of your credentials, projects, inventories, job templates and workflows have been restored. So credentials may require re-authenticating.

ICTU LTD is a company registered England and Wales (Company No. 09344913) 15 Queen Square, Leeds, West Yorkshire, England, LS2 8AJ
Copyright © 2024 ICTU LTD, All Rights Reserved.
exit