Skip to main content

How to customize Liquibase Docker image for Observability in AWS CloudWatch

Tylor Graves
Updated

Applies to:

  • Liquibase Pro
  • Liquibase Open Source (Community)

Summary:

Liquibase publishes an official docker image for many versions of Liquibase. The official docker image needs to be customized to automatically push logs to an observability tool. In this article, we will look at pushing logs to AWS CloudWatch.

Environment Conditions:

  • Docker installed

Steps:

  1. Create a new repository. 
  2. Add a CloudWatch configuration file
  3. Create a Dockerfile
  4. Add an entrypoint into the Dockerfile
  5. Build the Docker image
  6. Configure Log Groups and Log Streams in AWS CloudWatch
  7. Setup permissions for the IAM user
  8. Run Docker image
  9. Build your dashboards

CloudWatch Agent configuration file

  1. Create an AWS CloudWatch Agent configuration file. See Create the CloudWatch agent configuration file. Name this file default_linux_config.json. 
    1. {
           "agent": {
                   "run_as_user": "liquibase",
                   "region": "us-east-1"
           },
           "logs": {
                  "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                             "file_path": "/liquibase/liquibase-cw-logs/**.json",
                                             "log_group_name": "liquibaseLogs",
                                             "log_stream_name": "liquibaseLogStream2",
                                             "retention_in_days": -1
                                        }
                                 ]
                        }
                  }
           }
      }

Create a Dockerfile:

  1. Create a Dockerfile which extends the Liquibase image.   
    1. # Use the Liquibase official image as the base image
      FROM liquibase/liquibase:4.24.0
  2. Install AWS CLI and AWS CloudWatch Agent. The AWS CLI is used to authenticate the IAM user. This enable the IAM user to push logs to CloudWatch. The CloudWatch Agent scans for logs in a specific folder and pushes those logs. This will be used by the customized Liquibase launch script to send the logs to the logging endpoint after each invocation. 
    1. RUN apt-get update && apt-get install -y python3-pip ca-certificates curl && pip3 install awscli && \
         curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb && \
         dpkg -i -E ./amazon-cloudwatch-agent.deb && \
         rm ./amazon-cloudwatch-agent.deb
  3. Configure the dockerfile with environment variables to setup for IAM user.  
    1. # Tell Amazon CloudWatch we are in a container
      ENV RUN_IN_CONTAINER="True"
      ENV AWS_CONFIG_FILE=/home/liquibase/.aws/config
      ENV AWS_CREDENTIALS_FILE=/home/liquibase/.aws/credentials

      # Set your AWS access and secret key as build arguments
      ARG AWS_ACCESS_KEY_ID
      ARG AWS_SECRET_ACCESS_KEY
      ARG AWS_SESSION_TOKEN

      # Create the credentials file with build arguments
      RUN mkdir -p /home/liquibase/.aws/ && \
          echo "[AmazonCloudWatchAgent]\naws_access_key_id = $AWS_ACCESS_KEY_ID\naws_secret_access_key = $AWS_SECRET_ACCESS_KEY\naws_session_token = $AWS_SESSION_TOKEN"  /home/liquibase/.aws/credentials && \
          echo "[AmazonCloudWatchAgent]\nregion = us-east-1\noutput = json"  /home/liquibase/.aws/config && \
          chmod a+rw /home/liquibase/.aws/credentials /home/liquibase/.aws/config
  4. Create a directory for Liquibase logs. This is the same directory specified in the CloudWatch agent configuration file: 
    1. RUN mkdir -p /liquibase/liquibase-cw-logs && \
          chown -R liquibase:liquibase /liquibase/liquibase-cw-logs && \
          chmod -R a+rw /opt/aws/amazon-cloudwatch-agent
  5. Copy the CloudWatch agent configuration file into the image: 
    1. COPY default_linux_config.json /opt/aws/amazon-cloudwatch-agent/bin/

Add an entrypoint to the Dockerfile

  1. Create an entrypoint script. The goal of this script is to run a Liquibase operation and push logs to AWS CloudWatch. Name this script as entrypoint.sh. 
    1. #!/bin/sh
      set -e

      echo "Running command: liquibase $@"
      liquibase $@ || true 
      # wait for liquibase to generate the structured log. (maybe this can be deleted)

      echo "Starting CloudWatch Agent..."
      # starts CW agent in the backgorund
      /opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent &

      # wait for the agent to be up and running
      sleep 15

      echo "Sending logs to AWS CloudWatch..."
      sleep 10
      exit 0
  2. Add the entrypoint.sh script to the Dockerfile:
    1. COPY --chown=liquibase:liquibase entrypoint.sh /liquibase/

      # Start the CloudWatch agent
      ENTRYPOINT ["/liquibase/entrypoint.sh"]

Here is a complete Dockerfile

    1. # Use the Liquibase official image as the base image
      FROM liquibase/liquibase:4.24.0

      # Tell Amazon CloudWatch we are in a container
      ENV RUN_IN_CONTAINER="True"
      ENV AWS_CONFIG_FILE=/home/liquibase/.aws/config
      ENV AWS_CREDENTIALS_FILE=/home/liquibase/.aws/credentials

      # Set your AWS access and secret key as build arguments
      ARG AWS_ACCESS_KEY_ID
      ARG AWS_SECRET_ACCESS_KEY
      ARG AWS_SESSION_TOKEN

      # Set the user to root to install the AWS CLI and CloudWatch agent
      USER root

      # Install CloudWatch agent
      RUN apt-get update && apt-get install -y python3-pip ca-certificates curl && pip3 install awscli && \
          curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb && \
          dpkg -i -E ./amazon-cloudwatch-agent.deb && \
          rm ./amazon-cloudwatch-agent.deb

      # Create the credentials file with build arguments
      RUN mkdir -p /home/liquibase/.aws/ && \
          echo "[AmazonCloudWatchAgent]\naws_access_key_id = $AWS_ACCESS_KEY_ID\naws_secret_access_key = $AWS_SECRET_ACCESS_KEY\naws_session_token = $AWS_SESSION_TOKEN"  /home/liquibase/.aws/credentials && \
          echo "[AmazonCloudWatchAgent]\nregion = us-east-1\noutput = json"  /home/liquibase/.aws/config && \
          chmod a+rw /home/liquibase/.aws/credentials /home/liquibase/.aws/config

      # Create a directory for the CloudWatch agent logs
      RUN mkdir -p /liquibase/liquibase-cw-logs && \
          chown -R liquibase:liquibase /liquibase/liquibase-cw-logs && \
          chmod -R a+rw /opt/aws/amazon-cloudwatch-agent

      # Configure CloudWatch agent
      COPY default_linux_config.json /opt/aws/amazon-cloudwatch-agent/bin/

      # Set the user back to liquibase
      USER liquibase

      COPY --chown=liquibase:liquibase entrypoint.sh /liquibase/

      # Start the CloudWatch agent
      ENTRYPOINT ["/liquibase/entrypoint.sh"]

Build the Docker image

  1. Prior to building the Docker image, setup IAM user credentials by exporting AWS environment variables: 
    1. export AWS_ACCESS_KEY_ID=""
      export AWS_SECRET_ACCESS_KEY=""
      export AWS_SESSION_TOKEN=""
  2. Build the Docker image using docker build command: 
    1. $ sudo docker build -t liquibase-w-cloudwatchagent:4.24.0 \
         --build-arg AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
         --build-arg AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
         --build-arg AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \
         .
  3. You now have a new Docker image created tagged as liquibase-w-cloudwatchagent:4.24.0. 

Configure Log Groups and Log Streams in AWS CloudWatch

  1. In your AWS Management Console, go to CloudWatch console. Create a Log Group and Log Stream as configured in the CloudWatch agent configuration file.  
    1. ...
              "log_group_name": "liquibaseLogs",
              "log_stream_name": "liquibaseLogStream2",
      ...

Setup permissions for the IAM user

Setup these roles for the IAM user:

arn:aws:iam::aws:policy/CloudWatchLogsReadOnlyAccess
arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

Run Docker image

  1. Ensure that all build endpoints have network access to send logs to CloudWatch
  2. Run the docker image, using a docker run command similar to this. In this example, we assume that liquibase.docker.properties file and changelog.xml file are located in the host's /opt/cloudwatch/resources directory which is mounted into the container using the -v argument. Also note that additional Liquibase arguments are provided such that logs are generated as structured logs (--log-format=JSON_PRETTY and --log-level=INFO).
    1. sudo docker run --rm \
        -v /opt/cloudwatch/resources:/liquibase/changelog \
        liquibase-w-cloudwatchagent:4.24.0 \
        --log-format=JSON_PRETTY \
        --log-level=INFO \
        --log-file=/liquibase/liquibase-cw-logs/mylogfile.json \
        --defaults-file=changelog/liquibase.docker.properties \
        status --verbose

You can now replace the existing docker image in use by application teams with the new docker image (liquibase-w-cloudwatchagent:4.24.0). Once this is published new deployments should pull this latest version of the image and should have their logs forwarded to the Observability tool.

Build your dashboards

Build your CloudWatch dashboard. Your dashboard widgets can measure items such as measure the number of unique databases, number of deployments, pipelines being deployed to within a given period, etc. This dashboard can be used to evaluate Liquibase adoption among users of the Docker image.

Related Articles:

Related to

Related articles

Comments

0 comments

Article is closed for comments.