Strategies for using Flow Files

Applies to:

• Liquibase Pro
• Liquibase Open Source (Community)

Summary:

Liquibase flow files capability enables users to standardize their use of Liquibase across the organization. As such, flow files can be viewed as automation code that can be placed in a separate repository alongside other automation templates.

When Liquibase Pro is enabled in a pipeline that can perform many different operations, we need to develop a specific pattern for flow files. This pattern is designed for DevOps engineers' and end-users' ease of maintenance and use. 

Environment Conditions:

The flow feature requires Liquibase 4.17.0+. 

The flow feature also requires a Liquibase Pro license key.

Overview of Flow Files:

The liquibase flow command executes portable, platform-independent Liquibase workflows that can run anywhere without modification. Flow allows you to put multiple commands in a single YAML file, called a Flow File, so they can all run sequentially via a single liquibase flow command run in automation or manually. Flow files can contain Liquibase commands, shell commands, “include” child flow files, global and local variables with property substitution, Environment Variables, and more. (flow ).

The use of flow files in automation enables standardized and consistent use of Liquibase commands. 

A common scenario is to always perform a handful of other operations when running liquibase update operation. Therefore, the update operation (#3 below) can be sandwiched between other operations inside a flow file, as follows:

1. liquibase status

2. liquibase checks run

3. liquibase update

4. liquibase history (endStage)

Here is what a simple flow file would look like:

stages:
  Status Database:
    actions:
      - type: liquibase
        command: status
        cmdArgs: { status: "verbose" }

  Run Policy Checks:
    actions:
      - type: liquibase
        command: checks run
        cmdArgs: { scope: "changelog" }

  Update Database:
    actions:
      - type: liquibase
        command: update

endStage:
  actions:
    - type: liquibase
      command: history

Flow files should be treated as automation code, similar to YAML workflow files in GitHub or pipeline YAML files in Azure DevOps. The best practice would be to place flow files in a common repository, e.g., a template repository, or remote folders such as AWS S3, where Liquibase can look up using --search-path global argument or LIQUIBASE_SEARCH_PATH environment variable or liquibase.searchPath in liquibase.properties file. (search-path )

Simple Implementation using Flow Files

A simple implementation can be implemented using one or two flow files. One flow file for the update operation and another flow file for the rollback operation.

For example, a customer wishes to implement a database pipeline in GitLab. The default path would be to deploy to dev, test, and prod databases. However, they require an option to rollback each deployment. Here is what a Gitlab pipeline may look like:

In order to drive this pipeline, a number of environment variables can be established:

Limitations of the Simple Implementation

There are significant limitations to this approach:

1. There is only one type of rollback operation that can be implemented. Liquibase provides a variety of rollback operations for different scenarios (Liquibase Commands ). In simple implementations, teams will be allowed to use only one rollback command implemented in their pipeline.

2. There are no maintenance jobs or database comparison jobs available via this simple pipeline. End-users may want to perform a variety of operations on their database rather than just update or rollback. A simple pipeline may not be able to achieve their needs.

As such, there exists a need for a database pipeline to enable a broad set of use cases and also leverage Liquibase flow files.

A Better Implementation Using Flow Files

Let’s take a look at a solution using multiple flow files where there is a root flow file and child flow files. 

This approach requires using environment variables to drive Liquibase operations. The user only needs to export environment variables for commands he/she needs to run. They do not have to set environment variables for all commands.

# Setting env vars
export LBCOMMAND="update"

# Setting Liquibase command args
export LIQUIBASE_COMMAND_CONTEXT_FILTER="myschema"
export LIQUIBASE_COMMAND_CHECKS_SCOPE="changelog,database"

# Setting Liquibase global args
export LIQUIBASE_LIQUIBASE_SCHEMA_NAME="LIQUIBASE_USER"
export LIQUIBASE_LOG_LEVEL="fine"

├── liquibase.flowfile.yaml
├── liquibase.flowfile-checksrun.yaml
├── liquibase.flowfile-history.yaml
├── liquibase.flowfile-rollback.yaml
├── liquibase.flowfile-rollbackOneChangeset.yaml
├── liquibase.flowfile-rollbackOneUpdate.yaml
├── liquibase.flowfile-status.yaml
├── liquibase.flowfile-update.yaml
├── liquibase.flowfile-updateTestingRollback.yaml

########## LIQUIBASE FLOWFILE ##########
########## learn more http://docs.liquibase.com/flow ##########

## Run this file by executing "liquibase flow" or "liquibase flow --flow-file=liquibase.flowfile.yaml"

## Note: Any command which fails in any stage below result in the command stopping, and endStage being run.
## A flow file can have one or more stages, each with multiple "actions",
## or your flow file can have multiple stages with fewer actions in each stage.

stages:
  Status Database:
    actions:
      - type: liquibase
        command: status
      cmdArgs: { status: "verbose" }

  Run Policy Checks:
    actions:
      - type: liquibase
        command: checks run
        cmdArgs: { checks-scope: "changelog,database" }

##############################################################################
######## Invoke child flow file based on $LBCOMMAND environment variable #####
##############################################################################
  DB_Requested_Operation:
    actions:
      - type: liquibase
        command: flow
        cmdArgs: { flow-file: "liquibase.flowfile-${LBCOMMAND}.yaml" }
##############################################################################

## The endStage ALWAYS RUNS.
endStage:
  actions:
    - type: liquibase
      command: history

## A flow file can have one or more stages, each with multiple "actions",
## or your flow file can have multiple stages with fewer actions in each stage.

# Optional "LIQUIBASE_COMMAND_CONTEXT_FILTER" set to ""
# Optional "LIQUIBASE_COMMAND_LABELS" set to ""

stages:
  DB update:
    actions:
      - type: liquibase
        command: update       # This command will vary