Multi-Controller Setup

Distributed Operation

The Bowtie multi-Controller syncing mechanism is undergoing rapid development and so this section is subject to change between releases. Please reach out to Bowtie staff if you encounter issues configuring and operating multi-Controller deployments.

Installing Bowtie Controllers in a fault-tolerant, highly-available cluster is the recommended configuration that is well-supported by the Bowtie server software.

Each controller can act independently during partitions or partial outages and will synchronize and coordinate seamlessly during normal operations. No notion of a ‘primary’ controller exists.

Backups

To persist your data off live Controllers and retain historical backup snapshots for use in disaster recover scenarios, Backup and Restore is available.

Fail-over Behavior

A Bowtie deployment tolerates the loss of individual Controllers without operator intervention. Two mechanisms work together to keep Clients connected and policy enforced when a Controller becomes unavailable.

Every Controller is equal. Each Controller holds a complete, continuously synchronized copy of your organization’s configuration, policy, and identity state, and answers Client requests from that local copy. No primary or backup Controller exists, and no shared database sits behind the cluster that could fail on its own, so any Controller can fully serve any Client. If Controllers cannot reach one another during a network partition, each keeps operating from its own copy and synchronizes with the others again once connectivity returns.

Every Client connects to every Controller. Rather than connecting to one active Controller and holding the rest in reserve, a Client maintains a live tunnel to every Controller in its deployment at the same time. It continuously measures whether each Controller is reachable and how quickly it responds, then sends traffic through the Controller that responds fastest. A Client prefers to stay with its current Controller unless another is meaningfully faster, which avoids needless switching when several Controllers perform similarly.

When a Controller stops responding, the Client marks that tunnel unhealthy and moves traffic to the fastest remaining healthy Controller automatically. Because the tunnels to the other Controllers are already established, fail-over is fast and does not drop the Client from the network: traffic simply moves to a Controller the Client already reaches, with no new tunnel handshake and no operator action. With default settings, this transition completes within roughly two minutes of a Controller becoming unreachable; see Configuring Fail-over and Health Checks to make detection faster or more tolerant.

Geographically distributed deployments

Because a Client selects the closest responsive Controller for each site, you can place Controllers in different regions or physical locations to survive the loss of an entire site. Assign Controllers to sites so that Clients fail over to the nearest healthy Controller during a regional outage.

Configuring Fail-over and Health Checks

Fail-over works without any configuration, but several settings let you tune how Clients and Controllers detect and react to an unavailable Controller.

Client health checks. Two Bowtie Client options govern how quickly a Client decides that a Controller is unreachable and moves on:

• persistent_keepalive sets the interval, in seconds, between keepalive packets sent over each tunnel (default 25).

• unhealthy_tunnel_health_count sets how many consecutive health checks a tunnel must fail before the Client marks it unhealthy (default 5).

A Client treats a Controller as unreachable after approximately persistent_keepalive multiplied by unhealthy_tunnel_health_count seconds, or about two minutes with the default values. Lower these values for faster fail-over at the cost of greater sensitivity to brief network interruptions; raise them to tolerate transient loss. Set either option in the Client configuration file or on the command line as described in Advanced Configuration, and run bowtie-service --help for the complete list of options.

Controller partition recovery. Controllers continuously synchronize organizational operating data with each other. When a Controller detects that it has lost its sync connection to another Controller, it attempts to reconnect to that Controller while continuing to sync with Controllers that it remains connected to. Reconnection completes within roughly three minutes of the end of the network partition. This behavior is fully automatic and always active.

To install additional Controllers after the first you will prepare your first controller then follow the steps in Controller Setup for each Controller with the following additional steps.

General Clustering Guidance

For a new Controller to join an existing deployment, all members of the cluster must rely on a pre-shared key (PSK) for use in sync operations. This key should be kept secret and is used to authenticate sync operations between Controllers. You can generate a key with the following command in Linux:

uuidgen

PSK Generation

The uuidgen command-line application is bundled on all Controller appliances for convenience.

If you previously set up a Controller without explicitly generating a PSK, then one was generated on your behalf and can be retrieved from /etc/default/bowtie-server.

Two methods are available to expand your Bowtie Controller deployments:

• Unattended cluster expansion with pre-seeded configuration files. This method is well-suited for operators who prefer to manage infrastructure in an automated fashion with tools like cloud-init. This approach is also suitable for configuration over ssh.

• Guided cluster expansion via web-based Controller setup. Controllers created without any pre-seeded information offer two guided tracks for initial configuration: either as initial hosts for a new Bowtie deployment or as new members of an existing deployment.

Automated Cluster Expansion

Many administrators will prefer to use cloud-init or Terraform for these steps but they can also be performed manually with SSH or other console access and a text editor of your choice.

On your first controller, add or edit a file in /etc/bowtie-server.d/ to set the following values, adding them if they do not already exist:

BOWTIE_SYNC_PSK=<your sync PSK>

These values will also need to be set on every additional controller that you would like to join to the cluster. Every new controller should consider setting the SITE_ID value to the same ID as any other Controllers within the same site as well.

Proceed to configure and launch new Controllers after the first following any relevant guidelines in Controller Setup while including the aforementioned variables for syncing strategy and pre-shared key. Additionally, a temporary file should be created at /var/lib/bowtie on the new controller to point to an existing controller in the cluster. This file should be named should-join.conf and should contain the following:

entrypoint = "https://your-first-controller.example.com"

Below is an example demonstrating how to place these files using cloud-init:

#cloud-config
# ...other cloud-init parameters...

write_files:
- path: /var/lib/bowtie/should-join.conf
  content: |
      entrypoint = "https://your-first-controller.example.com"
- path: /etc/bowtie-server.d/custom.conf
  content: |
      SITE_ID=<your site ID>
      BOWTIE_SYNC_PSK=<your sync PSK>

Site ID

Multiple controllers in the same network segment that are used for HA purposes should have the same SITE_ID.

Controllers in different network segments should have different values for SITE_ID. You can use any UUID4 value for the SITE_ID.

Remember to follow Seeding Configuration if you would like to achieve a purely declarative start-up sequence, which will initialize the new Controller without any necessary manual intervention when joining your Bowtie cluster by automating endpoint configuration, TLS, and single sign-on.

During the bootstrapping process, the new Controller will connect to the existing controllers to discover the existing application state and other peers.

The new controller and existing controller must have matching BOWTIE_SYNC_PSK values and the new controller must be accepted into the control plane in the Administrative GUI under the /devices page or via the device acceptance API endpoint.

After the new controller is accepted, it will restart in a running state and can begin accepting traffic.

Guided Cluster Expansion

All requirements common to Controller Setup still apply – such as network configuration, machine resource allocation, and port requirements – but guided cluster expansion allows operators to provision Bowtie Controllers into existing environments without cloud-init intervention or post-creation steps over ssh. Initial configuration steps occur over a secured first-run web configuration wizard.

Proceed with the steps outlined in Controller Setup normally but without any cloud-init content. At the Configuration step, rather than selecting the “New Setup” tab, instead choose the “Join Existing” tab to proceed to join to an already-configured Controller.

Follow the guided setup to proceed. During the course of the configuration, you will configure a hostname, TLS, pre-shared key (PSK), existing Bowtie Controller endpoint, network configuration, and site configuration.

• Recall that the pre-shared key (PSK) should be common among all Controllers.

• Depending on your deployment topology, new Controllers in existing deployments may setup new sites or join existing ones. Choose whichever option is appropriate for your network.

At the conclusion of the setup steps, your Controller will redirect you to an initial login page. Because clustered Controllers share a common database, you may log in with an existing set of user credentials.

Debugging Clustering

The Bowtie server-side software runs under the bowtie-server service and can be queried as any other systemd-based Linux service. Consult its most recent logs with:

journalctl --unit bowtie-server --pager-end

Check the service’s status with:

systemctl status bowtie-server

Some additional sources of service configuration are listed below in case additional debugging steps are necessary:

• /etc/default/bowtie-server

  • Used to inject environment variables into the bowtie-server service. Primarily for internal use to set default values not provided by end users.

• /etc/bowtie-server.d/*

  • Files in this directory will be read to set environment variables as defined in man 5 systemd.exec. Use this directory to set environment variables on a per-file basis for options you may want to override.

• Use systemctl cat bowtie-server to inspect the running configuration for the bowtie-server.service unit.