****************** Bowtie Controllers ****************** A Bowtie *controller* is the server-side orchestrator that glues all the Bowtie :doc:`client ` machines together. The roles and responsibilities of a controller in your environment include: - Supporting administrative tasks by a privileged user to: - Update policies and their associated ACLs - Grant or revoke privileges for users that have joined the network - Managing the private network, which includes: - Distributing updated routes to other controllers and clients - Restricting or granting access to resources based upon defined ACL policies - Providing DNS for configured domains within the private network This page enumerates some of the nuances to bear in mind when operating the Controller appliance or guides outside the scope of :doc:`setup-controller`. Please refer to :doc:`operating/controller` for exhaustive documentation about all configuration options available for these appliances. Bootstrapping ************* A Controller’s initial startup logic proceeds as follows: - Run cloud-init - Check for the existence of ``/etc/hostname``. If present, bootstrap the appropriate settings for the Bowtie server software and supporting services like Grafana. - As part of this step, if ``/etc/ssl/controller.pem`` and ``/etc/ssl/controller-key.pem`` are present, the Controller will configure itself to serve TLS using using this certificate and key. - Start the Bowtie server daemon. The server daemon will ingest a few key files at start time: - If ``/var/lib/bowtie/init-users`` is present, create the first set of initial users based on this file’s contents. Read about :ref:`pre-seeding initial users ` for additional information. - If ``/var/lib/bowtie/should-join.conf`` is present, initialize clustering operations with the other Controller indicated in this file. Read about :ref:`joining other Controllers ` for additional information. - If ``/var/lib/bowtie/skip-gui-init`` is present, skip presenting :ref:`the initial setup start screen ` for new Controllers, which is useful for automation-driven provisioning. - Source environment variables for the daemon from ``/etc/default/bowtie-server`` and ``/etc/bowtie-server.d/*`` You may leverage this logic to perform initial configuration according to your taste: - Skip defining `cloud-init hostname settings `_ to avoid running the initial bootstrap process automatically. You can define this file later to invoke the same process. - Specifying all configuration options (hostname, certificates, and SSO files) in cloud-init user data permits for a declarative Controller setup without any manual steps required for installation. - Change log verbosity by dropping in environment variable settings into files in ``/etc/bowtie-server.d/*``. See :doc:`operating/controller` for more comprehensive server configuration documentation. .. todo:: I think we should permit for TLS termination at a reverse proxy but it would require configuring the Controller to listen over plain HTTP. Need to reconcile that. .. _backup-and-restore: Backup and Restore ****************** Bowtie Controllers support the ability to perform regular backups and also to restore from backups. These settings are defined via each Controller’s :ref:`backup strategy ` and controlled via the :ref:`backups` page. Backups are fundamentally :ref:`restic snapshots ` and may be used with basic ``restic`` commands if desired. Backups are primarily intended to facilitate data recovery in a disaster scenario - in order to achieve highly available or fault-tolerant deployments, you should preferentially architect your deployment to leverage :doc:`ha-controller` which will replicate your data across multiple hosts in the event that a Controller is irrevocably lost. Downtime can be avoided entirely in this scenario by running more than one Controller for clients to fail-over. We suggest using backup and restore for the following shape of use case: - To safeguard data in deployments that operate a single Controller - To backup historical application data in the event of configuration mistakes or accidental deletion - To retain old application database versions in the event of unexpected stability or software bugs and a full restore becomes necessary Note that Controllers cannot currently rely on backup snapshots to restore individual configuration items (like policies or users) at a fine-grained level -- backup operations can restore the entire application database, but not specific objects as represented over :ref:`the API `. This may change in future updates. .. _backup-settings: Backup Settings --------------- For some backup `repository `_ types, you will need to provide authentication credentials to successfully create and remove backup snapshots. For local backup paths, credentials are unnecessary. Similarly, if you launch a Controller in :ref:`aws` with an appropriate instance role with privileges granted to a backup S3 bucket, the underlying :ref:`restic ` libraries will authenticate without the need for static AWS credentials. For other cases that may require API keys or authentication tokens, two options are available: - Use the :ref:`secrets management page ` of the Control Plane to manage secret values with a web interface. - Alternatively, you may populate the file at ``/etc/default/backupd`` with key/value pairs that set your desired credentials. Use `the documentation for supported restic environment variables `_ to set the desired credentials for your repository type. For example, to set credentials for use with a Minio deployment located at the backup repository endpoint ``s3:https://minio.example.com/backups/bowtie/bt1``, populate the file at ``/etc/default/backupd`` with the following: .. code:: ini AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= Changes to this file will prompt the Controller’s backup daemon to restart and load these environment variables as necessary. The :ref:`secrets management page ` will operate on this same file. .. _backup-encryption: Backup Encryption ----------------- Controller backups are encrypted. Before using backup and restore, you must define a backup encryption/decryption key that will be used when either performing backups or restoring them (we refer to this value as a "key" in this documentation, but any sequence of random characters constitutes a valid key, which might also be labeled a password). This key should be an ASCII string. A long sequence of random characters generated from a password manager is a good example of a strong encryption key or the output of a cryptographically-strong command like ``openssl rand -hex 32``. Store this value in a secure location apart from the Controller in case of a disaster recovery scenario in the event that the Controller's filesystem is unreadable. To provide this encryption key to the Controller for backup operations, use the :ref:`secrets management page ` to load this ``ENCRYPTION_KEY`` variable onto the Controller. Alternatively, you may define it at the file path ``/etc/default/backupd`` in the following form: .. code:: ini # Replace this value with your real encryption key. ENCRYPTION_KEY=5352891d395f59cb64d72ea07270f8da Daemons that honor this configuration key on the Controller will restart as necessary to consume its value when it is defined or changed. .. _backup-restore: Backup Restore -------------- Two primary interfaces exist for restoring from backup: interactive restore and automated restore. .. _interactive-restore: Interactive Restore ~~~~~~~~~~~~~~~~~~~ The ``restore`` command is available from the command line to retrieve backup snapshots, move them into the appropriate filesystem paths, and start or stop the relevant system services in the correct sequence to facilitate successful restore. You may simply invoke the ``restore`` command which will prompt you for any missing values required to operate, like the desired remote repository URL and backup encryption key. Alternatively, consult ``restore --help`` for a list of accepted parameters to run the command with predetermined values, including additional actions such as the ability to list available backup snapshots and optionally restore specific backup snapshot IDs. .. _automated-restore: Automated Restore ~~~~~~~~~~~~~~~~~ Controllers will consume and honor values found in the file ``/etc/restore`` at boot and feed these values into the ``restore`` command outlined in :ref:`interactive-restore` if you would like to instruct a Controller to restore from a previous backup at startup. To use this feature, write out the file contents at ``/etc/restore`` with the following variables set: - ``REPOSITORY`` - Use the `restic-compatible repository endpoint `_ used when configuring the backup repository endpoint. - ``ENCRYPTION_KEY`` - Use the encryption key you configured when setting up the backup repository. In an automated Controller provisioning scenario, these files may be written with a mechanism like :ref:`seeding`. The automated restoration process will pause until ``cloud-init`` has finished in order to ensure the file has been written. After a successful restore, the automated process will *remove* the file at ``/etc/restore`` to avoid potentially overwriting the running Controller's database in the future if the Controller is rebooted. .. _backup-manual: Manual Backup Interaction ------------------------- Underneath the high-level tooling that Bowtie provides for automatic backups, `restic `_ is the fundamental tool that manages backup snapshots. Because backup snapshots are ultimately ``restic`` snapshots, you are free to interact with these backup snapshots as any other ``restic`` repository without negatively impacting the backup or pruning operations that Controllers may run. For example, assume that you have configured :ref:`Controller backups ` for storage in S3 at ``s3:s3.amazonaws.com/example/bowtie/controller/bt1`` using the backup encryption key ``correct-horse-battery-staple``. With ``restic`` installed, you may view existing snapshots with the command: .. code:: sh RESTIC_PASSWORD=correct-horse-battery-staple \ RESTIC_REPOSITORY=s3:s3.amazonaws.com/example/bowtie/controller/bt1 \ restic snapshots While this is a supported method to view, delete, and manage snapshots, we still recommend using the approaches outlined in :ref:`backup-restore` to restore a Controller from backup snapshots because the ``restore`` tool includes additional logic to move restored files into the correct paths and start/stop services in the correct sequence when restoring. This technique is also a viable approach if you have a need to retrieve your application data for uses such as manual recovery of specific pieces of data in tandem with a Bowtie representative. Self-Signed Certificate Bundles ******************************* Some environments may require trusting a self-signed certificate if HTTPS endpoints are intercepted. Follow the installation guide's :ref:`section about self-signed certificates to install one `. .. _controller-security: Controller Security ******************* As an edge device on your network, Bowtie takes the security of our software seriously. While most security features are built into the products we ship and configurable at the application layer, a few specific features are worth calling out specifically: Vulnerability Data ------------------ Controller artifacts under regular scanning as part of our release process. Although third-party tools like `Wiz `_ can scan Controller systems, our experience has shown that, with our detailed knowledge of Controller internals, Bowtie is better-suited to provider scanning results that we can triage to filter our false positives or noise. Refer to :ref:`vulnerability-data` for additional information about retrieving scan data about your specific Controller version and format. Audit Trails ------------ Controller server logs include the output of a variety of server daemons, but logs for the ``bowtie-server.service`` ``systemd`` service include logs with the annotation ``audit_event=true`` that can be targeted to record key events like user authentication or address assignment. Refer to :ref:`application-logs` for additional information. Updates ------- Bowtie strongly recommends opting-in to :ref:`unattended-updates` whenever possible to stay ahead of new vulnerabilities and the latest stability fixes. Generally speaking, Controllers can move upwards in version without breaking backwards compatibility with Client packages. .. _terraform: Terraform ********* Deploying Controllers via Terraform is supported. :ref:`cloud-init ` is included on all images to permit configuration via user-data at instance creation time as well. Paired with native cloud images for :ref:`aws`, :ref:`gce`, and :ref:`azure`, you can fully automate the installation of Bowtie Controllers. Examples are provided for both :ref:`terraform-aws` and :ref:`terraform-azure`. .. _terraform-aws: Deploying to AWS ---------------- When writing Terraform code to deploy and manage Controllers, you may want to leverage the ``aws_ami`` ``data`` resource in order to acquire the appropriate Bowtie Controller AMI ID. For example, the following code snippet demonstrates how to acquire a Controller AMI ID and create an AWS instance: .. literalinclude:: /examples/terraform-aws.tf :language: terraform :linenos: The referenced ``user-data.yaml.tpl`` would contain the content noted in :ref:`terraform-user-data`. Terraform resources may look slightly different for the *first* Controller versus all subsequent Controllers as laid out by :doc:`ha-controller`. Notably, the ``/var/lib/bowtie/should-join.conf`` file will instruct a new Controller to peer with an existing node and ``SITE_ID`` should be consistent within a site (such as a public cloud region or on-premise datacenter). .. _terraform-azure: Deploying to Azure ------------------ Deploying a Controller into Azure is similar to :ref:`the steps for AWS `: establish a new instance based on the latest Bowtie Controller ID, define the relevant cloud-init data, and load it as user-data for the new instance. .. literalinclude:: /examples/terraform-azure.tf :language: terraform :linenos: See :ref:`terraform-user-data` for example contents of the referenced ``cloud-init.yaml.tpl``. .. _terraform-user-data: Terraform cloud-init User Data ------------------------------ When creating cloud instances with Terraform, :ref:`seeding` provides a way to bootstrap configuration options for new Controllers. Use the following boilerplate as reference to configure the relevant portions of your deployment: .. literalinclude:: /examples/terraform-aws-cloud-init.yaml :language: yaml :linenos: .. _terraform-provider: Bowtie Terraform Provider ------------------------- To configure the Bowtie application itself beyond just the network Controller appliance, you may optionally use the `Bowtie Terraform Provider `_ to declaratively manage resources over the Bowtie API natively in Terraform. This is a powerful approach when paired with :ref:`building Controllers with Terraform ` to achieve declaratively managed infrastructure. The `Bowtie provider page `_ provides the latest documentation for supported resources and example code. An example is included here for convenience to demonstrate how to use the provider: .. literalinclude:: /examples/terraform-provider.tf :language: terraform :linenos: Additional items to remember when using the native Terraform Bowtie provider: - The provider is under active development. If you encounter APIs that may not have equivalent Terraform resources, let us know! `The provider is an open source project `_. - Many resources support the ``terraform import`` ability to populate Terraform state with existing resources already present over the control plane web API. Remember to import these resources if you would like to avoid creating duplicate resources. Provider ``apply`` Ordering ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Many teams may enforce infrastructure-as-code as a top-level Terraform file that declares defined state across many domains including providers like EC2, Route53, and more. While Bowtie resources can fit into this model, there are a few caveats to remember when Terraform refreshes state to generate a final plan for the ``apply`` step. The ``provider "bowtie" { }`` block must be able to successfully authenticate against a Controller endpoint when ``terraform apply`` attempts to manage resources. If you intend to deploy a new Controller in the same Terraform run in which you declare a Bowtie provider and Bowtie resources, you may need to amend your ``.hcl`` code to enforce ordering so that resources are called in the correct sequence. For example, consider a Terraform deployment in which an AWS instance with a Route 53 record serves as the Bowtie API endpoint. To delay calling the API endpoint until it has become available, you might consider code like the following, which injects a null ``terraform_data`` resource that blocks until the indicated API endpoint is reachable: .. code:: terraform # Create a `terraform_data` resource that does nothing except wait for # the API endpoint to become responsive. resource "terraform_data" "bowtie_api_wait" { # Loop until a curl command against the endpoint stored as # local.bowtie_endpoint responds. provisioner "local-exec" { command = <`_, you can ``apply`` only a subset of your Terraform resources to avoid instantiating providers that may not yet have ready endpoints. For example, given this example ``main.tf`` file that declares a module to manage AWS resources as well as a Bowtie deployment: .. code:: terraform module "us_west_2" { source = "/path/to/custom/module/aws" # You likely have additional module options here } module "our_bowtie_org" { source = "/path/to/custom/module/bowtie" # You likely have additional module options here } You can build the requisite Controller hosts first by providing the module name to ``terraform apply`` in the following form: .. code:: shell $ terraform apply -target=module.us_west_2 After your ``apply`` step has created any Controllers that can serve the Bowtie API, you can then either apply the Bowtie resources which should now succeed: .. code:: shell $ terraform apply -target=module.our_bowtie_org Alternatively, assuming that the Bowtie API is available, applying the entire Terraform state should also succeed: .. code:: shell $ terraform apply Caddy ***** Bowtie Controllers front all inbound traffic with `Caddy `_, a modern reverse proxy with secure defaults and automatic SSL/TLS management. In some rare cases, the reverse proxy configuration may need to be reset to restore known-working defaults. To do this, you can run the following script from the command line over an ``ssh`` session on the impacted Controller which recovers the stock Caddy configuration and sets the running configuration to its contents: .. code:: sh curl "http://127.0.0.1:2019/load" -H "Content-Type: text/caddyfile" --data-binary @/etc/caddy/caddy_config Confirm that the reverse proxy is functioning correctly by browsing to your Controller's HTTP(S) endpoint or viewing the running configuration with ``curl http://127.0.0.1:2019/config/ | jaq``. .. _helm: Helm **** The :ref:`Kubernetes installation documentation ` provides comprehensive information and examples regarding how to deploy the server-side component of Bowtie to your Kubernetes environment. The :ref:`kubernetes-troubleshooting` section of :doc:`troubleshooting` offers additional guidance. Collecting Observability Data ***************************** Reference :ref:`exporting-telemetry` for examples of defining additional `OpenTelemetry Collector `_ `exporters `_. The `opentelemetry-collector-contrib `_ package permits a wide range of exporter targets, so you do not necessarily need to run an identical observability stack on your own infrastructure (Prometheus for metrics, Loki for logs, and Tempo for traces). For example, you may define an `Elasticsearch exporter `_ and then append it to the Controller’s log pipeline to receive and store logs using your own Elasticsearch cluster in tandem to the Controller’s local Loki installation. This strategy is equally valid for metrics and traces as well. Note that Prometheus often operates on a pull-based `scraping `_ scheme and so each Controller makes ``:9090`` available that provides all metrics from this single endpoint. You may elect to implement `federation `_ in order to implement hierarchical scaling and receive all discrete metrics that are visible from the local :ref:`controller-grafana` installation. .. _audit-logging: Audit Logging ------------- For operators who need insight into auditing events such as user authentication or IP assignment activity, refer to :ref:`controller-logs` and specifically the section about audit logs under :ref:`application-logs`. .. _policy-verdict-tracking: Policy Verdict Tracking ----------------------- Every Controller :ref:`provides settings ` to optionally report on per-packet policy decision verdicts that occur when inbound traffic flows over the network control plane. This may take the form of metrics or logs (which is configurable) and operators may choose to view the resulting data locally by leveraging the Controller-local :ref:`controller-observability` stack or forwarding this data off-Controller by :ref:`exporting-telemetry`. In the case of policy verdict tracking, there are additional considerations to bear in mind: - Because the policy engine is consulted for all network traffic, the volume of metrics and logs **may be significant**. You should be prepared to handle log streams with significant amounts of output volume and metrics with a nontrivial number of active time series reported at any given time. - In general, endpoint device activity can be measured and queried based upon either kind of data -- metrics include metadata like device or user ID as part of their labels and logs may be counted in aggregate to estimate volume. Given operator flexibility, Bowtie suggests relying on **metrics** within environments that have established auditing requirements because metrics incur less processing overhead for Controllers, generate less overall volume of data, and still achieve similar levels of granularity given sufficiently sophisticated queries. Anecdotal performance benchmark measurements taken with policy verdict tracking enabled under pathological traffic patterns (for example, `loaded with `_ ``iperf3`` traffic) show that reporting metrics yields almost no negative performance impact, while reporting logs can noticeably reduce overall throughput. Whether metrics or log verdict tracking are enabled, either may be used from the local observability stack or exported for centralized aggregation: - For local consumption, use the :ref:`controller-grafana` installation to view *metrics* (exposed via ``prometheus`` or *logs* (exposed via ``loki``). - To forward these signals to a receiving endpoint, we strongly suggest relying on strategies similar to those outlined in :ref:`exporting-telemetry`. This will usually take the form of tapping into a ``receiver`` and constructing a ``pipeline`` that forwards along a ``metrics`` or ``logs`` stream to a designated ``exporter`` suitable for your environment. Because ``bowtie-server.service`` forwards logs and metrics via OTLP into the local ``opentelemetry-collector.service`` daemon, you may freely choose an `exporter `_ to collect these signals. Verdict Collection Examples ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following YAML document, when placed at ``/etc/otel.yaml``, will aggregate all logs from the Controller and deliver them to Sumologic. .. code:: yaml exporters: sumologic: endpoint: https://my-endpoint service: pipelines: logs: exporters: - sumologic Similarly, this configuration file will deliver all Controller metrics to Datadog: .. code:: yaml exporters: datadog: api: key: ${env:DATADOG_KEY} service: pipelines: metrics: exporters: - datadog .. _nix: Nix and NixOS ************* Controller appliances are based on the `NixOS `_ Linux distribution. If you find yourself interacting with Bowtie Controllers in a command line shell, there are some key differences to keep in mind when performing tasks like package installation or configuration management. .. admonition:: Message of the Day (``motd``) :class: hint Many of these suggestions are included in the `login shell message `_ for ease of discovery and are repeated here for sake of completeness. .. _package-management: Package Management ------------------ A variety of packages are available for ad-hoc installation as with a distribution like Debian or Red Hat, albeit with a different package manager. You may find package names by: #. Entering a missing command name and running it from the shell. A shell hook will inform you about whether the indicated command is available in the named nix package. #. Searching for specific file contents among all known packages with ``nix-locate``. For example, to find which package provides the executable ``ethtool``: .. code:: sh nix-locate --whole-name --at-root /bin/ethtool .. admonition:: Using ``nix-locate`` :class: hint Consult ``nix-locate --help`` for additional flags. #. Searching for package names explicitly. You can either search the ``nixpkgs`` repository directly: .. code:: sh nix search nixpkgs ethtool Or instead search the local database. This method may be slow. .. code:: sh nix-env -qaP ethtool Once you've found the desired package name, install it: - Using ``nix profile`` will install the package persistently so it is available across upgrades, logins, and cache rotations: .. code:: sh nix profile install nixpkgs#ethtool - Alternatively, you may enter an ephemeral shell with the desired program in-``$PATH``. The installed package will be cleaned up later on when the system undergoes regular cache clearing. .. code:: sh nix shell nixpkgs#ethtool Sandbox ------- A NixOS system does not include operating system-wide shared libraries in ``/usr`` or ``/lib``. If you find yourself in need of a remote executable that is not available as :ref:`package `, you will need to stub out a sandbox environment so that dynamically-linked executables can run. The ``sandbox`` command is provided for this purpose: .. code:: sh $ sandbox $ The shell will enter an ephemeral environment populated with functional, sandboxed ``/lib`` and ``/usr`` directories. You may download and run any executables or utilities and then exit the sandbox normally via ``exit`` or ``Ctrl-d``.