*************** Troubleshooting *************** .. spelling:word-list:: Wi Fi We did our best, and you still ended up here. Alas! We're here to help. .. _troubleshooting-controller: Bowtie Controller ================= For managed support, we suggest relying on either :ref:`controller-chat` or :ref:`cp-support-sos` for assistance from Bowtie representatives. You may use :ref:`controller-sos` as a fallback if the Controller web frontend is inoperative. For self-directed support diagnosis, reference the sections that follow (for :ref:`tls-troubleshooting`, :ref:`controller-monitoring`, and :ref:`kubernetes-troubleshooting`). .. _controller-chat: Chat Support ------------ Chat-based support is available for users with **Enterprise**-level support. If your control plane’s :ref:`controller-licensing` page indicates a support tier other than "Enterprise", reach out to a Bowtie representative if you would like to add Enterprise support to your license. See :ref:`enable-chat-support` for how to use this feature. Chat-based support interactions are facilitated by `Intercom `_ and contingent upon: .. _chat-requirements: - Your deployment granted **Enterprise** level support - At least one :ref:`support domain ` defined and validated by Bowtie - The initiating user’s control plane email address matching one of your trusted :ref:`support domains ` If all of these criteria are met, you may initiate a chat support session with a member of Bowtie staff after :ref:`enable-chat-support` by either: .. _chat-support-methods: - Navigating to the dedicated :ref:`controller-chat-page` page to begin a new session, or - Use the :ref:`control plane support ` option to load the support widget on all control plane pages In either case, an in-browser widget will appear on the lower-right of the browser window which you may click on to expand and use to start a new chat with a member of Bowtie support or see existing cases. .. _enable-chat-support: --------------------- Enabling Chat Support --------------------- To successfully enable :ref:`controller-chat`, follow these steps: - Choose one more or domains in your organization that you trust any control plane :ref:`org-users` with configured email addresses to initiate chat support sessions with Bowtie. This mandatory step ensures that only your deployment has the ability to open new and existing support cases under the indicated domain names. Enter these domains under :ref:`your support domains `. After clicking *Save*, they will be sent to Bowtie for approval. - Consult with your Bowtie representative about your granted *support tier* and claimed :ref:`support domains `. Bowtie must enable **Enterprise** level support for your license and confirm that you have ownership of any configured :ref:`support domains ` and make the changes to your license before you can retrieve a trusted license with these required fields. - After your license has been granted **Enterprise**-level support and any support domains trusted against your deployment, the next automatic or manual license synchronization (through the :ref:`controller-licensing` page) will fetch and store your updated license. Any users with email domains listed in trusted :ref:`support domains ` may :ref:`initiate a new chat session `. .. _controller-sos: Controller SoS -------------- Controller SoS provides the capability to instruct a Controller to aggregate a diagnostics bundle and deliver it to Bowtie support for assisted troubleshooting. The methods described in this section provide a route into diagnostic capabilities without any assumption that the :ref:`control-plane` is functional to ensure that support diagnostics can be generated independently. If the :ref:`control-plane` is non-functional, you should proceed to follow this section for assisted support. To get started quickly, follow the steps for your situation - collecting diagnostic bundles from the command line skips Controller ownership validation, so proceed with the first bullet point if you have console access to the host. - Run the ``sos`` command. - If you do not have console access to the Controller or are otherwise unable to run the ``sos`` command, browse to ``/sos`` under the IP address or domain that your Controller is hosted under. If a browser is unavailable, you may issue a request to your Controller’s HTTPS endpoint under the ``/sos`` path with a tool like ``curl``. If your Controller’s HTTPS endpoint is non-functional, you may alternatively use port ``911`` which bypasses the proxy layer. The responses from this endpoint will explain how to proceed. At some point following either of these two steps, a representative from Bowtie will need to trust your upload request, so you should be in communication with a Bowtie representative in preparation for that step. Read on if you would like to change parameters like valid collection windows or bundle expiration - by default, after validation, bundle generation is permitted once within the next 24 hours before requiring re-validation and will be auto-deleted by Bowtie in 30 days. The remainder of this section explains SoS functionality in greater detail for additional customization options and capabilities. :ref:`Authorization flow ` explains how the permissions model operates, and :ref:`controller-sos-api` explains the API and available options. .. include:: partials/serial-console.rst .. _sos-authorization: ----------------- SoS Authorization ----------------- If you submit SoS bundles via the ``sos`` command on the command line, the ability to execute this command grants the implicit ability to submit diagnostic support bundles. This section explains the requirement for additional validation steps when generating support bundles via the API-based method. Delivering Controller information to an external party like Bowtie is gated behind authorization steps in order to: - Avoid unintentionally disclosing organization information - Ensure that only operators may initiate and send diagnostics bundles The typical Controller web-based :ref:`control-plane` user system does not come into play here so as to avoid any reliance on authorization systems that may themselves be malfunctioning. In order to validate that an SoS request is legitimate, the Controller will emit a time-limited nonce on the Controller’s serial console output. After an operator retrieves this value from the Controller’s runtime platform (such as the EC2 serial console) and validates the ability to access the Controller’s serial console, support bundles may be generated a limited number of times for a period of time (these limitations are configurable and may be optionally bypassed by an optional token generated at validation time). Limits are set by default to avoid persistent availability of the ``/sos`` endpoint to generate support bundles indefinitely. Serial console read access is an imperfect metric to gauge privileged access by, but in the event that remote access is unavailable or unreliable, it provides a simple method to confirm administrative access to the underlying Controller infrastructure. .. _controller-sos-api: ------------------ Controller SoS API ------------------ The SoS API (either via ``/sos`` or requests issued to port ``911``) consists of the following endpoints. Note that the ``/sos`` path :ref:`requires authentication `. - ``/`` - ``GET`` - Will return the next steps to proceed with SoS bundle collection whether initiated from the command line or a browser. The root path accepts the following headers if they are given: - ``Authorization: Bearer ``: If set and the token is valid, the Controller will defer any time or use limits to generate a support bundle without re-validation. - The following ``GET`` parameters are accepted: - ``expiration``: Accepts an `ISO 8601 `_ formatted date/time string to control when support bundles will be deleted by Bowtie. By default, this value is 30 days. For example, to request that a support bundle undergo deletion one week from time of submission, you may generate a timestamp using the command: ``date --utc -Is --date '1 week'`` and URL-encode the resulting value to set with ``?expiration=``. - ``test``: When set to ``1``, will perform a test bundle collection without sending data to Bowtie. - ``stream``: When set to ``true`` will initiate bundle collection and stream responses back. - ``email``: When set, the support bundle will associate the given user email to help Bowtie support staff engage with the submitter. This field is optional. - ``note``: When set to a base64-encoded string, the submission endpoint will store this field as a free-form description of the reason the support bundle was submitted. This field is optional. - ``POST`` - Used to initiate support bundle collection from the command line with a command like ``curl --location --no-buffer --data '' {endpoint}``. Will redirect to the ``GET`` endpoint with appropriate headers set for streaming responses. All of the parameters accepted by the ``GET`` endpoint are accepted here and will be forwarded to the ``GET`` endpoint. The ``note`` field accepts a user-readable string instead of a base64-encoded string to facilitate easier submission when using ``curl``. - ``/validate[?]`` - ``GET`` - This will issue a *challenge* in the form of a time-limited nonce on the Controller’s serial console. - In a browser, this page will present a form to enter the limited-use nonce to proceed with the remaining steps. - Via the command line, the plaintext response will direct the client to the ``POST`` validation endpoint with an example command to validate ownership from the command line. - ``POST`` - When passed a valid authentication nonce in ``POST`` form-data under the ``nonce`` key, the Controller will permit diagnostics bundle collection and delivery for a limited duration or limited number of times, constrained by whichever is reached first. It accepts the following query string arguments: - ``limit_uses`` (Default: ``1``): restrict the ``/sos`` endpoint’s ability to generate and deliver support bundles by the given number of times. After this limit is reached, the Controller we require re-validation of ownership over the serial console. - ``limit_minutes`` (Default: ``1440``): restrict the ``/sos`` endpoint’s ability to generate and deliver support bundles within the given minute(s) limit. After this limit is reached, the Controller we require re-validation of ownership over the serial console. - ``include_secret`` (Default: ``0``): If set (to a truthy value like ``1`` or ``true``), the endpoint will return the trusted secret in the response body. We recommend to *not* set this parameter if using a non-HTTPS port, for example, if using port ``911`` over an untrusted network. - If validation responses include a trusted *secret* with the use of ``include_secret``, you may send it in subsequent requests to the root SoS endpoint to reliably generate support bundles without adhering to the time and usage-based limits. - ``/shared_secret`` - ``POST`` - When properly authenticated or validated via the serial console method, will generate a new shared secret if one does not exist. Note that, if you have command-line access to the Controller in question, you may elect to validate ownership of the Controller using the ``sos validate`` command. .. _sos-access: Controller ``/sos`` Access '''''''''''''''''''''''''' To avoid abuse from the public Internet, the ``/sos`` endpoint requires a logged-in user session. You may either use the endpoint after logging in to the :ref:`control-plane` or authenticate against the :ref:`controller-api` and use a session cookie with a tool like ``curl``. For example, the following sequence of commands logs in to a Controller at the endpoint ``https://bowtie.example.com`` with the username ``admin@example.com`` and password ``deadbeefcoffee``: .. admonition:: Capturing Authentication Cookies :class: hint We use the tool `httpie `_ here to simplify capturing authentication cookies, but the ``curl`` command also supports the ``--cookie`` and ``--cookie-jar`` flags. .. code:: shell http --session=bowtie https://bowtie.example.com/-net/api/v0/user/login \ email=admin@example.com \ password=deadbeefcoffee After this command, you may issue authenticated requests to the Controller with the saved session cookie. .. code:: shell http --session=bowtie https://bowtie.example.com/-net/api/v0/user/me Refer to :ref:`controller-api` for a complete list of available API endpoints. If the Controller Bowtie backend is non-functional and unable to authenticate user credentials, you may instead use the endpoint via port ``911`` which does not enforce a valid user session. For this reason, we suggest keeping access to port ``911`` restricted to trusted networks only. --------------- Example SoS Use --------------- Support bundles may be generated via either the :ref:`command line `, :ref:`browser `, or :ref:`REST API from the command line `: .. _sos-command-line-example: SoS Command Line Example '''''''''''''''''''''''' Invoke the ``sos`` command. If a Bowtie representative has already trusted the shared bundle secret, a diagnostic bundle will be generated and delivered to Bowtie. When run for the first time or waiting for Bowtie to trust the shared secret, this command will print the shared secret ID to pass along to our support team. The ``sos`` command accepts the ``validate`` command to optionally validate ownership of the Controller via the command line if you are using the API-based method. Consult ``sos --help`` for configurable options, including the ability to validate nonces generated via the :ref:`API method ` or select additional service logs as part of the diagnostics collection. .. _sos-browser-example: SoS Browser Example ''''''''''''''''''' .. admonition:: Privileges and Capabilities :class: tip If you are already logged into your Controller's :ref:`control-plane`, the browser-based SoS steps will skip validation and you will be forwarded directly to either create a shared secret or generate a support bundle directly, whichever is necessary first. If the :ref:`control-plane` is functional, we suggest using the native :ref:`controller-sos` instead. - Navigate to the domain name that your Controller is hosted under to ``/sos``. For example, the endpoint for a Controller running under ``https://controller.example.com`` would be ``https://controller.example.com/sos``. If the proxy layer terminating TLS and matching the domain name is non-functional, use the plain HTTP backup endpoint at ``http://controller.example.com:911`` (potentially by plain IP address if necessary). .. image:: _static/sos-validate.png - The initial ``/sos`` page will provide the instructions to follow. By default, the page will redirect to the ``/validate`` endpoint after generating a time-limited nonce emitted on your Controller's serial console to prove administrative ownership of the machine. - After retrieving the nonce, enter it into the form on the page to affirm ownership of the Controller. A valid nonce will return a success message with a link back to the ``/sos`` page for next steps. .. image:: _static/sos-validated.png - If necessary, the page will prompt you to create a shared secret with Bowtie. Proceed to create the shared secret with the indicated button. .. image:: _static/sos-create-secret.png - The page will provide a link back to the root page. **Make a note of the shared secret ID**, which you will need to share with a Bowtie representative. Follow the link to proceed. .. image:: _static/sos-secret-created.png - The next page will indicate whether the shared secret must first be trusted by Bowtie before initiating support bundle uploads. .. image:: _static/sos-awaiting.png Reach out to a Bowtie representative with your shared secret ID. Bowtie will trust the shared secret to enable your Controller to upload support bundles to our managed platform. - Load the root ``/sos`` after Bowtie has trusted the shared secret, which should present you with the following page. .. image:: _static/sos-collect.png Click the button to generate a support bundle and send it to Bowtie. If re-validation is necessary to gather more support bundles, the page will instruct you with next steps. - The resulting support bundle ID should be shared with Bowtie. Assisted troubleshooting can then begin in tandem with a member of the Bowtie team. Future diagnostic bundles may be generated without the shared secret step - only validation is necessary when performing the web-based method from this point on. .. _sos-api-example: SoS API Example ''''''''''''''' Given a Controller with the endpoint ``https://controller.example.com``, the steps to generate and send an SoS support request bundle to Bowtie are: - Open your platform’s serial console page for the Controller. For many on-premise hypervisors, this may be a page listing details for the Controller virtual machine. Most cloud providers list serial console output as a discrete page that you may navigate to and leave open in a browser tab. - Issue an initial request to the Controller’s SoS validation endpoint: .. code:: curl https://controller.example.com/sos/validate .. admonition:: API Access :class: tip Remember that if you elect to use ``/sos`` rather than port ``911``, :ref:`API authentication ` is necessary. Note that you may alternatively direct requests to http://controller.example.com:911 if the TLS-terminating reverse proxy is non-functional, but we suggest using the TLS-enabled endpoint if available to avoid plaintext transmission. - Retrieve the nonce from the Controller’s serial console output. - Issue a validation request to grant the Controller the ability to generate support bundles. If you would like a persistent secret to generate bundles in perpetuity, set ``include_secret`` and note down the printed secret for unrestricted use later if desired. .. code:: curl --form nonce=$token https://controller.example.com/sos/validate - Alternatively, retrieve the nonce and validate Controller ownership on the command line with the ``sos validate`` command. - If you have not yet already created a shared secret, do so now using the ``/shared_secret`` endpoint and note the resulting shared secret ID. .. code:: curl --data '' https://controller.example.com/sos/shared_secret - Reach out to a Bowtie representative to request permission to upload support bundles to our platform. Share the secret ID to identify your shared secret with our team. This step trusts the shared upload key to prevent abuse of the file upload capabilities of the support bundle API. - Generate a new support bundle and deliver it to Bowtie. This request sets the ``--no-buffer`` ``curl`` flag to stream output and the progress of the collection process, which will close the connection once the process completes. Set the ``expiration`` parameter at this step if you would prefer that your uploaded data be deleted within a time frame other than the default 30 days. .. code:: curl --location --no-buffer --data '' https://controller.example.com/sos - Contact your Bowtie representative with the listed bundle ID to collaborate on diagnostics efforts. Subsequent requests will either require re-validation or the ``Authorization`` header set to the trusted secret. Alternatively, with command-line access, subsequent calls to the ``sos`` command will generate support bundles without these validation steps. .. _sos-contents: ------------ SoS Contents ------------ The SoS diagnostics payload bundle includes a variety of different metrics and pieces of debugging information. The collection script intentionally avoids information that could lead to privileged access or sensitive information disclosure like password hashes, private keys, or personally identifiable information. While the total set of data collected may change with updates and improvements, in general, the data sent to Bowtie will include information like the following: - Service logs, including log output from the Bowtie API server, reverse proxy, Bowtie DNS service, and cloud-init logs. - Wireguard configuration, including device information, peer data, and network addressing information. Public keys are truncated to avoid unintentional disclosure and private keys are never included. - System profile (hardware specifications, current resource usage, hostname, filesystems, and network configuration) - Process tree - Metrics and other daemon instrumentation - A limited set of application database information. The collection script exports a subset of API objects like Controller information, site ranges, and organization configuration. Sensitive collections like user records are not exported. Note that, as part of the diagnostics process, Bowtie will proactively reach out to your Controller's public HTTP and HTTPS endpoints to confirm availability to affirm that capabilities like automatic TLS can function. .. _tls-troubleshooting: DNS/TLS ------- If you've encountered problems setting up DNS or TLS, check the following: - Is your controller instance available from any public address? - Have you confirmed :ref:`ports`? - If the application at ``/setup`` presented you with an error, please contact your Bowtie representative, who can engage with our engineering team to understand the error and provide assistance. .. _controller-monitoring: Monitoring ---------- Bowtie Controllers ship with a local, self-hosted observability stack. Reference :ref:`controller-observability` for complete documentation, including how to export this data via :ref:`OpenTelemetry ` for additional visibility within your own infrastructure. .. _kubernetes-troubleshooting: Kubernetes Help --------------- Consider the following steps when debugging a potentially problematic Kubernetes deployment: ----------- UDP Routing ----------- The Bowtie server process negotiates a `wireguard `_ connection between each client and thus requires that inbound port 443 UDP traffic be sent to the Bowtie ``Service`` object. The ``ingress-nginx`` Ingress Controller `supports inbound UDP port routing `_ and the :ref:`Helm Chart ` installs a `ConfigMap `_ that enables this. If you are not using ``ingress-nginx`` for inbound traffic routing, you may need to consult with your Ingress Controller’s documentation to determine how to properly route UDP traffic to your Bowtie ``Service``. .. _troubleshooting-client: Bowtie Client ============= .. _reset-dns: Resetting DNS ------------- The Bowtie Client includes a DNS server updates and your local network configuration to use it. This makes services in your private network accessible, and the local DNS server forwards public queries to your original DNS server (which is typically provided by your Internet service provider). Your original configuration will be restored whenever Bowtie is shut down, but if Bowtie is not shut down cleanly, these settings may not be restored. You may follow the instructions below to restore your DNS configuration if the Bowtie Client fails to do so (for example, if the Client daemon crashes.) --------------------- Shutting down Cleanly --------------------- To use the Bowtie Client itself to attempt to restore DNS, restart Bowtie and then shut it down cleanly. --------------------------- Shutting down on Windows 11 --------------------------- .. admonition:: Windows Documentation :class: todo This documentation will come soon, but you may reference the :ref:`MacOS documentation ` until then, as the processes are similar. In the meantime, use the ``Services`` app to query and disable the service. Delete Bowtie from ``\\HKCU\Software\Microsoft\Windows\CurrentVersion\Run`` to disable the tray applet starting on startup. .. todo:: Need to explain how to cleanly shut down Bowtie Client on Windows. .. _macos-shutdown: Shutting down on MacOS '''''''''''''''''''''' A helper script is available in ``/usr/local/bin/bowtie-ctl``. More information about this script may be printed via ``bowtie-ctl -h``. To shut down and disable the Bowtie client, run ``sudo bowtie-ctl disable``. Shutting down on Linux with ``systemd`` ''''''''''''''''''''''''''''''''''''''' Restarting and stopping Bowtie cleanly may be done using ``systemd``: .. code:: shell sudo systemctl restart bowtie-service sudo systemctl restart --user bowtie sudo systemctl stop bowtie-service sudo systemctl stop --user bowtie sudo systemctl disable bowtie-service sudo systemctl disable --user bowtie ---------------------- Resetting DNS Manually ---------------------- Alternately, you may choose to manually restore your DNS settings. Resetting DNS on Windows '''''''''''''''''''''''' .. todo:: Explain how to stop & disable ``bowtie-service`` on Windows .. todo:: TBD Step 0: explain how to ensure Bowtie is disabled and stopped on Windows #. Open Settings. #. Click on Network & Internet. #. Click the interface used by your Internet. This is usually labeled either “Ethernet” or “Wi-Fi”. #. Under the “DNS server assignment” section, click the Edit button. #. Select the “Automatic” option using the drop-down menu. Alternately, you can select the “Manual” option and manually enter a DNS server. For example, you may use “8.8.8.8” to use Google’s DNS service or “1.1.1.1” to use Cloudflare's. .. todo:: We should probably add some Windows screenshots explaining how to reset DNS manually. Resetting DNS on MacOS '''''''''''''''''''''' .. admonition:: MacOS Documentation :class: todo This documentation will come soon! .. todo:: OS X: guide for how to reset DNS manually. Resetting DNS on Linux with ``systemd`` ''''''''''''''''''''''''''''''''''''''' Ensure that the Bowtie Client ``bowtie-service`` service is stopped and disabled: .. code:: shell sudo systemctl stop bowtie-service sudo systemctl disable bowtie-service sudo systemctl stop --user bowtie sudo systemctl disable --user bowtie Then remove the DNS configuration file and restart ``systemd-resolved``: .. code:: shell sudo rm -rf /etc/systemd/resolved/bowtie-*.conf sudo systemctl restart systemd-resolved Cannot access the Internet when Bowtie is not running ----------------------------------------------------- Follow the steps at :ref:`reset-dns` in order to restore DNS settings. Deleting the Cache ------------------ The Bowtie Client maintains a cache of ``entrypoint`` values and other network data so that it can continue to work in the event of network issues. If you are experiencing issues with the Bowtie Client, you may delete this cache to prompt the Bowtie Client to reconfigure itself. The cache is located at the following filesystem paths: - Windows: ``C:\ProgramData\bowtie\cache`` - MacOS & Linux: ``/etc/bowtie/cache`` Removing an Entrypoint ---------------------- Entrypoints are cached, so Bowtie will continue to try and contact all entrypoints even if they have been removed from the configuration file. To remove an entrypoint, first remove its entry from the :ref:`client-configuration`, then follow instructions for `Deleting the Cache`_. .. _client-logs: Debug Log Information --------------------- The amount of log information created by the Bowtie client may be adjusted by passing one or more ``--verbose`` or ``--silent`` options to the executables or adding the ``verbose = N`` or ``silent = N`` options to the configuration file. ------- Windows ------- Log information for the Bowtie client is saved in ``C:\ProgramData\Bowtie``. ----- MacOS ----- Log information for the Bowtie client is saved in ``/var/log/bowtie``. These log files, along with related system information, may be saved into a compressed file that can assist with remote debugging through ``sudo bowtie-ctl inspect``. ----- Linux ----- Log information is printed to standard output and is usually collected by ``systemd-journald``. It can be inspected with: .. code:: shell sudo journalctl --unit bowtie-service journalctl --user --unit bowtie