Automating Reverse SSH Tunnels to Maximize Remote Connectivity

Introduction

"In my vacation home, I recently installed some cameras and temperature/humidity sensors which report to my Home Assistant server running in my main residence." 

This experience highlights the increasing reliance on remote access solutions. In his setup, a reverse SSH tunnel serves as the linchpin, facilitating secure connections from a Home Assistant server to a remote Linux machine for various tasks, including maintenance, software updates, and real-time monitoring of environmental conditions.

"To do that, I set up a wireguard VPN through a small Intel NUC running Ubuntu Linux and the IOT devices use that as their gateway."

This article explores his process to illustrate how this arrangement could help us streamline the management of devices.

Setting Up the Reverse SSH Tunnel

The process started with the establishment of a reverse SSH tunnel. By leveraging the SSH command with the '-R' option, he initiates a tunnel that enables connections from the Home Assistant server to the remote Linux machine, effectively bridging the geographical gap between the two locations.

ssh -R 1022:localhost:22 homeassistant

This configuration ensures secure access to the remote machine without exposing it directly to the internet, thereby safeguarding sensitive data and mitigating the risk of unauthorized access. Additionally, his utilization of Home Assistant as a centralized hub for managing smart devices brought about a seamless and interconnected ecosystem.

Navigating Connectivity Challenges

However, despite the initial success of the setup, periodic disruptions in the tunnel came about due to intermittent internet connections. This issue becomes particularly pronounced given the transatlantic nature of the link, as geographical distance and varying network conditions contribute to occasional lapses in connectivity.

"This happens approximately every 3-4 days because the connection is a transatlantic link."

These disruptions pose a considerable inconvenience, impacting his ability to perform essential tasks remotely while presenting challenges associated with maintaining connectivity.

Seeking Automation Solutions

To mitigate these connectivity challenges and enhance the reliability of the reverse SSH tunnel, he diverted his attention towards various automation solutions. Initially, he created an endless loop to automatically restart the tunnel, hoping to address potential disruptions and minimize downtime proactively.

"However, this doesn’t work because the ssh reverse tunnel does not connect and waits for connection."

This realization that SSH does not wait for connections and exits immediately after tunnel establishment, renders the manual restart approach ineffective as well.

Discovering Autossh for Enhanced Reliability

After this, he finds the 'autossh' package (learn more here), a tool expressly designed for monitoring SSH connections and automatically restarting them if they fail or stop passing traffic.

"The correct way to automatically restart an ssh reverse tunnel is by using an autossh package."

With its advanced features and intuitive interface, autossh exhibits greater reliability. After installing autossh on the remote NUC, they configure it to manage the reverse tunnel.

autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -N -R 5022:localhost:22 homeassistant

This command, with various options, reaffirms the reliability of the tunnel by dispatching keepalive packets and delineating reconnect thresholds, addressing concerns regarding connectivity disruptions. With autossh in place, we can rest assured knowing that the remote access infrastructure is secured against network instability, enabling one to focus on tasks without the looming fear of downtime.

Step-by-Step Guide to Setting Up and Automating a Reverse SSH Tunnel:

1. Initial Setup: Cameras and sensors reporting to a Home Assistant are set up. Due to security reasons, the remote Linux machine is not directly accessible from the internet.

2. Establish Reverse SSH Tunnel: On the Intel NUC machine (located remotely), initiate a reverse SSH tunnel by running the command:

ssh -R 1022:localhost:22 homeassistant

The -R option instructs SSH to open a reverse tunnel.

3. Remote Access: When maintenance or updates are required on the remote NUC, access it from the Home Assistant Server by running:

ssh -p 1022 localhost

This command allows logging in to the remote NUC via the established tunnel.

4. Addressing Connectivity Issues: Periodic disruptions could occur due to intermittent internet connections, particularly noticeable with a transatlantic link, causing the reverse tunnel to drop approximately every 3-4 days.

5. Automating Tunnel Restart: If attempts to restart the tunnel using a bash shell loop fail because SSH exits immediately after connection, install and utilize Autossh, a tool designed for monitoring and automatically restarting SSH connections.

6. Installing Autossh: Install Autossh on the remote NUC with Linux by running:

sudo apt install autossh

7. Starting Automatic Tunnel: Launch the automatic reverse tunnel with Autossh by executing the following command:

autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -N -R 5022:localhost:22 homeassistant

- The -M 0 option specifies not to use a port for monitoring connection status.

- -o "ServerAliveInterval 30" and -o "ServerAliveCountMax 3" options instruct Autossh to send keepalive packets every 30 seconds and attempt to reconnect if three consecutive keepalive packets fail.

- -N -f options tell SSH to create the tunnel in the background without executing any remote commands.

Following these steps ensures the establishment and automation of a reliable reverse SSH tunnel for seamless remote access to the Linux machine.

“Voila! Now, the ssh reverse tunnel will automatically restart if the connection goes down. For better monitoring, I ran the autossh command inside a tmux session, so I can connect to it and check what’s going on.”

Conclusion

Moral of the story? By embracing automated solutions like Autossh, we can improve our infrastructure against the uncertainties of network disruptions and ensure uninterrupted access to critical resources. As technology continues to evolve, the need for reliable remote connectivity remains an ongoing effort.

With Autossh, we can navigate the complexities of modern networking environments to unlock new, exciting possibilities.