Homelab Chronicles 10: Verifying the VPN and Failing

Having done all the prep work for the Unifi L2TP VPN, I was ready to test it out. I turned on my hotspot on my phone and had my laptop connect to it. Using the built-in Windows VPN client, I went ahead and put my VPN address in, username and password, and the pre-shared key. Then I hit connect.

And it connected! Quickly and on the first attempt!

Of course, that’s only half the battle. Could I reach local network resources? Would the VPN forward my web traffic?

Yes and No. Great.

At first, pings to local resources failed. But then I realized that I was still running a firewall rule in Unifi that blocked all inter-VLAN traffic. After I turned those rules off, those pings, including to the router and a Windows server, started working.

I could even connect to network drives—though only using the IP address, and not with a hostname. In a command line, I ran ipconfig /all, and the entry for the L2TP VPN adapter showed the correct namesevers for my network. Strange.

On the web front, it failed completely. In Edge (Chromium-based), trying to go to any website failed immediately. Even trying to go to ESXI’s portal, which simply uses an IP address failed. Same happened in Chrome and Firefox.

OK, well maybe it wasn’t getting out to the Internet. I tried ping 8.8.8.8 -t; that worked, so it was getting out to the Internet via the VPN. Then I tried pinging a domain, like espn.com or yahoo.com. Interestingly, it resolved the IP address and the ping succeeded.

I checked that all custom firewall rules in Unifi were turned off, not that I had many. And certainly none related to blocking web traffic.

Well, maybe Windows itself is doing some kind of firewalling. I don’t understand why it would appear to be blocking only Port 80 web traffic when connected to this VPN (I often use a VPN for work and Windscribe when travelling and they work flawlessly), but I completely turned off all firewalls. Still didn’t solve the issue.

At this point, I start scouring the Internet. It seemed like many others had similar issues, with even a few having basically the same problem. But there was never a solution or something that I hadn’t tried already or some configuration change that was applicable to me. A common problem was people who were on the same subnet locally and on the VPN. That didn’t apply to me since my phone hotspot was using a completely different subnet from anything I use.

I was starting to get annoyed. I had to refocus. What could it not be? Because the VPN actually did connect, it couldn’t have been the domain and DDNS stuff I was doing the other day. The username, password, and PSK were correct as well. It wasn’t any custom Unifi firewall rules that I had in place, since those only dealt with inter-VLAN routing and I turned them off, and was able to reach other devices on the network.

Could it be the computer itself?

I know the Windows VPN client is crappy. Though I’ve also used it before with other VPN connections and it was fine. But it’s always good troubleshooting to isolate the problem as best as possible.

That led me to pull out my aging 8-9yo Macbook Pro. I connected it to my hotspot, created a new VPN connection, set it to be highest in the network order, and also set it to route all traffic down the VPN, and then pushed the “Connect” button.

It connected. I tried pinging local resources; success. I tried connecting via SMB to local resources; success. I even opened a movie that I had stored on that network drive; it played. OK, looking good. Time for the moment of truth: I opened Edge and went to a website.

It loaded! I navigated to ESXI’s login page, which I connect to using an IP address. It loaded. I went through several of my bookmarks, went to YouTube, watched a video—it all worked!

But was web really going through the VPN? For all I knew, it could have been simply “falling back” to the regular WiFi hotspot connection. However, in MacOS, there are some colored bars that show when sending and receiving traffic is going through the VPN connection. And guess what? As I made requests in the web browser, I could see the bars lighting up, especially when traffic was inbound.

So I did set-up the VPN properly! It was working exactly how it should! But then why the hell was it not working on my main Windows laptop?

For good measure, I restarted the laptop. Then I deleted the VPN connection in Windows and remade it. It connected just fine, but like before, network resources could be reached, but not web traffic.

How about the VPN client? Maybe Windows’ client really is that bad and the culprit. I looked around for a third party client and someone on reddit recommended the Draytek Smart VPN Client. Downloaded and installed it. Entered in the VPN settings. It connected. But like before…Exact. Same. Thing. Happened.

Which leaves me here, after 3-4hrs of messing with this. I don’t understand why it works perfectly on MacOS, but far from perfectly on Windows. I don’t understand why local traffic, and even domain resolution for command line ping and tracert commands, work, but not web traffic. I don’t understand why Spotify “half-loaded.” Forgot to mention that. Like the items on the app’s home screen wouldn’t load, but songs I know I’ve never played and downloaded onto that laptop actually played.

So 3-4hrs later, I’m defeated. I’m frustrated. I don’t know what else to do or where else to look. Even Ubiquiti’s Unifi forums aren’t super helpful. Lots of really old posts that I don’t think necessarily apply here. YouTube had several videos on creating the VPN, but not addressing this specific problem. Reddit’s Unifi forum had plenty of questions, but no answers. I’m at a loss.

But I need a VPN solution. A friend told me about OpenVPN’s free Access Server service. I also have a friend who uses WireGuard and (mostly) swears by it. Some time last year, I did make an attempt to build a self-hosted OpenVPN server, though it was quite technical. I’ll start looking into one of those options.

Sigh.

Homelab Chronicles 09: Meandering in Ubuntu all for VPN

All I wanted to do was look into setting up VPN access on my Unifi USG. That’s actually pretty easy to do.

I set up a L2TP VPN in the Unifi Controller, no big deal. But then I remembered that I don’t have a static IP address at home. Like most households, I have a dynamic IP address. Of course, even dynamic IPs tend to be “sticky.” At my office about a year ago, I found out that our router was misconfigured and not using a static IP address like it was supposed to when VPN connections stopped working one day. But it had been like a year since it was initially (mis)configured! Sticky, indeed.

Same goes for residential; it’s not unusual for dynamic IPs addresses to last weeks or even months. However, I didn’t want to have to deal with my home VPN not working when I needed it due to “losing” my IP address.

Thank goodness for Dynamic DNS (DDNS). Fortunately, the Unifi Controller makes it easy to use DDNS services. Unfortunately, my host/registrar—Dreamhost, where this site is hosted—wasn’t included in the easy-to-set-up services list in the Controller. Typically for DDNS, if the router doesn’t have this function built-in, software can be downloaded that quietly runs in the background, periodically updating the DNS records with the current IP address. Dreamhost doesn’t have that though, because they don’t provide out-of-the-box DDNS service.

A quick Google search, however, revealed that some kind soul had created a bash script to run DDNS for Dreamhost. Which meant I’d have to host this on Linux. My Ubuntu VM that hosts the Unifi Controller seemed like a good place. No sense in spinning up another VM for something so lightweight.

This is where it snowballed. Mainly because I don’t have a whole lot of experience with Linux, especially in the command line.

Task 1 – Setting up XRDP

Whenever I need to go into that VM, I sign-in to ESXi and use the remote desktop in there. But it’s limited in resolution and sometimes ESXi signs me out. I needed a proper remote desktop program and have for awhile.

I use a proprietary RDP program for my Windows and Mac machines. But it no longer support Linux. Bummer. But Google tells me that setting up an RDP server for Ubuntu is pretty simple. By using XRDP.

The main change I did here was to use a different port for RDP. I remember in my MSP days that it was important to change the port away from the default of 3389 for security purposes. Knowing that some blocks of ports shouldn’t be used, but not knowing which, I once again turned to Google. An answer from Stack Overflow gave me the answer I needed:

  1. System Ports (0-1023): I don’t want to use any of these ports because the server may be running services on standard ports in this range
  2. User Ports (1024-49151): Given that the applications are internal I don’t intend to request IANA to reserve a number for any of our applications. However, I’d like to reduce the likelihood of the same port being used by another process, e.g., Oracle Net Listener on 1521.
  3. Dynamic and/or Private Ports (49152-65535): This range is ideal for custom port numbers. 
Stack Overflow – Best TCP port number range for internal applications

Dynamic/Private sounds the best, but realistically, User Ports are better. The former is sometimes used by the OS or applications for ephemeral purposes and I don’t want to run into an issue where something is using my RDP port temporarily, blocking my access. I selected a random port number in the User Ports range that wasn’t known to be used by any applications by the IANA.

After setting up XRDP, I attempted to use Windows’ RDC to connect…and it failed. All I saw was a black screen momentarily, before the RDP connection close. Apparently that’s because XRDP only works when if the user account is signed-out locally. I’d forgotten that RDP isn’t like a “virtual KVM” like TeamViewer or ConnectWise. RDP actually requires signing-in to the user account and starting a new session. And obviously an account can be signed-in one place at a time. Same as Windows RDP.

Once realizing that and signing out Ubuntu via ESXi, I was able to sign-in!

Screenshot of Ubuntu desktop with some windows open, including terminal, file manager, and Chrome.
Ubuntu looks a tad different in an RDP session.

So now that almost unrelated journey was over, it was time to get to the meat: setting up that bash script.

Task 2 – Running the Dreamhost Dynamic DNS Script

I’m not going to go through all the instructions since the script’s GitHub page lists them, but I’ll just briefly tell of things I got stuck on and how I got around it.

The command syntax is listed as

dynamicdns.bash [-Sd][-k API Key] [-r Record] [-i New IP Address] [-L Logging (true/false)]

I had already created the API key in Dreamhost and the new A Record (with a “fake” IP address) that I wanted the script to update. So just plug and chug at this point. The instructions for the -i flag said that if it was empty, the script would automatically use dig to find the external IP of the network. That’s obviously what I wanted since my home IP address could change. But then it kept giving me an error that the flag required an argument.

Ubuntu Terminal test showing the bash command and the error generated.
But I thought it’s supposed to have no argument…

Eventually, I tried not including that flag and it appeared to succeed! I checked out the DNS on the Dreamhost panel and the IP address was now showing my external IP. I tried updating the DNS with either various IPs via the script a few times to make sure it was working, and each time the A Record listed in the Dreamhost DNS showed the IP address I used.

Future configuration changes can be made in the config file that the script creates when it’s first run, instead of the command line. However, I couldn’t find where that was at first; it wasn’t in the same directory as the script itself. Reading through the script itself, it seemed that it created it in a hidden .config directory in the user home folder. The config file is conveniently called dynamicdns.

Naturally, I don’t want to have to manually run the script each time. The whole point of using DDNS services is that it’s periodic and automatic. If I have to run the script every time myself, might as well just forget all this and make the change manually in the DNS! Time to set up a Cronjob.

Task 3 – Automating via Cron

I’ve messed with Cron exactly once before. I can’t even remember why I did it and, therefore, didn’t remember how to use it. I followed this guide by Digital Ocean to install Cron (wasn’t sure if it was installed already). I chose to use nano as my editors because my experiences with vi have not been great.

Cron requires a schedule and the command or thing to run. I wanted to have the script run hourly, so that if my home IP did change, it’d get picked up relatively quickly (ignoring any delays in DNS propagation). Trying to understand how to format the schedule can be challenging…so back to Google, where it found the Crontab Guru.

Screenshot of Crontab Guru website.
Immature, I know.

Playing around with guru helped me better understand how the scheduling syntax worked, as opposed to reading examples. I quickly found that running the command hourly could be done with:

0 * * * *

Now all I needed was the command. Given that it was a bash script, I knew I’d need to use bash, followed by the path of the script. But what was the path of the script? It was in a subdirectory in my home directory, but what’s the path? I noticed that a lot of cron examples had something like ~/bin/filehere. What does the tilde mean?

Apparently it means the home directory. So after playing around a bit and testing—I set the schedule temporarily to run every 5 minutes and set the A Record IP to something else—I figured out the correct path after noticing the IP address in the DNS finally changed. This is what the complete cronjob looks like:

Screenshot of the cronjob after using command crontab -l.
Maybe I should’ve tossed the script in ~/bin/

All Done! Maybe?

Well, no. This was all done just so I could set-up VPN access via the Unifi Controller. Which is set up, but I haven’t had a chance to test! It’s late and I’ll be in the office later this week, so I’ll try it out then.

Even though this was just prep work, it was good opportunity to play around with Ubuntu and Linux more, in particular the CLI. I recently expanded size of the Ubuntu VM and had to do a little bit of CLI work, but it really wasn’t that much. And I can’t imaging I’ll be doing that very often. Before this, I think the last time I played around in Terminal was when I was setting up the Unifi Controller as a service, via systemctl. So not a whole lot of experience collectively, but I’m hoping to do more stuff like so I can get better acquainted. That’s what a Homelab is for, right?

I’ll report on the VPN at later date.