Thursday, May 17, 2012

Troubles with OpenVPN in Ubuntu 12.04

Update: Check out my solution to the problem described below.

Last year at work I set up a VPN using OpenVPN in Ubuntu Server 11.10.  There were a number of issues with supporting the specific topology of the local network I was trying to make accessible that made it tricky to get traffic from a connected client properly routed in and back.  To be upfront, I'm not a Linux guru.  I get by with the principles I've learned in the classroom, the tips I've been given from more enthusiastic friends and family, and from what I can glean from forums, help pages, and manual pages, and generally I do alright.

Back to my story.  So last year I honestly don't know what exactly it was that got things to work, which is bad, I know.  At one point I had two other guys helping me while we looked at the traffic going between a client and the VPN server with Wireshark on both ends.  We could see ping packets making it successfully to the server and we could see the responses generated by the server, but no replies were making it back to the client.  Weird.  In that case, I finally figured out that in the different configurations I had tried, at one point I had commented out the bridged interface, br0, from /etc/network/interfaces.  I un-commented those lines, things started working, and I never touched it for fear of breaking things.  Restarting the system kept things going, so it must have been some sort of persistent configuration.

Fast forward to a couple weeks ago, when my overconfidence and excitement at the release of Ubuntu 12.04 LTS got the better of me, overpowering my previous fear of breaking the VPN.  I think I said to myself something like the following. "Well, all the configuration files will carry over, so things can only get better by running 'sudo do-release-upgrade', right?  After all, this is a Long-Term Support release, so it's nice and stable."

Wrong.  Things broke.  So far, I've overcome one hurdle, but things still aren't working, so I have no idea how many other obstacles are in my way to being back up and running.  Here is the evidence I've gathered so far:
  • In /var/log/syslog, there was an error about the security level setting for running external scripts.  Adding the line "script-security 3" before calling any scripts made that go away and allowed OpenVPN to successfully start up.  The script I'm running is up.sh as described in the OpenVPN tutorial on the Ubuntu help pages, under the section "Prepare server config for bridging".
  • Initially after the release upgrade, the primary connection, eth0, no longer had any access to the outside world.  I could ping the router, but nothing outside the local network.  I found that setting the tap0 device to down and changing eth0 from static to dhcp seemed to fix this.  Unfortunately, later something reverted back and the server no longer has outside access again.  I tried the tricks I used before and even tried stopping OpenVPN with '/etc/init.d/openvpn stop' with no success, even after restarting networking and rebooting the server.
  • Even when the server could access external addresses things didn't seem to be routing packets correctly.  I found that by disabling the br0, eth1, or tap0 (I can't remember...), web pages loaded faster and everything seemed to work properly.  Running 'route -vn' was what gave me the idea in the first place because there multiple devices specified as the outgoing for the same destination range.  Unfortunately I didn't memorize the details and things aren't at the point they were previously to try and figure it out.
I'm sure there are some specifics of how OpenVPN works that would help me troubleshoot this, but my struggle with problems like this (which seem to present themselves more often in Linux because I'm doing more advanced things) is always trying to balance understanding the details of whatever I'm working with and just getting the dumb thing to work because I need it done last week or last month...

Hopefully I'll get to the bottom of all this and I'll be able to report my findings soon.  In the mean time, take a lesson from my book, and be a little more cautious than I was with updating to the latest release.