FreeNAS – getting hacks to survive an upgrade Part 2


***WARNING***

THIS DOES NOT MEAN YOU GET TO SHUT YOUR BRAIN OFF.  UPGRADING FREENAS COULD STILL RENDER YOUR CHANGES NULL AND VOID.  AFTER ANY FREENAS UPGRADE, VERIFY THAT YOUR CHANGES ARE STILL IN EFFECT EVEN WITH THIS STICKY HACK IN PLACE.

Part 1 of trying to get hacks to FreeNAS to survive an upgrade was a failure.  But never fear, I went back to the drawing board and had great success tonight. Basically I changed my approach slightly,  Instead of trying to make changes to /conf/base/etc/rc.conf , what ended up working was just making the changes to /etc/rc.conf on every boot.

This will have the added benefit of working even if the FreeNAS developers change their naming scheme again for things in /conf/base/etc.

Write our hacks.sh script

1. I have OpenVPN and pf Firewall enabled, neither of which are stock to FreeNAS.  You’ll need to adjust this script to meet your needs.  Open up your favorite text editor and save your hacks.sh file somewhere on your data drive on the FreeNAS box.

First thing is lets search /etc/rc.conf to see if our hacks are in place.  If they are, we exit.

#!/bin/sh
if grep "pf_enable" /etc/rc.conf
then
echo "Hacks in Place"
else

Now, the else section is where we add our hacks back in to /etc/rc.conf when they are not there.  I’m just using a simple echo command to write the line I want to add to the shell and then am adding it to the end of /etc/rc.conf.  I’m adding a blank line before my hacks, some notes about the hacks I’m adding and the config changes.  You will need to change the paths to pf_rules, openvpn_configfile and openvpn_dir to match your setup.

echo "" >> /etc/rc.conf
echo "#Turn on PF Firewall" >> /etc/rc.conf
echo "pf_enable='YES'" >> /etc/rc.conf
echo "pf_rules='/mnt/Files/joe/hacks/pf.conf'" >> /etc/rc.conf
echo "gateway_enable='YES'" >> /etc/rc.conf
echo "" >> /etc/rc.conf
echo "#Turn on OpenVPN" >> /etc/rc.conf
echo "openvpn_enable='YES'" >> /etc/rc.conf
echo "openvpn_if='tun'" >> /etc/rc.conf
echo "openvpn_configfile='/mnt/Files/openvpn/openvpn.conf'" >> /etc/rc.conf
echo "openvpn_dir='/mnt/Files/openvpn'" >> /etc/rc.conf

Next we want to start the services that we have added as hacks to FreeNAS.  For me that is pf and openVPN.  Last thing is to close out the if statement with fi.

service pf start
service openvpn start
fi

Here’s the full script.

#!/bin/sh
if grep "pf_enable" /etc/rc.conf
then
echo "Hacks in Place"
else
echo "" >> /etc/rc.conf
echo "#Turn on PF Firewall" >> /etc/rc.conf
echo "pf_enable='YES'" >> /etc/rc.conf
echo "pf_rules='/mnt/Files/joe/hacks/pf.conf'" >> /etc/rc.conf
echo "gateway_enable='YES'" >> /etc/rc.conf
echo "" >> /etc/rc.conf
echo "#Turn on OpenVPN" >> /etc/rc.conf
echo "openvpn_enable='YES'" >> /etc/rc.conf
echo "openvpn_if='tun'" >> /etc/rc.conf
echo "openvpn_configfile='/mnt/Files/openvpn/openvpn.conf'" >> /etc/rc.conf
echo "openvpn_dir='/mnt/Files/openvpn'" >> /etc/rc.conf
service pf start
service openvpn start
fi

Save it and exit.

2. Now we need to make our hacks.sh script executable and make it owned by root for good measure.  Make sure to change the path below to match your setup.

chmod 700 /mnt/Files/joe/hacks/hacks.sh
chown root /mnt/Files/joe/hacks/hacks.sh

Set up the FreeNAS GUI to run hacks.sh as an init script.

3. Log into the FreeNAS gui and go to System > Init/Shutdown Scripts.  Then click add Init/Shutdown Script

Screen Shot 2014-03-09 at 10.30.47 PM

You want to select script in the first field.  Browse to your hacks.sh script and select it in the second field and select Post Init in the third field.

Screen Shot 2014-03-09 at 10.31.06 PM

Hit OK.

4. Reboot and watch the magic.

Verification

5. After you’ve rebooted, check the status of your hacks.

service openvpn status

You should get this if you are running openVPN

openvpn is running as pid 17535.

service pf status

You should get this if you are running pf.  Notice that is says enabled.

No ALTQ support in kernel
ALTQ related functions disabled
Status: Enabled for 5 days 00:16:00           Debug: Urgent

State Table                          Total             Rate
current entries                       21
searches                      1236116133         2855.0/s
inserts                            75627            0.2/s
removals                           75649            0.2/s
Counters
match                            2299524            5.3/s
bad-offset                             0            0.0/s
fragment                               0            0.0/s
short                                  0            0.0/s
normalize                              0            0.0/s
memory                                 0            0.0/s
bad-timestamp                          0            0.0/s
congestion                             0            0.0/s
ip-option                          28950            0.1/s
proto-cksum                            0            0.0/s
state-mismatch                     94319            0.2/s
state-insert                           0            0.0/s
state-limit                            0            0.0/s
src-limit                             15            0.0/s
synproxy                               0            0.0/s

Final Thoughts

I set this up and tested it in a VM that started out as FreeNAS 9.2.0.  After setting this all up, I did a system upgrade using the GUI and upgraded to 9.2.1.2.  After rebooting, I checked /etc/rc.conf and my changes were in fact still there.  I then checked to see if pf and openVPN were running and they were.  SUCCESS!!!

I hope you all enjoy this.  Let me know of all the great ways you find to use this.

FreeNAS – getting hacks to survive an upgrade


Update:

The first attempt at getting this to work failed miserably. Not only did the shell script mangle the additions to rc.conf, but the script created a reboot loop that was pretty much impossible to break.  THIS NEEDS MORE WORK! 

GOT IT WORKING!!!  CHECK IT OUT HERE!

I’ve been customizing FreeNAS to do more than it is meant to do out of the box.  I want to run OpenVPN on it.  I want it to have a firewall enabled on it.  I’ve got those things working, but the issue I have is after every OS upgrade, I have to remember to go back in and re-enable all of them.

What we really need is a way to make modifications to FreeNAS sticky.  So, I was thinking, why not write a script that will launch when the system is initializing and check to see if the modifications are in place or not.  The FreeNAS developers are pretty good about keeping things that are configured in the GUI and moving them forward with the OS upgrades.

Here are the basic steps involved:

  1. Write a shell script that checks /conf/base/etc/rc.conf for our customizations
  2. If they are not present, add them to the end of /conf/base/etc/rc.conf, then reboot
  3. If they are there, boot normally
  4. Run the script at startup by creating an init script in the GUI

I am not much of a code writer…and honestly this is my first shell script….so it isn’t going to be pretty.  So far here is what I have.

#!/bin/sh
if grep ‘pf_enable’ /conf/base/etc/rc.conf
then
echo “Hacks in Place”
else
mount -uw /
echo ‘#Turn on pf firewall’ >> /conf/base/etc/rc.conf
echo ‘pf_enable=”YES”‘ >> /conf/base/etc/rc.conf
echo ‘pf_rules=”/mnt/Files/joe/hacks/pf.conf”‘ >> /conf/base/etc/rc.conf
echo ‘gateway_enable=”YES”‘ >> /conf/base/etc/rc.conf
echo ‘ ‘ >> /conf/base/etc/rc.conf
echo ‘#Turn on OpenVPN’ >> /conf/base/etc/rc.conf
echo ‘openvpn_enable=”YES”‘ >> /conf/base/etc/rc.conf
echo ‘openvpn_if=”tun”‘ >> /conf/base/etc/rc.conf
echo ‘openvpn_configfile=”/mnt/Files/openvpn/openvpn.conf”‘ >> /conf/base/etc/rc.conf
echo ‘openvpn_dir=”/mnt/Files/openvpn”‘ >> /conf/base/etc/rc.conf
fi

So, that’s where I am on this so far.  Next step will be to test it out in a vm and see what it does.  Hopefully it does what I expect and then it is just a matter of setting it up in the GUI under Init Scripts.

More to come, so stay tuned.

Secure FreeNAS 9.2.1.2 with a Firewall


The recent NTP reflection incident I was a victim of woke me up to the need for securing my FreeNAS boxes from outside connections.  Luckily, FreeNAS 9.2.1.2 comes shipped with the kernel extension for pf, so getting it working is pretty easy.

If you are going to do this on your FreeNAS box, please also check out my guide on getting hacks to FreeNAS to survive system upgrades.

DISCLAIMER – I AM NOT A SECURITY EXPERT, FOLLOW THIS GUIDE AT YOUR OWN RISK

I’ve been told that pf does not play nice with vimage jails.  It tends to cause kernel panics.

Get the basics done

1. Mount the filesystem so we can make some changes.

su
enter your root password
mount -uw /

2. Figure out what interfaces you have active so we can add them to the firewall rules.

ifconfig

Your output should look something like this:

re0: flags=8943<up,broadcast,running,promisc,simplex,multicast> metric 0 mtu 1500
options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
ether f4:6d:04:db:21:ba
inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
inet6 xxxx::xxxx:xxxx:xxxx:xxxx%re0 prefixlen 64 scopeid 0x6
inet xxx.xxx.xxx.xxx netmask 0xfffffff8 broadcast 50.241.46.71
nd6 options=23<performnud,accept_rtadv,auto_linklocal>
media: Ethernet autoselect (1000baseT )
status: active

ipfw0: flags=8801<up,simplex,multicast> metric 0 mtu 65536
nd6 options=9<performnud,ifdisabled>

lo0: flags=8049<up,loopback,running,multicast> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<performnud,auto_linklocal>

bridge0: flags=8843<up,broadcast,running,simplex,multicast> metric 0 mtu 1500
ether 02:df:7f:1c:ff:00
nd6 options=1
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair0a flags=143<learning,discover,autoedge,autoptp>
ifmaxaddr 0 port 12 priority 128 path cost 2000
member: re0 flags=143<learning,discover,autoedge,autoptp>
ifmaxaddr 0 port 6 priority 128 path cost 20000

epair0a: flags=8943<up,broadcast,running,promisc,simplex,multicast> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:09:09:00:0c:0a
nd6 options=1
media: Ethernet 10Gbase-T (10Gbase-T )
status: active

tun0: flags=8051<up,pointopoint,running,multicast> metric 0 mtu 1500
options=80000<LINKSTATE>
inet 10.8.0.1 –> 10.8.0.2 netmask 0xffffffff nd6 options=1
Opened by PID 17528

What we want to note are all the interface names.  Mine are re0, ipfw0, lo0, bridge0, epair0a and tun0

Create the firewall rules

3. Create the pf.conf file and edit it to your needs.  We are going to put it on our data drive so future FreeNAS upgrades don’t wipe it out. Adjust the path to match your setup.

mkdir /mnt/Files/hacks
vi /mnt/Files/hacks/pf.conf

Side note, vi has lots of commands, we just need to know a few:

x will delete the character your cursor is over, i will insert, esc will exit insert mode and to save and quit we use :wq

Here are the rules I am running.

#change this to match your primary ethernet interface, re0 or em0 are common, but there are others
ext_if="re0"
vpn_if="tun0"
table <bruteforce> persist
#These are all of the other interfaces we discovered in step 2
set skip on lo0
set skip on bridge0
set skip on ipfw0
set skip on epair0a
set skip on tun0
set block-policy return
scrub in all
#change xxx.xxx.xxx.xxx to the external IP of your FreeNAS box
nat on $ext_if from 10.8.0.0/24 to any -> xxx.xxx.xxx.xxx
#Lock it down
block in all
block out all
#Allow VPN traffic
pass on tun0 keep state
block quick from <bruteforce>
#Allow traffic in for ssh
pass in on $ext_if proto tcp from any to any port 22 flags S/SA keep state (max-src-conn 10, max-src-conn-rate 5/5, overload <bruteforce> flush global)
#Allow traffic in for web - delete or comment out if you don't want web traffic
pass in on $ext_if proto tcp from any to any port 80 flags S/SA keep state
pass in on $ext_if proto tcp from any to any port 443 flags S/SA keep state
#Allow traffic in for OpenVPN
pass in on $ext_if proto udp from any to any port 1194 keep state label "openvpn"
#Allow LAN traffic to connect to FreeNAS - change xxx.xxx.xxx.0 to match your network, ie 192.168.0.0 or 10.0.0.0
pass in on $ext_if from xxx.xxx.xxx.0/24 to any keep state
#Allow traffic out from the LAN
pass out on $ext_if from any to any keep state

Enable the Firewall

4. Edit /etc/rc.conf and add the following.

vi /etc/rc.conf

pf_enable="YES"
pf_rules="/mnt/Files/hacks/pf.conf"
gateway_enable="YES"

5. Start up the firewall and see if it works

service pf start

Your should get this as your output:

Enabling pf
No ALTQ support in kernel
ALTQ related functions disabled

Now check to make sure it is working:

service pf status

Your output should be something like this:

No ALTQ support in kernel
ALTQ related functions disabled
Status: Enabled for 0 days 00:04:55           Debug: Urgent

State Table                          Total             Rate
current entries                       29
searches                         1040038         3525.6/s
inserts                               95            0.3/s
removals                             109            0.4/s
Counters
match                               1093            3.7/s
bad-offset                             0            0.0/s
fragment                               0            0.0/s
short                                  0            0.0/s
normalize                              0            0.0/s
memory                                 0            0.0/s
bad-timestamp                          0            0.0/s
congestion                             0            0.0/s
ip-option                             22            0.1/s
proto-cksum                            0            0.0/s
state-mismatch                         0            0.0/s
state-insert                           0            0.0/s
state-limit                            0            0.0/s
src-limit                              0            0.0/s
synproxy                               0            0.0/s

6. Start IP Forwarding without restarting your server

sysctl net.inet.ip.forwarding=1

Then make sure to go into the FreeNAS gui, click on System, Click on Sysctls.  Then add that same value so it survives reboots.

Screen Shot 2014-05-14 at 10.43.22 AM

Make your changes stick

7.  Add your rc.conf changes to /conf/base/etc/rc.conf

vi /conf/base/etc/rc.conf

pf_enable="YES"
pf_rules="/mnt/Files/hacks/pf.conf"
gateway_enable="YES"

To really make these changes stick, follow my guide on the subject.

Clean Up

8. Make the filesystem read only again

mount -ur /

Final Thoughts

An added benefit of setting up a firewall this way is that it will let you route to other computers on your LAN over your VPN.  Hope you all enjoy this and let me know how things work out for you.

FreeNas Naughty NTP


Seems like FreeNas 9.2.0 and earlier shipped with ntpd enabled and listening for monitoring requests.  Recently got a notice that one of my machines was part of a DDoS attack on port 123 (NTP).  It was a NTP reflection attack and my freenas box was responding to the requests.

I should say, this will only affect you if your FreeNas box is internet facing or if you have port 123 forwarded to it.

So for a fix, ssh’d in and edited /etc/ntp.conf

Added:

disable monitor

Then stopped and restarted ntpd

service ntpd stop

service ntpd start

To verify that the problem is fixed,

ntpdc -n -c monlist 127.0.0.1

You should get a no connection error…or something like that.

Then, we want to:

mount -uw /

vi /conf/base/etc/ix.rc.d/ix-ntpd

scroll down to where you see EOF and right before that insert (i in vi):

disable monitor

Hit esc to exist insert mode and type :wq enter to save and quit

Make the file system read only again

mount -ur /

That should do it.  Your other option is to upgrade to 9.2.1 or higher.