Sysadmin Notes

Jump to: navigation, search

Notes on system administration gotchas, snafus, etc.


Linux Unable to See/Use My 2nd CPU Core

CPU #1 not responding - cannot use it.
powernow-k8: Found 1 AMD Athlon(tm) 64 X2 Dual Core Processor 5600+   
processors (1 cpu cores) (version 2.00.00)

Santa brought me an HP Pavilion a6230n for Christmas 2007. It came with an AMD Athlon(tm) 64 X2 Dual Core Processor 5600+ stepping 03, 3GB of RAM, 400 GB SATA, decent nVidia graphics, etc. When I installed both FC7 and FC8, I received the above warning messages in my log files. Linux was unable to use the 2nd core. Fixing this involved installing a BIOS update, for this model, that I downloaded from HP. Unfortunately, the BIOS update would only run under Windows Vista so I had to re-install Vista on the machine.

Fun with Dovecot IMAP, Postfix, Squirrel Mail, Apache, etc.

I recently lost a disk and paid to have it recovered. When I re-loaded my home directory, some of my file permissions got mangled.

When I tried to log into my Webmail facility, I received the following error message:

ERROR: Could not complete request.
Query: LSUB "" "*"
Reason Given: Permission denied

Fixing this problem involved finding all the Dovecot files in my home directory and changing the ownership and group back to my user rather than root.

Look for .subscriptions and .imap in your Mail directory (or mail) and check the ownership and permissions.

Fun with Java Keystores: How to import an existing private key and cert into a Java Keystore

I use SSL outside of Java for many things including Web servers, LDAP, SSL programming, etc. Consequently, most of my systems already have private keys and x509 certs. Java's keytool program makes it nearly impossible (as far as I can tell) to import pre-existing keys and certs into an existing or new keystore. Plus, the various Java keystore GUIs are hard to use (I cannot figure them out) or are not open source.

I came across this web page that describes how to do so. I summarize here just in case this web page goes away. We assume the private key is in key.pem and the cert is in cert.pem (both are in PEM format).

Convert the key and cert from PEM format to DER format using openssl command

Use openssl to convert from PEM to DER format.

openssl pkcs8 -topk8 -nocrypt -in key.pem -inform PEM -out key.der -outform DER
openssl x509 -in cert.pem -inform PEM -out cert.der -outform DER

Put key and cert into a new Java Keystore

Use the class to take the key and cert and place it in a newly constructed JKS keystore. I modified the ImportKey java source to use the keystore password changeit and to use the key alias importkey and to save the resulting keystore in the file jetty.keystore

java ImportKey key.der cert.der

Getting Apache to work with Subversion and LDAP Authentication

I use both LDAP and NIS internally for directory services although I am migrating everything to LDAP. I recently installed and created a Subversion repository for my source code. I wanted Apache 2.x to provide access to the repository and I wanted to use LDAP (LDAPS actually) to authenticate users so that I did not have to do yet another .htusers file.

To get everything working, I placed the following directives in conf.d/subversion.conf. I already had https working on the server and configured it to load SSL at startup.

LoadModule dav_svn_module     modules/
LoadModule authz_svn_module   modules/
<Location /repos>
   DAV svn
   SVNPath /columbia4/repos
   AuthType Basic
   AuthName "Subversion repository"
   AuthBasicProvider ldap
   AuthzLDAPAuthoritative on
   Require valid-user
   AuthzLDAPMethod ldap
   AuthLDAPUrl ldaps://,dc=org?uid

Notes for Administering OpenNMS

OpenNMS Admin Notes

Root Login on FC10 with Gnome 2.24 and FC11 with Gnome 2.26

The geniuses at Fedora and Gnome have disabled root login on the console in Fedora Core 10 (FC10). No matter what I did, I was unable to "fix" this using the docs I found on the web. On various Fedora mailing lists, allowing or not allowing root login generated a lot of discussion.

I re-enabled root login on FC10 Gnome 2.24 by editing the file /etc/pam.d/gdm and commenting out the line that says user != root.

#auth       required user != root quiet

On FC11 with Gnome 2.26 (FC12, and FC13 with 2.30), there are even more files to edit. gdm-password and gdm-fingerprint also need to be edited to comment out the user/root clause.

How to Change Network Device Names in FC10 and FC 11

Network device names (e.g. eth0, eth1) are seemingly assigned in an arbitrary and capricious manner in recent Fedora distributions. I like for my network interfaces to be assigned in a more predictable manner such as: motherboard ethernet should be eth0 while PCI ethernet cards should be ordered sequentially based on their slot number.

In past Fedora distributions, /etc/modprobe.conf could be edited to reflect device naming preferences. I supposed that may still be true in Fedora 10 and 11. However, one can also edit udev rules in /etc/udev/rules.d. For example, in my new router/firewall box with four ethernet interfaces, the following file can be edited to reflect my desired ethernet interface numbering preferences. From 70-persistent-net.rules

# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:17:31:8c:6d:be", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:41:e5:43:56", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:13:46:77:fe:b3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"
# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:04:5a:6b:aa:58", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Add Support for Remote X11 in Gnome

I read email on a different machine using mutt. I want to be able to have pictures and other content displayed on my desktop display via X. The default settings in more recent versions of Gnome disable X11/TCP.

%ps -aef | grep Xorg
rdk       1494 30140  0 11:22 pts/7    00:00:00 grep -i Xorg
root      4115  4114  0 Aug01 tty1     00:06:29 /usr/bin/Xorg :0 -nr -verbose -auth /var/run/gdm/auth-for-gdm-tJGw2e/database -nolisten tcp vt1

To disable the nolisten tcp, edit /etc/gdm/custom.conf and add DisallowTCP=false under the [security] section. Log out and back in and that should do the trick.

Fun with OpenVPN

I spent a lot of time getting OpenVPN to work so that I could do site-to-site and allow mobile clients to connect. Throw in iptables, routing issues, and OpenSSL and it gets really entertaining.

The OpenVPN server is a multi-homed router with several internal subnets. I wanted the OpenVPN server to make the internal subnets available to clients and other sites. One big problem is that the OpenVPN server was to export the very subnet it resides on to clients and sites. This caused a potential routing loop to occur. I also wanted to be able to access the client's subnet as well hence site-to-site.

The configuration solution was to have the OpenVPN server bind to my ppp0 interface only rather than an internal yet routable subnet interface. I also used a private CA to generate x509v3 certs and keys. Using a private CA allows me to overload client authentication onto the PKI provided by SSL.

After getting things working, I noticed that any sustained traffic, over the VPN, would cause the link to lock up. Searching through the net, I tried tuning the end-system TCP implementations, setting MTUs, etc. Adding the statement mssfix 1400 greatly reduced the congestion on the VPN link by essentially "helping" the end-system TCP implementations tune their maximum-segment-size.

I also added the following entry to both sides /etc/sysctl.conf file to increase buffers available to UDP.

vm.min_free_kbytes = 8192

The server's config, stripped down is:

proto udp
port 1194
dev tun
mssfix 1400
local ppp0-ip-address
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server.crt
key /etc/openvpn/server.key
dh /etc/openvpn/dh1024.pem
ifconfig-pool-persist ipp.txt
keepalive 10 60
user nobody
group nobody
status openvpn-status.log
client-config-dir ccd
# tell the underlying system to route to this subnet via VPN
# this subnet is located at the other site; my subnets are "pushed"
# to the other side via the ccd config file

The client (other site) config file:

local my-own-ppp0-ip-address
proto udp
port 1194
dev tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client.crt
key /etc/openvpn/client.key
resolv-retry infinite
keepalive 10 60
remote server-ip-address 1194
user nobody
group nobody
# server side iroutes my net and make it available to others                    
# via CCD config                        

The client-specific config file, located on the OpenVPN server is:

# client-specific config that gets pushed to specific client
# filename must match hostname/cert-CN in order to get read
push "route"
push "route w.x.y.z"
# iroute tells the VPN daemon to route packets to 10.0.1.x 
# to this particular client

Fun With Nvidia Graphics Cards

I bought and installed a nice PCI Nvidia graphics card but had problems telling Xorg to use the new card. It turned out that I needed to specify the BusID of the video card in my xorg.conf file. Dmesg returns the BusID in one format (with a period) and the config file wants a slightly different format. The two formats are subtly different hence the cause of my consternation.

# dmesg | grep nvidia
nvidia 0000:03:00.0: PCI INT A -> Link[LN0A] -> GSI 19 (level, low) -> IRQ 19

Here is the working device section from my xorg.conf file. Note the use of colons separating the BusID components. Dmesg puts a period before the last 0.

Section "Device"
    Identifier     "Videocard0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID 	   "PCI:03:00:0"

How to Upgrade VMware Server 2.0 on Linux

I recently had to upgrade my VMware VMserver 2.0 to 2.0.2 to install Windows 7. Rather than re-invent the wheel and re-learn how to do this, I found this website which outlines the fairly simple procedure.

Building an ESXi 4.1 "Whitebox" on the Cheap

I recently built a "whitebox" to run ESXi 4.1. I used the following components:

  • Samsung OEM SATA DVD reader/burner $18.00
  • 2 SATA drives 1 TB 3.5” Seagate Barracuda 7200 RPM 32MB cache $120.00 MicroCenter
  • Hand-me-down case
  • Coolmast 350Watt $28.00 MicroCenter
  • Total $656 not including tax.

Installing Solaris 11 x86 on ESX 4.1

At the time of writing (12/2010), ESXi 4.1 did not "officially" support the newly released Solaris 11 (x86). After some trial and error, I was able to install Solaris 11 x86 in a VM running on ESXi 4.1 using the following hardware configuration. (Note that the default hardware for Solaris 10 x86 did not work for Solaris 11.)

Component Config
Disk IDE (0:0)
Video Generic 800x600 24-bit 16 MB video RAM
CD/DVD IDE (1:0)
NIC E1000 (same as in host)
CPU 1 AMD x64
Other All other default settings

SIP Registration Attempts are Pounding My SIP Server

There are some scripts, making the rounds on the Internet, that probe SIP servers for open and mis-configured extensions. One such script performs two SIP registration attempts and then moves on to the next host if the SIP server is properly configured. Another script, what I call the "broken" script, encounters my SIP server and then just pounds away ad nausea until I manually put a filter rule to stop it. The "broken" script pounds so hard that it acts as a denial-of-service attack against my network.

In searching for a way to automatically block these attempts, I came across two such approaches.

  • fail2ban is a script that will scan your SIP server log file and then automatically add a rule to the local iptables config file.
  • iptables recent module can be configured to block various packets after some number are seen from a particular IP address

I do not like the first approach for several reasons. First, fail2ban is a separate script that must scan log files and then create and add an iptables rule. Given how many morons are out on the Internet, I'd end up with lots and lots of rules over time. And, I want to block the SIP registration attempts at my border firewall/router so that they do not even enter my network (my SIP server does not run on my border router/firewall).

The second approach seemed thus seemed the easiest. The downside is that there is some increased processing for each packet although the majority of the processing is only performed on SIP packets. Here are the iptables rules I added.

iptables -A FORWARD -p udp --dport 5060 -m recent --set --name SIPPKT
iptables -A FORWARD -p udp --dport 5060 -m recent --update --seconds 30 --hitcount 20 --name SIPPKT -j DROP

The above rules first create a set called SIPPKT and the then watch for greater than 20 matches in the last 30 seconds. If more than 20 matches come from the same IP address, in the last 30 seconds, subsequent matches are dropped silently on the floor. You could choose to log these drops with an additional rule but then your log file would be clogged entries. Your SIP server log file will probably still contain the first 20 attempts so you can still detect that you have been attacked.

One gotcha that you have to watch out for. You should place these rules before any state module rules that may let packets through the firewall if they have been previously accepted. That is, if you let a few of these SIP requests in, reply back to them essentially saying SIP-off, then future SIP packets will continue to be accepted from the offending machine because state rules permit them.

How do I use ImageMagick to correct the orientation of a directory of images?

Digital cameras now include orientation meta-information. Depending on the how you get your images off your camera, they may or may not be oriented correctly for viewing. If the images are not auto oriented, you can use the following commands to correct the orientation. However, the file access dates will be modified.

In csh, use the following command.

% foreach i (*.JPG *.jpg)
echo $i
convert -auto-orient $i $i

Deploying IPv6

I recently deployed IPv6 within my network and connected to the IPv6 Internet. While I studied the IPng protocol development effort in graduate school, it had been a while since working with IPv6. Plus, studying the protocol and deploying it in a working network are often two vastly different efforts. Consequently, I bought and read Running IPv6 by Iljitsch van Beijnum. The book is a pretty good tutorial for bring up and running IPv6 in a heterogeneous environment. Originally written in 2005, the book is starting to get a tad bit dated. Included below are notes from my deployment effort roughly divided into several categories.

Connecting to the IPv6 Internet and Obtaining Addresses

Ipv6 sage.png

IPv6 is currently deployed in parallel with IPv4 on the Internet. That is, you can consider IPv4 and IPv6 as separate, distinct protocols that are like ships in the night. Since my own ISP does not yet support IPv6 (that is, they do not route IPv6 and do not have their own address block), I was faced with the task of finding an IPv6 tunnel provider. Hurricane Electric (HE) provides free tunnel service as well as IPv6 address block assignment. I liked the price and heard good things about their IPv6 support tools. They even have an unofficial certification process that strives to educate network administrators. After deploying IPv6, I was able to achieve the rank of guru in their certification process.

Each tunnel is a point-to-point link, over (or encapsulated in) IPv4 and assigned a /64 IPv6 address. While it would seem a waste to assign a 64-bit subnet for just two endpoints, there are no shortages of addresses in IPv6 and the IPv6 community has reached consensus that netblocks are not assigned smaller than /64. My end of the tunnel was assigned ::2/64 and HE's was assigned ::1/64. In addition, I was assigned an IPv4 address for HE's side of the tunnel and I in turn gave them the IPv4 address of my side of the tunnel.

In addition to the tunnel address assignment, HE assigns a /64 to each registered user for use within their network. Because I have 4+ subnets in my network, I requested and received a /48 address assignment -- 2001:470:e499::/48. With this ::/48 assignment, I have 80 bits (128-48) of addresses I can potentially assign. By convention, the last 64-bits are for the host-portion of addresses so I can have 16-bits or 65,536 subnets!

Sage certificate.png

Path MTU

Because IPv6 packets, leaving my network, are encapsulated in an IPv4 tunnel, and because IPv6 headers are larger, I had to decrease the MTU through the tunnel and push that decrease back out to IPv6 clients. I hardcoded 1280 as the path MTU because ICMPv6 Path MTU discovery packets may have been getting dropped somewhere over the Internet and I was experiencing strange TCP connection problems when attempting to browse websites like YouTube. Once I lowered my tunnel MTU to 1280 (and propagated that MTU to IPv6 clients via router advertisement), my TCP connection problems disappeared.

If you are going to let your path MTU float, you need to make sure your firewall/router will generate, accept, and forward ICMPv6 Packet Too Big (Type=2) messages.


I run my own DNS server and use Zoneedit for backup. Thus, it was pretty straightforward to add support for IPv6. IPv6 addresses are specified using AAAA (quad-A) records. In my internal and external facing zones, I simply added AAAA records for hosts that had static IPv6 addresses. I also needed to tell my DNS server software to explicitly bind to IPv6 interfaces as it did not do that by default. That is, I want my DNS server to answer with both A and AAAA records whether its contacted via IPv4 or IPv6.

For hosts that use EUI-64 derived addresses, I will eventually add support for dynamic DNS updates either via DHCPv6 or via hosts updating their addresses directly.

Setting up reverse records was fairly simple both for internal and external queries.

First, I set up an zone for the reverse lookup (PTR) records. Since my ::/48 prefix is 2001:0470:e499, my zone is named Records are stored in reverse-nibble format -- each 4-bits is stored as a hexadecimal digit separated by colons, in reverse order.

The above configuration works fine for internal reverse lookups but how do I publish this info to the wider Internet despite not technically owning the address space? HE provides for reverse DNS delegation. In the process of registering my IPv6 tunnel, I provided the DNS name, IPv4, and IPv6 address of my DNS server for reverse-DNS delegation. When a PTR query is received for my address block, their DNS server will delegate the query/response to/from my DNS server. You just have to make sure that your firewall rules allow queries from the HE.

Firewall Rules

Because IPv6 packets are tunneled through IPv4, I had to add a few new entries in my IPv4 firewall rule set. IPv4 packets, that carry IPv6 messages, set their protocol field to 41. Also, each endpoint of my IPv4/IPv6 tunnel has a static IPv4 address assigned to it so I could further constrain my firewall rules.

The rules below instruct IPv4 iptables to allow IPv6 tunnel traffic to/from HE. The variable $IPTABLES is set to the path and name of the iptables binary and the $V6TUNNELSERV variable is set to the IPv4 address of the other end of the tunnel. $EXT_IF is the particular interface that the tunnel traverses (e.g. sit1).


The eventual goal (probably a long way off in the future) is to turn off IPv4 and migrate everything to IPv6. Any of the services I currently make available via IPv4 I also want to make available via IPv6. Consequently, I had to create a separate IPv6 firewall rules set to permit essentially the same services to traverse my firewall/router via IPv6. Since I'm using globally unique IPv6 addresses everywhere (even for internal machines), there is no need for network address translation (NAT). Regarding IPv6 firewall rules, one has to make sure to allow ICMPv6 to/from the Internet in order for Path MTU discovery to work. You still may wish to filter out ICMPv6 redirects though. I filter redirects coming from my external interface (my tunnel) but allow them internally. Below are statements from my firewall script that implement this policy.

# dont accept redirects from external interfaces sit0 and sit1
echo "0" > /proc/sys/net/ipv6/conf/sit0/accept_redirects
echo "0" > /proc/sys/net/ipv6/conf/sit1/accept_redirects

I am not going to post my IPv6 firewall rule set here because doing so might give away subtle configuration information that might aid bad guys. If you'd like a copy of my IPv6 firewall rule set, contact me via email. Let me just say that it was very similar to my IPv4 rule set.

Addressing Architecture

Each IPv6 address is 128-bits long -- 16 8-bit bytes. See this link for more information on IPv6 addresses. HE assigned me a /48 which means that I was assigned the first 48-bits and have control over the subsequent 80-bits. Since IPv6 standardized on /64 for end-system addresses, I was left with 16-bits for subnet assignment.

In IPv4, it is common to use RFC 1918 private addresses (e.g. for communication inside of an enterprise. IPv6 originally included an address scoping mechanism for site-local communication. I chose not utilize site-local addresses and instead have deployed globally-unique IPv6 addresses throughout my network. I did so because I can easily duplicate the functionality within my firewall via rules designed to protect internal systems. Besides, there is no network-address translation in IPv6. Site-local addresses are distinguished by the prefix fec0::/10.

For now, I'm distinguishing my subnets via bit-shifting. My subnets are 0x01, 0x02, 0x04, 0x08, etc. I did so because its easier to visually pattern-match these addresses when looking at network traces. Thus, the first few subnets are:

  • 2001:0470:e499:0001::/64
  • 2001:0470:e499:0002::/64
  • 2001:0470:e499:0004::/64
  • 2001:0470:e499:0008::/64

Since by convention, we assign ::1 to routers, the IPv6 address for the router on the above subnets is:

  • 2001:0470:e499:0001::1/64
  • 2001:0470:e499:0002::1/64
  • 2001:0470:e499:0004::1/64
  • 2001:0470:e499:0008::1/64

I'm using static IPv6 addresses for important systems like servers and some desktops. The rest of my systems will configure their IPv6 addresses using stateless address configuration. Those systems derive their globally unique address by first using EUI-64 to acquire a unique host-portion based on their IEEE MAC-layer address. Then, those systems listen for router advertisements and learn the ::/64 prefix for their subnet. Via router advertisements, they also learn the IPv6 address of their router.

Stateless address-configuration, however, does not provide the IPv6 address of DNS servers. Thus, that has to either be configured staticly or via some other dynamic means. Consequently, I am deploying DHCPv6 in order to answer queries about DNS servers. However, even in the absence of DNS over IPv6, clients can still use IPv4 to resolve IPv4 and IPV6 addresses.

Router Advertisement and Discovery

IPv6 includes several mechanisms to reduce configuration overhead. One such mechanism is router discovery. Since the distro running on my firewall/router did not include a router advertisement daemon, I had to download and install one via my distro's software update mechanism. radvd is a popular router advertisement daemon available on many UNIX variants.

I use this daemon to disseminate router and link MTU information. Here is the config file snippet for one of my subnets. Note the MTU assignment.

interface eth1
	AdvSendAdvert on;
	MinRtrAdvInterval 30;
	MaxRtrAdvInterval 100;
	AdvLinkMTU 1280;
	prefix 2001:0470:e499:0001::/64
		AdvOnLink on;
		AdvAutonomous on;
		AdvRouterAddr on;

Applications -- Email, Web, VoIP, etc.

Other than explicitly configuring my DNS server software to listen on IPv6 addresses, all my other applications were already configured to listen on IPv6 interfaces. Apache, Postfix, Sendmail, IMAP daemons, etc. were already listening on IPv6. However, one major application that I'm using is not yet IPv6 capable. The version of asterisk I use does not support IPv6 however newer versions do. Once I upgrade, I will enable and test IPv6. Other than that, I have not encountered any specific IPv6 application issues. I may discover otherwise but my web and email servers have been tested with IPv6 and passed. As of this writing, I have not yet received spam via IPv6 but I have received some legitimate email over IPv6.

Wireless and Mobile Devices

I have a separate wifi subnet within my network utilizing home-office-grade routers each configured to work in access-point (AP) mode. For the most part, IPv6 over wifi works just fine. However, I have noticed quite a bit of spurious IPv6-related log messages on my router/firewall box. In essence, while I've put these wifi devices in AP mode, they are still sending out IPv6 router-advertisements with different configuration information that I desire (e.g. MTU). I'm waiting confirmation from the manufacturer for configuration help.

iPhones, iPads, and various Apple i devices do support IPv6 but do not currently allow one to inspect nor change any IPv6 related configuration information. There are some entries in the App Store that provide IPv6 configuration information and allow one to ping6 devices. Apple and PC laptops running recent versions of Windows and MacOS work just fine with stateless address configuration.

Systemd Breaks OpenVPN Startup

The new systemd facility on recent distros of Linux breaks the starting up of OpenVPN. In the system log, you'll see something like:

Oct  4 13:23:35 peachtree openvpn[2761]: Error: private key password verification failed

The reason this error occurs is that systemd does not let openvpn, by default, interact with the shell to prompt for a password to access the private key. One way to start openvpn is to do so from command-line, as root:

openvpn --config /etc/openvpn/client.conf 

The above command starts openvpn and creates the VPN tunnel. I have not configured an OpenVPN tunnel through the NetworkManager GUI.

Another way to workaround the startup bug, is to edit /etc/init.d/openvpn and add the following:


Place the above line at the start of the script prior to setting the value of other variables and prior to including other files.

FreeNAS, Samba, LDAP, and Integrating with Existing Domain

I recently built a FreeNAS 8.0.3p1 machine to play with and use as a backup server for Windows machines. I currently have an LDAP server for my network and am running Samba on my fileserver. Everything works fine.

When I first configured the LDAP client on FreeNAS, I had problems getting it to authenticate properly. Here are my LDAP client params for FreeNAS to talk to my OpenLDAP server. Beyond the usual hostname and base DN, I needed to set the root bind dn and bind password.

  • User Suffix: ou=People
  • Group Suffix: ou=Group
  • Password Suffix: ou=People
  • Machine Suffix: ou=Hosts
  • Encryption Mode: SSL
  • Self-signed cert: for this field, I finally figured out that I should put the root cert for CACert since I use CACert for x509 certs within my network.
  • No change to aux parameters

Once I got the above set, I was able to authenticate users via LDAP and could log into the FreeNAS box with LDAP users.

However, when trying to mount windows shares, using LDAP users, I kept getting an error (discovered in log.smbd) that said that my primary group SID for my FreeNAS server did not match the primary group SID for my LDAP user. In my LDAP directory, each existing user has a field called sambaSID that gives the SID for my existing Samba server running on my main fileserver. If I ran net getlocalsid on the FreeNAS box, the SID that was reported was new. How to set the local SID? Using net setlocalsid would not make permanent changes.

It turns out, that the FreeNAS samba machine created an entry in my LDAP directory called sambaDomainName=NAS (my box is named nas). In this entry, there is an attribute called sambaSID and this value was causing the problems. Using an LDAP browser, I logged into the LDAP server and edited this entry and changed the SID to match my existing Samba server and the SIDs that were contained in my LDAP user entries. Now, Windows clients could mount my FreeNAS shares using their LDAP user. Yuck.

Fun with IPSEC, L2TP, VPNs, and iOS Devices

iOS devices like iPads and iPhones do not have an OpenVPN client easily available for them. However, I'd like to be able to VPN into my network from say an iPad. The solution is to configure L2TP tunnels using IPSEC since iOS devices support this type of VPN out of the box. Configuring this on my firewall/router was actually fairly straightforward once I assembled all the pieces.

Modern Linux distro have all the pieces you need. To configure this style of VPN'ing, you'll need the following components available from your distro's package repository.

  • openswan
  • xl2tpd
  • ppp

Once installed, configuring this morass of code requires a few config files.

  • /etc/ipsec.conf
  • /etc/ipsec.d/
  • /etc/ipsec.d/file.secrets
  • /etc/xl2tpd/xl2tpd.conf
  • /etc/ppp/chap-secrets
  • /etc/ppp/options.xl2tpd
  • IPtables rules to allow incoming IPsec/L2TP VPNs

Don't forget to have the xl2tpd daemon and ipsec start automatically at system boot. How you do this depends on your distro.

Also, don't forget to set the various sysctl entries you'll need. If you run the following command, it will let you know what you need to do.

ipsec verify


config setup

conn roadwarrior
     # apple iOS does not send delete notify so we need dead peer detection


  • Replace X.X.X.X with your external facing IP address
  • Dead peer detection is nice to have so you can re-connect faster after you iPhone/iPad goes to sleep and wakes back up.
  • Left side is your VPN server, right side is your client (e.g. iPhone/iPad).
  • The list of IP addresses under virtual-private are those to allow from the client size with exceptions prefaced by ! From the man page: the address ranges that may live behind a NAT router through which a client connects. This value is usually set to all the RFC-1918 address space, excluding the space used in the local subnet behind the NAT (An IP address cannot live at two places at once).
  • Tabs are important! Yuck.


X.X.X.X %any: PSK "some random arbitrary secret"


  • X.X.X.X is the same external IP address that placed in /etc/ipsec.conf
  • This pre-shared key is used for IPsec encryption and is different from the password used for PPP authentication.


[lns default]
ip range =
local ip =
; leave chap unspecified for maximum compatibility with windows, iOS, etc
; require chap = yes
;refuse chap = yes
refuse pap = yes
require authentication = yes
name = MyVPN
ppp debug = yes
pppoptfile = /etc/ppp/options.xl2tpd
length bit = yes


  • Experiment with the pap and chap settings if you have trouble connecting. Leave chap unspecified for maximum compatibility though.
  • PPP is used for the actual authentication for the layer-2 tunnel setup. You should not need the ppp daemon running at system boot. The xl2tpd daemon takes care of starting the PPP daemon which will then handle authentication and create the interface.
  • I use the address range 172.30.30/24 for my PPP interfaces.


ms-dns  Y.Y.Y.Y
asyncmap 0
name l2tpd
lcp-echo-interval 30
lcp-echo-failure 4
idle 1800
mtu 1280
mru 1280
connect-delay 5000


  • Y.Y.Y.Y is the IPv4 address of your nameserver.
  • I'm not yet tunneling over IPv6.


# client	server	secret				IP addresses
someuser	*	"a diffrent secret"		*


  • Tabs are important! Yuck again.
  • This secret is what you put into your client for password. Typically, you configure your client IPsec encryption key as the secret.


# accept packets from ppp l2tp/ipsec VPNs                                       

# let out packets from ppp l2tp/ipsec VPNs                                      

# accept l2tp/ipsec VPN tunnels                                                 
$IPTABLES -A INPUT -i eth0 -p udp -m policy --dir in --pol ipsec -m udp --dport 1701 -j ACCEPT
$IPTABLES -A INPUT -i eth0 -p udp --dport 4500 -j ACCEPT
$IPTABLES -A INPUT -i eth0 -p udp --dport 500 -j ACCEPT

# add this rule so that you can MASQUERADE/NAT for the clients
# this way, they can route all packets through the VPN if they desire


  • Once established, the VPN packets will be routed to/from ppp interfaces. So, you need to add rules to let ppp traffic in/out. You should be safe to assume that any ppp interface that is created is for a valid authenticated tunnel.
  • You also need to add rules, for your external interface, that allow packets into the IPsec key daemon often called "pluto".
  • In my case, eth0 is my external interface that VPN traffic is passing in/out of.

Its an ugly mess but it works!

Deploying a DHCP Slave Server for Failover

Turning on and configuring a slave or failover DHCP server is pretty easy to do with the ISC DHCP daemon. Assuming Linux and ISC DHCP daemon.

  • Install ISC DHCPD on slave/failover server taking care to make sure both servers are running similar or same version
  • Add a failover peer clause:
failover peer "failover-partner" {
  port 519;
  peer address;
  peer port 520;
  max-response-delay 60;
  max-unacked-updates 10;
  mclt 3600;
  split 128;
  load balance max seconds 3;

omapi-port 7911;
omapi-key omapi_key;
key omapi_key {
  algorithm hmac-md5;
  secret someveryveryverylongsecret;

  • Within each subnet you want the secondary to provide failover service, add a pool clause to that subnet declaration.
 pool {
    failover peer "failover-partner";
    range x.y.z.a x.y.z.b;
  • On the failover/slave DHCP server, copy the master/primary's dhcpd.conf file
  • Add failover peer statement taking care to remove some of the statements:
failover peer "failover-partner" {
  port 520;
  peer address;
  peer port 519;
  max-response-delay 60;
  max-unacked-updates 10;
  load balance max seconds 3;

omapi-port 7911;
omapi-key omapi_key;
key omapi_key {
  algorithm hmac-md5;
  secret someveryveryverylongsecret;
  • For each subnet that this slave is going to act as failover, add a pool clause.
 pool {
    failover peer "failover-partner";
    range x.y.z.a x.y.z.b;
  • Thats it; restart the DHCP daemons on both primary/master and slave/secondary. Use netstat to verify that a TCP connection is taking place between primary:519 and secondary:520.
  • Test by shutting down primary and observing if IP addresses are still issued. Or, wait for next failure and verify!

Configuring/Deploying a Replica (Slave) on Linux LDAP Server

Configuring the openldap SLAPD server is a real bear. And, the folks that support it via email, message boards, etc. are so charming and kind to non-LDAP people. But, given how dependent my network infrastructure is on LDAP, I decided that I had to deploy a slave/replica LDAP server. The incident that precipitated this decision was that my primary LDAP server went catatonic while on a weekend backpacking trip along the Georgia AT. Doing tech support, via cellphone, from a trail shelter was less than ideal. Fortunately, rebooting the machine solved the problem. With a secondary LDAP server, my users may never have known something died.

The primary LDAP server is the master or provider. The secondary LDAP server is a consumer or slave. To get a seconary/slave/consumer up and running, the process is roughly this:

  1. Prepare the primary server for slave connections by adding a new directory user just for the secondary
  2. Configure the primary server to load the sync provision software module
  3. Enable the syncprov module for each directory we wish to allow replication/sync'ing
  4. Install openldap packages on slave/replica server.
  5. Configure slave/replica server with credentials and config to sync to server
  6. Test

I found the following URL helpful for walking me roughly through the process:

But, I ran into permissions issues (dreaded error 50) when trying to add the sync provision module to my server. I ended up fixing those permissions problems by following the instructions in this URL.

Independently, I also configured the SSL parameters and general slapd server setup/initialization by following these instructions:

Personal tools