Editing Sysadmin Notes

Notes on system administration gotchas, snafus, etc.

== Linux Unable to See/Use My 2nd CPU Core ==

<pre>
CPU #1 not responding - cannot use it.
powernow-k8: Found 1 AMD Athlon(tm) 64 X2 Dual Core Processor 5600+   
processors (1 cpu cores) (version 2.00.00)
</pre>

Santa brought me an HP Pavilion a6230n for Christmas 2007.  It came with an AMD Athlon(tm) 64 X2 Dual Core Processor 5600+ stepping 03, 3GB of RAM, 400 GB SATA, decent nVidia graphics, etc.  When I installed both FC7 and FC8, I received the above warning messages in my log files.  Linux was unable to use the 2nd core.  Fixing this involved installing a BIOS update, for this model, that I downloaded from HP.  Unfortunately, the BIOS update would only run under Windows Vista so I had to re-install Vista on the machine.

== Fun with Dovecot IMAP, Postfix, Squirrel Mail, Apache, etc. ==

I recently lost a disk and paid to have it recovered.  When I re-loaded my home directory, some of my file permissions got mangled.

When I tried to log into my Webmail facility, I received the following error message:
<pre>
ERROR: Could not complete request.
Query: LSUB "" "*"
Reason Given: Permission denied
</pre>

Fixing this problem involved finding all the Dovecot files in my home directory and changing the ownership and group back to my user rather than root.

Look for ''.subscriptions'' and ''.imap'' in your ''Mail'' directory (or ''mail'') and check the ownership and permissions.

== Fun with Java Keystores: How to import an existing private key and cert into a Java Keystore ==

I use SSL outside of Java for many things including Web servers, LDAP, SSL programming, etc.  Consequently, most of my systems already have private keys and x509 certs.  Java's ''keytool'' program makes it nearly impossible (as far as I can tell) to import pre-existing keys and certs into an existing or new keystore.  Plus, the various Java keystore GUIs are hard to use (I cannot figure them out) or are not open source.

I came across [http://www.agentbob.info/agentbob/79-AB.html?branch=1&language=1 this] web page that describes how to do so.  I summarize here just in case this web page goes away.  We assume the private key is in ''key.pem'' and the cert is in ''cert.pem'' (both are in PEM format).

==== Convert the key and cert from PEM format to DER format using openssl command ====

Use openssl to convert from PEM to DER format.
<pre>
openssl pkcs8 -topk8 -nocrypt -in key.pem -inform PEM -out key.der -outform DER
openssl x509 -in cert.pem -inform PEM -out cert.der -outform DER
</pre>

==== Put key and cert into a new Java Keystore ====

Use the [http://www.agentbob.info/agentbob/80/version/default/part/AttachmentData/data/ImportKey.java ImportKey.java] class to take the key and cert and place it in a newly constructed JKS keystore.  I modified the ImportKey java source to use the keystore password ''changeit'' and to use the key alias ''importkey'' and to save the resulting keystore in the file ''jetty.keystore''
<pre>
java ImportKey key.der cert.der
</pre>

== Getting Apache to work with Subversion and LDAP Authentication ==

I use both LDAP and NIS internally for directory services although I am migrating everything to LDAP.  I recently installed and created a Subversion repository for my source code.  I wanted Apache 2.x to provide access to the repository and I wanted to use LDAP (LDAPS actually) to authenticate users so that I did not have to do yet another .htusers file.

To get everything working, I placed the following directives in ''conf.d/subversion.conf''.  I already had ''https'' working on the server and configured it to load SSL at startup.

<pre>
LoadModule dav_svn_module     modules/mod_dav_svn.so
LoadModule authz_svn_module   modules/mod_authz_svn.so
<Location /repos>
   DAV svn
   SVNPath /columbia4/repos
   AuthType Basic
   AuthName "Subversion repository"
   AuthBasicProvider ldap
   AuthzLDAPAuthoritative on
   Require valid-user
   AuthzLDAPMethod ldap
   AuthLDAPUrl ldaps://columbia.krupczak.org/dc=krupczak,dc=org?uid
</Location>
</pre>

== Notes for Administering OpenNMS ==

[http://www.opennms.org/index.php/Admin_Notes OpenNMS Admin Notes]

== Root Login on FC10 with Gnome 2.24 and FC11 with Gnome 2.26 ==

The geniuses at Fedora and Gnome have disabled root login on the console in Fedora Core 10 (FC10).  No matter what I did, I was unable to "fix" this using the docs I found on the web.  On various Fedora mailing lists, allowing or not allowing root login generated a lot of discussion.

I re-enabled root login on FC10 Gnome 2.24 by editing the file ''/etc/pam.d/gdm'' and commenting out the line that says user != root.

<pre>
#auth       required    pam_succeed_if.so user != root quiet
</pre>

On FC11 with Gnome 2.26 (FC12, and FC13 with 2.30), there are even more files to edit.  ''gdm-password'' and ''gdm-fingerprint'' also need to be edited to comment out the user/root clause.

== How to Change Network Device Names in FC10 and FC 11 ==

Network device names (e.g. ''eth0'', ''eth1'') are seemingly assigned in an arbitrary and capricious manner in recent Fedora distributions.  I like for my network interfaces to be assigned in a more predictable manner such as: motherboard ethernet should be ''eth0'' while PCI ethernet cards should be ordered sequentially based on their slot number.

In past Fedora distributions, ''/etc/modprobe.conf'' could be edited to reflect device naming preferences.  I supposed that may still be true in Fedora 10 and 11.  However, one can also edit ''udev'' rules in ''/etc/udev/rules.d''.  For example, in my new router/firewall box with four ethernet interfaces, the following file can be edited to reflect my desired ethernet interface numbering preferences.  From ''70-persistent-net.rules''

<pre>
# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:17:31:8c:6d:be", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:41:e5:43:56", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:13:46:77:fe:b3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"
# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:04:5a:6b:aa:58", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
</pre>

== Add Support for Remote X11 in Gnome ==

I read email on a different machine using mutt.  I want to be able to have pictures and other content displayed on my desktop display via X.  The default settings in more recent versions of Gnome disable X11/TCP.

<pre>
%ps -aef | grep Xorg
rdk       1494 30140  0 11:22 pts/7    00:00:00 grep -i Xorg
root      4115  4114  0 Aug01 tty1     00:06:29 /usr/bin/Xorg :0 -nr -verbose -auth /var/run/gdm/auth-for-gdm-tJGw2e/database -nolisten tcp vt1
</pre>

To disable the ''nolisten tcp'', edit ''/etc/gdm/custom.conf'' and add ''DisallowTCP=false'' under the ''[security]'' section.  Log out and back in and that should do the trick.

== Fun with OpenVPN ==

I spent a lot of time getting OpenVPN to work so that I could do site-to-site ''and'' allow mobile clients to connect.  Throw in ''iptables'', routing issues, and OpenSSL and it gets really entertaining.

The OpenVPN server is a multi-homed router with several internal subnets.  I wanted the OpenVPN server to make the internal subnets available to clients and other sites.  One big problem is that the OpenVPN server was to export the very subnet it resides on to clients and sites.  This caused a potential routing loop to occur.  I also wanted to be able to access the client's subnet as well hence site-to-site.

The configuration solution was to have the OpenVPN server bind to my ppp0 interface only rather than an internal yet routable subnet interface.  I also used a private CA to generate x509v3 certs and keys.  Using a private CA allows me to overload client authentication onto the PKI provided by SSL.

After getting things working, I noticed that any sustained traffic, over the VPN, would cause the link to lock up.  Searching through the net, I tried tuning the end-system TCP implementations, setting MTUs, etc.  Adding the statement ''mssfix 1400'' greatly reduced the congestion on the VPN link by essentially "helping" the end-system TCP implementations tune their maximum-segment-size.

I also added the following entry to both sides ''/etc/sysctl.conf'' file to increase buffers available to UDP.
<pre>
vm.min_free_kbytes = 8192
</pre>

The server's config, stripped down is:
<pre>
proto udp
port 1194
dev tun
mssfix 1400
local ppp0-ip-address
tls-server
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server.crt
key /etc/openvpn/server.key
dh /etc/openvpn/dh1024.pem
server 172.30.0.0 255.255.255.0
client-to-client
ifconfig-pool-persist ipp.txt
keepalive 10 60
ping-timer-rem
comp-lzo
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
client-config-dir ccd
# tell the underlying system to route to this subnet via VPN
# this subnet is located at the other site; my subnets are "pushed"
# to the other side via the ccd config file
route 10.0.1.0 255.255.255.0
</pre>

The client (other site) config file:
<pre>
client
local my-own-ppp0-ip-address
proto udp
port 1194
dev tun
tls-client
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client.crt
key /etc/openvpn/client.key
resolv-retry infinite
keepalive 10 60
ping-timer-rem
remote server-ip-address 1194
comp-lzo
user nobody
group nobody
persist-key
persist-tun
# server side iroutes my net and make it available to others                    
# via CCD config                        
</pre>

The client-specific config file, located on the OpenVPN server is:
<pre>
# client-specific config that gets pushed to specific client
# filename must match hostname/cert-CN in order to get read
push "route 10.0.0.0 255.255.255.0"
push "route w.x.y.z 255.255.255.240"
# iroute tells the VPN daemon to route packets to 10.0.1.x 
# to this particular client
iroute 10.0.1.0 255.255.255.0
</pre>

== Fun With Nvidia Graphics Cards ==

I bought and installed a nice PCI Nvidia graphics card but had problems telling Xorg to use the new card.  It turned out that I needed to specify the BusID of the video card in my ''xorg.conf'' file.  ''Dmesg'' returns the ''BusID'' in one format (with a period) and the config file wants a slightly different format.  The two formats are subtly different hence the cause of my consternation.

<pre>
# dmesg | grep nvidia
nvidia 0000:03:00.0: PCI INT A -> Link[LN0A] -> GSI 19 (level, low) -> IRQ 19
</pre>

Here is the working device section from my ''xorg.conf'' file.  Note the use of colons separating the BusID components.  ''Dmesg'' puts a period before the last 0.

<pre>
Section "Device"
    Identifier     "Videocard0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID 	   "PCI:03:00:0"
EndSection
</pre>

== How to Upgrade VMware Server 2.0 on Linux ==

I recently had to upgrade my VMware VMserver 2.0 to 2.0.2 to install Windows 7.  Rather than re-invent the wheel and re-learn how to do this, I found this website which outlines the fairly simple procedure.

[http://www.cyberciti.biz/tips/rhel-centos-upgrade-vmware-server.html http://www.cyberciti.biz/tips/rhel-centos-upgrade-vmware-server.html]

== Building an ESXi 4.1 "Whitebox" on the Cheap ==

I recently built a "whitebox" to run ESXi 4.1.  I used the following components:

* AMD Athlon II X4 640 Propus 3.0GHz 4 x 512KB L2 Cache Socket AM3 95W Quad-Core Desktop Processor ADX640WFGMBOX $100.00 http://www.newegg.com/Product/Product.aspx?Item=N82E16819103871	
					
* ASUS M4A785TD-V EVO AM3 AMD 785G HDMI ATX AMD Motherboard $100.00 http://www.newegg.com/Product/Product.aspx?Item=N82E16813131398						

* Kingston 4GB 240-Pin DDR3 1066 (PC3 8500) (4 x 4GB) $260.00 http://www.newegg.com/Product/Product.aspx?Item=N82E16820134927						

* Intel EXPI9301CTBLK 10/100/1000Mbps PCI-Express Network Adapter 1 x RJ45 $30.00 http://www.newegg.com/Product/Product.aspx?Item=N82E16833106033						

* Samsung OEM SATA DVD reader/burner $18.00							

* 2 SATA drives	1 TB 3.5” Seagate Barracuda 7200 RPM 32MB cache  $120.00 MicroCenter						

* Hand-me-down case								

* Coolmast 350Watt $28.00 MicroCenter

* Total $656 not including tax.

== Installing Solaris 11 x86 on ESX 4.1 ==

At the time of writing (12/2010), ESXi 4.1 did not "officially" support the newly released Solaris 11 (x86).  After some trial and error, I was able to install Solaris 11 x86 in a VM running on ESXi 4.1 using the following hardware configuration.  (Note that the default hardware for Solaris 10 x86 did not work for Solaris 11.)

{| border="1"
! Component 
! Config

|-
| align="center" | Disk 
| align="center" | IDE (0:0) 
|-
| align="center" | Video
| align="center" | Generic 800x600 24-bit 16 MB video RAM
|-
| align="center" | CD/DVD
| align="center" | IDE (1:0)
|-
| align="center" | NIC
| align="center" | E1000 (same as in host)
|-
| align="center" | CPU
| align="center" | 1 AMD x64
|-
| align="center" | RAM
| align="center" | 1 GB
|-
| align="center" | Other
| align="center" | All other default settings
|}

== SIP Registration Attempts are Pounding My SIP Server ==

There are some scripts, making the rounds on the Internet, that probe SIP servers for open and mis-configured extensions.  One such script performs two SIP registration attempts and then moves on to the next host if the SIP server is properly configured.  Another script, what I call the "broken" script, encounters my SIP server and then just pounds away ad nausea until I manually put a filter rule to stop it.  The "broken" script pounds so hard that it acts as a denial-of-service attack against my network.

In searching for a way to automatically block these attempts, I came across two such approaches.

* [http://www.fail2ban.org/wiki/index.php/Asterisk fail2ban] is a script that will scan your SIP server log file and then automatically add a rule to the local iptables config file.
* iptables ''recent'' module can be configured to block various packets after some number are seen from a particular IP address

I do not like the first approach for several reasons.  First, fail2ban is a separate script that must scan log files and then create and add an iptables rule.  Given how many morons are out on the Internet, I'd end up with lots and lots of rules over time.  And, I want to block the SIP registration attempts at my border firewall/router so that they do not even enter my network (my SIP server does not run on my border router/firewall).  

The second approach seemed thus seemed the easiest.  The downside is that there is some increased processing for each packet although the majority of the processing is only performed on SIP packets.  Here are the iptables rules I added.

<pre>
iptables -A FORWARD -p udp --dport 5060 -m recent --set --name SIPPKT
iptables -A FORWARD -p udp --dport 5060 -m recent --update --seconds 30 --hitcount 20 --name SIPPKT -j DROP
</pre>

The above rules first create a set called ''SIPPKT''  and the then watch for greater than 20 matches in the last 30 seconds.  If more than 20 matches come from the same IP address, in the last 30 seconds, subsequent matches are dropped silently on the floor.  You could choose to log these drops with an additional rule but then your log file would be clogged entries.  Your SIP server log file will probably still contain the first 20 attempts so you can still detect that you have been attacked.
 
One gotcha that you have to watch out for.  You should place these rules '''before''' any ''state'' module rules that may let packets through the firewall if they have been previously accepted.  That is, if you let a few of these SIP requests in, reply back to them essentially saying SIP-off, then future SIP packets will continue to be accepted from the offending machine because state rules permit them.