Difference between revisions of "Sysadmin Notes"

From Krupczak.org
Jump to: navigation, search
(→‎Router Advertisement and Discovery)
(→‎Firewall Rules)
Line 353: Line 353:
 
echo "0" > /proc/sys/net/ipv6/conf/sit1/accept_redirects
 
echo "0" > /proc/sys/net/ipv6/conf/sit1/accept_redirects
 
</pre>
 
</pre>
 +
 +
I am not going to post my IPv6 firewall rule set here because doing so might give away subtle configuration information that might aid bad guys.  If you'd like a copy of my IPv6 firewall rule set, contact me via email.  Let me just say that it was very similar to my IPv4 rule set.
  
 
=== Addressing Architecture ===
 
=== Addressing Architecture ===

Revision as of 20:49, 11 September 2011

Notes on system administration gotchas, snafus, etc.

Contents

Linux Unable to See/Use My 2nd CPU Core 

CPU #1 not responding - cannot use it.
powernow-k8: Found 1 AMD Athlon(tm) 64 X2 Dual Core Processor 5600+   
processors (1 cpu cores) (version 2.00.00)

Santa brought me an HP Pavilion a6230n for Christmas 2007. It came with an AMD Athlon(tm) 64 X2 Dual Core Processor 5600+ stepping 03, 3GB of RAM, 400 GB SATA, decent nVidia graphics, etc. When I installed both FC7 and FC8, I received the above warning messages in my log files. Linux was unable to use the 2nd core. Fixing this involved installing a BIOS update, for this model, that I downloaded from HP. Unfortunately, the BIOS update would only run under Windows Vista so I had to re-install Vista on the machine.

Fun with Dovecot IMAP, Postfix, Squirrel Mail, Apache, etc.

I recently lost a disk and paid to have it recovered. When I re-loaded my home directory, some of my file permissions got mangled.

When I tried to log into my Webmail facility, I received the following error message: 

ERROR: Could not complete request.
Query: LSUB "" "*"
Reason Given: Permission denied

Fixing this problem involved finding all the Dovecot files in my home directory and changing the ownership and group back to my user rather than root.

Look for .subscriptions and .imap in your Mail directory (or mail) and check the ownership and permissions.

Fun with Java Keystores: How to import an existing private key and cert into a Java Keystore

I use SSL outside of Java for many things including Web servers, LDAP, SSL programming, etc. Consequently, most of my systems already have private keys and x509 certs. Java's keytool program makes it nearly impossible (as far as I can tell) to import pre-existing keys and certs into an existing or new keystore. Plus, the various Java keystore GUIs are hard to use (I cannot figure them out) or are not open source.

I came across this web page that describes how to do so. I summarize here just in case this web page goes away. We assume the private key is in key.pem and the cert is in cert.pem (both are in PEM format).

Convert the key and cert from PEM format to DER format using openssl command

Use openssl to convert from PEM to DER format. 

openssl pkcs8 -topk8 -nocrypt -in key.pem -inform PEM -out key.der -outform DER
openssl x509 -in cert.pem -inform PEM -out cert.der -outform DER

Put key and cert into a new Java Keystore

Use the ImportKey.java class to take the key and cert and place it in a newly constructed JKS keystore. I modified the ImportKey java source to use the keystore password changeit and to use the key alias importkey and to save the resulting keystore in the file jetty.keystore 

java ImportKey key.der cert.der

Getting Apache to work with Subversion and LDAP Authentication

I use both LDAP and NIS internally for directory services although I am migrating everything to LDAP. I recently installed and created a Subversion repository for my source code. I wanted Apache 2.x to provide access to the repository and I wanted to use LDAP (LDAPS actually) to authenticate users so that I did not have to do yet another .htusers file.

To get everything working, I placed the following directives in conf.d/subversion.conf. I already had https working on the server and configured it to load SSL at startup. 

LoadModule dav_svn_module     modules/mod_dav_svn.so
LoadModule authz_svn_module   modules/mod_authz_svn.so
<Location /repos>
   DAV svn
   SVNPath /columbia4/repos
   AuthType Basic
   AuthName "Subversion repository"
   AuthBasicProvider ldap
   AuthzLDAPAuthoritative on
   Require valid-user
   AuthzLDAPMethod ldap
   AuthLDAPUrl ldaps://columbia.krupczak.org/dc=krupczak,dc=org?uid
</Location>

Notes for Administering OpenNMS

OpenNMS Admin Notes

Root Login on FC10 with Gnome 2.24 and FC11 with Gnome 2.26

The geniuses at Fedora and Gnome have disabled root login on the console in Fedora Core 10 (FC10). No matter what I did, I was unable to "fix" this using the docs I found on the web. On various Fedora mailing lists, allowing or not allowing root login generated a lot of discussion.

I re-enabled root login on FC10 Gnome 2.24 by editing the file /etc/pam.d/gdm and commenting out the line that says user != root. 

#auth       required    pam_succeed_if.so user != root quiet

On FC11 with Gnome 2.26 (FC12, and FC13 with 2.30), there are even more files to edit. gdm-password and gdm-fingerprint also need to be edited to comment out the user/root clause.

How to Change Network Device Names in FC10 and FC 11

Network device names (e.g. eth0, eth1) are seemingly assigned in an arbitrary and capricious manner in recent Fedora distributions. I like for my network interfaces to be assigned in a more predictable manner such as: motherboard ethernet should be eth0 while PCI ethernet cards should be ordered sequentially based on their slot number.

In past Fedora distributions, /etc/modprobe.conf could be edited to reflect device naming preferences. I supposed that may still be true in Fedora 10 and 11. However, one can also edit udev rules in /etc/udev/rules.d. For example, in my new router/firewall box with four ethernet interfaces, the following file can be edited to reflect my desired ethernet interface numbering preferences. From 70-persistent-net.rules 

# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:17:31:8c:6d:be", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:41:e5:43:56", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:13:46:77:fe:b3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"
# Networking Interface (rule written by anaconda)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:04:5a:6b:aa:58", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Add Support for Remote X11 in Gnome

I read email on a different machine using mutt. I want to be able to have pictures and other content displayed on my desktop display via X. The default settings in more recent versions of Gnome disable X11/TCP. 

%ps -aef | grep Xorg
rdk       1494 30140  0 11:22 pts/7    00:00:00 grep -i Xorg
root      4115  4114  0 Aug01 tty1     00:06:29 /usr/bin/Xorg :0 -nr -verbose -auth /var/run/gdm/auth-for-gdm-tJGw2e/database -nolisten tcp vt1

To disable the nolisten tcp, edit /etc/gdm/custom.conf and add DisallowTCP=false under the [security] section. Log out and back in and that should do the trick.

Fun with OpenVPN

I spent a lot of time getting OpenVPN to work so that I could do site-to-site and allow mobile clients to connect. Throw in iptables, routing issues, and OpenSSL and it gets really entertaining.

The OpenVPN server is a multi-homed router with several internal subnets. I wanted the OpenVPN server to make the internal subnets available to clients and other sites. One big problem is that the OpenVPN server was to export the very subnet it resides on to clients and sites. This caused a potential routing loop to occur. I also wanted to be able to access the client's subnet as well hence site-to-site.

The configuration solution was to have the OpenVPN server bind to my ppp0 interface only rather than an internal yet routable subnet interface. I also used a private CA to generate x509v3 certs and keys. Using a private CA allows me to overload client authentication onto the PKI provided by SSL.

After getting things working, I noticed that any sustained traffic, over the VPN, would cause the link to lock up. Searching through the net, I tried tuning the end-system TCP implementations, setting MTUs, etc. Adding the statement mssfix 1400 greatly reduced the congestion on the VPN link by essentially "helping" the end-system TCP implementations tune their maximum-segment-size.

I also added the following entry to both sides /etc/sysctl.conf file to increase buffers available to UDP. 

vm.min_free_kbytes = 8192

The server's config, stripped down is: 

proto udp
port 1194
dev tun
mssfix 1400
local ppp0-ip-address
tls-server
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server.crt
key /etc/openvpn/server.key
dh /etc/openvpn/dh1024.pem
server 172.30.0.0 255.255.255.0
client-to-client
ifconfig-pool-persist ipp.txt
keepalive 10 60
ping-timer-rem
comp-lzo
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
client-config-dir ccd
# tell the underlying system to route to this subnet via VPN
# this subnet is located at the other site; my subnets are "pushed"
# to the other side via the ccd config file
route 10.0.1.0 255.255.255.0

The client (other site) config file: 

client
local my-own-ppp0-ip-address
proto udp
port 1194
dev tun
tls-client
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client.crt
key /etc/openvpn/client.key
resolv-retry infinite
keepalive 10 60
ping-timer-rem
remote server-ip-address 1194
comp-lzo
user nobody
group nobody
persist-key
persist-tun
# server side iroutes my net and make it available to others                    
# via CCD config

The client-specific config file, located on the OpenVPN server is: 

# client-specific config that gets pushed to specific client
# filename must match hostname/cert-CN in order to get read
push "route 10.0.0.0 255.255.255.0"
push "route w.x.y.z 255.255.255.240"
# iroute tells the VPN daemon to route packets to 10.0.1.x 
# to this particular client
iroute 10.0.1.0 255.255.255.0

Fun With Nvidia Graphics Cards

I bought and installed a nice PCI Nvidia graphics card but had problems telling Xorg to use the new card. It turned out that I needed to specify the BusID of the video card in my xorg.conf file. Dmesg returns the BusID in one format (with a period) and the config file wants a slightly different format. The two formats are subtly different hence the cause of my consternation. 

# dmesg | grep nvidia
nvidia 0000:03:00.0: PCI INT A -> Link[LN0A] -> GSI 19 (level, low) -> IRQ 19

Here is the working device section from my xorg.conf file. Note the use of colons separating the BusID components. Dmesg puts a period before the last 0. 

Section "Device"
    Identifier     "Videocard0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID 	   "PCI:03:00:0"
EndSection

How to Upgrade VMware Server 2.0 on Linux

I recently had to upgrade my VMware VMserver 2.0 to 2.0.2 to install Windows 7. Rather than re-invent the wheel and re-learn how to do this, I found this website which outlines the fairly simple procedure.

http://www.cyberciti.biz/tips/rhel-centos-upgrade-vmware-server.html

Building an ESXi 4.1 "Whitebox" on the Cheap

I recently built a "whitebox" to run ESXi 4.1. I used the following components:

  • Samsung OEM SATA DVD reader/burner $18.00
  • 2 SATA drives 1 TB 3.5” Seagate Barracuda 7200 RPM 32MB cache $120.00 MicroCenter
  • Hand-me-down case
  • Coolmast 350Watt $28.00 MicroCenter
  • Total $656 not including tax.

Installing Solaris 11 x86 on ESX 4.1

At the time of writing (12/2010), ESXi 4.1 did not "officially" support the newly released Solaris 11 (x86). After some trial and error, I was able to install Solaris 11 x86 in a VM running on ESXi 4.1 using the following hardware configuration. (Note that the default hardware for Solaris 10 x86 did not work for Solaris 11.)

Component Config
Disk IDE (0:0)
Video Generic 800x600 24-bit 16 MB video RAM
CD/DVD IDE (1:0)
NIC E1000 (same as in host)
CPU 1 AMD x64
RAM 1 GB
Other All other default settings

SIP Registration Attempts are Pounding My SIP Server

There are some scripts, making the rounds on the Internet, that probe SIP servers for open and mis-configured extensions. One such script performs two SIP registration attempts and then moves on to the next host if the SIP server is properly configured. Another script, what I call the "broken" script, encounters my SIP server and then just pounds away ad nausea until I manually put a filter rule to stop it. The "broken" script pounds so hard that it acts as a denial-of-service attack against my network.

In searching for a way to automatically block these attempts, I came across two such approaches.

  • fail2ban is a script that will scan your SIP server log file and then automatically add a rule to the local iptables config file.
  • iptables recent module can be configured to block various packets after some number are seen from a particular IP address

I do not like the first approach for several reasons. First, fail2ban is a separate script that must scan log files and then create and add an iptables rule. Given how many morons are out on the Internet, I'd end up with lots and lots of rules over time. And, I want to block the SIP registration attempts at my border firewall/router so that they do not even enter my network (my SIP server does not run on my border router/firewall).

The second approach seemed thus seemed the easiest. The downside is that there is some increased processing for each packet although the majority of the processing is only performed on SIP packets. Here are the iptables rules I added. 

iptables -A FORWARD -p udp --dport 5060 -m recent --set --name SIPPKT
iptables -A FORWARD -p udp --dport 5060 -m recent --update --seconds 30 --hitcount 20 --name SIPPKT -j DROP

The above rules first create a set called SIPPKT and the then watch for greater than 20 matches in the last 30 seconds. If more than 20 matches come from the same IP address, in the last 30 seconds, subsequent matches are dropped silently on the floor. You could choose to log these drops with an additional rule but then your log file would be clogged entries. Your SIP server log file will probably still contain the first 20 attempts so you can still detect that you have been attacked.

One gotcha that you have to watch out for. You should place these rules before any state module rules that may let packets through the firewall if they have been previously accepted. That is, if you let a few of these SIP requests in, reply back to them essentially saying SIP-off, then future SIP packets will continue to be accepted from the offending machine because state rules permit them.

How do I use ImageMagick to correct the orientation of a directory of images?

Digital cameras now include orientation meta-information. Depending on the how you get your images off your camera, they may or may not be oriented correctly for viewing. If the images are not auto oriented, you can use the following commands to correct the orientation. However, the file access dates will be modified.

In csh, use the following command. 

% foreach i (*.JPG *.jpg)
echo $i
convert -auto-orient $i $i
end

Deploying IPv6

I recently deployed IPv6 within my network and connected to the IPv6 Internet. While I studied the IPng protocol development effort in graduate school, it had been a while since working with IPv6. Plus, studying the protocol and deploying it in a working network are often two vastly different efforts. Consequently, I bought and read Running IPv6 by Iljitsch van Beijnum. The book is a pretty good tutorial for bring up and running IPv6 in a heterogeneous environment. Originally written in 2005, the book is starting to get a tad bit dated. Included below are notes from my deployment effort roughly divided into several categories.

Connecting to the IPv6 Internet and Obtaining Addresses

He-ipv6-guru.png

IPv6 is currently deployed in parallel with IPv4 on the Internet. That is, you can consider IPv4 and IPv6 as separate, distinct protocols that are like ships in the night. Since my own ISP does not yet support IPv6 (that is, they do not route IPv6 and do not have their own address block), I was faced with the task of finding an IPv6 tunnel provider. Hurricane Electric (HE) provides free tunnel service as well as IPv6 address block assignment. I liked the price and heard good things about their IPv6 support tools. They even have an unofficial certification process that strives to educate network administrators. After deploying IPv6, I was able to achieve the rank of guru in their certification process.

Each tunnel is a point-to-point link, over (or encapsulated in) IPv4 and assigned a /64 IPv6 address. While it would seem a waste to assign a 64-bit subnet for just two endpoints, there are no shortages of addresses in IPv6 and the IPv6 community has reached consensus that netblocks are not assigned smaller than /64. My end of the tunnel was assigned ::2/64 and HE's was assigned ::1/64. In addition, I was assigned an IPv4 address for HE's side of the tunnel and I in turn gave them the IPv4 address of my side of the tunnel.

In addition to the tunnel address assignment, HE assigns a /64 to each registered user for use within their network. Because I have 4+ subnets in my network, I requested and received a /48 address assignment -- 2001:470:e499::/48. With this ::/48 assignment, I have 80 bits (128-48) of addresses I can potentially assign. By convention, the last 64-bits are for the host-portion of addresses so I can have 16-bits or 65,536 subnets!

Path MTU

Because IPv6 packets are encapsulated in an IPv4 tunnel, and because IPv6 headers are larger, I had to decrease the MTU through the tunnel and push that decrease back out to IPv6 clients. I hardcoded 1280 as the path MTU because ICMPv6 Path MTU discovery packets may have been getting dropped somewhere over the Internet and I was experiencing strange TCP connection problems when attempt to browse websites like YouTube. Once I lowered my tunnel MTU to 1280 (and propagated that MTU to IPv6 clients via router advertisement), my TCP connection problems disappeared.

If you are going to let your path MTU float, you need to make sure your firewall/router will generate, accept, and forward ICMPv6 Packet Too Big (Type=2) messages.

DNS

AAAA records and IPv6 transport for my nameserver.

Setting up reverse records internally.

Firewall Rules

Because IPv6 packets are tunneled through IPv4, I had to add a few new entries in my IPv4 firewall rule set. IPv4 packets, that carry IPv6 messages, set their protocol field to 41. Also, each endpoint of my IPv4/IPv6 tunnel has a static IPv4 address assigned to it so I could further constrain my firewall rules.

The rules below instruct IPv4 iptables to allow IPv6 tunnel traffic to/from HE. The variable $IPTABLES is set to the path and name of the iptables binary and the $V6TUNNELSERV variable is set to the IPv4 address of the other end of the tunnel. $EXT_IF is the particular interface that the tunnel traverses (e.g. sit1). 

$IPTABLES -A INPUT -i sit+ -s $V6TUNNELSERV -p 41 -j ACCEPT
$IPTABLES -A OUTPUT -o sit+ -j ACCEPT
$IPTABLES -A OUTPUT -o $EXT_IF -p 41 -j ACCEPT

The eventual goal (probably a long way off in the future) is to turn off IPv4 and migrate everything to IPv6. Any of the services I currently make available via IPv4 I also want to make available via IPv6. Consequently, I had to create a separate IPv6 firewall rules set to permit essentially the same services to traverse my firewall/router via IPv6. Since I'm using globally unique IPv6 addresses everywhere (even for internal machines), there is no need for network address translation (NAT). Regarding IPv6 firewall rules, one has to make sure to allow ICMPv6 to/from the Internet in order for Path MTU discovery to work. You still may wish to filter out ICMPv6 redirects though. I filter redirects coming from my external interface (my tunnel) but allow them internally. Below are statements from my firewall script that implement this policy. 

# dont accept redirects from external interfaces sit0 and sit1
echo "0" > /proc/sys/net/ipv6/conf/sit0/accept_redirects
echo "0" > /proc/sys/net/ipv6/conf/sit1/accept_redirects

I am not going to post my IPv6 firewall rule set here because doing so might give away subtle configuration information that might aid bad guys. If you'd like a copy of my IPv6 firewall rule set, contact me via email. Let me just say that it was very similar to my IPv4 rule set.

Addressing Architecture

How did I divide up my new addresses? Site-local? No. Dynamic DNS and EUI-64? DHCPv6 to distribute DNS server info.

Router Advertisement and Discovery

IPv6 includes several mechanisms to reduce configuration overhead. One such mechanism is router discovery. Since the distro running on my firewall/router did not include a router advertisement daemon, I had to download and install one via my distro's software update mechanism. radvd is a popular router advertisement daemon available on many UNIX variants.

I use this daemon to disseminate router and link MTU information. Here is the config file snippet for one of my subnets. Note the MTU assignment. 

interface eth1
{
	AdvSendAdvert on;
	MinRtrAdvInterval 30;
	MaxRtrAdvInterval 100;
	AdvLinkMTU 1280;
	prefix 2001:0470:e499:0001::/64
	{
		AdvOnLink on;
		AdvAutonomous on;
		AdvRouterAddr on;
	};
};

Applications -- Email, Web, VoIP, etc.

Config changes to important applications.

Retrieved from "https://www.krupczak.org/index.php?title=Sysadmin_Notes&oldid=3986"
Personal tools