Ethernet Bonding (Linux)

Bonding ethernet channels in Linux is easy. With the right configuration and hardware, it can be done in minutes.

In this example, I have a 4 port intel Gigabit card and I wanted to utilize this and my Network switches LACP/LAG capability. Lets assume we have the Network switch part already configured to have its ports utilize LACP/LAG.

On the RHEL 4.3 server that is hosting my samba file server below are the steps outlined to configure ethernet channel bonding/teaming on the RHEL 4.3 server.

Edit the /etc/modprobe.conf file and add the bonding kernel module and its mode.

# vi /etc/modprobe.conf

#regular network card (built-in)

alias eth0 e1000

alias eth1 e1000

#the bond interface 

alias bond0 bonding

#bond interface mode. mode=4 (802.3ad/link aggregation)

options bond0 mode=4 miimon=100 

#Intel 4 port Gigabit ethernet card

alias eth5 e1000

alias eth3 e1000

alias eth4 e1000

alias eth2 e1000

alias scsi_hostadapter ahci

alias usb-controller ehci-hcd

alias usb-controller1 uhci-hcd

Now make sure to edit the /etc/sysconfig/network-scripts/ifcfg-ethX (where X are the ethernet ports that will be part of the bond0 LACP interface) 

# vi /etc/sysconfig/network-scripts/ifcfg-eth2

DEVICE=eth2

USERCTL=no

ONBOOT=yes

MASTER=bond0

SLAVE=yes

BOOTPROTO=none

TYPE=Ethernet

Now we don't want to edit every single file as its a boring and error prone process

# for i in {3..5} ; do cp /etc/sysconfig/network-scripts/ifcfg-eth2 /etc/sysconfig/network-scripts/ifcfg-eth$i ; done

#for i in {3..5} ; do sed -i "s/DEVICE=eth2/DEVICE=eth$i/g" /etc/sysconfig/network-scripts/ifcfg-eth$i ; done

Now create the bond0 interface configuration file.

# vi /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0

IPADDR=192.168.1.30

NETWORK=192.168.1.0

NETMASK=255.255.255.0

USERCTL=no

BOOTPROTO=none

ONBOOT=yes

TYPE=Ethernet

Load the bonding kernel module.

# modprobe bonding

Now bring up the bond0 interface

# ifup bond0

Verify the bond0 interface

# cat /proc/net/bonding/bond0 

Ethernet Channel Bonding Driver: v2.6.1 (October 29, 2004)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

802.3ad info

LACP rate: slow

Active Aggregator Info:

        Aggregator ID: 1

        Number of ports: 1

        Actor Key: 17

        Partner Key: 1

        Partner Mac Address: 00:00:00:00:00:00

Slave Interface: eth2

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:15:17:6a:c1:e4

Aggregator ID: 1

Slave Interface: eth3

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:15:17:6a:c1:e5

Aggregator ID: 2

Slave Interface: eth4

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:15:17:6a:c1:e6

Aggregator ID: 3

Slave Interface: eth5

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:15:17:6a:c1:e7

Aggregator ID: 4

You can now try transferring files to/from the samba servers shared folders, once the transfer process is on-going, try unplugging the network cables that are part of the bond/LACP/LAG on the Network switch or on the configured 4 port Intel ethernet card.

Resetting lost password

Resetting lost password

If you've ever forgotten your user account's password in OS X, it is still possible to reset or change your lost password. All you need is to remember your username and then reboot your computer. From there it's command line work

You should know your username in order to perform the password reset as outlined below. The username is usually your computer name.

Hold Apple+S when booting to enter single user mode, then enter the commands below.

#sh /etc/rc

#passwd yourusername

#reboot

Note that the process above will reset all your keychain if you ever have to use it. Anyway, this is better than no password at all -

Adding clamav-milter antivirus headers to emails

Edit the /etc/clamav-milter.conf and change the configuration to "AddHeader Add".

[root@mail ~]# less /etc/clamav-milter.conf

# If this option is set to "Replace" (or "Yes"), an "X-Virus-Scanned" and an

# "X-Virus-Status" headers will be attached to each processed message, possibly

# replacing existing headers.

# If it is set to Add, the X-Virus headers are added possibly on top of the

# existing ones.

# Note that while "Replace" can potentially break DKIM signatures, "Add" may

# confuse procmail and similar filters.

# Default: no

#AddHeader Replace

AddHeader Add

Restart the clamav-milter daemon after changing the configuration file, then try sending emails from/to your user in your mailserver and check the header if the clamav-milter is already working. This is also one way to check if your clamav-milter integration with your email server is working correctly.

X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.95.3 at mail.example.com Received: (qmail 49613 invoked by uid 60001); 29 Mar 2010 08:59:04 -0000

Deleting pesky emails that are in the postfix queue (for whatever reason)

Sometimes, emails gets stuck on the email queue, retrying and retrying. A more unwanted scenario if there are spam mails in the queue hogging the system down (resources, bandwidth, etc). This spam might have been caused by incorrectly configured email server, a spam bot infected client that is trusted by your network, and maybe emails that are just really going nowhere for whatever reasons (again).

In Unix and Unix-like operating systems, there are quite a lot of ways to getting rid of those *unwanted* stuck emails on the email servers queue. In Postfix, you can use the postsuper -d to delete those unwanted emails on the queue. So if you have a couple of hundred emails to delete, you may want to automate this process as we don't basically want to waste most of our precious administrator time by just manually deleting this pesky emails. If you like reading manual pages, you can see that there is a fantastic one-liner to delete a number of emails from an unwanted sender, or in the case that this sender (particularly a spam) will never have mailbox on your email server so this gets stuck on the queue as being a MAILER_DAEMON pointing to non-existent address. Imagine you have thousands of this on the queue, this will easily bring your server down to its knees. What more bad is that you will be bombarded with phone calls from not so happy users.

In this kind of situation, you don't really need to panic as there will always be a way how to correct this thing. In Postfix, you can delete/purge this unwanted emails by using the command below:

Assuming that you already know the offending email address, you can substitute the "user@example.com" with the email address that is stuck in the queue. Also, take note of the correct "test" in the "if" clause, so you can basically change this accordingly to exactly match you situation.

# $7=sender, $8=recipient1, $9=recipient2

mailq | tail +2 | grep -v '^ *(' | awk  'BEGIN { RS = "" } { if ($8 == "user@example.com" && $9 == "") print $1 } ' | tr -d '*!' | postsuper -d -

Determine the offending email address as below using the mailq command:

mailq

-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------

3E90C13F88D3     4543 Mon Mar 29 20:03:56  ebhospeterloaninvestment@hotmail.com

(host mx1.free.fr[212.27.48.7] said: 451 too many errors from your ip (xxx.xxx.xxx.xxx), please visit http://postmaster.free.fr/ (in reply to DATA command))

                                         elmore@online.fr

                                         dannye@online.fr

                                         daphacorp@online.fr

                                         darci@online.fr

                                         daryl@online.fr

                                         daveen@online.fr

                                         debra@online.fr

                                         deeyn@online.fr

                                         delila@online.fr

                                         delilah@online.fr

                                         delmore@online.fr

                                         demetri@online.fr

OK, suppose that we have an email inside the queue that is from a spam address ebhospeterloaninvestment@hotmail.com (and being a spam, it will send to X number of recipients), we can adjust our one-liner command like below. 

NOTE: I added -n to the tail command as below or you will get this error message on the shell "tail: cannot open `+2' for reading: No such file or directory". Also, take note of the "if" clause.

# mailq | tail -n +2 | grep -v '^ *(' | awk  'BEGIN { RS = "" } { if ($7 == "ebhospeterloaninvestment@hotmail.com") print $1 } ' | tr -d '*!' | postsuper -d -

postsuper: 3E90C13F88D3: removed

postsuper: Deleted: 1 message

We now just deleted one offending email stuck on the email servers queue. And this will work on multiple emails sent by this address that are stuck in the email servers queue.

Cyrus-IMAP Logging region out of memory thing

One Wednesday morning, our email users reported that they are not receiving any emails, which is very unusual for a very busy day. I found out that the Cyrus-IMAP's database is flagging with errors as below:

[root@mail ~]# tail -n 100 -f /var/log/maillog

Mar 24 10:36:14 mail lmtpunix[4017]: executed
Mar 24 10:36:14 mail lmtpunix[4017]: DBERROR db4: Logging region out of memory; you may need to increase its size
Mar 24 10:36:14 mail lmtpunix[4017]: DBERROR: opening /var/lib/imap/deliver.db: Cannot allocate memory
Mar 24 10:36:14 mail lmtpunix[4017]: DBERROR: opening /var/lib/imap/deliver.db: cyrusdb error
Mar 24 10:36:14 mail lmtpunix[4017]: FATAL: lmtpd: unable to init duplicate delivery database
Mar 24 10:36:14 mail master[4664]: process 4017 exited, status 75
Mar 24 10:36:14 mail master[4664]: service lmtpunix pid 4017 in READY state: terminated abnormally
Mar 24 10:36:14 mail master[4018]: about to exec /usr/lib/cyrus-imapd/lmtpd

It turned out that the Cyrus-IMAP service runs out of log memory. So here comes the need to change the default setting and create a bigger cache and log memory regions of Cyrus-IMAP's database as documented below.

Changing the default Logging region of Cyrus-IMAP servers database is easy.

Create a DB_CONFIG file is you have not done it yet.

[root@mail ~]# cat > /var/lib/imap/db/DB_CONFIG 

set_cachesize 0 2097152 1

set_lg_regionmax 1048576

^D

Stop the cyrus-imapd server, this is very important.

[root@mail ~]# /etc/init.d/cyrus-imapd stop

Backup the database to be recovered.

[root@mail ~]# rsync -auvp /var/lib/imap/db /root/db_backup.date

Recover the database that was changed.

[root@mail ~]# db_recover -h /var/lib/imap/db

Once the command above is successfully done, we can now start the Cyrus-IMAP service.

[root@mail ~]# /etc/init.d/cyrus-imapd start

Check the new cache and memory statitics and wether the new settings are set correctly.

[root@mail ~]# db_stat -h /var/lib/imap/db -m

[root@mail ~]# db_stat -h /var/lib/imap/db -l

Shoot a few emails and check the logs how it goes. If users are no receiving emails correctly, the server is now back online. Keep monitoring until you are certain that everything is now back to normal.

Cyrus-IMAP user is over quota

One of my user's mailbox is already being flagged by cyrus-imapd as over quota, I need to do something before she gets furious.

My email server's queue shows below:

3069F13F88A1 3625 Thu Mar 25 01:20:21 sender@sender.com

(host mail.recipient.com[/var/lib/imap/socket/lmtp] said: 452 4.2.2 Over quota (in reply to RCPT TO command))

recipient@recipient.com

I now need to do my act and login to my email server, then use cyradm to administer my Cyrus-IMAP server as below.

[root@mail ~]# su - cyrus

-bash-3.2$ cyradm -user cyrus localhost

IMAP Password:

mail.recipient.com> lqr user.sales-tv

user.sales-tv STORAGE 150000/150000 (100.006%)

mail.recipient.com> sq user. recipient 200000

quota:200000

mail.recipient.com>

I used lqr as listquotaroot shorthand and sq as setquota shorthand.

Verify it the new quota is set, then exit the cyradm admin tool and exit the email server as well. Your users new quota is now set and she is now very happy again.

Checking running queries on MySQL server

You are administering a MySQL server and suddenly your users are not so very happy with what is going on. Basically, they are getting errors like "too many connections" or the content they are accessing is basically crawling at all. In this situations, one can monitor "what is going on" on a loaded MySQL server to find out what is causing hiccups/bogs or what queries are eating CPU cycles available to the a system.

Upon logging in to the mysql database's interactive shell, we can use the "show processlist;" command. This will list the running querries currently to the database of the logged in user.

One one of the interactive shell, we somehow run for example.

mysql> select sleep(100);

mysql> show processlist;
+----+------+-----------+------+---------+------+-----------+------------------+
| Id | User | Host      | db   | Command | Time | State     | Info             |
+----+------+-----------+------+---------+------+-----------+------------------+
| 23 | root | localhost | NULL | Query   |    2 | executing | select sleep(100)|
| 24 | root | localhost | NULL | Query   |    0 | NULL      | show processlist |
+----+------+-----------+------+---------+------+-----------+------------------+
2 rows in set (0.00 sec)

We can somehow kill hte process if we elect to, using the "kill 23" (23 being the process id).