Deleting pesky emails that are in the postfix queue (for whatever reason)

Sometimes, emails gets stuck on the email queue, retrying and retrying. A more unwanted scenario if there are spam mails in the queue hogging the system down (resources, bandwidth, etc). This spam might have been caused by incorrectly configured email server, a spam bot infected client that is trusted by your network, and maybe emails that are just really going nowhere for whatever reasons (again).

In Unix and Unix-like operating systems, there are quite a lot of ways to getting rid of those *unwanted* stuck emails on the email servers queue. In Postfix, you can use the postsuper -d to delete those unwanted emails on the queue. So if you have a couple of hundred emails to delete, you may want to automate this process as we don't basically want to waste most of our precious administrator time by just manually deleting this pesky emails. If you like reading manual pages, you can see that there is a fantastic one-liner to delete a number of emails from an unwanted sender, or in the case that this sender (particularly a spam) will never have mailbox on your email server so this gets stuck on the queue as being a MAILER_DAEMON pointing to non-existent address. Imagine you have thousands of this on the queue, this will easily bring your server down to its knees. What more bad is that you will be bombarded with phone calls from not so happy users.

In this kind of situation, you don't really need to panic as there will always be a way how to correct this thing. In Postfix, you can delete/purge this unwanted emails by using the command below:

Assuming that you already know the offending email address, you can substitute the "user@example.com" with the email address that is stuck in the queue. Also, take note of the correct "test" in the "if" clause, so you can basically change this accordingly to exactly match you situation.

# $7=sender, $8=recipient1, $9=recipient2

mailq | tail +2 | grep -v '^ *(' | awk  'BEGIN { RS = "" } { if ($8 == "user@example.com" && $9 == "") print $1 } ' | tr -d '*!' | postsuper -d -

Determine the offending email address as below using the mailq command:

mailq

-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------

3E90C13F88D3     4543 Mon Mar 29 20:03:56  ebhospeterloaninvestment@hotmail.com

(host mx1.free.fr[212.27.48.7] said: 451 too many errors from your ip (xxx.xxx.xxx.xxx), please visit http://postmaster.free.fr/ (in reply to DATA command))

                                         elmore@online.fr

                                         dannye@online.fr

                                         daphacorp@online.fr

                                         darci@online.fr

                                         daryl@online.fr

                                         daveen@online.fr

                                         debra@online.fr

                                         deeyn@online.fr

                                         delila@online.fr

                                         delilah@online.fr

                                         delmore@online.fr

                                         demetri@online.fr

OK, suppose that we have an email inside the queue that is from a spam address ebhospeterloaninvestment@hotmail.com (and being a spam, it will send to X number of recipients), we can adjust our one-liner command like below. 

NOTE: I added -n to the tail command as below or you will get this error message on the shell "tail: cannot open `+2' for reading: No such file or directory". Also, take note of the "if" clause.

# mailq | tail -n +2 | grep -v '^ *(' | awk  'BEGIN { RS = "" } { if ($7 == "ebhospeterloaninvestment@hotmail.com") print $1 } ' | tr -d '*!' | postsuper -d -

postsuper: 3E90C13F88D3: removed

postsuper: Deleted: 1 message

We now just deleted one offending email stuck on the email servers queue. And this will work on multiple emails sent by this address that are stuck in the email servers queue.

Cyrus-IMAP Logging region out of memory thing

One Wednesday morning, our email users reported that they are not receiving any emails, which is very unusual for a very busy day. I found out that the Cyrus-IMAP's database is flagging with errors as below:

[root@mail ~]# tail -n 100 -f /var/log/maillog

Mar 24 10:36:14 mail lmtpunix[4017]: executed
Mar 24 10:36:14 mail lmtpunix[4017]: DBERROR db4: Logging region out of memory; you may need to increase its size
Mar 24 10:36:14 mail lmtpunix[4017]: DBERROR: opening /var/lib/imap/deliver.db: Cannot allocate memory
Mar 24 10:36:14 mail lmtpunix[4017]: DBERROR: opening /var/lib/imap/deliver.db: cyrusdb error
Mar 24 10:36:14 mail lmtpunix[4017]: FATAL: lmtpd: unable to init duplicate delivery database
Mar 24 10:36:14 mail master[4664]: process 4017 exited, status 75
Mar 24 10:36:14 mail master[4664]: service lmtpunix pid 4017 in READY state: terminated abnormally
Mar 24 10:36:14 mail master[4018]: about to exec /usr/lib/cyrus-imapd/lmtpd

It turned out that the Cyrus-IMAP service runs out of log memory. So here comes the need to change the default setting and create a bigger cache and log memory regions of Cyrus-IMAP's database as documented below.

Changing the default Logging region of Cyrus-IMAP servers database is easy.

Create a DB_CONFIG file is you have not done it yet.

[root@mail ~]# cat > /var/lib/imap/db/DB_CONFIG 

set_cachesize 0 2097152 1

set_lg_regionmax 1048576

^D

Stop the cyrus-imapd server, this is very important.

[root@mail ~]# /etc/init.d/cyrus-imapd stop

Backup the database to be recovered.

[root@mail ~]# rsync -auvp /var/lib/imap/db /root/db_backup.date

Recover the database that was changed.

[root@mail ~]# db_recover -h /var/lib/imap/db

Once the command above is successfully done, we can now start the Cyrus-IMAP service.

[root@mail ~]# /etc/init.d/cyrus-imapd start

Check the new cache and memory statitics and wether the new settings are set correctly.

[root@mail ~]# db_stat -h /var/lib/imap/db -m

[root@mail ~]# db_stat -h /var/lib/imap/db -l

Shoot a few emails and check the logs how it goes. If users are no receiving emails correctly, the server is now back online. Keep monitoring until you are certain that everything is now back to normal.

Cyrus-IMAP user is over quota

One of my user's mailbox is already being flagged by cyrus-imapd as over quota, I need to do something before she gets furious.

My email server's queue shows below:

3069F13F88A1 3625 Thu Mar 25 01:20:21 sender@sender.com

(host mail.recipient.com[/var/lib/imap/socket/lmtp] said: 452 4.2.2 Over quota (in reply to RCPT TO command))

recipient@recipient.com

I now need to do my act and login to my email server, then use cyradm to administer my Cyrus-IMAP server as below.

[root@mail ~]# su - cyrus

-bash-3.2$ cyradm -user cyrus localhost

IMAP Password:

mail.recipient.com> lqr user.sales-tv

user.sales-tv STORAGE 150000/150000 (100.006%)

mail.recipient.com> sq user. recipient 200000

quota:200000

mail.recipient.com>

I used lqr as listquotaroot shorthand and sq as setquota shorthand.

Verify it the new quota is set, then exit the cyradm admin tool and exit the email server as well. Your users new quota is now set and she is now very happy again.

Checking running queries on MySQL server

You are administering a MySQL server and suddenly your users are not so very happy with what is going on. Basically, they are getting errors like "too many connections" or the content they are accessing is basically crawling at all. In this situations, one can monitor "what is going on" on a loaded MySQL server to find out what is causing hiccups/bogs or what queries are eating CPU cycles available to the a system.

Upon logging in to the mysql database's interactive shell, we can use the "show processlist;" command. This will list the running querries currently to the database of the logged in user.

One one of the interactive shell, we somehow run for example.

mysql> select sleep(100);

mysql> show processlist;
+----+------+-----------+------+---------+------+-----------+------------------+
| Id | User | Host      | db   | Command | Time | State     | Info             |
+----+------+-----------+------+---------+------+-----------+------------------+
| 23 | root | localhost | NULL | Query   |    2 | executing | select sleep(100)|
| 24 | root | localhost | NULL | Query   |    0 | NULL      | show processlist |
+----+------+-----------+------+---------+------+-----------+------------------+
2 rows in set (0.00 sec)

We can somehow kill hte process if we elect to, using the "kill 23" (23 being the process id).

Monitoring Unix(-like) systems performance

One of the day to day tasks of the Systems Administrator is to monitor the servers he/she manages and makes sure that the services the servers are serving are all working correctly. The SA should be able to interpret the output from the systems monitoring tools when the system is bogging down. This skill alone will dramatically help the SA pinpoint and gather clues on what are the usual suspects that "bogs" the server down. This is very critical to what steps should be taken when dealing with slow/overloaded servers. Service availability are very critical to every organization to have the business running smoothly and continuously.

Now, enter the realm of systems monitoring. On every Unix system, there are a few monitoring tools available and they grow in number as days go by. The most commonly used commands to monitor a Unix systems performance follows (classics):

uptime - Tell how long the system has been running. (More onto this really.)

procinfo - display system status gathered from /proc

who - show who is logged on (Also more onto this.)

top - display top CPU processes (Resource hungry though...)

Those above commands are very useful in determining the actual System LA (Load Average), this metrics can also be used in performance analysis and capacity planning.

The following link below, you will learn System Load Average in the more in depth manner. I would like to thank Dr. Neil Gunther for his wonderful insight. And also everybody that contributes to the Unix and Unix-like Operating Systems. And I would like to encourage every Systems Administrators to always read the man pages.

http://www.teamquest.com/resources/gunther/display/5/index.htm

Using "for" loop in automating repetitive tasks

The "for" loop is a very powerful tool in automating repetitive tasks. A good example would be cloning virtual hard drives in VirtualBox (I use this for my development and RnD as my sandbox, Thank you SUN!). Below shows the basic usage of the for loop on the bash cli.

In this example, I would like to clone my VirtualBox hard disk from one of my SuSE 10 servers hard disk, as I don't want to do repetitive installation. But instead, I will just modify the settings of my new "cloned" harddrive(s) tp the settings that will suite my new VirtualBox machine to my liking (or yours).

for i in {0..5} ; do VBoxManage clonehd SuSE.10.LVS.00.vdi SuSE.10.MYSQL.0$i.vdi ; done

The command above will iterate 'i' number of times as defined inside the range {0..5} which in this case will be 6 (0,1,2,3,4,5) that is.