This morning I found out that our company mail server, using qmail, is failing to communicate through secure http to ticket system, using RT
So, I logged into out mail server to verify the mail server to ticket server communication setup
cat /var/qmail/alias/.qmail-help
|822field x-spam-status | egrep -iw yes >/dev/null && exit 100 || exit 0
|822field from | egrep -i “MAILER-DAEMON|POSTMASTER|ABUSE” >/dev/null && exit 100 || exit 0
|822field to | egrep -i “MAILER-DAEMON|POSTMASTER|ABUSE” >/dev/null && exit 100 || exit 0
|822field cc | egrep -i “MAILER-DAEMON|POSTMASTER|ABUSE” >/dev/null && exit 100 || exit 0
|822field return-path | egrep -i “MAILER-DAEMON|POSTMASTER|ABUSE” >/dev/null && exit 100 || exit 0
|/opt/rt3/bin/rt-mailgate –queue help –action correspond –url https://ticket.company.net 2>/dev/null || exit 111
That means any mail goes to help@company.net
will eventually use
rt-mailgate
to make a secure http connection to the ticket system and create a ticket.
So the first thing that could go wrong is the mail server failing to receive the mail. To double check that I went ahead and send an email from my computer to help@company.net
and checked the log on the smtp side.
tail -f /var/log/qmail/smtpd/current | tai64nlocal
I noticed the smtp server received the email based on the following line in the log
2007-08-20 18:29:44.718444500 tcpserver: ok 22971 smtp.company.net:1.2.3.4:25 NO_REVERSE_DNS:2.3.4.5::60738
Where 1.2.3.4
is the company smtp server IP and 2.3.4.5
is my computer sending the email. And it says OK
. So we are good on smtp side. Now if you know qmail, you know that this mail then will be handed over eventually to qmail-local alias.
To verify that the mail did not fail on its way to the qmail-local I tried to look at the qmail send log
tail -f /var/log/qmail/current | tai64nlocal
And I see deferal in the delivery for the email that I sent to help@company.ent
2007-08-20 09:03:25.432989500 delivery 1804118: deferral:
Usually you should see a message right after the deferral: to explain why it is deferred like the followings i.e.:
2007-08-20 09:04:01.774745500 delivery 1804142: deferral: Sorry,_I_wasn’t_able_to_establish_an_SMTP_connection._(#4.4.1)/
2007-08-20 09:03:21.451940500 delivery 1804099: deferral: 4.4.8.50_does_not_like_recipient./Remote_host_said:_450_<blah
@blah.com>:_User_unknown_in_local_recipient_table/Giving_up_on_4.4.8.50./
But this time there is no message after deferral:
. So I am still clueless as to why it is being deferred instead of being a delivered successfully. Well at least I know that mail is trying to make through. So the next possible point where mail could fail is at the rt-mailgate. If you are not sure why, look at the beginning of the article which shows how the communication is setup in the .qmail-help file
To troubleshoot if rt-mailgate is working I decided to run it in debug mode and send some garbage to it to see how it responds
echo this is a test | /opt/rt3/bin/rt-mailgate –queue help –action correspond –url https://ticket.company.net –debug
I see few lines of error with Crypt::SSLeay
module. One of its library that is depending on openssl is failing to load. Why? looks like the openssl library itself changed to a newer version. It might have happend during ssh package upgrade which is very picky about the openssl library version. So that means either openssl library needs to be downgraded to a lower version to match the version Crypt::SSLeay
, which probably will break ssh, OR recompile Crypt::SSLeay
I decided to just recompile Crypt::SSLeay
and point to new openssl library during compile. I called CPAN
to install
Crypt::SSLeay
module. When it rains it pours. The perl module is failing to compile with default CPAN
install
method. It is looking for openssl/ssh.h
header file under /usr/local/ssl/include/openssl
. So instead of using CPAN
install
command I used CPAN
look
command which let me compile the module manually from source. I then modified the include variable INC
in the Makefile
(run perl Makefile.PL
first) and replaced include path from /usr/local/ssl/include/openssl
to /usr/local/ssl/include
. That is pretty much the only hack I needed to do during manul make
. After that I did a re make
and make install
and now I have a fresh new Crypt::SSLeay
. To make sure that fixed the rt-mailgate
, I feeded some garbage, like before, to it and I got a ok
and a Ticket
number. Eeehaa secure http communication between mail server and ticket system have been restored and the day is saved!