This morning I found out that our company mail server, using qmail, is failing to communicate through secure http to ticket system, using RT
So, I logged into out mail server to verify the mail server to ticket server communication setup
cat /var/qmail/alias/.qmail-help
|822field x-spam-status | egrep -iw yes >/dev/null && exit 100 || exit 0
|822field from | egrep -i “MAILER-DAEMON|POSTMASTER|ABUSE” >/dev/null && exit 100 || exit 0
|822field to | egrep -i “MAILER-DAEMON|POSTMASTER|ABUSE” >/dev/null && exit 100 || exit 0
|822field cc | egrep -i “MAILER-DAEMON|POSTMASTER|ABUSE” >/dev/null && exit 100 || exit 0
|822field return-path | egrep -i “MAILER-DAEMON|POSTMASTER|ABUSE” >/dev/null && exit 100 || exit 0
|/opt/rt3/bin/rt-mailgate –queue help –action correspond –url https://ticket.company.net 2>/dev/null || exit 111
That means any mail goes to help@company.net will eventually use
rt-mailgate to make a secure http connection to the ticket system and create a ticket.
So the first thing that could go wrong is the mail server failing to receive the mail. To double check that I went ahead and send an email from my computer to help@company.net and checked the log on the smtp side.
tail -f /var/log/qmail/smtpd/current | tai64nlocal
I noticed the smtp server received the email based on the following line in the log
2007-08-20 18:29:44.718444500 tcpserver: ok 22971 smtp.company.net:1.2.3.4:25 NO_REVERSE_DNS:2.3.4.5::60738
Where 1.2.3.4 is the company smtp server IP and 2.3.4.5 is my computer sending the email. And it says OK. So we are good on smtp side. Now if you know qmail, you know that this mail then will be handed over eventually to qmail-local alias.
To verify that the mail did not fail on its way to the qmail-local I tried to look at the qmail send log
tail -f /var/log/qmail/current | tai64nlocal
And I see deferal in the delivery for the email that I sent to help@company.ent
2007-08-20 09:03:25.432989500 delivery 1804118: deferral:
Usually you should see a message right after the deferral: to explain why it is deferred like the followings i.e.:
2007-08-20 09:04:01.774745500 delivery 1804142: deferral: Sorry,_I_wasn’t_able_to_establish_an_SMTP_connection._(#4.4.1)/
2007-08-20 09:03:21.451940500 delivery 1804099: deferral: 4.4.8.50_does_not_like_recipient./Remote_host_said:_450_<blah
@blah.com>:_User_unknown_in_local_recipient_table/Giving_up_on_4.4.8.50./
But this time there is no message after deferral:. So I am still clueless as to why it is being deferred instead of being a delivered successfully. Well at least I know that mail is trying to make through. So the next possible point where mail could fail is at the rt-mailgate. If you are not sure why, look at the beginning of the article which shows how the communication is setup in the .qmail-help file
To troubleshoot if rt-mailgate is working I decided to run it in debug mode and send some garbage to it to see how it responds
echo this is a test | /opt/rt3/bin/rt-mailgate –queue help –action correspond –url https://ticket.company.net –debug
I see few lines of error with Crypt::SSLeay module. One of its library that is depending on openssl is failing to load. Why? looks like the openssl library itself changed to a newer version. It might have happend during ssh package upgrade which is very picky about the openssl library version. So that means either openssl library needs to be downgraded to a lower version to match the version Crypt::SSLeay, which probably will break ssh, OR recompile Crypt::SSLeay
I decided to just recompile Crypt::SSLeay and point to new openssl library during compile. I called CPAN to install Crypt::SSLeay module. When it rains it pours. The perl module is failing to compile with default CPAN install method. It is looking for openssl/ssh.h header file under /usr/local/ssl/include/openssl. So instead of using CPAN install command I used CPAN look command which let me compile the module manually from source. I then modified the include variable INC in the Makefile (run perl Makefile.PL first) and replaced include path from /usr/local/ssl/include/openssl to /usr/local/ssl/include. That is pretty much the only hack I needed to do during manul make. After that I did a re make and make install and now I have a fresh new Crypt::SSLeay. To make sure that fixed the rt-mailgate, I feeded some garbage, like before, to it and I got a ok and a Ticket number. Eeehaa secure http communication between mail server and ticket system have been restored and the day is saved!