sendmail queue

[Len Wilson]

Hey everybody,

We're using sendmail on our new Unix based server to send our monthly enewsletters. Tried twice now. Both times the send fills up the sendmail queue, and it takes about 48 hours for it to empty out. In the meantime any other emails generated by the apps on our site go to the end of the line, which means people don't get their passwords and stuff like that.

I've tried using command lines to get sendmail to process the queu without success.

Any suggestions on how I can avoid queue craziness?

Thanks ahead of time.

Len

[Greg Atkinson]

Originally Posted by Len Wilson

Hey everybody,

We're using sendmail on our new Unix based server to send our monthly enewsletters. Tried twice now. Both times the send fills up the sendmail queue, and it takes about 48 hours for it to empty out. In the meantime any other emails generated by the apps on our site go to the end of the line, which means people don't get their passwords and stuff like that.

I've tried using command lines to get sendmail to process the queu without success.

Any suggestions on how I can avoid queue craziness?

Thanks ahead of time.

Len

If you find out - let me know. My newsletter takes about 2 days to go out!

Wow -- I don't know what's causing your problems. I wonder if it's the mail program you are using. My newsletter to 28,000 subscribers only takes about 5 hours to send in batches of 50. What programs are you using to send your newsletters?

Tony Kanago is great with servers. I'm sure he will chime in when he sees this (or you should hire him to fix it for you).

[Len Wilson]

hey man. how's it going? we use MaxBulk Mailer which I absoutely love. On our old ISP it would send in a few hours no problem. So it has something to do with sendmail getting overloaded, as best as I can tell.

I'll PM Tony about it.

Thx.

Yeah -- if you're using the same mailer program then that would leave the server.

Tony's the man!

[Tony Kanago]

Alright, just to save time later, I'll cover all the basics I can.

(Edit- If you are using a dedicated server provider, then leave out any intranet details, but please include connection speed from the provider. )

1. What kind of hardware do you have? (Please include fairly detailed information on the processor, specifically.)
2. What is your current network configuration regarding physical hardware? Include the path from the internet to the intranet, as well please. (For example, internet -> unix gateway -> router -> server)
3. What is your internet and intranet speed? (For example, 10mbps from your ISP, 100mbps over Intranet)
4. What is your software intranet configuration like? (DHCP, forwarded ports through what hardware, (canonical)hostname, any dynamic DNS?)
5. What flavor of Unix/Linux/Solaris/etc. are you using?
6. Are you using any strange configurations of the OS/Network with the OS?
7. Are you using any Firewall services? (APF on the server, APF on the gateway, hardware firewall, etc?
8. Can you upload a copy of your sendmail configuration file and post a link?
9. Are you using any secondary mail applications, such as qmail, exim, qpopper, procmail, etc?
10. Are you using any mailing list handler applications in conjunction with sendmail?
11. Can you verify that when mails are queued, they are actually being sent to /var/spool/mqueue, or wherever sendmail is sent to put this to?

This should be a good start.

[Len Wilson]

Wow, this is complicated! I'll try to answer as best as I can. Maybe you can point me where to find answers to this stuff in some places.

>
> 1. What kind of hardware do you have? (Please include fairly detailed
> information on the processor, specifically.)

CPU - GenuineIntel, Intel(R) Xeon(TM) CPU 2.80GHz, Version - psa v7.5.2_build75050128.10 os_RedHat 9

> 2. What is your current network configuration regarding physical hardware?
> Include the path from the internet to the intranet, as well please. (For
> example, internet -> unix gateway -> router -> server)
>
> 3. What is your internet and intranet speed? (For example, 10mbps from your
> ISP, 100mbps over Intranet)

I use comcast cable here at my home office to connect to the internet, 3.2mpbs, and our host is GoDaddy's Virtual Dedicated Server service; I can't find info on their speed. One thing says 4Gbps but I don't know what that really means for us. The problem isn't here, because I can send 16,000 emails in about 2.5 hours here. Their tech support told me vaguely that they have a cap of 40 at a time on sending email but he didn't tell me what the interval was.

> 4. What is your software intranet configuration like? (DHCP, forwarded ports
> through what hardware, (canonical)hostname, any dynamic DNS?)

Don't have an intranet here, like I said above. The virtual dedicated server service is a dedicated service with an IP that is simply a larger server partitioned for smaller businesses with high bandwidth needs, like us.

> 5. What flavor of Unix/Linux/Solaris/etc. are you using?

Linux 2.4.20-021stab028.3.777-enterprise

> 6. Are you using any strange configurations of the OS/Network with the OS?

Not that I'm aware of.

> 7. Are you using any Firewall services? (APF on the server, APF on the
> gateway, hardware firewall, etc?

No, not here. I'm sure GoDaddy does but I don't have access to that info.

> 8. Can you upload a copy of your sendmail configuration file and post a link?

Is this the sendmail.cf file? I type a find on that on the command line and it says no such file or directory. How do I find it? Sorry for the Unix newbie. I've just learned this past summer.

> 9. Are you using any secondary mail applications, such as qmail, exim,
> qpopper, procmail, etc?

This is part of my confusion. Qmail is installed and is handling smtp, or so I thought, but the sendmail queue is getting populated when I blast. Maybe I don't understand how the 2 interface.

> 10. Are you using any mailing list handler applications in conjunction with
> sendmail?

We blast using MaxBulk Mailer on our local Mac OS and send through smtp.

> 11. Can you verify that when mails are queued, they are actually being sent to
> /var/spool/mqueue, or wherever sendmail is sent to put this to?

Not really. I found a /var/spool/mail file and when I open it using vi it gives me /var/qmail/queue and a bunch of subdirectories but I don't really know what I'm looking for here.

I hope this is enough info to help answer some questions or at least ask other ones.

Thanks a ton.

[Tony Kanago]

For one, I'd suggest during a low traffic time doing the command 'updatedb' as root, (or toor, and the command updatedb is found at /usr/libexec/locate.update on some other distrobutions,) and using the 'locate' command to more simply find files. It's great if you aren't really used to the directory structure or regular expressions.
--If from this you get a great big flood of lines, you can always use the command 'locate <whatever> | more' to allow you to scroll through the output line or page at a time.--

A few more questions!
1. So are you running Sendmail or Qmail to send out files? I'm not sure I understand exactly how the mailing list is getting to the server to begin with, so let's just say we're talking about your site registrations. Are you telling your web application for registration to use sendmail?

2. Are you running any form of a Nameserver application such as BIND on this server in conjunction with the overall hosting platform, or is godaddy handling the nameservers for you?

3. Are you in fact using the mailman application in turn with sendmail?

4. Did sendmail come installed on your distrobution, have you updated it yet, and either way, what version is it?

5. Approximately how big is the queue usually, or how long is the mailing list?

6. What is the command being executed by cron to process the queue, or what command are you entering manually?

7. Try looking in '/etc/mail/sendmail.cf' for your configuration file, as well as using 'locate sendmail.cf | more' to find it otherwise. Then, please upload a copy somewhere for me to see.

More than anything, Sendmail uses a configuration file built for speed, not for readability. There are two formats for this file, and one is meant to be pseudo-compiled into something sendmail executable. So for this, also use any search command to locate all *.mc files, as well as *.m4. List those for me as well.


Sorry to ask so much of you for this, but problems with sendmail can cover a large spectrum.

8.

[Len Wilson]

Originally Posted by Tony Kanago

1. So are you running Sendmail or Qmail to send out files? I'm not sure I understand exactly how the mailing list is getting to the server to begin with, so let's just say we're talking about your site registrations. Are you telling your web application for registration to use sendmail?

Sendmail. Not sure what role Qmail has in it, honestly, but the queue I'm seeing the files in is in Sendmail.

Originally Posted by Tony Kanago

2. Are you running any form of a Nameserver application such as BIND on this server in conjunction with the overall hosting platform, or is godaddy handling the nameservers for you?

GoDaddy.

Originally Posted by Tony Kanago

3. Are you in fact using the mailman application in turn with sendmail?

No.

Originally Posted by Tony Kanago

4. Did sendmail come installed on your distrobution, have you updated it yet, and either way, what version is it?

GoDaddy really has sendmail shut down. I can't do any -q commands to mess with the queue, and just now I tried this to check version: "sendmail -d0.1 -bt < /dev/null" and it came back unknown.

Originally Posted by Tony Kanago

5. Approximately how big is the queue usually, or how long is the mailing list?

The list is around 16,000 and the queue is massive. I can't get a summary (see above) but I can see the list, and it goes beyond the memory's ability to display all for the first 24 hours. It fills up at least 7-8 thousand.

Originally Posted by Tony Kanago

6. What is the command being executed by cron to process the queue, or what command are you entering manually?

No commands. It just proceses it at its preset intervals, which I cannot change. (See above).

Originally Posted by Tony Kanago

7. Try looking in '/etc/mail/sendmail.cf' for your configuration file, as well as using 'locate sendmail.cf | more' to find it otherwise. Then, please upload a copy somewhere for me to see.

The mail dir is empty except for SpamAssassin. The locate command gave me this:

[root@Midnight_Oil etc]# cd mail
[root@Midnight_Oil mail]# ls
spamassassin
[root@Midnight_Oil mail]# locate sendmail.cf | more
warning: locate: could not open database: /var/lib/slocate/slocate.db: No such file or directory
warning: You need to run the 'updatedb' command (as root) to create the database.

A friend of mine wrote me and said:

Originally Posted by Scott

If they said, "virtual dedicated servers of 40 emails per interval", then I would think it doesn't matter if it's qmail or sendmail since it's at the server level either way.

I think when they said you "could set up a relay for outgoing and bypass this restriction", they meant you can have someone else host your outgoing email so that they can configure your account to direct all outgoing traffic to you other email server (away from them), which would bypass their restriction.

In other words, godaddy will not change their restrictions and you will have to go with someone else in order to get your outgoing emails sent out immediately.

So, it looks like we're stuck unless we use a service to send our newsletters.

Lemme know if you have any other ideas.

Thanks,
Len

[Brian Busch]

newer versions of sendmail support multiple queues. i wonder if you're 1) out of luck due to limits by their sla, or 2) they're controlling the bandwidth you use (via tcp window control) rather than "intervals" in sendmail sending.

ask tim: he's given me his root password before to help him find performance issues on his server - i'll offer the same to you so I can look around and see what I can find. a forum is less than ideal to get this done.

you have to have a master's degree in nonsense to understand the macros related to sendmail. when I started in Unix back in the early 80s it was incomprehensible. I know they've layered more stuff on top now to get it back to being only nearly impossible to understand, but I do have deep Unix knowledge - including tcp, & smtp.

[Tony Kanago]

One of the biggest key slowdowns in sendmail is when you used a nameserver on the same box. Too many cross-references going out and back in can not only slow it down, but take up bandwidth if they limit you to a small pipe, (ie. 1.5mbps at any given time.)

Also, I'm not sure exactly how GoDaddy's semi-dedicated plans work out, but it's issues like these that make me wonder.. Your friend has a completely valid point. I don't know what you're paying for the server at the moment, but I can almost guarentee it would benefit you to move to another provider, such as LayeredTech.com. (I've worked with them before, great provider.)

Considering that this is likely not an option, and you can't get into sendmail well, you can assume a few other things. You probably can't upgrade your operating system or kernel independantly..

More likely than not, this isn't a case of technical issues such as settings or pipe limitations, it's simply GoDaddy not allowing this much throughput with sendmail. Other than a new provider or an external service, I don't really know what you could do.

If you do look into a new provider, a place like LayeredTech would be a good place to at least start looking for prices. For someone with a moderate amount of power, 1000gb bandwidth, and a 10mbps pipe. For 10 dollars more, you can make it a 100mbps pipe, and have email shooting out like crazy. Full OS control, any OS you'd like, full customizable. Good response time for tech support, overall a nice place. 65$ monthly, 20$ setup fee.

Normally I wouldn't be trying to suggest an alternate provider, but at this point it seems that there is no alternative as far as the email goes. When it's interrupting your ability to provide the main services of your site, an extra few dollars a month seems like an acceptable loss.

[Brian Busch]

Tony - if this is Linux (I think he said Redhat), dns lookups would be cached, even an exhorbitant amount of email like this. Lots of domains. Without looking (that is, logging in), I'd say we probably have a situation of simple bandwidth control on port 25. There might be something in their control program, but with shared hosts like this - (CMN/Tim had the same situation - bandwidth control) you see software at the driver level controling the tcp window to keep usage within some sla/limits.

i'd hate to suggest another server, but you might be able to solve this short-term by finding an emailing service to get your Christmas stuff out, which gives you another month or so to either move or work out a solution with your provider.

Even a "smallish" pipe like a t1 (like the 1.5mbps you reference tony) wouldn't take this long (3 days?) to send out this much email - it must be throttled. Nothing we can do about it if it is.

you might be able to find a service for the bulk emails, and keep your current provider until you have time (money) to move. Then, the daily traffic will be no problem, since you're not flooding your bandwidth.