« For anyone wondering why I'm getting married... | Main | On single-sourcing »
So over on The Apple Phone Show, Liana Lehua posted an article entitled POP and IMAP, what's the Buzz?, that while well - intentioned, (wants to explain such things in an easy to understand manner), ends up looking like a badly done, (has real factual errors) article on IMAP that was really, in theory about Yahoo Push email.
If it seems confusing to you, well, it is to me as well. The fact that when I pointed out some of the errors, I got snarked at by Scott Bourne, my comment was deleted, and he managed to throw in a few comments about how I need to learn about targeting my audience. Evidently, I am incapable of relating to a non-technical audience. The people I've supported over the years would be amused by this, but oh well, Scott's more concerned with saving face than being right.
However, I am far more bothered that an article with real factual errors is being held as something it's not, so in the spirit of of the intarweb, I'm going to to do an overview of POP and IMAP, then talk a little about Yahoo's Push email. If you're fairly technical, this isn't the article you want, it's really for people who are perhaps dealing with things like IMAP for the first time, and want to know just what the heck is going on. I will do my best to avoid waxing technophile but it's kinda hard for me to avoid ;-)
POP
Of the three, POP, aka POP3 is the oldest email protocol. POP is really quite simple to understand, and conceptually, it works rather like going to the UPS store to get a package. When you connect to your email server, all email in your inbox is copied to your email program on your computer or iPhone, and it is deleted off the server. Just like when you get your package from the UPS store, it's in your possession, and no longer in the UPS store. POP is very simple to deal with from the user and the programmer perspective. It does have some disadvantages. If you have a slow connection, and a lot of email, well, you're going to be waiting a bit for all that to download to your computer or iPhone. If you have a lot of attachments, then you'll be waiting even longer, as the entire message has to completely download to your system before you can read it. If you aren't at the computer you normally read your email from/don't have your iPhone with you, then you can't read any of your old email. By the same token, any email you download to the computer you are at won't be easily available to the computer you normally use. Now, if you only have one computer that you ever use, and only read all your email in one place, this isn't a real problem.
But you have an iPhone. Now what? Well, as it turns out, POP has an option to kind of work around this. This options leaves the email on the server after you download it. In fact, this is the default for the iPhone, which makes sense. So now, as long as you have your email program set up to leave the email up on the server, there's no problem right? It's always available, life is good.
Well, only kind of. Remember, with POP, you have to download all the available email when you connect. If you leave your email on the server, that can start taking longer and longer. On a slow connection? It's even worse when you consider that spammers are now trying to use PDF attachments to spam you with. So if your email provider doesn't have a good anti-spam setup, then you get all those attachments downloaded to your iPhone. Then you get them all downloaded to your main Mac or Windows machine. To keep using our UPS analogy, this would be like every time you went to a different UPS store, they gave you another copy of every package you'd ever had delivered. No one needs that many packages. Another issue with POP is that your email storage is only as reliable as the machine it lives on. With POP, if you aren't careful about backing up, a hard drive crash could easily destroy all your email archives. Of course, we ALL back up, right? ;-)
So if you're using multiple computers or devices to read your email, (and if you're an iPhone user, this is probably you), and you have a slow connection, POP is still simple, but it's no longer so great. However, there is another answer: IMAP.
IMAP
IMAP was created to make up for POP's deficiencies. The most important difference between POP and IMAP is where your email lives. By default, with IMAP, your email lives on the server. Assuming your mail server is run by competent folks, this is actually a good thing. If nothing else, servers tend to have redundant drives, better backups, more reliable hardware, better power, etc., than your typical laptop/desktop/phone. Face it, you don't see a lot of Xserves taken out by small children, pets, or people tripping while carrying them to a meeting.
IMAP is also designed to be more efficient over bad connections than POP. In a perfect world, when you connect to an IMAP server, (Yes, I know Liana said IMAP = Push email. That's not correct as we'll see), you download a list of headers. Headers are the parts of the message that let you see who sent it, when, the subject of the message, and the like. Email headers are, as a rule, very small, and so a lot of them can be downloaded very fast even over a slow connection. As an example, way back in the dark ages, when I was running Netscape Communicator 4.5 on a Powerbook 3400c, I could download over 3,000 IMAP headers in under 5 minutes on a 33.6K modem connection. For all you people kvetching about EDGE speeds? EDGE is blindingly fast compared to that. The important part here is that by only downloading the headers, our perfect IMAP client lets you see how much email you have very quickly, without making you download the whole kit & kaboodle.
So now you want to read a message. You select the message, and our perfect IMAP client then only downloads the body of the message. Since the body of the message is the part you read, this makes sense. Rich text, HTML, plain text, doesn't matter. Even a fairly long message isn't going to have that much stuff in the body. So over a typical EDGE connection, this is not a long delay. Even over a modem, it's not really long. But what about attachments? What if you have a 2MB or greater attachment that you don't want to deal with? Well, this is one of the beauties of IMAP. You don't have to. The attachment isn't, or at least shouldn't be downloaded until you explicitly choose to. So you can read the message and blow off the attachment until you get to a faster machine, one with more storage, a faster connection, whatever. Keep in mind, I am talking about a "perfect" IMAP client. Not many clients are that perfect.
Now, when I say "download" here, I'm not talking about how POP does things. What I'm really doing is copying the information in the message to my local mac/iPhone/whatever. It's all still "physically" on the server. Why is this good? Because if someone sends me a big horkin' WMV file as an attachment, I don't want that on my iPhone. For one, I can't view it, for another, it's too friggin' big. With POP, I'd have to set my client(s) to always leave mail on the server. With IMAP, that's how it works until you move it off, either to a local folder on your client, or you delete it. This means that by default, that same message I saw on my iPhone will still be there when I get to my Mac. However, unlike POP, since IMAP is a smarter protocol, when I bring up my email client, it will show the message as being read already, since I did read it...on my iPhone. If we go back to our UPS example, this would be like the "Star Trek" version of UPS, where no matter where I went, I could get to any package in UPS's system nearly instantly, but I'd always have access to that one package, instead of making copies. I could get just the information I wanted on the package, or even open it up. Then when I was done, just walk out, and know that the package would still be available for me whenever I wanted. IMAP is the email equivalent of having your cake and eating it too. Only with less calories and fat.
If I really want to get all complicated, I can create email folders on the server and use rules to move my email to those folders out of my inbox. That way, even when I file my mail, it's still available from well, any IMAP client. If I have a really kick-butt ISP, then they have a server that lets me create server-side rules. These are rules that live and run on the server. They work just like rules in Mail, Entourage, Eudora, etc., but since they run on the server, they don't require me to leave my email program running to use them. So even my iPhone can reap the benefits of server rules. Server rules just leave POP in the dust.
Now, there are some disadvantages to IMAP. First, if you have an email provider that is stingy with space, IMAP makes it really easy to run out of space. This is why I have 2GB with over a GB dedicated just to email in my .Mac accounts. Secondly, deleting email can be more complicated with IMAP in that you can either say "Move it to a 'deleted items' folder" or 'Mark' it for deletion. (For a point of reference, the iPhone does the latter.) With the former, you have to be careful, because even though it's in a deleted items folder, it's still up on the server, and can still count against your email quota, if you have one. The latter is more obvious to see in an IMAP email client, (for example, in Microsoft Entourage, items 'marked' for deletion have a line through them), but you still have to take another step to really delete the email, aka "purging" items marked for deletion. On my Mac, I just have things moved to the deleted items folder, and I empty that when I quit my email client. For stuff I 'delete' on my iPhone, I just manually purge them in the email client. Off-line access can be tricky too. After all, the only way to have off-line access with IMAP is to cache the messages (and optionally attachments) locally. This would seem to make IMAP about as useful as POP, but remember, unlike POP, you haven't removed anything from the server, and most email clients don't require you to cache the entire message, attachments and all. So even IMAP caching can be more efficient than POP. Finally, a lot of email providers either don't do IMAP outside of webmail, (which almost requires IMAP), or they're so stingy on your server space that it's almost worthless.
Now, while IMAP is most definitely not "Push" email, it can emulate a lot of the features of a push email service/client, such as Goodlink or Microsoft Exchange. The way it does this is via the IDLE command. The IDLE command, at a VERY basic level, lets the client tell the server "Hey, when something happens, tell me". The server, when something happens, i.e. new email shows up says "Hey! Something happened, and here's what it is." So, as new email shows up, you can get it without having to do an actual email check. This is technically not Push email, as the client has to start the process, whereas with true Push, the server says "Yo, here's your email".
However, not all IMAP servers support IDLE. This is not a big deal, it's not required by the standard.
A clarification supplied by a reader who is smarter than I about such things. IDLE is not actually part of the IMAP standard. It is an optional extension to IMAP4 and IMAP4rev1. IDLE's not the only one here, there are a few IMAP features that are not a part of the actual IMAP4 standard. There are new versions of IMAP that have been proposed, such as P-IMAP and Lemonade that incorporate IDLE into the standard, but as it stands currently, the IMAP4 *standard* does not include IDLE. Much thanks to this reader, who wishes to remain anonymous.
Amusingly enough, Apple's .Mac servers don't support IDLE. (If they do, they hide it well.) Using Microsoft Entourage's ability to get the features an IMAP server supports, we see the following:
Now, there's a lot of stuff in there that's really only of use to email administrators, but what we don't see is the word "IDLE". What should we see on a server that does support IDLE? Check out the same information for my bynkii.com email server:
Looking at that screen shot, we see the second word from the right is "IDLE". So this email server does support it, whereas .Mac doesn't. If I had to guess, I would say that .Mac has a lot of email servers, and upgrading a large server farm is not something you do often, (or at all if you can help it). I'll also hazard that this kind of upgrade is not minor at all.
So where does this leave Yahoo!'s push email?
Yahoo! Push
As near as I can tell, Yahoo! is just using IDLE, although based on some of the comments I've seen, not very well. However, in Yahoo!'s defense, based on how the iPhone deals with my two accounts on servers that support IDLE, it may not be all Yahoo!'s fault. So, until I get better info, which can always happen, I'll go with Yahoo! using IMAP's IDLE feature to emulate "push" email. I would also REALLY like it if the iPhone actually used IDLE better. That would leave me with only my .Mac account being the one IMAP account I have that doesn't support IDLE.
And behold, the intarweb did bring people with patience, such as Conrad who did reference traffic-sniffing and say that Yahoo! does not in fact appear to use IDLE, but rather some other scheme which looks to involve sending the new message notification over the cell network. That wouldn't be unheard of, Microsoft did something similar for early versions of OTA, (Over The Air) ActiveSync. Now I'm almost really curious as to just what Yahoo! really is using, and I wish I knew someone over there who could tell me.
So that's a very basic look at POP, IMAP, and Yahoo! Push email on the iPhone. Hopefully, it helped you more than it confused you. If not, leave a comment, and I'll try to make it more clear.
On a personal note, this article was started as a bit of a finger to The Apple Phone Show, particularly after a few emails where my evidently odd insistence that one be correct even when talking to the less-technical crowd was viewed as "nitpicky" and an attempt to drag the article down into "pixel - pushing". I cannot possibly disagree more. Like everyone in my field, I started out knowing, well, nothing. I had to learn, and make the journey from ignorance into knowledge just like everyone else. Without accurate information along the way, that journey would have been impossible. I find it rather insulting in the extreme that anyone would defend inaccuracy for any reason, but especially in an article aimed at beginners, who, because they don't know any better, need to be able to trust that what they're reading is accurate, no matter how it is written, or at what audience it is aimed at.
While my normal audience is well able to catch any errors, I don't feel that a non-technical reader is any less deserving of accurate information than someone with a decade of experience. It's for that reason, along with the attitude I got from the Apple Phone Show that I have stopped reading that site.
okay, it was mostly the snark. Dude, when someone points out real factual errors, don't get your damned hackles up. Just admit it, correct it and move on. Yeesh.
Technorati Tags: Email, iPhone, TEH STOOPUD, WTF?
Posted by John C. Welch at 23:17 | Permalink
Comments
Traffic sniffed between the iPhone and Yahoo!'s server indicates that IDLE isn't used for push e-mail. Some iPhone owners, moreover, have manually logged into the IMAP server and verified that it doesn't support IDLE. The current theory is that the notification for new e-mail is instead being sent over the cellular network.
Posted by: Conrad Hirano | August 1, 2007 2:30 AM
Cool, thanks for that info Conrad.
Posted by: John C. Welch | August 1, 2007 7:03 AM
Yahoo! are apparently using some kind of out-of-band notification to tell the iPhone when mail arrives on the IMAP server:
"XYMPKI provides Yahoo! IMAP with information on the phone, that enables an alert about new email to be sent by an out of band alert mechanism (which we speculate is SMS)."
http://blog.isode.com/2007/07/iphone-signific.html
Posted by: Nick | August 1, 2007 12:08 PM
Excellent overview of the protocols! Our IT guy was completely unable to explain things this clearly.
My favorite description: "the "Star Trek" version of UPS"
Posted by: Lynch | August 1, 2007 12:48 PM
Well, I started with the UPS example, and wanted to keep it going, since it's something that people can relate to on a wide scale.
But thanks! I'm glad it was able to help.
Posted by: John C. Welch | August 1, 2007 1:44 PM
I have found the yahoo push to be unreliable at best, and seems to only work when EDGE (and not wifi) is the active connection. Does anyone else see that?
Posted by: Eytan Bernet | August 1, 2007 2:51 PM
IMAP IDLE is the "killer app" for the iPhone in my opinion. I'm kind of pissed off to see that the iPhone isn't doing IMAP IDLE, as I was pretty sure that's what Steve announced. Now that I think back, I think he said Push IMAP, which I probably assumed was IMAP IDLE.
Posted by: Charles Plater | August 1, 2007 4:00 PM
Eytan, as i'm finding out, Yahoo Push is pretty wierd. It uses out of band UDP data to tell the phone there's mail. Thus far, it's a pretty fucked up implementation.
Posted by: John C. Welch | August 1, 2007 9:49 PM
I find it odd that information on what Apple is doing with "push" imap is so hard to come by.
Perhaps this is an interim solution that Apple doesn't want other servers/isps using while they implement something more standards based.
Posted by: James Bailey | August 2, 2007 12:19 PM
Actually, I think it has more to do with Yahoo! not talking a whole lot about how their push email setup works. I'm working on another article talking more about push, and some of the proposed revs to IMAP in general. But Yahoo!'s implementation looks to be proprietary to them.
Posted by: John C. Welch | August 2, 2007 1:53 PM
Hello
I am very frustrated at trying to find the right device and service to get consistent "push email"--if there even is such a thing. I bought my first PDA ( the newest Blackerry) a couple of months ago only to return it after finding out that their email was only checked every 15 minutes after being told that it was instantaneous. I then bought an iPhone after being told again that I could get instantaneous email. Again after finding out again the mail was pulled in only every 15 minutes. I then took Apples solution to go with Yahoo "push email" and after several phone calls to both apple and yahoo it seems that 70% pushed email is the best I can expect. The business I am hoping to run using a PDA requires 100% of my mail to come through emmediately--or close to it.
And so my question is -----is there any service provider, program and PDA where I could experience consistent pushed email.
Thanks
Posted by: Darrol | August 15, 2007 7:27 AM
@darrol: yeah, If you subscribe to RIM's BIS OR BES on any blackberry device, you will experience the most amazing and lighting fast push email which, due to their own use of private servers, also supports just about every email service that offers pop only (i.e. gmail), my biggest regret having switched from a bbpearl to the iphone is the missing push mail. (god do I miss it)
Posted by: Anonymous | September 16, 2007 3:09 AM
A fairly easy way to get IDLE mail in small companies is to run your mail server on OS X server (I'm sure there are a few Mac heads among you). Included in the price of OS X server is a fairly easy to implement (POP/)IMAP server (why would you use POP when you can have IMAP?) that supports the IDLE command. The thing about IDLE is that the client has to maintain an open connection to the server. Presumably the connection is without traffic (i.e. doesn't cost you anything, hopefully - that's a worry if you don't have a generous data plan) until you get the notification from the server. I have set one up recently and was surprised that IDLE was included out-of-the-box.
Thing is, I find that most mobile phones have a half-baked IMAP client, that only really synchronizes the Inbox. I personally find that annoying. I want my SENT/TRASH/Personal Folders to be all available on all my devices and in sync. Until the iPhone is available here in Sweden (and even then, I'm not sure I'm ready for it - or it for me), I'm kind of loyal to SonyEricsson, but even on their flagship P1i, IMAP is rubbish.
Anyone with a great IMAP on mobile (apart from Blackberry)?
Posted by: Nico | September 24, 2007 1:04 AM
