Monthly Archives: April 2011

POSTMORTEM ON MXTOOLBOX EMAIL OUTAGE

4/25/2011

We screwed up. We cannot thank our customers enough for their patience.

As of Monday morning (4/25) we are fully operational, with all mailboxes recovered and operating normally. We are still in full response mode and by Monday evening we will have additional contingency plans in place. The objective of these activities is to ensure that if a similar event were to occur tomorrow, we would be in a position to respond much faster.

As it happens, we were also in the middle of a six month project to improve our redundancy and disaster recovery plans. The lessons learned during this outage will certainly be put to use as we build out this planned expansion.

We will be contacting our directly impacted customers over the next couple of days with additional information and to answer any questions.

On behalf of our entire team, thank you again for your understanding and patience.

Eric Rachal, President
MxToolBox, Inc.

Additional Technical information
MxToolBox hosts 15 email servers within the Amazon EC2 east coast region across 3 availability zones. On Thursday morning, April 21st, Amazon had major failures in two of these zones. During the event, many of our servers became unresponsive and we were unable to access our backups. Approximately 15% of our email customers were unable to send or receive email until Friday morning, and unable to access their previous mailbox data until 6:00am Monday morning. The MxToolBox.com website was not impacted.

On Friday morning we made the decision to move all effected mailboxes onto a standby server to restore mailflow and the ability to send and receive email, but without the restored mailbox data. In hindsight we should have done this much earlier. At the time we believed we would have access to our data volumes much sooner, allowing for a cleaner and quicker recovery path for our users.

Once we launched the backup mailboxes on the standby server and were heading into the weekend, we elected to slow down and proceed more cautiously. We tested and retested the process for merging the backup mailboxes with the restored data, with the objective of being fully operational for the start of business early Monday morning.

It is important to note that we don’t host all of our critical infrastructure “in the cloud” or with a single provider. We maintain critical email routing gateways, spam filtering services, LDAP directories, etc. on hardware we fully control.

April Newsletter: Cloud Based Web Security Mitigates Anti Virus Software Vulnerabilities

In today’s world, almost every business understands the value of running anti-virus software. The problem is more than 85% of malicious malware is distributed via the Web and each variant only has an average life-span of 5-7 hours. Small and medium-sized businesses are especially vulnerable to attacks because they are less likely to have the multiple layers of protection than larger enterprises.

Anti-virus software is still essential. It works with your computer to scan the processor’s memory for patterns that could indicate an infection. With 50,000 new malware samples discovered each day, antivirus software must try to stay current by having updates installed on every users computer whenever a new virus is identified. Daily scans search for viruses and stop any newly identified infections from tainting the computer while remembering the coding to prevent the specific virus from infecting the computer again. Without antivirus to remove viral corruptions, the virus can trick your computer to become a server to send the virus to other computers. However, anti-virus software typically does not update often enough to keep up with the malware problem today.

While anti-virus software will block many of the threats that attack your system, there are additional types of threats such as malware, spy-ware, hacking, and phishing that you need protection against . Earlier this month, there was the Epsilon Data Breach that set the stage for a large scale, targeted phishing attack. Every day, businesses who are only running anti-virus software are surprised to find out that they are under-protected when faced with the consequences of a web attack which can include data loss, viruses, trojans, malware, and spyware. 82% of organizations experience regular hacking and malware attacks and, even with current anti-virus software running, 62% of these organizations have had malware successfully infiltrate their network.

Cloud-based Web Security and policy filtering is an essential part of protecting your company from the inside out. The Internet is a powerful business tool. However, if left unsecured, it can be your greatest security risk. For example, your network can be breached by an employee inadvertently clicking on a (carefully disguised) link containing malware from a legitimate website. Currently, 80% of web-borne malware is distributed through legitimate websites.

MxToolBox is an expert in email security and we do all of our spam and virus filtering in the cloud. However, we’re seeing an undeniable need for companies to protect themselves from the increasing number of attacks coming through the web. In 2007, 10,000 malware samples were discovered daily. Today 50,000 unique malware samples are discovered each day: this is double from one year ago!

At MxToolBox, we want to be part of the solution and help companies protect themselves by blocking ALL objectionable content, malicious code and malware that is rife across the Internet.

Epsilon Data Breach: Could Lead to a Rise in Phishing Attacks

The recent Epsilon Press Release letting their clients know that a major security breach had occurred at their company has email security companies like MxToolBox on the alert. Epsilon is one of the largest distributor of permission-based emails in the world and that equates to around 40 billion marketing emails per year. They have revealed that millions of individual email addresses were exposed during the attack and breach of their servers. Among Epsilon’s customers are US Bank, JPMorgan Chase, TiVo, Capital One, the Home Shopping Network, LL Bean Visa Card, Ritz-Carlton Rewards, Best Buy, Disney Destinations, Walgreens, and many more. (net-security.org)

You may ask what exactly is phishing and why is it so dangerous? Phishing is a way of attempting to acquire sensitive information such as user names, passwords and credit card details by masquerading as a trustworthy entity in an electronic communication. Communications purporting to be from popular social web sites, auction sites, online payment processors or IT administrators are commonly used to lure the unsuspecting public. Phishing is typically carried out by e-mail or instant messaging, and it often directs users to enter details at a fake website whose look and feel are almost identical to the legitimate one (wikipedia.org).

Additional Resources
Original Epsilon Press Release regarding incident.
Epsilon Breach Affects JPMorgan Chase, Capital One
Massive Epsilon breach affects JPMorgan Chase, Capital One, US Bank and others
Seven of the top Fortune 10 use Epsilon