Why few companies do DNS well.  Why fewer can scale it.
IP is the dominant protocol of networking and networking heavyweights like Ethernet, TCP, and MPLS are emerging as clear standards, but networks are a long way from becoming a commodity as long as DNS is not properly implemented. Few companies do DNS well. Fewer have the skill set to scale it reliably and securely.

Best practices?   They’re rarely documented, never mind automated – DNS has replaced routing as the new “black art” of networking.   Today’s networks rely on an IP services backbone of DHCP, DNS, and RADIUS – critical network services components that dictate availability.   The enterprise network must be available, scalable and flexible to meet the needs of both small and large organizations because any network downtime has a direct financial impact.

The moral here is :      Single points of failure can never be a good thing.

If your site is monetized and/or mission critical  – secure DNS with a secure 3rd-party vendor with a robust, Anycast-ed DNS server network.

Then have a mirrored content server infrastructure at seaprate physical locations, load-balanced on an ongoing basis.

If any DNS server goes down, DNS still works.  If your hosting provide suffers and outage or your internal content server becomes unavailable, traffic continues relatively uninterrupted to that content at a difficult location.

And disaster-recovery is of course facilitated if it ever comes to that.

1. DNS Servers – An Internet Achilles Heel
   May 28, 2007 … Scan finds that hundreds of thousands of the servers that act as the white pages of the Net are vulnerable to attack.news.zdnet.com/2100-1009_22-144018.html – 107k – Cached – Similar pages –

2. Is DNS the Achilles heel in your MySQL installation? | MySQL …

   Is DNS the Achilles heel in your MySQL installation? Posted by Baron Schwartz. Do you have skip_name_resolve set in your /etc/my.cnf? If not, consider it. …www.mysqlperformanceblog.com/2008/05/31/dns-achilles-heel-mysql-installation/ – 44k – Cached – Similar pages –

3. DNS Servers: An Internet Achilles‘ Heel | EDUCAUSE CONNECT

   In a presentation at the Black Hat conference last week, security researcher Dan Kaminsky argued that domain name system (DNS) servers represent a broad …connect.educause.edu/Library/Abstract/DNSServersAnInternetAchil/36500 – 15k – Cached – Similar pages –

4. DNS servers – an Internet Achilles‘ heel

   “That is almost 10 percent of the scanned DNS servers,” Kaminsky said in a presentation last week at the Black Hat security event in Las Vegas. …www.gss.co.uk/news/article/2145/go – 36k – Cached – Similar pages –

5. DNS servers–an Internet Achilles‘ heel

   9 posts – Last post: Aug 4, 2005

   Report as spam Discussion – Post 1 of 8; DNS servers–an Internet Achilles‘ heel : According to a presentation at the Black Hat security …

   techrepublic.com.com/5208-6230-0.html?forumID=4&threadID=178944&start=0 – 55k – Cached – Similar pages –

6. DNS – The Internet’s Achilles‘ Heel?

   DNS – The Internet’s Achilles‘ Heel? Tuesday May 10, 2005. | Commentary | Every time a major Web site like Google experiences a DNS outage, critics of the …compnetworking.about.com/b/2005/05/10/dns-the-internets-achilles-heel.htm – 23k – Cached – Similar pages –

7. DNS servers–an Internet Achilles‘ heel – CNET News

   Scan finds that hundreds of thousands of the servers that act as the white pages of the Net are vulnerable to attack. A CNET article by Joris Evers, …att.com.com/DNS-servers–an-Internet-Achilles-heel/2100-7349_3-5816061.html – 79k – Cached – Similar pages –

8. ISN 2005/08: [ISN] DNS servers — an Internet Achilles‘ heel

   http://news.com.com/DNS+servers–an+Internet+Achilles+heel/2100-7349_3-5816061. html By Joris Evers Staff Writer, CNET News.com August 3, 2005 Hundreds of …lists.jammed.com/ISN/2005/08/0014.html – 9k – Cached – Similar pages –

9. VenChar: DNS – The achilles heel of the Internet

   August 03, 2005. DNS – The achilles heel of the Internet. CNET has a good article on the vulnerability of the Internet to DNS “cache poisoning” attacks. …www.venchar.com/2005/08/dns_the_achille.html – 32k – Cached – Similar pages –

It sure seems a bit more severe than Network solutions admitted to…

Potential Latency on Network Solutions DNS | Network Solutions …
There may be some latency on Network Solutions DNS Severs and some queries may be timing out. This may include instances when someone types a domain name into
Network Solutions – Small business… – http://blog.networksolutions.com/

By now it must be obvious to readers of this blog that DNS resolver code as it exists on the name-servers of most hosting providers is not the safe structure it was once assumed to be.

Just Google ” DNS Achilles Heel ” for articles that started appearing years ago:

DNS Achilles Heel

And in case you thought your VOIP phone lines were safe, notice Packet8′s outage…

Hacker: “I guess I won’t ask permission then…”

Did you know that your DNS server may respond to queries from websites that you visit, or even domains that you send mail to, rather than just your own domain? An open DNS Server, or open recursive, responds to queries for domains that the DNS server is not authoritative for, and does so for anyone (rather than just clients on your local network).

Now, when DNS servers and mailservers were originally put into use, they were all open. That’s just how the Internet was way back when.

Over the years, spammers started relaying through open relays, so the best practice became not to run open relay mailservers. For quite a few years now, “best practice” has been not to have a DNS server be both authoritative and caching (doing recursive lookups). But most DNS servers are still open.

Once again, getting folks to act in their intelligent self-interest is not as easy as it might seem . . .

The problem is that there are now DDoS attacks (attacks that send lots of data to a computer, so that it becomes overloaded) that use open DNS servers, using amplification (sending small packets to a computer that then sends large packets to the victim, making it possible to send more data to the victim). Specifically, a UDP DNS packet is sent with a forged source IP address (the one of the victim), and a query is made in a small packet (about 75 bytes) for a domain that has a very large response packet (using EDNS0, it can be 4,000 or more bytes).

The response packet then goes to the victim. The victim gets about 50 times as much data as the attacker is sending out. So with a dialup connection, they could saturate a T1 line. Or, with a broadband connection – - well, you get the picture . . . it’s not pretty.

Web-surfers are not known for being a patient lot. ( If they were, who would ever pay for broadband? )

So . . . can they get to your site, first time, every time? Is your site really accessible by everyone trying to get there, or are you “bleeding” traffic?

Do you know how many unique visitors out of every 100, hour after hour, are getting 404 Page Errors or Server Time-Out messages?

How do you know?

Here are 10 more questions. If you don’t have the answers off the top of your head, contact this blog for a free analysis and save yourself an hour of research.

What resolver code are you running? What version? Is it patched? If patched, is it slower than it used to be?

Are you running an open recursive server?

Are your name-servers on the same /24 address space, or separate? Is that good or bad?

Do your serial numbers match across all name-servers?

Is your delegation “LAME” ? (No, I don’t mean the MP3-ripping program)

Do your parent and authoritative name-servers agree? ( ” Mommy & Daddy! Please don’t fight! ” )

CLICK for FREE Analysis Report of Your DNS

From Robert McMillan in PC World:

By day’s end, Kaminsky had even turned his most vocal critic, Matasano’s Ptacek, who issued a retraction on this blog after Kaminsky explained the details of his research over the telephone. “He has the goods,” Ptacek said afterward. While the attack builds on previous DNS research, it makes cache poisoning attacks extremely easy to pull off. “He’s pretty much taken it to point and click to an extent that we didn’t see coming.”

The whole article: http://www.pcworld.com/businesscenter/article/148151/internet_bug_fix_spawns_backlash_from_hackers.html

Then in an interview for Kim Zetter’s Threat Level in WIRED’s blogs, I thought these remarks of Kaminsky’s were particularly salient:

DK: “People are allowed to be very, very skeptical. But, you know, don’t be so skeptical that you’re telling people to not patch.

This is a really bad bug. And for everyone who (says), Oh, I knew about this years ago . . . no, you didn’t. Stop pretending you did. Because every time you say it, another network doesn’t patch (their system).

This (attack takes) ten seconds to hijack the net. . . . Unless you like other people reading your e-mail, go patch. If you want to actually see Google and Yahoo and MySpace and Facebook and the entire web, if you actually want to see the correct web sites, go patch. The debate about whether this bug is new or old is ultimately useless. In ten seconds, the ISP DNS servers are taken over. . . .

There are a couple million name servers on the internet. There are many million more that are not physically on the internet but are behind firewalls. Ultimately any name server that is not patched is vulnerable and will probably eventually be attacked. The attack is just too good and too easy. My grandma’s going to be in the audience (at Black Hat). My grandma’s going to understand the bug.”

For the full interview see http://blog.wired.com/27bstroke6/2008/07/kaminsky-on-how.html

(2) getting DDoSed

(3) having your customers syphoned off to a scammer’s copycat site to give up their secure information, thinking they are on your (previously) trusted site?

Here’s the story: One of my colleagues had the misfortune to speak with a paid employee who flatly announced in the North American vernacular, “Paying for DNS is “retarded.” Really?

Never mind the insensitivity of such a term, calling the buyer “retarded” insults the IT Professional or businessperson who has chosen to insure his investment. This misguided “genius” insinuates that the buyer who actually invests in something whose basic function can be had gratis is worse than a fool.

Some kind of DNS name-server set-up is necessary for any hosting solution. It started out 3 decades ago, and is still largely offered as, a “free” component of one’s hosting infrastructure. So what? Implying that paying for it is somehow foolish, or beyond that — “a very bad buying decision that only a mentally-challenged individual could be conned into” — must only come from someone either ignorant ( especially after July 8th, 2008, when Kaminsky’s recursive exploit was announced and patches released ) or someone with no investment in having a domain available 100% of the time.

Take a hint from the the “big dogs” at the enterprise level. They spend plenty of money on making sure their sites are up, running and optimized. One way or another, they pay for DNS, and pay a lot. They pay for zone files and/or code to be written, for failover and load-balancing boxes, and for people to manage them. But they also have a lot, if not everything, riding on those sites’ being available every single time someone clicks on their link, banner, or ad, or types-in their domain name.

For small business owners, to mid-sized company IT professionals, outsourcing makes a lot of sense if you can fix the both problems of vulnerability and sluggish performance in one action…

Yes, there are horror stories about Overages and so on, but frankly much less so than in your own cell phone bill. The legacy phone comanies and wireless plans are notorious for generating crazy bills in a usage-based model. Typically, though, at a high-end service you will have a rep assigned to your account whose business it is to make sure you’re in the optimum tier of service. They get neither kudos nor commissions when you get hit with overages… but they are compensated when they stay on top of your account, keep in touch and upgrade you properly as your traffic grows, keeping your traffic costs down and keeping your doorway open to customers coming to buy from you.

This is the distinction between paying to be on as big a network as you can afford, with full-redundancy distributed over many countries and continents VS a typical hosting provider or someone else who purports to manage DNS, but can’t or won’t guarantee uptime (or even if they were to do so, there would be little at stake as far as actual refunded fees go). Most likely, what you ultimately want, though, is 100% DNS uptime. Period. If you can’t afford it, you can get some protection and even some “guarantees” out there. Practially speaking, the more skin a DNS provider has in the game, meaning, the most invested in its infrastructure, the most actual nameservers physically located behind the firewalls of as many large ISP’s as possible, the faster the average global response times, the greater ability to dilute DDoS attacks, and the more you should expect to pay. Conversely, the more you pay, the more you should expect. Simple, no?

Here is an excerpt from their blog:

——————————————————————–

Monday, July 14. 2008

“(Provider Name Withheld) main site outage“

As I’ve mentioned in the past, if your DNS is mission-critical, consider investing in it. Then logically, consider what it’s worth to you to be up and running at optimum effeciency, and consider how much to invest in it as you grow.

A typical hosting provider offers 2 name-servers connected by one ethernet cable VS

Mid range managed DNS provider 32 name-servers (over ? connections) VS

Premium -  *Hundreds* of name-servers in 14 node locations in bomb-resistant structures each with Quad GigE connections — 4 ISP backbone providers in each.

As the saying goes, “no one was ever fired by going with the best available solution for mission-critical business functions” … whereas if a lesser provider fails you, who gets the blame?

This has caused some frenzied activity world-wide for IT professionals charged with locating and updating all their servers running either code. Although it is a recursive-level exploit, most will likely opt to update authoritative servers as well, and they will likely have to go through this all again in a couple of weeks when the final versions of the patches are released.

Some experts are calling this perhaps the most significant DNS exploit in the past 10 years, and is most likely to capture the attention of hackers who wish to compromise or control computers. It may be glib to suggest this, yet it is true that avoiding the use of open-source resolver code for your mission-critical DNS applications, in general, is one way to side-step issues like these.

The Internet gets a patch, as DNS bug is fixed
NetworkWorld.com – Southborough,MA,USA
The Internet Software Consortium’s open-source BIND (Berkeley Internet Name Domain) software runs on about 80 percent of the Internet’s DNS servers. …
See all stories on this topic

Major fix to DNS vulnerability impacts Windows, Debian
BetaNews – USA
The real vulnerability is not in Windows or Linux but in BIND, the most widely deployed DNS software everywhere. A security feature in BIND creates a …
See all stories on this topic

DNS at Risk From Multivendor Cache Poisoning
InternetNews.com – USA
Unix and Linux distributions widely use the open source BIND DNS server, which ISC manages. Kaminsky, who takes credit for discovering the flaw, …
See all stories on this topic

Major DNS flaw could disrupt the Internet
NetworkWorld.com – Southborough,MA,USA
He says Yahoo was vulnerable because it uses an older version of BIND but had committed to upgrading to BIND 9.0. Kaminsky says there’s a way to check for …
See all stories on this topic

ISC Acts Quickly to Shield BIND User Base
Market Wire (press release) – USA
In addition to patches for the current versions of BIND9, ISC has also released beta versions of upcoming maintenance releases, BIND 9.5.1b1 and BIND …
See all stories on this topic

Debian Security Advisory – bind9 (DSA-1604-1)
Help Net Security – Croatia
The BIND 8 legacy code base could not be updated to include the recommended countermeasure (source port randomization, see DSA-1603-1 for details). …
See all stories on this topic

Debian Security Advisory – bind9 (DSA-1603-1)
Help Net Security – Croatia
This update changes Debian’s BIND 9 packages to implement the recommended countermeasure: UDP query source port randomization. This change increases the …
See all stories on this topic

[USN-622-1] Bind vulnerability
Bugtraq: [USN-622-1] Bind vulnerability.
www.derkeiler.com: Bugtraq – http://www.derkeiler.com/Mailing-Lists/securityfocus/bugtraq

USN-622-1: Bind vulnerability
By KeesCook
Details follow: Dan Kaminsky discovered weaknesses in the DNS protocol as implemented by Bind. A remote attacker could exploit this to spoof DNS entries and poison DNS caches. Among other things, this could lead to misdirected email and …
Ubuntu – news, usn – http://www.ubuntu.com/taxonomy/term/1+2/0