Summary of Controversy

 

Status

The DNSOP serverid proposal has not been adopted, nor deployed, nor has the server identification problem been solved as of January 2006. The DNSOP WG does not accept that it is necessary to alter the DNS protocol to solve this problem, and keeps working on the problem. The dnsop-serverid draft is now in its 5th revision. Identification of incorrectly operating servers remains a serious impediment and unsolved problem after almost 4 years of work on the problem.

History

November 1993, RFC 1546 "Host Anycasting Service" is issued.
Page 5:

It is important to remember that anycasting is a stateless service. An internetwork has no obligation to deliver two successive packets sent to the same anycast address to the same host.

IPv6 includes the notion of "anycast" addresses to improve multicast (RFC2460, 12/98) There are subsequently discussed problems with using the IPv6 anycast for DNS in IPv6. A significant problem is identification of malfunctioning servers

July 1998, RFC 2373 IP Version 6 Addressing Architecture is issued.

December, 1998 RFC 2460 Internet Protocol, Version 6 (IPv6) Specification is issued.

July 2, 1999, Ted Hardie submits draft-hardie-dnsop-shared-root-server-00 to the DNSOP group. This document later becomes RFC 3258. Abstract:

This memo describes a set of practices designed to enable the distribution of a root DNS server to multiple geographically and topologically distinct network locations. These practices presume that a single entity remains administratively and operationally responsible for each of the distributed servers.

This is essentially DNS Root Anycast. The term "shared unicast" is indistinguishable from Anycast, except that perhaps it doesn't trigger the "anycasting is a stateless service" alarms.

April, 2002 RFC 3258 Distributing Authoritative Name Servers via Shared Unicast Addresses issued.

On May 3, 2002, the dnsop-serverid-00 draft was introduced in the DNSOP WG by David Conrad in an attempt to address the problem of server identification. David Conrad is the sole listed author. server identification. This effort fails.

June 14, 2002 A typical IPv6 Anycast problem is presented by K. Robert Elz:

"How can one possibly tell if the server that will happen to respond to a particular anycast address right now is one of the "properly configured" ones or not?"

What this means is that it is impossible to identify a malfunctioning server which has returned an incorrect response, and thus impossible to correct the problem. It means that problems with anycast'd DNS servers are difficult or impossible to trace down and repair. This is a significant operational problem. And this serious problem still plagues Anycast DNS as of this writing.

The dnsop-serverid proposal introduced on the DNSOP WG to attempted to address the server identification problem by recommending the all server implementations implement a BIND proprietary scheme which identifies Unicast servers via the CHAOS Class query for "BIND.HOSTNAME". The draft proposes that "BIND.HOSTNAME" should be changed to "SERVER.ID" to make the query generic and vendor neutral. This scheme doesn't work with Anycast servers since the subsequent request may go to a different server.

July 5, 2002, DNSEXT message from Randy Bush noting the severity of the problem, notes that DNSSEC would be mandatory. (Bush apparently means using the DNSSEC signature facilities--its not clear that this signature uniquely identifiesAnycast'd DNS Servers). At this time, there seems general agreement by many that this is a severe problem

August, 2002, Bill "woody" Woodcock presentation on DNS Anycast to SANOG (South Asian Network Operators Group)

October 7, 2002, DNSOP message from Bill Manning that indicates DNSSEC isn't ready for production. Clearly lays out the consequences:

So... why are we considering experimenting with the live, production root system at this time? IMHO, this is lunacy. We have a working, experimental system in play where most (all) of these issues can be tested. Folks that have serious commercial interests in a stable system will not be amused when we start experimenting with the systems that they depend on.

October 9, 2002, DNSOP message from Brad Knowles that advocates production testing.

October 9, 2002, DNSOP message from Randy Bush hints that people are running Anycast without DNSSEC:

don't take some folk's comments too seriously. some of the folk screaming "caution" are deploying anycast roots without dnssec.

October 22, 2002, a DNSOP message from Randy Bush Bush proposes that if ISP create their own local copies of Anycast roots, that this might improve security by eliminating external route spoofing as a threat.

October 22, 2002, a DNSOP message from David Conrad Conrad doesn't want anyone else to operate Anycast roots. Conrad works for Nomimum, which shares an office with ISC. Some characterize ISC and Nomimum as "the BIND company". It is probably fair to say that that Nomimum is the commercial, for-profit software sales arm, while ISC functions as the non-profit research arm.

Conrad's asserts "there is an origin AS associated with the root IP address that can and should be locked down". The implication is that this improves security. However, this isn't true. Route spoofing attacks can include the "locked down" origin AS in the false BGP route data.

Conrad then asserts some very unusual claims:

Maybe it's just me, but this seems like a really bad idea. Might seem like a good idea to folks in authoritarian governments who want to muck about with the contents of the zone though.

Only authoritarian governments would think this is a good idea? Hmm. Well, it turns out that ISC sells root anycast copies to other ISPs.

Maybe that's why is seems like such a bad idea to Conrad.

October 23, 2002, DNSOP message from John Brown

October 29, 2002, a DNSOP message from Bill Woodcock Woodcock agrees that Anycast does not impove security against DoS attacks until there are a lot of instances:

Correct. It will merely sink attacks at the nearest instance. This is not particularly useful until there are a _lot_ of instances. For instance, if every major carrier ran instances near their customer edges, then all attacks would be sunk before they left any of those carriers, or before they even affected those carrier's internal backbones. That would be ideal, since it would localize the pain in the same locality as the fault. However, we're presumably quite a long ways away from being there. -Bill
Woodcock notes that in order to get benefits, carriers have to have copies near the customer edges. These carriers aren't "authoritarian governments".

November 5, 2002 Karrenberg proposes RIPE implement Anycast on K
Among other things, proposes charging fees to providers for this service.

November 16, 2002 Paul Vixie gives Anycast Plan to RSSAC

Anycast Plan (PV)
Sent copy of press release that ISC is deploying mirrors of root server with APNIC.
K also has a plan; see ripe's website. Have already had offers to host.
DRC: Would like members of the group to comment on rfc3258 revisions (T. Hardie document)
This addresses DDOS threat, not distributing bad data. That problem requires dnssec; this is solved with anycast.
But still would like to sign the root zone, especially for these anycasted machines.
Issue of route poisoning and re-direction is a concern, but is not something roots could resolve. DDOS is. Implementing dnssec is made somewhat more difficult with the anycasted servers, but not impossible. (Can solve with locking the IP to AS.)

November 17, 2002 ISC Announce Anycast agreement with APNIC
When problems are later found with Root DNS Anycast, attacks launched at APNIC meeting by Randy Bush and Peter Boothe

November 22, 2002, Paul Vixie casually mentions on DNSEXT WG (namedroppers) that 2 root server operators already plan to deploy "massive-scale bgp4 anycasting". (!) This is first mention of such plans with respect to IPv4 servers on the DNSEXT WG. While not entirely overlooked, this bombshell largely missed because the DNSEXT WG is embroiled in controversy over abuse of Dr. Dan Berstein

November 23, 2002 John Brown is an early supporter of Anycast. This is the second message about "anycast root DNS servers". It turns out that John Brown is a business partner of Suzanne Woolf.

February 4, 2003 Rob Austein recaps his talk at IETF 55. Austein summarize the Anycast debate in the message:

The one thing that (almost) all sides of the "are anycast roots a good idea?" debate agree on is that the sooner we get a working DNSSEC out there, the less scary anycast roots will be.
Implied in Austein's summary is that there is a discussion of Anycast Root servers somewhere else. Also implied is that some people have grave concerns about this practice.

February 9, 2003 Nanog 27, Presentation by Suzanne Woolf entitled "BGP4 Anycast for Root Name Service -- Threat or Menace?" Woolf is the Program Manager at ISC in charge of Anycast.

March 24, 2003 Paul Vixie Anycast status to RSSAC

April 18, 2003 Nanog Message Paul Vixie says "bgp-anycast isn't even controversial at this time"

July 13, 2003 RSSAC Meeting: What is Anycast and how does it affect security?

November 9, 2003 Anycast status report at RSSAC meeting Anycast discussion about RFC 2373 (IPv6)
Woolf reports that ICANN wants more reports on Anycast and DNSSEC.

December 1, 2003 Vixie on ICANN. Says ICANN not responsible for Root DNS stability. Says Root DNS Anycast preceded ICANN's existance. ICANN was incorporated on November 6, 1998, and took responsibility for DNS from the US Department of Commerce on November 25, 1998. But Vixie writes:

again there's a tense problem. there was no "switch to" anycast. the last time those thirteen (or eight) ip addresses were each served by a single host in a single location was some time in the early 1990's.
This seems to be contradicted by the Anycast Plan presented to RSSAC Nov 16, 2002. Vixie seems to be fabricating assurances.

December 8, 2003 IETF message about Anycast concerns by Eliot Lear Lear notes problem of server identification:

I expressed this concern privately to Paul Vixie who provided me a very satisfactory answer: you can query the name server for a record that will provide you uniquely identifying information. I'll let Paul describe this, but it amounts to the borrowing of an unused class for management purposes.

This method of identification fails to identify Anycast servers. A solution is sought in the serverid draft. The reason it fails is described in the introduction to the nsid draft which will be submitted September 12, 2005, which documents a proposal to add a new EDNSO request option to request that the server include identification information in the response. This demonstrates the over-statement of assurances by Vixie.

February 9, 2004

February 20, 2004 Patrick Gilmore responds on Nanog to question about "Windows Anycast" with link to Microsoft clustering. This is not stateful Anycast.

February 29, 2004 RSSAC status on Anycast. Two pronged approach to serverid problem. One is serverid draft. Other is nsid draft, which uses EDNSO extension to put server identification in response.

July 22, 2004 version 2 of dnsop-serverid is issued. Suzanne Woolf is added as an author. This effort fails.

August 1, 2004 Vixie tells RSSAC that Anycast is no longer controversial This is an overstatement. There is a great deal of controversy, and a number of serious problems that remain unsolved at this time.

Successful and wide deployment by about 1/2 servers, no longer controversial.
"everything seems to be ok" is the message.

[...]

One issue is unique server identification. still issue of dealing with 'which server served me' message info for when things go wrong. server id requirements being perused by Woolf and Austien in IETF DNSEXT wg. Work stalled, restarting.

Work has stalled because it is impossible to guarantee sending 2 sequential packets to the same Anycast server. This is the same issue as with TCP. This issue was first identified in RFC 1546.

September 29, 2004, DNSOP Message from Dean Anderson. Notes previous assertions about single paths are wrong. Posts Cisco link on PPLB, and suggests that DNS Root Anycast should be reconsidered.

September 30, 2004, DNSOP Message from Paul Vixie

September 30, 2004, DNSOP Message from Dean Anderson

September 30, 2004, DNSOP Message from John Brown Says he has no connection with ISC. It is later discovered that Brown is a business partner with Suzanne Woolf, of ISC. Brown is also associated with RIPE.

October 2, 2004, DNSOP, IETF Message from Paul Vixie. Says that PPLB is only link bundling technology. Refers to "other anycast services around the net, like woody's and rodney's," Woody is Bill Woodcock,

October 3, 2004 DNSOP Message by Iljitsch van Beijnum Acknowledges that Anderson has a point, notes that Anycast and PPLB is "quite deadly to UDP fragments

October 17, 2004 Verisign reports Anycast problems Data confirms Andersons concerns.
Reports "interesting behavior" in slide 27:

+ Looked at sites that were seen in two or more
anycast sites
+ 3.69% of all traffic over three days was seen at
two or more sites
+ More than expected
+ Looked at a couple of examples…
The "interesting behavior" is consistent with what would be expected from current (small) usage of PPLB, as Anderson asserted. PPLB is a significant benefit for high availability, and its use is expected to grow.
+ Expected to see a saw tooth distribution –
instead have a noisy distribution in many cases
+ Does not affect UDP
+ DO NOT RUN Anycast with Stateful Transport
Verisign emphatically says Anycast can't be run with stateful transport!

 

November 1, 2004, Abley (ISC) submits Anycast to GROW WG

November 7, 2004, RSSAC Meeting

Anycast: - Paul Vixie 
No new features, there is incremental growth with much the same architecture
Mark Kosters presents data from nanog on the "J" instance. Highights are: unexplained jitter
TCP concerns - does not seem to affect UDP
Stay the course. 
Mark Kosters: - still doing analysis to understand events
David Conrad: - any similar data from others? 
Paul Vixie:- no - but may be due to the nature of the way we do anycast. 
Lars Liman: - no - but we have not done the analysis. 
Daniel Karrenberg: - our measurements for "K" do not see what "J" has found. - may be where 
we are looking from?
Akira Kato: - M is still evolving - no data as of yet
Bill Manning: - v6 anycast is an open issue, per previous agenda item

December 20, 2004 Nanog Message Paul Vixie says DNS Anycast pioneered commercially in 1997. Claims people would have seen PPLB problems. If fact, that depends on whether TCP was used in 1997.

this is the second important point you raise. anycast isn't new. rodney pioneered it commercially in 1997 or so. it had been in campus-area use for at least six years by that time and perhaps 10 years depending on how you count. akamai has been using it since 1999 or so. if there were any stability problems like the pplb assertions made elsewhere in this thread, we'd've all been seeing them for a long time by now.
See message by Patrick Gilmore about Vixie fabricating claims about Akamai doing http Anycast. Akamai DNS Anycast doesn't have any stateful requirements or obligations. Stateless DNS Anycast is compatible with RFC 1546. This is not proof that stateful anycast is safe.

February 23, 2005 Randy Bush, Peter Boothe attack Verisign report at APNIC Meeting
APNIC is the first Anycast server setup by ISC.

• Verisign presentation "Life and Times of J-Root"
http://www.nanog.org/mtg-0410/pdf/kosters.pdf
• Foils 27 to 29, reported non-trivial routing jitter
and therefore suggested "DO NOT RUN anycast
with stateful transport."
• But for almost a decade, there have been reports
of successful delivery of stateful services over
anycast
Bush and Booth claim that "stateful services delievered for over a decade" is false. RFC1543 clearly explains that this is impossible. And Bush/Boothe do not offer any examples of such stateful anycast services to support their assertion. Ultimately no explanation is given for the TCP behavior noted by Verisign.

March 6, 2005 RSSAC Meeting

Randy Bush was invited to workshop at APNIC, and he had some comments about anycast DNS.
Daniel Karrenberg has not been able to replicate what he has observed, but it could be because of where on the net it is happening. It's really mostly about routing, not dns. Still get answers, but get them from rapidly changing instances. There may be more data presented at the CAIDA/WIDE workshop.

March 29, 2005 DNSOP Message by Vixie Says that HTTP Anycast is a bad idea. Says he keeps telling Akamai and speedera its a bad idea.

May 2, 2005 Paul Vixie calls Av8 Internet "dv8", and calls Anderson a crackpot. This is an ad hominem on Anderson's assertions about the PPLB/TCP DNS Anycast issue. Calling Anderson names does not make Anderson wrong about the TCP DNS Anycast issue. In 2006, Anderson is ultimately vindicated on the TCP DNS Anycast issue.

May 3, 2005 Nanog Message by Patrick Gilmore Says Akamai doesn't do http anycast. But Gilmore is convinced it would work.

On May 3, 2005, Nanog Message by Anderson complains about ad hominem. Harris ignores the complaint.

On May 3, 2005, Nanog Message by Anderson also responds directly to the attacks. Anderson explains his motivation and previous mistakes in dealing with such attacks. There is no ad hominem in this message.

On May 3, 2005, Regarding Anderson's response to the attacks, Harris makes another false accusation of ad hominem on Vixie by Anderson! With no hint of shame or irony, Harris actually includes the text of Vixie calling Anderson a crackpot, in her message to Anderson.

On May 3, 2005, Anderson refutes an explanation by Vixie of why DNS anycast is somehow safe. This is an entirely technical explanation. Vixie has made claims that are irrelevant to the assertions made about DNS Anycast.

On May 4, 2005, Harris again makes another false accusation of ad hominem by Anderson. But there is no ad hominem in Anderson's message. Anderson says Vixie is wrong for entirely technical reasons. Harris also claims that Anderson's message on May 3rd ignored her May 3rd warning. This too is false. Harris' off-list message was queued in with ordinary spam and other email, and was not read until May 4th. Harris is a participant in the fraud of TCP DNS Anycast stability.

So, effective May 4 2005, Harris again banned Anderson. Although the new "reformed" rules require a limit of 6 months, Anderson remains banned as of April 16th, 2006. It seems permanent.

On May 5, 2005, Daniel Karrenberg enters the DNS Anycast fray, and posts a preview of his upcoming paper. Karrenberg is responding to Anderson's message.

July 30, 2005 RSSAC Meeting

August 30, 2005 GROW Message by Anderson summarizes the problems with stateful Anycast and RFC 1812

Anycast update - Matt Larson

     the update from ICANN-GAC - Luxumbourg
     Bill Manning - recent reports of unauthorized copies of anycast 
     instances - they appear to exist.  Do we have good tools to detect
     and mitigate?
     Rob - IPv6 anycast has been fixed in the IETF.

September 12, 2005 first draft of DNS Name Server Identifier Option (NSID) submitted to the DNSEXT working group.

Existing ad-hoc mechanisms allow an operator to send follow-up queries when it is necessary to debug such a configuration, but there are situations in which this is not a totally satisfactory solution, since anycast routing may have changed, or the server pool in question may be behind some kind of extremely dynamic load balancing hardware. Thus, while these ad-hoc mechanisms are certainly better than nothing (and have the advantage of already being deployed), a better solution seems desirable.

Given that a DNS query is an idempotent operation with no retained state, it would appear that the only completely reliable way to obtain the identity of the name server which responded to a particular query is to have that name server include identifying information in the response itself.

After 3 years, it is finally discovered that RFC1546 is correct after all. Actually, a TCP connection has retained state. And yet, ISC/RIPE still claims that TCP operation isn't threatened by Anycast!

September 23, 2005 Anderson tells Harald Alvestrand that Root DNS Anycast threatens DNSSEC deployment

September 23, 2005 Threat by David Kessens on Anderson.

September 23, 2005 Kessens begins to carry out threat after Anderson complains about threat

September 23, 2005 Ad Hominem attack on Anderson by Bert Hubert

October 14, 2005 RFC3683 Action initiated against Anderson by Kessens. Dave Crocker is listed as sole author

October 28, 2005 version 5 of dnsop-serverid is submitted. The title has been changed to "Requirements for a Mechanism Identifying a Name Server Instance". This draft does not even bother to propose a solution, but rather describes the advantages and disadvantages of the BIND proprietary server identification scheme for Unicast servers. It then goes on to state some desirable characteristics of a "vendor neutral" solution.

November 6, 2005 RSSAC Meeting

Anycast status...   Liman * Brian

            Not significant changes - slow deployment for J,I,K - F continues

            scheduled plan. A,D,E, and L remain unicast.

January 17, 2006 Anderson receives RIPE source code used by Karrenberg. The RIPE code only does UDP queries, and doesn't do TCP or EDNSO testing. The reason Karrenberg doesn't see "J's" problems at the 11/04 and 3/05 RSSAC meetings is because they aren't testing anything but UDP! Of course they don't see TCP or other problems!