PING

In this episode of PING, APNIC’s Chief Scientist, Geoff Huston explores bgp "Zombies" which are routes which should have been removed, but are still there. They're the living dead of routes. How does this happen?Back in the early 2000s Gert Döring in the RIPE NCC region was collating a state of BGP for IPv6 report, and knew each of the 300 or so IPv6 announcements directly. He understood what should be seen, and what was not being routed. He discovered in this early stage of IPv6 that some routes he knew had been withdrawn in BGP still existed when he looked into the repositories of known routing state. This is some of the first evidence of a failure mode in BGP where withdrawal of information fails to propagate, and some number of BGP speakers do not learn a route has been taken down. They hang on to it.Because BGP is a protocol which only sends differences to the current routing state as and when they emerge (if you start afresh you get a LOT of differences, because it has to send everything from ground state of nothing. But after that, you're only told when new things come and old things go away) it can go a long time without saying anything about a particular route: if its stable and up, nothing to say, and if it was withdrawn, you don't have it, to tell people it's gone, once you passed that on. So if somehow in the middle of this conversation a BGP speaker misses something is gone, as long as it doesn't have to tell anyone it exists, nobody is going to know it missed the news.In more recent times, there has been a concern this may be caused by a problem in how BGP sits inside TCP messages and this has even led to an RFC in the IETF process to define a new way to close things out.Geoff isn't convinced this diagnosis is actually correct or that the remediation proposed is the right one. From a recent NANOG presentation Geoff has been thinking about the problem, and what to do. He has a simpler approach which may work better.Read more about BGP zombies at the APNIC Blog and the web:BGP Zombies at NANOG 93 (Geoff Huston, APNIC Blog February 2025)NANOG 93 presentation on BGP Zombies (Iliana Xygkou from Thousand Eyes, NANOG presentation)RFC9687 SendHold Timers (IETF RFC)

In this episode, Job Snijders discusses RPKIViews, his long term project to collect the "views" of RPKI state every day, and maintain an archive of BGP route validation states. The project is named to reflect route views, the long-standing archive of BGP state maintained by the University of Oregon, which has been discussed on PING.Job is based in the Netherlands, and has worked in BGP routing for large international ISPs and content distribution networks as well as being a board member of the RIPE NCC. He is known for his work producing the Open-Source rpki-client RPKI Validator, implemented in C and distributed widely through the OpenBSD project.RPKI is the Resource PKI, Resource meaning the Internet Number Resources, the IPv4, IPv6 and Autonomous System (AS) numbers which are used to implement routing in the global internet. The PKI provides cryptographic proofs of delegation of these resources and allows the delegates to sign over their intentions originating specific prefixes in BGP, and the relationships between the AS which speak BGP to each other.Why rpkiviews? Job explains that there's a necessary conversation between people involved in the operational deployment of secure BGP, and the standards development and research community: How many of the worlds BGP routes are being protected? How many places are producing Route Origin Attestations (ROA) which are the primary cryptographic object used to perform Route Origin Validation (ROV) and how many objects are made? Whats the error rate in production, the rate of growth, a myriad of introspective "meta" questions need to be asked in deploying this kind of system at scale, and one of the best tools to use, is an archive of state, updated frequently, and as for route views collected from a diverse range of places worldwide, to understand the dynamics of the system.Job is using the archive to produce his annual "RPKI Year in review" report, which was published this year on the APNIC Blog (it's posted to operations, research and standards development mailing lists and presented at conferences and meetings normally) and products are being used by the BGPAlerter service developed by Massimo CandelaRead about the rpkiviews archive on the APNIC Blog, and on the web:RPKI's 2024 Year in review - (Job Snijders, APNIC Blog January 2025)RPKIViews - (the RPKI views Web archive)

In his first episode of PING for 2025, APNIC’s Chief Scientist, Geoff Huston returns to the Domain Name System (DNS) and explores the many faces of name servers behind domains. Up at the root, (the very top of the namespace, where all top-level domains like .gov or .au or .com are defined to exist) there is a well established principle of 13 root nameservers. Does this mean only 13 hosts worldwide service this space? Nothing could be farther from the truth! literally thousands of hosts act as one of those 13 root server labels, in a highly distributed worldwide mesh known as "anycast" which works through BGP routing.The thing is, exactly how the number of nameservers for any given domain is chosen, and how resolvers (the querying side of the DNS, the things which ask questions of authoritative nameservers) decide which one of those servers to use isn't as well defined as you might think. The packet sizes, the order of data in the packet, how it's encoded is all very well defined, but "which one should I use from now on, to answer this kind of question" is really not well defined at all.Geoff has been using the Labs measurement system to test behaviour here, and looking at basic numbers for the delegated domains at the root. The number of servers he sees, their diversity, the nature of their deployment technology in routing is quite variable. But even more interestingly, the diversity of "which one gets used" on the resolver side suggests some very old, out of date and over-simplistic methods are still being used almost everywhere, to decide what to do.Read more about Geoff's research on DNS nameserver selection and diversity on the APNIC Blog:DNS nameservers: Service performance and resilience (Geoff Huston, APNIC Blog February 2025)

Welcome back to PING, at the start of 2025. In this episode, Gautam Akiwate, (now with Apple, but at the time of recording with Stanford University) talks about the 2021 Advanced Network Research Prize winning paper, co-authored with Stefan Savage, Geoffrey Voelker and Kimberly Claffy which was titled "Risky BIZness: Risks Derived from Registrar Name Management".The paper explores a situation which emerged inside the supply chain behind DNS name delegation, in the use of an IETF protocol called Extensible Provisioning Protocol or EPP. EPP is implemented in XML over the SOAP mechanism, and is how registry-registrar communications take place, on behalf of a given domain name holder (the delegate) to record which DNS nameservers have the authority to publish the delegated zone. The problem doesn't lie in the DNS itself, but in the operational practices which emerged in some registrars, to remove dangling dependencies in the systems when domain names were de-registered. In effect they used an EPP feature to rename the dependency, so they could move on with selling the domain name to somebody else.The problem is that feature created valid names, which could themselves then be purchased. For some number of DNS consumers, those new valid nameservers would then be permitted to serve the domain, and enable attacks on the integrity of the DNS and the web.Gautam and his co-authors explored a very interesting quirk of the back end systems and in the process helped improve the security of the DNS and identified weaknesses in a long-standing "daily dump" process to provide audit and historical data.Read more about RISKY BIZness and the supply chain attack on the web:The 2021 ANRP paper "Risky BIZness: Risks Derived from Registrar Name Management"2017 Grand Jury indictment of Zhang et al2022 IMC paper "Retroactive Identification of Targeted DNS Infrastructure HijackingThe prevalence, persistence, and perils of lame delegations (APNIC blog, 2021)

In the last episode of PING for 2024, APNIC’s Chief Scientist Geoff Huston discusses the shift from existing public-private key cryptography using the RSA and ECC algorithms to the world of ‘Post Quantum Cryptography. These new algorithms are designed to withstand potential attacks from large-scale quantum computers and are capable of implementing Shor’s algorithm, a theoretical approach for using quantum computing to break the cryptographic keys of RSA and ECC.Standards agencies like NIST are pushing to develop algorithms that are both efficient on modern hardware and resistant to the potential threats posed by Shor’s Algorithm in future quantum computers. This urgency stems from the need to ensure ‘perfect forward secrecy’ for sensitive data — meaning that information encrypted today remains secure and undecipherable even decades into the future.To date, maintaining security has been achieved by increasing the recommended key length as computing power improved under Moore’s Law, with faster processors and greater parallelism. However, quantum computing operates differently and will be capable of breaking the encryption of current public-private key methods, regardless of the key length.Public-private keys are not used to encrypt entire messages or datasets. Instead, they encrypt a temporary ‘ephemeral’ key, which is then used by a symmetric algorithm to secure the data. Symmetric key algorithms (where the same key is used for encryption and decryption) are not vulnerable to Shor’s Algorithm. However, if the symmetric key is exchanged using RSA or ECC — common in protocols like TLS and QUIC when parties lack a pre-established way to share keys — quantum computing could render the protection ineffective. A quantum computer could intercept and decrypt the symmetric key, compromising the entire communication.Geoff raises concerns that while post-quantum cryptography is essential for managing risks in many online activities — especially for protecting highly sensitive or secret data—it might be misapplied to DNSSEC. In DNSSEC, public-private keys are not used to protect secrets but to ensure the accuracy of DNS data in real-time.If there’s no need to worry about someone decoding these keys 20 years from now, why invest significant effort in adapting DNSSEC for a post-quantum world? Instead, he questions whether simply using longer RSA or ECC keys and rotating key pairs more frequently might be a more practical approach.Read more about Post-Quantum Cryptography and DNSSEC on the APNIC blog and the web.Post-Quantum Cryptography (Geoff Huston, APNIC Blog November 2024)[Podcast] Testing Post-Quantum Cryptography DNSSEC (Podcast July 2024)A quantum-safe cryptography DNSSEC testbed (Caspar Schutijser, APNIC Blog 2024)[Podcast] The SIDN Labs post-quantum DNSSEC testbed (Podcast August 2024)Quantum Computing and the DNS (Paul Hoffman, office of the CTO, ICANN April 2024)PING will return in early 2025This is the last episode of PING for 2024, we hope you’ve enjoyed listening. The first episode of our new series is expected in late January 2025. In the meantime, catch up on all past episodes.

This time on PING, Peter Thomassen from SSE and DEsec.io discusses his analysis of the failure modes of CDS and CDNSKEY records between parent and child in the DNS. These records are used to provide in-band signalling of the DS record, fundamental to the maintenance of a secure path from the trust anchor to the delegation through all the intermediate parent and grandparent domains. Many people use out-of-band methods to update this DS information, but the CDS and the CDNSKEY records are designed to signal this critical information inside the DNS, avoiding many of the pitfalls of passing through a registry-registrar web service.The problem is, as Peter has discovered, the information across the various nameservers (denoted by the NS record in the DNS) of the child domain can get out of alignment, and the tests a parent zone need to do checking CDS and CDNSKEY information aren't sufficiently specified to wire down this risk.Peter performed a "meta analysis" inside a far larger cohort of DNS data captured by Florian Steurer and Tobias Fiebig at the Max Planck Institute and discovered a low but persisting error rate, a drift in the critical keying information between a zones NS and the parent. Some of these related to transitional states in the DNS (such as when you move registry or DNS provider) but by no means all, and this has motivated Peter and his co-authors to look at improved recommendations for managing CDS/CDNSKEY data, to minimise the risk of inconsistency, and the consequent loss of secure entry path to a domain name.Read more about DNSSEC delegation at the APNIC Blog, and the IETF:Authenticated bootstrapping of DNSSEC delegations (NIls Wisiol, APNIC Blog March 2022)Measurement of CDS/CDNSKEY inconsistencies (IETF119 Presentation, March 2024)Generalised DNS NOTIFY (IETF Draft)

In his regular monthly spot on PING, APNIC’s Chief Scientist Geoff Huston discusses the slowdown in worldwide IPv6 uptake. Although within the Asia-Pacific footprint we have some truly remarkable national statistics, such as India which is now over 80% IPv6 enabled by APNIC Labs measurements, And Vietnam which is not far behind on 70% the problem is that worldwide, adjusted for population and considering levels of internet penetration in the developed economies, the pace of uptake overall has not improved and has been essentially linear since 2016. In some economies like the US, a natural peak of around 50% capability was reached in 2017 and since then uptake has been essentially flat: There is no sign of closure to a global deployment in the US, and many other economies.Geoff takes a high level view of the logisitic supply curve with the early adopters, early and late majority, and laggards, and sees no clear signal that there is a visible endpoint, where a transition to IPv6 will be "done". Instead we're facing a continual dual-stack operation of both IPv4 (increasingly behind Carrier Grade Nats (CGN) deployed inside the ISP) and IPv6.There are success stories in mobile (such as seen in India) and in broadband with central management of the customer router. But, it seems that with the shift in the criticality of routing and numbering to a more name-based steering mechanism and the continued rise of content distribution networks, the pace of IPv6 uptake worldwide has not followed the pattern we had planned for.Read more about the IPv6 transition at the APNIC BlogThe IPv6 Transition (Geoff Huston, APNIC Blog November 2024)The Transition to IPv6 are we there yet (Geoff Huston, APNIC Blog May 2022)

In this episode of PING, Vanessa Fernandez and Kavya Bhat, two students from the National Institute of Technology Karnataka (NITK) discuss the student led, multi-year project to deploy IPv6 at their campus. Kavya & Vanessa have just graduated, and are moving into their next stages of work and study in computer sciences and network engineering.Across 2023 and 2024 they were able to attend IETF118 and IETF119 and present on their project and it’s experiences to the IPv6 working groups and off-Working Group meetings, in part funded by the APNIC ISIF Project and the APNIC Foundation.This multi-year project is supervised by the NITK Centre for Open-source Software and Hardware (COSH) and has outside review from Dhruv Dhody (ISOC) and Nalini Elkins (Inside Products inc). Former students have also acted as alumni and remain involved in the project as it progresses.We often focus on IPv6 deployment at scale in the telco sector, or experiences with small deployments in labs, but another side of the IPv6 experience is the large campus network, in scale equivalent to a significant factory or government department deployment but in this case undertaken by volunteer staff, with little or no prior experience of networking technology. Vanessa and Kavya talk about their time on the project, and what they got to present at IETF.Read more information on the NITK and their IPv6 deployment project on the APNIC Blog, the IETF website and the APNIC Foundation pages:Migrating the NITK Surathkal Campus Network to IPv6 (APNIC Foundation)How Deploying IPv6 at NITK Led me to IETF (Vanessa Fernandez, APNIC Blog)IPv6 Deployment at NITK (IETF118 Presentation)

In his regular monthly spot on PING, APNIC’s Chief Scientist, Geoff Huston, discusses a large pool of IPv4 addresses left in the IANA registry, from the classful allocation days back in the mid 1980s. This block, from 240.0.0.0 to 255.255.255.255 encompasses 268 million hosts, which is a significant chunk of address space: it's equivalent to 16 class-A blocks, each of 16 million hosts. Seems a shame to waste it, how about we get this back into use?Back in 2007 Geoff Paul and myself submitted An IETF Draft which would have removed these addresses from the "reserved" status in IANA and used to supplement the RFC1918 private use block. We felt at the time this was the best use of these addresses because of their apparent un-routability, in the global internet. Almost all IP network stacks at that time shared a lineage with the BSD network code developed at the University of California, and released in 1983 as BSD4.2. Subsequent versions of this codebase included a 2 or 3 line rule inside the Kernel which checked the top 4 bits of the 32 bit address field, and refused to forward packets which had these 4 bits set. This reflected the IANA status marking this range as reserved. The draft did not achieve consensus.A more recent proposal has emerged from Seth Schoen, David Täht and John Gilmore in 2021 which continues to be worked on, but rather than assigning to RFC1918 internal non-routable puts the address into global unicast use. The authors believe that the critical filter in devices has now been lifted, and no longer persists at large in the BSD and Linux derived codebases. This echoes use of the address space which has been noted inside the Datacentre.Geoff has been measuring reachability at large to this address space, using the APNIC Labs measurement system and a prefix in 240.0.0.0/4 temporarily assigned and routed in BGP. The results were not encouraging, and Geoff thinks routability of the range remains a very high burden.Read more about 240/4 in the APNIC Blog, and the IETF Datatracker website:Looking for 240/4 addresses (Geoff Huston, APNIC Blog September 2024)Re-delegation of 240/4 from "future use" to "private use" (expired IETF draft, 2008)Unicast use of the formerly reserved 240/4 (active IETF draft, 2024)

In this episode of PING, Nowmay Opalinski from the French Institute of Geopolitics at Paris 8 University discusses his work on resilience, or rather the lack of it, confronting the Internet in Pakistan.As discussed in his blog post, Nowmay and his colleagues at the French Institute of Geopolitics (IFG), University Paris 8, and LUMS University Pakistan used a combination of technical measurement from sources such as RIPE Atlas, in a methodology devised by the GEODE project, combined with interviews in Pakistan, to explore the reasons behind Pakistan’s comparative fragility in the face of seaborne fibre optical cable connectivity. The approach deliberately combines technical and social-science approaches to exploring the problem space, with quantitative data and qualitative interviews.Located at the head of the Arabian Sea, but with only two points of connectivity into the global Internet, Pakistan has suffered over 22 ‘cuts’ to the service in the last 20 years, However, as Nowmay explores in this episode, there actually are viable fibre connections to India close to Lahore, which are constrained by politics.Nowmay is completing a PhD at the institute, and is a member of the GEODE project. His paper on this study was presented at the 2024 AINTEC conference held in Sydney, as part of ACM SIGCOMM 2024.Read more about GEODE, and Nowmay’s work:The GEODE projectPakistan, a case study in Internet fragilityThe Quest for a Resilient Internet Access in a Constrained Geopolitical Environment (AINTEC 2024 Paper)