In the past decade, the construction of Internet-scale distributed applications has been a battle between the immovable object of the Internet architecture and the irresistible force of the need for better application performance and control.
There are many science experiments that generate huge amounts of data, ranging from terabytes to petabytes and beyond. Some of these science applications require their data to be processed in complex workflows that involve different functions (data generation, analysis, and storage) and span multiple collaborating sites. While progress has been made in addressing these needs – particularly the Data Transfer Nodes (DTNs) developed by ESnet – more work is needed to provide the detailed management and specific data-handling functionality these complex workflows require.
The Internet has created a world of universal connectivity, where any two devices can communicate as long as they are both connected to the Internet. The Internet architecture is a miraculous feat of engineering, remaining largely unchanged while scaling from an early prototype to the centerpiece of the global communications infrastructure. Having grown to such an unprecedented scale, the Internet is now the victim of its own success in that the Internet’s core protocol, IP, is now embedded in every router and thus is essentially impossible to change in any fundamental way. Yet to achieve goals like better performance and greater security, it is clear that the Internet must eventually change. The central technical question facing the Internet is thus: can we fundamentally change the Internet without changing IP?
In this project, researchers at ICSI argue that the answer to this question is most definitely “yes”. Leveraging insights from the large private networks recently built by cloud and content providers and a long line of academic research, they describe an approach called the Extensible Internet (EI).