============================================================================= Note: This paper compiles and describes the routing principles used by the agency IP backbones in their relationships with other networks. It is also available via anonymous ftp to devvax.tn.cornell in the subdirectory [.pub]. The filename is [backbones] and it is availalbe in text or postscript version. This version was retrieved via anonymous ftp to devvax.tn.cornell. ============================================================================ IP Routing Between U.S. Government Agency Backbones and Other Networks Scott Brim Cornell University 1. Introduction This is an overview of how the agency backbones route IP (Internet Protocol) packets at this time, with any generalizations that can be made and statements of their differences. Also included are recommendations from the agency backbones about how other networks that connect to them can best set up their inter-administration routing. This paper has a sin- gle goal: to improve global engineering in the Internet. The backbone networks discussed here are NSFNET (NSF), ESNET (DoE), the NASA Science Net, also known as NSN (NASA), and ARPANET/MILNET (DoD). Also, al- though the agency backbones are moving toward supporting multiple protocol suites (ISO OSI, DECnet), only IP is considered here. This paper describes routing from the point of view of the US Government Agency back- bones. It does not try to say anything, for example, about how mid-level networks should interact with each other, or how backbone networks in other countries might connect to each other, except to the extent that these activities would affect the agency backbones. Also, I would like to caution the reader that the agency backbones and their relationships with other networks are always evolving, and that some of the information here may have been out of date very soon after it is written. Before making any critical decisions based on what is included here, check with the backbone network management organizations. This paper, done at the request of the FRICC, is almost entirely a collaboration between representatives of the organizations which manage these backbones. Major contributors were Hans-Werner Braun, Jeff Burgan, Mike Collins, Tony Hain, Milo Medin, Rebecca Nitzan, and Jessica Yu. Also we are grateful to Merit Inc., which manages NSFNET and which made the first attempt to define a routing architecture for their backbone in the mod- ern, non-hierarchical Internet; and to the members of the ESNET staff for asking us to re- view the ESNET IP routing model, which was our main inspiration to produce this paper. 2. Definitions Almost all of the terms used in this paper are well-understood in the networking communi- ty, but a few are open to interpretation and they are defined here in order to avoid confusion. These definitions should only be considered to apply to this paper. Administrative Domain (or simply, Domain or AD): In this paper administrative domain is used to mean an administrative entity which has policy, technical, and engineering control over a group of routers which use one or more IGPs to communicate within the group. An AD uses EGP(s) to communi- cate with routers outside of the group, and may in fact use different autonomous sys- tem numbers at different exterior boundaries. Connection: Two administrative domains are said to have a connection when they are actually passing EGP routing packets across a shared network (e.g., an ethernet); not just physically connected. Mid-level Network A mid-level network is not only a network in its own right, but also an administrative domain which offers transit services to other domains, its client organizations, in connecting them to other clients, other mid-level networks, and especially to back- bone networks. In this paper mid-level networks' attributes as domains are by far the most relevant. 3. General Principles 3.1 The Internet as a hierarchy For management purposes, there is a natural tendency to think of the Internet as a hierarchy, but at this point in time-with rapid proliferation and uncertainty about the best strategy for management-the Internet tends to grow as a mesh. As any portion of the Internet stabilizes it tends to become more clearly hierarchical, but there will always be areas where using a strictly hierarchical model for planning may, through dependence on false assumptions, lead to management problems. On the other hand the only well-established inter-domain routing protocol (EGP2) was de- signed for a strictly hierarchical environment, and therefore, in order to avoid loops and other routing instabilities, the backbones must treat the Internet as a hierarchy regardless of whether it is or not. They do this by carefully choosing which paths through the Internet they are willing to use, and by filtering what routing information they are willing to send to and receive from their neighbors. As a simple example of a problem to be avoided, consider a backbone sending routing information about a very remote network A into a mid-level network. The mid-level network is connected laterally to another mid-level network, which is connected back to the backbone. With no restrictions on the flow of routing information, if connectivity to A is lost, a route for it will still circulate through the three domains (the two mid-level networks and the backbone) and a routing loop may persist for a very long time-perhaps forever, depending on how the interfaces between the domains are engi- neered. Other typical conditions to be avoided are unintended sub-optimal routes, oscillat- ing routes, administratively unacceptable routes, and asymmetric routes (where traffic takes a different path depending on which direction it is heading between the two commu- nicating nodes). In the simplest case, the above example is solved by having the backbone filter routing in- formation received from the mid-level networks, and refuse to believe any announcement of a path to A, since any such announcement is either a suboptimal path or an echo of what the backbone passed to the mid-level networks. For further examples see RFC 1093. Re- commendations for further reading are at the end of this paper. In summary, connections which do not conform to the implicit hierarchy levels of the par- ticipating networks do exist, and in the current real world are sometimes necessary, but they should be avoided. If used, they must be used carefully, with the complete understanding and support of all topologically nearby networks which might be affected. Information about such connections should be propagated through the Internet only as far as is absolute- ly necessary. 3.2 Boundaries between domains In order to make control and tracking of the inter-administration flow of routing informa- tion possible at all, administrative boundaries must be clearly delimited. A well-defined ad- ministrative boundary requires that no IGP traffic pass between the two domains. Across these boundaries an exterior gateway protocol should be used; at this time the only one used widely is EGP2. Such boundaries should be weakened only with extensive coordination between the networks involved, with careful consideration of possible repercussions upon other topologically nearby nets. "Default" routes should never be passed across administrative boundaries. If a domain wishes to consider itself lower in a hierarchy than another, it may generate a "default" route itself, pointing in the direction of that other domain, based on some criteria such as the es- tablishment of EGP neighbor reachability, but generation of such a default should be under its exclusive control. Over the last few years it has been found that in a large, volatile Internet it is difficult if not impossible to strive for least-cost routing and still keep stability-route optimization has had to be sacrificed somewhat. Also EGP2 is only a reachability protocol, and there is little agreement on how to handle EGP2 "metrics", so least-cost routes cannot be built with con- fidence across EGP2 boundaries. Finally, least-cost routes may be administratively unac- ceptable, due to considerations, for example, of contractual agreements, or performance or reliability history. For these reasons, at administrative boundaries metrics representing "costs" should be considered only advisory. Except for ARPANET/MILNET, the US back- bone networks currently use pre-set metric values for the networks they hear about from other domains, ignoring whatever metrics might be sent. These metrics will have been pre- set according to administrative agreements between the networks sharing the connection, and while such agreements will be designed to promote Internet-wide connectivity, they may or may not take delay or throughput into consideration. 4. Interactions Between Backbones 4.1 Between two backbones Except for the ARPANET/MILNET mailbridges (the "mailbridges" control IP routing within ARPANET and MILNET), the backbones consider all EGP2 routing information they receive as reachability information only, and metrics are replaced with metrics pre- assigned in agreements between the backbone networks sharing the connection. The mail- bridges do take EGP2 metrics into consideration, and if they hear about a net through two different connections at two different metrics, they will use the path with the lowest metric. NSN, ESNET and NSFNET all filter nonsense which sometimes appears, such as routes to 127.0.0.1, 255.255.255.255, and so forth. Apart from that, ESNET and NSN have chosen to assume that each other, NSFNET, and the mailbridges have reliable routing information, and not to filter what they hears from them. While they do not filter, NSN and ESNET will accept a path from another backbone only if they do not already have a direct connection to it or a path through a mid-level networks to which they are directly connected. NSFNET filters all routing information from all of its neighbors. NSFNET routes internally by autonomous system number, not by network, and thus gen- erally recommends that a domain connected in more than one location use a different au- tonomous system number at each connection (see section 5.2 for more information). However, for backbone connections, NSFNET generates alternate autonomous system numbers internally for each connection, and thus the backbones can use the same autono- mous system number at their connections to NSFNET. The mailbridges announce via EGP2 all networks to which they have a path. NSFNET ac- cepts routes from the mailbridges only for those nets listed as having "connected" status by the DDN NIC. NSFNET is currently negotiating to have administrative agreements with ARPANET/MILNET just as they have with all other domains they connect to, whereby NSFNET will be given a list of networks for which they should be willing to accept reach- ability information from the mailbridges. NSN and ESNET do not use routing information from the mailbridges directly at this time (although they might in the future). Instead, when their mailbridge connections are func- tioning, they use a path to the mailbridges as a default route if they don't have a specific route for a destination; actual routes heard from the mailbridges are not redistributed inter- nally. To the mailbridges, NSFNET, ESNET and NSN announce the sites for which they have pri- mary responsibility, with the further limitation that from among the networks for which it is responsible, NSFNET only announces those for which announcement has been request- ed. The mailbridges take EGP2 metrics into consideration, and the other backbones use pre- set metrics when communicating with them. In order to express the preference of routing to a particular network through a particular mailbridge connection, NSFNET uses different metrics to designate one mailbridge as the primary path for that network, another second- ary, etc. (See [Yu et al., 1989] for further information about routing exchanges between NS- FNET and the mailbridges.) Where domains are connected and exchanging routing information at more than one point, it is potentially possible to heal a partition in one of them by allowing traffic to cross from one part of the partitioned domain, through the second domain, and back into another part of the partitioned one. Even though the backbones are multiply-connected to each other, the backbone managers have decided not to use this capability-if one of the backbones parti- tions, it will not be healed by another, although they will depend on the other backbones to make their client networks reachable by more remote parts of the Internet. Therefore the backbones will not be taking routing information heard from one connection to a backbone and passing it back into that backbone at another connection. 4.2 Between multiple backbones A major goal is to have all backbones reliably and multiply connected to all other back- bones, so that no backbone has to depend on any other domain to reach another backbone. For this purpose the agency backbones have established locations where all of the back- bones have nodes on a small, dedicated shared network. There are currently two of these sites, at NASA Ames Research Center (Mountain View, CA) and at the University of Maryland. These sites have uninterruptible power, 24 hour support coverage, and reason- ably easy access for maintenance purposes; ideally they should be at government institu- tions. The agency networks are looking for a third site in the center of the country. In theory the backbones are engineered such that every client network has one backbone which is primarily responsible for it, and if that backbone has connectivity to that network and is announcing it to the other backbones, no other backbone will announce it (to the other backbones). However, if, for example, ESNET were to have primary responsibility for a network and NSN were to hear about it from both ESNET and NSFNET, NSN would use the path through ESNET. NSN's and ESNET's order of preference for routing infor- mation is o direct connection to a network o direct connection to a mid-level network o via the other specialized backbone o via NSFNET o via ARPANET/MILNET (using a "default" route) 5. Backbones and Mid-level Networks 5.1 A single connection to one backbone Backbones and mid-level networks communicate via an exterior gateway protocol, current- ly EGP2. Metrics in EGP2 updates are ignored, and replaced by administratively preset val- ues, by all backbones except ARPANET/MILNET. A single connection may in fact involve multiple routers connected to one backbone router. Where multiple EGP2 peers connect to a single ESNET, NSN or NSFNET node with the same autonomous system number, the backbone node will compare the metrics it receives for a given destination network, but only in order to select which connection to use for traf- fic to a particular net-the metric used internally to the backbone will still be decided by administrative agreement. Except for ARPANET/MILNET, the backbones accept routes only for nets which have been listed in a prior agreement with the other participant in a connection point, and they apply metrics which are listed in those agreements. These agreements are similar across all the backbones and the mid-level network administrators are strongly encouraged to handle them similarly, and with care. For domains connected to only one backbone network, generation of a "default" route is useful, since through careful use of a default such a "stub" domain can cut down greatly on the amount of routing information it has to carry internally. 5.2 More than one connection to a backbone When a backbone network is connected to a mid-level network at more than one location, through more than one backbone router, the backbone will be able to exchange routing in- formation at all of those connections, but how multiple connections will be used must be negotiated with each backbone. For example ESNET may have several sites on its back- bone which are also connected to a mid-level network, but ESNET's initial position will be to pass ESNET backbone site reachability information to the mid-level network through only one of those potential connections. It is willing to use more than one connection if it feels that this can be done safely, but this must be negotiated. All of the backbones except ESNET support having a mid-level network use multiple con- nections as backups for each other (such that the mid-level network's client networks are announced as reachable through two or more connections). ESNET is not ready to support multiple connections at this time. The mailbridges pay attention to metrics used and, for a given network, the advertisement with the lowest metric will take precedence. As men- tioned, the other backbones ignore the metrics in the actual EGP2 announcements. The backbones generally do not try to keep track of the internal structure of other domains. Unless agreements have been made where, for example, different autonomous system num- bers and different metrics are used at different connection points, they simply assume that domains are monolithic, and to reach a network served by that domain they will route as dictated by administrative agreements, probably to the nearest connection point to that do- main. NSFNET currently routes internally by autonomous system number, and thus recommends that a mid-level network use different autonomous system numbers at each connection point. If the mid-level network does so, not only will it have more control over which client networks are reached through which NSFNET connections, and be able to use the NSFNET connections to back each other up, but also NSFNET will be able to heal the mid-level net- work when it is partitioned between the two connections, so that packets can flow from one part of the (partitioned) mid-level network, through NSFNET, and into the other part. How- ever, if the mid-level network wishes to use this feature, its IGP(s) must be able to discri- minate between externally- and internally-acquired routes, and to have externally-acquired routes be overridden by other externally-acquired routes which are better. Also, mid-level networks should not depend on this feature always being available-NSFNET has been considering ways to support flexibility without the need for a different autonomous system number at each connection point, and if some of them are adopted it may not be possible to offer partition healing anymore. A mid-level network which wants NSFNET potentially to heal its partitions should talk to NSFNET network management about engineering details. NSN will support multiple connections to a mid-level network, but it requires that the same autonomous system number be used at each connection, and will not heal partitions in the mid-level network. If a mid-level network is partitioned, NSN will make sure that the mid- level network's client networks continue to be reachable, but it will not support routing be- tween the two partitions the way that NSFNET will, since NSN will not re-advertise into a mid-level network any routing information they heard from it, even at a different connec- tion point. ESNET will support multiple connections in the same manner as NSN once they are ready to support more than one active connection to a mid-level network. Currently none of the backbones support comparing EGP2 metrics between connection points to a single mid-level network, even if the same autonomous system number is used at both (although NSFNET will compare metrics if two EGP2 peers connect to the same backbone node, as described above). NSN is considering doing so in the future 5.3 Connections to more than one backbone NSFNET is the only backbone which is willing to carry all traffic in support of research and engineering. NASA and DoE have sites which are of particular importance to them, and they do not intend to carry traffic for others except when others are communicating with those sites. Thus these backbones will only be passing information to mid-level networks about paths to the sites for which they are responsible. NSFNET will tell the mid-level net- works about all routes except those the mid-level network will be hearing about from other backbones. The mailbridges accept all routes. ESNET, NSN, and NSFNET will never pass on routing information from the mailbridges to a mid-level network until there is better route filtering capability in the mailbridges, in order to avoid the possibility of loops in the case where the mid-level network is also connected to the mailbridges. Even so, mid-level networks should be careful not to pass routing information they hear from any backbone back to the mailbridges. When a backbone is not connected to a mid-level network, another backbone which is con- nected will carry routing information from the non-connected backbone to the mid-level network. However, when two or more backbones are connected to the same mid-level net- work, there will be no overlap in which backbone advertises which remote networks at any time. When the more specialized backbones are connected to a mid-level network, they will be the only ones to announce networks of sites for which they have primary responsibility to that mid-level network, even if those sites are also connected (directly or indirectly) to other backbones. They may allow another backbone to announce their sites into the mid- level network if their connection to it breaks. To start with this will be done through static configurations; in the future the backbones hope it will be done dynamically by having the routers of the backbones communicate with each other. There are a number of sites which are contractors to NASA or DoE where the entire organization (campus, etc.) is restricted from using the agency backbone by backbone policy, or does not wish to do so for some reason. In these cases the groups with agency contracts will be asked to establish a separate network which can be attached to the agency backbone, and routed separately, even if con- nected to the rest of the organization. When a backbone is not connected to a mid-level network, it will depend on the other back- bones for paths to that mid-level network. However, when a backbone is connected to a mid-level network, then it will ignore routes to client networks in that mid-level network heard from other backbones. If a backbone connection to a mid-level network is lost, then it will be willing to use a path through the other backbones, but at this time ESNET and NSN would prefer the switchover to depend on manual intervention, at least until the real- world behavior of such a situation is better understood. They are concerned about how long transient false routing information will persist if routes are changed at the backbone level before things are allowed to stabilize at the middle level. Except for NSFNET, backbones are consistent in what they pass to each other, in that what- ever they announce to one backbone they announce to all of them. NSFNET, however, will not tell NSN about routing information it heard from a mid-level network if NSN already has a connection to that mid-level network. There may be special cases where this rule will not be followed, for example if a connection between NSN and a mid-level network is ten- uous. This rule may be relaxed in the future. 5.4 Connections through other mid-level networks Sometimes a mid-level network is connected to one or more backbones, and also has an agreement with a neighboring mid-level network, to which it is also connected and which also has connections to one or more backbones, for mutual backup support. If routing information about the client networks of both mid-level networks is passed to the backbones at both of the mid-levels' connections with them, then essentially the mid-level networks have extended their client base so that they each overlap the other. Conceptually the backbones are still dealing with two administrative domains, but the domains are now much larger. The backbones will treat the situation in this way, and will not concern them- selves with how the client networks happen to be members of both domains. All of the fea- tures and concerns that were mentioned above still hold (such as the dangers of using a simple IGP), but for the extended domains. As usual the backbones will refuse to believe routing information about which there is no administrative agreement, and (unless it is agreed otherwise) will continue to tell each mid-level network about the client networks reachable through the other mid-level network's connection(s). 6. Backbones and Sites Government institutions will in almost all cases have a connection to the backbone man- aged by their agency. Such government institutions are obligated to use their agency's net- work as their primary access path except to reach other clients of any mid-level network to which they might be connected. Any exceptions desired by the site must be agreed to by the management of that agency's backbone network. Sites which are not associated with a particular agency are considered NSF's clients and their primary access path will be via NSFNET. If a site is not connected directly to NSFNET or through a mid-level network which is connected to NSFNET, then another agency backbone should agree to take re- sponsibility for it. Site connections can be complicated in that a backbone might need to exchange IGP infor- mation with one of its sites in order to override routing information that a mid-level net- work might be injecting beyond the control of the backbone network or even the site. For example, many mid-level networks announce a "default" route with an IGP which may need to be overridden. Also, ESNET prefers that a DoE institution use ESNET to reach an- other DoE backbone site even if the institution and the other site share a mid-level network. On the other hand this may cause asymmetric routes if the other DoE backbone site is not a DoE institution (e.g. UCLA), and prefers routing to the first site across the mid-level net- work. Asymmetric routes are by no means a design goal, but they can occur. A backbone site on one of the more specialized backbones needs to work this sort of detail out with the backbone management, on a case-by-case basis. If a site has connections to multiple external networks, such as to both a mid-level network and a backbone, it is highly recommended that the site have all such external connections on a small, dedicated, shared network, independent of and gatewayed to the site's internal network, and that it use an EGP to talk to all external networks. In this way the site will be able to support traffic between external networks without having the reliability of that traf- fic being influenced by its own infrastructure. In addition, this will keep routing updates and other maintenance traffic passing between the backbone and the mid-level network(s) off the site's local networks. 7. Special considerations for non-US domains If a non-US domain connects to a US backbone, then that backbone has primary responsi- bility for it and will carry its traffic in addition to that of its own domestic sites. While the following guidelines are not followed for all existing international links, it is highly recommended that all new links do so, and that all old links be re-engineered to do so. The non-US domain should have only one primary path into the US, and for now (until we have more robust routing protocols and more experience with international connections) switching to a backup path should depend on manual intervention. Especially where more than one country is involved and joint troubleshooting may be difficult, we need to impose a hierarchical structure for the time being. This restriction may be relaxed where the back- bones are convinced there will not be problems. International connections should not be made through secondary or tertiary domains such as mid-level or university networks. Such connections should either be directly into a back- bone router (thus making the international link an integral part of the backbone) or to a small shared network dedicated to backbone connections. In order for the United States agency backbones to be able to abide by decisions of their government, they must have peer relationships with networks of other countries-the two networks must connect to each other in a direct enough fashion that the relationship between them can be managed. As mentioned above, cooperation between networks in two countries can be much more diffi- cult than within a country, and when lower-level networks are involved routing may be ex- tremely hard to keep under control. A mid-level network, for example, might pass on more routing information than its national backbone would like it to, and unintentionally pull traffic away from other international links. Security, of course, is also more difficult to guarantee in these situations. Except in cases where a network of another country is closely affiliated with one of the US agency backbones, and considers itself subsidiary to that backbone, all client networks from other countries must be announced to the US backbone to which they connect using EGP2 and a unique autonomous system number, which will be used to control routing with that country. If necessary, a separate router must be established at the connection to the US backbone in order to make this possible. If the international link does not terminate at a site where all the US backbones come together, as described in section 4.2, the US backbone to which the connection is made will arrange to have the non-US networks announced to the other US backbones at those sites using a separate autonomous system number, distinct from any used by the US backbone. 8. Further Reading Braun, H.W. NSFNET Routing Architecture. RFC 1095, April 1989. Braun, H.W. Models of Policy Based Routing. RFC1104, June 1989. Collins, M., and Nitzan, R. ESNET IP Routing. Available from ESNET. Mills, D.L. Exterior Gateway Protocol Formal Specification. RFC 904, April 1984. Yu, J., and Braun, H.W. Routing Between NSFNET and the DDN. RFC 1133, November 1989.