Carrier Routing Basics: Peering, Transit, and Public IXPs

Carrier Routing Basics: Peering, Transit, and Public IXPs

When two networks meet on the public Internet, they choose how to exchange traffic: by paying an upstream, by meeting as equals, or by gathering at a neutral switching fabric. Each model shapes cost, performance, and control. If you run or buy connectivity for an ISP, SaaS, CDN, or enterprise, knowing the differences keeps your routes fast and your bills predictable.

Think of transit as an all-roads pass, peering as a shortcut to specific neighborhoods, and public Internet exchange points as marketplaces where many neighbors trade traffic at once. The same Border Gateway Protocol (BGP) underpins all three, but incentives and policies change the paths packets take. Once you see the levers—attributes, communities, filters—you can tune for latency or savings without breaking reachability.

We’ll break down how each model works, what they cost, how BGP prefers one path over another, and the operational hygiene that prevents leaks, loops, and outages. Then we’ll sketch playbooks for common network types so you can mix transit, private peering, and IXPs with intent.

How Networks Interconnect

Transit is a commercial service where an upstream provider agrees to carry your traffic to the entire Internet and deliver Internet traffic back to you. You usually receive a default route, a full table, or both, plus SLAs, support, and billing based on a committed bandwidth level (with 95th percentile or flat rate models). The provider announces your prefixes to the world and forwards everything else toward you.

Peering is a voluntary agreement between two autonomous systems (ASes) to exchange traffic for their own and their customers’ routes, typically without payment (settlement-free) if the trade is balanced. Some peering is paid when traffic ratios, geography, or business leverage call for compensation. Peering is bilateral (a direct BGP session between two parties) or multilateral (via a route server at an exchange). It targets cost reduction and performance for “on-net” destinations.

Public IXPs are layer-2 switching fabrics where many networks colocate and interconnect. Instead of buying dozens of cross-connects, each participant buys one port on the shared Ethernet fabric and can set up many bilateral sessions or join route servers for one-to-many reachability. IXPs encourage local traffic to stay local, lowering latency and transit bills.

Peering: Bilateral and Multilateral

In bilateral peering, each side runs eBGP over a dedicated VLAN or cross-connect, sets import/export policies, and enforces filters. Policies range from open (peers with anyone meeting basic technical criteria) to selective (case-by-case) to restrictive (large thresholds for traffic volumes or presence). Traffic balance matters; extreme asymmetry can push parties toward paid peering or transit.

Multilateral peering uses one or more route servers (RS) operated by the IXP. You peer with the RS; the RS reflects routes from participants that opt in, simplifying management from N×(N−1) sessions to one or two sessions. You still apply inbound filters (IRR/RPKI-based) and set communities to influence export behavior. Many networks use both RS and a handful of key bilateral sessions for fine-grained control.

Transit: Upstream Connectivity

An upstream often offers multiple handoff options (10/25/100/400G), diversity across metro/regions, and features like blackhole communities for DDoS mitigation. You receive a default route plus the full table for best routing, or just a default if you prefer simplicity. Tier-1 providers have settlement-free reachability to the whole Internet via their peering mesh; Tier-2s buy some transit but may offer better pricing and regional performance. What matters to you is path quality, not labels.

Public Internet Exchange Points

IXPs supply the physical layer (switch fabric), a control plane (RS and looking glass), and a membership framework. Ports can be bundled via LAGs for capacity, and jumbo frames are common. Route servers honor common BGP communities to let you opt out of specific peers, prepend for certain ASNs, or set local-pref hints. Because IXPs concentrate traffic, watch for microbursts and buffer behavior on your NICs and switches, and size accordingly.

Routing Policy and Control

BGP picks a “best” path through a decision chain: highest local preference, shortest AS path, lowest origin type, lowest MED (if the same AS), eBGP over iBGP, lowest IGP metric to next hop, and tie-breakers. Operators exploit this to prefer peering over transit: set higher local-pref on peering routes than on transit. Use AS path prepending on your advertisements when you want to de-prefer specific exits on the wider Internet.

Communities act as policy signals. Inbound, you might honor a peer’s no-export community or blackhole community to drop traffic for an attacked /32 at your edge. Outbound, you can tag routes for your upstreams to avoid re-announcing to certain peers, change local-pref within the provider, or trigger selective prepends on their edge.

Hot-Potato and Cold-Potato Routing

Hot potato hands traffic to the next AS as soon as possible, minimizing your backbone cost but possibly increasing end-to-end latency if the handoff point is suboptimal. Cold potato keeps traffic on your backbone longer until closer to the destination, costing more but improving performance. CDNs and content networks lean cold potato where they run robust backbones; access ISPs often choose hot potato to conserve transit and backbone resources.

Traffic Engineering Tactics

Practical levers include splitting advertisements by region (scoped communities), deaggregating only where acceptable, tagging MEDs for multi-link peers, and steering return paths with more specific routes for critical services. Monitor actual flows; BGP is policy, but user experience depends on RTT, jitter, packet loss, and path stability. TE that ignores application metrics usually backfires.

Security and Route Hygiene

Baseline controls: RPKI origin validation to reject invalids, IRR-based prefix filters to limit what you accept and announce, and max-prefix limits on all sessions. Use BFD for fast failure detection where both sides agree, and MD5 or TCP-AO for session protection. Enforce martian and bogon filters on eBGP inputs.

Prevent spoofing with BCP-38/uRPF at edges so your network doesn’t originate forged source IPs. Watch for route leaks using well-known communities and monitoring (e.g., look for unexpected upstreams advertising your space). DDoS playbooks should include remotely triggered blackholing, scrubbing integrations, and communities advertised to upstreams for rapid filtering.

Operational Observability

Track BGP session state changes, route churn, flap dampening thresholds (if used), and per-prefix traffic. Export NetFlow/IPFIX to understand who benefits from which peer. Use looking glasses and route collectors to verify that the broader Internet sees your intended advertisements and prepends. Maintain an up-to-date peeringdb entry so potential peers can discover your locations, policies, and NOC contacts.

Cost, Contracts, and Scaling

Transit contracts revolve around commits, burst policy (95th percentile vs flat), and term discounts. Price per megabit falls with larger commits but beware over-committing and paying for unused capacity. Peering is “free” only after you include port fees, cross-connects, and colocation; it still often undercuts transit on a per-Mbps basis when traffic volumes are high to a set of partners. Paid peering sits between: dedicated capacity and predictable performance for a price below transit.

At IXPs, your cost drivers are port speed, cross-connects from your rack to the meet-me room or IXP switch, and any remote IXP transport if you extend to other metros. Route servers save time, but bilateral sessions with high-value peers can reduce path stretch and give finer policy control. Scale up using LAGs, diverse line cards, and separate peering and transit edge routers when table size and policy complexity grow.

Designing the Right Mix

Eyeball ISPs prioritize peering with big content and caches to lower last-mile congestion and transit spend, then add one or more transits for default and long tail. Content networks invest in IXPs and private interconnects near users, add cold-potato backbones, and keep a couple of upstreams for reachability insurance. SaaS and enterprises start with dual transit providers in diverse facilities, then add peering at a few strategic IXPs where user concentration justifies the port.

Ask three questions: Where are your users and heavy peers located, how much of your traffic is targetable with peering, and what reliability tier do your SLAs require? Pilot with one IXP port, measure offload and latency, then expand to other metros. Keep transit diverse (different carriers and physical paths) and test failover through planned maintenance—don’t wait for a bad day to discover asymmetric behavior or missing communities.

Peering Readiness Checklist

Publish accurate route objects and RPKI ROAs, document your import/export policies, and set max-prefix to sane values per session. Standardize templates for BGP neighbors, communities, and monitoring. Automate IRR/RPKI filter builds and redeploy on change. Keep an escalation-ready NOC with 24×7 contacts in your peeringdb profile. Finally, review port utilization and queueing monthly; upgrade before sustained peaks exceed 60–70% to preserve headroom for bursts.

Carrier Routing Basics: Peering, Transit, and Public IXPs (FAQ)

Transit is a paid “reach everywhere” service from an upstream, while peering is a direct exchange with specific networks to cut cost and improve performance for shared routes.

Check the exchange’s participant list and your router’s BGP neighbor table, then verify suspected partners with an ASN Lookup to confirm ownership and routing relationships.

Most networks use the route server for broad reachability, then add a few bilateral sessions with heavy-hitter peers to fine-tune policy, capacity, and troubleshooting visibility.

Enable RPKI/IRR filtering, set max-prefix limits, protect eBGP with authentication, and verify hostname ownership with a quick Reverse DNS Lookup when you audit traceroute hops.

When traffic volumes with a specific partner are large and predictable, paid peering can deliver dedicated capacity and stable latency at a lower price than equivalent transit.

Increase local-pref on received peering routes versus transit, and optionally prepend your AS toward transit neighbors to shift inbound paths in favor of peering where acceptable.

Confirm address plans, neighbor reachability, RPKI ROAs, and session templates, then run an IPv6 Test from a vantage host to validate connectivity and routing symmetry.

IXPs give you more steering options: you can blackhole locally with communities, sink traffic at an upstream scrubbing provider, or diversify handoff points to contain blast radius.