Enabling Competition & Innovation on a City Fiber Network case study The technology of network virtualization has enabled the city of Ammon, Idaho, to inexpensively open its fiber network to competing service providers. In Ammon, residential and business customers may choose from different providers and even receive multiple services simultaneously, create private networks within Ammon’s city network, and obtain city services and emergency alerts over the network even if they don’t have an Internet access subscription. The project, though in an early stage, represents a versatile technological and operational model for other public fiber networks. RCOESMPMOUNNSITIVIEES ddiiggiittaall jjuussttiiccee << >> ddaattaa sstteewwaarrddsshhiipp Paddy Leerssen David Talbot October 2017 Workers in Ammon, Idaho, install fiber in a trench (below right). The city-built fiber network serves business (top right) and is now being extended to residents who can choose providers using an online interface. (Photos courtesy of the city of Ammon.) ABSTRACT This report describes how the municipally owned fiber-optic network in Ammon, Idaho, uses a technology known as network virtualization to inexpensively allow retail service providers to compete for users and provide innovative services over a public network without any requirement for new hardware at the customer’s home or business. Among other novelties, Ammon allows users to instantly switch between services, receive more than one service at a time, and inexpensively create private sub-networks. Under network virtualization, functions previously performed by specialized hardware devices are instead performed by software. In the United States, such technology is most often used by private telecommunications companies in ways that reduce internal costs but leave those companies controlling all services over their networks. Ammon’s technology strategy—along with other aspects of Ammon’s financing and operational model—provides one model for other U.S. public entities and for policymakers seeking to increase service competition and innovation. Other models include building public "dark" (or unused) fiber for use by private entities, and using virtualization in a more limited way. Suggested Citation: Leerssen, Paddy, Talbot, David. Enabling Competition & Innovation on a City Fiber Network. (October 2017). Responsive Communities. Available at: cyber.harvard.edu/publications/2017/09/ fibercompetition KEY FINDINGS • Ammon’s network initially served government and business users. Construction of a residential network—paid for by a property assessment equal to $17 monthly for 20 years—began in September of 2016. As of August 2017 it had 145 residential customers, with more than 270 homes expected to be connected by November 2017 in the first connected neighborhood. • The city charges users a $16.50 monthly utility fee for a fast data connection to the city network. Users then choose from Internet service providers (ISPs) via an online dashboard for access to the wider Internet or specialized services. To make this possible, the city uses network virtualization software from Entrypoint Networks, a Salt Lake City company. • So far two ISPs, Direct Communications and Fybercom, offer retail residential service; their prices start at $10 per month for 100Mbps upload and download service to $50 per month for 1 Gbps upload and download with no contract. Unlike services from most ISPs, these speeds are guaranteed, not “best effort.” • Though in a very early stage, Ammon’s network provides novel capabilities: customers can take services from more than one provider at a time; businesses can form their own internal networks within Ammon’s network without using an ISP; and public safety communications can be given priority over all other network uses in emergencies. Enabling Competition and Innovation on a City Fiber Network | Responsive Communities Initiative 3 Introduction THE PROMISE OF “VIRTUALIZATION” these technologies, “[a] revolution is sweeping through the telecommunications world, leading to possibly one of the biggest upheavals in the hundred plus year old industry.”8 Internet access networks have traditionally depended on a variety of specialized hardware devices, ranging from the major routers and switches that dispatch traffic between network providers to the humble consumer routers in most private homes.1 “Network virtualization” describes methods of replacing some of this specialized hardware with software. This involves techniques called software-defined networking (SDN) and networking function virtualization (NFV) that can reduce the need for various devices to which fiber-optic cable is connected.2 Academic researchers have long proposed using these strategies as a way to impose new and improved architectures on the Internet as a whole.3 Commercially, network virtualization first found an application in the data center industry as a means to provide flexible and efficient cloud computing services.4 More recently, telecommunications companies have started using these technologies in order to cut costs internally. In 2015, AT&T announced that it plans to have 75 percent of its network virtualized by 2020.5 As of January 2017, AT&T claims to be ahead of schedule, with 34 percent of the network virtualized.6 CenturyLink and Verizon have announced similar plans.7 As a 2015 Deloitte report put it, thanks to Cost savings can be found in two areas. First, virtualized networks can be programmed from a distance without physical adjustments to individual pieces of hardware, significantly improving service response time and reducing operational costs.9 Second, virtualization can reduce capital costs because generic computing hardware can be used instead of proprietary equipment.10 John Stephens, senior executive president and CFO at AT&T, recently said, “Our virtualization and software-defined networks are already delivering material CAPEX [capital expenditure] savings. We will be adding 2.5 times more capacity at 75 percent of the capital cost compared to just a few years ago.’11 1See, generally: Deloitte Consulting, “Operationalizing SDN and NFV Networks”, May 2015 . See also Dialogic Consulting, “Exploratory study on Network Virtualisation”, July 26, 2016, < https://www.rijksoverheid.nl/documenten/rapporten/2016/07/26/dialogic-exploratory-study-on-network-virtualisation>. 2 SDN and NFV are conceptually distinct, yet highly synergistic. In practice they are usually implemented and discussed jointly as ‘SDN/NFV’, or simply as ‘network virtualization’. 3 Thomas Andersen et al., “Overcoming the Internet Impasse Through Virtualization”, April 2015, . 4 Ben Pfaff et al., “Extending Networking into the Virtualization Layer”, Hotnets 2009, . 5 Ray Sheffer, “How Network Virtualization Is Impacting AT&T’s Capital Expenditures”, Market Realist, May 17, 2016, . 6 Sean Buckley, “AT&T finds virtual, mobile business services growth, but poor economy and legacy losses pose challenges”, Fierce Telecom, January 24, 2017, . 7 CenturyLink, “Press Release: CenturyLink announces virtualization plans as it continues integrating its network into the cloud”, October 19, 2015. . David Chernicoff, “CenturyLink plans full network virtualization by 2018”, October 20, 2015, . 8 Deloitte Consulting 2015. 9 Ibid.; Thomas Gryta, “AT&T Targets Flexibility, Cost Savings With New Network Design”, Wall Street Journal February 24, 2014, https://www.wsj. com/news/articles/SB10001424052702303426304579402953146294792; 10 Sheffer 2016. 11 Ibid. Enabling Competition and Innovation on a City Fiber Network | Responsive Communities Initiative 4 VIRTUALIZATION & RETAIL COMPETITION Virtualization can also enable retail competition. In conventional networks, it is cumbersome and time-consuming for users to switch between ISPs, requiring connections to be rewired. (In the industry these costly interventions are often referred to as “truck rolls,” because they require employees to physically visit the premises.) These practical barriers, along with requirements that users sign new 12-month or longer contracts, discourage users from changing providers. With network virtualization, multiple providers can potentially offer their services via the same hardware at low cost, allowing users to switch instantly. And users can potentially not only choose from different providers but also take specialized services from more than one provider at a time. Unsurprisingly, private ISPs that own their physical networks—including all of the major U.S. providers—generally show no interest in this concept, as it would involve opening up their networks to competitors. Municipalities, however, are now exploring this feature of network virtualization as a means to improve user choice and retail competition. This can be achieved through something called an open-access network, in which an infrastructure provider (typically a municipality or other public agency, sometimes in partnership with a private company) builds a fiber network and then opens the network as a platform for ISPs to provide services. The network owner does not compete with these ISPs. Rather, it leases access to its fiber network to private retail providers. There are two basic ways the infrastructure provider (here, the city) can do this. The city can lease strands of so-called dark fiber, and require ISPs to install their own hardware—that is, to “light up” the fiber—before they can offer their services. Alternatively, the city can choose to in- stall its own network hardware and create its own “lit” fiber network. In conventional lit networks, such as in Chattanooga Tennessee, the city itself acts as an ISP selling Internet access connections and other telecommunications services to its residents. Ammon is running a lit network, but with a twist: the city merely connects users to a local network (with a speed of 1 Gbps), and connected users can choose between multiple ISPs from which to buy access to the wider Internet or other services that use Ammon’s fiber. For example, a service provider might offer a single service, such as remote data storage. Thanks to virtualization technology, these ISPs do not need to install any hardware at their customers’ premises; they can each offer their services via “virtual networks” running on the city’s hardware.12 The practice is growing in popularity in Europe. For example, many towns in Sweden—which have a long history of providing diverse services over publicly owned fiber infrastructure13 —are starting to use the technology. At the European Union level, regulators are exploring virtualization as a means for incumbents to share their infrastructure with competing providers.14 Several companies provide virtualization software and services for municipal networks. These include EntryPoint, based in Salt Lake City, Utah; Netadmin Systems, based in Sweden and owned by a unit of Constellation Software in Toronto; and Cos Systems, based in New York and Sweden. But despite activity around the world in providing retail competition using virtualization—and the growing numbers of companies providing solutions—the practice is still rare in the United States. Ammon is one of very few U.S. municipalities using virtualization to foster retail competition. What makes it particularly novel is its “virtualized open access” model, in which the city owns and operates the entire network, provides lit service 12 Garakheili and Sivaraman, “Virtualizing National Broadband Access Infrastructure”, CoNEXT December 2013, . 13 Marco Forzati et al., “Open access networks, the Swedish experience”, August 2010, . 14 Body of European Regulators of Electronic Communications (BEREC), “Annual Report 2015”, April 26, 2016, . Enabling Competition and Innovation on a City Fiber Network | Responsive Communities Initiative 5 to its residents as a public utility, and facilitates competition by presenting users with an online dashboard from which users can choose ISPs.15 Ammon essentially made a philosophical decision not to be an ISP; instead, it would leave the private market to provide whatever services customers might want, acording to Bruce Patterson, Ammon’s technology director. HOW AMMON GOT STARTED Ammon, which has a population of about 15,000, began planning a municipal fiber network in 2008 and entered into a partnership with EntryPoint. Initially, it built fiber for the city government’s internal purposes, after concluding it could save money compared with obtaining equivalent services from local commercial providers. The network cost $1 million to build, and saves Ammon $70,000 yearly, according to Patterson. That network was subsequently expanded to connect various private wireless ISPs and more than 30 local businesses. Then, in 2016, the city—which was already served by cable and DSL providers—started connecting its first residential neighborhood with fiber-to-the-home (FTTH) service. The city decided to fund this residential rollout through a “local improvement district” (LID) model, in which residents of selected neighborhoods are asked to opt in or out of the new network.16 Those who participate receive a fiber connection to the home at a one-time charge of $3,000, which can be either be paid up-front or amortized over 20 years as a special assessment tax on the property, resulting in a $17 monthly cost to the homeowner.17 nections completed by November 2017. It has also identified a second LID and intends to begin construction after the buildout in the first LID is finished. A recent report by Strategic Networks Group (SNG), a consultancy that performs econometric analyses of public technology investments, pointed to the low financial and political risk associated with expansion of the city’s network through the LID model, given that the network is financed by property owners receiving the service. While risk is low, long-term benefits could be significant, according to SNG, which estimated that over 25 years (a term that is less than the expected useful life of the fiber), the city and its schools, businesses, and residential subscribers could save $43.6 million, far surpassing the city’s and customers’ total network costs of $8.6 million. (This figure represents the city's $1 million investment plus an estimated $7.6 million that would be charged to customers under the LID model if between 70 percent and 75 percent of Ammon's potential customers took service.) The network could also aid local business productivity and help increase the tax base, according to SNG’s analysis.18 THE NETWORK AS VIRTUAL MARKETPLACE A key feature of the Ammon model is its creation of a virtual marketplace for competing service providers. The city itself—rather than a private ISP—operates the network, funded by a monthly utility fee of $16.50 charged directly to the user. (This $16.50 utility fee is in addition to the $17 monthly property tax assessment.) After a successful pilot program in the summer of 2016, the city started making home connections in its first LID later in the year. By the time this report was published, 270 out of 360 residences in the LID had signed up—a take rate of 75 percent. The city has currently reached 145 of these homes, and it plans to have all 270 con- In exchange, the city provides a 1 Gbps connection to the home, which includes access to municipal services and to other users on the Ammon network. It also provides a dashboard from which users can choose between available third-party Internet access services at the click of a mouse. In essence, the physical infrastructure and so-called lit services are run as a public 15 The city refers to its model as Open Access Virtual Infrastructure, or OAVI. 16 See, generally: IC 50-1703(a)(10) (“The LID Act”). 17 Kevin Trevellyan, “First Ammon fiber district goes live”, The Post Register January 31, 2017, . 18 Strategic Network Groups, “Broadband Benefits Assessment of the Ammon Fiber Network”, May 3, 2017, . Enabling Competition and Innovation on a City Fiber Network | Responsive Communities Initiative 6 utility in order to provide a virtual marketplace for services provided by third parties. Patterson compares it to an app store: “The market is not created by installing competing infrastructure. The market is created by a single infrastructure capable of supporting any number of virtual infrastructures.” Currently, the dashboard lists 15 different deals from two ISPs, Direct Communications and Fybercom. Their prices range from $10 per month for 100 Mbps to $50 to $105 per month (depending on the provider selected) for 1 Gbps. Even when the municipality’s $16.50 utility fee and the $17 property assessment are added, the maximum price of $105 is well below the $180 charged for 1 Gbps service by the local cable provider, CableOne.19 In contrast to their cable competitors, who offer “best effort” connections, bandwidth from Ammon’s providers is not shared with other users in the neighborhood, so subscribers are guaranteed full use of their bandwidth at all times. And most deals are offered on a pay-as-you-go basis, without cancellation fees or penalties. (Ammon even offers a “Test Drive” feature; a brief trial period during which users can try the advertised broadband service for free.) The virtual network software needed to create this digital marketplace is maintained and operated by EntryPoint, which receives $1.50 out of the $16.50 utility fee users pay to Ammon each month. This model aligns EntryPoint’s incentives with the city’s. From Ammon’s perspective, the partnership was necessary to attract expertise and to simplify management. Ammon’s virtual open-access model is quite new. Other municipal open-access networks, such as those in Huntsville, Alabama, and Westminster, Maryland, follow a different and simpler model in which the city limits its role to constructing, owning, and maintaining dark fiber while a private partner does the rest, installing and maintaining the network's electronics and providing the service.20 In Ammon, the costs for an ISP to enter the market are low. Ammon charges ISPs a flat rate of $49.50 per month for their use of the municipal network, regardless of how many users they attract. The city can afford to do this because its costs are low; users pay for the physical fiber connection and pay $16.50 monthly for access to the city’s network. Patterson says the city tried to ensure that the barriers to entry were as low as possible to encourage competition. Indeed, in Ammon, ISPs do not maintain any part of the network; that's the city's responsibility. Already, Ammon is seeing lively competition between the current providers. Shortly after the first residents were connected, monthly prices for a 1 Gbps connection from both providers dropped by $10, and both providers have also updated their offerings to include symmetrical upload speeds across the board. On July 1, 2017, one provider cut all pricing by 50 percent and the other is expected to make adjustments to its pricing in order to compete. And Patterson reports that Ammon has received inquiries from national carriers interested in providing service and is currently working with two additional providers who may begin offering users specialized phone and TV service. In theory, others could emerge, such as cloud-based storage services or home security or health monitoring services. But it remains to be seen how many providers will ultimately attempt to provide service. 19 Note: these connections are ‘symmetrical’, which means that the upload speeds are equal to download speeds. This can have useful applications for online storage, video streaming, and home businesses. 20 Westminster entered a public-private partnership with Ting, a local ISP. Currently, Ting is the sole network and service provider on the Westminster network, but has agreed to a structural separation of these operations on the longer term. At that point the main difference would be that the private partner, Ting, rather than the city, will own and maintain the network hardware. More experience with this method can be found in European municipal fiber projects, particularly in Sweden, which separated network and service providers before the development of network virtualization technology. See Forzati et al. 2010. Enabling Competition and Innovation on a City Fiber Network | Responsive Communities Initiative 7 Figure 1: Screenshot of the Ammon dashboard. (“No contract” indicates a pay-as-you-go deal, allowing users to switch providers whenever they want without fees or penalties.) VIRTUALIZATON ALLOWS USERS TO CREATE CUSTOM NETWORKS SERVING MULTIPLE LOCATIONS Ammon’s virtual network can reduce costs for innovators and their experiments. If other cities or towns follow the same model, virtual network users in Ammon could potentially connect directly with users in those other networks.22 A novel feature of the Ammon network is that it allows users to establish private connections with one another within the Ammon network without needing to enter into a service contract with any ISP. For example, a business can set up its own company network, connecting all local offices, stay-at-home workers, and remote backup sites. This creates a virtual network within Ammon’s network. Indeed, Ammon is encouraging startups to use these flexible, low-cost networks to launch novel products and services. The city’s THRIVE program (Tech Hub & Research Infrastructure Virtual Ecosystem) offers free local cloud hosting services for researchers and developers.21 By combining the network’s flexible, free connections with THRIVE’s flexible, free storage, MUNICIPAL INNOVATION ON THE AMMON NETWORK The city itself has already begun experimenting with new forms of connectivity. For instance, it has created a program that, in emergencies, offers free Internet access at the click of a mouse. Under this program, any user paying the monthly maintenance fee can get 45 minutes of basic Internet access from the city by clicking a button. This is meant as an emergency fallback for users who are unable to afford the services of a private ISP or are facing unexpected outages or technical difficulties.23 21 For more details, see: http://ammonthrive.org 22 Facilitating connectivity between virtualized networks (as opposed to within a single virtualized network) will require the development of ‘Software-Defined Change’ (SDEx) technologies and standards. This technology is currently being developed at e.g. Princeton, Georgia Tech, Virginia Tech and the University of Utah, with funding from the National Science Foundation and USIgnite. 23 This feature is part of Ammon’s efforts under the LifeLine program, which provides federal support for connecting low-income users to phone and broadband service. See, generally: Federal Communications Commission, “Lifeline Program for Low-Income Consumers”, . Enabling Competition and Innovation on a City Fiber Network | Responsive Communities Initiative 8 Another example is the ActiveShooter program, a partnership between local schools and the 911 dispatch center. If gunshots are ever detected in the schools, the network can automatically grant priority to the data traffic from the school’s security cameras, ensuring reliable high-speed streaming for first responders.24 In a conventional network, this would require dedicated fiber lines running from point to point; in the Ammon network, this bandwidth can be simply be allocated dynamically in rare instances when it might be needed, leading to a far more efficient use of network resources. AMMON’S VIRTUALIZATION APPROACH DIFFERS FROM THAT OF OTHER U.S. MUNICIPAL PROVIDERS Some other U.S. cities are building fiber networks and using virtualization to enable retail competition.  For example, the Utah Telecommunications Open Infrastructure Agency (UTOPIA), a consortium that serves 11 cities in Utah, is in the process of building out a fiber networkover which  private  ISPs compete. UTOPIA customers pay their city simply for the physical rollout, with the option of either leasing the fiber line and router from their city at $30 per month, buying the line up-front for $2,750, or financing over various periods of time. UTOPIA customers now number 16,000, and they have many choices—10 residential providers and more than 25 business service providers compete over the network. UTOPIA uses virtualization to a lesser extent than does Ammon. To get started, UTOPIA customers typically sign up through the consortium’s website or contact an ISP over the phone or by using the Internet from some other location. UTOPIA then schedules the installation of their fiber. Once users choose a provider, all services they receive are from that provider unless they cancel and enter into a new contract with a different provider. These changes are executed using virtualization technology. However, this is done by ISPs working with UTOPIA, not by the user sitting in front of a Web-based dashboard at home. In practice, this means that UTOPIA users can’t do certain things that are technically possible in Ammon. UTOPIA users can’t take services from more than one provider at a time; they can’t forgo an ISP service plan and still access city services and emergency alerts; and they can’t fashion their own sub-networks within the city’s service territory. And while many providers can and do compete over the UTOPIA network, they face moderately higher barriers  to entry: providers on the UTOPIA network must pay at least $5,000 to provide service. (In Ammon, these costs can be as low as $100.) Finally, the funding model for Ammon’s network is different. In UTOPIA’s model, infrastructure rollout is funded in part through municipally backed bonds; in Ammon’s LID model, the participating users bear all costs. STATE LAW & THE AMMON MODEL Twenty-one states have passed laws restricting or preventing local governments from providing Internet access. For example, Tennessee and North Carolina have prohibited municipal electric utilities from offering Internet access service beyond the boundaries of their electric service territory (and have also prohibited municipalities without electric utilities from offering service at all). In 2015, the FCC moved to preempt these laws as an undue restriction on local autonomy to provide communications services, but a federal appeals court later ruled that the FCC had overstepped its authority. This ruling has allowed U.S. states to continue restricting municipalities in this area. Although Idaho is not one of these 21 states, the City of Ammon decided to petition a state district court to confirm its municipal powers in this area before starting construction on its first LID. In February 2016, a district judge issued a decision affirming that Ammon had the legal right to construct, operate, and maintain the network.25 The court also confirmed that Ammon could rely 24 City of Ammon, “Ammon creates software to assist emergency responders in crisis event”, October 2, 2014, . 25 The City’s authority to apply a LID model for fiber networks was recognized in a judicial confirmation by the local District Court. The confirmation ruling is available online: http://www.localnetchoice.org/wp-content/uploads/2016/04/Ammon-Bonneville-County-Idaho-Seventh-Circuit-Findings-of-Fact-2-29-16.pdf Enabling Competition and Innovation on a City Fiber Network | Responsive Communities Initiative 9 on an Idaho statute that authorizes LID-based financing for public works. According to Patterson, the voluntary, opt-in nature of the LID model was central to the project’s success in the deeply conservative community of Ammon. CONCLUSION By providing virtualized fiber network access as a public utility, Ammon has created a platform that allows an extraordinary level of competition, innovation, and experimentation by businesses, local government, and residential users alike. And Ammon’s model provides very little, if any, financial risk to the city. Given that Ammon’s network is presently very small in extent—only 145 customers were taking service at the time of this report’s publication— it’s too early to conclude that the city’s approach is more successful than other types of open-access networks. But the city’s project is worthy of study by other municipalities and public entities seeking to promote retail competition and encourage new services and applications. Vertically integrated private ISPs have little incentive to open up their networks for virtualized access by competitors and innovators. ABOUT THE PROJECT Responsive Communities is a project of the Berkman Klein Center for Internet & Society at Harvard University. Responsive Communities addresses the most important issues of social justice, civil liberties, and economic development involving Internet access and government use of data. The initiative offers a forum for meaningful engagement across academia, government, and industry. Through a cross-disciplinary approach, including law, public policy, business, engineering, and design, and a combination of academic and applied efforts, we work to shape communities that are fully connected and truly responsive to people's needs. AUTHORS Paddy Leerssen is an LL.M. candidate at Harvard Law School. David Talbot is a fellow at the Berkman Klein Center for Internet & Society at Harvard University. ACKNOWLEDGMENTS The authors thank Susan Crawford, John A. Reilly Clinical Professor of Law at Harvard Law School, and Waide Warner, Lecturer on Law at Harvard Law School and a Senior Advisor at the CyberLaw Clinic of Harvard Law School, for their guidance in the preparation of this report. The authors also thank Chris Mitchell, Director of the Community Broadband Networks Initiative at the Institute for Local Self-Reliance, and Ashley Stelfox, Associate at Baller Stokes & Lide, a law firm based in Washington, DC, for invaluable assistance in reviewing the report. Finally the authors thank Bruce Patterson, Technology Director, City of Ammon; Jeff Christensen, CEO of EntryPoint Networks; Eric Lampland, President, Lookout Point Communications; Roger Timmerman, Executive Director at UTOPIA; and Robert Ricci, Associate Professor, University of Utah, for sharing time and source materials. This report was made possible by grants from the Ford Foundation and the Open Society Foundations. The use of virtualization technology to enable retail competition is rare in the United States, and Ammon’s use of virtualization is especially sophisticated. Municipalities and other public agencies could lead the way in advancing all forms of open-access models, including ones like Ammon’s and others in which cities build and own dark fiber and make it available for lease by private ISPs. Enabling Competition and Innovation on a City Fiber Network | Responsive Communities Initiative 10