IEEE Southeastern Michigan – Wavelengths | August 2023
(See Pages 18-22)
Case Study Abstract:
August 2003 North American Blackout: University of Michigan Response
Michael A. Anthony, P.E.
It was one of those late summer days that makes poets of us all; a clear-skyed, not-too-humid day between the end of Ann Arbor Art Fair and the beginning of students trickling back to campus for the new school year. On the three Ann Arbor campuses — Central, Hospital (Medical) and North — normally the milieu for 60,000 students, faculty and staff spread across 4000 acres — economic activity was significantly reduced, mild and pleasant. In academia in August, the weekend starts on Thursday.
Since the University’s Central Heating Plant (CHP) began operation sometime in the early 1900’s (in the lowest part of campus so that gravity would help condensate return to its origin) it had never had to reckon with anything like the colossal blackout of August 13, 2003. Over the previous 100-odd years the CHP evolved from a heating plant for steam and hot water into a large university Central Power House (CHP → CPP) which now generates 50 megawatts of electricity and a million pounds of steam through 10-odd miles of tunnels. In effect it evolved into a microgrid long before microgrids were cool.
When the “Old Main” hospital was demolished in the early 1990’s it was decided that Old Main’s replacement (the building that stands there now) would sever its dependence on CPP electricity; a decision influenced by political atmospherics around district energy at the time. Despite the proximate 1500 foot electrical distance between the University Hospital and the CPP an effective “voltage wall” was created; complicated when transformer manufacturing standards changed from 13.2 kV to 13.8 kV. The 4 percent voltage difference between the 13.2 kV secondary output of the DTE’s CPP transformers and the 13.8 kV secondary output of DTE’s University of Michigan Hospitals (UMH) transformers was not troublesome until the turn of the millennium.
In the minutes before the outage crossed the Ohio and Canadian borders DTE was aware of anomalies gathering pace with the regional transmission operator. The CPP was generating normally — about 26 MW of its own power and buying 17 MW from DTE Energy. Think of mix as the “home position” for energy production for a late summer afternoon. CPP operators try to tune the energy mix so that the “house power” requirement is met at the same time the generators operate in the sweet spot of their efficiency curve (typically 50 percent loading). Operators also want to maintain purchased power below the annual peak purchased power contract demand set by DTE which, if breached, present punitive demand charges.
Farther away from the CPP heating and cooling was provided by regional steam chillers that reduced the need for inefficient and unsightly split DX and window AC units. Cooling to the Hospital and North Campus, essential for patient care was completely dependent upon DTE power. Ninety-six hours of life safety power for UMH and ninety minutes of code minimum egress power but nothing after that; except where a few backup generators supported laboratory research (i.e. animals, freezers, incubators, special alarms, etc.) .
[2]
A daytime blackout in beautiful weather takes a few minutes to figure out. I was driving back to Central Campus from field work at the Argus Building on Ann Arbor’s Old West Side. I noticed traffic signals were dark. Drivers took turns at intersections. At that point I knew that the outage was at least an Edison problem; perhaps an errant squirrel. When, further down the west boundary of Central Campus I saw exterior egress lighting units energized, I knew Edison’s problem was also our problem. I tuned to WJR — “The Voice of the Great Lakes” — the go-to place on the radio dial for emergency communication protocols. Once back at the Plant Operations complex I found the dispatch center running on the generator we built for Y2K.
At that point 69 Michigan counties had dropped power to 2.1 million DTE Energy customers and 1.9 million Consumers Energy customers.
[3]
The life-saving success of the University of Michigan Plant Operations response to the August 14th blackout begins the year ahead of the turn of the century when it prepared for embedded system clocks that could not differentiate Year 1900 from Year 2000. Everything with a microprocessor — information and communications, security, and energy — required identification and risk evaluation. Lots of field work, spreadsheets and meetings. Lots of meetings. To many who knew memory board level programming, it seemed a trivial problem but others spread the belief that the “bug” could propagate incalculable economic disruption and societal breakdown. Thus, the genesis of the “Year 2000 Information and Readiness Disclosure Act” *.
At the beginning of 1999 Plant Operations was in compliance with all its paperwork obligations for Y2K but time was running out on non-paperwork defend-in-place contingencies for University Hospital. Week by week, the cost of reserving ten megawatts of mobile generation escalated beyond belief. When the pain point reached $2 million we were given the green light to find a way to supply power to UMH from the CPP:
Q. How do we get cabling between the two systems? (A. We found an unused breaker and empty conduits in a switchgear room that serviced the original 1928 University Hospital).
Q. How will UMH run with a 4 percent difference in voltage? (A. Electricians confirmed that most transformers voltage taps that would permit 7.5% adjustment.)
Q. How much will it cost? (A. Construction bids — including a one-off 4% autotransformer — came in less than $200,000)
Q. Do we still have time? (A. A Wisconsin-based manufacturer made quick work of it and delivered it before Thanksgiving)
Thus the “C.S. Mott Children’s Hospital Intertie” was born. As the clock struck midnight on the new millennium the catastrophic failures were nowhere to be found.
[4]
Many people in the upper quadrant of North America remember what they are doing on the Thursday, Friday and Saturday 1322 days later. (Times are approximate.*)
[Thursday 16:11] The outage initializes when a 345 kV transmission line sags into overgrown trees in suburban Cleveland. An improbable combination of man and machine errors in regional transmission operation centers became probable.
[Thursday 16:15] Power flows in a giant loop from southern Ohio to Pennsylvania to New York to Ontario to Michigan to northern Ohio. Detroit Edison and Consumers Energy drop 1,863 MW and 820 MW of generation, respectively.
[Thursday 16:16] Five minutes later the outage is essentially complete. DTE system frequency = 00.00 Hz. Transmission protective regimes did their job — they protected generators and transmission lines — but now about 50 million people in North America are without power.
At the University of Michigan CPP frequency is 60.0 Hz, nominal. University Emergency Command Center is running on generators. Plant Operations Zone managers scramble with radios and plain old telephone service. Electricians and mechanics fan out to buildings, resetting a bazillion alarms as necessary. Zone maintenance staff reports no one trapped in elevators.
[Thursday 17:00] Through WJR we learn there will be no quick return to normal. CPP operators are in communication with DTE primary service representatives and lock out its service switchgear. With UM system stable we were confident we could send UM generated power down the 13.2 kV distribution network to give the regional boiler operators the power they need to re-start the absorption chillers which, in turn, loaded the heat recovery gas turbines which created more steam to run through the steam generators. Additionally, a balance condenser mounted on the roof of the CHP– normally used to shave electrical demand peaks — was put into service to maintain steam pressure.
Iterating steam and electrical load between the CPP and the outlying boiler teams gets done in stages over the next few hours; proving the engineering beauty of district energy and what would later be identified as a microgrid.
[Thursday 18:00] Hospital Security calls the CPP to say UMH is getting mighty hot. After a few phone calls the Y2K protocol for relief was approved by the Director of Plant Operations — ” “Close the C.S. Mott Intertie” Campus electricians and UMH electricians scrambled, adding load incrementally, measuring voltage at key substations; eventually ramping up to about 4 MW — enough to provide power well above minimums to chillers for surgical units, birthing, clinics and critical laboratories, sterilization, etc.
For the next 18-odd hours Ann Arbor could hear the world without the ambient 60 Hz hum of electrical energy. You could hear the birds. Voices traveled farther. The skies were empty of airplanes. You could see stars. People made the best of it, gathering in the well-lit Diag which had morphed into a place of refuge for people you normally only see during the Hash Bash. People brought their sleeping bags and tents to be in the Diag light which felt safe to them. .
[Friday 21:00] DTE restoration accelerates with re-energization of its 120 kV distribution network. Initializing a cranking path for all the generators on a large grid like DTE’s is non-trivial; not unlike lighting a birthday cake with one candle to the next. By over-exciting its generators the CPP was able to provide a measure of reactive power to DTE to hasten grid support for the Ann Arbor region.
By early Saturday August 16 DTE had restored power and so had much of the Northeastern part of North America. An estimated 100 lives were lost (indirectly; mostly on the East Coast) and at a cost in the range of $10-$100 billion.
Through the whole ordeal the CPP remained the beating heart of the University of Michigan Ann Arbor campus.
[5]
Today’s power grid has been significantly improved since 2003 but some smaller outages are noteworthy. The September 2011 Southwest blackout affected 7 million. Hurricane Florence blackout affected 4.5 million. The February 2021 ERCOT blackout affected 4.5 million. During Hurricane Sandy in October 2012 and the Princeton University “microgrid” provided some local relief for above life-safety minimums though not nearly at the same scale as at the University of Michigan with the urgency of a 900-bed hospital.
Reflections & Recommendations:
- The “scary” part of the Y2K scare was authored by the federal government; which, in recent history, has proven sketchy in its judgements about national emergencies. Admittedly, the Y2K legislation provided impetus for upgrades that might not have been driven any other way.
- Amateur radio and plain old broadcast radio are an essential part of the US emergency alert system. Regulators should find other things to regulate. AM radio bands have many technical advantages over FM for coverage, penetration and resilience in four weather and should remain in all automobiles.
- Develop more solutions to manage water-electric-fuel interdependence with natural gas supply. You cannot maintain municipal water supply and sanitation systems with solar panels. Loss of backflow pressure can contaminate a community in seconds.
- Grid enhancement technology (smart grid and prosumer) innovations at the utility level are welcomed but more local contingencies can be developed by upscaling the backup systems already required by building safety codes and permitting proximate buildings to share household generators.
- Having even 1000 watts of power from a car battery for household defend-in-place; will dampen the likelihood of civil unrest if outages last weeks. Hybrid and conventional automobiles can be tricked out with DC/AC inverters and plugs to provide power to one or more households for weeks for less than $200 and with less noise. The largest energy storage system in the USA (in terms of BTU’s) is mobile and stored in the ~300-million road vehicles.
- Harden blackout damages and hasten recovery with resilient internet access. People can function in daylight without power but they cannot function either day or night without internet access. The iPhone did not arrive until four years later, Anything close to the normal order of business has been unthinkable since.
This is only a snapshot of what happened on one of America’s largest campus power systems. Confidentiality is essential for power security; but sharing “lessons learned” in some measure is an essential feature of academic freedom. There are many people who contributed to the University response — too many for a case study abstract. A more complete paper will include more engineering and management detail suitable for another IEEE publication.
Just as January 1, 2000 helped us with August 14, 2003, will August 14, 2003 prepare us for the next major regional contingency? It is hard to count something that does not happen. What is more certain is that if federal regulators persist in their janky energy policies by willfully building single points of failure it is only a matter of time.
References and Bibliography:
Final Report on the August 14, 2003 Blackout in the United States and Canada
Michigan Public Service Commission Report on August 14 2003 Blackout
North American Electric Reliability Council: Technical Analysis of the August 14, 2003, Blackout
Mike Anthony was the first full time high voltage engineer directly employed by any college or university in the United States. He worked there from 1982 until his retirement in 2016; after which he co-founded Standards Michigan (and 49-other state affiliates) which he leads today; while mentoring engineering students at the University of Michigan College of Engineering. He has been a technical committee member on the National Electrical Code since 1999, chairs the IEEE Education & Healthcare Facilities Committee and participates in the IEEE Power System Reliability and IEEE Forensics Committee. He has authored many papers for IEEE, for trade journals and is the author of three engineering textbooks published by McGraw-Hill.
