Foreword xvii
Preface xxi
Acknowledgments xxiii
1 An Overview of Reliability and Resiliency in Today's Mission Critical Environment 1
1.1 Introduction 1
1.2 Risk Assessment 5
1.2.1 Levels of Risk 6
1.3 Capital Costs versus Operation Costs 7
1.4 Critical Environment Workflow and Change Management 9
1.4.1 Change Management 10
1.5 Testing and Commissioning 11
1.6 Documentation and Human Factor 16
1.7 Education and Training 20
1.8 Corporate Knowledge Transfer - the Means to Securing Tomorrow's Critical Infrastructure 21
1.9 Operation and Maintenance 24
1.10 Employee Certification 25
1.11 Standards and Benchmarking 25
1.12 What is a Mission Critical Engineer 26
1.13 Conclusion 28
1.14 An Overview of Reliability and Resiliency in Today's Mission Critical Environment - Questions to Consider 28
2 Energy and Cyber Security and its Effect on Business Resiliency 31
2.1 Introduction 31
2.2 Risks Related to Information Security 36
2.3 Electro Magnetic Pulse and Solar Flares 42
2.4 How Risks Are Addressed 47
2.5 Use of Distributed Energy Resources and Generation 52
2.6 Documentation and Its Relation to Information Security 55
2.7 Smart Grid 57
2.8 Conclusion 60
2.9 Energy Security and Its Effect on Business Resiliency - Questions to Consider 60
3 Mission Critical Engineering with an Overview of Green Technologies 63
3.1 Introduction 63
3.2 Companies' Expectations: Risk Tolerance and Reliability 65
3.3 Identifying the Appropriate Redundancy in a Mission Critical Facility 67
3.4 Improving Reliability, Maintainability, and Proactive Preventative Maintenance 69
3.5 The Mission Critical Facilities Manager and the Importance of the Boardroom 71
3.6 Quantifying Reliability and Availability 71
3.6.1 Review of Reliability Terminology 72
3.7 Design Considerations for the Mission Critical Data Center 73
3.7.1 Data Center Certification 74
3.8 The Evolution of Mission Critical Facility Design 76
3.9 Human Factors and the Commissioning Process 77
3.10 Short Circuit & Coordination Studies 79
3.11 Introduction to Direct Current in the Data Center 84
3.11.1 Advantages of DC Distribution 85
3.11.2 Lighting Updates 87
3.11.3 DC Storage Options 87
3.11.4 Renewable Energy Integration 88
3.11.5 DC and Combined Cooling, Heat & Power 89
3.11.6 Safety Issues 91
3.11.7 Maintenance 91
3.11.8 Education & Training 92
3.11.9 Future Vision 93
3.12 Containerized Systems Overview 93
3.13 Mission Critical Engineering with an Overview of Green Technologies - Questions to Consider 95
4 Mission Critical Electrical System Maintenance & Safety 103
4.1 Introduction 103
4.2 The History of the Maintenance Supervisor and the Evolution of the Mission Critical Facilities Engineer 105
4.3 Internal Building Deficiencies and Analysis 107
4.4 Evaluating Your System 108
4.5 Choosing a Maintenance Approach 110
4.5.1 Annual Preventive Maintenance 111
4.6 Safe Electrical Maintenance 112
4.6.1 Standards and Regulations 112
4.6.2 Electrical Safety: NFPA 70E Arc Flash Mitigation 114
4.6.3 Personal Protective Equipment (PPE) 117
4.6.4 Lockout/Tagout 126
4.7 Maintenance of Typical Electrical Distribution Equipment 127
4.7.1 Thermal Scanning and Thermal Monitoring 127
4.7.2 15 KV Class Equipment 129
4.7.3 480 Volt Switchgear 130
4.7.4 Motor Control Centers and Panel Boards 131
4.7.5 Automatic Transfer Switches 131
4.7.6 Automatic Static Transfer Switches (ASTS) 132
4.7.7 Power Distribution Units 132
4.7.8 277/480 Volt Transformers 133
4.7.9 Uninterruptible Power Systems 133
4.8 Being Proactive in Evaluating the Test Reports 134
4.9 Designing for Safety and Reliability 135
4.10 Conclusion 136
5 Standby Generators 137
5.1 Introduction 137
5.2 The Necessity for Standby Power 138
5.3 Emergency, Legally Required, and Optional Systems 140
5.4 Standby Systems That Are Legally Required 141
5.5 Optional Standby Systems 142
5.6 Understanding Your Power Requirements 142
5.7 Management Commitment and Training 142
5.7.1 Lockout/ Tagout (LOTO) 143
5.7.2 Training 144
5.8 Standby Generator Systems Maintenance Procedures 144
5.8.1 Maintenance Record Keeping and Data Trending 145
5.8.2 Engine 145
5.8.3 Coolant System 145
5.8.4 Electrical / Control System 146
5.8.5 Generator 146
5.8.6 Automatic and Manual Switchgear 147
5.8.7 Load Bank Testing 147
5.9 Documentation Plan 148
5.9.1 Proper Documentation and Forms 148
5.9.2 Record keeping 148
5.10 Emergency Procedures 149
5.11 Cold Start 150
5.12 Non-Linear Load Concerns 151
5.12.1 Line Notches and Harmonic Current 151
5.12.2 Voltage / Frequency Drop 152
5.12.3 Voltage / Frequency Rise 152
5.12.4 Frequency Fluctuation 153
5.12.5 Synchronizing to Parallel 154
5.12.6 Automatic Transfer Switch 154
5.13 Conclusion 155
6 Fuel Systems Design and Maintenance 157
6.1 Introduction 157
6.2 Brief Discussion on Diesel Engines 158
6.3 Bulk Storage Tank Selection 159
6.3.1 Aboveground Tanks 159
6.3.2 Modern Underground Tanks and Piping Systems 160
6.3.3 Fuel Receiving Tanks 161
6.3.4 Generator Sub-Base Tanks 161
6.4 Codes and Standards 162
6.5 Recommended Practices for all Tanks 163
6.6 Fuel Distribution System Configuration 168
6.7 Day Tank Control System 170
6.8 Diesel Fuel and a Fuel Quality Assurance Program 174
6.9 Conclusion 186
7 Power Transfer Switch Technology, Applications, and Maintenance 187
7.1 Introduction 187
7.2 Transfer Switch Technology and Applications 189
7.3 Types of Power Transfer Switches 191
7.3.1 Manual Transfer Switches 191
7.3.2 Automatic Transfer Switches 191
7.4 Control Devices 204
7.4.1 Time Delays 204
7.4.2 In-Phase Monitor 205
7.4.3 Test Switches 206
7.4.4 Exercise Clock 207
7.4.5 Current, Voltage and Frequency Sensing 207
7.5 Design Features 207
7.5.1 Close Against High In-Rush Currents 208
7.5.2 Withstand and Closing Rating (WCR) 208
7.5.3 Carry Full Rated Current Continuously 208
7.5.4 Interrupt Current 209
7.6 Additional Characteristics and Ratings of ATS 209
7.6.1 NEMA Classification 209
7.6.2 System Voltage Ratings 209
7.6.3 ATS Sizing 209
7.6.4 Seismic Requirement 210
7.7 Installation & Commissioning, Maintenance, and Safety 210
7.7.1 Installation & Commissioning 210
7.7.2 Maintenance & Safety 212
7.7.3 Maintenance Tasks 214
7.7.4 Drawings and Manuals 215
7.7.5 Testing & Training 215
7.8 General Recommendations 218
7.9 Conclusion 219
8 The Static Transfer Switch 221
8.1 Introduction 221
8.2 Overview 222
8.2.1 Major Components 222
8.3 Typical Static Switch One Line 223
8.3.1 Normal Operation 223
8.3.2 Bypass Operation 224
8.3.3 STS and STS/transformer Configurations 225
8.4 STS Technology and Application 225
8.4.1 General Parameters 225
8.4.2 STS Location and Type 226
8.4.3 Advantages and Disadvantages of the Primary and Secondary STS/Transformer Systems 226
8.4.4 Monitoring, Data Logging, and Data Management 227
8.4.5 Downstream Device Monitoring 227
8.4.6 STS Remote Communication 228
8.4.7 Security 228
8.4.8 Human Engineering and Eliminating Human Errors 229
8.4.9 Reliability and Availability 230
8.4.10 Repairability and Maintainability 231
8.4.11 Fault Tolerance and Abnormal Operation 232
8.5 Testing 232
8.6 Conclusion 233
9 The Fundamentals of Power Quality 235
9.1 Introduction 235
9.2 Electricity
Basics 237
9.2.1 Basic Circuit 238
9.2.2 Power Factor 238
9.3 Transmission of Power 241
9.3.1 Life Cycle of Electricity 241
9.3.2 Single-Phase and Three-Phase Power Basics 243
9.3.3 Unreliable Power versus Reliable Power 245
9.4 Understanding Power Problems 245
9.4.1 Power Quality Standards 246
9.4.2 Power Quality Transients 249
9.4.3 RMS Variations 250
9.4.4 Causes of Power Line Disturbances 255
9.4.5 Power Line Disturbance Levels 261
9.5 Tolerances of Critical Loads 261
9.5.1 CBEMA Curve 263
9.5.2 ITIC Curve 263
9.5.3 Purpose of Curves 265
9.6 Power Monitoring 265
9.7 The Impact of Alternative Energy Generation 268
9.8 Conclusion 269
10 UPS Systems: Applications and Maintenance with an Overview of Green Technologies 273
10.1 Introduction 273
10.1.1 Green and Reliability Overview 273
10.2 Purpose of UPS Systems 275
10.3 General Description of UPS Systems 279
10.3.1 What is a UPS system? 279
10.3.2 How does a UPS system work? 279
10.3.3 Static UPS Systems 280
10.3.4 Online 281
10.3.5 Double Conversion 282
10.3.6 Double Conversion UPS Power Path 282
10.4 Components of a Static UPS System 284
10.4.1 Power Control Devices 284
10.5 Online - Line Interactive UPS Systems 291
10.6 Offline (Standby) 292
10.7 The Evolution of Static UPS Technology 293
10.7.1 Emergence of the IGBT 293
10.7.2 Two and Three-Level Rectifier/Inverter Topology 294
10.7.3 Silicon Carbide Replaces Silicon as UPS Semiconductor of Electricity 295
10.8 Rotary UPS Systems 299
10.8.1 UPSs Using Diesel 300
10.8.2 Hybrid UPS Systems 301
10.9 Redundancy, Configurations, and Topology 301
10.9.1 N 302
10.9.2 N+1 302
10.9.3 Isolated Redundant 303
10.9.4 N+2 303
10.9.5 2N 304
10.9.6 2(N+1) 305
10.9.7 Distributed Redundant / Catcher UPS 305
10.9.8 Eco-Mode for Static UPS 306
10.9.9 Availability Calculations 307
10.10 Energy Storage Devices 308
10.10.1 Battery 308
10.10.2 Flywheel Energy 314
10.11 UPS Maintenance & Testing 316
10.11.1 Physical Preventive Maintenance (PM) 318
10.11.2 Protection Settings, Calibration, and Guidelines 318
10.11.3 Functional Load Testing 319
10.11.4 Steady State Load Test 319
10.11.5 Steady State Load Test at 0%, 50% and 100% load: 320
10.11.6 Harmonic Analysis and Testing 320
10.11.7 Filter Integrity and Testing 321
10.11.8 Transient Response Load Test 322
10.11.9 Module Fault Test 322
10.11.10 Battery Run Down Test 322
10.12 Static UPS and Maintenance 323
10.12.1 Examples of Semi-Annual Checks and Services for UPS Systems 324
10.13 UPS Management 324
10.14 Conclusion 325
11 Data Center Cooling Systems 327
11.1 Introduction 327
11.2 Background Information 330
11.3 Cooling within Datacom Rooms 331
11.4 Cooling Process 332
11.4.1 Cooling Process in Datacom Space 332
11.4.2 Direct Expansion (DX) Systems 333
11.4.3 Chilled Water Systems 334
11.5 Cooling Final Dissipation 334
11.5.1 Air Cooled System 335
11.5.2 Water Side 335
11.6 The Refrigeration Process 337
11.6.1 Refrigeration Equipment - Compressors 337
11.6.2 Refrigeration Equipment - Chillers 338
11.6.3 Heat Rejection Equipment 342
11.6.4 Energy Recovery Equipment 353
11.6.5 Heat Exchangers 360
11.7 Components Inside Datacom Room 363
11.7.1 Computer Room Cooling Units 363
11.8 Conclusion 373
12 Data Center Cooling Efficiency, Concepts, & Technologies 375
12.1 Introduction 375
12.2 Heat Transfer Inside Data Centers 379
12.2.1 Heat Generation 379
12.2.2 Heat Return 380
12.2.3 Cooling Air 380
12.3 Cooling and Other Airflow Topics 381
12.3.1 Leakage 381
12.3.2 Mixing and its Relationship to Efficiency 382
12.3.3 Re-circulation 382
12.3.4 Venturi Effect 382
12.3.5 Vortex Effect 383
12.3.6 CRAC/CRAH Types 383
12.3.7 Potential CRAC Operation Issues 383
12.3.8 Sensible vs. Latent Cooling 384
12.3.9 Humidity Control 386
12.3.10 CRAC Fighting / Too Many CRACs 387
12.4 Design Approaches for Data Center Cooling 388
12.4.1 Hot Aisle/Cold Aisle 388
12.4.2 Cold Aisle Containment 388
12.4.3 In-Row Cooling with Hot Aisle Containment 388
12.4.4 Overhead Supplemental Cooling 389
12.4.5 Chimney or Ducted Returns 389
12.4.6 Advanced Active Airflow Management for Server Cabinets 390
12.5 Additional Considerations 390
12.5.1 Active Air Movement 390
12.5.2 Adaptive Capacity 390
12.5.3 Liquid Cooling 391
12.5.4 Cold Storage 392
12.6 Hardware & Associated Efficiencies 392
12.6.1 Server Efficiency 392
12.6.2 Server Virtualization 392
12.6.3 Multi-Core Processors 393
12.6.4 Blade Servers 393
12.6.5 Energy Efficient Servers 393
12.6.6 Power Managed Servers 393
12.6.7 Effect of Dynamic Server Loads on Cooling 393
12.7 Best Practices 394
12.8 Efficiency Problem Solving 394
12.9 Conclusion 396
12.10 Conversions, Formulas, Guidelines 396
13 Raised Access Floors 397
13.1 Introduction 397
13.1.1 What is an Access Floor? 397
13.1.2 What are the Typical Applications for Access Floors? 399
13.1.3 Why use an Access Floor? 399
13.2 Design Considerations 400
13.2.1 Determine the Structural Performance Required 400
13.2.2 Determine the Required Finished Floor Height 403
13.2.3 Determine the Understructure Support Design Type Required 404
13.2.4 Determine the Appropriate Floor Finish 405
13.2.5 Air Flow Requirements 406
13.3 Safety Concerns 409
13.3.1 Removal & Reinstallation of Panels 409
13.3.2 Removing Panels 409
13.3.3 Stringer Systems 411
13.3.4 Protection of the Floor from Heavy Loads 412
13.3.5 Grounding the Access Floor 417
13.3.6 Fire Protection 418
13.3.7 Zinc Whiskers 419
13.4 Panel Cutting (For all Steel Panels or Cement Filled Panels that do not Contain an Aggregate) 419
13.4.1 Safety Requirements for Cutting Panels 419
13.4.2 Guidelines for Cutting Panels 420
13.4.3 Cutout Locations in Panels; Supplemental Support for Cut Panels 420
13.4.4 Saws and Blades for Panel Cutting 420
13.4.5 Interior Cutout Procedure: 421
13.4.6 Round Cutout Procedure 421
13.4.7 Installing Protective Trim Around Cut Edges 421
13.4.8 Cutting and Installing the Trim 422
13.5 Access Floor Maintenance 423
13.5.1 Best Practices for Standard High Pressure Laminate Floor Tile (HPL) and for Vinyl Conductive & Static Dissipative Tile 423
13.5.2 Damp Mopping Procedure for HPL and Conductive & Static Dissipative Vinyl Tile 423
13.5.3 Cleaning the Floor Cavity 424
13.6 Troubleshooting 424
13.6.1 Making Pedestal Height Adjustments 425
13.6.2 Rocking Panel Condition 425
13.6.3 Panel Lipping Condition (Panel Sitting High) 425
13.6.4 Out-of-Square Stringer Grid (Twisted Grid) 426
13.6.5 Tipping at Perimeter Panels 427
13.6.6 Tight Floor or Loose Floor: Floor Systems Laminated with HPL Tile 427
13.7 Additional Design Considerations 428
13.7.1 LEED Certification 428
13.7.2 Energy Efficiency - Hot and Cold Air Containment 428
13.7.3 Airflow Distribution and CFD Analysis 429
13.8 Conclusion 437
14 Fire Protection in Mission Critical Infrastructures 439
14.1 Introduction 439
14.2 Hazard Analysis 441
14.3 Alarm and Notification 441
14.4 Early Warning Detection 444
14.4.1 Wireless Detection 445
14.5 Fire Suppression 445
14.5.1 Hybrid Fire Suppression Systems 448
14.5.2 Protecting Lithium Ion Batteries 449
14.6 Systems Design 450
14.6.1 Stages of a Fire 450
14.6.2 Fire and Building Codes 451
14.7 Fire Detection 452
14.8 Fire Suppression Systems 461
14.8.1 Water Mist Systems 467
14.8.2 Carbon Dioxide Systems 470
14.8.3 Clean Agent Systems 472
14.8.4 Inert Gas Agents 472
14.8.5 IG-541 473
14.8.6 IG-55 474
14.8.7 Chemical Clean Agents 474
14.8.8 Portable Fire Extinguishers 479
14.8.9 Clean Agents and the Environment 479
14.9 Conclusion 480
15 Managing Through Pandemics 481
15.1 Executive Summary: COVID-19's Impact on Critical Infrastructure Globally 481
15.2 Architectural Solutions and Air Purification Systems 482
15.2.1 HVAC Systems 482
15.2.2 UV Technology 482
15.2.3 Bipolar Ionization 485
15.2.4 Copper Doorknobs 485
15.2.5 Architectural Improvements to be Considered 486
15.3 Building Equipment Solutions and Technology 487
15.3.1 Cleaning vs. Disinfecting vs. Sanitizing 487
15.3.2 Intensify Cleaning Frequency and Measures 487
15.3.3 IR Scans 488
15.3.4 Rethinking the flush, the sink, and the hand dryer 488
15.3.5 Technology 489
15.4 Operations, Maintenance and Training 491
15.4.1 Personal Protection 491
15.4.2 Change in Operation 491
15.4.3 Data Center Betterment Opportunities 492
15.5 Site Protection: Safeguarding the Staff and Visitors 493
15.6 The Workforce of Tomorrow 494
15.7 Assessment Tasks - HVAC and Air Handling Units Filter Upgrades 495
15.8 Managing Through Pandemics -Questions to Consider 496
15.9 Conclusion 497
Appendix A Policies and Regulations 499
A.1 Introduction 499
A.2 Industry Policies & Regulations 501
A.2.1 USA PATRIOT Act 503
A.2.2 Sarbanes-Oxley Act (SOX) 505
A.2.3 Comprehensive Environmental Response, Compensation, and Liability Act of 1980
(also known as the Superfund Act) 506
A.2.4 Executive Order 13423: Strengthening Federal Environmental, Energy and Transportation Management 507
A.2.5 ISO27000 Information Security Management System (ISMS) 508
A.2.6 The National Strategy for the Physical Protection of Critical Infrastructures and Key Assets 513
A.2.7 2009 National Infrastructure Protection Plan 514
A.2.8 North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection Program 514
A.2.9 U.S. Security & Exchange Commission (SEC) 516
A.2.10 Sound Practices to Strengthen the Resilience of the U.S. Financial System 516
A.2.11 C4I Command, Control, Communications, Computers, and Intelligence 517
A.2.12 Basel II Accord 519
A.2.13 National Institute of Standards and Technology (NIST) 519
A.2.14 Business Continuity Management Agencies and Regulating Organizations 521
A.2.15 FFIEC - Federal Financial Institutions Examination Council 523
A.2.16 National Fire Prevention Association 1600 - Standards on Disaster/Emergency Management and Business Continuity Programs 524
A.2.17 Private Sector Preparedness Act 525
A.3 Data Protection 526
A.4 Encryption 528
A.4.1 Protecting Critical Data through Security and Vaulting 529
A.5 Business Continuity Plan (BCP) 529
A.6 Conclusion 531
Appendix B Consolidated List of Key Questions 535
Appendix C Airflow Management (A System Approach) 553
C.1 Introduction 553
C.2 Control is the Key 555
C.3 Obtaining Control 558
C.4 Air Management Technologies 565
C.5 Conclusion 570
Glossary 573
References 595
Index 609
