Typically, mission critical Network Operations Centers (NOCs) are associated with utility companies that monitor the flow of electricity or telephony systems or emergency operations centers that handle public security and transportation. Such organizations often build operations centers that run 24/7 and 365 days a year to ensure consistent service.
But what about financial organizations? Think of all the financial information that is stored in databases from credit cards to car loans, mortgages, personal loans, business loans and many more. Imagine if all that information was lost or could not be accessed. Isn’t that information critical?
AmeriCredit thinks so. Founded in 1992, AmeriCredit is a leading independent middle-market auto finance company. The company works with a network of auto dealers to provide loans to consumers who typically are unable to obtain financing from traditional sources. With more than one million customers and over $14 billion in managed auto receivables, AmeriCredit looks to innovative technology to provide best-in-class service to all its customers across the US and Canada.
AmeriCredit began operations with just one branch in Fort Worth, Texas in the early 1990s and now operates 90 branch offices throughout North America. The company’s growth and its commitment to innovation prompted the opening of a Network Operations Center in Texas to monitor its corporate data network.
The first control room they had was equipped with eight 27” computer monitors mounted on a wall. It was only able to accommodate four people and as the company grew more personnel were added in order to cover different aspects of the business, which required more space.
In 2002, AmeriCredit hired The Whitlock Group to build an innovative control room after a competitive review. The Whitlock Group is a well-established audio/visual systems integration firm based in Richmond, Virginia since 1955. With operations in Irving, Texas, the company designed and built sophisticated NOCs for clients ranging from Verizon Wireless to the Los Alamos National Laboratory.
Heading up The Whitlock team was account manager Craig Taylor who specializes in control room applications. Working closely with Jess Whitfield, senior network monitoring engineer for AmeriCredit, Taylor and his team developed a design to turn an old office space into an impressive control room with a distinctive “wow” factor.
In July 2002, the transformation began and the once empty room was turned into a sophisticated NOC over the next four months. With an industrial modern design in mind for the control room, The Whitlock Group showed Whitfield several different possible designs of furniture by Tresco. The company chose to have four consoles with slick curves in a black, white and metallic grey finish similar to furniture found in AT&T’s Network Operations Center.
There are four double consoles altogether in the room suitable for eight IT specialists. They’re laid out on two tiers with two consoles on each level. At each workspace, operators are equipped with two LCD monitors on which they are able to bring up any application that is accessible on the network. They can either study the network on their individual small computer monitors or share the information with their colleagues by displaying information on the large video wall. All of the operators in the room are able to control the images on the screens from their desktop making it easy to share information and gain a much larger overview of a situation.
“They wanted to be able to see a whole bunch of information at once,” said Taylor. “Some of the programs they monitor have a lot of icons and it takes a large display to get an instant snapshot of the overall condition of the network. In the absence of a large screen display it takes more investigation. You can’t see at a glance if anything’s wrong. You’d have to drill down and go across many screens.”
According to Taylor a common mistake made when building a NOC is to try to cut costs by purchasing a cheaper projector that would be a perfect solution for a conference room but would break down quickly in a 24/7 environment. Choosing the appropriate equipment to ensure reliability in continuous operation is essential.
The AmeriCredit video wall is comprised of 12, 50” Christie GraphXMASTER CX50-100U single-chip DLP™ display wall cubes, which are specifically designed for 24/7 mission critical environments. Arranged in a 2 x 6 array, the screens provide an overall image size of 60” in height and 240” in width when measured all together.
Having a large display wall enables AmeriCredit to monitor the status of all of its remote servers and locations via graphic depictions on screen.
“We’re spread out across the country,” said Whitfield. “The idea is to be able to get information on national events to our managers that might affect our ability to do business. If one of our branches isn’t functioning it will affect how loans are handled.”
In 2001, representatives from AmeriCredit attended a control room show hosted by The Whitlock Group. They were shown various projection solutions that could be used in a network operations center, including a display wall cube. Jess Whitfield also did quite a bit of research into the different display solutions available before choosing to go with display wall cubes.
“We knew we wanted to go with DLP technology,” said Whitfield. “The data we display is high resolution so we needed a display solution compatible to achieve consistently high quality images. It also seemed to be the most effective means for our size. The cubes fit into a small area so a dedicated room for rear projection was not required.”
All 12 of the display wall cubes are encased together in a custom-made black laminate base built by The Whitlock Group to create the appearance of a seamless video wall. Whitfield wanted the wall to appear completely seamless. It was not only important for the images on screen to align perfectly to create the appearance of one large screen but the seams and edges need to match up perfectly. So perfectly in fact that it would be possible to run a finger along the wall and not feel any physical deviations.
The Whitlock Group was faced with a challenge in the form of an uneven cement floor. They built the base for the wall and mounted all the cubes together but found that the floor cause the cubes to be off slightly so that it didn’t achieve the seamless appearance. With a strong eye for detail, Whitfield could stand at the back of the room and point out imperfections in the balance of the cubes, which most people would not notice. In order to get the display wall cubes level it was necessary to tear the wall down and redesign and rebuild the base to meet the exact specifications to achieve complete balance for the screens. In the end the screens were mounted within half a millimeter of each other.
“We had installed the project pretty much by the end of September except for the fact that we had to reconstruct the base,” said Taylor. “The custom base had to be redesigned to provide a level and stable surface for the cubes, which took till November. Having to compensate for a concrete floor that was not level took a lot of redesign.”
The final result was an impressive screen that stands as the focal point of the room. It acts almost like a large computer monitor that can be shared by all the operators. According to Whitfield the ability to display a high level overview of information enables greater efficiency in problem solving.
A short distance from the NOC is the nerve center of the corporation where 242 of the company’s servers reside in one room, including 18 RISC 6000 and 4 AS/400 servers by IBM as well as approximately 220 Compaq servers of varying models. All the information on AmeriCredit’s customers are stored on these servers. One of the primary responsibilities of the command center is to monitor that data center. If a problem occurs, the IT specialists will be able to pinpoint it and react quickly to restore service.
The fact that the servers are so close actually doesn’t make a difference to the network. “The servers could be in another state,” said Taylor. “It really doesn’t matter to us as long as the server is accessible to the network.”
Using a Christie FRC-5000 virtual display wall controller, operators are able to access and display any application that can typically be accessed on the AmeriCredit data network. Any number of applications can be opened and displayed on the wall at one time with the ability to layer, resize or move images on the screen based on the needs of the operators.
Redundancy and reliability were important issues to AmeriCredit in order to ensure consistent and uninterrupted system operation. The FRC-5000 installed in the NOC provides numerous redundancy and automatic failover features to prevent system interruptions.
All data transfer is managed by the FRC-5000’s Ultra160 SCSI RAID controller. RAID (Redundancy Array of Independent Disks) is a method that involves spreading information across multiple disk drives for greater redundancy, bandwidth and recovery from hard disk crashes. Despite information being shared among multiple drives the computer reads the information as if it is all stored on one large drive.
The Christie FRC-5000 at AmeriCredit utilizes a three-drive configuration with RAID 5, the most common type of RAID. If one drive fails the other two can reconstruct the data on a hot swapped third drive without having to power down the controller. This prevents service interruptions from information being lost.
Another feature of the FRC-5000 is that it comes standard with a four module hot swap power supply. If any of the power supplies fail, an alarm is sounded and the remaining three modules would provide sufficient power to keep the controller running. The faulty power supply can then be hot swapped.
There are also three chassis cooling fans that are constantly monitored. If the speed of one of the fans drops below optimal limits, an alarm goes off and the fan can be hot swapped without having to turn off the controller.
AmeriCredit insisted on having two FRC-5000 controllers though only one is used. The other is redundant to ensure complete reliability of the system.
Budgetary constraints of course played a role in choosing the equipment. The Whitlock Group reused as much of the existing equipment as they could from the original control room, including an Extron Crosspoint 12x8 RGBHV and stereo audio matrix routing switcher, which allows for twelve computer inputs and eight computer outputs.
In addition to being used to display computer graphics from the network, the video wall is also used to watch television channels such as the weather network and CNN in order to monitor events that could impact AmeriCredit’s network and branches across North America.
To monitor world broadcast events, The Whitlock Group installed a Crestron ST-TUNE integrated AM/FM/TV tuner, which provides full control of television tuning functions. There are also JBL LSR-25 powered studio reference monitor speakers that were chosen for their high fidelity sound quality. For presentation purposes and to review broadcast material a JVC SVHS SR-TS1U videocassette recorder and a Sony DVP-NS400D DVD player are installed in the NOC along with two RCA DSS DRD222RD satellite receivers.
The audio/visual equipment is mounted on rack rails that were custom designed and built into the black video wall console by The Whitlock Group. Operators are able to control the equipment via a user interface on a Crestron Pro2 Integrated control system virtual touch panel. This type of remote control was chosen to provide every operator the ability to control the images on the video wall from their desktop.
Steps from the command center is a corporate boardroom. With a large window facing the control room there is an excellent view of the technology driving AmeriCredit. The boardroom is a standalone room and is used regularly for meetings and public tours. The Whitlock Group was responsible for furnishing it as well.
There’s a large oval table in the center of the room where people can watch the activity in the control room. Or they can watch a presentation projected onto a large 60 x 80 inch Da-Lite screen that is electrically retractable into the ceiling. The projector, a Christie LX33, is also mounted on an electrically retractable device so that when not in use it can also be hidden from view in the ceiling.
The boardroom and control room are considered marketing showpieces. Analysts, customers and investors are regularly brought in to see the impressive command center. It’s become a very valuable tool for AmeriCredit in terms of providing a more effective environment to monitor the status of their network and demonstrating that the company is on the leading edge of providing excellent customer service through technological innovation.
“The control room has had a positive impact on our operators being able to do their jobs,” said Whitfield. “Instead of being bound into the system and only being able to view a limited amount of information at a time, the large screen gives them a high level overview of what’s going on. They now have more flexibility to resize and rearrange images based on problems that they’re working on. We’ve also had a lot of tours and everyone’s been very impressed with the control room. It’s definitely a “wow” factor.”