loud operator Joyent went through a major failure on Tuesday when a fat-fingered admin brought down an entire data center’s compute assets.

The cloud provider began reporting “transient availability issues” for its US-East-1 data center at around six-thirty in the evening, East Coast time.

“Due to an operator error, all compute nodes in us-east-1 were simultaneously rebooted,”

The problems were mostly fixed an hour or so later.

The cause of the outage was that an admin was using a tool to remotely update the software on some new servers in Joyent’s data center and, when trying to reboot them, accidentally rebooted all of the servers in the facility.

“The command to reboot the select set of new systems that needed to be updated was mis-typed, and instead specified all servers in the datacenter,” Joyent wrote.

A study recently conducted by Ponemon Institute and sponsored by Emerson Network Power (ENP) shows that on average, an unplanned data center outage costs more than $7,900 per minute. That number is a 41-percent increase over the $5,600-per-minute quantification put on downtime from Ponemon’s similar 2010 study. “Data center downtime proves to remain a costly line item for organizations,” ENP said when announcing the study’s results.

The ARM collective doesn’t just want to get into the data center. It wants to utterly transform it and help companies “manage down the legacy” of existing systems

How a data center is like a disk drive

Masters also said that the shift towards hyperscale computing is forcing the change in the data center. Big apps require massively scalable, cut-down systems where the redundancy is in the software and in the quantity of hardware, not in any particular server that is equipped with all kinds of redundancies, because this cuts overall acquisition and operation costs.

The companies that Red Hat is talking to in the hyperscale data center racket are looking at “fail in place” scenarios in which they treat a data center like a disk drive, and while they admire the compact and tuned nature of SoC server nodes with integrated switch fabrics, they are not even thinking about things at the rack level anymore, but at the data center level.

With a fail-in-place data center, you load it up with a few tens of thousands of nodes that have networking on each server node, pipe in external networking and power, and you never do maintenance on it. If a server fails, you mark it as bad, like a bad block on a disk drive, and you just leave it in there and let the network heal around it.

The warning came through loud and clear in our recent Regcast, Future-Proofing the Data Centre: if you want to build a private cloud, your teams must work together.

That, HP’s David Chalmers told us, means creating a service delivery team: some of you from the server team need to work with a small group from the storage team, and a group from the storage team is going to be hanging out with the networks team, and so on.

For next-generation data centre projects, you need dedicated cross-functional teams with complementary specialisations, aiming for a single service-delivery goal.

The benefits to users of coherent project management may be obvious, but as anyone who has tried to do it knows, it is not easy.

When we build cross-functional teams, establishing a common goal is the easy bit. What follows will determine whether that goal is realised effectively.

The server capacity testing is currently often lacking, since each service will have to draw up its own testing. The test preparation is hard work and requires a lot of expertise. Therefore, the test is often made only to the introduction, in which case merely to ensure that the performance is sufficient.

However, among other things, software updates, and service use changes affect the system’s ability to continue to serve the users. That is why frequently encountered situations where a network is down or the operation has slowed down considerably.

Source: http://m.tietoviikko.fi/Uutiset/Palvelinten+optimointi+s%C3%A4%C3%A4st%C3%A4isi+kymmenen+ydinreaktorin+verran+energiaa

“A thorough infrared scan analysis can indicate hot spots and anomalies in electrical equipment that might compromise data center network reliability caused by high heat.”

Koty then listed the following three ways IR scanning can support data center uptime.

1. Detecting worn bearings, which may indicate above-normal heat in electrical systems
2. Early-stage detection of irregularities in a data center’s support infrastructure
3. Non-invasively detecting problems hidden from the naked eye

CIOs increasingly think that on-premise IT systems are a waste of resources and cloud is the future

According to a new survey from Savvis, three out of five IT and business managers believe owning and operating in-house data centres will drive computing costs upwards and waste resources.

In addition, more than half of all CIOs believe they have wasted money on IT purchases, indeed a whopping 66 percent of US respondents have a purchase they regret making.

The dissatisfaction with in-house IT is leading to an increased take-up in cloud according to the research, which was carried out by Vanson Bourne.

Eighty five percent of companies are now using some form of cloud service.

Most organisations are moving to private cloud – 42 percent, compared to 22 percent of public cloud users.