Cloud Relocation takes planning

Planning for a Cloud Relocation

It is easy to create a new environment in a public IaaS cloud to develop or test a new application.   But it takes planning to relocate a set of applications and shutdown of a legacy infrastructure.   It also takes detailed implementation steps.   I plan on discussing those in future articles.   As CTO of dinCloud, Inc., and as a leader of T3 Dynamics, I have found that there is a limited number of IT leaders and practitioners that have a thorough understanding of the components of a cloud relocation.    

When you relocate applications to an IaaS cloud, the result is the equivalent of a data center transition with a partial outsourcing of IT.   So the topics listed below will not be surprising for experienced IT professionals.   Each, however has a different flavor because this transition is not like any other that has been done before.    I will follow-up with more details on each subject.

Cloud Relocation Planning Topics

• Business Justification
• ROI Cloud vendor selection
• Functional and Non-functional Test planning
• Application regression testing
• Performance testing
•  Security testing
• Failover testing
• Network planning
• Wide area network changes
• IP Address planning
• DNS architecture and vendor selection
• Application architecture planning
• Inventory of applications, data flow and service interdependencies
• Application changes (to remove legacy IP addresses) or re-architect for failover
• Server architecture planning (physical if private cloud; sizing if public cloud)
• Data architecture planning
• Data migration or synchronization
• Design for Disaster recovery
• ITIL Processes change planning
• Security change Planning
• Business Continuity / Disaster Recovery planning
• Plan to decommission of legacy servers and network

Amazon EC2 Outage and Cloud Strategy

Last Friday, Amazon experienced a partial outage of its cloud infrastructure.   Here the initial update and the closing updates:


Event Issue

"The problem started with a "networking event" that led to problems with how data is mirrored: We'd like to provide additional color on what were working on right now (please note that we always know more and understand issues better after we fully recover and dive deep into the post mortem). A networking event early this morning triggered a large amount of re-mirroring of EBS [Elastic Block Storage] volumes in US-EAST-1. This re-mirroring created a shortage of capacity in one of the US-EAST-1 Availability Zones, which impacted new EBS volume creation as well as the pace with which we could re-mirror and recover affected EBS volumes. Additionally, one of our internal control planes for EBS has become inundated such that it's difficult to create new EBS volumes and EBS backed instances. We are working as quickly as possible to add capacity to that one Availability Zone to speed up the re-mirroring, and working to restore the control plane issue. We're starting to see progress on these efforts, but are not there yet. We will continue to provide updates when we have them."

Closing update from Amazon:

As we posted last night, EBS (Elastic Block Store) is now operating normally for all APIs and recovered EBS volumes. The vast majority of affected volumes have now been recovered. We’re in the process of contacting a limited number of customers who have EBS volumes that have not yet recovered and will continue to work hard on restoring these remaining volumes…
We are digging deeply into the root causes of this event and will post a detailed post mortem.

One of the unfortunate realities of infrastructure and operations is that the goal will always be 100% uptime for all infrastructures but it cannot be achieved.   The SLAs for infrastructure and operations is very unlikely to be 100%.   The strategic question will always be what SLAs can be afforded, what is the impact to business agility for the target SLAs and what can be improved from a people, process and technology perspective to achieve the business goals and minimize cost. 

Because there are clear ties between performance, availability and security objectives and the success of outsource cloud infrastructure and operations, I believe that public cloud will outperform internal infrastructure over time.   This does not lessen the requirement for internal roles of architecture, end-to-end management of performance, availability and security, and vendor management.   These roles will increase in importance within organizations. 

The current Amazon issue re-emphasizes that a cloud strategy needs to include

• Clear and continuous risk management program for IT
• Enterprise change, incident, problem, release and configuration management process re-engineering
• End-to-end SLA and systems management
• Server provisioning process and technology
• Patching process
• Server configuration baselining and auditing
• Repurposing of servers
• Disaster recovery planning and testing

Private Cloud - why and how?

There is an explosion of change occurring in infrastructure and operations.   While it took almost a decade for virtualization to become main stream, cloud options are evolving much more rapidly.   There are two major business drivers – variable cost for consumers of IT resources and a need for increased IT agility.   All cloud options are built upon shared physical network, virtualized server and storage resources.   Cloud takes virtualization to the next level.   On top of virtualization it layers automated self-provisioning, chargeback for resource utilization, and service level agreements for cloud services that are in the service catalog.

The primary cloud discussions today center on when an enterprise will use public cloud and if it needs to implement private cloud as a stepping stone along the way or as a step-sibling for a longer period of time.   The growth of public cloud is large.   IDC estimates that the total expenditure on public cloud to be $29.5 billion by 2014.   There are some issues affecting the speed of public cloud adoption.   These are compliance concerns, data security, and cost.   As a competitive public utility, cloud cost will eventually go away as a concern.

In the meantime, Gartner believes that the most enterprises over the next couple of years will focus their attention on implementation of private cloud.   Today, there are three options for private cloud.   Enterprises can build their own private cloud (in their data centers or colocation sites), they can contract with a public cloud provider to create a physically separate private cloud for the enterprise (in cloud provider data centers or contracted colocation sites), or they can contract with a managed services provider to manage a private cloud in the enterprise data center.  

Whether an enterprise chooses to move to a public cloud or implement a private cloud, the approach to developing a strategy and implementation plan needs to follow the same methodology.   At a high level, the methodology has a four steps:

• Define an end-state that satisfies business requirements including the financial goals, service goals and resourcing/role goals.
• Identify the transition actions including development of services, financial changes, skill/role changes, ITIL process changes, and infrastructure changes.
• Plan and communicate the individual transition work streams.
• Communicate the overall program frequently and execute.

While the overall transformation creates business value and opportunity, each of the transition actions will create resistance.   Call me if you want to discuss this further.

Planning for Cloud Implementation

We have done a good amount of consulting on moving to public cloud (especially for companies that are not happy about the cost of their existing managed hosting vender).   On the initial discussion, one of the first questions is “what do I need to think about and how do I choose a cloud vendor?”

Moving to a public IaaS cloud vendor and to a lesser extent, a SaaS vendor is a typical data center move or implementation with a few twists and the usual issues that are easily forgotten.   While it is amazing simple and fast to build a new environment in the cloud, caring for it will take some planning, and may require changes to existing technology and processes.   While it may not be as formalized, even small organizations need to think through the issues.   Here is a checklist of items to think about:

Resiliency and Availability

Adding another node to your infrastructure network requires that you think through network configuration and redundancy, as well as server resiliency for servers that are in the new cloud environment.

Data considerations

Do you have constraints because of compliance, performance or support that affects where your data needs to be located.   A compliance requirement may force you to keep data in an internal data center and use it from the cloud.   A performance requirement may suggest a hybrid cloud with database servers in the cloud managed environment and web and application servers in the self-service environment.   Your database vendor or your performance requirements may not support a virtualized database server.

Compliance and Security

Does your IT implementation require that you have an intrusion detection or intrusion prevention system?   Is there a requirement that your infrastructure be located in a SAS-70 certified environment?   Are there requirements in your security policy that require multi-factor authentication?   Will you need to extend your vulnerability and penetration testing activities for the new site?

Identity management

How will enforce the user authentication and control policies for the new environment, e.g., when an employee or consultant leaves the organization?   Will you need to create a new AD domain and build a trust?  

Managing capacity

Monitoring performance and availability

Change, Configuration and Release Management

Will you need to add roles or workflow changes to the change management process?   What changes to you need to make to ensure that your configuration management database is current as you add and remove CIs from the new cloud environment?  Will you need to modify your release management process to push changes to the cloud?

IT service management

If there is a bump in the night, do you need to modify your incident management process to deal with workflow or contacts associated with the new environment?   Are there new services that you need to add to your service catalog to support users of the new environment?

Licensing

Will you need to extend software licensing to cover the new environment from with your vendors or will you acquire licenses through the cloud vendor or SaaS provider?

Testing

If you are moving applications or portions of applications to the new cloud environment, how will you approach functional and performance testing?

Disaster recovery

Will you build a DR site for the cloud implementation?   How will you approach data synchronization?   How does this affect your change, configuration and release management processes?

9 Rules of Crisis Management

I know that this email and blog entry is way too long.   But it is definitely worth reading and I think sort of fun.   So print it out and read part of it now and if you can put it down, part of it tomorrow.   I think you will enjoy it.   Don’t forget to send it to a couple of friends.   Share the fun.

When I was VP of IT at Barnes and Noble.com, we were on the bleeding edge of ecommerce.   We were one of 5 sites that were the biggest, most influential, most complex, fastest moving ecommerce sites on the Internet.   Like the others, we were breaking new ground constantly as we invented how to do everything.   Along with all the excitement came mistakes.   Some of the mistakes turned into crises that impacted the business.   Managing crises became one of my unwritten jobs.   I became a student of crises to learn how to deal with them better.   Here are 9 rules that I learned along the way:

Nine rules for managing crises

1. Train your people to pull the rip-cord early

2. Avoid the cube magnet

3. Always assume there is more than one unrelated problem occurring

4. Ignore the obvious

5. Schedule the fire-fighting effort for effectiveness

6. Help people get off the call

7. Fire the fire enthusiasts

8. Learn from every crisis

9. Engineer your way to sleeping at night

1. Train your people to pull the rip-cord early.    I found it fascinating that the onset of a crisis was naturally ignored by well – everyone.   I think people think maybe it will go away like a cold.   Just leave it alone.   Or maybe they think if I close my eyes the boogeyman can’t be there, or maybe someone else will fix it so I can focus on what is important to me right now.   Or maybe they are afraid that someone will shoot the messenger.   The problem of ignoring an event is that the longer you put off diagnosing the problem, the more noise enters the system from increased usage, planned changes, and unrelated events.   All of this clutter can make it much more difficult to gather the pertinent information to understand what happened to initiate the crisis.   Understanding is the first step to solving a problem.    Employees should be rewarded and encouraged for identifying issues and raising the alarm as soon as possible.   

2. Avoid the cube magnet.   I am sure you have seen this occur.   Shortly after the alarm is sounded, one of the best engineers announces, “I know what the problem is!”.   You see everyone stop their own investigation and start surrounding the engineer’s cube, admiring how fast she can type.   Uh-oh, you find out after a while that perhaps the engineer was wrong or perhaps she discovers some interesting information, but not the root cause of the event.   The result is lost time.   Rather than letting this happen, you should say, “Great, Sue!   Let us know what you found on the conference call, in the meantime, everyone should carry on with their own fact finding.”   

3. Always assume there is more than one unrelated problem occurring.   IT systems naturally grow to become complex systems of systems.   At any moment in time, there are unimportant events occurring that should, when time permits, (ya,right), should be cleaned up in the code or configuration of servers or SAN or network.   Because these are happening all the time, when you start looking for problems, you are guaranteed to find them.   Unfortunately, not necessarily the clues you need to find that relate to the root cause of the current crisis.   How should you deal with these?   Track them all.   You should absolutely look for how spurious metrics may be related to the problem at hand.   But, it is good to train the firefighters to keep an open mind.   Any two events, no matter how much you want them to be correlated, may not be.

4. Ignore the obvious.   “The cause cannot be the fuzzywuz master database.   There is no real-time relationship between that database and the order path”.     “There is no way that the test code could have ended up loaded on that production server.   There is no way that putting in that server name could have ended up pointing to that server”.   Regardless of how we think systems work, we may not completely understand the errors in systems, the mistakes that people make, or the configuration impacts of planned and unplanned changes.   I have always found that the root cause of problems made sense.   Not the sense that experts expect, but still very logical.   Net-net, it is not a personal thing, but crises need to be treated very logically to get resolved.   People, even the smartest ones may be wrong sometimes.   What they think is obvious, may not turn out so obvious.   

5. Schedule the fire-fighting effort for effectiveness.   Ok.  So we have declared a crisis.   Someone opens the “bridge”.   Everyone who could help gets on the call.   This will be effective for at least 30 minutes as data is shared.   Then, when you find that the root cause has not been discovered, it is time for more fact-finding.   In many cases, the best way to manage the diagnosis is a set of waves of sharing information then data collection, then sharing information.   To do this, it is best to plan the waves.   Sometimes it is 45 minute waves, sometimes longer.   This planned fire-fighting approach gives the experts the time that they need to accomplish tasks.   It is effective.

6. Help people get off the call.   The crisis has been going on.   The call has been open for 5 hours.   Everyone who could have offered any help at the beginning voluntarily or not, joined the call.   Now, it is likely that half of them no longer have any value on the call.   They want to get back to work, visit their family or go back to sleep.   To get off the bridge, a) the problem must be solved, b) an individual must be able to prove without a shadow of a doubt that they have nothing to offer, or c) the individual dies.   I have seen engineers feign their own death to get off conference calls.   Rather than forcing them the embarrassment of explaining how they were resuscitated through the wonders of medicine the next day, it is better to actively help people get off the call.   If it is reasonable that their area of expertise will not help, tell them to go to bed.   We know how to find them.

7. Fire the fire enthusiasts.   Every organization has them.   They are the first ones that volunteer to get on the bridge.   They are the only ones that know how System Z works.   They are so fast at fixing the problem when it occurs.   But, interestingly, they don’t ensure that the root cause gets fixed.   They live for crises.   It may seem like they are indispensible.  Temporarily they are.   In the long term, they tend to inhibit your ability to stabilize systems, reduce the mean time between failures, and improve the service levels to your customers.   If it is possible, find a way to give them the religion to improve process, and reduce defects.   But if they are adrenaline addicts, your only choice may be to replace them.   

8. Learn from every crisis.   I love to learn how to avoid pain.  There are many things to learn from crises.   These can include how the systems really work.   What are the thresholds of systems past which they become unstable.   What should be monitored in the future to foresee looming problems and avoid them.   What caused the defect that could be avoided in the development process.   How to improve change and release management.  What tools were most useful to you and which ones weren’t helpful.   How to improve crisis management in your organization.   

9. Engineer your way to sleep at night.   As far as I can tell, 90% of all significant crises start at 1am.   They follow the sun to ensure that it is always 1am wherever the engineers live.   This will ensure that there is no way that twenty people in your organization sleep at night.   It takes effort to fight this law of nature.   It can be done.   We all know how to do this.   We merely implement good quality assurance throughout the application development life cycle, we implement ITIL ITSM to manage change, configuration items, release management and service desk processes.   And we implement end-to-end monitoring to ensure that we learn to eliminate critical alarms and start managing warnings.   

At T3 Dynamics, we are rolling out Monitoring as a Service to complement our professional services of strategy, architecture, implementation and managed services for enterprise monitoring.   We are looking for additional beta testers for a no-charge implementation of monitoring so you too can sleep better at night.   Please let me know if you have an interest.

Top 3 drivers of an application monitoring strategy

There are a plethora of application monitoring tools on the market and a large number of open source solutions to enable monitoring in production.   With all this choice, how does one decide what to implement?   Of course, vendors would have you believe that their solution is the right one in all cases.   Until I join a vendor again :), I will tell you that the right solution for an organization depends on many criteria.  

The top 3 drivers will be

• the near/long term goal for monitoring (what you want to be able to achieve)
• the maturity level of the IT organization (how much you can digest in process enabling technology)
• the existing investment in monitoring tools (whether you just ate lunch and can't justify buying another meal)

Goal
Some IT leaders strongly believe that all you have to do is keep the servers, network and operating systems alive and the applications and end-user experience will follow.   Others believe that you must start by defining Service Level Agreements that translate into measuring end-user experience and then you need to be able to correlate this with infrastucture metrics.  I fall into the second camp, but the right solution for an organization is to the one that will actually be implemented and used.   If there will be no incentive from the top of the organization for measuring end-user experience, then the strategy will drive the solution toward infrastructure only solutions.   If the IT strategy is to progressively improve processes following an ITIL approach, then it will eventually need an integrated Capacity DataBase based on a CMDB and a real time performance management DB.   What is your strategy?

Maturity level
We have been discussing maturity levels of IT organizations for many years.   The Capability Maturity Model (CMM) was first published in 1989.   As it turns out, the first books on ITIL were also published in 1989.   While the two events occuring in the same year is interesting, they were ultimately initiatives to deal with quality and the capability of IT organizations.   The assumption of both initiatives was that until IT organizations were ready to adopt good process, they would not be able to deliver improved quality and predictable throughput.   The ability of an IT organization to effectively implement, care for and use application monitoring, diagnostic drill-down and correlation tools will depend on the level of maturity of the IT organization.  The ability to tightly integrate application monitoring tools and process with ITIL incident and problem management tools and process requires even more maturity.  

Existing investment
As organizations evolve, they always implement some monitoring tools.   Typically, less mature organizations will start by implementing point solutions (database monitoring, network monitoring, server heartbeat monitoring) that are not designed to be integrated.   In many situations, these point solutions will be accompanied by investments in integrated solutions.   I have been involved in many engagements where one of the first goals is to rationalize the investment in tools.   In almost all cases, the right strategy for an IT organization will include some, but definitely not all of the existing investment in monitoring technology.  

Good luck on your monitoring journey and let me know if I can help.