People, Process, Performance and Capacity

Feel like there is not enough IT to go around?

2011-05-17T13:19:00.000-07:00

The good news is that Spring seems to have sprung for IT. There are signs of activity in many areas, new job openings, technology sales and most importantly - new projects. There appears to be a drive from the business to allow IT to execute on much of the demand that has been on hold for a couple of years. The challenge is that IT has been on a significant diet for the same length of time.

Are you seeing any of these symptoms?

Projects that were expected to start near the beginning of the year, are just being kicked off
Even though development resources levels may be addressed, new projects may be constrained by infrastructure capacity that has resourcing issues and its own ramp up schedule
New projects appear to be affecting each other because they may have been initiated simultaneously and draw on the same pool of resources
You have a feeling that quality may be affected as IT pushes the envelope on risk to achieve delivery dates

If this is affecting your organization, what can you do as an IT leader?

Ensure that the project portfolio maps well to the business strategy – this is not the time to be working on any projects that aren’t the highest business priority.
Mitigate risk by recognizing the issues of resource constraints and interdependencies and focus on 100% execution through good planning – avoid the pressure for ready, fire, aim projects.
Leverage variable capacity opportunities offered through consulting resources and the new secret sauce: public cloud – consider short-term ramp up of consulting for burst capacity, moving more infrastructure resources to outsourced managed services and consider a long term shift to leveraging cloud for infrastructure where your organization can live with concerns for data security.

Taking on these issues now could help you live with the result in 12 months.

2011-04-26T10:53:00.000-07:00

Amazon EC2 Outage and Cloud Strategy

Last Friday, Amazon experienced a partial outage of its cloud infrastructure. Here the initial update and the closing updates:

Event Issue

"The problem started with a "networking event" that led to problems with how data is mirrored: We'd like to provide additional color on what were working on right now (please note that we always know more and understand issues better after we fully recover and dive deep into the post mortem). A networking event early this morning triggered a large amount of re-mirroring of EBS [Elastic Block Storage] volumes in US-EAST-1. This re-mirroring created a shortage of capacity in one of the US-EAST-1 Availability Zones, which impacted new EBS volume creation as well as the pace with which we could re-mirror and recover affected EBS volumes. Additionally, one of our internal control planes for EBS has become inundated such that it's difficult to create new EBS volumes and EBS backed instances. We are working as quickly as possible to add capacity to that one Availability Zone to speed up the re-mirroring, and working to restore the control plane issue. We're starting to see progress on these efforts, but are not there yet. We will continue to provide updates when we have them."

Closing update from Amazon:

As we posted last night, EBS (Elastic Block Store) is now operating normally for all APIs and recovered EBS volumes. The vast majority of affected volumes have now been recovered. We’re in the process of contacting a limited number of customers who have EBS volumes that have not yet recovered and will continue to work hard on restoring these remaining volumes…
We are digging deeply into the root causes of this event and will post a detailed post mortem.

One of the unfortunate realities of infrastructure and operations is that the goal will always be 100% uptime for all infrastructures but it cannot be achieved. The SLAs for infrastructure and operations is very unlikely to be 100%. The strategic question will always be what SLAs can be afforded, what is the impact to business agility for the target SLAs and what can be improved from a people, process and technology perspective to achieve the business goals and minimize cost.

Because there are clear ties between performance, availability and security objectives and the success of outsource cloud infrastructure and operations, I believe that public cloud will outperform internal infrastructure over time. This does not lessen the requirement for internal roles of architecture, end-to-end management of performance, availability and security, and vendor management. These roles will increase in importance within organizations.

The current Amazon issue re-emphasizes that a cloud strategy needs to include

Clear and continuous risk management program for IT
Enterprise change, incident, problem, release and configuration management process re-engineering
End-to-end SLA and systems management
Server provisioning process and technology
Patching process
Server configuration baselining and auditing
Repurposing of servers
Disaster recovery planning and testing

Private Cloud - why and how?

2011-03-22T16:37:00.000-07:00

There is an explosion of change occurring in infrastructure and operations. While it took almost a decade for virtualization to become main stream, cloud options are evolving much more rapidly. There are two major business drivers – variable cost for consumers of IT resources and a need for increased IT agility. All cloud options are built upon shared physical network, virtualized server and storage resources. Cloud takes virtualization to the next level. On top of virtualization it layers automated self-provisioning, chargeback for resource utilization, and service level agreements for cloud services that are in the service catalog.

The primary cloud discussions today center on when an enterprise will use public cloud and if it needs to implement private cloud as a stepping stone along the way or as a step-sibling for a longer period of time. The growth of public cloud is large. IDC estimates that the total expenditure on public cloud to be $29.5 billion by 2014. There are some issues affecting the speed of public cloud adoption. These are compliance concerns, data security, and cost. As a competitive public utility, cloud cost will eventually go away as a concern.

In the meantime, Gartner believes that the most enterprises over the next couple of years will focus their attention on implementation of private cloud. Today, there are three options for private cloud. Enterprises can build their own private cloud (in their data centers or colocation sites), they can contract with a public cloud provider to create a physically separate private cloud for the enterprise (in cloud provider data centers or contracted colocation sites), or they can contract with a managed services provider to manage a private cloud in the enterprise data center.

Whether an enterprise chooses to move to a public cloud or implement a private cloud, the approach to developing a strategy and implementation plan needs to follow the same methodology. At a high level, the methodology has a four steps:

Define an end-state that satisfies business requirements including the financial goals, service goals and resourcing/role goals.
Identify the transition actions including development of services, financial changes, skill/role changes, ITIL process changes, and infrastructure changes.
Plan and communicate the individual transition work streams.
Communicate the overall program frequently and execute.

While the overall transformation creates business value and opportunity, each of the transition actions will create resistance. Call me if you want to discuss this further.

Planning for Cloud Implementation

2011-03-08T14:14:00.000-08:00

We have done a good amount of consulting on moving to public cloud (especially for companies that are not happy about the cost of their existing managed hosting vender). On the initial discussion, one of the first questions is “what do I need to think about and how do I choose a cloud vendor?”

Moving to a public IaaS cloud vendor and to a lesser extent, a SaaS vendor is a typical data center move or implementation with a few twists and the usual issues that are easily forgotten. While it is amazing simple and fast to build a new environment in the cloud, caring for it will take some planning, and may require changes to existing technology and processes. While it may not be as formalized, even small organizations need to think through the issues. Here is a checklist of items to think about:

Resiliency and Availability

Adding another node to your infrastructure network requires that you think through network configuration and redundancy, as well as server resiliency for servers that are in the new cloud environment.

Data considerations

Do you have constraints because of compliance, performance or support that affects where your data needs to be located. A compliance requirement may force you to keep data in an internal data center and use it from the cloud. A performance requirement may suggest a hybrid cloud with database servers in the cloud managed environment and web and application servers in the self-service environment. Your database vendor or your performance requirements may not support a virtualized database server.

Compliance and Security

Does your IT implementation require that you have an intrusion detection or intrusion prevention system? Is there a requirement that your infrastructure be located in a SAS-70 certified environment? Are there requirements in your security policy that require multi-factor authentication? Will you need to extend your vulnerability and penetration testing activities for the new site?

Identity management

How will enforce the user authentication and control policies for the new environment, e.g., when an employee or consultant leaves the organization? Will you need to create a new AD domain and build a trust?

Managing capacity

Monitoring performance and availability

Change, Configuration and Release Management

Will you need to add roles or workflow changes to the change management process? What changes to you need to make to ensure that your configuration management database is current as you add and remove CIs from the new cloud environment? Will you need to modify your release management process to push changes to the cloud?

IT service management

If there is a bump in the night, do you need to modify your incident management process to deal with workflow or contacts associated with the new environment? Are there new services that you need to add to your service catalog to support users of the new environment?

Licensing

Will you need to extend software licensing to cover the new environment from with your vendors or will you acquire licenses through the cloud vendor or SaaS provider?

Testing

If you are moving applications or portions of applications to the new cloud environment, how will you approach functional and performance testing?

Disaster recovery

Will you build a DR site for the cloud implementation? How will you approach data synchronization? How does this affect your change, configuration and release management processes?

Cloud economics may surprise you

2011-02-11T16:16:00.000-08:00

The economics of Cloud may incentivize changes in architecture.   Here is an example:

Many companies aggregate security log files from servers and network devices into a single repository to facilitate alerting on events and to support forensic investigation of security events.   For some organizations, compliance requirements like PCI indirectly make this a requirement (it would be too onerous to satisfy Requirement 11 without implementing a SIEM). There are many Security Incident and Event Management systems (SIEM) that support this.   Security log aggregation can create a large amount of network traffic to the centralized database.

Many cloud providers allow an unlimited amount of inbound network traffic, but charge for outbound network traffic.   This could create a situation for a company, considering all costs, where it is less expensive to place the SIEM and other monitoring infrastructure in the Cloud rather than inside the walls of the organization’s data center.   This may become even more obvious as the company increases the number of servers it puts in the Cloud.

Let me know if you would like help analyzing the cost of cloud for your organization.

Cloud Computing will change IT Organizations

2011-02-01T16:55:00.000-08:00

At its core, cloud computing is outsourced infrastructure or application services. There will continue to be increasing adoption to allow IT to improve service levels, lower costs (if it can dial down services during periods of lower demand) and respond faster to change required by the business. In some cases, especially smaller organizations, a move to outsourced services can ensure that the number of jobs within an IT organization does not need to grow. This is good for business.

In medium and large organizations, the new ala carte menu options provided by various options in cloud computing will create new challenges for the IT organization. This will initiate a shift in job functions. There will be more outsourcing of jobs that are single-focus technical specialists (either to managed services or to services bundled with cloud offerings), but there will also be growth in need for architects, designers, development integrators, security specialists, compliance officers and IT managers of outsourcers within the IT organization. This is also good for the business because the leverage and value for the funded job position grows. IT has always created and lived with change and transformation. The cloud transformation, like all change, creates opportunities and challenges, but from the perspective of jobs, I anticipate that there will be continued net growth because as a whole, IT enables business.

Can you effectively outsource a NOC?

2011-01-15T08:16:00.000-08:00

Over the past 10 years, there has been significant growth of two types of outsourced NOC services – I think of these pure-play infrastructure services and targeted application services. Do they answer the question, “Can you effectively outsource a NOC?” I believe that these service offerings provide an answer topart of the question and at the same time can create a financial obstacle to solving the whole problem. But there is another approach.

Pure-play Infrastructure Services

These services include support for network, server operating systems, management of databases, backups and execution of scheduled tasks. To optimize this service offering, vendors focus on recruiting employees that are knowledgeable in specific horizontal technologies – Cisco, F5, Windows, Linux, Oracle, SQLServer, etc. They implement good ITIL process and supporting technology. What is lacking is an understanding of the applications that deliver the business value. This limits their ability to predict business impact from infrastructure issues and their ability to react to alarms on breaches of transaction and process availability and performance thresholds that directly affect the business.

Targeted Application Services

Outsourced services for SAP, Lawson, Oracle Fusion, Warehouse Management Systems, Content Management Systems, and other similar applications get closer to alignment with business value. However, many organizations have integrated multiple customized off-the-shelf systems and they have some homegrown systems. The challenge of this reality is that the integration and interdependencies of these systems has created a single overall lovable Frankenstein of a system for the organization. Solving the support question of any of the component systems is not sufficient to providing overall coverage for the business.

The financial obstacle of partial outsourcing

Sometimes I think about Pure-play Infrastructure Services and Targeted Application Services as skimming the cream from milk. If one is not trying to solve the whole support problem, it is the least expensive approach to get value for the outsourced dollar. This is why there is a clear business case for outsourcing vendors. They do provide economies of scale for the hired resources. The challenge for the buyer is that the dollars that pay for these services are not available to put into the pot to solve the whole problem.

So, can you effectively outsource a NOC?

Yes. There is an alternative approach that leverages outsourced resources focused vertically rather than horizontally. And, if an organization implements a monitoring system that crosses the infrastructure (data centers and cloud) and includes monitoring of the applications and their interdependencies, you can answer the whole question. A key advantage of implementing the monitoring (and a ticketing system) within the organization is that it provides some additional independence from outsouring vendors.

T3 Dynamics delivers professional services on monitoring and offers a SaaS offering for end-to-end monitoring.

What happens when you run out of Cloud?

2011-01-04T13:29:00.000-08:00

Cloud Computing is generating excitement in IT because it promises to improve responsiveness to the business and reducing the cost of infrastructure during non-peak periods. Some of the excitement comes from an expectation that that cloud capacity is not limited. It is a fallacy. The simple matter is that cloud infrastructure, whether you manage it yourself as a private cloud or it is outsourced in a public cloud is still based on a finite number of servers, a network that requires care when changing and a finite amount of storage at any one time. How do cloud vendors plan on dealing with this? The same way that your infrastructure team would deal with it - monitoring capacity utilization and forecasting the need to expand. There will always be limits to the power, space and possibly network connectivity to data centers, so cloud vendors will need a strategy for managing this also. While cloud provides a great solution to the speed of provisioning and cost reduction through improvements in capacity optimization, it still has some traditional underlying issues of capacity management.

Here are some strategies for managing your risk of running out of Cloud. They are not mutually exclusive

Negotiate to manage risk

Negotiate contracts that either give you guaranteed capacity or early notice of capacity limits. Guaranteeing capacity will require that you be able to forecast how much capacity you will require in the future. This option puts your capacity management gurus in the same situation that they have always been, except they now need to forecast yet another set of resources. Early notice of capacity limits is an alternative, if you can get this agreement from your cloud vendor. The notice period would need to be greater than the length of time it would take to negotiate alternative Cloud resources with your current vendor or another.

Use a 2 Vendor Approach

You could start with a strategy that you plan to have a primary and a secondary vendor for Cloud. You could choose to be your own second vendor as an option. The assumption of this approach is that your two vendors will not run out of capacity at the same time and you can either shift resources to the secondary vendor or you are always managing application capacity across your vendors.

Build for Cloud Flexibility

Build an application/infrastructure strategy that facilitates allocation to alternative Clouds. Being indifferent about where servers or services are to be deployed requires some critical architecture decisions. Some key requirements would be that the application can provide sufficient performance given the range of network latencies that are possible. Another requirement would be that the release management process and toolset supports deployment to alternative clouds.

Like earlier shifts in technology including mini-computers, desktop computers, client-server, and the rise of the Internet, there is no turning back from Cloud Computing. Once you start using it, you will be hooked. Now is a great time to start planning how you will implement it before you end up in a fog (sorry).

T3 Dynamics has been focused on leveraging Cloud Computing for its own business and the challenges of monitoring hybrid environments.

Coach-Referee-Player?

2010-12-07T13:49:00.000-08:00

I referee soccer.    On the surface, the game seems quite simple, but it is actually a dynamic system.   The system has some corollaries for the system underlying an IT organization.   There are three classes of participants in soccer - players, coaches and referees.    Together three roles form a complete system that produces and manages work.   The product of a professional soccer game is enjoyment for spectators.

In an IT organization, we have one team comprised of the development organization and another, the infrastructure and operations organization.   Together, they produce a product that enables the business.   While we all look for ways to ensure that the two IT teams work together, they naturally compete for opportunities to change the production environment, they compete for budget, and they have some conflicting process goals.   Each of the IT teams has players. (Have you ever watched a soccer game of young players where almost all players move together around and with the ball?   Lots of legs trying to kick it all at the same time?   There is an interesting analogy for an immature IT organization and a topic for a future email.)

I think the most fascinating part of an IT organization is how it is organized to support coaching and refereeing.   In most sports, the two roles are thought of as being in conflict.   In soccer, this should not be the case.   It does depend on the maturity of the coaches and referees.   The role of the referee in soccer is to maximize the opportunity for game flow.   The referee can exercise control to do this, but highly ranked and certified referees are taught to exercise control to the minimum extent possible.   Referees are given the latitude to do this through the laws of the game.   In soccer, a foul is only a foul if it is one in the opinion of the referee.    Depending on the maturity and skills of the players, and their desire to keep the game moving, a referee may decide that a foul in one game is not a foul in another.   Increased control is used to offset activities that will by nature decrease flow – time wasting activities, unfair and dangerous acts.   Because the role of the referee is to optimize the throughput of the entire system, I think it naturally becomes the role of the IT leader.   How do you optimize throughput of your organization?

The role of the soccer coach is to optimize the effectiveness of the team through selection of players, and encouragement to improve individual and coordinated team skills.   An IT coach for a development organization should be focused on the “set pieces” that support efficient development -- the application development methodology, documentation standards, etc.   An IT coach for the infrastructure and operations team would focus on ITSM processes.   What set pieces do you use to increase team efficiency?

Unquestionably, the most difficult role in IT would be the leader of a small IT organization.   The game does not change.   There are still requirements for development, operations, security, throughput, data management, player development, financial management, etc.   But in many organizations, this leader must wear the hat of player, coach and referee.   If an IT leader is comfortable and capable of wearing all these hats, they are incredibly unique and likely only fit in small organizations.   The solution for others is to work to move toward the roles of coach or referee, giving up the role of player rapidly and leveraging outsourcing to allow themselves to focus on the management roles.

I look forward to hearing your thoughts on this subject and happy to discuss how T3 Dynamics can help you with outsourcing of enterprise systems monitoring.

9 Rules of Crisis Management

2010-11-30T17:50:00.000-08:00

I know that this email and blog entry is way too long. But it is definitely worth reading and I think sort of fun. So print it out and read part of it now and if you can put it down, part of it tomorrow. I think you will enjoy it. Don’t forget to send it to a couple of friends. Share the fun.

When I was VP of IT at Barnes and Noble.com, we were on the bleeding edge of ecommerce. We were one of 5 sites that were the biggest, most influential, most complex, fastest moving ecommerce sites on the Internet. Like the others, we were breaking new ground constantly as we invented how to do everything. Along with all the excitement came mistakes. Some of the mistakes turned into crises that impacted the business. Managing crises became one of my unwritten jobs. I became a student of crises to learn how to deal with them better. Here are 9 rules that I learned along the way:

Nine rules for managing crises

1. Train your people to pull the rip-cord early

2. Avoid the cube magnet

3. Always assume there is more than one unrelated problem occurring

4. Ignore the obvious

5. Schedule the fire-fighting effort for effectiveness

6. Help people get off the call

7. Fire the fire enthusiasts

8. Learn from every crisis

9. Engineer your way to sleeping at night

1. Train your people to pull the rip-cord early. I found it fascinating that the onset of a crisis was naturally ignored by well – everyone. I think people think maybe it will go away like a cold. Just leave it alone. Or maybe they think if I close my eyes the boogeyman can’t be there, or maybe someone else will fix it so I can focus on what is important to me right now. Or maybe they are afraid that someone will shoot the messenger. The problem of ignoring an event is that the longer you put off diagnosing the problem, the more noise enters the system from increased usage, planned changes, and unrelated events. All of this clutter can make it much more difficult to gather the pertinent information to understand what happened to initiate the crisis. Understanding is the first step to solving a problem. Employees should be rewarded and encouraged for identifying issues and raising the alarm as soon as possible.

2. Avoid the cube magnet. I am sure you have seen this occur. Shortly after the alarm is sounded, one of the best engineers announces, “I know what the problem is!”. You see everyone stop their own investigation and start surrounding the engineer’s cube, admiring how fast she can type. Uh-oh, you find out after a while that perhaps the engineer was wrong or perhaps she discovers some interesting information, but not the root cause of the event. The result is lost time. Rather than letting this happen, you should say, “Great, Sue! Let us know what you found on the conference call, in the meantime, everyone should carry on with their own fact finding.”

3. Always assume there is more than one unrelated problem occurring. IT systems naturally grow to become complex systems of systems. At any moment in time, there are unimportant events occurring that should, when time permits, (ya,right), should be cleaned up in the code or configuration of servers or SAN or network. Because these are happening all the time, when you start looking for problems, you are guaranteed to find them. Unfortunately, not necessarily the clues you need to find that relate to the root cause of the current crisis. How should you deal with these? Track them all. You should absolutely look for how spurious metrics may be related to the problem at hand. But, it is good to train the firefighters to keep an open mind. Any two events, no matter how much you want them to be correlated, may not be.

4. Ignore the obvious. “The cause cannot be the fuzzywuz master database. There is no real-time relationship between that database and the order path”. “There is no way that the test code could have ended up loaded on that production server. There is no way that putting in that server name could have ended up pointing to that server”. Regardless of how we think systems work, we may not completely understand the errors in systems, the mistakes that people make, or the configuration impacts of planned and unplanned changes. I have always found that the root cause of problems made sense. Not the sense that experts expect, but still very logical. Net-net, it is not a personal thing, but crises need to be treated very logically to get resolved. People, even the smartest ones may be wrong sometimes. What they think is obvious, may not turn out so obvious.

5. Schedule the fire-fighting effort for effectiveness. Ok. So we have declared a crisis. Someone opens the “bridge”. Everyone who could help gets on the call. This will be effective for at least 30 minutes as data is shared. Then, when you find that the root cause has not been discovered, it is time for more fact-finding. In many cases, the best way to manage the diagnosis is a set of waves of sharing information then data collection, then sharing information. To do this, it is best to plan the waves. Sometimes it is 45 minute waves, sometimes longer. This planned fire-fighting approach gives the experts the time that they need to accomplish tasks. It is effective.

6. Help people get off the call. The crisis has been going on. The call has been open for 5 hours. Everyone who could have offered any help at the beginning voluntarily or not, joined the call. Now, it is likely that half of them no longer have any value on the call. They want to get back to work, visit their family or go back to sleep. To get off the bridge, a) the problem must be solved, b) an individual must be able to prove without a shadow of a doubt that they have nothing to offer, or c) the individual dies. I have seen engineers feign their own death to get off conference calls. Rather than forcing them the embarrassment of explaining how they were resuscitated through the wonders of medicine the next day, it is better to actively help people get off the call. If it is reasonable that their area of expertise will not help, tell them to go to bed. We know how to find them.

7. Fire the fire enthusiasts. Every organization has them. They are the first ones that volunteer to get on the bridge. They are the only ones that know how System Z works. They are so fast at fixing the problem when it occurs. But, interestingly, they don’t ensure that the root cause gets fixed. They live for crises. It may seem like they are indispensible. Temporarily they are. In the long term, they tend to inhibit your ability to stabilize systems, reduce the mean time between failures, and improve the service levels to your customers. If it is possible, find a way to give them the religion to improve process, and reduce defects. But if they are adrenaline addicts, your only choice may be to replace them.

8. Learn from every crisis. I love to learn how to avoid pain. There are many things to learn from crises. These can include how the systems really work. What are the thresholds of systems past which they become unstable. What should be monitored in the future to foresee looming problems and avoid them. What caused the defect that could be avoided in the development process. How to improve change and release management. What tools were most useful to you and which ones weren’t helpful. How to improve crisis management in your organization.

9. Engineer your way to sleep at night. As far as I can tell, 90% of all significant crises start at 1am. They follow the sun to ensure that it is always 1am wherever the engineers live. This will ensure that there is no way that twenty people in your organization sleep at night. It takes effort to fight this law of nature. It can be done. We all know how to do this. We merely implement good quality assurance throughout the application development life cycle, we implement ITIL ITSM to manage change, configuration items, release management and service desk processes. And we implement end-to-end monitoring to ensure that we learn to eliminate critical alarms and start managing warnings.

At T3 Dynamics, we are rolling out Monitoring as a Service to complement our professional services of strategy, architecture, implementation and managed services for enterprise monitoring. We are looking for additional beta testers for a no-charge implementation of monitoring so you too can sleep better at night. Please let me know if you have an interest.

Would you send your mother to a generalist for open heart surgery?

2010-11-23T10:04:00.000-08:00

I was speaking with a colleague last week who had recently lost a valued employee that managed the company’s SANs. He felt exposed. This wasn’t the first time he had lost employees who were specialists. He was thinking that the solution was to hire only generalists and give them on the job training on the many disciplines that one must manage in an IT environment – network, security, SAN, virtualization, Windows admin, Linux admin, monitoring, some DBA skills, scripting, and each of the flavors of vendor solutions of each of these disciplines.

Cost and transiency

The core of the issue that he faced is that a specialist may be the one that will save your mother’s life if she needs a heart valve replacement, but specialists they cost more than general practitioners. And finding the right one that wants to live or work in your location may be difficult. What’s more, many only want to do open heart surgery and if you want them to treat your mother’s arthritis, they may look for another job.

Tactical vs. Big bang outsourcing to the rescue?

While we may all need medical specialists from time to time, we do not consider hiring one as a full time employee. We outsource. Since we may not need a continuous relationship with a specific medical specialist, we don’t often interview his/her partners when we are looking for one. This is tactical outsourcing. It makes sense for one-off needs. Hopefully, your mother doesn’t need more than one heart valve replacement. In California, where we have Kaiser Permanente, we can consider outsourcing all of our medical needs to a single organization. There are options to do this in IT also. This Big Bang outsourcing is a big decision with big risk and significant transition requirements.

The targeted managed service option

The Internet allows diligent workers or service providers with good process and tools to work anywhere at any time for anyone. I can recommend great managed service providers for DBA services, for JD Edwards CNC services etc. I think of this as targeted managed services. The benefit of this approach is that one can get a guaranteed supply of specialized resource that is shared with other customers at a cost that can be less than a full time employee.

At T3 Dynamics, we are launching a new SaaS and managed service offering for enterprise end-to-end monitoring. Please let me know if you want to join the no-charge beta offering.

Implementing virtualization without monitoring?

2010-09-11T18:20:00.000-07:00

We often find at T3 Dynamics that clients are rapidly moving toward more virtualization, private clouds and public clouds. Everyone is doing virtualization with the battle in full swing between VMware and Hyper-V for servers and XenApp/XenDesktop for the desktop. With the current burst in capacity and density of server hardware, this is no surprise. What is surprising is how many of them are rapidly moving toward virtualization sprawl without addressing how they are going to manage the growth and how they are going to ensure performance and availability service levels to the business. We have been doing implementation of HP/Mercury monitoring/BSM and Microsoft SCOM implementation for years. It is possible to build a strategy for monitoring that is dynamic and fits within the ROI requirements of a company with proven technologies that enable ITIL ITSM.

Paying for too many drinks?

2010-08-26T10:00:00.000-07:00

A friend and colleague, Sasha Gilenson is president of Evolven. Evolven has a product that takes detailed snapshots of server configuration across the enterprise that can be used to investigate and diagnose change. It can be used for daily audits of infrastructure to support stability and compliance. I was speaking with him today about the tool and I was thinking that it also may be very useful to take daily snapshots of cloud consumption to audit the financial statements of cloud vendors, i.e. Amazon or Rackspace. Here is the company website http://www.evolven.com/. Do you think about this?

The future of the data center professional

2010-06-01T17:55:00.000-07:00

I believe we are moving toward outsourcing of infrastructure vis a vis IaaS clouds including outsourced private clouds. There are larger roles available in cloud utility providers. At the same time, internal organizations will need to strengthen their architect/outsourcing manager roles.

The new skills include more knowledge of various virtualization technologies, knowledge of APIs of specific clouds, negotiation skills with cloud providers, x-cloud architects, heterogenous infrastructure application/systems management knowledge and skills, new disaster recovery architects, and more specific consulting skills aligned with cloud independent managed service providers.

There is a bold new world that is rapidly evolving for data center professionals. The biggest concern might be being left behind.

The technological clock is lapping itself in our generation

2010-04-09T13:07:00.000-07:00

Adoption cannot keep pace.   Have you noticed that the youngest generation thinks email is passe? For them, email as a technology does not satisfy their needs most of the time. They know how to use email and are not afraid of it.   But, email is not instantaneous. The next generation (is it generation Z?) has grown up with SMS on their phones, video calling through Skype, chatting through Facebook (AOL IM also appears passe), and communicating through their headsets on XBox.   Amazingly, dialing a telephone may be passe also.   When my son wants to talk to friends, the first thing he will do is put on his Xbox headset because he thinks many of the friends are there and immediately available.

What is most amazing about the situation is not that the next generation is adopting new technology.   It is that they are skipping a technology that has yet to be adopted by the oldest generation.   It is almost like we can see time lapping itself in our generation. In the past, (think cars, radios, TVs, cell phones), the adoption of technology may have started in a younger generation, but the wave would eventually reach the oldest generation while the youngest was still using it.   I can remember the first remote control TV that my grand parents got.   At that time everyone was adopting remote controls.   Even in recent times, we have seen all generations adopt mobile phones.

I think the first discontinuity of technology adoption may be email, but I can see that there will be others in the future.

Data Center Consolidation

2010-03-24T18:33:00.001-07:00

Here is a slide presentation that I delivered on a webinar with BDNA today.

Data Center Consolidation

View more presentations from Barry Weber.

People are analog, not digital

2010-02-25T14:05:00.000-08:00

Most individuals in IT get a sense of comfort from the predictability of IT applications and systems. They know that even if there are configuration problems or defects in applications, these will also be predictable. Once finding that an event occurs, it is just a matter of following a methodical process to identify the root cause. While root causes can be obscure, they are ultimately logical outcomes.

People on the other hand, do not deliver discrete predictable outcomes. At times they can disappoint, but at other times, they can deliver ingenuity, innovation, unexpected effort, passion, love and impact far beyond the expected. This ability of people to deliver results and surprises is the wonder of who we are. Most executives and leaders recognize this about people. The surprising thing is that these days, in the hiring process, we seem to be operating in a way that assumes that people are as predictable and limited as IT applications and systems. Today, when there are job openings, those that are looking for talent seem to be consumed with

Finding the candidate that matches the job description regardless if an individual matching all components of the job description has ever been seen in the wild
Assuming that individual capabilities are as discrete as the functions built into applications

Part of the reason for this new behavior in hiring is that technology has ensured that a mountain of resumes can now be delivered for every advertised opening. Our ability to convey information about the opening is as rapid and expansive as a pandemic. The fallacy in the current situation is that building the mountain will not necessarily ensure that a resume exists in the stack that matches the stated requirements or that an individual who has applied has accurately conveyed what they can deliver. We will find that there will be a number of changes in the future.

We will deploy more technology to identify meaning and content in the mountain of resumes (like semantic search)
Some who are looking for talent will stop trying to advertise as widely to avoid the effort and cost of consuming the mountain
There will be more emphasis on leveraging technogy, like LinkedIn that exposes relationships to allow us to identify the analog target (a person) with a technologically enabled analog process.

Walmart takes on cable and satellite

2010-02-23T11:25:00.000-08:00

Internet streaming of movies to DVR is not new. It has not reached most households yet. But it was a wave that was slowly building. Until today. The announcement that Walmart has acquired VUDU significantly changes the balance in the market. Perhaps your new entertainment provider will be Walmart in the future.

The 3 foundation measurements of IT

2010-02-21T10:49:00.000-08:00

There are 3 foundation measurements of IT. These are the indicators of successful delivery to the business.

Requirements delivered for changes to application functionality. This can be measured as internal or external customer satisfaction, revenue growth attributed to application changes, function points, user stories, and use cases delivered.
Service level achievement. This can be measured as achieving Service Level Agreements and the metrics will typically be in application performance, application availability, and time to close issues.
Cost. Coupled with the first or second bullet point, it would be measured as ROI. Otherwise, it is typically a historically trended set of numbers.

The concepts are not complicated. Execution is the challenge.

The gap between being rational and being informed

2010-02-19T14:06:00.000-08:00

I was talking to a trusted colleague yesterday.   I was explaining that in almost all cases, I find IT people that would agree that systems should be high quality, highly available, and deliver the performance that the business expects.   If there was an event, they would want to be able to know that it was coming and rapidly diagnose the root cause from information provided as part of end-to-end application/system monitoring and configuration management information.

Organizations are filled with intelligent, talented, rational IT people.   Yet, when they implement systems, there are often limitations to the above stated goals.

I think there are two root causes for the disconnect between the goals and reality.   One of the root causes is hard to fix and the other is easy.   The hard to fix root cause is that we implement complex systems in complex ecosystem.   Systems are getting more complex with more moving parts as we move toward SOA, more vitualization, clouds, and more non-functional requirements in areas like privacy and security.   This is coupled with new and improved budget constraints and the age-old alignment questions between IT and the business.

The easy to fix root cause is that often, limitations in IT's ability to manage systems is not communicated.
I have seen many organizations that have invested in excellent application and infrastructure management tools and rapidly implement a portion of what they need. Unfortunately this is coupled with limited investment in internal transfer of knowledge on how to care and feed management systems.   And the release management process is not modified to ensure that changes to applications are coupled with changes to monitoring and management.   Often, those individuals that do know how to make changes move on and the internal knowledge gets diluted further.   As the visibility to manage applications decreases, this is not communicated. The result is that those that invested in tools to support the goals listed above do not know that things have changed.   They remain intelligent, rational IT professions. But now some of their decisions become impacted by an information gap. This is the gap between being rational and being informed.

Its the application, stupid - revisited

2010-02-07T14:55:00.000-08:00

In his blog story with the title "Its The Application, Stupid", Daniel Kushner explains that cloud computing brings exponential complexity with the touch of a button. He is right. Given that I know and trust at least one investor in Nolio, I am certain that the company is on track to help reduce the complexity of the release management task of "Application Service Automation".

Even with the best software release management tool, engineers will still need to define and test the application package. I believe that integration of application management with the change and release management processes is essential to improving service levels. In practice, this means that as applications are moved to QA, application monitoring changes also need to be tested in QA. As application changes are moved to production, application monitoring changes also need to move to production. Many might read Kushner's article and agree that automation of application changes will become more and more important as we move toward cloud computing. I agree. I also believe that automation of these changes need to include automation of the application management changes; and the integration of the application management changes with application changes will be just as important.

Top 3 drivers of an application monitoring strategy

2010-01-04T18:29:00.000-08:00

There are a plethora of application monitoring tools on the market and a large number of open source solutions to enable monitoring in production. With all this choice, how does one decide what to implement? Of course, vendors would have you believe that their solution is the right one in all cases. Until I join a vendor again :), I will tell you that the right solution for an organization depends on many criteria.

The top 3 drivers will be

the near/long term goal for monitoring (what you want to be able to achieve)
the maturity level of the IT organization (how much you can digest in process enabling technology)
the existing investment in monitoring tools (whether you just ate lunch and can't justify buying another meal)

Goal
Some IT leaders strongly believe that all you have to do is keep the servers, network and operating systems alive and the applications and end-user experience will follow.   Others believe that you must start by defining Service Level Agreements that translate into measuring end-user experience and then you need to be able to correlate this with infrastucture metrics. I fall into the second camp, but the right solution for an organization is to the one that will actually be implemented and used.   If there will be no incentive from the top of the organization for measuring end-user experience, then the strategy will drive the solution toward infrastructure only solutions.   If the IT strategy is to progressively improve processes following an ITIL approach, then it will eventually need an integrated Capacity DataBase based on a CMDB and a real time performance management DB.   What is your strategy?

Maturity level
We have been discussing maturity levels of IT organizations for many years.   The Capability Maturity Model (CMM) was first published in 1989.   As it turns out, the first books on ITIL were also published in 1989.   While the two events occuring in the same year is interesting, they were ultimately initiatives to deal with quality and the capability of IT organizations.   The assumption of both initiatives was that until IT organizations were ready to adopt good process, they would not be able to deliver improved quality and predictable throughput.   The ability of an IT organization to effectively implement, care for and use application monitoring, diagnostic drill-down and correlation tools will depend on the level of maturity of the IT organization. The ability to tightly integrate application monitoring tools and process with ITIL incident and problem management tools and process requires even more maturity.

Existing investment
As organizations evolve, they always implement some monitoring tools.   Typically, less mature organizations will start by implementing point solutions (database monitoring, network monitoring, server heartbeat monitoring) that are not designed to be integrated.   In many situations, these point solutions will be accompanied by investments in integrated solutions.   I have been involved in many engagements where one of the first goals is to rationalize the investment in tools.   In almost all cases, the right strategy for an IT organization will include some, but definitely not all of the existing investment in monitoring technology.

Good luck on your monitoring journey and let me know if I can help.

The promise of virtualization and cloud computing brings challenges

2009-11-13T12:02:00.000-08:00

I find that like most new technologies that offer great promise, the gotchas of virtualization and cloud computing rest in people issues. There are two basic promises of virtualization and IaaS cloud computing.

Virtualization - We can squeeze more out of underutilized CapEx in hardware by leveraging the ability to oversubscribe the resources across multiple virtual servers
Infrastructure as a Service Cloud Computing - We can automate the provisioning and de-provisioning of virtual instances to fine tune the utilization.

Before you read on, please understand that I think that virtualization and cloud computing are incredibly important to the next phase of evolution for IT enablement of business. I believe the promises are real and important.

So where are the people gotchas? As a discipline, experienced professional IT leaders recognize that it is not only admirable, but required to take on the Sisyphean challenge of implementing standards and process. Standards and process are the only consistently proven approaches to increase predictability of effort and systems, reduce defects, and the resulting long-term cost of ownership. There are three pressures that impede the implementation of standards and processes. a) It is often difficult to show immediate value to the business (a people/strategy sale/trust issue); b) Most people in the IT organization don't think of implementation of standards and process as the "fun" stuff (another people issue); c) Progress toward implementation of standards and process is driven by indivituals and progress is affected by the natural changes in career and life paths of these individuals.

So what has this to do with Virtualization and Cloud Computing? I believe that virtualization and IaaS cloud computing will tend to act as the grease for all of the counter pressures to implementing standards and processes. The ability to rapidly solve a business problem by adding another server or 10 or 800 in minutes or hours without spending capital will drive people to make changes, just this one time, without implementing or following new standards and processes. The personal challenge of tackling the increased complexity offered by virtualized servers, virtualized storage and virtualized switches will drive some engineers to relax the fight against entropy. There are probably 10x more career paths in IT that will come from virtualization and cloud computing. Who would ever thought that a specialization on how to implement virtualized iSCSI to support a VMWare ESXi cluster would have existed even 5 years ago? So people in IT will move for good personal reasons toward new opportunities and away from tuning the machine.

So it is about people. I believe we have the equivalent of a new atomic energy in IT coming from virtualization and cloud computing. This is a great thing that we will eventually harness the old way, by thinking about how we manage and motivate people. Understanding will come from experience and pain.

Google Wave will have an impact

2009-10-09T16:04:00.000-07:00

I just finished reviewing the long video of Google Wave. I was an early adopter of Lotus Notes. I recognized back in the early 90's that collaborative tools would become incredibly valuable as we needed to increase the productivity of knowledge workers and support anytime, any place work. There were a couple of core problems with Notes that the industry has not yet addressed. These centered on the requirement for multiple containers for collaboration and the limitations on real-time collaboration. I believe that Google with Wave is finally addressing these issues and this will be a game changing technology.

Capacity management depends on good asset management

2009-10-05T15:36:00.000-07:00

In many cases, capacity management needs to start with asset management. How does one know where the bottlenecks are unless one knows what can be bottlenecked? Availability will depend on capacity management. If you run out of SAN space (storage) associated with a server, availability will be impacted. Capacity management is dependent on asset management. To be able to measure the existing capacity and forecast it, you will need to know which applications use specific network, server and software assets. To be able to map applications to assets, one needs to know what the inventory is.

Good asset management can also ensure that the right licensing costs are being paid, whether there are any security holes from un-patched software, and whether there are assets that exist and may be under-utilized.

Want to know a fast way to get an accurate asset inventory? Visit the T3 Dynamics website and drop us a line - http://www.T3Dynamics.com

Business issues and APM activities

2009-09-29T09:18:00.001-07:00

Application performance management addresses 5 core issues. The activities of APM cross the ITIL service lifecycle stages. They become sub-components of strategy, design, transition, operations and continual improvement.

The importance of selecting and managing the right people

2009-09-10T08:36:00.003-07:00

I attended an excellent CIO networking event sponsored by Q-Connect last night. There were two speakers, Monte Gibbs and Joe Manna. Monte is an interim CTO, Web Technical Architect & Online Media Strategist. Monte spoke on the expectation that IT would deliver innovation to the the organization. Joe is CIO of Live Nation and COO of their ticketing operation. He spoke on the spinoff from Clear Channel and the start-up and maturation phases of Live Nation including 5-6 acquisitions, implementation of Oracle financials, and the launch of the ticketing app.

The discussion among the CIO/CTOs focused on the issues of how to address the expectation of innovation while keeping 20 other balls in the air (my interpretation) and how to create an environment that delivers continuous innovation in response to Monte's talk. After Joe's briefing, the discussion focused on how to transition managers from doers to delegators to support sustainable and predictable service levels for the IT organization.

In my mind, the common theme across both discussions was that a core solution to both issues has to do with selecting and managing the right people in the IT organization. With the right people and an environment that fosters innovation it is achievable. Excellent individual contributors cannot necessarily become great managers. If it were possible to prioritize, it is highly advisable that the CIO focus first on people, then enabling process and finally on technology. Most IT personnel are hired first for their technical prowess and it is assumed that they may not have the best people orientation. In selecting individuals for manager roles, we need to ensure that they have the capability to transition to a people focus as a primary orientation if we are to be able to groom these individuals to eventually replace us, the CIOs.

Funny Google Voice speech to text messages

2009-09-03T14:33:00.001-07:00

I use Google Voice and I am very pleased with the functionality it provides. The Voice mail speech to text function is one that has lots of opportunity for improvement. Here is a humorous sample of one of the messages they left for me:

Hi Brandy, This is Morgan Reed in for Glen. I'm calling from India. I just want to kind, almost, could you on 45th to. I hope you Oshkosh send me an email is for Becker timing. I'm okay. Beat anytime England updates, please conform your timing so that I can about to go out and talk and I look forward to speak to you. Thank you very much. Have a wonderful day.

Imagine all your users were retail customers

2009-08-24T15:26:00.004-07:00

Actually, they are. And they expect you to manage your internal systems as if they are customers of every other retail site they visit. They may not be able to go to another competitor for the information that is on your intranet or provided by internal applications, but you can bet they wish they could if the availability or performance is not great. So what would you do if the internal systems were retail systems.

You would monitor the end-user experience for health and performance
You would communicate clearly and with advanced notice if there were planned maintenance periods or surprise outages
You would make the look and feel of internal sites attractive and highly usable
You would find ways to minimize the mean time to repair when there are issues through integrated monitoring and training on how to diagnose issues

Performance management in the cloud

2009-08-20T11:29:00.005-07:00

Researchers from the University of New South Wales report that performance can vary by 20x. http://www.itnews.com.au/News/153451,stress-tests-rain-on-amazons-cloud.aspx

It would be good to see the details on the tests. But at a high level the article makes sense. All clouds are based on an over-subscription model. There are more virtual servers on host machines than equivalent hardware. The theory is that most hardware is utilized at the 10% level and there is excess capacity available. There is no guarantee that all the equivalent hardware that you want is available when you want it in any virtualized environment including the Amazon, Microsoft or Google clouds unless they are being managed.

One question is whether the 20x difference is significant, e.g., if response time varies from .5 seconds to 10 seconds. This may not be important to the business and the application that is placed in the cloud.

The article also says that that the clouds are not offering the tools necessary to manage service levels. In many cases, I believe that when one buys infrastructure as a service, e.g., Amazon EC2 and RackSpace, one also has to deploy application management tools in the cloud to manage availability, capacity and performance service levels. This is the same requirement that one has if one deploys real metal or virtual machines in a private environment. It will be interesting to see what Microsoft and Google offer for management tools for their web services in the cloud offerings.

Business Impact of good website and infrastructure monitoring

2009-08-02T07:40:00.002-07:00

There is a direct relationship between website availability, customer satisfaction and revenue.

In my roles leading infrastructure for Barnes & Noble.com (www.bn.com) and as CIO at Lieberman Research Worldwide, I have worked with the HP/Mercury Interactive Business Availability Center and Microsoft's System Center Operations Manager (SCOM) to get real-time feedback on customer online experience and correlate issues to underlying infrastructure and application architecture. My interest first began in the early days of the Internet when customers of Barnes & Noble.com stopped calling customer service if there was a website issue, they just clicked www.amazon.com.

There is a direct relationship between well architected monitoring and minimizing mean time to repair (MTTR).

It is not sufficient to implement monitoring to see system alerts or to monitor components of the infrastructure independently (network, databases, applications). It is common practice in less mature organizations for developers to send emails to themselves as the sole alerting mechanism if there are application errors. The challenge of independent monitoring approaches is that there may not be sufficient information to understand what a problem is or to prioritize issues with respect to business impact. IT energy is saved, problems are solved faster, and monitoring is not "over-implemented" if the architecture is thought out and the process of updating monitoring is integrated with the SDLC. What is your experience?

Balancing cost and risk of implementing APM

2009-07-27T10:05:00.003-07:00

IT leaders are consistently focused on decisions on the balance of cost and risk. The result of this balancing act on the application performance life cycle impact decisions on:

Do we do performance testing for the next release of an application set?
How big should my performance test environment be?
If my performance test environment is not the same size as production, will guesses on capacity and performance be accurate?
Do I leverage performance modeling as part of my APM life cycle?
Do we choose to implement open source server monitoring solutions or do I need a solution that is more robust? How much am I willing to spend?
When there is a critical event, can my team find the root cause quickly?
How much of the time of my engineers needs to be spent on trouble-shooting?

I find that having a strategy for the Application Performance Management Life Cycle helps the IT executive feel comfortable that the answers are optimized.

IT/Business Alignment analysis

2009-06-14T12:32:00.002-07:00

There is a lot written on alignment of IT with the business. I think alignment and governance start with understanding three things:

· The gap between the business need (new functionality and risk mitigation requirements) and the current state of IT value

· The flexibility of the IT resources to deliver on business value (skills, process, friction caused by effort to maintain existing systems)

· A prioritization of initiatives based on the need, ROI and budget

Single Sign On implementation

2009-05-29T11:06:00.002-07:00

Single sign implementation across companies has come a long way in the past few years. With the advent of the SAML protocol and various off the shelf implementations of service provider and identity provider code, it is now possible to implemented a federated SSO implementation in a short time with very low cost. I think a lot about how technology is evolving to provide more functionality and complexity without necessarily improving the quality of life. This evolution of single sign-on is a great one that makes life easier for end-users.

Four key issues of living with the "cloud"

2009-05-10T18:45:00.004-07:00

A review of the industry posts and my own view of living with the "cloud" indicates that we will need to address 4 key issues:

Security of data
Cost of network getting to the cloud
Managing capacity/performance
Monitoring for availability and diagnosing issues

I believe that with standard metrics of CPU, internal messaging and I/O, we can forecast the required capacity and associated performance by leveraging performance testing and capacity modeling in the lab. I have used a similar approach to defining capacity for real metal environments and I believe it provides reasonable estimates for private clouds. Let me know if you have an interest in discussing this.

Monitoring availability and diagnosing issues can be addressed with colocation of a subset of the production environment that is privately managed. Alternatively, if you are using Gigaspaces, Dynatrace has a product offering that they indicate allows you to view utilization of your Gigaspaces XAP implementation on Amazon's EC2. You would still need to manually correlate the end-user experience with the utilization shown in Dynatrace. See http://blog.dynatrace.com/2009/05/07/proof-of-concept-dynatrace-provides-cloud-service-monitoring-and-root-cause-analysis-for-gigaspaces/.

Cost of network getting to the cloud can be addressed in many ways and measured accurately.

I believe that security of data will be addressed with defined process and auditing of cloud providers in the same way that we address it today with other outsourced services, e.g., a SAS-70 audit.

2009-05-07T10:30:00.003-07:00

I have been focusing on the "cloud" a lot lately. I know most CIOs are. There are some basic misunderstandings about the cloud.

Cloud Misunderstandings

Misunderstanding - The cloud is the same as SaaS. Reality - SaaS is about shared application space. It is likely that an SaaS provider may be considering the cloud for hosting. This would give them the same basic benefits of the cloud - optimized cost/availability/capacity infrastructure. The cloud provides capacity aligned with utilization without a need to build out CapEX for burst capacity. The cloud is useful for custom application owners/developers.
Misunderstanding - The cloud has different management questions. Reality - The same questions apply that have pervaded IT since the 50's. a) What is my required capacity going to cost and how do I plan, monitor, and manage for this? b) How do I predict, monitor and manage performance and availability?
Misunderstanding - I don't need any production environment, I can move everything to the third party managed cloud. Reality - It is likely that you will need some production environment. Here are some questions. Can you place all of your data in the cloud or will you need to leave some of it near other applications? How much data needs to be transferred to and from the cloud? Do you need a PCI certified environment? Are you satisfied with the load balancing options available to you in the cloud? Each application moved to the cloud requires an individual assessment for the cost savings and functional requirements.

Managing entropy in IT

2009-04-29T10:38:00.002-07:00

I had dinner with Tony Scott, CIO of Microsoft on Friday night. We had a side conversation about how important it is to control change in applications, services and process. My experience is that this is one of the most significant areas that CIOs must focus. I believe the right definition is really about managing entropy. There is a natural tendency in IT to evolve to chaos. Not out of bad intention, but often out of expediency. Tony said that Steve Balmer thinks the same thing and often describes himselve as the Chief Simplification Officer.

2009-03-27T19:21:00.001-07:00

I recently put together an approach to really answer the questions:

1. Can my application scale
2. Where are the application bottlenecks
3. What size environment do I need for a target load

I did this because I needed to answer these questions quickly without spending a lot of money for my company. To make the challenge more fun I did not have a large test environment, did not have the skills for performance testing, did not have the skills for modeling and did not have the licenses for either. I also found a way to do this and deliver ongoing performance tuning, testing and modeling services for my company leveraging the initial effort and keeping all those ridiculous assumptions (limited test environment, no skills, no licenses). Interested?

Reducing time and CapEx in performance tuning, testing and capacity planning

2009-03-08T11:13:00.002-07:00

We needed to answer questions on the capacity or footprint required to support more than 1000 concurrent users of our client portal website. The question was "how can this be done with mimimal capex, the lowest cost of performance test resources and minimal time?". To solve the problem, I organized a group of services including performance testing service using HP LoadRunner and the Hyperformix discrete simulation capacity planning tool. The services included a bundling of the LoadRunner licenses, the use of existing injectors and controller over the Internet and a process of gathering the data to ensure that we had what we needed for the capacity model. In a very short period of time, we had completed a Deep Diagnostic run of the application and could see the call tree with costs associated with each step. We found that there were performance bottlenecks in the application code that were removed to optimize the data for the capacity model and then built the capacity model. It was fast, efficient and we got the answers we needed in a month. Let me know if you would like to discuss this.

Online presense

2009-01-03T10:46:00.001-08:00

I think that in the future one will have to pay to have an unlisted presense.

Fixing a Windows Mobile Calendar Sync issue

2008-12-13T10:17:00.002-08:00

I had a issue syncing my ATT Tilt (HTC Kaiser) Windows Mobile 6.1 phone to exchange. While entries from my Exchange calendar would sync with the device, the entries entered on the device would not sync to Exchange. It was caused by improper timezone entries in the records being synced from the WM 6.1 ATT Tilt to Exchange. This was diagnosed by turning on verbose logging on the sync. This is done by going to ActiveSync on the device, choosing Configure Server, go to Edit Server Setttings, choose Menu, Choose Advanced, Change Event Logging to Verbose and Saving. The log file is stored in the Windows\ActiveSync directory on the device.

Why do people choose IT as a career?

2008-11-30T11:46:00.002-08:00

I believe that most people who choose IT as a career want to achieve order and balance. While the most complex issues in life revolve around people, early entrants to the career think that it is about the machine or the system and how it can always be made predictable and regular. Ironically, IT people eventually learn that IT is about enabling people and one cannot escape the people part. Many never really understand the complexity of the systems and begin to believe the system cannot be made predictable. Those that get it all in balance eventually understand the meaning of the word system and also understand that people are part of the system.

Step-by-step

2008-07-13T17:39:00.002-07:00

We recently had a problem with one of our websites. The team is very sharp, but had difficulty finding the problem. It was dispatched quickly once we started making very small planned changes, testing after each one. Sometimes, you can cover large distances faster by taking small steps.

2008-05-22T09:49:00.002-07:00

When I was at company A, it became clear to me there is an opportunity to help people inside and outside IT to understand the word "system". It is used frequently in an offhand way, but the true meaning isnt always "grokked". Unless extreme measures have been taken to segregate network components, servers, applications, databases, web services, etc. They all affect each other in either direct of indirect ways. It is almost always true that changes someplace will affect all other interconnected components to some extent. The impact may be very small, or the impact may be just enough change to create an unexpected reaction.

This understanding can be very challenging for many in IT because there is so much specialization. Less experienced IT professionals may have incredible depth of knowledge in their speciality, yet have very little understanding of most other components of the system as a whole.

Many at company A believed that there was a relationship between a particular master read-only database and the performance and availability of the shopping cart. No, no said the developers of the shopping path code. That part of the system never had a connection with the master database. It could be proven, they said by taking it offline and there would be no impact on the shopping cart. This was absolutely true.... However, the master read-only database did have replicates that were used by the shopping cart. And as it turned out, if there was a problem with replication, the replicates could fail to provide data to the shopping cart code. By rebooting the master read-only database, one could make replication recover and the shopping cart, when affected would recover.

Please sing along - the hip bone is connected to the leg bone, the leg bone is connected to the knee bone.... An understanding of systems is important.

Entropy exists

2007-11-17T11:21:00.000-08:00

Managing IT is always a balancing act. Regardless of the industry or the IT budget, there will never be enough time and money to keep IT as well ordered as you would like. This is because the reason d'etre for IT is to create change to enable the business/organization. Change is naturally destabilizing. The approach to counterbalance this destabilization is to define good process and ensure that the process is appropriately resourced with either talented people or enabling IT or both. In a free market economy, business is always motivated to "optimize" the investment in IT toward improving ROI ande ROA. Managing IT is about finding the best possible balance between maximum ROI/ROA and minimizing the risk from change. This is often made more difficult by the limited understanding that business leaders have in what it takes to minimize risk. Since it will never be possible or reasonable to eliminate risk, the goal of IT management is to live with entropy as best as possible.

Microsoft presence vis Cisco presence

2007-11-04T09:35:00.000-08:00

Microsoft has re-invigored its unified messaging/presence products with new activities to take on Cisco in the VOIP space. Cisco has responded with a purchase of Webex. The battleground is now set for a determination if Microsoft is held to asynchronous messaging and Cisco is held to synchronous messaging. Will there be a merger in the real world to get to integrated presence? Do we care? I wonder if we are all so overloaded with communications and messaging that the discussion will be a very slow evolution because we don't really need integrated presence and messaging.

There is nothing simple about IT

2007-11-01T23:47:00.000-07:00

There are many dimensions to IT -

It's relationship to the business
The many types of technologies that must be managed
Issues of security
Managing change
Managing quality
Managing cost
Defining and managing policies for use of IT assets
Privacy
IT personnel
People outside IT
and the infinite number of vendors

Nothing about IT is simple anymore.

It's about people

2007-03-10T09:08:00.000-08:00

Change requests, capacity, technical challenges, incident management, service level agreements, outsourcing process - there are lots of challenges that consume the time of IT professionals. We grow up thinking that most of the challenges are first technical. Then as we mature we recognize that efficiency and effectiveness are affected by having processes under control. When you start trying to implement these processes, it becomes clear that everything depends on relationships with people and influencing people to understand and adopt processes. The opportunity to do work starts with people. Ultimately to be effective, it is about working with people. It is really all about people.

Personal Network Relationship Management

2007-02-13T15:47:00.000-08:00

Contact management, Customer relationship management, partner relationship management, candidate relationship management, job-seeker relationship management - there are lots of tools today to support management of relationships. However, none of them appear to do what I am finding I need to do.

What I want to do is to manage my personal network as a relationship management activity. There are pieces of this network in many places and meta-data in many places. This includes my Outlook addressbook(s), my LinkedIn network, my Plaxo view of the data and meta data. Until recently it also included my JibberJobber database. I envision a desktop tool that integrates with my email tool, my LinkedIn network, other peoples personal network management tools (e.g. through Plaxo), my Skype directory, my VOIP directory (Broadvoice). It would contain meta-data that includes all the personal details that Outlook can contain with information about how the people are related to me and to each other. It would also help me track what I promised to each of these contacts, what each promised me, reminders to do these things and reminders to stay in contact. It would be especially cool if it could help me by suggesting what I should communicate to them based on events in my life, who they are, and my relationship to them.

I can see that there are great improvements in each of the niche relationship management tools. What I am looking for is a tool that is based on the recognition that I really only have one network with many dimensions rather than many focused networks.

Where does PE fit in the organization?

2006-05-25T05:23:00.000-07:00

Performance modeling is closely tied to performance testing. I often think that it is an extension of performance testing to answer the question "What will my performance be in production?" One could assume that performance modeling as a service within an organization would fit in the performance testing group. However, I am finding a trend that larger organizations are placing the service within the centralized architecture group. This group often reports to the CTO. This makes sense to many CTOs because they often associate performance testing with measuring a requirement of an application. Performance modeling is more frequently associated with understanding of performance across applications - predicting impact of change to the system of systems.

What does it take to drive SAP?

2006-05-22T07:12:00.000-07:00

Performance prediction of upgrades to SAP systems is more and more important. Recently, one of our clients did a tech upgrade to their global SAP implementation and were surprised to find that in the week following the upgrade, they needed to add 6 IBM mainframe engines to get reasonable performance from SAP.

Last week, I was at Sapphire and heard SAP estimate that the move to unicode will require between 30 and 50% more CPU on application servers and database servers. I have heard many organizations say that hardware is cheap and all we need to do is to throw more hardware at the problem rather than taking the time to understand what performance will be in production. I anticipate as more and more IT shops work to manage their SLAs with the business they will find that predicting performance prior to doing production transports will become mandatory.

HyPerformix

2006-04-11T19:14:00.000-07:00

I have been spending a lot of time focused on the HyPerformix modeling tool, IPS Optimizer. I have been demonstrating what I know about the tool to many diverse individuals. The tool rapidly provides an "ah ha" to many. The Modeler component shows off the Wysiwig topology interface of a system model. Behind the covers is the "ADN" code that makes the discrete simulation model work. When completed, the model visually contains what I call plug-and-play calibrated components (routers, firewalls, app servers, db servers, mainframes, etc) that can be clicked and modified. This allows for rapid changes to simulation runs to answer the business or technical what-if questions.

Behind the GUI is the simulation code. Best practice is to let the Application Model Generator generate the code from the parameters that are entered on the AMG worksheet. These parameters associate

transactions to business scenarios
sub-transactions to transactions
transactions to sub-systems
sub-systems to servers

and

utilized capacity of the infrastructure to each of the lowest layer of transactions defined.

It is the association of real information on the relationship between transactions and resource consumption that makes the model tick and allows you to modify the model for the implementation that you are considering for production.

Basics of performance testing

2006-04-07T21:13:00.000-07:00

Not too long ago, I had the opportunity to consult for the CIO of a large multi-national. At the beginning of the engagement, he looked at me and said “I am not seeing any value from the effort my company puts into performance testing. I need you to tell me if whether it is the tools that are the problem or us.” They had a centralized “performance certification” process. In this process, every change to the production environment was required to be tested for performance against pre-established goals. The centralized certification team, received the change from the development team along with the defined criteria for testing. The criteria was typically like a requirement to ramp to 10 virtual users running a specific set of transactions to achieve a response time of better than 7 seconds. There was no criteria for any regression performance testing and rarely any requirement to simultaneously drive load against any other portion of the application set to simulate what happens in production. It isn’t very surprising that the tests did not match results in production.

Effective performance testing is expensive. It requires access to a typically constrained test environment, performance test subject matter expertise, infrastructure and application subject matter expertise. It consumes the most expensive resource - time. It can save you from the most embarrasing business and customer fiascos. But it is not worth the cost unless it is done right.

The smokestacks are exposed

2006-04-03T08:13:00.000-07:00

I think that the web and ecommerce has propelled performance testing to a higher priority in many organizations. One VP of a large financial trading organization tells of the change that ecommerce has created for brokers. It goes something like this:

”Experienced brokers used to know that it would take 20-30 seconds to get a response to queries that they made of their trading systems. It was predictable. So, they would find out what the customer wanted, initiate the query and then ask about the kids, then the wife, and if that went too quickly they would ask about the dog. When the conversation would end, of course, the response would be back and the customer would never know how long it took.”

Life is different today. Every customer can know how long it takes for transactions to complete end-to-end. Even the definition of end-to-end is evolving. In the world of ecommerce it really is order to settlement or order to receipt. In the past, as a transaction crossed from application area to another, there were separations of responsibility that were tightly managed. These smokestacks are being impacted by pressure from the business for an end-to-end view based upon the customer perspective. Performance now needs to be measured in this context in production and tested and predicted prior to release.

Performance Testing and Performance Modeling

2006-03-31T07:12:00.000-08:00

Many organizations are enforcing good performance testing activities. These include defining the goals of the test, characterizing the behavior of virtual users, and thoughtful establishment of a performance test environment and data. For complex environments there are typically two challenges – the cost of sizing an acceptable environment and test scheduling conflicts with end-to-end components such as a mainframe test environment.Performance testing delivers some key information:

Performance testing can help with identification of bottlenecks of the application in the test environment

It can help identify memory leaks during endurance tests
It can help identify tuning attributes of the test environment that may be applicable to the production environment
It can be very valuable to help identify the relationships between transactions and resource consumption to build a performance model

Performance modeling can extend the value of performance testing:

It can translate information found in the test environment to results that will occur in the production environment. Because modeling simulations can be done quickly to test a variety of questions, it can offer a broader spectrum of performance and cost trade-off answers in a shorter timeframe and at a lower cost than performance testing by itself. Changes to a model can be done incrementally. This supports a variety of performance question strategies from a top-down approach to a system by system approach.

Simulation modeling can extend performance testing and provide an accurate answer to performance and capacity requirements in production - avoiding over capitalizing the production environment or the test environment. The ROI is real and easy to measure.

Standard Performance Modeling Approach

The standard approach to gathering performance measures to build a performance simulation model is to leverage system resource measures that are native and correlate them with the workload of transactions of a performance test. In many cases, Genilogix leverages the integrated monitors of Mercury LoadRunner to gather this data.

Questions that can be answered with modeling

What will the response time and resource consumption for a given number of business functions in the production environment after transport or go-live?
What improvement would be seen in production by varying the type of hardware, the number of servers or the network topology?
Would consolidation of sub-systems onto a centralized instance impact performance?
What would be the expected impact in production if the number of users or the ratio of transactions changed?

Performance Modeling Methodology

At Genilogix, we use the HyPerformix IPS Performance Optimizer (Mercury Capacity Planning) and Mercury LoadRunner to enable performance modeling. A standard detailed methodology is followed to deliver rapid results form performance modeling engagements. This methodology includes a five step approach:

Define goals, objectives, and data requirements
Gather data
Build, verify and validate the model
Predict the performance and capacity utilization in production
Evaluate alternative scenarios

Thanks for stopping by

2006-03-30T06:03:00.000-08:00

I began focusing on performance of systems when I was VP of IT at Barnes & Noble.com. Certainly, I had spent a good deal of my career focusing on architecture of systems, application development, integration of systems ... before that. But the early days of the Internet explosion was a unique period in the history of IT. There were many more unknowns that needed to be solved in a shorter period of time than ever before. This was especially true for one of the fastest growing, most used websites in the world. It was a unique opportunity to understand performance testing, the interdependencies of applications, network, and servers. Like any good IT group working to satisfy the needs of the business, need to understand the complexity of the systems was coupled with a need to support rapid change.

At Barnes & Noble.com, we were very fortunate to have a geographically distributed website. We had something that was and continues to be rare in IT, a 50% sized performance test environment that was always kept current with production. With appropriate planning to avoid risk the business and to our customers, we could shift all web traffic to one site and performance test against the other. Most IT organizations only have a similar luxury when they are launching a new large system and can use the future production environment as a performance test environment prior to go-live.

We got to be very good at performance testing. We learned that our performance test strategy was more appropriately based on work volume rather than virtual users. We also learned to leverage external load injectors to understand the vagaries of the Internet while adjusting for the affect of our Akamai network that was used to serve images.

The challenges of understanding the performance of changes can be summed up in a few items:

Understanding the topology of the test and production environments
Understanding the user behavior and transaction load throughout the business cycles
Defining the goals of all performance engineering activities (development, testing, modeling, management)
Planning for the right amount of performance engineering
Acquiring the skills and knowledge of the end-to-end application system, network and supporting application architecture
Managing execution of performance engineering activities
Leveraging the typically virtual team of subject matter experts for analysis, adjustment and feedback to the development cycle

I currently manage the practice at Genilogix LLC for performance modeling. There are some significant AHA's about the relationship between performance testing and performance modeling that I hope to discuss next.