Stability and Service vs Costs
How do you provide an exceptional solution and service, that is stable, yet keep costs down?
The simple answer is, you can’t! The two just do not go hand in hand. Ask any experienced IT professional to guarantee 100% up-time on a shoestring budget, and they simply won’t be able to do it. Ask a developer or support desk to provide a great product and low case closure times with limited resources, and, again, it’s simply not possible.
The 3 main factors: Infrastructure costs – Staff support costs – Development costs
If its too good to be true, it often is!
I can write this because, we ourselves have fallen victim to promises made by a major telecommunications supplier providing hosted contact centre solutions. We were promised superior uptime by senior members of their staff, we were sent fancy Visio diagrams with superb infrastructure topography, data centre fail over, DR and it all looked very good. We thought they are enterprise company, they must be telling the truth? surely?
But, we were oversold, and we experienced significant periods of
down time because of this. In the end they admitted that everything they promised was not in place and they were, in fact, “working towards these goals”. So, how do you cut through the ‘smoke and mirrors’, as we now fondly refer to it as. It’s hard and was ultimately one of the main factors that contributed to us deciding to develop our own product. We wanted to:
• Deliver the solutions and services that we promised
• Do things the right way
• Not cut costs that could result in a negative impact on clients
A large enterprise client once told us that our service was too cheap. They didn’t believe we could provide what we said we could, at the price point we sell at. It was shocking to hear, but they understood the true cost of providing these services and solutions, because they had to as well.
Microsoft Azure Hosting
Our hosting choice is Microsoft AZURE virtual cloud platform, trusted by 90% of the fortune 500 companies. It is a premium price however, we think that its benefits far outweigh its costs.
Below are some of the technologies that AZURE lets us incorporate into our solutions and how they improve our service; In our experience no one else goes as far as we do to ensure both security, compliance and stability.
LRS – Local Redundant Storage
Data is stored in multiple places in the datacentre, so should there be a partial outage, our platform will stay up >>Click for Microsoft Article<<
GRS – Geo Redundant Storage
Data is stored in multiple geographical regions, so should there be a complete datacentre outage, our platform will stay up >>Click for Microsoft article<<
AZURE partition their datacentre with total redundancy in multiple places. Should a fault arise on a piece of hardware, we have a duplicate platform ready in a different fault domain. Microsoft guarantee (within reason) that both fault domains can’t break at once >>Click for Microsoft Article<<
Like GRS, fault regions are designed with natural disasters in mind. Should any region have a serious issue, such as typhoons, earthquakes etc, there is another DC ready that does not typically suffer from those disasters.
Processes and Procedures.
We deal with clients who require the upmost security for their data, such as banks and utility companies. They have to be sure their suppliers are as stringent as they are around IS policies and procedures Being ISO 27001 accredited with BSI, means we have very strict security policies and procedures. We follow them religiously around all aspects of the controls, such as access and change control, business continuity and Software Development Life cycle (SDLC), then auditing and testing the use of the controls continuously. Anybody who follows stringent ISMS policies and procedures and goes through the continual process of accreditation, understands again the cost implications of it. Following these security principles is significantly more expensive than not. But they are imperative in the age of compliance that we live in, the cost of not having strict security policies and procedures when we mange the infrastructure where the data is stored is immeasurable.
Staff – Skill level, Cost and Capacity
This topic is often ignored when looking into total costings of service and delivery of solutions, yet it is the largest direct cost. Unfortunately, with more features, comes the requirement for high staff skill levels to manage them. With the number of servers we currently monitor (over 400), pro-activeness, high availability and rapid fail over is key, These all require higher skill levels of staff. Highly skilled infrastructure staff, cost significantly more than staff who manage a one server per client setup, which is the case at many suppliers, so staff costs become a huge factor to consider. Developers are one of the highest earning technical resources in the market today. With an average salary of £40,000 per year (reference glassdoor.co.uk), costs can quickly add up when building a stable, feature rich solution. We employ a highly skilled team of professional engineers and developers who have each been working with us for over 10 years and are aware of the complexities and impact of operating contact centres. This team, I believe, is the best in the business. We are continually striving to improve and believe that the biggest risk for any business is failing to adapt to new technologies. Let’s look at the ‘too good to be true’ scenario. If a small provider has 10 developers (let’s forget support and other engineers) and offers a system for £35/month per user, covering the £400,000 cost of development alone would require 1200 licences. If you then assume the same for support and account management these numbers are closer to 3000 – 4000 licences to even start to break even. Therefore, sales pitches that offer “free development forever” or “High Availability”, are just that, sales pitches. The costs don’t add up for it to be business viable.
The software we supply plays a huge role in availability. We may be able to achieve high availability on hardware, by leaning on technological advancements from the likes of AZURE, AWS and VMware, but if the software we sell is unstable then that becomes irrelevant. We often come across large clients who are struggling with current suppliers due to stability and scalability problems within the software itself rather than the infrastructure it sits on. Everyone goes through growing pains in terms of scale, but the problems should not be permanent, and if they cannot be resolved quickly it might not be a sustainable solution. Quite often one of the following is true;
• The software is built upon a third-party product that can’t be changed
• The software is old and has become too large a project to update
• The software has ceased to add new features, normally due to a company buyout.
• They do not have the development resources to fix the problems or progress the product.
We write all our own software, meaning we are in complete control and we are not a re-seller, so we don’t build on the foundations of other products. This gives us the flexibility to continue to develop with the creation of new technologies, creating new features when required and adapting to changes in the market, such as compliance. We have our own UK development team which has grown year on year, so we are confident that when we say we can deliver, we can.
The Impossible Task
If someone says they never have a problem, they are simply not telling the truth! The backbone of the contact centre industry is not infallible. Wherever possible we try and mitigate the risks, for example by having multiple telecommunications suppliers, multiple data entry points and so on. Unfortunately if BT have a loss of service then there is nobody you can call using the BT network, even, if we are still 100% functional. Because of broken promises, we quite often get contacted by contact centres
looking to improve stability, resilience and overall support services to their business after having been let down, or over promised, by their existing supplier. This is often the case for larger sites, as they have often simply out-grown their current supplier. They are promised things like complete fail-over or high availability, and quite often this is not delivered. The fact is, it becomes harder to implement these services with larger clients. It requires more enterprise grade infrastructure and a much higher staff skill level and therefore, cost.
Costings based on a client that contacted us; With a previous provider our client on average experienced an outage twice a week, they have 150 staff and are an BPO. They are
not currently paying much for their contact centre system but had become increasingly frustrated with the downtime.
Two outages a week, Average cost per agent billed to client £21.50 30 mins interruption to business per instance £21.50 (billed amount per hour) x 0.5 (half an hour incident) x 150 (number of staff) x 8 (downtime 8 times a month)
= £12,900 direct cost £12,900 per month or £86 per Agent per Month!!!
Other factors to consider
• Loss of momentum for the business
• Loss of sales
• Impact to staff bonuses due to lost hours resulting in demotivated
• SLA breaches for clients
• Harder to hit targets
• Dropped/Silent calls caused by incident = Increased complaints
• OFCOM breaches from recalling people not dis-positioned correctly
• Frustration from staff and managers
• Loss in vendor confidence
The total cost of the outages becomes much higher than the simple monetary amount. Is the extra cost required to avoid these situations justified? We truly believe this to be the case which is why we achieve 99.99% availability across our client base.