bankingciooutlook

Tuning Strategy to Achieve Highly-Secured Digital Banking Sector

By Lou Senko, SVP / CIO Information Technologies, Q2ebanking

Lou Senko, SVP / CIO Information Technologies, Q2ebanking

Valuing the roadmap

Our journey at Q2 has been one from the story books—a journey that has been both exciting and at times, a little bit scary. Growing any business at over 300 percent in less than three years can be hard to manage; however, growing at that rate in the highly-regulated, highly-secure digital banking sector can strain even the most mature business group. Additionally, because we are adding nearly 100,000 consumer and business banking customers each month who rely on our solutions for critical services including checking balances, initiating wires and executing corporate payroll it means our investment into our people, processes, products and our partners must keep pace and deliver tremendous measurable value back to the organization.

Because all great businesses are primarily differentiated by their talent, we seek to align our development with the pressing needs of the business–which is always a challenge. As you might expect, the hyper-growth curve we are experiencing can create pressure on a young, developing and growing IT organization pressure to anticipate future needs of the business but also, pressure to provide engaging and beneficial development opportunities. Even if growth were normalized, forward planning can be difficult; however, with our extraordinary growth, forward planning and professional staff development can become nearly impossible as the business works to improve SLA compliance, develop critical skills, deliver improved reporting and execute to scale.

With this in mind, tuning a strategy to follow a design anticipating future requirements of the business and the growing competencies of the staff can be a delicate balance. Buying the ‘thing’ you need three years from today is simply not financially possible (or responsible). The ‘IT unicorn’ we all search for is a balance between anticipating the future and executing each and every day—building just a block or two in front of you.

At Q2, we have successfully partnered in our quest to locate the ‘IT unicorn’—to balance future needs with day-today execution. In our experience, partnership with a technology vendor we know and trust delivers both our employee development goals and support our business growth. In our case, a partner who knows ‘what good looks like’ at 4X our current volume is simply mission critical. Allowing for industry best practices to augment the knowledge base and experience of our staff while providing additional bandwidth required to successfully execute on critical projects in parallel and adding the experience to avoid the all-too-common need to reset and rework ill-fated steps to rip-and-replace the last project you just finished.

Reflecting on the evolution of our strategy and our team, I am pleased at the choices we have made over the past several years. From initial low-end iSCSI entry to our current fullblown storage fabric spanning multiple data centers, our journey began with a modest step into consolidated storage in 2012, where we made our first EMC purchase in the form of two VNX5500 SAN, running iSCSI. These quickly scaled up and in late 2012, our team determined it was time for a drastic capacity plan that could outpace the demand of the organization. Our current storage capacity was near zero, both processing and free-space was oversubscribed and all of this was happening with the legacy solution and new storage solutions running side-by-side. With much of the application and infrastructure intertwined moving into an external environment not viable so we were faced with a decision to invest; however, pre-IPO funding required careful allocation of resources necessary to bridge the gap between current state and longer-term roadmap.

With tremendous external commitment and passion for our outstanding customer support, we decided to gear-up and focus on turning hosting into a core competency. We immediately focused on investing in the people, technology and process to deliver a best-of-class platform for Q2’s robust application suite. We migrated from Tier 3 to Tier 4 data centers in late 2013/early 2014 which was the catalyst to shift the foundation technology -moving into an all-fiber channel network and addressing the explosive storage demand.

To enable the transition to the new environments while limiting the impact on our customer’s mission-critical applications suite, we invested in new VNX5600 SANs and upgraded the existing two VNXs. This decision, along with our talented team, supported the build out of our new data center environments and the transition of 300 customers to new environments over a six-month period. The company-wide effort meant late nights, dedication and commitment, but also delivered little downtime to our customers—a key outcome for the investment and partnership.

With a solid infrastructure established, and performance management optimization well underway, we shifted our focus to Disaster Recovery in late 2014. Moving to a production-to standby replication methodology, we added EMC’s Recover Point Appliances (RPAs) to synchronize the workloads between the data centers should a facility failure occur. Adding a single engine VPLEX to each environment offered the flexibility and agility to manage the two local VNX as a single data pool.

Continuing to map to our estimated hyper-growth in 2015, our next steps were to implement DataDomain+Avamar at the backup layer, then upgrade the VPLEX with a second engine and metro-replication. We then upgraded one of the VNX to the newer XtremIO All-Flash-Array (AFA), solidifying a high performance backbone fabric, virtualized between the data centers enabling the next level of availability and scaling.

Today, we have seen batch SQL workloads cut to a fifth of runtime, disk latency cut by 50X, and our overall database performance improve by nearly 50 percent. These improvements resulted from a combination of application tuning, manual database tuning and the investment in the new storage architecture designed to support the entire solution framework.

So, What’s Next?

At Q2, we know that continued growth is coming. In fact, due to our business model, the majority of our future growth continues to be very well understood which means we will continue to scale out the XtremIO, eventually replacing the remaining VNX footprint. We will also continue to eye ECS-type (elastic cloud storage) technologies –as our application evolves, our reliance on the monolithic database backend could potentially evolve and enable us to explore the options of hyper-converged infrastructures.

In closing, we have enjoyed and appreciated the EMC partnership. Not only have they anticipated our technology needs as we have moved along our roadmap, but they have also supported us with key insights and advice throughout our journey. As with any partnership, it is not always perfect; however, like any true partnership we have remained closely aligned through collaboration to remain focused on the substantial improvements to our business, our people, our processes and our technology.