PaaS testing

PaaS (Platform as a Service) is an application delivery model which provides services to design, develop, test, deploy and host applications. It plays a key role in cloud computing space similar to the other services, IaaS (infrastructure as a service) and SaaS (software as a service). Google App engine, vmForce and WSO2 Stratos are some of the popular PaaS offerings.
This post is not about detailed concepts of PaaS or cloud computing. Rather, I'm going to look at the testing and quality assurance aspects of PaaS. PaaS testing is not a widely discussed topic. There is no pre-defined model for PaaS testing. We need to explore the features of PaaS and derive an approach for testing PaaS apps. I will go through some basic components based on my experience of testing WSO2 Stratos opensource cloud platform.

Hosted services testing - Minimized deployment overhead

In simplest terms, PaaS testing is about testing a hosted platform. So, there is no deployment and configuration overhead for the test team. Testers are expected to access the hosted services remotely using web browser and carry out functional and non-functional testing.

Lesser test platform combinations

When you test a standalone product or a combination of products, you should try out the possible platform combinations such as different types of clustering setups, multiple DBMSs, Application Servers or various operating systems. However, when you test the same product suite or platform on cloud (PaaS), you are bound to one optimum configuration stack. You have one or two choices. There is no requirement to try all possible platform setups.

Multi-tenancy aspects

Multi-tenancy allows a single application to emulate multiple application instances. With multi-tenancy, a single application can be shared across many organizations. Therefore, series of tests should be carried out carefully to verify the multi-tenant aspects. For example, if company A logs in to a web application hosting service in your PaaS offering and deploy a web app, then the users of company B should not be able to locate it unless company A made it public. In other words, the configurations done by one tenant should not be exposed to other tenants.
These mission-critical aspects are some of the key requirements which should be taken into account in any PaaS testing model.

Performance and scalabilityAny hosted service should conform with the SLAs. Similar to a website, services included in a PaaS can be accessed by multiple users concurrently. Therefore, sufficient amount of performance testing must be carried out. The usual usage pattern of standalone technology platform can be dramaticallty changed when it is hosted as a service in cloud. For example, the general use cases of WSO2 Carbon SOA middleware platform are different from the standalone version when the platform is exposed as a service. WSO2 SOA middleware platform consists of 12 different products and all of them includes a management UI for administration purposes. In non-cloud based deployment, the multiple, concurrent users access to the management UIs are minimum since it is very rare that hundred of administrators accessing management console at once.
However, if this platform is hosted as a service in cloud (PaaS), then the concurrent user access to the management UIs are a highly desirable scenario.

It is a requirement that the hosted platform should be able to handle load seamlessly without affecting consumers. Auto-scaling is a key feature of WSO2 Stratos cloud platform. When new resources are needed, WSO2 Stratos transparently adds services and when load goes down, WSO2 Stratos automatically brings services down. Testers should ensure, with the load, the new EC2 instances spawning up and down correctly. The tools like hybridfox are very useful in these situations.

Data

If your PaaS provides users with the ability to store their data in cloud and various database centric operations, then the testing will be much complicated. As a tester, you must ensure the availability of data sources, accessibility and provisioning aspects. You must also take extra care that, if the data are geographically distributed in cloud, then it adheres to the legal requirements of the users of your PaaS.

Security

Security is the utmost important aspect of any cloud offering. It is recommended to have a separate group of security testers who do penetration testing and ethical hacking in order to ensure secure infrastructure of your PaaS. WSO2 Stratos cloud platform allows users to deploy their webapps, web services and various custom code. Therefore we must ensure that no vulnerable code are deployed and a code deployed by one tenant does not affect the operations of others or whole platform.

Virtualization

Public PaaS offerings are usually based on virtualized infrastructure such as Amazon EC2. Therefore, a full functional test cycle should be carried out on EC2 VMs inorder to make sure that there are no regressions due to virtual servers.

In addition to the above key areas, PaaS testing model should consists of the general web application testing aspects such as cross browser testing, accessibility testing, localization testing