Android Device Automation with Calabash

In this blog post/tutorial I will describe how to write a simple automated test for an android device and get it running on your machine. I will start with a quick crash course explaining the android emulator, how to get it up and running and how to install a native application (in this example instagram). Once the emulator is up and running, we will move onto 'calabash-android', how to install, configure, write and run a simple login test.

Installing and running the Android emulator

First thing is we need to download and extract the latest Android SDK, get it from here and unzip it into your area of choice, I would recommend putting it in your $HOME directory for now.

After you have downloaded and unzipped the SDK, you will need to create an AVD (Android Virtual Device). Do this by changing into the tools directory in the SDK root and running the android create command:

$ cd android-sdk-linux/tools $ ./android create avd -n android4.0 -t 20 -c 256M

If you want an explanation of what exactly the command line options are, then check this documentation from the Android developer site. In a nutshell we are creating an AVD, giving it a name of android4.0 specifying target (type of machine in our case a device with version 4.0 of the OS) and an SD card size for our device of 256M.

We have now created the AVD should now be able to boot it up with the following command:

$ ./emulator -avd android4.0

I find first time you issue this command on the AVD, it takes a while too boot just be patient after a while you should see something like this (depending on what you parameter value you used for -t).

At this point feel free to have a little mess around with the device, and get familiar with the controls as detailed here.

Installing an Android app to automate

The next step is to download and install an Android application, for this tutorial we are going to use Instagram, you can download it from here .

Once you have downloaded it you need to install it onto your emulator, the android SDK has a tool called 'abd' that lets you do this, cd into the platform-tools directory and issue the following command:

/android-sdk-linux/platform-tools$ ./adb install Instagram_4.0.2.apk

Where the Instragram_4.0.2.apk file is the location of your downloaded Instragram app. After the command has completed the Instagram application will have installed on your emulator and you should be able to see it in your application list:

We are now ready to install calabash and start automating!

Installing Calabash

Running the following gem install command (assuming you have Ruby installed, you haven’t go get RVM here).

 $ sudo gem install calabash-android

This will install the calabash-android gem and command line client, now create a new directory, navigate into it and issue the following command:

$ calabash-android gen

This command sets up your standard Cucumber folders and files, if you look in your directory now, you should see a 'features', 'step_definitions' and a 'support' directory. For now we are only interested in the features directory, navigate into there and you will see a file named 'my_first.feature'. Open it up and replace the contents with the following scenario text:

Feature: Login feature Scenario: As a valid user I can log into my app When I press "Sign In" Then I see "Forgot Password?" When I enter text "pablo" into field with id "username" When I enter text "pablospassword" into field with id "password" When I press "Sign In" Then I see "Follow friends and interesting people to see their photos here"

We are almost ready to run our first test, but before there are a couple things that I allways do to ensure my tests run. We first need to create a keystore file and then re-sign the instragram app so It can be used by calabash. To create the keystore file you will need a JRE tool called 'keytool'. If you have Java installed hopefully it should be in your PATH if not your going to have to hunt down where it is and then run it. Assuming that its in your PATH issue the following command:

$ keytool -genkey -v -keystore debug.keystore -alias androiddebugkey -storepass android -keypass android -keyalg RSA -validity 14000

Then Run:

$ calabash-android resign Instagram_4.0.2.apk

At this point you should be in the new directory you created with the calabash-android gen tool, e.g you should see a features directory. Now run the following command to run the scenario we just created, and watch the emulator:
 
$ calabash-android run Instagram_4.0.2.apk ADB_DEVICE=android4.0

The Instagram app will be removed and re-installed as part of the run, the application will open, go to the 'Sign In' page and attempt to login with some credentials I have included in the feature file. The tests will fail on the final step for two reasons.

1.  The account details I have used do not exist
2.  There is security on the Instagram app that does not let you Sign in from an emulator, running these on a actual device will solve the issue.

Summary

This post is intended to get you up and running, we have used some 'pre-defined steps' that make up part of the calabash-android implementation, you can view more of these 'canned' steps by navigating into the directory of where your calabash-android gem lives and viewing files in the gems 'lib' directory. There are steps which handle actions and gestures that you may find useful, however you will need to delve into the actual API to get the best from the tool.

In the coming weeks I intend to do a similair guide for calabash-ios and how to write page models using calabash, in the meantime check the following links to help you get more from the tool:

• API Documentation
• Documentation of canned/pre-defined steps
• Documentation on calabash locators

FT.COM Cross Site Scripting Example

I have been working with a number of client web applications, and it surprised me how many of them are/have been open to XSS to the point I'm actually worried.

My guess is that testers don't really understand what XSS is, how to exploit it and what risk it poses to the business, so I will try and give you a working example using the ft.com. They are not a client of mine but I did find their search was open to the XSS exploit. Don't worry I have contacted and reported the issue to them, so by the time you read this the site will be patched and immune to any sort of XSS bug.

Lets get started:

1. First lets find an input into the webserver, ft.com has a search form field we may be able to exploit.

2. We want to enter some malicious text into the search field, for example script tags to see if they break the page. First lets do a normal search and analyse how the page is returned, type 'Lenovo' into the search field and click search, after a short wait the results are returned:

A few things happen, the most obvious is that results are displayed, we can remove the search term and the page is not broken in anyway. Also look at the url you can see that there is a url parameter "http://search.ft.com/search?queryText=lenovo". Now lets see how the XSS exploit works.

3. The easiest way to see if the site is open to XSS is to enter a script tag into the search field. This normally takes trial and error to get the result you want, you may need to enter part of a tag  like "script>", or encoded text "%3C%2Fscript%3E" its just about having a play and learning how the application responds. For the ft.com search box I found a closing script tag </script> broke the page:

You can see some JavaScript being output just above the search results. A quick look at the markup in firebug and we can see where the script tag has been inserted, escaping some JavaScript code. Lets now do something really simple and say hello to the world using this code:

http://search.ft.com/search?queryText=)</script><a href="https://www.maltapersonaltraining.com">hello from pablo testerbar</a><script>

With a little more effort we can modify the page further, include some css and javascript on another server, add more mark-up, this really depends on how creative/malicious the attacker/hacker wants to be. Look at the url in the address bar, the attacker could now send this modified url out to unsuspecting victims (hiding the verboseness via something like tinyurl), and as its hosted by a reputable organisation the user maybe duped into thinking the content is legitimate.

I hope this gives you a quick and simple example of how an XSS attack works. My recommendation is that all testers should include this type of testing as part of their standard testing routine.

Useful links:

• Url Encoder - Does exactly what it says on the tin, quick way to encode/decode your text
• Another XSS example - An excellent write up on XSS using the Gaurdian website, gives you another example and well worth a read.

Breaking the Rules, Pushing the Limits

Testers should break the rules....

I come across a lot of Developer In Test roles these days, its a new trend that a large number of companies are adopting as they fully embrace Agile and BDD practices, or at least try. I think its good and productive to embrace Agile, its something that works and as a tester far better than waterfall era of giving everything to test in a short space of time at the end of a project. 

I am concerned however in the trend of the DIT role and what its doing to the test horizon, what worries me is this now huge lean on automation, and companies losing sight of what testing is really about, now it seems automating your Cucumber feature files are more important than actually finding bugs, and in some companies I have come across, leaving functional testing to the BA's whilst your DIT's just write automation code or even worse, making the developers that actually wrote the code now functionally test it (recipe of an apocalyptic ELE for your project).

Testing is about finding bugs, about hacking and breaking, its about pushing the application you are testing to the limit, and as a tester to get the golden egg, is to make the application fail, or even better crash. 

Can you find bugs automating? well if you are automating correctly yes, if you go beyond the UI layer and start interfacing into the service layer and back end systems you can learn a great deal about a system and how it works, and the most useful automation suites I have come across and developed do this, automation goes beyond the Selenium/Webdriver tests that have come to gain huge popularity and notoriety like Kim Kardasian after her sex tape, you have to get past the surface and beneath the sheets (no pun intended and certainly no links included).

Automation is your safety net, and if done correctly as part of the testing process a great and important safety net to have, but you must do it properly, and if you want to learn to do it properly follow these blogs, for me these are true experts in the automation field:

1. Nat on Testing - If i am having trouble automating a feature, i come here first 9/10 hes already done it and blogged about it.
2. The Evil Tester - Mr Richardson has a great Selenium/Webdriver course on Udemy, if you want to learn how to write automation code in java properly, I strongly suggest you take it.

Im writing this because I recently got dropped onto a project that was about to go live, there had been a huge push on automating the application, and indeed a lot of time and money invested in this task. When I got given the application to test, in the first 10 minutes I found a major bug, its one of the first things I test for when I get given an web application.....XSS. What really struck me even more was how easy it was find, I didn't have to pollute some AJAX requests going on behind the scenes, that I only found by looking in the Firebug Net tab, nope all I had to do was put some encoded js into the url after the search input parameter, it was really that simple. 

It certainly showed me that the testers were far to busy automating than finding bugs, they lost their identity, they were not the super hero testers that they should of been, they became developers writing automation code, a waste of testing talent.

So by all means automate, but remember why you are there, and why you got into testing in the first place, push the application to the limit, and break the rules. Ill leave you with a great clip from an average movie, here the test pilot and his pilot buddy are set the task to test out a couple of drone aircraft that apparently are suppose to wipe the floor with their human counterparts, study what Hal Jordan does to beat them, and try and take this metaphor with you to work, albeit without the green spandex.

All Testers Should Have A Database

SQLite Manager and testing data files.

I recently was given a task to test some data files, created by one system, processed by another then sent back to the system of origin in a transformed state (data munged with a few extra columns and data, standard ETL stuff).

In my experience, when data leaves its system of origin it pretty much goes into the unknown, it passes a boundary into a foreign system, and sometimes a foreign culture. It may come from a company culture where testing is of the utmost importance and enter into a culture where testing in itself is a foreign concept, needless to say it will not get treated in the manner in which it is accustom too.

We must be careful of making too many assumptions, a 'csv' file that contains data means different things to different people and more importantly different systems.For example files created on a Windows server may not be encoded correctly for a UNIX machine, some developers insist that csv values are contained in "double quotes", others prefer to omit the same, and some systems just will not handle special characters, no matter how much encoding and escape characters you include....Any ways back to the task.

After being ask to undertake this testing task alarm bells rang,I  thought I needed to dust of my old awk skills, but then I remembered my data-warehouse days, and how easy it was to test data once it was actually in a database. I had a SQLite instance running on my machine from some RoR stuff that I was messing around with, and decided to use it to test the data files, after all it was just gathering (binary) dust on my Lenovo.

At this point I could of imported the file using the SQLite shell, but I wanted to search for a quicker way so I did some digging and found a cool Firefox plug-in called SQLite Manager.

In a nut shell, you install it, open it up in Firefox, point it to your csv files and hit import, field names (if you have a header line) and table name (based on file name) get created automatically, great for beginners and experts alike, we all like to save some time. Here is a quick guide on how to get it up and running, please note you need a running instance of SQLite before you can use this plugin, here are some very good install instructions (you dont need to install the rubygems, but you should use ruby...it increases your street cred).

Here is what my csv file looks like:

 1. Open up SQLite Manager, for the sake of this example Ill create a new database.

2. Once the database is created hit Database option on the top left, and go all the way down to 'Import' this will bring up the import wizard, you should be presented with some options, below I am importing a postcodes.csv and set the options to import the file, as you can see you can even import data via SQL and XML.

3.  Hit 'OK' and you should get an option to modify the import before it begins, as you can see SQLite Manager has already figured out the column names for you, here you can change the table attributes

4. After a while, depending how many records are in your file, the import will complete and you will be ready to start running your sql queries against the file, just hit the 'Execute SQL' tab and test away.

I hope this gives you an idea of how easy it is to set up your own database, and how useful it can be for testing, here ive given you an example in how you can use it to test data files. Maybe you are doing performance testing, you could perhaps use it to store results and compare metrics between runs and builds, and with the power of SQL you could get some really good reporting out of it.

I advise all testers (especially consultants) to keep your own database on your Laptop/PC, you never know when it will come in handy, and something like SQLite uses less resource than your browser! dont believe me...check your process status's!.

Think Like a Tester, Train as a Developer.

Training is the key....

I consider myself a man of habit, somewhat stubborn and stuck in my ways especially when I was younger. In my first few jobs mainly working in data warehouses, I refused to work with graphical SQL clients, it was all sqlplus, db2, isql for example, and up until a few years ago I would not go near an IDE, I was tied In holy matrimony to vim. I have chilled out a lot since then and have become more open minded, and would not develop anything without my trusty intelliJ IDE, in addition I wish I had embraced these more modern tools sooner in my career, but why didn’t I?.

I always felt there was something pure about working with the command line, seeing lines of green output on a black screen always made me feel more technical, and indeed it made me more technical. I had to think about ORACLE environment variables, about where the database lived and what files I had to configure to connect to it, I forced myself to do it the hard way which in turn helped me develop stronger technical skills than the other testers I was working with.

As I have become older and wiser, I realise that doing things the hard way is probably not the best way, its time consuming and sometimes less productive. However training yourself and keeping up to date with the latest technologies and methodologies is essential if you want to become a better tester. Testing literature in my experience talks more about the mindset and the process of testing, there is nothing much out there that teaches you how to become a technical tester, which is why I tend to train myself as a developer, whilst still keeping a testing mindset.

I'm always taking on small projects in my own time, from setting up a media server on my trusty Pentium 4 machine with Ubuntu install, to building fully fledged data driven website's. Tinkering and messing around with technology is something that I enjoy. It is this exploration and playing that increases my technical skills and in turn makes me a better tester.

I'll give you an example, I have recently started playing with the web development framework 'Sinatra' to build a hotels booking website. When choosing a database, instead of going for the more traditional SQL distribution, I decided to run with MongoDB. The application also makes numerous calls to a RESTful hotels api, so sending, receiving and parsing those requests had to be coded. In a weekend I have managed to gain an understanding on how ruby sinatra works, how to install and configure Mongodb and developed my net/http programming knowledge further. This journey has now sparked an interest in html5 which i'll learn in depth so I can make a mobile version of the site.

I am learning specific technologies, but more importantly, learning how applications are built. Learning the inner workings, the cogs that drive applications, and how they fit together which gives me the inside track on how to break them. By undertaking development activities I have become a better technical tester.

Training in this way helps me develop skills that are essential to me as a tester I can think of the following, but this is by no means an exhaustive list:

1. Reverse engineering - No documentation? Not a problem to a true technical tester, they understand how software applications are built, and can decode the functionality and architecture with a little effort.
2. Technical Terminology - Technical testers can speak the language of the developer, they go deeper than the front end, in web apps for example they talk the language of requests and responses rather than 'actions' and 'expected results'.
3. Better Bug Reports - Technical tester can fire up a shell, they know where the logs are being spooled, they know how to view the stored procedures in the db, they know how to monitor process's and resources, and this all goes into a bug report when they raise one.
4. Automation - OK so not strictly a testing task in my opinion, but coding skills help testers to automate their testing if need be, and as more and more jobs require this skill it certainly comes in handy.
5. Adoption - Technical testers are quick to adopt new useful tools, even if they are not directly related to testing, anything to help them in their bug finding mission they will utilize.
6. Enjoyment - Technical testers are continually learning new skills and technologies, days seem to fly by, and they have a real passion in what they do, its almost like it isn’t even a job!.

There are many more benefits in developing your technical testing skills in this way, for me it beats just reading a text book cover to cover with no practical activity. Finding bugs is a lot about trial and error, trying new things and persistence, it's this 'hacker' mentality that separates the good testers from the bad, and therefore it's a mentality that all testers should adopt.