I’m currently in the process of testing my Coffee App and wanted to share the testing process with you. This is the first time I’ve done any significant public testing for one of my Apps. In the past I’ve just tested with a couple of close friends/family. With my Coffee App, I wanted to be sure that I could get a good mix of people who didn’t feel obliged to say nice things.
In this post I’m going to take you through the full process of Beta testing your App.
This post puts us back on track in my 8-step (9 if you count going to the pub) guide to App development success (or total failure – or more likely a shade in between). You can catch up on the previous posts in the series here.
1) App Development – Doing it properly
2) App Development – The Design Document
3) App Development – The Wireframes
4) App Development – The Artwork
5) App Development – The Code (Split into 3 development diary posts – Diary #1, Diary #2, Diary #3)
6) App Development – The Testing (You’re here)
7) App Development – The Final Art
8) App Development – The Marketing
9) Go to the Winchester and have a nice cold pint
The Tools
There was a time when sharing an incomplete App with others was a very manual process with a multitude of headaches. These days, there are some tools out there to help you. In addition to Apple improving Xcode and the provisioning portal, you now have access to free (currently) services such as TestFlight. (I’m sure there are others out there too, feel free to share them in the comments).
TestFlight aims to make App testing easy by managing your Beta testers, distributing your App to them (over-the-air), and collecting feedback. It’s an extremely slick service that removes many of the Admin headaches from the testing process.
Getting Beta testers
First up, sign up to TestFlight. You can do that here.
Once you’ve signed up to TestFlight, you’ll then need to create a team. TestFlight describes this as “…a means of grouping your builds and testers”. For my Coffee App, I decided to create a single team called “Bytesize Adventures” for testing all of my Apps.
Once you’ve done this, you’ll have an option to “Invite a Teammate”. You can either add them using their email addresses or, if you click “Start Recruiting Testers”, you can use a TestFlight hosted recruitment form. This has its own URL that you can share publicly.
Gathering Beta testers is not as easy as it sounds. If you’ve never discussed your App publicly before, you may find it difficult to get people interested. This is yet another good reason to blog/tweet/talk about your App early in development.
I decided to put a call out on twitter for anyone interested in Beta testing my App. I also posted a link on one of my development updates. Within a couple of days, 16 people had signed up – not bad at all.
People who have signed up appear in the TestFlight dashboard and you are given the option to approve or reject them. You’ll also notice that TestFlight automatically collects device details for each person (and crucially, the UUID for each device).
Once you’re ready to send your App to your testers, you need to get your hands dirty.
The Provisioning Portal
Whilst TestFlight handles a lot of things, it can’t handle your certificates or building and signing the App. These steps assume that you’ve already setup an App ID and that you’ve already created and installed a Distribution certificate.
First, in TestFlight, under Teams, you should have the option to select all of you testers and then in the “actions” drop-down, “export devices”.
Now, log into Apple developer connection and access the iOS provisioning portal. Go to the Devices section and “Upload Devices” (selecting the file you downloaded from TestFlight). This will load the UUID’s of all of your testers devices into the provisioning portal and register them as development devices against your account.
Still in the provisioning portal, you then need to go to the Provisioning section, hit the Distribution tab and create a new profile. When creating your profile, be sure to select the “Ad Hoc” distribution method and “select all” devices (this will enable the testers, and only the testers, to run the App).
Download the Ad Hoc provisioning profile and drag it onto the Xcode Dock icon – magic will ensue and the profile will be registered with Xcode.
Building for Ad-Hoc
I’d get a coffee if I were you because whilst this process isn’t particularly taxing it often throws up obscure issues. Thankfully the iPhone development community rock and, if you do face any issues, you’ll find a solution in no time.
First of all you’ll need to duplicate your release build. You do this by selecting your project in Xcode and then in the Configurations section, click the + icon and select “duplicate Release configuration”. You’ll also want to name it something like “Ad Hoc”.
You’re now going to need to create an Entitlements plist and tell the build about it. To do this, create a new file (use the Entitlements template under the code signing section). You can leave the name as Entitlements.plist.
Click on the Entitlements.plist file and add a key for “Can be debugged” (if it doesn’t already exist), and then set it to “NO”.
In you App target’s build settings, search for the Code Signing section and expand the Code Signing Entitlements key. You should then copy the Entitlements filename (Entitlements.plist) into the Ad Hoc key value.
Next, in the build settings for your project, scroll to Code Signing section and you should now see an Ad-Hoc key under the Code Signing Identity key. Its value needs to be set to the Ad Hoc provisioning profile you created above. It should appear in the drop-down (along with any other provisioning profiles you have).
Now you need to edit your schemes so that the Archive command builds for Ad-Hoc. To do this click the schemes drop-down in the toolbar and select “Edit Schemes”. Then in the left hand panel, select Archive. In the Build Configuration drop down in the right panel, select Ad Hoc.
The Archive command will now build based on the Ad Hoc distribution. Archiving is Xcode’s process for building and signing your App for distribution.
Ensure you have “iOS Device” selected in the Scheme drop-down (rather than the simulator), then select “Archive” from the Product drop-down menu. Once the process completes, the Organizer will launch with the new Ad Hoc build listed.
To share this with TestFlight you need to click the “Share” button, select .ipa as the content, and then select the Ad Hoc distribution profile from the drop down (this is the same one you selected in the build settings).
Choose a name and save destination and you’re done.
Sharing the App with your Beta testers
With our freshly created IPA, we now need to get this into TestFlight – which will take care of the distribution.
In TestFlight you need to select the “Builds” tab, then “Upload a build”. Select the IPA you created in the previous step and wait for it to upload. TestFlight has some basic verification and it will notify you if something appears to be wrong (In my case I had accidentally signed the shared build with a different certificate to the one I built it with – doh!).
Hold on before you hit “Confirm testers”.
You need to enter some release notes. This section is literally the email body that will go out to your testers. Here are a few tips I’ve made up (and used for my Coffee App Beta), take them or leave them.
- Be polite and thank your testers for their involvement
- Highlight specific areas of interest in your App (e.g. the awesome laser-shooting feature)
- Highlight areas that aren’t complete (I feel this is useful so as not to waste your testers time pointing out missing/broken features you are already aware of)
- Let your testers know how they can send their feedback to you (covered in more detail below)
- Say thanks again. Honestly, these people are giving up their time to help you so be nice.
When you finally click through to the next stage you’ll have another opportunity to target specific testers. Once you’ve done this the email and link is sent to your testers… and then TestFlight becomes even more awesome!
The builds section of TestFlight will show you exactly what stage each tester is at. It will tell you if the email has been sent successfully, if the email has been opened, if the App link has been clicked, and if the tester has installed the App.
13 of my 16 testers installed my App. The other 3 opened the email but didn’t proceed (for whatever reason). I think thats a fairly good install rate.
Managing Feedback
As I alluded to above, you should provide your testers with as many avenues for feedback as possible. TestFlight allows them to reply directly to the original email and it will collect their feedback in the TestFlight dashboard for you to view. I also provided my email address in the release notes in addition to a “contact me” link within the App itself (this will stay in the final release and initiates a new email to my support address).
Before we move on to managing the feedback, we need to discuss managing your expectations. I was fairly impatient and, having received no replies within the first 2 days, was a little disappointed. It took the best part of a week to receive any volume of feedback and even then, of my 13 testers who installed the App, only 4 sent me feedback (so about 1/3rd).
Knowing what I know now, I would probably have pushed for more testers to begin with. I think getting a sign-up form out early and promoting the Beta is crucial.
Moving onto managing the feedback itself, this is going to depend on the volume of responses. Because my pool of testers was small, I didn’t receive a flood of feedback to manage. However, I did still receive around 30 individual items to be looked at (an item being something like “have you thought of x” or “you should fix y”).
The first thing I did was to look for duplicate items across testers. If more than one person feels the same about a feature, chances are you should listen. In these instances I’ve pretty much just gone ahead and implemented the changes.
Next up was looking for conflicts. If a tester highlights a particular behaviour they’d like changed and a second tester contradicts it, chances are its just opinion. In these instances it depends on how many are contradicting. You could just go with the highest volume or you could try to come up with a compromise.
Finally you’ll have the one-off items that only exist in a single testers feedback. These are a mixture of uninformed opinion, impractical suggestions, minor fixes, good advice, wacky suggestions, and absolute gems! You have to pick out the gems and make judgement calls on the rest. Keep in mind your ultimate goal with the App and try not to deviate too much from your original vision.
How much testing?
I suppose this depends on the complexity of your App and how far along you are with development. I’ve completed my first Beta and am working on applying the feedback to my App. I then plan on a second Beta just to give people a last opportunity to provide feedback before launch. This second Beta will be complete with all features, all polish, and no bugs (hopefully).
You could go on testing endlessly but at some point you have to draw a line and submit the App. I’ve already begun moving some items into a “First update” task list to be handled after launch.
Conclusion
So there you have it. Beta testing doesn’t have to be painful and provides you with truly valuable feedback. My next post will focus on the final Artwork for my App and then we move on to Marketing (by which time my App will have been submitted).
(Crash test dummy photo courtesy of BusyPrinting)
Part 1 – Doing it properly
Part 2 – The Design Document
Part 3 – The Wireframes
Part 4 – The Artwork
Part 5a -The Code (Xcode, GIT, and Core Data)
Part 5b – The Code (Coffee sharing and Camera functionality)
Part 5c – The Code (A change in workflow)
Part 6 – The testing (You are here)
Part 7 – The Final Artwork
Part 8 – The Marketing
Part 9 – The Launch
Part 10 – The Numbers
Part 11 – The Postmortem
(Bonus) Part 12 – A review of the process