Trade Me - Senior Software Test Analyst

Dec-2014 - Aug-2016

Trade Me

Trade Me is the iconic consumer auction website in NZ, one of the most popular websites accessed from NZ. Trade Me are renowned for their agile implementation, and have a well established test guild.

I was a Test Analyst with the Motors group, in a small cross functional agile squad, testing software changes to the iconic NZ Trade Me website focussing on the Motors page and features.

References

• 2016 Jason Cullum Reference. Jason was my chapter Lead for my duration at Trade me
• 2016 Jason Cullum Annual Review.
• 2016 Mike Berry Reference. Mike was the Trade Me Motors Delivery Manager

Responsibilities

• Context driven, tool assisted exploratory testing, using session and thread based techniques
• Testing DB, UI, API, and architectural changes
• Leading the deployment of changes using Trade Me's continuous integration and continuous delivery processes
• Splunk system monitoring
• Agile Squad Mastering (Facilitating project inception, story grooming, planning & estimation, retrospectives, daily standups, sprintboard)
• Test automation for API (ReadyAPI/SoapUI) and UI changes (protractor) using BDD with Gherkin syntax.
• Test planning and peer test reviews
• Visual test tools (eg mind maps for test planning, and video capture of test sessions etc)
• Jira for test plans, managing test sessions, and defect workflow. Confluence wiki for test practice documentation.
• Metrics driven development - using A/B Testing
• Active contributor to test and agile guilds
• New staff induction and junior staff support

Agile

• Spotify model of squads, chapters and guilds.
• Small squads typically 2 Dev's, 1 Tester, ½ BA. The Product Owner (PO) gave direction but was considered outside the squad.
• Squads are trusted to ask for assistance when needed
• Squads perform regular retrospectives, are encouraged to use any mix of methodologies, and to continually look for new ways to improve their processes.

Testing

• Followed Context Driven Testing principles. Considering a wide scope for testing, looking for anything that might surprise someone that mattered.
• Shifted Left. Contributing early to discussions about UX, system architecture, and testability.
• Efficient, focussed on high risks, leaving acceptable risks.
• Used "Three Amigos" to share and understand the problem, the solution, and the testing.
• Peer reviewed test plans (session based exploratory test charters) in Jira
• Test early for early feedback. ie. before dev work was completed. Aiding alignment, informing what is left to complete story, avoids re-work.
• Dev walk through - UI and code - review test plan
• Demo to PO
• Testers deployed to production during twice daily release windows

Tools and Technologies

At Trade Me is used the following tools and technologies

• ReadyAPI for API testing & Automation using Groovy
• Jira for stories, test plans, session charters, bug tracking,
• mindmup, Xmind, SimpleMind for mind maps
• Confluence Wiki
• Mercurial for source control management
• Protractor for UI automation
• PowerShell scripts
• Icecream Screen Recorder for recording test sessions
• Microsoft SQL Server Management Studio for querying databases and profiling stored procedure calls
• Splunk for monitoring and live error analysis and error graphs
• Fiddler and Wireshark for network protocol analysis
• Chrome extensions (CJS Custom JavaScript, Bug Magnet, Clear Session, Responsive Web Ruler, WASP etc).
• Developer tools on tier 1 browsers

Projects

Motors Price Change

Partway through 2015, Trade Me announced to the NZX (where Trade Me is a listed company) that the motors pricing was changing, effective in just over two weeks. That put a fixed deadline on the change. To make matters a little harder again was that the pricing structure was being changed from a three tiers to four tiers. This was now a high profile, risky, time constrained project with potential for significant financial and reputational impacts. I had only been with the company for a few months and there were very few testers with any better experience at the Auckland office.

This project was scary from the outset but with help from my newly joined test chapter lead, Jason Cullum, we set out to implement and test the changes. It was clear we were not going to be able to cover as much of the testing scope as we would have liked to, and it was going to be a task of determining the areas to cover, the areas that we would not be able to cover and communicating the test coverage with the wider team for feedback and buy-in. We decided to draw up a large mindmap for this and it proved a very good visual tool to convey the information. We also looked for opportunities to call in help with specific parts, and we were able to borrow a tester from the Christchurch office for a few days.

In this project I was the lead tester, and managed the testing and comms with guidance and support from my chapter lead. I was pleased that we were able to quickly work out the agreed test coverage and then put the blinkers on to test as much as we could cover as fast as we could. In the end the price change went live without major issues and only a couple of minor issues were found in the untested areas, these were fixed quickly with fast follow updates.

Premium Packages - Motors Sell Process

The process Trade Me users use to create a new (or edit it) listing is called the sell process. A user specifies the listing's details, uploads some photos, they can promote their listing with some extras, they're shown a confirmation screen and presto! their listing is live for people to look at and buy. Parts of the sell process code is shared across the entire Trade Me site. The promotional extras include: feature to show the listing higher in search results, a bold title, a yellow border to highlight the search card, a subtitle, etc. Most of these were shown in an unattractive list on the extras page, and were in need of a face lift. The product owner, after investigations, decided to replace the entire concept of extras with a new model of four packages with much nicer graphics and UX. Each package would contain a suite of the extras growing from the basic package to bronze, silver and gold packages, each with more goodies.

This was a difficult project, requiring changes and testing through the entire stack from DB, core shared Sell Process architecture, to a whole lot of work at the UI for the new page design. We found many places where the extra's were being referenced eithe by name or by their cost and required updates and testing. There were a plethora of edge cases because the Sell Process code was also used for sellers to edit their listings. Making sure the page displayed properly across the tier 1 browsers and portable devices such as iPads also proved quite hard.

I was trying to help the squad focus on the most common use cases, trying to identify the value associated with certain features, or the cost of leaving small bugs to be fixed later. This was hard, but we developed over time a good model where we would do periodic demos of potential issues and progress to the product owner and together we decided on what was fixed now or deferred to later.

Due to size and fairly risky nature of this feature it was implemented with an on/off config key, and we were able to deploy smaller stories to production in a switched off (aka dormant) state. After several month's work we were finally able to switch the feature on in production, but after a few weeks it showed that it negatively affected TBC

Metrics Driven development of the Motors Sell Process Extras Page

The formula for calculating the confidence of the difference between the control and the variant (this should go into a blog at some point)

Statistical Significance for MDD