On data blind and data informed

This post is the story of a personal journey. It starts with a legacy product. Both powerful and flexible having grown to meet needs over time. But it is my view on steps to improve the process in my context - so your mileage may vary.

Phase 1: Transactional DB as source

Started with looking at the transactional data. Some things were obvious, one option in the UI is recorded as true/false in the DB. Some were more complicated and required more investigation. To back this up we conducted a user survey. Followed up by a workshop to find the most important user journey to concentrate on. This highlighted three related journeys. The key characteristic of activity in this phase was mining different structures of data. The format of the data coming from custom reports.

Getting to the questions to ask was guided by a user survey to almost the whole user base, a smaller workshop looking at the "job to be done" for the platform. Other than that the skills needed where accessible to most development companies or teams - SQL and Excel.

Phase 2: Audit metrics

During developing these we put in place metrics to track the different activities. This was across all clients in on pace. Even though these were basic metrics they did allow patterns of usage to emerge. For example, actions being triggered and then canceled. Repeatedly. We were able to follow up and check why this happened. Before it became a support or account management issue. This phase had standardised event data that was easily aggregated and sorted. But like the previous phase still tabular in nature. A key benefit here over the transactional data was explicit time stamping of events and users.

In this phase, we also started to use Google Analytics to make the collection of browser user statistics easier. One advantage that we gained over web server log file analysis (other than having to do log file analysis ;) was the browser window size and native screen size. This was useful for guiding the constraints on page design, and tracking how this changed over time.

Phase 3: Users behaviour


The next phase came in adding in more user journeys and variations on the three that already existed. the different combinations made it more difficult to see what was happening and who the interrelated functions were used in tabular data. So the next step was to use session recording. This also helps speed up doing usability testing by opening up a way of recording sessions and playing back. This is invaluable as the pace of development in the new framework speeds up. The feedback on how that solves the problem also needs to scale and improve in quality.

This does bring interesting challenges on workflow. Previous phases have ultimately ended up in some for of document - Excel spreadsheet or table in Word. So a factor in evaluating tools is "how can I make sure this interesting behaviour can be referenced". Can I get a permalink to go to a point in the playback? Will it integrate with our chat tool to let people know something has been found? ... will they then use it?

Phase 4+: Infinity and beyond...

What comes next very much depends on which process problem we are looking to solve next. One thing in our context that probably won't make sense is A/B testing, as in our B2B environment consistency and predictability are valued. Some of the things that it might include are:
  • ROI - Taking into account factors like whole-life costing of the new product development, revenue generation/protection and feature usage to get the most bangs per buck.
  • Process mining - looking at user behaviour to find correlations between tasks to form new packages or products.
  • In app feedback - fine grained/easy feedback, like the MSDN "was this article useful" to flag issues.
  • Visulisation - how information flows through the system, maybe animated to really tell a story.
  • Heat maps - Another way to tell a story about feature usage.
  • Further development of customer types and tailoring products for them.

Further reading

Comments

Post a Comment

Popular posts from this blog

CONFERENCE: TTI Summer Forum 2017 – Getting to Grips with GDPR

Image
T his week I was at the Travel Technology Initiative's Summer Forum . The subject was the new EU (and UK!) data protection laws. These are due to come into force in May 2018. It's a large topic where the individual member state's laws and guidance are evolving. There were three main points that I picked up from the presentations. First from Dai Davis , GDPR expert at Percy Crow Davis & Co. His lively presentation talked about how a large part of the change is in how rights communicated. Before this was by "fairness" through registration of usage with a central body. The shift is to transparency by informing individuals directly. This means that the consumer mindset could then shift to match how legislation is framed. For example, with the repetition of many companies holding personally identifiable information now having to inform what they are collecting and how they are processing it. Currently, awareness isn't high and the compensation not high eno
Read more

On HBX and online education

Image
I have written this blog post as a reflection on what has occupied me a lot in the past 3 months. After finishing my Post graduate Certificate in Technol ogy Manage ment with the O pen U niversity I was looking at what to do next in continuing my education - being a life long learner! I was looking at different online plat forms for interesting courses.  I was looking at sites like Coursera and  Future Learn , when I came across a new extension to Harvard Business School and a course they run called HBX CORe. This one paragraph pitch they provide sums it up pretty well: HBX CORe (Credential of Readiness) is a 120-150 hour certificate program on the fundamentals of business from Harvard Business School. CORe is comprised of three courses - Business Analytics, Economics for Managers, and Financial Accounting – developed by leading Harvard Business School faculty and delivered in an active learning environment based on the HBS signature case-based learning model. Sinc
Read more

On performance and environment

Image
Hobbies are a great way to relax and unwind from work. They are also an excellent opportunity to practice some skills and learn about yourself.  Photography shows a clear visual representation of how practice with tools can improve output . Running I have found has taught me a couple of lessons about performance and motivation. Here are my top four Goals are important   An arbitrary goal, doesn't fit in any plan but I feel good Yes, even arbitrary goals . As long as they have a narrative that you can use as a guide. They provide a sense of progress and small achievements to celebrate.  For example as an individual runner, it makes no real difference to my life running 10km just under an hour. Or a bit faster at 51 minutes. But the difference in the sense of pride and achievement in my progress was real.  Your environment matters Periodically I pause from running then have to restart. Sometimes this also involved a house move and I need to find a new route. On one of
Read more