BRIEFING: ThoughtWorks' QTB on Big Data

Some notes from this quarter's technology briefing from Thoughtworks. This session's topic was "Big Data". I was pleased that the topic was chosen as I am interest in Big Data and travel, especially how it can be used by my clients and to enhance the product that I work on.

 Caitlin McDonald from twitter has also created a Storify story from tweets during the event (with the added bonus is that I am in the background to someone's photo)

Session

The main speaker was David Elliman with Ashok Subramanian - David has also written a blog post called The Big in Big Data Misses the Point that presents some of the content covered or see the full presentation in English or German. The session started y looking at the origin of "information explosion" and how in the 1940s people were starting to get worried about the miles of shelf space would be needed by 2000 to store all the books produced. This was contrasted with the explosion of multimedia information produced now, for example pictures from camera phones or the daily output from the large hadron collider.

 Next basic architecture such as MapReduce was explained along with what makes "Big Data" - variety, volume and velocity - but also that it's what you have in your data, you don't always need a large data set. Sampling is important!

 Another point was that the relationship is that velocity and volume is inversely proportional, low volume allows more real time analysis towards batch processing with very large volumes. Sorting data takes around 80% of the time - understanding data and sorting models is key, don't expect to get a data dump and have instant insight as there is no one answer or model, it requires data scientist to understand question and go through models.

 Process described to get successful insight was:

  1. Start small
  2. Start with "?"
  3. Iteratively follow the value
Next some implementations of the Lamda architecture were described and tools discussed, finishing off with a talk about data lakes and their similarity to enterprise data warehousing - i.e. not very like big data. The quote of the day came in this session:

"Master data management is the enemy of innovation"
Overall a good session highlighting the importance of good analytics in getting insight from data. Having read their big data blog I was surprised by this as they prefer the term Big Data Analytics.

Venue

Shoreditch village hall is quite a nice venue, a little bit of a pain from Brighton and the room layout made mingling a bit harder. The sound was had a bit of feedback at the start but soon settled down. The food was great, a very tasty cheese burger (non-cheese and non-meat burgers were also available).
Location: Hoxton Hall, 193 Hoxton Street, Shoreditch, London N1 6RA, UK

Comments

Post a Comment

Popular posts from this blog

CONFERENCE: TTI Summer Forum 2017 – Getting to Grips with GDPR

Image
T his week I was at the Travel Technology Initiative's Summer Forum . The subject was the new EU (and UK!) data protection laws. These are due to come into force in May 2018. It's a large topic where the individual member state's laws and guidance are evolving. There were three main points that I picked up from the presentations. First from Dai Davis , GDPR expert at Percy Crow Davis & Co. His lively presentation talked about how a large part of the change is in how rights communicated. Before this was by "fairness" through registration of usage with a central body. The shift is to transparency by informing individuals directly. This means that the consumer mindset could then shift to match how legislation is framed. For example, with the repetition of many companies holding personally identifiable information now having to inform what they are collecting and how they are processing it. Currently, awareness isn't high and the compensation not high eno
Read more

On HBX and online education

Image
I have written this blog post as a reflection on what has occupied me a lot in the past 3 months. After finishing my Post graduate Certificate in Technol ogy Manage ment with the O pen U niversity I was looking at what to do next in continuing my education - being a life long learner! I was looking at different online plat forms for interesting courses.  I was looking at sites like Coursera and  Future Learn , when I came across a new extension to Harvard Business School and a course they run called HBX CORe. This one paragraph pitch they provide sums it up pretty well: HBX CORe (Credential of Readiness) is a 120-150 hour certificate program on the fundamentals of business from Harvard Business School. CORe is comprised of three courses - Business Analytics, Economics for Managers, and Financial Accounting – developed by leading Harvard Business School faculty and delivered in an active learning environment based on the HBS signature case-based learning model. Sinc
Read more

On performance and environment

Image
Hobbies are a great way to relax and unwind from work. They are also an excellent opportunity to practice some skills and learn about yourself.  Photography shows a clear visual representation of how practice with tools can improve output . Running I have found has taught me a couple of lessons about performance and motivation. Here are my top four Goals are important   An arbitrary goal, doesn't fit in any plan but I feel good Yes, even arbitrary goals . As long as they have a narrative that you can use as a guide. They provide a sense of progress and small achievements to celebrate.  For example as an individual runner, it makes no real difference to my life running 10km just under an hour. Or a bit faster at 51 minutes. But the difference in the sense of pride and achievement in my progress was real.  Your environment matters Periodically I pause from running then have to restart. Sometimes this also involved a house move and I need to find a new route. On one of
Read more