Big Data is a term that's being thrown around very loosely; it is slowly creeping its way into society and most recently even appearing on adverts.
Over the past few years there has been a realisation that numerous companies have expotential amounts of stored data that has been sitting dormant for weeks, months and even years before either being archieved, deleted or destroyed.
Big Data is about unlocking the value in this information - and it is big business!
There are two types of Big Data;
Batch Data: The processing of large sets of data that has been collected over a period of time be it a day or a year. This is then processed and stored in a very efficient format which can later be queried to extract specific information. This information can then be used at face value or a company can bring in a Data Scientist to do a task known as "Data Mining". This is the process of diving deeper by using algorithms to find trends in the data, allowing them to then understand what has happened and then forecast what will happen with suprising accuracy.
Real Time Data: Processing on an instantaneous basis as the information comes in, providing real-time insights which can be viewed through an interface to track trends around people behaviours from gaming, advertising, machine performance, sensors etc.
Big Data's value within a business is now being realised and therefore creating an increase in demand for Architects and Developers. This has led to shortfall in the qualiy of Developers and sparked an influx of people specialising in this field. To be a reputable Big Data Developer you need to have around 18 months experience working directly with Big Data technologies such as Hadoop, Spark, Sharp and Storm. With the industry booming at a rate faster than any other in the I.T. sector, the technologies involved are changing rapidly.
Big Data tech that is hot now, may not be in 6 months because new ones are emerging all the time to break into the industry. You need to be aware of what is "hot" and what's "not", and stay up to date with trends, blogs and the industry as a whole and move with it in whatever direction it takes to stop yourself getting lost for dust.
Big Data has its place in every industry, and I really do understand the great potential and benfit it can provide. From I.T. to shopping trends, scientific discoveries and research into deadly diseases, once perfected, it has the potential to change all our lives. For now though, we need to sit back and see what it can offer as we have just touched the surface of its potential. If you are an aspiring Developer, Analyst or Manager? Get stuck in, attend meet ups and conferences and follow the trend of something that could be a fantastic career for yourself.
The sort of skills that are highly transferable to the Big Data space are mathematics, Java Development, Database skills as well as Business Analyst and Project Management.