X
98323

New 451 Research survey reveals data volume and growth projections

November 22 2019
by Matt Aslett


Introduction


We don't need a dedicated research project to tell us that data volumes are on the rise. In every geography and industry, enterprises large and small are increasing their investment in data and analytics with a view to outsmarting the competition. Data is being generated and stored at greater volumes than ever before, and more of it is being used in analytics initiatives to generate meaningful business insight. 451 Research's latest Data & Analytics survey provides evidence of this ongoing trend, providing the data that illustrates data processing volume and growth projections.

The 451 Take

It is no surprise to see that data volumes are growing. It is interesting to note, however, the differences between the median and the mean, illustrating that there is significant variance among individual companies even if the overall trend – growing volumes of data – is the same. It is also no surprise to see that the volume of data being used for analytics is expected to grow faster than the rate of growth for all data under management – enterprises already have more data than they can analyze and there is arguably as much to be gained by shedding light on existing 'dark data' as there is from collecting and analyzing new data. As such, it is surprising that respondents expect the volumes of structured data under management to grow at the expense of unstructured or semi-structured (at least on average) data, although the results do indicate growth in the use of unstructured or semi-structured data for analytics purposes.

Data, data, everywhere


As noted above, we don't need a survey of enterprises to know that data volumes are growing. However, this survey provides details of the extent to which data volumes are growing. Respondents were asked about the total volume of data that they have under management today and the total volume of data they expect to have under management in two years. The results illustrate a broad range of responses: 10% of respondents have less than 1TB of data under management today, 43% have between 1TB and 999TB, 26% between 1PB and 499PB and 8% more than 500PB.

Figure 1
Figure 1: Current data volumes Source: Data & Analytics, 2H 2019

That equates to the median respondent having 638TB of data under management today, while the number of companies with huge data volumes results in the mean volume of data under management being significantly higher at 122PB.

The volume of data is expected only to grow. Two years from now only 5% of respondents expect to have less than 1TB of data under management, with 40% expecting to have between 1TB and 999TB, 30% between 1PB and 499PB and 11.4% more than 500PB.

That will equate to the median respondent having 821TB of data under management two years from now, an increase of 28.6%, while the mean volume of data under management will grow at a similar rate, 28.3%, to reach 156PB.

Figure 2
Figure 2: Mean and median data growth expectations Source: Data & Analytics, 2H 2019

Structured or unstructured?


Respondents were also asked further details about that total volume of data they have under management today, including what proportion of that data is structured rather than unstructured or semi-structured.

Structured data accounts for 350TB of the 638TB of data under management at the median respondent today, and is expected to grow to account for 572TB of the 821TB of data under management at the median respondent two years from now – equating to growth of 63.4%.

This level of growth is somewhat surprising given the growth in unstructured or semi-structured data we would anticipate based on anecdotal evidence, and actually implies that the volume of unstructured or semi-structured data at the median respondent is expected to decline.

The mean volume of structured data under management is currently 63PB and is expected to increase at 41.2% to reach 88PB two years from now. This would indicate an increase in the mean volume of unstructured or semi-structured data in the same period.

Data used for analytics


Respondents were also asked about the proportion of data that they have under management that is actually used for analytics. The median respondent is today using only 311TB of data for analytics today – slightly less than half of the total 638TB of data under management.

In other words, more than half of all data under management at the median company today is not used for analytics – or is dark data.

The volume of data used for analytics purposes is expected to rise rapidly to 595TB two years from now, an increase of 91.2%, to account for almost three-quarters of the 821TB of data under management at the median respondent.

The mean volume of data used for analytics is currently 73PB and is expected to increase at a lower rate of 43.6% to reach 104PB two years from now.

Figure 3
Figure 3: Data used for analytics Source: Data & Analytics, 2H 2019

Structured data accounts for 164TB of the 311TB of data used for analytics at the median respondent today, and is expected to grow to account for 375TB of the 595TB of data used for analytics at the median respondent two years from now – equating to growth of 129.4%. This also indicates growth in the use of unstructured or semi-structured data for analytics purposes (from 147TB to 220TB).

The mean volume of structured data used for analytics is currently 41PB and is expected to increase at 54.8% to reach 63PB two years from now. This would also indicate an increase in the mean volume of unstructured or semi-structured data used for analytics in the same period (from 32PB to 41PB).