The same tends to be true of most complex phenomena in real life: stock markets, economies, the success or failure of companies, war and peace, relationships, the rise and fall of empires. Short-term analyses aren’t only invalid – they’re actively unhelpful and misleading. Just look at the legions of economists who lined up to pronounce events like the 2009 financial crisis unthinkable right until it happened. The very notion that valid predictions could be made on that kind of scale was itself part of the problem.

It’s also worth remembering that novelty tends to be a dominant consideration when deciding what data to keep or delete. Out with the old and in with the new: that’s the digital trend in a world where search algorithms are intrinsically biased towards freshness, and where so-called link rot infests everything from Supreme Court decisions to entire social media services. A bias towards the present is structurally engrained in almost all the technology surrounding us, not least thanks to our habit of ditching most of our once-shiny machines after about five years.

What to do? This isn’t just a question of being better at preserving old data – although this wouldn’t be a bad idea, given just how little is currently able to last decades rather than years. More importantly, it’s about determining what is worth preserving in the first place – and what it means meaningfully to cull information in the name of knowledge.

What’s needed is something that I like to think of as “intelligent forgetting”: teaching our tools to become better at letting go of the immediate past in order to keep its larger continuities in view. It’s an act of curation akin to organising a photograph album – albeit with more maths. When are two million photographs less valuable than two thousand? When the larger sample covers less ground; when the questions that can be asked of it are less important; when the level of detail on offer instils not useful scepticism, but false confidence.

Many data sets are irreducible and most precious when complete: gene sequences; demographic data; the raw, hard knowledge of geography and physics. The softer the science, however, the more that scale is likely inversely to correlate with quality – and the more important time itself becomes as a filter. Either we choose carefully what endures, matters and meaningfully captures our receding past – or its imprint is silently supplanted by the present’s growing noise.

Time cuts several ways, for there is another crucial sense in which it remains a limiting factor: the availability of human time and attention. Corporations, individuals and governments alike have orders of magnitude more information available today than they did even a few years ago. Yet they don’t have any more available attention, board members, chief executives, elected officials or hours in the day. Better and better tools exist to help decision-makers ask meaningful questions of the information they possess – but you can only analyse what remains accessible. Mere accumulation is no kind of answer. In an era of bigger and bigger data, what you choose not to know matters just as much as what you do.