A guide to Regularly Recurring Reporting Panics

Data, the cliché runs, is the new oil. Not only in the sense that with appropriate refining it’s potentially valuable, but it’s also often hidden from sight. Buried beneath layers of organisational and IT infrastructure complexity, data is usually the software layer furthest removed from end-users and stakeholders, even as it powers almost everything they do. 

So, how can you tell if your data architecture is rotten and needs attention? One of the surest signs you need a data architecture revamp is if individuals or teams repeatedly ‘down tools’ to focus their attention on preparing reports. This is an anti-pattern found often enough in large organisations to deserve its own name. I call them Regularly Recurring Reporting Panics, or RRRPs.

Spotting RRRPs in the wild

Spotting RRRPs should be easy. It’s when the people involved, whether it’s business analysts, database administrators, or data scientists, need to stop what they’re doing and start gathering, collating, and reformatting data to populate a standard report template or dashboard on a monthly, quarterly, or annual basis. In the meantime, their backlogs build up and need to be addressed, hopefully, in the truncated period between the generation of one set of reports and the need to prepare the next.

The only difficulty in identifying RRRPs is a kind of cost-blindness. This is either because stakeholders and decision-makers only interest themselves in looking at the final reports and don’t give much thought to the costs of producing them, or because it’s assumed that RRRPs are just the natural and inevitable price of generating data insights. Many organisations have been in an RRRP cycle for so long that it’s taken as normal. After all, isn’t sinking extensive time and effort into reports just what business analysts do?

The price of panics

The costs of RRRPs are high. Most obviously, there are the overheads imposed by the time spent preparing the reports themselves, and the distraction of individuals and teams from other tasks. But a further concern is that the degree of uncertainty and manual re-work inherent in RRRPs, combined with deadline pressures, makes the reports they yield inherently error-prone. In other words, a lot of money and time are being spent generating reports that aren’t necessarily accurate. And of course, the time spent wrangling data into a predefined shape also means there’s seldom capacity left over for exploration, asking new questions, or generating new insights. The team is running to stand still.

In any given reporting period, these costs may be considered tolerable, but they increase linearly over time. Predictably, the price incurred in one reporting period will arise near-identically in the next. And this is not a cost that can or should be indefinitely sustained.

The rise of RRRP

If your organisation is suffering from RRRPs, it’s probably not because of a single catastrophic data problem, but a confluence of smaller issues. Although the list of possible difficulties is perhaps endless, most of the more common variants will fall under one of five categories:

Access management issues: data access to key systems is restricted to a small number of staff, who are then tasked with ferrying data that needs to be shared out of the system and storing it elsewhere.

Inconsistencies across systems: datapoints are stored in multiple systems, and their values may diverge. Manual intervention is needed to assess which value is correct, and ensure that this ends up in the final report.

Manual collation: for whatever reason, alignment of data isn’t safeguarded by automated, reliable mechanisms such as foreign keys. Instead, data is downloaded from multiple sources and then manually aligned to create a ‘coherent’ view of the data domain.

Poor data quality: a whole host of issues can present themselves here. Data might regularly be missing, requiring manual chasing by staff. Data validation may be poor or non-existent, leaving nonsensical values to be spotted and corrected by hand. Data might regularly be added to systems without obsolete information being deleted. Institutional memory, vague heuristics, and sometimes sheer guesswork are needed to determine where the true value is.

Zombie reporting: reporting requirements no longer reflect current business imperatives, but nobody feels they have the authority to remove the requirements. Entire legacy systems may thus be maintained to support obsolete needs.

These challenges then interact in complex ways, repeatedly creating a data mess requiring extensive manual intervention to clean up.

Stop feeding the beast

If you’ve worked in a data-related field for a large organisation for any length of time, you probably recognise some or all of the above problems. So what is to be done?

The first and most important step is to acknowledge that there’s a problem. Reporting should never be a significant line item for business-as-usual (BAU) expenses.

The next step is to start thinking seriously about data architecture. Unfortunately, every organisation’s RRRP is different, so there’s no one-size-fits-all solution. But if the bad news is that RRRPs tend to be caused by multiple, complex small causes interacting with each other, the good news is that architectural interventions can fix many of these in one fell swoop. Even the humble relational database has, if its features are well-managed, the capacity to address almost all the above issues. It’s a question of using them well, and of course, the last fifteen years have seen an explosion in technical options, their sophistication, and in their ease of use. Solutions to even the most intractable RRRPs are ready-to-hand, if only time and capacity are freed up to address them.

Taming RRRPs once and for all

Fortunately, it’s easy to assess just how much effort to expend on taming RRRPs. If they are minor, involving a couple of days of a single individual’s time, then perhaps only a few small tweaks to an existing architecture are justified. If a larger team is bogged down over a longer period, a more radical overhaul is in order. Your return on investment, in terms of the cost saved, is easy to calculate here.

And not only will your team thank you for it, now that they’ve been freed from these ravening demands, they’re in a better position to look further ahead. Once the RRRP has been tamed, they can stop panicking and start exploring, learning, and adding value. Everything a data professional wants and was almost certainly hired to do.