.webp)
In my last article, we explored the evolution of the EPM marketplace - which was driven by transformation efforts to connect business functions together through EPM, and to optimize the implementation of OLAP technologies for performance at scale, with many investing millions into these initiatives.
But as time went on, business conditions changed. We now live in a world of relative instability, where a business’s ability to react to changing conditions is a key competitive differentiator. As flexibility became more and more important, the people that built Pigment began to question the status quo of EPM.
The problems with OLAP
The adoption of OLAP in the 1980s as an EPM technology was transformational. But as time went on, the most advanced companies were the first to encounter limitations, particularly in the areas of data management, hierarchy and relationship management, and scalability.
Data management (Handling sparsity)
Multidimensional OLAP cubes are inherently optimized for dense data sets, where most combinations of dimension values contain data. However, in practice, many of these combinations may be empty - leading to sparsity.
.webp)
Despite this, storage is still allocated for every possible intersection of dimensions within the cube, even if no meaningful business data exists at those points.
Each time an aggregation or calculation is performed, each of these intersections is processed, which is why it makes sense to maximise the number of intersections which store data.
In practice what this means is that data sets are frequently manipulated to fill as many of these intersections as possible - this is usually achieved by merging levels of data within hierarchies to the lowest common denominator of levels for hierarchies across an organization. This inevitably requires a compromise in how granular analysis and reporting can be.
Consider the following example of assessing product revenue by region (purple) per month (blue). In this case the company offers a product which will only be sold in Canada in February, as a special Valentine’s Day promotion. To see this product’s revenue, Canada will need to be added as a distinct region in the OLAP cube. This means storage will be allocated to product revenue in Canada every month - even though there will only ever be data in February.
Adding this extra storage will impact processing times whenever this cube is accessed or changed, so it makes sense to try and minimise this storage if possible. In this case Canada may be merged with the United States to form a new Dimension of North America, thereby minimising the storage allocation required.
The issue here is that next year when the team needs to assess the impact of the product on Revenue, there’s no way to distinguish between Revenue in the United States vs Canada, potentially missing an opportunity to invest further in the Canada Region.
Hierarchy and relationship management
Because OLAP cubes require data to be pre-aggregated into the dimensionality that will be used for ongoing analysis and reporting this means that when those hierarchies change, they cannot be easily updated without affecting the entire cube.
It also makes it much more difficult to manage hierarchies that need to be represented in different structures for unique reporting needs. For example, if the finance team breaks workforce spend down by region, but the IT team needs to budget their workforce spend within projects that span multiple regions, there will need to be some mapping between employees, their region, and the projects they’re working on. If any changes are made, for example introducing a % of time spent per project, the mapping would need to be updated.
Businesses will typically rely on either IT teams or third party SMEs to maintain their implementation and to bring their data structures in-line when the business changes how it operates. This can be both expensive and time consuming, and means that in a lot of cases changes simply do not happen in a timely fashion, meaning teams make plans without an up to date view of how their business is operating.
Scalability
OLAP databases are very performant for retrieving data since they operate in-memory, but this also means that they are usually confined to an on-premises deployment, or rely on single-server architectures that inherently have a limit to their processing power.
The power of relational databases
Pigment has embraced recent advances in relational database technology, leveraging the inherent flexibility of this architecture, and establishing a unique implementation for data retrieval and processing that optimises this further for performance at scale.
Relational databases handle sparsity natively: storage is only ever allocated for the intersections of dimensionality where data exists, meaning that entire levels of Product hierarchies, departments and Regions can be mapped in Pigment, without compromising on granularity.
Managing and maintaining hierarchies is much more straightforward, since any record can be linked with another to by any attribute or property. This means tables of attributes can be easily added and updated, and connections established and used to retrieve complex intersections of records for fully flexible analysis and reporting.
The end result of all of this is that Pigment can be easily crafted into a true reflection of any business - a digital organisational twin.
But the challenge with relational databases has always been to increase their performance when performing these flexible data manipulations at scale. So how does Pigment solve this problem?
Pigment’s secret sauce
Without giving away all the product team’s secrets (we'll get more into that in the next post), the Pigment engine is designed for flexibility, and optimized for performance. This means that in addition to the inherent benefits of relational database architecture, Pigment is uniquely tuned to perform at scale.
Pigment achieves this through a multitude of approaches, including selecting the most advantageous infrastructure for each customer, processing calculations in the optimum sequence for end user experience, and by consolidating representations of data before end users even request them - making data retrieval and analysis smoother than ever before.
To book a demo of Pigment, click here.
