Know Your data. Cost per byte vs value per byte

Cost per byte vs. Value per byte: Rethinking Data Efficiency

We are living in an era where nothing gets erased (just archived). Let us dwell on cost per byte vs value per byte of such data. Every byte you store, move, or process has a cost. We focus on cost saving. Data engineering isn’t just about hoarding everything, it’s a calculated risk about understanding whether those bytes are worth to store them.

Pro hint – do not fall into the trap of 'let us grab everything and think about it later’. It does make sense until you figure out what is what but then remember to delete it ? Oh wait… it is so cheap nobody cares….

The basics

Cost per byte measures how long is your bar in an excel file name including the word „cost”. The amount of money your system spends handling data. This includes cloud storage, bandwidth, and/or compute time. For instance any cloud services charges per byte scanned, like aws dynamo db – meaning inefficient queries literally burn money byte by byte. You pay for things you do not get any value. Do you always get only that what You want or maybe „let’s grab it all and get what we need later”.

Let’s flip the narrative

We should include ( not replace ! ) value per byte metric. Sound a lot harder to calculate and kinda ephemeric . It should try and answer questions like : amount of insight, decision impact, or profit does out data deliver? Data and bytes are not equal. Some raw log.info of purchase cart could cost only microcents to store but add little business value. In the meanwhile a few kilobytes of well-curated, thought through analytic data could drive millions in revenue, strategic decisions and employee happines.

Thinking about that value will also triage the data and tell us what we really need and what we can safely scrape and forget about. Less data is easier to analyse and correlate.

What to do with cost per byte vs value per byte

Think about which byte could get the „Golden byte” award based on value which it gives us.

As data engineers we should aim for high value per byte and low cost per byte. All about the value.

• Store only necessary data — focusing on signal, not noise.
• There should be a position like „Data curator”… prove me wrong
• Using compression and partitioning to reduce cost per processed byte.
• Archive old data or just keep the analytics
• Applying analytics that maximize actionable value extracted from each dataset.

Let us stop repeating “big data” all the time. It is a lot more complex ! It is valuable data done efficiently.