Legacy historians ship with two knobs that most sites set once, at commissioning, and never look at again: exception deviation and compression deviation. Between them they decide, forever, which of your process values are worth remembering. This post is about what those knobs actually do, the day someone finds out, and why the argument that justified them expired around the time storage stopped being the expensive part.
Two knobs, set in 1998, still deciding
Exception deviation lives at the interface: a new value is only sent to the archive if it differs from the last sent value by more than the configured deadband. Compression deviation lives in the archive itself, classically as a swinging-door algorithm: the historian keeps extending a straight line through incoming points, and only writes a point when a new value swings outside the corridor the deviation defines. Everything inside the corridor is discarded on the assumption that a line drawn between the kept points is close enough.
Both are configured per tag, usually as a percentage of instrument span, usually by copying the value from the previous project. Both then operate silently for decades. The archive looks continuous. Trends draw smooth lines. Nothing anywhere says "the values between these points are reconstructions".
A worked example
Take a pressure transmitter on a pump discharge, ranged 0–40 bar, scanned once per second, with compression deviation set to 0.5% of span — a common, even conservative, choice. That is a corridor of 0.2 bar.
Now run the incident. The line is steady at 6.21 bar. A downstream valve slams, and over 30 seconds the pressure climbs to 6.34 bar, holds briefly, and settles back to 6.20 bar.
The excursion is 0.13 bar. The corridor is 0.2 bar. The swinging door never opens. Thirty seconds of genuine process behaviour — thirty raw samples — produce zero archived points. The archive records a serene straight line from a point logged some minutes before the event to a point logged some minutes after, and it records it with no annotation whatsoever, because as far as the algorithm is concerned nothing was lost: everything discarded was, by definition, within tolerance.
Six months later there is a seal failure investigation, and the reliability engineer pulls the trend. The trend says nothing happened. The operator remembers something happening. If the panel still has a paper chart recorder — and this is precisely the day someone digs it out — the pen trace shows the spike the £40,000 digital archive does not. The instrument was fine. The archive was configured to disbelieve it.
Multiply this by every tag, every deadband copied from the last job, every commissioning decision nobody has revisited since. The peaks are shaved, the valleys are filled, and the shave is always in the direction of "less interesting than reality".
The 1995 argument
Compression was never a data-quality feature. It was an economics feature, and in 1995 it was a good one: disks were small, expensive and slow, and a historian that kept every scan of every tag was unaffordable. Discarding 95% of points to fit the archive on the hardware of the day was a defensible engineering trade.
The arithmetic has not survived. A tag scanned at 1 Hz produces about 31.5 million points a year; with delta-of-delta timestamps and value compression that is on the order of 45–50 MB per tag-year — losslessly. A 20,000-tag site therefore generates roughly 1 TB of raw archive per year, which object storage will hold for around £15 a month. The complete, unshaved, second-by-second truth of a mid-sized plant costs less per month than the sandwiches at the meeting where the seal failure was discussed.
"We can't afford raw" was true once. Repeating it in 2026 is not economics; it is habit.
The honest alternative
OrdinateDB's position, stated as a non-goal in the project documents and implemented in the write path, is:
- Raw by default. There is no exception or compression deviation anywhere in the write path. What the collector saw is what the archive holds. Full stop.
- Lifecycle policies, not silent shaving. If old data is eventually thinned — and at some age, for some tags, it reasonably is — that happens through a lifecycle policy: a visible, versioned object that states what is kept, at what resolution, for how long, and who approved it. Thinning is a decision someone made and signed, not a default someone inherited:
policy discharge_pressure_retention v2 {
match: pump_*/discharge_pressure_bar
keep:
- raw for 5y
- mean(1s) thereafter
approved_by: c.osei 2026-04-02
}
- Disclosure at query time. Any query that touches a range a policy has thinned says so in the result — a
lossy_rangesfield alongside the values, with the policy that did it. The reliability engineer pulling that trend in 2032 sees not just the data but the fidelity of the data.
The details — policy syntax, approval trail, how disclosure propagates through aggregates — are on the archive page and in SPEC 03 §6.
Choose your losses in daylight
There is nothing wrong with deciding that ten-year-old vibration data can live on as one-second means. There is a great deal wrong with a 0.5% deadband, set by a contractor in a hurry during commissioning, deciding for thirty years which pressure spikes officially happened.
Loss is sometimes a reasonable engineering choice. Silence about loss never is. Keep raw, write your thinning down, and make every affected query confess — the storage bill will not be the thing that stops you.