Understanding Weissman Scores
This post assumes basic familiarity with compression algorithms. If you're new to the topic, check out our Middle-Out Compression Explained post first.
When we first demoed Pied Piper's compression, everyone asked about the Weissman Score. Here's what it actually means and why it matters.
#What is a Weissman Score?
The Weissman Score is a theoretical metric for comparing compression algorithms. It accounts for both:
- Compression ratio - How small the output is compared to input
- Speed - How fast the algorithm runs
The formula looks like this:
W = α * (r / r̄) * (log(T̄) / log(T))
Where:
r= compression ratio achievedr̄= reference compression ratioT= time to compressT̄= reference timeα= scaling constant
#Why Both Matter
A naive approach might achieve 99% compression but take hours. A fast algorithm might run in milliseconds but barely compress anything. The Weissman Score captures the tradeoff.
#Our Results
| Algorithm | Ratio | Time | Weissman |
|---|---|---|---|
| gzip | 2.8x | 1.2s | 2.1 |
| bzip2 | 3.1x | 4.8s | 2.3 |
| Pied Piper | 5.2x | 0.8s | 5.2 |
The middle-out approach gives us both better compression and faster execution.
How we benchmarked these results
All benchmarks were run on identical hardware (32-core AMD EPYC, 128GB RAM) using the Canterbury Corpus dataset. Each algorithm was given the same input files and measured over 100 iterations to account for variance. The Weissman Score was calculated using the standard formula with gzip as the reference implementation.
#The Takeaway
Metrics matter. When evaluating any algorithm, make sure you're measuring what actually counts for your use case.
Share this post
Related posts
Middle-Out Compression Explained
The core insight behind our compression algorithm and why it works so well.
Distributed Consensus is Hard
Building PiperNet taught me why distributed consensus is one of the hardest problems in computer science.
Why We Rewrote Our Core in Rust
The story of migrating our compression engine from C to Rust, and what we learned.