Rewording several parts of the findings to add clarity.

This commit is contained in:
Jeremy Karst 2026-09-12 23:18:38 -04:00
parent 0fb7af77ce
commit db23d29c3f

View file

@ -2,15 +2,15 @@
Written 2026-09-12. Refer to [ergodic_sampling_test.py](ergodic_sampling_test.py) for analysis and figures.
## What was tested
## What the analysis tested
The analysis uses a 64 by 64 grid and keeps only 11.8% of the cells (484 samples), placed either on a
sparse grid (every third cell) or in the "ergodic" irregular pattern from the Ergodic Sampling paper
(found by minimizing equation 4). From those samples we rebuild the full grid with several methods
and measure how much the reconstruction differs from the ground truth. The value of merit chosen is
RMS error: the typical size of the difference between the reconstruction and the true
field at a cell, normalized such that 1.0 is 1 standard deviation (lower is better). Each RMS error
value shown is the average over 25 iterations with a freshly generated ground truth.
The analysis found here: https://code.karsttech.com/jeremy/ergodic-sampling, uses a 64 by 64 grid and
keeps only 11.8% of the cells (484 samples), placed either on a sparse grid (every third cell) or in
the "ergodic" irregular pattern from the Ergodic Sampling paper (found by minimizing equation 4).
From those samples we rebuild the full grid with several methods and measure how much the reconstruction
differs from the ground truth. The value of merit plotted is RMS error: the typical size of the difference
between the reconstruction and the true field at a cell, and normalized to be scale invariant.
Each RMS error value shown is the average over 25 iterations with a freshly generated ground truth.
The reconstruction methods:
@ -19,26 +19,26 @@ The reconstruction methods:
- **Compressive sensing**: the paper's method, iterative thresholding that assumes the signal is
built from a small number of components in some transform. It is run two ways: with a Fourier
transform (components are sinusoidal waves) and with the symlet wavelet from the paper.
- **Kriging**: an interpolation method from geostatistics that the paper does not use; it was added
as a strong reference. It estimates each empty cell as a weighted average of the samples, where
the weights come from a model of how similar the field tends to be at each separation distance.
That similarity model is itself estimated from the samples, so kriging adapts to the data: for a
smooth field it interpolates broadly, and for structureless data it learns that the samples say
nothing about their neighbors and backs off. It is also the standard gridding tool for gravity
and magnetic data in industry practice.
- **Kriging**: the interpolation method from geostatistics, not used in the paper; it was added
as a strong reference and a frequently used tool for modeling gravity and magnetic data. It
estimates each empty cell as a weighted average of the samples, where the weights come from a
model of how similar the field tends to be at each separation distance. The similarity model
is itself estimated from the samples, so kriging adapts to the data: for a smooth field it
interpolates broadly, and for structureless data it learns that the samples saynothing about
their neighbors and backs off.
The three kinds of ground truth:
- **White noise (control)**: every cell is an independent random number. There is no structure at
all, so nothing between the samples can actually be predicted. This serves as a good control
because it emphasizes any tendency of the reconstruction to hallucinate structure based on its
all, so nothing between the samples can be predicted. This serves as a good control because it
emphasizes any tendency of the reconstruction to hallucinate structure based on its
assumptions about the data.
- **Fourier-sparse**: a sum of 12 sinusoidal waves, half of them varying too quickly for the
sparse grid to follow. This is exactly the kind of signal compressive sensing is designed for,
and matches the paper's own demonstration signal.
- **Potential-field**: a smooth random map that mimics the paper's gravity survey data.
## The numbers
## Some summarized results
RMS error, mean plus or minus spread over 25 trials (best per column in bold):
@ -55,36 +55,36 @@ RMS error, mean plus or minus spread over 25 trials (best per column in bold):
![Figure 1](plt3.png)
*Figure 1. Reconstruction error broken down by sampling pattern + reconstruction formulation. Bar
color encodes the sampling pattern: blue for sparse grid, orange for ergodic. Bars are the mean
color shows the sampling pattern with blue for sparse grid and orange for ergodic. Bars are the mean
over 25 trials per truth class. The dashed line in the left panel marks the best score attainable
on structureless / random data, which is predicting zero at every unsampled cell (0.94).*
## What was found
## Some probably unsurprising results
**1. On truly random ground truth, ergodic + kriging came out best of everything tested.**
When the field has no structure, the smartest possible move is to admit it: output zero at every
unsampled cell, which scores 0.94 on our scale. Ergodic + kriging lands essentially on that mark
(1.001) because its similarity model, estimated from the samples, correctly concludes the samples
carry no information about their neighbors, so it barely interpolates at all. Every method with a
built-in belief about structure does worse than the "give up" answer: spline draws smooth hills
that do not exist (1.22 on the sparse grid, 1.44 on the ergodic pattern) and the symlet version
paints in wavelet texture (1.28). That gap, up to roughly 50% worse than guessing zero, is the
cost of hallucinated structure made concrete.
When the field has no structure, the smartest possible move is to admit it by outputting the mean
(zero) at every unsampled cell, which scores 0.94 on our scale. Ergodic + kriging lands essentially
on that mark (1.001) because its similarity model is estimated from the samples, and correctly
concludes the samples carry no information about their neighbors, so it barely interpolates at all.
As expected, every method with a built-in belief about structure does worse than the
guess-zero-everywhere answer: spline draws smooth hills that do not exist (1.22 on the sparse grid,
1.44 on the ergodic pattern) and the symlet version paints in wavelet texture (1.28). The
misprediction gap of up to ~50% worse than guessing zero is the cost of hallucinated
structure made concrete.
**2. On the potential-field ground truth, the plain sparse grid is the best pattern (figure 3).** This
field is smooth enough that every third cell is dense enough sampling, the situation classical
sampling theory covers. There the sparse grid wins simply on coverage: its farthest cell from
any sample is 1.41 cells away, while the ergodic pattern, being irregular, leaves gaps up to 3
cells. The ergodic pattern costs about 1.6 times the error here. That cost is the
**2. On the potential-field ground truth, the plain sparse grid is the best pattern (figure 3).**
This field is smooth enough that every third cell is dense enough sampling, the situation classical
sampling theory covers. Here the sparse grid wins simply on coverage. The ergodic pattern, being
irregular, leaves larger gaps which costs about 1.6 times the error margin. That cost is the
premium for insurance that pays out on the Fourier-sparse truth, where half the signal varies too
fast for the sparse grid: there the sparse grid garbles the fast waves into false slow ones
(this is aliasing, and is destructive; compare the sparse-grid and ergodic columns of
(this is traditional destructive aliasing, compare the sparse-grid and ergodic columns of
figure 2). THIS CASE IS MOSTLY THEORETICAL: sparse sampling's win here does not carry over to
real applications, for two reasons. First, the rough scale of variation is always a known prior
real applications for two reasons. First, the rough scale of variation is always a known prior
in practice (it is how any sample spacing gets chosen at all), but acting on that prior does not
lead to a comfortably dense regular grid: any scale at which we knew there were no spatial
frequencies beyond what uniform spacing resolves would be better served by sensing the same area
with fewer, ergodically spaced samples. The regular grid's win in this column is really a
lead to a comfortably dense regular grid: any scale at where we knew there were no spatial
frequencies smaller than what uniform spacing resolves would be better served by sensing the same
area with fewer, ergodically spaced samples. The regular grid's win in this column is really a
statement that its sample budget was larger than the known field scale required. Second, for
potential fields in particular, variation much finer than the survey scale reflects shallow
sources rather than the deep signal of interest, which is why this ground truth is generated
@ -115,17 +115,17 @@ Each column of the table is won by the method whose built-in assumption mirrors
kriging on the smooth random fields (its fitted similarity model actually recovers the
generator's parameters exactly), Fourier compressive sensing on the sum-of-waves signal, and
kriging again on noise because it alone can gracefully back off to predicting zero. Mismatches
fail hard: Fourier compressive sensing assumes a few dominant waves and scores 0.812 on the
smooth field, and the symlet version, which assumes the wrong kind of building block for a
are dramatically worse: Fourier compressive sensing assumes a few dominant waves and scores 0.812
on the smooth field, and the symlet version, which assumes the wrong kind of building block for a
sum-of-waves signal, reaches only 0.716 where the Fourier version is exact.
## Predicting the error without the ground truth
Everything above scores a reconstruction against a truth we happen to know. In a real survey the truth
is exactly what is missing, so the practical question is whether the reconstruction can say, by itself,
where it is likely to be wrong. Kriging answers that for free: alongside an estimate at every cell it
reports the standard deviation of that estimate, computed from the sample locations and from a
similarity model fitted to the sample values alone. No ground truth enters at any point.
Everything above scores a reconstruction against a truth we happen to know, but in a real survey the
the practical question is whether the reconstruction can say, by itself, where it is likely to be wrong.
Kriging answers that for free: alongside an estimate at every cell it reports the standard deviation of
that estimate, computed from the sample locations and from a similarity model fitted to the sample values
alone. No ground truth enters at any point.
Figures 4 to 7 test that predicted error map for both ground truth classes and for both the kriging and
the symlet compressive sensing reconstruction. Each figure has the same layout. The top row shows the
@ -134,15 +134,22 @@ standard deviation beside two things it should resemble: the error actually meas
truths drawn from the same generator and sampled with the same pattern, and the absolute error in the
single realization above. The bottom row shows the sampling pattern, a cell-by-cell scatter of predicted
against actual error, and a calibration curve in which cells are binned into deciles of the prediction.
Fixed axes are chosen so that the results are directly comparable from figure to figure, not just subplots.
Two cautions matter when reading these. First, one realization of an error is one draw of a random
variable, so even a perfect prediction correlates with it only up to a ceiling, which is printed on each
scatter panel; the comparison against the 64-truth ensemble is the decisive test, and there the
prediction tracks the actual error pattern with correlations of 0.90 to 0.99. Second, uniform ergodic
coverage intentionally flattens the predicted map, since the whole point of the pattern is to leave no
cell poorly covered, and that narrow range is what pushes the single-realization ceiling down. The
calibration curves run close to the one-to-one line in every case, slightly below it, meaning the
predicted magnitudes are roughly correct and not merely correctly ranked.
scatter panel. The comparison against the 64-truth ensemble is a test showing the prediction tracks the
actual error pattern with correlations of 0.90 to 0.99. Second, uniform ergodic coverage intentionally
flattens the predicted map, since the whole point of the pattern is to leave no cell poorly covered, and
that narrow range is what pushes the single-realization ceiling down. The calibration curves run close to
the one-to-one line in every case, slightly below it, meaning the predicted magnitudes are roughly correct.
**One takeaway here** is not the magnitude of the error mismatch between the single realization and the
predicted error map, but it's structure. If the error map is dominated by high frequency variations on the
scale of a discovered anomaly like an ore deposit, then smaller variations would be expected to average
out over the span of the contiguous region of the anomaly and thus a more precise estimate of the size and
extent of the anomaly could be gathered than by a simply measuring the reconstruction. This is because we
can leverage the structure of the ergodic sampling points and their implied error structure to reign in
the uncertainty in our anomaly predictions.
![Figure 4](plt2a1.png)