Rewording several parts of the findings to add clarity.
This commit is contained in:
parent
0fb7af77ce
commit
db23d29c3f
1 changed files with 58 additions and 51 deletions
109
findings.md
109
findings.md
|
|
@ -2,15 +2,15 @@
|
|||
|
||||
Written 2026-09-12. Refer to [ergodic_sampling_test.py](ergodic_sampling_test.py) for analysis and figures.
|
||||
|
||||
## What was tested
|
||||
## What the analysis tested
|
||||
|
||||
The analysis uses a 64 by 64 grid and keeps only 11.8% of the cells (484 samples), placed either on a
|
||||
sparse grid (every third cell) or in the "ergodic" irregular pattern from the Ergodic Sampling paper
|
||||
(found by minimizing equation 4). From those samples we rebuild the full grid with several methods
|
||||
and measure how much the reconstruction differs from the ground truth. The value of merit chosen is
|
||||
RMS error: the typical size of the difference between the reconstruction and the true
|
||||
field at a cell, normalized such that 1.0 is 1 standard deviation (lower is better). Each RMS error
|
||||
value shown is the average over 25 iterations with a freshly generated ground truth.
|
||||
The analysis found here: https://code.karsttech.com/jeremy/ergodic-sampling, uses a 64 by 64 grid and
|
||||
keeps only 11.8% of the cells (484 samples), placed either on a sparse grid (every third cell) or in
|
||||
the "ergodic" irregular pattern from the Ergodic Sampling paper (found by minimizing equation 4).
|
||||
From those samples we rebuild the full grid with several methods and measure how much the reconstruction
|
||||
differs from the ground truth. The value of merit plotted is RMS error: the typical size of the difference
|
||||
between the reconstruction and the true field at a cell, and normalized to be scale invariant.
|
||||
Each RMS error value shown is the average over 25 iterations with a freshly generated ground truth.
|
||||
|
||||
The reconstruction methods:
|
||||
|
||||
|
|
@ -19,26 +19,26 @@ The reconstruction methods:
|
|||
- **Compressive sensing**: the paper's method, iterative thresholding that assumes the signal is
|
||||
built from a small number of components in some transform. It is run two ways: with a Fourier
|
||||
transform (components are sinusoidal waves) and with the symlet wavelet from the paper.
|
||||
- **Kriging**: an interpolation method from geostatistics that the paper does not use; it was added
|
||||
as a strong reference. It estimates each empty cell as a weighted average of the samples, where
|
||||
the weights come from a model of how similar the field tends to be at each separation distance.
|
||||
That similarity model is itself estimated from the samples, so kriging adapts to the data: for a
|
||||
smooth field it interpolates broadly, and for structureless data it learns that the samples say
|
||||
nothing about their neighbors and backs off. It is also the standard gridding tool for gravity
|
||||
and magnetic data in industry practice.
|
||||
- **Kriging**: the interpolation method from geostatistics, not used in the paper; it was added
|
||||
as a strong reference and a frequently used tool for modeling gravity and magnetic data. It
|
||||
estimates each empty cell as a weighted average of the samples, where the weights come from a
|
||||
model of how similar the field tends to be at each separation distance. The similarity model
|
||||
is itself estimated from the samples, so kriging adapts to the data: for a smooth field it
|
||||
interpolates broadly, and for structureless data it learns that the samples saynothing about
|
||||
their neighbors and backs off.
|
||||
|
||||
The three kinds of ground truth:
|
||||
|
||||
- **White noise (control)**: every cell is an independent random number. There is no structure at
|
||||
all, so nothing between the samples can actually be predicted. This serves as a good control
|
||||
because it emphasizes any tendency of the reconstruction to hallucinate structure based on its
|
||||
all, so nothing between the samples can be predicted. This serves as a good control because it
|
||||
emphasizes any tendency of the reconstruction to hallucinate structure based on its
|
||||
assumptions about the data.
|
||||
- **Fourier-sparse**: a sum of 12 sinusoidal waves, half of them varying too quickly for the
|
||||
sparse grid to follow. This is exactly the kind of signal compressive sensing is designed for,
|
||||
and matches the paper's own demonstration signal.
|
||||
- **Potential-field**: a smooth random map that mimics the paper's gravity survey data.
|
||||
|
||||
## The numbers
|
||||
## Some summarized results
|
||||
|
||||
RMS error, mean plus or minus spread over 25 trials (best per column in bold):
|
||||
|
||||
|
|
@ -55,36 +55,36 @@ RMS error, mean plus or minus spread over 25 trials (best per column in bold):
|
|||

|
||||
|
||||
*Figure 1. Reconstruction error broken down by sampling pattern + reconstruction formulation. Bar
|
||||
color encodes the sampling pattern: blue for sparse grid, orange for ergodic. Bars are the mean
|
||||
color shows the sampling pattern with blue for sparse grid and orange for ergodic. Bars are the mean
|
||||
over 25 trials per truth class. The dashed line in the left panel marks the best score attainable
|
||||
on structureless / random data, which is predicting zero at every unsampled cell (0.94).*
|
||||
|
||||
## What was found
|
||||
## Some probably unsurprising results
|
||||
|
||||
**1. On truly random ground truth, ergodic + kriging came out best of everything tested.**
|
||||
When the field has no structure, the smartest possible move is to admit it: output zero at every
|
||||
unsampled cell, which scores 0.94 on our scale. Ergodic + kriging lands essentially on that mark
|
||||
(1.001) because its similarity model, estimated from the samples, correctly concludes the samples
|
||||
carry no information about their neighbors, so it barely interpolates at all. Every method with a
|
||||
built-in belief about structure does worse than the "give up" answer: spline draws smooth hills
|
||||
that do not exist (1.22 on the sparse grid, 1.44 on the ergodic pattern) and the symlet version
|
||||
paints in wavelet texture (1.28). That gap, up to roughly 50% worse than guessing zero, is the
|
||||
cost of hallucinated structure made concrete.
|
||||
When the field has no structure, the smartest possible move is to admit it by outputting the mean
|
||||
(zero) at every unsampled cell, which scores 0.94 on our scale. Ergodic + kriging lands essentially
|
||||
on that mark (1.001) because its similarity model is estimated from the samples, and correctly
|
||||
concludes the samples carry no information about their neighbors, so it barely interpolates at all.
|
||||
As expected, every method with a built-in belief about structure does worse than the
|
||||
guess-zero-everywhere answer: spline draws smooth hills that do not exist (1.22 on the sparse grid,
|
||||
1.44 on the ergodic pattern) and the symlet version paints in wavelet texture (1.28). The
|
||||
misprediction gap of up to ~50% worse than guessing zero is the cost of hallucinated
|
||||
structure made concrete.
|
||||
|
||||
**2. On the potential-field ground truth, the plain sparse grid is the best pattern (figure 3).** This
|
||||
field is smooth enough that every third cell is dense enough sampling, the situation classical
|
||||
sampling theory covers. There the sparse grid wins simply on coverage: its farthest cell from
|
||||
any sample is 1.41 cells away, while the ergodic pattern, being irregular, leaves gaps up to 3
|
||||
cells. The ergodic pattern costs about 1.6 times the error here. That cost is the
|
||||
**2. On the potential-field ground truth, the plain sparse grid is the best pattern (figure 3).**
|
||||
This field is smooth enough that every third cell is dense enough sampling, the situation classical
|
||||
sampling theory covers. Here the sparse grid wins simply on coverage. The ergodic pattern, being
|
||||
irregular, leaves larger gaps which costs about 1.6 times the error margin. That cost is the
|
||||
premium for insurance that pays out on the Fourier-sparse truth, where half the signal varies too
|
||||
fast for the sparse grid: there the sparse grid garbles the fast waves into false slow ones
|
||||
(this is aliasing, and is destructive; compare the sparse-grid and ergodic columns of
|
||||
(this is traditional destructive aliasing, compare the sparse-grid and ergodic columns of
|
||||
figure 2). THIS CASE IS MOSTLY THEORETICAL: sparse sampling's win here does not carry over to
|
||||
real applications, for two reasons. First, the rough scale of variation is always a known prior
|
||||
real applications for two reasons. First, the rough scale of variation is always a known prior
|
||||
in practice (it is how any sample spacing gets chosen at all), but acting on that prior does not
|
||||
lead to a comfortably dense regular grid: any scale at which we knew there were no spatial
|
||||
frequencies beyond what uniform spacing resolves would be better served by sensing the same area
|
||||
with fewer, ergodically spaced samples. The regular grid's win in this column is really a
|
||||
lead to a comfortably dense regular grid: any scale at where we knew there were no spatial
|
||||
frequencies smaller than what uniform spacing resolves would be better served by sensing the same
|
||||
area with fewer, ergodically spaced samples. The regular grid's win in this column is really a
|
||||
statement that its sample budget was larger than the known field scale required. Second, for
|
||||
potential fields in particular, variation much finer than the survey scale reflects shallow
|
||||
sources rather than the deep signal of interest, which is why this ground truth is generated
|
||||
|
|
@ -115,17 +115,17 @@ Each column of the table is won by the method whose built-in assumption mirrors
|
|||
kriging on the smooth random fields (its fitted similarity model actually recovers the
|
||||
generator's parameters exactly), Fourier compressive sensing on the sum-of-waves signal, and
|
||||
kriging again on noise because it alone can gracefully back off to predicting zero. Mismatches
|
||||
fail hard: Fourier compressive sensing assumes a few dominant waves and scores 0.812 on the
|
||||
smooth field, and the symlet version, which assumes the wrong kind of building block for a
|
||||
are dramatically worse: Fourier compressive sensing assumes a few dominant waves and scores 0.812
|
||||
on the smooth field, and the symlet version, which assumes the wrong kind of building block for a
|
||||
sum-of-waves signal, reaches only 0.716 where the Fourier version is exact.
|
||||
|
||||
## Predicting the error without the ground truth
|
||||
|
||||
Everything above scores a reconstruction against a truth we happen to know. In a real survey the truth
|
||||
is exactly what is missing, so the practical question is whether the reconstruction can say, by itself,
|
||||
where it is likely to be wrong. Kriging answers that for free: alongside an estimate at every cell it
|
||||
reports the standard deviation of that estimate, computed from the sample locations and from a
|
||||
similarity model fitted to the sample values alone. No ground truth enters at any point.
|
||||
Everything above scores a reconstruction against a truth we happen to know, but in a real survey the
|
||||
the practical question is whether the reconstruction can say, by itself, where it is likely to be wrong.
|
||||
Kriging answers that for free: alongside an estimate at every cell it reports the standard deviation of
|
||||
that estimate, computed from the sample locations and from a similarity model fitted to the sample values
|
||||
alone. No ground truth enters at any point.
|
||||
|
||||
Figures 4 to 7 test that predicted error map for both ground truth classes and for both the kriging and
|
||||
the symlet compressive sensing reconstruction. Each figure has the same layout. The top row shows the
|
||||
|
|
@ -134,15 +134,22 @@ standard deviation beside two things it should resemble: the error actually meas
|
|||
truths drawn from the same generator and sampled with the same pattern, and the absolute error in the
|
||||
single realization above. The bottom row shows the sampling pattern, a cell-by-cell scatter of predicted
|
||||
against actual error, and a calibration curve in which cells are binned into deciles of the prediction.
|
||||
Fixed axes are chosen so that the results are directly comparable from figure to figure, not just subplots.
|
||||
|
||||
Two cautions matter when reading these. First, one realization of an error is one draw of a random
|
||||
variable, so even a perfect prediction correlates with it only up to a ceiling, which is printed on each
|
||||
scatter panel; the comparison against the 64-truth ensemble is the decisive test, and there the
|
||||
prediction tracks the actual error pattern with correlations of 0.90 to 0.99. Second, uniform ergodic
|
||||
coverage intentionally flattens the predicted map, since the whole point of the pattern is to leave no
|
||||
cell poorly covered, and that narrow range is what pushes the single-realization ceiling down. The
|
||||
calibration curves run close to the one-to-one line in every case, slightly below it, meaning the
|
||||
predicted magnitudes are roughly correct and not merely correctly ranked.
|
||||
scatter panel. The comparison against the 64-truth ensemble is a test showing the prediction tracks the
|
||||
actual error pattern with correlations of 0.90 to 0.99. Second, uniform ergodic coverage intentionally
|
||||
flattens the predicted map, since the whole point of the pattern is to leave no cell poorly covered, and
|
||||
that narrow range is what pushes the single-realization ceiling down. The calibration curves run close to
|
||||
the one-to-one line in every case, slightly below it, meaning the predicted magnitudes are roughly correct.
|
||||
**One takeaway here** is not the magnitude of the error mismatch between the single realization and the
|
||||
predicted error map, but it's structure. If the error map is dominated by high frequency variations on the
|
||||
scale of a discovered anomaly like an ore deposit, then smaller variations would be expected to average
|
||||
out over the span of the contiguous region of the anomaly and thus a more precise estimate of the size and
|
||||
extent of the anomaly could be gathered than by a simply measuring the reconstruction. This is because we
|
||||
can leverage the structure of the ergodic sampling points and their implied error structure to reign in
|
||||
the uncertainty in our anomaly predictions.
|
||||
|
||||

|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue