Rewording several parts of the findings to add clarity.
This commit is contained in:
parent
0fb7af77ce
commit
db23d29c3f
1 changed files with 58 additions and 51 deletions
109
findings.md
109
findings.md
|
|
@ -2,15 +2,15 @@
|
||||||
|
|
||||||
Written 2026-09-12. Refer to [ergodic_sampling_test.py](ergodic_sampling_test.py) for analysis and figures.
|
Written 2026-09-12. Refer to [ergodic_sampling_test.py](ergodic_sampling_test.py) for analysis and figures.
|
||||||
|
|
||||||
## What was tested
|
## What the analysis tested
|
||||||
|
|
||||||
The analysis uses a 64 by 64 grid and keeps only 11.8% of the cells (484 samples), placed either on a
|
The analysis found here: https://code.karsttech.com/jeremy/ergodic-sampling, uses a 64 by 64 grid and
|
||||||
sparse grid (every third cell) or in the "ergodic" irregular pattern from the Ergodic Sampling paper
|
keeps only 11.8% of the cells (484 samples), placed either on a sparse grid (every third cell) or in
|
||||||
(found by minimizing equation 4). From those samples we rebuild the full grid with several methods
|
the "ergodic" irregular pattern from the Ergodic Sampling paper (found by minimizing equation 4).
|
||||||
and measure how much the reconstruction differs from the ground truth. The value of merit chosen is
|
From those samples we rebuild the full grid with several methods and measure how much the reconstruction
|
||||||
RMS error: the typical size of the difference between the reconstruction and the true
|
differs from the ground truth. The value of merit plotted is RMS error: the typical size of the difference
|
||||||
field at a cell, normalized such that 1.0 is 1 standard deviation (lower is better). Each RMS error
|
between the reconstruction and the true field at a cell, and normalized to be scale invariant.
|
||||||
value shown is the average over 25 iterations with a freshly generated ground truth.
|
Each RMS error value shown is the average over 25 iterations with a freshly generated ground truth.
|
||||||
|
|
||||||
The reconstruction methods:
|
The reconstruction methods:
|
||||||
|
|
||||||
|
|
@ -19,26 +19,26 @@ The reconstruction methods:
|
||||||
- **Compressive sensing**: the paper's method, iterative thresholding that assumes the signal is
|
- **Compressive sensing**: the paper's method, iterative thresholding that assumes the signal is
|
||||||
built from a small number of components in some transform. It is run two ways: with a Fourier
|
built from a small number of components in some transform. It is run two ways: with a Fourier
|
||||||
transform (components are sinusoidal waves) and with the symlet wavelet from the paper.
|
transform (components are sinusoidal waves) and with the symlet wavelet from the paper.
|
||||||
- **Kriging**: an interpolation method from geostatistics that the paper does not use; it was added
|
- **Kriging**: the interpolation method from geostatistics, not used in the paper; it was added
|
||||||
as a strong reference. It estimates each empty cell as a weighted average of the samples, where
|
as a strong reference and a frequently used tool for modeling gravity and magnetic data. It
|
||||||
the weights come from a model of how similar the field tends to be at each separation distance.
|
estimates each empty cell as a weighted average of the samples, where the weights come from a
|
||||||
That similarity model is itself estimated from the samples, so kriging adapts to the data: for a
|
model of how similar the field tends to be at each separation distance. The similarity model
|
||||||
smooth field it interpolates broadly, and for structureless data it learns that the samples say
|
is itself estimated from the samples, so kriging adapts to the data: for a smooth field it
|
||||||
nothing about their neighbors and backs off. It is also the standard gridding tool for gravity
|
interpolates broadly, and for structureless data it learns that the samples saynothing about
|
||||||
and magnetic data in industry practice.
|
their neighbors and backs off.
|
||||||
|
|
||||||
The three kinds of ground truth:
|
The three kinds of ground truth:
|
||||||
|
|
||||||
- **White noise (control)**: every cell is an independent random number. There is no structure at
|
- **White noise (control)**: every cell is an independent random number. There is no structure at
|
||||||
all, so nothing between the samples can actually be predicted. This serves as a good control
|
all, so nothing between the samples can be predicted. This serves as a good control because it
|
||||||
because it emphasizes any tendency of the reconstruction to hallucinate structure based on its
|
emphasizes any tendency of the reconstruction to hallucinate structure based on its
|
||||||
assumptions about the data.
|
assumptions about the data.
|
||||||
- **Fourier-sparse**: a sum of 12 sinusoidal waves, half of them varying too quickly for the
|
- **Fourier-sparse**: a sum of 12 sinusoidal waves, half of them varying too quickly for the
|
||||||
sparse grid to follow. This is exactly the kind of signal compressive sensing is designed for,
|
sparse grid to follow. This is exactly the kind of signal compressive sensing is designed for,
|
||||||
and matches the paper's own demonstration signal.
|
and matches the paper's own demonstration signal.
|
||||||
- **Potential-field**: a smooth random map that mimics the paper's gravity survey data.
|
- **Potential-field**: a smooth random map that mimics the paper's gravity survey data.
|
||||||
|
|
||||||
## The numbers
|
## Some summarized results
|
||||||
|
|
||||||
RMS error, mean plus or minus spread over 25 trials (best per column in bold):
|
RMS error, mean plus or minus spread over 25 trials (best per column in bold):
|
||||||
|
|
||||||
|
|
@ -55,36 +55,36 @@ RMS error, mean plus or minus spread over 25 trials (best per column in bold):
|
||||||

|

|
||||||
|
|
||||||
*Figure 1. Reconstruction error broken down by sampling pattern + reconstruction formulation. Bar
|
*Figure 1. Reconstruction error broken down by sampling pattern + reconstruction formulation. Bar
|
||||||
color encodes the sampling pattern: blue for sparse grid, orange for ergodic. Bars are the mean
|
color shows the sampling pattern with blue for sparse grid and orange for ergodic. Bars are the mean
|
||||||
over 25 trials per truth class. The dashed line in the left panel marks the best score attainable
|
over 25 trials per truth class. The dashed line in the left panel marks the best score attainable
|
||||||
on structureless / random data, which is predicting zero at every unsampled cell (0.94).*
|
on structureless / random data, which is predicting zero at every unsampled cell (0.94).*
|
||||||
|
|
||||||
## What was found
|
## Some probably unsurprising results
|
||||||
|
|
||||||
**1. On truly random ground truth, ergodic + kriging came out best of everything tested.**
|
**1. On truly random ground truth, ergodic + kriging came out best of everything tested.**
|
||||||
When the field has no structure, the smartest possible move is to admit it: output zero at every
|
When the field has no structure, the smartest possible move is to admit it by outputting the mean
|
||||||
unsampled cell, which scores 0.94 on our scale. Ergodic + kriging lands essentially on that mark
|
(zero) at every unsampled cell, which scores 0.94 on our scale. Ergodic + kriging lands essentially
|
||||||
(1.001) because its similarity model, estimated from the samples, correctly concludes the samples
|
on that mark (1.001) because its similarity model is estimated from the samples, and correctly
|
||||||
carry no information about their neighbors, so it barely interpolates at all. Every method with a
|
concludes the samples carry no information about their neighbors, so it barely interpolates at all.
|
||||||
built-in belief about structure does worse than the "give up" answer: spline draws smooth hills
|
As expected, every method with a built-in belief about structure does worse than the
|
||||||
that do not exist (1.22 on the sparse grid, 1.44 on the ergodic pattern) and the symlet version
|
guess-zero-everywhere answer: spline draws smooth hills that do not exist (1.22 on the sparse grid,
|
||||||
paints in wavelet texture (1.28). That gap, up to roughly 50% worse than guessing zero, is the
|
1.44 on the ergodic pattern) and the symlet version paints in wavelet texture (1.28). The
|
||||||
cost of hallucinated structure made concrete.
|
misprediction gap of up to ~50% worse than guessing zero is the cost of hallucinated
|
||||||
|
structure made concrete.
|
||||||
|
|
||||||
**2. On the potential-field ground truth, the plain sparse grid is the best pattern (figure 3).** This
|
**2. On the potential-field ground truth, the plain sparse grid is the best pattern (figure 3).**
|
||||||
field is smooth enough that every third cell is dense enough sampling, the situation classical
|
This field is smooth enough that every third cell is dense enough sampling, the situation classical
|
||||||
sampling theory covers. There the sparse grid wins simply on coverage: its farthest cell from
|
sampling theory covers. Here the sparse grid wins simply on coverage. The ergodic pattern, being
|
||||||
any sample is 1.41 cells away, while the ergodic pattern, being irregular, leaves gaps up to 3
|
irregular, leaves larger gaps which costs about 1.6 times the error margin. That cost is the
|
||||||
cells. The ergodic pattern costs about 1.6 times the error here. That cost is the
|
|
||||||
premium for insurance that pays out on the Fourier-sparse truth, where half the signal varies too
|
premium for insurance that pays out on the Fourier-sparse truth, where half the signal varies too
|
||||||
fast for the sparse grid: there the sparse grid garbles the fast waves into false slow ones
|
fast for the sparse grid: there the sparse grid garbles the fast waves into false slow ones
|
||||||
(this is aliasing, and is destructive; compare the sparse-grid and ergodic columns of
|
(this is traditional destructive aliasing, compare the sparse-grid and ergodic columns of
|
||||||
figure 2). THIS CASE IS MOSTLY THEORETICAL: sparse sampling's win here does not carry over to
|
figure 2). THIS CASE IS MOSTLY THEORETICAL: sparse sampling's win here does not carry over to
|
||||||
real applications, for two reasons. First, the rough scale of variation is always a known prior
|
real applications for two reasons. First, the rough scale of variation is always a known prior
|
||||||
in practice (it is how any sample spacing gets chosen at all), but acting on that prior does not
|
in practice (it is how any sample spacing gets chosen at all), but acting on that prior does not
|
||||||
lead to a comfortably dense regular grid: any scale at which we knew there were no spatial
|
lead to a comfortably dense regular grid: any scale at where we knew there were no spatial
|
||||||
frequencies beyond what uniform spacing resolves would be better served by sensing the same area
|
frequencies smaller than what uniform spacing resolves would be better served by sensing the same
|
||||||
with fewer, ergodically spaced samples. The regular grid's win in this column is really a
|
area with fewer, ergodically spaced samples. The regular grid's win in this column is really a
|
||||||
statement that its sample budget was larger than the known field scale required. Second, for
|
statement that its sample budget was larger than the known field scale required. Second, for
|
||||||
potential fields in particular, variation much finer than the survey scale reflects shallow
|
potential fields in particular, variation much finer than the survey scale reflects shallow
|
||||||
sources rather than the deep signal of interest, which is why this ground truth is generated
|
sources rather than the deep signal of interest, which is why this ground truth is generated
|
||||||
|
|
@ -115,17 +115,17 @@ Each column of the table is won by the method whose built-in assumption mirrors
|
||||||
kriging on the smooth random fields (its fitted similarity model actually recovers the
|
kriging on the smooth random fields (its fitted similarity model actually recovers the
|
||||||
generator's parameters exactly), Fourier compressive sensing on the sum-of-waves signal, and
|
generator's parameters exactly), Fourier compressive sensing on the sum-of-waves signal, and
|
||||||
kriging again on noise because it alone can gracefully back off to predicting zero. Mismatches
|
kriging again on noise because it alone can gracefully back off to predicting zero. Mismatches
|
||||||
fail hard: Fourier compressive sensing assumes a few dominant waves and scores 0.812 on the
|
are dramatically worse: Fourier compressive sensing assumes a few dominant waves and scores 0.812
|
||||||
smooth field, and the symlet version, which assumes the wrong kind of building block for a
|
on the smooth field, and the symlet version, which assumes the wrong kind of building block for a
|
||||||
sum-of-waves signal, reaches only 0.716 where the Fourier version is exact.
|
sum-of-waves signal, reaches only 0.716 where the Fourier version is exact.
|
||||||
|
|
||||||
## Predicting the error without the ground truth
|
## Predicting the error without the ground truth
|
||||||
|
|
||||||
Everything above scores a reconstruction against a truth we happen to know. In a real survey the truth
|
Everything above scores a reconstruction against a truth we happen to know, but in a real survey the
|
||||||
is exactly what is missing, so the practical question is whether the reconstruction can say, by itself,
|
the practical question is whether the reconstruction can say, by itself, where it is likely to be wrong.
|
||||||
where it is likely to be wrong. Kriging answers that for free: alongside an estimate at every cell it
|
Kriging answers that for free: alongside an estimate at every cell it reports the standard deviation of
|
||||||
reports the standard deviation of that estimate, computed from the sample locations and from a
|
that estimate, computed from the sample locations and from a similarity model fitted to the sample values
|
||||||
similarity model fitted to the sample values alone. No ground truth enters at any point.
|
alone. No ground truth enters at any point.
|
||||||
|
|
||||||
Figures 4 to 7 test that predicted error map for both ground truth classes and for both the kriging and
|
Figures 4 to 7 test that predicted error map for both ground truth classes and for both the kriging and
|
||||||
the symlet compressive sensing reconstruction. Each figure has the same layout. The top row shows the
|
the symlet compressive sensing reconstruction. Each figure has the same layout. The top row shows the
|
||||||
|
|
@ -134,15 +134,22 @@ standard deviation beside two things it should resemble: the error actually meas
|
||||||
truths drawn from the same generator and sampled with the same pattern, and the absolute error in the
|
truths drawn from the same generator and sampled with the same pattern, and the absolute error in the
|
||||||
single realization above. The bottom row shows the sampling pattern, a cell-by-cell scatter of predicted
|
single realization above. The bottom row shows the sampling pattern, a cell-by-cell scatter of predicted
|
||||||
against actual error, and a calibration curve in which cells are binned into deciles of the prediction.
|
against actual error, and a calibration curve in which cells are binned into deciles of the prediction.
|
||||||
|
Fixed axes are chosen so that the results are directly comparable from figure to figure, not just subplots.
|
||||||
|
|
||||||
Two cautions matter when reading these. First, one realization of an error is one draw of a random
|
Two cautions matter when reading these. First, one realization of an error is one draw of a random
|
||||||
variable, so even a perfect prediction correlates with it only up to a ceiling, which is printed on each
|
variable, so even a perfect prediction correlates with it only up to a ceiling, which is printed on each
|
||||||
scatter panel; the comparison against the 64-truth ensemble is the decisive test, and there the
|
scatter panel. The comparison against the 64-truth ensemble is a test showing the prediction tracks the
|
||||||
prediction tracks the actual error pattern with correlations of 0.90 to 0.99. Second, uniform ergodic
|
actual error pattern with correlations of 0.90 to 0.99. Second, uniform ergodic coverage intentionally
|
||||||
coverage intentionally flattens the predicted map, since the whole point of the pattern is to leave no
|
flattens the predicted map, since the whole point of the pattern is to leave no cell poorly covered, and
|
||||||
cell poorly covered, and that narrow range is what pushes the single-realization ceiling down. The
|
that narrow range is what pushes the single-realization ceiling down. The calibration curves run close to
|
||||||
calibration curves run close to the one-to-one line in every case, slightly below it, meaning the
|
the one-to-one line in every case, slightly below it, meaning the predicted magnitudes are roughly correct.
|
||||||
predicted magnitudes are roughly correct and not merely correctly ranked.
|
**One takeaway here** is not the magnitude of the error mismatch between the single realization and the
|
||||||
|
predicted error map, but it's structure. If the error map is dominated by high frequency variations on the
|
||||||
|
scale of a discovered anomaly like an ore deposit, then smaller variations would be expected to average
|
||||||
|
out over the span of the contiguous region of the anomaly and thus a more precise estimate of the size and
|
||||||
|
extent of the anomaly could be gathered than by a simply measuring the reconstruction. This is because we
|
||||||
|
can leverage the structure of the ergodic sampling points and their implied error structure to reign in
|
||||||
|
the uncertainty in our anomaly predictions.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue