diff --git a/findings.md b/findings.md index c3c70d7..5b9f7c2 100644 --- a/findings.md +++ b/findings.md @@ -2,15 +2,15 @@ Written 2026-09-12. Refer to [ergodic_sampling_test.py](ergodic_sampling_test.py) for analysis and figures. -## What was tested +## What the analysis tested -The analysis uses a 64 by 64 grid and keeps only 11.8% of the cells (484 samples), placed either on a -sparse grid (every third cell) or in the "ergodic" irregular pattern from the Ergodic Sampling paper -(found by minimizing equation 4). From those samples we rebuild the full grid with several methods -and measure how much the reconstruction differs from the ground truth. The value of merit chosen is -RMS error: the typical size of the difference between the reconstruction and the true -field at a cell, normalized such that 1.0 is 1 standard deviation (lower is better). Each RMS error -value shown is the average over 25 iterations with a freshly generated ground truth. +The analysis found here: https://code.karsttech.com/jeremy/ergodic-sampling, uses a 64 by 64 grid and +keeps only 11.8% of the cells (484 samples), placed either on a sparse grid (every third cell) or in +the "ergodic" irregular pattern from the Ergodic Sampling paper (found by minimizing equation 4). +From those samples we rebuild the full grid with several methods and measure how much the reconstruction +differs from the ground truth. The value of merit plotted is RMS error: the typical size of the difference +between the reconstruction and the true field at a cell, and normalized to be scale invariant. +Each RMS error value shown is the average over 25 iterations with a freshly generated ground truth. The reconstruction methods: @@ -19,26 +19,26 @@ The reconstruction methods: - **Compressive sensing**: the paper's method, iterative thresholding that assumes the signal is built from a small number of components in some transform. It is run two ways: with a Fourier transform (components are sinusoidal waves) and with the symlet wavelet from the paper. -- **Kriging**: an interpolation method from geostatistics that the paper does not use; it was added - as a strong reference. It estimates each empty cell as a weighted average of the samples, where - the weights come from a model of how similar the field tends to be at each separation distance. - That similarity model is itself estimated from the samples, so kriging adapts to the data: for a - smooth field it interpolates broadly, and for structureless data it learns that the samples say - nothing about their neighbors and backs off. It is also the standard gridding tool for gravity - and magnetic data in industry practice. +- **Kriging**: the interpolation method from geostatistics, not used in the paper; it was added + as a strong reference and a frequently used tool for modeling gravity and magnetic data. It + estimates each empty cell as a weighted average of the samples, where the weights come from a + model of how similar the field tends to be at each separation distance. The similarity model + is itself estimated from the samples, so kriging adapts to the data: for a smooth field it + interpolates broadly, and for structureless data it learns that the samples saynothing about + their neighbors and backs off. The three kinds of ground truth: - **White noise (control)**: every cell is an independent random number. There is no structure at - all, so nothing between the samples can actually be predicted. This serves as a good control - because it emphasizes any tendency of the reconstruction to hallucinate structure based on its + all, so nothing between the samples can be predicted. This serves as a good control because it + emphasizes any tendency of the reconstruction to hallucinate structure based on its assumptions about the data. - **Fourier-sparse**: a sum of 12 sinusoidal waves, half of them varying too quickly for the sparse grid to follow. This is exactly the kind of signal compressive sensing is designed for, and matches the paper's own demonstration signal. - **Potential-field**: a smooth random map that mimics the paper's gravity survey data. -## The numbers +## Some summarized results RMS error, mean plus or minus spread over 25 trials (best per column in bold): @@ -55,36 +55,36 @@ RMS error, mean plus or minus spread over 25 trials (best per column in bold): ![Figure 1](plt3.png) *Figure 1. Reconstruction error broken down by sampling pattern + reconstruction formulation. Bar -color encodes the sampling pattern: blue for sparse grid, orange for ergodic. Bars are the mean +color shows the sampling pattern with blue for sparse grid and orange for ergodic. Bars are the mean over 25 trials per truth class. The dashed line in the left panel marks the best score attainable on structureless / random data, which is predicting zero at every unsampled cell (0.94).* -## What was found +## Some probably unsurprising results **1. On truly random ground truth, ergodic + kriging came out best of everything tested.** -When the field has no structure, the smartest possible move is to admit it: output zero at every -unsampled cell, which scores 0.94 on our scale. Ergodic + kriging lands essentially on that mark -(1.001) because its similarity model, estimated from the samples, correctly concludes the samples -carry no information about their neighbors, so it barely interpolates at all. Every method with a -built-in belief about structure does worse than the "give up" answer: spline draws smooth hills -that do not exist (1.22 on the sparse grid, 1.44 on the ergodic pattern) and the symlet version -paints in wavelet texture (1.28). That gap, up to roughly 50% worse than guessing zero, is the -cost of hallucinated structure made concrete. +When the field has no structure, the smartest possible move is to admit it by outputting the mean +(zero) at every unsampled cell, which scores 0.94 on our scale. Ergodic + kriging lands essentially +on that mark (1.001) because its similarity model is estimated from the samples, and correctly +concludes the samples carry no information about their neighbors, so it barely interpolates at all. +As expected, every method with a built-in belief about structure does worse than the +guess-zero-everywhere answer: spline draws smooth hills that do not exist (1.22 on the sparse grid, +1.44 on the ergodic pattern) and the symlet version paints in wavelet texture (1.28). The +misprediction gap of up to ~50% worse than guessing zero is the cost of hallucinated +structure made concrete. -**2. On the potential-field ground truth, the plain sparse grid is the best pattern (figure 3).** This -field is smooth enough that every third cell is dense enough sampling, the situation classical -sampling theory covers. There the sparse grid wins simply on coverage: its farthest cell from -any sample is 1.41 cells away, while the ergodic pattern, being irregular, leaves gaps up to 3 -cells. The ergodic pattern costs about 1.6 times the error here. That cost is the +**2. On the potential-field ground truth, the plain sparse grid is the best pattern (figure 3).** +This field is smooth enough that every third cell is dense enough sampling, the situation classical +sampling theory covers. Here the sparse grid wins simply on coverage. The ergodic pattern, being +irregular, leaves larger gaps which costs about 1.6 times the error margin. That cost is the premium for insurance that pays out on the Fourier-sparse truth, where half the signal varies too fast for the sparse grid: there the sparse grid garbles the fast waves into false slow ones -(this is aliasing, and is destructive; compare the sparse-grid and ergodic columns of +(this is traditional destructive aliasing, compare the sparse-grid and ergodic columns of figure 2). THIS CASE IS MOSTLY THEORETICAL: sparse sampling's win here does not carry over to -real applications, for two reasons. First, the rough scale of variation is always a known prior +real applications for two reasons. First, the rough scale of variation is always a known prior in practice (it is how any sample spacing gets chosen at all), but acting on that prior does not -lead to a comfortably dense regular grid: any scale at which we knew there were no spatial -frequencies beyond what uniform spacing resolves would be better served by sensing the same area -with fewer, ergodically spaced samples. The regular grid's win in this column is really a +lead to a comfortably dense regular grid: any scale at where we knew there were no spatial +frequencies smaller than what uniform spacing resolves would be better served by sensing the same +area with fewer, ergodically spaced samples. The regular grid's win in this column is really a statement that its sample budget was larger than the known field scale required. Second, for potential fields in particular, variation much finer than the survey scale reflects shallow sources rather than the deep signal of interest, which is why this ground truth is generated @@ -115,17 +115,17 @@ Each column of the table is won by the method whose built-in assumption mirrors kriging on the smooth random fields (its fitted similarity model actually recovers the generator's parameters exactly), Fourier compressive sensing on the sum-of-waves signal, and kriging again on noise because it alone can gracefully back off to predicting zero. Mismatches -fail hard: Fourier compressive sensing assumes a few dominant waves and scores 0.812 on the -smooth field, and the symlet version, which assumes the wrong kind of building block for a +are dramatically worse: Fourier compressive sensing assumes a few dominant waves and scores 0.812 +on the smooth field, and the symlet version, which assumes the wrong kind of building block for a sum-of-waves signal, reaches only 0.716 where the Fourier version is exact. ## Predicting the error without the ground truth -Everything above scores a reconstruction against a truth we happen to know. In a real survey the truth -is exactly what is missing, so the practical question is whether the reconstruction can say, by itself, -where it is likely to be wrong. Kriging answers that for free: alongside an estimate at every cell it -reports the standard deviation of that estimate, computed from the sample locations and from a -similarity model fitted to the sample values alone. No ground truth enters at any point. +Everything above scores a reconstruction against a truth we happen to know, but in a real survey the +the practical question is whether the reconstruction can say, by itself, where it is likely to be wrong. +Kriging answers that for free: alongside an estimate at every cell it reports the standard deviation of +that estimate, computed from the sample locations and from a similarity model fitted to the sample values +alone. No ground truth enters at any point. Figures 4 to 7 test that predicted error map for both ground truth classes and for both the kriging and the symlet compressive sensing reconstruction. Each figure has the same layout. The top row shows the @@ -134,15 +134,22 @@ standard deviation beside two things it should resemble: the error actually meas truths drawn from the same generator and sampled with the same pattern, and the absolute error in the single realization above. The bottom row shows the sampling pattern, a cell-by-cell scatter of predicted against actual error, and a calibration curve in which cells are binned into deciles of the prediction. +Fixed axes are chosen so that the results are directly comparable from figure to figure, not just subplots. Two cautions matter when reading these. First, one realization of an error is one draw of a random variable, so even a perfect prediction correlates with it only up to a ceiling, which is printed on each -scatter panel; the comparison against the 64-truth ensemble is the decisive test, and there the -prediction tracks the actual error pattern with correlations of 0.90 to 0.99. Second, uniform ergodic -coverage intentionally flattens the predicted map, since the whole point of the pattern is to leave no -cell poorly covered, and that narrow range is what pushes the single-realization ceiling down. The -calibration curves run close to the one-to-one line in every case, slightly below it, meaning the -predicted magnitudes are roughly correct and not merely correctly ranked. +scatter panel. The comparison against the 64-truth ensemble is a test showing the prediction tracks the +actual error pattern with correlations of 0.90 to 0.99. Second, uniform ergodic coverage intentionally +flattens the predicted map, since the whole point of the pattern is to leave no cell poorly covered, and +that narrow range is what pushes the single-realization ceiling down. The calibration curves run close to +the one-to-one line in every case, slightly below it, meaning the predicted magnitudes are roughly correct. +**One takeaway here** is not the magnitude of the error mismatch between the single realization and the +predicted error map, but it's structure. If the error map is dominated by high frequency variations on the +scale of a discovered anomaly like an ore deposit, then smaller variations would be expected to average +out over the span of the contiguous region of the anomaly and thus a more precise estimate of the size and +extent of the anomaly could be gathered than by a simply measuring the reconstruction. This is because we +can leverage the structure of the ergodic sampling points and their implied error structure to reign in +the uncertainty in our anomaly predictions. ![Figure 4](plt2a1.png)