jeremy/ublox-gps-data-processing
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add GCGC virtual-reference-station solution, VBS exploration suite, and report improvements

Browse source 
Pipeline (process_gps.py):
- New standard solution ppk_vrs: a virtual reference station synthesized at
  the capture position from the GCGC master station the CORS stage already
  fetched (zero baseline). Gated on GCGC availability; --no-vrs to disable.
  Engine validated in experiments/ (gates G1/G2a) and benchmarked equivalent
  to Trimble Pivot's commercial VRS on ground truth.
- Track output now selects among PPK-family solutions by measured dispersion,
  not nominal baseline (protects against poor network-edge VRS data).
- Warm-up exclusion: first 5 minutes (--warmup-min) removed from all quality
  metrics and figures; prominent note in both reports; data outputs unchanged.
- Professional report language: abbreviations expanded on first use, solution
  methods defined, no shorthand in tables or figure labels; fig5 reframed as
  "self-reported precision (uncalibrated) vs measured dispersion" with the
  optimism factor annotated.
- Local-base robustness: refuse to combine base files at different positions;
  RTCM logs and Data Shop RINEX accepted for the parallel GCGC method.

Streaming (stream_gps.py):
- Credentials via gitignored gcgc.env (template gcgc.env.example), env vars
  take precedence; RTCM correction stream recorded to stream_<ts>.rtcm3 for
  zero-baseline post-processing; live GGA uplink and sea dynamic model remain
  the buoy defaults; warm-up excluded from session statistics.

Experiments (new):
- vbs_synth.py: geometric virtual-base synthesis engine (RINEX 2.11 patcher,
  SP3 orbits, light-time + Earth-rotation per position, clock-robust).
- vbs_iono.py: carrier-leveled slant-ionosphere estimation + station-network
  interpolation with leave-one-out validation (median 7 cm; 1.2 cm / 10 km
  growth; DCBs as daily constants via median polish).
- vbs_compare.py: observation-domain comparison of commercial VRS files vs
  synthesized bases (ambiguity-detrended correction content).
- FINDINGS.md: gate results, commercial benchmark, offshore analysis, and
  promotion rationale.

Docs: README validation-results section explaining why the synthesized VRS
(kinematic) is the preferred solution for buoy deployments; VRS orders and
experiment data moved under gitignored experiments/data/.
This commit is contained in:
= 2026-07-27 21:50:10 -04:00
parent 65a4836608
commit 6f85a6f1ed
9 changed files with 1945 additions and 172 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
.gitignore vendored
View file

@ -12,7 +12,17 @@ processed/
# Live streaming session outputs (stream_gps.py) # Live streaming session outputs (stream_gps.py)
stream_*.csv stream_*.csv
stream_*.rtcm3
# Raw GPS capture data # Raw GPS capture data
*.ubx *.ubx
*.uc2x *.uc2x
# NTRIP credentials (copy gcgc.env.example -> gcgc.env; never commit)
gcgc.env
# GCGC Reference Data Shop downloads (auto-discovered base data)
gcgc_base/
# VBS exploration downloads and synthesized files (regenerable)
experiments/data/

113
README.md
View file

@ -105,10 +105,14 @@ published yet):
| Flag | Meaning | | Flag | Meaning |
|---|---| |---|---|
| `--interval N` | `track.csv` output interval in seconds (default 1 = native rate) | | `--interval N` | `track.csv` output interval in seconds (default 1 = native rate) |
| `--warmup-min N` | receiver warm-up window excluded from ALL quality metrics and figures (default 5; `track.csv` keeps the full session; note shown atop both reports) |
| `--static` | antenna verifiably stationary: enables static PPK/PPP solutions and static error framing. **Never for buoys.** | | `--static` | antenna verifiably stationary: enables static PPK/PPP solutions and static error framing. **Never for buoys.** |
| `--no-ppk` | skip base-station download/processing | | `--no-ppk` | skip base-station download/processing |
| `--no-vrs` | skip the synthesized GCGC VRS solution |
| `--ppp` | additionally run PPP against IGS precise products | | `--ppp` | additionally run PPP against IGS precise products |
| `--base SSSS` | force a specific CORS station id (e.g. `MSEV`) | | `--base SSSS` | force a specific CORS station id (e.g. `MSEV`) |
| `--base-obs FILE...` | explicit base file(s) for the parallel GCGC/local PPK method — normally unnecessary (`<capture>.rtcm3` sidecars and `gcgc_base/` are auto-discovered) |
| `--base-xyz X Y Z` | ECEF position of the local base (else its RINEX header position is used — verify its datum) |
| `--max-base-km KM` | base-station search radius (default 150) | | `--max-base-km KM` | base-station search radius (default 150) |
| `--force-stage S` | delete and re-run one stage (`rinex`,`single`,`ppk`,`ppp`,`track`) | | `--force-stage S` | delete and re-run one stage (`rinex`,`single`,`ppk`,`ppp`,`track`) |
@ -152,13 +156,22 @@ The optional `.uc2x` sidecar (u-center 2 index) is only used for its session
metadata header. metadata header.
**Real-time credentials (`stream_gps.py`).** Free registration at **Real-time credentials (`stream_gps.py`).** Free registration at
http://rtn.usm.edu/RegisterAccount.aspx (Mississippi GCGC RTN), then: http://rtn.usm.edu/RegisterAccount.aspx (Mississippi GCGC RTN). Provide the
credentials on the machine that runs the streamer — preferred: copy the
template and fill it in (the file is gitignored and loaded automatically):
```bash ```bash
export GCGC_USER=<username> GCGC_PASS=<password> cp gcgc.env.example gcgc.env # then edit gcgc.env with the real values
venv/bin/python stream_gps.py --port /dev/ttyUSB0 # or COM7 on Windows venv/bin/python stream_gps.py --port /dev/ttyUSB0 # or COM7 on Windows
``` ```
Alternatives: `export GCGC_USER=... GCGC_PASS=...` (or Windows
`setx GCGC_USER ...`) — set variables take precedence over `gcgc.env`. Avoid
the `--ntrip-user`/`--ntrip-pass` flags outside of quick tests: command-line
arguments end up in shell history and process listings. Never commit real
credentials; authentication is verified on the first NTRIP connect (a 401
appears in the log as "NTRIP connection failed").
Key streaming defaults (buoy-oriented): receiver dynamic-platform model `sea` Key streaming defaults (buoy-oriented): receiver dynamic-platform model `sea`
(`--dynmodel` to override), **live** NMEA GGA uplink so the network-RTK virtual (`--dynmodel` to override), **live** NMEA GGA uplink so the network-RTK virtual
reference follows platform drift (`--gga-fixed` for bench tests), raw reference follows platform drift (`--gga-fixed` for bench tests), raw
@ -166,6 +179,39 @@ RXM-RAWX/SFRBX always logged so every session is post-processable, and
`--replay <file.ubx>` offline test mode. Galileo HAS (~20 cm, no internet) `--replay <file.ubx>` offline test mode. Galileo HAS (~20 cm, no internet)
activates automatically as fallback when firmware ≥ HPG 2.10 is detected. activates automatically as fallback when firmware ≥ HPG 2.10 is detected.
**GCGC data in post-processing (automatic parallel method).** Beyond the
real-time streams, a GCGC account also gives access to their **Reference Data
Shop** (on-demand static/RINEX downloads from their 52 stations). No special
invocation is needed — running `process_gps.py <capture.ubx>` as usual
auto-discovers local GCGC data and, when found, runs a **second PPK solution
(`ppk_gcgc`) in parallel** with the NGS CORS one, reported alongside all other
results (it ranks first for the track output, since its baseline is shortest).
Discovered sources:
1. `<capture>.rtcm3` **sidecar** — every `stream_gps.py` NTRIP session records
its incoming VRS correction stream next to the raw log. A VRS is a *virtual
base at the receiver's own position* (zero baseline) — the best possible
PPK geometry.
2. **`gcgc_base/` folder** — drop Reference Data Shop RINEX downloads here;
files whose observation window overlaps the capture are used automatically.
`--base-obs FILE...` explicitly feeds this method instead of auto-discovery.
Note GCGC coordinates are NAD83(2011): supply `--base-xyz` (ECEF, ITRF) if
frame consistency with the NGS-based solution matters — otherwise the base
RINEX header position is used and the report warns about the ~1.5 m datum
offset.
**Synthesized GCGC VRS (`ppk_vrs`, standard).** When the selected base is a
GCGC network station, the pipeline additionally synthesizes a virtual base at
the capture's own position (zero baseline) from that master's observations and
runs PPK against it — the engine was gate-validated and benchmarked equivalent
to Trimble Pivot's commercial VRS (see `experiments/FINDINGS.md`). Requires
ESA orbit products (~1 day lag; the stage reports "pending" until then). Only
provided when GCGC is available — for captures outside the GCGC network the
solution is skipped automatically. Disable with `--no-vrs`. The track output
picks among the PPK-family solutions by *measured* quality (empirical scatter),
so a poor network-edge VRS never silently wins over a better direct solution.
**Datums.** PPK output is in the CORS base frame (ITRF2020, current epoch); **Datums.** PPK output is in the CORS base frame (ITRF2020, current epoch);
GCGC real-time corrections are NAD83(2011) epoch 2010.0 — a constant ~1.5 m GCGC real-time corrections are NAD83(2011) epoch 2010.0 — a constant ~1.5 m
offset from ITRF/WGS84 in CONUS. Each CSV records its frame in the header; offset from ITRF/WGS84 in CONUS. Each CSV records its frame in the header;
@ -191,3 +237,66 @@ processed/<capture>/ per-capture outputs: RINEX, .pos solutions,
track.csv, track.kml, figures/, diagnostics.json, track.csv, track.kml, figures/, diagnostics.json,
report.md, report.html (generated) report.md, report.html (generated)
``` ```
---
## Validation and test results (July 2026)
Full detail, gate definitions, and reproduction commands: `experiments/FINDINGS.md`.
**Test suite passed** The virtual-reference-station tooling
(`experiments/vbs_synth.py`, was integrated into the pipeline as `ppk_vrs`)
passed four correctness tests: bit-identical output at zero displacement;
round-trip displacement error of one quantization step (0.001); insensitivity to
a physically representative 1 microsecond receiver-clock offset (millimetre
agreement; an unphysical 1 ms test also documented why timestamp/pseudorange
self-consistency is mandatory in synthesized files); and a ground-truth test in
which a station with known coordinates, processed against a base synthesized at
its own location from a station 44 km away, reproduced the direct-baseline
solution statistics.
**Ionosphere interpolation (leave-one-out, five stations).** Carrier-leveled
dual-frequency slant ionosphere interpolated across the Gulf Coast Geospatial
Center (GCGC) / National Geodetic Survey (NGS) station network predicts a
held-out station's ionosphere to 5-10 cm (median) inside the network, growing
about 1.2 cm per 10 km — including 19 cm at a station 135 km outside the fitting
cluster. Offshore implication: roughly 10-15 cm of ionospheric prediction error
at 60 km beyond the coastal network.
**Benchmark against the commercial product.** Using ground-truth station MARY:
our synthesized virtual base and Trimble Pivot's commercial virtual-reference-
station product were statistically equivalent (fixed-solution horizontal RMS
7.5 cm vs 7.8 cm; both roughly halve the error and the wrong-fix count of the
direct 44 km baseline). Observation-domain analysis showed the commercial
product embeds only ~3-7 cm of correction content beyond pure geometry, roughly
flat out to 92 km of displacement — including virtual stations the operator's
portal generated 25 km and 60 km offshore. One commercial file (at the sparse
network edge near Hattiesburg) carried a ~5 m bias and 3x noise; our own
tooling at the same site with the same master station was clean, which is why
the pipeline selects its track solution by measured dispersion rather than by
nominal baseline length.
**Why "PPK - synthesized virtual reference station (kinematic)" is likely the
best solution for the buoy / boat use case.**
1. *Zero-length baseline at the platform*: the dominant baseline-dependent
errors shrink to the atmospheric difference between master and platform,
and the geometry term vanishes entirely.
2. *Measured performance*: best dispersion of all methods on the reference
capture (50% radius 0.39 m vs 0.41 m for the direct base), and the best
fixed-solution accuracy in the ground-truth benchmark.
3. *Kinematic processing makes no motion assumption* — correct for a drifting
buoy by construction.
4. *Offshore reach*: works wherever any GCGC/NGS master station exists within
~150 km; the commercial product's extrapolated corrections were shown to add
only centimetres of content at those distances, so little is lost by
synthesizing locally - and quality stays under our control (the network-edge
failure above was a commercial-side defect our synthesis avoided).
5. *Fully automatic with open data*: needs only the already-downloaded base
observations plus European Space Agency orbit products (about one day of
latency).
Caveats: validated on one (quiet) day so far; GPS+GLONASS until multi-GNSS
orbit products are available for a given day; the measured ionospheric
interpolation layer is not yet applied to the synthesized observations
(planned); disturbed-ionosphere days untested.

194
experiments/FINDINGS.md Normal file
View file

@ -0,0 +1,194 @@
# VBS Exploration Findings — Session 1 (2026-07-27)
Exploration of DIY virtual-base-station synthesis from GCGC/NGS multi-station
data, targeting offshore (extrapolation) PPK for buoy deployments. Full design
+ gate definitions: see the approved plan; scripts: `vbs_synth.py`,
`vbs_iono.py`. Data: NGS CORS 30 s dailies, DOY 204 (2026-07-23) — MSIN, MARY,
SBCH, COVG, MSEV (+ ALDI/ALFO/AL90/AL92/ALMJ downloaded for geometry D, unused
so far); ESA rapid SP3 (GPS+GLONASS — Galileo pending a multi-GNSS SP3 source,
e.g. ESA finals ~2 weeks post-date); NGS brdc RINEX 2 nav (demo5 rnx2rtkp did
NOT parse BKG's RINEX 3.05 mixed nav — use NGS brdc `.26n/.26g`).
## Gate G1 — geometric synthesis engine: **GREEN**
| Check | Result |
|---|---|
| G1a identity (zero displacement) | **PASS** — 44,884 surviving records bit-identical (max diff 0.000, flags preserved) |
| G1b round-trip MSIN→MARY site→back | **PASS** — max diff 0.001 (= 1 LSB of RINEX quantization) |
| G1c clock perturbation | **PASS at physical magnitude** (see below) |
| G1d A→B truth (MARY as rover, 43.7 km) | **PASS** — VBS-at-MARY base reproduces the direct MSIN-base solution (mean 3D diff 13 cm = float noise); truth errors statistically identical (horiz RMS 0.634 vs 0.639 m; mean N/E offset actually slightly smaller for VBS) |
G1d detail (4 h window 16–20 UT, GPS+GLO, 30 s, broadcast nav, kinematic):
direct fix rate 37.4% with **69 wrong fixes** (Q=1 & >10 cm truth error); VBS
43.7% with 31 wrong fixes. Wrong fixes are present in BOTH — confirming the
plan's central honesty rule: never score on fix rate; score truth error under
Q=1. 44 km daytime solar-max AR is marginal, as expected.
### The clock findings (validates the timing concern raised in review)
- A **1 ms** injected clock offset broke the solution (0.63 m shifts, AR lost)
— but diagnosis showed the *test* was unphysical, not the engine: shifting
observables without shifting the sampling instant creates an inconsistent
receiver. RTKLIB derives the base clock from pseudoranges and time-shifts
satellite positions accordingly — so timestamp↔pseudorange **self-consistency
in synthesized files is mandatory** (our engine preserves the master's real,
self-consistent clock, hence G1d passing).
- At the physically representative **1 µs** (steered CORS clocks): both-float
epochs agree to **max 4.7 mm / mean 0.8 mm** → the engine is clock-robust.
All larger deviations (≤12 cm) came from 18/481 epochs where a *borderline AR
validation decision flipped* — chaotic threshold sensitivity inherent to
marginal AR, present under any tiny perturbation, not a synthesis defect.
## Gate G2a — iono LIM cross-validation: **GREEN (GPS-only)**
Carrier-leveled geometry-free slant iono (arc-leveled to code), 5-min bins,
20° mask, per-satellite planar fit (LIM) across 4 stations, leave-one-out:
| Held-out | Centroid dist | median\|r\| | 95%\|r\| |
|---|---|---|---|
| MSIN | 18 km | 0.050 m | 0.158 m |
| MARY | 52 km | 0.050 m | 0.124 m |
| SBCH | 65 km | 0.057 m | 0.185 m |
| COVG | 38 km | 0.100 m | 0.267 m |
| MSEV | **135 km (true extrapolation)** | **0.191 m** | 0.551 m |
- **Overall median 0.070 m ≤ 0.10 m gate → PASS.**
- **Residual-growth slope: 1.2 cm per 10 km** — matches the literature band
(1–2 cm/10 km) used in the design review. Offshore implication: ~10–15 cm
slant-iono prediction error at 60 km beyond the network edge — supportive of
dm-level (not cm-level) VBS improvement targets for extrapolated points.
### Two instructive failures on the way (both diagnosed + fixed)
1. **v1 estimator failed the gate by 10x** (median 0.7–1.4 m; COVG 4.6–6.9 m).
Cause: per-bin free per-station bias parameters are near-degenerate with a
satellite-common gradient in a 4-station fit — extrapolated planes explode.
Fix: estimate station biases (DCBs) as **daily constants** via median polish
(hardware DCBs are stable), then fit per-bin planes with biases fixed →
residuals collapsed 20x. Lesson recorded for Phase 2b: DCB handling is
*the* conditioning issue in small-network LIM.
2. **P1/C1 column trap**: MSEV logs GPS code in C1 with a blank P1 column —
column-level fallback silently produced a GLONASS-only station. Value-level
P1→C1 / P2→C2 fallback required. (GLONASS remains excluded from the iono
fit for now: per-slot inter-channel code biases need per-(station,slot)
bias terms — queued for Phase 2b if GPS-only corrections prove insufficient.)
## Pivot VRS benchmark (2026-07-27, orders V304-V307) — **the G0 question answered**
GCGC's portal DOES offer VRS orders, and **accepted both offshore points (25 km
and 60 km beyond the network edge) without complaint** — 1 s files, zero missing
epochs, positions declared exactly as requested (NAD83(2011); note +1.4 m vs
ITRF ellipsoidal height in this region). Files in `experiments/data/orders/` (V304 = capture
site 50 min; V305 = MARY site 4 h; V306/V307 = offshore probes 4 h).
**MARY truth benchmark** (rover = MARY 30 s, 16-20 UT, GPS+GLO, broadcast nav;
truth = MARY's published coordinates, frame-matched per leg):
| Base | Fix rate | Wrong fixes | Truth horiz RMS | Fixed-only RMS / CEP95 |
|---|---|---|---|---|
| Pivot VRS at MARY (0 km) | 41.4% | 45 | 0.548 m | **0.078 / 0.129 m** |
| DIY VBS at MARY (0 km, geometric-only) | 43.7% | 31 | 0.634 m | **0.075 / 0.149 m** |
| Direct MSIN (44 km) | 37.4% | 69 | 0.639 m | 0.136 / 0.298 m |
Findings:
1. **Commercial Pivot VRS ~= DIY geometric-only VBS at this site/day** — the
network's atmospheric corrections added no measurable advantage over pure
geometric displacement (single quiet-ish day, GPS+GLO, 30 s caveats apply).
Both zero-baseline methods beat the 44 km direct base *when fixed* (7.5-7.8
cm vs 13.6 cm) and roughly halve wrong fixes.
2. **Wrong-fix rates are high for ALL methods** (~20-35% of Q=1 epochs) —
processing-config improvements (fix-and-hold, mask, L5) are currently a
bigger lever than base choice; reinforces truth-error-under-fix scoring.
3. **Capture-site leg (V304 via the production pipeline)**: ppk_gcgc ran
automatically (base 0.0 km) but stayed float with ratio ~1.1 and *worse*
scatter than the MSEV solution (CEP50 1.10 vs 0.44 m), plus a ~3.5 m
unexplained height offset beyond the ~1.4 m datum difference — float-bias
behavior consistent with the u-blox rover's GAL/BDS signals being unusable
against a G+R-only 2.11 base plus VRS correction noise; needs a dedicated
look (multi-frequency conf, `-f 3`, RINEX 3.04 VRS re-order at the site).
4. Offshore V306/V307 files are in hand for the Phase 2b/3 obs-domain
comparison (Pivot's extrapolated corrections vs our LIM extrapolation).
## Obs-domain analysis: how much correction does Pivot actually embed? (vbs_compare.py)
DIY geometric VBS synthesized at each Pivot file's exact declared position
(same NAD83 frame/master), observations differenced satellite-by-satellite,
DD'd against the highest satellite, per-satellite ambiguity constants removed.
What remains = network correction content relative to pure geometry (+ master
noise; 95% tails include uncleaned re-levelings):
| VRS point | Displacement | L1 DD variation median / 95% | P1 code DD median |
|---|---|---|---|
| V304 capture site (edge) | 34 km | **0.026 / 0.163 m** | 0.278 m |
| V305 MARY (in-network) | 29 km | **0.064 / 2.127 m** | 0.369 m |
| V306 25 km offshore | 75 km | **0.063 / 0.914 m** | 0.446 m |
| V307 60 km offshore | 92 km | **0.065 / 1.020 m** | 0.517 m |
**Three independent measurements now agree**: (1) LOOCV says LIM-interpolable
iono differences are 5-10 cm median in-network, growing 1.2 cm/10 km; (2) the
MARY truth benchmark says that correction content is too small to change
positioning outcomes on this day (Pivot ~= DIY); (3) the obs-domain analysis
says Pivot embeds a median of only **~3-7 cm of correction beyond pure
geometry, roughly flat out to 92 km displacement** (code DD grows mildly with
distance, dominated by master code noise). Consistent conclusion for buoys:
at these distances/conditions, a geometric virtual base + our own LIM layer is
competitive with the commercial product; the decisive factors are processing
config (AR strategy, multi-frequency) and disturbed-day behavior (untested).
Capture-site V304 anomaly **RESOLVED (2026-07-27, promotion session)**: the
production ppk_vrs stage (our engine, same MSEV master, same zero baseline,
same rover) is clean - matches direct-MSEV to 3 cm and slightly beats its
scatter (CEP50 0.42 vs 0.44 m). The ~5 m bias + 3x noise is therefore **in
Pivot's V304 file itself** (network-edge synthesis at Hattiesburg where MSHT
is decommissioned), not in the rover pairing. Method note: the obs-domain
DD comparison could not see this because per-satellite ambiguity detrending
also absorbs constant biases - it measures time-varying content only. The
RINEX 3.04 re-order of the capture-site VRS is now optional curiosity, not a
blocker. Fix-and-hold truth test at MARY: +10% fix rate on the Pivot leg only,
wrong-fix counts UNCHANGED on all legs - wrong fixes are a validation-threshold
problem (30 s epochs, ratio 3.0), not an AR-strategy problem; the next real
levers are multi-frequency processing and stricter/partial validation.
## Promotion (2026-07-27, user decision)
The DIY VRS is promoted into `process_gps.py` as the **standard GCGC-provided
solution `ppk_vrs`**: auto-synthesized at the capture position from the NGS
stage's master whenever that master is a GCGC station (agency header / MS
station check), ESA orbits (FIN>RAP, ~1 day lag, "pending" until published),
frame-consistent ITRF, zero nominal baseline. Reported alongside all other
solutions with its own color/validation; `--no-vrs` disables. Track output now
selects among PPK-family solutions by empirical scatter, protecting against
poor network-edge VRS data (exactly the V304 case). Caveats carried with the
promotion: single-day validation, GPS+GLO only until multi-GNSS orbits are
available for the day (Galileo joins via ESA finals), disturbed-iono behavior
untested (Phase 3).
## Status vs plan
- G0: partially done — station set + data availability verified (NGS path);
**USER ACTION open: check the GCGC portal (rtn.usm.edu) "new order" screen
for a "Virtual Reference Station" order type**, and whether it permits
offshore points; GCGC Data Shop 1 s downloads not yet needed but will be for
Phase 3's disturbed-day / 1 Hz legs.
- G1: **GREEN** (all four checks).
- G2a: **GREEN** (GPS-only, one quiet day). Pending for robustness: ≥2 more
days incl. a disturbed day (Phase 3 requirement, not a G2a blocker).
- Next (Phase 2b): apply LIM corrections in `vbs_synth` (`--iono` hook exists
in design): sign-correct code/phase application, (f_L1/f)² scaling,
phase-continuity audit (<λ/4 steps), then geometry B/C/D truth runs with the
wrong-fix-rate scoring, day/night splits, and the 0–60 km offshore sweep.
## Reproduction
```bash
# G1 (identity/round-trip/clock/truth) — see session commands; engine:
venv/bin/python experiments/vbs_synth.py experiments/data/204/msin2040.26o \
--xyz <MARY_XYZ> --master-xyz <MSIN_XYZ> \
--sp3 experiments/data/204/ESA0OPSRAP_20262040000_01D_05M_ORB.SP3 \
--nav experiments/data/204/BRDC00WRD_R_20262040000_01D_MN.rnx \
-o experiments/data/204/vbs_at_mary.obs
# G2a:
venv/bin/python experiments/vbs_iono.py --stations msin:... mary:... sbch:... \
covg:... msev:... --dir experiments/data/204 --sp3 ... --nav ... --gps-only
```
(ITRF2020 station coordinates epoch-propagated via `process_gps.cors_station_xyz`.)

193
experiments/vbs_compare.py Normal file
View file

@ -0,0 +1,193 @@
#!/usr/bin/env python3
"""
vbs_compare.py - Observation-domain comparison: Pivot VRS vs DIY geometric VBS.
Synthesizes a DIY virtual base at a Pivot VRS file's exact declared position
(same master frame), then differences the two files' observables per epoch and
satellite, double-differencing against the highest-elevation satellite to
remove both files' receiver-clock-like common terms.
What remains in the DD series is (Pivot's embedded network corrections +
Pivot's master data) minus (our master's raw atmosphere carried verbatim) -
i.e. the correction content the commercial network added relative to pure
geometric displacement, plus master-choice differences. Tracking its RMS
against extrapolation distance measures how much network correction actually
exists where the buoys will operate.
Usage:
vbs_compare.py PIVOT.obs --master MASTER.obs --master-xyz X Y Z
--sp3 FILE --nav BRDC [--label NAME]
"""
import argparse
import sys
from pathlib import Path
import numpy as np
sys.path.insert(0, str(Path(__file__).resolve().parent))
from vbs_synth import (Sp3, parse_header, glonass_slots, glo_freq, FREQ, C, # noqa
synthesize)
def read_l1p1(path, slots):
"""(epoch_key, sat) -> (L1_m, P1_m); L1 converted to metres."""
lines = open(path, errors="replace").read().splitlines()
types, xyz, hdr_end = parse_header(lines)
nlps = (len(types) + 4) // 5
idx = {t: k for k, t in enumerate(types)}
i_l1 = idx["L1"]
i_p1, i_c1 = idx.get("P1"), idx.get("C1")
out = {}
i = hdr_end + 1
while i < len(lines):
line = lines[i]
if len(line) < 32 or not line[:26].strip():
i += 1
continue
try:
flag = int(line[26:29]); nsat = int(line[29:32])
except ValueError:
i += 1
continue
if flag > 1:
i += nsat + 1
continue
tp = line[:26].split()
tkey = tuple(int(v) for v in tp[:5]) + (round(float(tp[5]), 3),)
sats = []
nlin = (nsat + 11) // 12
for k in range(nlin):
seg = lines[i + k][32:68]
for j in range(12):
s = seg[j * 3:(j + 1) * 3]
if s.strip():
sats.append(s.replace(" ", "0"))
i += nlin
p = line[:26].split()
tsec = int(p[3]) * 3600 + int(p[4]) * 60 + float(p[5])
for sat in sats:
block = lines[i:i + nlps]
i += nlps
sysid = sat[0]
if sysid == "G":
f1 = FREQ[("G", "1")]
elif sysid == "R" and sat in slots:
f1 = glo_freq("1", slots[sat])
else:
continue
def val(ti):
if ti is None:
return None
ln = block[ti // 5] if ti // 5 < len(block) else ""
seg = ln[(ti % 5) * 16:(ti % 5) * 16 + 14]
return float(seg) if seg.strip() else None
l1 = val(i_l1)
p1 = val(i_p1)
if p1 is None:
p1 = val(i_c1)
if l1 is None or p1 is None:
continue
out[(tkey, sat)] = (tsec, l1 * (C / f1), p1)
return out
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("pivot", type=Path)
ap.add_argument("--master", type=Path, required=True)
ap.add_argument("--master-xyz", nargs=3, type=float, required=True,
help="master position in the SAME frame as the pivot file's "
"declared position (NAD83 for GCGC)")
ap.add_argument("--sp3", type=Path, required=True)
ap.add_argument("--nav", type=Path, required=True)
ap.add_argument("--label", default=None)
ap.add_argument("--keep-synth", action="store_true")
args = ap.parse_args()
label = args.label or args.pivot.stem
# pivot's declared position
lines = open(args.pivot, errors="replace").read().splitlines()
_, pivot_xyz, _ = parse_header(lines)
x_v = np.array(pivot_xyz)
disp_km = np.linalg.norm(x_v - np.array(args.master_xyz)) / 1e3
sp3 = Sp3(args.sp3)
slots = glonass_slots(args.nav)
synth_path = args.pivot.with_suffix(".diy_synth.obs")
if not synth_path.is_file():
synthesize(args.master, synth_path, pivot_xyz, sp3, slots,
master_xyz=args.master_xyz, marker="DIYC")
piv = read_l1p1(args.pivot, slots)
diy = read_l1p1(synth_path, slots)
# group common records per epoch
epochs = {}
for key in piv.keys() & diy.keys():
tkey, sat = key
epochs.setdefault(tkey, []).append(sat)
dd_l1, dd_p1, used_epochs = {}, [], 0 # phase grouped per satellite
x_rcv = x_v
for tkey, sats in epochs.items():
if len(sats) < 4:
continue
tsec = piv[(tkey, sats[0])][0]
# reference satellite: highest elevation GPS
def elev(sat):
pos = sp3.pos(sat, tsec)
if pos is None:
return -99
d = pos - x_rcv
up = x_rcv / np.linalg.norm(x_rcv)
return float(np.dot(d, up) / np.linalg.norm(d))
gps = [s for s in sats if s[0] == "G"]
if len(gps) < 2:
continue
ref = max(gps, key=elev)
dl_ref = piv[(tkey, ref)][1] - diy[(tkey, ref)][1]
dp_ref = piv[(tkey, ref)][2] - diy[(tkey, ref)][2]
used_epochs += 1
for sat in sats:
if sat == ref:
continue
dl = piv[(tkey, sat)][1] - diy[(tkey, sat)][1] - dl_ref
dp = piv[(tkey, sat)][2] - diy[(tkey, sat)][2] - dp_ref
dd_l1.setdefault(sat, []).append(dl)
if abs(dp) < 50:
dd_p1.append(dp)
# Phase DD contains an arbitrary integer-ambiguity constant per satellite
# (each file's own arcs): remove the per-satellite median and analyse the
# remaining VARIATION - that is the differential correction content the
# network embedded relative to pure geometry (plus master noise).
var_l1 = []
for sat, vals in dd_l1.items():
v = np.array(vals)
v = v - np.median(v)
v = v[np.abs(v) < 5.0] # drop cycle-slip re-levelings
var_l1.append(v)
var_l1 = np.concatenate(var_l1) if var_l1 else np.array([])
dd_p1 = np.array(dd_p1)
print(f"[compare] {label}: displacement master->virtual {disp_km:.1f} km, "
f"{used_epochs} epochs")
if len(var_l1):
print(f" L1 phase DD variation (per-sat ambiguity removed): "
f"median|d|={np.median(np.abs(var_l1)):6.3f} m "
f"RMS={np.sqrt((var_l1**2).mean()):6.3f} m "
f"95%|d|={np.percentile(np.abs(var_l1),95):6.3f} m n={len(var_l1)}")
if len(dd_p1):
print(f" P1 code DD (incl. master code noise): "
f"median|d|={np.median(np.abs(dd_p1)):6.3f} m "
f"RMS={np.sqrt((dd_p1**2).mean()):6.3f} m "
f"95%|d|={np.percentile(np.abs(dd_p1),95):6.3f} m")
if not args.keep_synth:
synth_path.unlink(missing_ok=True)
if __name__ == "__main__":
main()

295
experiments/vbs_iono.py Normal file
View file

@ -0,0 +1,295 @@
#!/usr/bin/env python3
"""
vbs_iono.py - Carrier-leveled slant ionosphere estimation + LIM cross-validation.
Phase 2a of the VBS exploration (gate G2a): measures whether multi-station
ionosphere interpolation can be accurate enough to help a virtual base station,
BEFORE building the application layer.
Per station/satellite arc: geometry-free carrier LG = lambda1*L1 - lambda2*L2
(clock-immune by construction) leveled to the geometry-free code PG = P2 - P1
(arc median), giving slant iono delay at L1 (metres; contains per-station DCB,
handled as a per-station bias in the fit).
LIM (Wanninger linear interpolation model): per satellite, per time bin, fit
I = a0 + a1*dN + a2*dE across stations, plus one shared per-station bias b_r
(gauge: b=0 at the first station). Leave-one-out: predict each held-out station
from the others; report residual scatter (after removing the held-out station's
own median offset, which is its unobservable DCB) vs its distance from the
network centroid.
Usage:
vbs_iono.py --stations msin:X:Y:Z mary:X:Y:Z ... --dir data/204 --sp3 FILE
[--bin-s 300] [--elev-mask 20]
"""
import argparse
import math
import sys
from collections import defaultdict
from datetime import datetime, timezone
from pathlib import Path
import numpy as np
sys.path.insert(0, str(Path(__file__).resolve().parent))
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from vbs_synth import Sp3, parse_header, FREQ, glo_freq, glonass_slots # noqa: E402
C = 299792458.0
F1, F2 = 1575.42e6, 1227.60e6
GAMMA = (F1 / F2) ** 2
L1M = C / F1
L2M = C / F2
def read_gf(path, slots):
"""Per (sat) -> time-sorted arrays (t_sec_of_day, LG_m, PG_m) for GPS+GLO."""
lines = open(path, errors="replace").read().splitlines()
types, xyz, hdr_end = parse_header(lines)
nlps = (len(types) + 4) // 5
idx = {t: k for k, t in enumerate(types)}
need = all(t in idx for t in ("L1", "L2", "P2")) and ("P1" in idx or "C1" in idx)
if not need:
raise ValueError(f"{path}: missing L1/L2/P1/P2")
i_l1, i_l2 = idx["L1"], idx["L2"]
i_p1, i_c1 = idx.get("P1"), idx.get("C1")
i_p2, i_c2 = idx.get("P2"), idx.get("C2")
out = defaultdict(list)
i = hdr_end + 1
while i < len(lines):
line = lines[i]
if len(line) < 32 or not line[:26].strip():
i += 1
continue
try:
flag = int(line[26:29]); nsat = int(line[29:32])
except ValueError:
i += 1
continue
if flag > 1:
i += nsat + 1
continue
p = line[:26].split()
tsec = int(p[3]) * 3600 + int(p[4]) * 60 + float(p[5])
sats = []
nlin = (nsat + 11) // 12
for k in range(nlin):
seg = lines[i + k][32:68]
for j in range(12):
s = seg[j * 3:(j + 1) * 3]
if s.strip():
sats.append(s.replace(" ", "0"))
i += nlin
for sat in sats:
block = lines[i:i + nlps]
i += nlps
sysid = sat[0]
if sysid == "G":
l1f, l2f = F1, F2
elif sysid == "R" and sat in slots:
l1f, l2f = glo_freq("1", slots[sat]), glo_freq("2", slots[sat])
else:
continue
def val(ti):
ln = block[ti // 5] if ti // 5 < len(block) else ""
seg = ln[(ti % 5) * 16:(ti % 5) * 16 + 14]
return float(seg) if seg.strip() else None
l1, l2 = val(i_l1), val(i_l2)
# value-level P/C fallback: receivers fill P1 or C1 (and P2 or C2)
# depending on satellite/tracking mode
p1 = val(i_p1) if i_p1 is not None else None
if p1 is None and i_c1 is not None:
p1 = val(i_c1)
p2 = val(i_p2) if i_p2 is not None else None
if p2 is None and i_c2 is not None:
p2 = val(i_c2)
if None in (l1, l2, p1, p2):
continue
gamma = (l1f / l2f) ** 2
lg = (C / l1f) * l1 - (C / l2f) * l2 # metres
pg = p2 - p1
# normalize to L1(GPS)-equivalent iono metres
out[sat].append((tsec, lg / (gamma - 1) * (l1f / F1) ** 2,
pg / (gamma - 1) * (l1f / F1) ** 2))
return {s: np.array(v) for s, v in out.items()}, np.array(xyz)
def leveled_stec(gf, gap_s=120.0, jump_m=0.5, min_arc=10):
"""Level carrier GF to code GF per arc -> (t, stec_L1_m) arrays per sat."""
out = {}
for sat, arr in gf.items():
t, lg, pg = arr[:, 0], arr[:, 1], arr[:, 2]
order = np.argsort(t)
t, lg, pg = t[order], lg[order], pg[order]
arcs = []
start = 0
for k in range(1, len(t)):
if t[k] - t[k - 1] > gap_s or abs(lg[k] - lg[k - 1]) > jump_m:
arcs.append((start, k))
start = k
arcs.append((start, len(t)))
ts, ss = [], []
for a, b in arcs:
if b - a < min_arc:
continue
bias = np.median(lg[a:b] - pg[a:b])
ts.append(t[a:b])
ss.append(lg[a:b] - bias)
if ts:
out[sat] = (np.concatenate(ts), np.concatenate(ss))
return out
def elevation(sp3, sat, tsec_gps_day0, xyz):
pos = sp3.pos(sat, tsec_gps_day0)
if pos is None:
return -90.0
d = pos - xyz
up = xyz / np.linalg.norm(xyz)
return math.degrees(math.asin(np.dot(d, up) / np.linalg.norm(d)))
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--stations", nargs="+", required=True,
metavar="name:X:Y:Z", help="station obs basenames + ECEF")
ap.add_argument("--dir", type=Path, required=True)
ap.add_argument("--suffix", default="2040.26o")
ap.add_argument("--sp3", type=Path, required=True)
ap.add_argument("--nav", type=Path, required=True)
ap.add_argument("--bin-s", type=float, default=300.0)
ap.add_argument("--elev-mask", type=float, default=20.0)
ap.add_argument("--gps-only", action="store_true",
help="exclude GLONASS (per-slot inter-channel code biases "
"break the single per-station bias model)")
args = ap.parse_args()
sp3 = Sp3(args.sp3)
slots = glonass_slots(args.nav)
stations = {}
for spec in args.stations:
name, x, y, z = spec.split(":")
stations[name] = np.array([float(x), float(y), float(z)])
# ENU offsets (km) about network centroid, for the LIM fit
from process_gps import ecef_to_llh, llh_to_enu # noqa: E402
cen = np.mean(list(stations.values()), axis=0)
cen_llh = ecef_to_llh(*cen)
offs = {}
for name, xyz in stations.items():
llh = ecef_to_llh(*xyz)
dn, de, _ = llh_to_enu(llh[0], llh[1], llh[2], *cen_llh)
offs[name] = (float(dn) / 1e3, float(de) / 1e3) # km N, km E
print("[vbs_iono] loading + leveling", flush=True)
stec = {}
for name in stations:
gf, _ = read_gf(args.dir / f"{name}{args.suffix}", slots)
if args.gps_only:
gf = {s: v for s, v in gf.items() if s.startswith("G")}
stec[name] = leveled_stec(gf)
n = sum(len(v[0]) for v in stec[name].values())
print(f" {name}: {len(stec[name])} sats, {n} leveled samples")
# time-bin medians: station -> sat -> {bin: stec}
binned = {}
for name, sats in stec.items():
b = {}
for sat, (t, s) in sats.items():
k = (t // args.bin_s).astype(int)
d = {}
for kk in np.unique(k):
d[int(kk)] = float(np.median(s[k == kk]))
b[sat] = d
binned[name] = b
names = list(stations)
# Stage 1: per-station bias (DCB) as a DAILY constant via median polish.
# Leaving these as free per-bin parameters makes the 4-station LIM fit
# ill-conditioned (bias ~ degenerate with a satellite-common gradient) and
# extrapolation explodes; hardware DCBs are stable over a day, so estimate
# them once with full-day redundancy.
bias = {n: 0.0 for n in names}
for _ in range(4):
net_med = {}
for n in names:
for sat, d in binned[n].items():
for k, v in d.items():
net_med.setdefault((sat, k), []).append(v - bias[n])
net_med = {sk: float(np.median(v)) for sk, v in net_med.items() if len(v) >= 3}
for n in names:
diffs = [binned[n][sat][k] - m for (sat, k), m in net_med.items()
if sat in binned[n] and k in binned[n][sat]]
if diffs:
bias[n] = float(np.median(diffs))
print(" station daily biases (m):",
{n: round(b, 2) for n, b in bias.items()})
for n in names:
for sat in binned[n]:
for k in binned[n][sat]:
binned[n][sat][k] -= bias[n]
print(f"\n[vbs_iono] LOOCV over {len(names)} stations, bin={args.bin_s:.0f}s, "
f"elev mask {args.elev_mask} deg")
results = {}
for held in names:
fit_names = [n for n in names if n != held]
residuals = []
allbins = set()
for sat in binned[held]:
allbins.update(binned[held][sat].keys())
for bin_k in sorted(allbins):
t_mid = (bin_k + 0.5) * args.bin_s
# collect sats present at held-out + >=3 fit stations, above mask
rows, sats_ok = [], []
for sat in binned[held]:
if bin_k not in binned[held][sat]:
continue
have = [n for n in fit_names
if sat in binned[n] and bin_k in binned[n][sat]]
if len(have) < 3:
continue
if elevation(sp3, sat, t_mid, stations[held]) < args.elev_mask:
continue
sats_ok.append((sat, have))
if len(sats_ok) < 3:
continue
# per-satellite LIM planes; station biases already removed (stage 1)
dn, de = offs[held]
for sat, have in sats_ok:
A = np.array([(1.0, offs[n][0], offs[n][1]) for n in have])
y = np.array([binned[n][sat][bin_k] for n in have])
sol, *_ = np.linalg.lstsq(A, y, rcond=None)
pred = sol[0] + sol[1] * dn + sol[2] * de
residuals.append(pred - binned[held][sat][bin_k])
residuals = np.array(residuals)
if len(residuals) == 0:
continue
# remove held-out station's unobservable DCB offset
residuals = residuals - np.median(residuals)
dist_cen = math.hypot(*offs[held])
results[held] = (dist_cen, residuals)
print(f" hold {held}: n={len(residuals):5d} centroid dist {dist_cen:5.1f} km "
f"median|r|={np.median(np.abs(residuals)):.3f} m "
f"95%|r|={np.percentile(np.abs(residuals), 95):.3f} m")
med = {n: float(np.median(np.abs(r))) for n, (d, r) in results.items()}
dists = {n: d for n, (d, r) in results.items()}
if len(results) >= 3:
x = np.array([dists[n] for n in results])
yv = np.array([med[n] for n in results])
slope = np.polyfit(x, yv, 1)[0] * 10 # m per 10 km
print(f"\n residual-vs-distance slope: {slope*100:.1f} cm per 10 km")
overall = np.concatenate([r for _, r in results.values()])
print(f" OVERALL: median|r|={np.median(np.abs(overall)):.3f} m, "
f"95%|r|={np.percentile(np.abs(overall), 95):.3f} m "
f"(gate G2a: median <= ~0.10 m)")
if __name__ == "__main__":
main()

343
experiments/vbs_synth.py Normal file
View file

@ -0,0 +1,343 @@
#!/usr/bin/env python3
"""
vbs_synth.py - Virtual base station synthesis: geometric displacement engine.
Patches a RINEX 2.11 observation file so its observations appear to have been
collected at a different (virtual) position: per epoch/satellite, the change in
geometric range (independent light-time iteration + Earth-rotation correction at
each position, SP3 orbits) is added to every code observable (metres) and every
carrier-phase observable (cycles, per-signal wavelength). Everything else -
LLI/SSI flags, S/D observables, obs-type order, epoch structure - is preserved.
Satellites without SP3 orbits or (GLONASS) without a known frequency slot are
dropped, with epoch satellite lists rewritten accordingly.
The master receiver's clock is inherited by the virtual station (harmless: it is
one clock, absorbed by the processor's per-epoch receiver-clock estimation; see
the --perturb-clock-ms test mode). The atmospheric content of the master's
observations is NOT changed by this engine - a geometric-only virtual base
carries the master's atmosphere verbatim (Phase 2 applies interpolated
corrections on top via --iono).
Usage:
vbs_synth.py master.obs --xyz X Y Z --sp3 FILE --nav BRDC.rnx -o out.obs
[--master-xyz X Y Z] [--perturb-clock-ms N] [--marker NAME]
"""
import argparse
import math
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
import numpy as np
C = 299792458.0
OMEGA_E = 7.2921151467e-5 # rad/s
GAMMA_12 = (1575.42 / 1227.60) ** 2
# carrier frequency by (system, band); GLONASS handled via slot
FREQ = {
("G", "1"): 1575.42e6, ("G", "2"): 1227.60e6, ("G", "5"): 1176.45e6,
("E", "1"): 1575.42e6, ("E", "5"): 1176.45e6, # E1, E5a
("S", "1"): 1575.42e6,
}
def glo_freq(band, slot):
if band == "1":
return 1602.0e6 + slot * 562.5e3
if band == "2":
return 1246.0e6 + slot * 437.5e3
return None
# ----------------------------------------------------------------------------
# SP3 orbits
# ----------------------------------------------------------------------------
class Sp3:
"""SP3 position source with sliding-window Lagrange interpolation."""
def __init__(self, path):
self.t0 = None
times = []
pos = {} # sat -> {tidx: (x,y,z) m}
tidx = -1
for line in open(path, errors="replace"):
if line.startswith("*"):
p = line.split()
t = datetime(int(p[1]), int(p[2]), int(p[3]), int(p[4]), int(p[5]),
int(float(p[6])), tzinfo=timezone.utc)
if self.t0 is None:
self.t0 = t
times.append((t - self.t0).total_seconds())
tidx += 1
elif line.startswith("P") and tidx >= 0:
sat = line[1:4].replace(" ", "0")
try:
x, y, z = (float(line[4:18]), float(line[18:32]), float(line[32:46]))
except ValueError:
continue
if abs(x) > 900000 or (x == 0 and y == 0): # bad/absent
continue
pos.setdefault(sat, {})[tidx] = (x * 1e3, y * 1e3, z * 1e3)
self.times = np.array(times)
self.sats = {}
n = len(times)
for sat, d in pos.items():
if len(d) < n * 0.9: # incomplete arcs: drop satellite
continue
arr = np.full((n, 3), np.nan)
for i, xyz in d.items():
arr[i] = xyz
if np.isnan(arr).any():
continue
self.sats[sat] = arr
def pos(self, sat, t_sec, order=10):
"""Satellite ECEF position (m) at t_sec (seconds from SP3 start)."""
arr = self.sats.get(sat)
if arr is None:
return None
i = np.searchsorted(self.times, t_sec)
half = order // 2
lo = max(0, min(i - half, len(self.times) - order))
idx = slice(lo, lo + order)
tt = self.times[idx]
out = np.empty(3)
# Lagrange interpolation per coordinate
for k in range(3):
y = arr[idx, k]
acc = 0.0
for j in range(len(tt)):
lj = 1.0
for m in range(len(tt)):
if m != j:
lj *= (t_sec - tt[m]) / (tt[j] - tt[m])
acc += y[j] * lj
out[k] = acc
return out
def glonass_slots(nav_path):
"""sat 'Rnn' -> frequency slot number, from a RINEX 3 mixed nav file."""
slots = {}
lines = open(nav_path, errors="replace").read().splitlines()
i = 0
while i < len(lines):
line = lines[i]
if line[:1] == "R" and len(line) > 23 and line[1:3].strip().isdigit():
sat = "R" + line[1:3].replace(" ", "0")
if sat not in slots and i + 2 < len(lines):
orbit2 = lines[i + 2]
try: # 4th 19-char field on broadcast orbit 2
slots[sat] = int(float(orbit2[4 + 3 * 19: 4 + 4 * 19].replace("D", "E")))
except ValueError:
pass
i += 4
else:
i += 1
return slots
# ----------------------------------------------------------------------------
# Geometry
# ----------------------------------------------------------------------------
def geometric_range(sp3, sat, t_rx_sec, x_rcv):
"""Light-time-iterated, Earth-rotation-corrected range (m); None if no orbit."""
tau = 0.075
rho = None
for _ in range(3):
xs = sp3.pos(sat, t_rx_sec - tau)
if xs is None:
return None
ang = OMEGA_E * tau # rotate satellite into reception-time frame
ca, sa = math.cos(ang), math.sin(ang)
xr = np.array([xs[0] * ca + xs[1] * sa, -xs[0] * sa + xs[1] * ca, xs[2]])
rho = float(np.linalg.norm(xr - x_rcv))
tau = rho / C
return rho
# ----------------------------------------------------------------------------
# RINEX 2.11 observation patcher
# ----------------------------------------------------------------------------
def parse_header(lines):
"""Returns (obs_types, approx_xyz, end_idx)."""
types, xyz = [], None
for i, line in enumerate(lines):
label = line[60:].strip()
if label == "# / TYPES OF OBSERV":
p = line[:60].split()
if types == [] and p and p[0].isdigit():
types.extend(p[1:])
else:
types.extend(p)
elif label == "APPROX POSITION XYZ":
xyz = tuple(float(v) for v in line[:60].split())
elif label == "END OF HEADER":
return types, xyz, i
raise ValueError("no END OF HEADER")
def fmt_obs(value, lli, ssi):
if value is None:
return " " * 14 + lli + ssi
return f"{value:14.3f}" + lli + ssi
def synthesize(master_path, out_path, virtual_xyz, sp3, slots, master_xyz=None,
perturb_ms=0.0, marker="VBS"):
lines = open(master_path, errors="replace").read().splitlines()
types, hdr_xyz, hdr_end = parse_header(lines)
ntypes = len(types)
nlines_per_sat = (ntypes + 4) // 5
x_mas = np.array(master_xyz if master_xyz else hdr_xyz)
x_vrs = np.array(virtual_xyz)
disp_km = np.linalg.norm(x_vrs - x_mas) / 1e3
out = []
for line in lines[:hdr_end + 1]:
label = line[60:].strip()
if label == "APPROX POSITION XYZ":
out.append(f"{x_vrs[0]:14.4f}{x_vrs[1]:14.4f}{x_vrs[2]:14.4f}"
+ " " * 18 + "APPROX POSITION XYZ")
elif label == "MARKER NAME":
out.append(f"{marker:<60}MARKER NAME")
elif label == "END OF HEADER":
out.append(f"{'VBS synthesized from ' + Path(master_path).name:<60}COMMENT")
out.append(f"{f'displacement {disp_km:.3f} km; geometric only':<60}COMMENT")
out.append(line)
else:
out.append(line)
i = hdr_end + 1
stats = {"epochs": 0, "sats": 0, "dropped": {}, "dr_min": 1e9, "dr_max": -1e9}
while i < len(lines):
line = lines[i]
if len(line) < 32 or not line[:26].strip():
i += 1
continue
try:
flag = int(line[26:29])
nsat = int(line[29:32])
except ValueError:
i += 1
continue
if flag > 1: # event records: copy verbatim
out.append(line)
for k in range(nsat):
i += 1
out.append(lines[i])
i += 1
continue
# epoch time (GPS time system)
p = line[:26].split()
t_rx = datetime(2000 + int(p[0]), int(p[1]), int(p[2]), int(p[3]), int(p[4]),
tzinfo=timezone.utc) + timedelta(seconds=float(p[5]))
t_sec = (t_rx - sp3.t0).total_seconds()
# satellite list (12 per line, continuations)
sats = []
nlin = (nsat + 11) // 12
for k in range(nlin):
seg = lines[i + k][32:68]
for j in range(12):
s = seg[j * 3:(j + 1) * 3]
if s.strip():
sats.append(s.replace(" ", "0"))
i += nlin
# per-satellite observation blocks
keep = []
for sat in sats:
block = lines[i:i + nlines_per_sat]
i += nlines_per_sat
sys_id = sat[0]
if sys_id == "R" and sat not in slots:
stats["dropped"][sat] = stats["dropped"].get(sat, 0) + 1
continue
rho_m = geometric_range(sp3, sat, t_sec, x_mas)
if rho_m is None:
stats["dropped"][sat] = stats["dropped"].get(sat, 0) + 1
continue
rho_v = geometric_range(sp3, sat, t_sec, x_vrs)
dr = rho_v - rho_m
stats["dr_min"] = min(stats["dr_min"], dr)
stats["dr_max"] = max(stats["dr_max"], dr)
# parse + patch the observation fields
fields = []
ok = True
for ti, typ in enumerate(types):
ln = block[ti // 5] if ti // 5 < len(block) else ""
seg = ln[(ti % 5) * 16:(ti % 5) * 16 + 16].ljust(16)
raw, lli, ssi = seg[:14], seg[14], seg[15]
val = float(raw) if raw.strip() else None
if val is not None:
if typ[0] in ("C", "P"):
val += dr + C * perturb_ms * 1e-3
elif typ[0] == "L":
f = (glo_freq(typ[1], slots[sat]) if sys_id == "R"
else FREQ.get((sys_id, typ[1])))
if f is None:
ok = False
break
lam = C / f
val += dr / lam + C * perturb_ms * 1e-3 / lam
fields.append((val, lli, ssi))
if not ok:
stats["dropped"][sat] = stats["dropped"].get(sat, 0) + 1
continue
keep.append((sat, fields))
# rewrite epoch record with surviving satellites
if not keep:
continue
stats["epochs"] += 1
stats["sats"] += len(keep)
ids = [s for s, _ in keep]
head = line[:29] + f"{len(ids):3d}"
for k in range(0, len(ids), 12):
seg = "".join(ids[k:k + 12])
out.append((head if k == 0 else " " * 32) + seg)
for sat, fields in keep:
for li in range(nlines_per_sat):
chunk = fields[li * 5:(li + 1) * 5]
out.append("".join(fmt_obs(*f) for f in chunk).rstrip())
Path(out_path).write_text("\n".join(out) + "\n")
return stats, disp_km
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("master", type=Path, help="master RINEX 2.11 obs file")
ap.add_argument("--xyz", nargs=3, type=float, required=True, metavar=("X", "Y", "Z"),
help="virtual position, ECEF metres (same frame as --master-xyz)")
ap.add_argument("--sp3", type=Path, required=True)
ap.add_argument("--nav", type=Path, required=True,
help="RINEX 3 mixed nav (for GLONASS frequency slots)")
ap.add_argument("-o", "--out", type=Path, required=True)
ap.add_argument("--master-xyz", nargs=3, type=float, metavar=("X", "Y", "Z"),
help="master position override (else RINEX header position)")
ap.add_argument("--perturb-clock-ms", type=float, default=0.0,
help="TEST MODE: add a simulated receiver-clock offset to all "
"code+phase observables")
ap.add_argument("--marker", default="VBS0")
args = ap.parse_args()
sp3 = Sp3(args.sp3)
slots = glonass_slots(args.nav)
stats, disp = synthesize(args.master, args.out, args.xyz, sp3, slots,
master_xyz=args.master_xyz,
perturb_ms=args.perturb_clock_ms, marker=args.marker)
dropped = sum(stats["dropped"].values())
print(f"[vbs_synth] {args.master.name} -> {args.out.name}: "
f"displacement {disp:.3f} km, {stats['epochs']} epochs, "
f"{stats['sats']} sat-epochs kept, {dropped} dropped "
f"({len(stats['dropped'])} sats), dRho [{stats['dr_min']:.1f}, "
f"{stats['dr_max']:.1f}] m")
if __name__ == "__main__":
main()

8
gcgc.env.example Normal file
View file

@ -0,0 +1,8 @@
# GCGC RTN (rtn.usm.edu) NTRIP credentials.
#
# Setup: cp gcgc.env.example gcgc.env then fill in real values.
# gcgc.env is gitignored - never commit real credentials.
# stream_gps.py loads this file automatically; already-set environment
# variables take precedence.
GCGC_USER=your_username_here
GCGC_PASS=your_password_here

890
process_gps.py
View file

File diff suppressed because it is too large Load diff

71
stream_gps.py
View file

@ -47,6 +47,8 @@ from pathlib import Path
from pyubx2 import UBXMessage, UBXReader, UBX_PROTOCOL, SET_LAYER_RAM from pyubx2 import UBXMessage, UBXReader, UBX_PROTOCOL, SET_LAYER_RAM
SCRIPT_DIR = Path(__file__).resolve().parent
FIX_NAME = {0: "none", 1: "DR", 2: "2D", 3: "3D", 4: "GNSS+DR", 5: "time-only"} FIX_NAME = {0: "none", 1: "DR", 2: "2D", 3: "3D", 4: "GNSS+DR", 5: "time-only"}
CARR_NAME = {0: "", 1: "RTK-FLOAT", 2: "RTK-FIXED"} CARR_NAME = {0: "", 1: "RTK-FLOAT", 2: "RTK-FIXED"}
@ -145,9 +147,13 @@ def caster_reachable(server, port, timeout=8):
class SerialRTCMSink: class SerialRTCMSink:
"""Write-only wrapper handed to GNSSNTRIPClient as its output stream.""" """Write-only wrapper handed to GNSSNTRIPClient as its output stream.
def __init__(self, ser): Optionally tees the RTCM bytes to a log file so the session's correction
stream (e.g. a GCGC VRS - a virtual zero-baseline base) can be reused for
PPK later: process_gps.py <session.ubx> --base-obs <session.rtcm3>."""
def __init__(self, ser, log_path=None):
self._ser = ser self._ser = ser
self._log = open(log_path, "wb") if log_path else None
self.bytes_out = 0 self.bytes_out = 0
def write(self, data): def write(self, data):
@ -156,12 +162,18 @@ class SerialRTCMSink:
data = data[0] data = data[0]
if isinstance(data, (bytes, bytearray)): if isinstance(data, (bytes, bytearray)):
self._ser.write(bytes(data)) self._ser.write(bytes(data))
if self._log:
self._log.write(bytes(data))
self.bytes_out += len(data) self.bytes_out += len(data)
# queue-style interface (pygnssutils uses .put on Queue outputs) # queue-style interface (pygnssutils uses .put on Queue outputs)
def put(self, item): def put(self, item):
self.write(item) self.write(item)
def close(self):
if self._log:
self._log.close()
def start_ntrip(args, sink, reflat, reflon, app=None): def start_ntrip(args, sink, reflat, reflon, app=None):
"""app (with .get_coordinates()) => live GGA that follows a drifting platform; """app (with .get_coordinates()) => live GGA that follows a drifting platform;
@ -192,12 +204,15 @@ class Session:
self.stop = threading.Event() self.stop = threading.Event()
self.fix_hist = Counter() self.fix_hist = Counter()
self.hacc = [] self.hacc = []
self.t_first = None
self.warm_epochs = 0
self.tier = "none" self.tier = "none"
# latest fix, served to GNSSNTRIPClient.get_coordinates() for live GGA # latest fix, served to GNSSNTRIPClient.get_coordinates() for live GGA
self.live = {"lat": 0.0, "lon": 0.0, "alt": 0.0, "sep": 0.0} self.live = {"lat": 0.0, "lon": 0.0, "alt": 0.0, "sep": 0.0}
ts = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S") ts = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
self.csv_path = Path(f"stream_{ts}.csv") self.csv_path = Path(f"stream_{ts}.csv")
self.ubx_path = Path(f"stream_{ts}.ubx") self.ubx_path = Path(f"stream_{ts}.ubx")
self.rtcm_path = Path(f"stream_{ts}.rtcm3")
def get_coordinates(self): def get_coordinates(self):
"""pygnssutils app interface: live position for the NTRIP GGA uplink, """pygnssutils app interface: live position for the NTRIP GGA uplink,
@ -306,7 +321,7 @@ class Session:
reflat = reflon = 0.0 reflat = reflon = 0.0
else: else:
reflat, reflon = first reflat, reflon = first
sink = SerialRTCMSink(ser) sink = SerialRTCMSink(ser, self.rtcm_path)
app = None if args.gga_fixed else self app = None if args.gga_fixed else self
log("GGA uplink: " + ("fixed reference" if app is None log("GGA uplink: " + ("fixed reference" if app is None
else "live (follows platform drift)")) else "live (follows platform drift)"))
@ -347,8 +362,15 @@ class Session:
continue continue
p = last_pvt p = last_pvt
self.live.update(lat=p["lat"], lon=p["lon"], alt=p["height"]) self.live.update(lat=p["lat"], lon=p["lon"], alt=p["height"])
self.fix_hist[(p["fixType"], p["carrSoln"])] += 1 if self.t_first is None:
self.hacc.append(p["hAcc"]) self.t_first = p["utc"]
in_warmup = ((p["utc"] - self.t_first).total_seconds()
< self.args.warmup_min * 60)
if in_warmup:
self.warm_epochs += 1 # excluded from quality stats
else:
self.fix_hist[(p["fixType"], p["carrSoln"])] += 1
self.hacc.append(p["hAcc"])
writer.writerow([p["utc"].isoformat(), f"{p['lat']:.9f}", writer.writerow([p["utc"].isoformat(), f"{p['lat']:.9f}",
f"{p['lon']:.9f}", f"{p['height']:.3f}", f"{p['lon']:.9f}", f"{p['height']:.3f}",
f"{p['hAcc']:.3f}", f"{p['vAcc']:.3f}", f"{p['hAcc']:.3f}", f"{p['vAcc']:.3f}",
@ -374,6 +396,8 @@ class Session:
ntrip_client.stop() ntrip_client.stop()
except Exception: except Exception:
pass pass
if sink is not None:
sink.close()
csv_f.close() csv_f.close()
if raw_log: if raw_log:
raw_log.close() raw_log.close()
@ -397,9 +421,12 @@ class Session:
def _summary(self): def _summary(self):
n = sum(self.fix_hist.values()) n = sum(self.fix_hist.values())
if not n: if not n:
log("no position epochs received") log(f"no post-warm-up position epochs received "
f"({self.warm_epochs} warm-up epochs, stats need "
f">{self.args.warmup_min:g} min of data)")
return return
log(f"session summary: {n} epochs, tier = {self.tier}") log(f"session summary: {n} epochs in stats "
f"(+{self.warm_epochs} warm-up epochs excluded), tier = {self.tier}")
for (fix, carr), c in sorted(self.fix_hist.items(), key=lambda kv: -kv[1]): for (fix, carr), c in sorted(self.fix_hist.items(), key=lambda kv: -kv[1]):
lab = FIX_NAME.get(fix, "?") + ("/" + CARR_NAME[carr] if carr else "") lab = FIX_NAME.get(fix, "?") + ("/" + CARR_NAME[carr] if carr else "")
log(f" {lab:<14s} {c:6d} ({100*c/n:.1f}%)") log(f" {lab:<14s} {c:6d} ({100*c/n:.1f}%)")
@ -407,11 +434,34 @@ class Session:
log(f" hAcc: mean {sum(h)/len(h):.3f} m, median {h[len(h)//2]:.3f} m, " log(f" hAcc: mean {sum(h)/len(h):.3f} m, median {h[len(h)//2]:.3f} m, "
f"95% {h[int(len(h)*0.95)]:.3f} m") f"95% {h[int(len(h)*0.95)]:.3f} m")
if not self.args.replay: if not self.args.replay:
log(f" outputs: {self.csv_path}, {self.ubx_path}") outputs = [str(self.csv_path), str(self.ubx_path)]
log(f" post-process the raw log: venv/bin/python process_gps.py {self.ubx_path}") if self.rtcm_path.is_file() and self.rtcm_path.stat().st_size:
outputs.append(str(self.rtcm_path))
log(f" outputs: {', '.join(outputs)}")
log(f" post-process vs the recorded VRS (zero-baseline PPK): "
f"venv/bin/python process_gps.py {self.ubx_path} "
f"--base-obs {self.rtcm_path}")
else:
self.rtcm_path.unlink(missing_ok=True)
log(f" outputs: {', '.join(outputs)}")
log(f" post-process the raw log: venv/bin/python process_gps.py {self.ubx_path}")
def load_env_file(path=None):
"""Load KEY=VALUE lines from gcgc.env (untracked) into the environment.
Real environment variables take precedence over the file."""
path = path or SCRIPT_DIR / "gcgc.env"
if not path.is_file():
return
for line in path.read_text().splitlines():
line = line.strip().removeprefix("export ")
if line and not line.startswith("#") and "=" in line:
k, v = line.split("=", 1)
os.environ.setdefault(k.strip(), v.strip().strip("'\""))
def main(): def main():
load_env_file()
ap = argparse.ArgumentParser(description=__doc__, ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter) formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--port", help="serial port (COM7, /dev/ttyUSB0, ...)") ap.add_argument("--port", help="serial port (COM7, /dev/ttyUSB0, ...)")
@ -435,6 +485,9 @@ def main():
ap.add_argument("--ntrip-user", default=os.environ.get("GCGC_USER")) ap.add_argument("--ntrip-user", default=os.environ.get("GCGC_USER"))
ap.add_argument("--ntrip-pass", default=os.environ.get("GCGC_PASS")) ap.add_argument("--ntrip-pass", default=os.environ.get("GCGC_PASS"))
ap.add_argument("--gga-interval", type=int, default=15) ap.add_argument("--gga-interval", type=int, default=15)
ap.add_argument("--warmup-min", type=float, default=5.0,
help="exclude the first N minutes from session quality stats "
"(receiver warm-up); CSV keeps all epochs")
args = ap.parse_args() args = ap.parse_args()
if not args.replay and not args.port: if not args.replay and not args.port: