ublox-gps-data-processing/README.md

# u-blox GPS Module Data Procesing (u-blox ZED-X20P)

Tooling for evaluating u-blox ZED-X20P all-band GNSS modules, targeting **buoy
deployments**. Converts raw u-center capture files (`.ubx`) into corrected
positions and produces a full diagnostics report, and supports live streaming
with real-time corrections.

Two operating modes:

| Mode | Script | Accuracy | Correction source |
|---|---|---|---|
| Post-processing | `process_gps.py` | dm-cm (PPK) | NGS CORS base data, auto-fetched (free, open) |
| Real-time streaming | `stream_gps.py` | 1-3 cm (RTK fix) | GCGC RTN NTRIP (free registration) or Galileo HAS (FW ≥ HPG 2.10) |

Both consume/produce standard formats (RINEX 3, `.pos`, CSV, KML) via RTKLIB
(demo5 fork) and widely used open-source Python libraries (pyubx2, pygnssutils,
numpy, pandas, matplotlib).

## Motion policy (important)

**Processing never assumes the antenna was stationary.** Buoys drift slowly and
drift within a few metres is statistically indistinguishable from measurement
noise. Everything defaults to kinematic; pass `--static` to `process_gps.py` 
only when the antenna is independently known to have been fixed (e.g. bench/roof test).

## Quick start

```bash
bash setup.sh                                    # one-time environment bootstrap
venv/bin/python process_gps.py <capture.ubx>     # process a capture
# results in processed/<capture-stem>/  (report.html is the human summary)
```

## Requirements

Linux with `python3` (3.10+), `gcc`/`make`, `git`, `curl`, and network access
for `setup.sh` downloads. No sudo required and everything installs into the
project directory (`venv/`, `tools/`).

---

## setup.sh

Environment bootstrap (safe to re-run; each step skips if already satisfied):

1. **Python venv** (`venv/`): pyubx2, pygnssutils, pyserial, numpy, pandas,
   pyproj, matplotlib.
2. **RTKLIB demo5 v2.5.1** built from source into `tools/bin/`
   (`convbin`, `rnx2rtkp`, `pos2kml`). The demo5 fork is required: stock
   RTKLIB 2.4.3 silently drops all BeiDou and GPS L5 / Galileo E5 observations
   from this receiver (~30% of the measurements). If the build fails, setup
   falls back to the stock apt package and creates a `tools/STOCK_RTKLIB`
   marker — `process_gps.py` then adds a prominent data-loss warning to every
   report until a demo5 build succeeds.
3. **Antenna calibration models** into `tools/antex/`: `igs20.atx` (IGS, covers
   CORS base-station antennas) and `ngs20.atx` (NGS absolute calibrations,
   covers the rover antenna). The rover antenna is resolved automatically and
   written to `tools/antenna.json` (see *Antenna configuration* below).
4. **CRX2RNX** (Hatanaka RINEX decompressor) — fallback for CORS stations that
   only publish compressed `.d` observation files.
5. **NGS CORS station catalog** cache (used for nearest-base auto-selection).
6. Self-check summary of every tool and the resolved antenna.

## process_gps.py

```
venv/bin/python process_gps.py <capture.ubx> [options]
```

Stages (idempotent — outputs are cached, so re-running later only fills in
what was previously unavailable, e.g. CORS/IGS data that had not been
published yet):

1. **inventory** — parses the `.ubx` (RXM-RAWX/SFRBX raw observations,
   NAV-PVT/SAT/SIG, MON-RF) and the `.uc2x` sidecar's session metadata.
   A capture without raw observations (e.g. u-center debug-mode logs) gets a
   diagnostics-only report and a clear notice.
2. **rinex** — RINEX 3.04 obs+nav via demo5 `convbin`; verifies epoch count and
   BDS/L5 presence; extracts broadcast ionosphere parameters.
3. **single** — standalone single-point solution (~1 m, works offline).
4. **ppk** — differential post-processing against the nearest operational NGS
   CORS station (auto-selected by distance from the capture's own mean
   position; MSEV at 34 km for the Hattiesburg test site). Hourly base files
   are fetched automatically (~1 h publication lag; if not yet published the
   stage reports "re-run later" and everything else proceeds). Base antenna
   type is read from the base RINEX header; base coordinates are ITRF2020,
   velocity-propagated to the capture epoch.
5. **ppp** (`--ppp`) — precise point positioning against IGS rapid/final
   orbit+clock products (anonymous download from BKG/ESA, ~1 day lag).
   Kinematic PPP by default per the motion policy; note RTKLIB's PPP-kinematic
   engine is weak and may yield no solution — PPK is the reference method.
6. **track** — `track.csv` (UTC + decimal-degree lat/lon + per-epoch quality
   and 1-σ sd columns + onboard comparison columns) and `track.kml`.
7. **diagnostics** — `diagnostics.json`, `report.md`, `report.html`
   (self-contained, figures embedded) covering: constellation metrics (per-band
   C/N0, cycle slips, residuals), ionosphere (broadcast Klobuchar + measured
   dual-frequency slant delay/TECU), positional error metrics (RMS/R50/R95
   dispersion, formal σ, inter-solution offsets), **validation of every
   computed solution against the receiver's own onboard NAV-PVT positions**
   (epoch-matched, PASS/FAIL), timing (epoch jitter, tAcc, leap seconds), and
   RF health (jamming indicator, AGC, antenna supervisor state).

### Options

| Flag | Meaning |
|---|---|
| `--interval N` | `track.csv` output interval in seconds (default 1 = native rate) |
| `--warmup-min N` | receiver warm-up window excluded from ALL quality metrics and figures (default 5; `track.csv` keeps the full session; note shown atop both reports) |
| `--static` | antenna verifiably stationary: enables static PPK/PPP solutions and static error framing. **Never for buoys.** |
| `--no-ppk` | skip base-station download/processing |
| `--no-vrs` | skip the synthesized GCGC VRS solution |
| `--ppp` | additionally run PPP against IGS precise products |
| `--base SSSS` | force a specific CORS station id (e.g. `MSEV`) |
| `--base-obs FILE...` | explicit base file(s) for the parallel GCGC/local PPK method — normally unnecessary (`<capture>.rtcm3` sidecars and `gcgc_base/` are auto-discovered) |
| `--base-xyz X Y Z` | ECEF position of the local base (else its RINEX header position is used — verify its datum) |
| `--max-base-km KM` | base-station search radius (default 150) |
| `--force-stage S` | delete and re-run one stage (`rinex`,`single`,`ppk`,`ppp`,`track`) |

## Antenna configuration

Receiver-antenna phase-center corrections (PCO/PCV) matter at the cm level and
are applied during PPK/PPP via ANTEX calibration files.

- **Automatic**: `setup.sh` downloads `ngs20.atx`/`igs20.atx` and searches for
  the project's antenna (ArduSimple calibrated quad-band, NGS calibration
  AS-ANT3BCAL01). The resolved 20-character ANTEX type string is written to
  `tools/antenna.json`:

  ```json
  { "antex_name": "AS-ANT3BCAL     NONE", "source": "ngs20.atx" }
  ```

- **Different antenna**: edit `antex_name` in `tools/antenna.json` to the exact
  ANTEX type string of your antenna (padding/spacing matters, copy it verbatim
  from the `TYPE / SERIAL NO` line of an antenna block in
  `tools/antex/ngs20.atx` or `igs20.atx`). Browse available calibrations at
  https://geodesy.noaa.gov/ANTCAL/ . If your antenna has an NGS calibration but
  a different model string, also update the `ANT3B` search pattern in `setup.sh`
  so future setups re-resolve it automatically.
- **No calibration available**: set `antex_name` to `null` (or leave it
  unresolved) — processing continues without rover PCO/PCV and the report
  carries a warning. Expect a few cm of systematic error, mostly in height.
- **Base antenna** needs no configuration: its type is read from the CORS RINEX
  header and looked up in the same ANTEX files.
- **Antenna height**: assumed 0 (positions refer to the antenna reference
  point). For a surveyed monument, subtract your ARP offset downstream.

## Other important settings and inputs

**Capture requirements.** Input `.ubx` files must contain raw observations:
UBX-RXM-RAWX + UBX-RXM-SFRBX enabled (plus NAV-PVT; NAV-SAT/NAV-SIG/MON-RF
enrich diagnostics). In u-center, do **not** record with the "debug messages"
option — it floods the log with undocumented TRK/TUN/DBG/SEC messages and, in
past sessions here, raw output was configured in flash but not active in RAM.
The optional `.uc2x` sidecar (u-center 2 index) is only used for its session
metadata header.

**Real-time credentials (`stream_gps.py`).** Free registration at
http://rtn.usm.edu/RegisterAccount.aspx (Mississippi GCGC RTN). Provide the
credentials on the machine that runs the streamer — preferred: copy the
template and fill it in (the file is gitignored and loaded automatically):

```bash
cp gcgc.env.example gcgc.env    # then edit gcgc.env with the real values
venv/bin/python stream_gps.py --port /dev/ttyUSB0     # or COM7 on Windows
```

Alternatives: `export GCGC_USER=... GCGC_PASS=...` (or Windows
`setx GCGC_USER ...`) — set variables take precedence over `gcgc.env`. Avoid
the `--ntrip-user`/`--ntrip-pass` flags outside of quick tests: command-line
arguments end up in shell history and process listings. Never commit real
credentials; authentication is verified on the first NTRIP connect (a 401
appears in the log as "NTRIP connection failed").

Key streaming defaults (buoy-oriented): receiver dynamic-platform model `sea`
(`--dynmodel` to override), **live** NMEA GGA uplink so the network-RTK virtual
reference follows platform drift (`--gga-fixed` for bench tests), raw
RXM-RAWX/SFRBX always logged so every session is post-processable, and
`--replay <file.ubx>` offline test mode. Galileo HAS (~20 cm, no internet)
activates automatically as fallback when firmware ≥ HPG 2.10 is detected.

**GCGC data in post-processing (automatic parallel method).** Beyond the
real-time streams, a GCGC account also gives access to their **Reference Data
Shop** (on-demand static/RINEX downloads from their 52 stations). No special
invocation is needed — running `process_gps.py <capture.ubx>` as usual
auto-discovers local GCGC data and, when found, runs a **second PPK solution
(`ppk_gcgc`) in parallel** with the NGS CORS one, reported alongside all other
results (it ranks first for the track output, since its baseline is shortest).
Discovered sources:

1. `<capture>.rtcm3` **sidecar** — every `stream_gps.py` NTRIP session records
   its incoming VRS correction stream next to the raw log. A VRS is a *virtual
   base at the receiver's own position* (zero baseline) — the best possible
   PPK geometry.
2. **`gcgc_base/` folder** — drop Reference Data Shop RINEX downloads here;
   files whose observation window overlaps the capture are used automatically.

`--base-obs FILE...` explicitly feeds this method instead of auto-discovery.
Note GCGC coordinates are NAD83(2011): supply `--base-xyz` (ECEF, ITRF) if
frame consistency with the NGS-based solution matters — otherwise the base
RINEX header position is used and the report warns about the ~1.5 m datum
offset.

**Synthesized GCGC VRS (`ppk_vrs`, standard).** When the selected base is a
GCGC network station, the pipeline additionally synthesizes a virtual base at
the capture's own position (zero baseline) from that master's observations and
runs PPK against it — the engine was gate-validated and benchmarked equivalent
to Trimble Pivot's commercial VRS (see `experiments/FINDINGS.md`). Requires
ESA orbit products (~1 day lag; the stage reports "pending" until then). Only
provided when GCGC is available — for captures outside the GCGC network the
solution is skipped automatically. Disable with `--no-vrs`. The track output
picks among the PPK-family solutions by *measured* quality (empirical scatter),
so a poor network-edge VRS never silently wins over a better direct solution.

**Datums.** PPK output is in the CORS base frame (ITRF2020, current epoch);
GCGC real-time corrections are NAD83(2011) epoch 2010.0 — a constant ~1.5 m
offset from ITRF/WGS84 in CONUS. Each CSV records its frame in the header;
comparisons in the report account for the reference used. Onboard receiver
positions are WGS84-aligned.

**Data-source timing.** CORS hourly files: ~1 h lag, expire after ~2 days
(daily files remain, decimated to 30 s after 30 days). IGS rapid products:
~1 day lag. The pipeline caches everything it fetches under
`processed/<stem>/` and tells you when a re-run will find more data.

## Repository layout

```
setup.sh                 environment bootstrap (run first)
process_gps.py           post-processing pipeline (Mode 2)
stream_gps.py            real-time streaming client (Mode 1)
run.sh                   convenience wrapper
notes/                   manual workflow notes + combine_rinex.sh helper
tools/                   built binaries, ANTEX models, caches   (generated)
venv/                    Python environment                     (generated)
processed/<capture>/     per-capture outputs: RINEX, .pos solutions,
                         track.csv, track.kml, figures/, diagnostics.json,
                         report.md, report.html                 (generated)
```

---

## Validation and test results (July 2026)

Full detail, gate definitions, and reproduction commands: `experiments/FINDINGS.md`.

**Test suite passed** The virtual-reference-station tooling
 (`experiments/vbs_synth.py`, was integrated into the pipeline as `ppk_vrs`)
passed four correctness tests: bit-identical output at zero displacement;
round-trip displacement error of one quantization step (0.001); insensitivity to
a physically representative 1 microsecond receiver-clock offset (millimetre
agreement; an unphysical 1 ms test also documented why timestamp/pseudorange
self-consistency is mandatory in synthesized files); and a ground-truth test in
which a station with known coordinates, processed against a base synthesized at
its own location from a station 44 km away, reproduced the direct-baseline
solution statistics.

**Ionosphere interpolation (leave-one-out, five stations).** Carrier-leveled
dual-frequency slant ionosphere interpolated across the Gulf Coast Geospatial
Center (GCGC) / National Geodetic Survey (NGS) station network predicts a
held-out station's ionosphere to 5-10 cm (median) inside the network, growing
about 1.2 cm per 10 km — including 19 cm at a station 135 km outside the fitting
cluster. Offshore implication: roughly 10-15 cm of ionospheric prediction error
at 60 km beyond the coastal network.

**Benchmark against the commercial product.** Using ground-truth station MARY:
our synthesized virtual base and Trimble Pivot's commercial virtual-reference-
station product were statistically equivalent (fixed-solution horizontal RMS
7.5 cm vs 7.8 cm; both roughly halve the error and the wrong-fix count of the
direct 44 km baseline). Observation-domain analysis showed the commercial
product embeds only ~3-7 cm of correction content beyond pure geometry, roughly
flat out to 92 km of displacement — including virtual stations the operator's
portal generated 25 km and 60 km offshore. One commercial file (at the sparse
network edge near Hattiesburg) carried a ~5 m bias and 3x noise; our own
tooling at the same site with the same master station was clean, which is why
the pipeline selects its track solution by measured dispersion rather than by
nominal baseline length.

**Why "PPK - synthesized virtual reference station (kinematic)" is likely the
best solution for the buoy / boat use case.**

1. *Zero-length baseline at the platform*: the dominant baseline-dependent
   errors shrink to the atmospheric difference between master and platform,
   and the geometry term vanishes entirely.
2. *Measured performance*: best dispersion of all methods on the reference
   capture (50% radius 0.39 m vs 0.41 m for the direct base), and the best
   fixed-solution accuracy in the ground-truth benchmark.
3. *Kinematic processing makes no motion assumption* — correct for a drifting
   buoy by construction.
4. *Offshore reach*: works wherever any GCGC/NGS master station exists within
   ~150 km; the commercial product's extrapolated corrections were shown to add
   only centimetres of content at those distances, so little is lost by
   synthesizing locally - and quality stays under our control (the network-edge
   failure above was a commercial-side defect our synthesis avoided).
5. *Fully automatic with open data*: needs only the already-downloaded base
   observations plus European Space Agency orbit products (about one day of
   latency).

Caveats: validated on one (quiet) day so far; GPS+GLONASS until multi-GNSS
orbit products are available for a given day; the measured ionospheric
interpolation layer is not yet applied to the synthesized observations
(planned); disturbed-ionosphere days untested.