ublox-gps-data-processing/README.md

u-blox GPS Module Data Procesing (u-blox ZED-X20P)

Tooling for evaluating u-blox ZED-X20P all-band GNSS modules, targeting buoy deployments. Converts raw u-center capture files (.ubx) into corrected positions and produces a full diagnostics report, and supports live streaming with real-time corrections.

Two operating modes:

Mode	Script	Accuracy	Correction source
Post-processing	process_gps.py	dm-cm (PPK)	NGS CORS base data, auto-fetched (free, open)
Real-time streaming	stream_gps.py	1-3 cm (RTK fix)	GCGC RTN NTRIP (free registration) or Galileo HAS (FW ≥ HPG 2.10)

Both consume/produce standard formats (RINEX 3, .pos, CSV, KML) via RTKLIB (demo5 fork) and widely used open-source Python libraries (pyubx2, pygnssutils, numpy, pandas, matplotlib).

Motion policy (important)

Processing never assumes the antenna was stationary. Buoys drift slowly and drift within a few metres is statistically indistinguishable from measurement noise. Everything defaults to kinematic; pass --static to process_gps.py only when the antenna is independently known to have been fixed (e.g. bench/roof test).

Quick start

bash setup.sh                                    # one-time environment bootstrap
venv/bin/python process_gps.py <capture.ubx>     # process a capture
# results in processed/<capture-stem>/  (report.html is the human summary)

Requirements

Linux with python3 (3.10+), gcc/make, git, curl, and network access for setup.sh downloads. No sudo required and everything installs into the project directory (venv/, tools/).

---

setup.sh

Environment bootstrap (safe to re-run; each step skips if already satisfied):

1. Python venv (venv/): pyubx2, pygnssutils, pyserial, numpy, pandas, pyproj, matplotlib.
2. RTKLIB demo5 v2.5.1 built from source into tools/bin/ (convbin, rnx2rtkp, pos2kml). The demo5 fork is required: stock RTKLIB 2.4.3 silently drops all BeiDou and GPS L5 / Galileo E5 observations from this receiver (~30% of the measurements). If the build fails, setup falls back to the stock apt package and creates a tools/STOCK_RTKLIB marker — process_gps.py then adds a prominent data-loss warning to every report until a demo5 build succeeds.
3. Antenna calibration models into tools/antex/: igs20.atx (IGS, covers CORS base-station antennas) and ngs20.atx (NGS absolute calibrations, covers the rover antenna). The rover antenna is resolved automatically and written to tools/antenna.json (see Antenna configuration below).
4. CRX2RNX (Hatanaka RINEX decompressor) — fallback for CORS stations that only publish compressed .d observation files.
5. NGS CORS station catalog cache (used for nearest-base auto-selection).
6. Self-check summary of every tool and the resolved antenna.

process_gps.py

venv/bin/python process_gps.py <capture.ubx> [options]

Stages (idempotent — outputs are cached, so re-running later only fills in what was previously unavailable, e.g. CORS/IGS data that had not been published yet):

1. inventory — parses the .ubx (RXM-RAWX/SFRBX raw observations, NAV-PVT/SAT/SIG, MON-RF) and the .uc2x sidecar's session metadata. A capture without raw observations (e.g. u-center debug-mode logs) gets a diagnostics-only report and a clear notice.
2. rinex — RINEX 3.04 obs+nav via demo5 convbin; verifies epoch count and BDS/L5 presence; extracts broadcast ionosphere parameters.
3. single — standalone single-point solution (~1 m, works offline).
4. ppk — differential post-processing against the nearest operational NGS CORS station (auto-selected by distance from the capture's own mean position; MSEV at 34 km for the Hattiesburg test site). Hourly base files are fetched automatically (~1 h publication lag; if not yet published the stage reports "re-run later" and everything else proceeds). Base antenna type is read from the base RINEX header; base coordinates are ITRF2020, velocity-propagated to the capture epoch.
5. ppp (--ppp) — precise point positioning against IGS rapid/final orbit+clock products (anonymous download from BKG/ESA, ~1 day lag). Kinematic PPP by default per the motion policy; note RTKLIB's PPP-kinematic engine is weak and may yield no solution — PPK is the reference method.
6. track — track.csv (UTC + decimal-degree lat/lon + per-epoch quality and 1-σ sd columns + onboard comparison columns) and track.kml.
7. diagnostics — diagnostics.json, report.md, report.html (self-contained, figures embedded) covering: constellation metrics (per-band C/N0, cycle slips, residuals), ionosphere (broadcast Klobuchar + measured dual-frequency slant delay/TECU), positional error metrics (RMS/R50/R95 dispersion, formal σ, inter-solution offsets), validation of every computed solution against the receiver's own onboard NAV-PVT positions (epoch-matched, PASS/FAIL), timing (epoch jitter, tAcc, leap seconds), and RF health (jamming indicator, AGC, antenna supervisor state).

Options

Flag	Meaning
--interval N	track.csv output interval in seconds (default 1 = native rate)
--warmup-min N	receiver warm-up window excluded from ALL quality metrics and figures (default 5; track.csv keeps the full session; note shown atop both reports)
--static	antenna verifiably stationary: enables static PPK/PPP solutions and static error framing. Never for buoys.
--no-ppk	skip base-station download/processing
--no-vrs	skip the synthesized GCGC VRS solution
--ppp	additionally run PPP against IGS precise products
--base SSSS	force a specific CORS station id (e.g. MSEV)
--base-obs FILE...	explicit base file(s) for the parallel GCGC/local PPK method — normally unnecessary (<capture>.rtcm3 sidecars and gcgc_base/ are auto-discovered)
--base-xyz X Y Z	ECEF position of the local base (else its RINEX header position is used — verify its datum)
--max-base-km KM	base-station search radius (default 150)
--force-stage S	delete and re-run one stage (rinex,single,ppk,ppp,track)

Antenna configuration

Receiver-antenna phase-center corrections (PCO/PCV) matter at the cm level and are applied during PPK/PPP via ANTEX calibration files.

• Automatic: setup.sh downloads ngs20.atx/igs20.atx and searches for the project's antenna (ArduSimple calibrated quad-band, NGS calibration AS-ANT3BCAL01). The resolved 20-character ANTEX type string is written to tools/antenna.json:

  { "antex_name": "AS-ANT3BCAL     NONE", "source": "ngs20.atx" }
  

• Different antenna: edit antex_name in tools/antenna.json to the exact ANTEX type string of your antenna (padding/spacing matters, copy it verbatim from the TYPE / SERIAL NO line of an antenna block in tools/antex/ngs20.atx or igs20.atx). Browse available calibrations at https://geodesy.noaa.gov/ANTCAL/ . If your antenna has an NGS calibration but a different model string, also update the ANT3B search pattern in setup.sh so future setups re-resolve it automatically.

• No calibration available: set antex_name to null (or leave it unresolved) — processing continues without rover PCO/PCV and the report carries a warning. Expect a few cm of systematic error, mostly in height.

• Base antenna needs no configuration: its type is read from the CORS RINEX header and looked up in the same ANTEX files.

• Antenna height: assumed 0 (positions refer to the antenna reference point). For a surveyed monument, subtract your ARP offset downstream.

Other important settings and inputs

Capture requirements. Input .ubx files must contain raw observations: UBX-RXM-RAWX + UBX-RXM-SFRBX enabled (plus NAV-PVT; NAV-SAT/NAV-SIG/MON-RF enrich diagnostics). In u-center, do not record with the "debug messages" option — it floods the log with undocumented TRK/TUN/DBG/SEC messages and, in past sessions here, raw output was configured in flash but not active in RAM. The optional .uc2x sidecar (u-center 2 index) is only used for its session metadata header.

Real-time credentials (stream_gps.py). Free registration at http://rtn.usm.edu/RegisterAccount.aspx (Mississippi GCGC RTN). Provide the credentials on the machine that runs the streamer — preferred: copy the template and fill it in (the file is gitignored and loaded automatically):

cp gcgc.env.example gcgc.env    # then edit gcgc.env with the real values
venv/bin/python stream_gps.py --port /dev/ttyUSB0     # or COM7 on Windows

Alternatives: export GCGC_USER=... GCGC_PASS=... (or Windows setx GCGC_USER ...) — set variables take precedence over gcgc.env. Avoid the --ntrip-user/--ntrip-pass flags outside of quick tests: command-line arguments end up in shell history and process listings. Never commit real credentials; authentication is verified on the first NTRIP connect (a 401 appears in the log as "NTRIP connection failed").

Key streaming defaults (buoy-oriented): receiver dynamic-platform model sea (--dynmodel to override), live NMEA GGA uplink so the network-RTK virtual reference follows platform drift (--gga-fixed for bench tests), raw RXM-RAWX/SFRBX always logged so every session is post-processable, and --replay <file.ubx> offline test mode. Galileo HAS (~20 cm, no internet) activates automatically as fallback when firmware ≥ HPG 2.10 is detected.

GCGC data in post-processing (automatic parallel method). Beyond the real-time streams, a GCGC account also gives access to their Reference Data Shop (on-demand static/RINEX downloads from their 52 stations). No special invocation is needed — running process_gps.py <capture.ubx> as usual auto-discovers local GCGC data and, when found, runs a second PPK solution (ppk_gcgc) in parallel with the NGS CORS one, reported alongside all other results (it ranks first for the track output, since its baseline is shortest). Discovered sources:

1. <capture>.rtcm3 sidecar — every stream_gps.py NTRIP session records its incoming VRS correction stream next to the raw log. A VRS is a virtual base at the receiver's own position (zero baseline) — the best possible PPK geometry.
2. gcgc_base/ folder — drop Reference Data Shop RINEX downloads here; files whose observation window overlaps the capture are used automatically.

--base-obs FILE... explicitly feeds this method instead of auto-discovery. Note GCGC coordinates are NAD83(2011): supply --base-xyz (ECEF, ITRF) if frame consistency with the NGS-based solution matters — otherwise the base RINEX header position is used and the report warns about the ~1.5 m datum offset.

Synthesized GCGC VRS (ppk_vrs, standard). When the selected base is a GCGC network station, the pipeline additionally synthesizes a virtual base at the capture's own position (zero baseline) from that master's observations and runs PPK against it — the engine was gate-validated and benchmarked equivalent to Trimble Pivot's commercial VRS (see experiments/FINDINGS.md). Requires ESA orbit products (~1 day lag; the stage reports "pending" until then). Only provided when GCGC is available — for captures outside the GCGC network the solution is skipped automatically. Disable with --no-vrs. The track output picks among the PPK-family solutions by measured quality (empirical scatter), so a poor network-edge VRS never silently wins over a better direct solution.

Datums. PPK output is in the CORS base frame (ITRF2020, current epoch); GCGC real-time corrections are NAD83(2011) epoch 2010.0 — a constant ~1.5 m offset from ITRF/WGS84 in CONUS. Each CSV records its frame in the header; comparisons in the report account for the reference used. Onboard receiver positions are WGS84-aligned.

Data-source timing. CORS hourly files: ~1 h lag, expire after ~2 days (daily files remain, decimated to 30 s after 30 days). IGS rapid products: ~1 day lag. The pipeline caches everything it fetches under processed/<stem>/ and tells you when a re-run will find more data.

Repository layout

setup.sh                 environment bootstrap (run first)
process_gps.py           post-processing pipeline (Mode 2)
stream_gps.py            real-time streaming client (Mode 1)
run.sh                   convenience wrapper
notes/                   manual workflow notes + combine_rinex.sh helper
tools/                   built binaries, ANTEX models, caches   (generated)
venv/                    Python environment                     (generated)
processed/<capture>/     per-capture outputs: RINEX, .pos solutions,
                         track.csv, track.kml, figures/, diagnostics.json,
                         report.md, report.html                 (generated)

---

Validation and test results (July 2026)

Full detail, gate definitions, and reproduction commands: experiments/FINDINGS.md.

Test suite passed The virtual-reference-station tooling (experiments/vbs_synth.py, was integrated into the pipeline as ppk_vrs) passed four correctness tests: bit-identical output at zero displacement; round-trip displacement error of one quantization step (0.001); insensitivity to a physically representative 1 microsecond receiver-clock offset (millimetre agreement; an unphysical 1 ms test also documented why timestamp/pseudorange self-consistency is mandatory in synthesized files); and a ground-truth test in which a station with known coordinates, processed against a base synthesized at its own location from a station 44 km away, reproduced the direct-baseline solution statistics.

Ionosphere interpolation (leave-one-out, five stations). Carrier-leveled dual-frequency slant ionosphere interpolated across the Gulf Coast Geospatial Center (GCGC) / National Geodetic Survey (NGS) station network predicts a held-out station's ionosphere to 5-10 cm (median) inside the network, growing about 1.2 cm per 10 km — including 19 cm at a station 135 km outside the fitting cluster. Offshore implication: roughly 10-15 cm of ionospheric prediction error at 60 km beyond the coastal network.

Benchmark against the commercial product. Using ground-truth station MARY: our synthesized virtual base and Trimble Pivot's commercial virtual-reference- station product were statistically equivalent (fixed-solution horizontal RMS 7.5 cm vs 7.8 cm; both roughly halve the error and the wrong-fix count of the direct 44 km baseline). Observation-domain analysis showed the commercial product embeds only ~3-7 cm of correction content beyond pure geometry, roughly flat out to 92 km of displacement — including virtual stations the operator's portal generated 25 km and 60 km offshore. One commercial file (at the sparse network edge near Hattiesburg) carried a ~5 m bias and 3x noise; our own tooling at the same site with the same master station was clean, which is why the pipeline selects its track solution by measured dispersion rather than by nominal baseline length.

Why "PPK - synthesized virtual reference station (kinematic)" is likely the best solution for the buoy / boat use case.

1. Zero-length baseline at the platform: the dominant baseline-dependent errors shrink to the atmospheric difference between master and platform, and the geometry term vanishes entirely.
2. Measured performance: best dispersion of all methods on the reference capture (50% radius 0.39 m vs 0.41 m for the direct base), and the best fixed-solution accuracy in the ground-truth benchmark.
3. Kinematic processing makes no motion assumption — correct for a drifting buoy by construction.
4. Offshore reach: works wherever any GCGC/NGS master station exists within ~150 km; the commercial product's extrapolated corrections were shown to add only centimetres of content at those distances, so little is lost by synthesizing locally - and quality stays under our control (the network-edge failure above was a commercial-side defect our synthesis avoided).
5. Fully automatic with open data: needs only the already-downloaded base observations plus European Space Agency orbit products (about one day of latency).

Caveats: validated on one (quiet) day so far; GPS+GLONASS until multi-GNSS orbit products are available for a given day; the measured ionospheric interpolation layer is not yet applied to the synthesized observations (planned); disturbed-ionosphere days untested.