ublox-gps-data-processing/README.md

# u-blox GPS Module Data Procesing (u-blox ZED-X20P)

Tooling for evaluating u-blox ZED-X20P all-band GNSS modules, targeting **buoy
deployments**. Converts raw u-center capture files (`.ubx`) into corrected
positions and produces a full diagnostics report, and supports live streaming
with real-time corrections.

Two operating modes:

| Mode | Script | Accuracy | Correction source |
|---|---|---|---|
| Post-processing | `process_gps.py` | dm-cm (PPK) | NGS CORS base data, auto-fetched (free, open) |
| Real-time streaming | `stream_gps.py` | 1-3 cm (RTK fix) | GCGC RTN NTRIP (free registration) or Galileo HAS (FW ≥ HPG 2.10) |

Both consume/produce standard formats (RINEX 3, `.pos`, CSV, KML) via RTKLIB
(demo5 fork) and widely used open-source Python libraries (pyubx2, pygnssutils,
numpy, pandas, matplotlib).

## Motion policy (important)

**Processing never assumes the antenna was stationary.** Buoys drift slowly and
drift within a few metres is statistically indistinguishable from measurement
noise. Everything defaults to kinematic; pass `--static` to `process_gps.py` 
only when the antenna is independently known to have been fixed (e.g. bench/roof test).

## Quick start

```bash
bash setup.sh                                    # one-time environment bootstrap
venv/bin/python process_gps.py <capture.ubx>     # process a capture
# results in processed/<capture-stem>/  (report.html is the human summary)
```

## Requirements

Linux with `python3` (3.10+), `gcc`/`make`, `git`, `curl`, and network access
for `setup.sh` downloads. No sudo required and everything installs into the
project directory (`venv/`, `tools/`).

---

## setup.sh

Environment bootstrap (safe to re-run; each step skips if already satisfied):

1. **Python venv** (`venv/`): pyubx2, pygnssutils, pyserial, numpy, pandas,
   pyproj, matplotlib.
2. **RTKLIB demo5 v2.5.1** built from source into `tools/bin/`
   (`convbin`, `rnx2rtkp`, `pos2kml`). The demo5 fork is required: stock
   RTKLIB 2.4.3 silently drops all BeiDou and GPS L5 / Galileo E5 observations
   from this receiver (~30% of the measurements). If the build fails, setup
   falls back to the stock apt package and creates a `tools/STOCK_RTKLIB`
   marker — `process_gps.py` then adds a prominent data-loss warning to every
   report until a demo5 build succeeds.
3. **Antenna calibration models** into `tools/antex/`: `igs20.atx` (IGS, covers
   CORS base-station antennas) and `ngs20.atx` (NGS absolute calibrations,
   covers the rover antenna). The rover antenna is resolved automatically and
   written to `tools/antenna.json` (see *Antenna configuration* below).
4. **CRX2RNX** (Hatanaka RINEX decompressor) — fallback for CORS stations that
   only publish compressed `.d` observation files.
5. **NGS CORS station catalog** cache (used for nearest-base auto-selection).
6. Self-check summary of every tool and the resolved antenna.

## process_gps.py

```
venv/bin/python process_gps.py <capture.ubx> [options]
```

Stages (idempotent — outputs are cached, so re-running later only fills in
what was previously unavailable, e.g. CORS/IGS data that had not been
published yet):

1. **inventory** — parses the `.ubx` (RXM-RAWX/SFRBX raw observations,
   NAV-PVT/SAT/SIG, MON-RF) and the `.uc2x` sidecar's session metadata.
   A capture without raw observations (e.g. u-center debug-mode logs) gets a
   diagnostics-only report and a clear notice.
2. **rinex** — RINEX 3.04 obs+nav via demo5 `convbin`; verifies epoch count and
   BDS/L5 presence; extracts broadcast ionosphere parameters.
3. **single** — standalone single-point solution (~1 m, works offline).
4. **ppk** — differential post-processing against the nearest operational NGS
   CORS station (auto-selected by distance from the capture's own mean
   position; MSEV at 34 km for the Hattiesburg test site). Hourly base files
   are fetched automatically (~1 h publication lag; if not yet published the
   stage reports "re-run later" and everything else proceeds). Base antenna
   type is read from the base RINEX header; base coordinates are ITRF2020,
   velocity-propagated to the capture epoch.
5. **ppp** (`--ppp`) — precise point positioning against IGS rapid/final
   orbit+clock products (anonymous download from BKG/ESA, ~1 day lag).
   Kinematic PPP by default per the motion policy; note RTKLIB's PPP-kinematic
   engine is weak and may yield no solution — PPK is the reference method.
6. **track** — `track.csv` (UTC + decimal-degree lat/lon + per-epoch quality
   and 1-σ sd columns + onboard comparison columns) and `track.kml`.
7. **diagnostics** — `diagnostics.json`, `report.md`, `report.html`
   (self-contained, figures embedded) covering: constellation metrics (per-band
   C/N0, cycle slips, residuals), ionosphere (broadcast Klobuchar + measured
   dual-frequency slant delay/TECU), positional error metrics (RMS/R50/R95
   dispersion, formal σ, inter-solution offsets), **validation of every
   computed solution against the receiver's own onboard NAV-PVT positions**
   (epoch-matched, PASS/FAIL), timing (epoch jitter, tAcc, leap seconds), and
   RF health (jamming indicator, AGC, antenna supervisor state).

### Options

| Flag | Meaning |
|---|---|
| `--interval N` | `track.csv` output interval in seconds (default 1 = native rate) |
| `--static` | antenna verifiably stationary: enables static PPK/PPP solutions and static error framing. **Never for buoys.** |
| `--no-ppk` | skip base-station download/processing |
| `--ppp` | additionally run PPP against IGS precise products |
| `--base SSSS` | force a specific CORS station id (e.g. `MSEV`) |
| `--max-base-km KM` | base-station search radius (default 150) |
| `--force-stage S` | delete and re-run one stage (`rinex`,`single`,`ppk`,`ppp`,`track`) |

## Antenna configuration

Receiver-antenna phase-center corrections (PCO/PCV) matter at the cm level and
are applied during PPK/PPP via ANTEX calibration files.

- **Automatic**: `setup.sh` downloads `ngs20.atx`/`igs20.atx` and searches for
  the project's antenna (ArduSimple calibrated quad-band, NGS calibration
  AS-ANT3BCAL01). The resolved 20-character ANTEX type string is written to
  `tools/antenna.json`:

  ```json
  { "antex_name": "AS-ANT3BCAL     NONE", "source": "ngs20.atx" }
  ```

- **Different antenna**: edit `antex_name` in `tools/antenna.json` to the exact
  ANTEX type string of your antenna (padding/spacing matters, copy it verbatim
  from the `TYPE / SERIAL NO` line of an antenna block in
  `tools/antex/ngs20.atx` or `igs20.atx`). Browse available calibrations at
  https://geodesy.noaa.gov/ANTCAL/ . If your antenna has an NGS calibration but
  a different model string, also update the `ANT3B` search pattern in `setup.sh`
  so future setups re-resolve it automatically.
- **No calibration available**: set `antex_name` to `null` (or leave it
  unresolved) — processing continues without rover PCO/PCV and the report
  carries a warning. Expect a few cm of systematic error, mostly in height.
- **Base antenna** needs no configuration: its type is read from the CORS RINEX
  header and looked up in the same ANTEX files.
- **Antenna height**: assumed 0 (positions refer to the antenna reference
  point). For a surveyed monument, subtract your ARP offset downstream.

## Other important settings and inputs

**Capture requirements.** Input `.ubx` files must contain raw observations:
UBX-RXM-RAWX + UBX-RXM-SFRBX enabled (plus NAV-PVT; NAV-SAT/NAV-SIG/MON-RF
enrich diagnostics). In u-center, do **not** record with the "debug messages"
option — it floods the log with undocumented TRK/TUN/DBG/SEC messages and, in
past sessions here, raw output was configured in flash but not active in RAM.
The optional `.uc2x` sidecar (u-center 2 index) is only used for its session
metadata header.

**Real-time credentials (`stream_gps.py`).** Free registration at
http://rtn.usm.edu/RegisterAccount.aspx (Mississippi GCGC RTN), then:

```bash
export GCGC_USER=<username> GCGC_PASS=<password>
venv/bin/python stream_gps.py --port /dev/ttyUSB0     # or COM7 on Windows
```

Key streaming defaults (buoy-oriented): receiver dynamic-platform model `sea`
(`--dynmodel` to override), **live** NMEA GGA uplink so the network-RTK virtual
reference follows platform drift (`--gga-fixed` for bench tests), raw
RXM-RAWX/SFRBX always logged so every session is post-processable, and
`--replay <file.ubx>` offline test mode. Galileo HAS (~20 cm, no internet)
activates automatically as fallback when firmware ≥ HPG 2.10 is detected.

**Datums.** PPK output is in the CORS base frame (ITRF2020, current epoch);
GCGC real-time corrections are NAD83(2011) epoch 2010.0 — a constant ~1.5 m
offset from ITRF/WGS84 in CONUS. Each CSV records its frame in the header;
comparisons in the report account for the reference used. Onboard receiver
positions are WGS84-aligned.

**Data-source timing.** CORS hourly files: ~1 h lag, expire after ~2 days
(daily files remain, decimated to 30 s after 30 days). IGS rapid products:
~1 day lag. The pipeline caches everything it fetches under
`processed/<stem>/` and tells you when a re-run will find more data.

## Repository layout

```
setup.sh                 environment bootstrap (run first)
process_gps.py           post-processing pipeline (Mode 2)
stream_gps.py            real-time streaming client (Mode 1)
run.sh                   convenience wrapper
notes/                   manual workflow notes + combine_rinex.sh helper
tools/                   built binaries, ANTEX models, caches   (generated)
venv/                    Python environment                     (generated)
processed/<capture>/     per-capture outputs: RINEX, .pos solutions,
                         track.csv, track.kml, figures/, diagnostics.json,
                         report.md, report.html                 (generated)
```