193 lines
9.6 KiB
Markdown
193 lines
9.6 KiB
Markdown
# u-blox GPS Module Data Procesing (u-blox ZED-X20P)
|
||
|
||
Tooling for evaluating u-blox ZED-X20P all-band GNSS modules, targeting **buoy
|
||
deployments**. Converts raw u-center capture files (`.ubx`) into corrected
|
||
positions and produces a full diagnostics report, and supports live streaming
|
||
with real-time corrections.
|
||
|
||
Two operating modes:
|
||
|
||
| Mode | Script | Accuracy | Correction source |
|
||
|---|---|---|---|
|
||
| Post-processing | `process_gps.py` | dm-cm (PPK) | NGS CORS base data, auto-fetched (free, open) |
|
||
| Real-time streaming | `stream_gps.py` | 1-3 cm (RTK fix) | GCGC RTN NTRIP (free registration) or Galileo HAS (FW ≥ HPG 2.10) |
|
||
|
||
Both consume/produce standard formats (RINEX 3, `.pos`, CSV, KML) via RTKLIB
|
||
(demo5 fork) and widely used open-source Python libraries (pyubx2, pygnssutils,
|
||
numpy, pandas, matplotlib).
|
||
|
||
## Motion policy (important)
|
||
|
||
**Processing never assumes the antenna was stationary.** Buoys drift slowly and
|
||
drift within a few metres is statistically indistinguishable from measurement
|
||
noise. Everything defaults to kinematic; pass `--static` to `process_gps.py`
|
||
only when the antenna is independently known to have been fixed (e.g. bench/roof test).
|
||
|
||
## Quick start
|
||
|
||
```bash
|
||
bash setup.sh # one-time environment bootstrap
|
||
venv/bin/python process_gps.py <capture.ubx> # process a capture
|
||
# results in processed/<capture-stem>/ (report.html is the human summary)
|
||
```
|
||
|
||
## Requirements
|
||
|
||
Linux with `python3` (3.10+), `gcc`/`make`, `git`, `curl`, and network access
|
||
for `setup.sh` downloads. No sudo required and everything installs into the
|
||
project directory (`venv/`, `tools/`).
|
||
|
||
---
|
||
|
||
## setup.sh
|
||
|
||
Environment bootstrap (safe to re-run; each step skips if already satisfied):
|
||
|
||
1. **Python venv** (`venv/`): pyubx2, pygnssutils, pyserial, numpy, pandas,
|
||
pyproj, matplotlib.
|
||
2. **RTKLIB demo5 v2.5.1** built from source into `tools/bin/`
|
||
(`convbin`, `rnx2rtkp`, `pos2kml`). The demo5 fork is required: stock
|
||
RTKLIB 2.4.3 silently drops all BeiDou and GPS L5 / Galileo E5 observations
|
||
from this receiver (~30% of the measurements). If the build fails, setup
|
||
falls back to the stock apt package and creates a `tools/STOCK_RTKLIB`
|
||
marker — `process_gps.py` then adds a prominent data-loss warning to every
|
||
report until a demo5 build succeeds.
|
||
3. **Antenna calibration models** into `tools/antex/`: `igs20.atx` (IGS, covers
|
||
CORS base-station antennas) and `ngs20.atx` (NGS absolute calibrations,
|
||
covers the rover antenna). The rover antenna is resolved automatically and
|
||
written to `tools/antenna.json` (see *Antenna configuration* below).
|
||
4. **CRX2RNX** (Hatanaka RINEX decompressor) — fallback for CORS stations that
|
||
only publish compressed `.d` observation files.
|
||
5. **NGS CORS station catalog** cache (used for nearest-base auto-selection).
|
||
6. Self-check summary of every tool and the resolved antenna.
|
||
|
||
## process_gps.py
|
||
|
||
```
|
||
venv/bin/python process_gps.py <capture.ubx> [options]
|
||
```
|
||
|
||
Stages (idempotent — outputs are cached, so re-running later only fills in
|
||
what was previously unavailable, e.g. CORS/IGS data that had not been
|
||
published yet):
|
||
|
||
1. **inventory** — parses the `.ubx` (RXM-RAWX/SFRBX raw observations,
|
||
NAV-PVT/SAT/SIG, MON-RF) and the `.uc2x` sidecar's session metadata.
|
||
A capture without raw observations (e.g. u-center debug-mode logs) gets a
|
||
diagnostics-only report and a clear notice.
|
||
2. **rinex** — RINEX 3.04 obs+nav via demo5 `convbin`; verifies epoch count and
|
||
BDS/L5 presence; extracts broadcast ionosphere parameters.
|
||
3. **single** — standalone single-point solution (~1 m, works offline).
|
||
4. **ppk** — differential post-processing against the nearest operational NGS
|
||
CORS station (auto-selected by distance from the capture's own mean
|
||
position; MSEV at 34 km for the Hattiesburg test site). Hourly base files
|
||
are fetched automatically (~1 h publication lag; if not yet published the
|
||
stage reports "re-run later" and everything else proceeds). Base antenna
|
||
type is read from the base RINEX header; base coordinates are ITRF2020,
|
||
velocity-propagated to the capture epoch.
|
||
5. **ppp** (`--ppp`) — precise point positioning against IGS rapid/final
|
||
orbit+clock products (anonymous download from BKG/ESA, ~1 day lag).
|
||
Kinematic PPP by default per the motion policy; note RTKLIB's PPP-kinematic
|
||
engine is weak and may yield no solution — PPK is the reference method.
|
||
6. **track** — `track.csv` (UTC + decimal-degree lat/lon + per-epoch quality
|
||
and 1-σ sd columns + onboard comparison columns) and `track.kml`.
|
||
7. **diagnostics** — `diagnostics.json`, `report.md`, `report.html`
|
||
(self-contained, figures embedded) covering: constellation metrics (per-band
|
||
C/N0, cycle slips, residuals), ionosphere (broadcast Klobuchar + measured
|
||
dual-frequency slant delay/TECU), positional error metrics (RMS/R50/R95
|
||
dispersion, formal σ, inter-solution offsets), **validation of every
|
||
computed solution against the receiver's own onboard NAV-PVT positions**
|
||
(epoch-matched, PASS/FAIL), timing (epoch jitter, tAcc, leap seconds), and
|
||
RF health (jamming indicator, AGC, antenna supervisor state).
|
||
|
||
### Options
|
||
|
||
| Flag | Meaning |
|
||
|---|---|
|
||
| `--interval N` | `track.csv` output interval in seconds (default 1 = native rate) |
|
||
| `--static` | antenna verifiably stationary: enables static PPK/PPP solutions and static error framing. **Never for buoys.** |
|
||
| `--no-ppk` | skip base-station download/processing |
|
||
| `--ppp` | additionally run PPP against IGS precise products |
|
||
| `--base SSSS` | force a specific CORS station id (e.g. `MSEV`) |
|
||
| `--max-base-km KM` | base-station search radius (default 150) |
|
||
| `--force-stage S` | delete and re-run one stage (`rinex`,`single`,`ppk`,`ppp`,`track`) |
|
||
|
||
## Antenna configuration
|
||
|
||
Receiver-antenna phase-center corrections (PCO/PCV) matter at the cm level and
|
||
are applied during PPK/PPP via ANTEX calibration files.
|
||
|
||
- **Automatic**: `setup.sh` downloads `ngs20.atx`/`igs20.atx` and searches for
|
||
the project's antenna (ArduSimple calibrated quad-band, NGS calibration
|
||
AS-ANT3BCAL01). The resolved 20-character ANTEX type string is written to
|
||
`tools/antenna.json`:
|
||
|
||
```json
|
||
{ "antex_name": "AS-ANT3BCAL NONE", "source": "ngs20.atx" }
|
||
```
|
||
|
||
- **Different antenna**: edit `antex_name` in `tools/antenna.json` to the exact
|
||
ANTEX type string of your antenna (padding/spacing matters, copy it verbatim
|
||
from the `TYPE / SERIAL NO` line of an antenna block in
|
||
`tools/antex/ngs20.atx` or `igs20.atx`). Browse available calibrations at
|
||
https://geodesy.noaa.gov/ANTCAL/ . If your antenna has an NGS calibration but
|
||
a different model string, also update the `ANT3B` search pattern in `setup.sh`
|
||
so future setups re-resolve it automatically.
|
||
- **No calibration available**: set `antex_name` to `null` (or leave it
|
||
unresolved) — processing continues without rover PCO/PCV and the report
|
||
carries a warning. Expect a few cm of systematic error, mostly in height.
|
||
- **Base antenna** needs no configuration: its type is read from the CORS RINEX
|
||
header and looked up in the same ANTEX files.
|
||
- **Antenna height**: assumed 0 (positions refer to the antenna reference
|
||
point). For a surveyed monument, subtract your ARP offset downstream.
|
||
|
||
## Other important settings and inputs
|
||
|
||
**Capture requirements.** Input `.ubx` files must contain raw observations:
|
||
UBX-RXM-RAWX + UBX-RXM-SFRBX enabled (plus NAV-PVT; NAV-SAT/NAV-SIG/MON-RF
|
||
enrich diagnostics). In u-center, do **not** record with the "debug messages"
|
||
option — it floods the log with undocumented TRK/TUN/DBG/SEC messages and, in
|
||
past sessions here, raw output was configured in flash but not active in RAM.
|
||
The optional `.uc2x` sidecar (u-center 2 index) is only used for its session
|
||
metadata header.
|
||
|
||
**Real-time credentials (`stream_gps.py`).** Free registration at
|
||
http://rtn.usm.edu/RegisterAccount.aspx (Mississippi GCGC RTN), then:
|
||
|
||
```bash
|
||
export GCGC_USER=<username> GCGC_PASS=<password>
|
||
venv/bin/python stream_gps.py --port /dev/ttyUSB0 # or COM7 on Windows
|
||
```
|
||
|
||
Key streaming defaults (buoy-oriented): receiver dynamic-platform model `sea`
|
||
(`--dynmodel` to override), **live** NMEA GGA uplink so the network-RTK virtual
|
||
reference follows platform drift (`--gga-fixed` for bench tests), raw
|
||
RXM-RAWX/SFRBX always logged so every session is post-processable, and
|
||
`--replay <file.ubx>` offline test mode. Galileo HAS (~20 cm, no internet)
|
||
activates automatically as fallback when firmware ≥ HPG 2.10 is detected.
|
||
|
||
**Datums.** PPK output is in the CORS base frame (ITRF2020, current epoch);
|
||
GCGC real-time corrections are NAD83(2011) epoch 2010.0 — a constant ~1.5 m
|
||
offset from ITRF/WGS84 in CONUS. Each CSV records its frame in the header;
|
||
comparisons in the report account for the reference used. Onboard receiver
|
||
positions are WGS84-aligned.
|
||
|
||
**Data-source timing.** CORS hourly files: ~1 h lag, expire after ~2 days
|
||
(daily files remain, decimated to 30 s after 30 days). IGS rapid products:
|
||
~1 day lag. The pipeline caches everything it fetches under
|
||
`processed/<stem>/` and tells you when a re-run will find more data.
|
||
|
||
## Repository layout
|
||
|
||
```
|
||
setup.sh environment bootstrap (run first)
|
||
process_gps.py post-processing pipeline (Mode 2)
|
||
stream_gps.py real-time streaming client (Mode 1)
|
||
run.sh convenience wrapper
|
||
notes/ manual workflow notes + combine_rinex.sh helper
|
||
tools/ built binaries, ANTEX models, caches (generated)
|
||
venv/ Python environment (generated)
|
||
processed/<capture>/ per-capture outputs: RINEX, .pos solutions,
|
||
track.csv, track.kml, figures/, diagnostics.json,
|
||
report.md, report.html (generated)
|
||
```
|