Initial commit of fully working GPS processing pipeline. See README for more information.
This commit is contained in:
=
commit
65a4836608
8 changed files with 2620 additions and 0 deletions
18
.gitignore
vendored
Normal file
18
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,18 @@
|
||||||
|
# Python environment
|
||||||
|
venv/
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
|
||||||
|
# Bootstrapped tooling: RTKLIB build, ANTEX models, CORS catalog cache
|
||||||
|
# (fully reproducible with setup.sh)
|
||||||
|
tools/
|
||||||
|
|
||||||
|
# Pipeline outputs (regenerate with process_gps.py)
|
||||||
|
processed/
|
||||||
|
|
||||||
|
# Live streaming session outputs (stream_gps.py)
|
||||||
|
stream_*.csv
|
||||||
|
|
||||||
|
# Raw GPS capture data
|
||||||
|
*.ubx
|
||||||
|
*.uc2x
|
||||||
193
README.md
Normal file
193
README.md
Normal file
|
|
@ -0,0 +1,193 @@
|
||||||
|
# u-blox GPS Module Data Procesing (u-blox ZED-X20P)
|
||||||
|
|
||||||
|
Tooling for evaluating u-blox ZED-X20P all-band GNSS modules, targeting **buoy
|
||||||
|
deployments**. Converts raw u-center capture files (`.ubx`) into corrected
|
||||||
|
positions and produces a full diagnostics report, and supports live streaming
|
||||||
|
with real-time corrections.
|
||||||
|
|
||||||
|
Two operating modes:
|
||||||
|
|
||||||
|
| Mode | Script | Accuracy | Correction source |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Post-processing | `process_gps.py` | dm-cm (PPK) | NGS CORS base data, auto-fetched (free, open) |
|
||||||
|
| Real-time streaming | `stream_gps.py` | 1-3 cm (RTK fix) | GCGC RTN NTRIP (free registration) or Galileo HAS (FW ≥ HPG 2.10) |
|
||||||
|
|
||||||
|
Both consume/produce standard formats (RINEX 3, `.pos`, CSV, KML) via RTKLIB
|
||||||
|
(demo5 fork) and widely used open-source Python libraries (pyubx2, pygnssutils,
|
||||||
|
numpy, pandas, matplotlib).
|
||||||
|
|
||||||
|
## Motion policy (important)
|
||||||
|
|
||||||
|
**Processing never assumes the antenna was stationary.** Buoys drift slowly and
|
||||||
|
drift within a few metres is statistically indistinguishable from measurement
|
||||||
|
noise. Everything defaults to kinematic; pass `--static` to `process_gps.py`
|
||||||
|
only when the antenna is independently known to have been fixed (e.g. bench/roof test).
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash setup.sh # one-time environment bootstrap
|
||||||
|
venv/bin/python process_gps.py <capture.ubx> # process a capture
|
||||||
|
# results in processed/<capture-stem>/ (report.html is the human summary)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
Linux with `python3` (3.10+), `gcc`/`make`, `git`, `curl`, and network access
|
||||||
|
for `setup.sh` downloads. No sudo required and everything installs into the
|
||||||
|
project directory (`venv/`, `tools/`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## setup.sh
|
||||||
|
|
||||||
|
Environment bootstrap (safe to re-run; each step skips if already satisfied):
|
||||||
|
|
||||||
|
1. **Python venv** (`venv/`): pyubx2, pygnssutils, pyserial, numpy, pandas,
|
||||||
|
pyproj, matplotlib.
|
||||||
|
2. **RTKLIB demo5 v2.5.1** built from source into `tools/bin/`
|
||||||
|
(`convbin`, `rnx2rtkp`, `pos2kml`). The demo5 fork is required: stock
|
||||||
|
RTKLIB 2.4.3 silently drops all BeiDou and GPS L5 / Galileo E5 observations
|
||||||
|
from this receiver (~30% of the measurements). If the build fails, setup
|
||||||
|
falls back to the stock apt package and creates a `tools/STOCK_RTKLIB`
|
||||||
|
marker — `process_gps.py` then adds a prominent data-loss warning to every
|
||||||
|
report until a demo5 build succeeds.
|
||||||
|
3. **Antenna calibration models** into `tools/antex/`: `igs20.atx` (IGS, covers
|
||||||
|
CORS base-station antennas) and `ngs20.atx` (NGS absolute calibrations,
|
||||||
|
covers the rover antenna). The rover antenna is resolved automatically and
|
||||||
|
written to `tools/antenna.json` (see *Antenna configuration* below).
|
||||||
|
4. **CRX2RNX** (Hatanaka RINEX decompressor) — fallback for CORS stations that
|
||||||
|
only publish compressed `.d` observation files.
|
||||||
|
5. **NGS CORS station catalog** cache (used for nearest-base auto-selection).
|
||||||
|
6. Self-check summary of every tool and the resolved antenna.
|
||||||
|
|
||||||
|
## process_gps.py
|
||||||
|
|
||||||
|
```
|
||||||
|
venv/bin/python process_gps.py <capture.ubx> [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
Stages (idempotent — outputs are cached, so re-running later only fills in
|
||||||
|
what was previously unavailable, e.g. CORS/IGS data that had not been
|
||||||
|
published yet):
|
||||||
|
|
||||||
|
1. **inventory** — parses the `.ubx` (RXM-RAWX/SFRBX raw observations,
|
||||||
|
NAV-PVT/SAT/SIG, MON-RF) and the `.uc2x` sidecar's session metadata.
|
||||||
|
A capture without raw observations (e.g. u-center debug-mode logs) gets a
|
||||||
|
diagnostics-only report and a clear notice.
|
||||||
|
2. **rinex** — RINEX 3.04 obs+nav via demo5 `convbin`; verifies epoch count and
|
||||||
|
BDS/L5 presence; extracts broadcast ionosphere parameters.
|
||||||
|
3. **single** — standalone single-point solution (~1 m, works offline).
|
||||||
|
4. **ppk** — differential post-processing against the nearest operational NGS
|
||||||
|
CORS station (auto-selected by distance from the capture's own mean
|
||||||
|
position; MSEV at 34 km for the Hattiesburg test site). Hourly base files
|
||||||
|
are fetched automatically (~1 h publication lag; if not yet published the
|
||||||
|
stage reports "re-run later" and everything else proceeds). Base antenna
|
||||||
|
type is read from the base RINEX header; base coordinates are ITRF2020,
|
||||||
|
velocity-propagated to the capture epoch.
|
||||||
|
5. **ppp** (`--ppp`) — precise point positioning against IGS rapid/final
|
||||||
|
orbit+clock products (anonymous download from BKG/ESA, ~1 day lag).
|
||||||
|
Kinematic PPP by default per the motion policy; note RTKLIB's PPP-kinematic
|
||||||
|
engine is weak and may yield no solution — PPK is the reference method.
|
||||||
|
6. **track** — `track.csv` (UTC + decimal-degree lat/lon + per-epoch quality
|
||||||
|
and 1-σ sd columns + onboard comparison columns) and `track.kml`.
|
||||||
|
7. **diagnostics** — `diagnostics.json`, `report.md`, `report.html`
|
||||||
|
(self-contained, figures embedded) covering: constellation metrics (per-band
|
||||||
|
C/N0, cycle slips, residuals), ionosphere (broadcast Klobuchar + measured
|
||||||
|
dual-frequency slant delay/TECU), positional error metrics (RMS/R50/R95
|
||||||
|
dispersion, formal σ, inter-solution offsets), **validation of every
|
||||||
|
computed solution against the receiver's own onboard NAV-PVT positions**
|
||||||
|
(epoch-matched, PASS/FAIL), timing (epoch jitter, tAcc, leap seconds), and
|
||||||
|
RF health (jamming indicator, AGC, antenna supervisor state).
|
||||||
|
|
||||||
|
### Options
|
||||||
|
|
||||||
|
| Flag | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `--interval N` | `track.csv` output interval in seconds (default 1 = native rate) |
|
||||||
|
| `--static` | antenna verifiably stationary: enables static PPK/PPP solutions and static error framing. **Never for buoys.** |
|
||||||
|
| `--no-ppk` | skip base-station download/processing |
|
||||||
|
| `--ppp` | additionally run PPP against IGS precise products |
|
||||||
|
| `--base SSSS` | force a specific CORS station id (e.g. `MSEV`) |
|
||||||
|
| `--max-base-km KM` | base-station search radius (default 150) |
|
||||||
|
| `--force-stage S` | delete and re-run one stage (`rinex`,`single`,`ppk`,`ppp`,`track`) |
|
||||||
|
|
||||||
|
## Antenna configuration
|
||||||
|
|
||||||
|
Receiver-antenna phase-center corrections (PCO/PCV) matter at the cm level and
|
||||||
|
are applied during PPK/PPP via ANTEX calibration files.
|
||||||
|
|
||||||
|
- **Automatic**: `setup.sh` downloads `ngs20.atx`/`igs20.atx` and searches for
|
||||||
|
the project's antenna (ArduSimple calibrated quad-band, NGS calibration
|
||||||
|
AS-ANT3BCAL01). The resolved 20-character ANTEX type string is written to
|
||||||
|
`tools/antenna.json`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "antex_name": "AS-ANT3BCAL NONE", "source": "ngs20.atx" }
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Different antenna**: edit `antex_name` in `tools/antenna.json` to the exact
|
||||||
|
ANTEX type string of your antenna (padding/spacing matters, copy it verbatim
|
||||||
|
from the `TYPE / SERIAL NO` line of an antenna block in
|
||||||
|
`tools/antex/ngs20.atx` or `igs20.atx`). Browse available calibrations at
|
||||||
|
https://geodesy.noaa.gov/ANTCAL/ . If your antenna has an NGS calibration but
|
||||||
|
a different model string, also update the `ANT3B` search pattern in `setup.sh`
|
||||||
|
so future setups re-resolve it automatically.
|
||||||
|
- **No calibration available**: set `antex_name` to `null` (or leave it
|
||||||
|
unresolved) — processing continues without rover PCO/PCV and the report
|
||||||
|
carries a warning. Expect a few cm of systematic error, mostly in height.
|
||||||
|
- **Base antenna** needs no configuration: its type is read from the CORS RINEX
|
||||||
|
header and looked up in the same ANTEX files.
|
||||||
|
- **Antenna height**: assumed 0 (positions refer to the antenna reference
|
||||||
|
point). For a surveyed monument, subtract your ARP offset downstream.
|
||||||
|
|
||||||
|
## Other important settings and inputs
|
||||||
|
|
||||||
|
**Capture requirements.** Input `.ubx` files must contain raw observations:
|
||||||
|
UBX-RXM-RAWX + UBX-RXM-SFRBX enabled (plus NAV-PVT; NAV-SAT/NAV-SIG/MON-RF
|
||||||
|
enrich diagnostics). In u-center, do **not** record with the "debug messages"
|
||||||
|
option — it floods the log with undocumented TRK/TUN/DBG/SEC messages and, in
|
||||||
|
past sessions here, raw output was configured in flash but not active in RAM.
|
||||||
|
The optional `.uc2x` sidecar (u-center 2 index) is only used for its session
|
||||||
|
metadata header.
|
||||||
|
|
||||||
|
**Real-time credentials (`stream_gps.py`).** Free registration at
|
||||||
|
http://rtn.usm.edu/RegisterAccount.aspx (Mississippi GCGC RTN), then:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export GCGC_USER=<username> GCGC_PASS=<password>
|
||||||
|
venv/bin/python stream_gps.py --port /dev/ttyUSB0 # or COM7 on Windows
|
||||||
|
```
|
||||||
|
|
||||||
|
Key streaming defaults (buoy-oriented): receiver dynamic-platform model `sea`
|
||||||
|
(`--dynmodel` to override), **live** NMEA GGA uplink so the network-RTK virtual
|
||||||
|
reference follows platform drift (`--gga-fixed` for bench tests), raw
|
||||||
|
RXM-RAWX/SFRBX always logged so every session is post-processable, and
|
||||||
|
`--replay <file.ubx>` offline test mode. Galileo HAS (~20 cm, no internet)
|
||||||
|
activates automatically as fallback when firmware ≥ HPG 2.10 is detected.
|
||||||
|
|
||||||
|
**Datums.** PPK output is in the CORS base frame (ITRF2020, current epoch);
|
||||||
|
GCGC real-time corrections are NAD83(2011) epoch 2010.0 — a constant ~1.5 m
|
||||||
|
offset from ITRF/WGS84 in CONUS. Each CSV records its frame in the header;
|
||||||
|
comparisons in the report account for the reference used. Onboard receiver
|
||||||
|
positions are WGS84-aligned.
|
||||||
|
|
||||||
|
**Data-source timing.** CORS hourly files: ~1 h lag, expire after ~2 days
|
||||||
|
(daily files remain, decimated to 30 s after 30 days). IGS rapid products:
|
||||||
|
~1 day lag. The pipeline caches everything it fetches under
|
||||||
|
`processed/<stem>/` and tells you when a re-run will find more data.
|
||||||
|
|
||||||
|
## Repository layout
|
||||||
|
|
||||||
|
```
|
||||||
|
setup.sh environment bootstrap (run first)
|
||||||
|
process_gps.py post-processing pipeline (Mode 2)
|
||||||
|
stream_gps.py real-time streaming client (Mode 1)
|
||||||
|
run.sh convenience wrapper
|
||||||
|
notes/ manual workflow notes + combine_rinex.sh helper
|
||||||
|
tools/ built binaries, ANTEX models, caches (generated)
|
||||||
|
venv/ Python environment (generated)
|
||||||
|
processed/<capture>/ per-capture outputs: RINEX, .pos solutions,
|
||||||
|
track.csv, track.kml, figures/, diagnostics.json,
|
||||||
|
report.md, report.html (generated)
|
||||||
|
```
|
||||||
42
notes/RTK Processing Notes.txt
Executable file
42
notes/RTK Processing Notes.txt
Executable file
|
|
@ -0,0 +1,42 @@
|
||||||
|
### THESE ARE NOTES FROM PREVIOUS WORK, KEPT FOR REFERENCE. NONE OF THESE STEPS ARE REQUIRED TO USE THE GPS DATA PROCESSING SCRIPTS IN THE PARENT DIRECTORY.
|
||||||
|
|
||||||
|
## Download the CRX2RNX tool precompiled binary
|
||||||
|
|
||||||
|
# Download the pre-compiled Linux binary
|
||||||
|
cd /tmp
|
||||||
|
wget https://terras.gsi.go.jp/ja/crx2rnx/RNXCMP_4.2.0_Linux_gcc_64bit.tar.gz
|
||||||
|
|
||||||
|
# Extract
|
||||||
|
tar -xzf RNXCMP_4.2.0_Linux_gcc_64bit.tar.gz
|
||||||
|
|
||||||
|
# Install to system path
|
||||||
|
sudo cp RNXCMP_4.2.0_Linux_gcc_64bit/CRX2RNX /usr/local/bin/
|
||||||
|
|
||||||
|
# Make executable
|
||||||
|
sudo chmod +x /usr/local/bin/CRX2RNX
|
||||||
|
|
||||||
|
# Create lowercase symlink (rnx2rtkp looks for lowercase)
|
||||||
|
sudo ln -s /usr/local/bin/CRX2RNX /usr/local/bin/crx2rnx
|
||||||
|
|
||||||
|
# Test
|
||||||
|
crx2rnx -h
|
||||||
|
|
||||||
|
|
||||||
|
## Download CORS corrections from here: https://www.ngs.noaa.gov/CORS/
|
||||||
|
|
||||||
|
## If base station CORS files are compressed (.25d format), decompress them
|
||||||
|
ls msin*.25d | xargs -n 1 -P 4 crx2rnx
|
||||||
|
|
||||||
|
## Combine base station observations into one file
|
||||||
|
./combine_rinex.sh msin_combined.obs msin*.25o
|
||||||
|
|
||||||
|
## Perform corrections on a .obs file using base station files as reference
|
||||||
|
cd CORS\ STATION\ MSIN\ NOV\ 2025/
|
||||||
|
|
||||||
|
rnx2rtkp \ -t -u \
|
||||||
|
-o ../../A/DATA/result.pos \
|
||||||
|
../../A/DATA/20251120_033544.006_UTC_GPS_A_combined.obs \
|
||||||
|
msin_combined.obs \
|
||||||
|
../../A/DATA/20251120_033544.006_UTC_GPS_A_combined.nav
|
||||||
|
|
||||||
|
## The result.pos file now contains PPK corrected GPS data in LLH format
|
||||||
141
notes/combine_rinex.sh
Executable file
141
notes/combine_rinex.sh
Executable file
|
|
@ -0,0 +1,141 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# combine_rinex.sh - Combine RINEX observation files (supports both 2.x and 3.x)
|
||||||
|
|
||||||
|
if [ $# -lt 2 ]; then
|
||||||
|
echo "Usage: $0 output.obs input_files..."
|
||||||
|
echo "Example: $0 combined.obs msin*.25o"
|
||||||
|
echo "Example: $0 combined.obs msin3240.25o msin3250.25o msin3260.25o"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
output="$1"
|
||||||
|
shift
|
||||||
|
|
||||||
|
# Sort files chronologically (RINEX naming is sortable)
|
||||||
|
files=$(printf '%s\n' "$@" | sort)
|
||||||
|
|
||||||
|
if [ -z "$files" ]; then
|
||||||
|
echo "Error: No input files specified"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get first file
|
||||||
|
first_file=$(echo "$files" | head -1)
|
||||||
|
|
||||||
|
# Verify all files exist
|
||||||
|
echo "Verifying input files..."
|
||||||
|
for file in $files; do
|
||||||
|
if [ ! -f "$file" ]; then
|
||||||
|
echo "Error: File not found: $file"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
file_count=$(echo "$files" | wc -w)
|
||||||
|
echo "Found $file_count files to combine"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Get header from first file
|
||||||
|
header_lines=$(grep -n "END OF HEADER" "$first_file" | cut -d: -f1)
|
||||||
|
|
||||||
|
if [ -z "$header_lines" ]; then
|
||||||
|
echo "Error: No RINEX header found in $first_file"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Detect RINEX version from first line
|
||||||
|
rinex_version=$(head -1 "$first_file" | awk '{print substr($1,1,1)}')
|
||||||
|
|
||||||
|
if [ "$rinex_version" = "3" ]; then
|
||||||
|
echo "Detected: RINEX 3.x format"
|
||||||
|
epoch_pattern="^>"
|
||||||
|
elif [ "$rinex_version" = "2" ]; then
|
||||||
|
echo "Detected: RINEX 2.x format"
|
||||||
|
epoch_pattern="^ [0-9][0-9] [ 0-9][0-9] [ 0-9][0-9]"
|
||||||
|
else
|
||||||
|
echo "Warning: Could not detect RINEX version, assuming 2.x"
|
||||||
|
epoch_pattern="^ [0-9][0-9] [ 0-9][0-9] [ 0-9][0-9]"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Copy header from first file
|
||||||
|
head -n $header_lines "$first_file" > "$output"
|
||||||
|
echo "✓ Added header from $first_file"
|
||||||
|
|
||||||
|
# Append data from all files (skip their headers)
|
||||||
|
total_lines=0
|
||||||
|
for file in $files; do
|
||||||
|
data_start=$(($(grep -n "END OF HEADER" "$file" | cut -d: -f1) + 1))
|
||||||
|
data_lines=$(tail -n +$data_start "$file" | wc -l)
|
||||||
|
tail -n +$data_start "$file" >> "$output"
|
||||||
|
total_lines=$((total_lines + data_lines))
|
||||||
|
echo "✓ Added data from $file ($data_lines lines)"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Count epochs based on detected version
|
||||||
|
if [ "$rinex_version" = "3" ]; then
|
||||||
|
epochs=$(grep -c "^>" "$output" 2>/dev/null || echo "0")
|
||||||
|
else
|
||||||
|
epochs=$(grep -c "^ [0-9][0-9] [ 0-9][0-9] [ 0-9][0-9]" "$output" 2>/dev/null || echo "0")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get file size
|
||||||
|
file_size=$(ls -lh "$output" | awk '{print $5}')
|
||||||
|
|
||||||
|
echo "========================================="
|
||||||
|
echo "✓ Combined file created: $output"
|
||||||
|
echo " RINEX Version: $rinex_version.x"
|
||||||
|
echo " Size: $file_size"
|
||||||
|
echo " Files combined: $file_count"
|
||||||
|
echo " Total data lines: $total_lines"
|
||||||
|
echo " Total epochs: $epochs"
|
||||||
|
echo "========================================="
|
||||||
|
|
||||||
|
# Verify reasonable results
|
||||||
|
if [ $epochs -eq 0 ]; then
|
||||||
|
echo ""
|
||||||
|
echo "⚠ WARNING: No epochs detected!"
|
||||||
|
echo " This may indicate a problem with the combined file."
|
||||||
|
echo " Checking first few lines of output..."
|
||||||
|
echo ""
|
||||||
|
head -40 "$output"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Show time range if epochs found
|
||||||
|
if [ $epochs -gt 0 ]; then
|
||||||
|
echo ""
|
||||||
|
echo "Time range:"
|
||||||
|
|
||||||
|
if [ "$rinex_version" = "3" ]; then
|
||||||
|
# RINEX 3.x format: > YYYY MM DD HH MM SS.SSSSSSS
|
||||||
|
first_epoch=$(grep "^>" "$output" | head -1)
|
||||||
|
last_epoch=$(grep "^>" "$output" | tail -1)
|
||||||
|
first_time=$(echo "$first_epoch" | awk '{printf "%s-%02d-%02d %02d:%02d:%06.3f", $2, $3, $4, $5, $6, $7}')
|
||||||
|
last_time=$(echo "$last_epoch" | awk '{printf "%s-%02d-%02d %02d:%02d:%06.3f", $2, $3, $4, $5, $6, $7}')
|
||||||
|
else
|
||||||
|
# RINEX 2.x format: YY MM DD HH MM SS.SSSSSSS
|
||||||
|
first_epoch=$(grep "^ [0-9][0-9] [ 0-9][0-9] [ 0-9][0-9]" "$output" | head -1)
|
||||||
|
last_epoch=$(grep "^ [0-9][0-9] [ 0-9][0-9] [ 0-9][0-9]" "$output" | tail -1)
|
||||||
|
first_time=$(echo "$first_epoch" | awk '{printf "20%02d-%02d-%02d %02d:%02d:%011.7f", $1, $2, $3, $4, $5, $6}')
|
||||||
|
last_time=$(echo "$last_epoch" | awk '{printf "20%02d-%02d-%02d %02d:%02d:%011.7f", $1, $2, $3, $4, $5, $6}')
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo " First: $first_time"
|
||||||
|
echo " Last: $last_time"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Verify single header (no corruption)
|
||||||
|
header_count=$(grep -c "END OF HEADER" "$output")
|
||||||
|
if [ $header_count -ne 1 ]; then
|
||||||
|
echo ""
|
||||||
|
echo "⚠ WARNING: Found $header_count headers (expected 1)"
|
||||||
|
echo " The file may be corrupted!"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "✓ File validation passed"
|
||||||
1584
process_gps.py
Normal file
1584
process_gps.py
Normal file
File diff suppressed because it is too large
Load diff
3
run.sh
Executable file
3
run.sh
Executable file
|
|
@ -0,0 +1,3 @@
|
||||||
|
source ./venv/bin/activate
|
||||||
|
python3 process_gps.py -h
|
||||||
|
python3 process_gps.py 2026-7-23_124722_serial-COM7.ubx
|
||||||
193
setup.sh
Executable file
193
setup.sh
Executable file
|
|
@ -0,0 +1,193 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# setup.sh - One-time environment bootstrap for the GPS processing pipeline.
|
||||||
|
#
|
||||||
|
# Installs (all local to this directory, no sudo required):
|
||||||
|
# venv/ Python env: pyubx2, pygnssutils, pyserial, numpy, pandas, pyproj
|
||||||
|
# tools/bin/ RTKLIB demo5 v2.5.1 (convbin, rnx2rtkp, pos2kml) [stock apt fallback]
|
||||||
|
# tools/antex/ igs20.atx + ngs20.atx antenna models; tools/antenna.json (rover antenna)
|
||||||
|
# tools/bin/CRX2RNX Hatanaka RINEX decompressor (fallback for .d CORS files)
|
||||||
|
# tools/cache/ NGS CORS station catalog
|
||||||
|
#
|
||||||
|
# Idempotent: re-running skips anything already installed.
|
||||||
|
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
cd "$SCRIPT_DIR"
|
||||||
|
|
||||||
|
TOOLS="$SCRIPT_DIR/tools"
|
||||||
|
VENV="$SCRIPT_DIR/venv"
|
||||||
|
DEMO5_TAG="v2.5.1"
|
||||||
|
DEMO5_REPO="https://github.com/rtklibexplorer/RTKLIB"
|
||||||
|
|
||||||
|
PASS=()
|
||||||
|
WARN=()
|
||||||
|
|
||||||
|
say() { printf '\n=== %s ===\n' "$*"; }
|
||||||
|
ok() { printf ' [ok] %s\n' "$*"; PASS+=("$*"); }
|
||||||
|
warn() { printf ' [WARN] %s\n' "$*"; WARN+=("$*"); }
|
||||||
|
skip() { printf ' [skip] %s (already present)\n' "$*"; }
|
||||||
|
|
||||||
|
fetch() { # fetch <url> <dest>
|
||||||
|
curl -fsSL --max-time 300 -o "$2" "$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 1. Python venv
|
||||||
|
say "1/6 Python virtual environment"
|
||||||
|
if [ -x "$VENV/bin/python" ]; then
|
||||||
|
skip "venv"
|
||||||
|
else
|
||||||
|
python3 -m venv "$VENV" || { echo "FATAL: venv creation failed"; exit 1; }
|
||||||
|
fi
|
||||||
|
if "$VENV/bin/python" -c "import pyubx2, pygnssutils, serial, numpy, pandas, pyproj, matplotlib" 2>/dev/null; then
|
||||||
|
skip "python packages"
|
||||||
|
else
|
||||||
|
"$VENV/bin/pip" install --quiet --upgrade pip
|
||||||
|
"$VENV/bin/pip" install --quiet pyubx2 pygnssutils pyserial numpy pandas pyproj matplotlib \
|
||||||
|
|| { echo "FATAL: pip install failed"; exit 1; }
|
||||||
|
fi
|
||||||
|
"$VENV/bin/python" -c "import pyubx2, pygnssutils, serial, numpy, pandas, pyproj, matplotlib" \
|
||||||
|
&& ok "venv with pyubx2/pygnssutils/pyserial/numpy/pandas/pyproj/matplotlib" \
|
||||||
|
|| { echo "FATAL: package import check failed"; exit 1; }
|
||||||
|
|
||||||
|
mkdir -p "$TOOLS/bin" "$TOOLS/antex" "$TOOLS/cache" "$TOOLS/src"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 2. RTKLIB demo5
|
||||||
|
say "2/6 RTKLIB demo5 $DEMO5_TAG (convbin, rnx2rtkp, pos2kml)"
|
||||||
|
build_demo5_app() { # build_demo5_app <appname>
|
||||||
|
local app="$1"
|
||||||
|
local gccdir="$TOOLS/src/RTKLIB/app/consapp/$app/gcc"
|
||||||
|
[ -d "$gccdir" ] || return 1
|
||||||
|
# -lgfortran in some makefiles is vestigial; try without it first.
|
||||||
|
if ! make -C "$gccdir" -j"$(nproc)" LDLIBS="-lm" >"$TOOLS/src/build_$app.log" 2>&1; then
|
||||||
|
make -C "$gccdir" clean >/dev/null 2>&1
|
||||||
|
make -C "$gccdir" -j"$(nproc)" >>"$TOOLS/src/build_$app.log" 2>&1 || return 1
|
||||||
|
fi
|
||||||
|
cp "$gccdir/$app" "$TOOLS/bin/$app"
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ -x "$TOOLS/bin/convbin" ] && [ -x "$TOOLS/bin/rnx2rtkp" ] && [ -x "$TOOLS/bin/pos2kml" ]; then
|
||||||
|
skip "RTKLIB binaries"
|
||||||
|
[ -f "$TOOLS/STOCK_RTKLIB" ] && warn "using STOCK RTKLIB (BDS/L5 signals will be missing from RINEX)"
|
||||||
|
else
|
||||||
|
if [ ! -d "$TOOLS/src/RTKLIB" ]; then
|
||||||
|
git clone --depth 1 --branch "$DEMO5_TAG" "$DEMO5_REPO" "$TOOLS/src/RTKLIB" \
|
||||||
|
|| warn "git clone of demo5 RTKLIB failed"
|
||||||
|
fi
|
||||||
|
built=true
|
||||||
|
for app in convbin rnx2rtkp pos2kml; do
|
||||||
|
if build_demo5_app "$app"; then
|
||||||
|
printf ' built %s\n' "$app"
|
||||||
|
else
|
||||||
|
warn "demo5 build failed for $app (see tools/src/build_$app.log)"
|
||||||
|
built=false
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if $built; then
|
||||||
|
rm -f "$TOOLS/STOCK_RTKLIB"
|
||||||
|
ok "RTKLIB demo5 $DEMO5_TAG built from source"
|
||||||
|
else
|
||||||
|
# Fallback: stock RTKLIB 2.4.3 from the Ubuntu archive (works, but silently
|
||||||
|
# drops BeiDou + L5/E5 observations from this receiver's raw data).
|
||||||
|
echo " falling back to stock RTKLIB 2.4.3 from apt archive..."
|
||||||
|
tmpd="$(mktemp -d)"
|
||||||
|
if (cd "$tmpd" && apt-get download rtklib >/dev/null 2>&1 && dpkg -x rtklib_*.deb x); then
|
||||||
|
cp "$tmpd"/x/usr/bin/{convbin,rnx2rtkp,pos2kml} "$TOOLS/bin/"
|
||||||
|
touch "$TOOLS/STOCK_RTKLIB"
|
||||||
|
warn "STOCK RTKLIB installed - BDS/L5 will be MISSING from RINEX output"
|
||||||
|
else
|
||||||
|
warn "stock RTKLIB fallback also failed - pipeline cannot run"
|
||||||
|
fi
|
||||||
|
rm -rf "$tmpd"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
for app in convbin rnx2rtkp pos2kml; do
|
||||||
|
[ -x "$TOOLS/bin/$app" ] || warn "$app missing from tools/bin"
|
||||||
|
done
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 3. Antenna models
|
||||||
|
say "3/6 Antenna calibration models (ANTEX)"
|
||||||
|
if [ -s "$TOOLS/antex/igs20.atx" ]; then skip "igs20.atx"; else
|
||||||
|
fetch "https://files.igs.org/pub/station/general/igs20.atx" "$TOOLS/antex/igs20.atx" \
|
||||||
|
&& ok "igs20.atx (IGS antenna models incl. CORS base antennas)" \
|
||||||
|
|| warn "igs20.atx download failed"
|
||||||
|
fi
|
||||||
|
if [ -s "$TOOLS/antex/ngs20.atx" ]; then skip "ngs20.atx"; else
|
||||||
|
fetch "https://geodesy.noaa.gov/ANTCAL/LoadFile?file=ngs20.atx" "$TOOLS/antex/ngs20.atx" \
|
||||||
|
&& ok "ngs20.atx (NGS absolute calibrations incl. ArduSimple)" \
|
||||||
|
|| warn "ngs20.atx download failed"
|
||||||
|
fi
|
||||||
|
# Resolve the rover antenna (NGS calibration AS-ANT3BCAL01, ArduSimple) to its
|
||||||
|
# exact 20-char ANTEX type string, consumed by process_gps.py as ant1-anttype.
|
||||||
|
"$VENV/bin/python" - "$TOOLS" <<'EOF'
|
||||||
|
import json, re, sys, pathlib
|
||||||
|
tools = pathlib.Path(sys.argv[1])
|
||||||
|
result = {"antex_name": None, "source": None, "candidates": []}
|
||||||
|
for atx in ("ngs20.atx", "igs20.atx"):
|
||||||
|
p = tools / "antex" / atx
|
||||||
|
if not p.is_file():
|
||||||
|
continue
|
||||||
|
for line in p.open(encoding="utf-8", errors="replace"):
|
||||||
|
if "TYPE / SERIAL" in line and "ANT3B" in line[:20].upper():
|
||||||
|
name = line[:20].rstrip()
|
||||||
|
result["candidates"].append({"name": name, "file": atx})
|
||||||
|
if result["antex_name"] is None:
|
||||||
|
result["antex_name"], result["source"] = name, atx
|
||||||
|
(tools / "antenna.json").write_text(json.dumps(result, indent=2))
|
||||||
|
if result["antex_name"]:
|
||||||
|
print(f' [ok] rover antenna resolved: "{result["antex_name"]}" ({result["source"]})')
|
||||||
|
else:
|
||||||
|
print(" [WARN] no ANT3B antenna found in ANTEX files - PPK will run without rover PCO/PCV")
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 4. CRX2RNX
|
||||||
|
say "4/6 CRX2RNX (Hatanaka RINEX decompressor, fallback for .d CORS files)"
|
||||||
|
if [ -x "$TOOLS/bin/CRX2RNX" ]; then
|
||||||
|
skip "CRX2RNX"
|
||||||
|
else
|
||||||
|
tmpd="$(mktemp -d)"
|
||||||
|
if fetch "https://terras.gsi.go.jp/ja/crx2rnx/RNXCMP_4.2.0_Linux_gcc_64bit.tar.gz" "$tmpd/rnxcmp.tar.gz" \
|
||||||
|
&& tar -xzf "$tmpd/rnxcmp.tar.gz" -C "$tmpd" \
|
||||||
|
&& cp "$tmpd"/RNXCMP_*/bin/CRX2RNX "$TOOLS/bin/CRX2RNX" 2>/dev/null \
|
||||||
|
|| cp "$tmpd"/RNXCMP_*/CRX2RNX "$TOOLS/bin/CRX2RNX" 2>/dev/null; then
|
||||||
|
chmod +x "$TOOLS/bin/CRX2RNX"
|
||||||
|
ok "CRX2RNX 4.2.0"
|
||||||
|
else
|
||||||
|
warn "CRX2RNX install failed (only needed for CORS stations without plain .o files)"
|
||||||
|
fi
|
||||||
|
rm -rf "$tmpd"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 5. CORS catalog
|
||||||
|
say "5/6 NGS CORS station catalog"
|
||||||
|
if [ -s "$TOOLS/cache/itrf2020_geo.comp.txt" ]; then
|
||||||
|
skip "station catalog"
|
||||||
|
else
|
||||||
|
fetch "https://geodesy.noaa.gov/corsdata/coord/coord_20/itrf2020_geo.comp.txt" \
|
||||||
|
"$TOOLS/cache/itrf2020_geo.comp.txt" \
|
||||||
|
&& ok "CORS station catalog (ITRF2020 coordinates)" \
|
||||||
|
|| warn "CORS catalog download failed (process_gps.py will retry at run time)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 6. Self-check
|
||||||
|
say "6/6 Self-check summary"
|
||||||
|
for app in convbin rnx2rtkp pos2kml; do
|
||||||
|
if [ -x "$TOOLS/bin/$app" ]; then
|
||||||
|
printf ' %-10s %s\n' "$app" "$("$TOOLS/bin/$app" -h 2>&1 | grep -m1 -iE 'synops|usage|version' || echo present)"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
[ -x "$TOOLS/bin/CRX2RNX" ] && printf ' %-10s %s\n' "CRX2RNX" "present"
|
||||||
|
[ -s "$TOOLS/cache/itrf2020_geo.comp.txt" ] && printf ' %-10s %d stations\n' "catalog" "$(wc -l < "$TOOLS/cache/itrf2020_geo.comp.txt")"
|
||||||
|
[ -s "$TOOLS/antenna.json" ] && printf ' %-10s %s\n' "antenna" "$(cat "$TOOLS/antenna.json" | "$VENV/bin/python" -c 'import json,sys; print(json.load(sys.stdin)["antex_name"])')"
|
||||||
|
|
||||||
|
echo
|
||||||
|
if [ ${#WARN[@]} -gt 0 ]; then
|
||||||
|
echo "Completed with ${#WARN[@]} warning(s):"
|
||||||
|
printf ' - %s\n' "${WARN[@]}"
|
||||||
|
else
|
||||||
|
echo "Setup complete - no warnings."
|
||||||
|
fi
|
||||||
|
echo
|
||||||
|
echo "Mode 1 (real-time RTK) additionally requires free registration at"
|
||||||
|
echo " http://rtn.usm.edu/RegisterAccount.aspx (Mississippi GCGC RTN)"
|
||||||
|
echo "then: export GCGC_USER=<user> GCGC_PASS=<pass>"
|
||||||
446
stream_gps.py
Normal file
446
stream_gps.py
Normal file
|
|
@ -0,0 +1,446 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
stream_gps.py - Mode 1: continuous real-time positioning for the u-blox ZED-X20P.
|
||||||
|
|
||||||
|
Best-available correction tier is selected automatically:
|
||||||
|
1. Network RTK via NTRIP (default: Mississippi GCGC RTN, free registration at
|
||||||
|
http://rtn.usm.edu/RegisterAccount.aspx) -> onboard RTK engine, 1-3 cm.
|
||||||
|
Credentials: env GCGC_USER / GCGC_PASS (or --ntrip-user/--ntrip-pass).
|
||||||
|
2. Galileo HAS (E6-B, no internet needed) -> ~20-25 cm. Requires firmware
|
||||||
|
HPG >= 2.10 (auto-probed via MON-VER; this project's module reports 2.10).
|
||||||
|
3. Neither available -> the onboard multiband+SBAS solution (~1 m) cannot be
|
||||||
|
improved in real time; per project policy the enhanced mode is SKIPPED
|
||||||
|
(run with --log-only to still record a .ubx for post-processing).
|
||||||
|
|
||||||
|
Every session simultaneously logs raw RXM-RAWX/SFRBX to a .ubx file, so a
|
||||||
|
streaming session can always be post-processed later with process_gps.py.
|
||||||
|
|
||||||
|
Buoy deployment defaults: the receiver's dynamic platform model is set to "sea"
|
||||||
|
(--dynmodel to override) and the NTRIP GGA uplink sends the LIVE position so the
|
||||||
|
VRS reference follows platform drift (--gga-fixed to disable for a fixed antenna).
|
||||||
|
|
||||||
|
Outputs: live console status, stream_<ts>.csv (decimal-degree lat/lon),
|
||||||
|
stream_<ts>.ubx (raw capture).
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
venv/bin/python stream_gps.py --port /dev/ttyUSB0 (Linux)
|
||||||
|
venv/bin/python stream_gps.py --port COM7 (Windows)
|
||||||
|
venv/bin/python stream_gps.py --replay capture.ubx (offline test)
|
||||||
|
|
||||||
|
Datum note: NTRIP corrections define the output frame. GCGC RTN broadcasts
|
||||||
|
NAD83(2011) epoch 2010.00 - constant ~1.5 m from WGS84/ITRF in CONUS; the CSV
|
||||||
|
header records which tier (and therefore frame) was active.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import os
|
||||||
|
import queue
|
||||||
|
import signal
|
||||||
|
import socket
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from collections import Counter
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from pyubx2 import UBXMessage, UBXReader, UBX_PROTOCOL, SET_LAYER_RAM
|
||||||
|
|
||||||
|
FIX_NAME = {0: "none", 1: "DR", 2: "2D", 3: "3D", 4: "GNSS+DR", 5: "time-only"}
|
||||||
|
CARR_NAME = {0: "", 1: "RTK-FLOAT", 2: "RTK-FIXED"}
|
||||||
|
|
||||||
|
# UBX-CFG keys (u-blox X20 HPG interface descriptions; NAVCOR from HPG 2.10 JSON)
|
||||||
|
K = {
|
||||||
|
"UART1_BAUD": 0x40520001,
|
||||||
|
"UART1_IN_RTCM3": 0x10730004,
|
||||||
|
"USB_IN_RTCM3": 0x10770004,
|
||||||
|
"PVT_UART1": 0x20910007, "PVT_USB": 0x20910009,
|
||||||
|
"HPPOSLLH_UART1": 0x20910034, "HPPOSLLH_USB": 0x20910036,
|
||||||
|
"RAWX_UART1": 0x209102a5, "RAWX_USB": 0x209102a7,
|
||||||
|
"SFRBX_UART1": 0x20910232, "SFRBX_USB": 0x20910234,
|
||||||
|
"NAVCOR_ENABLE_HOST": 0x100d0001, # HPG >= 2.10
|
||||||
|
"NAVCOR_ENABLE_GAL_HAS": 0x100d0002, # HPG >= 2.10
|
||||||
|
"DYNMODEL": 0x20110021, # CFG-NAVSPG-DYNMODEL
|
||||||
|
}
|
||||||
|
|
||||||
|
# CFG-NAVSPG-DYNMODEL values; default "sea" for the buoy deployment
|
||||||
|
DYNMODEL = {"portable": 0, "stationary": 2, "pedestrian": 3, "automotive": 4,
|
||||||
|
"sea": 5, "airborne1g": 6, "airborne2g": 7, "airborne4g": 8, "wrist": 9}
|
||||||
|
|
||||||
|
|
||||||
|
def log(msg):
|
||||||
|
print(f"[stream_gps] {msg}", flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------
|
||||||
|
# Receiver configuration
|
||||||
|
# ----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def valset(ser, ubr, cfg, label):
|
||||||
|
"""Send CFG-VALSET (RAM) and wait briefly for ACK; returns True on ACK-ACK."""
|
||||||
|
msg = UBXMessage.config_set(layers=SET_LAYER_RAM, transaction=0, cfgData=cfg)
|
||||||
|
ser.write(msg.serialize())
|
||||||
|
t_end = time.time() + 2.0
|
||||||
|
while time.time() < t_end:
|
||||||
|
try:
|
||||||
|
_, parsed = ubr.read()
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
if parsed is None:
|
||||||
|
continue
|
||||||
|
if parsed.identity == "ACK-ACK":
|
||||||
|
return True
|
||||||
|
if parsed.identity == "ACK-NAK":
|
||||||
|
log(f" NAK for {label}")
|
||||||
|
return False
|
||||||
|
log(f" no ACK for {label} (continuing)")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def poll_fw_version(ser, ubr):
|
||||||
|
"""Poll MON-VER; returns firmware string like 'HPG 2.10' or None."""
|
||||||
|
ser.write(UBXMessage("MON", "MON-VER", 2).serialize()) # 2 = POLL msgmode
|
||||||
|
t_end = time.time() + 3.0
|
||||||
|
while time.time() < t_end:
|
||||||
|
try:
|
||||||
|
_, parsed = ubr.read()
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
if parsed is not None and parsed.identity == "MON-VER":
|
||||||
|
sw = getattr(parsed, "swVersion", b"")
|
||||||
|
sw = sw.decode(errors="replace") if isinstance(sw, bytes) else str(sw)
|
||||||
|
return sw.strip("\x00 ")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def configure_receiver(ser, ubr, want_has, dynmodel):
|
||||||
|
"""Enable RTCM input, HP output, raw logging; optionally HAS. RAM layer only."""
|
||||||
|
base_cfg = [
|
||||||
|
(K["UART1_IN_RTCM3"], 1), (K["USB_IN_RTCM3"], 1),
|
||||||
|
(K["PVT_UART1"], 1), (K["PVT_USB"], 1),
|
||||||
|
(K["HPPOSLLH_UART1"], 1), (K["HPPOSLLH_USB"], 1),
|
||||||
|
(K["RAWX_UART1"], 1), (K["RAWX_USB"], 1),
|
||||||
|
(K["SFRBX_UART1"], 1), (K["SFRBX_USB"], 1),
|
||||||
|
(K["DYNMODEL"], DYNMODEL[dynmodel]),
|
||||||
|
]
|
||||||
|
ok = valset(ser, ubr, base_cfg, f"base output/input config (dynmodel={dynmodel})")
|
||||||
|
if want_has:
|
||||||
|
ok_has = valset(ser, ubr, [(K["NAVCOR_ENABLE_HOST"], 0),
|
||||||
|
(K["NAVCOR_ENABLE_GAL_HAS"], 1)], "GAL HAS enable")
|
||||||
|
return ok, ok_has
|
||||||
|
return ok, False
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------
|
||||||
|
# NTRIP
|
||||||
|
# ----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def caster_reachable(server, port, timeout=8):
|
||||||
|
try:
|
||||||
|
with socket.create_connection((server, port), timeout=timeout):
|
||||||
|
return True
|
||||||
|
except OSError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
class SerialRTCMSink:
|
||||||
|
"""Write-only wrapper handed to GNSSNTRIPClient as its output stream."""
|
||||||
|
def __init__(self, ser):
|
||||||
|
self._ser = ser
|
||||||
|
self.bytes_out = 0
|
||||||
|
|
||||||
|
def write(self, data):
|
||||||
|
# GNSSNTRIPClient may hand us (raw, parsed) tuples or raw bytes
|
||||||
|
if isinstance(data, tuple):
|
||||||
|
data = data[0]
|
||||||
|
if isinstance(data, (bytes, bytearray)):
|
||||||
|
self._ser.write(bytes(data))
|
||||||
|
self.bytes_out += len(data)
|
||||||
|
|
||||||
|
# queue-style interface (pygnssutils uses .put on Queue outputs)
|
||||||
|
def put(self, item):
|
||||||
|
self.write(item)
|
||||||
|
|
||||||
|
|
||||||
|
def start_ntrip(args, sink, reflat, reflon, app=None):
|
||||||
|
"""app (with .get_coordinates()) => live GGA that follows a drifting platform;
|
||||||
|
without app, falls back to a fixed reference position."""
|
||||||
|
from pygnssutils import GNSSNTRIPClient
|
||||||
|
client = GNSSNTRIPClient(app=app)
|
||||||
|
kwargs = dict(
|
||||||
|
server=args.ntrip_caster, port=args.ntrip_port, https=0,
|
||||||
|
mountpoint=args.ntrip_mount, datatype="RTCM",
|
||||||
|
ntripuser=args.ntrip_user, ntrippassword=args.ntrip_pass,
|
||||||
|
ggainterval=args.gga_interval,
|
||||||
|
ggamode=0 if app is not None else 1, # 0 = live from receiver, 1 = fixed
|
||||||
|
reflat=reflat, reflon=reflon, refalt=0.0, refsep=0.0,
|
||||||
|
output=sink,
|
||||||
|
)
|
||||||
|
ok = client.run(**kwargs)
|
||||||
|
return client if ok else None
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------
|
||||||
|
# Main streaming loop
|
||||||
|
# ----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class Session:
|
||||||
|
def __init__(self, args):
|
||||||
|
self.args = args
|
||||||
|
self.q = queue.Queue()
|
||||||
|
self.stop = threading.Event()
|
||||||
|
self.fix_hist = Counter()
|
||||||
|
self.hacc = []
|
||||||
|
self.tier = "none"
|
||||||
|
# latest fix, served to GNSSNTRIPClient.get_coordinates() for live GGA
|
||||||
|
self.live = {"lat": 0.0, "lon": 0.0, "alt": 0.0, "sep": 0.0}
|
||||||
|
ts = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
|
||||||
|
self.csv_path = Path(f"stream_{ts}.csv")
|
||||||
|
self.ubx_path = Path(f"stream_{ts}.ubx")
|
||||||
|
|
||||||
|
def get_coordinates(self):
|
||||||
|
"""pygnssutils app interface: live position for the NTRIP GGA uplink,
|
||||||
|
so the VRS reference follows a drifting platform (buoy)."""
|
||||||
|
return dict(self.live)
|
||||||
|
|
||||||
|
def reader_thread(self, stream, raw_log):
|
||||||
|
"""Serial/file -> raw .ubx tee -> parsed NAV messages -> queue."""
|
||||||
|
ubr = UBXReader(stream, protfilter=UBX_PROTOCOL, quitonerror=0)
|
||||||
|
while not self.stop.is_set():
|
||||||
|
try:
|
||||||
|
raw, parsed = ubr.read()
|
||||||
|
except Exception:
|
||||||
|
if self.args.replay:
|
||||||
|
break
|
||||||
|
continue
|
||||||
|
if raw is None:
|
||||||
|
if self.args.replay:
|
||||||
|
break
|
||||||
|
continue
|
||||||
|
if raw_log:
|
||||||
|
raw_log.write(raw)
|
||||||
|
if parsed is None:
|
||||||
|
continue
|
||||||
|
if parsed.identity in ("NAV-PVT", "NAV-HPPOSLLH"):
|
||||||
|
self.q.put(parsed)
|
||||||
|
self.q.put(None)
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
args = self.args
|
||||||
|
# ---------------- input: serial or replay file ----------------------
|
||||||
|
if args.replay:
|
||||||
|
stream = open(args.replay, "rb")
|
||||||
|
ser = None
|
||||||
|
log(f"REPLAY mode from {args.replay} (no config writes, no NTRIP)")
|
||||||
|
else:
|
||||||
|
import serial
|
||||||
|
from serial.tools import list_ports
|
||||||
|
if not args.port:
|
||||||
|
ports = [p.device for p in list_ports.comports()]
|
||||||
|
sys.exit(f"--port required. Detected ports: {ports or 'none'}")
|
||||||
|
ser = serial.Serial(args.port, args.baud, timeout=1)
|
||||||
|
stream = ser
|
||||||
|
log(f"connected {args.port} @ {args.baud}")
|
||||||
|
|
||||||
|
cfg_ubr = UBXReader(ser, protfilter=UBX_PROTOCOL, quitonerror=0)
|
||||||
|
fw = poll_fw_version(ser, cfg_ubr)
|
||||||
|
log(f"firmware: {fw or 'unknown'}")
|
||||||
|
fw_has = bool(fw and any(
|
||||||
|
fw.split("HPG")[-1].strip().startswith(v)
|
||||||
|
for v in ("2.1", "2.2", "3.")))
|
||||||
|
|
||||||
|
# ------------- correction tier selection -------------------------
|
||||||
|
ntrip_ok = False
|
||||||
|
if not args.log_only and args.ntrip_user and args.ntrip_pass:
|
||||||
|
ntrip_ok = caster_reachable(args.ntrip_caster, args.ntrip_port)
|
||||||
|
if not ntrip_ok:
|
||||||
|
log(f"NTRIP caster {args.ntrip_caster}:{args.ntrip_port} unreachable")
|
||||||
|
elif not args.log_only:
|
||||||
|
log("no NTRIP credentials (set GCGC_USER/GCGC_PASS or --ntrip-user/-pass)")
|
||||||
|
|
||||||
|
want_has = not ntrip_ok and not args.log_only and fw_has
|
||||||
|
if ntrip_ok:
|
||||||
|
self.tier = f"NTRIP RTK ({args.ntrip_caster}/{args.ntrip_mount})"
|
||||||
|
elif want_has:
|
||||||
|
self.tier = "Galileo HAS (E6-B, ~20 cm)"
|
||||||
|
elif args.log_only:
|
||||||
|
self.tier = "log-only (onboard solution)"
|
||||||
|
else:
|
||||||
|
log("no correction source available that improves on the onboard "
|
||||||
|
"solution (NTRIP unreachable/no credentials; HAS needs FW >= "
|
||||||
|
"HPG 2.10). Skipping enhanced streaming per project policy.")
|
||||||
|
log("use --log-only to record raw data for post-processing anyway.")
|
||||||
|
ser.close()
|
||||||
|
return 2
|
||||||
|
|
||||||
|
ok_base, ok_has = configure_receiver(ser, cfg_ubr, want_has, args.dynmodel)
|
||||||
|
log(f"receiver configured (base={'ok' if ok_base else 'partial'}"
|
||||||
|
+ (f", HAS={'ok' if ok_has else 'failed'}" if want_has else "") + ")")
|
||||||
|
|
||||||
|
log(f"correction tier: {self.tier}")
|
||||||
|
|
||||||
|
# ---------------- outputs -------------------------------------------
|
||||||
|
raw_log = None if args.replay else open(self.ubx_path, "wb")
|
||||||
|
csv_f = open(self.csv_path, "w", newline="")
|
||||||
|
csv_f.write(f"# stream_gps.py session - correction tier: {self.tier}\n")
|
||||||
|
csv_f.write("# datum: NAD83(2011) if GCGC NTRIP tier, else WGS84/ITRF\n")
|
||||||
|
writer = csv.writer(csv_f)
|
||||||
|
writer.writerow(["utc", "lat_deg", "lon_deg", "ellip_height_m",
|
||||||
|
"hAcc_m", "vAcc_m", "fixType", "carrSoln", "numSV"])
|
||||||
|
|
||||||
|
# ---------------- threads -------------------------------------------
|
||||||
|
t_read = threading.Thread(target=self.reader_thread, args=(stream, raw_log),
|
||||||
|
daemon=True)
|
||||||
|
t_read.start()
|
||||||
|
|
||||||
|
ntrip_client, sink = None, None
|
||||||
|
if ser is not None and self.tier.startswith("NTRIP"):
|
||||||
|
# need a first fix for the GGA reference position
|
||||||
|
reflat, reflon = args.lat, args.lon
|
||||||
|
if reflat is None:
|
||||||
|
log("waiting for first fix to seed NTRIP GGA position ...")
|
||||||
|
first = self._wait_first_fix(timeout=60)
|
||||||
|
if first is None:
|
||||||
|
log("no fix within 60 s; using 0/0 GGA (VRS may reject)")
|
||||||
|
reflat = reflon = 0.0
|
||||||
|
else:
|
||||||
|
reflat, reflon = first
|
||||||
|
sink = SerialRTCMSink(ser)
|
||||||
|
app = None if args.gga_fixed else self
|
||||||
|
log("GGA uplink: " + ("fixed reference" if app is None
|
||||||
|
else "live (follows platform drift)"))
|
||||||
|
ntrip_client = start_ntrip(args, sink, reflat, reflon, app=app)
|
||||||
|
if ntrip_client is None:
|
||||||
|
log("NTRIP connection failed (check credentials/mountpoint) - "
|
||||||
|
"continuing with onboard solution only")
|
||||||
|
self.tier += " [FAILED - onboard only]"
|
||||||
|
|
||||||
|
signal.signal(signal.SIGINT, lambda *_: self.stop.set())
|
||||||
|
|
||||||
|
# ---------------- consume + display ---------------------------------
|
||||||
|
last_pvt = {}
|
||||||
|
try:
|
||||||
|
while not self.stop.is_set():
|
||||||
|
try:
|
||||||
|
m = self.q.get(timeout=1.0)
|
||||||
|
except queue.Empty:
|
||||||
|
continue
|
||||||
|
if m is None:
|
||||||
|
break
|
||||||
|
if m.identity == "NAV-PVT":
|
||||||
|
last_pvt = dict(
|
||||||
|
utc=datetime(m.year, m.month, m.day, m.hour, m.min, m.second,
|
||||||
|
tzinfo=timezone.utc),
|
||||||
|
fixType=m.fixType, carrSoln=m.carrSoln, numSV=m.numSV,
|
||||||
|
lat=m.lat, lon=m.lon, height=m.height / 1000.0,
|
||||||
|
hAcc=m.hAcc / 1000.0, vAcc=m.vAcc / 1000.0)
|
||||||
|
elif m.identity == "NAV-HPPOSLLH" and last_pvt:
|
||||||
|
# high-precision refinement: lat/lon + Hp components (deg)
|
||||||
|
last_pvt["lat"] = m.lat + m.latHp
|
||||||
|
last_pvt["lon"] = m.lon + m.lonHp
|
||||||
|
last_pvt["height"] = (m.height + m.heightHp) / 1000.0
|
||||||
|
last_pvt["hAcc"] = m.hAcc / 1000.0
|
||||||
|
last_pvt["vAcc"] = m.vAcc / 1000.0
|
||||||
|
continue
|
||||||
|
if not last_pvt:
|
||||||
|
continue
|
||||||
|
p = last_pvt
|
||||||
|
self.live.update(lat=p["lat"], lon=p["lon"], alt=p["height"])
|
||||||
|
self.fix_hist[(p["fixType"], p["carrSoln"])] += 1
|
||||||
|
self.hacc.append(p["hAcc"])
|
||||||
|
writer.writerow([p["utc"].isoformat(), f"{p['lat']:.9f}",
|
||||||
|
f"{p['lon']:.9f}", f"{p['height']:.3f}",
|
||||||
|
f"{p['hAcc']:.3f}", f"{p['vAcc']:.3f}",
|
||||||
|
p["fixType"], p["carrSoln"], p["numSV"]])
|
||||||
|
n_epochs = sum(self.fix_hist.values())
|
||||||
|
if not sys.stdout.isatty() and n_epochs % 60 != 1:
|
||||||
|
continue # avoid log spam when piped
|
||||||
|
fixlab = FIX_NAME.get(p["fixType"], "?")
|
||||||
|
if p["carrSoln"]:
|
||||||
|
fixlab += "/" + CARR_NAME[p["carrSoln"]]
|
||||||
|
rtcm = f" rtcm {sink.bytes_out//1024} kB" if sink else ""
|
||||||
|
end = " " if sys.stdout.isatty() else "\n"
|
||||||
|
status = (f"\r{p['utc']:%H:%M:%S} {fixlab:<12s} "
|
||||||
|
f"{p['lat']:+.9f} {p['lon']:+.9f} h={p['height']:8.3f} m "
|
||||||
|
f"hAcc={p['hAcc']:6.3f} m sv={p['numSV']:2d}{rtcm}{end}")
|
||||||
|
sys.stdout.write(status)
|
||||||
|
sys.stdout.flush()
|
||||||
|
finally:
|
||||||
|
self.stop.set()
|
||||||
|
print()
|
||||||
|
if ntrip_client is not None:
|
||||||
|
try:
|
||||||
|
ntrip_client.stop()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
csv_f.close()
|
||||||
|
if raw_log:
|
||||||
|
raw_log.close()
|
||||||
|
if ser:
|
||||||
|
ser.close()
|
||||||
|
self._summary()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def _wait_first_fix(self, timeout):
|
||||||
|
t_end = time.time() + timeout
|
||||||
|
while time.time() < t_end:
|
||||||
|
try:
|
||||||
|
m = self.q.get(timeout=1.0)
|
||||||
|
except queue.Empty:
|
||||||
|
continue
|
||||||
|
if m is not None and m.identity == "NAV-PVT" and m.fixType >= 2:
|
||||||
|
self.live.update(lat=m.lat, lon=m.lon, alt=m.height / 1000.0)
|
||||||
|
return (m.lat, m.lon)
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _summary(self):
|
||||||
|
n = sum(self.fix_hist.values())
|
||||||
|
if not n:
|
||||||
|
log("no position epochs received")
|
||||||
|
return
|
||||||
|
log(f"session summary: {n} epochs, tier = {self.tier}")
|
||||||
|
for (fix, carr), c in sorted(self.fix_hist.items(), key=lambda kv: -kv[1]):
|
||||||
|
lab = FIX_NAME.get(fix, "?") + ("/" + CARR_NAME[carr] if carr else "")
|
||||||
|
log(f" {lab:<14s} {c:6d} ({100*c/n:.1f}%)")
|
||||||
|
h = sorted(self.hacc)
|
||||||
|
log(f" hAcc: mean {sum(h)/len(h):.3f} m, median {h[len(h)//2]:.3f} m, "
|
||||||
|
f"95% {h[int(len(h)*0.95)]:.3f} m")
|
||||||
|
if not self.args.replay:
|
||||||
|
log(f" outputs: {self.csv_path}, {self.ubx_path}")
|
||||||
|
log(f" post-process the raw log: venv/bin/python process_gps.py {self.ubx_path}")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__,
|
||||||
|
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
ap.add_argument("--port", help="serial port (COM7, /dev/ttyUSB0, ...)")
|
||||||
|
ap.add_argument("--baud", type=int, default=38400)
|
||||||
|
ap.add_argument("--replay", type=Path, help="read a .ubx file instead of serial "
|
||||||
|
"(offline parser/CSV test mode)")
|
||||||
|
ap.add_argument("--log-only", action="store_true",
|
||||||
|
help="skip corrections; just log raw data + onboard positions")
|
||||||
|
ap.add_argument("--lat", type=float, help="initial latitude for NTRIP GGA (else first fix)")
|
||||||
|
ap.add_argument("--lon", type=float, help="initial longitude for NTRIP GGA")
|
||||||
|
ap.add_argument("--gga-fixed", action="store_true",
|
||||||
|
help="send a FIXED GGA reference instead of live position "
|
||||||
|
"(only for a truly stationary antenna; default is live, "
|
||||||
|
"so the VRS reference follows a drifting buoy)")
|
||||||
|
ap.add_argument("--dynmodel", default="sea", choices=sorted(DYNMODEL),
|
||||||
|
help="receiver dynamic platform model, set in RAM at startup "
|
||||||
|
"(default: sea, for buoy deployment)")
|
||||||
|
ap.add_argument("--ntrip-caster", default="rtn.usm.edu")
|
||||||
|
ap.add_argument("--ntrip-port", type=int, default=2101)
|
||||||
|
ap.add_argument("--ntrip-mount", default="VRS_GNSS_RTCM34_NAD83")
|
||||||
|
ap.add_argument("--ntrip-user", default=os.environ.get("GCGC_USER"))
|
||||||
|
ap.add_argument("--ntrip-pass", default=os.environ.get("GCGC_PASS"))
|
||||||
|
ap.add_argument("--gga-interval", type=int, default=15)
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
if not args.replay and not args.port:
|
||||||
|
ap.error("one of --port or --replay is required")
|
||||||
|
sys.exit(Session(args).run())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Loading…
Add table
Reference in a new issue