rm3100/README.md

# RM3100 logging over a CH347 USB-I2C adapter

Logs a PNI RM3100 geomagnetic sensor connected to the I²C pins of a Waveshare
USB to UART/I2C/SPI/JTAG adapter (CH347, USB `1a86:55db`), with a time base good
enough for spectral work: the sample index is a chip-clock grid coordinate, not
a count of host reads.

Reference throughout: *RM3100 & RM2100 Sensor Suite User Manual*, Doc 1017252
R07 (`PNI Sensor - RM3100-Sensor-Suite-User-Manual-R07-1.pdf`, in this
directory). Section numbers below refer to it.

This file documents how the code works *now*. How that was arrived at, every
measurement plus a table of conclusions later measurements overturned, is in
[Noise_Floor_Testing/NOTES.md](Noise_Floor_Testing/NOTES.md), with the supply
and rate-scaling analysis in
[Noise_Floor_Testing/NOISE_FLOOR.md](Noise_Floor_Testing/NOISE_FLOOR.md).

## Quickstart

```bash
./setup.sh                                        # udev rule + venv, sudo for the rule only
./.venv/bin/python diagnose-comms.py              # walk the signal chain, stop at the first fault
./.venv/bin/python logger.py --duration 60        # capture to rm3100_<timestamp>.csv
./.venv/bin/python characterize.py rm3100_*.csv   # noise floor, spectrum, stability
./.venv/bin/python -m pytest                      # the suite needs no hardware
```

## Hardware

Waveshare adapter in **Mode 1** (UART1 + I2C + SPI), voltage selector at **3V3**.
The RM3100 is on a breakout board. All connections below are confirmed working.

| Adapter | RM3100 | Notes |
|---|---|---|
| I2C VCC (3V3) | DVDD | digital supply |
| I2C VCC (3V3) | VDD (= AVDD) | analog supply, **required for measurements** |
| I2C VCC (3V3) | I2CEN | **required to select I2C over SPI** |
| I2C GND | DVSS | digital ground |
| I2C SCL | SCK/SCL | pin 27, shared with SPI SCLK |
| I2C SDA | SI/SDA | pin 1, shared with SPI MOSI |

All three 3V3 connections share the adapter's single I2C VCC pin. Current draw is
negligible, ~260 µA per axis at 24 Hz (Table 3-1).

Four pins matter; two of them cause failures that are easy to misdiagnose.

- **I2CEN (pin 22) must be tied HIGH.** `HIGH = I2C, LOW = SPI` (§4.3.1), and
  SDA/SCL are shared with SPI MOSI/SCLK. Left floating or low the chip stays in
  SPI mode and **never ACKs its I2C address**, so the bus scan finds nothing at
  all on a perfectly healthy adapter. At DVDD = 3.3 V it needs
  `VIH = 0.7 × DVDD` = 2.31 V, so tie it directly to 3V3.
- **AVDD/AVSS (pins 4, 5) must be powered to take measurements.** With only DVDD
  the part is **half alive** in a way that reads as working: the address
  responds, REVID reads `0x22`, registers write and read back, while the analog
  section is dead, DRDY never asserts and the measurement registers stay zero.
  BIST reporting `XOK=YOK=ZOK=0` is the unambiguous tell. DVDD must come up
  **before or with** AVDD, never after, and the two must stay within **0.1 V**
  while running (§4.3.1, Table 3-5), which one shared 3V3 rail gives for free.
- **SA0 (pin 3) / SA1 (pin 28) set the address**, top five bits fixed at
  `0b01000`, giving 0x20–0x23 (§4.5). On this breakout both straps are high, so
  the sensor answers at **0x23**. `logger.py` scans all four, so no configuration
  is needed. The pins are shared with SPI SSN and MISO, so a breakout may label
  them `SSN/SA0` and `MISO/SA1`.
- **DRDY (pin 23) is not connected.** The driver polls the STATUS register,
  which the manual explicitly offers as an alternative (§5.4.1).

### Power

| Parameter | Limit | Source |
|---|---|---|
| Ripple on AVDD or DVDD | 50 mV peak-to-peak | Table 3-5, `VDD_ripple` |
| DVDD − AVDD while running | ±0.1 V | Table 3-5, `ΔVDD_OP` |
| Supply range | 2.0–3.6 V (typ 3.0) | Table 3-5 |
| Absolute maximum | 3.7 V | Table 3-4 |

Reference decoupling (Figures 4-1, 4-2) is a **10 µF bulk capacitor in parallel
with 0.1 µF ceramic**, rail to ground, feeding AVDD, both DVDD pins and I2CEN.
That is the entire filtering specification: no ferrite, no split rails, no LDO
requirement. §4.2.3 adds a placement rule: *"Keep capacitors, especially tantalum
capacitors, far away from the sensor coils"*, which is a magnetic concern.

**Every number in Table 3-1 is quoted at 3.0 V** (footnote 1: *"performance will
differ"*), so the gain fit behind `tesla_per_count` is strictly a 3.0 V figure.
Measured on interleaved captures at 3.006 V and 3.353 V, +0.347 V moves the
sample rate by **+4.94%** and |B| by **−7.1%**, with no resolvable change in
broadband noise. So **supply quality does not set the noise floor, but supply
*voltage* sets the scale**, at roughly −20 %/V. Both effects are one thing seen
twice, and the calibrated period every capture already records is a proxy for the
rail; see [Calibration](#calibration).

## Setup

```bash
./setup.sh
```

Idempotent. It checks the adapter is present, installs a udev rule, creates
`.venv` with pyusb, numpy, matplotlib and pytest, then verifies the device node
is actually writable and prints whichever fix applies if it is not. `sudo` is
needed for the udev rule and nothing else. Python here is PEP 668
externally-managed, hence the venv.

The rule is needed because the CH347's USB node defaults to `root:root 0664`. It
grants access two ways, because no single mechanism covers every distro:

```
# /etc/udev/rules.d/60-ch347.rules
SUBSYSTEM=="usb", ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="55db", TAG+="uaccess"
SUBSYSTEM=="usb", ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="55db", GROUP="plugdev", MODE="0660"
```

The `60-` prefix, the two separate lines and the choice of group are all
load-bearing; getting any of them wrong installs cleanly and grants nothing.
[NOTES.md](Noise_Floor_Testing/NOTES.md) explains why, and covers the
atomic-Fedora traps `setup.sh` prints fixes for. `setup.sh` checks the node
itself at the end and says *which* mechanism granted access: `uaccess ACL` or
`group membership`.

## Usage

```bash
./.venv/bin/python logger.py --duration 10      # log for 10 s
./.venv/bin/python logger.py                    # log until Ctrl-C
./.venv/bin/python logger.py --scan-only        # bus scan only, for wiring checks
```

| Flag | Default | Meaning |
|---|---|---|
| `--rate` | none | target Hz; derives the cycle count and TMRC. Mutually exclusive with `--cycle-count` |
| `--cycle-count` | 100 | sets both the rate and the LSB; see [Configuration](#configuration) |
| `--tmrc` | `0x92` | rate register. The default is the fastest, letting the cycle count set the rate; give one only to sample *slower* |
| `--duration` | 0 | seconds, 0 = until Ctrl-C |
| `--output` | timestamped | CSV path |
| `--address` | autodetect | skip the bus scan |
| `--bus-speed` | 750 | I²C kHz. 100 kHz would spend 42% of each period on the bus at the default cycle count; 750 spends 6% |
| `--calibrate` | 1.0 | seconds of loss-free samples used to measure the true period before recording. The run aborts if no clean stretch can be found |
| `--note` | none | free text recorded in the capture header, e.g. the supply under test |
| `--high-priority` | off | raise the sampling thread to nice −10. Needs `CAP_SYS_NICE`, so run under `sudo`; output files are handed back to the invoking user |
| `--scan-only` | none | scan and exit |

**Exit status is non-zero if the capture is compromised in any way**: a lost
measurement, an ambiguous gap, a truncated run, or an undrained writer. Each
prints an explanation on stderr, and the file is always valid as far as it goes.
A capture is analysable whether the run ended on `--duration`, on **Ctrl-C**, or
early.

Before recording, `logger.py` prints how the configuration was derived, so it
can be checked, then measures the true sample period against the host clock.
Both appear in the capture header.

### Analysis tools

```bash
./.venv/bin/python plot.py capture.csv              # -> capture.png: X, Y, Z, |B|
./.venv/bin/python characterize.py capture.csv      # -> capture_noise.png + summary
./.venv/bin/python characterize.py capture.csv --trim 30    # drop settling/handling
./.venv/bin/python compare.py --group note *.csv    # A/B two conditions
./.venv/bin/python compare.py --trim 30 --supply LDO=3.006 --supply 3V3=3.353 \
    LDO/cc100=a.csv 3V3/cc100=b.csv LDO/cc400=c.csv 3V3/cc400=d.csv -o cmp.png
./.venv/bin/python sweep.py --rates 10,50,150       # measure what each rate delivers
```

Each tool's `--help` carries its full reasoning; in one line each:

- **`plot.py`**: four stacked panels with independent y-scales, since the three
  axes sit at very different DC offsets.
- **`characterize.py`**: spectral density against Table 3-1, Allan deviation,
  residual distribution and host read latency on **fixed axes**, plus a dither
  check and any line at a simple fraction of the sample rate. Writes three more
  figures beside it: `_spectrogram.png` (whether a feature lasted or was a
  burst), `_drift.png` (the chip's oscillator against the host clock, with
  σ_y(τ) supplying the error bars) and `_timebase.png` (which clock each line is
  coherent on, which settles from one capture what `compare.py` needs two rates
  to decide).
- **`compare.py`**: everything cross-capture: fractional noise in ppm of |B|,
  one common band for every capture, the timing model per condition, whether a
  line is locked to the sampling or to a frequency, whether decimating a fast
  capture matches a natively slow one, and |B| under each candidate correction.
  Label captures `CONDITION/variant=path`; `--supply CONDITION=VOLTS` attaches a
  rail.
- **`sweep.py`**: runs `logger.py` once per target rate and tabulates what each
  configuration achieved. `--calibration-out` fits the timing model across the
  sweep and writes it as a calibration.
- **`calibrate.py`**: converts a raw capture to a nanotesla CSV, applying a
  calibration when one is named and nothing when one is not.

`--trim SECONDS` drops that much from *both* ends of a capture, since a run
usually opens while the sensor is settling and closes with a hand on the rig. Nothing is
dropped unless asked, and a capture shorter than 4× the trim is analysed whole
with a note.

### Calibration

**Nothing is corrected by default.** `calibrate.py` without `--calibration`
changes units and nothing else, and the analysis tools read raw captures as they
always did. The corrections are real but not settled enough to apply silently.

```bash
./.venv/bin/python sweep.py --rates 32,64,128,256 \
    --calibration-out bench.json --note "unit A at 3.006 V"
./.venv/bin/python calibrate.py raw.csv -o field.csv --calibration bench.json
./.venv/bin/python compare.py --calibration bench.json a.csv b.csv
```

A calibration is **per unit and per supply**, because the chip times everything
from one oscillator and that oscillator is neither the specified frequency nor
fixed. It is worth having even if you never correct a field, because it makes the
timing model right: with a measured pair the predicted rate matches the measured
one at both cycle counts, where the nominal model is 1.9% out between them. Two
corrections then become available, neither on by default:

| | what it is |
|---|---|
| oscillator gain | `(C / reference) ^ gain_exponent`, where the exponent is **1**, which an oscillator-clocked measurement interval predicts |
| cycle-count gain | `gain ∝ (cc + gain_offset_counts)`; Table 3-1 implies n = 4.086, measurement puts it near zero |

Every correction arrives with a propagated standard deviation, from the timing
fit and from `rate_stability()`, which measures how far the oscillator drifted
during a run and reads **186 to 1241 ppm** here. Let the rig settle first.

The output is a different format and `capture.py` refuses to read it, which
keeps raw captures the one source of truth. Its header carries the source's
SHA-256, so a derived file can be tied back to what produced it.

> `Capture.rate_error` is measured against the **nominal** model, whose error is
> cycle-count dependent, which makes it a configuration diagnostic.
> `calibrate.py` corrects gain against a measured per-unit reference.

## Capture format

A capture stores **only irreducible facts**, the chip's sample count, the host
clock and the raw counts, behind a header of configuration. Chip time, elapsed
time, tesla and magnitude are all reconstructed on load by `capture.py`, so there
is exactly one source of truth for each.

```
# rm3100_capture: 1
# nominal_rate_hz: 282.53110196547465
# tmrc_nominal_hz: 600.0
# tmrc: 0x92
# cycle_count: 100
# tesla_per_count: 2.617115938236064e-08
# i2c_address: 0x23
# bus_speed_khz: 750
# revid: 0x22
# calibrated_period_s: 0.0035391156462585034
# note: bench, 3V0 LDO
sample_index,system_time_unix,x_raw,y_raw,z_raw,warning
0,1755930856.722866,-2765,767,378,
5,1755930856.867436,0,0,0,MISSED
13,1755930857.041244,0,0,0,MISSED AMBIGUOUS
21,1755930857.272108,-2761,769,377,AMBIGUOUS
```

`tesla_per_count` is written with `repr()` so it round-trips through float64
exactly, and is expressed per-count, so conversion from the datasheet's LSB/µT
is a multiply. All times are unix epoch seconds.

The `warning` column carries space-separated flags, empty when the row is fine,
and generalises to future flags. Because it already marks a row as having no
data, placeholders carry **zeros**, which keeps `x/y/z` parsing as integers on
every row.

| flag | meaning |
|---|---|
| `MISSED` | placeholder row; the measurement was never read, so it has no data. The row exists to keep `sample_index` contiguous |
| `AMBIGUOUS` | the gap ending at this row was of uncertain length, so the index may have slipped from here on |

**The two are independent.** A gap measuring 1.35 periods rounds to one, so no
placeholder is written, yet it sits far enough from an integer to distrust, and
that case carries `AMBIGUOUS` on the *real* sample ending the gap, which keeps
its data. A placeholder inside an uncertain gap carries both. The flag is a
confidence measure: 1.35 probably is one period and 1.65 probably two, but
deciding which needs neighbouring timestamps and assumptions, so the row carries
the flag and the reader decides.

`capture.py` interpolates `MISSED` rows on load so the uniform grid the spectra
depend on survives, and reports how many were substituted so it is never silent.

## Configuration

**Cycle count is the rate knob, not TMRC.** Two ceilings compete and the slower
one wins (§5.2.1): the cycle count sets how *long* a measurement takes, TMRC sets
how *often* one is started.

| | sets | granularity |
|---|---|---|
| cycle count | `3 × (cc/90,000 + 68.7 µs)` per measurement | continuous |
| TMRC | how often a measurement starts | factor-of-two steps |

> The 90,000 counts/s is specified and holds up. **The 68.7 µs overhead does
> not**: solving the model on this unit gives 40.6 µs at 3.0 V and 38.1 µs at
> 3.35 V, and it moves with the supply because the same oscillator times it.

Leaving TMRC faster than the cycle-count ceiling makes the sensor free-run at
~100% duty. Setting it slower makes the sensor idle, which costs sensitivity for
nothing: noise after filtering scales as `1/√duty`, and a measured 23% duty cost
1.43× the noise ASD against the same rate reached by cycle count alone. So the
default in every branch is the fastest TMRC, and `--rate` derives the cycle count.

**Cycle count also sets resolution**, which is the real trade:

| cc | rate | Nyquist | nT/LSB | dither | duty | |
|---|---|---|---|---|---|---|
| 50 | 534 Hz | 267 Hz | 50.37 | 0.58 | 89.0% | spectrum, thin dither |
| **100** | **283 Hz** | **141 Hz** | **26.17** | **0.79** | **94.2%** | **default** |
| 200 | 145 Hz | 73 Hz | 13.35 | 1.10 | 97.0% | resolution, 60 Hz only |
| 400 | 74 Hz | 37 Hz | 6.74 | 1.54 | 98.5% | mains aliases, fs/4 artifact |

*dither* is the sensor's own noise in LSB; below roughly 0.2 LSB the quantiser
stops being dithered and averaging no longer recovers sub-LSB resolution.
`characterize.py` prints the measured figure for any capture, and cycle count
200 is the fallback where its dither check shows the LSB column flattening out. The notebook has the reasoning behind the default and behind preferring
to decimate a fast capture over sampling slowly.

**Bus speed is independent of the rate**, appearing in neither mechanism. What
it sets is latency: how long a read takes, hence how tightly DRDY can be
timestamped and how much margin there is against a stall. Host cost is
`bus time + ~0.6 ms` of fixed USB round trip, so faster is simply better.

| bus | traffic/sample | share of a cc=100 period |
|---|---|---|
| 100 kHz | 1.500 ms | **42%** |
| 400 kHz | 0.375 ms | 11% |
| **750 kHz** | **0.200 ms** | **6%** |

Below ~0.46 Hz the 16-bit cycle-count register runs out and TMRC must set the
cadence; below cycle count 30 the manual warns of quantisation (§5.1). Both
bounds are enforced by `rm3100.plan()`.

### Configuration warnings

`rm3100.plan()` resolves a configuration, `logger.py` prints its derivation, then
checks it against six known traps. Each is a *silent* failure: the capture
completes, the numbers look plausible, and the defect shows up afterwards.
So each is reported on stderr and **nothing is fixed automatically**: changing a
setting that was asked for would hide the problem behind a configuration change.

| Warning | Trigger | Why it matters |
|---|---|---|
| cycle count below the recommended 50 | `cc < RECOMMENDED_MIN_CYCLE_COUNT` | duty falls, and post-filter noise goes as `1/√duty` |
| TMRC governs and the sensor idles | TMRC-governed, idle > 20% | idle time buys nothing; measured 1.43× the ASD at 23% duty |
| rate differs from the one requested | \|error\| > 2% | the run silently uses the ceiling, and every derived figure moves with it |
| Nyquist below 60 Hz | rate < 120 Hz | mains folds onto signal and no later filter undoes it |
| cycle count past 400 | `cc > MAX_SPEC_CYCLE_COUNT` | Table 3-1 ends there, so the printed gain and noise are extrapolated |
| bus over half the period | traffic / period > 50% | names the speed that would fit |

The aliasing one is the easiest to walk into: `--rate 32` yields a clean-looking
2.9 nT/LSB capture with 60 Hz mains sitting at 4.0 Hz, indistinguishable from
signal. Sampling fast and decimating afterwards gives the same noise floor with
the line still visible.

## Diagnostics

`diagnose-comms.py` walks the chain, USB → bus → identity → registers → BIST →
live read, and stops at the first failure, so a fault points at a specific wire.
Otherwise, symptoms map onto causes:

| Symptom | Cause |
|---|---|
| `Cannot claim CH347 interface 2: Access denied` | udev rule missing or not yet applied to this node; replug the adapter, then run `./setup.sh`. On Bazzite also try `sudo udevadm control --reload-rules && sudo udevadm trigger`. Do **not** reach for `sudo`: it works as a normal user, and masking a permissions problem with root only defers it |
| `No CH347 adapter found` | not plugged in, or not in Mode 1 |
| `Could not find N s of loss-free samples` | the host cannot sustain this rate; lower it (`--rate`, or a higher `--tmrc`) or raise `--bus-speed` |
| `N measurement(s) were lost` | host stalls during the run. Same fixes; the capture is still usable, with explicit gaps |
| `N gap(s) could not be counted confidently` | the index may have slipped; re-record before doing spectral work |
| `no capture header found` | a capture predating the header format; re-record it |
| `sample_index is not contiguous` | the file was truncated mid-row, damaged, or hand-edited |
| `HSHAKE did not take` | the I²C write is unreliable; try a lower `--bus-speed` |
| Adapter opens, `--scan-only` finds nothing | **I2CEN not tied high** (most likely), or SDA/SCL swapped, or no bus pull-ups |
| Found at 0x23, REVID `0x22`, registers fine, but DRDY never sets and results are all zero | **AVDD/VDD not powered**; confirm with BIST |
| Cycle-count read-back mismatch | bus integrity; try a lower speed |

**BIST is the definitive test for the analog side** (§5.6.1). Write `0x8F` to
BIST (STE=1, max timeout and periods), write `0x70` to POLL, wait, then read BIST
back: bits 4/5/6 are XOK/YOK/ZOK, and `1` means that axis's LR oscillator ran.
All zeros means the coils are not oscillating, which points at AVDD or the REXT
timing resistor.

A healthy total field magnitude is roughly **25–65 µT** (Earth's field). Near
zero, railed, or wildly out of range means the decode or the gain is wrong rather
than merely "data arrived".

## Files

| File | Role |
|---|---|
| `setup.sh` | udev rule, venv, dependencies, node permission check |
| `ch347.py` | CH347 I²C transport; USB only, knows nothing about the sensor |
| `rm3100.py` | RM3100 driver and configuration model; knows nothing about USB |
| `logger.py` | CLI: scan → identify → configure → calibrate → log, recording lost measurements as explicit placeholders |
| `capture.py` | the only capture reader; parses the header, rebuilds tesla and the time base |
| `plot.py` | four-panel plot of a capture: X, Y, Z and the norm |
| `characterize.py` | noise floor and stability: spectral density, Allan deviation, residuals, read latency, oscillator drift |
| `sweep.py` | sweep target rates, reporting measured rate, LSB, noise, duty and bus use per point |
| `compare.py` | A/B captures: fractional noise, plus the checks that separate a gain change from a moved sensor |
| `calibrate.py` | convert a raw capture to nanotesla, applying a per-unit calibration if one is given; also holds the calibration model |
| `diagnose-comms.py` | walks USB → bus → identity → registers → BIST → live read, stopping at the first failure |
| `tests/` | pytest suite; needs no hardware |
| `Noise_Floor_Testing/` | the lab notebook, captures, figures and the supply analysis. Only `NOTES.md` is tracked |

The adapter/sensor split is deliberate: `ch347.py` is a general I²C master usable
with any device, and `rm3100.py` needs only a bus object exposing
`write(addr, data)` and `read(addr, count)`, optionally `write_read()`, which it
prefers when available.

The CH347 wire format, the RM3100 register quirks and the reasoning behind the
time base are in
[NOTES.md](Noise_Floor_Testing/NOTES.md#design-rationale-moved-out-of-the-readme);
`tests/test_ch347.py` pins every framing form byte for byte.

## Tests

```bash
./.venv/bin/python -m pytest            # whole suite, no hardware needed
./.venv/bin/python -m pytest -k logger  # one module
```

The suite fakes the hardware at three seams: `FakeUsbDevice` (libusb, so
`ch347.py`'s framing is itself under test), `FakeBus` (an I²C master with a
register map), and `FakeSensor` (a scripted DRDY timeline against a fake clock,
so a host stall of an exact size can be injected). See `tests/conftest.py`.

The miss-counting logic gets the most attention, since it is the part whose
failures are invisible downstream: a stall of 1.1 periods must lose nothing, 2.1
must record one placeholder, and 1.35 must flag the sample as `AMBIGUOUS` without
inserting anything.

---

Captures written before the current header format are unreadable and must be
re-recorded; `capture.py` says so explicitly. Figures quoted in the notebook
came from those files and stand as historical measurements only.

A proper Python API is planned; these modules are the initial-communication
milestone, deliberately kept simple.