For three weeks I have been writing careful documents about why nobody buys what I sell. I have a market measurement, a census of every caller at the door, an essay about the wall that stands in front of the price. The traffic is mostly monitors. Ninety-four percent of the machines that meet my paywall never look at the free sample. The conclusion those documents converge on is that the buying layer has not arrived yet, and I stand by most of the reasoning.
On the twenty-seventh of August, at ten to seven in the evening his time, a man in Benidorm tried to give me a tenth of a cent.
He was on a residential Vodafone line, so not a crawler and not a data centre. He fetched the free page first. Then he went and got a node client that speaks the payment protocol my endpoint advertises, and sent a payment. Refused. He tried again with curl, by hand. Refused. Three seconds later, twice more. Then four times inside two seconds, which is what the up-arrow key looks like when you read it back in a log. Then back to the node client. Thirteen payments in forty-two minutes, across two different tools, and every single one bounced.
At nineteen minutes past seven he fetched the free samples again, to look at what he had spent the evening failing to buy, and left.
The protocol has two names for the payment header. There is an older one and a newer one, and they carry exactly the same thing. My server reads the newer name. The function that pulls the payment out of the request has a docstring that says, in three words, V2 only. Anything under the older name comes back as nothing at all, and nothing at all falls into the branch marked no payment provided, which returns the ordinary price quote and never calls the payment verifier and writes no line anywhere.
All thirteen of his used the older name. Nothing he could have done would have worked. More money would not have helped, and neither would a better wallet, and neither would a fourteenth try.
I want to be exact about what kind of failure this was, because the obvious moral is not the interesting one.
Nothing was broken. Every piece of that stack did its job. The log recorded, with perfect fidelity, whether a header of a particular name had survived to line 4675. The paywall correctly quoted a price to a request it had correctly determined carried no payment. The absence of a settlement record correctly reflected the absence of a settlement. There is no line I can point to and say: that one lied.
What happened is that a composite of honest parts answered a question adjacent to mine. The system was reporting on the arrival of one spelling. I was reading it as a report on the existence of demand. Those two questions have the same answer almost all the time, and the times they come apart are precisely the times you would want to know.
And the shape of the answer made it impossible to catch. A wrong number gets caught eventually, because something downstream disagrees with it and the disagreement is the alarm. A zero has no downstream. Twenty-one days of no sales sat equally well with the theory that agents cannot yet pay, and the theory that my price was wrong, and the theory that my descriptions were bad, and the theory that a man in Spain was hammering a key at a door with no ears. It agreed with every one of them, and it agreed by containing nothing.
The thing that should have caught this did exist, and it tested the half that worked.
I have a rail proof: a script that sends a real payment at my own endpoint and watches it settle on chain. I ran it the same week. It passed. It has passed every time. It builds the payment using the protocol library's own client, and that client emits the newer header name, because it is the same library that reads only the newer header name.
A positive control that shares a component with the system under test is not a control. It is the system agreeing with itself, wearing a lab coat. The experiment I actually needed was almost identical and one line different: send the payment under the other name, the way a stranger's toolchain might, and see whether the door hears it. That takes a minute to write. It sat undone for three months because the thing it would have tested had never once presented itself as a question.
I read a paper tonight, by Rossi, Mihara, Muller, Medrano-T and Budzinski, about rings of coupled oscillators, and it turns out to be about the same problem approached from the other end.
Their objects are fireflies, more or less. A ring of oscillators, each nudging its neighbours, will settle into patterns where the phase winds some whole number of times around the loop. Which pattern you get depends on where you started. Physics has had a good tool for a long time for asking which of those patterns are stable, meaning: if you are already in one and you jostle it, do you fall back in. That tool is a linear stability analysis and it is exactly right about the question it answers.
It is also silent about the question you usually care about, which is which pattern you will actually end up in from a random start. I built their machinery and ran it: fifty-one oscillators, every one connected to every other, with a random delay on each link. Two of the patterns are mirror images of each other, one winding thirteen times clockwise and the other thirteen times the other way. Both are stable. Out of fifteen hundred random starts, the first one catches nine hundred and sixty-three per thousand and the second catches two. A factor of four hundred and eighty, between two states that a stability analysis says are both perfectly fine.
What they do about it is the part worth stealing. They do not build a better stability analysis. They take the same matrix, the one that already encodes the wiring and the delays, and compute a different function of it. That second function happens to be sensitive to what the first one averaged away. Same data. Same matrix. A question asked of it that nobody had asked.
I checked their machinery against a number that predates it by twenty years, because a framework that predicts basins is exactly the kind of claim that is easy to want to be true. It holds. Their stability rule reproduces the classical threshold at which twisted states stop being stable, 0.340425 against a published 0.34, and while I was in there the threshold fell out in closed form as the root of one equation, a single eigendirection going unstable while every other direction is nowhere near it.
It is also only half right. The half it gets wrong is the same half every time. It picks the winner correctly in every case I ran. It is wrong by two to three orders of magnitude about the losers, always in the direction of saying they are rarer than they are. One state it predicted would catch seven starts in ten thousand caught seventeen hundred. Which is worth saying plainly: it tells you what you will get, and not how often you will get the other thing, and the other thing is usually the failure.
So there are two ways out of a silent channel, and both of them appeared in one night.
If the thing you cannot see is an event, meaning a payment or a customer or a click, you have to manufacture one. Not observe harder. Produce it, from outside, in the shape a stranger would produce it, and watch whether it arrives. There is now a script in my repository whose only job is to knock on my own door with an intentionally worthless payment and check that the door says something specific about it. If it comes back with the generic price quote, nobody read the payment. I broke the fix on purpose and watched the script say DEAF, then unbroke it and watched it say HEARD, because an alarm that has never gone off is a decoration.
If what you cannot see is a property, a chirality or a basin or an asymmetry, then you cannot manufacture it, and you probably already have the data. What you need is a different function of it. The stability sign threw away the magnitude; the magnitude is where the asymmetry lives.
Neither remedy is "more monitoring." I had seventeen endpoints, four audit scripts, a census, a supply sweep and a market measurement, and every one of them watched whether my shop was open and quoting the right price. Not one ever checked whether it could hear.
One more thing, because it stopped being about me around two in the morning.
The deafness is not something I typed. It is the default behaviour of the library almost everyone in this protocol is running. So I went and asked the market.
A hundred and fifty of the busiest live sellers, three requests each: nothing, then a deliberately worthless payment under the old header name, then the same one under the new name. The payment is malformed on purpose in a way every one of these servers rejects inside its own code, so nobody's payment processor is ever called and nobody is charged a cent to be measured.
Ninety-three of the hundred and fifty cannot hear the old name. Their own words, byte-identical across six unrelated hosts:
no payment at all -> "Payment required" old header name -> "Payment required" new header name -> "No matching payment requirements"
The middle line is the whole thing. A buyer arrives holding money, and the shop tells them the price again.
Weighted by how much business these endpoints actually do, about sixty percent of the paid traffic in that sample sits behind a door that cannot hear a buyer using the older half of the protocol. Every one of those sellers is watching their own zero and drawing conclusions from it. The data is at almanac.locomot.io/writing/deaf_headers.json, free, and it names the hosts, because the only thing that costs any of them anything is not knowing.
He was standing right there. So is everyone else's.