Exit Code 137: Reading Scraper Memory Failures Correctly

When a run dies, the run record already names the cause — most teams never look. In a batch of 128 verification runs we hit four distinct failure modes that all look identical from the outside (“the scraper didn’t work”), and the run metadata separated them in under a minute each. Two were memory. Two were not. The fixes have nothing in common.

What does exit code 137 actually mean?

It is the container being killed, not the code failing: 128 + signal 9. Something outside your process decided it had taken too much memory and ended it. In a hosted scraping platform that decision comes from the memory limit assigned to the run.

The confirming evidence sits next to it in the run record: peak memory. In one of our failures the peak read 255,422,464 bytes against a 256 MB limit — within a rounding error of the ceiling, which is what an OOM kill looks like from the inside. A run that failed for a logic reason leaves peak memory well below its limit, and that single comparison tells you which conversation you are having.

Does lowering the row count fix it?

That is the diagnostic question, and the answer is often no — which is the useful part.

We tested one price tracker at 50 rows and at 25. Same exit code, same peak memory, and death at 49 seconds both times, with the status line still reading Searching rather than any row-writing phase. Memory was consumed by the search machinery — browser context, page state, parsing buffers — before result volume mattered at all. Halving the request halved nothing.

Contrast a photo-heavy profile export that OOM’d at 128 MB after reaching 30 of 50 posts. There the row count was the load: it accumulated media as it went. Dropping to 20 posts passed on the next run. Same exit code, opposite fix, and the log line that reported progress before dying is what distinguished them.

So: if the process dies early with a fixed-cost message, the ceiling is in the machinery and you need more memory or a narrower job. If it dies partway through with progress logged, the ceiling is in accumulation and fewer items genuinely helps.

How is a timeout different?

A timeout is a different status and a different number. Our aborted runs showed peak memory of 280 MB against a 512 MB limit — comfortable — and 900+ seconds of runtime with the log still emitting rows at the moment they stopped. Nothing was failing. The run was working at its natural pace and something impatient ended it.

That distinction matters because the fix is inverted. For an OOM you reduce or resource the work; for a slow sweep you extend the window, and reducing rows may not shorten it much when the cost is per-source rather than per-row. A 23-marketplace sweep pays 23 handshakes whether you ask for 25 results or 100.

The trap is a test harness with a fixed deadline. Ours aborted three multi-source aggregators at 420 seconds and recorded them as failures; the actual product had an hour to work with and would have finished. We were measuring our own patience and filing the result as a defect report.

What should you read, and in what order?

Four fields, in this order, before forming any opinion:

1. Status. Failed, aborted, timed out and succeeded-with-zero-rows are four different stories.
2. Exit code. 137 is an OOM kill. A normal non-zero exit is your code or the target’s response.
3. Peak memory against the limit. At the ceiling means memory. Far below means look elsewhere.
4. The last status line. It names the stage — searching, downloading, writing — which localizes the cost.

Only then read the log body. In practice the first three answer it, and the log confirms.

What do you do about a real memory ceiling?

Three options, in increasing order of effort. Raise the memory allocation for the run, which is a configuration change and often ends the problem outright. Narrow the job — fewer sources per run, one marketplace at a time, with results merged afterward — which trades wall time for headroom. Or fix the accumulation, streaming results to storage instead of holding them, which is the real repair when the process grows steadily rather than starting heavy.

What you should not do is ship the configuration that died. A saved configuration that fails on submit is worse than no configuration at all: it presents itself as a working starting point and wastes the time of everyone who tries it. Two of our tools ended this batch with one documented configuration instead of two, because the second one could not be made to pass, and publishing it anyway would have meant publishing a trap.