Most collected data dies in a dataset nobody opens. The gap between “the Actor runs” and “the team acts” is a delivery layer: a schedule, a store, a filter, and a notification that arrives where people already work. That layer is small — usually four nodes — but the details decide whether anyone trusts the alerts a month later.
What should the pipeline actually do?
Split the job into archive and alert, because they have opposite requirements. The archive wants everything, forever, unfiltered — it is what makes trend analysis possible later. The alert wants only what is new and relevant right now, or people stop reading it.
A minimal shape that satisfies both:
Schedule → Actor run → flatten rows → ┬→ append to spreadsheet (archive)
└→ diff + digest → chat message (alert)Every production pipeline we run follows this fork. Skipping the archive means you can never answer “when did this change?”; skipping the diff means the alert repeats itself until it is muted.
How does the data get out of Apify?
Three routes, in increasing order of control: the dataset export, the platform integrations, and the API.
| Route | Best for | Trade-off |
|---|---|---|
| Manual export (JSON/CSV/Excel) | One-off analysis | No automation |
| Platform integration nodes (n8n, Make, Zapier) | Scheduled workflows without code | Node capabilities lag the raw API |
REST API (run-sync-get-dataset-items) | Custom services, agents | You own retries and scheduling |
For scheduled team workflows, the integration nodes win on maintenance cost. Apify publishes a verified n8n community node (@apify/n8n-nodes-apify) that can start a run and return the dataset in one step, which is the whole extraction half of the pipeline.
How do you flatten rows for a spreadsheet?
Decide the column set before you write the first row, and never let it drift. Actor datasets are nested — location objects, arrays of platforms, social link maps. A spreadsheet needs flat, stable columns.
A workable transform for job data, for example, projects to ten columns: found date, title, company, location, remote flag, job type, salary string, platform, posted date, URL. Everything else stays in the raw dataset for later.
Two rules keep the sheet usable:
- One row per entity, one column per fact. Do not stuff a JSON blob into a cell; you will never filter on it.
- Always carry a collection timestamp and a source URL. Without the first you cannot build history; without the second you cannot verify a claim.
What makes a chat digest worth reading?
Digest, do not firehose. One message per run, containing the top items, beats one message per item. The pattern that survives contact with a busy channel:
- A count in the first line — “7 new listings today” — so the message is skimmable without opening it.
- Five to fifteen items maximum, each one line: what it is, who it is from, and a link.
- The link goes to the source, not to your dashboard. People act from where they read.
- An explicit empty state — “no new matches today” — because silence is ambiguous. A quiet channel might mean nothing happened, or might mean the pipeline broke three days ago.
The last point is the one teams learn the hard way. A pipeline that only speaks when it has news is indistinguishable from a dead pipeline.
How do you stop repeated alerts?
Diff against what you have already delivered, using a stable key. The naive version — alerting on everything the run returned — floods the channel on day two because most sources return the same records repeatedly.
Choose the key by entity:
| Data type | Stable key | Not a key |
|---|---|---|
| Job listings | platform + source URL, or platform job id | title + company (the same role reposts) |
| Products | product id or platform + SKU | title (sellers rewrite titles) |
| Places | place id | name + address string |
| Messages/posts | platform message id | text hash (edits break it) |
Store delivered keys in the archive sheet, then filter the current run against them. This is also why the archive comes first in the fork: it doubles as the deduplication memory.
What should be scheduled, and how often?
Match the schedule to how fast the source actually changes, not to how often you want news. Job boards and marketplaces update meaningfully on a daily cadence; social trends move hourly; property listings move weekly in most markets. Running a daily collector every hour multiplies cost and produces no additional signal.
For a jobs pipeline built on All Jobs Scraper, a morning run is enough — the Houston nursing listings example task shows the kind of narrow, scheduled query that produces a readable daily digest rather than a wall of noise.
What breaks, and how do you notice?
Three failure modes account for nearly all silent pipeline death:
- Credential expiry — an OAuth token to the spreadsheet or chat tool lapses. The Actor still runs; delivery stops.
- Schema drift — a field is renamed upstream and the flatten step starts writing empty columns. The sheet grows, the data is blank.
- Zero-result runs — the source returns nothing for a transient reason and the digest reports “no new items,” indefinitely.
The cheap defence is a heartbeat: have the pipeline post its empty state, and check the archive’s row count weekly. If the count stops moving while the schedule keeps firing, something upstream is broken. For deeper checks — field coverage, duplicate rates, freshness distribution — see How to Validate Scraped Data Quality.
For the API-first version of this pipeline, including run states and retry semantics, read Using the Apify Actor API in Production.
Continue in the directory
Turn the guide into a real sample run.
Open the current AgentX contract, check pricing and fields, then validate a narrow output.