Scraped data is production-ready only when it passes checks tied to the consumer’s actual contract. Row count is not enough. A useful gate measures provenance, completeness, validity, uniqueness, freshness, schema stability, and a manually reviewed sample.
What is the minimum quality contract?
Write a contract for each dataset:
| Dimension | Example rule |
|---|---|
| Provenance | At least 99% of accepted rows have a source URL or stable source ID |
| Completeness | Required consumer fields meet field-specific thresholds |
| Validity | Dates, currencies, URLs, enums, and numeric ranges parse correctly |
| Uniqueness | Duplicate definition and acceptable rate are explicit |
| Freshness | Collection time and source time fall inside the required window |
| Consistency | Normalized units and entity definitions match the documented schema |
| Traceability | Every transformed row maps to raw input and Actor run |
Thresholds depend on the use case. A research sample and an automated customer-facing product should not share an unexplained standard.
Why must source evidence come first?
Without a URL, source ID, collected timestamp, and query/run context, a row cannot be verified. Provenance also distinguishes genuine source changes from parser or normalization defects.
Keep raw Actor output immutable. Store normalized and enriched records separately with transformation versions.
How should completeness be measured?
Calculate missingness per field, source, country, query, and run—not only across the entire table. Overall completeness can hide one broken source.
Classify missing states where possible:
- not present on source;
- not supported by this source;
- extraction failed;
- not requested;
- unavailable due to access state;
- unknown.
Do not replace unknown values with zero, false, or an inferred default merely to improve a completeness percentage.
Which validity checks catch real defects?
Examples:
- URL parses and expected host/source relationship is plausible;
- timestamps parse and are not impossibly far in the future;
- minimum value does not exceed maximum value;
- currency is recognized when a price exists;
- latitude and longitude fall inside valid ranges;
- mileage, area, and compensation have units;
- enum values belong to a versioned vocabulary;
- text fields are not repeated boilerplate or error pages.
Keep validation errors as data so operators can inspect patterns.
How should duplicates be evaluated?
Define duplicate at the right entity level. Two rows can be duplicate source listings, two observations of one listing, or two source representations of one real-world entity.
Measure exact duplicate rate separately from probable entity-cluster rate. Never discard probable duplicates before preserving the raw rows and matching confidence.
What is schema drift?
Schema drift includes fields appearing or disappearing, types changing, enum vocabularies expanding, nested structures moving, and the semantic meaning of a field changing.
Compare each run with a versioned expected schema and recent successful runs. Warn on additions as well as removals: a new field can break strict consumers or represent a useful capability that needs mapping.
Apify Actors expose input and output contracts that can evolve. The official input and output documentation explains the platform model; the specific Actor page remains authoritative.
Which anomalies should trigger review?
- row count collapses or spikes relative to comparable runs;
- one source disappears;
- one field becomes mostly empty;
- all timestamps become identical;
- text length distribution changes sharply;
- duplicates increase;
- country, currency, or category vocabulary changes;
- success status is returned with an empty dataset where data was expected.
Use comparable baselines. A seasonal query can legitimately change.
Why is manual review still required?
Automated checks prove consistency with rules, not truth. Review a stratified sample that includes every source, unusual values, missing fields, and probable duplicates. Open source URLs and compare visible content with the dataset.
Record reviewer decisions and feed recurring defects back into validators.
What should the final quality report contain?
Publish run ID, Actor/build, input version, row count, source distribution, field coverage, validity failures, duplicates, anomalies, sampled URLs, reviewer notes, and pass/warn/fail decision.
Apify datasets can be accessed through API clients, REST, Console, and exports. The dataset documentation provides the current storage and retrieval details.
Apply this gate to the workflows in the job data, social intelligence, video knowledge, and market signals hubs.
Continue in the directory
Turn the guide into a real sample run.
Open the current AgentX contract, check pricing and fields, then validate a narrow output.