Provided by DataForSEO
Find non-indexable pages in a website crawl with DataForSEO
Crawled pages that search engines cannot index, with the reason (robots.txt, meta tag, HTTP header, attribute or redirects). Reads one page of results of a finished OnPage crawl Run by its task id (free within 30 days of the crawl); the crawl is never repeated here.
How to run it
Step 1Set up
Once: paste this into your agent's chat. The Skill installs the CLI and logs in by itself. For an agent that cannot use a shell, connect it over MCP instead.
set up https://glasser.ai/SKILL.md
Show me how to use the DataForSEO endpoint /v3/on_page/non_indexable on Glasser.glasser run -p dataforseo -e '/v3/on_page/non_indexable' -i '{"id":"09110940-2417-0216-0000-5231f6fc6986"}' --endpoint-version 3Uses endpoint version 3. Inspect the current price before running. Install and log in first on Get Started.
Once per Run:
IDEMPOTENCY_KEY=$(uuidgen)Run it, and copy this same block to send it again:
curl -X POST https://api.glasser.ai/v1/runs \
-H "Authorization: Bearer gl_..." \
-H "Idempotency-Key: $IDEMPOTENCY_KEY" \
-H "Content-Type: application/json" \
-d '{"provider":"dataforseo","endpoint":"/v3/on_page/non_indexable","input":{"id":"09110940-2417-0216-0000-5231f6fc6986"},"endpoint_version":3}'Reuse the same Idempotency-Key to send it again: the second send is a read of the first Run. A new key is a new Run and a new Charge. Uses endpoint version 3; inspect the current price before running.
Step 2Try a specific example task
When you need to investigate a website problem, you might review crawl findings that relate to the issue you are trying to understand.
Copy this prompt into your agent to review crawl findings for your website issue.
Use dataforseo:/v3/on_page/non_indexable on Glasser.
Find returned crawl findings relevant to the website problem I describe.Endpoint structure
{
"type": "object",
"$schema": "http://json-schema.org/draft-07/schema#",
"required": [
"id"
],
"properties": {
"id": {
"type": "string",
"pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"
},
"limit": {
"type": "integer",
"maximum": 1000,
"minimum": 1
},
"offset": {
"type": "integer",
"maximum": 2000000,
"minimum": 0
},
"filters": {
"type": "array",
"description": "DataForSEO filter expression, passed through unchanged: one condition [field, operator, value], or conditions joined by \"and\" / \"or\" ([[a], \"and\", [b]]). Field names come from this endpoint's result items; the number of conditions allowed is in the endpoint's docUrl."
}
},
"additionalProperties": false
}{
"id": "09110940-2417-0216-0000-5231f6fc6986"
}This Endpoint publishes no output schema. You get the Provider's own response body, unchanged — we do not warrant a shape we cannot guarantee.
How pricing works
| Data returned or no match | The same price with or without matching data. | $0.00 |
|---|---|---|
| Provider error | The Provider failed to answer. | $0.00 |
| Timed out | The Run exceeded its time limit. | $0.00 |
| Internal failure | An internal error prevented completion. | $0.00 |