Creating API for GPT Crawler

Technical Questions
1

Hello everyone I have a question. I am looking to re-create this an API call inside if my GPT but I am having a hard time. I have created a schema that seems to work, but I can never match the parameters for some reason. I have other GPT project where I can make a API call to my local server. Once the code is live i run lt --port 3000 or 5000 and generate a url to use in the shema. I never get an error about the API but the parmeters not being met. Wondering if I can get some help. Also happy to share the shema as well. Thanks.

{
“openapi”: “3.0.0”,
“info”: {
“title”: “Crawler API”,
“description”: “API to manage web crawling operations.”,
“version”: “1.0.0”
},
“servers”: [
{
“url”: “https://kind-facts-dress.loca.lt”,
“description”: “Production server”
}
],
“paths”: {
“/crawl”: {
“post”: {
“summary”: “Initiate a crawling operation”,
“description”: “Starts a crawler with provided configuration”,
“operationId”: “crawlOperation”,
“requestBody”: {
“required”: true,
“content”: {
“application/json”: {
“schema”: {
“type”: “object”,
“properties”: {
“url”: {
“type”: “string”,
“format”: “url”,
“description”: “Starting URL for the crawler”
},
“depth”: {
“type”: “integer”,
“description”: “Depth of the crawl”
},
“otherConfig”: {
“type”: “string”,
“description”: “Other configuration parameters”
}
},
“required”: [“url”]
}
}
}
},
“responses”: {
“200”: {
“description”: “Crawling initiated successfully”,
“content”: {
“application/json”: {
“schema”: {
“type”: “object”,
“properties”: {
“message”: {
“type”: “string”,
“example”: “Crawler initiated successfully”
}
}
}
}
}
},
“400”: {
“description”: “Invalid request parameters”
},
“500”: {
“description”: “Server error”
}
}
}
}
}
}

2

Hello @Chowderr,

Welcome to the builder.io forum post.

The schema you’ve provided defines a POST endpoint at /crawl which takes in a JSON request body with parameters url, depth, and otherConfig, and returns a success message upon successful initiation of crawling.

Though this issue is not something we can provide accurate support for, here are a few things to check:

  1. Endpoint URL: Make sure you’re sending the API request to the correct URL. In your schema, it’s defined as https://kind-facts-dress.loca.lt/crawl.
  2. Request Body: Ensure that your request body matches the schema defined in your OpenAPI specification. It should include the required parameter url and can optionally include depth and otherConfig.
  3. Request Headers: Check if any specific headers are required by your API. If not mentioned in your schema, you might not need any additional headers.
  4. HTTP Method: Since this endpoint is a POST request, make sure you’re using the correct HTTP method when making the API call.
  5. Response Handling: Ensure that you’re correctly handling the response from the API. If the request is successful, you should expect a status code of 200 and a JSON response with a message indicating success.

Hope this helps!

Thanks,

3

Do you work for builder with that response? Also I am not sure what you mean “by support”.

4

Hello @Chowderr,

I’m here to assist you with general troubleshooting steps. However, it’s important to note that while I can offer guidance, my expertise regarding GPT may be limited, so I might not be able to provide pinpoint accurate solutions in that area. Therefore, wait until someone from our community can help you with this.

5

I appreciate your help, I think I will be able to figure it out a bit later. I do have another question, I want to scrape a site but when I do I get no content or text. Trying to figure out how to solve this problem.