Random JSON errors from Gatsby build process suggesting failed API call

Builder public api key
77d1fa194e53407a809b1a8f4c82ac81 and a81f6b87d78b4a96ba0a5423c40bcab5

What are you trying to accomplish
Have Gatsby pull in data to be used as part of the build

Screenshots or video link


Code stack you are integrating Builder with
Gatsby

Reproducible code example
This occurs randomly. I can run the same build again with no code changes and its successful. I can also run the build and handful of times and have it fail on different queries on different pages. I have 2 projects running and both have the same random errors occur, though frequent enough to happen a couple of times a day.

Heres another thats the same but looks to have returned a different error in the background:
image

One more for good measure. This one actually states that its a connection issue:

Thanks for the detailed report @Ash - we are digging in and will report back when we have some findings!

1 Like

Ok! We believe we have found and squashed the issue you were running into @Ash - can you confirm if you ever hit this error again?

That’s amazing news Steve.
I’ll run a number of builds todays and let you know of anything that pops up.

@steve Looks to still be occurring:
image
image
image

Hi Ash,

As Steve pointed out we feel reasonably certain we have squashed the Unexpected token < in JSON at position 0 error. But you seem to be experiencing I and E at position 0 as well.

Few questions:

  1. Because the previous error was location-specific, could you provide us with your approximate location, so we can see which data center your request is going to?
  2. How persistent is this. Does this fail consistently or just really.
  3. (I don’t know how technical you are) Would it be possible for you to somehow capture the response and send it to us?

Also, any way you can think of we could reproduce this problem?

@Misko Its unfortunate that I didn’t have the other errors at the time of the initial ticket creation, however, if you have a closer look at my second comment on this thread, you can see there that the Unexpected token I had been flagged there, so I had highlighted that there are many different error responses returning from your system. I assure you too that the Unexpected token E has occurred many times in the past also. Which error it is that comes up is random so it was luck of the draw for what screenshots I could provide you really.

The errors in my latest comment had come from a build using the first API key that I provided above if that helps determine which data center that particular account is connected to?

As mentioned, its persistence is pretty random. For example, the second API key had a number of builds fired between the time you flagged a fix was in place and now, and we have only seen one failed build out of about 9 builds. The error that occurred in that build was:

The error that occurred about 12 hours ago using the first API key is actually the error you claim to have repaired:

As for capturing what the actual response is behind it, I would LOVE to as I’d like to see what it is for myself to know if there’s anything on our side we can do to prevent it, however, after much research, I don’t believe Gatsby logs the errors coming from GraphQL, or even what occurs during the node creation of the data. We are using your gatsby package @builder.io/gatsby to create those nodes to use in these queries if that helps any? If you have any ideas on how and where we can somehow log whats happening behind this, I’m all ears - I’ve not found any way of doing so though.

Probably goes without saying, this is a pretty annoying problem. It is wasting clients build minutes, is unreliable in knowing whether changes can be pushed live in a timely manner, and the worst part is that its unpredictable - as mentioned, we can redeploy after an error and it works, or we have to redeploy 5-6 times before it will successfully go through (which is a strong indication that its not the code). Not ideal when clients have time-sensitive builds that need to happen and have to wait half a day sometimes before we get a successful build through. I’d find it hard to believe that we’re the only accounts affected by this also - if I’m honest.

@Misko @steve Are there any updates on this? We’re still experiencing these errors, albeit, in the last couple of days, I will say we’ve had more successful builds than failed ones (that’s inclusive of the weekend though).

Hi Ash,

So in order for us to figure out what is going on, we need to see the communication between your applications and our servers. I wrote up steps on how to do this here: GitHub - BuilderIO/http-debug-proxy: This project contains a set up to create a HTTP proxy for the https://cdn.builder.io/ endpoint. Please collect them and send me the resulting logs.

This is the first time we are trying to do this, so if you run into trouble please document it and let us know so that we can improve this for others.

As soon as you send us the logs, we should have a better idea of what is going on and how we can resolve this.

Thanks @Misko - Not sure if there might be something wrong with it as my local test build errors within 10 seconds.
Heres a ZIP of the folder it created: Download - TransferNow

To confirm - if I remove the proxy, the error doesn’t seem to occur and the build continues as normal.

Pulling the HTML out of the error and rendering it, I get:


Not sure if that’s an error from the error as that message isn’t actually present in the raw markup, but thought I’d highlight.

Turns out I had a mistake in my instructions. include graphql endpoint in the url by teleaziz · Pull Request #1 · BuilderIO/http-debug-proxy · GitHub The URL needs to contain /api/v1/graphql. Could you give it a try again.

Thanks @Misko - that looks to have done the trick… at least from allowing the build to continue.

Unfortunately, it doesn’t look like its capturing the error though. I have created another ZIP as instructed: Download - TransferNow however I’ve eyeballed the JSONs created and the error (screenshot enclosed in the ZIP also) or the data pertaining to the error doesn’t seem to exist in any of the files, not that I could see anyway.

Hi Ash,

Yes, you are right. There is nothing out of ordinary in the logs. As far as we can tell the back-end service is working as intended. At this point, we need to be able to reproduce the issue locally to be able to help you further. Could you try to create a minimal reproduction for us?

– Misko

@Misko at this point, I’m prepared to just give you read access to the repo so you can see this for yourself.

Any way we can take this conversation private to be able to set this up?

@Ash I have replied to you on your email on file associated with the organization. Let me know if you did not get it.

Thanks @Misko - will continue this convo via email.

@Ash We just pushed a fix to our infrastructure that should address the issue you were facing. Please give it a try and let us know if you are still having problems building your site with Gatsby.

Thanks @sami - I’ll keep an eye on it today and report back. Was access to our repo helpful in tracking down what the issue was?