Migrating millions of data into Builder.IO

Technical Questions
1

Hi team,

I am investing on how to migrate our Old data to the builder.IO.

So far what I have learned is, to migrate the data, first I need to create data models in builder.IO and then use Builder.IO write API’s to post data of any data model.

But my concern is how can i migrate our millions of data into Builder.IO, for example we have millions of Blogs in our database that we want to migrate into Builder.IO.

Following are my questions:

  1. The Write API eg: “https://builder.io/api/v1/write/blogs” can take only 1 record at a time, does that mean if i have to migrate my 1 milion of blog post to builder.IO then I have to hit Builder.IO Write API 1 million times?

  2. Can Write API accept multiple records at a time?

  3. Do you have any other tool which migrate our Old data to Builder.IO effectively.

  4. Does Write API counts in the billing?

  5. Can you provide or develop the new API which can accept multiple records at a time for us to migrate the data if we are in the enterprise plan?

Thanks

Sushil Suthar

2

Hi @sushil.suthar,

  1. Yes, that’s right, If you have 1 million blog posts to migrate to builder.io then you will have to hit Write API 1 million times

  2. Currently, write API can’t accept multiple records.

  3. We do have an Admin CLI / SDK which customers can / have used before (builder/packages/admin-sdk at main · BuilderIO/builder · GitHub)

  4. Write API requests that do not account for billing.

  5. We might be able to, but we can’t promise, in past, we have always recommended just writing a script using the Write API to migrate large sets of data, but of course, you can also write a middleware on your own to balance all those requests.

Let us know if you have any further questions. Thank you!

3

Hi Manish,

Thanks for your response,

1 million hit, That’s a bad news for us, because this is very common requirement which builder.io doesn’t have, inserting multiple records in one API call.

If i take 1 second for 1 blog post to insert in builder.io for overall process, fetching from our database then making builder.IO write API call, overall if it takes 1 second then 1 million records will take around 11 days to complete, and that will be a very bad practice, even 0.5 seconds will also take 5 days.

Do you have any other approach to reduce the time?

4

Hi @sushil.suthar,

Currently, we don’t have any other approach but feel free to submit a feature request at Builder.io Ideas