How to build up the cache?

Technical Questions
1

We have a register of companies with around 250k entries.
I did read within the documentation, that we can set the cacheSeconds when fetching the data.

However: How would we build up the cache in the first place?

The only solution that comes to my mind is to build a crawler or rsync script that requests every entry, so the cache is build up.

Is this the only way / the way how to build up the cache?

2

Hello @christopherdosin,

To build up a cache for a large set of entries in Builder.io, like your register of companies with around 250,000 entries, there are several strategies you can consider beyond just crawling or using rsync.

  1. Use Pagination with API Calls: You can fetch entries in batches using the limit and offset attributes provided by Builder.io. This allows you to paginate through your data without overwhelming the server. For example, you can set a limit of 100 entries per request and adjust the offset for each subsequent call.

  2. Set cacheSeconds Appropriately: By setting the cacheSeconds parameter, you define how long the content should remain in the cache. This is beneficial in scenarios where content does not change often. A higher value will lead to better cache management across multiple requests.

  3. Use CDN Caching: Leverage CDN caches using the staleCacheSeconds setting, which allows serving stale content while newer content is being fetched and cached in the background. This is useful for ensuring high availability and performance.

  4. Automated Caching Strategy: Implement scripts that periodically make requests to your data model, ensuring entries are fetched at regular intervals. This could be a Node.js script that iterates through each entry using the pagination technique mentioned.

  5. Consider Background Workers: Use background workers or cron jobs that request your Builder.io content periodically, spreading the load over time to avoid spikes and ensuring that updates are captured as they happen.

  6. Utilize Builder.io’s Edge Caching: Builder employs edge caching which can serve cached content from the nearest edge node, improving access speed for frequently accessed content.

Hope this helps! Let us know if you have any further questions.

Thanks,