An experiment: what do `includeRefs` and `noTraverse` really do?

Technical Questions
1

Overview

Symbols and references: two of Builder’s more mysterious features.

I’ve been trying to get to the bottom of what has appeared to me like inconsistent results when using symbols with reference fields for my clients’ projects in Builder. Furthermore, because each feature touches on some of Builder’s more advanced capabilities, testing the two in combination is a good proxy for understanding Builder’s rendering model as a whole.

To better serve my clients and to understand how Builder renders symbols and references, I set up an experiment.

Methodology

I created a Next.js project that renders a Builder page containing a symbol with a reference field on it. The Next.js page fetches the Builder page’s content and renders it during SSR with <BuilderComponent>.

Additionally, the Next.js page accepts four query parameters, which affect rendering in the following ways:

  • includeRefs=true|false: sets the content API request’s includeRefs option to true or false (defaults to false)
  • noTraverse=true|false: sets the content API request’s noTraverse option to true or false (defaults to true)
  • builderComponentIncludeRefs=true|false: sets <BuilderComponent>'s includeRefs option to true or false (defaults to false)
  • builderComponentNoTraverse=true|false: sets <BuilderComponent>'s noTraverse option to true or false (defaults to true)

These four parameters were chosen because of their importance in affecting the outcome of how a symbol with a reference field would render, according to the documentation and various forum posts.

The most important use case for a symbol with a reference input field is to use the content of the item pointed at by the reference from within the symbol. Since content inputs are consumed within a symbol by accessing properties of state, I decided to measure the effect of these four parameters on the final value of the symbol’s state.projectRef property after rendering (projectRef was the name of the symbol’s reference field). I would observe this value by binding a text block in the symbol to state.projectRef.

The objective was to try all 16 possible combinations of the query parameters, record how each combination affected the value of state.projectRef in the symbol, and compare the values to determine which parameters had an effect.

Code

Methodology

I manually visited the test page in my browser at http://localhost:3000/one-project, passing the values for all 16 combinations of the four test parameters into the query string of the URL.

I copied and pasted the “Project ref (state.projectRef): …” text rendered in the symbol for each combination, formatting the raw results with an online JSON formatting tool and diffing them with an online JSON diffing tool.

I came to the conclusions below by manually looking through the results of the diffs. The formatted JSON output from each combination is listed below under Raw data.

Conclusions

  • No content was for the symbol’s reference was inlined unless the containing page’s content API request had noTraverse: false. When the content API request had noTraverse: false set, the entire reference field for the symbol was inlined.
  • Setting includeRefs or noTraverse on <BuilderComponent> had no effect.
  • The results from setting includeRefs: true, noTraverse: false vs. includeRefs: false, noTraverse: false on the Content API request were identical, except for the rev (revision) parameter and the order of the properties.
    • The impression that different inlined responses varied materially in their content appears to be due to the shifting order of properties in the returned JSON object.
  • Therefore, it appears that includeRefs has no effect and only setting noTraverse on the content API request has any effect on whether the content for a symbol’s reference field is inlined and passed to that symbol’s state.
  • No hydration errors were observed during the test.

Caveats

  • This experiment only considered the results of rendering a symbol on a page from the perspective of that symbol’s state. includeRefs may have some effect on the raw response of the content API request from the page. It’s possible that the symbol itself transforms the content API response received by the page, negating potential differences in the initial payloads when using includeRefs: true vs. includeRefs: false.

Further questions

  • Does includeRefs have an effect on the raw content API response that’s not seen in the symbol’s final state? If so, how does the symbol transform the response in such a way that includeRefs doesn’t have an effect on the final result?
  • Is the content for the symbol’s reference field inlined within the content API response for the page, or is it inlined later—for example, by the symbol?
  • What happens if we rerun this experiment with a symbol containing another symbol?
  • What happens if we rerun this experiment with a symbol containing a slot containing another symbol?
  • What happens if we rerun this experiment using <BuilderContent> instead of <BuilderComponent>, comparing the results of the content parameter of the render function?

Table

Legend

x: fully-inlined symbol content
o: no inlined content

Content API includeRefs: true, noTraverse: true Content API includeRefs: true, noTraverse: false Content API includeRefs: false, noTraverse: true Content API includeRefs: false, noTraverse: false
<BuilderComponent> includeRefs: true, noTraverse: true o x o x
<BuilderComponent> includeRefs: true, noTraverse: false o x o x
<BuilderComponent> includeRefs: false, noTraverse: true o x o x
<BuilderComponent> includeRefs: false, noTraverse: false o x o x

Raw data

Click here

http://localhost:3000/one-project?includeRefs=false&noTraverse=false&builderComponentIncludeRefs=false&builderComponentNoTraverse=false

Content API

  • includeRefs: false
  • noTraverse: false

<BuilderComponent>

  • includeRefs: false
  • noTraverse: false 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project",
  "value": {
    "data": {
      "url": "/projects/example-project-1",
      "title": "Example project 1",
      "image": "https://cdn.builder.io/api/v1/image/assets%2F825df0b059fe4a2587c53008d27a7ef9%2Fe672345ab06d448db71fcc111ca4fd62",
      "description": "The first example project."
    },
    "name": "Example project 1",
    "modelId": "bb05894f7ec397f96c4582ee9743040c351408b7da97ee6fd2151eb33d431d63",
    "@originContentId": "b1ebe36a28d0493182bb06e0f7c1775c",
    "@originOrg": "825df0b059fe4a2587c53008d27a7ef9",
    "meta": {
      "lastPreviewUrl": "",
      "kind": "data"
    },
    "rev": "auqykk0flf",
    "lastUpdated": 1664820885747,
    "testRatio": 1,
    "@originModelId": "b594225b65bd44bf887fec35a56754f5",
    "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
    "lastUpdatedBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "published": "published",
    "firstPublished": 1664820701357,
    "createdDate": 1664820643177,
    "createdBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "variations": {},
    "query": [
      {
        "value": "/projects/example-project-1",
        "property": "urlPath",
        "operator": "is",
        "@type": "@builder.io/core:Query"
      }
    ]
  }
}

http://localhost:3000/one-project?includeRefs=false&noTraverse=false&builderComponentIncludeRefs=false&builderComponentNoTraverse=true

Content API

  • includeRefs: false
  • noTraverse: false

<BuilderComponent>

  • includeRefs: false
  • noTraverse: true 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project",
  "value": {
    "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
    "variations": {},
    "data": {
      "url": "/projects/example-project-1",
      "title": "Example project 1",
      "image": "https://cdn.builder.io/api/v1/image/assets%2F825df0b059fe4a2587c53008d27a7ef9%2Fe672345ab06d448db71fcc111ca4fd62",
      "description": "The first example project."
    },
    "lastUpdated": 1664820885747,
    "@originOrg": "825df0b059fe4a2587c53008d27a7ef9",
    "rev": "syb3rds8vbn",
    "testRatio": 1,
    "name": "Example project 1",
    "firstPublished": 1664820701357,
    "published": "published",
    "lastUpdatedBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "modelId": "bb05894f7ec397f96c4582ee9743040c351408b7da97ee6fd2151eb33d431d63",
    "@originModelId": "b594225b65bd44bf887fec35a56754f5",
    "@originContentId": "b1ebe36a28d0493182bb06e0f7c1775c",
    "createdDate": 1664820643177,
    "query": [
      {
        "operator": "is",
        "value": "/projects/example-project-1",
        "@type": "@builder.io/core:Query",
        "property": "urlPath"
      }
    ],
    "meta": {
      "kind": "data",
      "lastPreviewUrl": ""
    },
    "createdBy": "7CR6nekYycX9cktjX9sWyiP655l2"
  }
}

http://localhost:3000/one-project?builderComponentIncludeRefs=true&builderComponentNoTraverse=false&noTraverse=false

Content API

  • includeRefs: false
  • noTraverse: false

<BuilderComponent>

  • includeRefs: true
  • noTraverse: false 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project",
  "value": {
    "data": {
      "url": "/projects/example-project-1",
      "title": "Example project 1",
      "image": "https://cdn.builder.io/api/v1/image/assets%2F825df0b059fe4a2587c53008d27a7ef9%2Fe672345ab06d448db71fcc111ca4fd62",
      "description": "The first example project."
    },
    "name": "Example project 1",
    "modelId": "bb05894f7ec397f96c4582ee9743040c351408b7da97ee6fd2151eb33d431d63",
    "@originContentId": "b1ebe36a28d0493182bb06e0f7c1775c",
    "@originOrg": "825df0b059fe4a2587c53008d27a7ef9",
    "meta": {
      "lastPreviewUrl": "",
      "kind": "data"
    },
    "rev": "auqykk0flf",
    "lastUpdated": 1664820885747,
    "testRatio": 1,
    "@originModelId": "b594225b65bd44bf887fec35a56754f5",
    "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
    "lastUpdatedBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "published": "published",
    "firstPublished": 1664820701357,
    "createdDate": 1664820643177,
    "createdBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "variations": {},
    "query": [
      {
        "value": "/projects/example-project-1",
        "property": "urlPath",
        "operator": "is",
        "@type": "@builder.io/core:Query"
      }
    ]
  }
}

http://localhost:3000/one-project?builderComponentIncludeRefs=true&noTraverse=false

Content API

  • includeRefs: false
  • noTraverse: false

<BuilderComponent>

  • includeRefs: true
  • noTraverse: true 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project",
  "value": {
    "meta": {
      "kind": "data",
      "lastPreviewUrl": ""
    },
    "testRatio": 1,
    "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
    "data": {
      "url": "/projects/example-project-1",
      "title": "Example project 1",
      "description": "The first example project.",
      "image": "https://cdn.builder.io/api/v1/image/assets%2F825df0b059fe4a2587c53008d27a7ef9%2Fe672345ab06d448db71fcc111ca4fd62"
    },
    "lastUpdatedBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "@originContentId": "b1ebe36a28d0493182bb06e0f7c1775c",
    "createdBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "createdDate": 1664820643177,
    "@originModelId": "b594225b65bd44bf887fec35a56754f5",
    "variations": {},
    "rev": "0483p1ekkgum",
    "firstPublished": 1664820701357,
    "name": "Example project 1",
    "published": "published",
    "@originOrg": "825df0b059fe4a2587c53008d27a7ef9",
    "query": [
      {
        "property": "urlPath",
        "@type": "@builder.io/core:Query",
        "value": "/projects/example-project-1",
        "operator": "is"
      }
    ],
    "lastUpdated": 1664820885747,
    "modelId": "bb05894f7ec397f96c4582ee9743040c351408b7da97ee6fd2151eb33d431d63"
  }
}

http://localhost:3000/one-project?includeRefs=false&noTraverse=true&builderComponentIncludeRefs=false&builderComponentNoTraverse=false

Content API

  • includeRefs: false
  • noTraverse: true

<BuilderComponent>

  • includeRefs: false
  • noTraverse: false 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project"
}

http://localhost:3000/one-project?includeRefs=false&noTraverse=true&builderComponentIncludeRefs=false&builderComponentNoTraverse=true

Content API

  • includeRefs: false
  • noTraverse: true

<BuilderComponent>

  • includeRefs: false
  • noTraverse: true 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project"
}

http://localhost:3000/one-project?includeRefs=false&noTraverse=true&builderComponentIncludeRefs=true&builderComponentNoTraverse=false

Content API

  • includeRefs: false
  • noTraverse: true

<BuilderComponent>

  • includeRefs: true
  • noTraverse: false 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project"
}

http://localhost:3000/one-project?includeRefs=false&noTraverse=true&builderComponentIncludeRefs=true&builderComponentNoTraverse=true

Content API

  • includeRefs: false
  • noTraverse: true

<BuilderComponent>

  • includeRefs: true
  • noTraverse: true 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project"
}

http://localhost:3000/one-project?includeRefs=true&noTraverse=false&builderComponentIncludeRefs=false&builderComponentNoTraverse=false

Content API

  • includeRefs: true
  • noTraverse: false

<BuilderComponent>

  • includeRefs: false
  • noTraverse: false 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project",
  "value": {
    "@originModelId": "b594225b65bd44bf887fec35a56754f5",
    "createdBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "firstPublished": 1664820701357,
    "meta": {
      "kind": "data",
      "lastPreviewUrl": ""
    },
    "variations": {},
    "modelId": "bb05894f7ec397f96c4582ee9743040c351408b7da97ee6fd2151eb33d431d63",
    "query": [
      {
        "@type": "@builder.io/core:Query",
        "operator": "is",
        "value": "/projects/example-project-1",
        "property": "urlPath"
      }
    ],
    "name": "Example project 1",
    "lastUpdated": 1664820885747,
    "lastUpdatedBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
    "testRatio": 1,
    "data": {
      "image": "https://cdn.builder.io/api/v1/image/assets%2F825df0b059fe4a2587c53008d27a7ef9%2Fe672345ab06d448db71fcc111ca4fd62",
      "title": "Example project 1",
      "description": "The first example project.",
      "url": "/projects/example-project-1"
    },
    "rev": "wg1fnanwtx",
    "createdDate": 1664820643177,
    "@originContentId": "b1ebe36a28d0493182bb06e0f7c1775c",
    "published": "published",
    "@originOrg": "825df0b059fe4a2587c53008d27a7ef9"
  }
}

http://localhost:3000/one-project?includeRefs=true&noTraverse=false&builderComponentIncludeRefs=false&builderComponentNoTraverse=true

Content API

  • includeRefs: true
  • noTraverse: false

<BuilderComponent>

  • includeRefs: false
  • noTraverse: true 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project",
  "value": {
    "name": "Example project 1",
    "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
    "published": "published",
    "data": {
      "url": "/projects/example-project-1",
      "image": "https://cdn.builder.io/api/v1/image/assets%2F825df0b059fe4a2587c53008d27a7ef9%2Fe672345ab06d448db71fcc111ca4fd62",
      "description": "The first example project.",
      "title": "Example project 1"
    },
    "variations": {},
    "meta": {
      "kind": "data",
      "lastPreviewUrl": ""
    },
    "query": [
      {
        "operator": "is",
        "value": "/projects/example-project-1",
        "property": "urlPath",
        "@type": "@builder.io/core:Query"
      }
    ],
    "modelId": "bb05894f7ec397f96c4582ee9743040c351408b7da97ee6fd2151eb33d431d63",
    "lastUpdatedBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "createdDate": 1664820643177,
    "@originContentId": "b1ebe36a28d0493182bb06e0f7c1775c",
    "rev": "zxpi0ibz75m",
    "firstPublished": 1664820701357,
    "@originOrg": "825df0b059fe4a2587c53008d27a7ef9",
    "createdBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "lastUpdated": 1664820885747,
    "testRatio": 1,
    "@originModelId": "b594225b65bd44bf887fec35a56754f5"
  }
}

http://localhost:3000/one-project?includeRefs=true&noTraverse=false&builderComponentIncludeRefs=true&builderComponentNoTraverse=false

Content API

  • includeRefs: true
  • noTraverse: false

<BuilderComponent>

  • includeRefs: true
  • noTraverse: false 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project",
  "value": {
    "firstPublished": 1664820701357,
    "query": [
      {
        "property": "urlPath",
        "@type": "@builder.io/core:Query",
        "operator": "is",
        "value": "/projects/example-project-1"
      }
    ],
    "data": {
      "url": "/projects/example-project-1",
      "title": "Example project 1",
      "description": "The first example project.",
      "image": "https://cdn.builder.io/api/v1/image/assets%2F825df0b059fe4a2587c53008d27a7ef9%2Fe672345ab06d448db71fcc111ca4fd62"
    },
    "@originModelId": "b594225b65bd44bf887fec35a56754f5",
    "name": "Example project 1",
    "@originOrg": "825df0b059fe4a2587c53008d27a7ef9",
    "variations": {},
    "lastUpdatedBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "modelId": "bb05894f7ec397f96c4582ee9743040c351408b7da97ee6fd2151eb33d431d63",
    "published": "published",
    "lastUpdated": 1664820885747,
    "@originContentId": "b1ebe36a28d0493182bb06e0f7c1775c",
    "testRatio": 1,
    "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
    "createdBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "createdDate": 1664820643177,
    "rev": "b0mqcbannv",
    "meta": {
      "lastPreviewUrl": "",
      "kind": "data"
    }
  }
}

http://localhost:3000/one-project?includeRefs=true&noTraverse=false&builderComponentIncludeRefs=true&builderComponentNoTraverse=true

Content API

  • includeRefs: true
  • noTraverse: false

<BuilderComponent>

  • includeRefs: true
  • noTraverse: true 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project",
  "value": {
    "createdDate": 1664820643177,
    "@originContentId": "b1ebe36a28d0493182bb06e0f7c1775c",
    "testRatio": 1,
    "lastUpdated": 1664820885747,
    "lastUpdatedBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "published": "published",
    "createdBy": "7CR6nekYycX9cktjX9sWyiP655l2",
    "rev": "ot0i7o45wr",
    "@originModelId": "b594225b65bd44bf887fec35a56754f5",
    "firstPublished": 1664820701357,
    "variations": {},
    "@originOrg": "825df0b059fe4a2587c53008d27a7ef9",
    "name": "Example project 1",
    "data": {
      "title": "Example project 1",
      "description": "The first example project.",
      "url": "/projects/example-project-1",
      "image": "https://cdn.builder.io/api/v1/image/assets%2F825df0b059fe4a2587c53008d27a7ef9%2Fe672345ab06d448db71fcc111ca4fd62"
    },
    "meta": {
      "lastPreviewUrl": "",
      "kind": "data"
    },
    "modelId": "bb05894f7ec397f96c4582ee9743040c351408b7da97ee6fd2151eb33d431d63",
    "query": [
      {
        "@type": "@builder.io/core:Query",
        "property": "urlPath",
        "operator": "is",
        "value": "/projects/example-project-1"
      }
    ],
    "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917"
  }
}

http://localhost:3000/one-project?includeRefs=true&noTraverse=true&builderComponentIncludeRefs=false&builderComponentNoTraverse=false

Content API

  • includeRefs: true
  • noTraverse: true

<BuilderComponent>

  • includeRefs: false
  • noTraverse: false 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project"
}

http://localhost:3000/one-project?includeRefs=true&noTraverse=true&builderComponentIncludeRefs=false&builderComponentNoTraverse=true

Content API

  • includeRefs: true
  • noTraverse: true

<BuilderComponent>

  • includeRefs: false
  • noTraverse: true 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project"
}

http://localhost:3000/one-project?includeRefs=true&noTraverse=true&builderComponentIncludeRefs=false&builderComponentNoTraverse=false

Content API

  • includeRefs: true
  • noTraverse: true

<BuilderComponent>

  • includeRefs: true
  • noTraverse: false 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project"
}

http://localhost:3000/one-project?includeRefs=true&noTraverse=true&builderComponentIncludeRefs=true&builderComponentNoTraverse=true

Content API

  • includeRefs: true
  • noTraverse: true

<BuilderComponent>

  • includeRefs: true
  • noTraverse: true 
{
  "@type": "@builder.io/core:Reference",
  "id": "22c38c12a88abe0d3e42888b19330658e2a9629ba16f8194e11391f6836f8917",
  "model": "project"
}
2

It would be great if someone from the Builder team could confirm my conclusions. It’s especially surprising that includeRefs appeared to have no effect on whether the symbol’s reference field was inlined.

If that’s the case, what does includeRefs: true actually do?

3

Getting a bit more nuance in real world cases.

I have a setup on a client’s website similar to the test case that I created, where the content API fetches the page’s content during SSR and it’s rendered into a <BuilderComponent> in Next.js. The page has a symbol with a reference field named project.

My testing shows that, as expected, including/excluding includeRefs from the content API request has no effect on the end result, and noTraverse: false must be included in the request.

However, includeRefs: false (or just simply excluding includeRefs, since it defaults to false) in <BuilderComponent options={...}> has an interesting result. I added console.log(state.project) into the symbol’s Builder.isBrowser block of its custom JS to demonstrate the effect of this option on the symbol’s state:

CleanShot 2022-10-04 at 10.43.33
CleanShot 2022-10-04 at 10.43.331617×356 203 KB

Notice how the project reference’s value property is inlined at first, then disappears. When I add includeRefs: true to <BuilderComponent options={...}>, the value property remains inlined:

CleanShot 2022-10-04 at 10.52.55
CleanShot 2022-10-04 at 10.52.551604×229 143 KB

Based on this observation, and a close reading of the docs, I hypothesize that:

  • noTraverse determines whether any symbols will be rendered server-side on Builder’s servers during a content API request and inlined into the response.
  • When Builder’s content API renders a symbol, it always evaluates any reference fields and inlines the data. There’s no option to turn this behavior off.
  • includeRefs determines whether any reference fields directly associated with the content item itself—not its children—will be evaluated. This behavior would affect, for instance, model fields and content inputs.
  • The page’s content API request includes the symbol’s inlined reference value because noTraverse: false triggers a content API render of the symbol and all its content inputs, including the reference field. At first, this data gets passed to the symbol and populates its state.
  • Later, the symbol itself initiates a second, client-side evaluation of its fields and inherits the includeRefs value set on <BuilderComponent>. When this value is true, it reevaluates the reference field, fetching the reference’s value and populating state. When false, it skips evaluating the reference and deletes the content API-provided value.

Some circumstantial evidence to support my hypothesis.

Below is a screenshot of the Network tab inside Chrome Dev Tools with <BuilderComponent options={{includeRefs: true}}>. Note that there are 12 requests to the content API for the project model (NOTE: sometimes I got 16 or even 22 requests. It’s not clear why so many client-side requests were issued):

CleanShot 2022-10-04 at 11.13.43
CleanShot 2022-10-04 at 11.13.431903×407 184 KB

Next is a screenshot of the Network tab with <BuilderComponent options={{includeRefs: false}}>. Note that there are now four requests:

CleanShot 2022-10-04 at 11.02.32
CleanShot 2022-10-04 at 11.02.321902×328 139 KB

I’m guessing that the discrepancy is due to the symbol not evaluating the reference client-side when includeRefs: false.

But why the inconsistent result between my use case here and the test case above? I’m not sure.

4

The more that I work with refs and symbols, the more that I feel that all ref and symbol content should be fetched by default. Evaluating refs and traversing through the content tree should be opt-out, not opt-in.

I can’t think of a situation in which I just wanted a raw ref or a raw symbol without the content data pointed to by that ref or symbol. I’m sure that such situations exist, but it shouldn’t be the default.

I’ve filed an idea to make this the default for v3 of the content API, if one should ever be released: If/when there's a v3 content API, make saner | Builder.io Ideas. Please vote if you agree!

5

howdy @ersin! so we do have some updates planned here, but a few notes that might help:

  • includeRefs=true in the API includes references (Rerence field type) values
  • options={{}} in BuilderComponent is only applied if the component itself does the fetching, not if you fetch with builder.get() and pass content as a prop. the preferred way is using builder.get(), we will be deprecating fetching within the react component (which include the options prop) over time
  • noTraverse=true tells the API to traverse the content to apply dynamic data like symbols, HTTP requests, custom server side js code, etc. at the API level it is off by default, but the SDKs send noTraverse=false aka augment the data
  • the reason why API calls don’t include references or traversal by default is it is computationally expensive, as well as problematic if you are doing reads and writes in the API (you don’t want to write back to us with refs/symbols/etc included). most use cases for using the content APIs directly is fetching large lists of content to either modify and write back, or to scan for basic thing like “give me all of our Builder page URLs” which don’t need any of these refs/traversal. the SDKs are most commonly used for fetching single entries at a time, and need all refs and traversal, so those default to sending noTraverse=false (and soon will default to includeRefs=true too)

over time, the v3 API aims to consolidate these params, likely there will just be one includeRefs param that will do everything (traverse and apply refs, symbols, other dynamic data). this will be off by default but for SDKs will pass this to be on by default

also slotted for the v3 content API is recursive refs as well (so includeRefs=true will include all refs not just the first layer)

6

Hi @steve! Thanks for these notes. A few points:

  • includeRefs=true in the API includes references (Rerence field type) values

As my testing above shows, it’s not so clear cut. includeRefs had no effect at all on resolving symbol input fields that were references. The only thing that mattered was noTraverse. I have noticed that includeRefs has an effect on top-level content item fields.

These kinds of discrepancies make the data flow confusing and give the impression that refs can’t be relied on. Knowing how Builder works, I think the issue is that the SDK and the content API logic aren’t always aligned on assumptions about user intent but the SDK depends on the content API. When there’s a misalignment, like you point out with the use case for includeRefs in the content API vs. the SDK, confusion ensues.

  • options={{}} in BuilderComponent is only applied if the component itself does the fetching, not if you fetch with builder.get() and pass content as a prop. the preferred way is using builder.get(), we will be deprecating fetching within the react component (which include the options prop) over time

I figured that something like this is true, thanks for confirming. I think what’s confusing here is that it’s not clear that <BuilderComponent> is something that would ever fetch content on its own, let alone under which circumstances. I’m still kinda fuzzy on this myself and will need to go back into the docs.

I’m glad to hear that it’s being deprecated. Sounds like the new approach is in line with the functional ergonomics of the v2 SDKs (fewer side effects === :+1:).

  • noTraverse=true tells the API to traverse the content to apply dynamic data like symbols, HTTP requests, custom server side js code, etc. at the API level it is off by default, but the SDKs send noTraverse=false aka augment the data

Did you mean noTraverse=false?

Are you saying that noTraverse has implications beyond just symbols? If so, those should be documented. I’m not sure what the “etc.” in your response means precisely :slight_smile:.

  • the reason why API calls don’t include references or traversal by default is it is computationally expensive

I figured this was probably the reason, but in my experience there has never been a case where I don’t want all the refs for my content resolved. By not including the resolved refs by default, all that happens is that I have do the same work myself. In fact, it’s probably less efficient because now your servers have to process multiple network requests from my back end.

Once this PR is merged, your SDK will have a utility function to do exactly what I’m describing: https://github.com/BuilderIO/builder/pull/1238. So why not save the hassle and just turn it on by default?

(OK, having re-read your last bullet point, I get it. Point taken.)

Also, seems like this is an ideal use case for a revamped GraphQL API that could auto-resolve and only fetch whatever fields I tell it to.

as well as problematic if you are doing reads and writes in the API (you don’t want to write back to us with refs/symbols/etc included)

Isn’t the write API separate? The changes I’m advocating for would only affect the read API.

Also, why not let the user write back to you with refs/symbols included? I was basically trying to do exactly that here, only manually inside the Visual Editor’s JSON view: Reference field on a "repeat for each" symbol

most use cases for using the content APIs directly is fetching large lists of content to either modify and write back, or to scan for basic thing like “give me all of our Builder page URLs” which don’t need any of these refs/traversal. the SDKs are most commonly used for fetching single entries at a time, and need all refs and traversal, so those default to sending noTraverse=false (and soon will default to includeRefs=true too)

OK! This clarifies many things.

I may be wrong, but I don’t think it’s documented that the SDK turns on certain options by default, like noTraverse=false. I myself didn’t know that until now.

One of the annoying things about the current SDK docs is that you have to kinda guess the correct options by cross-referencing the content API doc section on query params with whatever IntelliSense pops up when typing builder.get(. Especially annoying that some things are at the top-level of the builder.get options object (e.g., { fields: 'data.url,data.title' }), while other things are implicitly nested within the undocumented options property (e.g., {options: { noTraverse: false }}). Or that something like fields expects a comma-separated list of params vs. something like sort that expects nested objects.

To get back to your original point, now that I think about it, probably the most common use case for content API calls is something like getStaticPaths in Next.js where you just want a list of all your content item URLs, which you’re right, has nothing to do with refs. The division of labor you outlined makes perfect sense: SDKs with resolved content, naked API requests without.

over time, the v3 API aims to consolidate these params, likely there will just be one includeRefs param that will do everything (traverse and apply refs, symbols, other dynamic data). this will be off by default but for SDKs will pass this to be on by default

Yes Yes Yes

also slotted for the v3 content API is recursive refs as well (so includeRefs=true will include all refs not just the first layer)

Ship It Ship It Ship It

7

ah! symbol input refs not resolving is not the intended behavior, will get a ticket in to fix that

and yeah when we consolidate that API transformations we can better document that, what the param is and what exact impact it has

re: write API, what I mean is a common use of using the raw content API (outside of SDK usage) is to fetch a lit of items, transform them, and write them back w/ the write API. when you write back to our API you will not want to send resolved symbols/references/etc but rather the original raw form

1 Like
8

ah! symbol input refs not resolving is not the intended behavior, will get a ticket in to fix that

The weird thing is that noTraverse: false was enough to get those symbol inputs to resolve. There was no situation in which I was able to resolve symbol content but not resolve reference field inputs on that content. It was all-or-nothing: no symbols whatsoever, or symbols with all their fields, including refs, resolved.

My expectation based on the docs is that noTraverse: false by itself would only resolve the symbols but leave refs on those symbols unresolved. The end result happened to be what I wanted and is basically the behavior you’re proposing for includeRefs in the v2 SDKs, but given the current docs, it’s inconsistent and is part of why using refs feels unpredictable.

Btw the doc in question: Content API - Builder.io

From the above:

[noTraverse:] Default is true, pass false to include Symbol json in the response, which is helpful if you want to render your page all at once like the case in Server-side rendering

No mention of any of the non-symbol stuff that you listed in your original comment:

  • noTraverse=true tells the API to traverse the content to apply dynamic data like symbols, HTTP requests, custom server side js code, etc. at the API level it is off by default, but the SDKs send noTraverse=false aka augment the data
9

Again @steve, I think you meant to say “noTraverse=true tells the API to not traverse the content…”