seo: how to avoid indexing the URLs requested by AJAX without also blocking the page from which the call is made?


We use AJAX to include dynamic context-based content in a normal HTML page. The answer to this HTML call is an HTML segment without HTML headers.

The problem is that Google is indexing these pages.

We try to use robots.txt to avoid indexing, but Google complains in (GSC) because it is actually some content (although not too interesting), and still indexes and provides these URLs in the results.

We tried to add an HTTP response header "noindex" to the AJAX page, but that led to the de-indexing of the main HTML page, since it inherited the X-Robots-HTTP tag.

So what is the right way to do this?