Metadata files can be static or dynamic
Each metadata file convention can be defined using a static file (e.g. opengraph-image.jpg) or a dynamic variant that uses code to generate the file (e.g. opengraph-image.js).
Next.js · API reference · all subjects
file-conventions/metadata
23 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.
Next.js automatically serves metadata files with hashes and updates head elements
Once a metadata file is defined, Next.js will automatically serve the file with hashes in production for caching and update the relevant head elements with the correct metadata, such as the asset's URL, file type, and image size.
Configure proxy.ts matcher to exclude metadata files
When using proxy.ts alongside metadata files, configure the matcher to exclude the metadata files.
Metadata files are special Route Handlers cached by default
Special Route Handlers like sitemap.ts, opengraph-image.tsx, icon.tsx, and other metadata files are cached by default in Next.js.
robots.ts TypeScript example
Example of generating robots dynamically with TypeScript: import type { MetadataRoute } from 'next' export default function robots(): MetadataRoute.Robots { return { rules: { userAgent: '*', allow: '/', disallow: '/private/', }, sitemap: 'https://acme.com/sitemap.xml', } }
robots.js JavaScript example
Example of generating robots dynamically with JavaScript: export default function robots() { return { rules: { userAgent: '*', allow: '/', disallow: '/private/', }, sitemap: 'https://acme.com/sitemap.xml', } }
rules property accepts single rule or array of rules
The rules property in the Robots object can be a single rule object or an array of rule objects for customizing behavior per user agent.
userAgent as string or array in rules
The userAgent field in a rule can be a string for a single bot or a string array to apply the same rule to multiple bots, for example ['Applebot', 'Bingbot'].
allow and disallow in rules accept string or array
The allow and disallow fields in a rule can each be a string or array of strings for specifying multiple paths.
Multiple user agents example
Example with array of rules customizing for specific user agents: import type { MetadataRoute } from 'next' export default function robots(): MetadataRoute.Robots { return { rules: [ { userAgent: 'Googlebot', allow: ['/'], disallow: '/private/', }, { userAgent: ['Applebot', 'Bingbot'], disallow: ['/'], }, ], sitemap: 'https://acme.com/sitemap.xml', } }
Non-standard directives via other field
Use the other field on a rule to pass non-standard directives not in the Robots Exclusion Standard, such as Request-Rate (Seznam) or Clean-param (Yandex). The other field is a Record where keys preserve their casing and array values emit one line per entry, scoped to the rule's User-Agent block. Next.js does not validate directive names or values; refer to the target search engine's documentation.
Non-standard directives example
Example using the other field for non-standard directives: import type { MetadataRoute } from 'next' export default function robots(): MetadataRoute.Robots { return { rules: [ { userAgent: '*', allow: '/' }, { userAgent: 'SeznamBot', allow: '/', other: { 'Request-Rate': '10/1m', }, }, ], } }
Robots object type definition
The Robots type has the following structure: type Robots = { rules: | { userAgent?: string | string[] allow?: string | string[] disallow?: string | string[] crawlDelay?: number other?: Record<string, string | number | Array<string | number>> } | Array<{ userAgent: string | string[] allow?: string | string[] disallow?: string | string[] crawlDelay?: number other?: Record<string, string | number | Array<string | number>> }> sitemap?: string | string[] host?: string }
crawlDelay field in rule
The crawlDelay field in a rule is an optional number specifying the delay in seconds between requests from a crawler.
sitemap property accepts string or array
The sitemap property in the Robots object can be a string or array of strings for specifying one or more sitemap URLs.
host property in Robots object
The host property in the Robots object specifies the preferred host for the website.
robots.js Route Handler caching behavior
A robots.js file is a special Route Handler that is cached by default unless it uses a Request-time API or dynamic config option, which would make it dynamic.
robots introduced in v13.3.0
The robots file convention was introduced in Next.js v13.3.0.
other field for non-standard directives added in v16.3.0
The other field for non-standard per-agent directives was added in Next.js v16.3.0.
robots.txt location and purpose
Place a robots.txt file in the root of the app directory to tell search engine crawlers which URLs they can access on your site. The file must match the Robots Exclusion Standard.
Static robots.txt example
A static robots.txt file placed at app/robots.txt uses standard robots.txt format. Example: User-Agent: * Allow: / Disallow: /private/ Sitemap: https://acme.com/sitemap.xml
robots.js/robots.ts dynamic generation
Add a robots.js or robots.ts file that exports a default function returning a Robots object. This file is a special Route Handler cached by default unless it uses a Request-time API or dynamic config option.
global-not-found.js metadata
For global-not-found.js, you can export a metadata object or generateMetadata function to customize the title, meta tags, and other head tags for the 404 page. Next.js automatically injects <meta name="robots" content="noindex" /> for pages returning 404 status, including global-not-found.js pages.