What is the difference between robots.txt and meta tags?

What is the difference between robots.txt and meta tags?

Robots.txt is a file that tells search engines which pages to crawl or not, while meta tags provide specific instructions on individual pages.

Can you give me an example of how to use a robots.txt file?

Sure! A simple robots.txt file might look like this:

User-agent: * Disallow: /private/

This tells all search engines not to crawl the 'private' directory.

And what about meta tags? How do they work?

Meta tags are placed in the HTML of a webpage. For example, a meta tag for robots might look like this: <meta name='robots' content='noindex'>, which tells search engines not to index that page.

Which one should I use for better SEO?

Both are important! Use robots.txt to manage crawling at a site-wide level and meta tags for page-specific instructions.