There are two tiny files with disproportionate power over whether your site shows up in search engines. One is the sitemap, the other is the robots instructions file. Both can be opened straight from a browser, neither takes up more than a few kilobytes, and yet an error in either one can keep your entire site out of search results without anything looking broken at a glance.
The robots.txt file: what they're allowed to crawl
It always lives at the root of the domain and it's the first thing a search engine checks before crawling your site. Its job is to indicate which areas it may visit and which it should skip.
There's a huge confusion worth clearing up once and for all: blocking an address here does not remove it from Google. It only prevents the engine from reading it. If that address has links pointing to it from other sites, it can still appear in results with a strange title and no description, because the engine knows it exists but can't see what's inside. To genuinely remove a page from results you have to let them read it and place a no-index instruction on the page itself. Blocking and hoping it disappears is the most common conceptual error in technical SEO.
What's actually worth blocking: administrative areas, internal system folders, internal search results that generate infinite combinations, addresses with filter parameters that multiply near-identical pages, and carts or user account areas. What you must never block: stylesheets and scripts, because the engine needs to see them to understand how your page looks on a phone, and images if you want them to appear in image search.
The sitemap: what you want them to find
If the robots file says where not to go, the sitemap does the opposite: it's the list of every address you want indexed, served on a plate. It doesn't guarantee indexing, but it dramatically speeds up discovery, especially on new sites, large sites, and pages that barely receive internal links.
Each entry can include the last modification date, which is the single most useful field because it tells the engine where it's worth returning. The other two fields the format allows, change frequency and priority, have been largely ignored by search engines for years; include them or don't, it changes nothing.
As a site grows, one single file becomes unwieldy. The format supports a sitemap index: a main file pointing to several child files, each with its own list. That's the correct way to organize a site with thousands of addresses, and it also lets you separate by content type or by language, which makes it far easier to spot where the problem is when something isn't getting indexed.
Multilingual sites
If your site exists in two languages, each address should declare its alternate versions. That's done in the header of each page or inside the sitemap itself, and it prevents the most typical problem of bilingual sites: the engine treating both versions as duplicate content, or showing the English version to someone searching in Spanish.
The rule is that declarations must be reciprocal: if the Spanish page says its English version is at a given address, the English page has to say the same in reverse. A declaration that isn't returned is simply ignored.
The mistakes I've seen most often
- The full block left over from development. The whole site is blocked while it's being built, and in the rush of launch nobody removes it. The site works perfectly and appears nowhere. It is, without exaggeration, the most expensive error on this list.
- Staging addresses in the sitemap. The sitemap gets generated before the site moves and ends up pointing at the development address.
- Including pages you don't want indexed. If an address is in the sitemap and simultaneously carries a no-index instruction, you're sending the engine contradictory signals.
- Outdated sitemaps. A hand-made sitemap from two years ago, with half its addresses dead, does more harm than good.
- Blocking styles and scripts. The engine sees your page as an unstyled skeleton and can't evaluate whether it's mobile-friendly.
- Not submitting it. The sitemap exists, it's correct, and nobody registered it in webmaster tools. It gets discovered anyway, but far more slowly.
How to verify they're right
Verification is simpler than it sounds and takes five minutes. Open your domain followed by /robots.txt in a browser and read it line by line; if there's a line blocking everything, there's your problem. Then open your sitemap address and check that it loads without errors and that the addresses are production ones. Finally, go into webmaster tools, submit the sitemap, and use the inspection tool to confirm that an arbitrary page can be indexed. That tool tells you exactly what the engine sees, which is almost never the same as what you see.
What these files cannot do for you
It's worth setting expectations, because both files get credited with powers they don't have. A sitemap does not make pages rank. It doesn't even guarantee they get indexed: it's an invitation, not an order. If a page is thin, duplicated, or offers nothing anyone was searching for, listing it in a sitemap changes nothing. Search engines index what they judge worth storing, and that judgment happens after crawling, not before.
Likewise, the robots file doesn't protect anything. It's a public document any person or bot can read, and well-behaved crawlers respect it while malicious ones read it as a map of exactly where the interesting folders are. Listing your admin directory there to keep it out of search results is effectively advertising it. Anything that genuinely needs protection needs a password, not a polite request.
Used correctly, these two files do one job very well: they make crawling efficient. On a small site that barely matters. On a site with thousands of addresses, efficient crawling is the difference between new content being found in hours or in weeks.
An architecture recommendation
If you get to choose, make the sitemap generate itself. A static file you have to regenerate by hand always goes stale, without exception. A dynamic sitemap built by reading the site's real content is never out of date, because it's assembled the moment someone requests it. It's more work the first time and zero work forever after.
On the server side, what you need is the ability to generate and serve these files without friction: access to the configuration file for rewrite rules, cPanel for comfortable editing, and a server that responds quickly when the engine crawls thousands of addresses, because if it takes too long, the engine slows its pace and takes weeks to cover your whole site. That's why I use BanaHosting: LiteSpeed with LSCache, pure SSD on Dell servers with RAID-10 and full cPanel, from $4.95 a month with free migration and a 30-day guarantee.