OUR SERVICE AREA
DM. Digital serves businesses across the USA and is available Open 24/7 for SEO Agency in the USA. We handle Google Business Profile, Local SEO, Website Design, Content Strategy, Analytics & Performance and AI Visibility - data-driven, professional, and ROI-focused.
Get your free SEO audit. We'll help you get more leads, visibility, and clients for your local business.
Need a website for your business?
Grow Local creates a complete, SEO-optimized website in seconds. Fill a few fields - AI does the rest. $99/month.
A bakery owner near downtown Reno called us last month, worried. She had noticed odd traffic in her website reports - bots with names like GPTBot and PerplexityBot hitting her pages at 3 a.m. She wanted to know if these visitors were stealing her recipes or, worse, spreading wrong hours to customers who ask an AI where to buy fresh sourdough.
Her questions are the same ones we hear from shop owners across the Truckee Meadows and beyond. AI tools now read websites the way search engines have for years, and most small business owners have no idea what is being pulled or how to control it. Two small text files - robots.txt and llms.txt - sit at the center of this.
Think of robots.txt as the front-door sign for a website. It tells every web crawler where it may go and where it should stay out. The file has been around since the mid-1990s, long before anyone typed a question into ChatGPT.
For a local business, this small file carries real weight. When Google's crawler arrives, it checks robots.txt first before reading any pages. Get it right and your shop shows up when people search nearby. Get it wrong and you can vanish from results overnight.
Here is why the file matters for a local website:
The file speaks a simple language built on two main commands. The Allow directive gives a crawler permission to read a page or folder. The Disallow directive tells it to stay away. Bots read these lines top to bottom before they crawl a single page.
A line like Disallow: /admin/ keeps crawlers out of a login area. A line like Allow: /services/ makes sure your service pages stay open for reading. Each rule starts with a User-agent line naming which bot the rule applies to, or an asterisk that means every bot.
This is where many owners trip up. A single typo can block a whole folder of important pages. We always test each rule before it goes live, because one wrong slash can hide a plumber's emergency service page from the exact people searching for help at midnight.
The order of rules matters too. Search engines read the most specific rule that matches a URL. A well-written file uses clear, deliberate lines so there is no guessing about what a crawler can touch.
Here is the part that surprises people. robots.txt is a request, not a lock. Reputable bots like Googlebot honor it, but a rude scraper can ignore every line and crawl anyway. The file has no teeth against bad actors.
There is also a difference between blocking crawling and blocking indexing. Crawl blocking stops a bot from reading a page. It does not always stop that page from showing up in search results if other sites link to it.
To truly keep a page out of search results, an owner needs a noindex tag on the page itself, not a robots.txt rule. In fact, blocking a page in robots.txt can prevent Google from ever seeing the noindex tag, which defeats the purpose. The two tools solve different problems.
We explain this to clients often. If you want a page truly private, robots.txt is the wrong tool. Passwords and noindex tags do that job. robots.txt is about steering polite crawlers, nothing more.
The file always sits at the root domain. That means it lives at yoursite.com/robots.txt, never in a subfolder. If it is placed anywhere else, crawlers will not find it and will assume they can read everything.
Because it sits in one spot and controls the whole site, editing it deserves care. We have seen brand-new sites launch with a leftover block from the developer that hid the entire business from Google for weeks. Nobody noticed until the phone stopped ringing.
The biggest danger comes during a website redesign. Developers often block all crawlers while building a site so the unfinished version does not show up in search. The mistake is forgetting to remove that block after launch. If a new site suddenly loses traffic, robots.txt is the first place we check. Whoever manages the site should treat this file as a switch that can turn the whole business on or off in search.
The llms.txt file is the new kid on the block. It was proposed in 2024 as a way to help AI language models understand a website more clearly. Where robots.txt is decades old, this idea is barely a year into real use.
The goal is simple. A large language model like the one behind ChatGPT reads messy web pages full of menus, ads, and scripts. An llms.txt file offers a clean, plain summary of what a business does, so the AI gets the facts right when someone asks about it.
For local businesses, this could shape how AI tools describe their shop, hours, and services. It is worth learning even if adoption is still early. Our AI visibility service helps owners set this up correctly from the start.
The two files do very different jobs. Looking at llms.txt vs robots.txt side by side clears up the confusion fast. One controls access, the other explains meaning.
robots.txt is a traffic cop. It tells crawlers which roads are open and which are closed. It says nothing about what the pages actually contain, only where a bot may travel.
llms.txt is more like a tour guide. It does not block anything. Instead, it hands an AI model a short, readable summary of the business and points to the most useful pages. It answers the question "what is this site about" in plain words.
An owner can use both files together with no conflict. robots.txt manages crawling behavior across all bots. llms.txt gives AI tools a head start on understanding the business so their answers stay accurate.
The llms.txt format is written in markdown, which is just plain text with a few simple symbols. That makes it easy for a non-technical owner to read and edit. No coding degree required.
A basic file starts with the business name as a heading, followed by a one or two sentence summary. For a Reno coffee shop, that might read: a family-owned cafe near Midtown serving locally roasted coffee and fresh pastries seven days a week.
Below the summary, the file lists links to top pages with short notes. A line might point to the menu page, another to the location and hours page, another to the catering services page. Each link gets a brief plain-language description of what it holds.
The whole point is clarity. Rather than making an AI dig through a cluttered homepage, the file spells out the services, the service area, and the hours. When done well, it reads like a clean cheat sheet about the business.
Here is our honest take on adoption. As of now, no major AI tool has publicly confirmed it reads llms.txt as a hard rule. The standard is young and still finding its footing among the big players.
That said, setting one up costs almost nothing and carries no downside. As AI search grows, early adopters may find their business described more accurately when tools do begin honoring the file. It is a low-effort bet on where things are heading.
We think of it like claiming a Google Business Profile early. The businesses that acted before it was common got a head start. The same pattern tends to repeat with new web standards.
For a local shop with a small website, an llms.txt file takes an afternoon to write. If it helps even one AI tool describe the business correctly to a future customer, the small effort pays off. We recommend it as a smart, low-risk move.
DM. Digital helps local service businesses dominate Google with custom-built websites.
Once an owner starts watching their traffic, the bot names pile up fast. These AI crawlers visit websites to gather text for training models and answering live questions. Knowing them by name helps an owner decide who to welcome.
Each bot leaves a fingerprint called a bot user agent. That is the identity string a crawler sends when it visits. Names like GPTBot show up plainly in server records once you know where to look.
Recognizing these visitors is the first step toward controlling them. Our technical SEO team reviews these logs regularly for local clients to separate helpful bots from bandwidth-eating scrapers.
OpenAI runs two bots that do different jobs. GPTBot is the training crawler. It reads web pages to help build and improve the models behind ChatGPT. When it visits, it collects text to make the AI smarter over time.
ChatGPT-User works differently. It is the live browsing agent that fetches a page in real time when a user asks ChatGPT to look something up. If someone asks the AI about a specific coffee shop, this bot may visit that shop's site to grab current details.
The difference matters for an owner deciding what to allow. Blocking GPTBot keeps content out of future training data. Blocking ChatGPT-User stops the live agent from pulling current info when a real person asks about the business.
Most local shops benefit from allowing the live agent. When a customer asks ChatGPT for a nearby bakery, you want your accurate hours and menu to be what the AI reads. That is a referral waiting to happen.
Google added a separate control called Google-Extended. This lets an owner allow normal Googlebot crawling for search while choosing whether Google may use the content for its AI training and Gemini answers. It splits the two decisions apart.
This is a helpful distinction. An owner can stay fully visible in Google Search while opting out of AI training if they wish. Blocking Google-Extended does not hurt regular search rankings, which surprises many people.
Bingbot plays a double role. It powers Bing search and also feeds Microsoft Copilot, the AI answer engine tied to Bing. Blocking Bingbot would remove a business from both Bing results and Copilot answers at once.
For a local business, we generally advise leaving Bingbot open. Bing still drives real traffic, and Copilot appears inside Windows and Edge for millions of users. Cutting it off closes two doors for a small upside. You can read Google's own guidance on these crawlers on their Google crawlers documentation.
The list keeps growing. PerplexityBot powers Perplexity, an AI answer engine that cites its sources with links. This one is worth attention because it often sends clickable referrals back to the sites it quotes.
ClaudeBot gathers content for Anthropic's Claude models. Amazonbot collects data for Amazon's Alexa and shopping features. Each has its own user agent name and its own purpose on the web.
Not all of them behave the same way. Reputable bots like PerplexityBot and ClaudeBot state that they honor robots.txt rules. Some lesser-known scrapers ignore the rules entirely and crawl whatever they can reach.
This split is why blocking rules only go so far. A polite bot reads your file and obeys. A rude one reads nothing and takes what it wants. Owners need to know which is which before deciding how hard to push back.
Every visit to a website gets recorded in server logs. These records list the IP address, the time, the page requested, and the user agent string. That string is where bot names appear.
To find them, an owner can open their hosting control panel and look for raw access logs or a traffic report. Searching those logs for names like GPTBot or PerplexityBot shows exactly when each visited and what they read. Many hosts now flag known bots automatically.
The trick is telling friendly crawlers from scrapers. A friendly bot names itself honestly and crawls at a reasonable pace. A scraper may hide behind a fake browser name or hammer the site hundreds of times a minute, eating bandwidth.
If an owner sees one address requesting thousands of pages in minutes, that is a red flag. We help clients set up log reviews and traffic alerts so aggressive bots get caught before they slow the site down for real customers.
Now for the decision that matters most. Which bots earn a spot at the table? The choice comes down to a trade-off between visibility and control.
The right answer to allow AI crawlers depends on a business's goals. A shop that wants maximum reach welcomes almost everyone. A studio protecting original work may pull back. Either way, the goal is protecting local search visibility while keeping unwanted crawlers in check.
Our advice for most local shops leans toward openness, with a few smart exceptions. Below is how we sort the decision for clients across cities like Reno and beyond.
This one is not up for debate. Googlebot and Bingbot must always be allowed. Blocking them is the fastest way to make a business disappear from the exact searches that bring in customers.
When someone near Midtown Reno searches for a plumber or a pizza place, Google's crawler is what decides whether your business shows up. Block that bot and you erase yourself from the map pack and the blue links at once. No amount of good service can fix invisibility.
We have audited sites where a leftover block cost a business months of search visibility. The owner could not figure out why the phone went quiet. One line in robots.txt was the whole story.
The rule is simple. Never block search engine crawlers unless you truly want to be invisible. For a local business chasing customers, that is never the goal. Keeping these bots welcome is the floor, not the ceiling, of a healthy site.
AI answer bots deserve a real look. Tools like Perplexity and ChatGPT increasingly answer questions like "best taco spot near the University of Nevada." If your site is readable to those bots, your business can be named in the answer.
That is the promise of AI referrals. When Perplexity cites a source, it links back, sending a curious searcher straight to the site. Those brand mentions put a business in front of people at the moment they are deciding where to go.
The trade-off is that these bots read your content to form their answers. For most local shops, that is a fair deal. The information they read - hours, services, location - is the same info you want customers to have anyway.
We usually recommend allowing the reputable AI answer bots for local businesses. The referral upside beats the small cost of being read. Our content strategy team helps shape pages so AI tools quote the right details.
Blocking has its place. If a business sells proprietary content - custom designs, paid courses, original research - it may want to block bots from feeding that work into AI training for free. That is a fair reason to draw a line.
Gated pages behind a login or paywall also belong off-limits to crawlers. There is no benefit to letting a bot read content that only paying members should see. These pages add nothing to public visibility.
Heavy server load is another trigger. If an aggressive bot crawls thousands of pages and slows the site for real visitors, blocking it protects the customer experience. A slow site loses sales, and no crawler is worth that.
The point is to block with intent, not fear. Blanket blocking every AI bot out of worry can quietly shrink a business's reach. We help owners weigh each case so the blocks they set actually serve their goals.
Time to get practical. Setting up these files does not require a developer for a basic local site. With a few safe defaults, an owner can handle both in an afternoon.
The goal of a clean robots.txt setup is to welcome search engines, hide admin areas, and point bots to the sitemap. A good llms.txt setup gives AI tools a clear picture of the business. Both files sit at the root of the domain.
Below are starter examples any local shop can adapt. Our website structure service handles this for clients who would rather leave it to a team.
A solid sample robots.txt keeps things simple. It starts by allowing all reputable crawlers, then blocks the folders no bot needs, then points to the sitemap. That last line helps bots find every page fast.
A basic version reads: User-agent: * on the first line, Allow: / on the next, then Disallow: /wp-admin/ to hide the login area, and finally Sitemap: https://yoursite.com/sitemap.xml at the bottom. Those few lines cover most local sites.
The sitemap line is the piece many owners forget. It is a direct map of every important page, and adding it helps search engines crawl a site more completely. For a shop with service pages, location pages, and a blog, that matters.
We tailor the blocked folders to each site's setup. A WordPress site hides different areas than a Shopify store. The core idea stays the same - open the public pages, close the back office, and hand over the sitemap.
An llms.txt file should read like a friendly briefing. Start with a clear business description in one or two sentences. Say what the business does, where it is, and who it serves.
Next, list the core service pages with short notes. A landscaping company might link to its lawn care page, its irrigation repair page, and its seasonal cleanup page. Each link gets a plain description so the AI knows what it holds.
Include the service area in plain terms. Instead of just naming the city, name the neighborhoods - serving Midtown, Old Southwest, Somersett, and the areas near Rancharrah. Naming specific areas helps AI tools match the business to nearby searches.
Finish with hours and contact details written simply. Open Monday through Saturday, 8 a.m. to 6 p.m., closed Sunday. When the file spells this out clearly, AI tools are far more likely to quote the right hours to a customer asking about the business.
Once the files are live, testing confirms they work. Google Search Console is the free tool every owner should use. It shows how Google reads a site and flags pages that are blocked by mistake.
Search Console includes a robots.txt tester feature that lets an owner paste a URL and see whether it is allowed or blocked. Running the most important pages through it catches errors before they cost traffic. We do this after every change we make.
A quick manual check helps too. Just type yoursite.com/robots.txt into a browser to see the live file. If it loads the correct rules, the file is in place. If it shows a 404 error, the file is missing or misplaced.
Regular checks matter because sites change. A new page, a plugin update, or a redesign can alter these files without warning. We build file reviews into our monthly work so nothing important slips through. You can learn more about robots.txt standards from the Google Search Central robots guide.
DM. Digital helps local service businesses dominate Google with custom-built websites.
We have cleaned up the same handful of errors dozens of times. These robots.txt mistakes quietly drain rankings and AI mentions. The frustrating part is that owners often have no idea anything is wrong.
Most of these local SEO errors are easy to check today. An owner can open their own robots.txt file and look for the warning signs below. Catching one early can save months of lost traffic.
Here are the mistakes we see most often and how to spot them on any site.
The worst mistake is a single line: Disallow: / with nothing after the slash. This disallow all command tells every crawler to skip the entire site. The business simply vanishes from search.
This usually happens during launch. Developers block the site while building it so the unfinished version stays private. The problem is forgetting to remove the block once the site goes live.
These launch mistakes can go unnoticed for weeks. The site looks perfect to visitors who type the address directly. But new customers searching Google never find it, and the phone stays quiet.
The fix takes seconds once you spot it. We check for this line first on every new client site. If a business recently relaunched and traffic dropped, this is almost always the culprit worth ruling out immediately.
Some older sites block folders holding CSS, images, and JavaScript files. It seems harmless. In reality, it breaks how Google sees the page.
Google renders a page the way a visitor would, loading the design and scripts to judge the layout. When those files are blocked, the crawler sees a broken, half-loaded page. This is render blocking, and it confuses the search engine about what the page really offers.
The damage hits hardest on mobile. Google uses mobile-first indexing, so a page that renders poorly can slide down in mobile rankings. For a local business, where most searches happen on phones, that is a serious hit.
The fix is to remove any Disallow rules covering these design files. Let Google load the full page as a customer sees it. Our mobile-first design service makes sure nothing important gets blocked from the crawler.
Crawler settings are only half the local picture. A clean robots.txt does nothing for the map results if the Google Business Profile is a mess. The two work together, not as substitutes.
The local map pack - those three business listings with the map at the top of local searches - draws heavily from the Google Business Profile. Hours, phone number, category, and reviews all live there, not in robots.txt.
We see owners obsess over site files while their profile lists old hours or a wrong address. That mismatch costs more customers than any crawler rule. Both need attention.
A strong local presence pairs a clean website with an accurate, active profile. Our Google Business Profile optimization keeps that listing sharp while the site files handle the crawlers. Together they cover the full local search picture.
All these decisions add up to something real. The bots an owner allows shape whether a business appears in map results, AI overviews, and voice searches. It is the bigger picture behind the small text files.
Today's local search results are not just blue links. They include AI overviews at the top of Google, AI answer engines, and voice search replies from phones and smart speakers. Each of those pulls from content that crawlers can read.
Getting the crawler choices right puts a business in more of those answers. Our local SEO service ties these pieces together for shops that want the full reach.
When someone asks an AI for a local recommendation, the tools name specific businesses. Being one of those names is the payoff for allowing AI bots and writing clear content. This is the heart of AI recommendations.
Picture a visitor to Reno asking ChatGPT for a good breakfast spot near the Riverwalk. The AI answers with a few names. The businesses that allowed crawling and described themselves clearly are the ones most likely to be listed.
This new practice has a name - answer engine optimization. It means shaping a site so AI tools can find, understand, and quote the business accurately. It builds on classic SEO but aims at AI answers instead of just search rankings.
The businesses winning here are not always the biggest. They are the ones with clear, well-organized pages that an AI can read without confusion. A small shop with a clean site can outshine a larger competitor that blocks the bots.
Nothing hurts more than an AI telling a customer the wrong hours. Accurate info across every tool starts with NAP consistency - the same name, address, and phone number everywhere the business appears online.
When AI tools crawl a site and find matching details on the Google Business Profile and local directories, they trust the data more. Conflicting info makes the AI guess, and guesses lead to wrong answers that send customers to a closed door.
Adding structured data helps too. This is code that labels information clearly - marking the phone number as a phone number and the hours as hours. It removes the guesswork so tools quote the right details.
We keep this data consistent across a client's entire online footprint. Our citation management service fixes mismatched listings so every AI tool and search engine sees one clear, correct set of facts.
Every owner has to decide how open to be. Wide visibility means being quoted often across AI tools. Tight control means keeping certain content private. Most local businesses land somewhere in the middle.
The visibility trade-off is real but manageable. A restaurant wants its menu and hours quoted everywhere. That same restaurant might keep an internal recipe blog behind a login. Different content calls for different rules.
The answer is a clear content policy. Decide which content is public and meant to attract customers, and which is private or proprietary. Then set robots.txt and llms.txt to match that plan.
For most local shops, leaning toward visibility wins. The public info that draws customers is exactly what you want AI tools to share. We help owners draw that line so the right content stays open and the sensitive parts stay protected.
DM. Digital helps local service businesses dominate Google with custom-built websites.
The bots visiting a local website are not something to fear. They are the new front door to customers who ask Google and AI tools where to go. Getting robots.txt and llms.txt right decides whether a business greets those customers or hides from them.
For most local shops, the smart move is openness with a few careful limits. Allow the search engines, welcome the reputable AI answer bots, block only what truly needs protecting, and keep every file at the root of the domain. Pair that with an accurate Google Business Profile and consistent business details, and a shop shows up wherever people are searching.
If reading server logs and editing text files sounds like more than you want to take on, our team handles all of it for local businesses. We set up these files correctly, keep your information consistent, and make sure you stay visible in both search and AI answers. Contact us today for a consultation and let us make sure the right crawlers can find your business.
The robots.txt file controls which crawlers can reach the pages on a website. It acts like a traffic cop, opening some roads and closing others. The llms.txt file does something different - it helps AI language models understand and summarize a site by offering a clean, plain-language description of the business. One manages access, the other explains meaning. A business can use both files together without any conflict between them.
An llms.txt file is optional today. No major AI tool has confirmed it reads the file as a strict rule yet. That said, setting one up takes little effort and carries no downside. As more tools adopt the standard, an early file could help a business appear accurately in AI answers. We view it as a low-risk move worth making now, much like claiming a business profile before it became common practice.
For most local shops, we do not recommend blocking GPTBot or similar bots. Allowing them lets AI tools cite your business when a customer asks for a nearby recommendation, which can drive real referrals. The main reason to block AI crawlers is protecting proprietary content like paid courses or original designs. If your public pages hold the same info you want customers to see, allowing these bots usually helps more than it hurts.
No. Blocking AI bots does not raise Google rankings. Search rankings are decided by Googlebot, which is a separate crawler from the AI training bots. In fact, blocking AI answer bots can reduce your visibility in AI-powered search features like AI overviews and answer engines. If the goal is better visibility, blocking these bots works against you rather than helping. Focus on clean content and an accurate profile instead.
Both files belong at the root domain of the website. That means the correct file location is yoursite.com/robots.txt and yoursite.com/llms.txt. If either file is placed in a subfolder, crawlers will not find it and will assume no rules apply. You can confirm they are in place by typing the address into a browser. If the file loads with your rules, it is set up correctly and ready to work.
No. The robots.txt file is not real security. It is a polite request that reputable bots honor, but bad scrapers can ignore it entirely. To truly keep a page private, use a password or a noindex tag on the page itself. Blocking a page in robots.txt can even prevent Google from seeing a noindex tag. For real page privacy with sensitive data, always rely on passwords and proper access controls, not robots.txt.
Check your server logs or analytics for the user-agent names of known bots. Most hosting control panels offer raw access logs or a traffic report where names like GPTBot appear. Searching those records shows when each bot visited and what pages it read. Many hosts now flag bot traffic automatically. If you see one address requesting thousands of pages in minutes, that is a sign of an aggressive scraper worth investigating.
No. Reputable bots like GPTBot, PerplexityBot, and ClaudeBot state that they follow crawler rules in robots.txt. However, some lesser-known scrapers ignore the rules completely and crawl whatever they can reach. This means robots.txt only stops the polite bots. To block aggressive scrapers, a business may need extra steps like firewall rules, rate limiting, or server-level blocks. For most local sites, the reputable bots are the ones that matter most.
Yes. robots.txt supports per-bot rules using each bot's user agent name. You can write a rule that allows Googlebot and Bingbot while blocking a specific AI training bot. Each rule starts with a User-agent line naming the bot, followed by Allow or Disallow directives. This lets a business fine-tune exactly who gets access. We often set up these targeted rules for clients who want search engines open but certain AI crawlers limited.
We recommend a file review at least every few months for an active site. More importantly, always check these files after any website redesign or new page launch. Redesigns are the most common time a blocking rule gets left in place by mistake, hiding the whole site from search. A quick check after major changes catches errors before they cost traffic. Building these reviews into a regular schedule keeps a business safely visible.
Last updated on
Why trust DM. Digital?
Founded in 2015, DM. Digital is an SEO Agency serving businesses across the USA. All content is reviewed by our licensed technicians.
DM. Digital helps local service businesses dominate Google with custom-built websites.
AI assistants use star rating thresholds to pick which local businesses to recommend. Learn the 4.0 floor, the 4.5 sweet spot, review count targets, and freshness rules that get you named first.
Free websites often cost local businesses more than they save through ads, upgrade fees, and missed calls. Learn the hidden costs and better options.
AI Overviews now answer roughly 40 percent of searches. Learn which queries trigger them, how Google's AI picks local businesses, and how to earn a spot.