Feature Guide

Site Crawler & Site Profile

A complete technical checkup of your website — run automatically, kept current as your site evolves.

You cannot fix problems you cannot see. The Site Crawler scans your website the way a search engine does and turns what it finds into a plain-language site profile: every page mapped, your technology stack identified, common SEO issues flagged, and your site’s metadata (the titles and descriptions search engines display) cataloged in one place. It is the technical checkup most small businesses never get — done automatically.

For an online store, that means learning that twelve product pages are missing descriptions before Google notices. For a law office, it means discovering that a practice-area page is not linked from anywhere and effectively invisible. The crawler surfaces these quiet problems while they are still cheap to fix.

The site profile it builds is also the foundation the rest of Crafty Meerkat stands on. Your built-in AI Advisor reads the profile to understand what your site actually contains — so weekly reports, Stats Chat answers, and Prompt Builder documents reflect your real site structure, not assumptions. A current profile makes every AI feature sharper.

How it works

The crawler starts at your homepage and follows your internal links, visiting your pages the way a visitor or a search engine would. Along the way it maps your page structure, detects the platforms and tools your site is built with, checks each page for common SEO issues, and records metadata. It also spots broken links — internal and external links that lead to 404 errors or unreachable pages — and reports them so you can fix them before visitors hit them.

You can refresh the profile manually anytime with one click — useful right after you publish new pages or finish a redesign. On Growth and Agency plans, the crawler also re-runs automatically every day, so your profile keeps up as your site evolves without you having to remember anything.

Step by step

1
Open your Site Profile
Select your site in the dashboard, then choose Profile in the sidebar. If a crawl has already run, you will see your current profile; otherwise, start one.
2
Review the page map and technology stack
Check that every important page was found. A page missing from the map is usually a page with no links pointing to it — which means visitors and search engines struggle to find it too.
3
Work through flagged issues
The profile lists SEO issues and broken links in plain language. Fix broken links first (they frustrate real visitors today), then work through metadata gaps like missing page titles and descriptions.
4
Re-crawl after you make changes
Click the manual refresh after publishing fixes to confirm they took effect. On Growth and Agency plans, the daily automatic re-crawl keeps the profile current on its own.

Tips from the den

◆Run a manual re-crawl immediately after launching new pages or a redesign — do not wait to find out something broke.
◆Treat the broken-links list as your highest-priority finding; dead links cost you visitors and search ranking at the same time.
◆Check the page map for “orphaned” pages that nothing links to — adding a link from your navigation or a popular page is often a free traffic win.
◆Keep the profile current before generating a Prompt Builder document; the improvement plan is only as good as the site picture behind it.

See it live — no signup

The demo dashboard runs on a real site with real traffic.

Open the live demo →

Common questions

How often does the crawler run?

Everyone can refresh the site profile manually anytime with one click. On Growth and Agency plans, the crawler additionally re-runs automatically every day, so the profile stays current without any effort on your part.

Will crawling slow down my website?

No. The crawler visits your pages politely, the way a single careful visitor would — it does not hammer your server, and your visitors will not notice anything.

What SEO issues does the crawler check for?

It checks for the common, high-impact problems: missing or duplicate page titles and descriptions, metadata gaps, broken internal and external links, and structural issues like pages that nothing links to. Each finding is explained in plain language.

How does the site profile improve the AI features?

The profile tells your built-in AI Advisor what your site actually contains — its pages, structure, and technology. Weekly reports, Stats Chat, and Prompt Builder all read it, so their recommendations reference your real site instead of generic assumptions.