rizalfarhan.
Back to Study Cases
R&D•2024•5 min read

Technical SEO Crawler & Auditor

Web ScrapingPostgreSQLSEONode.js

Objective

Audit 1,000+ pages of the Bali Realty site for broken links and missing meta tags, with trend analysis over time.

Developer Approach

Build a custom crawler in Node.js using Cheerio. Store results in a PostgreSQL database so you can compare runs and track fixes over time.

Technical Optimization

Use recursion with depth limits and a Sitemap.xml parser so the crawler discovers orphaned pages that aren't linked in the main navigation, giving a complete picture of site health.

Key Learnings

  • Sitemap + recursion with depth limits find orphaned pages
  • Storing results in PostgreSQL enables trend and delta analysis
  • Cheerio in Node.js is well-suited for HTML audit pipelines