docs: update README.md and DEVELOPMENT.md to match latest robots.txt policy

Author: chengjiahao1234

DEVELOPMENT.md

@@ -770,9 +770,10 @@ headline: "Text with [link](OPENINGS_LINK)"
 **Security & Protection:**
 - ✅ Enhanced 404 page with navigation buttons and Bootstrap icons
 - ✅ Comprehensive `robots.txt` with crawler access control
-  - Allows: Googlebot, Bingbot, Slurp with 10-second crawl delay
-  - Blocks: MJ12bot, AhrefsBot, SemrushBot, DotBot, PetalBot, DataForSeoBot
-  - Restricts: `/images/`, `/assets/`, `/css/`, `/js/` directories
+  - Allows: major search engines (Googlebot, Bingbot, Slurp, DuckDuckBot, etc.)
+  - Blocks: known heavy scraper / SEO bots (AhrefsBot, SemrushBot, MJ12bot, DotBot, PetalBot, DataForSeoBot)
+  - Avoids `Crawl-delay` (ignored by Googlebot and may trigger Search Console warnings)
+  - Restricts: internal build artifacts only (`/_site/`, `/bin/`)
 - ✅ Apache security configuration (`.htaccess`)
   - Directory browsing disabled
   - Security headers (X-Frame-Options, X-XSS-Protection, etc.)
@@ -970,7 +971,7 @@ headline: "Text with [link](OPENINGS_LINK)"
    - Test theme compatibility
 
 3. **Security Review:**
-   - Review `robots.txt` blocked crawlers
+   - Review `robots.txt` blocked scraper list (and confirm sitemap URL)
    - Update `.htaccess` security headers
    - Check GitHub Pages security settings
 

README.md

@@ -328,7 +328,7 @@ bundle install
 
 ## Security Features
 
-- ✅ **Crawler Protection:** `robots.txt` controls search engine access
+- ✅ **Crawler Protection:** `robots.txt` allows major search engines and blocks known heavy scraper bots
 - ✅ **Custom 404 Page:** User-friendly error handling with navigation
 - ✅ **DDoS Protection:** GitHub Pages + Cloudflare CDN
 - ✅ **Security Headers:** Content security and XSS protection