wstg-info-03

Review Webserver Metafiles for Information Leakage

Files1

View on GitHub

SKILL.md

GitHub

wstg-info-03

Test ID

WSTG-INFO-03

Test Name

Review Webserver Metafiles for Information Leakage

High-Level Description

Web servers contain various metafiles that can inadvertently expose sensitive information about the application structure, hidden directories, restricted areas, and internal configurations. Files such as robots.txt, sitemap.xml, security.txt, and META tags are often overlooked but can provide attackers with valuable reconnaissance data. This test involves systematically reviewing these metafiles to identify information leakage and potential attack vectors.

What to Check

Metafiles to Examine

  • robots.txt - Spider/crawler directives
  • sitemap.xml - Site structure mapping
  • sitemap_index.xml - Sitemap index files
  • security.txt - Security policy disclosure
  • humans.txt - Team/contributor information
  • crossdomain.xml - Flash cross-domain policy
  • clientaccesspolicy.xml - Silverlight policy
  • .well-known/ directory contents
  • HTML META tags - Robot directives
  • Open Graph and Twitter Card metadata
  • ads.txt - Authorized digital sellers
  • app-ads.txt - Mobile app advertising
  • assetlinks.json - Android app links
  • apple-app-site-association - iOS app links

Information to Identify

  • Hidden or restricted directories
  • Administrative panels and login pages
  • API endpoints
  • Backup directories
  • Development/staging paths
  • Internal IP addresses
  • Technology stack indicators
  • Team member names and roles
  • Security contact information
  • PGP/GPG keys

How to Test

Step 1: Retrieve and Analyze robots.txt

# Download robots.txt
curl -s https://target.com/robots.txt

# Save to file
curl -s https://target.com/robots.txt -o robots.txt

# Check for multiple user-agent sections
curl -s https://target.com/robots.txt | grep -i "user-agent\|disallow\|allow\|sitemap"

robots.txt Directives to Analyze

Directive Purpose Example
User-agent Specifies which bot User-agent: *
Disallow Paths to exclude Disallow: /admin/
Allow Explicitly allowed paths Allow: /public/
Sitemap Sitemap location Sitemap: /sitemap.xml
Crawl-delay Delay between requests Crawl-delay: 10

Common Sensitive Paths in robots.txt

/admin/
/administrator/
/backup/
/config/
/database/
/db/
/dev/
/internal/
/login/
/logs/
/private/
/secret/
/staging/
/test/
/tmp/
/upload/
/uploads/
/wp-admin/
/api/
/cgi-bin/

Step 2: Analyze Sitemaps

# Get main sitemap
curl -s https://target.com/sitemap.xml | xmllint --format -

# Check common sitemap locations
for path in sitemap.xml sitemap_index.xml sitemap1.xml sitemaps/sitemap.xml; do
    echo "=== $path ==="
    curl -s "https://target.com/$path" | head -20
done

# Extract all URLs from sitemap
curl -s https://target.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+'

# Find nested sitemaps
curl -s https://target.com/sitemap.xml | grep -i "sitemap"

Sitemap Types

Type Purpose
sitemap.xml Main page listing
sitemap_index.xml Index of multiple sitemaps
video-sitemap.xml Video content
image-sitemap.xml Image content
news-sitemap.xml News articles

Step 3: Check security.txt

# RFC 9116 compliant location
curl -s https://target.com/.well-known/security.txt

# Alternative location
curl -s https://target.com/security.txt

# Extract useful information
curl -s https://target.com/.well-known/security.txt | grep -i "contact\|encryption\|policy\|hiring\|acknowledgments"

security.txt Fields

Field Description
Contact Security team contact
Encryption PGP key location
Acknowledgments Hall of fame page
Policy Disclosure policy URL
Hiring Security jobs page
Expires File expiration date
Preferred-Languages Preferred languages

Step 4: Examine humans.txt

# Get humans.txt
curl -s https://target.com/humans.txt

# Look for names and roles
curl -s https://target.com/humans.txt | grep -i "team\|developer\|designer\|lead\|manager"

Step 5: Scan .well-known Directory

# Common .well-known files
for file in \
    security.txt \
    openid-configuration \
    assetlinks.json \
    apple-app-site-association \
    change-password \
    dnt-policy.txt \
    host-meta \
    host-meta.json \
    mta-sts.txt \
    nodeinfo \
    webfinger \
    matrix/client \
    matrix/server \
    acme-challenge \
    pki-validation \
    traffic-advice; do
    echo "=== .well-known/$file ==="
    curl -s "https://target.com/.well-known/$file" | head -10
done

Step 6: Analyze HTML META Tags

# Get page source and extract META tags
curl -s https://target.com | grep -i "<meta"

# Look for robot directives
curl -s https://target.com | grep -i "name=\"robots\""

# Extract Open Graph data
curl -s https://target.com | grep -i "og:"

# Find generator information
curl -s https://target.com | grep -i "generator"

META Tag Analysis

<!-- Robot directives -->
<meta name="robots" content="noindex, nofollow" />
<meta name="googlebot" content="noindex" />

<!-- Generator (CMS info) -->
<meta name="generator" content="WordPress 6.0" />

<!-- Technology indicators -->
<meta name="csrf-token" content="..." />
<meta name="viewport" content="..." />

Step 7: Check Additional Metafiles

# Cross-domain policy files
curl -s https://target.com/crossdomain.xml
curl -s https://target.com/clientaccesspolicy.xml

# Advertising files
curl -s https://target.com/ads.txt
curl -s https://target.com/app-ads.txt

# App association files
curl -s https://target.com/.well-known/assetlinks.json
curl -s https://target.com/.well-known/apple-app-site-association
curl -s https://target.com/apple-app-site-association

Step 8: Google Dorking for Metafiles

# Find robots.txt via Google
site:target.com inurl:robots.txt

# Find sitemaps
site:target.com inurl:sitemap filetype:xml

# Find exposed directories from robots.txt
site:target.com inurl:admin OR inurl:backup OR inurl:config

Tools

Command-Line Tools

Tool Description Usage
curl HTTP client curl -s https://target.com/robots.txt
wget File retrieval wget https://target.com/robots.txt
xmllint XML parser xmllint --format sitemap.xml
Gobuster Directory fuzzer gobuster dir -u target.com -w wordlist.txt
ffuf Fast web fuzzer ffuf -u https://target.com/FUZZ -w wordlist.txt
httpx HTTP toolkit httpx -path /robots.txt -l targets.txt

Automated Scanners

Tool Description
Burp Suite Spider/crawler module
OWASP ZAP Automated spider
Nikto Web server scanner
Parsero robots.txt analyzer

Online Tools

Service URL Purpose
Google Search Console search.google.com/search-console robots.txt tester
Bing Webmaster Tools bing.com/webmasters robots.txt validator
SEO Site Checkup seositecheckup.com Sitemap analysis

Example Commands/Payloads

Comprehensive Metafile Scan Script

#!/bin/bash
TARGET=$1

echo "=== METAFILE SCANNER ==="
echo "Target: $TARGET"
echo ""

# robots.txt
echo "[+] Checking robots.txt..."
curl -s "https://$TARGET/robots.txt" -o robots.txt
if [ -s robots.txt ]; then
    echo "Found robots.txt:"
    cat robots.txt
    echo ""
    echo "Disallowed paths:"
    grep -i "disallow" robots.txt | awk '{print $2}'
fi

echo ""

# sitemap.xml
echo "[+] Checking sitemap.xml..."
curl -s "https://$TARGET/sitemap.xml" -o sitemap.xml
if [ -s sitemap.xml ]; then
    echo "Found sitemap.xml"
    echo "URLs count: $(grep -c "<loc>" sitemap.xml)"
fi

echo ""

# security.txt
echo "[+] Checking security.txt..."
for path in ".well-known/security.txt" "security.txt"; do
    response=$(curl -s "https://$TARGET/$path")
    if [[ ! -z "$response" && ! "$response" =~ "404" ]]; then
        echo "Found at /$path:"
        echo "$response"
        break
    fi
done

echo ""

# humans.txt
echo "[+] Checking humans.txt..."
curl -s "https://$TARGET/humans.txt"

echo ""

# .well-known directory
echo "[+] Scanning .well-known/..."
for file in security.txt openid-configuration assetlinks.json apple-app-site-association; do
    status=$(curl -s -o /dev/null -w "%{http_code}" "https://$TARGET/.well-known/$file")
    if [ "$status" == "200" ]; then
        echo "Found: .well-known/$file"
    fi
done

Parsero - robots.txt Analyzer

# Install
pip install parsero

# Basic usage
parsero -u https://target.com

# Check disallowed entries
parsero -u https://target.com -sb

# Output to file
parsero -u https://target.com -o output.txt

Burp Suite Crawler Configuration

  1. Target > Site map > Right-click target
  2. Spider this host
  3. Spider > Control > Start
  4. Review discovered paths in Site map

OWASP ZAP Spider

# Using ZAP CLI
zap-cli quick-scan -s all -r https://target.com

# Spider specific URL
zap-cli spider https://target.com

Extract URLs from Sitemap (Python)

#!/usr/bin/env python3
import requests
import xml.etree.ElementTree as ET

def parse_sitemap(url):
    response = requests.get(url)
    root = ET.fromstring(response.content)

    # Handle namespace
    ns = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}

    urls = []
    for url_elem in root.findall('.//ns:loc', ns):
        urls.append(url_elem.text)

    return urls

# Usage
urls = parse_sitemap('https://target.com/sitemap.xml')
for url in urls:
    print(url)

Remediation Guide

1. robots.txt Best Practices

# Good - Generic exclusions
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/

# Bad - Reveals sensitive paths
User-agent: *
Disallow: /admin-secret-panel/
Disallow: /backup-20240115/
Disallow: /api/v2/internal/

Recommendations:

  • Do not use robots.txt as a security mechanism
  • Use authentication for sensitive areas instead
  • Consider not listing truly sensitive paths at all
  • Monitor for unauthorized access to "disallowed" paths

2. Sitemap Security

  • Exclude sensitive URLs from sitemaps
  • Keep sitemaps updated (remove deprecated pages)
  • Don't include internal/development URLs
  • Consider requiring authentication for full sitemaps

3. META Tag Hardening

<!-- Remove generator information -->
<!-- Bad -->
<meta name="generator" content="WordPress 6.0" />

<!-- Good - Remove entirely or use generic -->
<meta name="generator" content="Custom CMS" />

<!-- Restrict indexing for sensitive pages -->
<meta name="robots" content="noindex, nofollow" />

4. security.txt Guidelines (RFC 9116)

# Include useful information
Contact: mailto:security@example.com
Encryption: https://example.com/.well-known/pgp-key.txt
Policy: https://example.com/security-policy
Preferred-Languages: en
Expires: 2025-12-31T23:59:59.000Z

# Sign the file with PGP for authenticity

5. humans.txt Considerations

  • Avoid exposing full names
  • Don't include email addresses
  • Use roles instead of personal details
  • Consider removing if not necessary

6. .well-known Directory

  • Audit all files in .well-known
  • Remove unnecessary entries
  • Restrict access to sensitive configurations
  • Monitor for unauthorized additions

Risk Assessment

CVSS Score

Base Score: 5.3 (Medium)

CVSS Vector: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N

Metric Value Description
Attack Vector Network Accessible via internet
Attack Complexity Low Simple file retrieval
Privileges Required None No authentication needed
User Interaction None No user interaction required
Scope Unchanged Impact scope unchanged
Confidentiality Low Directory/path disclosure
Integrity None No integrity impact
Availability None No availability impact

Severity Levels

Finding Severity Description
Standard robots.txt Info Normal crawler directives
Sensitive paths in robots.txt Low Hidden admin/backup paths revealed
Internal IPs in metafiles Medium Network architecture exposed
Credentials in metafiles High Direct security compromise
Full team roster in humans.txt Low Social engineering vector

CWE Categories

CWE ID Title Description
CWE-200 Exposure of Sensitive Information to an Unauthorized Actor Information disclosure via metafiles
CWE-538 Insertion of Sensitive Information into Externally-Accessible File or Directory Sensitive data in public files
CWE-548 Exposure of Information Through Directory Listing Path disclosure
CWE-1230 Exposure of Sensitive Information Through Metadata Metadata-based information leak

References

OWASP References

Standards

Tools

Checklist

[ ] robots.txt retrieved and analyzed
[ ] Disallowed paths documented
[ ] Sitemap.xml retrieved and parsed
[ ] All sitemap URLs extracted
[ ] security.txt checked (both locations)
[ ] humans.txt retrieved
[ ] .well-known directory scanned
[ ] HTML META tags analyzed
[ ] crossdomain.xml checked
[ ] clientaccesspolicy.xml checked
[ ] ads.txt checked
[ ] App association files checked
[ ] Hidden paths verified for accessibility
[ ] Sensitive information documented
[ ] Risk assessment completed
[ ] Remediation recommendations prepared

Related in security

git-workflow

Git workflow guidance for commits, branches, and pull requests

by agno-agi
37,144

clerk-observability

Implement monitoring, logging, and observability for Clerk authentication. Use when setting up monitoring, debugging aut...

by jeremylongshore
1,061

juicebox-migration-deep-dive

Advanced Juicebox data migration strategies. Use when migrating from other recruiting platforms, performing bulk data im...

by jeremylongshore
1,061