User-Agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /_next/
Disallow: /search

User-Agent: Scrapy
User-Agent: python-requests
User-Agent: Go-http-client
Disallow: /

Host: https://aicatalog.com.ua
Sitemap: https://aicatalog.com.ua/sitemap.xml