User-agent: * Disallow: /assets/files/ Disallow: /members/ Disallow: /files/ User-Agent: W3C-checklink Disallow: