#spider sepcifics for HBLN website User-agent: * Disallow: /library/ Disallow: /tmp/ # these will soon disappear Disallow: /phpwcms_filestorage/ Disallow: /images/ Disallow: /files/ Disallow: /downloads/ Disallow: /picture/ User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Disallow:/