On November 3, I added a check to Scarcity’s speech-language pathologist page. I matched each school against the accreditor’s directory to see how many degrees came from accredited programs. The answer was 99%. I took the match down the same day.
The directory is on a professional association’s website. That part of the site had no robots.txt, the file a site uses to tell automated tools what they can’t touch. I took that to mean it was fine. It wasn’t. The site’s terms of use ban scrapers and other automated copying, and they ban republishing site content without written permission. I hadn’t read the terms before the download.
So I removed the match from the page and the downloads, added a switch in the code that keeps it off, and drafted a request for permission. None of the page’s counts changed. The match was a check on the numbers, not part of them.
Then I checked the other directories I was using. One accreditor offers a public CSV download, and its robots.txt allows automated access. Another publishes its accredited list as a PDF. A third blocks automated access in its robots.txt, so I hadn’t pulled from it.
Two days later I wrote the rules down, so I wouldn’t have to decide each case on the fly:
- Public data is there to be used. Cite it, credit it, and link to it. Facts don’t need permission.
- Ask first in two cases only. The site’s terms or robots.txt block automated access. Or a document says it can’t be reproduced, and I’d be republishing part of it, like a table row by row.
- Read a site’s terms of use as well as its robots.txt.
- Keep every raw download exactly as it came, so anyone can check a number against the file it came from.
If you’re building anything with other people’s data, read the terms before you write the script.
More on this: Scarcity, the project this came from.
Get new essays and notes by email.
One email when I publish something new.
