I downloaded a list I wasn’t allowed to republish

A site with no robots.txt still had terms that banned automated copying. I took the data down, then wrote rules for what we use and when we ask first.

Etching-style drawing of a wooden card catalog with one drawer pulled out and a rust-colored tag hanging from its handle

On November 3, I added a check to Scarcity’s speech-language pathologist page. I matched each school against the accreditor’s directory to see how many degrees came from accredited programs. The answer was 99%. I took the match down the same day.

The directory is on a professional association’s website. That part of the site had no robots.txt, the file a site uses to tell automated tools what they can’t touch. I took that to mean it was fine. It wasn’t. The site’s terms of use ban scrapers and other automated copying, and they ban republishing site content without written permission. I hadn’t read the terms before the download.

So I removed the match from the page and the downloads, added a switch in the code that keeps it off, and drafted a request for permission. None of the page’s counts changed. The match was a check on the numbers, not part of them.

Then I checked the other directories I was using. One accreditor offers a public CSV download, and its robots.txt allows automated access. Another publishes its accredited list as a PDF. A third blocks automated access in its robots.txt, so I hadn’t pulled from it.

Two days later I wrote the rules down, so I wouldn’t have to decide each case on the fly:

  • Public data is there to be used. Cite it, credit it, and link to it. Facts don’t need permission.
  • Ask first in two cases only. The site’s terms or robots.txt block automated access. Or a document says it can’t be reproduced, and I’d be republishing part of it, like a table row by row.
  • Read a site’s terms of use as well as its robots.txt.
  • Keep every raw download exactly as it came, so anyone can check a number against the file it came from.

If you’re building anything with other people’s data, read the terms before you write the script.

More on this: Scarcity, the project this came from.

Written by Aaron Edgell

Growth executive. Builder. Writer.

Get new essays and notes by email.

One email when I publish something new.

Previous: