Secrets of a seventeen year old scraper
Even if you don’t follow Avi Schiffmann, you’re probably familiar with the 17-year-old’s work: he’s the creator of http://nCoV2019.live and http://2020Protests.com, the two leading tracking tools for covid cases and protests you can join.
In an interview with Tanya Basu for MIT Tech Review, Schiffmann describes how he taught himself to build these dashboards when he made a site that scraped his school’s athletic stats portal and made it legible and useful.
When he decided to use those skills to build his covid tracker, he put a call out on some coding sites and attracted a dozen high-school aged volunteers, many in Asia, who helped write the scrapers that pull in the data for http://nCoV2019.live.
As the project grew more sophisticated and then begat his protest tracker, he used online tutorials and communities to acquire the knowledge he needed to overcome new challenges (“The thing is, you can learn anything online”).
Since the first days of the maker movement, it’s been clear that the ability to search and discuss is the major thing that differentiates “makers” from earlier generations of tinkerers, like the radio and electronics hobbyists that kept Modern Mechanix in business.
Schiffmann is a self-confessed “bad student” with a 1.7GPA and 60% attendance, which he attributes to his consuming passion for his programming projects. It’s a testament to how much of pedagogy turns on getting out of the student’s way when their passions are inflamed.
I read my first novel - Alice in Wonderland - one day in second grade. I pulled it off the shelf before class started and sat on the carpet to read it, and my teacher, Bev Panikkar, saw that I was engrossed in it and didn’t call me to class.
She let me sit there for two consecutive school days while I read, and while I was kicking off a lifelong passion for literature (also, I married a woman called Alice!).
As interesting as the pedagogical and makerish implications of Schiffmann’s story, I’m also fascinated by the role that scraping plays in making these essential information resources.
Scraping went from a honorable practice (the core of Google Search, for example) to a potential felony over the course of decades, as companies that made their fortunes scraping others turned around and sued, banned and blocked anyone who returned the favor.
But scraping is one of the key tools for attaining Adversarial Interoperability, which once kept tech dynamic and responsive to users, and the absence of which has contributed greatly to its stagnation and corruption.