The idea quickly evolved into a real goal, to make good on the promise of free and open policing data. By freeing policing data from antiquated and difficult-to-access county data systems, and compiling that data in a rigorous way, we could create a valuable new tool to level the playing field and help provide community oversight of police behavior and activity.
In the almost 3 years since the first post, something amazing has happened.
The idea turned into something real. Something called The Police Data Accessibility Project. (https://www.pdap.io)
More than 2,000 people joined the initial community, and while those numbers dwindled after the initial excitement, a core group of highly committed and passionate folks remained. In these 3 years, this team has worked incredibly hard to lay the groundwork necessary to enable us to realistically accomplish the monumental data collection task ahead of us.
Let me tell you a bit about what the team has accomplished in these 3 years.
Established the community and identified volunteer leaders who were willing and able to assume consistent responsibility.
-Gained a pro-bono law firm to assist us in navigating the legal waters. Arnold + Porter is our pro-bono law firm.
-Arnold + Porter helped us to establish as a legal entity and apply for 501c3 status
-501c3 status granted
-We've carefully defined our goals and set a clear roadmap for the future
-Hired first full-time staff.
-PDAP was awarded a $250,000 grant by The Heinz Endowments
So now, I'm asking for help, because scraping, cleaning, and validating 18,000 police departments is no easy task.
The first is to join us and help the team. Perhaps you joined initially, realized we weren't organized yet, and left? Now is the time to come back. Or, maybe you are just hearing of it now. Either way, the more people we have working on this, the faster we can get this done. Those with scraping experience are especially needed. The second is to either donate, or help us spread the message. The more donations, the more data we can gather. I want to thank the r/privacy community especially. It was here that things really began.
TL;DR: I accidentally started a movement from a blog post I wrote about policing the police with data. The movement turned into something real because of r/privacy and hackernews: (Police Data Accessibility Project). 3 years later, the groundwork has been laid, non-profit established, full-time staff hired, and $250,000 in grant money and donations so far!
Last time this question was asked on HN was in 2017 (https://news.ycombinator.com/item?id=15694118), a lot has changed in the last 5 years in the world of web scraping (legal landscape, antibot unblockers, data type specific APIs, etc), so I thought it may be a good idea to refresh this question and see what are the most popular tools used by the HN community these days.
The idea quickly evolved into a real goal, to make good on the promise of free and open policing data. By freeing policing data from antiquated and difficult-to-access county data systems, and compiling that data in a rigorous way, we could create a valuable new tool to level the playing field and help provide community oversight of police behavior and activity.
In the almost 3 years since the first post, something amazing has happened.
The idea turned into something real. Something called The Police Data Accessibility Project. (https://www.pdap.io)
More than 2,000 people joined the initial community, and while those numbers dwindled after the initial excitement, a core group of highly committed and passionate folks remained. In these 3 years, this team has worked incredibly hard to lay the groundwork necessary to enable us to realistically accomplish the monumental data collection task ahead of us.
Let me tell you a bit about what the team has accomplished in these 3 years.
Established the community and identified volunteer leaders who were willing and able to assume consistent responsibility.
-Gained a pro-bono law firm to assist us in navigating the legal waters. Arnold + Porter is our pro-bono law firm.
-Arnold + Porter helped us to establish as a legal entity and apply for 501c3 status
-501c3 status granted
-We've carefully defined our goals and set a clear roadmap for the future
-Hired first full-time staff.
-PDAP was awarded a $250,000 grant by The Heinz Endowments
So now, I'm asking for help, because scraping, cleaning, and validating 18,000 police departments is no easy task.
The first is to join us and help the team. Perhaps you joined initially, realized we weren't organized yet, and left? Now is the time to come back. Or, maybe you are just hearing of it now. Either way, the more people we have working on this, the faster we can get this done. Those with scraping experience are especially needed.
The second is to either donate, or help us spread the message. The more donations, the more data we can gather.
I want to thank the r/privacy community especially. It was here that things really began.
TL;DR: I accidentally started a movement from a blog post I wrote about policing the police with data. The movement turned into something real because of r/privacy and hackernews: (Police Data Accessibility Project). 3 years later, the groundwork has been laid, non-profit established, full-time staff hired, and $250,000 in grant money and donations so far!
Scrapers so far Github https://github.com/Police-Data-Accessibility-Project/Scraper...
Discord if you would like to join the efforts: https://discord.com/invite/wMqex8nKZJ
*This is US centric