Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It's all part of DocumentCloud, which means API access to the text of all that Guantanamo data. See how easy this can be:

    curl http://www.documentcloud.org/api/search.json?q=group:nytimes

    "total": 2821,
    "page": 1,
    "per_page": 10,
    "q": "group:nytimes",
    "documents": [{
        "id": "86275-isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo",
        "title": "ISN 10015 - Abd al Rahim al Nashiri - JTF-GTMO Detainee Assessment",
        "access": "public",
        "pages": 15,
        "description": null,
        "source": null,
        "created_at": "Sun, 24 Apr 2011 15:50:19 +0000",
        "updated_at": "Mon, 25 Apr 2011 15:12:53 +0000",
        "canonical_url": "http://www.documentcloud.org/documents/86275-isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.html",
        "resources": {
            "pdf": "http://s3.documentcloud.org/documents/86275/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.pdf",
            "text": "http://s3.documentcloud.org/documents/86275/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.txt",
            "thumbnail": "http://s3.documentcloud.org/documents/86275/pages/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo-p1-thumbnail.gif",
            "search": "http://www.documentcloud.org/documents/86275/search.json?q={query}",
            "page": {
                "text": "http://www.documentcloud.org/documents/86275/pages/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo-p{page}.txt",
                "image": "http://s3.documentcloud.org/documents/86275/pages/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo-p{page}-{size}.gif"
            }
        }
    },
    ...
    ]}
Look at documents[0].resources.text:

    curl http://s3.documentcloud.org/documents/86275/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.txt
There you go. Now go build that tree and report back. :-)


I'm afraid that only a handful (< 20, so far) of the newly released documents have been redacted and published thus far. You won't get too far trying to build that tree.


Sure but I can start writing code now, in anticipation of the full set.


So I was fiddling a bit. Sadly, despite being able to skip the OCR step, there's obviously a lot of OCR errors in the text as is, which would make reliably stripping out data somewhat more difficult.

The reality is that I don't really have the time to commit to this unless everything worked perfect. Ah well, maybe someone else will do it.


Oh, very cool. Thanks for the heads up-- If I get some free time I'll do this.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: