ThePythonicCow
8th August 2013, 22:24
Here's the best article I've seen so far explaining how intelligence agencies work with large software and Internet companies to spy on us humans: ‘Big Data’ Dynamo: How Giant Tech Firms Help the Government to Spy on Americans (Global Research) (http://www.globalresearch.ca/big-data-dynamo-how-giant-tech-firms-help-the-government-to-spy-on-americans/5344211)
Here are some snippets from this article:
...Indeed, a top secret NSA Inspector General’s report published by The Guardian, revealed that the agency “maintains relationships with over 100 US companies,” adding that the US has the “home field advantage as the primary hub for worldwide telecommunications.”
...
Three of those data-slurping programs, UPSTREAM, PRISM and X-KEYSCORE, shunt domestic and global communications collected from fiber optic cables, the servers of Apple, Google, Microsoft and Yahoo, along with telephone data (including metadata, call content and location) grabbed from AT&T, Sprint and Verizon into NSA-controlled databases.
But however large, a database is only useful to an organization, whether its a corporation or a spy agency, if the oceans of data collected can be searched and extracted in meaningful ways.
To the growing list of spooky acronyms and code-named black programs revealed by Edward Snowden, what other projects, including those in the public domain, are hiding in plain sight?
Add Google’s BigTable and Yahoo’s Hadoop to that list. Both are massive storage and retrieval systems designed to crunch ultra-large data sets and were developed as a practical means to overcome “big data” conundrums.
...Who might also have a compelling interest in cataloging and searching through very large data sets, away from prying eyes, and at granular levels to boot? It should be clear following Snowden’s disclosures, what’s good for commerce is also a highly-prized commodity among global eavesdroppers.
Despite benefits for medical and scientific researchers sifting through mountains of data, as Ars Technica pointed out BigTable and Hadoop “lacked compartmentalized security” vital to spy shops, so “in 2008, NSA set out to create a better version of BigTable, called Accumulo.”
Developed by agency specialists, it was eventually handed off to the “non-profit” Apache Software Foundation. Touted as an open software platform, Accumulo is described in Apache literature as “a robust, scalable, high performance data storage and retrieval system.”
“The platform allows for compartmentalization of segments of big data storage through an approach called cell-level security. The security level of each cell within an Accumulo table can be set independently, hiding it from users who don’t have a need to know: whole sections of data tables can be hidden from view in such a way that users (and applications) without clearance would never know they weren’t there,” Ars Technica explained.
...
(Since that Ars piece appeared, we have since learned that NSA is now conducting what is described as “three-hop analysis,” that is, three degrees of separation from a target’s email or phone number. This data dragnet “could allow the government to mine the records of 2.5 million Americans when investigating one suspected terrorist,” the Associated Press observed).
...
Once collected, data is separated into digestible fragments (phone numbers, email addresses and log ins), then reassembled at lightning speeds for searchable queries in graphic form. Information gathered in the hopper includes not only metadata tables, but the “full log,” including what spooks call Digital Network Intelligence, i.e., user content.
And while it may not yet be practical for NSA to collect and store each single packet flowing through the pipes, the agency is already collecting and storing vast reams of data intercepted from our phone records, IP addresses, emails, web searches and visits, and is doing so in much the same way that Amazon, eBay, Google and Yahoo does.
...
Additionally, NSA is feverishly working to bring supercomputers online “that can execute a quadrillion operations a second” at the Multiprogram Research facility in Oak Ridge, Tennessee where enriched uranium for nuclear weapons is manufactured, as James Bamford disclosed last year in Wired.
As the secret state sinks tens of billions of dollars into various big data digital programs, and carries out research on next-gen cyberweapons more destructive than Flame or Stuxnet, as those supercomputers come online the cost of cracking encrypted passwords and communications will continue to fall.
Stanford University computer scientist David Mazières told CNET that mastering encrypted communications would “include an order to extract them from the server or network when the user logs in–which has been done before–or installing a keylogger at the client.”
This is precisely what Microsoft has already done with its SkyDrive cloud storage service “which now has 250 million users worldwide” and exabytes of data ready to be pilfered, as The Guardian disclosed.
One document “stated that NSA already had pre-encryption access to Outlook email. ‘For Prism collection against Hotmail, Live, and Outlook.com emails will be unaffected because Prism collects this data prior to encryption’.”
...
Here are some snippets from this article:
...Indeed, a top secret NSA Inspector General’s report published by The Guardian, revealed that the agency “maintains relationships with over 100 US companies,” adding that the US has the “home field advantage as the primary hub for worldwide telecommunications.”
...
Three of those data-slurping programs, UPSTREAM, PRISM and X-KEYSCORE, shunt domestic and global communications collected from fiber optic cables, the servers of Apple, Google, Microsoft and Yahoo, along with telephone data (including metadata, call content and location) grabbed from AT&T, Sprint and Verizon into NSA-controlled databases.
But however large, a database is only useful to an organization, whether its a corporation or a spy agency, if the oceans of data collected can be searched and extracted in meaningful ways.
To the growing list of spooky acronyms and code-named black programs revealed by Edward Snowden, what other projects, including those in the public domain, are hiding in plain sight?
Add Google’s BigTable and Yahoo’s Hadoop to that list. Both are massive storage and retrieval systems designed to crunch ultra-large data sets and were developed as a practical means to overcome “big data” conundrums.
...Who might also have a compelling interest in cataloging and searching through very large data sets, away from prying eyes, and at granular levels to boot? It should be clear following Snowden’s disclosures, what’s good for commerce is also a highly-prized commodity among global eavesdroppers.
Despite benefits for medical and scientific researchers sifting through mountains of data, as Ars Technica pointed out BigTable and Hadoop “lacked compartmentalized security” vital to spy shops, so “in 2008, NSA set out to create a better version of BigTable, called Accumulo.”
Developed by agency specialists, it was eventually handed off to the “non-profit” Apache Software Foundation. Touted as an open software platform, Accumulo is described in Apache literature as “a robust, scalable, high performance data storage and retrieval system.”
“The platform allows for compartmentalization of segments of big data storage through an approach called cell-level security. The security level of each cell within an Accumulo table can be set independently, hiding it from users who don’t have a need to know: whole sections of data tables can be hidden from view in such a way that users (and applications) without clearance would never know they weren’t there,” Ars Technica explained.
...
(Since that Ars piece appeared, we have since learned that NSA is now conducting what is described as “three-hop analysis,” that is, three degrees of separation from a target’s email or phone number. This data dragnet “could allow the government to mine the records of 2.5 million Americans when investigating one suspected terrorist,” the Associated Press observed).
...
Once collected, data is separated into digestible fragments (phone numbers, email addresses and log ins), then reassembled at lightning speeds for searchable queries in graphic form. Information gathered in the hopper includes not only metadata tables, but the “full log,” including what spooks call Digital Network Intelligence, i.e., user content.
And while it may not yet be practical for NSA to collect and store each single packet flowing through the pipes, the agency is already collecting and storing vast reams of data intercepted from our phone records, IP addresses, emails, web searches and visits, and is doing so in much the same way that Amazon, eBay, Google and Yahoo does.
...
Additionally, NSA is feverishly working to bring supercomputers online “that can execute a quadrillion operations a second” at the Multiprogram Research facility in Oak Ridge, Tennessee where enriched uranium for nuclear weapons is manufactured, as James Bamford disclosed last year in Wired.
As the secret state sinks tens of billions of dollars into various big data digital programs, and carries out research on next-gen cyberweapons more destructive than Flame or Stuxnet, as those supercomputers come online the cost of cracking encrypted passwords and communications will continue to fall.
Stanford University computer scientist David Mazières told CNET that mastering encrypted communications would “include an order to extract them from the server or network when the user logs in–which has been done before–or installing a keylogger at the client.”
This is precisely what Microsoft has already done with its SkyDrive cloud storage service “which now has 250 million users worldwide” and exabytes of data ready to be pilfered, as The Guardian disclosed.
One document “stated that NSA already had pre-encryption access to Outlook email. ‘For Prism collection against Hotmail, Live, and Outlook.com emails will be unaffected because Prism collects this data prior to encryption’.”
...