I think we can now confidently say we’re now at the point where Big Tech needs to be seriously regulated. Facebook is in the hot seat right today, but as New York Times research into Amazon and Google patents show, there’s plenty more to be nervous about.
A scene from a future Amazon hopes to bring about, as described in a November patent:
You’re taking to a friend on the phone or “within a detectable distance of [a] device” like Alexa. You say
The vacation was wonderful. I really enjoyed Orange County and the beaches. And the kids loved the San Diego zoo.
The device is running a “sniffer algorithm” that’s looking for “trigger words,” “often a verb indicating some level of desire or interest” such as like, love, enjoy, downloaded, hate, returned.
For each identified potential trigger word, the device can capture adjacent audio that can be analyzed, on the device or remotely, to attempt to determine one or more keywords associated with that trigger word.
So when you said “loves,” it triggers the device to analyze your conversation and extract the words “San Diego zoo.” It’d does the same thing for your friend, who replied,
When we went to Southern California, I fell in love with Santa Barbara. There were so many great wineries to visit.
So at the end of the call, your device could send you a discount for a San Diego season zoo pass and her a wine of the month discount. And all of the info about these keywords, including the context of the call, could be stored as part of a profile of you and your friend.
Google’s even more ambitious. They don’t just want to extract info from every conversation you have, they want every drop of info they can get. In their September 2016 patent about a home monitoring system, they envision the following, either in a homeowner’s— or renter’s— home could be closely monitored. According to the patent,
an audio signature matching a dining chair movement across a floor may suggest that an occupant is sitting in the chair (e.g., because the occupant may have presumably moved the chair to sit in it). Indeed, video inputs may confirm and/or identify that occupants are sitting in the chair and/or at the table. Additionally, smart device inputs may be used to obtain a number of contextual clues, such as utensil movement, conversation content, vapor detection, etc. For example, in one embodiment, the vapor sensors may detect the presence of food within the dining room zone, which may indicate that a meal is being consumed in the dining room….. an audio signature of keyboard clicking, a desk chair moving, and/or papers shuffling etc. may indicate that someone is working…. an audio signature and/or video signature may be associated with the sounds and/or images of teeth brushing in the zone 618. Next, additional characteristics may be determined (e.g., the sink being left on, a duration of teeth brushing, a speed of teeth brushing, etc.). These findings may me reported and/or recorded within the system (e.g., for subsequent control and/or reporting by the household policy manager [program ]) .
And all of this info could be sliced and diced by demographic info.
Demographic information may include, for example: occupant information such as: number of occupants, gender of occupants, age of occupants, ethnicity of occupants.
And the techniques for gathering this data could be pretty sophisticated.
By way of example, a video monitoring camera placed in the kitchen of the home can perform image processing on several days or weeks worth of captured data to determine how many different individuals it sees on a regular basis, to establish how many occupants live in the house.
What could be done with all this data? Well, the patent suggests, you could use it to identify children’s behavior.
For example, characteristics of audio signatures, such as speech patterns, pitch, etc. may be used to discern child occupancy. Next, the occupants may be monitored, specifically listening for low-level audio signatures (e.g., whispering or silence), while the occupants are active (e.g., moving or performing other actions). Based upon the detection of these low-level audio signatures combined with active monitored occupants, the system may infer that mischief (e.g., activities that should not be occurring) is occurring….. For example, it may be expected that certain activities be performed in quiet, thus indicating that the quiet activity is unlikely to be mischief. For example, reading a book, mediating, etc. are oftentimes performed in quiet…..
For example, audio monitoring, optical monitoring, infrared monitoring, etc. may be used to discern occupancy and undesirable activities of the occupants. In one embodiment, the contextual data may include audio signatures indicating “bully” keywords such as derogatory name-calling, elevated voices, etc. Accordingly, the system may monitor for and detect the use of such “bully” keywords. Additionally, in some embodiments, the contextual data may include audio signatures indicating the use of foul language.
And not just kids.
returning to the “I’ll be home by 5:00” assertion, the system may determine a current location of the household member and how long it would take to get from the current location to the occupant’s house (block 394 of FIG. 46). If the occupant’s house can be reached by 5:00, a determination is made as to whether the assertion/promise has been met (decision block 356 of FIG. 45 and decision block 400 of FIG. 46)…. If the assertion has been met, routine monitoring proceeds (block 358 of FIG. 45 and block 402 of FIG. 46)…. If the assertion/promise cannot be met (e.g., the occupant’s house cannot be reached by 5:00), a finding that the assertion cannot be met may be reported and/or recorded (block 360 of FIG. 45 and block 398 of FIG. 46).
What an abusive spouse — or an authoritarian regime — could do with this kind of automated micro surveillance is not a topic covered by the patent.
The system could also be used to monitor group behavior:
For example, if the household indicated a goal to spend more time doing activities together and the system monitoring indicates that the household is spending less time together or marginally more time together, the system may provide a reminder of the household goal to one or more members of the household (e.g., via an audible and/or visual alert in the household, via a text message provided to the user’s smartphone, etc.)…. If sufficient progress toward the goal is attained, a progress reward may be provided to one or more members of the household. For example, in the togetherness goal mentioned above, if the family spends 20 additional minutes together in a week, when the goal is to spend an additional hour together a week, a progress reward may be provided to the family.
And you could use it to see if you’re keeping up with the Joneses.
In one embodiment, household bragging rights may be a reward, by providing a neighborhood message to other participating households, stating that the household is progressing towards and/or has attained the particular goal.
These visions of a world in which Big Tech has the ability record any scrap of info about us aren’t from some random email an idiot marketing manager sent to their boss or videotaped off the cuff remarks at some conference. At least a few people at Amazon and Google thought they were nifty enough to apply for patents. That’s not remotely the same as official business plans. But the fact that there’s a corporate culture at places like Amazon and Google where these public legal documents would be considered OK should give us pause.
There are a lot of great, caring people who work at Big Tech companies who’d be appalled at the iPanOpticon sentiments behind these patents. But given the resources, power, and reach that Big Tech now has, we need to make damn sure that while they “move fast and break things” they don’t end destroying our liberty and our society’s future.