Normcore Tech

Share this post

The data lake overfloweth

vicki.substack.com

The data lake overfloweth

Delete this tweet, actually all of them

Vicki Boykis
Jul 17, 2020
9
3
Share this post

The data lake overfloweth

vicki.substack.com

Art:The Second Skin, Friedensreich Hundertwasser, 1986

A while ago, Tim created a Twitter bot called Your Old Tweets (sadly no longer in service due to something Twitter claims was a spam policy violation) that would search through your tweets on a daily basis and fetch everything you’d tweeted on that same day every year you were on Twitter. 

It was wonderful because it allowed me both to reminisce, and to delete all the stupid crap I’ve tweeted since 2009. 

My Old Tweets

Twitter in 2009 was a wild and small place, entirely full of strangers using their real names to tweet stuff like what the weather was like in Washington, D.C. Back then, it felt more like hanging out your front porch and chatting with the neighbors.

But today, due to context collapse and the size of the platform, as well as the inherent conflict and animosity built in,  it feels like every tweet is a minefield. Amplifying all of this are the social and economic dynamics of cancel culture (if you only read one thing about it, make it this one.)

It reminded me of this amazing 2003 paper/talk that I came across just recently, a group is its own worst enemy, which I think everyone doing anything online should read,  where Clay Shirky argues that we are still in the very early stages of understanding social software, and that in any social software setting, people will follow the rules of people in a group. 

And people in a group both act in their own self-interest, and in that of a forming group. Shirky lists several behaviors, based on analyzing previous real-world psychology studies, that people in a group will always engage in once the group forms, that thwart the forward momentum of the group: 

  • the group conceives of its purpose as the hosting of flirtatious or salacious talk or emotions passing between pairs of members

  • The group starts identifying and vilifying external enemies (“Nothing causes a group to galvanize like an external enemy.” )

  • And 3, the group starts engaging in religious veneration—the nomination and worship of a religious icon or a set of religious tenets. The religious pattern is, essentially, we have nominated something that’s beyond critique. (i.e. “Lord of the Rings cannot be criticized in a Lord of the Rings fan website”)

Once you know these patterns, it’s so obvious that Twitter, and really all social platforms these days, is meant to amplify all of them, which makes it impossible for cohesion, which means there is the possibility that every single thing you tweet will be controversial to someone.

Let’s face it. Social online is just plain hard, and I even though I very rarely agree with pg anymore, I  agree with him on this one: 

Twitter avatar for @paulg
Paul Graham @paulg
@jesslivingston Here's a little known fact about the history of Y Combinator. The single biggest source of stress, for me at least, was not picking startups or advising them or Demo Day or even fighting with people on the startups' behalf. It was running HN. Don't start a forum.
8:43 PM ∙ Jul 11, 2020
1,082Likes91Retweets

Having thought long and hard about this, and inspired by Chris’s tweets, I’ve been having this idea that I want to delete all my old tweets. 

Previously, I was doing this manually. Every time Your Old Tweets would show me the day’s retro, I would go through and delete anything that I thought could remotely be considered controversial, insensitive, or just plain dumb today. Every day for the better part of two years, I read and cleaned up my old tweets, agonizing over what to keep and what to delete. 

Because these tweets have been my diary. It’s sad, but those tweets are the main markers I have of being young and just-married and living in Washington, DC, what I cared about then, what I thought, and also of what Twitter was back in 2009, which was a nice, small chatroom for you and your closest online friends (doesn’t matter that most of my tweets had zero interaction.)

It’s also fun to see my programming skills progress throughout the years. 

The Past is a Foreign Country Made of JSON

But no matter how much I love coming back to these, more and more, I’ve been thinking that our past is, particularly in today’s climate, an enormous liability. There is a quote in Outlander, one of my absolute favorite books,  “The past is a foreign country. They do things differently there.” And there is no way I can compare the online atmosphere in 2020 to what it was like in 2009, which is radically different than what it was in 2000. 

The only thing that is the same is our streams of log data, quietly following us around and changing its context throughout time so that at times it might be benign, at times it might be extremely worthy of cancellation. 

So I downloaded my Twitter archive (which you can also do pretty easily!) and started looking through it to get an idea of how to get rid of it. What you get is a WHOLE bunch of stuff. 

Twitter avatar for @cabel
Cabel @cabel
Holy smokes — Twitter Data Export has been massively upgraded. In January I exported my data: 42MB. Today, I get a 3.03GB (!!!) json-based archive of everything — followers, DMs, and most importantly, ALL MY MEDIA! Photos and videos! THIS IS VERY GOOD 🎉
Image
3:03 AM ∙ Apr 26, 2019
1,531Likes369Retweets

There is a very friendly README file that tells you everything included in the download, and then you are on your own with .js files. 

I won’t bore you with the technical details, although I probably will do a separate technical post on this stuff, but basically, in order to access your archive, you need to understand: 

  • HTML

  • Javascript variables

  • JSON

  • How a README works

  • And a lot of terminology around tracking web events: 

Twitter avatar for @vboykis
Vicki Boykis @vboykis
Playing with my Twitter archive (which is really well documented and good!) It's wild how much you have to know to have any understanding of your data online. A layperson who wants their tweets has to parse JSON and understand that "inference" is a fancy word for ad targeting.
Image
2:00 AM ∙ Jul 13, 2020
68Likes5Retweets

And, further, to truly have fine-grained control over delete your tweets, you need to either use Chris’s script or build your own app. Either way you need to: 

  • Know a general-purpose programming language with access to Twitter’s API

  • Understand secret tokens

  • Understand API throttling

  • Have some way to test tweet deletion before you do so

  • Have access to GitHub

For me, all this stuff is as familiar as breathing air. Logs, JSON, parsing data, deleting embarrassing joke content, etc. 

Twitter avatar for @vboykis
Vicki Boykis @vboykis
JSON Derulo.
6:00 PM ∙ Jun 5, 2012

The Data Lake is Full

As I was researching all of this, a couple thoughts came to me. First, only people who work in software or know someone who does can go about doing this (unless you use some website to delete your tweets, but the website might not give you the fine-grained control you want). 

And second, that, for most people swimming in the enormous cesspool that is the data lake of our digital lives today, realistically, it’s easier to work through security by obscurity than to have to curate all of this stuff. Security by obscurity is a concept from..you guessed it..computer security. The basic theory behind it is that if you hide your most important stuff out in the open in an unassuming way, it only becomes vulnerable when someone realizes that it’s important, but they have to dig through all the content to find that thing.

This is where we all are now, I think: figuring out which content to hide and which content to keep, which content is a liability and which content is an asset. And even if we’re not thinking about it,as most people are not, we experience it every time there is, say a security breach where there could be a leak in tweets or private messages, or data breaches in systems of companies like Marriott or Anthem or LinkedIn or Quest Diagnostics or Labcorp. Or…

And, I was just taking a look at my tweets. But what about my Facebook posts? All of my Google activity (which you can also download through takeout, and which Sundar now says we can delete, in theory )

Twitter avatar for @sundarpichai
Sundar Pichai @sundarpichai
Privacy is at the heart of everything we do, and we’ll keep challenging ourselves to do more with less. As a next step, today we’re changing our data retention practices to make auto-delete the default for our core activity settings.
blog.googleKeeping your private information privateUpdates to our privacy and security work, including changing our data retention practices for our core products to keep less data by default.
4:56 PM ∙ Jun 24, 2020
5,303Likes783Retweets

All of my cell phone data that’s since been passed countless times to marketers?  All of my medical data, floating out there somewhere in the ether? All of my chats? My private messages across 10-11 different social platforms and 10+ years?  Just as I squash one thing, it will be impossible to take care of the rest of them, and you and I and all of our friends and family will keep generating more data until we flip the Big Bit from one to zero, so to speak. 

I could go on and on and on. But, it’s impossible to delete and control all of this stuff and logs are a liability, and they’re growing exponentially. 

The easiest way to mitigate this today is, if you can’t delete it, generate a lot of data, until either (hopefully) regulations start telling companies not to collect more and we become more empowered over our own data, or until the log-filled waters of the data lake overflow. 

What I’m reading lately:

  1. Reading recs:

    Twitter avatar for @mims
    Christopher Mims @mims
    Good morning! I feel like the thing so many of us need more than ever is *perspective.* The sort that comes from putting down our phones and journeying inward. Or, that old shortcut: reading a good book. So I'm going to start a mega-thread of books sustaining me in this dark time
    12:17 PM ∙ Jul 9, 2020
    209Likes23Retweets
  2. Even more reading recs

  3. Tech interviews are bad and cause anxiety

  4. Has anyone ever used clustering?

  5. Really great replies here:

    Twitter avatar for @jaltma
    Jack Altman @jaltma
    What’s something your parents did that you think was important for your development?
    4:12 AM ∙ Jul 10, 2020
    4,368Likes559Retweets
  6. This is a really nifty video and you don’t even have to know much about ML:

    Twitter avatar for @criticalneuro
    Ida Momennejad @ICML @criticalneuro
    Teaching tensors of different ranks and basis sets take time, you always need to use visual aid to teach how to imagine it. I usually draw book shelves to visualize it. This video does a better 3D visualization job, especially the last 3.5 minutes. youtube.com/watch?v=f5liqU…
    Image
    2:08 PM ∙ Jul 10, 2020
    152Likes21Retweets

The Newsletter:

This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out once a week to free subscribers, and once more to paid subscribers. If you like it, forward it to friends and tell them to subscribe!

Swag: Stickers. Mug. Notepad.

The Author:
I’m a machine learning engineer. Most of my free time is spent wrangling a preschooler and a baby, reading, and writing bad tweets. Find out more here or follow me on Twitter.

3
Share this post

The data lake overfloweth

vicki.substack.com
3 Comments
Chris Billingham
Jul 18, 2020Liked by Vicki Boykis

It's interesting as I was using "My Old Tweets" bot to do exactly the same thing (over a year rather than two) and did enjoy the very slow drip drip of removing things a tweet or two at a time whilst reading someone of my less filtered nonsense from the beginning to time (2009). A friend of mine came to me the other week and asked how to mass delete a bunch of old stuff, my retort "You can either slowly do it over a year, or learn to program". The conversation ended at that point.

Expand full comment
Reply
1 reply by Vicki Boykis
Benjamin Rouif
Jul 19, 2020

Great post ! Did you hear about Jumbo they have a very interesting approach to data privacy and helping people to manage better their social media history ?

BTW Spotify is also sharing inferences when you ask for your data and I had to explain to my parents what inference mean.

Expand full comment
Reply
1 more comment…
TopNewCommunity

No posts

Ready for more?

© 2023 Vicki Boykis
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing