Open thread: Keybase's next steps

About a day after I published the post on Keybase and their inaction against spammers on their platform, Keybase put out this blog post, “Dealing with Spam on Keybase.” They outline changes they’ve been working on, including the ability to block and report people, and what they deem the “nuclear option”: the ability to whitelist people you chat with, which they say “breaks Keybase’s social features.”

What do you think of the post and changes in general? In the post, they also answer the question of “Why didn't you do this sooner?” with “There were other things about Keybase holding it back far more than spam. We were working on those things.” What do we think those things are? That’s a lot of questions, so any of them, or your thoughts on Keybase in general, will do for the open thread. :)

As always, looking forward to a good discussion!

Previous open threads:

View 5 comments →

Keybase and the chaos of crypto

A little privacy. Some blockchain. And a lot of anarchy, spam and abuse.

Today, it’s nearly impossible to have a truly private conversation online. Someone is always listening: either the company providing the chat service, or governments (Normcore post), or third parties who want to scoop up data. 

For example, you could try Facebook Messenger. But Facebook automatically scans these messages for abuse and combs through them manually for other, random pieces of information. You could try Whatsapp, but that’s now owned by Facebook which means, in spite of end-to-end encryption, Facebook could potentially still be accessing that content.  (Whatsapp founder Jan Koum specifically left Facebook over privacy concerns.) You could use iMessage, which is in theory secure (if you trust Apple’s newfound market-based commitment to privacy) unless you message someone on Android. Signal is wonderful but a huge headache to use. Telegram is my messenger of choice, but their cryptographic algorithm is still under question

The Origins of Keybase

What to do? In 2015, Max Krohn and Chris Coyne, two of the founders of dating site OKCupid, found themselves wondering the same thing. They left OKCupid and started working on an app that was oriented towards a need for privacy.

When my co-founder Chris Coyne and I were working on OkCupid, we were surprised by how willing our users were to entrust their most deeply personal data to the site. We had good reason to collect that data – so we could match people – but we spent a lot of time and effort thinking carefully about how to best protect it. We knew that privacy would be a growing concern because, at that time, people hypothetically knew that the internet wasn’t secure, but they also widely believed that individuals didn’t have the wherewithal or interest to exploit the insecurities. 

They initially built a directory of PGP keys (cryptographic keys that you can use so that no one can decrypt your communication with someone else) that became popular through Twitter. They then decided to flesh out the concept so that users could also use the platform to have secure, encrypted chats, and so that they could also chat with people who had verified their identities with Keybase through other platforms like GitHub, Twitter, and Reddit. 

Eventually, they built out social features:  

On top of this foundation, we’ve also built encrypted chat, file sharing, and Git on a single platform that works across all of your devices – so, you can think of us as Slack/Dropbox combined and encrypted (or Signal with support for large groups). 

And then the VCs came calling. Andreessen Horowitz invested in 2015. If you’ve been reading Normcore long enough, you know what happens when VCs come around: it’s time to grow quickly or die. VC investment has, so far, ruined Reddit (Normcore post), Twitter (Normcore post), and Imgur (paid Normcore post). That’s exactly what’s happening now with Keybase (with a bonus splash of cryptocurrency fever).

I first created a Keybase account in 2017, when the company was introducing interesting features like Teams (a group chat) and support for exploding messages, which are deleted from the platform after you view them. These are great for sharing pictures you don’t want to become public, or deleting what you’ve said so the internet doesn’t remember your every word. 

The founders were emphatically pro-privacy in the press, and they were introducing cool features. Everything was going great. 

Then Stellar got involved. 

Our Series A was from Andreessen Horowitz in 2015. That money lasted us until about 2018, when we would have done a Series B. But instead of doing a Series B, we were lucky enough to enter into a multi-year partnership with SDF. We’ve been able to make our money last because we try to keep our expenses low on most fronts. For example, we’re in a WeWork instead of building our own space. We spend on hiring and supporting the best people we can find to build Keybase. Chris and I are still very much involved in the day-to-day engineering. *Fun fact: We have only one employee who’s not an engineer or designer. 

Our future plan to monetize is based on a freemium model – similar to how we monetized OkCupid, except Keybase won’t have ads. For the time being, we’re really just focused on getting as many users as possible. When we reach a point when we need more revenue, we’ll start charging our corporate users for certain features. 


Stellar is, essentially, a platform that converts cryptocurrency into money like dollars and Euros, because transferring cryptocurrency to fiat currency like dollars is a complicated process.

If you’re a little hazy on the world of cryptocurrency and blockchain, you’re not alone. Before I did the research for this post, I basically imagined it as an enormous scam glued together by buzzwords. I still do, but at least now I know about the internals. 

It first helps to understand how modern money works, at least in America. Money is, basically, a medium of exchange. You can exchange dollars for a Normcore Tech subscription (cough). Or some ice cream. Or Amazon kipple (Normcore post). And so on. Money has no intrinsic value of its own. But the government backs money, which is why it’s called fiat currency. We are all exchanging money for stuff based on the belief that the US government is stable and will continue to be so. All of the money is issued by the Federal Reserve (and printed by the Bureau of Engraving and Printing), which tracks and controls how much is in circulation. Banks keep the money and verify transactions. This means that any monetary transactions with banks, credit cards, stocks, etc, are tracked by the government and financial institutions. 

Cryptocurrency is, in theory, also money. Or, more accurately, it’s a digital medium of exchange that doesn’t go through any central processing authority. It has nothing to do with the Federal Reserve or any banks. Because there is no central governing body that regulates and tracks it, the transactions are verified by math instead.

This math is the blockchain, also called a ledger. It’s basically a really long and complicated appendable log (Normcore link). The Federal Reserve issues dollars. Cryptocurrency is issued when any given entity decides to give out money, in this case, the Stellar Foundation’s Lumens. As the site says, “One hundred billion Lumens were created the instant Stellar went live, as part of the protocol’s design.”  

You can get Lumens by buying them on cryptocurrency exchanges, like Coinbase. Or, other people can give them to you and you can put them in your crypto wallet. Or, the organization creating the currency can give them to you directly.

That’s all well and good, but how can you transfer the Lumens to something you can use, like dollars or Euros? Here’s where it gets a little complicated. You first have to convert the Lumens to another, more established cryptocurrency, like Bitcoin or Ethereum, and then sell those on an exchange like Coinbase for dollars. (Coinbase just started supporting Stellar in March.) It’s a lot of steps and you’re gonna have to pay taxes on it, which also gets hairy. 

So while Lumens are cash in theory, the steps to get from Lumens to actual dollars are long and varied.


But where did Stellar come from? There are already so many cryptocurrencies out there - why another one? Here’s where it gets interesting. Stellar, both the currency and the foundation, was started by Jed McCaleb, one of the founders of Mt. Gox, a bitcoin exchange run out of Japan. In 2014, Mt. Gox filed for bankruptcy after it was attacked by a third-party. There is a whole fun backstory that you can read in your free time because if I go deep on this we’ll never get to the actual point of this newsletter. 

Ok! So Mt. Gox went bankrupt and McCaleb, a serial entrepreneur, started Stellar, which was funded by the payment company Stripe, to the tune of $3 million dollars. 

The new company’s board consists of ex-PayPal executive Keith Rabois, Y-Combinator’s Sam Altman, WordPress owner and Automattic CEO Matt Mullenweg, and AngelList co-founder Naval Ravikant.

“Stellar’s goal is to build a great transport layer for transmitting monetary value,” Stripe CTO Greg Brockman wrote in the Stellar blog.

“Figuring out how to efficiently move money is something we support very strongly: Stripe spends a lot of effort integrating different banking and finance protocols in various countries (17 at last count).”

Brockman said the Stellar team is bullish about the cryptocurrency space in general and expects multiple systems to prosper.

“When the opportunity arose to help Stellar, we enthusiastically agreed. A couple of months ago, Stripe contributed US$3m to help get the project going. In return, we received 2pc of the stellars.”

(BTW, remember Brockman? He later became the CTO at OpenAI (Normcore post).)

So now, effectively Stellar became a cryptocurrency backed by Stripe, which means that Stripe invested in Keybase. Who invested in Stripe though? Andereessen Horowitz (which, as a reminder, now funds Substack, the platform Normcore is coming to you live from - and which is also powered by Stripe payments), and other VC firms.

So basically, we have Keybase first funded by A16Z, then again by A16Z (but indirectly, through Stripe, by way of the Stellar Foundation). If all of this sounds slightly sketchy and annoying, welcome to the world of VC and cryptocurrency in 2019. 

The Airdrop

How did this impact me? Because not long after Keybase announced their partnership with Stellar (aka funding), they announced something known as an Airdrop of Lumens to all Keybase users

The Stellar Development Foundation and messaging service Keybase announced Monday a 2 billion XLM airdrop, the largest in Stellar’s five-year history, worth roughly $120 million. Over the next 20 months, Keybase users can expect monthly airdrops of 100 million XLM.

Keybase currently has 300,000 active users across its various services, according to the company.

“All you have to do is have an authenticated Keybase account, and your XLM will appear in your wallet – automatically, every month, for as long as the airdrop continues,” according to a draft blog post shared with CoinDesk.

Stellar Development Foundation CEO Denelle Dixon tells CoinDesk that Keybase is a key part of its larger adoption strategy:

“The foundation’s mandate is to deliver lumens [XLM] into the world. This is part of that plan. The up to 2 billion is a really great way. And we will have checks along the way.”

It doesn’t take a cryptography expert to imagine what would happen when you combine what sounds essentially like “free money”, the confusing internals of cryptocurrency, and a messaging platform where you only have to verify your identity using a cryptocurrency wallet: rampant platform spam and abuse. 

I didn’t predict it, but the founders of Keybase should have seen this coming. They cut their teeth on the social internet at OKCupid. They had they seen what being a woman on a social site was like - very hard, with incoming messages often flooding women’s inboxes and overwhelming them. 

They also understood that drastically changing a platform was the Wild, Wild West. You never knew what would happen when you tried something on the internet. Co-founder Christian Rudder (who I was at an event with a couple years back), hosted a quirky and somewhat controversial blog showcasing his learnings about humans after analyzing OKCupid data. (The site was deleted and later resurrected.) One of the articles read,

But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out.

But somehow they didn’t understand that crypto would lead to chaos.

The Aftermath of the Airdrop

The way the Airdrop was supposed to work was this: First, Keybase users received a surprise airdrop, which ended up being the then-equivalent of $20 in Lumens; after that, everyone who joined the Keybase platform would receive something like the equivalent of $40 if they signed up and connected to an existing GitHub or Hacker News user account. And they would continue to receive Lumens every month thereafter. 

Even after those major caveats and theoretically some level of airdrop design, the Keybase platform exploded in pandemonium. 

First, a series of bad actors tried to game the system by brute-forcing both Hacker News and GitHub, and Keybase had to stop qualifying new users based on those accounts,

UPDATE #2 September 16: HackerNews and GitHub were both under attack by bad actors who hoped to game this by stealing other users' accounts. In cooperation with both of those sites, we've halted Space Drop registration for anyone who wasn't a Keybase user before Sept. 9. SDF and Keybase will discuss, while whining and moaning, how this giveaway can continue. (COME ON PEOPLE OF THE INTERNET, STOP SUCKING SO MUCH!)

Then, people overwhelmed Max and Chris with questions about the Lumens. 

And, while they were firefighting, the enormous influx of new users meant that current users started to get a lot of spam related to bad actors coming onto the platform - a kind of very horrible eternal September. All of a sudden, I started receiving lots of random messages from people without profile pictures. 

These were people I didn’t know contacting me on a pretty frequent basis, and I had no way to opt-in to their messages. Pretty ironic for a platform built on privacy. 

I wasn’t the only one. After asking around, it seems that everyone I knew on Keybase was getting these. But of course, women with female names and profile pictures got a lot more than most men I knew.  Other women online were experiencing it, too:

Meanwhile, I was trying to do some investigative journalism and find out why randos were sending me messages. As you can imagine, it didn’t go too great. 

The most interesting chat was this one, where I realized that people were finding me from the “Consider following” tab on Keybase, which as far as I can tell from the Keybase code on GitHub comes from suggested contacts based on the social networks you’re connected to. Users had previously asked Keybase to disable this feature

For women in particular, this kind of encounter is accompanied by with wariness and a sense of invasion of personal space, especially in an online world where they are already constantly wary of possible harassment and stalking. 

And, not only were people getting direct messages, but they were getting added to random teams of strangers. As a result, people were asking Keybase for a way to proactively report and block people, both on Twitter and through GitHub issues. Meanwhile, there was, throughout all of this, no response from Keybase when people contacted them directly. 

In fact, someone had already done some sleuthing on this issue and written up a fantastic blog post, with meticulous documentation, on the teams issue. He had contacted Max and Chris directly and gotten this in response: 

The reply I received from Chris Coyne on November 13th (Highlighting by me):

Hi Dave - thanks for reaching out in a responsible way with what you see as a security issue.

We currently see team additions as analogous to email thread additions or phone messaging — lowering the friction makes for easier/healthier group forming, and if you’re not interested you can jump out. We’re actually rolling out shortly some newer/better tools around dealing with this, so it will be very obvious to you that you can choose not to be in the team upon being added….and it will therefore feel a bit more like an invitation.

We’re constantly revisiting this and we might also add an advanced setting that lets users specify rules around themselves getting invited/added to convos.

It’s all well and good for Keybase to suggest that lowering friction creates more rainbows and world harmony. In the meantime, the women I know on Keybase are having to do the very manual process of blocking people, leaving group chats they didn’t want to be part of, and changing their profile pictures so it’s not obvious that they’re women.

What to make all of this? 

Let’s put aside the issue that neither of the founders has ever had the experience of being a woman online and understand the impact of this kind of behavior directed at them. And that they either naively, or deliberately, do not see this as a problem (similar to Zuck’s constant and extremely naive argument that a default-open world is always a good thing). It’s an important issue, and one that deserves a post of its own. Because, as we already know, Silicon Valley (and every other industry in America)  is terrible at making products for women

Let’s take a look instead at the root of the business problem. From a Normcore perspective, it’s alarmingly easy to understand what’s going on here. Keybase, once they were funded by A16Z, was given a mandate of hypergrowth and gaining users as fast as possible. Now that it’s funded by Stellar, it’s obligated to get more users on the Lumen cryptocurrency platform. Where A16Z left off, Stellar is now at the reins. Stellar’s perspective is simple: The more people that use their cryptocurrency, the more credibility there is, the more it’ll rise in the cryptocurrency rankings to be as legitimate as Bitcoin (which is the cryptocurrency probably everyone knows). Stellar doesn’t care whether these people are harassing anyone. All it cares about is activity on the Keybase platform, and, eventually, on Coinbase. What’s a better way to have people find out about your platform than to offer what people interpret as free money through airdrops?

Keybase, as of September, had 300,000 active users. Its Airdrop page now says it’s up to almost 300,000 registered for this current Airdrop, making the active at least userbase 600k. That’s an increase of 200% over the course of a couple months. That’s one way to cash in on the Stellar Foundation money and ensure more people are talking about and using Lumens. It’s also a really fast way to lose control of your userbase.

Second, from Keybase’s perspective, the company wants to make it easy to talk to people, regardless of who those people are, because that also draws people to the platform. Enabling blocks and checks enables friction, which they don’t want. They said as much in the reply to the incident report. In this way, even though Keybase is a platform that bills itself as a privacy play, it’s absolutely no different than Facebook or Twitter, which also struggle with the balance between allowing harassment and allowing their userbase to grow. 

The founders of Keybase have been focused on getting the technical parts of encryption, cryptowallets, and the like right. But they’ve missed the bigger point: every technology platform is eventually ruled by social rules just as much or even more than by code. Eventually, once they’re large enough, platforms become more than just pipes, and the creators of those platforms need to decide whether to take responsibility for the content on them. As every Normcore reader knows by now (rolling their eyes, waiting for me to say it),good things don’t scale. As soon as you can’t or are unwilling to police a growing social platform, you can’t enforce the quality.

Being at the intersection of economic incentives and societal internet problems is a very bad time and very difficult to reconcile. It’s no wonder that, instead of addressing this, Keybase is focusing on airdrops. 

In the meantime, the amount of abuse on the platform continues to grow, with an entire thread full of people angry about spam and harassment and now leaving the platform.

In the long-term, Keybase founders will have to decide what’s more important to them: a heavily monetized and growing, but anonymous base of cryptocurrency users (maybe including some of the same kind of bad actors that eventually brought down Mt. Gox), or a maybe less profitable but privacy-forward chat that actually is a healthy platform for all users.

It seems like, for now, they’ve chosen door A.

Art: Rosalyn Drexler, Marilyn Pursued By Death (1967)

What I’m reading lately 

  1. Teaching about cryptocurrency in North Korea? You hate to see it

  2. Python’s source code

  3. Please quote this tweet with something everyone in your field knows”

  4. Zuck interviews economist Tyler Cowen and Stripe’s Patrick Collison 

About the Newsletter

This newsletter is about issues in tech that I’m not seeing covered in the media or blogs and want to read about. It goes out once a week to free subscribers, and once more to paid subscribers. If you like it, forward it to friends!

Select previous free Normcore editions:
What’s up with Russia’s Internet· I spent $1 billion and all I got was this Rubik’s cube· Die Gedanken sind frei · Neural nets are just people· Le tweet, c’est moi· The curse of being big on the internet· How do you like THAT, Elon Musk?·Do we need tech management books? ·Two Python Paths
Select previous paid Normcore editions:
Imgur is bad now · Eric Schmidt and the great revolving door· No photos please · Deep thoughts of Cal Newport

About the Author:
I’m a data scientist in Philadelphia. Most of my free time is spent wrangling a preschooler and an infant, reading, and writing bad tweets. I also have longer opinions on things. Find out more here or follow me on Twitter.

We're still in the steam-powered days of machine learning

ML platforms are complicated, unique, and, so far, hard to reproduce

The reveal of the ridiculous Cybertruck design last week made me curious about the history of cars. If you look at pictures of cars from the early days (as I, a Normal Person, did last Friday night), you’ll see some insane ideas. Before we got to the Ford Model-T that standardized car production, people iterated on a ton of crazy stuff. For example, here is a steam car from the 1700s:

By Photo et photographisme © Roby. Grand format sur demande - own work. Avec l'aimable permission du Musée des Arts et Métiers, Paris., CC BY-SA 3.0,

Here’s a steam-powered bus.

Here’s an electric-powered tricycle:

And here is the creme de la creme, the Oruktor Amphibolos, hailing from my own fair city of Philadelphia,

built by Evans for the Philadelphia Board of Health as a solution to the Board’s concerns about dredging and cleaning the city’s dockyard and removing sandbars. It could drive along on four wheels or steam down the river using its rear-mounted paddle

It took some time for people to experiment and agree on what a car even was, what features it had, and how it needed to work. For example, for a long time in the beginning, quite a few cars ran on steam, until gasoline began to overtake them (thanks in part to Henry Ford’s standardization of the assembly line, which made non-gasoline cars harder to produce. ) Eventually, all the cars standardized to the form we know today: a closed car, powered by gasoline, with four wheels, four windows, seating 4-8 people. No paddles. Even the godawful Cyberthing follows this model.

Today, when we talk about modern technologies, machine learning comes up pretty frequently as the future-iest of them all. It’s often seen as an exciting (ahem) vehicle to move people to the future - to predict cancer, to improve prisons, to iterating on French fries.

But, we are still very much in the weird, early steam-powered tricycle days of machine learning.

What prompted me to think about this was that earlier this month, Lyft released an open-source machine learning platform, called Flyte. Flyte is a an:

end-to-end solution for developing, executing, and monitoring workflows reliably at scale.

Let’s back up a step.

What do we mean by machine learning platforms? Simply put, they’re a systematized, automated way to take raw data, transform it, run a model on it, and show some sort of results that lead to decision-making, either for internal or external customers.

Let’s start with some examples.

Serving Netflix recommendations is a machine learning platform. The Facebook News Feed incorporates machine learning. Amazon recommendations. On and on. Anything that serves you personalized content online is (likely) powered by some kind of machine learning.

On the webpage, the creators of Flyte say that it currently powers Lyft Pricing, which generates the ride price you see when you pull up the app. For example, here’s a price estimate from me to Suraya, a very tasty Middle Eastern restaurant in Philadelphia that has delicious hummus, but almost zero parking spaces nearby.

When I pull it up, The price the app gives for a single-passenger Lyft is $17.56. It’s just a single number. But to pull that number and display it the moment I need a Lyft takes a lot of work.

To start with, let’s think about some of the things we’d need to get an accurate price estimate for an on-demand car, given that we know where all the other cars in the app are (actually, a variant of this is probably an interview question that Lyft asks).

Here are some of the things that go into figuring out what is known in data science lingo as dynamic pricing. Lyft’s own site says:

Our goal at Lyft is to connect you with best-in-class service at an affordable price when and where you need it. To achieve this goal, prices for rides are dynamically calculated based on a variety of factors including route, time of day, ride type, number of available drivers, current demand for rides, and any local fees or surcharges.

So here we have a bunch of variables that we need to collect to make a model:

  • time of day

  • length of route

  • ride type

  • number of drivers available

  • current driver demand

  • local fees

And here’s what else I think goes into it (I’m completely spitballing here based on my previous transportation/data science work. Lyft people, feel free to email me and tell me I’m wrong (and maybe send a free coupon for 15% off my next ride):

  • geographic location (Philadelphia or Pasadena?)

  • difficulty of route (Highway or city traffic?)

  • weather

  • Some other things that Lyft may use internally but won’t tell you about because it might get bad press (i.e. driver rating, rider rating, how many Ubers are potentially in the area, and more. )

Collecting all this data is not easy, particularly when you need to update it in real-time. So Lyft might have a graph database that has calculations of coordinates and distances, driver data, current driver demand and rider data. It needs to get all of this data into one place, crunch it through an algorithm that generates a price, and spit that price out to the front-end web app.

This is a really, really simplified view of what’s going on (and almost completely illegible, but hey, they pay me to build these platforms, not draw them)

We have the initial data sources: a graph database updating driver location among route ride segments, maybe a graph database of pricing based on those route segments, a table full of driver data (ratings, reviews, car make, etc.) and one of rider data (ratings, etc.).

All of this is an enormous amount of data and needs to be consolidated and moved into one place. Some of it will be streamed through pipelines like Kafka, some will be through web service requests, some will be loaded directly from a database.

All of this data is then aggregated in a centralized machine-learning store, which then generates a clean dataset for modeling. That dataset is plugged into a working statistical model, which trains and picks the most important features (the variables that make price go up or down), tunes hyperparameters (the metadata about the model, such as how many times you want to run it), and, as a prediction, spits out the number $17.56.

That number then needs to travel via to the iOS app, where we, the end consumer, finally see it.


That’s a lot of work. And, what I’ve described is basically an idealized state, where you have everything functioning. If you really want to torture yourself, do an image search for “machine learning platform architecture” and see how complicated some of the architectures are.

In reality, there are so many problems that can arise in machine learning platforms. This Google paper, “Machine Learning: The High Interest Credit Card of Technical Debt” — which is a classic and I think everyone working with data should read — describes many of them.

Just look at Adam’s tweet creating an idealized outline of a book that would cover all the steps of getting machine learning into production - look at how many things can break!

Data Problems

First, your data sources might start producing incorrect data. Or, the data might be dirty: for example, what if your system has Philadelphia as Phla, or Phila, or Philly? Or all four! How do you tie that to other data? Your data might be in different formats: Some in JSON from web services,some in CSV outputs from SQL query results, some in serialized, compressed formats like Avro which then needs to be converted to Parquet, and on and on.

Remember that data cleaning makes up the majority of data work.

Model Problems

Then, you might get something called model drift:

Nothing lasts forever. Not even a carefully constructed model on mounds of well-labeled data. Its predictive ability decays over time. Broadly, there are two ways a model can decay. Due to data drift or due to concept drift. In case of data drift, data evolves with time potentially introducing previously unseen variety of data and new categories required thereof. But there is no impact to previously labeled data. In case of concept drift our interpretation of the data changes with time even while the general distribution of the data does not

Or, you might have incorrect model feedback loops, which is what happens when you end a statistical experiment too early and come to the wrong conclusion.

Or, perhaps the model is the wrong one for the shape of the data and you need to cycle through a couple different options.

Software Problems

Then, on top of the data issues there are your garden variety software engineering issues: data not getting ingested quickly enough, data feeds breaking, data incompatibility issues between formats, data warehousing problems, web service connection issues, configuration issues, and if you’re on a cloud platform, firewall and permissions issues between all the ways these services connect.

(Most of my career in machine learning so far has been spent fixing data compatibility and connection issues (and troubleshooting AWS Lambdas.))

I’ve only really scratched the surface here.

Today’s ML Platforms

All of these issues is something the concept of a machine learning platform hopes to solve. What you’re doing when you develop a platform, is you, in theory, wrap all of that stuff together into a single system where compatibility is not an issue and there is no messy glue code.

Essentially, you’re taking all the parts you need for the car to function - tires, windows, combustion engine, windshield wipers, and putting it together into the monstrous Cybertruck form.

Today, there are lots of machine learning platforms that are popping up aside from Flyte. Uber, Lyft’s direct competitor, has Michelangelo. Facebook has FBLearner Flow. Databricks has MLFlow. There’s Sagemaker on AWS, Azure, BentoML, and CometML and many, many more.

Some, like Michelangelo, Stripe’s Railyard, and Twitter’s Cortex, are proprietary. Some, like BentoML and Flyte are open-source. Some, like CometML are SaaS products, some, like Flyte are self-hosted. Some focus on the actual modeling process, some focus on hyperparameter tuning, some focus on algorithms, some focus on data ingest and cleaning, some focus on visualization.

There really is no good, generalized single system of best practices for creating machine learning platforms. There’s not even a textbook on it. (Other than Designing Data-Intensive Applications, which is good but doesn’t necessarily fit the bill.)

Much like the steam-powered tricycle, each machine learning project is beautiful and weird and inefficient in its own way. Every company has its unique set of needs, stemming from the fact that each company’s data and business logic run differently. In a recent Hacker News threads on how models are deployed in production, there were over 130 different answers, ranging from using scikit-learn, to Tensorflow, to pickling objects, to serializing them, to AWS Lambdas, to Kubernetes, to even C#.

The other amazing thing about machine learning platforms is that none of this stuff is new: people have been trying to build machine-learning type cars for over 20 years now.

Just take a look at this Usenix paper FROM 2002, trying to analyze logs:

Server systems invariably write detailed activity logs whose value is widespread, whether measuring marketing campaigns, detecting operational trends or catching fraud or intrusion. Unfortunately, production volumes overwhelm the capacity and manageability of traditional data management systems, such as relational databases. Just loading 1,000,000 records is a big deal today, to say nothing of the billions of records often seen in high-end network security, network operations and web applications. Since the magnitude of the problem is scaling with increases in CPU and networking speeds, it doesn't help to wait for faster systems to catch up.

This paper discusses the issues involving large-scale log management, and describes a new type of data management platform called a Log Management System, which is specifically designed to cost effectively compress, manage and analyze log records in their original, unsummarized form.

We’re not any closer to a complete, efficient, generalizable solution now (although we sure are using a lot of Kubernetes these days, including in Flyte). Yet.

So what?

All of this is to say, these systems that are serving us music recommendations, books we’d like to buy, News Feed items from friends, Airbnb, whatever, are still sometimes very fragile, very brittle, and very much experimental (although there are some, as Normcore readers can attest, that are generally very good.)

What does this mean? It means, first, don’t believe the hype that any one single machine learning platform will solve your needs. And second, be very, very careful with the high-interest credit card of technical debt: machine learning. Make sure that if you or your company has or is thinking about such a system, that it’s very carefully monitored, tagged, and that results are constantly evaluated, and that it has tests.

And, if you’re a consumer, think about all the stuff that has to happen for a recommendation or action to make its way to you, and be skeptical. The system is not always right. We are still very much in the early, experimental days of what machine learning platforms need to be, and don’t let anyone tell you different.

But, does this mean you should panic and that machine learning is worthless? Also absolutely not. (This is Normcore, after all.) There are plenty of platforms that work well, that are stress-tested, and reasonable. They just probably won’t work exactly the same way for your company.

But we as an industry need to keep trying, experimenting, and, most importantly, documenting and sharing these systems, how to run them, and their results publicly.

Because, if we don’t iterate, how else are we ever going to get from a steam-powered paddleboat, to a Tesla?

Art: Car Clothing, Dali, 1941

What I’m reading now:

  1. Love, love, love this article explaining “moral grandstanding” - why social media is toxic

  2. The data enrichment industry, if you want to depress yourself this Thanksgiving.

  3. Everyone should learn Latin (not biased at all here, having taken 4 years of it.)

  4. Am I the asshole for regifting my kids their tablets for Christmas?”

  5. Delivering cows in Yakutia

  6. How do you design a social media site to avoid misinformation?

  7. Best 2019 Thanksgiving tweets

About the Newsletter

This newsletter is about issues in tech that I’m not seeing covered in the media or blogs and want to read about. It goes out once a week to free subscribers, and once more to paid subscribers. If you like this newsletter, forward it to friends!

Select previous free Normcore editions:
What’s up with Russia’s Internet · I spent $1 billion and all I got was this Rubik’s cube· Die Gedanken sind frei · Neural nets are just people· Le tweet, c’est moi· The curse of being big on the internet· How do you like THAT, Elon Musk?·Do we need tech management books? · Two Python Paths
Select previous paid Normcore editions:
Imgur is bad now · Eric Schmidt and the great revolving door· No photos please · Deep thoughts of Cal Newport

About the Author:
I’m a data scientist in Philadelphia. Most of my free time is spent wrangling a preschooler and an infant, reading, and writing bad tweets. I also have longer opinions on things. Find out more here or follow me on Twitter.

Loading more posts…