Broadly speaking, platforms exist on a scale from Netflix to Discord. That is, completely entertainment-based to completely connection-based.

Two platforms might have all the same features — user generated content, liking and following, an algorithmic feed — yet work very differently in social terms. The key difference is something like: whether or not the platform succeeds in building new relationships between people.

Many, many people have found friendship and community on X/Twitter (shout out to TPOT). But in the last few years there’s been a vibe shift. The place has felt less friendly, less community-oriented. It’s hard to know if this is just me and my network, or if it’s a more general effect, but if it’s true, what might be causing it? Even better, how would we design a social media algorithm that is truly social?

What I’ve learned from reading the X source code, and from comparing it to the previous architecture, is how two algorithms could feel very different, even if they are optimizing for almost the same thing.

Fewer friends in For You

I’ve been trying to understand my different, less friendly experience of X/Twitter over the last few years. I’ve been on the platform since 2008 and have built up a large network of people I follow, so the first thing I wondered is how often posts from these people appear in my algorithmic feed.

This is the sort of thing I used to write code for, but instead I found a simple prompt for Claude Cowork that used my browser to figure out the percentage of posts in my For You feed that come from people I follow. This turned out to be 16%. I convinced two other people to try it, and they got 5% and 11%. (Try it yourself and leave a comment below!)

So, something like 10% of your For You feed is from people you follow. But it wasn’t always thus. Twitter told us in 2023,

We find candidates from people you follow (In-Network) and from people you don’t follow (Out-of-Network). Today, the For You timeline consists of 50% In-Network Tweets and 50% Out-of-Network Tweets on average, though this may vary from user to user.

Whatever else has changed, the fraction of posts in the For You feed from people you follow seems to have declined precipitously. This may be an intentional parameter change, or it may be the result of optimizing the fraction of in-network content so as to maximize engagement. Either way, you won’t see much from who you follow on the For You tab.

This follows a general trend which I call the TikTok-ification of social media. Starting around 2022 or so, many platforms started injecting more “out-of-network” or “unconnected” posts in their feeds. Facebook, Instagram, and LinkedIn all did this, clearly X has done it, probably others too. And that’s… fine I guess, but it’s not really going in the direction of “social” media because it doesn’t build social connections (as opposed to one-way parasocial interactions with creators). “Personalized television” would be a better description.

Retrieving posts by social similarity

I’m not against out-of-network posts, not at all. I think recommendations from all over the social graph can be a great way to discover new content and, ideally, make new friends. But it’s challenging to choose a few good posts out of the few billion per day that are posted to X/Twitter. The first, biggest filtering stage is called candidate generation (we explained the stages of a recommender in our last post).

Here’s how Old Twitter did candidate generation (as of March 2023, presumably for much longer before that):

How can we tell if a certain Tweet will be relevant to you if you don’t follow the author? Twitter takes two approaches to addressing this.

Our first approach is to estimate what you would find relevant by analyzing the engagements of people you follow

In other words, this method generates candidate posts by looking at what the people you follow liked (or retweeted, replied to, etc.) This is a classic, simple algorithm, and a version of this called LinkLonk powers the popular For You feed on BlueSky (just one of the many 3rd party feeds users can choose from over there).

The other big way Twitter used the social graph was by clustering all of its users into 145,000 or so overlapping communities with the SimClusters algorithm, which was published in 2020.

This gives every user a vector describing which communities they belong to, derived from the structure of the social graph. When a user engages with a post, their community vector gets added into that tweet's embedding vector, creating a time-decayed average. Thus the post’s embedding gradually comes to represent “the kind of user who likes this post” in community space.

Each user also has an “interested in” vector, representing the communities that the people they follow are most “known for.” A post is a good candidate for the user if the user’s interest vector is close to the post’s engagement vector in community space. In other words, if people from the communities of people they follow like it.

For all the gory details, see the SimClusters readme and this analysis.

Grok’s new way

As I dug through the code for the new Grok-based recommender (Claude was very helpful here) I discovered that it doesn’t use the social graph when picking out-of-network content. It again uses a shared embedding space for posts and users, but the embedding for a post is defined directly by which users engage with it — not which communities.

The transformer-based architecture predicts probability of user engagement directly (for 15 different actions) and posts are retrieved based on their probability of engagement. Again, see the readme and this analysis. The candidate posts are then ranked by multiplying these engagement probabilities by action-specific weights, and taking the sum.

Old Twitter also ranked posts by predicting engagement and then taking a weighted sum. They even published the weights (sadly, the Grok algorithm repo now contains only dummy values). This approach to ranking is totally standard stuff. GreenEarth does it (you can customize the weights of course) and every production recommender I’ve ever seen does it too.

The difference between Old Twitter and Grok Twitter, then, is not the core ranking algorithm. They both try to maximize engagement! Instead, I wonder if Grok’s new candidate generation just doesn’t end up pulling in as many posts from the communities you care about. With billions of posts per day, it’s possible that the old and new algorithms end up selecting for really different parts of the content universe, even if they both produce about the same total engagement. Or maybe the new algorithm increases engagement, but that has come at the cost of moving the product somewhat away from “connecting” and toward “entertaining.”

Communities are made of triangles

All of this matters because meeting friends-of-friends is how communities are built in the real world. This is called triadic closure, and there’s empirical evidence for it: if A is friends with B and C, then B and C are likely to become friends too.

In the world of social networks, this means following someone that someone you follow follows. But you probably won’t do that unless you first see a post from them. Probably several posts; repeated exposure is how relationships form. And both reposts from friends (in-network posts) and posts from people that your existing community likes (ala LinkLonk and SimClusters) provide ample opportunity to repeatedly encounter your potential new internet BFF.

It’s easy to count triadic closure — just look for when someone follows a friend of a friend. I suspect that tuning the feed to increase this metric would create a community-building social media algorithm. Such triangles are key to how the Facebook “people you may know” recommender works (per Meta, and this study).

Here at GreenEarth, we’re certainly going to try it. Measuring the X feed composition and spelunking through the old and new Twitter code has given us valuable ideas about how community dynamics might play out algorithmically, and it’s influencing our candidate generator design. We’re not sure that we’re right about all of this, but we do know that we’re committed to building social media that is actually social. Stay tuned.

What if anyone could create a new algorithm for social media?

Many people have complained about social media feeds, including me. ATProto, the open protocol behind BlueSky, allows anyone to build a feed algorithm, and there are now thousands of feeds available in the BlueSky feed store. However, none of them rival the sophisticated personalized recommenders available on major platforms.

GreenEarth is a project to change that. Our goal is to make state-of-the art personalized feed technology available to everyone. Key features on our roadmap include interactive customization through LLM prompts (“show only the funniest cat videos”), engagement prediction models, and selectable candidate generators.

If you want classic reverse chron you are welcome to keep it, but modern recommender algorithms have many advantages. There is ample evidence that they greatly improve the user experience for a lot of people and thereby significantly boost retention, so you may want your ATProto app to include them. But we can do way better than the status quo of fixed, opaque, non-customizable feeds. The point is choice and control!

This post explains our goals, architecture, and how to get involved.

Goals

The GreenEarth project was founded with three goals:

Create a production “prosocial” feed. In particular, we want to productize the Prosocial Ranking Challenge, a large experiment where we showed that the right algorithms can make people hate each other less.

Create open recommender infrastructure. We aim to support not just our prosocial feed but a wide variety of feed types. “Open” means both open source and full support for algorithmic transparency, control, and research.

Support ATProto startups. In particular we are proud to partner with Graze.social (“create personalized feeds on ATProto, no code required”) and SkyLight (“the unbannable short form video platform”).

What we’re building

This is an infrastructure project which doesn’t have a unified user experience. Instead, our mission is to support a variety of projects which want to use algorithmic social recommenders in different ways. To do this, we are building a configurable recommendation architecture that looks like this:

This is a fairly standard recommendation pipeline, broadly similar to what you will find in use at any major platform.

Starting with all possible items of content (billions) we first run a user-configurable set of “candidate generators” which each use a different approach to produce a much smaller set of items (hundreds) that might score highly in the final recommendation. There are many ways to do this, from just collecting every post from the people you follow, to searching the entire inventory for topics that you’ve previously liked, or analyzing your social network (Twitter’s source code includes several). The key point is that candidate generators have to run in sub-linear time on the available inventory, because their job is to select just a few posts out of an arbitrarily large set. So, this step is typically based on search engine technology, vector databases, etc.

These candidates will be merged and sent to a number of “signal generators,” each of which scores each post on one or more attributes. Our initial plan is to build two signal generators: an LLM scorer and an engagement predictor.

The LLM scorer will take a post and a prompt to produce a single “relevance” score. Users (or feed designers) will provide a prompt describing how to score post content, e.g. on a 10 point scale. You will be able to run multiple prompts against the same content to generate and combine different signals. We have prototyped a fast approximation algorithm that will allow interactive editing of prompts — change the wording and you’ll see your feed change. We think this is a killer feature (and btw we started building it before Elon announced something similar for X).

The engagement model will output the probability that a particular user will like, reply, follow, block etc. when they see a particular post. But isn’t optimizing for engagement the cause of all our algorithmic ills? Well, yes and no (see: What’s Right and What’s Wrong with Optimizing for Engagement). Engagement signals can sometimes select low-quality content, but they’re also essential for personalization. If you want to build any sort of recommender that gives people more of what they “like” (rather than just what’s popular overall) then engagement prediction is essential. So, we’re going to support it using a standard two tower model.

The result will be a vector of “signals” about each candidate post. The next step, scoring, will turn that vector into a single number indicating the rank of each post. The simplest way to do this is to assign linear “weights” to each signal indicating its importance, which is a surprisingly common approach. After scoring, posts are simply sorted by their final score to generate an initial ranking.

Signal generation and scoring are designed to be be highly parallel, so that they can run independently on each post. This can result in a lack of diversity, for example many posts in a row might be on the same topic or by the same author. The final step, re-ranking, takes the feed and potentially reorders, inserts, or deletes posts to ensure a good mix.

All of these steps will be user configurable. That is, you will be able to specify:

• which candidate generators to use (e.g. just post from friends, or everyone)

• the prompt(s) for LLM scoring

• how all the LLM and engagement signals are combined into a final post score

• which reranking or diversification algorithm to run

This may not be the simplest thing to get your head around if you’ve never seen it before. It might seem a little different than specifying keywords or user lists (though you can do that too, as a custom query for our search-based candidate generator). But we chose this because it’s a generalization of a tried-and-true production architecture that scales well. The set of candidate generators, prompts, scoring weights, and reranking rules is a design space that we are hoping many people will be able to create wonderful things with.

Using GreenEarth

There are several different types of “users” for our infrastructure. Here’s what we call them:

• Consumers are the people scrolling generated feeds through BlueSky or another ATProto app like SkyLight. Some feeds may offer a UI for consumer customization, e.g. by editing prompts or adjusting scoring weights.

• Feed builders are people who get deep into creating custom feeds. They might write code to call the GreenEarth API, or use graphical feed building tools like Graze.social.

• Customers are the people who call the GreenEarth API directly.

But before any of this can happen we need to continuously ingest, analyze, and index every post in the ATProto universe in real time. That’s a big task all by itself, and thus the entire recommendation pipeline is a fairly hefty service. This is, of course, why no one has built it before. We will be offering GreenEarth both as open source code and, in association with our partners, a paid API with access to live data.

Timeline, team, and funding

We are aiming to have an MVP recommender system up in Q2 2026.

GreenEarth was founded by Renee DiResta (Georgetown University), Glen Weyl (Microsoft Research), and Jonathan Stray (UC Berkeley). Our core team includes Ian Baker as Director of Engineering (Berkeley) and Nikhil Garg as Director of Research (Cornell Tech) — plus a growing team of contributors.

We are currently funded by a seed grant of $300,000 from Project Liberty. Thank you!

How to get involved

We are building open social media algorithm infrastructure for everyone. We want a world where people have control over their feeds, and those feeds are both open source and open data (with appropriate privacy controls) so we can all understand how these algorithms work and what they are doing to us.

This is a big vision! We don’t have anywhere near the resources we need to accomplish it. You can help! We are currently focussed on building out our open source development team. Eventually, we will be inviting academic researchers to collaborate with us on open social media science!

If you’d like to be a part of GreenEarth, the easiest way is to join our discord and start asking questions! See also our Github repo.