PEP 572: The Walrus Operator

You might have heard about PEP 572. It caused a bit of drama in the Python community.

But before we talk about this PEP, and what happened, we need to talk about "Python Governance".

Python Governance is the answer to question "How do we govern the language of Python?"

That is, how do we make changes to the language, how do we implement new features, and that kind of thing. How do we make decisions about Python?

When Python was first created, it's creator installed himself as the "BDFL": the Benevolent Dictator For Life.

"Dictator" meaning that he can do whatever he wants to the language, "benevolent" meaning that he's generally going to be looking out for us as Python users.

This is the creator of Python, Guido van Rossum, and he's our BDFL.

But one thing to note is that Python was created more than 20 years ago, and in that time, it's really grown, and Guido hasn't been making every decision himself.

Instead what we have is a process. The way that decisions are made within the Python language is with a PEP: a Python Enhancement Proposal.

This is kind of like amendments to the constitution of a country, which similarly determines how a country is governed.

Each PEP builds on the previous one, and makes changes to "the law of Python". So we started with PEP 1, which tells us what a PEP is and how to write it, and we grew from there.

Can anyone name a PEP? Someone in the audience inevitably shouts: "PEP 8!"

Right, PEP 8, I figured you'd say that. So PEP 8 is probably the most well-known PEP. It's a style guide for Python code.

The author of this PEP? It's Guido van Rossum, as you might expect. This was a pretty early PEP, and it was authored by Guido himself.

Some other interesting and well-known PEPs: There's PEP 20, the Zen of Python. It's what you get if you type import this in your Python REPL.

The author of the Zen of Python is Tim Peters, who's one of Python's core contributors.

My favorite PEP is PEP 566, Metadata for Python Packages 2.1.

The reason I like this PEP so much?

I am the author of this PEP.

When I wrote this PEP, this is what the process was generally like. First, I came up with a draft for the PEP in a format in which all PEPs should be made.

Then, it went for acceptance, had some discussion, and someone accepted it.

And then, it was implemented. Sort of a three-step process.

One thing to note is that Guido himself did not approve my PEP directly.

In fact, I'm quite sure he has never read it, and he has no idea who I am. But that's fine!

Instead, what we have are BDFL Delegates.

These are people that Guido has entrusted the ability to make decisions about the language, in areas which he doesn't really care about, doesn't know that much about, or thinks someone else is a better steward for that part of Python.

In my example, Daniel Holth was the delegate for this packaging-related PEP.

So the flow of power kind of looks like this: there's Guido, and then there's the delegates beneath him, and they derive their power from Guido, and then all those delegates are approving (or rejecting) PEPs.

(And sometimes Guido is still approving PEPs himself as well).

So, you might have heard about PEP 572. This is a relatively recent PEP, and the title of the PEP was "Assignment Expressions".

The reason you might have heard about it is because it caused a little bit of drama.

This PEP is centered around just these two little characters: :=

These two characters represent a new operator for Python, and it has a name... but I'm not going to tell you what the name of it is though.

Instead, what I'm going to do is call it the "walrus operator" because it kind of looks like a walrus lying on it's side.

And by the way, when I gave this talk once before, some folks were really concerned that this walrus was dead.

He's not dead, he's just resting. Don't worry.

So instead of explaining to you word-for-word the contents of this PEP and what it aims to do, I'm going to give you some examples to give you some ideas about why we might need the walrus operator.

One is for balancing lines of code (LOC) and complexity (big-O).

Let's take this example. Let's say I'm setting the variable foo to the list which contains f(x), f(x)**2, f(x)**3.

Let's say that f(x) is a really expensive, deterministic computation. Maybe each call takes a full minute to run, and so we don't want to re-run it every time. Here, we're calling it three times, but it's going to give us the same result every time.

What you'd normally do as a Python user is break that function call out to a new line, set a new variable y to the result of that call to f(x), and then you'd reuse that variable y inside the list.

Some people don't like this because it's two lines of code where you could have had one line of code and a little more complexity.

If you had the walrus operator, you could do this. You could set y := f(x) inside the list expression, and value of that gets bound to y, and you can reuse that later in that expression, so you can reuse a value that's expensive to compute, and keep it on a single line.

Another example for why you might want to use the walrus operator is avoiding inefficient comprehensions.

Let's look at this for loop: we have an empty list of results, we're iterating over data, we're calling f(x) on each item in the data, and we get a result from it. If the result is truthy, we're appending it to the list of results, and if it's not, we're doing nothing with it.

Normally, when I see a for loop that looks like this, with an empty list initialized above it like that, that's kind of a code smell, and I say "this should probably be turned into a list comprehension".

And you'd be right, if you turned this into a list comprehension, it would look like this, it would produce the same results, it would be less lines, it would look a little cleaner...

...and it would also be more inefficient, because you'd be calling f(x) twice now instead of calling it once.

With the walrus operator, you can do something very similar to what we did before: you can set y := f(x) in the filter clause, and then you can re-use it and share that sub-expression in the other part of the comprehension.

Another example: unnecessary variables in scope.

If you've ever used the re (regular expression) module in Python, you've probably written something kind of like this.

You call pattern.search() for some piece of data, and you get back something that is either None, or it's a regular expression match object, and you can do things with it. So you always have this check for None in your code.

What happens here is that match is always going to exist farther down: as your program passes this if condition, match is still going to be there, and it's kind of a little bit annoying, because you only need match inside that if block, but it's sitting outside of it.

With the walrus operator, you can move match into the if statment, and it's less lines and more clearly within the conditional.

Another example: processing streams in chunks.

This is a pretty common pattern: if we have a big file, but we only want to process part of it at a time, we'll work on it in chunks. We take the first chunk from the file, and then we can set up a while loop, processing that chunk over and over until file.read() eventually gives us something falsey like an empty string.

This works fine, but one thing that is kind of "not pretty" is that the chunk = file.read(8192) line with a magic number is being repeated twice, and if I want to change the value of that chunk, I have to change that number on two lines, and not one.

And maybe that could live in a variable, and I could put it outside this block, but then I'm still duplicating a line here.

We can make it even simpler with the walrus operator. Here, we can put chunk inside the while statement similar to before, and then it behaves exactly the same way.

You might say "why?": why do we have to do this?

All those examples from before were totally fine, and if I paid attention to where I was using expensive or duplicated statements, I could find ways to avoid them entirely.

One argument for why is that maybe fewer lines are better.

Programmers kind of hate change, so minimizing your diffs by at least one line is worth it. You want to minimize the amount of code you have to grok, to review, to explain to another developer.

So you might say if I can turn these two lines...

...into one line, my coworkers are going to love me, right?

Well, maybe.

One of the proponents of this PEP looked through some Python code bases and found this example, where instead of writing these two lines...

...the programmer had combined them into a single line like this, in the interest of minimizing their diffs, perhaps.

But again, this has become more inefficient, because re.match(data) is being called twice where before it would only have been called once.

With the walrus operator, the programmer could have kept it on one line, but also maintained the same efficiency of two lines.

You might be looking at all this and be thinking "hmm, this kind of looks like the assignment operator".

It's important to note that the walrus operator is almost nothing like the assignment operator. In fact, everywhere you can use the assignment operator, you cannot use the walrus operator, and vice versa.

So for example, with the assignment operator, you can have multiple targets, which lets you set multiple variables to the same value on one line.

You can't do that with the walrus operator.

Another example is setting the value of a dictionary or an index in a list.

You can't do this with the walrus operator.

Same thing with properties. You can assign to a property with an assignment operator, you can't with the walrus operator.

Another difference is comma priority: in this example, with the assignment operator, x will become the tuple (1, 2), while with the walrus operator, x will only become 1, since this is actually defining x inside the creation of a (x, 2) tuple.

And, finally, we can't do things like like the augmented assignment operator, where we can add and reassign in a single operation.

We can't do that with the walrus operator, and actually I don't even know what we'd call it at that point, like maybe...

...a walrus...

...but with a pope hat?

So that's PEP 572. Let's talk a bit about the reception of this PEP.

Like I said before, most people hate change, and programmers especially hate change.

One of the reactions was related to backwards compatibility: where will this work?

The thing is, it's not backwards compatible. If you write code using the walrus operator, you cannot use it in older versions of Python, which didn't support that syntax. So it's not a breaking change to the syntax, but it's not backwards compatible.

Another concern was teachability: what do we call this? I've been avoiding calling it by it's real name, I've been calling it the "walrus operator", but we shouldn't really call it that, that's jargon, we need a real name for it that tells us what it does. We can't call it the assignment operator, that's the equals symbol. We could call it the "becomes" operator... how about the "named expression" operator?

So that's actually the name of it, the "named expression" operator, but it's still challenging. If I told you "hey, we're adding a named expression operator to Python", you probably wouldn't understand what that means. What is a named expression?

Another common complaint was attractiveness: a lot of people looked at this and said "wow, that's really ugly or weird or different, I hate it!"

There was a lot of discussion about this: there were long mailing list threads debating the merits of this PEP.

Like, really long threads.

There were polls, where core contributors were polled to see if they liked it or not.

Some of the core contributors weighed in with opinions of their own.

Tim Peters, who was the author of the Zen of Python and actually a co-author of this PEP, said that the proposal would have allowed a modest but clear improvement in quite a few bits of code.

Barry Warsaw said that since it changes the syntax of the language, people were tending to focus on that, rather than understanding the deeper issues here.

Łukasz Langa said that he thought dictators should dictate: he thought that Guido should just do whatever he wanted and ignore everyone else.

And in fact, Guido himself said that he had to stop reading the threads so that he wouldn't go insane.

So, after all this discussion, Guido accepted this PEP...

And then he sent an email to the core contributors mailing list, stepping down as BDFL, and going on a permanent vacation.

He said that now that PEP 572 is done, he doesn't ever want to have to fight so hard for a PEP and find that so many people despise his decisions.

So he removed himself entirely from the decision process.

In my talks, I like to share reactions on Twitter that people have, so here's a few about Guido stepping down.

This one's a bit funny: long before the PEP was proposed, this person said all they wanted for their birthday was the chunk example I showed before.

And then after Guido stepped down, they realized that thanks to PEP 572, they got their wish... but at what cost?

Hynek says PEP 572 rocks: what a farewell gift.

This person says "TFW reddit says PEP 572 lost the Zen of Python, but the author of the Zen is a co-author of the PEP!"

This person shared that Guido is stepping down as BDFL, and after quoting the first line of Guido's resignation email, says that he's shocked by the vitriol we throw at the people who run important free and open source software projects.

And as an example of that, there was this tweet: "PEP 572 is a trash feature, and I'm sad it was the straw that broke Guido's back".

I want to be really clear here: If we're talking about straws breaking people's backs...

...the straw that broke Guido's back was not trash PEPs that were heaped upon him by PEP authors...

...it was having your work called trash by people on the internet.

I think when we're on the internet, we kind of tend to forget who we're talking to. Even in the Python community, I think we're forgetting that dictators are people too.

Guido is the creator of our language, but he is a person, and he has thoughts feelings just like the rest of us. And just because he's the super-famous creator of our language doesn't mean that we get to treat him poorly.

And in fact, it's not just true for Guido, it's true for all people that work on open source software in our community: maintainers are people too.

Really, just think about it like this: people are people too.

I think this is what we're forgetting here is that everyone that you talk to on the internet is a real person.

I think, as a community, we need to step back and think, how are we acting with each other?

We were so mean and vitriolic that the creator of our language stepped away from us!

I'm going to pre-empt some questions about this.

The first is, what does this mean for Python?

I gave this talk before and someone asked "is this going to be the PEP that ends all PEPs?"

And in a way, this is kind of true.

Like I said before, the power to approve PEPs come from him. So with him stepping down, we had no BDFL-delegates, and we couldn't approve any PEPs and couldn't change the language.

That said, it's going to be OK. Right after this happened, a bunch of smart people got to work on the governance problem.

When Guido stepped down, he specifically entrusted those people with figuring out what we were going to do next, and they got to work.

They created a new PEP, PEP 8000, which outlined exactly what we were going to do to come up with a new governance model for Python.

They also created PEP 8001, which outlined the voting process for how we're going to decide which governance model to use.

They also performed a survey, of the governance models of other similar projects and ecosystems, to see if we could emulate them or learn from them in some way, and put that in PEP 8002.

Then, there were a whole bunch of proposals for the new model:

• PEP 8010: The Technical Leader Governance Model
• PEP 8011: Python Governance Model Lead by Trio of Pythonistas
• PEP 8012: The Community Governance Model
• PEP 8013: The External Governance Model
• PEP 8014: The Commons Governance Model
• PEP 8015: Organization of the Python Community
• PEP 8016: The Steering Council Model

...there was a vote on them...

...and the results were announced in December of 2018.

The model that was accepted was PEP 8016, the steering council model, which was proposed by Nathaniel Smith and Donald Stufft, and called for a council of five members voted on by the core contributors.

Then, the core contributors had a second vote to determine who would be on the council based on nominations, and we elected our steering council.

This is steering council that was elected at a panel at PyCon US in 2019. From left to right, we have:

• Barry Warsaw
• Brett Cannon
• Carol Willing
• Guido van Rossum
• Nick Coghlan

OK, on to some more Q&A.

Will this become part of Python?

The answer to this is that it already has: most of you probably aren't using Python 3.8, but you could be using it right now.

Emily Morehouse, a core developer, has already implemented this in 3.8, so you can install Python 3.8 and go and play with this today.

You might say: but I don't like it!

That's totally fine. You are entitled to your opinion! If you don't like it, then just don't write it. Nobody is going to force you to use the walrus operator.

And if someone on your team is telling you that you should be using it, it's your job to have a discussion with them and the rest of your team to decided if you'll use it (like any other optional feature, library, etc).

Another question you might have: is Guido coming back?

This depends on what you mean. Is Guido coming back to Python?

Of course. He's here, he's here for us, and he's not actually on a permanent vacation on a beach somewhere, or working on some other programming language. He's still involved.

Maybe you mean "Is Guido going to come back as BDFL?".

The answer to that is definitely no: We have solved the governance process, we chose a new model, we elected people to that committee, and so there will no longer be a BDFL for Python.

Another question: Will this happen again?

Again, this kind of depends on what you mean.

Maybe you're asking if another PEP will ever be this controversial.

I'm going to say "probably not". I think this PEP was an outlier: it was a very all syntax change that was easily understandable by anyone that looked at it, and that meant that it was easy for folks to make a quick gut reaction.

However, maybe your question is "Will people continue to be negative on the internet?".

My answer to that is... I don't know. I hope not: I'd really like it if we all could learn from this and just treat each other a little better and be more open and receptive to new ideas. I hope this all makes us take a step back and realize we could all be a little nicer.

However, so many people use Python now, and for a lot of those people, the language already feels perfect. And they don't want it to change.

So maybe this is the new status quo: maybe we're all going to get really upset every time we try to make some fundamental change to the language.

But I really hope it's not: I hope we can entrust the people that know the most about our language to make good decisions for us, that we aren't so reactionary online, and that we can treat those dictators, those maintainers, those people like they're actually people.

I hope we can do better.

Thanks!

• GitHub: @di
• Social: @di.dev