Static Typing in Python

I want to start with a pop quiz: is Python dynamically or statically typed?

Maybe you think it's dynamically typed. Maybe you think, hmm, this is a talk about 'Static Typing in Python', so it's probably statically typed?

Or maybe you think this is a trick question?

Yeah, it's a trick question.

The answer is that Python is dynamically typed, but it can optionally be as statically typed as you want it to be.

And if that doesn't make any sense to you, that's OK, you're in the right talk, and we'll break down everything that's necessary to understand that answer.

To understand that, we need to understand types in Python, and type systems in general.

We'll talk about dynamic typing in Python, then we'll talk about static typing in Python.

Once we understand that, we'll talk about how to use static typing in Python, when you should use it, and when you maybe shouldn't use it as well.

Let's talk about types, and specifically let's talk about type, the builtin keyword in Python.

In Python, in our REPL, we can type something like this: type(42), and we get back something that says it's of the class int. Since forty-two is an integer, this makes sense.

We can do the same thing for floats...

...strings...

...lists, etc.

The type builtin tells us what type a given object is.

You might say, "oh, I recognize these things, str, int, float, this is what I use to change one type into another."

And yeah, you could have a variable like this, a = 42...

...you call float() on it and you get a float...

...you call str() on it and you get a string...

...and you can call list() on it and you get a really ugly list.

Who's seen a really ugly list like this before, where a string is getting turned into a list? That's a type error!

You've had a type error in your code, where something that was expecting an iterable, like a list, and got a string instead, and Python did this.

The thing to note here is that things like int look like all the types available to us, but they're just classes which correspond to builtins.

Here int is just a keyword that corresponds to a class. When you do isinstance(42, int), it's just doing class matching, and is telling you whether it's a member of that class or not.

But there are other types as well, right? Perhaps you've gotten a NoneType error. NoneType is the type of None, but we don't see NoneType as a keyword in Python, we just see it in our stack traces.

Similarly there's a function type, but we don't use the keyword function to define a function in Python, we say def: function_name() etc.

There's also an ellipsis type, and actually, there's a lot.

In Python if you import types...

...and you call dir() on types, there's this whole list of things that exist in Python that could have a type, like CodeType, you can see FunctionType is in there, there's all sorts of types.

You could instantiate a function with these classes, it's just a class which takes arguments. But we don't, because it'd be really messy, but these types are available.

When we say that Python is a dynamically typed language, what does it mean?

First of all, it means that a variable can be any type. If I assign something to be a variable, it can contain any of these types that are available to us.

For example, I could import random, and set the variable a to be the random choice between an integer, a float and a string.

What is the type of a here if I evaluate this?

Well, it depends, it's non-deterministic. It depends on which object was randomly chosen from that list.

Could be a string, could be int, could be float, doesn't matter. That variable can contain any of those types.

Dynamic typing also means that the arguments and return values for a function can also be any type as all. The same is true for for a variable as it is for a functions arguments as it is for the return value for the function.

If I had a function like this, frobnicate which takes three arguments and returns a + b + c, how do we know what types this is going to expect? Does anyone have a guess what a, b and c should be here?

If I define the frobnicate function, and I call it with 1, 2, and 3, it correctly adds them all up to 6.

But, I can also call this same function with strings. And it would concatenate these strings instead.

This function would accept either of these types as arguments, and it would still work, right? It's still a valid Python function.

What I can't do is mix these types, I can't add integers and strings just with the + operator, so this would result in a TypeError: I'm trying to do something and the types are incompatible.

That's confusing, this function is confusing. How can we fix it?

One thing we could do is write really long and detailed docstrings that outline every single type that the function expects, what they are, and what it returns.

Who writes docstrings like this, anybody? For those of you with your hands up, is your employer paying you enough to write docstrings like this?

Because it's a lot of work! And while this is great to read, the thing about this is that writing this docstring has zero effect on whether this function is actually being called correctly or not.

You're telling the developer that's eventually going to come and try and use it, that this is what you expect, but there's no guarantee that they're actually going to do that.

Another thing we could do: we could assert on the type of everything that's passed to this function, do our business logic, and then assert on the return type before we actually return it.

I gave this talk once before and I said "nobody does, this, this is insane!" Because the thing is, this actually adds overhead to the execution of this function. There's a small cost you have to pay to do all these assertions.

And after I gave this talk, someone came up to me and said "yeah, actually, we do do this..." And while it works -- it's a valid way to assert on the types for your function -- it's so many extra lines, and what if you forgot an assertion? It's on you to remember that.

What do we do instead? In Python, what we do is called "duck typing"

Which means that if it walks like a duck, and it quacks like a duck, it is probably a duck.

This means that how we use a variable helps us determine what it's type may be.

For example, in this first line, I'm setting foo to be [f(x) for x in bar]. We could probably reasonably assume that bar is an iterable, maybe a string, a list, or a set.

On the second line, we're comparing bar to zero, so bar is probably a number, like an integer or a float. It's unclear, but we have a pretty good idea about what it is.

And the last one, that could be a function, it could be a class that's poorly named (doesn't start with a capital letter), but we don't know. It could be anything.

This brings us to static typing. Static typing is basically the opposite of dynamic typing. It means that the type of a variable, the arguments to a function and the return type of a function are statically defined and they cannot change, they have to be a specific thing.

Here are some examples of that same frobnicate function in other languages, and I'm curious if people can recognize these languages.

Here's the first example, anyone know?

That's C.

How about this one?

That's Java, it's a dead giveaway with the public static int.

Anyone know this one?

This is Rust: Rust has really fine-grained integer types, so this is an unsigned 8-bit integer.

How about this one? The number is the hint here.

This is TypeScript, which is typed JavaScript. In JavaScript there's only one number type, everything is a number, so that's how you know.

You can sorta put languages into one of two categories: there are dynamically typed languages, and there are statically typed languages.

And I have to put a little asterisk here next to Python here, which is why we're having this talk. Because Python is, and was originally, a dynamically typed language, but it is now also kinda statically typed.

Also apparently Ruby is going to get a type system kind of like Python's but not until the end of next year, so I won't include it yet.

Like I said before: Python is a dynamically typed language, but it can optionally be as statically typed as you want it to be.

This wasn't always true. When Python was first created, it was purely a dynamically typed language.

And the story of Python becoming a language that is optionally statically typed is also the story of type-checking at Dropbox.

Dropbox is a large company with millions of lines of Python code. At that scale, having that many lines of untyped code is a liability. It becomes exponentially harder for your developers to interpret what this code does when they inherit new code bases, when they're doing refactoring, etc.

Pretty early on in their journey of having that much Python code, they realized that it's not great that Python is dynamically typed in this case, because it makes it hard for us to work with it.

However, there were some things leading up to it as well.

The first is PEP 3107. We got this in Python 3.0 in 2006, and this allowed us to do something something like this.

I could take a function like this...

...and I could add any metadata that I wanted to the arguments and the return type of this function. Anything that's valid Python could be put in after the colon here.

And what I would get, if I defined that function and called __annotations__ to get that attribute off of it, would be the evaluation of all those annotations, in a dict, including the return annotation. Now, these are just made up things, they don't have any real meaning.

And that was the thing, it was basically a nice feature, and there were a lot of things that could be done with it.

All of them boiled down to "maybe we can provide typing information with this", and there were a lot of different ways we could do that.

We could also just use it as documentation, but really it was looking forward to having type annotations for functions.

This allows us to actually write a function like this which is actually interesting. Here, you can see the type for each argument, and the return type, and then you would have annotations that would tell you what the developer was expecting when they wrote that code.

But, it still doesn't give us a way to actually evaluate whether that function's being used correctly, it just gives us annotations, and now an attribute that tells us what those annotations were.

And, this is just for functions: can't do variables here at all.

Around the same time, Jukka Lehtosalo was working on his PhD thesis at the University of Cambridge.

His research was about the unification of statically typed and dynamically typed languages.

His goal was to be able to use the same language for everything from really tiny scripts, to a sprawling multi-million line codebase, and add as much typing as you want.

He wanted to have this gradual growth from something that was completely untyped, and allow you to slowly mix in and add static typing to it.

Sounds kind of interesting. Would be great to not have to do it all at once, especially if you have millions of lines.

In 2011 he published his PhD thesis, and basically his conclusion was this:

If you try to add a static type system to a dynamically typed language, it could be totally invasive. You'd have to change every tool in the ecosystem, every interpreter, everything that checks syntax, etc. This was his conclusion.

However, he proposed, that if you could create an optional, pluggable type system that doesn't actually affect the runtime of the program, this could be added to everything without adding significant burden to the ecosystem.

This sounds really great. What he did was take his PhD work, and went to PyCon US in 2013, and he introduced something called Mypy.

If you've heard of Mypy before, what he introduced was not what you've heard of before.

In the abstract for this talk, he describes Mypy as "an experimental variant of Python that supports writing programs that seamlessly mix static and dynamic typing".

This wasn't a just a type checker (although it included a type checker), it was actually a variant of Python.

In his research, he couldn't use Python directly, so he created a new language that kinda looked like Python that allowed him to actually do his research about type systems.

Mypy the language looked like this. It kinda looks like Python, but you see that it has int fib, that's defining the type of that function, etc.

The point is, Python at the time, even with function annotations, it wasn't enough for him to actually use Python to do static typing research.

He presented his project at PyCon, and he said afterwards, he chatted with Guido about it, and Guido convinced him to get rid of this variant of Python, to just try to do it in pure Python, since it would make a lot more sense, and we'd get some benefit from it as well.

So the variant went away, and Mypy became just the type checker that was included with it, and that is actually what we think of and what we'll see become Mypy today.

Shortly after that, Guido published PEP 483, which is his theory of type hints: the idea of how types and static typing should work in Python. This is some really broad and general ideas about typing in Python in general.

The first tenet of this was that typing should be optional, and you can see how this is sort of coming from Jukka's research. Essentially: adding an annotation to some Python code shouldn't affect the runtime. It should have zero effect on the actual runtime of your program.

This means that an annotated function should behave exactly the same as one that doesn't have any type annotations.

I think this is representative of some lessons we learned from the transition from Python 2 to Python 3: we want this to be an easy transition from untyped to typed code.

And we also want to be able to do it piecemeal: like Jukka said in his paper, we want to do gradual typing, you want to be able to add typing to certain files in your codebase and not have to do it all at once.

In addition, we should have variable annotation: just functions is not enough, we need a way to specify the types for individual variables in Python as well.

That means that in addition to having the function annotation...

...we should be able to say that the type of this variable I'm assigning to is an integer, and then return that.

This also means that we can do type hinting for Python 2. As long as we have this sort of comment notation for variables, even those that have old versions of Python should be able to use static typing. This allowed us to do all the same types of annotations before function annotations existed in Python 3.

That means I can have the same function in Python 2 and Python 3, and just by turning that function annotation into a comment, it has the exact same behavior.

This PEP also introduced some special type constructs, these are some fundamental things we need to do static typing.

These take all the existing types and allows us to construct new types as well.

These are types like Any, which would match the type of anything at all, Union, which is the union of two types, Optional, which is an alias for the union of NoneType and any existing type, Tuple, Callable, etc., which match the corresponding objects in Python.

Then we could write something like this: a frobnicate function that takes an integer, an integer, and then optionally could take an int or a float, and then the return value, again, could be either an int or a float.

Once we have these special type constructs, we have a more powerful type system here.

The PEP also defines some container types, which allows us to define types inside container classes like dictionaries and lists.

That would look something like this. If I had a list of integers, I could define the type of that container class and the values that will be stored inside it, and the same for dictionaries: I could have a dictionary where they key is always a string and the value is always an integer.

The PEP also gives us generic types, for things that behave in a generic way.

In Python we have things like an iterable. An iterable is this whole class of objects that have the features of something that can be iterated on.

We also have types like Iterable that we can use to type things which we don't really care if it's a list, or if it's a dict: we just want to be able to iterate over it.

Finally, with this theory of type hints, we can do type aliases using Union.

So if we wanted to be more like JavaScript, we could have a single number type that unifies all numbers in Python.

That was the theory of type hints, and then we had PEP 484, which standardizes everything in PEP 483, and basically standardizes around what Mypy was currently doing.

In short, this PEP was "how to build a type checker for Python". It introduces a typing module, it introduces a lot of details about edge cases and specific use cases. It leans really heavily on what Mypy was already doing.

We got that in Python 3.5, which had PEP 484 support, including the typing module.

Then we got PEP 526, which let us do inline variable annotations.

This allowed us to do something like this: before, for variables, we had to use these type comments...

...and now we can do it inline. So primes would look like this.

One problem with the comment annotation syntax was that it's hard to initialize a type for a variable that doesn't have a value yet.

With inline variable annotations, we could do this, where just say "this variable is a string, I don't care, it doesn't have a value yet"

And the same thing for class types. Here, stats is a class variable, I can now annotate it correctly with inline annotations.

We got that in Python 3.6, and that was almost everything we needed to do static typing in Python, but we needed a type checker.

And that was Mypy, and it's the last piece of the puzzle. It's not included with core Python, it's a third-party tool.

The idea is that we have a couple different types of type checkers: we have either dynamic or static type checkers.

A static type checker is going to type-check your code at rest. It's going to look at your source and not actually evaluate any of it, it's going to do it totally statically. Whereas a dynamic type checker is going to be kind of like those asserts we were doing before: it's going to happen at runtime, it's going to check those types as your program executes.

So Mypy is a static type checker for statically typed Python.

Like I said, it's just a tool, you can pip install it, it's available on PyPI.

If you run it, it looks something like this: you install it, create a file with some static types, and if they're wrong, it'll warn you. It will tell you "hey, you told me this argument should be an integer, but you're calling it with a string, that's a type error.

There are actually a bunch of type checker besides Mypy, like I said, there are static and dynamic ones. The static ones kind of align with all the large Python shops that exist, so Mypy is mostly owned by Dropbox these days, Google has Pytype, Facebook has Pyre, Microsoft has Pyright.

PyCharm actually has a type checker built into the IDE. And actually you can use any of these type checkers with whatever your IDE or development environment is as well, most of them integrate into some Python environment.

Disclaimer, I work at Google, so one of the questions that I often get about type checking is "what is the difference?" Like, if these all implement PEP 484, what's the difference between them. I've actually only used Mypy and Pytype, so I'll tell you what the difference between those is.

The answer's not a whole lot, basically it comes down to a philosophy.

The philosophy is that Pytype will only give you a type error if it will actually become a runtime exception. Whereas Mypy is going to be a lot more strict about the actual usage of types. I'll show you some examples.

Here I have two functions: one returns a string, and the other takes the call to that function and adds it to an integer.

If I were to run this, it would be a type error: I'm trying to add a string to an integer, and that's a problem.

If I run it with Mypy, it actually passes. It says that this is fine, and the reason is because it is unable to do type inference across multiple functions.

However, if I ran this with Pytype, it would actually say "yeah, if you ran this, at runtime you would get a type error, and that's a problem". And you would get an error.

And actually the reverse is true as well. Pytype is going to be more lenient in situations where it's not going to produce a runtime error.

Here's an example of a function, it takes a list, the list has at string inside of it, and I append an integer in it, iterate over it, and print it.

This will succeed if I try to run this. It's totally valid, although it might be a little confusing to our developers to mix strings and integers here.

If I run this, it correctly prints a list of strings.

Pytype says there's no errors here.

But Mypy's going to complain because you're trying to append an integer to a list of strings and it's incompatible.

You might now say: "why?"

This is all great, it's great that we have this in Python, why do I want to use it?

First I'll say when you shouldn't use static typing, and the answer is: basically never.

It's not going to hurt you, it's not painful to use.

One case when you might not want to use static typing is if you're going to use it to replace your unit tests. Static typing is not a replacement for unit tests.

A lot of time unit tests end up being something that ends up looking like you're the types of your program, but it's not. And actually you probably need both static typing and unit tests simultaneously.

When should you use static typing? Basically, use it as much as possible. Use it liberally, use it in these situations:

Maybe when you're millions-of-lines scale. Like I said before, that much dynamically typed Python is a liability, and so if you have millions of lines of Python code, you probably already know this, but you should probably start statically typing it.

Dropbox found that at their scale, dynamic typing produced needlessly hard to understand code that impacted developer productivity.

That's why all these big shops like Google and Facebook have invested in static typing, because it actually makes their developers lives easier. And the lack of static typing is a liability at that scale.

You can imagine a graph that looks like this, where as your lines of code go up, your desire to add type annotations goes up, and the ease of doing it goes down.

You are probably here, unless you're Facebook, Dropbox or Google.

This is a good time to add static typing.

This is probably when you're actually going to do it, but keep that in mind when you're evaluating when to start adding static typing. It's easier to do it earlier.

You should use static typing when your code is confusing.

Let's be honest, we've all written confusing code. You can think of static typing as machine-verified documentation.

You add typing to your Python function that is extremely confusing, and it will tell your developers what that function should expect and return. And it will also allow you to verify that it's actually being called that way.

If you feel the need to document the input and output of a function, that's probably a hint that you should just statically type it.

You should use static typing when your code is for pubic consumption. Let's say you're publishing a module on PyPI: adding type annotations help the developers who use your module know how to use it. And it also means that if they're using static typing in their codebase, they'll really appreciate that you're publishing types for your codebase as well.

Another good time to use static typing is before doing a big migration or doing a big refactor. Go and add static types to everything that you're about to change, and then change it, and see if you get a bunch of type errors. Because if you do, it means that you're calling some function wrong, or that you've missed some part of your migration.

And finally, you can also use static typing to just experiment with static typing. Like I said, it doesn't hurt, it's pretty easy to just add some static types, as long as you're in the latest version of Python that supports it.

To conclude: here's how you can use static typing in Python in just five easy steps.

First, optionally, you can migrate to a newer version of Python.

Like I said, type comments will allow you to statically type older Python 2 code, but really we should probably all be migrating to Python 3.6 or above. We now have Python 3.8, so you should at least be on 3.6.

Second, you can install a type-checker locally.

I don't care which one it is, install one, install multiple, it doesn't matter, and integrate it into your IDE. This will allow you to start getting type notifications in your code as soon as you add just one line of static typing.

And then you can start optionally typing your codebase.

You can start with the hardest function you have, the one that's the most impossible to understand, or you can start with the easiest one, the simplest function would be the easiest one to type. Just pick a critical area and start there.

And then, what you can do when you have a little static typing is that you can start running your same type checker with your linting.

I'm assuming you're doing linting, everyone's probably just running black now. Just add type checking as well, just run Mypy at the same time.

And finally you can convince all your coworkers to join you in the glory of static typing.

If you need help convincing them, you can share this talk with them. 😉

Thanks!

• GitHub: @di
• Social: @di.dev