.center.middle # Optional Typing in Python .right[Andrey Vlasovskikh, JetBrains] .right[Minsk Python Meetup] .right[2013-11-29] --- # About * Andrey Vlasovskikh * St. Petersburg, Russia * JetBrains * PyCharm, IdeaVim * FProg.ru * Functional programming meetups --- # Python Problems * Performance * _Interpreter_ * CPU/IO parallelism * Mobile * iOS, Android * Python 3 * Slow adoption * _Tooling_ * _Smart editing_ * _Static analysis_ --- # Plain Text Editing * Do we really need tools for Python? --- # Plain Text: Pros * Universal editing experience * No extra dependencies * Python is easy to read and write * PEP 8 values * Cool guys don't need tools --- # Plain Text: Cons * Cool guys don't need syntax highlighting? * It's real! See the Go language * The dot completion makes editing even easier * Less run-time errors * Context-sensitive rename/move --- # Python Tooling * Tools could do better * PyCharm, PyLint, PTVS and others suck * Compared to Java, Objective-C * Tools require static types * Affects all dynamic languages * Except Erlang, Dart, TypeScript, etc. * Because of optional typing --- .center.middle # The JavaScript Story --- # JavaScript is Quite Bad * Too many quircks * See the [WAT talk](https://www.destroyallsoftware.com/talks/wat) * Lacks structure * No modules, classes * Languages to replace it * CoffeeScript, ClojureScript, Dart, TypeScript --- # TypeScript * A superset of JavaScript * TypeScript = JavaScript + modules + classes + optional types * Optional types * For tooling and documentation * DefinitelyTyped: TypeScript definitions for existing JS libs --- # TypeScript Example module Sayings { class Greeter
{ greeting: T; constructor(message: T) { this.greeting = message; } greet() { return this.greeting; } } } var greeter = new Sayings.Greeter
("Hello, world"); --- # Dart * Performance * Fixing the structure is not enough * New semantics for a faster VM * You can't mess with objects --- # Dart Example import 'dart:math' show Random; void main() { print(new Die(n: 12).roll()); } class Die { static Random shaker = new Random(); int sides, value; String toString() => '$value'; Die({int n: 6}) { sides = n; } int roll() { return value = shaker.nextInt(sides); } } --- .center.middle # Optional Typing for Python --- # Related Work * Timeline * 2002: ReStructured Text and Epydoc docstrings * 2003: PEP 316 programming by contract (deferred) * 2007: Sphinx docstrings * 2005: Optional typing blog posts by BDFL * 2006: PEP 3107 function annotations * 2008: Python 3.0 * 2012: rightarrow, typeannotations * 2013: Mypy, Python skeletons * Mostly run-time checking, recently static analysis * BDFL: More static analysis for Python --- # Typing * Common optional type system for Python * An initiative by Mypy project and JetBrains * Work in progress * Activities * Typing – an optional typing library * Python skeletons – a database of type annotations for libraries * Typing support for PyLint, PyCharm, etc. * Mypy – a faster optionally typed Python variant --- # Python is Good Enough * No new languages are needed * Types can be added via Python function annotations or decorators * Types for libraries are more important than for user code * The user should benefit from typed libraries * Everything has any type by default * Python program = valid optionally typed program --- # Simple Class-based Types from datetime import date def until_new_year(today: date) -> int: delta = date(today.year + 1, 1, 1) - today return delta.days * Variables that always hold values of a single type * Python 3 function annotations * Enough for many practical cases * Otherwise a variable is untyped --- # Run-time Semantics from datetime import date def until_new_year(today: date) -> int: delta = date(today.year + 1, 1, 1) - today return delta.days * New function attribute * No run-time checking, no overhead .no-highlight >>> until_new_year.__annotations__ {'today':
, 'return':
} --- # Static Semantics from datetime import date def until_new_year(today: date) -> int: delta = date(today.year + 1, 1, 1) - today return delta.days * How to handle types of values statically * Types should resolve to `class` statements * Type checking based on instance/subclass relations * Possible type inference for constructors / literals / assignments --- # Skeletons for Libraries * PEP 8: No stdlib annotations in the near future * Crowd-sourced Python skeletons database * [JetBrains/python-skeletons](https://github.com/JetBrains/python-skeletons) on GitHub * Simplified example of `datetime` skeleton class date: def __init__(self, year: int, month: int, day: int): pass def __sub__(self, other: date) -> timedelta: pass @property def year(self) -> int: pass ... --- # Apply Skeletons and Type Inference from datetime import date def until_new_year(today: date): delta = date(today.year + 1, 1, 1) - today return delta.days until_new_year(.error[(2013, 11, 29)])..error[days] * Typed skeletons for * `date.year`, `int.__add__`, `date.__sub__`, `timedelta.days` * Inferred types for * `today.year + 1`, `delta`, `return` of `until_new_year` --- # More Complex Types * Either class-based or untyped isn't enough * What's the type of `x` in these examples? # Duck typing def f(x): return x.read(10) # Collections x = [1, 2, 3].pop() # Higher-order functions def f(x, y): return x(y) # Function overloading / union types def f(x): b = x.encode('utf-8') if isinstance(x, str) else x return b.strip() --- # Types for Duck Typing * Abstract base classes (ABCs) since Python 2.6 * Overloading `isinstance()` and `issubclass()` in a metaclass * Unreadable, not declarative, hard for static analysis * The worst part is `ABCMeta.register()` class Sized(metaclass=ABCMeta): @classmethod def __subclasshook__(cls, C): if cls is Sized: if any('__len__' in B.__dict__ for B in C.__mro__): return True return NotImplemented ... >>> isinstance([1, 2, 3], Sized) True --- # Protocols * Structural subtyping for Python * Like interfaces in Go, TypeScript * Both run-time and static semantics from typing import Protocol class Sized(Protocol): @abstractmethod def __len__(self) -> int: pass >>> isinstance([1, 2, 3], Sized) True def not_empty(x: Sized) -> bool: return len(x) > 0 --- # Use Protocols for Duck Typing * Very simple to write and read class MyFileLike(Protocol): @abstractmethod def write(self, data: bytes) -> int: pass @abstractmethod def close(self): pass * Protocols are just better for stdlib ABCs * `collections.abc` to `typing.collections` * `numbers` to `typing.numbers` --- # Generics * Generic functions, collections, function types * Type variables and parameteric types are enough * Both run-time and static semantics from typing import List, Function, T, V from typing.collections import Iterable def singleton(x: T) -> List[T]: return [x] def map(f: Function[[T], V], xs: Iterable[T]) -> List[V]: ... --- # Internals of Generics * Define `__getitem__` for classes * Generics disappear at run-time T = typevar('T') # Some object class GenericMeta(type): def __getitem__(self, key): return self class Protocol(metaclass=GenericMeta): ... class Iterator(Protocol[T]): @abstractmethod def __next__(self) -> T: pass >>> Iterator[T] is Iterator True --- # Generics for Existing Types * Disappear only after `__getitem__` * Run-time semantics may be improved class typealias: def __init__(self, type): self.type = type def __getitem__(self, key): return self.type List = typealias(list) >>> List is list False >>> List[T] is list True --- # Typing for Python 2 * Python 2 and Python 3 * Type annotations are almost isomorphic to this decorator def annotations(**kwargs): def wrapper(f): if 'returns' in kwargs: kwargs['return'] = kwargs.pop('returns') f.__annotations__ = kwargs return f return wrapper def non_empty(x: Sized) -> bool: ... @annotations(x=Sized, returns=bool) def non_empty(x): ... --- # Typing: Work in Progress * Dynamic semantics * Do we need `isinstance` for types? * Do we need run-time checking of function calls? * Classes that refer to themselves * Impossible due to scoping rules * Use strings: `'C'` or refs: `ref('C')`? * Function overloading / union types * Several type signatures per function? * `@singledispatch` from Python 3.4? * Union types: `A | B`? * The use of `None` --- # Type System Summary * Untyped values * Class-based types * Protocol types * Generic types * Function types * Overloading / union types (WIP) --- # Mypy Language * Mypy – faster optionally typed Python variant * [mypy-lang.org](http://mypy-lang.org/) * Python program = Mypy program * Optimizations based on optional typing annotations * Python semantics and native semantics def fib(n: int) -> int: a, b = 0, 1 while a < n: a, b = b, a + b return a --- # Mypy Compared to * Cython: CPython extensions made easier * Language not compatible to Python * C language types, not for general use * PyPy: Faster via JIT compilation * ~6 times faster on benchmarks * Not enough data for further optimizations? * JavaScript world * Python + Typing ~ TypeScript * Mypy ~ Dart * But syntax remains compatible --- # Typing: Wrap Up * Optional type system for Python * Python skeletons for annotating libraries * Support for PyLint, PyCharm, etc. * Faster Mypy language --- .center.middle # Q&A [code-quality@python.org](mailto:code-quality@python.org) [github.com/JetBrains/python-skeletons](https://github.com/JetBrains/python-skeletons) [mypy-lang.org](http://mypy-lang.org/)