Some Python programmers like to think of their language as flawless. I personally know some Python programmers claiming that Python is superior to other languages in its clean design and unmatched elegance.
This however isn’t true. Python has at least as many deep and fundamental flaws as most other languages, despite parts of its community claiming something different.
Scoping in Python is as broken as you’d expect a half-baked implementation of lexical scoping atop of a half-baked implementation of dynamic scoping to be.
Take a look at this Scheme code.
(define (modifier x)
(lambda (f)
(set! x (f x))
x))
The equivalent in Python would be the following.
def modifier(x):
ref = {'contents': x}
def closure(f):
ref['contents'] = f(ref['contents'])
return ref['contents']
return closure
Non-global variables from a parent scope in Python are accessible as read-only. (Except when they aren’t, but more on that later) This means, that only the values and not the variables themselves can be mutated, which means more needless boilerplate (don’t worry, we’ll talk about nonlocal soon).
“But what about using classes?” I hear you say. Can’t we fake closures using them? As it turns out, no we can’t.
Many Python programmers would write the above “function” like this:
class modifier:
def __init__(self, x):
self.x = x
def __call__(self, f):
self.x = f(self.x)
For those of you like me, who are blind to it when they see it: This is so called elegance.
You may wonder where the problem with our elegant solution is. Like all other Python programmers, let’s just assume it does exactly the same and continue our journey through the surprisingly wonderful world of Python programming and briefly take a look at classes.
class A:
x = 5
a = A()
a.x = 10
print(a.x)
This is cool, isn’t it? Let’s go ahead and add a method.
class A:
x = 5
def ten(self):
self.x = 10
a = A()
a.ten()
print(a.x)
Nice! As it turns out, functions defined inside of a class simply become methods, where the first argument becomes the object itself. Knowing that, let’s make a Counter class using our modifier function/class.
class Counter:
update = modifier(0)
def __call__(self, counter):
return counter + 1
counter = Counter()
print(counter.update(), counter.update(), counter.update())
If modifier is defined as our true modifier function, this code works as you’d expect it to and prints “1 2 3.”
Using our modifier class however it doesn’t. (See, I told you it won’t work)
This is the kind of consistency you can expect from a well designed language like Python, and remember, that we just defined a “Counter” class so far as things already start to go downhill regarding clean language design.
The reason why this works with our true function is because Python interprets the assignment of a function inside a class as a method definition and does its first-arg magic. Since our class only creates an object, that looks like a function (since it is a callable) Python interprets it as a slot definition and tells us, that it is missing the first argument (that we expected to be the object itself) when we call the update slot (not method).
Explicit is better than implicit.
I mentioned earlier, that it is not possible to mutate a variable from the outer scope without boilerplate. Well, as it turns out, there is a semi-reliable way to do so. It’s Python 3’s nonlocal keyword and it rocks!
It allows us to write our modifier function like this.
def modifier(x):
def closure(f):
nonlocal x
x = f(x)
return x
return closure
As it turns out, this works perfectly fine! You can make every non-local variable available as readwritable in the current scope, if it is in a parent scope. The only problem is that Python is a bit strange about what exactly counts as a nonlocal.
Consider the following example (tested under v3.6.2).
x = 5
def f():
y = 10
def g():
nonlocal x, y
Now, does this code work? The answer is simple: perhaps. It all depends on its own level of indentation, because Python does not believe a global variable counts as a non-local.
Global variables are not non-locals, hence they are non-non-local, which makes them …?
By now you may have acquainted a taste for clean language design, but in case you haven’t, let me tell you: this is an instance of it.
I have to admit, that bashing Python based on its implementation of lexical scoping maybe wasn’t fair. Lexical scoping might just not be pythonic, so let’s talk about dynamic scoping in Python for a brief second, because Python’s implementation of dynamic scoping is extremely intriguing.
It is one of the lesser known features of the language, so I’ll give you a whirlwind introduction ot it.
See, in Python, a function not only as access to the variables in lexical parent scopes, but also to all of the variables in its callers. And for what its worth, Python takes its duty to support that kind of scoping to new extremes.
Let’s take a look at my all-time favorite function for a moment: eval. “eval” is a function (for the pythonic definition of “function”), that takes a string and evaluates it as an expression. It then returns its result. What makes eval interesting though, is that the variables in that string expression (which it – the function defined somewhere deep down in the Python implementation of your choice – evaluates) are dispatched to their definition as visible for the caller of eval.
That means, that Python functions can access their caller’s scope and even modify it.
Let’s see what fun things we can do with this:
def f(g, x):
y = 'hello world'
return g(x)
insanity = 'y'
print(f(eval, insanity))
This code snippet indeed prints out “hello world.”
Now I know, that many people think of eval as evil and that their opinion on dynamic scoping isn’t all that positive and open, but I personally don’t see a problem with supporting an obscure feature: there are use-cases for this.
So let us try to make our own eval function out of pure fun. (Something you can also have with an inherently broken system)
I start with the function signature:
def my_eval(string):
...
Python’s support for call stack manipulation is so good, we only have to write a C-extension in order to get that call-stack and …
I’m just kidding, Python ships with a powerful module called inspect, which allows you to get non-local variables on the call stack … as readonly. (Things being readonly except when they aren’t seems to be a common theme of Python)
This means we will basically have to write a C-extension, which provides us with an incallereval, which we can use to complete the code-snippet shown above.
There should be one – and preferably only one – obvious way to do it.
Speaking of obvious ways to do something, let’s take a look at the following code snippet in which we (try to) create a chain of functions, that print some values in a list. (Note, that the task is not about printing values, it is about creating functions)
values = map(lambda n: n + 1, range(5))
countdown = lambda: print(0)
for value in values:
def countdown(nextstep = countdown):
print(value, end = " ")
nextstep()
countdown()
At least this piece of Python code will do exactly what you expect it to and print “5 5 5 5 5 0.”
Yay? This is because for/in doesn’t make ‘value’ local to its body (watch my wording: I haven’t said “make a new child scope,” since that wouldn’t be possible for a whole list of other design oddities, like not being able to mutate variables from outer scopes without nonlocal boilerplate).
This however isn’t an issue because there is an – I suppose elegant – solution to this problem:
values = map(lambda n: n + 1, range(5))
countdown = lambda: print(0)
for value in values:
def closure(value = value):
global countdown
def countdown(nextstep = countdown):
print(value, end = " ")
nextstep()
closure()
countdown()
And now we get the expected “5 4 3 2 1 0.”
Well, unless of course your printer is a local variable, in which case you have to use the nonlocal keyword to access it instead of the global keyword required to access globals. (Clean design hitting us again, I suspect)
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
But hey, this isn’t that bad. At least you get access to the last element of the object you were iterating on using the loop variable after it finished. (I bet you use this feature all the time)
Now for what it’s worth and in Python’s defence, the alternative would be to introduce a kind of semi-scope to the language, which holds local variables, though where assigning to any variable but a few truly-locals would set them to the outer scope, which isn’t the cleanest language design either. You see, Python messed that one up much much earlier.
Next thing on the list: Object orientation. (Or how not to worry about it and happily override without any awareness whatsoever)
Let’s make a simple Python class.
class A:
x = 5
def f(self):
print(self.g(self.x))
def g(self, x):
return x + 1
a = A()
a.f()
As code prints 6 as you might expect. Now, imagine for a moment, that you weren’t the author of this class. (and I bet you use classes you aren’t the author of) You most likely don’t know all of the variables it provides and uses internally, and you most likely don’t know all of the methods it provides and uses internally.
But maybe you are a risk seeker and still inherit from it and write the following.
class B(A):
def __init__(self, x):
self.x = x
def g(self, x):
return x + 2
b = B(0)
b.f()
Well congratulations, you have successfully messed up the assumptions of A’s author, because your variable x collides with the A’s variable x and even your method causes A’s one to be overridden (in the worst case, without you noticing it).
Note, that this does not apply to methods prefixed with two underscores.
In oder to avoid that, you’d have to either be aware of all of the variables and methods of a parent class or you would have to use some weird naming conventions, like prefixing every variable and method with a class name.
It seems to me, that the conveniences of dynamic scoping are deeply intermingled into all the other aspects of the Python language as well!
But can we really blame the designers of Python for that? After all, coming up with a good object system is hard, some of them.
The ignorant programmer might thing, that it would be possible to just provide a makeobject function, that creates a fresh minimal object, and setparents function, that alters the delegation tree, and just go on from there, so that the A class from above would look like the following.
def A():
self = makeobject()
self.x = 5
self.f = lambda: print(self.g(self.x))
self.g = lambda x: x + 1
return self
a = A()
a.f()
The problem with this is that classes aren’t just functions, so a class decorator would have to be made. Also, syntax for setting nested structures through “def” should exist for convenience, then. (This however is not pythonic, as it would add to the consistency between “def” and “=”).
@makeclass(class)
def A(self):
self.x = 5
def self.f():
print(self.g(self.x))
def self.g(x):
return x + 1
Being ignorant, she might also think, that defining B would then be as easy as the following.
@makeclass(class, A)
def B(self, x):
self.x = x
def self.g(self, x):
return x + 2
b = B(0)
b.f()
She might also think, that explicit overriding would then be possible through a providerof primitive, that would get the concrete object in the tree, which implements that certain functionality, so that a version of B, where x shall be overridden would look like this, assuming a parentofclass.
@makeclass(class, A)
def B(self, x):
providerof('x', parentofclass(self, A)).x = x
def self.g(self, x):
return x + 2
She might also assert, that this system includes metaclasses, since “class” could simply be replaced by any other metaclass, whose constructor would be used as a decorator and that adding a static method would be as free of magic as simply adding a method to the class object using def.
Also, she might think, that this system is free of all of the mentioned inconsistencies, like the function/callable differentiation.
She’d be right about all of that. It really can be that simple.
Even the implementation of class and makeclass would not be black magic.
class = makeobject()
def class.__call__(parents, constructor, classobject = None):
if classobject is None:
classobject = makeobject()
setparents(classobject, class)
classobject.__bases__ = parents
# ...
def classobject.__call__(*args, **kwargs):
self = makeobject()
# Initialize the parent structure (a bunch of tree hopping and makeobjects)
constructor(self, *args, **kwargs)
return self
return classobject
class((class, object), class.__call__, class)
def makeclass(*parents):
return lambda constructor: class(parents, constructor)
You see, Python isn’t the well designed language proponents like to call it. It has its quirks too, and some of them won’t be fixed any time soon.