python - Accessing class variables from a list comprehension in the class definition -

how access other class variables list comprehension within class definition? following works in python 2 fails in python 3:

class foo:     x = 5     y = [x in range(1)] 

python 3.2 gives error:

nameerror: global name 'x' not defined 

trying foo.x doesn't work either. ideas on how in python 3?

a more complicated motivating example:

from collections import namedtuple class statedatabase:     state = namedtuple('state', ['name', 'capital'])     db = [state(*args) args in [         ['alabama', 'montgomery'],         ['alaska', 'juneau'],         # ...     ]] 

in example, apply() have been decent workaround, sadly removed python 3.

class scope , list, set or dictionary comprehensions, generator expressions not mix.

the why; or, official word on this

in python 3, list comprehensions given proper scope (local namespace) of own, prevent local variables bleeding on surrounding scope (see python list comprehension rebind names after scope of comprehension. right?). that's great when using such list comprehension in module or in function, in classes, scoping little, uhm, strange.

this documented in pep 227:

names in class scope not accessible. names resolved in innermost enclosing function scope. if class definition occurs in chain of nested scopes, resolution process skips class definitions.

and in class compound statement documentation:

the class’s suite executed in new execution frame (see section naming , binding), using newly created local namespace , original global namespace. (usually, suite contains function definitions.) when class’s suite finishes execution, its execution frame discarded local namespace saved. [4] class object created using inheritance list base classes , saved local namespace attribute dictionary.

emphasis mine; execution frame temporary scope.

because scope repurposed attributes on class object, allowing used scope leads undefined behaviour; happen if class method referred x nested scope variable, manipulates foo.x well, example? more importantly, mean subclasses of foo? python has treat class scope differently different function scope.

last, not least, linked naming , binding section in execution model documentation mentions class scopes explicitly:

the scope of names defined in class block limited class block; not extend code blocks of methods – includes comprehensions , generator expressions since implemented using function scope. means following fail:

class a:      = 42      b = list(a + in range(10)) 

so, summarize: cannot access class scope functions, list comprehensions or generator expressions enclosed in scope; act if scope not exist. in python 2, list comprehensions implemented using shortcut, in python 3 got own function scope (as should have had along) , example breaks.

looking under hood; or, way more detail ever wanted

you can see in action using dis module. i'm using python 3.3 in following examples, because adds qualified names neatly identify code objects want inspect. bytecode produced otherwise functionally identical python 3.2.

to create class, python takes whole suite makes class body (so indented 1 level deeper class <name>: line), , executes if function:

>>> import dis >>> def foo(): ...     class foo: ...         x = 5 ...         y = [x in range(1)] ...     return foo ...  >>> dis.dis(foo)   2           0 load_build_class                    1 load_const               1 (<code object foo @ 0x10a436030, file "<stdin>", line 2>)                4 load_const               2 ('foo')                7 make_function            0               10 load_const               2 ('foo')               13 call_function            2 (2 positional, 0 keyword pair)               16 store_fast               0 (foo)     5          19 load_fast                0 (foo)               22 return_value          

the first load_const there loads code object foo class body, makes function, , calls it. result of call used create namespace of class, __dict__. far good.

the thing note here bytecode contains nested code object; in python, class definitions, functions, comprehensions , generators represented code objects contain not bytecode, structures represent local variables, constants, variables taken globals, , variables taken nested scope. compiled bytecode refers structures , python interpreter knows how access given bytecodes presented.

the important thing remember here python creates these structures @ compile time; class suite code object (<code object foo @ 0x10a436030, file "<stdin>", line 2>) compiled.

let's inspect code object creates class body itself; code objects have co_consts structure:

>>> foo.__code__.co_consts (none, <code object foo @ 0x10a436030, file "<stdin>", line 2>, 'foo') >>> dis.dis(foo.__code__.co_consts[1])   2           0 load_fast                0 (__locals__)                3 store_locals                        4 load_name                0 (__name__)                7 store_name               1 (__module__)               10 load_const               0 ('foo.<locals>.foo')               13 store_name               2 (__qualname__)     3          16 load_const               1 (5)               19 store_name               3 (x)     4          22 load_const               2 (<code object <listcomp> @ 0x10a385420, file "<stdin>", line 4>)               25 load_const               3 ('foo.<locals>.foo.<listcomp>')               28 make_function            0               31 load_name                4 (range)               34 load_const               4 (1)               37 call_function            1 (1 positional, 0 keyword pair)               40 get_iter                           41 call_function            1 (1 positional, 0 keyword pair)               44 store_name               5 (y)               47 load_const               5 (none)               50 return_value          

the above bytecode creates class body. function executed , resulting locals() namespace, containing x , y used create class (except doesn't work because x isn't defined global). note after storing 5 in x, loads code object; that's list comprehension; wrapped in function object class body was; created function takes positional argument, range(1) iterable use looping code, cast iterator.

from can see difference between code object function or generator, , code object comprehension latter executed immediately when parent code object executed; bytecode creates function on fly , executes in few small steps.

python 2.x uses inline bytecode there instead, here output python 2.7:

  2           0 load_name                0 (__name__)               3 store_name               1 (__module__)    3           6 load_const               0 (5)               9 store_name               2 (x)    4          12 build_list               0              15 load_name                3 (range)              18 load_const               1 (1)              21 call_function            1              24 get_iter                     >>   25 for_iter                12 (to 40)              28 store_name               4 (i)              31 load_name                2 (x)              34 list_append              2              37 jump_absolute           25         >>   40 store_name               5 (y)              43 load_locals                       44 return_value         

no code object loaded, instead for_iter loop run inline. in python 3.x, list generator given proper code object of own, means has own scope.

however, comprehension compiled rest of python source code when module or script first loaded interpreter, , compiler not consider class suite valid scope. referenced variables in list comprehension must in scope surrounding class definition, recursively. if variable wasn't found compiler, marks global. disassembly of list comprehension code object shows x indeed loaded global:

>>> foo.__code__.co_consts[1].co_consts ('foo.<locals>.foo', 5, <code object <listcomp> @ 0x10a385420, file "<stdin>", line 4>, 'foo.<locals>.foo.<listcomp>', 1, none) >>> dis.dis(foo.__code__.co_consts[1].co_consts[2])   4           0 build_list               0                3 load_fast                0 (.0)          >>    6 for_iter                12 (to 21)                9 store_fast               1 (i)               12 load_global              0 (x)               15 list_append              2               18 jump_absolute            6          >>   21 return_value          

this chunk of bytecode loads first argument passed in (the range(1) iterator), , python 2.x version uses for_iter loop on , create output.

had defined x in foo function instead, x cell variable (cells refer nested scopes):

>>> def foo(): ...     x = 2 ...     class foo: ...         x = 5 ...         y = [x in range(1)] ...     return foo ...  >>> dis.dis(foo.__code__.co_consts[2].co_consts[2])   5           0 build_list               0                3 load_fast                0 (.0)          >>    6 for_iter                12 (to 21)                9 store_fast               1 (i)               12 load_deref               0 (x)               15 list_append              2               18 jump_absolute            6          >>   21 return_value          

the load_deref indirectly load x code object cell objects:

>>> foo.__code__.co_cellvars               # foo function `x` ('x',) >>> foo.__code__.co_consts[2].co_cellvars  # foo class, no cell variables () >>> foo.__code__.co_consts[2].co_consts[2].co_freevars  # refers `x` in foo ('x',) >>> foo().y [2] 

the actual referencing looks value current frame data structures, initialized function object's .__closure__ attribute. since function created comprehension code object discarded again, not inspect function's closure. see closure in action, we'd have inspect nested function instead:

>>> def spam(x): ...     def eggs(): ...         return x ...     return eggs ...  >>> spam(1).__code__.co_freevars ('x',) >>> spam(1)() 1 >>> spam(1).__closure__ >>> spam(1).__closure__[0].cell_contents 1 >>> spam(5).__closure__[0].cell_contents 5 

so, summarize:

• list comprehensions own code objects in python 3, , there no difference between code objects functions, generators or comprehensions; comprehension code objects wrapped in temporary function object , called immediately.
• code objects created @ compile time, , non-local variables marked either global or free variables, based on nested scopes of code. class body not considered scope looking variables.
• when executing code, python has globals, or closure of executing object. since compiler didn't include class body scope, temporary function namespace not considered.

a workaround; or, it

if create explicit scope x variable, in function, can use class-scope variables list comprehension:

>>> class foo: ...     x = 5 ...     def y(x): ...         return [x in range(1)] ...     y = y(x) ...  >>> foo.y [5] 

the 'temporary' y function can called directly; replace when return value. scope is considered when resolving x:

>>> foo.__code__.co_consts[1].co_consts[2] <code object y @ 0x10a5df5d0, file "<stdin>", line 4> >>> foo.__code__.co_consts[1].co_consts[2].co_cellvars ('x',) 

of course, people reading code scratch heads on little; may want put big fat comment in there explaining why doing this.

the best work-around use __init__ create instance variable instead:

def __init__(self):     self.y = [self.x in range(1)] 

and avoid head-scratching, , questions explain yourself. own concrete example, not store namedtuple on class; either use output directly (don't store generated class @ all), or use global:

from collections import namedtuple state = namedtuple('state', ['name', 'capital'])  class statedatabase:     db = [state(*args) args in [        ('alabama', 'montgomery'),        ('alaska', 'juneau'),        # ...     ]]