Many functions in Python use or generate in-memory lists for iteration.  This is fine for relatively small list, but for large lists it becomes a problem.

Lazy Lists

A lazy list is when the next item is supplied when needed, and not before.  If we stop processing before reaching the end of the list then those unused items will never be generated or use up resources. It is also possible to run an endless list. Lazy lists are implemented by supplying an iterator to the list, rather than the whole list itself.

Simple Example

def simp():
    yield 10
    yield 20
    yield 30

it=simp()
print(it.next()) # 10
print(it.next()) # 20
print(it.next()) # 30

The function simp returns a generator , calling next() return the next value

We can use the generator like a list object:

for item in simp():
    print(item)

Instead of building a big list, just generate the next value:

For example, return a list:

def nogen(num):
    res=[]
    i=0
    while i<num:
        i+=1
        res.append(i)
    return res

s=nogen(10);
print (s)
# [1, 2, 3, 4, 5, 6, 7, 8, 9]

for i in nogen(5):
   print(i)

And with generator:

def withgen(num):
    i=0;
    while i<num:
        i+=1
        yield i

s=withgen(10);
print (s)
# <generator object withgen at 0x1093786e0>

for i in s:
    print(i)

Note: I didn’t use a for loop because in python 2 range() function returns a list and in python 3 returns a generator (use xrange() in python 2 to get a generator)

Endless sequence 

Sometimes we receive data from a stream and want to handle it like a sequence , with generator its easy:

def simp():
    num = 0;
    while True: # endless loop
        num+=10
        yield num

it=simp()
print(it.next()) # 10
print(it.next()) # 20
print(it.next()) # 30

You can replace it with a function that receives a message from a socket and return it etc.

Co-routines via Enhanced Generators

You can use the send() generator method to sends a value back to the generator function, which can be picked-up as the return value from yield().

When the send() method is not used, yield returns None.

Example

def simp():
    firstnum = 0
    while True:
        num = firstnum;
        while True:
            num+=10
            firstnum = yield num
            if firstnum:
                break

it=simp()
print(it.next()) # 10
print(it.next()) # 20
print(it.next()) # 30

print(it.send(200)) # 210
print(it.next()) # 220
print(it.next()) # 230

As you can see, using send we change the next value. The inner loop is finished, num gets the value we sent and continue

List Comprehensions as Generators

An alternative syntax to yield, we can use list comprehension , use parentheses instead of squared brackets. You can’t send value to the generator

Example

def fn():
    return (item for item in [1,2,3,4,5])


x=fn(); # returns 

print(x.next()) # 1
print(x.next()) # 2
print(x.next()) # 3

Implementing a Custom Iterable Class

You can write a new class that behaves like an iterator. For example if your class holds a data structure and we want to provide a way to iterate over its internal structure without knowing how it implemented.

Example

class simpIter(object):
    def __init__(self, limit):
        self.limit = limit
        self.all = 0
        self.nums = []

    def __iter__(self):
        return self

    # Python 3 compatibility
    def __next__(self):
        return self.next()

    def next(self):
        if self.all < self.limit:
            cur = self.all
            self.all += 1
            return cur
        else:
            raise StopIteration()

it = simpIter(5)
for n in it:
    print n 

s = sum(simpIter(10))
print (s) # 45

We use a simple list inside the class but we can change it to any other data structure without worry as long as we reimplement the inner class functions