Doing some factoring out:

tot = disjoint + common_inner + common_outer + inner_self + outer_self
    = (N1 * (N1 - 1) * N2 * (N2 - 1) / 2) + 
      (2 * N1 * N2 * (N2 - 1) / 2) + 
      (2 * N2 * N1 * (N1 - 1) / 2) + 
      (N1 * N2 * (N2 - 1) * (N1 - 2) / 2) + 
      (N2 * N1 * (N1 - 1) * (N2 - 2) / 2)
    = (N1 * N2 / 2) * (
                        (N1 - 1) * (N2 - 1) + 
                        2 * (N2 - 1) +
                        2 * (N1 - 1) +
                        (N2 - 1) * (N1 - 2) + 
                        (N1 - 1) * (N2 - 2)
                      )
    = (N1 * N2 / 2) * (
                        (N1 * N2 - N1 - N2 + 1) +
                        (2 * N2 - 2) +
                        (2 * N1 - 2) +
                        (N1 * N2 - N1 - 2 * N2 + 2) +
                        (N1 * N2 - N2 - 2 * N1 + 2)
                      )
    = (N1 * N2 / 2) * (3 * N1 * N2 - 2 * N1 - 2 * N2 + 1)

This result is not consistent with the final equation given to us, which is N1 * N2 * (N1 ** 2 + N1 * N2 + N2 ** 2) / 2.

The diagram also seems to be wrong.

# FOR N1=3, N2=3, M=2

# Two disjoint cuts
disjoint = N1 * (N1 - 1) * N2 * (N2 - 1) / 2
# Yields 3 * (3 - 1) * 3 * (3 - 1) / 2 = 18

# Two cuts, each of which connects the two polygons, which meet at a common vertex of the inner polygon
common_inner = 2 * N1 * N2 * (N2 - 1) / 2 
# Yields 2 * 3 * 3 * (3 - 1) / 2 = 18

# Two cuts, each of which connects the two polygons, which meet at a common vertex of the outer polygon
common_outer = 2 * N2 * N1 * (N1 - 1) / 2
# Yields 2 * 3 * 3 * (3 - 1) / 2 = 18

# Two cuts, one of which connects the two polygons, one of which connects the inner polygon with itself
inner_self = N1 * N2 * (N2 - 1) * (N1 - 2) / 2 
# Yields 3 * 3 * (3 - 1) * (3 - 2) / 2 = 9

# Two cuts, one of which connects the two polygons, one of which connects the outer polygon with itself    
outer_self = N2 * N1 * (N1 - 1) * (N2 - 2) / 2
# Yields 3 * 3 * (3 - 1) * (3 - 2) / 2 = 9

# Total of the parts:
tot = disjoint + common_inner + common_outer + inner_self + outer_self
# Yields 72, not 117, as is provided by the total equation n1 * n2 * (n1 ** 2 + n1 * n2 + n2 ** 2 - 1) // 2

Since the diagram is wrong, I don't really know what the terms of the kata are, and I'm not sure how to proceed.

Additionally, can you draw multiple lines from any given p0 to p1? That isn't shown in the example diagram, but given the rules as written, it shouldn't be out of bounds.

Lines may begin and end at the same polygon, but may they also begin and end at the same point?

There are no random tests ;-)