Understanding the midpoint polygons of self-intersecting quadrilaterals requires use of the oriented, or algebraic, area. For a triangular region, this means that the area is positive if we order the points in such a way that we move counter-clockwise around the region, and negative if we move clockwise.

Figure 3. The triagle on the left is oriented in the counter-clockwise direction and so has positive area; the triangle on the right has negative area.

This actually gives us the oriented area for any planar polygon; we just subdivide it into triangles, calculate their areas, and sum them up. In fact, it doesn't even matter if parts of our triangles lie outside the region, as in the quadrilateral below:

If we travese the quadrilateral in the order that the vertices are numbered, we go clockwise around the red region, so it has negative area. We go counter-clockwise around the blue region, so it has positive area.

By dividing the region into two triangles along the diagonal (which lies outside the original polygon), we can easily calculate the total area of the quadrilateral. The triangle 123 has negative area, and the triangle 134 has positive area. Thus the purple region outside the quadrilateral is counted once positively and once negatively, and so does not affect our calculation, leaving us with nothing but the red and blue regions, which have the correct sign.

Once we have this decomposition, a simple adaptation of the proof for non-convex quadrilaterals shows us that the area ration for self-intersecting quadrilaterals is also 1/2.