The question is of selection as order is NOT important here. Selecting vertices A and B is same as B and A.

Number of ways to select two vertices is 8C2 =28. This is our total sample space or S.

Of these 28, 8 will be the sides of the octagon itself
Excluding them, the rest of the 20 are diagonals.
(We can also use the formula nC2 - n =Diagonals in a polygon)

Thus the required probability = 20/28 = 5/7 (OPTION E)