The structure was originally designed, with two rows of seats about its four sides, to accomodate as spectators only the trainers, patrons and present and past officials of the games, the building was enlarged by Argyrius, who gave the games in 332AD, and then by Libanius' uncle Phasganius, who gave them in 336, each doubling the former capacity of the seats. The resulting admission of students, workmen and idlers of all sorts destroyed, Libanius says, the sacred character of the contests, which were even set at a later hour to suit the convenience of the new spectators". Downey adds that Proculus, the comes Orientis of 383-4 proposed a further expansion of the seating. This last proposal is what spurred Libanius to his outburst against riff-raff at the plethron!
The Xystos was another structure used for the Games. Malalas relates that this was built (or rebuilt) by Commodus (180-192AD). This is described in some commentaries as a covered track used for sports. In a footnote A.F. Norman suggests some sort of linkage with the Theatre of Zeus and references Roland Martin in Festugiere but my reading of Martin quite clearly states the Theatre of Zeus was in Daphne, moreover Norman attributes Martin as claiming that the Xystos and Plethron were in some way connected with the theatre. Maybe our French is faulty but R. Martin says in reference to the Plethron "où il est bien distingué des deux theatres". That sounds like the contrary view.
Both of these structures were in or around the Forum, which is usually named in honour of Valens, for his rebuilding activities.
I have uploaded Richard Foerster's edition of Libanius' Oration X: "On the Plethron" and it is available here: