Following up on the topic of stopped horn in Brahms, there is a related issue in the famous “March to the Scaffold” movement of the Symphonie Fantastique of Hector Berlioz. In some editions the part is marked Con Sordino. In some recordings it is played muted, in others it is open. What did Berlioz want?
Berlioz also wrote an extremely influential book on orchestration, his Grand Traite d’Instrumentation et d’Orchestration Modernes (1843). In it he clearly addresses a “dangerous abuse” carried out by some horn players.
Many composers show themselves opposed to this new instrument, because, since its introduction into orchestras, certain horn-players, using the pistons for playing ordinary [natural] horn parts, find it more convenient to produce by this mechanism, as open notes, notes intentionally written as closed notes by the author. This is, in fact, a dangerous abuse; but it is for orchestral conductors to prevent its increase; and, moreover, it should not be lost sight of that the horn with pistons, in the hands of a clever player, can give all the closed sounds of the ordinary horn, and yet more; since it can execute the whole scale without employing a single open note. Since the use of the pistons, by changing the key of the instrument, gains the open notes of other keys, in addition to those of the principal key, it is clear that it must also secure the closed notes.
This passage is especially relevant to his own Symphonie Fantastique (1832), movement IV, where in a note to the published score (1845) Berlioz requested at the beginning of the movement that the horns “produce the stopped tones with the hand without using the valves” (“faites les sons bouchés avec la main sans employer les cylindres”); this instruction is almost universally ignored today. [See Hector Berlioz, Fantastic Symphony (New York: W. W. Norton, 1971), ed. Edward T. Cone, 122, for this marking, not seen in all editions of the work (I first saw this noted by Jeff Snedeker in his dissertation on the valved horn in France; see pages 77-79)].
So what Berlioz wants instead of muted horn is for the horn players to produce the combination of open and closed notes that would occur on the natural horn. It really is an interesting effect in this context, which is one reason I love period instrument orchestral recordings such as the recording of the Symphonie Fantastique done by Roger Norrington and the London Classical Players. My copy is on vinyl (!) but it is still available as a CD or download (at Amazon for example) and very worth checking out. Also I have a more extended article on Berlioz and the early valved horn which may be accessed here in my Horn Articles Online site; this article (based on materials from my dissertation) also lists the full sources for the quotes above.
As to what to do today, while Berlioz clearly wanted a different effect, I would be inclined toward doing what the part set you use says, playing it either muted or open. The original effect is unfortunately almost impossible to duplicate without natural horns.