Take Candidate A. I don't particularly care what his values or politics are. He wants to run as a Democrat and registers as a Democrat. Fine. He signs the paper. Perhaps he's far left, far right, dead center, or some motley assortment of what are seen as conflicting views. Don't care. He's filed the paperwork.
Now, a party has to be built on its membership, loosely defined enough to include regular voters and regular contributors as well as those who regularly participate in party activities in a way that makes it a viable organization at the local level in a way that allows it to coordinate and help maintain the state and national organizations.
Notice, that's a lot of constraints on what the party as a whole can say. If it adopts a platform that a large portion of the party disapproves of, it may be virtuous and pure in the minds of some but it will stop being a functioning party. More commonly, it finds a compromise to allow the party to exist. In the absence of the party, the other side automatically wins, so it does turn into a question of whether the (temporarily) perfect is the enemy of the good. Surely the compromise views of that party would still be distinct from the opposition, and therefore superior in a relative way. Half a loaf is better than none.
It's possible for the party to cede a certain geographic area (since elections are geographic critters) to the opposition and just say, "We are unable to field a candidate or maintain an organization; let the other side have the field." That's fine in limited amounts. Let it go too far and the field yielded to the opposition might well become bigger and bigger. However, this state of affairs must be temporary. Seldom is the populace so united as to be entirely in one party; it does seem that we're wired for discontent (a phrase I read today). If one party dominates and all others yield, a new party will arise in opposition. This has happened in the past. Perhaps the party of the pure hangs in there in impotent perfection, more virtual than virtuous, perhaps it folds entirely.
So we're forced to say that ultimately the litmus test isn't something that's decreed top-down or by a minority, however vocally gifted or powerful. The litmus test that matters is going to be who the membership vote for in most of the territory and most of the elections that are held. And the party is ultimately going to be a kind of compromise--some percentage may want it to be further right on some issues, some percentage further left, but by and large the overwhelming majority is okay with the overall trend. If not, they'll leave, and if the loss of membership is too great it will cause the party to reconsider as it loses votes, organizational assistance, and funding. Perhaps it will reduce it to a trivial party and make room for a third: "creative destruction", so to speak.
Most of the time, though, the party simply adapts to maintain a sufficiently large plurality, at the least.
So if Candidate A with whatever values he brings to the podium is unacceptable to the voters in the party he claims to be a part of, he will lose in the primary. No other litmus test is necessary. If he's dead-on with the local membership, he will win, and by winning likely gain the support of a majority of those actively voting. This has to have an affect on the leadership, because he's a low-level leader at that point and a portion of the people that the party claims to represent support him. He'll take those values to the general election if he wins, and if he wins in the general's he's a more important leader and represents a larger portion of the population--including, possibly, people who were in the opposition. That helps build the party, even if it might pull it from the extremes at times--at other times, it'll pull away from the opposition to be more extreme.
Repeat this over the entire country, at local, state, and national levels, and you will get a mix of values represented. That's inevitable. The key is finding a way for all those people, with diverse values, to get along. And the key is the same as with any other diversity: You have to accept that people are different and that's not necessarily evil or bad, and you have to accept that they bring a strength that a portion of the party you are in think is worth having around. And there will be people you just don't like because they're "too diverse."
The one legal litmus test says he has to have the right letter after his name for being on a ballot. The real one is, "Does a majority of the party vote for him?" This isn't a litmus test that is imposed or set up by committee or ideology. It's the same litmus test that holds for Democrats now as in 1850 and 1950, whatever the changes in the values and platform, as well as for the Whigs and Republicans and every other political party that's existed in a democratic system. It's the same litmus test that will exist in 2050 if there is a US and political parties, and in 2150.
There is another option: Schism, because one group thinks the other group is heretical. Perhaps it's changed too much, perhaps it hasn't changed enough. It pays to remember that the word "heresy", so prominent in ecclesiastical circles at times, means "faction" as opposed to the unity that the church is to represent.
When people say "no litmus test," they mean imposed on the party that's ultimately going to be democratic in a way that is not democratic but demagogic.
= new reply since forum marked as read
