Prerequisite-using Token rebuild

Note: You should be familiar with the Background and Terminology provided on the PREREQ Cleanup page.

Purpose of this work

The scope of this work is to enable (but not actually implement) #2 in the Background section of PREREQ Cleanup. This work performs a set of changes to the lsttokens. There are 2 main reasons for this change: (I) Extract the (common) Prerequisite processing, to localize that information and make it resiliant to changes in individual token formats. (II) Enable the use of subtokens in all forms of Prerequisites

Detailed Reasoning

(I) Develop a Common Processing Location

Today, when a token needs Prerequisite processing, it has to perform that function itself. This means a set of checking to ensure that processing is properly performed with and without Prerequisites, and they are always applied to all objects. This delegates a material (though not crazy) amount work to each token that wants to support prerequisites, in terms of error checking, etc. This work would be potentially doubled by the addition of a Requirement system. To avoid having such a change impact all of the tokens, the processing of these types of items can be centralized to offload the complexity of Prerequisite processing from the individual tokens.

(II) Enable use of other characters in Prerequisites

With the current system, we have tokens of the form PRExxx:<stuff>, such as: PRELEVEL:5 In the new system, this could take two forms, a Requirement: REQ:LEVEL|5 or a True Prerequisite: PRE:LEVEL|5

Note three things: (a) The PRE and the REQ system are both tokens with subtokens, just like ADD, AUTO, and CHOOSE have subtokens, so it's familiar to LST users. (b) The PRE and REQ systems can share their subtokens (to ensure one set of work provides 2 benefits) and this can be done more easily if the PRE and REQ are the tokens and the subtokens are explicit (rather than deriving a tok of token names based on the Prrequisties that exist) (c) The form of REQ: and PRE: shown above are incompatible with today's tokens that chain prerequisites.

What do I mean by (c)? Today we have things like: ABILITY:FEAT|AUTOMATIC|Dodge|PRELEVEL:5 The PRExxx can get rather complicated (see PREMULT!) but has a rather hard and fast rule: It does not include a PIPE. Since PRE and REQ use a subtoken, they may have a PIPE, and we'd like to at least keep the option to use the PIPE in order to stay consistent with other subtoken-using tokens. In order to do this, we need to ensure the PIPE in the PRE/REQ token is never passed into the token for processing. (Note to the data team: This is NOT committing the new form, we have a NEWTAG process for that and such things end up on _experimental. This is strictly keeping options open.)

Implementation

Strategically, the target is to be able to support tokens of the form: ABILITY:FEAT|AUTOMATIC|Dodge|PRE:LEVEL|5 For now, we are not using PRE/REQ, so we are targeting ABILITY:FEAT|AUTOMATIC|Dodge|PRELEVEL:5

To discuss implementation, we will use a simpler token, namely DR. This is found in the package plugin.lsttokens as DrLst.java For purposes of this discussion, we will ignore the .CLEAR processing.

The token starts out with: StringTokenizer tok = new StringTokenizer(value, "|"); ...which as pointed out in the detailed reasoning of (II)(c) will potentially split in a bad place with the PIPE (just reinforcing that point)

We later get to the Prerequisite processing:

if (tok.hasMoreTokens())
{
  String currentToken = tok.nextToken();
  Prerequisite prereq = getPrerequisite(currentToken);
  if (prereq == null)
  {
    return ParseResult.INTERNAL_ERROR;
  }
  dr.addPrerequisite(prereq);
}

Which is followed by the load into the context and the return of SUCCESS.

The goal here is to completely eliminate the Prerequisite processing code shown above from DrLst.java.

We do this by: (A) Performing pre-processing on the incoming String value to remove the Prerequisites and store them until the token has processed. (B) Changing the result returned from the token to indicate where Prerequisites should be applied (if legal) (C) Applying the Prerequisites after the token has returned

(A) Pre-processing

Let's say the incoming String is: 5/Light|PRELEVEL:5 Today, the entire String is passed in. In the new implementation, we only want to pass in: 5/Light

So prior to the String being passed in, we do something of the form:

List<Prerequistite> preList = new ArrayList<Prerequisite>();
while (true)
{
  int pipeLoc = tokenString.lastIndexOf(Constants.PIPE);
  if (pipeLoc == -1) { break; }
  String preCandidate = tokenString.substring(pipeLoc + 1);
  if (//not valid PRExxx) { break; }
  preList.add(//valid preqrequisite)
  //Needs a test to ensure tokenString is not empty and if it is throw an LST_ERROR
  tokenString = tokenString.substring(0, pipeLoc);
}
//Do normal token processing here (parseToken)

(B) Change the returned result

Today DrLst returns ParseResult.SUCCESS. This changes that line to: return new ParseResult.PREREQ_ALLOWED(dr); Note what is happening here. All of the objects to which the prerequisites are attached (anything that was parsed by the token) are to be returned in the ParseResult.PREREQ_ALLOWED. This means one object (as in DR) or a list of objects (such as you might encounter in ABILITY). So the system needs to be able to handle either situation. Ownership transfer of the returned list is not significant as you will see in (C).

(C) Apply to the prerequisites to the result

Let's pick up with (from (A)):

ParseResult result = //Do normal token processing here (parseToken)
if (result.isPreReqAllowed() && !preList.isEmpty())
{
  prereqObjects = ((ParseResult.PREREQ_ALLOWED) result).getPrerequisiteObjects();
  if (prereqObjects != null && !prereqObjects.isEmpty())
  {
    for (Prerequisite pre : preList)
    {
      for (PrereqObject po : prereqObjects) 
      {
        po.addPrerequisite(pre);
      }
    }
  }
}
else if (!preList.isEmpty())
{
//Need an error here since we found PRExxx but the token says they aren't allowed
}

Notes about the Implementation

Gradual Conversion Possible

The design above obviously assumes it can unconditionally strip PRExxx tokens and all of the places they will be legal will return ParseResult.PREREQ_ALLOWED. This may or may not be reasonable depending on the time frame in which the implementing developer wishes to work. It is always possible to perform an identifying check (such as adoping a "tagging" interface) to know whether to perform the "strip" of the PRExxx for a given token. This is acceptable as a temporary measure, but that interface should be removed once the transition to all Prerequisite supporting tokens using ParseResult.PREREQ_ALLOWED is complete

Unit Tests

Note that it is likely that many of the unit tests for these tokens (Which are generally dependent upon assuming the token takes PRExxx stuff) will break, and need to be rebuilt.