Anomalies or Observed Discrepancies in the Model
An example of one concern that was hard to evaluate or account for without further and much more refined research can be illustrated by the anomaly between the asapm/IPMA C and B levels. While the hours of experience remain the same, the difficulty of the projects managed increases. That increasing level of difficulty is not captured in this model, although theoretically, it would be captured during the assessment process. Also, the asapm/IPMA C requires both an exam and an assessment, while the asapm/IPMA Level B does not. This results in the IPMA C, with a PSCOR of 11106 actually scoring higher than the asapm/IPMA Level B, with a PSCOR of 11033, while the asapm/IPMA B is a higher-level credential than the asapm/IPMA C.
A related issue we find in the asapm/IPMA C is that this exam requires both multiple choice and written short answer questions. We find a similar issue with AACE's Certified Cost Engineer Credential (CCC/E). Not only does this exam require multiple choice questions that require fairly extensive calculations to derive the right answer, but one quarter of the value of all AACE exam based credentials require a narrative or short essay report. Are written narrative or essay type answers more difficult than multiple-choice questions? While we may believe intuitively that they are, the model presented here is not sufficiently refined to pick up those nuances.
Another challenge we faced in creating a relative scoring model was considerable inconsistency in how the value of a 4-year degree was determined by the various organizations.
Again, we will use both PMI and AACE certifications as examples based on our first-hand knowledge. PMI requires 7500 hours of experience for anyone with less than a 4-year degree and 4500 hours of experience for a person with a 4-year degree. Simple subtraction yields the value of a 4-year degree according to PMI as being 7500 - 4500 = 3000 hours of work experience. Compare this to AACE who require 8 years of full time experience, i.e. assuming working full time at 40 hours per week x 50 weeks per year equals 16,000 hours. Since AACE allows 4 years of the 8-year requirement to be fulfilled by a 4 year degree, then by performing the same calculation, the equivalent value for a 4-year degree for AACE is 16,000 - 8,000 hours = 8,000 hours.
How can this wide variation, 3,000 equivalent working hours for a Bachelor degree for PMI versus 8,000 hours accepted by AACE for a 4-year degree, be kept from skewing the results? To mitigate this large discrepancy in how the different organizations calculate or determine the value a degree, we chose to create a standardized value for a degree and to apply that in all cases where a degree was required. Note, however, that not all certifications required a degree - see the previous calculations. Based on the level-of-effort, the bachelor's degree (BDEG) was equated to 5,000 hours, while the Masters Degree (MDEG) was set at 1920 hours.
Now is it fair or reasonable to equate an hour spent in university to an hour spent working? Probably not, but what is or should be the relative weighting?
Lastly, if we take the PSCOR for all credentials and compare it against just the level of effort for all credentials, we find that the level of effort comprises on average, 85%, while the testing and assessment process only comprises 15% of this scoring model. Again, does this make sense or should the testing and assessment process count for more? Intuitively, it seems as though the answer should be yes, but that too will have to be the subject of more research. Moreover, we have to ask whether that means that the academic and experience is being weighted too high or that the testing and assessment processes are too low and need to be more robust to increase their weighting factor?
Obviously, more research is needed.
|
Malcolm Gladwell's "Outliers"
