Letting CAT2009 out of the bag (and 40 Million USD in outsourcing)

I have some serious problems with the way IIM has been restructuring entrance tests. The fiasco on the first day of the computerized CAT 2009 (Common Admission Test) proves that there is some substance in these fears.

• 40 Million Dollar outsourcing contract to Prometric
  40 million dollars is not small change for IIM or Prometric. It’s well above 160 Crore Rupees.  The contract includes creating the tests and every other step in the testing process. The value of the deal was a surprise to me. [Promteric is an American company who's presence in India has been limited to certification for professionals wanting to work in the US and in delivering the GMAT and similar examinations to Indian students wanting to study in America.]
  

• The Model
  Based on my understanding, Prometric has testing centres which are operated by agents/franchisees.Example of list of centres (look for the word "franchisee").
  
  The model will call upon franchisees to run testing centres. Unless the checks and balances are very tight, cheating may become possible. Run a google search for GMAT and cheating and you will see that there are weak links in the chain.
  

• Computerization creates an uneven playing field
  A vast majority of Indians did not have the privileged upbringing that I did. A computer is still a distant luxury. One class of students may have an unfair advantage in terms of familiarity and access to technology. This in-turn helps them in preparation. To use an analogy from sports, many Indian sportsmen are unable to compete at an international level because of a lack of basic training infrastructure. Are we handicapping a group of students in this rat race by moving to a computerized model. Is there a cultural bias here?
  
  
• Right to Information
  The contract between Prometric and the IIMs is not in the Public Domain. Considering that IIMs are a Government supported institution this is odd. A recent RTI application to IIM-A by an RTI enthusiast was shot down. I can imagine that there could be a few confidential clauses like the location of “test engines” but these could be blacked out. Transparency is obviously not something that the IIM’s are keen on. The RTI application is now with the CIC.
  

• An NDA with candidates
  There are noises on the web that candidates appearing for the test are signing an NDA. What this NDA says is that the students cannot discuss questions and question patterns with anyone. This throws water on the concept of collective learning and learning through other peoples experiences. Of greater concern is the fact that, despite having a contract valued at 40 Million USD, IIM is not able to get Pro-metric to develop a question bank large enough to avoid a need for a NDA. To put it crudely, what they are saying is, we have a limited number of questions..... so please do not tell anybody else what we asked you. Why pay them so much then?
  
  
• Outsourcing a key competency?
  You don't have to be an IIM-grad to know that key competencies should not be outsourced. IIM has done well because it has been able to select bright candidates. It has paid a lot of attention to the quality of questions and patterns of questions in the paper. I can understand outsourcing the exercise of conducting the exam, but why outsource the task of building the test database? That should be done in-house even if it requires investment. (Note: I have never been a fan of the CAT model - I am an "essay" type and not "multiple choice").
  
  
• Cultural Bias?
  GMAT has had a cultural bias. However, that is fair, since it is meant for Admission to American Schools and the majority of the test takers are American. No complaints there. However, I wonder if they will be able to ensure that they avoid this cultural bias when they are producing questions for Indian students. For example many Indian students who appear for CAT would have studied in a vernacular medium and the vocabulary used in the questions has to take this into account. I hope Prometric and IIM have factored this bias into their contract.
  
  
• Incomplete Model?
  Since CAT results act as cut-off for the interviews, the Model is incomplete. Prometric's American clients, for example, call for exhaustive essays from candidates. These essays are used in conjunction with test scores to judge whether or not a candidate is suitable for admission. In the IIM scenario, there is too much emphasis on the CAT scores and too little emphasis on other aspects that are unlikely to come out in a group discussion or a quick fire personal interview. 

In a nutshell, a) I think IIM has to be far more transparent about it's relationship with vendors for CAT excercise b) I am not convinced that the tests and computerized model is unbiased to under-privileged applicants c) The admission process needs to include more qualitative checks.

Disclosure -

Sour Grapes? I admit I have never been a fan of IIMs. In 2003-4 I attended an interview for IIM-Indore and the questions were right out of a "dummies guide to intervieweing". I attended the interview and the exam at the insistence of family. In 2007 a colleague at an MNC that I worked for said that I was suffering from a "sour grapes" complex when it came to the IIMs. So I went ahead and registered with IIM-A for an MDP. At the end of the programme, my opinion had not changed one bit. They are good but nowhere near great.