Friday, January 31, 2014

Bounded Rationality

It has occurred to me that it would be irresponsible to not address the namesake of this blog at some point. Therefore, in brief here is an explanation of bounded rationality.

Economists generally prefer to view people as perfectly rational individuals.  A perfectly rational being is predictable and easily modeled.  However, at some point even the most die hard academic had to admit that people make suboptimal, or even downright poor decisions.  The acknowledgement of this fact led to a number of changes in economics, the most influential being the rise of Homo Economicus and bounded rationality.

Homo Economicus is a theoretical tool economists use in creating models.  He is a representation of humanity that is perfectly rational, narrowly focused and purely self interested.  He maximizes personal utility at every opportunity and never makes foreseeable mistakes.  Essentially Homo Economicus is a theoretical human who's behavior is easily defined in nearly any situation.  Economists all acknowledge that Homo Economicus is a fiction.  However, in many cases his behavior doesn't diverge all that much from what occurs in reality.  Thus we keep him around in a back closet and shamefully drag him out whenever we need to create a new model of auction pricing, consumer behavior or whatever we're working on that week.

The other concept, and namesake of this blog, is bounded rationality.  Bounded rationality is simply the idea that although humans are basically rational, a great deal of obstacles get in the way of that rationality.  Emotion, time constraint, lack of information, and inability to process sufficient information all are major obstacles to making optimal decisions.

For example, suppose Alice was shopping for health insurance. She has four choices with the following information:  Plan A has a lifetime cost of $6000  and a lifetime benefit of $5000, Plan B has a lifetime cost of $8000 and a lifetime benefit of $10,000, Plan C has a lifetime cost of $3000 and a lifetime benefit of $1000, and Plan D has a lifetime cost of $2000 and no lifetime benefit.

Given only this information it's easy to conclude that Plan B is the best. It's the only plan which provides a benefit that exceeds costs.  However, Alice may not choose Plan B for a variety of reasons.  The first and largest reason she may not choose Plan B is although we have defined future benefits Alice has no way to anticipate them so easily.  She can reasonably approximate future costs of the plans but healthcare utilization is an unpredictable factor.  Thus she can't know for sure what level of benefit each plan will provide.  Here her lack of knowledge has limited her ability to make an optimal decision.

Imagine Alice was only given a short time to choose her health insurance.  This time constraint would very likely further impede her ability to choose wisely.  Rather than conducting comprehensive research on costs and benefits Alice would probably fall back on a heuristic shortcut such as, "choose the cheapest plan that meets my anticipated minimum needs."  While this will often times lead to a selection which is "good enough" it's a decision method which has discarded the goal of optimal selection.  Thus time constraint has led to a non optimal selection method.

Alice may have had a difficult week at work and is too stressed to make a careful selection. She simply wants the decision made so she can put it behind her so she selects mostly at random.  Here emotion has greatly impaired Alice's rationality.

Finally, suppose each plan had innumerable payment schedules, reimbursement plans, copays, deductibles, and similar minutiae which populate health insurance plans.  Alice may very well not be capable of processing all of that information and making intelligent comparisons between plans.  In this case Alice's inability to process information adequately puts her in a situation where her decision is more an informed guess than an optimal selection.

Clearly there are many factors which may impair a person's rational thinking.  By saying individuals exhibit bounded rationality we are simply stating that a person is generally rational within the limitations of their knowledge and abilities.  When that knowledge is incomplete or those abilities insufficient to the task they may exhibit behavior which is clearly suboptimal from the standpoint of Homo Economicus.

That's all for this week. Until next time stay safe and rational.

Thursday, January 23, 2014

Researcher Integrity

Two years ago Adam Wilkinson was forced to retract a prominent fMRI study regarding language processing due to suspected fraud.  What follows is an interview conducted in November of last year in which he discusses that study.

Interview


Interviewer: Thank you for speaking with me Adam.  Obviously the retraction of your language processing paper in 2012 was a significant career setback. Some people allege that you knowingly manipulated data in order to get the results that would merit publication.  How do you respond to those allegations?

Adam:  At no time did I modify data in any way in order to manipulate results.

Interviewer: So no data was changed at any point?

Adam: That is correct.

Interviewer: Was any data discarded for any reason?

Adam: Some data sets were discarded as is routine during a study of this nature.

Interviewer: And what were the reasons for ignoring those data sets?

Adam: In some cases post scan surveys revealed characteristics of subjects that made them unsuitable for the experiment. Other data sets were removed due to excessive head motion or an inability to normalize their data with our chosen methods.

Interviewer: So in other words some people moved around too much or had previously unknown neuorological conditions?

Adam: Yes. Or their head orientation or brain shape did not easily conform with a standard fitting model.

Interviewer: If you had included those data sets would your results have remained the same?

Adam:  To include those data sets we would have had to adjust our analytic framework. It's possible that doing so would have altered the end results.

Interviewer: Several labs have attempted to replicate your findings but have as yet been unable. Do you believe your initial study to be in error?

Adam: In the time since publication it has become evident that our initial publication was likely incorrect.  Unfortunately there is always a small chance of a false positive in research and this time we got unlucky.

Interviewer: Your critics say that in this case you made your own luck.  They claim you selectively eliminated data and chose analytical methods which would lead to a positive result. In particular Nick Harrison has been quoted as saying, "Nearly any data set and analytic method except those used would have lead to a negative result." What's your response to that?

Adam: No doubt there are statistical techniques which would change the results of the study. I reject the idea that I chose my method in order to manipulate the result.

Interviewer: Was your method of analysis determined before data was collected?

Adam: It was not.

Interviewer: Was your hypothesis preregistered?

Adam: It was not.

Interviewer: So it would have been possible for you to choose a protocol and data set combination that would lead to a positive result.

Adam: Possible, but it did not occur.

Interviewer: How many subjects were included in this study?

Adam: Due to budget limitations we were only able to include fourteen participants.

Interviewer: How many participants had their data discarded for any reason?

Adam: Two.

Interviewer: Is twelve subjects enough to reach significant conclusions?

Adam: Due to the high cost of fMRI studies twelve subjects is not unusual. It's possible that it led to an under-powered study but it is not unusual.

Interviewer: Does a small group of participants make it easier or more difficult to change results by altering analytic protocols and discarding data sets.

Adam: Experiments with few participants are altered by a greater degree when a small number of subject's data is discarded.  In that way it would make altering the results easier.

Interviewer: Were you under pressure to publish something significant at the time of publication?

Adam: In academic research there is always a pressure to publish.

Interviewer: So there was no unusual pressure at that time?

Adam: The high cost of fMRI studies makes all such studies a significant investment. It would have been disappointing to have invested so much and not discovered anything of consequence.  Beyond that, no.

Interviewer: Some have claimed that you didn't initially share your raw data in an effort to cover up fraud. Was that the case?

Adam: Data sharing in the field is a rarity more than the norm.  As soon as the publication was called into question I made the raw data freely available to anyone who requested it.

Discussion


Luckily Adam Wilkinson and his research are fictional.  He was made up in order to demonstrate the ease by which a researcher can guide their studies results.

In most fMRI studies some data is discarded.  It's an unfortunate side effect of using such a precise instrument on subjects who have difficulty remaining still.  An unscrupulous researcher, such as Adam, can determine which data sets will move results towards a predetermined end when eliminated. 

Similarly there are dozens of analytical methods which a researcher can use to perform statistical comparisons.  Each of these methods will give slightly different results and some will almost certainly give a positive result. Here's an example.

Let's say Adam had thirty people scheduled to receive scans.  After five people have been scanned he begins to check his data between scans to see if he has a statistically significant result. On the twelfth scan the data barely tips into the statistically significant range and Adam cancels the remaining scans.  The data for scans 1-11 didn't show what Adam was looking for but due to an anomaly on the 12th scan Adam got lucky and the data worked out for him. Had scans 13-30 been conducted the data would have returned to the mean and Adam wouldn't have the result he sought.  Since Adam didn't preregister his hypothesis or protocol he can peek at the data for any number of statistical tests and stop whenever any of them dip into the range of significance. Unfortunately after the fact there's really no way to tell if Adam did this or not.

FMRI data is by it's nature very "noisy". It requires quite a bit of pre-processing before it can be analyzed in a meaningful fashion.  How this pre-processing is done can wildly change results.  Adam can adjust his pre-processing parameters in a large variety of ways in order to manipulate his results.  As long as Adam keeps his parameters reasonable no one will question this aspect of his research.

FMRI studies look at a particular region of interest in the brain.  If Adam is conducting research on language processing than there's a general region that should be his focus. But the exact defined area of that region is a matter of Adam's discretion. Does he outline a region that's just a little more anterior than usual thus including a bit of a the visual cortex? If so he can show activation in his region of interest that really is attributable to the visual cortex, not any sort of language processing. This sort of deception is detectable but only with close inspection by a proficient expert. Even then Adam could present it as a simple error that wouldn't change results.

What comparison does Adam make? Does he compare individuals to their own baseline scan or to a group normalized scan? Does he group subjects into male and female and conduct comparisons within those groups only? Perhaps divisions are created based on gender and age or on age alone.  Each comparison made allows Adam another chance at a statistically significant result. Since he didn't state his hypothesis before starting the experiment he can test as many things as he can imagine until one by chance comes up as significant.

It's easy to see that given all these factors a researcher has thousands of possible end results.  The sheer volume of possibilities nearly guarantees that some null hypothesis can be rejected and the experimenter can find a publishable result.  Most importantly all of this can be done without "changing" any of the data. 

Adam's answers in the interview would in most cases be typical of a researcher in his field.  Small studies, without preregistered protocols, and with hidden data sets are routine in fMRI research.  Unfortunately this presents a ripe opportunity for researchers to manipulate their protocols in order to meet publishing pressures.  

Many researchers would argue that fMRI research is by it's nature exploratory and observational.  However, it is dishonest to make such a claim while at the same time hiding behind the all mighty p-value.  Either you are hypothesis testing, in which case a p-value is valid and you know your hypothesis before hand or you're conducting an observational study and a p-value isn't relevant.  In this case you can't have your cake and eat it too.

That's all for this week. Until next time stay safe and rationale.

Saturday, January 11, 2014

The Economics of Cutthroat Kitchen

During the holiday season a coworker insisted I watch a new Food Network show called Cutthroat Kitchen. The show follows a standard chef game show format in which four chefs must cook a given menu item.  A guest judge then determines which chef's dish was the worst and that chef is eliminated.  The show proceeds through three rounds, eliminating one chef a round until the final chef is crowned the victor.

Here is the interesting twist meant to keep audiences coming back for more.  The chefs are given their prize ($25,000) at the start of the show and may spend that money in auctions during each round.  Auctions are generally for sabotages which can be given to other contestants to make their life more difficult.  For example, if the item to be cooked was meatloaf and mashed potatoes a sabotage might be to force another contestant to make mashed potatoes with potato peels or to make meatloaf with unground meat.  When losers are eliminated they return their prize money to the host whereas the winner keeps whatever they didn't spend over the course of the show.

As someone who cooks frequently the show has no educational value.  The techniques or tricks the chefs use aren't discussed.  Instead the focus is put on how a chef deals with the obstacles heaped upon him.  While amusing, it's unlikely your average viewer will ever need to cook a pancake on top of a tin can.  Thus, unless your cookware is stolen by the international culinary mafia you probably won't learn much.

As an economist however, the auctions are fascinating.  It's rare to see so blatantly how bad we are at making financial decisions under pressure.  Here is a fictional, but representative, example of how cutthroat kitchen often goes. Remember each contestant has $25,000 to spend in auctions. Any amount they don't spend will be their prize if they win.

Allen, Beth, Candice and Dave arrive at the set.  They're told they'll be cooking a pizza. As usual they have thirty minutes to complete the task after they grab their ingredients from the shows pantry.  Everyone scrambles to grab the items they'll need to make a crust and sauce as well as some cheese and toppings to cover the pizza with.

Shortly after cooking begins the host starts to auction off sabotages. The first auction (of an expected four) is for the ability to force another contestant to make their sauce out of unripe tomatoes. Beth and Dave are both terrified of having to use unripe tomatoes and so bid against each other until Beth wins the auction for $5500.  She awards the tomatoes to Candice.

Auction two is won by Dave who forces Allen to have to grate his own cheese from a block rather than using preshredded. Auction three is won again by Dave who forces Beth to choose from some common ice cream toppings instead of pizza toppings for his pizza. Finally auction four is won by Candice who gets revenge on Beth by making her cook her pizza on a griddle instead of in an oven.

The end result is Allen has to grate his cheese from a block. Beth has to use ice cream toppings instead of pizza toppings and cook her pizza on a griddle. Candice has to use unripe tomatoes in her sauce and Dave is unsabotaged.  Additionally, Allen has spent nothing, Beth is down $5500, Candice has spent $5000, and Dave has spent $10,000. Judging commences and Candice is eliminated as the judges felt her pizza was the least delicious.

Round two commences and Dave is eliminated. Allen and Beth remain with $22,000 and $18,000 respectively.  After the final round Beth is crowned the victor with winnings of $10,500.

Using this fictitious series of events we can point to a number of poor decisions contestants made along the way.  In the first round Beth and Dave were willing to spend a great deal in order to avoid getting the under ripe tomato sabotage.  This is a very common behavior as expressed by contestants.  They often are willing to bid on an auction just to guarantee they are not stuck with the sabotage.  However, in the first round losing an auction generally only costs a contestant a one third chance of being hit with the penalty (as the winner could award it to any of the three other contestants). Given this probability the prices contestants pay is borderline absurd. Contestants routinely bid as if losing an auction is a guarantee of receiving a penalty, a belief which is clearly incorrect.

Another oddity is that contestants are often most willing to spend during the first round of the contest.  Bids of $4000 to $6000 are not uncommon in the first round while in rounds two and three items will routinely sell for under $3000.  This is partially attributable to the fact that contestants that are loose with their money quickly deplete it or are eliminated thus leaving only more reluctant bidders with the assets needed for such lofty bids.  Still, it's difficult to understand why these big spenders expend so many resources so early, thus leaving themselves vulnerable in later rounds where they're more likely to be hit with sabotages.  A possible, if unlikely, explanation is they're attempting to put themselves in a position of weakness so that other players will award sabotages to stronger opponents.  However, putting oneself in a position of difficulty in order to avoid difficulty seems rather nonsensical.

The most bizarre behavior is when contestants spread penalties relatively evenly among the other contestants.  Each round one person is eliminated.  The logical behavior is as soon as a person receives a sabotage all other sabotages should be heaped upon them.  This eliminates animosity in later rounds (as the contestant will have been eliminated) and is the best method to ensure continuation through the rounds.

It would seem there are two possibly strong methods of bidding intelligently on the show given logical behavior (which the contestants never seem to exhibit).  Either win the first bid or do not bid at all.  If you choose to win the first bid you then award it to another player who then is signaled as weak to the remaining players. If that player does not win the second auction it should logically be awarded to them as well and they are now in a position where they are at best as weak as the next weakest player if they win the next two auctions and award them both to the same person. Thus the sabotaged player is forced into winning the second auction or having a very high probability of elimination.  This method is aggressive and potentially resource intensive if played against another player with the same strategy.  However, when facing three passive players (as described below) the resource cost is low.  The considerable downside is whoever you award the sabotage to essentially must bid against you in the second auction or risk the piled on sabotages. If they win the second auction they're most likely awarding their sabotage to the person who sabotaged them in the first auction (you).  So while you may gain a cheap sabotage to use against someone else, you also likely gain an expensive sabotage used on you. Overall it is a gain against the player you sabotaged, but a loss against the two players who are uninvolved.

The second, and likely preferable strategy is far simpler and likely superior.  Do not bid on the first auction.  The winner only has a one third chance of awarding the sabotage to you.  Two thirds of the time you will be able to keep all your money and receive no sabotage for the first round (as all sabotages should go to the first person sabotaged or very likely the person who sabotaged them). In the unfortunate event that you are chosen for the first sabotage you are put in a situation where you must make another player as weak as you, meaning you must start bidding and winning auctions to weaken them.  However, with the exception of a likely higher cost of auctions two, three and possibly four in the first round as compared to auction one you are at no particular disadvantage against the player you choose to target.  Overall your expected earnings increase a great deal as well as your position in later rounds of the contest.

It would be interesting to conduct interviews with contestants in order to establish their motivations for how they award sabotages. Certainly the tendency to spread the penalties around seems to indicate contestants think more of equity than how best to improve their odds of victory.  Still, at times I wonder if at times their consideration amounts to no more than, "I don't want to be stuck with that. I'll buy it then give it away to someone more or less at random."  Regardless of motivation, the show is an excellent demonstration of how not to think about strategic decisions.

That's it for this week. Until next time stay safe and rationale.

Friday, January 3, 2014

The 25%

As Americans we are constantly reminded how fortunate we are that we're allowed to keep most of the money we earn.  After all, in many nations the public toils all year long and the greedy government takes the fruit of their labor to pay for wasteful social programs.

The reality of course is a bit less extreme.  The United States is near the middle of the pack when it comes to taxation as measured as a percentage of GDP (26%).  Many Middle Eastern and Asian countries tax far less (Saudi Arabia 5%, Hong Kong China 13%)  while many European nations tax more (Sweden 46%, Italy 43%).  Thus in terms of keeping the results of our efforts we're neither terribly fortunate nor victimized.

The upper crust of France however will be finding their baguettes quite a bit smaller for the near future.  Last Sunday the French high court upheld a 75% marginal tax on salaries over one million Euros ($1.3 million). The tax is largely symbolic as it is planned to last only two years and generate a relatively meager amount of revenue.  However, it certainly sends a powerful message to the public, "There is no limit to what the government can take from you."  Much like the Cyprus deposit confiscations this move can only terrify the wealthy while solving precious little.

I do not mean to give the impression that I believe taxes or governments are evil.  Nothing could be further from the truth. Taxes are a necessary component of modern society.  Further, this new French tax is hardly going to cause any millionaires to starve or even prevent them from upgrading their yacht next year (their first million euros are taxed normally). But, this move is a classic example of political machinations that will have real detrimental economic impact.

French president Francois Hollande himself has said that the tax is symbolic and unlikely to substantially impact France's problematic revenue shortfalls.  Why then institute a tax which will undoubtedly drive many wealthy French from the country, thus removing their tax revenue from the system entirely?  In the long term it's likely that tax revenue will decrease as once the tax expires two years hence those that emigrate now will still be gone.  Not to mention the extra reluctance of any wealthy considering moving into France.

What is gained by such a symbolic maneuver?  Hollande claims it's about economic fairness.  However, I doubt many people would claim that confiscating 75% of anyone's wages is particularly fair, especially if they receive nothing more than those who are taxed far less.  More likely this act of political theater will simply increase tensions between the haves and the have-nots while doing nothing to solve France's fiscal issues.

It makes sense that the wealthy should bear more of the burden of taxation.  Thus the logic of a progressive tax system such as we enjoy in the United States.  But to tax any person for three quarters of their wage, even if it is only for Euros earned over one million, is ridiculous.  To do so when it does nothing to solve revenue shortfalls is bordering on criminal.

That's all for this week. Until next time stay safe and rationale (and probably out of France).