Friday, December 5, 2008

Echoing stuff in your .bash_profile can mess up sftp and scp

In my .bash_profile I typically print out my PATH so I can see it when I login. This can cause problems when trying to use an SSH-based file transfer protocol to get files out of this machine.

Using sftp you might see something like "Received message too long 1819238756"
Using scp you will likely see nothing at all - it just won't work.

Comment out all echo statements in your profile and the problem should be resolved.

For more info:

Wednesday, November 26, 2008

Automatic litterbox reviews: It makes YOU the biggest LitterMaid of all!

Americans have a strong faith that technology will improve their lives, so it's always fun to see our patience get tapped by a poorly designed product. My favorite example of this is the line of LitterMaid automatic or "self-cleaning" cat litter boxes. When I need a good laugh I just sort the latest LitterMaid reviews on Amazon by "Rating: Low to High". Here are some choice quotes:

LitterMaid LM500 Automated Litter Box

I read all the reviews saying this thing sucked, and I thought, Well I'm an engineer, I can handle it but I was Wrong. I even got the older model LM500 hoping that it would work, but it didn't.
In the automatic rake thing, there are several plastic parts put, not glued or secured together. The first couple of times the rake went offline (because of excessively large dump - this happened at least 4-5 times per day despite filling litter to lower than the mark and using premium litter etc), the little plastic parts would pop out and the rake would go offtrack. Once this started happening, the plastic pieces would pop off every time. I tried glueing it, rubberbands, etc. Nothing worked. I feel really stupid for having read all the reviews and still buying it. Look at the content of the reviews. People describe intimately what is wrong with the product (rake broken, off track, etc) and how they tried to fix it. This should tell you how involved the owner got to try and get it to work. For me this meant getting cat piss all over myself, litter in my eyes, and messing with it everyday until the day, only three weeks after I got it, when it refused to work whatsoever.
Now I have this gigantic plastic piece of cat piss smelling garbage sitting around that I have to repackage and mail back. This is just not worth it. My brother has one, and it works, but it's just luck of the draw whether you get it to work or not. But judging from all the negative reviews, odds are against you.
I got more [poop] flicked in my face in 1 month then in 15 years in the veterinary business.
Emptying the poop containers flicks sand and poop in you face and mouth. Also, litter rake would clump and you had to clean it by hand. The plastic must have been poop bonding material because you could never get it clean. After 2 weeks , it started throwing poop clumps acrosss the floor. Somehow the rake got stuck and preassure woud build and thwack! Poop and urine would fly. don't waste your money
In particular, whatever plastic is used seems to have an affinity for soiled cat litter, as it sticks to everything, especially the rake, making a mess. Remember the idea is that you don't have to deal with handling cat litter directly anymore, right?

I am now on the 2nd Cat Tent as the cat jumped on it one time and the first one broke and I had to use duck tape to fix it. THis new cat tent is better but is very weak for the price u pay for the plastic and plastic materal that is the tent.
Now we're cleaning the litter box more than ever! We groan with despair when we know we have to go up to see the condition that it's in. To look and see poo stuck to the rake along with clumps of other stuff. Not to mention the loose litter all over the edges of the box and on the floor outside the box. We bought the crappy tent that goes with it but that tore during assembly and now there's loose litter on the bottom of the tent. The entire area must be vacuumed every other day. If you don't mind your cats stepping in clumps of broken up pee, then this product is for you!
The motor was SO noisy that we could hear it go on when we were all the way upstairs, the noise scared our new kitten so much, he refused to use it. Our most mischievious little feline devil discovered what he needed to do to turn the motor on and that became his favorite sport, he'd activate it and watch the rake go back and forth, so the lifespan of the batteries was about 3 days in our house. Our older cat, who's very fastidious, even for a cat, steadfastly refused to use it at all because it smelled. He'd simply go to the spot where the old box used to be and wail. I should point out that the smell was distributed throughout our home when the box was raking. We needed to replace the motor after only 3 weeks because the casing broke, its a complicated process that takes more time than it should since you're presummed to be without a litterbox when it occurs. Cleaning it was a bigger chore than a regular litterbox and took over half hour because you had to dismantle parts of it to clean it completely. The motor couldn't be submerged in water, so taking it to the backyard and hosing it down like I did my regular box wasn't an option, it was scrape, scrub, scrap, scrub, scrape, scrub, etc every 2 or 3 days. It was a disgusting and smelly job, and you'd end up with flecks of litter all over your shirt and FACE. My husband and I would actually argue about who got to clean the mess. Many times the rake would just scoop whatever litter wasn't stuck to the sides and head onto the floor, but since we had the tent, it was scooped inside the tent, so on a daily basis, that had to be dismantled and swept out, because if we didn't, one of the cats would squeeze between the box and tent to use the litter that was deposited there.
Everything about cleaning this unit causes you to have to get cat waste all over yourself.
First off it scared my cats to death. One of them refused to use the litter box ever again, even after switching back to a conventional scoop one. Next, the litterbox would literally fling poo. I would give it 5 stars if we were rating it on range!

The machine only lets you put in a shallow amount of litter or it will clog as the tines rake through it. This means you have to really watch the litter level. If it is too high the machine jams and the motor runs back and forth over and over again trying to push against a pile of litter. The trouble is, with such a shallow amount of litter [1] if your cat is in the habit of using the same place in the box everytime the litter gets REALLY shallow, and [2] even with a full load of litter, the litter does not clump when your cat has a large pee. The pee soaks to the bottom and sticks, and the litter stays slightly gummy, and sticks to the tines of the rake, and the machine jams. Yes I used premium litter. To make matters worse, my cat liked to pee right on the edge of the litter by the tines where the floor of the box ramps up and there is nearly no litter. instant mess.

If your cat's stool is soft that day, and/or your cat does not properly cover it, the poo smears and sticks all over the rake and it jams. And it is horrible to clean. I have has cat feces jammed way up under the tines on the side where the motor is.
It is as if Littermaid tried to design a product to expose you to as much used cat litter as possible.
the rake has just recently adopted a new trick - CATAPAULTING large clumps of poop across the kitchen floor! It's amazing how far those clumps can fly. When I was making a sandwich just now I found some small litter clumps that had landed on the kitchen counter.
Really look at the picture and think how you would like to clean poop from between all those tines on the rake.
I hate this product.
My wife and I received one of these as a gift and were initially very excited. The excitement, however, wore off when the cat stopped using the box and instead used the basement floor directly in front of the box.

LitterMaid LM900 Mega Self-Cleaning Litter Box
Mine was quite entertaining when I first purchased it because it came with a unique, added bonus feature. Every time the rake would get up to the disposal container, it would hang up and then forcefully catapult it's precious cargo across the room. I would be sitting there watching turd showers as they fell like meteors after being flung from inside my closet.
This thing is, no pun intended, a total piece of crap.
My cats don't like the noise it makes, and I think they've gotten scared of it, since one of them has started to poop just outside of the box, and the other one started to pee on the bathroom rug (he's 5 years old, and has never had a problem before)
I have admit this is a great idea, again underline/capitalize the word IDEA.

Well... these are made for cats to use, are they not?
My cat, who also enjoys just setting off the sensor so that the rake will pass through the litter, has learned that a great way to get me out of the bed in the middle of the night is to build up a little mountain of litter so that when the rake passes through it and attempts to return to its start position it cannot and the machine will continue to run back and forth, making a sound very similar to nails on a chalk board, until someone (me) evens out the litter.
I read several reviews before buying this product. They seemes very mixed, but I hate changing the litter box, and I thought, "Hey, that person who talks about the cat poop flying 10 feet in the air has GOT to be lying!" Well, sir, that person was not. I, too, have seen the flying cat poop. In fact, the day some hit me in the face was the day I decided to return this!
Countless times I have been awaken at midnight by the loud whine of this contraption struggling to empty the contents until it finally stops after four or five rounds of noise unable to achieve its goal
I don't know what kind of toxoplasmosis you can get when it flings a chunk of cat litter in your eye, but I'll know soon.
I have had for about 5 months now, and this morning at 4am, it broke. The whole arm with the rake and the motor broke clean off, and I can see no way to repair it. However, after it broke, the motor continued to run and run until I got up and turned it off...I am surprised after reading all of the bad reviews that LitterMaid has not improved their product. I am also surprised they are even still in business.
What you have to understand is that the rake mechanism that sweeps up the clumps of litter runs along a line of sprocket-like plastic ridges...ridges that get jammed with litter every time it operates.
is lovely if the waste isn't in any way still damp but if it is watch out! With the rake-like feature it dug through the waste and stuck in the scooper all the time. Then it would reverse and push the unscooped waste into the mechanisim and it was just gross!
The Incubus, or, How a Litterbox Ruined My Life
This is possibly the worst product ever fashioned by human hands. When it does work, which is rarer than the appearance of most astronomical phenomenon, all the tines of the rake serve to do is spear chunks of cat feces and carry them back to the return position, leaving nothing in the waste receptacle. Spending just five minutes scraping it off is enough to make me want to stab myself with one of said tines, that I might become infected with a deadly strain of ecoli, and be permanently freed from this abortion of a product.
I have asked the Litter Box God to please tell me the "Littermaid secret". I give up. The secret is apparently I AM THE LITTERMAID!...I really regret not asking for a new poop shovel and taken the $100 and my cats could be pooping in gold dust. Wow. I did not know I was that mad. Sorry.
I have spent more money on litter boxes than clothes.
Louder. Louder Than Hell. My wife and I bought one of these infernal machines and next to nails on a chalkboard and Ashley Simpson, the shrieking that comes from the slow-dying motor is absolutely one of the worst noises I can think of. I have been woken from a dead sleep many nights to the tune of "clogged feces." If you haven't heard this tune, let me describe it for you. Combine a five year old playing the electric violin for the first time with Fran Drescher and add in a few seconds of dead silence in two or three 15 second intervals.

Tuesday, November 25, 2008

Calculating an N50 from Velvet output

In sequencing circles the N50 length is a useful heuristic for judging the quality of an assembly. Here is my definition of N50 length, which you may or may not find intuitive:
N50 length is the length of the shortest contig such that the sum of contigs of equal length or longer is at least 50% of the total length of all contigs
For example's sake imagine an assembler has created contigs of the following length (in descending order):
91 77 70 69 62 56 45 29 16 4
The sum of these is 519bp, so the sum of all contigs equal to or greater than N50 must be equal to or greater than 519/2 or 259.5
We can see by brute force that
91+77+70+69=307 (that'll do)
so the N50 for this assembly is 69bp

Another way to look at this:
at least half the nucleotides in this assembly belong to contigs of size 69bp or longer.

N50 vs N50 length

Technically N50, as opposed to N50 length, refers to the ordinal of that last contig that pushes it over the brink - in this example 4 (since 69bp is the 4th largest contig). Unfortunately, a higher N50 implies the opposite of a longer N50 length. Some papers refer to N50 length as L50, while most have simply followed the lazy convention of dropping "length" off of "N50 length". I think it is important to include units with your N50 to minimize confusion.

Contig N50 vs Scaffold N50

Another distinction is often made between contig N50 and scaffold N50. Contigs are "contiguous segments", while scaffolds (aka supercontigs) consist of contigs separated by gaps. Scaffolds are constructed using paired-end information at the read level and, in major sequencing projects, paired BAC ends. Because the scaffolds sequences are filled with varying quantities of empty N's, the scaffold N50 should not solely be used as a comparative score of assembly quality.

Velvet, when used with sane expCov settings, is very conservative with regard to scaffolding - so much that the contigs.fa N50 can be virtually considered a contig N50, as opposed to a scaffold N50. More aggressive programs, such as SOAPdenovo, produce separate contig and scaffold files.

Velvet N50 from stats.txt

Velvet is a popular assembler for short sequences that uses DeBruijn graphs and Eulerian graph theory instead of a repetitive align-consensus-align approach. Although it returns an N50 in the course of assembling, I wanted to derive it from the contigs themselves. These contigs are summarized in a table called stats.txt

Using R and its cumulative summation function we can easily compute N50.

Strategy: Order the contigs by decreasing size and find the first value for which the cumulative summation is at least half the total sum.

Thanks to Barry Rowlingson for providing this solution

n50<-contigs[cumsum(contigs) >= sum(contigs)/2][1]

Beware the kmer

Note: The length in the stats.txt file is given as length=lgth+kmer-1, where kmer is the kmer value chosen for that assembly. The N50 length given in the Log file also appears to be in kmers. You cannot convert an N50 in kmers to bp by adding kmer-1. The math doesn't work like that - you need to convert each contig to bp before recalculating N50.

Finally, you can calculate N50 from sequences in the contigs.fa, but this file only contains contigs longer than 2-kmers by default. The contigs.fa bp-N50 will sometimes approximate the kmer-N50 in the Log file, but that is not a rule to depend on.

Monday, November 24, 2008

Writing a decent cover letter

Times are tough, and many of us will soon find ourselves back on the job market. Many people ask me how I've gotten so many jobs for which I am clearly not qualified. A lot of the credit goes to the excellent cover letters I have written over the years. A decent cover letter sets the tone for a successful interview. I'll try to boil down the key practices that I try to emphasize throughout the entire interview process:
  1. Be yourself - be honest about your strengths and weaknesses
  2. Don't let a position's requirements stand in your way
  3. Make your expectations clear
Here is a cover letter I helped my friend Erskine write for a lab tech position that recently opened up:
To whom it may concern,
I am very interested in your Research Specialist position. Although I have none of the skills you listed, I am confident my overwhelming sense of entitlement will win you over.

A series of bizarre and unfortunate lab accidents has left me without the use of the left side of my body. I also have syphillus, but that is a long story. Anyway, when the insurance money ran out I decided it was time to get back in the game. Word to your mother.

The only thing I request, other than the $85k/yr, is that the lights in the lab be dimmed when I am working - too much stimulation gives me seizures and/or provokes violence in me. I am also profoundly racist, so please do what you need to do in that area. I will reveal my gender to you, in private, when I feel the time is right.

I have tried to enlist references but their phones appear to be blocking my number so I doubt you would have better luck. The gang at Top2Bottom knows me pretty well so just ask around.

To make a long story short, Erskine got the job!

Friday, November 14, 2008

Errata to XML for Bioinformatics by Ethan Cerami

This book is a great introduction to parsing XML and XML-based webservices for bioinformatics. Unfortunately a lot of the URLs in book have changed and some of the services listed in it have already died. I will try to provide the updated examples below as I continue through the book.

If anyone has the source code to this book please send it to me!

pg 9,1200
change to,1200

pg 12 table 1.1
BioML (looks abandoned)

pg 46
change to
it is not clear that EBI still produces BSML output, more info at

pg 142
TIGR DAS Viewer Sounds a lot like gbrowse
BioJava DAS Client

pg 153,30876100
you need to use hg16 or newer

pg 162
looks like wombase is using biomart queries instead of das
thanks to Michael Han, Wormbase Group, Sanger Institute

pg 249
change to[@symbol=BRCA2]

Wednesday, October 8, 2008

WSJ: Nearly 100 in 100 Grocery Store Shoppers "Under Water" on Their Purchases

From the Wall Street Journal:
Continual depreciation of consummables in the US economy has left nearly 100% of grocery store shoppers owing more than the value of their recently purchased products. Buyers of food and beverages including meats, poultry, fish, eggs, beer, candy, soda, bread, milk, fruit, vegetables, and snacks have experienced severe losses on the resale market. "Nearly every item we sold today has lost value," says ShopRite produce manager Doreen Marshall, "It would be nearly impossible to recoup these investments on eBay."

Thursday, October 2, 2008

Groovy classes and scripts

Groovy has done a lot to make Java more palatable, but at the same time has introduced some confusion as to what developers can get away with at the most fundamental level.

Living on The Main Line
In Java, main methods are typically, if not always
public static void main(String[] args)
In Groovy, methods are public by default and args is typed dynamically, so you can get away with
static main(args)  //ok
but not
static main() //not ok
void main() //not ok
void main(args) //not ok
attempting to run a class with these bad main method declarations, the compiler will invariably return
Caught: groovy.lang.GroovyRuntimeException: This script or class could not be run.
It should either:
- have a main method,
- be a JUnit test, TestNG test or extend GroovyTestCase,
- or implement the Runnable interface.
IntelliJ is clever in that it can recognize the proper declaration on the fly. If you missing the required main method declaration you will get the normal Groovy class file type icon

If you have a valid main method you will see the runnable arrow

Groovy scripts
I have some sturdy bags I take with me to the grocery store. If I buy lot of food they just give me a paper bag in which to put the extra items. Every item is treated the same regardless of which bag they put it in.
A Groovy script is essentially any file with some code outside the confines of a class. Scripts can contain their own classes but all the stuff outside those is given its own class based on the filename and placed inside a working main(args) method. They just give you another bag - it's no big deal.

The following Groovy class and Groovy script are equivalent.

Groovy class with main method
class aGroovyClass {
def myString = "I am an instance variable";

static main(args) {
def myObject = new aGroovyClass();//def required
println myObject.myString;
Groovy script with class
class aGroovyClassWithinAScript {
def myString = "I am an instance variable";
myObject =
new aGroovyClass(); //no def needed
println myObject.myString;

Multiple main methods - you had to ask

Can classes inside a script have their own main method? Umm, yes, but that is pretty confusing (even to IntelliJ). I would have to explicitly call that main method from my class to use it - main is not a constructor.

Class with its own main method.

IntelliJ again
Both Groovy scripts and Groovy classes with main methods should be run as Groovy Script in the Run/Debug Configuration. Sometimes IntelliJ will default to attempting to run it as a Java application. If you attempt to run a Groovy script as a Java application you will get the following error:
Exception in thread "main" java.lang.NoSuchMethodError: main

Tuesday, September 23, 2008

Is this where the $700 billion figure comes from?

There are about 51 million first mortgages in the United States right now — but only about 1.4 million of them are either referred for foreclosure or in foreclosure, said Mortgage Bankers Association chief economist Jay Brinkmann. In other words, fewer than 3 percent of all American homes are in foreclosure.
If the average home is $1M and the bank can recover 50% of the value (or $500k and only recover 25%???) - that means $700B will in fact buy out all these bad mortgages.

I bet someday they'll reveal that's really where this stupid number came from.

Friday, September 19, 2008

Best name for a software project
And to think I was waiting for dick-wad to get out of beta.

Beware of LPDEST

When LPDEST is set in your shell profile it will make that printer the one used by lp, even if another default has been chosen systemwide. Fair enough - every user should be able to choose their favorite printers. However, this confusingly (IMO) alters what is returned by lpstat as "system default destination". For example:
(imagine somewhere hidden in .bash_profile is LPDEST=wrongPrinter1)

>lpstat -d
system default destination: wrongPrinter1
#ok let's fix this
>sudo lpadmin -d rightPrinter2
>lpstat -d
system default destination: wrongPrinter1
>sudo lpstat -d
system default destination: rightPrinter2
#root and me see two different "system defaults" on the same machine
>lpadmin -d
Permission denied: not in group 14
#i guess only root can change the real system default
>export LPDEST=rightPrinter2
>lpstat -d
system default destination: rightPrinter2

Perhaps most hair ripping will be that the GUI print settings will show rightPrinter2 as the default but most programs will print to wrongPrinter1. I see the origins of a good April Fools prank here. A better design would allow lpstat to be cognizant of user and default system preferences and label them accordingly.
(I use Solaris10 YMMV)
For more info:

Thursday, September 18, 2008

What wrong with this Groovy method?

def setSymmetricDifferenceTheWrongWay(s1, s2) {
def sd = s1;
def tmp = s1;
return sd;

Remember Lists are still just objects in Groovy as the are in Java, not primitives. Assigning a new reference to an object does not copy the object. You need to clone them.
def setSymmetricDifferenceTheRightWay(s1,s2){
def sd= s1.clone();//remember these are objects, assignment without clone is not copying
def tmp=s1.clone();
return sd;

Senior Bioinformaticist

Senior Bioinformaticist
Company: Vanderbilt Microarray Shared Resource Location: Nashville, TN 37232
Salary/Wage: 32,000.00 - 60,000.00 USD /year Status: Full Time, Employee
Job Category: Biotech/R&D/Science Occupations: Biological/Chemical Research;Mathematical/Statistical Research
Relevant Work Experience: 1+ to 2 Years Career Level: Experienced (Non-Manager)
Industry: Biotechnology/Pharmaceuticals

Apply Online
Monster recommends using Apply Now. Learn more

I'd hate to see what they are offering the junior bioinformaticist.

Wednesday, September 17, 2008

BerkeleyDB with Java

This is assuming you are not running the pure BerkeleyDB Java edition
Make sure you rebuild BerkeleyDB with
../dist/configure --enable-java
must include /usr/local/BerkeleyDB/lib

If you do not rebuild you will get java.library.path errors even though is clearly installed. The BerkeleyDB guide is not very helpful in this regard. Why wouldn't Java access be enabled by default. It is very frustrating.


I really wanted to get the Infragistics NetAdvantage to work as I would like a "modeless popup" - as opposed to modal. I want one that does not disable the background.

In the tutorial:
To install and deploy the sample application on Sun Java™ System Application Server 9.1: [a bunch of instructions ]

In the "Known Issues and Breaking Changes":

Known Issue with JSF 1.2 and Sun Application Server 9.1

The NetAdvantage for JSF 2008 volume 2 with support for JSF 1.2 release does not work in Sun® Application Server 9.1

Gee thanks for the heads up. Do Not Taunt Happy Fun Ball.

Wednesday, September 10, 2008

Are we really making progress with JSF?

This is a tutorial on how to open a link in a new window using Java Server Faces. Hello 1995.

Why do the developers of JSF think they can dispense with hyperlinks? We are having major pains here because our end users can't actually bookmark pages. Now I need to use a very foreign package (Tomahawk's Extensions Filter) just to pass an argument to a JSF page in a new window.

Yes javascript is not a lot of fun to write or debug, but sometimes the medicine is worse than the disease. Someone needs to do a comprehensive shootout between the rich JSF implementations and extJs. There must be some middle ground to abstracting javascript without losing so much basic functionality.

Tuesday, September 9, 2008

Perl to Groovy Translator

Yes I am actually looking for one as I have decided to stop writing Perl completely and start migrating over old scripts as the need arises. This is a cumbersome task so it would be nice to have something that did the most basic translation to keep me from find-and-replace hell.

Why groovy over python or ruby?
1. Groovy has seamless access to all the Java libraries without making me deal with the sometimes unbearably verbose Java syntax.
2. Grails, the Groovy web framework, has quickly taken advantage of work done on Rails. I see little reason to use Ruby just for Rails unless Grails simply cannot catch up soon. I don't see much work on the web framework side of Python. Even the GWT is Java based.
3. IMO the IDE support is better for Groovy already.

However I must really be in the minority on this one from the google results:
perl-to-python: 48000 hits
perl-to-ruby: 3800 hits
perl-to-groovy: 1 hit

I am interested in seeing more Groovy benchmarks come out. Honestly most of the bottlenecks I have encountered in my career are database related, so I am not that concerned if a script takes 50% longer to run.

Will there ever be a Groovy for Bioinformatics textbook? I suspect before that happens Java will incorporate Groovy syntax to the point most will simply call it Java 8 or perhaps ABJ - "Almost Bearable Java"

Monday, September 8, 2008

What was the name of that hot air balloon accident movie?

The one where a hot air balloon balloon accident survivor becomes obsessed another hot air balloon accident survivor because he let go of the rope?
A. Hot Air Balloon Accident Movie
B. Why Did You Let Go of the Rope?
C. Rising Suspicion
D. Enduring Love

Gnome settings manager crashes at startup

This problem took a few hours for me to fix as I got sidetracked doing fsck stuff for awhile, somehow believing my disk was corrupt. I hope this saves someone else some trouble:
Gnome would crash on start up when I logged in as myself - it would crash with
The GNOME Session Manager (process ####) has crashed due to a fatal error (segmentation fault)
I could log in as root
I could start up Gnome apps like firefox
Others could log in with no problem

The problem is related to my LD_LIBRARY_PATH
I had /opt/sfw/lib BEFORE /usr/local/lib

Obviously one of the newer libraries or possibly a corrupted version of the same library was messing things up. From doing a compare of these directories in UltraEdit, I suspect one of the following culprits:

Putting /usr/local/lib first in the LD_LIBRARY_PATH corrected the problem. I would use /usr/local/lib for everything but there was a history of that being automounted when we were using Sparc machines.

If Firefox 2 crashes at the default browser dialog

and leaves an error related to
from the commmand line (I am running Solaris 10 x86)
Make sure you are running
/opt/sfw/lib/firefox/firefox-bin (or whatever your path is)
and not simply
/opt/sfw/lib/firefox/firefox, which is a startup script that may contain some fixed paths like
which you may not have installed into