Limited or Lazy: Materials and Methods are Lacking Methods.

Imagine asking a friend for a recipe to their homemade chocolate cake and getting one of the following two responses: First: Collect the material, mix with a KitchenAid Stand Mixer (Model KSM150PSOB) for 65 seconds with a whisk attachment, and bake using a GE wall oven (Model PK916SRSS) using the default settings. That is not going to be an easy recipe to follow.  What are the ingredients? Can I get the same results with a different mixer or oven? Default settings, these two words are the bane of my existence, they are meaningless and should be taboo from scientific writing.  The second response: It’s my grandmothers recipe (see Grandmas Cookbook, 1954).

What do these analogies have to do with science?  The more I read my peers’ papers the more I see trending towards incomplete Material and Methods sections. Typically, I find that there is a fairly even mix of the above mentioned strategies to explaining what was done in a paper.  Personally, I don’t find either explanation useful.

Below I will highlight some guidelines for writing Methods sections, some examples of useful and less useful descriptions, and some potential solutions for people writing their own papers.

The Goal: Before we start we need to know what we are aiming to achieve.  This may be the reason this problem has been perpetuating.  I will use Day’s How to Write and Publish a Scientific Paper as our benchmark.

…careful writing of the [Materials & Methods] is critically important because the cornerstone of the scientific method requires that your results, to be of scientific merit, must be reproducible; and, for the results to be adjudged reproducible, you must provide the basis for repetition of the experiments by others.

That is a perfect reflection of what someone should expect when reading about methods. Let’s look at two examples from published research and see how our peers have done.

Example 1: A method describing how oomycete cultures were prepared for experimental manipulation (Volz & Beneke, 1966)

Hyphal tips were removed from stock cultures and placed on agar medium containing 0.4 ~o maltose and 0.1 ~o peptone. After approximately one day, 5 mm discs of hyphae were cut out of the medium from the perimeter of the colony with a cork borer. Nine of these discs were then placed equidistantly in Petri plates. Distilled water was added to slightly submerge the plugs. Hemp seed halves were then placed on top of each agar disc.

Example 2: A method describing how gene predictions were generated (Denoeud et al., 2011).

GeneID [96] and SNAP [97] ab initio gene prediction software were trained on 300 genes from the training sets.

These examples were chosen to highlight the growing problem with our ability to report in the digital age.  As a group we are, generally, very good at putting our benchwork into text with enough detail that others could follow our experiments (Ex. 1).  As technology has taken a front seat with current research I feel that we haven’t developed a successful strategy for documenting our computer based work.

Added to the lack of effective communication we have journal page limitations, thus, further forcing us into brevity.  However, I don’t think that we can continue to use that as an excuse. With more and more journals going to online formats and the ability to create supplemental documents we must still adhere to the Scientific Method and provide detailed explanations.  According to Day, but not to the reviewers, Example 2 should not have been “adjudged reproducible.” I cannot take their gene set and return the same results using the information they supplied.  I have used SNAP in my own research and here are the steps I did to create one file:

What Example two MAY look like from my use of SNAP

  1. Moved to SNAP folder on computer for next steps
  2. Used inputs as described in MAKER readme; entered on Terminal command line
  3. ./fathom “filename”.ann “filename”.dna -gene-stats
  4. ./fathom “filename”.ann “filename”.dna -validate
  5. ./fathom “filename”.ann “filename”.dna -categorize 1000
  6. ./fathom uni.ann uni.dna -export 1000 -plus
  7. mkdir params
  8. cd params
  9. move ‘forge’ into params
  10. ./forge ../export.ann ../export.dna
  11. move ‘forge’ out of params
  12. cd..
  13. perl “filename” params > “filename”.hmm

I don’t think that ‘I used SNAP’ qualifies as “basis for repetition.” That is 11 different commands send to SNAP.  The example above shows only one program and only one file manipulation. Most modern genome papers use and average of 6 different pieces of software; and of those 6 most require other programs to function, compounding the problem further.  How do we report on these computational processes in a concise AND useful manner?

Solution. My first suggestion: error on the side of too much information.  If you’ve used a program you need to describe your settings.  For the commands found in my SNAP example above, I did exactly what was described in the MAKER readme file. In that case there would be no issue with stating “Custom HMM’s were created using SNAP as described in the MAKER readme.” However, if I’ve deviated in anyway from that procedure I must discuss those changes.

My second suggestion: if page limitations are an issue submit supplemental documents, or indicate where someone can go to get the detailed information required.  Dequard-Chablat et al. (2011) provide an excellent example of this.  Most journals don’t even print hard copies any more and all have SI online, do your part to make useful information available without worrying about destroying the rainforest.

My third suggestion: if you are going to cite previous work(s) make ensure the reference has done a proper job of detailing procedures.  Perpetuating a terrible description is lazy and a cop-out; do the right thing and provide a decent protocol (your paper could become the new “see” paper and up your citations). Do not daisy chain methods citations! If you say ‘procedures followed as in Calvin & Hobbes (2001)’ and I go to that paper and it links me to another reference it becomes very annoying and time consuming.

Conclusions: Ninety percent of the people that read your publication will not even look at the methods, so I know that putting in the proper amount of information seem futile. However, for the 10% that need to know what you did give half-hearted details hinders the progression of their work, and ultimately slows down our progress as scientists. We’ve all built off of previous efforts and know how difficult it can be to recreate our own experiments lets make things easier for our peers and future graduate students by doing what we are supposed to, and be as transparent as possible.

Day, RA. How to Write & Publish a Scientific Paper. 4th Edition. Phoenix: The Oryx Press, 1994.

Volz, P., & Beneke, E. (1966). An aquatic fungal bio-assay method for detection of carcinostatic agents Mycopathologia et Mycologia Applicata, 30 (2), 97-114 DOI: 10.1007/BF02130356

Denoeud F, Roussel M, Noel B, Wawrzyniak I, Da Silva C, Diogon M, Viscogliosi E, Brochier-Armanet C, Couloux A, Poulain J, Segurens B, Anthouard V, Texier C, Blot N, Poirier P, Ng GC, Tan KS, Artiguenave F, Jaillon O, Aury JM, Delbac F, Wincker P, Vivarès CP, & El Alaoui H (2011). Genome sequence of the stramenopile Blastocystis, a human anaerobic parasite. Genome biology, 12 (3) PMID: 21439036

Déquard-Chablat M, Sellem CH, Golik P, Bidard F, Martos A, Bietenhader M, di Rago JP, Sainsard-Chanet A, Hermann-Le Denmat S, & Contamine V (2011). Two nuclear life-cycle-regulated genes encode interchangeable subunits c of mitochondrial ATP synthase in Podospora anserina. Molecular biology and evolution PMID: 21273631

This entry was posted in Uncategorized. Bookmark the permalink.

4 Responses to Limited or Lazy: Materials and Methods are Lacking Methods.

  1. Pingback: Material e métodos « Mario A. Lira Junior

  2. Scott says:

    This should be disseminated far and wide.

  3. Sylva says:

    Great post! Very timely given the whole Gould – Morton thing right now:

  4. Lindsay says:

    Thank you for this post!

    I have this trouble, too, when reading very technical articles about genomic or gene-expression analyses. (For me, the software thing is less of a bother because I know I won’t understand *that* anyway, but sometimes I do want specifics of, say, exactly what kind of array they’re using, so that I can go find out how the probes in that array are spaced.)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s