$ cat file_to_covert.fq | perl -e \
'$i=0;while(<>){if(/^\@/&&$i==0){s/^\@/\>/;print;}elsif($i==1){print;$i=-3}$i++;}' \
> output.fasta

Thanks Edwards Lab!

Reblogged from Manuel Corpas' Blog:

File and directory naming conventions may sound like a trivial thing to do but I have found that their complexity increases exponentially when many components are run. To illustrate my current approach to tackling this problem, I present here a simple example. Suppose a project (‘project_name’) that runs two programs, ‘program_1′ and ‘program_2′. Each time the pipeline is run, input files may vary and so I create a new ‘job_name’ for each run. I have come up with this directory architecture:

/project_name
/project_name/data
/project_name/data/job_name_1
/project_name/data/job_name_1/input_data_type_1
/project_name/data/job_name_1/input_data_type_2
/project_name/data/job_name_1/input_data_type_3
/project_name/results
/project_name/results/job_name_1/program_1
/project_name/results/job_name_1/program_1/output_1
/project_name/results/job_name_1/program_1/output_2
...
/project_name/results/job_name/program_2/output_1
/project_name/results/job_name/program_2/output_1
...

What would happen if instead of running 2 programs as I did above I run 5 or 6? And what if for each input data file I had replicates? What about maximising the number steps taken in parallel? You can start to see that the thing really gets complicated.

File and directory naming conventions is something that I am teaching myself, but any directives or systematic methods taught during my computer science student years would have come in handy now. In future bioinformatics lectures I teach I will definitively challenge my students to think about this issue very carefully.

• Work around a server, one process, Gigabyte datasets.

Life in early 2012:

• Work around a cluster, many processes, Terabyte datasets.

I remember the old days, when I had to pipette to run an experiment. Today I do not have to pipette, I run a command or pipeline in a computer terminal connecting remotely to a cluster of a few thousand nodes. Sometimes it might be quicker to run a PCR than running my workflow script.

I consider a privilege being “drown” in data. Why? Because this is the future. More data brings more hypotheses and more hypotheses bring more knowledge. One either learns to surf the waves or a tsunami ends up catching one soon enough.

How does it feel from the inside? It feels exciting, overwhelmingly exhilarating! It feels like wanting to surf in a sea of data yet happy to be able to barely keep afloat: this is the inevitable fate of those genome bioinformaticians dealing with Next Generation Sequencing data.

What next in my todo list?

Cloud computing. I am counting the days when my experiments will be run in the cloud, not the cluster.

By Sam Johnston (CC BY-SA 3.0 license)

I look forward to welcoming you to the data feast. Will you join?

Chromosome 1. Comparison of Corpas mum and dad 23andMe genotypes using SNPedia's visualization tool.

It seems that Ben has taken this idea further and designed a blanket that incorporates chromosomal patterns for a complete 23andMe genotype. I quote here the description for this blanket from the Ben’s blog:

First Gift is a precious blanket which compares the digital DNA data of a child with their parents. If the child’s genes are edited, these changes will mask the parent’s DNA with synthesized DNA. The blanket itself represents a sacred and fragile heirloom, where tampering with it could potentially lead to frayed edges and uncertain outcomes. This first genetic gift will be with the child for life, and will also be inherited by future generations.

Although technically speaking this visualization shows comparisons between any two individuals, and not between the two parents and child as it is mentioned in the blog, I am still amazed at the craftiness and ingenuity of this idea. And since it uses data from the Corpas family dataset, I hereby report it in Manuel Corpas’ Blog. Here is another unpredicted and surprising effect from publishing our family genomes on the Internet.

To finish this blog entry, I borrow from Ben a complete profile view of my sister’s genotype patterned in his ‘First Gift’ blanket. According to him, the weaving of the blanket was done at the Tilburg Textile Museum in Amsterdam.

Blanket showing genotype pattern comparisons in my sister's 23andMe genotype. Each shape in theory corresponds to a chromosome comparison, although I still need to understand which chromosome represents each of the 25 shapes above (Humans have 22 autosomes + 1 sexual pair + mitochondrial).

In my new role at TGAC I will be leading the group of computational biologists working on plant and animal genome analyses within the Genome Analysis Team. Our aims will include the organization of the analysis of sequence generated at TGAC, engaging with internal and external collaborators, nationally and internationally. A lot of the work will focus on (but will not solely be restricted to) the analysis of RNA-seq, ChIP-seq and Bisulphite sequencing data for the purposes of understanding how genes are regulated.

Coming Opportunities

I will be expecting to have openings in my group in the near future for student projects (Masters and PhD) as well as research associates and technicians. Meanwhile, if you are interested in joining or simply discussing ideas or potential projects in the the broad areas of transcriptomics, epigenomics and gene regulation, you are always welcome to drop me a line.

In addition, I release under a CC0 license the following sequence-derived data from OGT’s services: a) the aligned and processed BAM file, b) the BAM file index and c) the compressed VCF file. The BAM file (.bam) is the binary version of a tab-delimited text file that contains sequence alignment data. The BAM file index (.bai) provides fast random access to the BAM file. The compressed VCF file (.vcf.gz) describes variant calls in text format. These format types are industry standard and can be used in a variety of research contexts involving genome visualization and analysis.

Looking at the summary metrics in OGT’s report, my personal exome produces:

• 30,702 variations to the reference genome (GRCh37)
• 5,565 non synonymous coding variations with consequences
• A minimum of 61.42% of the on-target regions, covered with a depth of at least 20x (remember that this data was sequenced by the BGI).
• A total of 2.54 Gigabases of sequence data read and aligned at high quality.

Figure 1 is a screenshot of the OGT report showing the summary of all variants identified, including those in dbSNP release 132.

Figure 2 summarizes all novel variants identified by OGT, filtering those in dbSNP release 132.

Download of BAM and VCF Files

You are allowed to use my personal exome’s BAM and VCF files under a completely free license CC0. You can add this data to any database or resource with no need for attribution. Any usage or finding derived from this data communicated back to me will be shared (if considered noteworthy) through this blog or publication, with due attribution or request for coathorship in papers.

• Aligned BAM file download
• Index BAM file download
• VCF file download

In this interview I talk about my experiences in analyzing my genome and the genomes of my family. Here I also introduce the motivation and future plans for the analysis of my personal exome (i.e. all coding regions in my genome).

Some further information regarding my family experience of personal genomics can be found in a recent publication.

What is an exome?

An exome is the ~1% of my genome that encodes for proteins.

Why do I release my personal exome?

When my family and myself made our genotypes available through the Internet, we immediately received results from researchers around the world who took our data for analysis and came back with interesting results. As a result of this, we have been able to learn much about ourselves. I have reported this in a previous entry on this blog entitled “Benefits for Publishing Family Genomes on the Internet“. I now follow the same principle: if I make my exome available for people to analyse it, I can expect that some researchers may come back with interesting results.

What data do I actually release?

I release my 4 FastQ files that were given to me by my sequencing provider. This is the same kind of information that 23andMe gives in their current exome analysis offer. This information basically consists of raw reads that need to be aligned to a reference assembly. Once aligned, interesting variation data can be inferred.

What do I ask in return?

Nothing. I do appeal though to the good will of potential users to report back to me anything interesting they might find.

How big are the files?

They are huge. On average they are about 0.6 Gb per file and I have 4 of these. That means that it can take several hours for each file to be downloaded. Be patient!

Where can I get them?

Here:

1. File 1
2. File 2
3. File 3
4. File 4

How did I get my personal exome sequenced?

Completely independently. If you want to know the story on how I did it, please refer to my blog entries “Getting My Genome Sequencing Done” Part I and Part II. As it is implied there, I managed to get my personal genome sequenced by knocking on quite a few doors and then finding someone who would sponsor me to do so. In fact, part of this exercise’s aim was to prove that it is possible now a days for ordinary citizens to get their genomes sequenced if they so wish.

In order to have the most up-to-date version for the Human Genome set of genes and features sometimes it is necessary to convert from one assembly to another. In the past I have written a tutorial on how to remap from NCBI36 to GRCh37 human assemblies using liftOver. In this tutorial I present a simple step-by-step guide for feature remapping using NCBI’s remapping tool.

Important:

Please make sure you know in advance the assembly to which your aberration data is currently mapped to. If by mistake you remap an aberration already in GRCh37 to GRCh37 you will get new coordinates for the region mapped to the wrong coordinates.

The NCBI provides a web facility to convert coordinates from one assembly into another. To convert coordinates using their genome remapping service do the following:

1. Make sure that your data is in BED format,  e.g. “chr3            100000 999990 myId0000123” -> CNV aberration in NCBI36/hg18

• Please note that each field is separated by a tab and each line by a character return. Please follow this strictly or the remapping tool may throw an error.
• Add as many lines as aberrations you would like to remap

1. Go to the NCBI Remap page:

• http://www.ncbi.nlm.nih.gov/genome/tools/remap/

1. Select “Organism for source data” Homo Sapiens, “Source Assembly” NCBI36 (hg18) and “Target Assembly” GRCh37 (hg19)
2. Please leave all “Remapping Options” (Minimum ratio of bases that must remap, etc) with default values
3. Select for “Input format” BED, “Output format” Same as input
4. Paste your aberration in the input box where it says “Paste data here” and hit submit at the bottom of the page
5. Wait until results are returned
6. To retrieve results download “Mapping Report”, which is in excel format or alternatively Mapping report Sample in the results page

Please note that your aberration may remap to more than one location. I recommend that you manually check the coordinates and select the most appropriate of the doubly remapped aberration in the new assembly. Please also note that your aberration may not remap because the region is partially or entirely deleted in the new assembly or split in GRCh37. In this case I recommend that you use another start or end point position, maybe use the start/end of alternative probes until you find a region where it maps.

Another possibility could be to look at the genes for the region in the old assembly and select a region in GRCh37 that includes the same genes as in NCBI36. Each of these solutions requires careful deliberation and may not be applicable to your particular case (e.g. genes in different chromosomes would not allow remapping based on genes).