Showing posts with label Data management. Show all posts
Showing posts with label Data management. Show all posts

Wednesday, 31 August 2016

Data management - Post 3 - File organisation and naming protocols

By Maxine Whitfield

A few weeks earlier: "I'll just 'save to desktop' now since I'm in a rush, but I'll sort everything really, really soon..."

Folders within folders, within other folders... 

I can’t emphasise enough how important it is to have your data and associated documents sorted and named logically. Structure your directory of folders hierarchically, but don’t rely on the structure of the directory to give you critical information about a file. If the file is copied elsewhere, you’ll lose the structure. So, for example, it makes more sense to have the following folder system...

WinterSiteB  >>  WinterSiteB_SociableWeaver  >>  WinterSiteB_SociableWeaver_Tb

... rather than

WinterSiteB  >>  SociableWeaver  >>  Tb


File naming

Your file names should be relevant and meaningful, and provide the “what, when, where and how” of the the file. Be consistent, and rather tha using spaces, use underscores or capital letters to make computer searches for files more successful. If you are using dates, use the format YYYY_MM_DD, or YYYYMMDD. Names of authors/data collectors should be full initials, or surname first, followed by initials.

Versions of the same file

Your dissertation/thesis is going to go through many rounds of comments, first from your supervisor\s and then from your external examiners towards the end of your degree. If you try to do everything from one document, you will lose a vast amount of information through the editing process. You have to save each version separately, so that you have the means to go “back in time”, and follow the evolution of your study as it becomes more refined. Save your first version with the suffix “before comments” and the date you sent it to your supervisor for review e.g. Ch1_before comments_20140426. When your supervisor sends back a chapter full of comments and edits, save that version exactly the way it is, also with a date and your supervisor’s initials at the end of the file name, e.g. Ch1_AEM_20140513. You can then start working in the document with your supervisor’s comments, and save this version with your initials following your supervisor’s e.g. Ch1_AEM_MCW_20140530. On a side note, don’t be tempted to “accept all changes” as Microsoft Word allows you to do – you won’t learn anything about scientific writing that way.  


Papers from journals

There are a few options (see here) when it comes to reference managers. I initially chose Mendeley simply because it is free, but I am really enjoying it as a “one stop shop” for all my literature needs. You can import a PDF directly from the internet using the web plugin, and then read, highlight and annotate the paper itself in the built-in Mendeley PDF viwer. While initially the PDF remains wherever you saved it and Mendeley saves a shortcut, the Mendeley file organiser tool (under preferences) can actually locate all the papers (scattered around your downloads folder and elsewhere) and reorganise them into a central folder wherever you want. It will even rename the PDF’s however you prefer, with author_year_keywords_journal being my favoured order. This way, if you clear out your downloads folder or delete the PDF’s that are clogging up your desktop screen, you don’t suddenly lose your meticulously highlighted papers. You only have to do this once – after the initial organisation Mendeley will automatically save a perfectly named copy of any file you import into this very aesthetically pleasing folder. Try to make a habit of downloading all your papers via your reference manager – it might seem like a schlep to wait the extra minute while Mendeley processes the PDF and to check that the author, year etc. are all correct, but you’ll be much happier later when you can insert the citations easily into your dissertation, without having to sift through hundreds of papers in one batch.

Data management - Post 2 - Storage and backing up

By Maxine Smit

This was me after I lost two pages of world-changing writing mid-way through my honours year. (I tried to close Microsoft Word, it politely asked me if I wanted to save my changes, and I clicked no. I'm smart that way.) 


Every postgrad I know has at some point in their scientific career lost an unhealthily large portion of work due to losing a flash stick or forgetting to press “save” before their archaic laptop crashed.

Adjust the settings on Word/Excel/whatever program you are working in, so that it automatically saves every five minutes or so – don’t rely on your manic coffee-saturated brain to do something as menial as pressing save, when it is frantically trying to encapsulate a million escapee thoughts into words. You will forget to press the button, and your laptop will at some point freeze or crash.

Cloud storage is a life-saver. Download Dropbox, save all your data and documents there, and work directly from Dropbox wherever you are, from any computer. It will automatically sync and save (as long as you are connected to the internet) every time you save your document, so you don’t ever have to worry about backing up again. If you have a cap on your data bundle and need to use it sparingly, just use the “pause syncing” option for a while, and unpause it once or twice a day to back up your files (Dropbox replaces and saves the whole file each time you press save, so it can be a very data-hungry process if you’re saving a large file every five minutes). 

Don’t try to have a version of a file on your personal hard drive as well as on Dropbox – you will end up making changes to one and forgetting to update the other, leading to conflicting versions. Dropbox allows you to share documents with other Dropbox users, so make sure that you have protocols in place for shared folders – if someone makes a change in one of your files, there should be a document wherein they record what they did and when they did it. Dropbox allows you 2 GB of storage for free, after which you have to pay around R1500 a year for a full TB. Prioritise your data files, stats and writing, and rather back up all your papers from scientific journals on an external hard drive or flash – you can always download papers again. In addition, most referencing programs have a cloud storage option, so that your PDFs and citations can all be restored should something happen to your computer.  

AJP: Dropbox also offers an additional safety net for your files - it keeps every version of a file for 30 days. So, if Mendeley or Endnote corrupts your entire Word document, you can go onto Dropbox Online and revert the file back to a previous version. Saved my bacon (or rather sanity) more than once! 

Data management - Post 1 - Intro

By Maxine Whitfield

I still get emails from my MSc supervisor, who I have nothing but the utmost respect for, but whose name in my inbox is usually accompanied by a heart-quickening, “fight or flight” response, and feelings of complete terror. 

Until recently, we were preparing manuscripts for publication (thankfully they're all published now), and he usually had a question about how I analysed my data. Despite the fact that I wrapped up the final edits on my dissertation a mere year and a half ago, it often took me a full day of frantic searching in documents scattered to the furthest corners of my laptop, under filenames such as “dissertation final”; “dissertation REALLY final” and “dissertation December 2015 kill me now”, before I could answer his simple question. I spent hours poring through spreadsheets of data. Did I end up including those outliers in the mixed model? I need to hunt down the R file from that analysis… What the heck would I have named it? 



My naming system for folders and files left much to be desired. I shared this Dropbox folder with my supervisor. It must have terrified him.


I had some kind of hunch about the relationship between expected and observed metabolic rates, and spent a crazy afternoon excitedly exploring my data. Yikes. A couple of years down the line and I honestly have no clue what is going on in this spreadsheet.

The point I am trying to make here, is that as a naïve, happy-go-lucky honours graduate entering my Master's degree, I had absolutely no idea just how much data collection, exploration, analysis and writing I would do in the next couple of years, and I had even less of an idea of how to go about managing it all. The point of this blog series, therefore, is to arm the bright-eyed, bushy-tailed future postgrads who read it with the tools to manage their data effectively, so that when publishing their incredible results, they can coolly respond to any co-author’s questions with negligible blood-shed. 

DataONE, an online repository for worldwide ecological and environmental data, has a fantastic set of tutorials on the various aspects of data management. For each of these blog posts, I'll put a link up to their page of tutorials and exercises, and suggest which ones correspond to the topics I'm discussing in the post. See Lesson 01 here.