Data analysis is a big subject, I’m therefore going to approach it through a number of posts. Here we’re going to talk about what I think you probably need for a good package, and some of the down sides of what are the most commonly used in the non-professional scientist world for data analysis. Next time I’ll move onto what I think if (for most applications) the best tools for the job and then in a third post I’ll give some alternatives that are just as good and provide a few reasons why you might want to use some other or varied tools.
So, what do I think you need and why? Well to begin with you need to ability to deal with a lot of data, anything that can’t handle a few thousand numbers is out straight away. You need to be able automate things. For the most part if you need to run some analysis to one dataset you will need to run it to many datasets, this is pretty much the nature of beast. If you are not crazy you will want some way of running the same process many times. Automation, you therefore need some type of scripting language.
Along similar lines to the automation if you are producing some kind of report, which everyone will have to from time to time, you will want all of your figures to look the same. So again you want to be able to write a script to automate your figure design. Not only will you want them to look the same but look nice. There are some other features of figures which may not be immediately apparent to those approaching this kind of thing for the first time. Journals often require figures to look specific ways, you need something that is versatile enough to allow this. But you have an added difficulty that if you are rejected from one journal, which is not uncommon, then you need some easy method to redraw your figures using the design parameters defined by the next journal you are going to submit to, again scripting and automation are your friends here. The quality of your figure is the final thing to consider, it is probably most appropriate to use some scalable vector image format for the best print quality of your figures.
So what packages do most non-scientists use for this kind of thing? Excel… or one of the similar products available in openoffice.org or libre office. These are good applications for what they are designed for, spreadsheets. They are not normally the most suitable for the kind of analysis described above. Automation is difficult, scripting is irritating, and it is normally very difficult to apply these tools to many datasets one after the other. Figures are also often a big problem, they tend to look rather unprofessional. There are other problems with these types of applications, they are not good at high level maths. Complex numbers are not handled particularly well or intuitively along with matrices and vectors. You may think that I’m being overly harsh here but DO NOT USE A SPREADSHEET FOR DATA ANALYSIS! Don’t believe me? Use one of the many programs designed for this kind of thing and see how easy it is compared. I’ll run through a few of these in my next post…
As a follow up to my last post I’m going to talk about another modelling package…
Having said what I said last time (my last post) about finite element method packages I have come across a very nice tool. It’s finite difference time domain as opposed to finite element method, but it works in a kinda similar way but also provides some temporal information. This is very useful if you are working with pulsed systems. This package is called meep (http://ab-initio.mit.edu/wiki/index.php/Meep).
During my time doing a PhD, and some time before, I have used, come across and talked about a great many software packages. These range from simple little tools written for linux to scarily complicated 3D modelling applications running on windows and everything in between and roundabouts…
I’m going to highlight a few of them and talk a bit more generally about scientific software in a short series of post. There will be both some recommendations and some general commentary.
There has been quite a large amount of attention about he curiosity rover in the last week or so and deservingly so. I want to talk briefly about the computational ability of the device and how that might compare to something a little cheaper that might lead to some interesting projects.
This isn’t really needed but there are two groups I’d like to publically congratulate, the team behind curiosity, all of them. From the guy who makes sure the bins are empty to the project director. Well done on getting your guy down I hope you get even part of the success of Spirit and Opportunity (I would wish you as much success but they were extraordinary).
The second group is all of the athletes and contestants at the Olympics. I’ve never been a fan really but I have enjoyed the last few days! Well done for getting to the games, it’s a great achievement even if you do not get a medal. I think a special congratulations must go to those members of Team GB who have done so well and made everyone in the UK proud. Not only have you done well but when interviewed you’re all so nice! Well done, and thank you.
Throughout my time doing my PhD I have discovered things do not work. I’ll expand on this a little more below, but for now lets just go with it. How do you cope? How do you get up in the morning? I’ll try to answer these questions.. I guess it really comes down to hope…
Sorry about the lack of many posts recently, hopefully the reasons for this will become clear over the next few blog posts.
I thought I’d restart my blogging with a brief discussion of some events recently. I’m sorry about the lack of science content here but a big chunk of the physics is commercially sensitive so I cannot talk about it.
–Edit, sorry this is a republish after I decided to add a section at the bottom–
I’ve recently written in the region of 75,000 words. That’s quite a few for a physicist. I’m pretty sure that, excluding my dissertation (which was around 10,000 words so still a long way off) the longest previous piece of written work I’ve completed in the last 10 years is 2000 words, maybe 5000 at a push.
I thought I’d share a few of the things I learnt as I wrote, the things I did, how I did them and to a certain extent why. I don’t expect this to be a particularly long post [edit: turns out it's longer than I expected] but hopefully interesting to those about to start writing some long piece of work. I’m not saying these tips will be good for everyone, but I found them helpful.
The following is a reworking of a previous post, for this I would apologise, but am pretty sure you mostly do not care.
Let me start by talking a bit about The Big Bang Theory (TBBT) tv show. I like the show for the most part, it seems to work well and I only normally feel slightly surprised when they do things like putting on safety goggles to use liquid nitrogen, safe yes, often carried out, no. The characters have attributes that I could easily point to many of my colleagues and say “they act like x”. Therefore I would say I am neither a fan boy nor a critic of the show, if it’s on I’ll watch it, but I tend not seek it out.
More generally now, there seems to be a perception with the non-scientists of the world that we are an elitist group who spend all of their time agreeing with each other. This is often visible to me in the form of online forum posts but recently TBBT also alarmed me with the views of one of its physicists. I bet you can’t guess which?
This is very much not the case. Science is a hot bed of arguments and discussion. I have recently seen a public display of this with a well known (in the UK) public figure and a blogger I hold in high regard. I’ll go into this in a little more depth below. First I will explain some of the problems I have with the character from TBBT.
Let me start by welcoming you all to my new blog.
For a while I’ve been a reader of SFN, I might participate, I might not. But a while ago I started a blog,
All of which has now been reproduced here. I’ve not had much time to add things to it, but hopefully that will change.
A few posts ago now I wrote one based loosely on a post I read on the forums, after doing so a member here (Klaynos) has been bugging me on and off about moving my blog to SFN, so here I am, posts and all.
Hope you all enjoy it somewhat.