Visualising the weather

by Martin Jucker (See Martin's Weather (a)live page here).
The weather is all around us. It’s the first subject of (small) talk around the world.

Weather is also part of my job, although we don’t like to call it weather, because that’s what the forecasters do. Those beautiful people on TV.

We instead are (beautiful) scientists, so we tend to talk about the various phenomena rather than ‘weather’. There’s convective activity, storms, cyclones, breezes, updrafts, downdrafts, low cloud, high cloud, thin cloud, thick cloud, precipitation, reflectivity, and so on. But in the end, if we are honest, it’s just weather.

The internet is another thing that is all around us. Only a very small number of things are not connected to it in some way, and the things we see or read on the internet are probably the number two subject, after the weather, that we talk about. The internet works with computers and I spend all of my work time in front of one, being connected to thousands of processors thousands of kilometres away, and using it to (small) talk with family overseas.

I also like visualising stuff. My stuff mostly. I use it to try and explain to my family and friends what I am doing. There is no use talking about Eliassen-Palm flux, the CFL criterion, SciPy, or up- and downdrafts. Keep it simple, make it visual, make it beautiful.

That's when I asked the question, why not combine all of the above? Why not put the weather on the web? Why not make it beautiful?

If I do this, it won’t be about science. It’ll be about creating something nice and interactive. Something that connects my work, my interests, and, well, the world. So, how do we do this?

 

How do I do this?

And there’s the first pitfall: You cannot start a visualisation with the question “how do I do this?”.

Let me repeat: You cannot.

The first, and most important question is “what do I want to show?”. Everything else follows from there. Trust me.

 

What do I want to show?

It’s the key question to be answered. Most people that approach me about visualisations don’t even recognise they don’t know the answer to that question (other than “something cool” of course).

But if you don’t know exactly what you want, you won’t know which data you need or which tool to use. So, first thing: Look at what’s out there!

 

What have others done?

There are some amazing examples out there, which is good and bad news. There are good ideas to drive inspiration, but also the moment when you realise you are not the first one to do this (surprise, surprise).

In my view, the gold standard for casual (i.e. non-commercial) weather visualisation in two dimensions is earth.nullschool.net. It’s mesmerising and it’s beautiful. I could spend entire days staring at this.

How does this guy do this?

Well, the author Cameron Beccario lists all his sources, and has put his stuff on github. Yay, open source! Looking at his information, it becomes clear that the data comes from the Global Forecasting System (GFS). That’s good to know but beyond that... quite simply, this guy is a wizard and I have no idea how he does what he does, even looking at his code. Fortunately, that doesn’t matter that much, as I don’t want to copy something that exists anyway.

So, now that I have an example that I like. What does that tell me about what I want to show?

 

What would I like to see?

First thing: What was I looking for when searching the web in the previous step?

Those will be important key words defining my interests. Well, ‘weather’ was one of course. Another one was ‘3D’, but nothing came up from that search.

Wait, nothing came up!

I know how to do 3D, and I was looking for 3D stuff on the web. Bingo. I’ll do 3D weather.

What does ‘weather’ mean to me? It means clouds, rain, temperature and winds. All the other stuff is detail, really. The one obvious 3D thing here are the clouds, so they will be my centrepiece. There is no point looking at the weather if I don’t know if it’s raining or not. So precipitation must be there too.

As to temperature? I think I can do without that - it’ll be chilly in winter and warm in summer. There is no need for a weather chart to know that.

Winds. I want winds - a sunny day in summer with winds coming down from polar regions is a lot cooler than a sunny day in summer with winds coming in from the desert. Also, the one component making the example I found previously so mesmerising is wind. So, I want winds.

Finally, I need to define the domain. I have family and friends all over Europe, the US, and of course Australia. I lived on three different continents in both hemispheres. So there’s no question, I want global coverage. Great, we can now define what we want to see: - 3D renderings, clouds, precipitation, wind and global coverage

 

Where do I get the data from?

I am on a low budget. Actually, the budget is exactly zero, so everything needs to be freely available, including the data. That rules out any meteorological institution in the world, as that data is their business.

From my work, I know that reanalysis products are freely available (I used them often myself for my research). But I want to have clouds and I want global coverage, so, resolution is an issue.

Ultimately, I want to upload this onto a website at regular time intervals, so automatic access to the data is a prerequisite.

Looking at the different reanalysis products available on the web, it becomes clear why earth.nullschool.net is using GFS/NCEP data. These guys are doing everything to make their data available, including command line scripts to automatically download the relevant data. Pretty cool.

In addition, it’s the dataset with the highest resolution (0.25 degrees), and it has all the fields I need. Bingo. And, there’s a big bonus: They even provide forecasts. If that isn’t amazing. Thank you, NCEP!

 

How do I visualise the data?

The software

I’ve got the data I need, I know what I want to show and it so happens that I also have the perfect software for visualising data in 3D. I published it myself. What a coincidence.

I’ve been using ParaView together with pv_atmos for a while now, and already have some examples on my website. So I know how to convert data into appealing 3D renderings.

Ok, so I can get the data, I can visualise it. We’re good then, right? Actually, I think we are at more-or-less at day two of the process. It takes months to get to the final product. So no, we’re not there yet. Anyone can get to this point. It’s what comes now that makes the difference.

 

The devil lies in the details

I now have the data, the software, and the list of variables to plot. Unfortunately, this is where real work begins. A lot of decisions will have to be taken.

The most important ones are:

  • Surface geometry: Show as a sphere, box, or some more sophisticated map projection? The Earth is a sphere, but the problem with spheres is that you can’t show the entire surface at once on a 2D website. On the other hand, sphere-to-plane projections show the whole globe at once, but there is no way around distortions and/or discontinuities (Alaska is really close to Siberia!). 
  • Colour schemes: Death to rainbow! But what else? I want to show three different variables, plus altitude, sea and topography. So, I need a variety of distinguishable colour schemes, and they should all look nice as well.
  • Aspect ratio: I want to show the altitude as a third dimension, but I can choose the scaling of that dimension with respect to the horizontal dimensions. Real scaling as on Earth is not practical, as it would simply look 2D (compare Earth’s radius of over 6’000km to the troposphere depth of 10km). 
  • Rendering method: There is volume rendering, isosurfaces, arrows, glyphs, streamlines, etc. I will have to choose one for each field, and make it look good when all fields are shown together. 
  • Rendering power: Again, the budget is zero. So a simple desktop computer will have to suffice. If I want this to become automatised, and run continuously, everything needs to be able to run within one night. 
  • Bandwidth: If this should end up on a website, I can’t produce huge files. 
  • Robustness: The whole output should eventually go onto a website, and be updated continuously, without human interference. Thus, everything will have to be robust and segmentation faults and stuff must be avoided at all cost.

What can I say? With great power comes great responsibility. These decisions really are 80% trial and error, and personal taste. 

Here is what I went for and why:

  • Surface geometry: I really like the globe. Earth’s a sphere, so let’s make it a sphere. So for the global weather visualisation, I produce the full sphere, but created an animation which rotates the globe. This way all of it is visible, if not at the same time. Unfortunately, that means that the tropics are most easily visible, and the midlatitudes with their weather systems are a little more difficult to observe. But that’s life, sorry.  In order to make up for that, I add a second section where I concentrate on a few regions of the globe. I can even keep the spherical geometry, as it is almost flat for small sections of the globe anyway. 
  • Colour schemes: Clouds have to be white. There’s no way around this. So starting from there, I’ve gone back and forth, back and forth, and back again. Finally I settled on something I like, I hope you do too.
  • Rendering: I would have loved to use volume rendering for the clouds, but the drain on my simple desktop computer is just too large. So isosurfaces for clouds it is, arrows for winds, and surface colour for precipitation.
  • Bandwidth: I had to split up Asia, as it simply is too big. I also had to take out the Himalayas, as the webgl files became way too large due to the topography. 
  • Robustness: Everything is written in either bash or python. That means maximum portability, and also stability in terms of software updates.

 

How do I show the data online?

Now I have data, visualisation, and automatisation, so I need a place to make it visible. Fortunately, I’ve already spent quite some effort into developing my own website at martinjucker.com. And there is already some webgl stuff, so I know I can put the interactive 3D elements onto my website.

But, I want it to look nice and be up to the standard of website technology. That means lots of coding in javascript. And again, it will have to be robust code.

A few elements that would need to be included are: 

  • Keep track of time. And if possible (yes), keep track of the user’s time zone and show the information accordingly.
  • Enable changing the time. I have hourly data, so I want the user to be able to go backward and forward in time interactively, and the correct files will have to be loaded.
  • Make it work on mobile devices as well as desktops.
  • Make it work on devices which don’t have webgl. Those are slowly dying out, but they still exist in numbers.
  • Let the user be able to switch randomly between regions, time, 2D and 3D content.
  • Make it shareable on social media.

All of this is possible of course. And anyone reading this with good knowledge of javascript will probably think it’s child’s play. I do think I know a bit about javascript. Now. Not so much when I started this project.

Fortunately, coding languages are languages, and if you learned one, learning one more is not that difficult. Setting up the website is where most of the time went. Working on the visualisation itself ended up to be only a very small part of the total work that needed to be done.

 

Conclusion

Was it worth it? Well, have a look yourself at martinjucker.com/weather. I like it, and I like having been able to do it.

There is a big component of “because it’s there”: I had the idea, and knew I had the means. So I tried. And succeeded.

Good feeling.

But just like the research I do every day, it’s not a finished product. I am counting on users to tell me what they liked, but most importantly what they don’t like, so that I can improve the experience, bit by bit.

I’m also hoping to inspire some of those geeks out there who know how to do this stuff a million times better than myself. Go and do something better, I’d be the first to congratulate you on it.

As a guideline, here the main points I learned from the experience:

  1. If you want to visualise “something cool”, make sure you define exactly what that means. This has to be the first step.
  2. Take your time. The first idea is very often not the best. But the good idea might not be born without the (bad) ideas and trials before that.
  3. Be realistic. No point in planning for that huge idea if the resources don’t allow it.
  4. Have fun along the way!
UNSW logo ANU logo Monash logo UMelb logo UTAS logo