MetService Vs NIWA: the Model is Broken

This issue has been bubbling away in the back of my brain for a few years now, however, it was when I listened to the podcast ‘The Detail’ from RNZ recently that I really began to think about it and do a bit of research.

The New Zealand Meteorological Service has existed in various forms since 1861. Its current guise, Meteorological Service of New Zealand (MetService) was established in 1992 as a state-owned enterprise (SOE). At the same time the National Institute of Water and Atmospheric Research (NIWA) was formed as a Crown Research Institute (CRI). The rationale was that MetService would focus on the day-to-day weather forecasting and NIWA would focus on climate research. The fundamental problem here is these are not different things. They are essentially the long-term and short-term, strategic and tactical, whatever you want to call it, aspects of the same problem. And to make matters worse, as an SOE and CRI respectively, both organisations have a responsibility to recoup some of their costs.

MetService does this through its various forecasting products. Interestingly, the marine forecast is free at the point of use, whereas commercial aviation operators must pay for the weather information they are legally required to obtain, but this is an issue for another post.

I assume NIWA moved into providing weather forecasts in order to chase a lucrative revenue stream, although NIWA has stated publicly that weather forecasting always been part of their mandate. This is where, for me at least, the waters get a little muddy. According to a January 2022 Official Information Act request by the NZ Herald:

The establishment of MetService and NIWA in 1992 separated the operational and research components of New Zealand’s weather function, performed by the Ministry of Transport (MoT) at the time.

With respect to the functions of MetService and NIWA it states the following.

MetService:

The rationale to create MetService as a commercial entity was to leverage its expertise in weather forecasting to support its operations, reduce the cost of weather services to taxpayers, and pay dividends to the Crown.

MetService helps people stay safe and make informed decisions, based on the weather. MetService supports the Minister of Transport in meeting statutory and regulatory requirements associated with the Meteorological Services Act 1990 and New Zealand’s obligations to the United Nations with respect to meteorology. It does this through a contract with the Minister of Transport. The contract includes providing severe weather forecasts and warnings for New Zealand land and marine areas, supporting search and rescue, and representing New Zealand at the World Meteorological Organisation (WMO).

NIWA:

NIWA was created as part of a government initiative to restructure the science sector, when the CRIs were established to undertake scientific research for the benefit of New Zealand. This framework was set up to incentivise CRIs to develop commercial revenue streams and reduce the reliance on core Crown funding and contestable research funding.

NIWA’s purpose is to enhance the economic value and sustainable management of New Zealand’s aquatic resources and environments, to provide understanding of climate and the atmosphere and increase resilience to weather and climate hazards to improve safety and wellbeing of New Zealanders.

On first reading this, my take away is MetService provides the day to day forecasting and NIWA looks after climate and atmospheric research. This research will help MetService with better forecasts and MetService’s observation data can be used by NIWA in their research - a symbiotic relationship.

Then if you look at the second paragraph regarding NIWA, I guess ‘increase resilience to weather and climate hazards to improve safety and wellbeing of New Zealanders’ could be construed to mean provide a weather forecast. Couple that with the requirement for a CRI to provide a financial return and the potential income stream from providing forecasting and you see how NIWA moved into this space.

What should have happened was someone in government telling these two organisations to stay in their lane and if there was any confusion as to what exactly these were tidying this up. Instead, in 2001 ministers encouraged the two organisations to work more closely together. Then in 2006 an independent panel recommended merging the two organisations, but this did not happen. In 2009 a memorandum of understanding was signed by the two organisations to work more closely together. As far as I understand, this has since expired and was not renewed. To make matters worse, NIWA then set up NIWA Weather in 2013.

Now if all of this was not enough of a shambles, in 2015 NIWA entered into a partnership with the UK MetOffice to help with the computing resources required to run the Unified Model which the UK MetOffice developed. In order to help do this, NIWA purchased three new supercomputers which are more powerful than anything MetService has. All the while, MetService is still the ‘official’ weather forecaster for the country and represents New Zealand at the World Meteorological Organisation.

In 2018 MBIE (the Ministry for Business Innovation and Employment) and Treasury looked into whether New Zealanders could get open access to the data collected by both organisations and it was concluded no, due to the commercial drivers of these organisations. Then in 2019 the Commerce Commission looked into the pricing of each organisation but decided to take no action.

So to summarise, MetService is the ‘official’ forecaster, but NIWA partners with the UK MetOffice and has more resources. Both compete in the forecasting market and both make it incredibly hard for the public to freely obtain data from them. Oh, and both are one hundred percent owned by the Government - that is you and I, the taxpayer. We help fund these organisations. They compete for commercial revenue and we cannot freely obtain a lot of that data. What is going on here? And why on earth is it still going on? From the inquiries, investigations and cajoling by ministers to work together over decades, it’s quite clear there is a problem.

Luckily, there is a little hope. The Government recently announced there will be another review into the rivalry between MetService and NIWA, commencing in September 2023 and concluding in February 2024. So what could come out of this? Who knows, there isn’t really a good track record here, is there? What should happen then?

It’s quite clear having two organisations with such an overlap does not work, so number one they should be recombined into one taxpayer funded entity, as they were previously. Personally, I am in favour of the smallest government possible and a free market, but some things do fall into a natural monopoly, especially in a small country such as New Zealand. The collection of meteorological data on a national scale is perhaps a good example of this. The data should then be free to use by all. Government can still have an ‘official’ voice for weather warnings and produce a commercial weather product if they wish. Along with this, anyone with the knowledge and inclination can use the public data to produce their own forecast, for profit if they want, or simply as a hobby. The market will quickly decide which of these is the best and produces the best value for money to the consumer. This is similar to the model in the United States with NOAA (National Oceanic and Atmospheric Administration) and the National Weather Service.

So the bottom line, why should you care about all this? Because the way it is right now is hopelessly inefficient and confusing - the model is broken.

Incidentally, if you are interested in finding out more about weather forecasting and the computer models used, listen to episode 326 of the excellent podcast omega tau: Weather Forecasting at the ECMWF.