Global warming, python and statistics part 3

Last year and in the previous two parts, we collected and analyzed the data using various tools such as sliding averages, linear regression and sine fitting. In this last part, we will predict what the climate may look like in the future (prediction). 

So far, we have established that the current climate follows a linear trend (long term trend) superposed to a seasonal cyclic variation. The highs and lows are distributed around these in an almost normal fashion. I want to insist that the weather is not a random variable: it follows the very strict rules of physics and weather specialists use complex models based on equations from thermodynamics and fluid mechanics. However, in the long run, the climate seems to behave like a random variable whose mean varies over time.

In order to not introduce any bias in the model, the observations until the end of 2014 will be used.

Let's try to predict the weather. If you recall, a normally distributed random variable will be between μ-2σ and μ+2σ about 90% of the time. Our model is then that the temperature will be, with 90% certainty, between the trend + seasonal cycle +/- 2σ. For the first month or so of 2015, this looks like this.

The solid red lines represent the two limits (90% confidence) for the high temperatures, the two blue lines the same for the low temperatures. The dotted lines represent the average. The dots are the observed temperatures from the GHCND data set. 

On average, the year has been rather on the low side: most of the measures for the high temperatures are near or below the average, with a few measures clearly below the low line. For the low temperatures, the situation is more balanced with about the same number of points above and below the average. It is to be noted that two values are clearly below the lower confidence bound. 

As of February the 1st, we have

Temperature	Maximum	Minimum
High	13.3C	-6.0C
Low	5.0C	-13.2C

The maximum between the high and low temperatures for the same day is 15.4C, the minimum 1.6C.

Conclusion

Python, numpy, scipy and matplotlib are probably the best tools to start some data exploration: python is reasonably performant, numpy and scipy provide the necessary statistical and mathematical tools, and matplotlib is a good graphic library. 

The code for generating the various graphes can be found here.

"The Imitation Game"

It is estimated by historians that British mathematician Alan Turing helped, through his contributions at Bletchley Park, shorten World War II by at least two years. This movie his about that period of his life, though the beginning is set at the time of his arrest for gross indecency", the charge then in vigor in the UK for being homosexual.

The movie does not pretend to be an historical account nor a documentary on Turing's life, and as such viewers familiar with the "real" Turing will found some elements to be inaccurate. It is however a good story that proves I thoroughly enjoyed.

And the violent content was very limited, for a change.

"Flatland"

Here is a review by Jean: http://www.goodreads.com/review/show/447710813

"Data Mining Tools for Malware Detection"

Here is a review by Jean: http://www.goodreads.com/review/show/1054850667

An Art Movement Where Art and Science Collide

Science Friday is one of my favorite podcasts. Filled with tons of interesting interviews, comments and reports, this is my weekend "radio station."

In last Friday's, one of the topics was "An Art Movement Where Art and Science Collide". The part that totally baffled me was the two tunes: one is from Johann Sebastian Bach, the other was generated by an algorithm written by David Cope (here are some other references: 1, 2 and 3). While I had no difficulties identifying the "real" Bach (I know that piece for having played it), I was really impressed by the likeliness to the real works composed by JS Bach.

I do remember, in the olden days of the 3DO, a program by Sid Meyer called "CPU Bach" that composed works "in the style of" Johann Sebastian Bach. This one goes a step further. I wonder if we should create a "Music Turing Test"

There is another piece on David Cope's page on the USCS website. Though I think that some parts would not have been written that way by Johann Sebastian Bach, the end result is truly amazing.

PoS malware found targeting mass transit systems

The security company InterCrawler has found a new malware strain that targets the mass transit systems. 

In the report, a sentence had both my eyebrows raise and my jaw drops at the same time:

"During ongoing POS investigations it was determined that some operators of Point-of-Sale terminals have violated their own internal security policies and have used their terminal for gaming and WEB-surfing, checking e-mail from it, sending messages, and viewing social networks. These cases have a common denominator of weak passwords and logins, many of which were found in large 3rd party credential exposures."

This is almost 2015 and still people operating Point of Sale terminals are still incapable of realizing that their actions can result in huge dramas. 

To the casual reader, this seems bad. To the security-minded, this is even worse: it means that these machines had, at the time of the breach, access to the Internet. This is in direct violation of the PCI standard. 

Last Friday was Black Friday in the US, I am curious to discover how many retailers were compromised and how much money cybercriminals have amassed.

AstroViz - Colliding Galaxies

If you don’t know the American Museum of Natural History, now is a good tile to be acquainted. Among the various research groups, the astrophysics, led by Dr. Tyson, is very active and has a nice visualization section.

Recently, it has been announced that our galaxy and the Andromeda galaxy are on a collision course. While this is the correct technical term, the result will mostly be that the 2 galaxies will go through each other a few times before merging and creating an even bigger galaxy. That is in about 3 billion years.

The AMNH has created a nice visual on this: Colliding Galaxies. Enjoy!