IP Informer - a program to look for several IPs in several black lists

So, you have a list of IP addresses you would like to quickly check against several black lists such as Abuse.ch 's Zeus Tracker, Spy Eye Tracker, Palevo Tracker, Feodo Tracker or the Malware Domain List. Here is a small tool in Google's go language that does exactly that: it takes a file containing a list of IP addresses (one per line) and checks each of them into these lists. Given that the black lists are defined in a configuration file, it is very easy to add or remove specific lists.

I keep adding new features, so feel free to check the GitHub repository from time to time.

Happy Birthday Fortran!

For some of us, the name "Fortran" or "FORTRAN" evokes a computer language closely associated with massive super computers and complex mathematical models. For others, it is reminiscent of a war with C++ for the supremacy in the scientific computing world.

It has been continuously developed since its initial publication in 1957, and the latest revision came out in 2010, with another minor revision planned for 2015. Fortran is not dead, far from that, even if it has a though competition from other languages such as Haskell, Clojure or even Python.

Wikipedia has an extensive history of the language.

The first example of code in Fortran I will present is the determination of the fraction that generates a given pattern.

Let's take 0.1278, where the underlined part repeats ad infinitum. The fraction needed to obtain this value is 211/1650. For the rest of this post, I will call the part that repeats the repeated part and the part that does not repeat the prefix. The algorithm to find the fraction is well known, let's focus on the code.

It contains three parts: computing the non reduced fraction, computing the greatest common denominator (gcd) of the numerator and denominator and reducing the fraction to a numerator and a denominator that are relatively prime. Let's start with the gcd.

For this, I use Euclid's algorithm. The code in Fortran 95 to achieve this is

function gcd(a, b) result(c)
! Returns the GCD of a and b
 integer :: a,b,c,u,l,m
 if ( a > b ) then
   u = a
   l = b
 else
   u = b
   l = a
 end if
 do while (l > 0)
    m = modulo(u,l)
    u=l
    l=m
 end do
 c=u
 return
end function

It consists of a few parts: having u contains the largest value, l the smallest then looping until u modulo l  is 0. The simplification subroutine is even simpler.

subroutine simplify(a, b, c, d)
! Returns the fraction a/b in its simplified form
! c/d where c and d are relatively prime
 integer, intent(in) :: a,b
 integer, intent(out) :: c,d
 integer :: n,gcd
 n=gcd(a,b)
 c=a/n
 d=b/n
 return
end subroutine


Now, the core of the problem is solved by two other functions - one that takes care of fractions with a prefix, the other one of fractions without a prefix. There are some issues in the code presented, but at this point and for a simple presentation, this is not important.

subroutine findfractionpref(pref, rept, mpref, a, b)
! Returns the fraction a/b such as its division gives the pattern prefreptreptrept ! ....
! With the necessary multiplier
 integer, intent(in) :: pref, rept, mpref
 integer, intent(out) :: a, b
 integer :: d1, d2, num, den
 d1=1+floor(log10(real(pref)))
 d2=1+floor(log10(real(rept)))
 num=((pref*10**(d2)+rept)-pref)
 den=10**(d1+d2)-10**(d1)
 if ( mpref > 0) then
  den=den*10**mpref
 else
  num=num*10**mpref
 end if
 call simplify(num, den,a , b)
 return
end subroutine

subroutine findfractionnopref(rept, a,b)
! Returns the fraction a/b such as its division gives the pattern reptreptrept...
 integer, intent(in) :: rept
 integer, intent(out) :: a,b
 integer :: d1, num, den
 d1=1+floor(log10(real(rept)))
 num=rept
 den=10**d1-1
 call simplify(num,den,a,b)
 return
end subroutine 

Lastly, the main part of the program, used to read the various information and call the necessary subroutines.

program fractionfinder
 implicit none
 integer :: pref,rept,a,b
 character :: c
 do
  print *, 'Does your fraction include a prefix (yY/nN) or Q to quit (qQ)?'
  read(*,'(A1)'), c
  if ((c == 'y').or.(c == 'Y')) then
   print *, 'Prefix part?'
   read(*, '(I12)'), pref
   print *, 'Repeated part?'
   read(*,'(I12)'), rept
   call findfractionpref(pref,rept,0, a,b)
   print *, 'The requested fraction is ', a, '/', b
  else if ((c == 'n').or.(c == 'N')) then
   print *, 'Repeated part?'
   read(*, '(I8)'), rept
   call findfractionnopref(rept,a,b)
   print *, 'The requested fraction is ', a, '/', b
  else if ((c == 'q').or.(c == 'Q')) then
   goto 100
  end if
 end do
100 print *, 'Bye bye!'
 stop
end program



Many thanks go to Rae Simpson for the help she provided with some of the terms in this post!

New gTLD Applications

Do you know what "cruise", "pamperedchef", "通用电气公司" and "click" have in common? They are proposals for new generic top-level domain ("gTLD") names. We all know the "com", "org","uk", "be" and other "net" names such as google.com or wikipedia.org, but some companies and entities fell a bit constrained by having to use overly generic categorizations in either commercial, organizational, governmental or generic entities.

A first wave of requests created some domain specifics TLD, such as .aero, .biz, and so forth. Mikko Hypponen, the charismatic CRO of the Finnish security company F-Secure, recently posted on twitter a few examples of the next wave. He also posted the wiki page that includes all the new gTLD applications.

Surprisingly enough, there is no ".omg", ".lolcat" or ".canIhazchezburger". It is also interesting to see that the internationalized domain names are making an appearance in the TLDs.

Global warming, python and statistics part 2

In the previous post, we stopped after establishing the long term trends for the high and low temperatures. This gave a very general overview of how the temperatures are evolving on average over a long period of time, in this case a bit more than a century.

Let's refine a bit and determine how the temperature evolves for the same day of the year, namely January the first, April the first, July the first and October the first, for the various years in the dataset.

The temperature curve for each day in the year is quite specific and separated from the other days, both for the high and low temperatures. The weather on earth is a cycle of period about 1 year. 

Now, let's detrend the original data: this means removing the long term growing trend to have a data set composed uniquely of the short term variations. We will also plot the curve representing the averages over a year for each day of the year (all the January 1st, all the January 2nd ...)

For the averages over a year, the detrended datasets were used. To account for the loss of the average in the detrending process, the initial dataset average has been added back. A sinusoidal fitting has been added for the high and low temperature curves. As it appears, that fits quite well.

At this point, we have (a) a long term linear trend and (b) a yearly cycle. These information are useful to give an idea of how temperatures evolve, but they are only averages. Let's have a look at how the temperature is distributed around that average. 

In order to do that, the relevant days of the year (i.e. January 1st) will be corrected to remove the long term trend and the yearly cycle. This will leave the difference with the average temperature.  Let's do this with the same days as before, January 1st, April 1st, July 1st and October 1st. 

Be careful though: the long term trend AND the sinusoidal fitting both contain the average value - if these two are subtracted from the data, the result will be that the average value will be removed twice. In this case, I decided to ignore the offset value from the sinusoidal fitting. Other techniques are possible. 

Let's check how the temperature is distributed around the trend/cycle.

From this, it seems that the distribution is fairly normal. The parameters for the stacked distribution are mean μ=8.301619e-02C and the standard deviation σ=4.411646e+00C.

Now, let's be careful: even if the distribution of the differences around the trend+cycle looks like it is a normal distribution, do not confuse this with a normal random variable: the weather, while it looks strangely random, is not a variable independent of everything else, including its history: the temperature at day D will influence the temperature at day D+1. That is one of the reasons why the weather forecast is possible. If temperature were a purely random variable, there would be no forecasting of the weather. 

To be perfectly complete, there is a slight difference between the distributions for the high temperatures and the low temperatures. 

Temperature set	Mean [C]	Standard Deviation [C]
High	1.684078e-01	4.777461e+00
Low	-2.375521e-03	4.015207e+00

However, from now on, I will consider it as a "somehow random variable."

[To be continued]

Global warming, python and statistics

Global warming ... some people believe in it, others don't. A growing number of scientists show data to prove its reality, detractors show other data sets. However, debating of who is right is not the point here.

The interest - for me at least - of global warming is that large sets of weather data are available. In this article, I will use the Global Historical Climatology Network - Daily data set (here for the readme.txt) hosted by the National Oceanographic and Atmospheric Administration. This data set contains the daily measures for various atmospheric parameters such as maximum and minimum temperatures, precipitation and so forth, for various stations identified by their ID. In the data set I downloaded, each file contains the results for a single station.

The data is line by line, each line representing a month, with fixed length. From the readme file, the structure is as follow:

------------------------------
Variable Columns Type
------------------------------
ID 1-11 Character
YEAR 12-15 Integer
MONTH 16-17 Integer
ELEMENT 18-21 Character
VALUE1 22-26 Integer
MFLAG1 27-27 Character
QFLAG1 28-28 Character
SFLAG1 29-29 Character
VALUE2 30-34 Integer
MFLAG2 35-35 Character
QFLAG2 36-36 Character
SFLAG2 37-37 Character
. . .
. . .
. . .
VALUE31 262-266 Integer
MFLAG31 267-267 Character
QFLAG31 268-268 Character
SFLAG31 269-269 Character
------------------------------

Non possible days (such as February 30) and missing measures have a value set to -9999. For this, I am interested in two variables: TMAX and TMIN. they are expressed in tenth of Celsius. 

As said, the file is organized line by line, with each line representing a month's worth of measures. Each line as 31 entries of the type value+3 flags. The function readFile() reads each line and return a tuple of 4 numpy arrays: the dates for the measures of the High temperatures, the high temperature measures, the dates for the measures of the Low temperatures and the low temperatures.

Let's start with the weather station in Central Park, New York NY.

The reason for returning the dates as well is that there may be missing measurements, i.e. days for which the Tmax, the Tmin or both may be missing from the file. 

Here comes the first graph with two subplots: the Tmax is the top one.

It is quite difficult to say anything about the data besides that "they look the same shifted by about 10C". In order to remove some of the "hairy" behavior, we will apply a sliding average on the data, with 7, 30, 182, 365 and 3650 days, a week, roughly a month, roughly six months, a year and roughly ten years. 

A sliding average is simply the average calculated over the last N points. This is used to smooth out the possible variations due to either the randomness or the seasonal variations. Let's use an example.

The following is a straight line (slope: 1, intercept: 10) with a superimposed Gaussian noise (mean: 0, deviation: 5). The X-range goes from 0 to 10, using 1001 points. Here is a graph (not "the" graph - guess why!)

From it, it "looks like" there is an increase. However, this is not really visible. Let's apply a sliding window to it, with size 10, 20 and 50.

Even at sample size 10 (in blue), the trend is visible. At sample size 50 (in red), the trend is impossible to miss.

This sliding window is performed in python with the numpy.convolve() function. This function does the convolution of two vectors (one-dimension arrays).

Back to our temperature samples, let's use this tool to smooth the temperatures. So, what do we expect?

The sliding average over a week won't probably change a lot - the weather cycle on earth is about a year, or 52 weeks. The same for a month. With a sliding window of six weeks, I expect the seasonal cycle will start showing. I really expect the trends to be visible with the year and ten years averaging. 

With the year sliding window, the trend appears but is still quite noisy. With 10 years, there is no doubt left: a trend appears for both the high and low temperatures. A visual estimation gives that for both measures, over the period the average has risen by about 2.25°C. Let's use a better method than "it looks like."

In the statistician's toolbox, a good tool to estimate the trend of a dataset is  linear regression or linear fitting, usually using the least-square method (minimising the vertical distance).  I will not explain all the values returned by scipy.stats.linregress(). Of these, I will use the slope, the intercept and the R-value.

Data source (High)	Slope [°C/day] (High)	Intercept [°C] (High)	R-value [-] (High)	Slope [°C/day] (low)	Intercept [°C] (low)	R-value [-] (low)
Raw dataset	4.5933e-05	-1.6242e+01	6.4263e-02	3.2326e-05	-1.4788e+01	5.0001e-02
7-day sliding window average	4.5997e-05	-1.6287e+01	6.8779e-02	3.2374e-05	-1.4821e+01	5.2816e-02
30-day sliding window average	4.6027e-05	-1.6303e+01	7.1649e-02	3.2403e-05	-1.4837e+01	5.5005e-02
182-day sliding window average	4.6947e-05	-1.6924e+01	1.1339e-01	3.3059e-05	-1.5273e+01	8.7093e-02
365-day sliding window average	4.7696e-05	-1.7464e+01	6.8588e-01	3.3669e-05	-1.5716e+01	5.6593e-01
3650-day sliding window average	4.8371e-05	-1.7851e+01	9.0518e-01	3.2297e-05	-1.4709e+01	7.9373e-01

The first notable point is that the slopes are different for the low and for the high by about 1.6e-5 °C/day. Second - and it was expected - the R-value increases with the sliding window length: as the window increases, the data set is closer and closer to a line. As a result, the linear regression model matches more and more.

If the values for the slope don't look that much, remember that they are per day: over the course of 100 years, this represents about 1.68°C for the high temperature and 1.18°C for the low. The average temperature has risen by about 1.43°C over a century.

[To be continued]

Possible tetraquark particles spotted at the LHC

Scientists at the LHC may have found another type of matter in the form of tetraquarks, or particles made of 4 quarks. If confirmed, this would be an important step in understanding how matter behaves at its most fundamental level. 

The paper on arXiv:  arXiv:1404.1903v1 [hep-ex]

Chinese Physicists provide a lower bound for the speed of "spooky action at a distance"

This concerns the speed at which information is transferred between two entangled particles. The paper is really interesting and the lower bound - if correct - is four orders of magnitude higher than the speed of light.

The paper on arxiv: arXiv:1303.0614v2 [quant-ph].