A Technique for real-time Network Traffic Classification

This time I have decided to try out Medium! Click on the following link to read my article:

https://medium.com/@vipulchaskar/a-technique-for-real-time-network-traffic-classification-55fec70206f#.wjj4m6lyn

Running python commands from inside a bash shell

Hello all,

In this post I am going to show you how you can run python commands from inside a shell. There are already a few ways of doing so -

• Write your python code in a .py file and run "python yourfile.py"
• Open up IDLE shell by just typing "python" and writing your command in that shell.

There is also another option of python command -

$ python -c 'your_command_here' 

which will run the python command you specify in quotes. 

However, the problem with above mentioned methods is, they are time consuming. If you want to quickly perform a list comprehension or anything equivalent in python, you have to either write a script or manually open a python interactive terminal and write your command there. 

I am going to show you a little tweak you can do in your shell to execute python code (or parts of it) directly from command line. It works as below -

$ p '<your_python_code_here>'

You type "p" followed by the python statement you want to execute in quotes. For example,

Vipuls-MacBook-Pro: ~ $p "[x**2 for x in range(1,11)]" 

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Follow the steps below to add this tweak in your shell.

1. Open a terminal and go to your home directory.
2. Create a directory called bin (~/bin) and inside this directory, create a script called "p" (or you can give it any name you like)
3. Copy the following snippet in this script:

   #!/bin/bash 

   python -c "print $1"

4. Pretty straightforward. Give execute permissions to this script: 
             $ chmod +x ~/bin/p
5. Now you need to add this little script to your PATH variable such that shell can find it. To make your shell be able to find this little command, edit the bashrc file (~/.bashrc) to add the following line :

   export PATH=$PATH:~/bin

6. On some distributions, the name of .bashrc file might be different. (for example, on my Mac OS X, I had to edit .bash_profile file for these changes to take effect.)
7. After you have added above entry, you may need to restart the terminal.
8. Now that we have added this little tweak, type the following command to test it.

         $p 2+3

This should show you the result i.e. 5. To run regular python commands, you need to enclose them in double quotes to prevent the shell from interpreting them.

$p "<your_python_command_here>"

Now you can try putting advanced python commands in there. 

This little script "p" that we wrote doesn't actually support advanced python features which python interactive shell offers. You can modify this script to the following :

#!/bin/bash 

echo "import os; 

os.system(\"clear\") 

$1" > tempfile 

python -i < tempfile

This is a little more sophisticated version of our script. It starts the python interactive shell, clears the screen (such that the initial version information that gets printed is wiped out), executes the command we passed in this interactive shell and exits. 

This can be useful in that you can pass shell environment variables to python, manipulate them in python, and get the result back. For example,

Vipuls-MacBook-Pro: ~ $mongodbpath=/Users/vipulchaskar/mongodb/mongodb-osx-x86_64-3.2.1/bin

Vipuls-MacBook-Pro: ~ $p "'$mongodbpath'.split('/')"

['', 'Users', 'vipulchaskar', 'mongodb', 'mongodb-osx-x86_64-3.2.1', 'bin']

Vipuls-MacBook-Pro: ~ $

I honestly don't have further idea about how this can be more useful or how it can be extended further :P If you can think of any improvements for this, please leave a comment below.

Questions/comments/suggestions are welcome :)

Pattern in magic squares

I was recently playing around with octave - a programming language for numerical computation - when I came across an interesting property in magic squares.

Magic square is a specific arrangement of unique integers in a square grid, such that sum of numbers in any row, any column or any diagonal is the same. Read more about magic squares here.

Magic squares are usually represented by matrices - square matrices. Following is an example of a 3x3 magic square:

Notice that the addition of digits across any row, any column or diagonal is same - 15 in this case.

Row 1: 8 + 1 + 6 = 15

Column 2: 1 + 5 + 9 = 15

Diagonal 1: 8 + 5 + 2 = 15

These magic square matrices are always of dimension i x i where i is an integer greater than or equal to 3.

For example, following are magic squares of dimension 5x5 and 9x9 respectively.

 

You can verify that sum of digits across any row or column or diagonal adds up to 65 - for 5x5 square and 369 for 9x9 square.

Now, consider this,

For a 3x3 magic square, we assign shades of gray colour to numbers from lightest to darkest. That is, 1 will have the lightest colour (white) and 9 will have the darkest colour (black). All the numbers in between will have the corresponding shade. With this colour assignment, the 3x3 magic square will look something like this:

The bar on the right indicates the number and its corresponding shade. Notice that this figure corresponds to the following 3x3 magic square. "1" in the middle on first row has lightest shade (white) while "9" in the middle of last row has darkest shade (black). All the other numbers have their corresponding weighted shades.

Interesting patterns emerge when we apply the same colour map for magic squares of higher dimensions.

For the illustrations below, I have considered magic squares of dimensions from 3x3 to 100x100.

I observed 3 distinct patterns for magic squares in this range of dimensions (3-100). One for all magic squares with odd number of dimension (3x3, 5x5, 7x7, 9x9 ... 99x99 ) and two for alternate even number dimensions. One for 4x4, 8x8, 12x12, 16x16, 20x20 ... so on. And another for 6x6, 10x10, 14x14, 18x18, 22x22 ... so on.

In other words,

Pattern 1 (for odd number dimensions) applies to magic squares of dimensions : 3x3, 5x5, 7x7, 9x9 ...

Pattern 2 (for 1st set of even number dimensions) applies to magic squares of dimensions : 4x4, 8x8, 12x12, 16x16, 20x20 ...

Pattern 3 (for 2nd set of even number dimensions) applies to magic squares of dimensions : 6x6, 10x10, 14x14, 18x18, 22x22 ...

The sample of patterns I found for these 3 patterns is as follows:

Pattern 1 (Odd numbers)

Notice how you can see two distinct oblique lines where dark shade lies on upper left side (indicating larger numbers) and light shade lies on lower right side (indicating smaller numbers). This figure is of a 51x51 square.

Pattern 2 (First set of even numbers) 

In the second pattern, you can see checkered distribution of numbers. That is, a group of large numbers (box of dark shade) are surrounded on four sides by groups smaller numbers (box of light shade) and vice versa. The prominence of difference between these squared boxes seem to diminish as we approach the middle horizontal line. The colours appear to converge. This figure is of a 52x52 square.

Pattern 3 (2nd set of even numbers)

In this 3rd pattern, you can see 8 distinct vertical rectangular regions. Also, there are 4 oblique lines which seem to separate darker shades (larger numbers) on top from lighter shades (smaller numbers) at bottom. But the differentiation caused by oblique lines is not as prominent as in the first pattern above. Also we can see a few small dents on left side. This is not error in data - this is part of pattern itself. This figure is of a 50x50 square.

_______________________________________________________________________

As a PoC, I have created the following animations which indicate how these patterns grow as the dimensions of magic square go on increasing. In each of the animation, the number in larger font at bottom of the square indicates the dimension of that magic square. The bar on the right indicates the colour scale - range of colour and their corresponding numbers.

Pattern 1 (Odd numbers)

 As the dimensions increase, (3x3, 5x5, 7x7, 9x9 ... ) the number of elements or the number of cells in the figure go on increasing but notice that the overall pattern more or less remains the same.

Pattern 2 (First set of even numbers)

As the dimensions increase (4x4, 8x8, 12x12, 16x16 ...), you can see that number of checkered boxes go on increasing, but they always seem to converge at the middle horizontal row.

Pattern 3 (2nd set of even numbers)

With the increase in dimensions (6x6, 10x10, 14x14, 18x18 ...), you can see that the four vertical rectangles as well as three oblique lines become more and more clear. The overall pattern remains the same.

______________________________________________________________________

This observation was made only for magic squares upto dimension of 100x100 but I think the pattern should continue for squares of higher dimensions as well.

To see this running for yourself, run the following piece of code in Octave:

for i=3:100, 

>     imagesc(magic(i)), colormap(flipud(gray)), colorbar; 

>     xlabel(i); 

>     sleep(0.5); 

> end;

As of now, I am not aware of any mathematical/statistical implications or corollaries of these patterns of magic squares. I am yet to find any literature or resources that explain this phenomena in magic squares.

You are welcome to share below any additional findings, related resources or comments about this.

Creating a makeshift media server

Hi everyone,

Here I am going to explain a quick trick to turn your machine into a makeshift media server (for the lack of a better term). This will allow you to access media files on your machine such as movies, songs, pictures etc. from anywhere inside the wifi that machine is a part of.

Basically, you will be able to view movies, songs etc. from your laptop on any other device (such as smartphone, tablet) without having to explicitly transfer them. This is assuming that both the devices are on same network.

Open a terminal and cd into the directory where your media files reside.

In that directory, execute the following command

python -m SimpleHTTPServer 9999

This will start an HTTP server at the current directory. 9999 is the port number which it will listen to and you can change it to any other port number.

Running the HTTP server will make all the media files in current directory accessible from anywhere in the network by making an HTTP request to this machine. Note down the IP address of this machine.

Now, open the browser on your smartphone or tablet, and type the following URL:

http://<ip_of_your_machine>:9999

For example, in my case, it is http://192.168.0.104:9999/
Port number should be the same as one you gave in above command.

This will show you the list of all files in the directory where Simple HTTP Server is started. Click on any file and browser will invoke media player to play that file.

Press Ctrl+C on the terminal to close this HTTP server.

You can use this trick to watch movies from your laptop on your mobile without having to keep bulky laptop in front of you.

Thank you.

Unix timestamps and the year 2038 problem

Recently I read a couple of news articles about the Y2038 bug. That is, the year 2038 problem. The news -as often- had exaggerated this problem to the level that they claimed computers might be wiped out. So, I decided to write an article to explain what the issue really is.

32-bit unix systems represent time values as a signed 32 bit integer. So, if you want to store timestamp in your program, a signed 32 bit integer is used for that. The actual value stored in this integer is 'number of seconds passed since a known and fixed moment in the past'. This moment is known as "epoch" and is fixed on - 00:00:00 UTC on 1 January 1970. So basically, unix timestamps store a specific time as number of seconds passed since epoch to that moment.

For example, if I want to store this time - 00:00:00 - 1 January 1972 - in a unix timestamp, it will simply be written as integer - 63072000. This is the number of seconds between 1 January 1970 and 1 January 1972 i.e. number of seconds in 2 years.

Now, let's consider binary representation of this.
Since timestamps are 32 bit signed integers, they will be represented something like this -
0 0000000 10111010 00101110 10011110
Note that the leftmost digit is sign digit.

At each second, the value of this timestamp increases by 1. After some years, at 03:14:07 UTC on 19 January 2038 to be precise, value of this timestamp will go something like this

0 1111111 11111111 11111111 11111101          at 03:14:05
0 1111111 11111111 11111111 11111110          at 03:14:06
0 1111111 11111111 11111111 11111111          at 03:14:07
1 0000000 00000000 00000000 00000000          at 03:14:08
1 0000000 00000000 00000000 00000001          at 03:14:09

Did you notice what happened here? the integer representing time filled up all the memory allocated to it (31 bits to be precise) and wrote into the 32nd bit which is used to indicate sign. This is a typical integer overflow problem. 

After this point, since the sign bit is now set, unix systems will start reporting time as -1, -2, -3, -4 and so on...
This is obviously wrong and many systems which read or are dependant on time will crash.

Manifestations of this problem can occur in more places than you think. Embedded systems generally use this 32-bit timestamp and run the risk of running into Y2038 problem. Moreover, from my knowledge, they are not easily upgradable.

Also many file systems and databases implement the 32 bit timestamps. One specific example is MySQL, which has an inbuilt function to return the current 32bit timestamp. Needless to say, those who won't upgrade their MySQL deployments might see their applications go crazy on d-day :)

The solutions to this problem are not without a lot of compatibility issues. Changing the data type of timestamps from 32 bit signed integer to 32 bit unsigned will extend the validity of timestamps to ~2100, but this won't work with systems which need to store time before epoch - 1970. Another approach - changing the representation from 32-bit to 64-bit would cause many more incompatibilities at the binary level.

Thank you for reading. Do let me know if you want to see an article on some specific topic :)