A Technique for real-time Network Traffic Classification

This time I have decided to try out Medium! Click on the following link to read my article:

https://medium.com/@vipulchaskar/a-technique-for-real-time-network-traffic-classification-55fec70206f#.wjj4m6lyn

Running python commands from inside a bash shell

Hello all,

In this post I am going to show you how you can run python commands from inside a shell. There are already a few ways of doing so -

• Write your python code in a .py file and run "python yourfile.py"
• Open up IDLE shell by just typing "python" and writing your command in that shell.

There is also another option of python command -

$ python -c 'your_command_here' 

which will run the python command you specify in quotes. 

However, the problem with above mentioned methods is, they are time consuming. If you want to quickly perform a list comprehension or anything equivalent in python, you have to either write a script or manually open a python interactive terminal and write your command there. 

I am going to show you a little tweak you can do in your shell to execute python code (or parts of it) directly from command line. It works as below -

$ p '<your_python_code_here>'

You type "p" followed by the python statement you want to execute in quotes. For example,

Vipuls-MacBook-Pro: ~ $p "[x**2 for x in range(1,11)]" 

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Follow the steps below to add this tweak in your shell.

1. Open a terminal and go to your home directory.
2. Create a directory called bin (~/bin) and inside this directory, create a script called "p" (or you can give it any name you like)
3. Copy the following snippet in this script:

   #!/bin/bash 

   python -c "print $1"

4. Pretty straightforward. Give execute permissions to this script: 
             $ chmod +x ~/bin/p
5. Now you need to add this little script to your PATH variable such that shell can find it. To make your shell be able to find this little command, edit the bashrc file (~/.bashrc) to add the following line :

   export PATH=$PATH:~/bin

6. On some distributions, the name of .bashrc file might be different. (for example, on my Mac OS X, I had to edit .bash_profile file for these changes to take effect.)
7. After you have added above entry, you may need to restart the terminal.
8. Now that we have added this little tweak, type the following command to test it.

         $p 2+3

This should show you the result i.e. 5. To run regular python commands, you need to enclose them in double quotes to prevent the shell from interpreting them.

$p "<your_python_command_here>"

Now you can try putting advanced python commands in there. 

This little script "p" that we wrote doesn't actually support advanced python features which python interactive shell offers. You can modify this script to the following :

#!/bin/bash 

echo "import os; 

os.system(\"clear\") 

$1" > tempfile 

python -i < tempfile

This is a little more sophisticated version of our script. It starts the python interactive shell, clears the screen (such that the initial version information that gets printed is wiped out), executes the command we passed in this interactive shell and exits. 

This can be useful in that you can pass shell environment variables to python, manipulate them in python, and get the result back. For example,

Vipuls-MacBook-Pro: ~ $mongodbpath=/Users/vipulchaskar/mongodb/mongodb-osx-x86_64-3.2.1/bin

Vipuls-MacBook-Pro: ~ $p "'$mongodbpath'.split('/')"

['', 'Users', 'vipulchaskar', 'mongodb', 'mongodb-osx-x86_64-3.2.1', 'bin']

Vipuls-MacBook-Pro: ~ $

I honestly don't have further idea about how this can be more useful or how it can be extended further :P If you can think of any improvements for this, please leave a comment below.

Questions/comments/suggestions are welcome :)

Pattern in magic squares

I was recently playing around with octave - a programming language for numerical computation - when I came across an interesting property in magic squares.

Magic square is a specific arrangement of unique integers in a square grid, such that sum of numbers in any row, any column or any diagonal is the same. Read more about magic squares here.

Magic squares are usually represented by matrices - square matrices. Following is an example of a 3x3 magic square:

Notice that the addition of digits across any row, any column or diagonal is same - 15 in this case.

Row 1: 8 + 1 + 6 = 15

Column 2: 1 + 5 + 9 = 15

Diagonal 1: 8 + 5 + 2 = 15

These magic square matrices are always of dimension i x i where i is an integer greater than or equal to 3.

For example, following are magic squares of dimension 5x5 and 9x9 respectively.

 

You can verify that sum of digits across any row or column or diagonal adds up to 65 - for 5x5 square and 369 for 9x9 square.

Now, consider this,

For a 3x3 magic square, we assign shades of gray colour to numbers from lightest to darkest. That is, 1 will have the lightest colour (white) and 9 will have the darkest colour (black). All the numbers in between will have the corresponding shade. With this colour assignment, the 3x3 magic square will look something like this:

The bar on the right indicates the number and its corresponding shade. Notice that this figure corresponds to the following 3x3 magic square. "1" in the middle on first row has lightest shade (white) while "9" in the middle of last row has darkest shade (black). All the other numbers have their corresponding weighted shades.

Interesting patterns emerge when we apply the same colour map for magic squares of higher dimensions.

For the illustrations below, I have considered magic squares of dimensions from 3x3 to 100x100.

I observed 3 distinct patterns for magic squares in this range of dimensions (3-100). One for all magic squares with odd number of dimension (3x3, 5x5, 7x7, 9x9 ... 99x99 ) and two for alternate even number dimensions. One for 4x4, 8x8, 12x12, 16x16, 20x20 ... so on. And another for 6x6, 10x10, 14x14, 18x18, 22x22 ... so on.

In other words,

Pattern 1 (for odd number dimensions) applies to magic squares of dimensions : 3x3, 5x5, 7x7, 9x9 ...

Pattern 2 (for 1st set of even number dimensions) applies to magic squares of dimensions : 4x4, 8x8, 12x12, 16x16, 20x20 ...

Pattern 3 (for 2nd set of even number dimensions) applies to magic squares of dimensions : 6x6, 10x10, 14x14, 18x18, 22x22 ...

The sample of patterns I found for these 3 patterns is as follows:

Pattern 1 (Odd numbers)

Notice how you can see two distinct oblique lines where dark shade lies on upper left side (indicating larger numbers) and light shade lies on lower right side (indicating smaller numbers). This figure is of a 51x51 square.

Pattern 2 (First set of even numbers) 

In the second pattern, you can see checkered distribution of numbers. That is, a group of large numbers (box of dark shade) are surrounded on four sides by groups smaller numbers (box of light shade) and vice versa. The prominence of difference between these squared boxes seem to diminish as we approach the middle horizontal line. The colours appear to converge. This figure is of a 52x52 square.

Pattern 3 (2nd set of even numbers)

In this 3rd pattern, you can see 8 distinct vertical rectangular regions. Also, there are 4 oblique lines which seem to separate darker shades (larger numbers) on top from lighter shades (smaller numbers) at bottom. But the differentiation caused by oblique lines is not as prominent as in the first pattern above. Also we can see a few small dents on left side. This is not error in data - this is part of pattern itself. This figure is of a 50x50 square.

_______________________________________________________________________

As a PoC, I have created the following animations which indicate how these patterns grow as the dimensions of magic square go on increasing. In each of the animation, the number in larger font at bottom of the square indicates the dimension of that magic square. The bar on the right indicates the colour scale - range of colour and their corresponding numbers.

Pattern 1 (Odd numbers)

 As the dimensions increase, (3x3, 5x5, 7x7, 9x9 ... ) the number of elements or the number of cells in the figure go on increasing but notice that the overall pattern more or less remains the same.

Pattern 2 (First set of even numbers)

As the dimensions increase (4x4, 8x8, 12x12, 16x16 ...), you can see that number of checkered boxes go on increasing, but they always seem to converge at the middle horizontal row.

Pattern 3 (2nd set of even numbers)

With the increase in dimensions (6x6, 10x10, 14x14, 18x18 ...), you can see that the four vertical rectangles as well as three oblique lines become more and more clear. The overall pattern remains the same.

______________________________________________________________________

This observation was made only for magic squares upto dimension of 100x100 but I think the pattern should continue for squares of higher dimensions as well.

To see this running for yourself, run the following piece of code in Octave:

for i=3:100, 

>     imagesc(magic(i)), colormap(flipud(gray)), colorbar; 

>     xlabel(i); 

>     sleep(0.5); 

> end;

As of now, I am not aware of any mathematical/statistical implications or corollaries of these patterns of magic squares. I am yet to find any literature or resources that explain this phenomena in magic squares.

You are welcome to share below any additional findings, related resources or comments about this.

Creating a makeshift media server

Hi everyone,

Here I am going to explain a quick trick to turn your machine into a makeshift media server (for the lack of a better term). This will allow you to access media files on your machine such as movies, songs, pictures etc. from anywhere inside the wifi that machine is a part of.

Basically, you will be able to view movies, songs etc. from your laptop on any other device (such as smartphone, tablet) without having to explicitly transfer them. This is assuming that both the devices are on same network.

Open a terminal and cd into the directory where your media files reside.

In that directory, execute the following command

python -m SimpleHTTPServer 9999

This will start an HTTP server at the current directory. 9999 is the port number which it will listen to and you can change it to any other port number.

Running the HTTP server will make all the media files in current directory accessible from anywhere in the network by making an HTTP request to this machine. Note down the IP address of this machine.

Now, open the browser on your smartphone or tablet, and type the following URL:

http://<ip_of_your_machine>:9999

For example, in my case, it is http://192.168.0.104:9999/
Port number should be the same as one you gave in above command.

This will show you the list of all files in the directory where Simple HTTP Server is started. Click on any file and browser will invoke media player to play that file.

Press Ctrl+C on the terminal to close this HTTP server.

You can use this trick to watch movies from your laptop on your mobile without having to keep bulky laptop in front of you.

Thank you.

Unix timestamps and the year 2038 problem

Recently I read a couple of news articles about the Y2038 bug. That is, the year 2038 problem. The news -as often- had exaggerated this problem to the level that they claimed computers might be wiped out. So, I decided to write an article to explain what the issue really is.

32-bit unix systems represent time values as a signed 32 bit integer. So, if you want to store timestamp in your program, a signed 32 bit integer is used for that. The actual value stored in this integer is 'number of seconds passed since a known and fixed moment in the past'. This moment is known as "epoch" and is fixed on - 00:00:00 UTC on 1 January 1970. So basically, unix timestamps store a specific time as number of seconds passed since epoch to that moment.

For example, if I want to store this time - 00:00:00 - 1 January 1972 - in a unix timestamp, it will simply be written as integer - 63072000. This is the number of seconds between 1 January 1970 and 1 January 1972 i.e. number of seconds in 2 years.

Now, let's consider binary representation of this.
Since timestamps are 32 bit signed integers, they will be represented something like this -
0 0000000 10111010 00101110 10011110
Note that the leftmost digit is sign digit.

At each second, the value of this timestamp increases by 1. After some years, at 03:14:07 UTC on 19 January 2038 to be precise, value of this timestamp will go something like this

0 1111111 11111111 11111111 11111101          at 03:14:05
0 1111111 11111111 11111111 11111110          at 03:14:06
0 1111111 11111111 11111111 11111111          at 03:14:07
1 0000000 00000000 00000000 00000000          at 03:14:08
1 0000000 00000000 00000000 00000001          at 03:14:09

Did you notice what happened here? the integer representing time filled up all the memory allocated to it (31 bits to be precise) and wrote into the 32nd bit which is used to indicate sign. This is a typical integer overflow problem. 

After this point, since the sign bit is now set, unix systems will start reporting time as -1, -2, -3, -4 and so on...
This is obviously wrong and many systems which read or are dependant on time will crash.

Manifestations of this problem can occur in more places than you think. Embedded systems generally use this 32-bit timestamp and run the risk of running into Y2038 problem. Moreover, from my knowledge, they are not easily upgradable.

Also many file systems and databases implement the 32 bit timestamps. One specific example is MySQL, which has an inbuilt function to return the current 32bit timestamp. Needless to say, those who won't upgrade their MySQL deployments might see their applications go crazy on d-day :)

The solutions to this problem are not without a lot of compatibility issues. Changing the data type of timestamps from 32 bit signed integer to 32 bit unsigned will extend the validity of timestamps to ~2100, but this won't work with systems which need to store time before epoch - 1970. Another approach - changing the representation from 32-bit to 64-bit would cause many more incompatibilities at the binary level.

Thank you for reading. Do let me know if you want to see an article on some specific topic :)

How does idle scan work?

        Idle scan, also called as 'Zombie scan' or 'Side channel attack' is a technique of port scanning. It is termed to be very stealthy or totally blind port scan. This is because the attacker's actual IP address is never disclosed to the target being scanned. The scan takes place indirectly through a zombie host. The attacker need not have control over zombie at all. However there is one condition for any host to act as a zombie -- it should be idle. That means, the 'zombie' host should not be talking on the network when the scan is being done. It should not have any connections already open which are sending or receiving packets. Otherwise the scan will be unsuccessful. Why is that so.. we'll see in a short while.

        Idle scan exploits the "Identification" field in IP header (IPID). It is based on the fact that this IPID is incremented by 1 for each packet that a host sends. Although many modern operating systems are immune from this attack (they randomize the IPID field), there are still other network-connected devices like printers which implement the bare essential TCP/IP. Idle scan also makes use of the TCP 3 way handshake - the standard "SYN" - "SYN/ACK" - "ACK" pattern -  and also the way open and closed ports react differently to the "SYN" packets. An open port will respond to a "SYN" TCP packet with a "SYN/ACK", while a closed port will respond with an "RST". Also remember that an unsolicited "RST" is not entertained, it is simply ignored with no reply.

        Okay, with that covered, now we'll see what exactly happens when an idle scan takes place. Note that we need to have address of a "zombie" host which is idle. 

See the above diagram, the sequence of incidents that take place are as follows:

• Attacker first sends a TCP SYN packet to the Zombie which is up and idle i.e. not talking on the network.
• The zombie replies with a SYN/ACK, in turn, disclosing its IPID value in this packet.
• The attacker notes down this IPID value, which in this case, is 100. (Actual IPID's are much larger, considering the 2 byte size of this field)
• Attacker then sends spoofed TCP SYN packet to the victim, with source IP of the Zombie. (in this case, it is 10.10.10.20). This SYN request is to the port which is to be scanned. 

Following till now? Here onward, there are two cases, either the port on the victim is open or closed. 

Case - I (Victim port is open):

• If the port is open, the victim will send a SYN/ACK to the zombie (because the source ip in the SYN packet was spoofed to be zombie's).
• Zombie will go like, "hey 10.10.10.30, we didn't have any connection open" and reply back with an RST (Hence, incrementing its IPID by 1)
• After some time (usually, milliseconds) attacker will again send a SYN packet to the zombie.
• But the zombie was expecting an "ACK" since it had sent "SYN/ACK" to the attacker earlier. So, it will send an "RST" packet back to the attacker, with IPID incremented by 1 (Now to 102).
• The attacker now goes "Aha! The first IPID was 100 and now it is 102. That means that zombie must have sent one packet (IPID: 101) in between, to the victim. This packet must have been a TCP RST, because victim had sent to him, a SYN/ACK which zombie wasn't expecting! And since victim sent him a SYN/ACK. The victim's port is open! Yay!". Notice the kind of reverse thinking that goes into here. We trust that zombie must not have sent any other packet except for its "RST" reply to the unsolicited "SYN/ACK" that it got from victim.

That's why it is important to have a zombie that is not talking! The situation will be more clear when we consider the second case.

Case - I (Victim port is closed):

• If the port is closed, victim will reply with an RST to the zombie (again, because of the spoofed IP).
• Zombie is surprised to see an RST from victim when they didn't even have any connection open. Unsolicited RST's are ignored.. so zombie won't reply back to the victim.
• Again attacker sends a SYN packet to the zombie, or any packet for that matter, it is really interested only in the IPID of the packet that it gets back.
• From the packet that zombie sends back to the attacker, the IPID is obtained, which in this case is 101.
• The attacker says, "hmm.. lets see, first IPID was 100 and now its 101. That means zombie didn't send any packet to anyone in between. It implies that it must have either got an RST from victim (port was closed), or nothing at all (port was filtered)! Since the IPID is not incremented in between, the port is either closed or filtered.. in any way not interesting to me!"

Hence, as you can see that idle scan is a very sophisticated kind of scan where the attacker's original IP address is not revealed to the victim. It can also be used to find out the trust relationships between zombies and the victim. Victim or target may react differently to different zombies based on the trust relationship between them, if any. 

Popular port scanning tools like hping and nmap can be used to perform idle scans. The command for nmap goes like this:

user# nmap -Pn -p 80 -sI 10.10.10.20 10.10.10.30

-sI is option for idle scanning followed by the zombie's address. Here, we are scanning for port 80. I am not sure if hping gives you directly such option.. I guess you have to manually note down and compare the IPIDs..

Thank you for reading my article and I hope you found it informative :)

Open Redirector Vulnerabilities

          Open redirector is a type of web application vulnerability. It is very easy to understand and exploit.

Working:

Open redirect occurs when a page takes a URL as a parameter (through form or anything) and redirects the user to that URL without validation. Here is a sample PHP code which is vulnerable to Open Redirect.

<?php
$vuln = $_GET["url"];
header('Location: ' . $vuln);
?>

          It accepts a GET request parameter called 'url' and redirects the user to the address specified in this parameter. For example, save the above code as "redirect.php" on your local www root directory and access the page from your browser as follows:

http://127.0.0.1/redirect.php?url=http://www.google.com

This page will simply redirect you to the address specified (in this case, google.com) without checking its malicious or not. You can replace www.google.com with any other URL and it will work just fine.

Applications:

Open redirect vulnerabilities are mainly used for phishing. The victim is given a specially crafted link to the open redirector page. This open redirector page gets the malicious link through parameter where it redirects the user to. For example,

http://www.example.com/redirect.php?url=http://www.malicious.com

In this case, example.com is a trusted domain. Innocent users get tricked into believing that the link leads them to example.com (and therefore, it is trusted) whereas in reality, they will get redirected to malicious.com which can have phishing traps waiting for them. 

URL Obfuscation:

From the above link, it is very easy to determine that there is something phishy with the link. The malicious site which this website will redirect to is clearly visible and readable. Hence, there are some techniques of URL obfuscation that can be used to hide this malicious address. Here are few ways of doing so...

• Directory Traversal
• Hex Encoding

In directory traversal, you add random directory names in the PATH part of URL such that malicious address is further pushed to the right and victim is likely to ignore that as a part of long and complex link. For example, 

http://www.example.com/security/auth/ODialogue/../../../redirect.php?url=http://www.malicious.com

Hex encoding is a way of encoding characters of a URL into hexadecimal, separated by a % sign. It can completely encode the malicious address present in the URL such that user has no way of knowing which malicious address is there just by looking at the link. For example,

127.0.0.1/redirect.php?url=%68%74%74%70%3A%2F%2F%77%77%77%2E%67%6F%6F%67%6C%65%2E%63%6F%6D

In this link, I have hex encoded the address "http://www.google.com". It can be, of course, replaced by any other address.

Prevention:

To avoid open redirects, it is necessary to code the redirecting pages in such a way that they will validate the address before redirecting to it. Many websites do this by implementing "trusted redirects" (I dont know the actual name for it. If you do, please mention in comments!). They have a database of trusted web addresses i.e. the addresses which they will trust in redirecting their users to. So whenever the redirect page is called, it checks if the address it got from parameters exists in its trusted database. If it is present, the redirector will happily forward the user to that address. If it is not, they will probably show a warning or take the user back to previous page. This is indeed a good practice but there is a little workaround for it ;)

Bypassing Trusted Redirects:

For as much as I have seen, most of the trusted redirects will only check the domain part of the address that they are forwarding to. If the domain part matches with something in the table of trusted websites, you are good to redirect to. But what if one of your trusted websites has an XSS vulnerability? ;)

With an XSS vulnerability, you are able to replace the current page with any other address you wish. For example, consider the URL:

127.0.0.1/xss.php?search=<script>window.location='http://www.google.com';</script>

It will inject the javascript code into the page which will redirect you to the address specified. (in this case, google.com)

Now consider, what if we pass this exact URL to the open redirect vulnerable page that we discovered earlier? Here is the specially crafted URL.

http://www.example.com/redirect.php?url=http://www.trusted.com/xss.php?search=<script>window.location='http://www.malicious.com';</script>

Here, example.com is our own (assume, reputed) site whose users are going to be phished.
trusted.com is a website whom we trust redirect our users to.
malicious.com is the attacker's phishing page.

When a victim executes above URL, the following actions take place in sequence:

1.   The browser connects to example.com and asks for redirect.php page. It also passes everything after the first "?" sign as parameter.
2.   The redirect.php finds the 'url' parameter along with the value which is "http://www.trusted.com/...".
3.   It finds trusted.com in its own list of trusted sites and happily redirects user to this URL, not bothering to see what the rest of the URL is.
4.   The browser connects to trusted.com and requests xss.php page (which is vulnerable to XSS :P ). It also passes everything after the second "?" sign as parameter.
5.   The xss.php receives the "search" parameter along with the value which is Javascript code to be injected.
6.   Since xss.php is vulnerable, somewhere in its output page (which will be sent to the browser) it will have the value of "search" parameter, which is the injected code.
7.   The browser receives the page with injected code, it starts parsing and loading the page. It comes across the injected Javascript code.
8.   Since there is no way for browser to know that it is injected javascript, it starts executing it.
9.   The execution of window.location... statement will stop loading the current page and instead it will load URL mentioned in the window.location property instead. (This URL is the attacker's malicious page)
10.   The browser finally contacts malicious.com and the malicious page (which is most likely a phishing page) is displayed to the user.
(phew!)

So, as you can see, this is a very sophisticated way of bypassing trusted redirects. The user gets routed from 2 different sites before landing on our malicious page! :D Please note that this works on Firefox. Doesnt work on chrome. Never bothered testing on IE ;) . 

THIS INFORMATION IS STRICTLY FOR EDUCATIONAL PURPOSES AND I AM NOT RESPONSIBLE FOR ANY TROUBLE YOU FIND YOURSELF IN WITH IT.

Thank you for reading, good night :)