Saturday, March 5, 2016
Unix timestamps and the year 2038 problem
Recently I read a couple of news articles about the Y2038 bug. That is, the year 2038 problem. The news -as often- had exaggerated this problem to the level that they claimed computers might be wiped out. So, I decided to write an article to explain what the issue really is.
32-bit unix systems represent time values as a signed 32 bit integer. So, if you want to store timestamp in your program, a signed 32 bit integer is used for that. The actual value stored in this integer is 'number of seconds passed since a known and fixed moment in the past'. This moment is known as "epoch" and is fixed on - 00:00:00 UTC on 1 January 1970. So basically, unix timestamps store a specific time as number of seconds passed since epoch to that moment.
For example, if I want to store this time - 00:00:00 - 1 January 1972 - in a unix timestamp, it will simply be written as integer - 63072000. This is the number of seconds between 1 January 1970 and 1 January 1972 i.e. number of seconds in 2 years.
Now, let's consider binary representation of this.
Since timestamps are 32 bit signed integers, they will be represented something like this -
0 0000000 10111010 00101110 10011110
Note that the leftmost digit is sign digit.
At each second, the value of this timestamp increases by 1. After some years, at 03:14:07 UTC on 19 January 2038 to be precise, value of this timestamp will go something like this
0 1111111 11111111 11111111 11111101 at 03:14:05
0 1111111 11111111 11111111 11111110 at 03:14:06
0 1111111 11111111 11111111 11111111 at 03:14:07
1 0000000 00000000 00000000 00000000 at 03:14:08
1 0000000 00000000 00000000 00000001 at 03:14:09
Did you notice what happened here? the integer representing time filled up all the memory allocated to it (31 bits to be precise) and wrote into the 32nd bit which is used to indicate sign. This is a typical integer overflow problem.
After this point, since the sign bit is now set, unix systems will start reporting time as -1, -2, -3, -4 and so on...
This is obviously wrong and many systems which read or are dependant on time will crash.
Manifestations of this problem can occur in more places than you think. Embedded systems generally use this 32-bit timestamp and run the risk of running into Y2038 problem. Moreover, from my knowledge, they are not easily upgradable.
Also many file systems and databases implement the 32 bit timestamps. One specific example is MySQL, which has an inbuilt function to return the current 32bit timestamp. Needless to say, those who won't upgrade their MySQL deployments might see their applications go crazy on d-day :)
The solutions to this problem are not without a lot of compatibility issues. Changing the data type of timestamps from 32 bit signed integer to 32 bit unsigned will extend the validity of timestamps to ~2100, but this won't work with systems which need to store time before epoch - 1970. Another approach - changing the representation from 32-bit to 64-bit would cause many more incompatibilities at the binary level.
Thank you for reading. Do let me know if you want to see an article on some specific topic :)
Saturday, April 5, 2014
How does idle scan work?
Idle scan, also called as 'Zombie scan' or 'Side channel attack' is a technique of port scanning. It is termed to be very stealthy or totally blind port scan. This is because the attacker's actual IP address is never disclosed to the target being scanned. The scan takes place indirectly through a zombie host. The attacker need not have control over zombie at all. However there is one condition for any host to act as a zombie -- it should be idle. That means, the 'zombie' host should not be talking on the network when the scan is being done. It should not have any connections already open which are sending or receiving packets. Otherwise the scan will be unsuccessful. Why is that so.. we'll see in a short while.
Idle scan exploits the "Identification" field in IP header (IPID). It is based on the fact that this IPID is incremented by 1 for each packet that a host sends. Although many modern operating systems are immune from this attack (they randomize the IPID field), there are still other network-connected devices like printers which implement the bare essential TCP/IP. Idle scan also makes use of the TCP 3 way handshake - the standard "SYN" - "SYN/ACK" - "ACK" pattern - and also the way open and closed ports react differently to the "SYN" packets. An open port will respond to a "SYN" TCP packet with a "SYN/ACK", while a closed port will respond with an "RST". Also remember that an unsolicited "RST" is not entertained, it is simply ignored with no reply.
Okay, with that covered, now we'll see what exactly happens when an idle scan takes place. Note that we need to have address of a "zombie" host which is idle.
See the above diagram, the sequence of incidents that take place are as follows:
Case - I (Victim port is open):
Case - I (Victim port is closed):
Hence, as you can see that idle scan is a very sophisticated kind of scan where the attacker's original IP address is not revealed to the victim. It can also be used to find out the trust relationships between zombies and the victim. Victim or target may react differently to different zombies based on the trust relationship between them, if any.
Popular port scanning tools like hping and nmap can be used to perform idle scans. The command for nmap goes like this:
Thank you for reading my article and I hope you found it informative :)
Idle scan exploits the "Identification" field in IP header (IPID). It is based on the fact that this IPID is incremented by 1 for each packet that a host sends. Although many modern operating systems are immune from this attack (they randomize the IPID field), there are still other network-connected devices like printers which implement the bare essential TCP/IP. Idle scan also makes use of the TCP 3 way handshake - the standard "SYN" - "SYN/ACK" - "ACK" pattern - and also the way open and closed ports react differently to the "SYN" packets. An open port will respond to a "SYN" TCP packet with a "SYN/ACK", while a closed port will respond with an "RST". Also remember that an unsolicited "RST" is not entertained, it is simply ignored with no reply.
Okay, with that covered, now we'll see what exactly happens when an idle scan takes place. Note that we need to have address of a "zombie" host which is idle.
- Attacker first sends a TCP SYN packet to the Zombie which is up and idle i.e. not talking on the network.
- The zombie replies with a SYN/ACK, in turn, disclosing its IPID value in this packet.
- The attacker notes down this IPID value, which in this case, is 100. (Actual IPID's are much larger, considering the 2 byte size of this field)
- Attacker then sends spoofed TCP SYN packet to the victim, with source IP of the Zombie. (in this case, it is 10.10.10.20). This SYN request is to the port which is to be scanned.
Case - I (Victim port is open):
- If the port is open, the victim will send a SYN/ACK to the zombie (because the source ip in the SYN packet was spoofed to be zombie's).
- Zombie will go like, "hey 10.10.10.30, we didn't have any connection open" and reply back with an RST (Hence, incrementing its IPID by 1)
- After some time (usually, milliseconds) attacker will again send a SYN packet to the zombie.
- But the zombie was expecting an "ACK" since it had sent "SYN/ACK" to the attacker earlier. So, it will send an "RST" packet back to the attacker, with IPID incremented by 1 (Now to 102).
- The attacker now goes "Aha! The first IPID was 100 and now it is 102. That means that zombie must have sent one packet (IPID: 101) in between, to the victim. This packet must have been a TCP RST, because victim had sent to him, a SYN/ACK which zombie wasn't expecting! And since victim sent him a SYN/ACK. The victim's port is open! Yay!". Notice the kind of reverse thinking that goes into here. We trust that zombie must not have sent any other packet except for its "RST" reply to the unsolicited "SYN/ACK" that it got from victim.
Case - I (Victim port is closed):
- If the port is closed, victim will reply with an RST to the zombie (again, because of the spoofed IP).
- Zombie is surprised to see an RST from victim when they didn't even have any connection open. Unsolicited RST's are ignored.. so zombie won't reply back to the victim.
- Again attacker sends a SYN packet to the zombie, or any packet for that matter, it is really interested only in the IPID of the packet that it gets back.
- From the packet that zombie sends back to the attacker, the IPID is obtained, which in this case is 101.
- The attacker says, "hmm.. lets see, first IPID was 100 and now its 101. That means zombie didn't send any packet to anyone in between. It implies that it must have either got an RST from victim (port was closed), or nothing at all (port was filtered)! Since the IPID is not incremented in between, the port is either closed or filtered.. in any way not interesting to me!"
Hence, as you can see that idle scan is a very sophisticated kind of scan where the attacker's original IP address is not revealed to the victim. It can also be used to find out the trust relationships between zombies and the victim. Victim or target may react differently to different zombies based on the trust relationship between them, if any.
Popular port scanning tools like hping and nmap can be used to perform idle scans. The command for nmap goes like this:
user# nmap -Pn -p 80 -sI 10.10.10.20 10.10.10.30-sI is option for idle scanning followed by the zombie's address. Here, we are scanning for port 80. I am not sure if hping gives you directly such option.. I guess you have to manually note down and compare the IPIDs..
Thank you for reading my article and I hope you found it informative :)
Monday, February 10, 2014
Open Redirector Vulnerabilities
Open redirector is a type of web application vulnerability. It is very easy to understand and exploit.
Working:
Open redirect occurs when a page takes a URL as a parameter (through form or anything) and redirects the user to that URL without validation. Here is a sample PHP code which is vulnerable to Open Redirect.
This page will simply redirect you to the address specified (in this case, google.com) without checking its malicious or not. You can replace www.google.com with any other URL and it will work just fine.
Applications:
Open redirect vulnerabilities are mainly used for phishing. The victim is given a specially crafted link to the open redirector page. This open redirector page gets the malicious link through parameter where it redirects the user to. For example,
In this case, example.com is a trusted domain. Innocent users get tricked into believing that the link leads them to example.com (and therefore, it is trusted) whereas in reality, they will get redirected to malicious.com which can have phishing traps waiting for them.
URL Obfuscation:
From the above link, it is very easy to determine that there is something phishy with the link. The malicious site which this website will redirect to is clearly visible and readable. Hence, there are some techniques of URL obfuscation that can be used to hide this malicious address. Here are few ways of doing so...
Hex encoding is a way of encoding characters of a URL into hexadecimal, separated by a % sign. It can completely encode the malicious address present in the URL such that user has no way of knowing which malicious address is there just by looking at the link. For example,
In this link, I have hex encoded the address "http://www.google.com". It can be, of course, replaced by any other address.
Prevention:
To avoid open redirects, it is necessary to code the redirecting pages in such a way that they will validate the address before redirecting to it. Many websites do this by implementing "trusted redirects" (I dont know the actual name for it. If you do, please mention in comments!). They have a database of trusted web addresses i.e. the addresses which they will trust in redirecting their users to. So whenever the redirect page is called, it checks if the address it got from parameters exists in its trusted database. If it is present, the redirector will happily forward the user to that address. If it is not, they will probably show a warning or take the user back to previous page. This is indeed a good practice but there is a little workaround for it ;)
Bypassing Trusted Redirects:
For as much as I have seen, most of the trusted redirects will only check the domain part of the address that they are forwarding to. If the domain part matches with something in the table of trusted websites, you are good to redirect to. But what if one of your trusted websites has an XSS vulnerability? ;)
With an XSS vulnerability, you are able to replace the current page with any other address you wish. For example, consider the URL:
It will inject the javascript code into the page which will redirect you to the address specified. (in this case, google.com)
Now consider, what if we pass this exact URL to the open redirect vulnerable page that we discovered earlier? Here is the specially crafted URL.
Here, example.com is our own (assume, reputed) site whose users are going to be phished.
trusted.com is a website whom we trust redirect our users to.
malicious.com is the attacker's phishing page.
When a victim executes above URL, the following actions take place in sequence:
1. The browser connects to example.com and asks for redirect.php page. It also passes everything after the first "?" sign as parameter.
2. The redirect.php finds the 'url' parameter along with the value which is "http://www.trusted.com/...".
3. It finds trusted.com in its own list of trusted sites and happily redirects user to this URL, not bothering to see what the rest of the URL is.
4. The browser connects to trusted.com and requests xss.php page (which is vulnerable to XSS :P ). It also passes everything after the second "?" sign as parameter.
5. The xss.php receives the "search" parameter along with the value which is Javascript code to be injected.
6. Since xss.php is vulnerable, somewhere in its output page (which will be sent to the browser) it will have the value of "search" parameter, which is the injected code.
7. The browser receives the page with injected code, it starts parsing and loading the page. It comes across the injected Javascript code.
8. Since there is no way for browser to know that it is injected javascript, it starts executing it.
9. The execution of window.location... statement will stop loading the current page and instead it will load URL mentioned in the window.location property instead. (This URL is the attacker's malicious page)
10. The browser finally contacts malicious.com and the malicious page (which is most likely a phishing page) is displayed to the user.
(phew!)
So, as you can see, this is a very sophisticated way of bypassing trusted redirects. The user gets routed from 2 different sites before landing on our malicious page! :D Please note that this works on Firefox. Doesnt work on chrome. Never bothered testing on IE ;) .
THIS INFORMATION IS STRICTLY FOR EDUCATIONAL PURPOSES AND I AM NOT RESPONSIBLE FOR ANY TROUBLE YOU FIND YOURSELF IN WITH IT.
Thank you for reading, good night :)
<?phpIt accepts a GET request parameter called 'url' and redirects the user to the address specified in this parameter. For example, save the above code as "redirect.php" on your local www root directory and access the page from your browser as follows:
$vuln = $_GET["url"];
header('Location: ' . $vuln);
?>
http://127.0.0.1/redirect.php?url=http://www.google.com
This page will simply redirect you to the address specified (in this case, google.com) without checking its malicious or not. You can replace www.google.com with any other URL and it will work just fine.
Applications:
Open redirect vulnerabilities are mainly used for phishing. The victim is given a specially crafted link to the open redirector page. This open redirector page gets the malicious link through parameter where it redirects the user to. For example,
http://www.example.com/redirect.php?url=http://www.malicious.com
In this case, example.com is a trusted domain. Innocent users get tricked into believing that the link leads them to example.com (and therefore, it is trusted) whereas in reality, they will get redirected to malicious.com which can have phishing traps waiting for them.
URL Obfuscation:
From the above link, it is very easy to determine that there is something phishy with the link. The malicious site which this website will redirect to is clearly visible and readable. Hence, there are some techniques of URL obfuscation that can be used to hide this malicious address. Here are few ways of doing so...
- Directory Traversal
- Hex Encoding
http://www.example.com/security/auth/ODialogue/../../../redirect.php?url=http://www.malicious.com
Hex encoding is a way of encoding characters of a URL into hexadecimal, separated by a % sign. It can completely encode the malicious address present in the URL such that user has no way of knowing which malicious address is there just by looking at the link. For example,
127.0.0.1/redirect.php?url=%68%74%74%70%3A%2F%2F%77%77%77%2E%67%6F%6F%67%6C%65%2E%63%6F%6D
In this link, I have hex encoded the address "http://www.google.com". It can be, of course, replaced by any other address.
Prevention:
To avoid open redirects, it is necessary to code the redirecting pages in such a way that they will validate the address before redirecting to it. Many websites do this by implementing "trusted redirects" (I dont know the actual name for it. If you do, please mention in comments!). They have a database of trusted web addresses i.e. the addresses which they will trust in redirecting their users to. So whenever the redirect page is called, it checks if the address it got from parameters exists in its trusted database. If it is present, the redirector will happily forward the user to that address. If it is not, they will probably show a warning or take the user back to previous page. This is indeed a good practice but there is a little workaround for it ;)
Bypassing Trusted Redirects:
For as much as I have seen, most of the trusted redirects will only check the domain part of the address that they are forwarding to. If the domain part matches with something in the table of trusted websites, you are good to redirect to. But what if one of your trusted websites has an XSS vulnerability? ;)
With an XSS vulnerability, you are able to replace the current page with any other address you wish. For example, consider the URL:
127.0.0.1/xss.php?search=<script>window.location='http://www.google.com';</script>
It will inject the javascript code into the page which will redirect you to the address specified. (in this case, google.com)
Now consider, what if we pass this exact URL to the open redirect vulnerable page that we discovered earlier? Here is the specially crafted URL.
http://www.example.com/redirect.php?url=http://www.trusted.com/xss.php?search=<script>window.location='http://www.malicious.com';</script>
Here, example.com is our own (assume, reputed) site whose users are going to be phished.
trusted.com is a website whom we trust redirect our users to.
malicious.com is the attacker's phishing page.
When a victim executes above URL, the following actions take place in sequence:
1. The browser connects to example.com and asks for redirect.php page. It also passes everything after the first "?" sign as parameter.
2. The redirect.php finds the 'url' parameter along with the value which is "http://www.trusted.com/...".
3. It finds trusted.com in its own list of trusted sites and happily redirects user to this URL, not bothering to see what the rest of the URL is.
4. The browser connects to trusted.com and requests xss.php page (which is vulnerable to XSS :P ). It also passes everything after the second "?" sign as parameter.
5. The xss.php receives the "search" parameter along with the value which is Javascript code to be injected.
6. Since xss.php is vulnerable, somewhere in its output page (which will be sent to the browser) it will have the value of "search" parameter, which is the injected code.
7. The browser receives the page with injected code, it starts parsing and loading the page. It comes across the injected Javascript code.
8. Since there is no way for browser to know that it is injected javascript, it starts executing it.
9. The execution of window.location... statement will stop loading the current page and instead it will load URL mentioned in the window.location property instead. (This URL is the attacker's malicious page)
10. The browser finally contacts malicious.com and the malicious page (which is most likely a phishing page) is displayed to the user.
(phew!)
So, as you can see, this is a very sophisticated way of bypassing trusted redirects. The user gets routed from 2 different sites before landing on our malicious page! :D Please note that this works on Firefox. Doesnt work on chrome. Never bothered testing on IE ;) .
THIS INFORMATION IS STRICTLY FOR EDUCATIONAL PURPOSES AND I AM NOT RESPONSIBLE FOR ANY TROUBLE YOU FIND YOURSELF IN WITH IT.
Thank you for reading, good night :)
Implementation of Apriori Algorithm
Apriori algorithm is used for finding frequently occurring items and associative rule mining from from an input database which is transactional. Associative rule mining and Apriori algorithm are part of a bigger domain of data mining. Data mining is basically the process of discovering patterns in large data sets. There can be many applications of apriori algorithm e.g. market basket analysis. Here is a link I found particularly useful for learning Apriori Algorithm and associative rule mining:
http://www.cs.uic.edu/~liub/teach/cs583-spring-12/cs583.html
I have implemented the first two passes of Apriori Algorithm as a part of an academic assignment. It was for constructing attack graphs which are used for threat prediction and vulnerability assessment. The algorithm is implemented in python and its very simple. It can be extended for k passes of the algorithm.
Here is the code:
minsup = 0.3
minconf = 0.8
def count_first(transactions):
adict = {}
for t in transactions:
for item in t:
if item in adict:
adict[item] += 1
else:
adict[item] = 1
return adict
def find_frequent(Candidate, minsup, no_of_lines):
adict={}
for key in Candidate:
if ((float)(Candidate[key]/no_of_lines)) >= minsup:
adict[key] = Candidate[key]
return adict
def candidate_gen(keys):
adict={}
for i in keys:
for j in keys:
if i != j and (j,i) not in adict:
adict[tuple([min(i,j),max(i,j)])] = 0
return adict
def add_frequency(Candidate, transactions):
for key in Candidate:
for t in transactions:
if key[0] in t and key[1] in t:
Candidate[key] += 1
return Candidate
f = open("testcase.txt","r")
transactions = []
no_of_lines=0
for line in f:
split_line = line.split()
transactions.append(split_line)
no_of_lines = no_of_lines + 1
print(no_of_lines)
#First iteration
C1 = count_first(transactions)
F1 = find_frequent(C1,minsup,no_of_lines)
#Second iteration
C2 = candidate_gen(F1.keys())
C2 = add_frequency(C2,transactions)
F2 = find_frequent(C2,minsup,no_of_lines)
print(F2)
It accepts input data from the file "testcase.txt". In this file, each new line represents one transaction. Each such transaction contains items represented by 'integers' and separated by spaces. So, essentially, each line is a collection of integers separated by spaces.
The program makes two passes of apriori over input data. It outputs pairs of frequent item sets along with their 'support' metric values.
Comments and suggestions are welcome :)
Sunday, January 12, 2014
Why are social networking sites a hit?
Before a couple of decades, no one would have thought something like online social networking sites would become phenomenal. The fever and reach of social networking websites has increased exponentially in the last decade and will continue to do so. People are appreciating it and the creators of these sites are enjoying huge success in relatively short period of time. It has been the new buzzword around for quite some time. Everyone of us is
signed up on at least one or the other social networking sites. People are appreciating it. In this article, I will try to probe into reasons which makes social networking a huge success. This writing is largely applicable to facebook and similar sites.
If you ask yourself or ask around about reasons of using social networking, you are likely to get a couple of common answers:
1. You can remain connected to your friends and family across the world.
2. You can share the stuff you like with your friends.
If you dig a little deeper, you can get more answers like 'keeping up to date with happenings with our friends' and similar. These reasons are perfectly valid, however, I think it goes beyond that. When asked why do they use social networking, people generally think superficially and answer what they are consciously able to observe - sharing, connecting and expressing their opinions. Notice that these reasons are related to what people observe. I believe that actual reasons can be found out using psychology rather than sociology.
Being human beings, we all have certain kinds of desires or feelings. Social networking websites exploit exactly these desires. Bitter it may sound, but it is the truth. There are lot of different desires a typical person can unconsciously have - lust, jealousy, hatred, narcissism, appetite for recognition and fame and the need to look socially cool. (I am not remarking these as good or bad). The websites like facebook give a perfect platform where these desires or feelings are given scope for satisfaction.
A surprisingly large number of facebook users stalk someone. This perfectly compliments their feelings of jealousy or hatred without disrepute of their social image.
Stalking can also be out of lust. This can be prominently observed in Indian users where there is significantly large cultural and lifestyle gap between people.
Consider the recent mushrooming of photographers and their clients on facebook. Lot of people are getting professionally photographed and edited. This can be attributed to the appetite for recognition and fame. Good photos get more 'likes' which satistifes the appetite for recognition.
Take narcissism and self-obsession. Yes, a lot of us share this trait and facebook gives a perfectly good platform to upload your good looking photos and showing off which cool places you've been to. Especially, photos with an attractive person of opposite sex are the first ones to make it to facebook.
Desire of lust is a perfect reason for photos of attractive females getting more 'likes' than their male counterpart.
Jealousy and depression are addictive. A person feeling jealous or depressed continues to do acts which makes him/her more jealous (stalking, for example). The worst sufferers are people with social awkwardness, low self-esteem or those who have inactive social life, typically introverts. They get exposed to the happening life of people around them which leads to jealousy which makes them more addicted to it. The vicious cycle continues. There is another term for it - "Fear of Missing Out" (FOMO). Wikipedia defines FOMO as -
Fear of missing out or FOMO is a form of social anxiety — a compulsive concern that one might miss an opportunity for social interaction, a novel experience, profitable investment or other satisfying event.
It is commonly found in people who have unsatisfied psychological needs such as love and respect.
All in all, Social networking sites compliment many of the unconscious desires we all have. Everything mentioned above leads to more and more use of them, thereby, making them a huge hit.
This post is not intended for any specific person or group. It is just a my neutral observation of the world around me. Please note that my views expressed here are not applicable to everyone out there, but they hold true for masses of young people on internet.
signed up on at least one or the other social networking sites. People are appreciating it. In this article, I will try to probe into reasons which makes social networking a huge success. This writing is largely applicable to facebook and similar sites.
If you ask yourself or ask around about reasons of using social networking, you are likely to get a couple of common answers:
1. You can remain connected to your friends and family across the world.
2. You can share the stuff you like with your friends.
If you dig a little deeper, you can get more answers like 'keeping up to date with happenings with our friends' and similar. These reasons are perfectly valid, however, I think it goes beyond that. When asked why do they use social networking, people generally think superficially and answer what they are consciously able to observe - sharing, connecting and expressing their opinions. Notice that these reasons are related to what people observe. I believe that actual reasons can be found out using psychology rather than sociology.
Being human beings, we all have certain kinds of desires or feelings. Social networking websites exploit exactly these desires. Bitter it may sound, but it is the truth. There are lot of different desires a typical person can unconsciously have - lust, jealousy, hatred, narcissism, appetite for recognition and fame and the need to look socially cool. (I am not remarking these as good or bad). The websites like facebook give a perfect platform where these desires or feelings are given scope for satisfaction.
A surprisingly large number of facebook users stalk someone. This perfectly compliments their feelings of jealousy or hatred without disrepute of their social image.
Stalking can also be out of lust. This can be prominently observed in Indian users where there is significantly large cultural and lifestyle gap between people.
Consider the recent mushrooming of photographers and their clients on facebook. Lot of people are getting professionally photographed and edited. This can be attributed to the appetite for recognition and fame. Good photos get more 'likes' which satistifes the appetite for recognition.
Take narcissism and self-obsession. Yes, a lot of us share this trait and facebook gives a perfectly good platform to upload your good looking photos and showing off which cool places you've been to. Especially, photos with an attractive person of opposite sex are the first ones to make it to facebook.
Desire of lust is a perfect reason for photos of attractive females getting more 'likes' than their male counterpart.
Jealousy and depression are addictive. A person feeling jealous or depressed continues to do acts which makes him/her more jealous (stalking, for example). The worst sufferers are people with social awkwardness, low self-esteem or those who have inactive social life, typically introverts. They get exposed to the happening life of people around them which leads to jealousy which makes them more addicted to it. The vicious cycle continues. There is another term for it - "Fear of Missing Out" (FOMO). Wikipedia defines FOMO as -
Fear of missing out or FOMO is a form of social anxiety — a compulsive concern that one might miss an opportunity for social interaction, a novel experience, profitable investment or other satisfying event.
It is commonly found in people who have unsatisfied psychological needs such as love and respect.
All in all, Social networking sites compliment many of the unconscious desires we all have. Everything mentioned above leads to more and more use of them, thereby, making them a huge hit.
This post is not intended for any specific person or group. It is just a my neutral observation of the world around me. Please note that my views expressed here are not applicable to everyone out there, but they hold true for masses of young people on internet.
Tuesday, December 3, 2013
Profiling based on Social Media Behavior
Rise of social media and networks is a hot topic of the decade. People have been flocking to social media sites like never before. Large proportion of traffic flowing on the internet is carrying updates from social networks. These social networks have become platform for everything from showcasing your lunch food to clamoring against barbaric governments. In essence, these digital platforms are gaining deeper roots in society.
If you look at them from an Analyst's perspective, you will find a golden pot of data on social sentiments. This huge collection of user generated opinions, likes, expressions and patterns waiting to be explored and analysed. There is an entire field called social media analytics or simply, data analytics that seeks to tap this golden pot. Data analytics is instrumental for advertising and marketing. The data collected from a user's behavior on social networking sites or e-commerce sites is processed to throw targeted advertising at them. That is, highly customized and tailored adverts based on the customer's interests, age, sex, geographical location and browsing history. This field has been gaining lot of attention these days but its focus has been only one - targeted advertising. Clearly, application of social media analytics in marketing has lot of benefits.
I believe there is another application of analyzing social media behavior - behavioral profiling. And I am not talking in terms profiling-for-targeted-advertising or profiling-for-product-suggestions. This behavioral profiling can be thought of as a unique 'digital footprint of user's behavior'. I will explain what it exactly stands for in the next paragraph. Please note that is is a relatively new idea and I have found very little to no material available about this.
I think every regular user of social media has their own unique way of using it and communicating through it. Each regular user can have different interests. It can be understood by analyzing-
1. the kind of pages they visit
2. the kind of content they like
3. the kind of group they are part of
4. the profiles they visit
5. the kind of content they comment on
6. the people they interact with
Each user can generate different data sets for above mentioned parameters. This largely requires judging the user and finding these traits.
Also I think users can be distinguished by analyzing their chat logs or messages sent. 'Chatting styles' can vary from person to person. Some characteristics based on which the chats or messages can differ are -
1. Capitalization used
2. Use of trailing dots after a sentence
3. Short-forms used
4. Use of punctuation and places at which it is used
5. Use of smileys, their type and frequency
6. The frequency of splitting sentences
7. Frequency of common words
8. Spelling mistakes
For example, some users may religiously follow capitalization rule for nouns and some may not use capitalization at all. Sometimes, all the first letters in a sentence of a chat log maybe capital, indicating that the user was probably chatting through a smartphone because most smartphone keypads make first letter capital while typing. Some novice users may use lot of trailing dots and some may put two or three after their sentences. Some people may use a lot of short forms while chatting, some may use them only for few words. A few may always use one short form for a particular word which almost becomes a trademark to their chats. People can also be distinguished by the amount of punctuation and smileys they use. Some may put an exclamation mark after many of their messages, others may heavily use their favorite smiley. Some people may type long chats and then hit 'send' while some others may hit 'send' after every few words. Some users may involuntarily repeat a word or phrase many times. Someone may spell a word wrongly and it may be exhibited in many of their chats. Some may type very fast and some might be very slow. If you are a regular user, I am sure you will find at least one of these traits in one of your friends.
The point I am trying to get across is, all these chatting traits can be recorded to form a unique identity of each user. It is very easy to fake the display identity (i.e. name) on the social networking sites, but still these chatting styles remain unique to the human. These traits cannot be changed unless the user is conscious and aware about this type of monitoring. This brings us to the applications of such analysis...
This type of analysis is of very little use to the marketers and advertisers but it can be very useful for intelligence agencies. As I said, criminals and terrorists often spoof their identities but may be unaware of these subtle details that go unnoticed in their chats. Intelligence and government can exploit this to identify suspects. They can try matching these behavioral traits of a known criminal with that of a suspect that is being monitored and listened onto.
Again, the application is limited to intelligence agencies in detecting fraud and criminals but I believe it can be immensely useful to them. Also this kind of intelligence gathering requires access to private data (i.e. chat logs, user behavior info etc) which is not revealed to public but a federal agency may access.
Please note that this is a relatively new idea on which I found no documentation available. I have tried to explain it the best I can. Feel free to contact me if you have anything to say. Also I will be glad to receive some pointers if there is any work being done on this.
Thank you :)
Thursday, October 31, 2013
Analysis of searches for Abandonwares
Greetings to all!
My first semester is finished and we have a fortnight of preparation leave before the final exam. It was pretty busy, specially near the end of semester for completion of all the remaining stuff. During this semester we were supposed to make a management system project in vb6. Yes you read that right. VB6. As you might be aware, many engineering colleges in India are still using very old softwares written in the 1990's for their academics. Even though we were free to choose any latter versions of visual basic, most of the students preferred to go with vb6 for their project. This is because we were supposed to use that software for all our college assignments. Even the official support from microsoft for vb6 ended a few years ago.
This made me curious about the current usage of such old softwares like vb6, turbo c, oracle database 8/9 etc. So, since we're having some free time in hand during PLs, I did some analysis on google trends about these softwares (Or as they are called -Abandonwares- the softwares which are abandoned or rarely used). It came up with some interesting findings which I am going to share here with u all...
1) Turbo C
This screenshot shows how the usage of turbo c has been prominently declining, especially during 2005-2006. Since then it has been steadily going down. Below that, we can see the top countries which are googling for 'turbo c'. Philipines at the top followed by India and its neighbours. Striking fact is that Philipines and India are rapidly developing in IT related sectors.
This chart is for just another related search term "turbo c" (notice the space). As it shows, it is most googled for from India.
This graph shows comparative statistics for "turboc" search term from four different countries.
Developing countries-
1. India (blue)
2. Brazil (green)
Developed countries-
1. USA (red)
2. UK (yellow)
As you can see, the search volume from developed countries is negligibly low, and even for Brazil, it has considerably gone down.
This shows the results for the same search term "turboc" from different parts of India. As there are many engineering colleges in southern India, you can see those states leading this chart.
This graph shows the time series of the same search term for India. What I find interesting is that, it reflects the time range followed by the semester pattern in most engineering colleges in India. In January, the spring semester begins which is reflected by the rise you can see in Jan 2013. The semester usually ends in April and in may we have summer breaks. This is outlined by the dip in graph after Apr 2013. In June-July, the fall semester starts which is again indicated by the rise in searches for "turboc".
2) Visual Basic 6
The world seems to be steadily giving up on Visual Basic 6. Here what we have is almost linear line indicating how the searches for vb6 has been declined.
Coming to the comparative graph of the previous 4 countries. USA, UK and Brazil seems to have gave up on vb6 and moved on to more recent versions. While India is seen catching up very fastly. And again the south states are leading from India - Tamilnadu, Pondicherry and Kerala being first, second and third respectively.
Similarly for the search term "visual basic 6". We dont see much difference here.
Here I got little more curious about what my fellow mates are searching about vb projects and did a little more digging...
This graph shows interest over time for the search term "vb project". Again it outlines the time range for semester pattern followed - Jan to Apr and Jul to Nov - Indicated by crests and dips at May and Dec. The time series graph for the search term "projects in vb" has a very identical shape to this one.
I checked the interest of different cities in india for this "projects in vb" search term and found that Coimbatore is the winner at 100 and chennai coming after far margin at 18. My city, Pune, is at 3rd position.
According to google trends, these 5 are related searches for the term "projects in vb" -
1. Code project
2. Vb projects
3. Library Management System
4. Hotel Management System
5. Hospital Management System
India seems to be the only out of the 4 countries we are considering, googling for "vb project".
3) Windows XP
We use windows xp in many of the academic institutions in our country and the following graph reflects the same. Although there has been fast transition to newer operating systems.
The legends are same as followed in previous graphs.
4) Oracle 9i
Again, being an old software, the searches for oracle 9i are declining throughout the world. The part of the graph after 2005 is very similar to the graph of inverse function (y = 1/x).
Again, India is the leading googler for the search term "oracle 9i".
But there is silver lining too. We have moved on to the newer versions of oracle database and searches for oracle 9i have been significantly reduced. Almost at par with other countries after 2011.
5) Internet Explorer
Ok this is not an old software, still newer versions are coming but just for fun, here are the top countries searching for IE :P
And this last graph shows the comparison for the 4 countries we are considering... We can see surge in searches for the years in which major versions of IE were released. Although there are some other factors responsible for less number of searches, people's interest in IE seems to be waning! :P
Thank you for reading my article. I hope you found it informative. :)
My first semester is finished and we have a fortnight of preparation leave before the final exam. It was pretty busy, specially near the end of semester for completion of all the remaining stuff. During this semester we were supposed to make a management system project in vb6. Yes you read that right. VB6. As you might be aware, many engineering colleges in India are still using very old softwares written in the 1990's for their academics. Even though we were free to choose any latter versions of visual basic, most of the students preferred to go with vb6 for their project. This is because we were supposed to use that software for all our college assignments. Even the official support from microsoft for vb6 ended a few years ago.
This made me curious about the current usage of such old softwares like vb6, turbo c, oracle database 8/9 etc. So, since we're having some free time in hand during PLs, I did some analysis on google trends about these softwares (Or as they are called -Abandonwares- the softwares which are abandoned or rarely used). It came up with some interesting findings which I am going to share here with u all...
1) Turbo C
This screenshot shows how the usage of turbo c has been prominently declining, especially during 2005-2006. Since then it has been steadily going down. Below that, we can see the top countries which are googling for 'turbo c'. Philipines at the top followed by India and its neighbours. Striking fact is that Philipines and India are rapidly developing in IT related sectors.
This chart is for just another related search term "turbo c" (notice the space). As it shows, it is most googled for from India.
This graph shows comparative statistics for "turboc" search term from four different countries.
Developing countries-
1. India (blue)
2. Brazil (green)
Developed countries-
1. USA (red)
2. UK (yellow)
As you can see, the search volume from developed countries is negligibly low, and even for Brazil, it has considerably gone down.
This shows the results for the same search term "turboc" from different parts of India. As there are many engineering colleges in southern India, you can see those states leading this chart.
This graph shows the time series of the same search term for India. What I find interesting is that, it reflects the time range followed by the semester pattern in most engineering colleges in India. In January, the spring semester begins which is reflected by the rise you can see in Jan 2013. The semester usually ends in April and in may we have summer breaks. This is outlined by the dip in graph after Apr 2013. In June-July, the fall semester starts which is again indicated by the rise in searches for "turboc".
2) Visual Basic 6
The world seems to be steadily giving up on Visual Basic 6. Here what we have is almost linear line indicating how the searches for vb6 has been declined.
Coming to the comparative graph of the previous 4 countries. USA, UK and Brazil seems to have gave up on vb6 and moved on to more recent versions. While India is seen catching up very fastly. And again the south states are leading from India - Tamilnadu, Pondicherry and Kerala being first, second and third respectively.
Similarly for the search term "visual basic 6". We dont see much difference here.
Here I got little more curious about what my fellow mates are searching about vb projects and did a little more digging...
This graph shows interest over time for the search term "vb project". Again it outlines the time range for semester pattern followed - Jan to Apr and Jul to Nov - Indicated by crests and dips at May and Dec. The time series graph for the search term "projects in vb" has a very identical shape to this one.
I checked the interest of different cities in india for this "projects in vb" search term and found that Coimbatore is the winner at 100 and chennai coming after far margin at 18. My city, Pune, is at 3rd position.
According to google trends, these 5 are related searches for the term "projects in vb" -
1. Code project
2. Vb projects
3. Library Management System
4. Hotel Management System
5. Hospital Management System
India seems to be the only out of the 4 countries we are considering, googling for "vb project".
3) Windows XP
We use windows xp in many of the academic institutions in our country and the following graph reflects the same. Although there has been fast transition to newer operating systems.
The legends are same as followed in previous graphs.
4) Oracle 9i
Again, being an old software, the searches for oracle 9i are declining throughout the world. The part of the graph after 2005 is very similar to the graph of inverse function (y = 1/x).
Again, India is the leading googler for the search term "oracle 9i".
But there is silver lining too. We have moved on to the newer versions of oracle database and searches for oracle 9i have been significantly reduced. Almost at par with other countries after 2011.
5) Internet Explorer
Ok this is not an old software, still newer versions are coming but just for fun, here are the top countries searching for IE :P
And this last graph shows the comparison for the 4 countries we are considering... We can see surge in searches for the years in which major versions of IE were released. Although there are some other factors responsible for less number of searches, people's interest in IE seems to be waning! :P
Thank you for reading my article. I hope you found it informative. :)
Subscribe to:
Posts (Atom)







