Tot Attempting To Intercept Encrypted Ssh Connections

[Bill]

I regularly log into Unix-based computers in different locations around the globe from Thailand.

Today for the first time, I noticed warning messages when trying to log in to several computers.

After some investigation, I have determined that someone upstream of me -- most likely TOT as they are my provider -- is attempting to intercept ALL SSH connections on the standard port (22).

If you are a SSH user, you can verify this for yourself by noting that any attempts to connect to ANY computer via SSH will result in an answer -- even if the other end does not accept SSH connections. You can also see this by telnet-ing to port 22 on any IP address and you will see something like the following:

[Linux:~] root% telnet google.com 22

Trying 209.85.171.100...

Connected to google.com.

Escape character is '^]'.

SSH-2.0-OpenSSH_5.1

Also, if you have "strict checking" turned on (which you should), you will receive a warning message if you try to connect to a computer you have previously connected to with SSH. Do NOT ignore this message! Even if you are able to successfully log on, your password and the entire contents of your session could be recorded by your ISP and/or the Thai government. This is a serious invasion of privacy.

I have been reading about some new network monitoring devices being developed by Cisco and Juniper back in the US which are capable of these kinds of attacks (man-in-the-middle attacks). I am surprised to learn that this kind of technology may already be deployed here. Just to be certain, I also tried this on two different computers on my LAN and although they both run different versions of Linux on them, they exhibit identical behavior when I try to connect with SSH.

If you are a SSH user in need of a workaround, I suggest you configure it to listen on a different port in order to bypass this most egregious invasion of privacy.

[stumonster]

I thought the bluecoat proxy that is used by TRUE was capable of the cert based man in the middle attacks

http://en.wikipedia.org/wiki/Blue_Coat_Systems

[steffi]

Is that only possible when they can install a certiicate on your client machine?

Is there anyway they can perform a man in the middle attack without any software being installed on your own computer?

So is there anyway ToT can proxy the ssh connection in such a way that they they are reencrypting the data back to your client?

I thought the bluecoat proxy that is used by TRUE was capable of the cert based man in the middle attacks

http://en.wikipedia.org/wiki/Blue_Coat_Systems

[Bill]

Is that only possible when they can install a certiicate on your client machine?

Is there anyway they can perform a man in the middle attack without any software being installed on your own computer?

So is there anyway ToT can proxy the ssh connection in such a way that they they are reencrypting the data back to your client?

SSH caches the fingerprint of the keys of systems you have connected to in the past.

The default behavior is to notify you, the user, if the key changes.

If you accept the new key, then it will replace the old one on your computer. However, you should NOT accept a new key UNTIL you have verified with the owner of the machine you are connecting to that he changed the SSH keys. If he did not, then you may be the victim of a man-in-the-middle attack.

Once you accept the key, the man-in-the-middle attack can proceed in the future without any further notification to you -- very bad.

The man-in-the-middle attack requires no special software (trojan, virus, etc.) be installed in your computer. It is an attack that most end-to-end encryption software is subject to. Certificates (as used by SSL) are one way to protect against it (if you don't ignore certificate warnings). A "web of trust" is another way used by the popular encryption program PGP. SSH can use a variety of mechanisms, but the most common is the simple caching of previously-used keys because it requires no additional configuration by the user.

If you want to know the details for how a man-in-the-middle attack works, do a Google search and you will find many detailed descriptions.

If you are concerned that you may have been compromised by a man-in-the-middle attack, here are some steps you can take to mitigate any potential damage and reduce the likelihood that it will happen again in the future.

1. Edit the "known_hosts" file in your ssh directory (typically stored in .ssh subdir of your home directory under Linux). Remove any keys associated with the host you logged into as well as the IP address of that host.

2. Arrange for the password to be changed on the host you logged into. Don't use SSH to do this until you have secured it (see next steps).

3. If possible, arrange for the remote host to listen for SSH connections on a port other than port 22. Then connect with the "ssh -p <portno> ..." option specifying the new port number in place of <portno>.

4. If you cannot change the port number that SSH listens on (or add an additional port), then consider connecting first to a VPN service, and then connecting via SSH to any remote hosts you need to access. This will prevent your local ISP from performing a man-in-the-middle attack on you, assuming you configure your VPN so that all internet traffic is routed through it.

Don't ignore security warnings issued by SSH, or certificate warnings issued by your web browser. A lot of people just click their way through these warnings without giving them a second thought, and this is exactly how their private information can be compromised.

This is the last straw for me with ToT. I'm going to see about switching to TT&T next week. Hopefully they aren't doing the same thing.

[jackk]

I have TOT and I SSH a lot everyday to several servers and don't have any problem. I can't SSH to google or any other servers I don't own.

I think that if TOT wanted to sniff your traffic you would likely never know about it.

Maybe you should check your own computer for virus ? A friend of mine, who uses windows, asked me to find out how and why all his servers (4 of them) were hacked. They all had some nasty PHP scripts uploaded but he couldn't find how the hacker uploaded them. As I didn't find anything in its servers logs, I asked him to bring his own computer : he had a nasty windows virus that, each time he was uploading to his website via FTP, was also uploading all that PHP cr@p to his 4 severs

If it works with FTP, it could also work with SSH.

[surface]

what could they hope to get out of this if they aren't doing a man in the middle attack? this is the point of ssh, is it not? as long as you don't fall for a mitm attack, you shouldn't have anything to worry about. we all know the mict love to monitor internet traffic, so there really shouldn't be anything surprising here.

they might have noticed a lot of ssh traffic from you, perhaps going to hosts that have been known at some point to be serving anti-whatever web pages, and they decided to monitor your activities more closely. guess you'll find out next time you pass through immigration.

did you notice anything on port 23 or other common ports?

maybe it's time to start ssh'n though a vpn.

Edited April 25, 2009 by surface

[nikster]

Never attribute to malice that which can be explained by incompetence.... I don't know what would be causing this behavior on your side but ssh is pretty safe and pretty hard to pull off a man in the middle attack.

For one thing, if you use public key authentication then the server will encrypt his outgoing connection in a way that a man in the middle cannot read. Since your public key is deployed on the server by you, there is no way a man in the middle can modify it - it's already on the server. The MIM can't inject his own. At that point the MIM doesn't have a clue as to what the server is saying.

If you use password based authentication, and you accept the server's "new" key, then yeah, possible.

mithril-XL:~ nik$ telnet google.com 22
Trying 209.85.171.100...
telnet: connect to address 209.85.171.100: Operation timed out
Trying 74.125.45.100...
telnet: connect to address 74.125.45.100: Operation timed out
Trying 74.125.67.100...
telnet: connect to address 74.125.67.100: Operation timed out
telnet: Unable to connect to remote host

[nikster]

ps: https also prevents this kind of attack because the server certificate won't match with the records of the cert authority. I am not sure it's 100% safe from an attack by an ISP though - they could possibly spoof the cert authority response too. But if you were to intercept all https traffic you'd need a Google-size server farm at least

[Bill]

ps: https also prevents this kind of attack because the server certificate won't match with the records of the cert authority. I am not sure it's 100% safe from an attack by an ISP though - they could possibly spoof the cert authority response too. But if you were to intercept all https traffic you'd need a Google-size server farm at least

Https only prevents the attack if the users pay attention. Next time you're at an internet cafe, look over some people's shoulder's (from a respectful distance, of course). I think you'd be surprised how many people simply "click through" any warnings they get about mismatched certificates. This is exactly what would happen if they were subjected to a MITM attack, from an ISP or otherwise. At least most modern browsers make you click through several pages of warning instead of simply clicking on "ignore" as was the case in many older browsers.

From doing some google searching, it appears that several companies (including Cisco) offer hardware for performing MITM attacks against SSL encryption. I wasn't aware of any for use against SSH, which is why I was so surprised when it appeared to be happening to me.

[Bill]

Maybe you should check your own computer for virus ? A friend of mine, who uses windows, asked me to find out how and why all his servers (4 of them) were hacked. They all had some nasty PHP scripts uploaded but he couldn't find how the hacker uploaded them. As I didn't find anything in its servers logs, I asked him to bring his own computer : he had a nasty windows virus that, each time he was uploading to his website via FTP, was also uploading all that PHP cr@p to his 4 severs

If it works with FTP, it could also work with SSH.

Viruses or trojans are quite rare on linux platforms, especially when they are not being used as servers. With workstations, the user has to do something stupid in order to become infected by them. However, on the off chance that my computer may have been compromised, I tried a second Linux platform here that runs a different version of Linux. The behavior was identical, leading me to conclude that a virus is very unlikely to be the cause of my trouble.

[Bill]

Never attribute to malice that which can be explained by incompetence.... I don't know what would be causing this behavior on your side but ssh is pretty safe and pretty hard to pull off a man in the middle attack.

For one thing, if you use public key authentication then the server will encrypt his outgoing connection in a way that a man in the middle cannot read. Since your public key is deployed on the server by you, there is no way a man in the middle can modify it - it's already on the server. The MIM can't inject his own. At that point the MIM doesn't have a clue as to what the server is saying.

If you use password based authentication, and you accept the server's "new" key, then yeah, possible.

mithril-XL:~ nik$ telnet google.com 22
Trying 209.85.171.100...
telnet: connect to address 209.85.171.100: Operation timed out
Trying 74.125.45.100...
telnet: connect to address 74.125.45.100: Operation timed out
Trying 74.125.67.100...
telnet: connect to address 74.125.67.100: Operation timed out
telnet: Unable to connect to remote host

Given that purpose-built hardware already exists for accomplishing this with SSL-based encryption, it doesn't seem like it's much of a stretch to pull it off with SSH as well. When used with password-based authentication, which is the most commonly used type of authentication in my experience (probably because it requires no additional configuration), it is certainly vulnerable to a MITM attack. Using public-key, kerberos, or other types of authentication (SSH supports quite a few), prevents this type of attack, but requires that you have a secure channel by which to communicate the authentication credentials ahead of time. Still, this is good advice when feasible for yet another way to work around the problem.

As for my "problem," it is still with me this AM:

[Linux:~] root% telnet google.com 22
Trying 74.125.67.100...
Connected to google.com.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.1

[Linux:~] root% telnet thaivisa.com 22
Trying 203.174.85.146...
Connected to thaivisa.com.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.1

[nikster]

Well even password based is safe if you don't accept the "new" server SSH key. Did you accept a new key?

Is the FBI after you?

The first thing I would do is test the whole thing with a more trusted channel and make sure all your SSH keys are correct. That is, you connect via GPRS / DTAC, or if you don't trust the whole of Thailand, connect via VPN to a US server and make sure your SSH keys are correct. Once you are reasonably certain your SSH keys are indeed correct, proceed to use your TOT connection and _check_ if it's actually trying to give you an incorrect key.

If and only if that happens, I'd dare speculate on a MIM attack and grand conspiracy scheme. Otherwise, it could be anything. A MIM attack is not exactly the first thing I'd think of, personally.

Edited April 26, 2009 by nikster

[Bill]

Well even password based is safe if you don't accept the "new" server SSH key. Did you accept a new key?

Is the FBI after you?

The first thing I would do is test the whole thing with a more trusted channel and make sure all your SSH keys are correct. That is, you connect via GPRS / DTAC, or if you don't trust the whole of Thailand, connect via VPN to a US server and make sure your SSH keys are correct. Once you are reasonably certain your SSH keys are indeed correct, proceed to use your TOT connection and _check_ if it's actually trying to give you an incorrect key.

If and only if that happens, I'd dare speculate on a MIM attack and grand conspiracy scheme. Otherwise, it could be anything. A MIM attack is not exactly the first thing I'd think of, personally.

I accepted the new key, but at that point warning bells started to go off in my mind and did not complete the login. I subsequently removed the newly accepted key from my known_hosts files -- following the advice I posted above on how to recover from a MItM attack.

I believe my SSH server's keys in my known_hosts file are fine. I have spot checked several with a secondary channel and they matched.

I stand by what I originally said that ToT appears to be conducting a MItM attack against me. It could also be a "misconfigured" router at ToT, which routes all accesses to port 22 of any host to be redirected to a particular server (to what end?). That would already be very suspicious. I have no idea how widespread this attack or "misconfiguration" is, which is precisely why I mentioned it here so that other people with ToT could check for themselves and react appropriately.

By the way, there is an interesting article on a successful MItM attack against SSH posted at

http://xperts.sce.carleton.ca/2004-05/MITMV/

[g00dgirl]

I also noticed various problems with secure connections in the past week or so! On TT&T Maxnet. I could not establish SSH connections on port 22 with putty to Unix servers, sometimes I also got the warning about the key being unknown, unfortunately I ignored it because I had just also rebuilt my system so no keys were cached locally and it would be usual to get that message in this case.

I also could not connect to Plesk via https on port 8443, the page would just not load.

I also could not connect to several instant messaging services like icq and jabber.

Many strange things happened in the past week, but since yesterday everything seems to be back to normal.

[Crushdepth]

From doing some google searching, it appears that several companies (including Cisco) offer hardware for performing MITM attacks against SSL encryption.

I don't think anyone can break SSL encryption at present. However, there are a couple of ways people can work *around* SSL:

* If someone installs their own Certificate Authority in your browser, then they can generate their own certificates for SSL sites and your browser will accept them as valid. If your connection passes through their proxy then they can also transparently decrypt your traffic without triggering any warnings. This is *often* done in workplaces that are concerned about employees leaking data etc.

* If a site you are connecting to uses a self-signed certificate (ie. not issued by a recognised CA) then you don't really have any way to tell whether its the 'real' site or not without some other means of verification.

* A research group demonstrated a successful attack against SSL certificates signed by MD5 hash a few months ago. They managed to forge a valid certificate from a mainstream CA and to designate it as a CA in its own right, allowing them to create their own valid certificates for pretty much any SSL site. The attack was only effective against CAs that had been particularly stupid and predictable about the way they generated their certificates, allowing the researchers to anticipate nearly all the content of the certificate. Pretty much all the CAs dumped MD5 signatures straight away after that anyway.

[Linux:~] root% telnet google.com 22

Trying 74.125.67.100...

Connected to google.com.

Escape character is '^]'.

SSH-2.0-OpenSSH_5.1

[Linux:~] root% telnet thaivisa.com 22

Trying 203.174.85.146...

Connected to thaivisa.com.

Escape character is '^]'.

SSH-2.0-OpenSSH_5.1

So *any* SSH/telnet connection you open is apparently 'accepted', even if its to some bullshit site you just made up that probably doesn't even have SSH? That's weird. Can you see the IP of the site 'accepting' the connection? That might be interesting.

Edited April 26, 2009 by Crushdepth

[Crushdepth]

Just a thought, maybe you are a victim of DNS poisoning. Suggest you try changing to OpenDNS (208.67.222.222 and 208.67.220.220):

* A lot of Thai DNS servers never got patched against that major DNS problem that was found a while back.

* Some ISPs definitely implement selective DNS poisoning against sites deemed to be 'illegal'.

* There is also a recent virus that makes infected computers behave as bogus DHCP servers - they will hand out the addresses of poisoned DNS servers to any computer that asks for IPs. Your own (Linux) computer doesn't need to be infected to be a victim - there just needs to be an infected computer somewhere on your network. Specifying some trusted DNS servers (OpenDNS) will stop this.

* Prasert recently showed that it is possible to intercept ADSL traffic on TOT because their routers are hopelessly overloaded and slow to respond. Wouldn't suprise me if this is being exploited by bad people.

[g00dgirl]

The problems have started again today, emails over secure ports are not going out, nwo they even found a way to block VPN connections, 3 out of 3 VPN servers in different countries, I cant establish a connection to either one anymore!

I also found this article from last year

http://www.zdnetasia.com/news/internet/0,3...62042297,00.htm

Net lags suggest Thai govt monitoring traffic

By Don Sambandaraksa, Bangkok Post

Friday, June 06, 2008 10:43 AM

Thailand's Internet is stuttering with a series of unexplained outages and slowdowns that suggest that the government is running a far-reaching program to monitor its citizens' online activities, one similar to the U.S. Carnivore e-mail policeware program.

This can be seen in the way YouTube is now all but unusable for TOT subscribers, and how sending large e-mail messages through a foreign server on port 25 often fails, while encrypted, non-standard ports or VPN access over the same network works fine.

A former security-consultant-turned-businessman in Thailand, speaking on condition of anonymity, said that the entire situation was seriously damaging business confidence and may be on the verge of being illegal, especially for foreign businesses operating in Thailand.

In many industries, a user has a duty to alert his company or his authorities if he knows that data has been compromised. But what if the leak is to a foreign government? That said, in most countries, national security laws override privacy laws.

There are legal ramifications on contract law, especially if the businessman is doing business with the Thai government or military.

"If you are going to do it, do it properly," the former security consultant suggested.

China and Singapore, for instance, use a monitoring software package from a company called Xacct Technologies from Israel which is capable of far more than e-mail logging and can scale better than the system that he believes is in place now. Another packet monitoring system that is in use and can scale well is Phorm.

He first noted earlier this year that e-mail sending slows down as the work day progresses and fails almost entirely around lunchtime. His company uses a corporate e-mail server located overseas. Analysis of the traffic suggest that the authorities are intercepting anything on standard SMTP port 25, regardless of the destination IP address. He said he has the IP numbers he suspects to be the sniffing machines as the latency incurred there is far too long to be a regular switch.

To circumvent this monitoring, users can simply use a VPN to access their corporate network overseas, use SSL encrypted e-mail ports or even encrypt on an end-to-end fashion. Gmail remains secure when accessed via HTTPS, although he did question what went on behind the scenes when Thailand lifted the ban on Google's YouTube and if any agreements had been struck.

That said, the former security consultant said that there was a legitimate need for governments to monitor e-mail for national security but that the way the Thai government had done it had failed miserably.

"How hard is the system to circumvent? I have to circumvent it as otherwise I have no way of getting my e-mail, by going to http://www.mail2pda.com or using HTTPS for Gmail. Then what's the point? Already the government has lost the ability to gather intelligence."

Rather it should have been done professionally and be totally invisible without the terrorists or the public knowing.

From the business point of view, it was just another in a long list of questionable decisions "that makes Thailand look positively Mickey Mouse", he said.

Rather than announce to the world that they were intercepting and monitoring e-mail and thus making everyone use encryption, it would have been much better if the government had kept quiet and had done traffic pattern analysis on individuals to learn more about their network, he said.

For instance, if one person was using encryption all the time, the government should keep a close eye on him and who he contacts, but by botching up this project, it means that everyone has to use encryption and VPN and thus the government has lost its ability to gather information and protect the people.

The consultant also questioned the legality of the recent Thai cybercrime law in the context of European privacy laws.

In a seminar soon after the law was passed, police said that as long as a server host or ISP could provide a name to an activity, that would satisfy the 90-day log retention requirements.

However, according to European privacy law, putting a name to an activity requires user consent. Quite how this would affect Europeans doing business here, or Thais doing business in the EU, was still unclear, he said.

Another question was who had access to the information being gathered. In the past, the US government launched Echelon and Carnivore, projects aimed at wiretapping the Internet in the name of security. But at least they had clear objectives and responsibilities, unlike the clandestine Thai system that appeared to be in place, he noted.

The consultant saw three possible scenarios: "If the reason is to spy by the government, then you (a journalist) are at great risk personally. If it was decided it was important to monitor all e-mail and someone in government screwed up, this is fine and people should be happy. But if the reason is commercial and you are exposing someone's email to somebody in the government, this is bad," he said. Finally, he had a piece of advice he would like to pass on to the government: "The effort involved in censorship is far greater than the effort required to monitor, and monitoring gives you more information. Censorship blocks, monitoring gains intelligence."

"In Europe, authorities infiltrated a major paedophile ring. They worked for 18 months, infiltrating and learning about the network, which operated in cells." Many of the operatives are still in therapy. But by taking that route, they were able to expand their reach through the cells and gather evidence they could use in court. In the end, they arrested hundreds of people and more importantly, they pulled 26 children out of that network who were being abused.

"If they had taken the approach that this server has paedophilia and it has to be shut down, they might cut off access to 10 paedophiles but they would not have saved any children," he said.

[Prasert]

I can't reproduce the problem, but I am very interested in learning where this attack is done. If the OP is willing, contact me! I will provide a userid and password on a ssh server in Europe, so the OP can login. I can then determine where the connection is actually coming from. After this, I'll remove the account so no further harm can be done.

Redirecting traffic is very simple on routers like Cisco and Juniper. A list is configured to match specific traffic, e.g. traffic from any host to any host on tcp port 80 (which is a website).

The router then changes the next-hop of all sessions that match this list to a different IP address, usually the IP address of a server where the monitoring is done. This server accepts the session, and proxies it to the original destination.

Squid is a transparent proxy that works this way. The end-user cannot circumvent this.

The webserver however, sees a connection coming from this proxy server. In the same way, I can determine where the connection is coming from.

[Bill]

This morning, I've noticed that the problem has disappeared. All of my SSH connections appear to be working normally now.

[Linux:~] root% telnet google.com 22
Trying 74.125.45.100...
telnet: connect to address 74.125.45.100: Operation timed out
Trying 74.125.67.100...
telnet: connect to address 74.125.67.100: Operation timed out
Trying 209.85.171.100...
telnet: connect to address 209.85.171.100: Operation timed out
telnet: Unable to connect to remote host

For me, this whole experience has served to remind me of the importance of not ignoring key fingerprint or certificate warnings EVER. While the man-in-the-middle attack might have been an academic curiosity a few years ago, I'm concerned that the availability of of-the-shelf hardware for implementing it will lead to widespread usage around the world by unscrupulous ISPs, employers, and of course governments which see threats to their power everywhere.

[nikster]

By the way, there is an interesting article on a successful MItM attack against SSH posted at

http://xperts.sce.carleton.ca/2004-05/MITMV/

Interesting, I guess the main idea in all of this is that twisting DNS lookups is much easier than breaking the encryption. I don't think this is the end of DNS vulnerabilities - it looks like that that is a major issue with internet security. And an ISP can spoof DNS towards its clients quasi per default. If the ISP decides to spoof DNSs, then in theory I also don't think OpenDNS would provide a way around it. The ISP could at a lower level tweak IP addresses to its own liking as well. What if all access to the OpenDNS IP addresses is in reality redirected to a server under ISP control?

As for the paper, I don't really understand how they propose to get around public key authentication. In the systems I have used, I had to manually upload my own public key to the server and enter it in the right place. The act of uploading the key is vulnerable to a MIM attack, but after it's been established, it's safe. The server didn't even offer password authentication. There is no way to get a public key to the server automatically. That means the MIM server can't get their own public key to the serve => no MIM attack.

In addition, the whole reason for the warning in the ssh client when the server has a new key is the man in the middle attack. The designers of ssh and public key encryption systems in general were absolutely aware of the problem, and that is their solution. I am pretty sure secret services and other organizations that really need strong security would hand-deliver paper copies of the server key checksum.

As for why telnet 22 acts weirdly... this is TOT, and Thailand. Which is likelier: A grand conspiracy theory, or somebody trying some new software configuration on the live production network?

[nikster]

For me, this whole experience has served to remind me of the importance of not ignoring key fingerprint or certificate warnings EVER. While the man-in-the-middle attack might have been an academic curiosity a few years ago, I'm concerned that the availability of of-the-shelf hardware for implementing it will lead to widespread usage around the world by unscrupulous ISPs, employers, and of course governments which see threats to their power everywhere.

Word.

The warnings are there for a reason.