On The Internet, Nobody Knows If You’re A Dog - An Interview with Christian Folini
Interview with Christian Folini:
Cyber Security Dispatch: Season 3, Episode 1
Show Notes:
Today on the show we welcome with Christian Folini of netnea. Christian is the author of The ModSecurity Handbook (2nd edition) and Co-Lead of the OWASP ModSecurity Core Rule Set project. He is also the program chair of the Swiss Cyber Storm conference, Vice-President of Swiss Cyber Experts and a partner and consultant at netnea.com. In this episode, we discuss Christian’s interesting path from PhD in Medieval History to becoming an expert in computer science. He also shares his fascinating work with the Swiss voting system and how E-voting is alive and happening in that country today. Will a system like this ever be possible in the US? Find that here too. We delve into some interesting discussions drawing parallels between Medieval social history and the what is happening with the internet today, in terms of open source technology. He also explains (in a way your grandparents will even understand) the usage of firewalls, whitelisting, blacklisting, IP addresses, and malicious/non-malicious traffic and how all these things works. We also hear more about his strategies for reverse proxy and stopping D-DOS. A background in humanities has really served Christian well in the art of explanation, making this episode full of great imagery, good humor and information that even the dog next door might appreciate. Please note this episode was recorded in February 2018, prior to various developments in Swiss online voting which took place later in the year.
Key Points From This Episode:
How Christian came to study both Medieval History and Computer Science.
Learn more about Christian’s unique PhD in German Mysticism.
Christian shares his unique passion for global cyber security theory.
Are their links between Medieval history and what is happening with the internet today?
Discover more about the balkanization of the internet and net neutrality.
Parallels between Medieval social connections and internet social connections.
Christian’s view on open source and how the ModSecurity Project fits into that.
Christian explains how a firewall works and the two main types of firewall.
Top five things that might make traffic look malicious or none malicious.
Whitelisting, blacklisting and IP addresses: Can they really be trusted?
E-voting: Why Switzerland is going all in while the rest of the world backs out.
Is it possible to fully secure identification in an E-voting system?
Why the world appears to be falling back on a physical verification process.
Christian walks us through what an E-voting process looks like.
Learn more about Christian’s strategies for reverse proxy and D-DOS.
And much more!
Links Mentioned in Today’s Episode:
ModSecurity – https://www.modsecurity.org/
Christian Folini on LinkedIn – https://www.linkedin.com/in/christian-folini-588ba278/
Christian Folini on Twitter – https://twitter.com/ChrFolini
ModSecurity Handbook – https://www.amazon.com/ModSecurity-Handbook-Second-Christian-Folini/dp/1907117075
Unix – http://www.unix.org/what_is_unix.html
OWASP – https://www.owasp.org/index.php/Main_Page
Apache – https://httpd.apache.org/
ModSecurity Core Rule Set – https://www.owasp.org/index.php/Category:OWASP_ModSecurity_Core_Rule_Set_Project
Ivan Ristic – https://www.linkedin.com/in/ivanr/
SSL Labs – https://www.ssllabs.com/
SQL (Programming Language) Union Keyword – https://www.w3schools.com/sql/sql_union.asp
Trustwave – https://www.trustwave.com/home/
Enigma Conference – https://www.usenix.org/conference/enigma2016
E-voting via Swiss Post – https://www.evoting.ch/en
Facebook Postcard Ad Verification – https://www.reuters.com/article/us-usa-election-facebook/facebook-plans-to-use-u-s-mail-to-verify-ids-of-election-ad-buyers-idUSKCN1G10VD
Introduction:
Welcome to another edition of Cyber Security Dispatch, this is your host Andy Anderson. In this episode, On The Internet, Nobody Knows If You’re A Dog, we talk with Christian Folini of netnea. We discuss his interesting path from a PhD in Medieval History to an expert in computer science. He shares his work with the Swiss voting system and how E-voting is alive and happening in that country today.
TRANSCRIPT
[0:00:32.5] Andy Anderson: I was reading through your background and you have an interesting background and being a history major myself, I kind of want to start there just to hear about your interest in medieval history and how that maybe led you to this space?
[0:00:46.9] Christian Folini: Okay, thank you. Yes, I’ve always been interested in computers and in history. When I started to go to university, I kind of tried to get into computer science and use history as a second branch. They wouldn’t allow that. They told me to do math or physics as a second branch. I was like “No, I’m not interested in that.”
Then I asked the history guys, “Would you allow computer science as a secondary branch” and they said “Yeah, absolutely, go for it. Get as far away from history as possible because it opens your horizon. That’s very interesting for us because we have so many English literature, German literature people around, let’s have a change of pace.” That’s what I did.
Actually, I did very little computer stuff at university, it was just the secondary branch and then I really specialized into medieval history. I continued with a PhD on Dominican mysticism or German mysticism, which is kind of a hardcore social history, church history, or theology, say around 1300. That is a fairly specialized field I would say.
[0:02:02.0] AA: Yeah.
[0:02:03.8] CF: Yeah, very interesting, probably not in this podcast and during the PhD, I really grew tired of that and I returned to my initial interest in computer stuff. I told myself - every day, I tried to run away from my PHD and do computer stuff - well, in the long run, you’ll be happy to get more computer experience, this might save your job after you finish your ivory tower PhD in Medieval History.” That’s how it worked out.
I applied in a museum or two and I didn’t get the job but then I slipped into a small computer company - a local computer company - which did Unix consulting and I stayed with them ever since. It’s been 15 years with [inaudible] and my partner now, we’re five guys running the company and we do network monitoring and security.
During the period, I more and more specialized in security. I used to do Unix system administration as a contractor and then I got exposed to ModSecurity early on, the module for a patch - this is a web application fiber module which started in about 2002 or 2003. By 2005, I ran my first Apache website with ModSecurity enabled.
I got hooked immediately. I stayed with that technology. Technology wise, I’m very deep into web server, web application security and as time went on, I started to look outside of this narrow technological field and started to talk to people about the theory of security, or what they think about the industry; how does the security industry work locally here in Switzerland where I live; started to build up a network and eventually, I got to be one of the co leads of the ModSecurity Core Rule Set Project, which is hosted under the umbrella of OWASP, the open web application security project. It’s now more a moderating role. Not so much technology anymore down to the screen but more working with people and I kind of see, that’s very interesting as well. Now I work deep deep in technology half of my work time and to the rest of the time is organizing conferences, talking to people, writing blog posts, stuff like that.
[0:04:45.6] AA: Yeah.
[0:04:47.3] CF: That’s how I got here.
[0:04:51.3] AA: Yeah, so many interesting things that I want to ask about. I mean, I think, just going back, starting kind of with the medieval history background. I think that’s such an interesting time in kind of European history because you sort of have the fall of these great empires and the beginning of like formation of some nation states and different fiefdoms and sort of control of you know, different geographic regions and individuals.
I mean, if you sort of changed some of the words out, it sounds a little bit like what’s happening in the internet, you know? Different kind of controls of spheres and you know, there are some common kind of languages. I mean, the church is sort of active, right? Connecting people to some degree, right?
You know, I certainly see the parallels, at least on the sort of grand scale, are you finding that those sort of stories are running in the assessment of how things are working?
[0:05:51.3] CF: That is an interesting thought. I think you can make the argument that early and middle ages were somehow organized in a more global way when you talk about big empires and then breaking down into smaller fiefdoms - that will be something you could relate to a global free internet and then breaking down into smaller areas.
We’ve called this the balkanization of the internet, so where you wouldn’t longer be able to access all of the internet. If we talk about net neutrality - net neutrality I think plays into it as well. I think seeing big parallels between older history and internet development - that's a bold thesis , yeah. But there are parallels perhaps.
[0:06:45.6] AA: Yeah, that’s true.
[0:06:47.2] CF: What I found interesting in the work of the medieval history, it is a period where like the lay man or the small people, the non-nobles, become much more important - it’s around 1200, 1300, 1400’s - the cities are getting a lot of power and they’re playing the role politically now and especially within the church, there are now new monasteries built into the cities. Monasteries used to be a noble thing out in the wilderness for important people.
Now, suddenly, you have sons and daughters of people on city who found monasteries in groups and this is a movement which explodes - it really explodes within 50 years, you have 10 times as many religious people like monks in monastery, then you used to have 50 years before.
That’s a huge movement sweeping all over Europe. I used to look at the social life and the social connections within these groups. I think that is very interesting and that working with people and power of people, it’s a grassroots movement and that’s something which are still very interesting.
When people get together and build something by themselves. I see this in history on and again, I think you can see internet and technology as something similar to that, or the open source software is often a bottom up thing.
[0:08:28.3] AA: Yeah, I mean, I definitely sort of, I’d love to kind of you know, transition and hear – OWASP is such an interesting sort of organization and the sort of general open source community. You know, for many people who are kind of outside of the computer industry, they see the sort of giant companies, you know, almost bordering on monopolistic control and you know - they may not appreciate the power and the history of the sort of open source within that industry. How that influences the adoption of technology and sort of the power of different groups and companies within it. I’d love to kind of get just your perspective on OWASP and how maybe the ModSecurity Project fits into that.
[0:09:20.6] CF: Yeah, OWASP is a huge organization with so many projects. In the core it boils down to maybe a dozen of flagship projects which are really actively maintained and our OWASP ModSecurity Core Rule Set is one of these.
As you just mentioned, it’s an interesting thought or a way to approach this the way you did it because this Core Rule Set Project is a rule set for Web Application Firewalls and web application firewalls, it’s a highly commercial competitive market.
You have big security companies doing a lot of research and development to produce very expensive products which they market globally or locally - it really depends a bit - and there is like a single open source offering which attempts also a multipurpose or a general WAF.
This is ModSecurity which is in its core, an Apache module but it also runs on Nginx and IIS (Internet Information Services) ; this is the engine and this was developed by Ivan Ristic who later founded the SSL (Secure Sockets Layer) Labs. Then this was acquired, by an Israeli company and later traded over to Trustwave which continues it as an open source project, but that’s only the engine, then you need to write the rules and they don’t seem to be too much interested in the rules themselves.
There is an open source project outside of Trustwave who works on these rules and this runs on their OWASP. We really have a community here of people who say, “We don’t want to rely on the commercial or proprietary offering for these rules.” - probably a black box and I think the commercial box now is quite black box in their nature; but, this ModSecurity thing is completely transparent and these rules really allow us to control exactly which traffic enters our web application and which doesn’t. As it happens, when you run a security field like that which is actually fairly complex in its ruleset, then you get false positives.
[0:11:46.0] AA: Yeah.
[0:11:47.8] CF: You have legitimate traffic which is suddenly blocked by your rules and being completely transparent and open in nature, the cogwheel set of OWASP allows you to really see why a request was blocked and they have to necessary tools to allow it in the future.
This is a lot harder when you were on a commercial product which is not so upfront about the rules which are actually hidden, and it gets a lot worse when you do machine learning and artificial intelligence because then the machine decides on its own and doesn’t really tell you why it made decisions.
[0:12:22.6] AA: Right. You’re truly getting into that sort of Kafkaesque kind of world of you know, “Why is it doing it?” - “Well, we have no idea, talk to someone else,” right? You just you know –
[0:12:34.7] CF: Yeah, it learns this by itself.
[0:12:36.6] AA: Right, truly. I mean, the implications of AI and machine learning are you know, incredibly positive but also quite scary. I mean, I think all of us has seen the terminator movies, right? A lot of people just go right to there. I think the other sort of potential scary scenarios that you think about are manifold, right? Just getting a driver’s license or you know, accessing your email or any of those sorts of things as you increasingly hand control to machines who the people who built them don’t even know how they work anymore. It’s frightening.
[0:13:17.8] CF: Yeah, absolutely.
[0:13:17.5] AA: You know, for someone, you know, our audience is kind of spans the range form like people who have been decades in the space to people who are newer to the world of security. I think it might be helpful just to take sort of 30 seconds and explain kind of how a firewall works or the different flavors of firewall, because it feels like it’s one of those terms that gets thrown around a lot but if you asked, you know, 10 people to define it, you might get 10 different answers.
“You want to be as secure as possible but you do not want to lock out any legitimate users. On the other hand, you also do not want to let an attacker pass. You need to tell the two apart and that’s the tricky bit because the attacker will try to look innocent while a benign traffic can look really dangerous just because or because the developers of the application didn’t really think of security, or they just didn’t think of your funny ruleset who makes assumptions about benign and malicious.”
[0:13:50.1] CF: I hope not but yes, a traditional firewall is a network firewall. It comes with – in a simple principle, fairly simple rule set which says, we block everything and then we open a web traffic. We open the HTTP ports to allow that traffic, that’s fairly simply if you leave it at that.
A web application firewall is much closer to the web server or the browser if you want. The traffic which runs between the browser and the web server, the application server is being inspected. Of course that’s really complicated traffic, that’s a lot of stuff.
Then, you need to make a decision if the given request - so this HTTP web thing is request based - if a given a request is it benign or malicious? And you have a little bit of context and then you make your decision, or the ruleset has to make that decision.
You want to be as secure as possible but you do not want to lock out any legitimate users. On the other hand, you also do not want to let an attacker pass. You need to tell the two apart and that’s the tricky bit because the attacker will try to look innocent while a benign traffic can look really dangerous just because or because the developers of the application didn’t really think of security, or they just didn’t think of your funny ruleset who makes assumptions about benign and malicious.
This can get really hairy in details well obviously it can. But, on the other side, the core rulesets is now in a state where we would say, default installations usually just work. If you have false positives - so legitimate request being blocked - that is rare and in between and then we have to do the documentation to help you up, what to do about it.
[0:15:58.5] AA: For someone who is not as sort of familiar with thinking through that ruleset as you are, you know, what are the kind of top three to five things that might make traffic look malicious or non-malicious. You know, obviously, the domain from where that traffic is originating, I know is one that often get cited, but walk us through kind of a few of those kind of core ones?
[0:16:23.6] CF: We look at in detail at parameters being transferred. When you fill out the forum being the log in forum, a registration form - anything - you send to the server, then we look at the individual arguments and try to make a decision.
We just had a fault where somebody was living at Union Street, there’s nothing wrong about living at Union Street but as it happens, Union is an SQL keyword. It seems we had a poor rule which would alert on keyboard Union which happens to be a standard English word. This caused – it wasn’t the keyboard alone but it was the combination of a number and then Union Street and all combined, it made for a rare but it made for a false positive in the default installation. The rule was not good enough and we had to update this rule. I think that’s a typical thing. We try to identify SQL statements because that could be an SQL injection attack, these are really widespread on the internet. We try to identify these and then at times, we’re a bit too aggressive.
[0:17:37.5] AA: Yeah, many fold, right? Where the traffic’s coming from potentially, are they essentially trying to put code where they should be just putting like text, like an address or a name or whatnot, right? How do you think about one of the things that was down this weekend kind of talking with a lot of the election commissioners in the US, because if you’re reading any US papers, right, sort of top of mind right now is potential election hacking.
They were thinking about you know, challenges of how they’re denying traffic and you know, “Should we just be whitelisting access or blacklisting and how do we think about Americans living…?” You know, typically we’d just say like any traffic coming from outside the US, we’re just going to block, but then, we have troops or individuals living overseas and they’re actually trying to access our systems, so they can vote absentee from Europe, from all over the place. How do you sort of think about - in this world where you are - I mean, just the explosion of kind of IP addresses and sort of –
Also, the tools to make people maybe not look like where they are versus where they might actually be sitting. You know, sort of VPN and proxy services and all those sorts of things.
[0:18:49.5] CF: Well I think, you cannot really trust an IP address on key location but then, when you are under attack, then maybe you don’t really care too much anymore about locking out people. I think that is especially the case with denial of service attacks. When everybody’s firing at you, then you stop caring about individual IP addresses no longer being allowed to connect to you because you want to keep up the majority of users and that usually is domestic users.
I’m living in Switzerland, really a small country. Locking out the rest of the world does 90% of the job under an attack. Foreign internationals will no longer be able to connect, which is a pity but during the attack, Swiss companies usually accept that.
[0:19:37.4] AA: Yeah.
[0:19:39.8] CF: Swiss traffic is usually something you can handle. I think that must be much more difficult if you’re living in a big country and locking out non-US traffic kind of split maybe half of the IP addresses. I don’t think that’s very helpful and as you said - people do Tor nowadays, VPN.
You cannot really trust an IP address. Generally, wouldn’t trust them, the Geo location too much. On a more general level, you mentioned whitelisting and blacklisting. This ruleset I mentioned is a blacklist on that. It tries to be generic in nature, but it tries to identify bad things.
At turning it around and coming up with a white listing rule set and this is what the network firewall usually does as you mentioned before. Everything is blocked and then we allow two or three things. That is much more secure.
[0:20:35.9] AA: Yeah.
[0:20:37.6] CF: That’s generally, I advise people to do that and rulesets like that as well. It’s just very difficult to do with web traffic because that’s so complicated. Then, as it happens, it’s just interesting you mentioned E-voting. I’m actually working on a high secure E-voting project. E-voting is very hot topic in Switzerland and while the rest of the world seems to step out of E-voting right now and pulling back, Switzerland is moving all in.
Our politicians say, “We have tested this for 15 years and now let’s really do this because we’re Switzerland and we know how to do secure stuff.” In the Swiss context, E-voting means online voting. You vote from home over the internet. That is something which sounds dangerous for people outside Switzerland but actually, we do voting by mail on paper in Switzerland.
Like 90% of people fill out a form themselves then put them in an envelope and send it by post. That’s how voting works in Switzerland. Doing this electronically is not such a big step. Filling it out at home is already accepted. In this E-voting context, whitelisting seems to be a very good approach because voting is not terribly complicated from the applications standpoint.
I mean, you select a few candidates, don’t you? This is something where you can actually do whitelisting. Whereas running a full blown web shop or an online forum with whitelisting is really hard. I think you need to come up, make up your mind how many resources you want to spend on a security and then try to get as much security out of these resources as possible.
This can mean, yeah: here with your whitelisting; over there, we buy your product; and over there, source a service or we use an open source product which brings this.
[0:22:44.2] AA: Yeah, there’s no one single solution, right? It’s really putting together a number of them.
[0:22:49.1] CF: Absolutely. I think we’re past the point where you buy a product and you hope it solves your problems; it doesn’t.
[0:22:57.8] AA: No.
[0:22:58.7] CF: With knowledge, experience and the people who really have an idea of how they want to secure something and then the source the right tools.
[0:23:08.4] AA: Yeah there’s a lot there. I would love to kind of just circle back to E-voting because it’s something that I’ve just – we’ve just been spending a ton of time thinking about it, talking about, et cetera. One of the challenges I think in the US - I mean, there’s two. We have a decent amount of absentee E-voting as well - essentially vote by mail, that’s where we call it -and some states, you know, they’re seeing more and more of that happening.
I don’t know the exact numbers, but you know, 30, 40, 50% potentially isn’t unrealistic. One of the things – essentially the identification step there is that it’s somewhat tied to an address, right? We’re going to send this to your home, right? Not impossible to essentially –
The concern that we have is that you know, you’re essentially like, the person who is voting is not the actual person, right? You’re having sort of fake people vote, or you’re creating individuals, right? Now, with addresses, you know, you start to see, well, it just becomes sort of logistically challenging, right? Because you got to somehow intercept those, either you control a lot of PO Boxes although sometimes I don’t know the rules whether you can send it to a PO Box, but you now have to just control a lot of addresses and be able to accept mail - which is not impossible but just logistically painful, right? It’s just challenging and potentially expensive. Then, as you think about like essentially internet voting online, you know, the ability to extenuate - accept a ballot. How do you kind of think about that step, essentially, the identification step of people in an online world and making sure that that’s kind of secure and I apologize if the question is not clear, that’s my fault, not yours.
“ I think identification and trusting somebody via the internet, that’s one of the core issues, isn’t it? In the virtual world, how is that joke: “On the internet, nobody knows if you’re a dog.””
[0:25:01.1] CF: Okay, I think I can make sense of it. It’s true, it wasn’t exactly clear for me but let me respond and then you can follow up.
[0:25:09.5] AA: That’s good.
[0:25:10.6] CF: I think identification and trusting somebody via the internet, that’s one of the core issues, isn’t it? In the virtual world, how is that joke: “On the internet, nobody knows if you’re a dog.”
[0:25:21.2] AA: Right, of course.
[0:25:24.1] CF: I think that falls down to E-voting as well. I think as long as it is tied to a paper ballot which somebody sends home to you, an attacker would need to steal the paper ballots and actually, that is happening in Switzerland and it’s accepted. Every couple of years, something bad happens and then there is a scandal - it’s always a local scandal.
Right now, we have one running where somebody stole like 70 paper ballots and filled them out himself. Faked the signature and they finally – no, that’s alway the same hand writing signatures. He’s likely to face jail time.
[0:26:09.2] AA: Yup.
[0:26:11.1] CF: That is the weakness of the system which is accepted in Switzerland and I mean, 70 votes - okay, that doesn’t usually turn in the election. As long as the E-voting is tied to this - may letter be of sent to somebody at home. It will probably, the limited to the same amount of fraud. Things get extremely different when we get rid of the paper -
[0:26:40.7] AA: Yup.
[0:26:40.9] CF: - and send it via email or any means of electronic transfer and where we no longer identify somebody based on credentials printed on paper but credentials sent via any sort of text message. Switzerland hasn’t done that step for very good reasons; then things, fraud is getting much more easy to do and you can pull this off: the stealing of ballots on the large scale. Well, it’s now on paper, you have a logistical problem. You would have to go to Swiss post and try to steal letters - an essential logistic platform. That’s kind of unlikely to pull off. People would notice when – if they don’t get their paper ballot then they notice hopefully.
A few of them might not, if you go into the hundreds, somebody will complain.
[0:27:40.6] AA: I mean, I was hoping you had a solution for me, I was hoping you can get to figure it out, sort of where we are.
[0:27:49.0] CF: No, I’m not solving US voting problems unfortunately.
[0:27:53.3] AA: You know, we got time.
[0:27:56.9] CF: Okay, it’s only a couple of years out. I think there is very good research in the states. I mean, the scientists, they know all about it, politicians just need to start to pay attention and really do what – I get a feeling all the scientists or scholars are agreeing what to do. Just the politicians need to accept that now and get to work and bring up the resources or the funds to finance these more secure systems.
The whole question of paper ballot, yes, because if it gets physical then it’s hard to do fraud on a big scale. Didn’t Facebook just announce that they will verify political ads via a mail postcard?
[0:28:45.6] AA: Yeah, that was right.
[0:28:46.5] CF: Isn’t that, we were giving up on identifying people in the online world because it’s too hard, too difficult, and we’re falling back on a physical verification process; because this is known to work and it has been working for hundreds of years. That’s why we are bringing it back for really hard problems and for voting, I wouldn’t trade this off. I think we need to keep this because here democracy is at stake and that’s really risky.
I actually don’t see definite solutions to identifying people in the online world ruling out any possibility of fraud. I think we can’t. We haven’t found a system that really works.
[0:29:33.5] AA: Yeah, I think it is essentially the second form. In some ways it is like multifactor authentication right? That is your second form is that mailed postcard and we know that the post is not a 100% secure, but again as you mentioned the scale and the logistical challenge of doing it at a scale where it would be statistically significant that you could impact an election in a way that would change the outcome, is it possible?
I mean we had a small runoff in the state here which was literally a tie - like zero vote difference and so in that scene it can’t happen but it just becomes challenging. The second piece I guess that I would be curious about on the E-voting front is thinking about – so there is the identification step but then there is also like that the vote you cast is actually that there is nothing happening in that process itself whether that was sort of a man in the middle attack.
How do you assure in the US at least most of the E-voting systems as they go into an online world, where actually again paper is figuring quite prominently in that process. So you are optically scanning the ballots, so you have essentially duplicate records, one on paper one on one that becomes digitized and so knowing that the digitized records could be manipulated but the very fact that you could always return to paper and you could audit it.
And do those sorts of things makes the success of those digital kind of efforts of manipulation that much more challenging and potentially unsuccessful. So how did you guys think about if that online process is happening, the sort of duplicate record step?
[0:31:29.0] CF: Well I am not really a cryptographer and I am not working on the protocols, I am just securing the web server but this is around here –
[0:31:35.9] AA: It’s other people’s problems.
[0:31:37.8] CF: Yeah, true and they work with return codes. So you enter the selection of your candidate and then the system returns a return code to you, which you can check on your papers; so the paper brings an individual return code which you can expect when you enter that candidate. So for every candidate you have a personalized return code and the man in the middle is not able to tell you this return code because only the web server is able to tell you this.
And then from a statistical view upon in survey, it is interesting that it doesn’t take a lot of people to check this and then Athena who worked a lot in open source E-voting software is he did a presentation last year at the Enigma Conference where he would quote about one in 1,000 people needs to check this code and then a threat would surface. If only for a million people and for a thousand people, if all the computers guys did this then it will surface.
They would notice there is something amiss, “The return code is not what is written on my paper ballot, there must be a man in the middle attack.”
[0:32:55.2] AA: Yeah, interesting so I am clearer or listeners who aren’t familiar with that process. So I vote for Snoopy for president or prime minister or senate whatever and then I submit that with my E-ballot and then it comes back and then if I voted for Snoopy, I should get a code that says “puppy” or something cryptographically more, probably unique, but essentially – and if Garfield was the other candidate and I got back the code that said “kitty.” I am like, “Whoa something funny is going on here” and so got you.
[0:33:33.1] CF: Absolutely and what is important here that is your return code. So your friend would get a different return code, that is very important and only the central application can know these codes and they are pre-calculated and we need to make sure they are not being stolen, etcetera but that is how our system boils down to this is the protection from the man in the middle.
[0:33:55.7] AA: Got you and literally every piece of paper, the same way it has a unique kind of voter number that that’s how I maybe log into the system, right? Also the codes are all unique and so then cryptographically you have – and then just because some of our listeners are truly paranoid, how do you know that the central system that essentially holds all of that repository of codes hasn’t been compromised? If they are able to do a man in the middle attack, you just say that they would also have to own that central repository?
[0:34:28.1] CF: Yes, essentially that and that is a question of democratical resign.
[0:34:33.4] AA: Got you, okay. Yeah, interesting and you can think about redundancies that those don’t sit next to each other that the transmission doesn’t go necessarily through that central repository?
[0:34:43.8] CF: Yes and you need multiple people joining for the key and so in the system I am working on, the vote is encrypted and they are not able to decrypt your vote before the end of the E-voting period and then multiple people have to come together to join the key to decrypt the votes and in this process the anatomization happens. So the IP address and the vote are separated in this process and it is only afterwards that they get to see the votes.
[0:35:23.7] AA: Got you, interesting, yeah and so you have that multiple – that the image that we think about multi-individual authentication is like it take two keys to turn on the nuclear missile that sort of image, got you. It sounds like you guys have done a lot of thinking. If someone were to kind of see or read further on what you guys are up to, is there a place that they can do that publicly available?
[0:35:51.4] CF: I think that the E-voting documentation of Swiss Post is translated into English. So that would be a good place to start because they do a lot of publications to explain this to people what they’re trying to build; these systems now are being build. They are allowed to be used for a certain percentage of all voters and to go to a 100% of voters, it takes additional certification stamps and no system has done this certification so far in Switzerland.
So this is active work, what I have just explained is actively being done or what I actually told you is already in place and additional security measures are being designed right now.
[0:36:42.9] AA: When is your next major – for those of us who aren’t in Switzerland and sort of not familiar with your political calendar, when is the next major vote that you’ll start to see this stuff be used?
[0:36:53.2] CF: Okay you are maybe familiar - but we are a direct democracy in the sense of California but then we vote four times a year. So every Swiss citizen is voting four times a year, so we are constantly voting. That’s why we have so much experience with voting and this is such an important thing and then we get to test this a lot because it happens four times a year. It makes it a lot easier than to elect a president every four years, and our next major parliament selection is on 2019.
So we are one and a half year away from the next major elections but only four weeks away from the next public vote.
[0:37:37.8] AA: Wow, has there been any concern about foreign adversaries attacking your system or anything like that?
[0:37:43.6] CF: Yeah, that is part of the threat model.
[0:37:45.7] AA: Okay but historically you haven’t seen any like there is no –
[0:37:50.0] CF: None we’ve seen so far. We’re Switzerland; we are neutral.
[0:37:53.8] AA: Right, good to know. Thanks for reminding me, all of us, right? Well you know there is so much other stuff I would love to talk about, but I feel like I have been peppering you with questions. So is there anything you particularly wanted to kind of cover or let people know about? You know I always think to just give you the opportunity to talk about something that you do, you are excited about, or want to make sure you get out in the world?
[0:38:19.4] CF: Okay, what else could we cover? No actually I like you asking questions and I have plenty of time. You want to throw the ball? That is a lot of better than me thinking of good topics.
[0:38:32.1] AA: Sure.
[0:38:32.5] CF: It’s been fun so far.
[0:38:33.4] AA: No, it was great particularly the voting stuff is so top of mind for what we’ve been – I spent three days of a holiday weekend sitting with election commissioners.
[0:38:43.8] CF: I think for the publication, you might want to cut this down a bit because it is getting on and on. Generally, I think yes it is an interesting topic.
“Let me put it this way, so denial of service attacks where somebody or a group of people are flooding you with traffic is a constant threat on the internet. So you can do all sorts of attacks against somebody but denial of service is one class of attacks which will always work if it is big enough. ”
[0:38:54.2] AA: Yeah and what we might do is we can cut it down and we’ll just make it a multi-parter too so that it doesn’t have 20 pages to read but how about your DDOS (Distributed Denial of Service) stuff. So I would like to hear more about how you think about strategies for stopping DDOS and maybe touch on some of the reverse proxy stuff as well?
[0:39:14.0] CF: Okay, that’s a good question and let me think what I want to tell you.
[0:39:21.4] AA: This is where you’re like, “Hmm what can I say and what I can’t?”
[0:39:25.5] CF: Yeah absolutely. That is exactly the point here. Let me put it this way, so denial of service attacks where somebody or a group of people are flooding you with traffic is a constant threat on the internet. So you can do all sorts of attacks against somebody but denial of service is one class of attacks which will always work if it is big enough. With denial of service attack, you either attack bandwidth, so that side is no longer able to communicate with the outside world, just completely jammed; or you attack the memory of the servers, so they ran out of memory, they are no longer able to compute; or you attack the processor with the specially crafted request. This is always a threat and you need to protect against it and then people don’t. People are not protecting against that because it happens a lot on the internet but then it is a big pond with lots of fish and why would it hit you?
So unless people have been attacked before, they tend not to protect against denial of service attacks and then once it really happens, there is a lot of panic because nobody has ever seen the like and it is extremely chaotic and you can count the dollars go out the window especially if you are an online shop and you know one hour of downtime costs you so much money. So it is really dangerous and it’s very good that you are well advised to prepare for that and how you prepare for that.
I think you need to know your architecture and you need to know your weak spots - how to learn about your weak spots - you need to do exercises. You need to try and do denial of service attack against your site yourself. Like with a lot of security things you need to try out the attack and then you will learn if your defense works or it doesn’t. But people are afraid of assimilating denial of service attack that’s not how it’s done. That is how they don’t know where the weak spots are and when it hits then there’s a problem.
Generally, you can cut down your service to be more lean, to be faster, to be more resilient so to make the best out of what you have, to make good use of your bandwidth or your servers but this will obviously only take you so far and then it’s also a resource problem. So it takes a lot, optimization takes a lot of time and a lot of resources and a lot more people and then when you’re no longer able to do this yourself and you need support, then you turn to specialists who will handle the DDOS for you.
This will work like an accountant delivery network nowadays meaning you hand over your domain record to somebody else and they do a filtering of the traffic for you and they are competing offerings there and you pay them like an annual fee or by the hour and you just hope that they are able to swallow the attack. What also works, and we’ve touched on this before, is for small countries to say, “Yeah during the attack let us just get rid of all the foreign traffic.”
And getting rid of all the foreign traffic in case of Switzerland means getting rid of 99% of the vote or getting rid of 99% of the internet. This means that foreign risk to your side are no longer able to connect; that is a pity but that is acceptable during the attack - if you have a local or domestic offering - and the attacker will realize that the fun is over because your site is staying up and open online and there is no longer able to bring you down and this usually stops many attacks.
The experience shows in Switzerland where people were globbed with Geo location and defense, attacks stop, within minutes or within hours very often and then you allow the foreign traffic again and everything is fine. It makes the hard cut to say, “No let us concentrate on the domestic traffic,” somebody in upper management needs to make that decision and it is not an easy decision but usually - or in my experience - this brings an end to the attack.
But obviously it has only work when you can concentrate on local traffic or you have built up your whole set up across the globe and you are able to separate it where you say, “Okay we are trying to keep East Asia up and we allow Southern America to drop during the attack,” stuff like that and you handle this with routing your traffic.
[0:44:17.4] AA: And essentially you use a cloud deployment where one of the brokers who essentially takes over your domain name and then that is some of the service they provide. They say, “Hey if this to now service happens we are going to spin up additional capacity in different public cloud providers all across the world and so we can essentially handle it,” or just make sure that they can’t overwhelm it.
[0:44:44.5] CF: Yes, again somebody needs to swallow the traffic and then they need to wash it. We call this washing of web traffic.
[0:44:53.2] AA: And you do that by just starting to cut off – like you had start to identify essentially malicious traffic and saying and cutting it off based on either the IP address or the nature of the activity, right?
[0:45:09.5] CF: The nature of activity. As it happens most DDOS attacks are still network based so you can look at the pattern of the network usage and a classical DDOS attack extremely simple is which just opens a lot of connections and then we do essentially nothing but we open a lot of connections and if it is a stupid server, it would just die because there are too many connections for it and the support server they would recognize that and say that if somebody is only opening dozens or hundreds or thousands of connections then which is no longer accepting the IP address.
[0:45:44.2] AA: And is that just that they start to handshake and don’t finish it, is that what they do?
[0:45:49.2] CF: Yeah, absolutely.
[0:45:51.4] AA: It’s like you stick your hand out and –
[0:45:52.6] CF: Or they are off to attack, exactly. Too many hands to shake.
[0:45:56.5] AA: Right, got you. Good stuff, how about reverse proxy?
[0:46:00.6] CF: Generally as an architecture feature or?
[0:46:03.8] AA: Yeah, for someone who is not familiar with the term; what it is and what it does, I think just start there?
“Reverse proxy... is put in front of [the application server] and would separate the application server from the internet. It is a bit like the clerk in a bank; so that is the person you talk to as a customer but that is not the person who gets to access the big bank safe. He doesn’t have direct access to the database. Now he’s just a clerk in front with limited authority. He can take requests, go to the back office and it is the back office who does all at once handling of the customer’s request”
[0:46:10.5] CF: Okay, sure. In the standard application architecture, you would have the core of your assets in the database that is very well guarded by multiple layers of security if you’ve done this right and then in front of the application and in front of the database is the application server, which runs the application - that’s where the intelligence is - and then you maybe don’t want to expose this onto the internet. I mean things change quite a lot with the cloud.
But let’s take to this classic view of you have an application server and on the database and we want to protect that one. An application, a gateway server, or a reverse proxy, which is a term usually used, is put in front of that and would separate the application server from the internet. It is a bit like the clerk in a bank; so that is the person you talk to as a customer but that is not the person who gets to access the big bank safe. He doesn’t have direct access to the database. Now he’s just a clerk in front with limited authority. He can take requests, go to the back office and it is the back office who does all at once handling of the customer’s request - I think that is a very good image and then they return the response to the clerk, he goes to the front and hands it over to the customer and that is how reverse proxy works. So he accepts requests, goes to the back-end application, the back-end application processes the request, and the response is returned via temporary proxy again.
So it has a function to architecture that it separates the different servers and it has a security functionality: if you use the reverse proxy as a web application firewall, so the web application firewall - especially in the case of ModSecurity - is deployed on this reverse proxy server to do the filtering of the request, identifying the malicious request and now this goes on. Their identification comes into play here; so you separate on the conceptual level: you have your authenticated users and your not authenticated users. Non-authenticated users could be anybody via the internet globally and now you could say, we don’t want to have unauthenticated users be allowed to access the application. They are only allowed to access the authentication service and afterwards, we connect them to the reverse proxy application server.
Now a funny thing happened. This setup is very - or it is a standard setup which you do in Switzerland. Swiss Banks are doing it like this, you go to reverse proxy and you have to identify yourself on the reverse proxy and that is like school book setup, that is how you would do this properly but then it is fairly complicated. I tend to think that outside Switzerland you have a different setup that the reverse proxy allows a narrow channel of authentication request back to the application. So unauthenticated users are allowed to pass through the reverse proxy onto the central application server of the bank if we are talking about online banking.
I think that is more risky, but the simpler setup because you have all of the customer data in the same server if you want. This setup, so the Swiss side is different than we do access layer or reverse proxy authentication very often, and this had on the industrial level an interesting effect that non-Swiss companies have a hard time penetrating the Swiss market with their technology because they are not specialized with the setup.
So within the small country of Switzerland, you have multiple providers of commercial reverse proxies that support this special architecture and the big American companies are not really penetrating the Swiss market and especially not the high secure market because they don’t have the necessary experience to support this architecture. I am not really sure if this is interesting to your audience, but I think it is an interesting fact that you can construct this differently.
[0:50:57.6] AA: Do you think are you seeing particularly as you are seeing just the increased – particularly I think of ransomware as an emerging threat in the space? And as we think about your analogy it becomes perhaps a little bit cruel to think about what is happening but the reverse proxy, or “clerk” isn’t worth that much and he doesn’t have that much; so if you ransomware him or if you destroy him you’re like, “Okay, well we’ll just hire a new clerk,” right?
So are you seeing this architecture kind of spread more widely or interest in it increasing because of its potential to kind of isolate the more valuable pieces of the architecture and the database and the application?
[0:51:49.3] CF: I don’t know and I don’t think we have the data to support this. Things are changing with the cloud, so the new architecture is coming in. I don’t think the cloud setups are more secure on the architectural level. I rather think that people are neglecting security on the architectural level because they think now we are in the cloud somebody else is handling security for us, which is absolutely not the case but you could get this idea by watching advertisement. But you are even more exposed in the cloud and you would be well advised to take good care of your architecture.
Touching on ransomware, yes, I think a simple reverse proxy in the cloud I mean that would be a container and you just throw it away if it gets exploited or corrupted or you do it in a way where you say, “No we replace the reverse proxies once per hour,” why not? I mean I think that would be a smart move if you have the resources to do that.
Then you don’t really care too much anymore but then again, if you have a complicated set up where this disregards proxy and needs to do authentication as well, then it is probably a more expensive clerk, which you are not replacing so easily - invest more resources into it and then you need to take better care of it.
[0:53:14.3] AA: These are valuable clerks because when he does authentication then he has access to a database which has the credentials and identification of those individuals right? So there is –
[0:53:26.0] CF: Usually not direct access but he has some means to access to be done.
[0:53:32.9] AA: Is it essentially like - if you think of that reverse proxy, is it possible that he has essentially access to two different databases or applications, one would be the identification application in database and the other is essentially whatever the actual customer data or whatever the main what’s going to happen post identification. That’s how the architecture looks so it is like two sided, he has two bosses that he talks to.
First: the one that says “Is this the guy or the gal” and then the next: is like, “Okay, we give him the thumbs up now he can actually talk to the next person,” is that how it is working?
[0:54:13.1] CF: Yeah, that is exactly how it’s working.
[0:54:14.3] AA: Very cool. Well Christian this has been great. Really just interesting stuff that you are working on and I appreciate you explaining concepts in a way that I think – I was talking about somebody else and they said I want to explain things in a way that my mom can understand. So I think you definitely pass that bar, right?
[0:54:37.4] CF: Thank you very much, that is what I try to do.
[0:54:40.1] AA: And that is a huge compliment, I think to be able to avoid jargon and make it understandable and approachable but at the same time I don’t feel that we stayed at a surface level. I don’t think we didn’t really get into how things actually work. We just managed to do it in a way that was understandable which was really valuable.
[0:54:57.5] CF: Yeah, very cool that is one of my ideas. I think you need to dig into technology to really explain security, but you need to do it in a way that non-techies are able to grasp it and technical people at times have a hard time doing that but as I have a not very technical background initially, that’s what I am bringing into this industry. So I am an arts and humanities scholar and in the end we learned to explain things and thank you for the compliment. Yes I think I try to make it really understandable and I am glad this worked out in this interview.
[0:55:34.5] AA: Yeah, that’s great. So I would love to know anything you think you’d love to link to so I can link to any of these, you know why don’t you follow up with any resources that you think would be interesting particularly like that Swiss post voter stuff.
[0:55:47.2] CF: Yeah, let’s do that. I will send you a couple of links that you could pick the ones you think are interesting.
[0:55:53.7] AA: Yeah, if you find yourself in the States anytime soon, I owe you several beers or glasses of wine, or coffee, or whatever your poison is so.
[0:56:05.0] CF: That would be so nice, yeah.
[0:56:06.9] AA: Awesome.
[0:56:07.8] CF: It was great talking to you.
[0:56:08.7] AA: Yeah, this is great and I am in –
[0:56:10.0] CF: Thank you very much Andy and we’ll hear from one another, great.
[0:56:13.0] AA: Yeah sounds great. Cheers.