Main Page
Connecting a Fortinet VPN to Amazon AWS VPC
There is a lot of spotty information out there on the Internet on how to connect a Fortinet VPN router to an Amazon AWS VPC VPN, but a lot of it is confusing, wants you to use the GUI, is outdated, or simply doesn’t work that well. It took me a bit to get all of the pieces put together, but here’s the basic steps involved:
- Enable asymmetric routing – this allows packets to go out through one of the tunnels and come back through the other
- Create interface based VPN tunnels (phase1 and phase2)
- Configure the wan1 sub-interfaces automatically created in step 2
- Configure BGP
- Configure firewall rules
So here’s a generic configuration that does this. If you right click on the VPN gateway in the AWS Console and download the “Generic” configuration, you can easily change the values in this config.
Also, you need to make sure that the policy numbers I put in for the firewall configuration (policies 200-203) do not conflict with any existing policy numbers you have configured. If they do, just pick a different number; the number doesn’t matter. Note that these policies allow all traffic in and out of your internal network and the VPC. After you get it working, you’ll probably want to tighten those policies up quite a bit.
So without further ado:
config system settings
set asymroute enable
end
config vpn ipsec phase1-interface
edit "amazon1"
set interface "wan1"
set dpd enable
set dhgrp 2
set proposal aes128-sha1
set remote-gw <CHANGE: Tunnel #1 Outside Virtual Private Gateway>
set psksecret <CHANGE: Tunnel #1 Pre-Shared Key>
set dpd-retryinterval 10
next
edit "amazon2"
set interface "wan1"
set dpd enable
set dhgrp 2
set proposal aes128-sha1
set remote-gw <CHANGE: Tunnel #2 Outside Virtual Private Gateway>
set psksecret <CHANGE: Tunnel #2 Pre-Shared Key>
set dpd-retryinterval 10
next
end
config vpn ipsec phase2-interface
edit "amazon1"
set dhgrp 2
set pfs enable
set phase1name "amazon1"
set proposal aes128-sha1
set replay enable
next
edit "amazon2"
set dhgrp 2
set pfs enable
set phase1name "amazon2"
set proposal aes128-sha1
set replay enable
next
end
config system interface
edit "amazon1"
set vdom "root"
set ip <CHANGE: Tunnel #1 Inside Customer Gateway> 255.255.255.255
set type tunnel
set remote-ip <CHANGE: Tunnel #1 Inside Virtual Private Gateway>
set interface "wan1"
next
edit "amazon2"
set vdom "root"
set ip <CHANGE: Tunnel #2 Inside Customer Gateway> 255.255.255.255
set type tunnel
set remote-ip <CHANGE: Tunnel #2 Inside Virtual Private Gateway>
set interface "wan1"
next
end
config router bgp
set as <CHANGE: BGP Customer Gateway ASN>
config neighbor
edit <CHANGE: Tunnel #1 Inside Virtual Private Gateway>
set remote-as <CHANGE: Tunnel #1 BGP Virtual Private Gateway ASN>
next
edit <CHANGE: Tunnel #2 Inside Virtual Private Gateway>
set remote-as <CHANGE: Tunnel #2 BGP Virtual Private Gateway ASN>
next
end
config network
edit 1
set prefix <CHANGE: Your Local Net> <CHANGE: Your Local netmask>
next
end
config redistribute "connected"
end
config redistribute "rip"
end
config redistribute "ospf"
end
config redistribute "static"
end
set router-id <CHANGE: Tunnel #1 Inside Virtual Private Gateway>
end
config firewall policy
edit 200
set srcintf "internal"
set dstintf "amazon1"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
next
edit 201
set srcintf "amazon1"
set dstintf "internal"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
next
edit 202
set srcintf "internal"
set dstintf "amazon2"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
next
edit 203
set srcintf "amazon2"
set dstintf "internal"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
next
end
Conditionally Installing Packages With Puppet
If you want to install a package using puppet only if another package is already installed, you can use puppet’s virtual resources to accomplish this. The proper way to do this is two define your two classes and then realize the virtual package in the dependent class. For example, if I wanted to install php5-dev only if gcc was installed, I would make two modules: a gcc module and a php5 module.
In the php5 module:
class php5($type) {
package { 'php5-common':
ensure => installed,
}
package { 'php5-cli':
ensure => installed,
require => Package['php5-common'],
}
@package { 'php5-dev':
ensure => installed,
tag => 'develpkgs',
}
}
The ‘@’ symbol defines the php5-dev package as a virtual resource, so it doesn’t actually get realized when the puppet manifest is compiled unless some other module realizes it. To realize it, we go into our gcc module:
class gcc {
package { 'gcc': ensure => installed, }
package { 'g++': ensure => installed, }
package { 'make': ensure => installed, }
Package <| tag == 'develpkgs' |>
}
This will search through all of your modules and realize any virtual resource that is tagged with ‘develpkgs’. So for example, if you have another module called mysql and you want to install the mysql development package:
class mysql {
package { 'mysql': ensure => installed, }
package { 'mysql-server': ensure => installed, }
@package { 'libmysqlclient-dev':
ensure => installed,
tag => 'develpkgs',
}
}
Using LAME to Concatenate MP3 Files
I needed a way to concatenate multiple MP3 files of varying bitrate/sample rate/channels and I needed it to be scriptable to handle pretty much any permutation of various input MP3 formats.
I came up with a simple script that does just that. It’s certainly not ideal, because it requires re-encoding everything 2 (more) times, but it works well enough for me. Of course, the input files can be anything lame supports, so you can pass in AIFF files which makes this a little better.
The goal was to take a short intro audio file, a long content audio file, and a short outro audio file and pull them all together. To do this, I first transcode each audio file to an MP3 with known sample rate, channels, and bitrate. Then I decode that newly encoded file to PCM. Finally, it encodes the PCM to a new MP3 file with my desired final MP3 settings.
for f in intro.mp3 content.mp3 outro.mp3 ; do
lame -m m -b 192 --resample 44.1 $f - | lame --decode -t --mp3input - -
done | lame -r -m m -s 44.1 --resample 22.05 - outfile.mp3
I’m going to have to do this same thing with video using ffmpeg in the near future. I have a feeling that’s going to be a lot more difficult.
Thanks to this guy for sending me down this path.
Dencor Energy Control Systems ? Bad Idea Or Worst Idea?
I’m going to deviate slightly from what I normally post about on here, but I guess this is somehow tangentially related to technology. I bought a new house a few months back and it had a Dencor Energy Control System in it. Of course, I had no clue what this system does (and frankly, I’m still not entirely sure), but it wasn’t that big of a deal until recently.
Basically, the system consists of a programmable interface inside the house and a relay disconnect outside of the house. I’ve spoken to two different electricians about the system and they both say they also know nothing about it. The system that I have was installed by the original builders back in the late 70s or so, so we’re talking about pretty old technology here. I’m sure things have progressed since then, but that’s not really the point of this post.
The problem is that I have 3 electrical outlets on different breakers that mysteriously stopped working. This may or may not be related to this Dencor Energy Management System, but since I have no idea how this thing works, it seemed like a good thing to investigate. When I first bought the house, I was kind of curious how the system worked, but when I called the Dencor headquarters, they told me it was going to cost me $20 or $30 to get a copy of the manual. I wasn’t that curious.
So now that I can’t charge my razor or my fancy electronic toothbrushes once these outlets died, I decided to try again and I emailed the president of Dencor Energy Control Systems, Matt Essig, with this email:
I purchased a home back in February and it seems the original builder installed Dencor energy management systems throughout the neighborhood (back in the 70s). We’ve recently had a handful of outlets on various breakers stop working and I can’t figure out any reason why other than possibly this system. I’ve asked all of my neighbors if they know how this thing works and no one knows anything about it.
It says DDS-809 on the outer cover and on the circuit board it lists 809-1002.
I spoke with someone a few months back and they said you would have to charge me $20 or $30 for a manual for this, but that seems a bit extreme just to buy some instructions for a product.
I can find no information about this system online and your website isn’t very informative. If you have a manual for this, can’t you just scan it and post it on your website or at the very least email it to me. Or if that’s too much effort, simple photocopies of the manual pages would be fine and I can stop by and pick it up since I live in south Denver. I’ve attached a picture of the control panel (sorry it’s blurry.. i can get a real picture if needed) and I can provide photos of the relay box in the back of the house if that helps too.
If I can’t figure out how it works, my next step is going to be trying to figure out how to disable the whole system without killing myself by electric shock.
-eric
That seems pretty reasonable to me. But then something strange happened. Here is the email exchange between me and Matt Essig, the president of Dencor Energy Control Systems.
Eric,
The manual and spec sheet are attached.
I know actually charging for products and services when you are a for profit business in a market driven economy seems odd but maybe your approach is the right one; when I’m at the grocery store I’m going to insist they give me everything for free because the prices they charge are excessive.
We stopped producing the 809 decades ago; in 20 years would you support a product you stopped developing and selling, or giving away, today? How about Microsoft? Oracle? Thought so….
Maybe you should disconnect the system and watch your power bills go up (assuming the system is currently programmed properly)…
Matt
Since Matt is a big fan of free market economics, I figured I’d teach him a thing or two. So I responded with this:
Matt,
Thank you for the manual.
I am well aware of how markets work, but it seems you are not. In a market driven economy, customer service is incredibly important. This is increasingly more important now that the masses have such innovations as the Internet in order to share information about how companies treat their customers.
I see that you’re beginning to understand this since you responded to “Sandra’s” 2008 post on ripoffreport.com just a week or two ago on July 5th, 2011. I agree that Sandra was a being a bit unreasonable, but given your response to me, I can see why she might be a tad bit upset with you.
Now there’s a pretty distinct difference between what I’m asking of your company versus what you suggested I should ask at a grocery store. I think a more apt analogy would be me contacting the grocery store to help me out with instructions on how to microwave a pizza I bought. Or even better yet, contacting the *manufacturer* of the pizza… say Red Baron (via the toll free number on the back of the box that says “questions?”) and asking them how to microwave it. Now granted, I’m not going to ask how to microwave a 20 year old pizza, but we’ll discuss that next.
You see, you think I want something for free, but I am not asking you to give me any actual product or service for free unless you consider the instruction manual for your real product yet another product. That’s quite the stretch. But you asked quite an interesting question. Can I, in fact, find support for say…. Windows 3.1? You bet your ass I can. As bad as Microsoft support is, they appear to be doing a better job than your company. It’s unfortunate that you happened to pick the industry I am in for your examples.
Here is Microsoft supporting 20+ year old products:
Oh? I can download an updated vshare.386 binary for Windows 3.1? Yep… right here: http://www.microsoft.com/download/en/details.aspx?id=16991
Holy crap! Look at this! Windows 3.0 instructions on editing an autoexec.bat and config.sys file? Wow, that brings back some memories of the 80s…. http://support.microsoft.com/kb/85194
Of course, there are plenty more examples, but I think that should be sufficient for now. If you want me to give you some more examples (maybe HP printer manuals from the 80s?) I could certainly dig that up as well if you’d like. But at any rate, that’s not really the issue anymore now, is it? I possibly would have hired someone to come fix and/or upgrade the system, which of course, would benefit you, because as you well know, in a market driven economy if people can make money working on your products, your product’s future value increases in non-real terms (hint: think advertising).
But back to the point: The issue now is that your level of customer service has made my decision quite easy. I will post my email to you as well as your email back (and this one too) in its entirety on my website. I think others would be glad to hear how the president of Dencor responds to requests from users of their products.
I’ve also noticed that you seem to have a bit of litigious streak in you. You can contact “Christian Onsager, at Onsager Staelin & Guyerson” and let them know that you want to file suit against me when I post this information online as well. There’s no need for a John Doe subpoena though, you can have them serve notice directly to me at the following address:Eric [redacted]
[redacting my actual address here as well]Remember…. all I asked for was a simple manual. And again, thanks for the manual as well as the incredibly quick response.
Eventually, however, you’ll learn one of the greatest lessons of the market driven economy: Don’t be a dick to your customers.
-eric
Now I assumed that’d be the end of the story. Only an idiot would respond to that email. But Mr. Matt Essig, the president of Dencor Energy Management Systems didn’t want to leave it at that. He said he would sue me if I posted these emails:
Eric,
If you would like to post the emails on your website then go ahead. The email was meant for you, and you only, hence it was addressed to you. I will litigate over this if you choose to do so…just try me.
Matt
Well, not to let him down and of course I haven’t been sued in a long time, here we are. So I responded with this:
Matt,
Seriously? Emails are certainly not confidential. Furthermore, Colorado doesn’t even require two-party consent for recording and publishing of phone calls, let alone other electronic communications. You may want to contact your attorneys before you continue digging yourself into a bigger hole. You would think that for a president of a company, you would be a little better informed about the ramifications of your communications and your business conduct in general. But again, you have my address. Instead, you sound like a petulant toddler trying desperately to undo the damage that you’ve already done. Feel free to have your legal team serve notice of a lawsuit.
I will contact you again when I post the information online with a web address where you can find your emails and my commentary on my dealings with you today.
Kindest Regards,
Eric
P.S. The manuals you sent don’t mention anything about programming the system. I appreciate the documentation you provided, but if you could send the actual programming manual, that’d be incredibly helpful. Thanks again.
Well Mr. Matt Essig of Dencor Energy Management Systems, your move. Best regards and I would appreciate that manual if you could foward over a copy. You have my physical and email address.
Also, I’ve sent him a link to this post. I look forward to hearing from you again Mr. Essig
yTransit and GTFS revisited
It’s been a long time since I last looked at GTFS. Since then, I’ve gotten tons of emails and comments on the blog post about my failed little yTransit project. A Google engineer in the Czech Republic working on their transit team even contacted me, but still nothing from actual transit companies.
However, it’s been a little while and I think there may be a glimmer of hope for this project yet. I’m guessing (hoping) that since smartphones have become increasingly popular, more people in the industry are getting a bit more interested in the technology. I called the Summit County, Colorado transit system (Summit Stage Transit) this morning and talked to the dispatcher.
I told him that I was interested in getting their transit schedule into Google Maps and he didn’t just say “uuhhhh… what?” He actually said “I think we’d be very interested in that.” He told me the person I needed to speak with wasn’t in the office at that moment, but he’ll be in later today and that I should leave a voicemail.
Now remember, I had originally contacted Summit Stage Transit way back in 2009 and they weren’t interested and didn’t return my calls. So at least this time around, I actually got a favorable response. That is huge progress!
We’ll see if this goes anywhere, but if they’re able to help me get some requirements built, I might actually be able to make this happen.
So officially, the project is still dead pending resuscitation by John at Summit Stage Transit who is supposed to return my call this afternoon.
Update: I spoke with John and I have a meeting scheduled for the July 20th to discuss their needs and requirements. This thing might happen after all.
Solaris Licensing Changes: The Real Story
As you should already know, Sun was purchased by Oracle. Not too long ago, someone noticed a licensing change on the Solaris license website. A slow rumble of rumors has been building up about what those changes mean. Well, I contacted our Sun account manager to get the definitive answer, and here it is:
- The old Solaris subscriptions, the way people got software support for 3rd party hardware, are no longer available for purchase. Existing contracts are honored.
- Solaris support now comes through a contract on the hardware (Oracle SUN hardware)
- The license and accompanying entitlement from the web, without a contract and without hardware, only entitle the downloader to non-commercial, non-production, or personal use in perpetuity. Production use and evaluation for production are good for 90 days.
- When you purchase hardware, you receive an addendum to the entitlement that grants that piece of hardware perpetual, non-transferable license and entitlement to Solaris.
- For hardware purchasers, this is the same (in net effect) as always.
- For non-hardware purchasers – 3rd party, gray market, etc. – there is no legal way to obtain a permanent entitlement or to obtain support.
Personal Use
So lets get the easy one out of the way first. Solaris is still free for personal use. So that should satisfy the 0.0001% (yes, that number is an anatomical extraction) of the Solaris users that use Solaris for non-commercial activity.
Non-Sun Servers
Let’s move on to people that run Solaris on non-Sun servers: No Solaris for you, not yours! Items 1 and 6 make it clear that there is no possible way to legally run Solaris on non-Sun servers. Period. End of story.
Sun Servers without a Support Contract
Now lets talk about people that run Solaris on Sun servers, but do not purchase a hardware support contract: Some Solaris for you, but only a little! Item 4 says (and I clarified it with them), that purchasing new Sun hardware gives you a binary license only for the version of Solaris that’s available at the time of the hardware purchase. It does not entitle you to future upgrades or updates.
Sun Servers with a Support Contract
For people running Solaris on Sun hardware with a Sun hardware support contract, your support contract grants you rights to run future versions of Solaris.
nVidia Overscan Correction fixed in Latest Drivers
My solution for fixing overscan on nvidia cards is obsolete! I did find out just a few days ago that my solution does actually work.
The person that I was originally helping with this problem decided to give Linux another shot. He tested it out and reported that it did indeed fix his overscan problems.
However… for no particular reason I decided to check out the nVidia settings control panel again. When I opened it up in Ubuntu 10.04, I noticed this (and tested it to make sure it works, which it does):
Solaris ZFS vs. Linux with Hardware Raid
I’ve had to start using Xen virtualization for a current project we’re working on. I always hate switching back to Linux servers because all of our fancy tools and scripts for automation are written for Solaris since we only have a handful of Linux servers.
At any rate, I’ve got Xen all figured out and really started to dig into Linux’s LVM for the first time. There’s some similarities between LVM and ZFS, but most noticeably LVM doesn’t deal with RAID at all. You have to set up manual Linux software RAID and put a VolumeGroup on the RAID meta-device. So I set up a nice software RAID5 device, created a VolumeGroup, and off I went.
The write performance was horrendous.
So I begrudgingly went into the RAID controller BIOS and set up hardware RAID5 and put LVM on top of that. After the installation, I decided to see how fast this was compared to ZFS raid1z (which is more or less RAID5).
The machines are identical:
- Dual 6 Core Opteron
- Sun STK RAID Controller (Adaptec) — 256MB cache, write-back cache mode enabled
- 16 Gigs of memory
Here’s the results:
Linux — 21GB Write
# time dd if=/dev/zero of=/root/test bs=10240 count=2009600
2009600+0 records in
2009600+0 records out
20578304000 bytes (21 GB) copied, 146.226 seconds, 141 MB/s
real 2m26.377s
user 0m4.068s
sys 1m53.823s
Linux — 1GB Write
# time dd if=/dev/zero of=/root/test bs=10240 count=102400
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB) copied, 2.69437 seconds, 389 MB/s
real 0m2.702s
user 0m0.108s
sys 0m2.584s
Solaris — 21GB Write
# time dd if=/dev/zero of=/zonepool/test bs=10240 count=2009600
2009600+0 records in
2009600+0 records out
20578304000 bytes (21 GB) copied, 55.3566 s, 372 MB/s
real 0m55.412s
user 0m0.913s
sys 0m27.012s
Solaris — 1GB Write
# time dd if=/dev/zero of=/zonepool/test bs=10240 count=102400
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB) copied, 1.25254 s, 837 MB/s
real 0m1.257s
user 0m0.046s
sys 0m1.211s
837MB/s for burst writes on raidz1! ZFS is too awesome.
Here’s the controller configurations:
Linux Controller Configuration
Solaris Controller Configuration
Patch for the VastHTML WordPress Forum Server
So, I’ve made a number of fixes to the VastHTML WordPress forum server plugin. It has some pretty big bugs, and I don’t know if the project is being maintained anymore or not. At any rate, the fixes I’ve made should have been considered critical and should have been fixed long ago by whoever is maintaining it, but I digress…
I’m not going to support people trying to apply this patch. If you don’t know what a diff is and you don’t know what the patch command does, you’re probably out of luck. If you want me to fix all of the problems in this code and release it, pay me a bunch of money…
Also, the security problems in their code makes babies cry… but that’s for another day.
Lastly, to make the search actually work, you need to connect to your wordpress mysql database and issue this SQL statement:
alter table wp_forum_posts add fulltext key `text` (`text`);
Here's the patch: vasthtml-forum-server.diff
Here's what it fixes (in no particular order):
- RSS feeds now contain the username of the poster instead of "feeds@r.us"
- All & characters in the links have been properly changed to & as they should be
- Page 2+ of your forums will work
- Page 2+ of posts will work
- The number of replies shown in the topic list is properly set to number of posts - 1
- The title delimeter is changed from » to "|" (don't remember why i did this, but there ya go)
- The search form/box uses HTTP GET instead of POST so your back button works without complaining about having to resubmit your request
- You can press enter in the search box to submit
- A $ followed by a number doesn't get filtered out
- Apostrophes in posts/titles get their slashes properly stripped
I may have fixed other things in this patch and forgot about it. This works for me... your mileage may vary.
Threaded/Parallel Web Crawler (or Web Server Killing Software)
Short Version
Parallel URL Fetcher – If you want to put load on a webserver by crawling it, this is what you’re looking for. No java, no python, just a nice small, fast C program.
Long Version
It’s time to re-evaluate our HTTP caching software. At present we use Apache mod_cache (disk cache) and we’ve run into some problems.
Apache mod_cache + ZFS + millions of URLs and hundreds of gigs of cache files = bad
I’m not sure which of these guys is the culprit in this one. But I do know that when the ZFS dataset holding Apache’s cache gets to a certain size, disk I/O requests go through the roof. By clearing the cache (and freeing up that I/O), we see a good 5%-10% (extremely significant) jump in traffic.
At any rate, this prompted us to start looking into alternatives to Apache. The obvious first choice is Squid in accelerator mode. So I got Squid all set up in our offline datacenter, fixed the little things, and was ready the beat the crap out of it with web requests.
I can easily request all of our 500k+ “static” URLs, but those pesky URLs with arguments aren’t quite that easy. I needed a crawler. Something like wget –mirror but much, much, much faster.
After a lot of searching, I found a few python apps that failed to compile on Solaris, had deprecated/old dependencies, required specific python, etc. Python is starting to feel more and more like Java. Either the developers are horrible or the language interpreter is too picky to work properly (think…. JRE 1.2.5 build 1482???? no no no, you need build 1761!!!).
Speaking of Java, I also found a Java app (JCrawler) that looked perfect for what I needed. It certainly claimed to be “perfect.” It actually worked better than the Python apps that failed to build/run properly, but it didn’t actually work. It just kept spawning threads until it ran out of memory.
I was almost to the point where I thought I would have to write one myself, until I clicked on a link and a bright light from the heavens shone down on my monitor and a choir started singing in the background.
I had found the Parallel URL Fetcher. It was exactly what I needed. It was like wget, but ran parallel requests. It didn’t compile on Solaris either, but adding timeradd() and timersub() macros fixed that real quick.
I don’t think it supports Keep-Alive requests either, which would have been nice, but either way it rocked through some URLs. After letting it run for a few hours, I had my Squid server maxed out at 100Gigs of cache and ready for some I/O testing.
