yPass.net

I’ve had to start using Xen virtualization for a current project we’re working on. I always hate switching back to Linux servers because all of our fancy tools and scripts for automation are written for Solaris since we only have a handful of Linux servers.

At any rate, I’ve got Xen all figured out and really started to dig into Linux’s LVM for the first time. There’s some similarities between LVM and ZFS, but most noticeably LVM doesn’t deal with RAID at all. You have to set up manual Linux software RAID and put a VolumeGroup on the RAID meta-device. So I set up a nice software RAID5 device, created a VolumeGroup, and off I went.

The write performance was horrendous.

So I begrudgingly went into the RAID controller BIOS and set up hardware RAID5 and put LVM on top of that. After the installation, I decided to see how fast this was compared to ZFS raid1z (which is more or less RAID5).

The machines are identical:

• Dual 6 Core Opteron
• Sun STK RAID Controller (Adaptec) — 256MB cache, write-back cache mode enabled
• 16 Gigs of memory

Here’s the results:

Linux — 21GB Write

# time dd if=/dev/zero of=/root/test bs=10240 count=2009600
2009600+0 records in
2009600+0 records out
20578304000 bytes (21 GB) copied, 146.226 seconds, 141 MB/s

real    2m26.377s
user    0m4.068s
sys     1m53.823s

Linux — 1GB Write

# time dd if=/dev/zero of=/root/test bs=10240 count=102400
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB) copied, 2.69437 seconds, 389 MB/s

real    0m2.702s
user    0m0.108s
sys     0m2.584s

Solaris — 21GB Write

# time dd if=/dev/zero of=/zonepool/test bs=10240 count=2009600
2009600+0 records in
2009600+0 records out
20578304000 bytes (21 GB) copied, 55.3566 s, 372 MB/s

real    0m55.412s
user    0m0.913s
sys     0m27.012s

Solaris — 1GB Write

# time dd if=/dev/zero of=/zonepool/test bs=10240 count=102400
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB) copied, 1.25254 s, 837 MB/s

real    0m1.257s
user    0m0.046s
sys     0m1.211s

837MB/s for burst writes on raidz1! ZFS is too awesome.

Here’s the controller configurations:

Linux Controller Configuration
Solaris Controller Configuration

eric General, Solaris linux, video, zfs

I would be remiss to not have a blog post about Netflix on the PS3. As much as I post about streaming video to the PS3 and as much as I love Netflix, I can’t resist chiming in on this one.

First of all, I don’t care what the CEO of Netflix says, but having to put in a disc to stream movies sucks. The streaming app should be an installable application that sits on the XMB.

It’s not a matter of being lazy, it is a matter of convenience. Back on the PS2, when I started working on streaming video to Sony devices using BroadQ (oh yeah… btw, i’ve been working on this for about 6 or 7 years now), it was annoying to have to load in the BroadQ disc to stream the movies. I can’t imagine it will be any less annoying 7 years later when the system has a hard drive that is perfectly capable of storing the application.

That said, I’m excited this is finally happening. There’s little doubt that Microsoft opened up the checkbook to prevent interoperability. Netflix will be available for the PS3 almost exactly one year after XBox. For games, I can understand these exclusive agreements. For third party services such as Netflix, I think it’s a dick move on Microsoft’s part. I view it as yet another good reason not to support their console.

As a Netflix subscriber, I think it’s a bad move by both Netflix and Microsoft. This should have happened long ago.

eric PlayStation3 movies, ps3, streaming, video

As a follow up to unblocking Hulu on the PS3, I’ve gotten tons of of responses. While that solution will successfully work in Windows, setting up squid on a Microsoft OS can be a painful task. Windows makes the simplest things incredibly difficult.

But squid was definitely designed for Linux, so that’s not incredibly surprising.

Thankfully, for the windows users out there, Jonathan Morales has this to contribute:

You can set this up in half the time using windows and an old program called proxomitron. You might want to post this for those who just want it to work easily:

step 1) Download proxomitron from http://www.proxomitron.info
step 2) open proxomitron and uncheck everything except the outgoing header filters
step 3) open outgoing header filters and uncheck them all, but find one of the user_agent ones and modify it as such:

1. change the header name to user-agent:firefox win32
2. (optional) in the url match, put *.hulu.com – this will only activate the header if you go to that url
3. header value match should be *
4. change the value to Mozilla/5.0 (Windows NT 5.1; en-US) Firefox/3.0.11
5. make sure that the outgoing checkbox is checked but not the in

step 4) on the proxomitron main window go to config, click the access tab and set to allow connections from your local network e.g. 192.168.0.1 to 192.168.0.255

step 5) go to file>save default settings, and then close and reopen proxomitron

step 6) On the ps3 network settings, set it to use a proxy server and point it to the ip address of your windows machine running proxomitron, port 8080.

Other comments:

it’s a good practice to set your ip address on the windows machine that runs proxomitron to be static, so that there’s no chance it’ll change and your ps3 starts pointing to an ip address that doesn’t exist or is assigned to another computer.

eric General, PlayStation3 hulu, ps3, streaming, video

An interesting little project I’ve been working on is time lapse photography. I picked up a used Canon Powershot A520 pretty cheap, and set up a laptop with Ubuntu to communicate with the camera. I’m still working on the best angle to minimize the power lines out front, but I’ve got a good start going.

What you’ll need:

• A Linux machine (a laptop really helps)
• gphoto2 >= 2.4.5 (note that you can upgrade jaunty’s gphoto2 with the karmic packages to get this version)
• A camera that supports remote capture
• An AC power outlet near were you want to take your photos (and an AC adapter for your camera unless you have really awesome batteries).
• jpeg2yuv and ffmpeg (with libx264 support)
• Something relatively interesting to take pictures of

This is what I ended up with:

So here’s what I did:

1. Connect the USB cable to the camera
2. Run the following command (in a while loop in case it crashes):
   

   while true ; do
       gphoto2 --capture-and-download -I 30
   done

3. Wait about 8 hours or so
   1. If you’re impatient like me, you can nfs mount the laptop after about 45 frames (about 20 minutes) and get a preview.
   2. You can rsync the laptop’s nfs mounted directory locally so you don’t have to copy the files over (most likely) wireless every time you want to encode the latest version
4. Collect all of your images and make sure that each frame is numbered sequentially.
5. Create an MPEG with jpeg2yuv by piping the output to ffmpeg:
   

   starframenum=XXXX # put the number of the first image in the sequence here
   jpeg2yuv -b $startframenum \
           -v 0 \
           -j the/path/to/your/images/IMG_%04d.JPG \
           -f 15 \
           -I p | ffmpeg -threads 2 -y -i - \
           -vcodec libx264 \
           -b 2500k \
           -acodec libfaac -ab 48k -ar 48000 -ac 2
           -s 1024x768 -f mp4 \
           outputfile.mp4

In the options above, the important ones are ffmpeg “-f” which is the framerate. You can change this to speed up and or slow down your movie. The “-s” option is the size. Keep in mind that the width and height of your images needs to be a multiple of 16 (ie, 640×480, 1024×768, 1920×1152, etc). Note that 1080 is not divisble by 16. 1280×720 will work for widescreen (16:9) hi-def though. Lastly, the “-b” option is the video encoding bitrate. Increase it for better quality and decrease it for smaller output movie files.

eric General linux, video

First things first: If you’re only minimally using your PS3 as a multimedia device, the Logitech Harmony remotes and Playstation 3 adapter are not for you. If you use your PS3 as a media hub, then it is a must have item.

My experience started when I realized my cable box DVR remote didn’t have a 30 second skip feature. I figured if I was going to buy a universal remote, it needed to be universal. After a bit of research, I ended up with the Logitech Harmony 880. From Amazon, with the PS3 adapter, it was about $180.

My first instinct was “Did I really just spend $180 bucks on a remote control.” Yeah, that’s a good chunk of change. If you’re looking for a cool toy to play with, I recommend it, but if you’re trying to save money, having a few different remotes and/or using the PS3 controller isn’t that big of a deal.

So if you were to ask me if I got my $180 bucks worth, my answer would be, “no.” However, that doesn’t mean that this thing isn’t really cool.

Instead of the standard style of universal remotes, Logitech has given us “activity based” universal remotes. Depending on the activity that you choose, the remote buttons react differently. For example, with a standard universal remote, if you want to switch from your DVD player to your cable box, you push the TV button, select the appropriate input, then push the cable button to control the cable box. If you have a receiver/amplifier that needs to change, you push that button and change the inputs as required.

With the Harmony remotes, you select an activity. For example, this video shows me switching from watching a show on a cable box to moving to my movie collection streamed from the PS3 (requires Firefox 3.5):

The important part to note is that I’ve only pressed one button. I set up the remote with a “Watch Movies” button. I push the button, it turns on the PS3, switches the TV input, and automatically navigates to the movies in the PS3 XMB. If I want to go back to watching TV, the remote turns off the PS3, switches the input back to the cable box, and brings up the guide.

Of course, all of this is configurable and you can create whatever macros you want. All in all, it’s a nice device. If you’re a fan of a single remote, this is a must have item. If you want a cool toy to play with, this is a must have item. If you just use the media features occasionally, skip it.

eric PlayStation3 movies, ps3, video

With Firefox 3.5, we get our first real chance to use the <video> tag. And man, it is awesome. If you’re using IE, Firefox 3.0, or Chrome, you won’t be able to see it. It seems there’s all kinds of debate about the codecs that the browser should support. Right now, Firefox uses ogg theora/vorbis.

Apple is apparently complaining about the ogg format. They say there are possible patent issues they want to avoid which is a weird claim since ogg is specifically designed to be free of such restrictions. More likely, they want to implement their own proprietary .mov format or possibly mp4. Either way, I’m sure it boils down to Apple wanting more money. This would also hamper open source project’s ability to use the video tag since there is no way to license the corporate proprietary formats.

Microsoft is strangely quiet about the whole thing. I’m guessing they’ll put out only WMV support and call it a wrap. They may just embed WMP into the webpage to handle the player. I wonder how long it will take to initialize that.

Anyhow, here is the video tag in action (OGG Theora variable bit rate, Vorbis: 96kb/s 2 channel):

Hopefully Sam and The Chin won’t mind me using this clip for a demonstration.

On a side note, is there a way to make Windows Media Player play H.264 encoded movies? Does it not support that capability?

eric General html, ogg, video

Update Jan 6, 2010: Some users are reporting that they’re getting an error telling them they need to update Flash. I imagine we’re going to have to wait for Sony to update the PS3 firmware with a new Flash version.

---

Over the weekend Hulu stopped working for PS3 users. How did they block the PS3 users? With the dumbest method they could find. They test the User-Agent string in the HTTP request. Well, luckily, we can use a proxy server and just rewrite it. Windows users may want to look here

1. Install squid (“sudo apt-get install squid” for ubuntu users…. for Windows users, google for: squid windows)

2. Edit squid’s default config (/etc/squid/squid.conf on ubuntu) with these changes (or download my configuration: squid-hulu.conf):

Search for “acl localnet src” and set it to your internal network. You can remove the other localnet definitions if you’re not using them as shown here:

#acl localnet src 10.0.0.0/8
#acl localnet src 172.16.0.0/12
acl localnet src 192.168.0.0/16

Note: For 99.9999% of you, the correct line will be “acl localnet src 192.168.0.0/16″

After the “acl localnet src” line, add this:

acl hulu url_regex -i ^http://.*.hulu.com/.*

Search for “http_access allow localhost” and add “http_access allow localnet” as such:

http_access allow localnet
http_access allow localhost

Add the following two lines pretty much anywhere in the file (the end of the file works just fine):

header_access User-Agent deny hulu
header_replace User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)

3. Start squid (sudo /etc/init.d/squid start)

4. Go to PS3 Internet settings. Do manual configure. Go through the settings as normal (the defaults should be fine) until you get to proxy server. Set it to be your PC’s IP address port 3128.

Your PS3 is now a Windows machine running Firefox (as far as Hulu is concerned) and you can use Hulu again. What a dumb method of restricting access.

UPDATE – Jul 30 2009: To all of you guys out there saying “Why don’t you just use PlayOn or TVersity?” My response is:

---

from eric to MediaMall Support
date Mon, Jun 29, 2009 at 10:00 AM

I assume you're saying that there's no plan for a Linux version then?

from MediaMall Support to eric
date Mon, Jun 29, 2009 at 10:03 AM

Correct.


---

That was the response I received. TVersity is quite obviously intertwined with Windows and there is no hope there either. The users that support the invasive DRM (by using it) are responsible for open protocols not being available. If you want to know why DRM is successful, you need only look in the mirror.

UPDATED – Aug 10 2009: As requested in the comments, the instructions now make it so that only hulu will get the header replaced. Also, the comments section has tips and hints for Windows users and squid version 3.0+. See:

Some more Windows Info

Squid 3.0+ Info

eric General, GTFS, PHP, PlayStation3 hulu, ps3, streaming, video

If you have a ton of movies like I do, scrolling through the full list on the PS3 is painful. After about 100 movies or so, you realize that you need a better method of doing it.

Well if you’re on Linux, you are in luck! As I started categorizing stuff by hand, I ran into another unrelated problem. While I was searching for the solution, I came across IMDB-to-MPEG. I had to hack around a bit in the PHP code to get it to work the way I wanted, but man did it save a lot of time. I fixed a number of things and submitted it back to the author, so hopefully we’ll see a new version soon.

Basically, you give it a movie name and it queries IMDB for the movie. Based on the IMDB Genres and Ratings, it creates your symlink tree for you. So in the PS3 XMB, when you go t your movie server you have all of your movies categorized by Genre so you can get a list of all Action movies, for example. Of course, since it’s all just symlinks, you can have the same movie covering multiple genres. So for a military crime drama — such as A Few Good Men — it would be listed under all 3 genres of Military, Crime, and Drama.

But it also had another interesting feature. It creates a MPEG video file that you can play from your uPnP client that gives you IMDB info, like the plot, the year, the ratings, etc.

This is what the original looked like: About Army of Darkness.m4v

This is what my latest version looks like: About Army of Darkness.m4v

And if you have an HTML 5 browser, here it is with the video tag (in Ogg Video format):

I wish I knew a good algorithm for making motion more fluid. The animation frames look jerky. I’m sure I need to blur or leave trails or something, but my attempts have all been failures.

eric General, PHP movies, streaming, video

I’ll start by summing up. If you have a big screen TV (40+ inches) and 802.11g, don’t try it. I have tinkered with just about every method of encoding files to fit into the 10 to 13 Mbit/sec max of wi-fi and it just doesn’t work (well). If you insist on trying to do this on 802.11g, keep reading.

If you’re looking to do this with any sort of respectable audio/video quality, buy a couple of 802.11n Ethernet bridges. For the price, the best I could find was the “NETGEAR HD/Gaming 5 GHz Wireless-N Networking Kit (WNHDEB111)” kit on Amazon for about $102.00 + shipping.

Update: Even More Information Here

That said, if you have a PS3 and are determined to use the built-in wireless 802.11g to stream movies, here’s some info:

1. Forget H.264 and the MP4 container. It won’t work. You have to reduce the bitrate to such low levels that the compression artifacts are incredibly noticeable.

2. Get a copy of Handbrake for ripping/encoding DVDs (the built-in PS3 preset will not work with Wi-Fi).

3. Your maximum bitrate is 1500kbit with 2 channel MP3 audio at 160kbit. Even at this low bitrate, you may still have problems. If you’re watching on a screen around 30″ to 35″, 1500kbit should be acceptable. On larger screens, you’ll still notice artifacts.

4. Your maximum sustained throughput with any over the counter 802.11g wireless router is going to be about 1.5MBytes/sec (about 12Mbits/sec).

Settings for the Handbrake Encoder

Here’s your settings for a normal 720×480 widescreen movie using Handbrake:

Container

For the “Container” drop-down box, select AVI.

Picture Tab

1. Set the De-Comb as needed. If you have a DVD that still has noticeable interlacing artifacts with De-Combing on, use Deinterlace->Slower instead, only use these if you can see the interlacing.

2. Click on the preview frame. Select everything except for Anamorphic. Check the boxes for Optimal for source, Align Dimensions, Keep Aspect, and Autocrop)

Video Tab

1. Video Code: MPEG-4 (FFMPEG)

2. Framerate: Same as Source

3. Uncheck 2-Pass encoding

4. Bitrate: 1500. If the video still stutters, you’ll need to drop this more.

Audio Tab

1. Track: Your desired language

2. Codec: MP3 (lame)

3. Bitrate: 160 (or 128 if you don’t mind sub-par audio)

4. Sample Rate: 48

5. Mix: Dolby Pro Logic II

Chapters Tab

1. It’s not supported, so disable chapter markers

That’s all you need to set. You can save these as a preset called PS3-WiFi.

Some other notes:

DLNA Servers

Windows

1. On Windows, there’s tons of them. Most will work fairly well. I liked SimpleCenter, mostly because it’s free (as in cost), it’s the first one I downloaded, it’s easy to use, and it worked.

Linux

1. On Linux, PS3 Media Server is junk. If all you want to do is watch a movie and don’t plan on ever using fast forward, rewind, or scene select it may work for you. It supports tons of options specific to the PS3, but on Linux it looks, feels, and runs like clunky Java software (oh, and you’ll need a good 750 megs of memory free to run it).

2. Twonky Media Server seems to work well, but it’s not free. It has some quirks with finding new content that you place into the media directory. Most notably, you have to restart the process for it to see them.

3. MediaTomb is free, but has a fairly ugly interface. I didn’t test very much with it, but it can be downloaded via Synaptic on Ubuntu. MediaTomb is the way to go. It’s rock-solid, fast, and easy enough to enable PS3 support (you have to modify the config slightly… just search for PS3 in the config file and follow the instructions). If you’re on Linux, this is your best bet. As far as stability, it’s way better than Twonky.

Lastly, it’s $100 bucks to get a pair of 802.11n bridges to connect your PS3. You’ll get the ability to stream HD content, you won’t ever have to rely on transcoding on the fly, and you won’t have to save your movies at such a low quality setting. Stop being a cheapskate and do your part to help the economy recover.

eric General, PlayStation3 movies, ps3, streaming, video