Main Page
DVI to HDMI overscan (screen edge cutoff) on an HDTV
Well I learned something new recently. I have a friend that's making the Ubuntu switch and he called me up with a bizarre problem. He's using an nVidia card (although other cards have the same issue) with a DVI out port to a DVI->HDMI converter to an HDMI input on a 26" HDTV that he uses as a monitor.
He called me up and described the problem and I confessed that I had never heard of this before. All 4 sides of his sides of his screen were getting cut off. He could only see part of his menu bars at the top and bottom and the left/right edges were cut off as well. After some Googling, I at least found the name for the problem: overscan.
And once I figured out the name, that's when my Google searches became eye openers. There are a lot of people out there with overscan problems and there are very few solutions in Linux. The Windows nVidia drivers allow dynamic overscan correction inside of their driver toolbox. The X server nVidia drivers have no options (for DVI out... for TV out there apparently are).
The problem, as I understand it, is that the PC is sending a DVI PC style output, but the TV is reading a HDMI TV style input. As such, the TV thinks it's receiving a TV signal and acts accordingly. If your TV has a DVI input, it should treat that as a PC input and give you 1:1 pixel mapping (which is what you're looking for). If not, you'll need to adjust for the overscan on the PC side. Some TVs even have an option to treat an HDMI signal as if it were PC. Check your TV's manual.
Anyhow, there are a lot of people asking for help for this issue but is very hard to find any actual information. If you're looking for a way to fix this (and you're ready to spend quite a while doing it), you should read this:
Ubuntu Forums: Nvidia, Modelines, Overscan...8.10
Basically it's trial and error to get the correct X server config's Modeline. It's mindboggling that no one (especially nVidia, which seems to care about Linux a little bit) has put out any definitive information on this topic.
More terms to know:
1-to-1 pixel mapping: If your HDTV (as a monitor) supports this option, chances are this will solve your problem. This means that every pixel sent by the PC will be mapped to a pixel on the screen (i.e. disable overscan).
Full Pixel: This is the same as 1:1 pixel mapping
Modelines: Definitions of video modes that control the display size in the X server
Overscan: Part of standard TV input where a percentage of the edges of the screen are cut off. Not noticeable for normal TV viewing, but very noticeable on a PC desktop.
EDID: Monitor/TV device information telling the PC what modes are supported (stored in the monitor and not configurable)
Good luck.
Got a PS3? Want Hulu Back? Easy enough...
Over the weekend Hulu stopped working for PS3 users. How did they block the PS3 users? With the dumbest method they could find. They test the User-Agent string in the HTTP request. Well, luckily, we can use a proxy server and just rewrite it.
1. Install squid ("sudo apt-get install squid" for ubuntu users.... for Windows users, google for: squid windows)
2. Edit squid's default config (/etc/squid/squid.conf on ubuntu) with these changes (or download my configuration: squid-hulu.conf):
Search for "acl localnet src" and set it to your internal network. You can remove the other localnet definitions if you're not using them as shown here:
#acl localnet src 10.0.0.0/8
#acl localnet src 172.16.0.0/12
acl localnet src 192.168.0.0/16
Search for "http_access allow localhost" and add "http_access allow localnet" as such:
http_access allow localnet
http_access allow localhost
Add the following two lines pretty much anywhere in the file (the end of the file works just fine):
header_access User-Agent deny all
header_replace User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)
3. Start squid (sudo /etc/init.d/squid start)
4. Go to PS3 Internet settings. Do manual configure. Go through the settings as normal (the defaults should be fine) until you get to proxy server. Set it to be your PC's IP address port 3128.
Your PS3 is now a Windows machine running Firefox (as far as Hulu is concerned) and you can use Hulu again. What a dumb method of restricting access. You can get a bit fancier with the acl's to make it so ONLY Hulu URLs get the User-Agent replaced. It's not that hard, but until I have a problem, I don't see any point in bothering.
UPDATE: To all of you guys out there saying "Why don't you just use PlayOn or TVersity?" My response is:
from eric to MediaMall Support
date Mon, Jun 29, 2009 at 10:00 AM
I assume you're saying that there's no plan for a Linux version then?
from MediaMall Support to eric
date Mon, Jun 29, 2009 at 10:03 AM
Correct.
That was the response I received. TVersity is quite obviously intertwined with Windows and there is no hope there either. The users that support the invasive DRM (by using it) are responsible for open protocols not being available. If you want to know why DRM is successful, you need only look in the mirror.
Detecting Xlib's Keyboard Auto-repeat Functionality (and how to fix it)
If you've ever messed around with listening for XWindows keyboard events, you may have noticed something that's odd. XWindows has a very strange way of dealing with keyboard auto-repeating. Let's take a scenario:
Hold down the "h" key on the keyboard for a few seconds and you'll get something that looks like this:
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
Now, if you try this and watch carefully, you'll notice a couple of things. The first thing to notice, is that after the first character, there is a slight delay before it starts repeating. That delay is about a half of a second. After that initial delay, it repeats every 30ms. These delays are configurable in X. It may be somewhere in your window manager control panel, but the xset command will tell you everything you need to know:
$ xset q
Keyboard Control:
auto repeat: on key click percent: 0 LED mask: 00000000
auto repeat delay: 500 repeat rate: 30
auto repeating keys: 00fffffffffffbbf
fbdfffefffedffff
9fffffffffffffff
fff7ffffffffffff
Now, when you grab the keyboard in X using XGrabKeyboard or XGrabKey (so you'll get the keyboard events), something strange happens with all those h's. When holding down a key, you would at first expect to see 2 events. The KeyPress event and the KeyRelease event. But in order for auto-repeat to work, there must be more. For each auto-repeating character, you should get an additional KeyPress event. This solves the auto-repeat problem, because you can write some simple code that can easily detect auto-repeat and ignore it if you just care if the user is holding down a key.
That's exactly how it works in Windows (or so I'm told). In Windows, you get the following events given our string of h's:
P = KeyPressEvent
R = KeyReleaseEvent
h h h h h h h h P P P P P P P P R
Just what we'd expect, a bunch of KeyPress Events and one KeyRelease event. But that's not how it works in X. In XWindows, you get a KeyRelease for every KeyPress Event. In X, it looks something like this:
h h h h h h h h PRPRPRPRPRPRPRPR
If you google around on the Internet for a bit, you'll notice a number of people coming to this realization and a handful of ways to get around it (none of which worked for my needs). It seems like such a simple thing at first glace. Then there's hints that lead to nowhere. For example, a fake KeyRelease event is paired with a corresponding KeyPress event. If it's a fake, it'll have the same serial number in the X event structure as the next KeyPress event.
That's great, except there's no really good way to know if the LAST KeyRelease is the actual release. There's a hack on a Sun forum to peek into the next event in the event queue, but that doesn't work reliably either, because it's very possible that the next event will not be the corresponding KeyPress and/or the next event hasn't even been queued yet. You could bypass the first problem by using XMaskEvent or XCheckMaskEvent to search the event queue for only keypress events. But if the paired KeyPress event isn't there when your event handler runs, then what?
So, the solution I've come up with (I think), is the following:
When you get a KeyRelease event, use XQueryKeymap to query the X server and find out what keys are currently pressed down. If the key you're looking for is pressed down, ignore the Release event. With this method, there's a chance that by the time the event handler runs, the user has let up the key and pressed it down again, but for my purposes, this scenario doesn't matter.
What a pain. When I finish up my event handler, I'll post up a simple version that can be easily modified.
UPDATE: Here's some code to deal with this. Put this inside your KeyRelease XEvent handler. This assume that xevent->xkey.keycode is the keycode for the key you are checking.
char pressed_keys[32];
XQueryKeymap(Xwindow, pressed_keys);
isPressed = (pressed_keys[xevent->xkey.keycode >> 3] >> (xevent->xkey.keycode & 0x07)) & 0x01;
if (! isPressed) {
// the key has physically been released
}
Categorizing Your Movie Collection with IMDB
If you have a ton of movies like I do, scrolling through the full list on the PS3 is painful. After about 100 movies or so, you realize that you need a better method of doing it.
Well if you're on Linux, you are in luck! As I started categorizing stuff by hand, I ran into another unrelated problem. While I was searching for the solution, I came across IMDB-to-MPEG. I had to hack around a bit in the PHP code to get it to work the way I wanted, but man did it save a lot of time. I fixed a number of things and submitted it back to the author, so hopefully we'll see a new version soon.
Basically, you give it a movie name and it queries IMDB for the movie. Based on the IMDB Genres and Ratings, it creates your symlink tree for you. So in the PS3 XMB, when you go t your movie server you have all of your movies categorized by Genre so you can get a list of all Action movies, for example. Of course, since it's all just symlinks, you can have the same movie covering multiple genres. So for a military crime drama -- such as A Few Good Men -- it would be listed under all 3 genres of Military, Crime, and Drama.
But it also had another interesting feature. It creates a MPEG video file that you can play from your uPnP client that gives you IMDB info, like the plot, the year, the ratings, etc.
This is what the original looked like: About Army of Darkness.m4v
This is what my latest version looks like: About Army of Darkness.m4v
I wish I knew a good algorithm for making motion more fluid. The animation frames look jerky. I'm sure I need to blur or leave trails or something, but my attempts have all been failures.
jsmin as a PHP extension - JavaScript Minify the Fast Way
Yahoo's performance docs and Google's page speed best practices recommend that both external and inline javascript code is minified. Two popular tools for minifying JavaScript are JSMIN and YUI Compressor.
I needed to have an efficient solution that can minify javascript on the fly. Unfortunately these tools do not work because a process would need to be forked every time minification needed to be performed. In PHP, one possible solution is to use jsmin-php, but again this is a very slow option. jsmin-php is written in PHP code and parsing a JavaScript file in PHP is relatively slow.
So I decided to develop a php extension based on jsmin.c developed by Douglas Crockford. It creates a native PHP function included as a PHP extension (written in C as opposed to PHP). The speed and efficiency improvement over jsmin-php is over 25x!
Try it out!
Get it here
Install instructions (make sure you have development tools installed as well as PHP and its development components):
tar zxf php-jsmin-1.0.tgz
cd php-jsmin-1.0
phpize
sh ./configure
make
make install
(optional) in php.ini add the following line: extensions=jsmin.so
Sample code:
if (!extension_loaded('jsmin')) {
dl('jsmin.so');
}
header('Content-Type: text/javascript');
echo jsmin(file_get_contents('my.js'));
I hope you find it useful.
More PS3 Media Streaming Information
It appears that a lot of people are trying to figure out how to get media streaming to work for the PS3. I figured I would add a bit more information to my previous post on this topic.
Again, I highly recommend getting a 802.11n access point configured. It will make your life much easier. The PS3 internal WiFi card only support 802.11g. With 2 802.11n access points, you plug an ethernet cable from the PS3 to the 802.11n access point. The other access point is then connected via an ethernet cable to your router (or if you have an 802.11n router... you only need one).
Basically your setup would look like this:
This set up will give you a nice solid connection between your PC and your PS3. If you can actually run a wire, you'll be better off, but 802.11n is close enough. If you do decide to use 802.11g, you really need to read that previous entry that I posted. You're basically shooting yourself in the foot. You also need to make sure you have the latest system update. Since you should have at least a basic connection at this point, that shouldn't be an issue.
Now, if you don't have control over the types of media you're trying to stream, you need to stop reading here and go look into something that does transcoding on the fly. If you're trying this on Linux, you can try to use PS3 Media Server. Personally, I thought it was junk. If you're on Windows, Some people say TVersity is the way to go. I don't use these methods. I encode all of my own media so I can guarantee I have a decent copy. I encode all of the director's commentary tracks into my M4Vs (you can select the track in the PS3 playback menu) and I make sure they're at a quality level that is acceptable to me. If you're pirating video off the of Internet, you probably don't have this luxury.
If you are going to use 802.11g, you MUST use the AVI container. You will not be able to use the MP4 (a.k.a. M4V) container. The PS3 supports DivX video and MP3 audio content inside of the AVI container. On WiFi G, that is your only option.
If you use 802.11n or a wired connection, you have a few more options. My recommendation would be to use the M4V container with h.264 encoding. The only problem with encoding using h.264 is that it is the encode process is REALLY slow. However, M4V is quickly becoming the standard and AVI is fading away. If you encode all of your movies in AVI format, you're going to be upset when it's obsolete in a few years. Might as well plan for the future so you don't have to reencode everything later.
If you're downloading movies from the Internet, you need to make sure they're in the appropriate format. If there's DRM attached to those movies, you may have issues. None of my M4V files have DRM (I avoid it like the plague), so if that's the case for you... good luck.
OK, so now we'll assume you have one of the following types of files:
1. An AVI containing DivX encoded video and MP3 encoded audio.
2. An MP4 containing h.264 encoded video and AAC encoded audio.
So first things first, set up your PS3. If you're going to use DivX, you'll need to make sure it's enabled in the XMB (a.k.a. the PS3 menu). In the XMB, go to Settings -> System Settings and you should have a DivX option in there. Make sure it's enabled and you have the required code. There's other options in there to enable WMA playback as well. That's all you need for your PS3.
Now for the computer: You need a DLNA server.
For Windows: There's tons of them. I know for a fact that SimpleCenter works quite well with the PS3. That said, just about any DLNA server should work if you don't like the SimpleCenter interface. I'm pretty sure that Windows Media Player has some DLNA options as well, but I haven't used them.
For Linux: Ubuntu users have it easy. mediatomb can be installed via Synaptic or just using Applications -> Add/Remove... There's a couple of others. I tried 3 or 4 of them. mediatomb is the most stable. There's a list of binaries for other distributions on the mediatomb download page. It may even be in your package manager as well. For the PS3, there's one change you need to make to the mediatomb configuration file.
Next, you need to point them to the location of your media files. Once you've done that, go back to your PS3 and go to "Video" in the XMB. You should see your media server listed in the menu already. Go into the media server and select whatever is you want to watch or listen to.
An App To Manage Bus/Rail Routes in GTFS Format -- that no one wanted
TL;DR version: I wrote a proof of concept GTFS data manager. I suck at sales or something. Small/medium transit companies don't care about GTFS? The link to the app is at the bottom of this post.
The full version:
So before I actually started learning ExtJS (and the whole reason I started in the first place), I had this grand idea that there was a market for software that managed GTFS data for small and medium sized public transit organizations. The thought was that these transit companies didn't use the incredibly powerful -- and incredibly expensive -- route scheduling apps. To go along with that, their IT departments are probably really small and underfunded.
For example, my local transit agency is using formmail.pl to submit feedback and their website has that wonderful tag that defines low tech:
<meta name="GENERATOR" content="Microsoft FrontPage 6.0">
And the link to their formmail form is also 9 grades of pure awesome:
Anyway, I threw together a little ExtJS application as a proof of concept and started reading and learning how ExtJS actually works (because my code for the proof of concept was horrible). I then started reaching out to some small transit companies around the country targeting the ones that actually had halfway decent maps of their routes and stops. By the way, Amarillo, TX has the most detail online including pretty much everything you would need to create the GTFS data feed.
I figured for a relatively small monthly fee somewhere around $50 to $75 a month or so, we could give them an application that lets them manage their routes, trips, stops, and publish a GTFS feed for them with their data. Or perhaps you could charge based on the size of the transit company... like $x for y number of routes. Anyway, it's actually not an incredibly difficult program to write. At $50 bucks a month you'd never be a millionaire, but it would be some nice extra side income if you could get a bunch of these small agencies signed up.
I figured a good way to get started would be to offer it for free to whichever company I could find to accept it first. Amarillo was an obvious first target. Summit County, Colorado had quite a bit of data as well. My local transit agency was missing GPS locations of their stops, but it wouldn't be too hard to collect that. Probably $100 bucks in gas or so driving around collecting GPS coordinates. There were a good number that had pretty much all of the data online already, they just needed the format. That would have been perfect, because I could format the data for them semi-manually while I was writing the app at the same time. This would give me a perfect test case and they'd get free service.
After a number of email inquiries and some phone calls, I got absolutely nowhere. One must really suck at sales when you can't even give away a product to someone. Either that, or there is simply no interest. I didn't get a single return phone call or email from anyone, and I was offering it for FREE. Not even a nibble. Nada. Nothing. Either these companies don't understand the benefit of pushing their GTFS data to Google or they just don't care. (Judy Phelps @ City of Amarillo, I'm still waiting to hear from you!)
So the next question was how could you provide the service for free to transit company and somehow monetize the content. Well, these types of questions are certainly not my forte either, but we kicked around a few ideas and ultimately decided that there's really just no good way to do it and make any halfway decent amount of money. I don't think you could even cover the costs of the hosting. You could write your own web portal type thing to host their agency website for them, which isn't a bad idea... but again, the cost would ultimately fall on the transit agency and that was a black hole for my attempts at contacts.
So, after lots of reading and learning ExtJS, figuring out the GTFS format, the Google Maps API, learning JavaScript again (I hadn't touched it since Netscape 4.x or so.. it's a bit different now), and crunching some numbers to make the costs work, I've decided to give up... for now.
The proof of concept app works OK as a GTFS data browser (it's buggy), but I never implemented any modification methods. I did make a shape editor to generate GTFS shapes, but you can't actually save it. The data that you would need to generate the shape in GTFS format is stored in memory though. The data currently in there was imported from a good number of public GTFS feeds. There's about 15 or so feeds included I think. Anyway, you can mess around with it here:
http://www.ypass.net/yTransit/
I figure I'll probably end up finishing it (read: rewrite from scratch) at some point anyway, because it was fun to work on and it'll help me learn ExtJS. Then I'll just open source the code. Don't hold your breath. Without any monetary incentive, this is low priority.
If you have any comments or questions about it, don't hesitate to leave a comment below.
Google Wave and the Death of Email
Normally I'm not a big fan of the marketing hoopla surrounding announcements about upcoming technology. But after I watched the 90 minute Google Wave demonstration, I was awed. It's not every day you hear about someone discussing replacing something as ubiquitous as email.
The fatal flaw in most of Google's offerings is that companies lose control of their data. This is the problem with cloud computing as well. There's all sorts of privacy and legal issues when you use remote data storage. This is why I've always felt cloud computing is something of a pipe dream.
Google Wave, on the other hand is different. It's not a Google service, per se. It's an open protocol more along the lines of a mixture of email and IM. It appears to be based on a modified version of XMPP. Here's a summary:
1. Take everything you know about email and throw it away.
2. Email becomes a wave (side note: people that use email as an IM client are annoying)
3. IM becomes wave.
4. Wiki becomes a wave.
5. Blogs become a wave.
6. Documents (text, spreadsheet, etc) become a wave.
This has the potential to completely replace the way we view communication on the Internet. The demo (in the link above) is nothing short of amazing. But the best part is that it's an open protocol and each company can run their own wave server. This means it is actually usable in the corporate world.
There's a couple of news articles out there that try to explain it using in text format, but if you really want to wrap your head around the concept, it's best to watch it in action in the demo. If you're any kind of tech geek, it will be 90 minutes well spent.
Now I'm ready to set up a Wave server... no more MS Outlook.
Streaming Movies over 802.11g WiFi to a PS3
I'll start by summing up. If you have a big screen TV (40+ inches) and 802.11g, don't try it. I have tinkered with just about every method of encoding files to fit into the 10 to 13 Mbit/sec max of wi-fi and it just doesn't work (well). If you insist on trying to do this on 802.11g, keep reading.
If you're looking to do this with any sort of respectable audio/video quality, buy a couple of 802.11n Ethernet bridges. For the price, the best I could find was the "NETGEAR HD/Gaming 5 GHz Wireless-N Networking Kit (WNHDEB111)" kit on Amazon for about $102.00 + shipping.
Update: Even More Information Here
That said, if you have a PS3 and are determined to use the built-in wireless 802.11g to stream movies, here's some info:
1. Forget H.264 and the MP4 container. It won't work. You have to reduce the bitrate to such low levels that the compression artifacts are incredibly noticeable.
2. Get a copy of Handbrake for ripping/encoding DVDs (the built-in PS3 preset will not work with Wi-Fi).
3. Your maximum bitrate is 1500kbit with 2 channel MP3 audio at 160kbit. Even at this low bitrate, you may still have problems. If you're watching on a screen around 30" to 35", 1500kbit should be acceptable. On larger screens, you'll still notice artifacts.
4. Your maximum sustained throughput with any over the counter 802.11g wireless router is going to be about 1.5MBytes/sec (about 12Mbits/sec).
Settings for the Handbrake Encoder
Here's your settings for a normal 720x480 widescreen movie using Handbrake:
Container
For the "Container" drop-down box, select AVI.
Picture Tab
1. Set the De-Comb as needed. If you have a DVD that still has noticeable interlacing artifacts with De-Combing on, use Deinterlace->Slower instead, only use these if you can see the interlacing.
2. Click on the preview frame. Select everything except for Anamorphic. Check the boxes for Optimal for source, Align Dimensions, Keep Aspect, and Autocrop)
Video Tab
1. Video Code: MPEG-4 (FFMPEG)
2. Framerate: Same as Source
3. Uncheck 2-Pass encoding
4. Bitrate: 1500. If the video still stutters, you'll need to drop this more.
Audio Tab
1. Track: Your desired language
2. Codec: MP3 (lame)
3. Bitrate: 160 (or 128 if you don't mind sub-par audio)
4. Sample Rate: 48
5. Mix: Dolby Pro Logic II
Chapters Tab
1. It's not supported, so disable chapter markers
That's all you need to set. You can save these as a preset called PS3-WiFi.
Some other notes:
DLNA Servers
Windows
1. On Windows, there's tons of them. Most will work fairly well. I liked SimpleCenter, mostly because it's free (as in cost), it's the first one I downloaded, it's easy to use, and it worked.
Linux
1. On Linux, PS3 Media Server is junk. If all you want to do is watch a movie and don't plan on ever using fast forward, rewind, or scene select it may work for you. It supports tons of options specific to the PS3, but on Linux it looks, feels, and runs like clunky Java software (oh, and you'll need a good 750 megs of memory free to run it).
2. Twonky Media Server seems to work well, but it's not free. It has some quirks with finding new content that you place into the media directory. Most notably, you have to restart the process for it to see them.
3. MediaTomb is free, but has a fairly ugly interface. I didn't test very much with it, but it can be downloaded via Synaptic on Ubuntu. MediaTomb is the way to go. It's rock-solid, fast, and easy enough to enable PS3 support (you have to modify the config slightly... just search for PS3 in the config file and follow the instructions). If you're on Linux, this is your best bet. As far as stability, it's way better than Twonky.
Lastly, it's $100 bucks to get a pair of 802.11n bridges to connect your PS3. You'll get the ability to stream HD content, you won't ever have to rely on transcoding on the fly, and you won't have to save your movies at such a low quality setting. Stop being a cheapskate and do your part to help the economy recover.
The GNU people have gone off their collective rocker
I wanted to find out why more and more libraries are now licensed under the GPL instead of the LGPL. In my search, I found GNU's article telling people not to use the LGPL. I thought that was odd.
Today, I had the need for an RSS reader/writer library for work. MagpieRSS did almost everything I needed. It parses pretty much any RSS feed and has fairly loose parsing methods. I thought: well that's easy enough, I can just write a function to turn the MagpieRSS object back into RSS XML. I'd need to add <enclosure> functionality as well. I could submit the code back to the MagpieRSS people and MagpieRSS would be an RSS writer as well.
That's when I saw the license... GPL. So, that makes MagpieRSS useless for this project. That means I have to write my own RSS reader and writer. It's not that big of a deal to write an RSS parser. It's not like RSS is incredibly complicated. For that matter, it's not that big of a deal to write an RSS writer, either. But as we all know, why reinvent the wheel?
So now I have to write my own RSS parser and writer which I'll then release under a normal license like BSD or LGPL. Now there will be competing open source projects that provide the same functionality thanks to GNU's short-sightedness.
People are not thinking their cunning plans all the way through. By preventing commercial use of libraries, they lose the support that commercial users also provide.
In short, the GNU (and MagpieRSS) people are being annoying fanatics.
