Fork me on GitHub

Sigil.org Blogging about technology, beer, and whatever the hell I want

23Dec/102

Sadism: the order of the day at Disney

Posted by sarumont

At risk of increasing the percentage of rants on my already ill-populated blog, I feel the desire to share this. Twitter's 140 characters were not sufficient; Facebook's 420 characters didn't foot the bill; a blagorant it must be, then. Disclaimer: I do not endorse the piracy of copyrighted content.

In Disney's infinite pursuit of profit, regardless of how said pursuit may impact their customers, the Tron Legacy soundtrack cannot be had in its entirety from one source. Or two. Or three. Four is the number of sources which you must use, and the number of sources is four.

If you happen to be one of the 307,000,000 or so US residents, you have to shop with amazon.co.uk or amazon.ca to import the Deluxe version of the soundtrack. This buys you five tracks (12 minutes, 53 seconds of music) more than the only version available from amazon.com.

Source number two? Disney's bosom buddy, Apple. Two more tracks (6 minutes, 5 seconds of music) can be had from Apple's iTunes store. These tracks will appear in Apple's non-free AAC encoding, compressed and possibly DRM-laden (I'm not 'up to date' with Apple's DRM policy. I know they announced dropping DRM completely for music at some point in the past, but I wouldn't be surprised if they backtracked on that. At least for special friends like Disney).

Moving right along, Amazon's digital music store gets a taste. A relatively tiny taste, as it were, of one track (2 minutes, 41 seconds). This lone track will be delivered to you in stunningly compressed, non-free MP3 format! DRM? Not here, fortunately.

The last stop on the Tron Legacy soundtrack pain train is Nokia's Ovi store. Who's what what? Yeah...thought so. Nokia. Finnish mobile handset maker. They're pretty big over across the pond. In the States? Not so much. They hold 8% of the mobile phone market share here. Even if you are one of the approximately 22.8 million Americans with a Nokia handset, I'm betting there's at least a fifty-fifty chance you haven't created an Ovi account. This is what you will need to grab the last gem and add another 2 minutes, 50 seconds to your collection. Oh...and you'll need their proprietary software to boot! Need I mention that this track is compressed? If I were a betting man, I'd bet on DRM, too.

Wow. That's pretty close to a metric shitton of effort to put forth to GIVE A COMPANY YOUR MONEY. Let's recap, shall we? You order a CD from a foreign country and wait at least ten days (as of this writing, Amazon UK estimates delivery w/ standard shipping between January 6th and 10th; the soundtrack is out of stock with Amazon Canada). To retrieve the other 24 minutes and 29 seconds of soundtrack, you make a purchase with three merchants, possibly creating as many accounts and installing as many as two pieces of software.

Seriously? File this one under "cruel and unusual punishment." Thanks, Disney, for punishing your potential customers who would gladly open their wallets a bit wider to get all 31 of these tracks in a single, two-disc set. In an uncompressed format. Without DRM.

As a side note, most things in the natural world take the path of least resistance: lightning, water, and humans who have heard of The Pirate Bay. Downloading a single torrent provides a whole hell of a lot less resistance than Disney's boondoggle. And in minutes rather than weeks. Way to encourage piracy, geniuses.

End of line.

Filed under: technology 2 Comments
2Sep/100

Twice the bits, twice the time

Posted by sarumont

When I built a zfs-based RAID array last year, I switched to the 64-bit build of FreeBSD. Knowing that 64-bit software is inherently evil, I set up a 32-bit jail for coding. At some point, I was dragged into doing UI work requiring GWT's devmode. I took the easy way out - abandoning my jail. Comparing compile-time numbers to a new machine a colleague built yesterday, it occurred to me to try a 32-bit JDK.

My work revolves around our codebase and our customer's codebase. Both are architected in a very minimalistic fashion, using ant to build. There are a couple xjc tasks, some javac annotation preprocessing and some code generation, but the bulk of the build process is pure javac. I'll shutup now; you're here for the numbers. Here they are ('real' time from time:

64-bit OpenJDK 6 b20, FreeBSD 8.1-STABLE

Codebase 0:


18.36 real
17.59 real
17.57 real
18.60 real
17.97 real
------------
(avg: 18.02)

Codebase 1:


23.06 real
23.37 real
22.34 real
22.64 real
22.34 real
------------
(avg: 22.75)

32-bit OpenJDK 6 b20, FreeBSD 8.1-STABLE

Codebase 0:


12.99 real
13.64 real
12.77 real
12.38 real
12.42 real
------------
(avg: 12.84)

Codebase 1:


15.35 real
15.44 real
15.20 real
15.17 real
15.14 real
------------
(avg: 15.26)

Compiling with a 64-bit javac is 140% and 149% SLOWER than the 32-bit javac. Roughly 1.5x! How much time have I wasted compiling with a 64-bit javac during the last year? I don't want to approximate that math. If I worked in a real office, I'd have a reason for compiling with 64-bit javac. Needless to say, JAVA_HOME has already been changed.

If there's a more general point to be had here, it would be that those extra bits can indeed affect performance in a human-perceivable and negative fashion. Always be sure to evaluate your tools to
determine where you are wasting time.

15Mar/102

Backups with ZFS and Amazon’s S3

Posted by sarumont

Amazon's S3 provides unlimited, relatively cheap storage, accessible via HTTP (using REST/SOAP). This is ideal for personal offsite backups. The barrier to entry is low as are the costs.

Sun's ZFS turns everything you know about managing filesystems on its head. When I recently upgraded my storage array, I decided to load my machine with extra RAM for ZFS rather than buying a dedicated RAID card. One of the best features of ZFS is the low-cost snapshots. You can snapshot a filesystem, and said snapshot will not take any space on disk until the original is modified. To put it another way, the snapshots only store the (block-level) diffs.

The icing on the snapshot cake is the ability to send a snapshot as a stream. This can be piped over the network (i.e. ssh) or simply output to a file (then bzipped and uploaded to S3!). ZFS also allows you to send a differential of two snapshots.

To put this into action, I have written a script which will create a snapshot, bzip it, encrypt it and upload it to S3. I use a threshold to determine whether I should upload the full or incremental to save space/bandwidth (and time, as my cable upstream isn't that great...). To interface with S3, I'm using s3tools.

I realized afterward that s3tools have GPG encryption built in, but I think it is simpler to use openssl and a passphrase for this use-case.
Here's the script:

#!/bin/sh
#
# Copyright 2010 Richard Kolkovich. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without modification, are
# permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice, this list of
# conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice, this list
# of conditions and the following disclaimer in the documentation and/or other materials
# provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY Richard Kolkovich ``AS IS'' AND ANY EXPRESS OR IMPLIED
# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL Richard Kolkovich OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# The views and conclusions contained in the software and documentation are those of the
# authors and should not be interpreted as representing official policies, either expressed
# or implied, of Richard Kolkovich.
#
ZFS=/sbin/zfs
BZIP=/usr/bin/bzip2
OPENSSL=/usr/bin/openssl
MD5=/sbin/md5
S3CMD=/opt/s3cmd/s3cmd
BACKUP_DIR=/tank/backup
TEMP=$BACKUP_DIR/tmp
PASSFILE=$BACKUP_DIR/.password
if ! test -d $TEMP; then
    mkdir $TEMP
fi
# Backup a given zfs store
# arguments: name, zfs fs, threshold (0 to always use incremental), S3 bucket
backup() {
    NAME=$1
    FS=$2
    THRESHOLD=$3
    BUCKET=$4

    # incremental
    SNAP=$FS@incremental
    SUFFIX=incremental
    $ZFS destroy $SNAP
    $ZFS snapshot $SNAP
    $ZFS send -i $FS@full $SNAP > $TEMP/$NAME
    export BLOCKSIZE=1024
    SIZE=$(( `du $TEMP/$NAME | awk '{print $1}'` * 1024 ))

    if test $THRESHOLD -eq 0 || test $SIZE -gt $THRESHOLD; then
        rm $TEMP/$NAME
        $ZFS destroy $SNAP
        SNAP=$FS@full
        SUFFIX=full
        $ZFS destroy $SNAP
        $ZFS snapshot $SNAP
        $ZFS send $SNAP > $TEMP/$NAME

        # a new full invalidates old incrementals
        $S3CMD del s3://$BUCKET/$NAME-incremental.bak
        $S3CMD del s3://$BUCKET/$NAME-incremental.bak.md5
    fi

    # compress
    $BZIP $TEMP/$NAME
    FILE=$TEMP/$NAME.bz2

    # encrypt snapshot
    $OPENSSL enc -aes-256-cbc -salt -pass file:$PASSFILE -in $FILE -out $FILE.bak
    rm $FILE
    FILE=$FILE.bak
    $MD5 $FILE > $FILE.md5

    # send snapshot to S3
    $S3CMD put $FILE s3://$BUCKET/$NAME-$SUFFIX.bak
    $S3CMD put $FILE.md5 s3://$BUCKET/$NAME-$SUFFIX.bak.md5

    # clean up
    rm $FILE
    mv $FILE.md5 $BACKUP_DIR/$NAME.md5
}

And I call backup() thusly:
backup "private" "tank/private" 52428800 "my.backup.bucket"

Filed under: coding, technology 2 Comments
2Dec/091

Done with Seagate

Posted by sarumont

I've been a proponent of Seagate drives since somewhere around my second or third PC build. I think my first "high capacity" (13.2GB) drive was a Seagate; I then went the Deathstar route with a 40GB Hitachi/IBM followed by an 80 (which, apropos, arrived on the day the 40 died). After that, I actually thought about brand before buying hard drives. I think I stepped up with Seagate in the following order: 160GB, 320GB and dual 400GB drives. When my 400GB mirror became full, I decided to upgrade once again.

If it's worth building, it's worth over-engineering the piss out of. I decided to roll with 4x1.5TB drives in a ZFS raidz pool (RAID5-equivalent capacity). I knew that initial shipments in Seagate's 1.5TB 7200.11 series had firmware issues aplenty; these had been resolved by the time I decided to build this array (early August 2009). I was doubly re-assured by the drive's placement on the Maximum PC Best of the Best list (where it reigned, I believe, until the recent arrival of the WD 2TB Caviar Black). I pulled the trigger on August 4.

After basking in the glow of 4.5 (4.0, effective)TB of empty space and the awesomeness that is zfs, I moved the contents of my 400GB array over. Everything went as smoothly as could be expected. About two days after this, however, I awoke to a S.M.A.R.T. error on one drive accompanied by its subsequent detachment from my system. I quickly started the RMA process at Newegg only to have a second drive begin exhibiting the Click of Death (before I had shipped the other drive back). ZFS's raidz can tolerate one drive failure, so I immediately shut my machine down to stymie the impending doom of the second failing drive. I amended my RMA to be a refund for two drives and ordered two new drives with expedited shipping (makeshift advanced replacement). With my array once again happy, I continued on my merry way.

Before October's passing, it decided to leave me a gift: the dreaded Click of Death had begun on another drive. Being out of Newegg's 30-day zone, I RMA'd the drive through Seagate. The drive was still accessible, but since access was painful, I shut my machine down. I paid the $19.99 for an advanced replacement and return shipping label, reslivered the new drive into the array and was off to the races (again). On a side note here, I decided that I would sell my (original) Raptor when I built this array, assuming it would be fast enough for OS/applications.

Fast-forward to yesterday. December has given me an early Christmas present: ANOTHER FAILING DRIVE! As I wondered why my machine was running sporadically slow yesterday, I discovered while refilling my coffee once that the Click of Death had once again visited me (my new headphones are designed to block out noise, so I didn't hear it until taking them off...). I did my due diligence and ran the long test provided by Seatools; this is just a bootable S.M.A.R.T. agent. The long test is the same as the S.M.A.R.T. long offline test (which would not complete using smartmontools, as the drive would disconnect after extremely long seek delays). This morning, I called Seagate Warranty Support, determined to at the very least not pay $19.99 for an advanced replacement AGAIN.

I initially suggested to the customer service representative that Seagate refund my money for all four drives. I would be much happier buying four new drives from a different manufacturer, such as the WD Caviar Black. Obviously, this suggestion was dismissed. I battled this rep for a good 20 minutes to get the advanced replacement with no charge. Upon requesting a supervisor, she told me her supervisor was on another call but suggested I talk to tech support before they could approve my request. I wasted time talking to a helpful tech support rep. who agreed that any clicking warrants drive replacement. He made his notes on my account and transferred me back to customer service.

I was now speaking to a different rep. than I had initially. She read the notes on my account, again presented my two options (I ship, they ship or advanced replacement) without waiving the $19.99 fee. She told me she could not approve that, so I waited in the queue for her supervisor.

The supervisor was pleasant but worthless. He danced to the same tune - they could not send me an advanced replacement for free. I don't typically let my emotions get the better of me, so I was cool-headed but forceful with him. At this point, I didn't care about Seagate's obligations or policies. I was a loyal customer who had experienced a 57% failure rate in four months of ownership of a drive with an advertised annual failure rate of 0.34%; I wanted him to go out of his way to make me happy. The best he said he could do was to send me a shipping label to ship my drive in, free of charge. That would result in at least a week turnaround which is, to me, unacceptable.

Once I had become exhausted with attempting to garner a concession based on sympathy for my (hopefully) unusual experience with faulty drives, I turned to logic. I asked this supervisor what the difference was between giving me the advanced replacement for free and covering my shipping costs. He stated the $19.99 was their cost for the packaging materials and shipping via UPS, and that's "just how it is". I asked him to explain, logically, how it could be any different. He's offering to cover shipping both ways. They are going to provide their own packaging materials even if I send my drive first. He did not refute these points, but he still could not explain why the options were really any different cost-wise.

They aren't. The only difference between these options is shipping order. In both cases, Seagate was going to pay for shipping from them to me, shipping from me to them and packaging materials for the replacement drive. As this gentleman did not have a supervisor, I am now waiting 24-48h for the department he reports to (Seagate Global Customer Service or something...) to call me back.

I would also like to note that Newegg's packaging and UPS's shipping are suspect here. I was fairly pleased with the packaging - each drive was individually wrapped in a copious amount of bubble wrap and the box was filled with packaging peanuts, but various sources on the Interweb swear against Newegg when it comes to magnetic hard drives. I've seen this same dogma applied to UPS shipping, too. In the future, I will attempt to avoid both these potential pitfalls. My friend and colleague Stephen Shelton ordered four of the same drive from Dell a week before I did. They came in foam packaging designed for four hard drives (just like the first replacement I received from Seagate); he has had no problems with his drives. At the same time, "normal" shipping abuse should be considered by manufacturers - how else are these drives going to get to end users?

I'm beyond the realm of logic now. I've spent more time than the $19.99 is worth between being on the phone and not having my workstation running. Now, I fight a battle of principle. I was, even with this failure, willing to place blame at Newegg and/or UPS. Seagate has thus far shown me they don't care enough about retaining me as a customer and proponent of their products to concede a bullshit $19.99 fee. For this, they will lose my business in the future.

Filed under: technology 1 Comment
9Nov/090

Fun with Asterisk

Posted by sarumont

I spend approximately 10-15% of my time wearing the IT hat for both my company and the company whom we are currently consulting. We are running FreePBX for both companies' phone systems. Our setups are relatively simplistic, so I have had little problems managing everything. The first real hurdle I ran into was the inability of Asterisk to allow the same extension to log on from multiple devices. I'm pretty sure this is not a SIP restriction, but I may be wrong. Either way, my goal is not to bash FreePBX or Asterisk.

Initially, I over-engineered a solution to this problem while simultaneously making it more difficult to administer and use. The setup required a ring group for each user's extension. This ring group contained an extension for each phone the user needed. To have a common voicemail box, there was yet another extension for voicemail. Convoluted? Yes, it was. It did work, however, for our use-case. We have three employees, and only one of us needed a multi-homed extension at the time. I set it up and forgot about it.

More recently, I set up another FreePBX system for our customer. At this point, I discovered the Follow-Me module. The original point of using this module was to allow calls to ring through a user's mobile phone. I also adapted my previous abomination to use Follow-Me, making for a much simpler setup. It was simpler, yes, but it was still a bit convoluted.

This system worked well until I was tasked with on-demand call recording. I determined how to enable this in a jiffy with some quick Googling. The problem? Access to these recordings was tied to the extension that initiated the recording.

To solve this, I re-visited "deviceanduser" mode in FreePBX. This is the route I should have taken in my earlier ventures - it decouples devices and users. A device can register via SIP to the server, and a user represents an extension and voicemail box. This can be enabled by setting AMPEXTENSIONS=deviceanduser in /etc/amportal.conf. Be sure to comment out the existing setting.

To enable on-demand recording, add 'w' to the dial command string and 'W' to the outbound dial command string. These settings are found in General Settings in the FreePBX UI.

Add the following to /etc/asterisk/features_general_custom.conf:

featuredigittimeout=3000
courtesytone=beep

This makes sure the PBX will register your keypress (waiting for up to 3 seconds) and beeps to confirm. Feel free to disable the beep.

To enable access to the recordings from the ARI:

ln -s /var/spool/asterisk/monitor/ /var/www/html/recordings/monitor
yum install sox

Finally, tie everything to the user. For every device a user has, add the user's extension in the 'accountcode' field on the device.
Edit /var/www/html/recordings/modules/callmonitor.module. Add:

OR accountcode = '" . $_SESSION['ari_user']['extension'] . "'

After the line:

OR dst = '" . $_SESSION['ari_user']['extension'] . "'

On my FreePBX, this was at line 614.

Now, all calls recorded from any device for a given user will show up in that user's ARI for download.

Filed under: technology No Comments
11Nov/081

When IT goes awry

Posted by sarumont

Information technology (IT) is a field born of increasing complexity and choice in computer technology. Before computer science was considered a discipline, physicists and mathematicians programmed computers. Before there was IT, computer scientists managed hardware and software; they even built their own hardware at some point. Now, IT is a loosely defined discipline which ends up, in the business world, governing and managing choices having to do with technology.

According to Merriam-Webster, technology is the practical application of knowledge especially in a particular area. Throughout history, technology has been incorporated into society as its usefulness becomes apparent. The plow made farming much easier, so farmers began using it. The plow evolved, becoming better with each iteration. The plow is relatively simple next to a computer, but its path to ubiquity is one the computer and its ilk are following.

Following the plow example, who is focused on the plow? The farmer most certainly is not. As long as the plow does its job, the farmer could care less about it. His goal is to produce as much as he can with the highest possible efficiency. The plow is a tool. It enables him to farm and increases his efficiency, thus providing value. The farmer's customers don't care about the plow. They want to buy or barter food from the farmer. How he produced the food is of no concern to them. This leaves the plow manufacturer to care about the plow. The plow manufacturer thrives by selling plows; she cares very much about the plow. But does she use plows to make other plows?

When a company reaches a certain size, the task of managing technology requires a dedicated resource; the IT department is born. This size varies from company to company, depending on that company's specific needs, but the underlying trend is there. We'll assume here that the company in question is Spacely Sprockets, so they do not, in fact, sell any form of IT-related services.

What is the role of technology at Spacely Sprockets? It is a tool. Technology enables them to more efficiently and effectively produce sprockets. In this case, the best IT department is an IT department who is never seen or heard from. Ideally, Spacely's IT department has configured their systems to be completely invisible. Computers never crash, phones never disconnect and printers never make archaic requests such as "PC Load Letter". Sound too good to be true? It is. Though this level of perfection can probably never be attained, it can be approached.

On this sliding scale of IT perfection, there is a point where IT becomes inhibitory of the company's business. I fear many companies in the world are at or below this point. Why? I believe incompetence and arrogance account for most cases.

Incompetence (or its brother, ignorance) can cause solutions to be implemented poorly or not at all. Part of the job of an IT department is to recognize where technology could improve business in any way. If no one in the IT department is able to recognize the need for a given technology, it's unlikely anyone will. These flaws can also yield half-baked policies which end up inhibiting production.

Arrogance is the lower volume offense here, but it can be just as damaging. One case of this type of behavior is the technique of building a fortress. Ever known someone whose job security was that no one knew how to manage what they had built? That's a fortress. Did this person build the fortress with the sole intention of keeping people out? Not necessarily. There are circumstances where some constrained resource (people or time, usually) can limit transparency. I'd say these situations should at least yield a wiki page in the corporate intranet about the system (don't have a corporate intranet/wiki? see the previous paragraph...). Arrogance, in some form, can once again yield inhibitory policies. These are typically knowingly inhibitory, meant to draw attention to the power held by the IT department.

What can be done about these problems? Well, to some extent, these are character flaws. Those in charge of hiring can screen the extreme versions of these people very quickly. Once they are in power, though, those in charge of them must have the ability and willingness to deal with potential problems as they are identified. The worst case scenario has an arrogant IT department gone rampant with a CEO who will not lay down the law. IT is just another department. If your sales team isn't producing, you take action. If your IT department isn't producing, you need to take action. IT is an integral part of your business but should not control it.

Filed under: technology 1 Comment