http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup

This is extremely cool, and means that I will most definately finally get that Open Solaris server up and running. 10 GB Ethernet with NFS is starting to look like a viable option instead of expanding our SAN. The cost of NICs and switch can be offset quickly by the cost expanding the FC-Switches and the cost of the SAN-software…

Some more discussion on the subject:

http://tech.slashdot.org/story/09/11/02/2117206/ZFS-Gets-Built-In-Deduplication?from=rss

Filipp over at Unflying Object has some ideas on integrating the authentication for the Wiki and a Forum on a Mac OS X Server:

http://unflyingobject.com/blog/posts/1014

Quite straightforward, but not working for me yet, probably due to my “golden triangle”-setup with our AD-server.

Oh, and Spooks must the best TV-show in a long, long time. Been watching this on the iPod in the bus while commuting. Just brilliant, and quite realistic. Quite similar to The Song Of Ice And Fire in that they seem pretty happy to kill off any of the leads at any time…

Archiving these Autodesk archives is boring and manual work. Each archive usually consists of two files: project_name and project_name_1.seg, and I just copy the name of the project to be the name of the backup-job, then choose the two files and run the job. And all this takes time because the GUI for BRU is not the most agile one available. So I decided to write a little script I could run, just giving the folder where the archives are as a parameter.

This is probably not very efficient, and there are many ways to make it better. But it save quite a lot of time for me.

#!/usr/bin/env bash
# 2009-11-03
# juso@iki.fi

PASSWD="passwd"
SERVER="bru.servername"

if [ -z "$1" ]; then
 echo usage: $0 directory
 exit
fi

folder=$1
cd $folder

for file in *_1.seg
do
 file2=${file%_1.seg}
 upload=`(echo ${PASSWD}
 sleep 5
 echo 'backup -j "'$file2'" -t "Full" -D "destination" -o "append" -v
["/localhost'$folder'/'$file2'", "/localhost'$folder'/'$file'"]'
 sleep 5
 exit) | bru-server.cmd username $SERVER`
 echo $upload
done

Some disclaimers: I have not actually had time to test the script exactly as is (because our Bru-server is quietly clogging away at the 25 backup jobs I fed to it a while ago), but it should work. It is also my very first stab at Bru-automation.

Adding a file called

~/Library/Preferences/nsmb.conf

and adding the following lines:

[default]
minauth=none

on the workstation fixed the problem.

All this after midnight while watching Liverpool beat Stoke. Thanks to Google and Ubuntu forums:

http://ubuntuforums.org/archive/index.php/t-917156.html ]]> http://www.thingamagic.net/jussi/2009/08/20/dreambox-and-the-zen-of-discovery/feed/ 0 http://www.thingamagic.net/jussi/2009/06/03/monitoring-which-folders-are-wasting-space/ http://www.thingamagic.net/jussi/2009/06/03/monitoring-which-folders-are-wasting-space/#comments Wed, 03 Jun 2009 11:46:14 +0000 Jussi Uosukainen http://www.thingamagic.net/jussi/?p=67 Most of my work seems to be freeing up space on the different shared HDs we have around the facility. After trying several  apps that show how folders use space graphically (GrandPerspective used to be my favourite) and testing different scripts, I have found out that Disk Inventory X suits my workflow the best. It allows me to graphically see where the space is used, and shows me when the folder in question has been created and modified. Helps in finding all those ancient DPX-sequences that have been forgotten in the miasma that is oour SAN.

After a couple of weeks testing, Nagios has worked very well. Alarms have been sent consistently and to the right addresses. Pretty soon after my first real Incinerator problem, I noticed the real problem with my setup: for each incident where the  incinerator nodes crashed, I got 8 separate emails. And after I fixed the issue, I got another 8 emails telling that the problem is fixed.

I thought I could work around the problem by creating a servicegroup with the incinerator nodes in it, and only enable alarms for the group as a whole. But you cannot assign alarms for servicegroups, only individual services. Nagios comes with check_cluster, but setting it up seemed like quite a bit of work, with wrappers etc.

The Solution

After some searching (and getting a nagios specific book, Nagios: System and Network Monitoring, 2nd Edition), I came across check_multi. It is a simple plugin that I installed on the Lustre mediaserver. I moved the commands that check the nodes from nrpe.cfg to a separate .cmd file, and added a new command into nrpe.cfg, that used the cmd file to run the check_multi command. Then I just added this as a nrpe command on the nagios server.

Now I get the status to Nagios neatly under one service, but I can still check the individual status of each node.

I have also been testing Cacti. My main interest is to get more data from switches and routers (especially since QLogic is asking 10000€ for the software to monitor their Fiberchannel switches with the newest firmware). I will write something about that later.

Lustre Frameserver from Nagios

We have been looking at setting up an open source monitoring solution at the office for quite some time (I remember having a discussion about Nagios on my first day at work), but looking at the Nagios docs made me think that setting up all those .cfg files was going to really suck, so I looked at alternatives.

Over the christmas holidays I installed Zenoss (http://www.zenoss.com/), mainly because it promised to crawl thru our networks and be a breeze to set up using the webGUI. It did do the crawling as promised, but setting anything further than the basic settings was really, _really_ painful. Sure, it was nice with the couple of windows machines it recognised immediately, but otherwise it was pretty useless.

It took me several months to finally bite the bullet, but last week I got around to finally installing Nagios at the office. After setting up the basic checks for the localhost ( a RHEL server we had reserved for this use) in a couple of hours, I was quickly feeling pretty proficient with all the different .cfg files, and started venturing into the unknown…

RAID-Chassis

Getting information from out ten-odd RAID-subsystems would be quite important. I first thought I had struck gold when I found check_promise_vtrak from http://www.consol.com/apple/nagios-plugins/check-promise-vtrak/ . Set up was a breeze, and I quickly got it working from the CLI. But the plugin refuses to function from Nagios, it only returns (null), which ends as a critical error and an email in my box. Probably a small fix is needed in the plugin itself, because plainly it is not returning anything readable by Nagios.

Mac OS X Servers

Using the basic plugins and nrpe (http://nagios.sourceforge.net/docs/1_0/addons.html) I was able to check all the basic data on our servers. I would like to monitor the actual services themselves (usually AFP and SMB, with some DNS and OD on some machines). As above, I thought check_osx_services (http://www.nagiosexchange.org/cgi-bin/page.cgi?g=Detailed%2F1497.html;d=1) was going to fix all my problems. Again I was foiled at the start: this plugin wouldn’t work properly even from the CLI.

Autodesk Lustre /w Incinerator

Since this is the system that is keeping me the busiest in normal times, I wanted Nagios to help me out here.

Lustre /w Incinerator is a complex system, consisting of a workstation, frameserver, 8 rendering nodes, an ethernet network for commands and an Infiniband network for moving those frames around. With so many moving parts, there are way too many failure points here. Lately we have had a lot of issues with the renderd service (that handles the rendering and contact with the server) crashing  on the nodes by itself. I have simple scripts that allow me to restart the service on all the nodes at once, and it takes just seconds to run. The difficult part is getting the info when the nodes have crashed. Installing the normal checking tools on the nodes was not an option for a couple of reasons. Firstly: I don’t want to have too many extra things installed on the nodes and secondly: the nodes are not actually on any other network that the incinerator network, so there is no access to them from the Nagios server.

I have installed nagios plugins and nrpe on the frameserver. These check the normal things on the server (root disk space, CPU loads, Processes etc.). I also created a specific check to handle Browsed, the process that handles serving the frames to the workstation and the nodes. After some searching I discovered a plugin called check_process_by_ssh (http://www.nagiosexchange.org/cgi-bin/page.cgi?g=Detailed%2F2013.html;d=1)

which allowed me to formulate a suitable nrpe command to execute (from nrpe.cfg on the frameserver):

command[check_node1]=/usr/local/nagios/libexec/check_process_by_ssh -H node1 renderd

I then added the check to my Linux definitions:

define service{
        use generic-service
        host_name frameserver
        service_description Incinerator Node 1
        check_command check_nrpe!check_node1
        }

This worked fine from the CLI, but nagios didn’t get thru and said the status was critical. After some thought I realized the problem: nagios executes all scripts as the user nagios, and I was doing my testing as root. The Nagios user sisntä have the needed SSH authentication settings, so I copied the needed file (id_rsa) to a suitable folder, and modified the command on the frameserver:

command[check_node1]=/usr/local/nagios/libexec/check_process_by_ssh -H node1 -k /usr/local/nagios/keys/id_rsa -u root renderd

Now the checks work without a hitch, and I get an email about the nodes being down before an operator has been wondering what is wrong for an hour…

Due to some illnesses in our support staff, I ended doing a big upgrade job short-handed. This is what I learned:

• When Autodesk support say they have shortened support hours, it also meant that they are quite understaffed, which reflected in both the reply times, and even the quality of some answers
• The Lustre upgrade scripts couldn’t handle the straight upgrade to 2009SP2 from 2007. Several things got broken on the way, like the Incinerator Manager website, which could no longer start or stop the nodes.
• The Lustre/Incinerator licensing scheme is quite difficult to understand at times, and you have to make sure your temporary license contains licenses for the nodes as well.
• It is hard to test systems you have no idea of how to use
• Working 17 hours shifts is not fun
• A good remote desktop & SSH remote connection to the office is very nice at times
• Always upgrade Autodesk-workstations from the local screen, else you will miss out on some essential settings, and get no warning
• writing bash scripts would be a nice skill to have, thanks Filipp
• Lustre 2009 is a lot more picky about settings. We used use our SAN with the frames mounted directly from workstation-SAN-mount, but in 2009 we need to use the frames over Infiniband from the Frameserver in order to get the nodes to work

http://www.red.com/epic_scarlet/

New cameras

I guess most people were happy with the announcement: 8 different cameras (”brains” and a whole slew

 of attachments, accessories and modules. The “brains” come in two different flavors, Scarlet is the smaller, “lowend” (called professional) unit, Epic is the “master professional”, high end unit. Both come with several models, with different lens mounts and sensor sizes.

If you have not acquainted yourself with RED-technology, look at the presentation I made earlier this month: http://www.thingamagic.net/jussi/?p=7 Here is a recap: the RED One is a digital cinema camera, that captures images in up to 4K resolution (4096×2580) using a proprietary “raw” format, called Redcode or r3d. The camera is very cheap compared to other professional digital motion picture cameras, with the price starting at 17500 USD.

At the same time I found this interesting article, Redfacts. It is especially interesting considering that it is the first piece to really heavily criticize the RED camera that I have read. Not being too familiar with any of the systems discussed in the paper (Red One or Sony F23), I will not go into details of the paper too heavily. It did however raise some valid points, which I would like to ponder about.

Redcode

Redcode is a proprietary compression technology. It is heavily compressed, with bitstreams of either 25MB/s or 36 MB/s. Redcode can contain a 2K stream at 113fps or 4K at 30fps. As a comparison a datastream of 4K in an uncompressed form (like the DPX-files we use for post production) would be around 885 MB/s, and this with 10bit color, not the 12bit Red captures. This means only 1/25th of the data is left in the Redcode file. In the still-camera world, a raw-file means that all the information from the sensor is put in the file. Red has somehow turned that around, and here “raw” means that they have discarded most of the data available.

Processing Redcode needs a very powerful computer. Some NLEs support the files directly, but most systems require the file to be conformed into a format that tools in the post-production workflow can actually use. Sony makes most of this processing in the camera (which is definitely one of the reasons there is such a huge difference in the cost of the camera). Putting the processing outside of the camera itself seems like a prudent choice, with all the cost benefits and versatility connected to normal desktop computers.

Resolution

4K is the current highend standard for Digital Cinema, with 2K (2048×1300) being the norm for post production work and display. Most DCI projectors that theaters are equipping themselves with are 2K, the HD screens customers are buying for their homes are nearly 2K (2K=2048×1080, HD=1920×1080). In fact, the biggest digital films to date (Star Wars episodes II & III) were filmed with HD resolution cameras.

Red One was the first camera with the capability to shoot 4K inside any reasonable budget. 4K has been more prone to come into the postproduction workflow originating from normal 35mm film via a scanner. Now we will see more and more projects done in 4K, because Red makes acquisition possible and relatively affordable. This is reasonable, since 4K can be handled by most of the vfx and grading solutions used in a modern workflow, and it can be printed to film using up-to-date film printers. My colleagues also say that there is a visible difference material printed with 4K to 2K when projected from 35mm film.

The new Red cameras vary in resolution from 3K to 28K. Yes, 28K. Everybody can probably guess that the 28K camera is meant for very specialist uses. It creates a image with 261 Megapixels (28000*9334).

I just can’t figure out why the normal Epic cameras that are designed for motionpicture work also have a resolution between 5K (5120×2700) and 9K(9334×7000). Yes, we can print 8K to film using our Celco Fury, but will any one see the difference between 4K and 8K? There will be few uses for 9K in it’s native resolution, which means that the images from these cameras will be downsampled into 4K or 2K resolutions for display. Sure, the end result will be an extra crispy image with hardly any noise, but I’m still left wondering…

Red also came out with a new, higher datastream versions of Redcode, Red42 (Scarlet) and Red225 (Epic). This means that we are cramming a huge amount of data from the sensor into a comparatively tiny space. This means there will be more compression on the images than there was before. Earlier I came up with 1:25 with Red36 and 4K, now we will see compression ratios more to the tune of 1:78 (6K Scarlet) or 1:35 (9K Epic).

8K resolution is not a real option for a postproduction workflow either. A single 10bit 8K picture in DPX is about 249MB, which means a data stream of almost 6 GB/s. We are not likely to see realtime 8K performance from any workstation and hard drive combination in the near future. This means that the data will be downscaled to 4K in order for it to make it thru the workflow.

Wouldn’t everyone be better off using the same compression techniques with smaller resolutions? Capturing 4K @ 225MB/s would only result in a compression ratio of about 1:4, which is a huge difference. Less compression is always better, no matter how good the compression is. Does it really make more sense to oversample the image by a magnitude of 4 and then discard 97% of that data?

I appreciate any comments and ideas.

RED – Camera

I was planning on concentrating on the RED part of the presentation, but decided I had to go thru the basics first, else there wouldn’t really be a point to the whole exercise.

The actual presentation will be on thursday the 13th of November.

/jussi