Lustre Frameserver from Nagios

We have been looking at setting up an open source monitoring solution at the office for quite some time (I remember having a discussion about Nagios on my first day at work), but looking at the Nagios docs made me think that setting up all those .cfg files was going to really suck, so I looked at alternatives.

Over the christmas holidays I installed Zenoss (http://www.zenoss.com/), mainly because it promised to crawl thru our networks and be a breeze to set up using the webGUI. It did do the crawling as promised, but setting anything further than the basic settings was really, _really_ painful. Sure, it was nice with the couple of windows machines it recognised immediately, but otherwise it was pretty useless.

It took me several months to finally bite the bullet, but last week I got around to finally installing Nagios at the office. After setting up the basic checks for the localhost ( a RHEL server we had reserved for this use) in a couple of hours, I was quickly feeling pretty proficient with all the different .cfg files, and started venturing into the unknown…

RAID-Chassis

Getting information from out ten-odd RAID-subsystems would be quite important. I first thought I had struck gold when I found check_promise_vtrak from http://www.consol.com/apple/nagios-plugins/check-promise-vtrak/ . Set up was a breeze, and I quickly got it working from the CLI. But the plugin refuses to function from Nagios, it only returns (null), which ends as a critical error and an email in my box. Probably a small fix is needed in the plugin itself, because plainly it is not returning anything readable by Nagios.

Mac OS X Servers

Using the basic plugins and nrpe (http://nagios.sourceforge.net/docs/1_0/addons.html) I was able to check all the basic data on our servers. I would like to monitor the actual services themselves (usually AFP and SMB, with some DNS and OD on some machines). As above, I thought check_osx_services (http://www.nagiosexchange.org/cgi-bin/page.cgi?g=Detailed%2F1497.html;d=1) was going to fix all my problems. Again I was foiled at the start: this plugin wouldn’t work properly even from the CLI.

Autodesk Lustre /w Incinerator

Since this is the system that is keeping me the busiest in normal times, I wanted Nagios to help me out here.

Lustre /w Incinerator is a complex system, consisting of a workstation, frameserver, 8 rendering nodes, an ethernet network for commands and an Infiniband network for moving those frames around. With so many moving parts, there are way too many failure points here. Lately we have had a lot of issues with the renderd service (that handles the rendering and contact with the server) crashing  on the nodes by itself. I have simple scripts that allow me to restart the service on all the nodes at once, and it takes just seconds to run. The difficult part is getting the info when the nodes have crashed. Installing the normal checking tools on the nodes was not an option for a couple of reasons. Firstly: I don’t want to have too many extra things installed on the nodes and secondly: the nodes are not actually on any other network that the incinerator network, so there is no access to them from the Nagios server.

I have installed nagios plugins and nrpe on the frameserver. These check the normal things on the server (root disk space, CPU loads, Processes etc.). I also created a specific check to handle Browsed, the process that handles serving the frames to the workstation and the nodes. After some searching I discovered a plugin called check_process_by_ssh (http://www.nagiosexchange.org/cgi-bin/page.cgi?g=Detailed%2F2013.html;d=1)

which allowed me to formulate a suitable nrpe command to execute (from nrpe.cfg on the frameserver):

command[check_node1]=/usr/local/nagios/libexec/check_process_by_ssh -H node1 renderd

I then added the check to my Linux definitions:

define service{
        use generic-service
        host_name frameserver
        service_description Incinerator Node 1
        check_command check_nrpe!check_node1
        }

This worked fine from the CLI, but nagios didn’t get thru and said the status was critical. After some thought I realized the problem: nagios executes all scripts as the user nagios, and I was doing my testing as root. The Nagios user sisntä have the needed SSH authentication settings, so I copied the needed file (id_rsa) to a suitable folder, and modified the command on the frameserver:

command[check_node1]=/usr/local/nagios/libexec/check_process_by_ssh -H node1 -k /usr/local/nagios/keys/id_rsa -u root renderd

Now the checks work without a hitch, and I get an email about the nodes being down before an operator has been wondering what is wrong for an hour…

Due to some illnesses in our support staff, I ended doing a big upgrade job short-handed. This is what I learned:

• When Autodesk support say they have shortened support hours, it also meant that they are quite understaffed, which reflected in both the reply times, and even the quality of some answers
• The Lustre upgrade scripts couldn’t handle the straight upgrade to 2009SP2 from 2007. Several things got broken on the way, like the Incinerator Manager website, which could no longer start or stop the nodes.
• The Lustre/Incinerator licensing scheme is quite difficult to understand at times, and you have to make sure your temporary license contains licenses for the nodes as well.
• It is hard to test systems you have no idea of how to use
• Working 17 hours shifts is not fun
• A good remote desktop & SSH remote connection to the office is very nice at times
• Always upgrade Autodesk-workstations from the local screen, else you will miss out on some essential settings, and get no warning
• writing bash scripts would be a nice skill to have, thanks Filipp
• Lustre 2009 is a lot more picky about settings. We used use our SAN with the frames mounted directly from workstation-SAN-mount, but in 2009 we need to use the frames over Infiniband from the Frameserver in order to get the nodes to work

http://www.red.com/epic_scarlet/

New cameras

I guess most people were happy with the announcement: 8 different cameras (”brains” and a whole slew

 of attachments, accessories and modules. The “brains” come in two different flavors, Scarlet is the smaller, “lowend” (called professional) unit, Epic is the “master professional”, high end unit. Both come with several models, with different lens mounts and sensor sizes.

If you have not acquainted yourself with RED-technology, look at the presentation I made earlier this month: http://www.thingamagic.net/jussi/?p=7 Here is a recap: the RED One is a digital cinema camera, that captures images in up to 4K resolution (4096×2580) using a proprietary “raw” format, called Redcode or r3d. The camera is very cheap compared to other professional digital motion picture cameras, with the price starting at 17500 USD.

At the same time I found this interesting article, Redfacts. It is especially interesting considering that it is the first piece to really heavily criticize the RED camera that I have read. Not being too familiar with any of the systems discussed in the paper (Red One or Sony F23), I will not go into details of the paper too heavily. It did however raise some valid points, which I would like to ponder about.

Redcode

Redcode is a proprietary compression technology. It is heavily compressed, with bitstreams of either 25MB/s or 36 MB/s. Redcode can contain a 2K stream at 113fps or 4K at 30fps. As a comparison a datastream of 4K in an uncompressed form (like the DPX-files we use for post production) would be around 885 MB/s, and this with 10bit color, not the 12bit Red captures. This means only 1/25th of the data is left in the Redcode file. In the still-camera world, a raw-file means that all the information from the sensor is put in the file. Red has somehow turned that around, and here “raw” means that they have discarded most of the data available.

Processing Redcode needs a very powerful computer. Some NLEs support the files directly, but most systems require the file to be conformed into a format that tools in the post-production workflow can actually use. Sony makes most of this processing in the camera (which is definitely one of the reasons there is such a huge difference in the cost of the camera). Putting the processing outside of the camera itself seems like a prudent choice, with all the cost benefits and versatility connected to normal desktop computers.

Resolution

4K is the current highend standard for Digital Cinema, with 2K (2048×1300) being the norm for post production work and display. Most DCI projectors that theaters are equipping themselves with are 2K, the HD screens customers are buying for their homes are nearly 2K (2K=2048×1080, HD=1920×1080). In fact, the biggest digital films to date (Star Wars episodes II & III) were filmed with HD resolution cameras.

Red One was the first camera with the capability to shoot 4K inside any reasonable budget. 4K has been more prone to come into the postproduction workflow originating from normal 35mm film via a scanner. Now we will see more and more projects done in 4K, because Red makes acquisition possible and relatively affordable. This is reasonable, since 4K can be handled by most of the vfx and grading solutions used in a modern workflow, and it can be printed to film using up-to-date film printers. My colleagues also say that there is a visible difference material printed with 4K to 2K when projected from 35mm film.

The new Red cameras vary in resolution from 3K to 28K. Yes, 28K. Everybody can probably guess that the 28K camera is meant for very specialist uses. It creates a image with 261 Megapixels (28000*9334).

I just can’t figure out why the normal Epic cameras that are designed for motionpicture work also have a resolution between 5K (5120×2700) and 9K(9334×7000). Yes, we can print 8K to film using our Celco Fury, but will any one see the difference between 4K and 8K? There will be few uses for 9K in it’s native resolution, which means that the images from these cameras will be downsampled into 4K or 2K resolutions for display. Sure, the end result will be an extra crispy image with hardly any noise, but I’m still left wondering…

Red also came out with a new, higher datastream versions of Redcode, Red42 (Scarlet) and Red225 (Epic). This means that we are cramming a huge amount of data from the sensor into a comparatively tiny space. This means there will be more compression on the images than there was before. Earlier I came up with 1:25 with Red36 and 4K, now we will see compression ratios more to the tune of 1:78 (6K Scarlet) or 1:35 (9K Epic).

8K resolution is not a real option for a postproduction workflow either. A single 10bit 8K picture in DPX is about 249MB, which means a data stream of almost 6 GB/s. We are not likely to see realtime 8K performance from any workstation and hard drive combination in the near future. This means that the data will be downscaled to 4K in order for it to make it thru the workflow.

Wouldn’t everyone be better off using the same compression techniques with smaller resolutions? Capturing 4K @ 225MB/s would only result in a compression ratio of about 1:4, which is a huge difference. Less compression is always better, no matter how good the compression is. Does it really make more sense to oversample the image by a magnitude of 4 and then discard 97% of that data?

I appreciate any comments and ideas.

RED – Camera

I was planning on concentrating on the RED part of the presentation, but decided I had to go thru the basics first, else there wouldn’t really be a point to the whole exercise.

The actual presentation will be on thursday the 13th of November.

/jussi