Confronting Change

In earlier posts, I’ve discussed working with the Hydra spectrograph on the WIYN Telescope at Kitt Peak National Observatory. It’s a piece of instrumentation that allows astronomers to observe up to 90 objects simultaneously during one telescope pointing. The WIYN telescope can see one square degree of the sky at a time. Hydra has a set of optical “buttons” which can be placed on a metal plate at the position where an object will be in the field of view. These buttons send data to the spectrograph two floors below via fiber optic lines. The device that places these buttons is a robot, which resembles the claw from crane machine. It picks up each button from a parked position and puts it where it belongs.

Sticker on Hydra’s side

This type of multi-object spectrograph is somewhat common now, but Hydra was one of the first of these types of spectrographs built in the early 1990s. In fact, the instrument is older than the WIYN telescope and I helped to commission the instrument in its first version at the Mayall 4-meter. Precision machines like the Hydra gripper suffer a lot of wear and tear in 30 years, so a little over a year ago, Hydra began an upgrade process. The engineers worked with Prod Design and Analysis in El Paso, Texas to upgrade the gripper with new technology. The folks at Prod were struck by how much the gripper resembled a crane machine’s claw and added the sticker you see in this first photo.

Hydra Control Rack

As you might imagine, I’ve become quite familiar with Hydra, working with it off and on for almost thirty years. However, making major changes to the instrument meant learning new procedures. The original Hydra gripper was an analog device that “knew” where buttons were by counting steps using a device called an encoder and then remembering those steps to go back and pick the button up again. If there was some electrical noise or a mechanical problem that caused the count to be less precise, it could miss picking up the button. The upgraded Hydra also uses encoder counts, but it has added new programmable logic controllers and video analysis. It goes to the button, takes a picture, analyzes that pictures and then adjusts it’s position, if needed, to get the button. All this new logic has meant that we have a new control cabinet in the telescope to house electronics. The hope is that this will make positioning more accurate. However, it has meant learning new ways to monitor the process of the gripper at work and recovering the gripper if problems occurred.

This past week was one of the first times I’ve had to use the upgraded Hydra without one of the engineers on hand in case problems occurred. I made sure to review the manual, because it had been several months since my introduction to the upgrades and I reviewed the troubleshooting procedures. The first night of observing started out quite smooth. We had four field setups without any problem whatsoever. Then on the fifth field configuration, a heart-stopping problem occurred. The Hydra control program crashed while the gripper was carrying a button to a new position. Of course, these buttons and their attached fibers are all very delicate and if Hydra forgets the button’s position, there’s a risk that the attached fibers could get tangled. This could prove to be a very expensive problem.

In the old days, a problem like this automatically meant going into the dome, which is a cold proposition on a winter night! Then with a long stick , carefully reaching into the instrument and releasing the button from the gripper jaws, which allow you to open the instrument. After that, you’d have to manually place this fragile, optically sensitive button and fiber back into its stow position. In short, it’s a delicate procedure to do when you’re cold and on a lift in a dark dome!

The new Hydra Handpaddle

Now we have a handpaddle, which lets us talk to the gripper directly. I went to the Troubleshooting guide, refreshed my memory and followed the instructions. For the most part they worked. I was able to control the gripper and set the button down in a controlled way. Unfortunately, we’re still working out some bugs, so I still had to open Hydra and check the button’s real position, but it was a much quicker, safer operation.

Confronting change in procedures that had grown familiar and routine was definitely scary and a little challenging. The things that helped me manage my discomfort were focusing on the familiar parts of the routine, reviewing the new procedures before starting work, and then when a problem did happen, I took a deep breath and used the instructions and my experience to solve the problem the best I could. This past week, the upgraded Hydra moved from a machine that presented me with a little anxiety to a machine I look forward to understanding better.