Confronting Change

In earlier posts, I’ve discussed working with the Hydra spectrograph on the WIYN Telescope at Kitt Peak National Observatory. It’s a piece of instrumentation that allows astronomers to observe up to 90 objects simultaneously during one telescope pointing. The WIYN telescope can see one square degree of the sky at a time. Hydra has a set of optical “buttons” which can be placed on a metal plate at the position where an object will be in the field of view. These buttons send data to the spectrograph two floors below via fiber optic lines. The device that places these buttons is a robot, which resembles the claw from crane machine. It picks up each button from a parked position and puts it where it belongs.

Sticker on Hydra’s side

This type of multi-object spectrograph is somewhat common now, but Hydra was one of the first of these types of spectrographs built in the early 1990s. In fact, the instrument is older than the WIYN telescope and I helped to commission the instrument in its first version at the Mayall 4-meter. Precision machines like the Hydra gripper suffer a lot of wear and tear in 30 years, so a little over a year ago, Hydra began an upgrade process. The engineers worked with Prod Design and Analysis in El Paso, Texas to upgrade the gripper with new technology. The folks at Prod were struck by how much the gripper resembled a crane machine’s claw and added the sticker you see in this first photo.

Hydra Control Rack

As you might imagine, I’ve become quite familiar with Hydra, working with it off and on for almost thirty years. However, making major changes to the instrument meant learning new procedures. The original Hydra gripper was an analog device that “knew” where buttons were by counting steps using a device called an encoder and then remembering those steps to go back and pick the button up again. If there was some electrical noise or a mechanical problem that caused the count to be less precise, it could miss picking up the button. The upgraded Hydra also uses encoder counts, but it has added new programmable logic controllers and video analysis. It goes to the button, takes a picture, analyzes that pictures and then adjusts it’s position, if needed, to get the button. All this new logic has meant that we have a new control cabinet in the telescope to house electronics. The hope is that this will make positioning more accurate. However, it has meant learning new ways to monitor the process of the gripper at work and recovering the gripper if problems occurred.

This past week was one of the first times I’ve had to use the upgraded Hydra without one of the engineers on hand in case problems occurred. I made sure to review the manual, because it had been several months since my introduction to the upgrades and I reviewed the troubleshooting procedures. The first night of observing started out quite smooth. We had four field setups without any problem whatsoever. Then on the fifth field configuration, a heart-stopping problem occurred. The Hydra control program crashed while the gripper was carrying a button to a new position. Of course, these buttons and their attached fibers are all very delicate and if Hydra forgets the button’s position, there’s a risk that the attached fibers could get tangled. This could prove to be a very expensive problem.

In the old days, a problem like this automatically meant going into the dome, which is a cold proposition on a winter night! Then with a long stick , carefully reaching into the instrument and releasing the button from the gripper jaws, which allow you to open the instrument. After that, you’d have to manually place this fragile, optically sensitive button and fiber back into its stow position. In short, it’s a delicate procedure to do when you’re cold and on a lift in a dark dome!

The new Hydra Handpaddle

Now we have a handpaddle, which lets us talk to the gripper directly. I went to the Troubleshooting guide, refreshed my memory and followed the instructions. For the most part they worked. I was able to control the gripper and set the button down in a controlled way. Unfortunately, we’re still working out some bugs, so I still had to open Hydra and check the button’s real position, but it was a much quicker, safer operation.

Confronting change in procedures that had grown familiar and routine was definitely scary and a little challenging. The things that helped me manage my discomfort were focusing on the familiar parts of the routine, reviewing the new procedures before starting work, and then when a problem did happen, I took a deep breath and used the instructions and my experience to solve the problem the best I could. This past week, the upgraded Hydra moved from a machine that presented me with a little anxiety to a machine I look forward to understanding better.


My friend Darla Hallmark sells buttons that say, “The problem with troubleshooting is that trouble often shoots back.” In my job operating telescopes at Kitt Peak National Observatory, I often get to see the truth of that statement. Here I am in my natural habitat at the control station of the WIYN telescope.

My actual title at Kitt Peak is “Senior Observing Associate” and my job is more than being a telescope driver. I see myself as the person whose job it is to make sure the astronomers who use the telescopes actually get the data they hope to obtain. At night, especially at the WIYN telescope, I’m often the only person in the building. It’s quite common for observers using the telescope to control the cameras over the internet and talk to me all night on a Skype connection.

The observatory has a daytime staff of engineers, electricians, mechanics and more. Most of them are tucked snug in their beds when I’m working through the night. So, if something goes wrong, I’m the guy who has to fix it, or find a workaround until the next day when the daytime staff returns to work. I think its a real testament to the design and maintenance of the telescopes at Kitt Peak that serious problems don’t crop up all that often, but when they do, they can be a challenge.

We had one such problem this week at WIYN. We were using the Hydra spectrograph. Instead of an eyepiece or a camera looking directly at the sky, there is a metal plate. Fiber optics in magnetized housings sit on that metal plate and face the sky. A robot within Hydra can move those around so they’re in a position to capture light from distant objects. This week, each fiber was placed to catch light from galaxies approximately 11 billion light years away. As you can imagine, you need to place that fiber in just the right place to catch that tiny bit of light. This is what the inside of the Hydra spectrograph looks like. You can see the fibers on the left-hand side. The robot that moves the fibers is on the right.

The problem we had was that some of these fibers were missing the light. To confuse matters, not all the fibers were missing the light. We saw light from some galaxies. We saw light from all the stars that let us do fine corrections to our pointing on the sky. My first thought was that there was a calculation error and not all the fibers were being placed correctly. The astronomer looking at these galaxies checked and eliminated that possibility. Next, we used a camera on the robot to watch the fibers as they were being moved to see if they were being placed where we put them. The robot did just what it was supposed to do.

The final step in this procedure is that the metal plate on the left gets warped, because the telescope’s focal plane isn’t actually flat. We watched the fibers as the plate was warped. The fibers in the center “jumped.” That’s not supposed to happen. As of this writing, I’m not sure why warping the plate made some fibers jump but not others, but the obvious workaround is not to warp the plate. What this means is that some galaxies will be better focused than others, when we take data, but since we’re taking spectra, that’s not a showstopper. We just care that the light makes it down the fiber. Once that happens, the astronomer can see what elements exist in that galaxy and get information about how far away it is and how fast its moving. As the weeks goes on, that team of engineers and technicians will take the information I learned about the problem and work to find a solution.

If you enjoyed this behind-the-scenes look at my job operating telescopes, you might enjoy my novel, The Astronomer’s Crypt. It tells the story of ghosts, gangsters, astronomers, and a dangerous Apache spirit colliding at a New Mexico observatory on a dark and stormy night. You can learn more about the novel and watch a cool trailer at:

If spooky stories aren’t your thing, but you’ll be in Phoenix, Arizona on Thursday, May 9, you can join me at the next meeting of the Phoenix Astronomical Society, where I’ll be talking about the DESI project on the Mayall 4-meter telescope and sharing some behind the scenes photos of the installation. You can get more details about the meeting at:

The Robots Are Taking Over

I spent this past weekend at Bubonicon in Albuquerque, New Mexico. Yesterday, I moderated a panel called “Are Robots Still Scary? Danger Will Robinson!” As it turns out, I suggested this panel to the event organizers because it occurred to me that I work with a robot quite a bit at Kitt Peak National Observatory and it’s a far cry from the anthropomorphic robots of pulp sci fi and probably more irritating at times than scary. The robot I’ve spent most of my time with is called the Hydra gripper. The reason it can be irritating, is that I’m one of the guys who has to go fix it when it breaks down!


The gripper is on the right-hand side of the photo above. Its job is to pick up the fibers on the left side and position them on the plate so that they line up with objects on the sky. When the telescope is pointed at the target, light from the objects will go downstairs to a spectrograph, where it will be broken into a spectrum and projected on a camera. Astronomers can take that light and analyze it to understand the chemical composition of the objects they’re studying. These objects can range from stars, to galaxies, to nebulae. Some objects are nearby, others are among the most distant in the universe.

The reason for this post’s title, is that we’re about to get another robotic spectrograph at Kitt Peak. This one is called DESI (which stands for Dark Energy Spectrographic Instrument). Hydra allows us to take spectra of upwards of 200 objects at a time. DESI will let us take spectra of 5000 objects. You can read more about DESI in this press release from the Department of Energy’s Berkeley Lab. The goal of the instrument is to get spectra of all the known galaxies obtainable by the 4-meter in order to understand the phenomenon that’s been dubbed “dark energy.” Along the way, we’ll build an incredible database of spectra available to the entire astronomical community.

This week, I’ll be helping to test a prototype of DESI on the Mayall 4-meter telescope. The fibers of DESI are so closely packed that they aren’t moved around by a system like a gripper. Instead, each fiber is a little robot that turns on its own to optimize its position on the sky. Because of this project and the number of people it takes to get a project like this off the ground, DOE has helped to fund a new, larger control room for the Mayall. You can see the original on the left below. The new control room is on the right.

New-Old Control rooms

As it turns out, ghosts and stories of haunted observatories feature prominently in my forthcoming novel, The Astronomer’s Crypt. Our new control room is one of the “haunted” spaces in the Mayall. It used to be an old lounge and there were two rocking chairs that would sometimes be seen to rock on their own even when no one is in there. It remains to be seen if there are any ghosts, or if this motion was just due to sway of the building. If there are ghosts in the building, I hope they like company, because there’s going to be a lot of it in the coming years!