'Dirty hack' restores Cluster mission from near loss

Since a pair of spectacular dual launches in 2000, the four Cluster satellites have been orbiting Earth in tightly controlled formation. Each of the 550 kg satellites carries an identical payload to investigate Earth's space environment and its interaction with the solar wind -- the stream of charged particles pouring out from the Sun.

Among each satellite's 11 instruments, five comprise the Wave Experiment Consortium (WEC), which makes important measurements of electrical and magnetic fields. All four sensors must work together to make carefully orchestrated observations -- the loss of any one could seriously affect the unique 'four-satellite science' delivered by the mission.

On 5 March, the WEC package on Cluster's number 3 satellite, Samba, failed to switch on. Ground controllers at ESA's European Space Operations Centre, in Darmstadt, Germany, immediately triggered a series of standard recovery procedures, none of which succeeded.

Even worse, no status information could be coaxed out of the instruments.

Dangerous scenario for orbiting satellite

"With no status data and no response from the instrument, we suspected either that the device's five power switches were locked closed or a failure caused by an electrical short circuit, one of the most dangerous faults on any satellite," said ESA's Jürgen Volpp, Cluster operations manager.

Over the next several weeks, working closely with the satellites' builder, the WEC scientists and manufacturer, and other ESA teams, the Cluster control team diagnosed the problem, eventually making use of some onboard software that had been dormant since just after launch over 10 years ago.

The result ruled out a short circuit and pointed an accusing finger at the five power switches being locked in the 'closed' position.

Tests in 1995 had simulated what might happen if three of the five switches locked close, but no one ever considered how to recover from all five being locked -- such a situation had not been deemed possible.

Armed with this information and a great deal of ingenuity, the team painstakingly designed a recovery procedure and tested it on one of Samba's functioning sister satellites.

Solution based on a 'dirty hack'

"The solution was based on a 'dirty hack' -- jargon referring to any non-standard procedure -- but we really had no other option," said Jürgen.

Finally, on 1 June, in a very tense mission control room, a series of commands was radioed up. To immense relief, these flipped the power switches to 'on and the recalcitrant WEC came back to life.

Cluster has since returned to normal operation and measures are being taken to prevent this failure from happening again.

"When everything goes as planned, flying a mission can be routine," said ESA's Manfred Warhaut, Head of Mission Operations. "But when unexpected trouble occurs, and there's nothing in the manuals, you really want to have an experienced and talented team on hand to solve the problem."