A. Super-Quick summary for experienced robot operators……………………. 1

B. Restarting robot pc…………………………………………………………………… 1

C. Connection to robot pc……………………………………………………………… 3

D. Copying files to/from the robot pc……………………………………………….. 3

Connecting from the robot computer in the dome:…………………………. 4

E. Starting Labview………………………………………………………………………. 4

F. Loading magnets on to the field plate…………………………………………… 5

G. Unloading magnets from the field plate………………………………………… 9

H. Recovery Mode……………………………………………………………………… 11

I. Resuming configuration of a plate after an abort/stop…………………….. 12

J. Calibration Options………………………………………………………………….. 13

K. Other important Information…………………………………………………….. 15

L. Troubleshooting……………………………………………………………………… 17

M. What to do if the E-stop is triggered by the controller:………………….. 21

     A. Super-Quick summary for experienced robot operators

       For full details, or inexperienced robot operators, you must follow the sections below.

1. GetCalibVals. Restart labview

2. cp new robot_shifts file  (C:\Robot\ robot_shifts_abs_xxxxxxxxxxxx.csv) to Z:\Robot_tile_files

3. Configuring: run python .\prepare_files_for_robot.py  .\<date>\<robot file name as downloaded from Data Central e.g. Robot_G15_T110> .csv .\robot_shifts_abs_<datetime>.csv 

Uncofiguring: run python .\prepare_files_for_robot.py   .\<date>\ <robot file name as downloaded from Data Central e.g. Robot_G15_T110>.csv .\robot_shifts_abs_<datetime_now>.csv  –robot_labview_file .\<date>\Robot…..CorrectionsApplied_<datetime_when originally configured>.txt

4. HolePositionsFile ->   Robot_tile_files\ParkingPosns_final_<datetime_now>

5. PickUpNPutDownSeqFile -> Robot_tile_files\<date>\Robot_…….CorrectionsApplied.csv     (the original CorrectionsApplied file whether configuring or unconfiguring)

6. click ReadHolePositions and either ReadSequence (if configuring) or LoadLastSeq (if unconfiguring)

7. Check Dishview looks correct.

8. click MoveInRdSeq

B. Restarting robot pc

-If the pc needs to be restarted, follow these steps, but if after doing so, there is an error in Labview about “Invocation error or anything about the robot Aerotech stages, then redo these steps after first following the trouble shooting steps (listed in the Troubleshooting section) discussing the resets in InTime.

-Restart PC by shutting down any software, and power -> restart. The PC will not restart, it will just shut down. Go to the pc box at next to the robot and pull out the power cord from the back of the box and then after a few secs put it back in.

-Return to the control room and connect to the robot pc following the “connection to Robot PC” section below.

-Start A3200 Motion Composer (pinned to the start menu). Wait until it has all drives shown in a table as in the following picture. DO NOT jog or drive the z axis EVER in Motion Composer, the z axis should only ever be moved by using the homing button.

-BEFORE enabling the drives, go up to the robot and check the arrows are aligned on the rotation stage as in the next picture with the cable hanging out to the LEFT (not wrapped around).

-Back in Motion Controller: Enable the all drives by clicking on the green and grey circle icon just to the right of each axis name, for the x, y, z and theta axis only. Then home them by clicking on the house button STARTING with the Z axis then theta then x and y.

 C. Connection to robot pc

From the Control room:

  1. On the windows pc to the left of the “data reduction” screens, login using the hector account with RobotisGreat! Password and then go to the start menu and the remote desktop application is pinned in the start menu.
  2. Computer = 10.88.90.117
  3. Log in with .\hector (or may be saved by default as HECTORROBOT\Hector) and password “hectorisgreat”.
  4. Ignore any error box about windows expiring.
  5. Once in the remote Hector pc window, to open the live camera view:  click the camera icon at the bottom of the window (at the TaskBar).

D. Copying files to/from the robot pc

The robot pc is intentionally blocked from talking to the external network to prevent windows updates. Therefore files cannot be transferred to/from the pc using the cloud.

To copy the robot tile files to the robot pc, first copy the robot and tile files to aatlui because they will need to be loaded into the control task for observing. Then from there copy the robot file to the robot pc using the following:

There exists a directory on the robot PC   C:\transfer

aatliu has been configured to automatically mount this directory with the correct credentials, and the aatinst user has the ability to manipulate files within it. The directory can be found on aatliu at /hectorrobot/transfer

On aatliu you will be logged in as “aatinst”. To transfer files to the hector robot PC, simply copy them to /hectorrobot/transfer. On the robot pc, such files will appear in c:\transfer  This also works in reverse if you want to copy files from the PC back to aatliu.

Note that after downloading the files from DataCentral then extract the files in the Downloads Directory (of aatliu). Then make a Directory in /hectorrobot/transfer that they can be written to (e,g, with the year-month of observations) and for copying to that directory, you may need to use ‘sudo cp ~/Downloads/…….’ due to permissions.

One trick: The transfer directory is automatically mounted on aatliu when an attempt is made to access it. After a few minutes of not being used, the system will automatically unmount the folder. From the user’s point of view this mounting and unmounting is automatic – it just happens. One thing that often catches people out with such automounting is that if the directory has not been recently accessed, it is not mounted and hence it appears to not exist. So for example, if you do an “ls /hectorrobot” the listing may show up as empty, and many people conclude that the transfer directory is missing and assume something has broken. You just need to have faith, access the directory as though it exists and the automounter will do its thing.

 Sharing a directory in this way does create a dependency between aatliu and the robot PC. The automounter limits this dependency to only exist while the shared directory is being used. But be aware, say the robot PC crashes or is shut down, then someone on aatliu accesses the shared directory, it obviously cannot work, and the command may hang waiting for the missing mount.

[OLD instructions if connecting from the “Data Reduction” Linux machine:

In an LXTerminal, type

ssh –X hector@aatlxh password RobotisGreat!

rdesktop –g 1920×1080 10.88.90.117

Other user message – hit OK

Type HECTORROBOT\Hector and the hectorisgreat password

Start => Remote desktop]

Connecting from the robot computer in the dome:

Ctrl+alt+del,  account: Hector,  password:hectorisgreat

E. Starting Labview

TO BRING THE LABVIEW WINDOW TO THE FRONT ON SCREEN DO NOT JUST CLICK ANYWHERE ON THE WINDOW AS YOU WOULD DO WITH ANY OTHER PACKAGE – ONLY CLICK ON THE TOP BAR OF THE WINDOW TO ACTIVATE THE WINDOW OR LABVIEW WILL HAVE ERRORS.

DO NOT CLICK OR CHANGE ANYTHING IN LABVIEW THAT IS NOT EXPRESSLY IN THIS GUIDE. THE ROBOT WILL BE PERMANENTLY DAMAGED IF CHANGES OUTSIDE OF THIS GUIDE ARE MADE. ENSURE YOU ARE SELECTING THE CORRECT BOX OR BUTTON BEFORE YOU GO AHEAD.

ACCIDENTALLY LOADING A FILE IN THE WRONG PLACE OR PRESSING THE WRONG BUTTONS IS VERY LIKELY TO CAUSE PERMANENT DAMAGE TO THE ROBOT.

If Labview is not already running, or if restarting to clear an error:

  1. To start Labview, click the Labview icon at the bottom of the screen (yellow arrow type symbol) “NI LabVIEW 2018 AP1 (32-bit)”. This will open a Labview menu.
  2. Click on “Hector_Robot_MainUI-YYMMDD-xxxxxxx.vi” (using whatever the latest date version is)
  3. In the Labview window that opens, hit the top left arrow (that says “run” when you hover over it) to run Labview. The arrow turns filled black when Labview is running. Labview needs time to initialise everything. To check when that is done, go to the Maintenance tab (See Fig 4.) and wait until the USBReadVal box shows “lit1:OK” in the top slot.

When Labview shows a pop-up window, you must click the buttons in the pop-up window. Don’t press ‘close window’ (‘x’ mark top-right corner of the pop-up windows) as you do in other window applications.

In the Manual control tab (see Fig. 1 below), in the “Status” panel (central to screen) it should say “Homed” and be green, and there should be no error message in “Error” panel if the Aerotech stages are correctly enabled without any errors. If they are not Homed, or if the pc has been rebooted, then you must reset or enable the robot gantry by hitting the “reset” button on the Manual Control tab lower right.  Or, if “Status” is “Homed” but there are error messages, click “ClearError” and re-run Labview (stop and run Labview). When it asks to align the rotating cylinder – manually rotate the robot z-axis cylinder to align the arrows on the cylinder and on the mount to within about 30 degrees of each other with the connecting cable hanging loose to the left side (not wrapped between the cylinder are the light). Check that the boxes at the bottom of the screen for each axis say “homed”. On the dish view, check the robot position mark is near top left corner.

F. Loading magnets on to the field plate

  1. Check Labview is running by checking the ‘Arrow’ mark on top left of image below, which should be solid black. If it is empty arrow, click the arrow to run Labview.
  2. Run the calibration that measures the current thermal expansion of the robot compared to the plate: Hit the “Get Calibvals” button with red text near the centre top of ‘Calibration’ tab (see Fig. 3 below). The green light beside that button will come on then Labview will read the fiducial marker position file, and the related sequence file, take 4 photos of fidicial markers. Wait until it goes off to indicate the metrology process is complete (takes about 3 mins). This generates the ‘robot_shift_abs’ file (in C:Robot/).
  3. The CPU will be at 100%. To fix that:
  4. Open “Task Manager” to check CPU.
  5. Close “Hector_Robot_MailUI.Ivoroj – project explorer window by selecting the labview icon at the bottom of the screen and switching to the project explorer window. Select File – Exit. Select “quit” in the box that warns about aborting Vi’s. Never quit by hitting the X in the screen corner, only ever quit by using File -> Exit.
  6. Re-start Labview by clicking on the“Hector_Robot_MainUL-<latest date>”.
  7. Hit the top left arrow in labview screen, wait for lit1:OK at ‘Maintenance’ tab (Fig 3).
  8. Run the metrology calibrations:

To run the calibration and metrology corrections in python (correct method as of Feb 2022), rather than running them in Labview as was done previously (but should no longer be done in Labview):

  1. Make new directory for today’s date in the format YYYYMMDD, in Z:\Robot_tile_files\
  2. Download the tile file for the next field from the Data Central Cloud Hector storage, called Robot_FinalFormat_tile….csv and put it in Z:\Robot_tile_files\<date>\
  3. Copy C:\Robot\robot_shifts_abs_<date just made>.csv Z:\Robot_tile_files\ (don’t copy the robot_shifts_abs file without date).
  4. Next run the standalone python function to implement metrology, in the PowerShell terminal by this method :
  5. cd Z:\Robot_tile_files 
  6. Run Python  .\prepare_files_for_robot.py  .\<date>\ <tile file name as downloaded from data central e.g. Robot_G15_T110>.csv .\robot_shifts_abs_<date just made>.csv

Or if needing to put in temperature values from above e.g. for 17 and 21 degrees:

Python  .\prepare_files_for_robot.py  .\<date>\R <tile file name as downloaded from data central e.g. Robot_G15_T110> .csv .\robot_shifts_abs_<date just made>.csv –T_observed 17 — T_configured 21

  • This will generate a new file with metrology, roll, telecentricity magnet hole offset and temperature correction calibrated, from the original robot tile-file (and the decentre of pick up arm correction will be added shortly [Feb 2022]). The file has “CorrectionsApplied” appended to the original tile name (e.g. Robot_FinalFormat_tile…. CorrectionsApplied.csv”. This is the generated metrology calibrated file to be loaded in ‘PickUpNPutDownSeqFile’ in step V below.
  • Copy the resulting csv file to a name with less than about 60 characters (if it isn’t already) or it will not be readable by Labview below, BUT leave the extension “CorrectionsApplied” on the end.
  • Back in Labview: Turn the Labview calibration options to OFF: “PuARotOffset”, “Y Shear” and “DoMetrology Calibration” at ‘Calibration’ tab at Labview (see Fig. 3). Turn off “CalibBeforeMv” far top, scroll right, click so the rectangle light is not illuminated. Must be done before loading files.
  • Load storage position file from ‘ManualControl’ tab (Fig. 1).
    • Load the “ParkingPosns_final_yymmddhhmmss.csv” (from the Robot_tile_files directory) with the current date stamp into HolePositionFile.
    • Check “RdStatus?” button is ON – to the right of the ReadHolePositions button (right and illuminated).
    • Click ‘ReadHolePositions’.
    • When the robot reads the file, LED right to ‘ReadHolePostions’ will lit.
  • Load the tile file for the field to be configured.
    • Choose a tile file in ‘PickUpPutDownSeqFile’ box. Tile file is of the format:             Robot_tile_files\<date>\Robot_…….CorrectionsApplied.csv     (the original CorrectionsApplied file whether configuring or unconfiguring).
    •   If the input box shows ../.. that means the file name you tried to read was too long. Copy the file to a shorter name.
    • Click ‘ReadSequence’.
    • When the robot reads the file successfully, LED right to ‘ReadSequence’ will light up and ‘RvseOrder’ will be set to OFF (slide to left with grey).
    • Check the Dish View tab to ensure it makes sense. The magnets in the holding positions should look to be in the right place (clearly not way out of line of the pattern) and the filled magnets indicating where magnets actually are, should agree with where they really are on the holding positions. The plate should show all magnets unfilled because there are no magnets on the plate, and the unfilled positions should match the tile you are attempting to load.
    • If an error is given that there are not enough magnets and says to unload magnets first, but the previous plate has been unloaded, then there may be one or more magnet colours that do not have enough left. In the Hector pipeline there is a python function that checks pairs of fields that will be observed consecutively to confirm there are enough magnets to do both (since two plates will be holding magnets at the same time). If that was run, then you should never see an error about the number of magnets, in which case double check you are loading the right robot tile file. If that function shows there will not be enough magnets for a pair of plates (will happen about 1% of the time), then swap out one field to observe on the next night, so the pair of field this night and the pair of fields next night will not require too many magnets of any type.
  •  Check the ‘MvFrmSeq’ button is “0” (to the far right, and right of the RvrseOrdr button). When you do full configuration it must start from ‘0’. If you find a problem during loading, press ‘Kill’ and write the number in ‘RunSeq’. Then restart Labview by clicking empty arrow under Labview menu bar (it will turn to dark arrow when Labview run) and wait for “lit1” in the Maintenance tab then reload the files (click ‘ReadHolePositions’ and ‘ReadSequence’) and switch off the calibration switches in the Calibration tab and turn off CalibBeforeMv, then once all back to the same point as it was, THEN type the number in ‘MvFrmSeq’ to restart loading from the stopped point. Currently it is more reliable to start from a number 2 ahead of the RunSeq number and to put aside the magnet that errored and put it back on the plate at the end using the Recovery code.
  • Start loading of magnets on to the field plate:
    • Check there are no errors in the Maintenance tab under USBReadVal. Scroll down through the last loaded values.
    • Click  ‘MoveInRdSeq’
    • Labview will show a pop-up window asking “will you start Loading/Unloading without calibration” click OK because the calibration was applied earlier by the python code.
    • When finishing, below pop-up window will appear. Leave that in place for now.
    • If the pick up arm is in the way so the plate cannot be lifted from the robot plate holder, then click cancel on the recovery mode (note you can reactivate recovery mode shortly by hitting the “RecoveryMode” button to the mid right of the Dish View tab if it is needed for bumped magnets). Then to move the robot arm out of the way, go to the manual tab and to the lower right there is a “move to” button with values to the right of it. Set to x and y values to be both 10, and ***DO NOT CHANGE THE Z VALUE OR THE PICK UP ARM WILL CRASH INTO THE PLATE**. Then hit “move to” and the robot arm will move away from the plate.
    • Load plate from robot holder to the telescope and plug hexabundles.
    • If no magnets were bumped, choose ‘Cancel’ because the loading is successful.
    • If any magnets were bumped then remove all hexabundles, put the plate back on the robot and hit OK in the box to begin the recovery procedure.

Figure 1  Manual control tab of the Hector Robot main GUI, with highlighting parts mentioned in the text.

Figure 2  Dish View tab of the Hector Robot main GUI

G. Unloading magnets from the field plate

  1. Check Labview is running by checking the ‘Arrow’ mark on top left of image below, which should be solid black. If it is empty arrow, click the arrow to run Labview.
  2. Run the calibration that measures the current thermal expansion of the robot compared to the plate: Hit the “Get Calibvals” button with red text near the centre top of ‘Calibration’ tab (see Fig. 3). The green light beside that button will come on then Labview will read the fiducial marker position file, and the related sequence file, take 4 photos of fidicial markers. Wait until it goes off to indicate the metrology process is complete (takes about 3 mins). This generates the ‘robot_shift_abs’ file (in C:Robot/).
  3. The CPU will be at 100%. To fix that:
  4. Close “Hector_Robot_MailUI.Ivoroj – project explorer window by selecting the labview icon at the bottom of the screen and switching to the project explorer window. Select File – Exit. Select “quit” in the box that warns about aborting Vi’s. Never quit by hitting the X in the screen corner, only ever quit by using File -> Exit.
  5. Re-start Labview by clicking on the “Hector_Robot_MainUI.lvproj”. Double click “Hector_Robot_MainUL-<latest date>”.
  6. Hit the top left arrow in labview screen, wait for lit1:OK at ‘Maintenance’ tab (Fig 3).
  7. Run the metrology calibrations (can be done while waiting for lit1:OK):

To run the calibration and metrology corrections in python (correct method since Feb 2022), rather than running them in Labview as was done previously (but should no longer be done in Labview):

  1. Identify the directory name in Z:\Robot_tile_files\ that corresponds to when the this plate was tiled. The files in that directory should be used and edited as follows.
  2. Copy C:\Robot\robot_shifts_abs_<date just made>.csv Z:\Robot_tile_files\ (don’t copy the robot_shifts_abs file without date).
  3. Next run the standalone python function to correct the original files used to tile the plate to the metrology corrected values for the current temperature, as well as correct the holding positions file to put the magnets in the plate-coordinates accurate holding positions by finding the equivalent virtual robot positions. In the PowerShell terminal use this method :
  4. cd Z:\Robot_tile_files 
  5. Run    python .\prepare_files_for_robot.py   .\<date>\ <tile file name as downloaded from Data Central e.g. Robot_G15_T110>.csv . \robot_shifts_abs_<datetime_now>.csv  –robot_labview_file .\<date>\Robot…..CorrectionsApplied_<datetime_when originally configured>.txt

e.g. python .\prepare_files_for_robot.py .\20230908\RobotA3716_T031.csv  .\robot_shifts_abs_230910011043 –robot_labview_file .\20230908\Robot_A3716_T031_CorrectionsApplied_230909031230.txt

(NOTE the date/time extension is the time that the plate was originally configured. The last file is the file generated by Labview when it configured the plate. The first file is the exact same name as the last file but without the date stamp on the end.)

  • Note the output file names listed towards the end of the text generated on-screen from python – those file names will need to be loaded in the following steps. This will generate a new file with corrections for the parking positions and the unplugging positions. The file has copied the previous file that Labview made when (plugging the plate) from its name of “Robot_A3716_T031……CorrectionsAppliedyymmddhhmmss”  to “Robot_A3716_T031……CorrectionsAppliedyymmddhhmmss_After_Loading” and then written the new corrected file to the original name so Labview will identify it as the right file to use to unconfigure the plate.
  • Check all 4 calibrations including CalibBeforeMv are off.
  • Check RdStatus is on.
  • The file called “ParkingPosns_final_yymmddhhmmss.csv” with the current date stamp, should then be loaded in ‘HolePositionFile’.
    • Click ‘ReadHolePositions’.
    • When the robot reads the file, LED right to ‘ReadHolePostions’ will lit.
  • For ‘PickUpPutDownSeqFile’ box Load the file for the field to be unconfigured.
    • Choose a tile file, which was used to load magnets, normally second last file. Note this is the original tile file “Robot…. CorrectionsApplied.csv”, not the one just generated with the yymmddhhmmss ending. Labview will find that new file based on the stem name of the original file to which yymmddhhmmss was appended.
    • Click ‘LoadLastSeq’ and check the date stamp on the file name shown in the box should be when that plate was configured originally. click ‘OK’ on pop-up window. If for some reason, a different file is planned to be used, the astronomers should be very sure that there is a reason for that. If the file selected is not the second last file used (i.e. 2 plates being configured) then it will give a warning.
    • When the robot reads the file successfully, LEDs right to ‘ReadSequence’ and ‘LoadLastSeq’ will light up and ‘RvrseOrdr’ will be set to ON (slide to right with red).
  • Check the “DishView” tab to be sure the magnets are plotted where they actually are and the holding positions look right (if the dish view has magnets located clearly wrong in the holding positions then you put the wrong file in when running the python. If the magnets shown do not match the plate actual magnets then you put in the wrong Robot tile file).
  • When doing a usual full unconfiguration of a plate, ‘MvFrmSeq’ must be ‘0’. If you find a problem during loading, press the red round button (stop button) top left to stop running the Labview code.  Note down the number in ‘RunSeq’ (Right panel towards top below blue “Magnets” button). Then follow directions in section “H. Resuming configuration of a plate after an abort/stop”
  • Start unloading of magnets on to the field plate by clicking  ‘MoveInRdSeq’.

H. Recovery Mode

  1. In recovery mode, you can replace/place a bumped magnet accurately to where it is intended. To use recovery mode, there are two ways to start it; 1) when finishing loading/unloading magnets to/from the plate, a pop-up window will ask you if you want to go into recovery mode, 2) manually go into the recovery mode by clicking ‘RecoveryMode’ button in ‘DishView’ tab.
  2. Note that the recovery mode only works for the tiling process only. It will not work with any magnets that are not involved in the current loading/unloading process.
  3. To place a magnet in recovery mode, first click the magnet where you want to place the magnet in ‘DishView’, then ‘MgtType’ and ‘MgtNum’ just right to ‘DishView’ show the magnet type and magnet number, which will be placed at the clicked point.

‘PremovSeqs’: When a cluster of magnets are placed very closely, during locating the recovering magnet, it may collide with other magnets so some close magnets must be cleared prior to putting down the recovering magnet. ‘PremovSeqs’ shows the sequence numbers to pre-move those close magnets.

  1. If the magnet is what you want to recover, click ‘Recovery “RUN”’ button. Then, pop-up windows will show instructions. Follow these instructions.
  2. Make sure to put the circular magnet into the pick up arm with the high side (coloured dot) facing right.
  3. If you need to place more magnets in recovery mode, repeat the above procedure.
  4. When you finish the recovery mode, click ‘RecoveryMode’ button again to return to normal operation mode.

I. Resuming configuration of a plate after an abort/stop

If the red button (or kill) is pushed to stop the Labview code running during configuring or unconfiguring a plate, OR if the Labview stops itself due to an unreolvable timeout, then the sequence can be resumed by following these steps.

  1. Firstly make a note of the RunSeq value in the right panel (just below the blue Magnets button). That is the step number that it stopped on.  These numbers refer to movements of the robot not to magnet numbers.
  2. Retrieve the magnet that is in the pick up arm if there is one. The way to do that will depend on the situation when it stopped:
  3. If the magnet is being held by the pick up arm with the pin in with the magnet either on the plate/holding position or above the plate, then EVEN if the magnet had just been put down in the right spot, OR if it was about to be picked up,  first you need to raise the pickup arm up (with magnet) up off the plate by homing the z axis (details below) then release the pin by clicking the  button in the Maintenance tab marked CCentrePin if it is a circular magnet, or RCentrePin if it is a rectangular magnet or PushDnPin for the two side pins down on a rectangular magnet.  Then gently lift the magnet out of the pick up arm by hand but being very careful that it does not jump to another magnet (if it does then the other magnet is likely bumped and must be removed from the plate or holding position). Put the retrived magnet aside. The next time that a plate configuration would have used that magnet, then it will need to be given back to the robot using the recovery mode at the end of running that configuration.
  4. If the magnet is being held by the pick up arm but the pin is not down, then if it is above the plate, carefully remove the magnet from the pick up arm by hand and place it to the side off the plate holder ready to be given back as in the last point. If the magnet is on the plate and aligned in the pick up arm but without the pin down, then use the method in the last point to put the pin down using the Maintenance buttons, then home the z axis and then drive the pin up, then retrive the magnet. Do not home the z axis without the pin pushed down into the magnet casing or it will be lifted up unconstrained and will jump from the pick up arm to another magnet.
  1. TO HOME THE ROBOT AXES :
  2. To home the z axis first to ensure it is high enough before any move: THE z AXIS AHOULD ONLY EVER BE DRIVEN TO ZERO, AND THAT SHOULD BE DONE ONLY BY HOMING IT. DO NOT ATTEMPT TO DRIVE THE Z AXIS TO ANY OTHER POSITION OR IT WILL BE DAMAGED. After any forced stop or issue,  home the z axis first before the other axes to ensure the arm moves upwards first before moving sideways to avoid collisions. In the Windows start menu open A3200 Motion Conposer if it is not already open. DO NOT jog or drive the z axis EVER in Motion Composer, only use it to enable/disable (see details below) or home the z axis. To home the z axis, check there are no faults (any boxes turned red) and if so see below. On the third row, marked as Z axis in the first column, click the picture of the “home” in the third column beside the Z axis. Once homed the Status column should say “homed”.
  3. If Labview has been restarted and is showing no errors and is operational, and all axes need homing, then click the red “RESET” button on the Manual Control tab (lower right). This will enable and home all 4 axes (x, y, z, theta).
  4. If Labview is not responding and the manual control tab shows errors in the Status and/or Error boxes that are to do with Firewire or controllers or axes, and restarting Labview and restarting the pc then restarting Labview doesn’t fix those errors, then the Robot drives may need to be reset in the A3200 Motion Composer as discussed in the first section of this document.
  1. Rerun Labview by clicking the empty arrow under the Labview menu bar (it will turn to dark arrow when Labview run). Check there are no errors in the Maintenance tab as the USBReadVal’s scroll by and wait until Lit1:OK. Check there are no errors in the Error box lower centre of the Manual Control tab. If there are then you will need to hit ClearError and it is safer to then rerun Labview (Red stop button, black arrow -> wait for lit1:ok). If the errors persist then see Troubleshooting steps below. Next load the two files (‘HolePositionFile’ and ‘PickUpPutDownSeqFile’-click ‘ReadHolePositions’ and ‘LoadLastSeq’) as was done originally.  If you start from exactly the RunSeq number then it may be one step out of sync with where the magnets are on the plate and crash into a magnet. To be safe, start the sequence from two after that number if you are not sure of the exact move it will make based on the following . NOTE on unplugging the pick up sequence is on an even number and put down is an odd number in the RunSeq . NOTE that when it is displaying an odd number it is putting down, then straight after it has come back up, the number changes to even as it then travels to where it needs to pick up. However when you put in an odd number in the MvFrmSeq box it says it had no magnet in the arm and asks do you want it to put down anyway….therefore the RunSeq and MvFrmSeq appear to be out of step by 1 index. Put an even number in the MvFrmSeq box for the robot to start by picking up. [CHECKED!]
  2. When you then hit “MoveInRdSeq” while watching dish view, it will think for a minute then start moving the magnets virtually in the dish view (without moving the pick up arm) until the dish view matches where you asked it to start from. Then it will move the pick up arm and continue on from where it was up to. Since it was started one step ahead of where it had ben stopped, there is likely to be one magnet that was not positioned. If you are configuring, then position that magnet after the field is finished configuring, by using the Recovery mode. If you are unconfiguring, then that magnet may have been left on the plate by the robot, and should be removed by hand and put aside. The next plate that requires that magnet will be missing a magnet after being configured and that magnet will then be added back in using Recovery mode.

J. Calibration Options

  1. There is a main option that selects if any calibration is applied or not. The option is placed on far-right upper side, marked with a red arrow in below picture – called “CalibBeforeMv”. When it is ON (bright green), calibration will be done just before loading/unloading magnets onto/from the field plate with the tile file. When it is OFF, no calibrations will be applied. This should always be OFF now because the calibration is now done in the separate python file.
  2. If this option was ON, there are 3 option switches applied to calibration in ‘Calibration’ tab; ‘PuARotOffset’, ‘Y Shear(c,r)’, ‘DoMetrology Calibration?’ as shown in below. These should always be OFF. Details of what those three are for are given further below, but as they should not be used, that is only for historical note.

Fig 3. Calibration Tab image with the calibrations to be turned off highlighted.

Fig 4. Maintenance Tab image

K. Other important Information

  1. If the robot needs to be stopped rapidly – hit the round red button to the top left of the Labview screen. If that does not stop the robot then hit the “KILL” button. Red button stops after current movement, KILL should stop faster. It is recommended to use red button at first instance to stop loading/unloading. All software stop methods take time and the stopping time depends on various factors. A hardware Emergency Stop button at the robot will immediately stop the robot, but is not accessible from the Control room (Feb 2022).
  2. After pressing hardware Emergency Stop button, the button is locked. Slightly rotate the button to release it. And, run Labview by pressing the empty arrow, reset the robot system by clicking ‘Reset Robot’ button with red ‘RESET’ on right middle area of ‘ManualControl’ tab.
  3. When Labview starts, it takes about 3 to 5 minutes to finish test of the system and to ready to receive command from users. Check the ‘Maintenance’ tab and wait ‘lit1: OK’.
  4. Not normally done, but IF you need to read in the files WITHOUT reading the past history – I.e. assuming all the magnets are in the holding positions not on any plate), turn off (to the left) the “RdStatus?” knob which is to the right of READHOLEPOSITIONS button. Normally the “RdStatus?”should be ON.
  5. Setting temperature values when running the python script: Typically the temperature value setting in the python script will not need to be used. The temperature difference between the expected temperature when the plate was configured and when it would be observed that was used in the pipeline when generating the tile files. If that temperature difference is going to be wrong (as known at the time the plate is being configured) then this step is the way to correct that. Note it is not correcting for absolute temperature only for the temperature difference. Since Invar has a very low thermal expansion, the temperature difference will typically be set as 0 degrees in the pipeline and here it only needs to be run if the temperature difference between plugging and observing is looking like it will be > 10 deg (10 deg is maximum of 5um shift in the galaxy centre position for objects at the edge of the plate).
  6. Do not edit the ‘T_observed’ and ‘T_confgured’ on ‘TCalibParams’ in the right middle of ‘Calibration’ tab. Ensure they are left as the default values of  -10000. These were used when the Labview code ran the temperature correction, but that is now implemented in the python code.
  7. If running in manual mode then must manually put the centre pin down before clicking on the empty magnet position in the dish view. Make sure to put the circular magnet into the pick up arm with the high side (dot) facing right.
  8. When loading and unloading magnets onto/from the field plate, even numbers of ‘RunSeq’ are always a pick-up operation and odd numbers are always a put-down operation. When you need to type the sequence number in ‘MvFrmSeq’ after stopping loading/unloading, please check carefully if you need to start pick-up or put-down operation. To start from pick-up operation, the number in ‘MvFrmSeq’ must be even number (0, 2, 4, ….) and for put-down, it must be odd number (1, 3, 5, …), otherwise pick-up arm will be damaged.
  9. When Labview shows a pop-up window, you must click the buttons in the pop-up window. Don’t press ‘close window’ (‘x’ mark top-right corner of the pop-up windows) as you do in other window applications.
  10. If you are not sure how to do, please ‘KILL” and start from running ‘Labview’. Don’t try to something that is not shown in this manual. It will cause an unexpected result.
  11. The filled round/rectangular marks in ‘DishView’ mean that there is a round/rectangular magnet. The empty round/rectangular marks show there is no magnet but a magnet will be placed.
  12. Normally ‘DishView’ shows most recent status of magnets in the plate and in the storage place but when Labview or PC restart, you can read the status by reading ‘HolePositionFile’ with ‘RdStatus?’ option ON (right side with green colour); ON is a default value. When the loading/unloading is finished, the status is automatically saved.
  13. The status of magnets in storage and plate will be saved automatically when configuring/unconfiguring of a tile finishes without any error. If you stop loading/unloading magnets for any reason and finish the loading/unloading in manual or recovery mode, you need to make Labview save the status for future. To do this, enter an enough large number (for example, if the number of sequence for the loading/unloading, which can be checked in ‘NumOfSeq’, ‘ReadWriteData’ tab, is 108, then you can type 108 or larger number) in ‘MvFrmSeq’ box, and click ‘MoveInRdSeq’ button. Return this number to ‘0’ when you done.
  14. Don’t forget to type number ‘0’ in the ‘MvFrmSeq’ if you typed there any other number during loading/unloading the plate in any reason.
  15. Magnet availability/status in storage area.
    1. When Labview reads ‘HolePositionFile’ by clicking ‘ReadHolePositions’, Labview will read status of magnets.
    1. There is an option called ‘RdStatus’ next to ‘ReadHolePositions’ button in ‘ManualControl’ tab. With this option ‘ON’, Labview will read automatically saved status file, which is automatically saved after loading/unloading of magnets onto/from the field palate finishes successfully.
    1. With ‘RdStatus’ ‘OFF’, Labview doesn’t read the status file and assumes that all magnets in the storage area are available for next configuration.
    1. When you need to initially load all magnets in the storage area (to set up the original array of magnets), turn off ‘RdStatus’ option and read the ‘HolePositionFile’. Then load magnets in the storage area by clicking ‘Loading Magnets’ button just top right of ‘tab’ structure. (This should normally only be needing to be done by the instrument team – Talk to Julia first.).
    1. When you configure a tile onto the field plate by clicking ‘MoveInRdSeq’, then Labview will save the current status of magnets so you can read it at next reading of ‘HolePositionFile’ with option ‘RdStasus’ ON.

Details of the calibration options that we no longer use:

  1. ‘PuARotOffset’ is for the pick-up arm rotation error compensation, which will compensate the error due to the off-centred of the pick-up arm in the rotation head. This is now done in the python code instead and the Labview corrections have not been confirmed to be the same as the python, but the python has been confirmed to be correct.
  2. ‘Y Shear(c,r)’ will compensate the shear force to y-axis due to the strong magnets (the “roll”). This is now done in the python code instead and the Labview corrections have not been confirmed to be the same as the python, but the python has been confirmed to be correct.
  3. With ‘DoMtrology Calibration?’ option ON, the metrology results will be applied to the calibration. This is now done in the python code instead and the Labview corrections have not been confirmed to be the same as the python, but the python has been confirmed to be correct.

These calibrations could be applied separately (but you shouldn’t have a reason to use them at all). All 3 options are OFF by default.

Startup Instructions for Hector Robot PC

These instructions explain the procedure for if the Hector Robot PC is restarted.

Press and hold down the power button on the front right of the pc box until the light goes off. Then wait until it comes on again and goes off again, then press it once without holding and the pc should start.

Begin temperature logging (optional and currently we don’t use the temperature logger)

  • On the desktop, double click the icon labeled ‘Start_Temperature_Logging’. This will begin a coolterm session, and you should see temperatures start appearing in the coolterm window. 
  • These are being logged to the file Desktop\HectorRobotTemperatureLog.txt . Open this file and make sure these newly measured temperatures have been appended to the end of the file.
  • You can now safely minimise (NOT CLOSE!) the coolterm window.

Startup After Major Power Failure

If all power is lost (and the building’s no-break circuit fails), AND the local UPS and the bench fails (eg battery runs out after extended outage):

  • If not automatically started, switch on PC using power button on front of PC case
  • Switch on the metrology light power supply by pressing Output on the front of the power supply – see picture below. The power is on when the LED above ‘Output’ is lit.

L. Troubleshooting

1.Error on pc in Labview “Not enough memory to complete this operation”.

a. Shut down any software that is running and is not hung.

b. Restart the pc by pulling out the power cord on the back and then putting it in again (don’t restart by pressing the button on the front of the box).

c. Do the following steps to reconnect the drivers:

  • Start Menu -> INtime -> INtime Configuration
  • Open Intime device manager
  • Right-click on INtime T1394         bus OHCI TI as shown above, select Pass to WIndows.
  • Press exclamation mark (top left) to save new settings. Restart the pc.
  • After restart, reallocate it from same device manager (but now it will be under the Windows devices heading as shown below) by right-clicking and choosing “Pass to INtime with legacy IRQ”.
  • Press exclamation mark (top left) to save new settings.

d. Restart Labview. Rerun the labview program by hitting the arrow and waiting for “Lit1:OK”.

e. In the Manual tab, hit the “Reset” button lower right, to initialise the robot gantry, and say yes to the pop-up box asking to confirm that operation. Then check the rotating cylinder arrows are aligned on the physical axis and hit confirm in the pop up box.

f. Turn off the calibration 4 buttons.

2.Any error about “Invoke….” or long error in Labview with “invocation error” as part of it, complaining about the motion composer or anything to do with the robot gantries.

a. Restart labview and try moving the robot in the manual control tab to say x=10, y=10 and if the robot does move then go to step b, but if the robot doesn’t move then  restart Labview again and try moving it again and if it still doesn’t move then shut down the pc and restart following the instructions in Troubleshooting example 1 above including the instructions to reconnect the INtime and restart again.

b. If the Status box (lower centre) on the “Manual”  tab says “disabled”, this means the robot gantry axes are disabled. To re-enable them, on the Manual tab, click the big RESET button to the lower right. It will ask you to confirm that you want to reset and click yes. It will then ask you to align the robot rotation cylinder. To do so, go up to the robot and by hand rotate the rotation cylinder to align the arrows on the side (as you would have been shown during training). Then come back to Labview and hit ok and let it continue on to zero and enable the gantries. You should see the pick up arm cylinder move in all 4 axes directions slightly as it re-homes. If this works then continue on with running Labview. If the axes do not respond and there are new errors then shut down the pc and restart following the instructions in Troubleshooting example 1 above including the instructions to reconnect the INtime and restart again.

 3. If you are going through any of the point 1 or 2 fixes and they do not solve the problem because you continue to get errors related to the pc not talking to the robot or not recognising drives or “communication error? Variants AND you cannot restart Labview to the point where you can press the RESET button (as described above), then it means that the gantries are not enabled and Labview cannot fix that and the solution is to enable the gantries outside of Labview. However, operating the gantries outside of Labview carries a high risk of breaking the robot. The Run leads have all been trained on how to do this and will do this for you. NOTE to run leads: Full instructions in email to run leads 28th June 2023. (as described above), then it means that the gantries are not enabled and Labview cannot fix that and the solution is to enable the gantries outside of Labview. However, operating the gantries outside of Labview carries a high risk of breaking the robot. The Run leads have all been trained on how to do this and will do this for you.

4. If the robot arm crashes onto a magnet and has an overcurrent error.  First step is to make the robot safe by getting the arm off the magnet because it could have significant force. Usually that will mean driving the robot z-axis upwards by homing it, BUT FIRST it is essential to look at the situation and the disrupted placement of magnets and ensure that driving the robot up is not going to make the situation worse. DO NOT try to pull out any magnets that are wedged between the pick up arm and the plate (or another magnet). Move the robot away first to remove the force. 

How to move the robot away: Labview will have shut down with the overcurrent. The best approach is to use Motion Composer but this should ONLY BE  done by the run leads who have been trained to use Motion Composer (because a lot more permanent damage can be done from the incorrect use of Motion Composer). The Run lead should  note that in Motion composer there is an over current error and that the z-axis (usually z) axis is in fault due to the overcurrent. First “Acknowledge” the error on the axis using the button towards the top left that clears the errors. Then enable the z axis (button on the z-axis row, with the green circles) then home the z-axis. Do not try to manually drive the z axis, only use the home option.

Once the robot is clear of the problem magnet/s, remove the magnets from the plate including any magnets that may have been bumped in the incident. Check they have no damage and put them aside for now.

Any issue as described here will have bent the pick up arm. The pick up arm will need to be changed. This can only be done by those trained to do this. If Tom W or Julia are at site they can do it straight away. Otherwise, during the day the site staff can call Julia and Julia will work through this with one of the site staff that have been instructed on this. Either way, calling Julia is the first step. Note that some extra magnets may need to be removed from the plate to give clear access to change the pick up arm without risk of bumping magnets.

Once the pick up arm has been replaced, then the field that was being configured/unconfigured needs to be completed or the robot’s files tracking the magnets’ positions will be incorrect. If the robot was unconfiguring a field then redo the unconfiguration that was being run, but this time start from the run number that is was at when it errored. The magnets related to the error are now off the plate and so when “move in read sequence” is hit (once the run number is first entered in the box to the right), the robot will pause briefly then catch up the dish view plot as if the moves are all done before that run number and will continue on from that run number. That should then do an ‘air swing’ because the magnet has been removed, but is a chance to see that the robot is operating correctly and the pins are moving as expected. It will then continue to complete the full plate, but will be doing ‘air swings’ for any magnets that had to be removed from the plate. It will continue on to completion and write out the correct file. Any removed magnets will be put back on the next plate they are required on, using Recovery mode after configuring that plate (note they will not all be required for the very next plate that is configured. You need to just check each configured plate is not missing any and if it is, put the right one on at the end with Recovery mode.

IF in the above scenario the plate has no magnets left on it after removing those in the collision then you still need to rerun the unconfiguration but setting the start run number to be 108 so it will think it has completed the plate and will write out the correct updated files for where it thinks the magnets are.

If a similar overcurrent error happened during configuring the plate, the same steps apply, but you will need to continue on to the end of the plate configuration and then after that immediately use recovery mode to put back on the magnets that had to be removed in the above steps.

M. What to do if the E-stop is triggered by the controller:

  1. Check what move number it was up to in Labview (so you know where to start again). Labview should have stopped itself, but if not stop it.
  2. Check if there is obviously anything wrong. Most likely you will see nothing wrong at all and it will simply be that the controller was not happy with the actuator position. If there is something obviously wrong then that needs to be sorted out depending on what it is (perhaps with a phone call to me if it is not obvious). If there looks to be nothing wrong, it has just stopped and all is looking safe then go to step 2.
  3. First get the magnet out safely – using the techniques already used in the past. The issue is if the magnet is held above the other magnets and the actuator resets then the magnet will jump to the other magnets, so first the magnet needs to be secured. Note that you will not be able to move the robot until after the controller is reset so you can’t just move it away from other magnets.
  4. Once it is made safe then you need to reset the Controllino: Click the “Manual Control Actuators.stc”  on the desktop. Click “Connect”. That will reset the Controllino and hence the pins so will release any held magnet. It will also enable the next step. Then click “Disconnect”.  You must disconnect this before starting up LABVIEW or crashed metal will happen . Then exit the “Manual Control Actuators.stc”.
  5. In Motion Com poser all axes will be red in fault state due to the e-stop.  Click the green tick top left to acknowledge the faults. Note that if you haven’t done the previous step then the tick will do nothing – must do the previous step first.
  6. Manually rotate the cylinder so the arrows line up. Then enable the X, Y, Z, THETA axes by clicking the green/white circle icon next to each of them. Then home the z-axis first (so it doesn’t move across magnets when in the down z position) using the house pic buttons, then home the X, Y, and THETA axes.
  7. Restart Labview.
  8. In Labview, Maintenance tab, operate the pins up and down (RCentrePin, PushdnPin, CcentrePin) to check that they are all moving freely with no magnet in. They may do nothing on the first click if they are already up and the click says UP. Then click DOWN. Do a Down then Up of each button separately. If they move fine then put them UP and continue.  If they don’t then, try only once more, and if they don’t move smoothly then contact Julia or Barnaby. If they move smoothly then continue on.
  9. Continue configuring from the move number it was up to following the instructions that are already in the robot instructions.