ClusterAlg.gif (577 bytes) The ATLAS Level-1 Calorimeter Trigger

[Home] [Architecture] [Meetings] [Participants] [Publications] [Software] [Search]

ATLAS-UK Level-1 Calorimeter Trigger Meeting

Wednesday 10th October 2001 - RAL

Present: Bruce Barnett, Ian Brawn, Adam Davis, James Edwards, Eric Eisenhandler (chair), John Garvey, Norman Gee, Tony Gillman, Stephen Hillier (minutes), Murrough Landon, Gilles Mahout, Ed Moyse, Viraj Perera, Richard Staley, Peter Watkins, Alan Watson.


Click for                                             Click for
summary                                          transparencies
                                                      (all pdf)

Minutes of previous meeting

Hardware status
Common Merger Module....................................Ian 10'
Cluster Processor Module............................Richard 10'
CP chip tests using GTM...............................James 10'
CP/JEP ROD prototype..................................Bruce 10'
Timing Control Module..................................Adam 10'
Short-term schedule....................................Tony 10'

Hardware-related items
CANbus status...............................Eric (for Dave) 10'
More on TTC timing jitter..............................Alan 10'

Software status
Online software status.............................Murrough 15'
Hardware Simulation and test vectors..................Steve 10'
How to read out the slice............................Norman 10'

Physics simulation
Off-line trigger simulation status.......................Ed 10'
Level-1 items from the Lund physics workshop..........Steve 15'

Out of the rut
Highlights of the LEB workshop.......................Gilles 15'
Malvern forum on microelectronics......................Tony 15'

Coming reviews
ATLAS System Status Overview introduction..............Eric 10'
ATLAS System Status Overview talk......................Tony 15'
PPARC review of ATLAS and CMS..........................Eric 10'

Coming meetings
ATLAS week.............................................Eric  5'
RAL joint meeting arrangements.......................Norman 15'
NIKHEF T/DAQ workshop..................................Eric  5'

Any other business
Dates of next meetings

Hardware status summaries

Common Merger Module - Ian

The board was being worked on in the Drawing Office by Dan Beckett. Progress was generally good, with all components placed, and most tracks routed, although the density of pins into the FPGAs was causing some problems. One major hiccup had occurred with the placement of the backplane connectors, which had been displaced due to a misunderstanding. The fault was only found by accident, although should have been picked up by the manufacturer at a later stage. Ian was also in contact with Sam about another connector placement problem which showed up in a disagreement between the Gerber files and the backplane specifications. Time was critical in finishing the work as Dan was only available for a little longer. It was estimated that the module should be ready in early December, with Ian and Viraj working on it until Christmas.

Cluster Processor Module - Richard

The schematics were now final, and the net-list had been checked for hanging wires. In the drawing office, the components were nearly all placed, with no chips placed underneath the PCB. The board should be routed soon. Richard was working on finishing the firmware and updating the documentation. A provisional schedule was presented with a projected date for layout to finish in mid-December. This would lead to probable start of tests in Birmingham in February.

It was pointed out that we should not assemble all four modules immediately. Norman commented that we should make sure all lessons from the ROD module have been learnt as regards test points and clear labelling of chips. Also the front panel layout and meaning should be checked for consistency with the CMM, and the backplane connector issue should be addressed. The CMM design should also be put through the net-list check before production. Investigations of BGA re-working should also continue with the old boards.

CP chip test using GTM - James

James had tried the CP-FPGA code in the GTM with two different configurations. Firstly a simple configuration with a small set of test-vectors was tried, and secondly Ian's more complex setup. James had succeeded in getting all 108 channels to synch after some debugging, and some of the test-vectors worked, but problems were observed in the right hand cells. With Ian's setup, lots of errors were seen, but there was no obvious pattern to the problem. This needs more work to understand the problems, but James will have to suspend CP chip work for a while to work on debugging and extending the ROD code. James showed a comparison of the resource usage in the FPGA devices used on the GTM and planned for the CPM. The numbers are similar, but timing margins should be better for the CPM.

Peter asked how can further progress be made on debugging the GTM. Use of ChipScope should clarify where the problems are located, and even if it takes a large number of iterations to fix complex FPGA code, at least the situation is better than a non-functional ASIC!

CP/JEP ROD prototype - Bruce

Bruce reported on recent work with the DSS/ROD setup. There are three known problems with the DSS, one of which Bruce had recently re-confirmed in the lab. There were nine known problems with the ROD, one of which has now been attributed to software, but most of which still need solving. Bruce described the new test setups made possible by the arrival of the SLIDAD, SLIMOD and ODIN cards. These allow more detailed checking of the S-Link output and protocol as was only previously possible in the RoIB setup at CERN. Using these setups, it was confirmed that James' firmware changes had fixed one of the known problems with the Link Full Flag (LFF). However, although the flow control now works, there was still some suspicion of bit errors occurring. This needs more work as Bruce has only recently had the time to put the system together. Input from James is also required now he has time to work on RODs again.

Finally Bruce summarised the tracking and documentation status. It has been suggested that error tracking for individual modules and module types could be done through route cards and problem reports as used by TD at RAL. An example of these for the ROD tests was shown. Bruce had also made the ROD status reports available on the web with password protection. Murrough thought that it would be good to make passwords the same across the level-1 system (as far as possible).

Timing Control Module - Adam

Adam described the TCM's functions and showed the layout of the front panel indicators. He then detailed the problems that had been found on the board. Most of these problems originated from mis-labelled connections in the schematics which could have been found by the net-list checking program. Some of the bad connections and timing problems could be fixed in firmware, but others required wire modifications, nine in all. The status after all these fixes is that most things have been checked and work. The major component still to be worked on is the CANbus - Adam hasn't yet checked the TTC distribution, but Bob had checked this before he left.

One option to consider in making up more boards is that seven of the wire modifications could be avoided by programming the address decoding differently, leaving just two wire mods. Alternatively the boards could be re-manufactured with the fixes. Adam summarised the options and it was felt that it would be better just to go ahead with the modifications rather than spend more money on new boards. Adam showed the new CPLD code that had already been designed to fix the problems.

Short-term schedule - Tony

Tony showed the current version of the schedule for the UK slice elements, but warned that he felt it was still too optimistic. It is now unlikely that the CPM would be debugged by the end of February, and the backplane was still being delayed as Sam was having discussions with the manufacturers about track widths. It would probably not arrive in the UK until Christmas at the earliest. To allow three months of slice testing in the UK, the modules would not go for tests in Heidelberg until summer 2002, but this will probably tie in well with the current PPM schedule, where the current holding item is the ASIC, which had still not been submitted for production.

Hardware-related items

CANbus status - Eric (for Dave)

Dave had been trying to get the Fujitsu chips to talk together over CANbus by adapting the existing ELMB software. He has succeeded in getting data across, but has been having problems with nested interrupts, which he identified as a problem with the ADC return from an interrupt. He has written new code which which he thinks has solved the problem. He should then be in a position to try talking to the TCM over CANbus - Adam is very to try this soon.

Dave had recently heard about a new device from Analog Devices which should have better software support than the Fujitsu chips, and also are half the size. The new device does not have much flash memory, but might fit into our requirements by having one of these on most modules, and an ELMB on the TCM. The status of ELMB support in ATLAS was queried, and Eric said that although the official line was still there was no support for other devices, he had heard that we are not the only group who feel the ELMB is not well suited to our needs.

More on TTC timing jitter - Alan

Alan had been trying to put some numbers on tolerance to a possible drift in the LHC clock signal arriving at the ATLAS pit due to its propagation on a long fibre from the LHC RF area. The problem cannot be corrected by the TTC system as it is the input to the TTC from LHC that might drift relative to the beam crossing time. The problems this might cause are a reduction in the efficiency of BCID in both saturated and non-saturated cases, and a reduction in Et resolution. Previous studies suggest a +/- 5ns tolerance, with the saturated pulses being a very hard limit. Note these studies were done for EM-like pulses, and the situation could be worse for the faster hadronic signals. Some of this tolerance will already be needed to cope with already foreseen sources of jitter and timing mis-alignment in the detectors and tower summing.

Nick had specifically asked what our tolerances would be for jitter, short-term drift and long-term drift. The worst of these is short-term drift, for which we have no correction mechanism - long-term drift can be corrected by a re-calibration. Assuming all other sources are less than 3ns (TDR figure) then we can accept about 2ns jitter or drift for a period less than a re-calibration. The actual numbers depend on the assumed tolerable error rate, with saturated pulses being both the most error-prone and probably the most important to get right. Alan noted that the situation would be far worse in a high-luminosity LHC scenario with a halved beam-crossing time.

Currently the status is that Nick is collecting requirements from many groups, and there will be discussions about experimental needs and what LHC can do for us. Nick also asked if we could monitor any drift, to which the answer is probably yes, given enough statistics, although the detectors themselves are in a better position to do this. However the general conclusion of the meeting was that many detector groups must be very sensitive to this drift and there ought to be some means of minimizing it, either from a better LHC signal, or use of beam-pickup. For the moment the best way forward is to set a sufficiently tough requirement on the drift.

It was noted that a member of the Heidelberg group is currently monitoring calorimeter signals at the CERN test-beams in order to compile a library of test pulses as a function of detector type and amplitude. These will be useful both for off-line studies and in the slice test.

Software Status

Online software status - Murrough

Murrough started by describing recently reviewed documents on software requirements and organisation. There was some question as to whether these should be reviewed externally. The software tasks were broken down into a list with those responsible named, and there was another list of tasks with no-one yet assigned.

Murrough then briefly described recent work in the software area, along with planned developments. Bruce had done a lot of work on setting up the Concurrent CPUs, and this was well documented and the CPUs now running nicely. Mainz have ordered another CPU identical to the three Bruce had worked on, and these four should be enough for the non-preprocessor part of the slice. Murrough has been working on the run control and database system using the current version of the standard online software. The skeleton is in place and running both at QMW and RAL, but many issues have to be resolved before it can be used to control the hardware. Note that we have a milestone to integrate, at some level, with the online software before the end of 2001.

The remaining tasks were separated into short, medium and long term. In the near future, much HDMC development is needed for stand-alone module tests. In the slightly longer term, for serious module tests, integration of the online, calibration, test-vectors and monitoring is needed. This must be extended in the longer term to cover the whole system in a seamless way.

There have been lots of software meetings and a successful video conference with Oliver which should be repeated. Informal meetings will be held at CERN during the ATLAS week, and a series of pre-joint meeting discussions are planned for November. The manpower situation is looking better - there is a new software person at Mainz (Thomas Trefzger), a new diploma student at Heidelberg, and a new PhD student at Mainz, who could however be concentrating on firmware or other areas. so there are now about 5 FTEs with a good spread throughout the level-1 groups. However things will still be tough, especially with time divided between writing software and testing hardware. Tony asked how the ratio of effort needed to assigned looked (probably similar to before) and whether there was anything we could do to manage the effort required when hardware arrived or just take things as they come.

Hardware simulation and test vectors - Steve

Steve summarized the simulation and test-vector work done since the last meeting. This included the development of a model for the GTM, addition of several sets of CP-FPGA test vectors and some code administration matters. Some time had been taken in actually helping to debug James' FPGA code using the test-vectors. There was still much work to be done, the most urgent being the need for documentation and ideas on how to integrate the framework into the online and test software.

More details about the test vectors themselves was presented. Many of the aspects of the CP algorithm are now fully exercised, but notably the BC-demux treatment is not fully tested. Steve then described his experience with debugging the FPGAs, commenting on how difficult it is to sort out anything but the most gross errors without resorting to looking inside the 'black box' with either the VHDL simulation, or tools such as ChipScope for the actual hardware. Debugging will be much harder than in the past because of the difficulty in accessing intermediate data and the turnaround time of new FPGA code. Simulation at the board level rather than the chip level may help to uncover some problems before they are seen in hardware, and Norman suggested that a block-by-block analysis of the FPGA functionality may help to pinpoint faults.

How to read out the slice - Norman

Norman presented the current plans for how to collect data from the RODs in the slice system. He briefly showed the known inventory of parts then went on to describe recent thoughts on readout of the CTPD and CMM outputs, the problem being that a) the CTPD does not have enough inputs for all the CMM outputs and b) the CTPD does not have a ROD equivalent. In order to read the CTPD inputs, a CPU with an S-Link mezzanine is required to collect the data from the FIFOs, and an even more messy system with DSS modules and special firmware would be required for the CMM outputs. It is proposed to rely on the slice readout for CMMs in general, with separate tests to ensure the real-time outputs are correct.

With this information in mind, an estimate of the S-Link numbers and data rates can be made using certain assumptions. Norman's numbers were eight S-Links with a total data rate of about 400 Mb/s. In order to house eight S-Links, PCI expansion crates would be needed but the data rates claimed for the PCI bus are too small to cope with the full slice. It will be possible to do debugging and some tests at low rates and with just a few modules being read, but to test at high rate, we can only check a subset of events, and we need a mechanism to select those events - note software could only check a fraction of the events in real time anyway. If a random selection of events is made, we only get a relatively low sensitivity to errors. Use of the DSS to select events with errors would increase sensitivity, but it is not easy to feed the DSS flag back into the selection.

There are two possible ways to read events at 100kHz. One is via ROD crate DAQ, ignoring the ROS, the alternative is to use the full ROS/ROBIN architecture to selectively read out events of interest. RHUL is currently developing ROBINs that could be used in this scheme, but they would have to be thoroughly tested before we could be sure of using them. However, this setup should be far more like the final system than any other solution. Norman then detailed how to run a system in this configuration in terms of triggering bursts of events and a mechanism for selecting data to be studied. It is potentially very worrying that we will have to use a large empire of new hardware and software in order to work this solution.

Physics simulation

Off-line trigger simulation status - Ed

Ed said that the code development was progressing with informal code reviews with Murrough. Some re-writing of the old code was required, and the jet trigger should be ready by the next meeting. Other areas that need work are the Et trigger, outputs to CTP and the trigger menu.

Athena still appears to be fairly poorly documented and difficult to install - working outside CERN appears to be very difficult. Ed had been complimented that his web pages were some of the most useful available! The release cycle seems to be poorly defined, with the current version not supporting StoreGate, a mechanism Ed had recently changed the code to use.

Level-1 items from the Lund physics workshop - Steve

Steve gave a quick summary of the Lund workshop, highlighting topics of direct interest to the group. In the general introduction, the current plans for startup of LHC and also the detector staging for 2006 were shown. There was also a talk on prospects for Super-LHC, with the likely scenario being a reduction in beam crossing time. This prompted some discussion of the ability of both trigger and detectors to cope with this change. In the detector performance section, an analysis of the potential for muon tagging at level-1 with the tile calorimeter was presented.

On the physics side, one topic of note was the Higgs production via Weak Boson Fusion which has prompted the recent flurry of activity in the forward jet tagging area. Benjamino Di Girolamo presented his analysis of the additional level-1 rate from forward jet tagging, which was very reasonable at 0.22Hz. The PESA meeting concentrated on new analysis of HLT rate in previously ignored areas, specifically thinking about luminosity and prescaled triggers.

Out of the rut

Highlights of the LEB workshop - Gilles

Gilles presented some of the material from the LEB most relevant to our project. The news from LHC was that they can't say yet that they can't make it on time! There were some questions raised on our talks, one questioning whether we could put so many S-Links on a ROD, and another the use of G-Links. There were a many talks of interest about various aspects of FPGAs, including experience from other experiments.

There was a interesting talk about the challenges and future of microprocessors. The scale is still increasing rapidly, although problems of yield, stability of power supply and sensitivity to particles are becoming more of a problem as devices become smaller. Another problem is that the compilers are not keeping up with the designs. On FPGAs, one point to note is that the lifetime of devices is so short that ours are already considered mature. There were useful talks about BGA reworking, JTAG strategies, and some interesting ideas on building FPGA redundancy into a board, and automated checking of the loaded code. On the final day a discussion was held where many of the topics were relevant to us, including how to keep up with rapidly developing technology and the problems of manpower stability.

Malvern forum on microelectronics - Tony

Tony showed some of the highlights from the Instrumentation Department Forum, at which a few outside visitors like himself were able to see what developments were being worked on at RAL. There was an impressively diverse range of detector/device development going on, covering a wide range of physics research. There were several interesting developments in the area of fundamental research in space with several projects being supported. There was also a nice presentation of particle physics at CMS and many talks on precision detectors of various types.

Coming Reviews

ATLAS System Status Overview introduction - Eric

Eric previewed the ASSO to be held on the Tuesday of the ATLAS week. Other than the main set of presentations, the process also consists of some prior consultation, with the emphasis being on management and integration issues. On the day of the meeting all parts of the system will be presented, but the time assigned to level-1 is short, giving both Paul Hanke and Tony only 10-12 minutes each to summarise the situation for the preprocessor and cluster processors. The review panel have split the system into major components for each reviewer to concentrate on - Bill Cleland is the reviewer for level-1. Much supplementary information has been requested on the PBS and schedules as background. The aim is to be constructive and produce a list of actions where problems are identified.

Bill Cleland suggested an outline for the level-1 talks to concentrate on deliverables, organisation, interfaces, milestones, QA and results. Preliminary talks were presented at a video conference including Bill and Nick, at which Bill already picked up on a few points. The areas of concern include software manpower, TileCal receivers, PRR timings for ASICs and MCMs, documents specifying signal organisation, QA, calorimeter calibration liaison and lack of documents in EDMS.

ATLAS System Status Overview talk - Tony

Tony presented his version of the talk updated according to Bill's wishes from the video conference. This version closely followed Bill's outline, and can be found by clicking on the link above. Comments were invited, and the final version, as delivered at the ASSO on Tuesday 16th October can be found by clicking here.

PPARC review of ATLAS and CMS - Eric

Imminent administrative work is also required for the PPARC mid-project review of ATLAS and CMS. All projects in ATLAS must submit a preliminary report to the UK ATLAS group meeting on 26th October to serve as a basis for the full UK ATLAS status. One of the most important inputs is the manpower estimates, which are tied to grant reviews.

Another important area is to report on the progress on deliverables. This is a little awkward for us since the deliverables as defined to PPARC at the start of the project have been changed considerably in our current architecture and sharing of responsibilities between the UK and other institutions. This will have to be explained in a relatively short status update. The slippage in schedule for LHC, ATLAS and the individual projects means that in general, progress and expenditure is behind the original estimates.

A careful re-assessment and presentation of estimated costs is required since this early in production it is difficult to be sure about module costs, and it is clear we need more engineering effort on FPGA firmware design. Costs will be examined very closely in view of the recent news of a short-fall in funding for LHC. Eric had an almost complete draft which would be circulated for comments before submission to the ATLAS collaboration board.

Coming meetings

ATLAS week - Eric

Many people were going to attend the next T/DAQ plus ATLAS week on 15th to 19th October and several official and unofficial meetings were planned. Other than the ASSO review, there would be the usual meetings plus lots of smaller gatherings to discuss specific issues. These include a talk with Bill Cleland about cabling issues, Yuri Ermoline about a patch panel to the CTPD, software discussions and level-1 dataflow.

RAL joint meeting arrangements - Norman

Norman commented that the registration was going slowly. Accommodation would not be in Coseners House, but instead in the Travel Inn in Abingdon, which is actually cheaper. The agenda will take shape soon, and the Thursday/Friday issue for the meal/management meeting would be decided. Software discussions are to be held before the main meeting.

NIKHEF T/DAQ workshop - Eric

The T/DAQ group would be holding their annual `outside CERN' meeting in NIKHEF, in the week after the RAL joint meeting. As usual the emphasis is on HLT, but a few UK level-1 people would go (at least Murrough and Norman). There is one level-1 specific meeting plus DIG and the institutes board.

Any other Business

Eric was congratulated on reaching a major age landmark and suitable celebrations were held.

Next meetings

    8th - 10th November .............. Joint meeting at RAL

Wednesday 28th November .............. UK meeting at RAL

This page was last updated on 16 December, 2013
Comments and requests for clarification may be sent to