Common Tool Technical Group Meeting August 12, 2010 Attending

advertisement
Common Tool Technical Group Meeting
August 12, 2010
Attending:
PDBe: Tom, Jawahar
RCBB: Jasmine, Zukang, Raul, Vladimir, Dimitris, Terry, Martha, and John
BMRB: Steve
PDBj: Takanori
Note change: The next Technical Team meeting will be Friday, August 20, 2010 at
9:00AM EDT, 2:00PM UK, 10:00PM Japan and 8:00AM Wisconsin. Agenda and list of
required participants to follow.
Meeting Summary and ACTION ITEMS:
WFE/WFM overview:
Level 1 summarizes the status of all depositions in play for a specific annotator.
Level 2 enables the full workflow overview and management for a single
deposition, as well as providing the inter-module GO BACK functionality.
Level 3 provides the log file drill down for a specific workflow module (ie
sequence alignment or ligand processing)
1. WFM Level 2 design allows for the depiction and navigation of multiple
workflow classes (ligand, sequence, …) for a single deposition. Level 2 will
include the “Go Back” functionality. (See: Tom and Luana’s screen shots of
scenarios for a working sequence example, included with the agenda).
a. The workflow graphical displays will be done using the JavaScript
Raphael library. Tom will provide a mockup of this graphical display by
next week. The back end of the graphical displays will take a couple of
weeks and will not be part of the August delivery. This work will be
assigned to the new PDBe systems engineer.
b. How will GO BACK be depicted (Jasmine)?
ACTION ITEM: Tom to mock up the interface functionality to
include/demonstrate GO BACK.
c. Tasks are most typically done serially (80:20 rule) with multiple steps
conducted in the background before annotator intervention.
ACTION ITEM: Tom will mock up a depiction of the business logic and
assumed order of tasks for the 80% scenarios.
d. In the special case events that the annotators need to change the
automatic serial order – how will this be managed at level 2? Interactions
between workflows need to be addressed. An example of this would be a
new modified amino acid or nucleotide which would have to be handled
in ligand processing before sequence processing could be completed.
ACTION ITEM: Tom will mock up a draft of this interface functionality.
e. Can tasks be started from the Level 2 display? Yes, the WFM can run
both in batch mode and in incremental mode.
f. How will the workflow detect if it should be run (Does entry contains a
ligand)? The business logic will define the automated sequence and the
exceptions that require annotator interventions. One model is that each
major set of tasks will run as far as possible automatically and then
indicate through the WFM that annotator input is needed.
g. How will major tasks be scheduled to minimize annotator waiting time?
Long tasks like reference sequence database searches should be done
automatically in the background. Many of these tasks will in fact be run
during the deposition process prior to submission. Level one will only
provide status information for each deposition – wait, running, or
finished.
h. Graphical depictions will also be provided with the Level 3 display
drilling down into specific functional components (Sequence or Ligand
processing).
i. A global report summarizing all issues for an entry is needed to provide
the annotators with the context necessary for processing an entry. How
will this be addressed in the WFM?
ACTION ITEM: Tom to provide further mockups for group review.
2. Log file management. Tom has captured logging output in named files
(deposition id, workflow class, workflow instance) that can be viewed by the
WFM.
ACTION ITEM: This should be available for view/testing by August 17.
3. Sequence module testing. Jasmine reported results of sequence module testing
by the RCSB annotation group. This level of activity initiated by the RCSB staff
“load test” was more than the “test” instance system could contain (17
concurrent processes). The memory load distribution and management
implication of this unintended load test need to be better understood.
This result raises the following issues/questions:
a. First off the users had no way of knowing that the process that they were
waiting for completion of had been “killed”. How will the interface alert
users of such occurrences in the future?
ACTION ITEM: Tom to suggest solution.
b. If the workflow server died because the Linux operating system detected
a VM over load - what are the full implications to our design? Questions
about exactly how the Linux system deals with VM overload have been
provided by Steve Mading – in his email description, on Aug 12, to the full
team, - of what appears to be a dysfunctional family reality show. More
discussion to follow.
c. What exactly happened during this load test? Since log files are not
available we will need to rerun this exercise once the capture of log files
is in place - ETA: August 17.
ACTION ITEM: Jasmine to coordinate with Tom and John to rerun the
“load test” scenario.
d. What are the minimum memory requirements for the component tasks
over a range of depositions sizes?
e.
f.
g.
h.
ACTION ITEM: John will provide some estimates of memory
requirements for the executables used in the sequence module.
How many active processes should we expect under normal
circumstances now and in the anticipated near future growth of
depositions and processing capacity? (as a reference point – RCSB
currently employs 6 production servers with load balancing functionality
and monitoring. )
What queue management strategy should we consider for this
application?
The resource requirements (at least some estimate of this) for each task
needs to be known to the workflow engine in order to make this
regulation possible. See 3.a and 3.b above.
What are the optimal hardware capacity requirements for the anticipated
production load of the new D&A system? Based on the data we gather for
the completed component modules we will be able to better estimate this.
What are the unanticipated implications of designing a distributed load
management system.
ACTION ITEM: A plan will be developed for a load and parallel processing
test in September. Tom & John to present plan by Sept 2.
Termination status of the sequence processing module - There is a distinction
between saving results, pause may re-enter, and sequence processing
completed. Jasmine to define the terminology for the buttons defining the
possible termination states. Raul to investigate capturing the window close
event to update the status.
Some “Run Engine” actions seem to take a very long time to complete. Some
explanation of the problem may be found in the level four logs.
4. Ligand module progress. Zukang will setup the ligand search code on the
shared server by the end of day Tuesday, August 17.
ACTION ITEM: Jawahar and others will begin to do testing on Wednesday,
August 18, so as to be able to report preliminary results at the next meeting.
Jawahar will contact Zukang directly if there are questions about how to test.
5. Next Week Schedule - Dimitris will visit Rutgers on Thursday and Friday next
week to work with the annotation group on inhibitor ligand processing. Kim
will also be visiting Rutgers on Tuesday next week. Kim will be available if
there are questions about inhibitor ligand processing or DOHLC.
Download
Related flashcards

Computer science

25 cards

ARM architecture

23 cards

MSN

28 cards

Web development

38 cards

Create Flashcards