The Python conguration language

Rick Wilkinson, Benedikt Hegner

Outline

Introduction to the Python language Work our way through a typical job cong Input, modules and services, paths, output Available tools Not going back to refer back to old congs (some backup slides for that though)

Reminder - the Framework

all this gets steered by conguration les...

The Python language I

Widely used language Object-oriented Nice syntax Lots of extensions Interfaces with C++ (boost, SWIG) Will be the only cong supported in upcoming CMSSW_2_1_0 Adds a lot of exibility for the user. So for you as well :-)

The Python language II

No C++ style memory management needed (has garbage collection) Things are usually passed by reference - Except native numbers and strings - Be careful about this Control ow based on indentation Comments start with a sharp (#) Will be the only cong supported in upcoming CMSSW_2_1_0
but lets forget about programming Python for now and see how the new congurations look like...

CMS Python Cong Types I

Most of the objects youll create will be of a CMS-specic type. To make them known to the interpreter you do:

import FWCore.ParameterSet.Config as cms

Objects are then created with a syntax like:

jets = cms.EDProducer(JetReco, coneSize = cms.double(0.4), debug = cms.untracked.bool(True) )

Warning: The comma between the parameters is very important! Forgetting the comma after line n results in a syntax error reported for line n+1. So not straight forward to track down!

CMS Python Cong Types II

!"#.$%&'().*%+,!"# $%&'() *%+,!"#" #$%&' /)&01 8/)&01 /)&@A 8/)&@A 7(8C=, C((= -&3/)D $#,& F)+8&*9D 52,)&F6 B/=,F)$9&' ()*%)'+#&' 2/)&01 28/)&01 2/)&@A 28/)&@A 27(8C=, 2-&3/)D E$#,& EF)+8&*9D E52,)&F6 &,&(-#+). $3(4,-#,;8,)4, $9&' 5)7$9&' #4',78=, #,;8,)4,$=94, '(=7,3 ' =7 *)!-/&' 56$3(784,3 56<)9=%>,3 56B/=&,3 5#$3(784,3 5##(834, #(834, #,4#(834, G((+,3 #,32/4, H8&+8&"(78=,
@

*)!+0+&1' 8)&394:,7 5#$3,?,3

Conguration les I
Denition of terms: python module

A python le that is meant to be included by other les Placed in Subsystem/Package/python/ or a subdirectory of it CMS le name convention: Denition of a single framework object: _cfi.py A fragment of conguration commands: _cff.py A full process denition: _cfg.py To make your module visible to other python modules: Be sure your SCRAM environment is set up Go to your package and do scram b or scram b python Needed only once Correctness of python cong les is checked on a basic level every time scram gets used. Currently only counting as warnings try Configuration/StandardSequences

Conguration les II
Denition of a cfg le

Controls the nal job to be run Contains a cms.Process object named process File name convention: _cfg.py Usually placed in a packages python/or test/ Can be checked for completeness (from the cong point of view) doing

python myExample_cfg.py

Can by run using cmsRun

cmsRun myExample_cfg.py

The Process object

A usual process denition

Process

process = cms.Process(RECO)
An input source

process.source = cms.Source(PoolSource, ...)

Some modules

process.photons = cms.EDProducer(JetProducer)
Some services

process.tracer = cms.Service(Tracer)
Some execution paths

process.p1 = cms.Path(process.a * process.b)

Maybe output modules

process.out = cms.OutputModule(...
Even full processes can be imported from elsewhere
10

How to Import Objects I

To fetch all modules from some other module into local namespace

from Subsystem.Package.Foo_cff import *

(looks into Subsystem/Package/python/Foo_c.py) To fetch only single objects you can do

from Subsystem.Package.Foo_cff import a, b

To use objects without putting them in the local namespace

import Subsystem.Package.Foo_cff as foo With foo.a and foo.b you can now access the objects
Dont forget that all imports create references, not copies: changing an object at one place changes the object at other places

11

Cloning

Sometimes you need to add a module which has almost the same parameter as another one You can copy the module and change the parameters that need to be modied

import ElectroWeakAnalysis.ZReco.zToMuMu_cfi as zmumu zToMuMuGolden = zmumu.zToMuMu.clone( massMin = cms.double(40) )

Changing while cloning should be preferred wrt clone + later replace as it is a much safer practice.

12

How to Import Objects II

To load everything from a python module into your process object you can say:

process.load(Subsystem.Package.Foo_cff)

Technical detail. This is identical to

import Subsystem.Package.Foo_cff process.extend(Subsystem.Package.Foo_cff)

13

Framework Modules

EDProducer - add objects into the event EDAnalyzer - analyze objects from the event EDFilter - can stop an execution path and put objects into the event EDLooper - For multi-pass loopng over events OutputModule - Write events to a le. Can use lter decisions

14

Framework Services

Examples of Framework Services are - geometry, calibration, MessageLogger Types are: - ESSource Provides data which have an IOV (Interval of Validity) - ESProducer Creates Products when IOV changes - Service Not IOV dependend. E.g. MessageLogger In case of conicts preference can be changed with ESPrefer:

es_prefer_trackerAlignment = cms.ESPrefer("PoolDBESSource","trackerAlignment")

15

Copying all the parameters from a PSet

Give the PSet name directly after module type Has to happen before the named parameters

KtJetParameters = cms.PSet( strategy = cms.string(Best) ) ktCaloJets = cms.EDProducer(KtCaloJetProducer, KtJetParameters, coneSize = cms.double(0.7)

As parametes get copied a later change of KtJetParameters will not get picked up

16

Sequences, Paths and Schedules

Sequence Denes an execution order and acts as building block for more complex congurations and contains modules or other sequences

trDigi = cms.Sequence(siPixelDigis + siStripDigis)

Path

Denes which modules and sequences to run.

p1 = cms.Path(pdigi * reconstruction)

EndPath A list of analyzers or output modules to be run after all paths have been run
outpath = cms.EndPath(myOutput)

Schedule Denes the execution order of paths. If not given rst all paths, then all endpaths.
process.schedule = cms.Schedule(process.p1, process.outpath)

17

Sequence operators
+ as follows Use if the input of the previous module/sequence is not required

trDigi = cms.Sequence(siPixelDigis + siStripDigis)

* as depends on

If module depends on previously created products

p1 = cms.Path(pdigi * reconstruction)

Enfored and checked by scheduler Precedence over +

outpath = cms.EndPath(myOutput)

Combining By using () a grouping is possible

(ecalRecHits + hcalRecHits) * caloTowers

18

Filters in Paths

When an EDFilter is in a path, returning False will cause the path to terminate Two operators ~ and - can modify this. ~ means not. The lter will only continue if the lter returns False. Text - means to ignore the result of the lter and proceed regardless jet500_1000 = cms.Path( ~jet1000filter + jet500filter + jetAnalysis )

19

Tracked / Untracked
Tracked parameters Parameters that change the content of the event Will be saved in the event data provenance

Cannot be optional - if asked for and not found, exception will be thrown - constructs to circumvent this policy are highly discouraged

a = cms.double(3.0)
Untracked parameters Parameters that dont affect the results, e.g. debug level Can have default values - So theyre used for optional parameters, but not really considered safe practice.

a = cms.untracked.double(3.0)
20

Modifying parameters

You are free to reach inside any parameter and change it

from Subsystem.Package.foo_cff import * foo.threshold = 4.0

Or via the process object

process.foo.threshold = 4.0
If the right hand side of the expression is a CMS type, a new parameter can be created

# this line will fail because g4.SimHits.useMagmeticField = # this line will create a new # with wrong name g4.SimHits.useMagmeticField =

of typos False parameter cms.bool(False)

21

Translating Old Cong Files

Standard congs in the release have been translated centrally In FWCore/ParameterSet/python there are

translate.py
- creates a python le based on your cong le - creates any included les which may be needed

comments.py

Another tool is edmTestCfgTranslation - translates your cong - tests whether the old and the new ParameterSets as read in by the SW framework match

22

Tools I
edmPythonSearch

A grep-like syntax to search for identiers within imported les

> edmPythonSearch minPt Reconstruction_cff

... RecoMuon.MuonIdentification.muons_cfi (line: 19) : ... minPt = cms.double(1.5),

There are a few feature requests pending...

23

Tools II
edmPythonTree

Gives an indended dump of which les are included by which les (initial version for the old congs by Karoly Banicz and Sue Anne Koay)

> edmPythonTree Reconstruction_cff

+ ... Simulation_cff + Configuration.StandardSequences.Digi_cff + SimCalorimetry.Configuration.SimCalorimetry_cff

Here a few feature requests pending as well...

24

Tools III
python

The Python interpreter helps you inspecting your congs > python -i Reconstruction_cff

Ctrl-D shuts it down

Try it out!

25

Support and Documentation

Look at what the automatic translation spits out There is a twiki page in the SWGuide:
https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideAboutPythonConfigFile

In case of questions and comments send an e-mail to hn-cms-edmFramework Experts are distributed over different time zones Have fun!

26

Backup

27

Plugin types

old cong
source

python cong
Source EDAnalyzer

module

EDFilter EDProducer OutputModule

service es_source ...

Service ESSource ...

Reminder: Since 1_5_0 there are stricter rules what kind of modules are allowed in Paths and EndPaths!
28

Sequences, paths ...

old cfg

path p1 = {foo, bar & foobar }

precedence of * over + maybe more intuitive! python

p1 = Path( foo * bar + foobar )

new feature: can be modied

29

Including other les

old cong Python cong

include "RecoJets/JetProducers/data/CaloTowerSchemeB.cfi"

from RecoJets.JetProducers.CaloTowerSchemeB_cfi import *

no data

30

Using blocks

old cong
block Record = { untracked string composer = Beethoven }

Python cong
Record = cms.PSet( composer = cms.untracked.string( Beethoven ) )

include SomePlace/Record.cff module MyModule = SomeFilter { using Record }

from SomePlace.Record_cff import Record MyModule = EDFilter(SomeFilter, Record )

31

A basic example
old cong
process = foo { [...] module udsfilter = JetFilter{ ...} module jets = JetReco{ ...} process.jets = cms.EDProducer("JetReco",...) [...] replace udsfilter.jetType = 2 [...] path mypath = (jets, udsfilter) [...] process.udsfilter.jetType = 2 [...] process.mypath = cms.Path(process.jets*udsfilter)

python cong
process = cms.Process("foo") [...] udsfilter = cms.EDFilter("JetFilter",...) process.udsfilter = udsfilter

$ cmsRun config.cfg

$ cmsRun config_cfg.py

32

Replace statements

old cong
module udsfilter = JetFilter{ ...} replace udsfilter.jetType = 2

python cong
udsfilter = cms.EDFilter("JetFilter",...) udsfilter.jetType = 2

replace musician += Madonna replace painters += { Picasso, da Vinci}

musician.append(Madonna) painters.extend( (Picasso, da Vinci) ) musician.pop() # dont like Madonna

33

Compatibility between the different conguration options

module foo = SomeFilter { double MeanX = double MeanY = double MeanZ = untracked bool

0. 0. 0. log = true}

each format can be converted into the other one

process = Process("test") process.include("foo.cfg") [...] process.dumpConfig()

process test = { module foo = SomeFilter { ... } ...

34

Module conguration
old cong
type name = value

Python
name = type( value )

module foo = FooFilter {}

foo = EDFilter(FooFilter)

process.foo = EDFilter("SomeFilter", MeanX = double(0), MeanY = double(0), MeanZ = double(0), log = untracked.bool(True))

module foo = SomeFilter { double MeanX = double MeanY = double MeanZ = untracked bool

0 0 0 log = true}

35