Many of us have been thrown into production issue, equipped (in the best case ;)) with only vague reproduction instructions and get stuck looking into logs and have only one wish: if I could only magically enable more detailed logging and replay the scenario… just without putting restart procedure into motion… One can argue for having DEBUG level enabled all the time but this is impractical for at least two reasons: too much information is as bad as too little and log files will explode with size. Well, our prayers were answered! Since CollabNet TeamForge Git Integration 8.6.1 release one can modify log level of all or particular classes on the fly.
Example: enable DEBUG for everything
There are two handy SSH Gerrit commands that allow viewing (logging ls-level) and setting/resetting (logging set-level) log level on demand. But there is one catch: user needs to have Administrate Server capability. So get it for yourself or make sure that during hot period there is someone who has it. After being warned, let’s go back to our typical scenario: having only a vague idea what is wrong (out of mysterious error stack trace and some reproduction instructions) one can enable DEBUG for all classes to get more context:
ssh -p 29418 admin@hostname gerrit logging set DEBUG
set-level and short
set (call above) versions of command can be used and by default they produce no output (was kind of a strange to me, but it means that everything went fine) and in case of error one gets something like:
Capability administrateServer is required to access this resource
Get yourself Administrate Server capability!
Note that the logging level will stay set until Gerrit is restarted or one calls reset command that will restore log levels of all classes to their initial values (according to the configuration that was provided during Gerrit start):
ssh -p 29418 admin@hostname gerrit logging set reset
If you don’t believe commands that produce no output (I don’t ;)) here is how one can display current level:
ssh -p 29418 admin@hostname gerrit logging ls
Again one can use longer
ls-level and shorter
ls (call above) version and output would be similar to that:
: INFO /api: INFO /gerrit: INFO /git/api: INFO GitApplication: INFO com.collabnet.gerrit: INFO ... com.google: INFO com.google.gerrit: INFO com.google.gerrit.audit.AuditService: INFO com.google.gerrit.common.ChangeHookRunner: INFO com.google.gerrit.common.SiteLibraryLoaderUtil: INFO com.google.gerrit.common.Version: INFO com.google.gerrit.gpg.GpgModule: INFO com.google.gerrit.httpd.DirectChangeByCommit: INFO com.google.gerrit.httpd.RunAsFilter: INFO ...
Example: enable DEBUG for certain package or …
After getting as much as possible with DEBUG level enabled for all packages, one can proceed to narrow it down to particular package:
ssh -p 29418 admin@hostname gerrit logging set DEBUG org.eclipse.jgit
Note that it will affect all classes within this package as well as subpackages – see
logging ls command output for package in question:
... org.eclipse.jgit.internal.storage.file.ObjectDirectory: DEBUG org.eclipse.jgit.internal.storage.file.RefDirectory: DEBUG org.eclipse.jgit.util.FS: DEBUG ...
In fact while
set command is executed check is being performed if given parameter is included in package qualified class name. Knowing that one can set logging to DEBUG for all classes that have name starting with given prefix regardless their packages:
ssh -p 29418 admin@ctf-centos72-dev-box gerrit logging set DEBUG Jar
Here is ls command output for given pattern:
... com.google.gerrit.server.plugins.JarPluginProvider: DEBUG org.eclipse.jetty.util.resource.JarFileResource: DEBUG org.eclipse.jetty.util.resource.JarResource: DEBUG ...
In order to limit it to particular class it is advised to provide class name qualified with full package name e.g.
What comes next?
Check CollabNet blog for more Gerrit productivity hacks!