Category: blog

Scientific project management

Scientific projects are unusually difficult to plan mainly due to their unpredictability in both outcome and effort that needs to be put into it to achieve them. Having to manage a number of them simultaneously with only a limited amount of time available for each one is a challenge to the productivity as a scientist. Here are a couple of practices I implement to ensure a productive use of my time while keeping the necessary flexibility in carrying out my scientific research. I use them as guidelines and best practices, but I am not always living up to these standards myself.

Overall vision

  • 3-5 year vision of what I want to achieve
  • Goals should be as little dependent on external factors and and on luck as possible, i.e. rather than ‘getting this job’, have a vision like ‘becoming an expert/contribute in this particular field’.
  • The purpose is to define how much time to spend on each topic, and not so much on goals with specific measures of success (those are more important for the shorter term)
  • Changes to the list can be made, but only if this wish for a change is persistent.

Project list

  • Concrete spelled-out projects, usually ending with at least one publication, sometimes more
  • Clear measure of success
  • List has a clear hierarchy in priority
  • There are a number of ‘potential projects’ lingering there for months (sometimes years) until they become real projects.

Hierarchy of time plans

  • Starting from a yearly plan (inspired by project list, with relative importance/time informed by vision list)
  • Plans in successively smaller timescales (for me: year, quarter, month, week, day)
  • Each item has to be part of a longer term plan item, but more specific
  • Daily plan filled by items of weekly plan
  • Retrospective & planning session at the end of each plan period
  • Measure fraction of completed tasks to adjust future plans

“Urgent list”

  • Separate list for spontaneous requests
  • Including all email & informal requests
  • Enough dedicated time in daily plan for ‘work on urgent list’ (say 2-3h/day)
  • Only check communication (email, slack, …) during ‘work on urgent list’ to ensure other items can be worked on undisturbed; 3 times a day works quite well for me (as a compromise between availability and undisturbed working).
  • During weekly planning, urgent list items can move to next weekly plan

Physical or digital?

This really depends on personal preferences.

I have my yearly, quarterly and monthly plan, as well as the vision and project list in the same physical notebook. The weekly plan is integrated as a ‘sprint’ to a scrum workflow handled by a project management software. This is highly useful because it allows automated generation of measures of productivity on a weekly basis, but also fairly time-consuming. I use a loose sheet of paper to create a plan for the following day every evening. My ‘urgent list’ is a simple Kanban board.

Python programming for scientists

A minimalist guide to learning Python for scientific use

Why another guide to Python programming? The main reason is that scientific programming is different from commercial software development, in particular for those who need results quickly and might not have the desire or the time to become a Python expert. The following guide is an ordered list of things I consider the most useful things to know for scientific code development in Python. It assumes the ability to write and run a single-file python script (or a Jupyter Notebook), and ends with writing and publishing your own package.

Mastering all of these topics takes years of practice (and I don’t claim I do), so start with the basics, and once you are familiar with one topic, move on to the next.

  • The most important libraries: numpy and matplotlib.pyplot. Get familiar with their functionality and how to use them.
  • Algorithms and data structures:
    • Learn a few of the basic algorithms and check if there is a library implementation.
    • Difference between arrays and dictionaries, and how to use both is very important.
  • Use functions: once a block of code exceeds the space on your screen, refactor parts to a function (recursively, if needed).
    • Make use of the flexibility in return type.
    • Use keyword arguments and default values wherever appropriate.
  • Use the time module to determine slowest part of code.
  • Learn simple optimization techniques:
    • Avoid loops.
    • Save intermediate results of calculation in files (numpy or pickle) or in memory.
    • Use of library functions whenever possible.
  • Create modules
    • In particular, packages (section 6.4 in ‘create modules’ link)
  • Documentation of functions and classes.
  • If you don’t to this already: use version control, e.g. git for your project
  • Get familiar with classes and object-oriented programming in Python. But:
    • Learn about design patterns and design principles. Using classes without having heard about these can be counterproductive.
    • Stick to the PEP8 style guide naming conventions
  • Unit tests via pytest
  • More advanced optimization:
    • Cython/using compiled code
    • Parallelism in python e.g. mpi4py
  • Create your own package
    • setuptools to install packages as a library
    • Continuous integration/ automated test and style check
    • Versioning (maybe a bit over-the-top for most purposes, but one suggestion here)
    • Creating a python package (e.g. for pip)

The question now is how to learn all of this. First and foremost, you learn to code by coding. There is no way around this. However, sometimes it is easy to fall back to somewhat complicated but known ways to implement things. Therefore spending some time trying new things is a good investment in your programming skills. Search engines are essential for concrete problems. For improving code structure, reading more experienced programmers’ code can be helpful. Once you are more advanced, I would therefore recommend looking at implementations of some of the major libraries and trying to understand them.

© 2022 Rainer Weinberger

Theme by Anders NorenUp ↑