Consider using docker build --no-cache .

[jamesaoverton]

We're still trying to figure out the best way to use Docker with DROID. For many projects we use the ODK image and then want to install a few more Ubuntu and Python packages. Here's a current example Dockerfile:

FROM obolibrary/odkfull

COPY requirements.txt /tools/ihcc-requirements.txt
RUN pip install -r /tools/ihcc-requirements.txt

RUN apt-get install aha dos2unix sqlite3

When the requirements.txt file is changed, using the Rebuild container button doesn't always pick up those changes. I've tried a bunch of things and the behaviour I'm seeing is inconsistent. Sometimes Docker rebuilds everything and sometimes it uses a cache, like this:

Sending build context to Docker daemon  175.7MB

Step 1/4 : FROM obolibrary/odkfull
 ---> 742aa3c8a1a2
Step 2/4 : COPY requirements.txt /tools/ihcc-requirements.txt
 ---> Using cache
 ---> 10327ccbb6d3
Step 3/4 : RUN pip install -r /tools/ihcc-requirements.txt
 ---> Using cache
 ---> c155fcf2c209
Step 4/4 : RUN apt-get install aha dos2unix sqlite3
 ---> Using cache
 ---> 5bc57b61a1f0
Successfully built 5bc57b61a1f0
Successfully tagged ihcc-master:latest
e8b4037000439cf79073fc38c24c92cf2036d802647094ce6494905741b481b7
IHCC-master

It think it would be better to specify --no-cache option here: https://github.com/ontodev/droid/blob/master/src/droid/branches.clj#L203

[matentzn]

Its odd - when requirements.txt changes it should always rebuild everything after the COPY command! I did however observe the behaviour that occasionally the whole image is rebuild - note that ODK base image has been changing frequently these last month, and if you do FROM obolibrary/odkfull without a tag (a version tag), you will rebuild whenever the base image changes - which may be at the most surprising times!

[lmcmicu]

Adding the --no-cache option seems like a relatively straightforward change to make, but I'll have a closer look a bit later this morning at the DROID code to confirm.

[lmcmicu]

I was worried that using the --no-cache option would force a complete pull of okdlibrary/odkfull whenever a new branch is created. But it doesn't. This leaves me somewhat confused about what the --no-cache option is really for (and I have found essentially no documentation on it).

I agree with Nico that it is odd that docker is not detecting that requirements.txt has changed.

Question: What is the exact use case here? Are we checking out a new branch or doing something else? If it is the former, note that requirements.txt needs to be updated on the remote branch and not just in the local directory. DROID creates a new local branch (in a new local directory) by first pulling from the remote. If changes are made to requirements.txt but are not pushed, then DROID will not pick those up when creating a new image for a new branch.

[lmcmicu]

Hmm... nevermind the first part of my previous comment.

[lmcmicu]

It didn't register to me at first that you mentioned the Rebuild button. In that case only the local file matters and whether the changes have been pushed or not is irrelevant.

In any case it is easy to add the --no-cache option to the docker command line. I can't foresee any detrimental impacts of adding it, and it seems to better reflect what we want to do with the Rebuild button anyway, and I've verified that it works correctly. If you would like to add this change to the code let me know.

[jamesaoverton]

I didn't realize that Docker would check for changes to COPY files. That's good, but I don't think it always works.

Let's not make any changes to DROID for now.