How can I safely create a nested directory in Python ?

How can I safely create a nested directory in Python?

Asked on December 18, 2018 in Python.
Add Comment


  • 3 Answer(s)

    Here the solution is,

    There are two answers with best qualities.

    Try os.path.exists and consider os.makedirs for the creation.

    import os
    if not os.path.exists(directory):
      os.makedirs(directory)
    

    As noted in comments and elsewhere, there’s a race condition – if the directory is created between the os.path.exists and the os.makedirs calls, the os.makedirs will fail with an OSError. Unfortunately, blanket-catching OSError and continuing is not foolproof, as it will ignore a failure to create the directory due to other factors, such as insufficient permissions, full disk, etc.

    One option would be to trap the OSError and examine the embedded error code

    import os, errno
     
    try:
      os.makedirs(directory)
    except OSError as e:
      if e.errno != errno.EEXIST:
        raise
    

    Alternatively, there could be a second os.path.exists, but suppose another created the directory after the first check, then removed it before the second one

    Answered on December 18, 2018.
    Add Comment

    Here the solution is,

    Python 3.5+

    import pathlib
    pathlib.Path('/my/directory').mkdir(parents=True, exist_ok=True)
    

    pathlib.Path.mkdir as used above recursively creates the directory and does not raise an exception if the directory already exists. If you don’t need or want the parents to be created, skip the parents argument.

    Python 3.2+:

    Using pathlib :

    Install the current pathlib backport named pathlib2. Refer the Python 3.5+ section and use it as the same

    If using Python 3.4, even though it comes with pathlib, it is missing the useful exist_ok option. The backport is intended to offer a newer and superior implementation of mkdir which includes this missing option.

    Using os:

    import os
    os.makedirs(path, exist_ok=True)
    

    os.makedirs as used above recursively creates the directory and does not raise an exception if the directory already exists. It has the optional exist_ok argument only if using Python 3.2+, with a default value of False. This argument does not exist in Python 2.x up to 2.7. As such, there is no need for manual exception handling as with Python 2.7

    Python 3.5+:
    import pathlib
    pathlib.Path(‘/my/directory’).mkdir(parents=True, exist_ok=True)
    pathlib.Path.mkdir as used above recursively creates the directory and does not raise an exception if the directory already exists. If you don’t need or want the parents to be created, skip the parents argument.

    Python 3.2+:
    Using pathlib:

    If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.

    If using Python 3.4, even though it comes with pathlib, it is missing the useful exist_ok option. The backport is intended to offer a newer and superior implementation of mkdir which includes this missing option.

    Using os:

    import os
    os.makedirs(path, exist_ok=True)
    os.makedirs as used above recursively creates the directory and does not raise an exception if the directory already exists. It has the optional exist_ok argument only if using Python 3.2+, with a default value of False. This argument does not exist in Python 2.x up to 2.7. As such, there is no need for manual exception handling as with Python 2.7.

    Python 2.7+:
    Using pathlib:

    If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.

    Using os:

    import os
    try:
    os.makedirs(path)
    except OSError:
    if not os.path.isdir(path):
    raise
    

    While a naive solution may first use os.path.isdir followed by os.makedirs, the solution above reverses the order of the two operations. In doing so, it prevents a common race condition having to do with a duplicated attempt at creating the directory, and also disambiguates files from directories.

    Note that capturing the exception and using errno is of limited usefulness because OSError: [Errno 17] File exists, i.e. errno.EEXIST, is raised for both files and directories. It is more reliable simply to check if the directory exists.

    Alternative:

    mkpath creates the nested directory, and does nothing if the directory already exists. This works in both Python 2 and 3.

    import distutils.dir_util
    distutils.dir_util.mkpath(path)
    

    Per Bug 10948, a severe limitation of this alternative is that it works only once per python process for a given path. In other words, if you use it to create a directory, then delete the directory from inside or outside Python, then use mkpath again to recreate the same directory, mkpath will simply silently use its invalid cached info of having previously created the directory, and will not actually make the directory again. In contrast, os.makedirs doesn’t rely on any such cache. This limitation may be okay for some applications.

    Answered on December 18, 2018.
    Add Comment

    Let us look at the solution:

    By executing try except and the right error code from errno module gets rid of the race condition and is cross-platform:

    import os
    import errno
     
    def make_sure_path_exists(path):
      try:
        os.makedirs(path)
      except OSError as exception:
        if exception.errno != errno.EEXIST:
          raise
    

    Also, we try to create the directories, but if they already exist we ignore the error. On the other hand, any other error gets reported. For example, if you create dir ‘a’ beforehand and remove all permissions from it, you will get an OSError raised with errno.EACCES (Permission denied, error 13).

    Answered on December 18, 2018.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.