How can I explicitly free memory in Python?
Unluckily (depending on your version and unleash of Python) some styles of objects use “free lists” that square measure a neat native improvement however might cause memory fragmentation, specifically by creating a lot of and a lot of memory “earmarked” for under objects of a definite sort and thereby unprocurable to the “general fund”.
The only extremely reliable thanks to make sure that an outsized however temporary use of memory will come back all resources to the system once it’s done, is to possess that use happen during a subprocess, that will the memory-hungry work then terminates. below such conditions, the OS can do its job, and fain recycle all the resources the subprocess might have gobbled up. fortuitously, the data processing module makes this sort of operation (which wont to be rather a pain) not regrettable in fashionable versions of Python.
In your use case, it appears that the most effective approach for the subprocesses to accumulate some results and however guarantee those results square measure out there to the most method is to use semi-temporary files (by semi-temporary I mean, NOT the type of files that mechanically depart once closed, simply normal files that you just expressly delete once you are all finished them).
The del statement may well be of use, however IIRC it is not bound to free the memory. The docs square measure here … and a why it is not discharged is here.
I have detected folks on UNIX operating system and Unix-type systems forking a python method to try to to some work, obtaining results then killing it.
This article has notes on the Python employee, however i believe lack of memory management is that the draw back to managed memory.