File Creation Shenanigans
Lately I’ve been working on building a concurrent CLI download manager with Go. Initially I wrote the following function to download a file:
// Excerpt from Download: HTTP and progress-reporting code omitted.
if _, err := os.Stat(outputPath); err == nil {
return fmt.Errorf("output file already exists: %s", outputPath)
} else if !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("check output file: %w", err)
}
// ... fetch the HTTP response ...
partPath := outputPath + ".part"
partFile, err := os.Create(partPath)
if err != nil {
return fmt.Errorf("create file: %w", err)
}
// ... stream into partFile, close it, and check both operations for errors ...
if err := os.Rename(partPath, outputPath); err != nil {
_ = os.Remove(partPath)
return fmt.Errorf("rename file: %w", err)
}
Initially, I check whether the destination file exists. If it does, I return an error. However, this check does not reserve the file name. Another process could create the destination while my download is still running. After receiving the HTTP response headers, I construct a temporary path by appending .part to the destination filename. I then use os.Create to open a file for writing, creating it if it does not exist or truncating it if it already exists. The response body is then streamed into the temporary file. After streaming completes successfully, I close the temporary file and check for errors before renaming it to the destination filename.
This was not a good solution to downloading a file. Since this function is intended to run concurrently, downloads targeting the same path can interfere with each other’s temporary files, and another process can create the destination before I rename the temporary file. Even downloading one file at a time would not protect against a separate program writing to the same paths.
On a Linux machine, two entries in the same directory can’t have the same name. If my downloader calls os.Create on a .part path that already exists, it truncates that file to zero bytes, discarding any data that has been written to it.
On Linux, os.Rename can also replace an existing destination, even if another process created it after my initial check.
Below, I propose a better approach:
// Excerpt from Download: HTTP and progress-reporting code omitted.
if _, err := os.Stat(outputPath); err == nil {
return fmt.Errorf("output file already exists: %s", outputPath)
} else if !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("check output file: %w", err)
}
// ... fetch the HTTP response ...
partFile, err := os.CreateTemp(filepath.Dir(outputPath), ".godl-*.part")
if err != nil {
return fmt.Errorf("create file: %w", err)
}
partPath := partFile.Name()
// ... stream into partFile, close it, and check both operations for errors ...
if err := os.Link(partPath, outputPath); err != nil {
_ = os.Remove(partPath)
return fmt.Errorf("publish file: %w", err)
}
if err := os.Remove(partPath); err != nil {
return fmt.Errorf("file saved, but temporary file cleanup failed: %w", err)
}
I began same as before, I checked if the file exists, handled HTTP requests but rather than creating a temporary file by appending .part I used Go’s os.CreateTemp. os.CreateTemp creates a temporary file by accepting a directory string and a pattern string. If the pattern contains an *, the last * is replaced with a random string. Otherwise, the random string is appended to the pattern. I then store the temporary file’s path in partPath. After streaming the response into the temporary file, closing it, and checking both operations for errors, I create a new hard link with os.Link and remove the temporary filename.
Why is this approach better? os.CreateTemp creates a unique temporary file. With multiple concurrent downloads, each download writes to its own temporary file, preventing one download from overwriting another’s partial data. os.Link creates a hard link at outputPath, giving the same file another name. If outputPath exists, it returns an error. Changes made through either name affect the same file. This prevents the downloader from overwriting an existing destination, even if another process creates it during the download. I then remove the temporary filename, leaving the downloaded data accessible through outputPath.
The lesson here is that when concurrent operations create files, the file-handling logic should be carefully inspected. Tests should cover not only successful downloads, but also what happens when multiple writers compete for the same paths. One limitation of this technique is that it requires a filesystem that supports hard links.
Some nice resources worth checking out:
I think moving forward I will try my best to update this blog twice weekly as it helps reinforce concepts I learn over time. Feel free to reach out to me via my contacts for any questions or if you’d like to chat!